Big Data: 98 per cent as good as gut feel

The Human Face of Big Data.Via Brian's Books

Not so long ago I was given a demonstration of a very clever bit of kit to analyse big data. By gathering evidence about your expressed preferences shared through social media it could make predictions about the preferences you don't articulate.

Scary clever.

The person presenting proudly told us how their analysis had identified the right kind of music to attract the target market to a new energy drink.
Let's say they picked Artist X as the ideal brand amabassador.

Funnily enough, Artist X was exactly the person the product's brand manager had come up with from their own 'analysis'. I'm sure they had some evidence. Some charts with brand positions. Some brand truths. All that malarky. But they had much more. They had stuff no one had written down, no one could easily define. They had what they felt.

And where was all this processed? In the 'gut' of the brand manager - someone living and breathing and believing the brand.

The vendors of the clever big data cruncher marvelled at how Artist X could be proven to be a 98% fit with the requirements of the brand.

I fear they may be wrong.

I suspect the brand manager was 100% right and that the dimensions of data captured by the tool has gaps - gaps that only guts can currently fill.

You process big data every day - with staggering accuracy. When you drive a car or walk along a crowded street you are processing huge amounts of information and making decisions on it. In real time. If only big data could do that? It's getting better. Our weather forecasts are one example. But it's not 100%. There's still room for your gut - and it's important room.

One final example. When I started out as a local newspaper man, my first sub-editing job was to go through what were called the village columns. We'd call it user generated content today. Reports from local correspondents of village fetes, whist-drives, bowls matches, bring and buys, church services - that kind of micro-local news. It was mostly hand-written (semi-scrawled) and often on unlined paper. And it was my task to correct, amend, headline, and guestimate. I'd mark it all up with instructions for the guys who would set the pages (using ems and picas, font names and point sizes).

And then I'd have to estimate how much I needed to cut or add, what depth I should set images to (for example) to fill the number of pages I had been allocated, accounting for the adverts booked on them. It was a task that was not unlike estimating the number of individual straws in a decent sized haystack.
Yet, within a few weeks, I could mark up all that separate, different-looking, miscellaneous copy and have the output fit within 5 or 6cms of the final column each week. All done without the aid of any big data analysis.

Big data has a hugely important role for organisations (and for governments) moving forward. Who controls it and to what ends it is used will occupy us for years to come. But throughout that process let's not forget the extra value that something human adds.