Big Data: 98 per cent as good as gut feel

The Human Face of Big Data.Via Brian's Books

Not so long ago I was given a demonstration of a very clever bit of kit to analyse big data. By gathering evidence about your expressed preferences shared through social media it could make predictions about the preferences you don't articulate.

Scary clever.

The person presenting proudly told us how their analysis had identified the right kind of music to attract the target market to a new energy drink.
Let's say they picked Artist X as the ideal brand amabassador.

Funnily enough, Artist X was exactly the person the product's brand manager had come up with from their own 'analysis'. I'm sure they had some evidence. Some charts with brand positions. Some brand truths. All that malarky. But they had much more. They had stuff no one had written down, no one could easily define. They had what they felt.

And where was all this processed? In the 'gut' of the brand manager - someone living and breathing and believing the brand.

The vendors of the clever big data cruncher marvelled at how Artist X could be proven to be a 98% fit with the requirements of the brand.

I fear they may be wrong.

I suspect the brand manager was 100% right and that the dimensions of data captured by the tool has gaps - gaps that only guts can currently fill.

You process big data every day - with staggering accuracy. When you drive a car or walk along a crowded street you are processing huge amounts of information and making decisions on it. In real time. If only big data could do that? It's getting better. Our weather forecasts are one example. But it's not 100%. There's still room for your gut - and it's important room.

One final example. When I started out as a local newspaper man, my first sub-editing job was to go through what were called the village columns. We'd call it user generated content today. Reports from local correspondents of village fetes, whist-drives, bowls matches, bring and buys, church services - that kind of micro-local news. It was mostly hand-written (semi-scrawled) and often on unlined paper. And it was my task to correct, amend, headline, and guestimate. I'd mark it all up with instructions for the guys who would set the pages (using ems and picas, font names and point sizes).

And then I'd have to estimate how much I needed to cut or add, what depth I should set images to (for example) to fill the number of pages I had been allocated, accounting for the adverts booked on them. It was a task that was not unlike estimating the number of individual straws in a decent sized haystack.
Yet, within a few weeks, I could mark up all that separate, different-looking, miscellaneous copy and have the output fit within 5 or 6cms of the final column each week. All done without the aid of any big data analysis.

Big data has a hugely important role for organisations (and for governments) moving forward. Who controls it and to what ends it is used will occupy us for years to come. But throughout that process let's not forget the extra value that something human adds.

Don't let Prism wreck the network - it belongs to you

The row over the ultimate data hoover that is Prism (catch up here with a quick guide of what it is and how legal or otherwise it may be) is, as usual, being characterised as a two-sided affair with battle lines drawn.
On one side we have our Governments - those who would protect us from whatever it is we are expected to live in terror of right now.
On the other are those who feel invaded and violated - that their data is the representation of their sovereign self - the protectors of our privacy.
Now, I'm not too concerned about the privacy thing.
I wrote in 2010:

"Privacy seeks to obscure truth. Often to an individual's personal benefit. Perhaps we are just going to have to get used to living in a state of truth - with the wider - by necessity less personal - benefits that may offer." (... the impact of an information revolution on privacy)

I've long argued that privacy is a relatively new and less-than-normal state for humankind. It's something that emerged when we started sleeping in separate rooms away from the cattle - around about the invention of the fireplace and chimney (read Bill Bryson At Home - A short History of Private Life - for more on that. It's a concept that is only some 500-600 years old. Culturally significant but in no way some kind of human need or even desire.
And I'm not big on kow-towing to terror either. Neither argument does it for me.
What I'm really concerned about in the Prism affair is that the fear of sharing it could generate could deliver a blow to the self-organising, open future we are on course for.
To self organise (the single biggest disruption to the centrally-organised status quo) we must express our meta data. That is, we must share what we think, as widely with each other, as often as possible. This is ultimately how we find each other at time of greatest need - how we find people who are seeking to solve the same problem as we are - right now.
One extra node on your network doubles its value - as I'm keen on repeating. We add each other through discovery.
So, if there's any conspiracy theory at play here that could cause us a collective problem, it's the one that's making you think twice about sharing, the one making you consider disconnecting and stopping sharing.
The centre would love that.
Stand firm. The network needs you.