Big Data, Meet Small Data

I’ve been following the panting exuberance of big data apostles for the past few years, rolling my eyes at most of it. Sure, it can be interesting, but maybe my age is showing when I say, “So what?” to most of it.

What finally pushed me over the edge enough to comment on it was a tweet I saw from none other than our friends at SAP, saying (I’m paraphrasing) “You have lots of data; let SAP analyze it.” But of course. There’s gold in them thar data… at least to SAP.

Just because we have lots of data, does it mean they have to be analyzed? What is the question, what is the problem, what is the opportunity being addressed? As Einstein said, “Not everything that can be counted, counts.”

It has become easier and easier to create, capture, and store data. Just ask the NSA. Or Target. Sure, data inventory doesn’t take up physical space, but it does take time, equipment, and people to manage it. And perhaps it is inventory... excess, over-produced inventory. What does lean tell us about inventory?

Sure, there is potential value in linking disparate data sets to come up with interesting and perhaps profitable conclusions. Unfortunately, the most common outcome I’ve seen is an increasing number of KPIs, dashboard indicators, and the like tied to data. And once again, an increasing number usually creates a dilution of focus. Just ask those companies that have a 2-in. thick strategic plan.

A year ago Rufus Pollock penned an article for The Guardian that described how the real revolution isn’t big data, but the increased accessibility to small data.

“Just as we now find it ludicrous to talk of ‘big software’—as if size in itself were a measure of value—we should, and will one day, find it equally odd to talk of ‘big data,’” he says. “Size in itself doesn’t matter—what matters is having the data, of whatever size, that helps us solve a problem or address the question we have.”

Bingo. Just like the 5S of lean or micro meditation of mindfulness, data is just a tool. First identify the problem, question, or opportunity. Only then determine what data, and analysis, is necessary.

“Meanwhile we risk overlooking the much more important story here, the real revolution, which is the mass democratization of the means of access, storage, and processing of data,” continues Pollock. “This story isn’t about large organizations running parallel software on tens of thousand of servers, but about more people than ever being able to collaborate effectively around a distributed ecosystem of information, an ecosystem of small data.”

Smaller data, more interconnected and accessible.

Pollock adds, “This next decade belongs to distributed models, not centralized ones; to collaboration, not control; and to small data, not big data.”

What size are your data?

First published May 16, 2014, on Kevin Meyer’s blog.

About The Author