In Defense of Big Data

The last few weeks have seen a number of interesting blog posts and news articles have lamented the term “Big Data”. VCs, bloggers, and I’m sure many others are getting worried the term is being oversold, and certainly is over-used. Questions abound: are benefits as good as they seem, and is there really a treasure trove of value within Big Data?

Arguments abound for why the term is becoming lacklustre in 2013. And while we at Canopy Labs don’t necessarily see ourselves as as Big Data startup (we prefer to see ourselves as making analytics more user friendly), we do feel the term warrants a defense. We see three, in particular.

Defense #1: Big Data is becoming available to any business

Today’s data-oriented enterprise is different than those of the past. Critics are correct in saying that large banks, large retailers, and large insurance companies have had Big Data problems for decades… And they’ve even solved their problems through Master Data Management (MDM) strategies, data warehousing, and other techniques that won our hearts and then became boring, much like Big Data is now.

What’s different today is that Big Data is available to small businesses and enterprises as well, whereas earlier technologies were limited to the Fortune 500s of the world. Every Fortune 500 firm had a Big Data problem in 1999, and most even had them in the 1980s. Today, however, even a business with 100,000 customers can collect more data than they can hope to analyze. This is both powerful and dangerous — it’s only in the early 2010s that businesses of any size have enough data to analyze, model, and predict what large swaths of their target populations are up to. At the same time, they need to make intelligent decisions around investments in analytics and data-oriented decisions. It is here that Big Data can make a Big Difference.

Defense #2: Big Data is still evolving

Justin LaFayette of Georgian Partners recently wrote about the importance of apps for Big Data. Similarly, we’ve argued that analytics (and as such, Big Data) struggles with usability issues more than infrastructure issues.

In this age of economic tumult and rapid innovation, we should not forget that Big Data is young, as far as society- and technology-wide trends go. The tools that make Big Data accessible are themselves quite young — Hadoop and Mongo are both about 5 years old, and analytic tools like Mahout have yet to enter popular use. Fine machine learning and statistical platforms like R and Weka have yet to be parallelized (if they ever will be).

In essense, the maturation of these tools and trends will lead to Big Visualization, Big Analytics, and the general appification and SaaS-ification of Big Data. That is where the particularly exciting benefits of data will come from. If we’re experiencing Big Data fatigue, it might be our own overblown expectations. Give it a few more years, I say, before truly judging the fad or trend.

Defense #3: Does the end justify the means?

The use of Big Data in business has always been a fuzzy concept. Unfortunately, the analytics industry has had such difficulties across lots of trends. Neural networks, artificial intelligence, data warehousing, and (back in the day) Taylorian scientific management concepts have all been touted as solutions to solve every major business problem. They too fell out of favour, yet are also consistently used in the background — most of today’s businesses use some sort of dashboarding (read: scientific management) and want to centralize their data. Big Data will suffer a similar fate: to be pushed aside for more trendy key words while also entering our subconscious approaches to business management.

Regardless of whether the Big Data trend/fad deserves the praise and attention it has gotten, the marketing effort behind the term (refer to McKinsey, IBM, Accel, etc.) has led to billions of dollars in investment in analytic science, mathematics, and machine learning. I would argue that if for no other reason, we should love Big Data for what it affords — a business world stimulating research in scientific fields that deserve funding and attention, because they might actually lead to important discoveries.

In 2008, it was trendy to lament the fact that society’s best mathematicians were becoming investment bankers. At least today, they’re starting companies that predict flu outbreaks, help pharmaceuticals organize their data, and tell us how to better spend our money.

In Conclusion…

We’re not a Big Data startup, in the formal sense of the word. We are, however, fans of analytics, machine learning, and a data-oriented world view. Before we push Big Data into the dark, dank, forgotten basement where “MDM” and “hyperlocal startups” have been living for the last decade, let’s accept that the term has led to a great deal of good: it’s lowered the barrier to entry for smaller enterprises wanting to use data, is evolving new applications and ideas around analytics, and has stimulated a great deal of research and investment in math and science.

Written by Wojciech Gryc

Leave a Reply

Your email address will not be published. Required fields are marked *