Big Data is a Big Buzzword this year in marketing analytics circles. But what does it mean? Is Big Data really a Big Deal for Direct Marketers?

Big Data refers to data sets so large, unstructured and complex that they exceed the scope of our typical tools, and new ways of crunching them must be employed. This is accomplished by distributing multiple databases over multiple servers. These servers crunch away simultaneously in real time to deliver answers faster than conventional relational databases could manage.

This concept is, in fact, not new. In fact, Big Data was sent up in Douglas Adams’ iconic work from the late 70’s, “Hitchhiker’s Guide to the Galaxy”.

In Adams’ story, the world’s most powerful computer is tasked with calculating “the answer to the ultimate question of life, the universe, and everything.” The computer dutifully chugs away for 7.5 million years and finally spits out the answer: “42”. But the long-awaited answer doesn’t illuminate – because nobody at that point in time remembers what the original question was … including the computer.

Computer processing power has changed a lot since Adams wrote this story thirty-five years ago. But the key truth illustrated in the “Hitchhiker’s Guide” story hasn’t changed a bit.

Despite its enormous potential, Big Data can’t move your business forward until you ask it the right questions and stand ready to act on the answers. The success of Big Data starts with you: the marketer; the client, the business “owner.” The key element is having a clear idea of the end goal, or objective.

Too often, this is how the conversation goes:

Stakeholder: “Here’s all the data from our ABC, QRS and XYZ systems dating back to the Carter Administration. Let’s analyze it and see what the data says.”

Big Data Wrangler: “Love to! So, you’d like me to tell you what the data says about…what?”

Stakeholder: “You know, about the business.”

Big Data Wrangler: “Okay, let me ask another way. What kinds of things do you wish you knew? What cool things could you do if you knew them? How do you see yourself using the new insights this data can provide to achieve your goals?”

Stakeholder: “Well, that would depend on what the data says.”

Bit of a stalemate there, right? You, the business owner, will first need to think about what the data might do for you strategically, in the big picture. In order to get actionable data, the business questions need to be asked BEFORE the crunching begins – not afterward. What kind of questions?

Let’s look at some very impactful questions others have asked of their data:

• Which ball players should we sign to get the most wins within our salary budget?
• Who is more likely to get a disease based on myriad lifestyle factors?
• How can we more accurately identify fraud from unstructured comment data?
• Can we use known terrorists’ purchase behavior to identify other terrorists?
• How can we use real-time online data to predict who will churn?

Until recently, the ability to harness Big Data was cost-prohibitive to all but the largest organizations with big budgets. But with the advent of cheap storage, cloud computing and open-source tools like Hadoop and MapReduce distributed file processing, the ability to leverage these data sets is getting more accessible to businesses with more modest means.

The promise of Big Data has spawned the belief that a modern business should keep every piece of data, no matter how small, in perpetuity. After all, who can tell which data elements will be gems of insight later on? That’s an understandable conclusion, but it’s wrong. Too much data can actually confuse and obscure the right path.

A 2011 study by The Economist, sponsored by data analytics giant SAS, suggests that businesses hone in on what they NEED to know, instead of what it’s possible to know. Once those questions have been identified, choose a set of metrics (ideally no more than a dozen) that will answer those questions. Then collect and keep the data that feeds those metrics, and don’t worry so much about the rest. That’s a much more sensible and accessible way to leverage Big Data.

Not all complex analysis requires Big Data techniques. But asking the right questions before you begin your analysis is the right thing to do, regardless of how big your data set is.

This post will also appear on the “greenbananas.net” blog.