About a year ago, Mike Wheatley wrote, “Big Data has become such a big buzzword in tech these days that anyone who’s anyone is desperate to get in on the action.” [“Gartner: Big Data Finally Going Mainstream,” Silicon Angle, 12 March 2013] Wheatley was referring both to organizations eager to jump into the stream of cash that has been predicted to be spent on Big Data analytics and to organizations that are collecting mountains of data. Wheatley notes, “The sheer amount of data that’s being collected by companies around the world is truly astonishing. What with websites and devices gathering petabytes of data on a daily basis, many large corporations today are sitting on a veritable gold mine of information that’s just waiting to be analyzed for insights.” He goes on to report that a Gartner study concluded that “businesses today feel that they can no longer afford to sit there twiddling their thumbs – now is the time to act if they’re not to be left behind.”

Ned Smith reports that his interviews with information technology experts reveal, “The opportunities [offered by Big Data analytics] are significant.” From those interviews, he was able to extract “four vectors of insight” that companies should expect Big Data to deliver. They are: “First, big data will deliver a full, start-to-finish model of a problem. Second, big data is real-time – providing immediate feedback on management decisions. Third, it offers a level of detail drawn from the whole population rather than inferred from a sample. Finally, it offers an understanding of how elements relate to one another, including a clearer picture of causalities.” [“IT ‘Brains’ Reveal Four Vectors of Big Data,” Business News Daily, 16 April 2013] Smith’s experts are a little optimistic assuming that all Big Data analysis is real-time. We’re not quite there yet; but, clearly, that is where many business leaders would like to end up. Frankly, all Big Data analysis doesn’t need to be real-time. Even non-real-time Big Data analysis is not as easy as the hype would lead you to believe. As Wheatley reports, “It won’t be anything like an easy ride for companies as their initiatives play out amidst disruptive forces that churn out newer and ever more demanding data types.” Most of those demanding data types involve unstructured data. The following video produced by Telefonica Digital provides a good overview of why so many experts are hyping Big Data’s potential.

Wheatley notes that the Gartner report stresses the urgency that companies are feeling to adopt Big Data analytics as part of their business structure. He explains:

“According to Gartner, companies feel that Big Data is a race that has to be seen through at all costs – Big Data tech has become crucial for two reasons, namely necessity and conviction. Companies can no longer afford to ignore the opportunities that simply cannot be met with traditional data streams and practices. Meanwhile, companies feel forced to act due to the never-ending media hype around Big Data.”

Since companies are feeling compelled to enter the Big Data arena, New Vantage Partners, a data and analytics consulting company, wanted to know “what large organizations are actually doing with big data initiatives.” They asked that question to nearly 100 senior executives from Fortune 1000 companies and reported their findings in a study entitled, “Big Data Executive Survey 2013: The State of Big Data in the Large Corporate World.” [“How Big Data is Influencing Big Companies,” Renee Boucher Ferguson, MIT Sloan Management Review, 25 November 2013] Ferguson notes that the 2013 survey is the second in the series which began in 2012. She reports, “In the first survey, 85% of respondents said their organizations were just beginning to explore big data, with only 35% investing $1 million or more.” The latest survey, however, supports Gartner’s conclusion that Big Data has gone mainstream. As Ferguson writes, “The current survey’s respondents have painted a different picture.” She goes on to report some of highlights of this year’s survey findings. They include:

• “91% have a big data initiative planned or in progress
• “60% have at least one big data initiative completed
• “32% have an initiative in production
• “88% expect to spend > $1 million on big data by 2016
• “50% expect to spend > $10 million on big data by 2016
• “14% expect to spend > $50 million on big data by 2016

“The survey results are especially interesting for organizations considering an investment in big data — time, technology or dollars — but worried about a lack of talent to push projects forward. … Between 60% and 70% of respondents listed their top areas of investment this year as: development of more sophisticated analytics; more effective integration of existing data sources; and the creation of analytical sandboxes to support data discoveries.”

Thomas C. Redman, author of Data Driven: Profiting from Your Most Important Business Asset, believes that most companies aren’t data driven and that many are fooling themselves into thinking that they are. [“Are You Data Driven? Take a Hard Look in the Mirror.” Harvard Business Review Blog Network, 11 July 2013] He notes that being data driven is important because “recent academic work shows that companies that regard themselves as ‘data driven,’ are measurably more profitable than those that aren’t.” Redman offers a dozen traits that he believes true data driven companies share in common. They are:

“The data-driven:

• Make decisions at the lowest possible level.
• Bring as much diverse data to any situation as they possibly can.
• Use data to develop a deeper understanding of their worlds.
• Develop an appreciation for variation.
• Deal reasonably well with uncertainty.
• Integrate their ability to understand data and its implications and their intuitions.
• Recognize the importance of high-quality data and invest to improve.
• Are good experimenters and researchers.
• Recognize that decision criteria can vary with circumstances.
• Recognize that making a decision is only step one.
• Work hard to learn new skills and bring new data and new data technologies (big data, predictive analytics, metadata management, etc.) into their organizations.
• Learn from their mistakes.

All of these traits are important. And most are self-evident.”

Redman goes on to offer four other insights concerning data-driven companies. He writes:

“First, data-driven companies work to drive decision-making to the lowest possible level. … Pushing decision-making down frees up senior time for the most important decisions. And, just as importantly, lower-level people spend more time and take greater care when a decision falls to them. It builds the right kinds of organizational capability and, quite frankly, appears to create a work environment that is more fun. Second, the data-driven have an innate sense that variation dominates. Even the simplest process, human response, or most-controlled situation varies. While they may not use control charts, they know that they have to understand that variation if they are going to understand what is going on. … Third, the data-driven place high demands on their data and data sources. They know that their decisions are no better than the data on which they are based, so they invest in quality data and cultivate data sources they can trust. As a result, when a time-sensitive issue comes up they are prepared. High-quality data makes it easier to understand variation and reduces uncertainty. Success is measured in execution, and high-quality data makes it easier for others to follow the decision-makers logic and align to the decision. Further, as one executes, one acquires more data. So the data-driven are constantly re-evaluating, refining their decisions along the way. They are quicker than others to pull to plug as when the evidence suggests that a decision is wrong. To be clear, it doesn’t appear that the data-driven ‘turn on a dime’; they know that is not sustainable. Rather, they learn as they go. Now take that hard look in the mirror. Look at the list above and give yourself a point for every trait you follow regularly and half a point for those you follow most — but not all — of the time. Be hard on yourself. If you can only cite an instance or two, don’t give yourself any credit. Unless you’re one of the rare few that truly score seven or more, you need to improve.”

It should be clear from the previous discussion that, even though Big Data analytics continue to penetrate business operations, we remain in the infancy of the Big Data era. The learning curve has been (and will continue to be) steep; but, we still have a lot to learn. The Gartner study captured the urgency and uneasiness that companies are feeling about Big Data and how analysis of that data can help them thrive in the years ahead. A bit of discomfort is a good thing because it keeps us on our toes, makes us ask questions, and keeps us motivated to find answers.