Steve Lohr, technology columnist for the New York Times, writes: “The quest to find decision-making insights in the modern data flood is certainly an appealing notion. After all, there is so much of data, from the traditional stuff inside corporate databases to e-mail, Web-browsing patterns, social-network messages and sensor data. Information drives decisions, so more of it ought to open the door to better decisions. The World Economic Forum has declared that data is a new asset class. All that is the intellectual and marketing tailwind behind the concept known as big data.” [“Big Data: Sorting Reality From the Hype,” 30 September 2011] In both his headline and opening paragraph, you can sense Lohr’s skepticism. As President and CEO of a company that is, in fact, trying “to find decision-making insights in the modern data flood,” I have to pay attention to what the skeptics are thinking.
The source of Lohr’s skepticism was a Forrester study that had just been released. Lohr reports that the study “provides some leavening perspective on the big data phenomenon.” It turns out, however, that the authors of the study conclude, “that big data is a real and significant trend.” They write: “Big data technology, while early-stage, is not vapor-ware.” Vaporware is a pejorative term that describes computer hardware or software that is announced to the general public but never actually materializes. The “leavening perspective” to which Lohr refers deals with the fact that a lack of talent to deal with big data is holding back many companies. As a result, he writes, “big data is an applied science project in most companies.” He reports that “the major potential constraint is not the cost of the computing technology but the skilled people needed to carry out these projects — the data scientists.” As anyone in the industry knows, finding good programmers who can deal with big data remains a challenge. Lohr concludes:
“Big data is about finding patterns in the proverbial noise of vast, unstructured data sets. The big data tools, [Boris Evelson, a Forrester analyst and coauthor of the report, with Brian Hopkins,] noted, are not themselves costly. Much of the software is based on open-source Hadoop, a framework for handling diverse data and probing it with distributed, parallel-processing computing clusters. There are commercial versions from companies including Cloudera, I.B.M., EMC and Hortonworks. And business intelligence software makers, like Microstrategy, are integrating their offerings with big data tools. And there are cloud-based services emerging for big data applications. Yet if the tools are comparatively low-cost, the skills needed are specialized and technical. Exploring for patterns in the data is not yet for the corporate rank and file. ‘In big data today,’ Mr. Evelson said, ‘it’s all about programming. You need Java programmers, computational statisticians and mathematicians.”
To that list of skilled workers, I would add ontologists. As I explained in Part 4 of these Big Data Dialogues, the Enterra Solutions® approach to big data uses a unique ontology that can help identity relationships in ways that other approaches can’t. Forrester analysts believe that the future of start-up big data firms is very bright. In fact, “Forrester expects that by Q3 2012, companies such as Teradata, Oracle, SAP/Sybase, Microsoft and HP/Vertica will acquire Hadoop start-ups such as Cloudera, MapR Technologies, DataStax, HStreaming and Outerthought.” [“Forrester predicts data warehouse suppliers will integrate Hadoop big data platform,” by Cliff Saran, Computer Weekly, 17 October 2011]
McKinsey consultants Brad Brown, Michael Chui, and James Manyika believe that the business sector is about to enter the “Big Data Era.” They assert that “radical customization, constant experimentation, and novel business models will be new hallmarks of competition as companies capture and analyze huge volumes of data.” [“Are you ready for the era of ‘big data?” McKinsey Quarterly, October 2011] They begin their report with case study of a company that was losing business to a competitor. The reason, it turns out, was that the competitor had embraced big data technology. Brown, et al. write:
“By constantly testing, bundling, synthesizing, and making information instantly available across the organization—from the store floor to the CFO’s office—the rival company had become a different, far nimbler type of business. What this executive team had witnessed first hand was the game-changing effects of big data. Of course, data characterized the information age from the start. It underpins processes that manage employees; it helps to track purchases and sales; and it offers clues about how customers will behave.”
Brown and his colleagues explain why big data is important. They write:
“Over the last few years, the volume of data has exploded. In 15 of the US economy’s 17 sectors, companies with more than 1,000 employees store, on average, over 235 terabytes of data—more data than is contained in the US Library of Congress. Reams of data still flow from financial transactions and customer interactions but also cascade in at unparalleled rates from new devices and multiple points along the value chain. Just think about what could be happening at your own company right now: sensors embedded in process machinery may be collecting operations data, while marketers scan social media or use location data from smartphones to understand teens’ buying quirks. Data exchanges may be networking your supply chain partners, and employees could be swapping best practices on corporate wikis. All of this new information is laden with implications for leaders and their enterprises. Emerging academic research suggests that companies that use data and business analytics to guide decision making are more productive and experience higher returns on equity than competitors that don’t.”
That kind of argument helps ease the skepticism that Lohr expressed at the beginning of this post. Whereas Lohr writes that increased access to information “ought to open the door to better decisions,” the McKinsey analysts confirm that it does. They go on to report that their own research shows “that ‘networked organizations’ can gain an edge by opening information conduits internally and by engaging customers and suppliers strategically through Web-based exchanges of information.” They continue:
“Over time, we believe big data may well become a new type of corporate asset that will cut across business units and function much as a powerful brand does, representing a key basis for competition. If that’s right, companies need to start thinking in earnest about whether they are organized to exploit big data’s potential and to manage the threats it can pose. Success will demand not only new skills but also new perspectives on how the era of big data could evolve—the widening circle of management practices it may affect and the foundation it represents for new, potentially disruptive business models.”
Normally, the word “disruptive” causes chills to run up the spines of supply chain professionals. In this case, however, being disruptive is a good thing because it promises a better future. Saul Klein, an entrepreneur and a partner in a private equity firm, asserts that “successful disruptive businesses” deliver not only good service and a quality product but “something that is fundamentally better than what is already available.” [“Ask the experts: Make yourself disruptive,” Financial Times, 26 April 2011] That is what big data technologies promise to deliver — something that is fundamentally better than what is already available.
Brown and company go on to ask “five big questions about big data.” (We’ll discuss those questions in tomorrow’s post.) The questions concern “important ways big data could change competition: by transforming processes, altering corporate ecosystems, and facilitating innovation.” They don’t claim to have all the answers, after all, they acknowledge, “these are still early days for big data, which is evolving as a business concept in tandem with the underlying technologies.” Before asking the first of their five questions, they identify what they believe are “big data’s key elements”: “First, companies can now collect data across business units and, increasingly, even from partners and customers. … Second, a flexible infrastructure can integrate information and scale up effectively to meet the surge. Finally, experiments, algorithms, and analytics can make sense of all this information.” Like past waves of technology that have ushered in new eras, Brown and company believe “the era of big data also could yield new management principles.” The reason is simple, “future competitive benefits may accrue to companies that can not only capture more and better data but also use that data effectively at scale.”
How big are those benefits? “A study from IBM shows that companies which excel at finance efficiency and have more mature business analytics and optimisation can experience 20 times more profit growth and 30% higher return on invested capital.” [“What is big data and how can it be used to gain competitive advantage?” by Cliff Saran, Computer Weekly, 1 August 2011]
If you’re still skeptical about the world being on the cusp of the big data era, you should listen to the prediction that Sanjay Mirchandani, chief information officer at EMC, made to Saran. He told Saran that he “believes there is a perfect storm” gathering that will be “driven by affordable IT and the ability to gather information.” “The onus on IT,” he says, “is to leverage data.” Saran concludes:
“Experts agree that big data offers a big opportunity for businesses. The technology appears relatively immature and the idea of a complete big data solution is a bit of a pipedream. There are products that can handle aspects of big data, like analysis of large datacenters, but this is not the complete story – businesses should begin to pilot big data projects to evaluate how to benefit from the additional insight it can potentially offer.”
The fact that Saran reports that “the technology appears relatively immature” should instill excitement rather than skepticism. There are already some pretty amazing things being done with big data and, if Saran is correct, we’ve only begun to understand its potential.
