According to Daniel Kahneman, winner of the 2002 Nobel Prize in Economics, “Intuition works less often than we think.”[1] He made that assertion last fall during a New York City meeting of the Santa Fe Institute’s Business Network. “What about the chess master’s intuition? The financier’s expertise? The doctor’s perspective that cannot be captured by data and models? … Kahneman shared anecdotes along with research from Paul Meehl suggesting that the ‘outside view’ of non-causal and statistical models for evaluation trounced the ‘inside view’ favored by intuition in every recorded example. Where intuition performs best, he said, is when it is anchored in the data-informed ‘outside view’ and disciplined by immediate feedback.” Kahneman’s observations only strengthen arguments about the value of cognitive computing in today’s business world and how actionable insights can improve decision making.

Larry Hardesty agrees that intuition (or subject matter expertise) still has a role to play in big data analytics; but, it is not the principal role. “Big-data analysis consists of searching for buried patterns that have some kind of predictive power,” Hardesty writes. “But choosing which ‘features’ of the data to analyze usually requires some human intuition. In a database containing, say, the beginning and end dates of various sales promotions and weekly profits, the crucial data may not be the dates themselves but the spans between them, or not the total profits but the averages across those spans.”[2] Or it may be the most crucial data may be weather patterns, holidays, or something else. The variables are many and way beyond the ability of a human to deal with. Fortunately, cognitive computing systems — like the Enterra Enterprise Cognitive System™ (ECS), a system that can Sense, Think, Act, and Learn® — can handle as many variables as needed to provide the correct “outside view” for better decision making. And, as Hardesty notes, advanced analytics systems can accomplish in hours what it takes months for a team of human analysts to accomplish.

Big data analytics, however, is more about the analysis than the size of the data. Last fall, Peter Sondergaard (@PeterSonderg), Senior Vice President and Global Head of Research at Gartner, told a group of conference participants, “Big data was so yesterday; it’s all about algorithms today. … Big data is not where the value is. Sure, data is necessary, but it’s transient. By itself it will not be transformative. … Data is inherently dumb. It doesn’t do anything unless you know how to use it, how to act on it, because algorithms is where the real value lies. Algorithms define action.”[3] Sondergaard went on to say:

“The algorithmic economy will power the next great leap in machine-to-machine evolution in the Internet of Things. … Products and services will be defined by the sophistication of their algorithms and services. Organisations will be valued, not just on their big data, but the algorithms that turn that data into actions, and ultimately impact customers.”

Ian Swanson, co-founder and CEO of DataScience, reports results from a study published by Accenture and General Electric, which concluded, “Eighty-nine percent of corporations believe that not leveraging big data will result in lost market share.”[4] Analysts at Flash Global add, “Big data utilization is the wave of the future for all aspects of business including supply chain management. There is simply too much information available in today’s world to ignore all but what can be comprehended by a single person.”[5] As Shakespeare would say, “There’s the rub.” Even a team of very bright individuals can’t comprehend all of the variables that could affect the outcomes of decisions made by business executives. But that’s exactly the area in which cognitive computing excels. Paul Walsh, Vice President of Weather Strategy at The Weather Company (which was recently acquired by IBM), and Joel Cawley, General Manager of IBM Information and Insight as a Service, assert that “data mashup” is one of the most important business trends in play today.[6] They explain:

“[Data mashup] is the process of combining, correlating and analyzing different internal and external data sources, such as sales data, trend predictions, weather data and social data. Through mashups, people can derive powerful insights from the data to improve sales and merchandising, streamline processes, better understand customer behavior, and drive success. Data mashups gain their name from the music DJ practice of taking two or more songs and merging them to make a new mix. The concept of data mashups is not new, but advances in cognitive analytics and the ability to access multiple data sources in real-time, are putting this practice at the top of the 2016 trends list. … According to Gartner, over the next few years, leading organizations will link analytic initiatives to financial objectives, increase investments in advanced analytics, and incorporate a wider range of exogenous data. Gartner predicts that by 2019, 75% of analytics solutions will incorporate 10 or more exogenous data sources from second-party partners or third-party providers.”

In the future, notes Michael Fitzgerald, no single company will possess all of the potentially important data it needs to make breakthrough discoveries or obtain valuable actionable insights. “CIOs need to recognize that no one company will be able to create such powerful data by itself,” he writes. “That demands Mashup IT.”[7] One of the important characteristics of being able to mashup data is that data created for one purpose (like a census) can be repurposed for other activities (like trade promotions). The future belongs to systems that can collect disparate data, integrate it, analyze it, and mash it into new applications that provide actionable insights. No matter how bright a subject matter expert may be, his or her intuition will never be better than a system providing cutting edge big data analytics.

Footnotes
[1] Santa Fe Business Network, “The problem with intuition: An interview with Daniel Kahneman, Santa Fe Institute, 5 November 2015.
[2] Larry Hardesty, “Big Data Analysis System Replaces Human Intuition with Algorithms, Outperforms Human Teams,” Scientific Computing, 19 October 2015.
[3] Aimee Chanthadavong, “Big data is useless without algorithms, Gartner says,” ZDNet, 27 October 2015.
[4] Ian Swanson, “Five Data Science Predictions for 2016,” insideBIGDATA, 21 December 2015.
[5] “Is Big Data Analytics a Key to a Successful Supply Chain in the Next 5 Years?” Flash Global, November 2015.
[6] Paul Walsh and Joel Cawley, “How Weather-Commerce Mashups Are Empowering Retailers,” Longitudes, 3 February 2016.
[7] Michael Fitzgerald, “Mashup Analytics: The Future Of Big Data,” InformationWeek, 16 July 2013.