There has been an enormous amount of hype about big data. The World Economic Forum declared big data a valuable resource like oil or gold. People forget, however, that extracting oil or gold is often extremely difficult and many prospecting efforts result in failure. Extracting value from big data can also be difficult and many companies have reported their big data projects have failed or produced disappointing results. Little wonder some companies are getting discouraged. A survey published by Syncsort found, “While more organizations are prioritizing big data analytics in IT initiatives, many are facing a series of problems connected to the gathering and harvesting of this information.”[1] When expectations go unmet, it’s easy to become discouraged. Reporting on the Syncsort survey, Brandon Vigliarolo (@bviglia), notes, “Responses given to the question of why enterprises are ineffective at getting big data insights are a bit disheartening: 53% say their team lacks the IT skills or staff to work on extracting meaningful insights from data, 50% say they lack the tools to feed downstream apps with the right data at the right time, 44% simply lack the time to sort through data, and 31% say their organization hasn’t invested enough in analytics platforms.”[2]

Kalev Leetaru (@kalevleetaru), Senior Fellow at the George Washington University Center for Cyber & Homeland Security, believes too many researchers have succumbed to big data hype. He asserts, “Data science has become more and more about hype rather than results. The ‘big data’ era was supposed to be about giving voice to our massive datasets. It seems we are less and less interested in listening to those voices.”[3] As a result, he asks, “In a world in which we seem increasingly uninterested in what our data actually says, is there any hope left for the future of ‘big data’?” To be fair, Leetaru is primarily addressing academic rather than business researchers. From my experience, businesses care less about hype and care more about results. Nevertheless, considering increasing discouragement about big data projects, Leetaru’s question is relevant: Is there any hope left for the future of big data? My answer is an unequivocal yes.

The importance of big data for business

Businesses need to make myriad decisions each and every day. Bain analysts, Michael C. Mankins and Lori Sherer (@lorisherer), assert if you can improve a company’s decision making you can dramatically improve its bottom line. They explain, “The best way to understand any company’s operations is to view them as a series of decisions.”[4] They add, “We know from extensive research that decisions matter — a lot. Companies that make better decisions, make them faster and execute them more effectively than rivals nearly always turn in better financial performance. Not surprisingly, companies that employ advanced analytics to improve decision making and execution have the results to show for it.” Their assertions about the importance of advanced analytics would obviously be meaningless if companies had no data to analyze. Fortunately, most companies face the opposite challenge — they often have too much data. Priya Kantaria (@KantariaPriya), explains that data comes from numerous sources of variable value. “Big Data,” she writes, “is a combination of structured data and unstructured data at scale and at speed. Structured data can be compartmentalized into fields, such as age, height or gender, in a relational database. Unstructured data refers to data that cannot be analyzed in a relational database, such as video or tweets. The trick is ensuring the accuracy, reliability and legibility of this data. Big Data therefore typically has four dimensions: volume because there is a lot of it, velocity, as it needs to be analyzed quickly, variety because it comes in many different formats, and veracity because it can be difficult to understand.”[5]

Overcoming big data discouragement

Business leaders might not get so discouraged about their big data results if they realized others were experiencing similar challenges. Herbert Chia, a Venture Partner of Sequoia Capital China, believes every company must go through five different stages of big data development.[6] Those states are:

• Stage 1. “In stage one, everyone is talking about big data, but only a few companies truly own big data resources or have the capability to harness them. Most people are still skeptical about big data, and only a handful of companies are willing to commit huge resources.”
• Stage 2. “In stage two, the massive collection of user behavior data becomes possible. … Despite the overwhelming amounts of data, applications [are] few and far between. Leading data-driven companies [look] for better ways to move forward.”
• Stage 3. “In stage three, the emergence of deep learning and the wide application of videos and language recognition substantially boost the applications of big data technology in areas such as financial technology, healthcare and smart cities.”
• Stage 4. “In stage four, data becomes a key resource for companies and governments seeking innovation. But concerns about data security and customer privacy start to surface. Data governance becomes crucial. Meanwhile, the complexity of data sources makes people realize that a proper management system and heavy investment in data middle-office technology are needed in order to keep data updated and in good quality.”
• Stage 5. “In stage five, the negative impacts of data technology and artificial intelligence become key concerns. Potential job losses and unfair competition are some of the issues that might need to be addressed.”

Currently, many companies find themselves somewhere between stages two and four. Hopefully, there will be a stage six in which companies and governments have proper policies and regulations in place to protect individuals so the full value of big data analytics can be realized. When big data analytics are done correctly, Abe Ankumah, Co-Founder and Chief Executive Officer at Nyansa, Inc., insists the benefits are well worth the effort it takes to overcome big data challenges. “Beyond simply improving network operations,” he writes, “this data can provide a better understanding into improving business operations. Insights into user behavior networks can also be applied to improving revenue performance.”[7] As a result, he recommends, “Companies should explore the use of tools that apply artificial intelligence and machine learning to network data sources to answer complex questions that simply can be performed by humans. … By applying this scheme to the enterprise network, companies can now use big data to make real business decisions. … The answers are buried in a host of sources across the network — all of which must be examined and compared to provide an accurate picture of how the network is performing from the end user point of view and what recommendations, if any, can be provided to make things better.”

Concluding thoughts

I predict big data results will eventually live up to the hype. It will take patience and perseverance on the part of companies, but results will come. Aleksei Antonov (@AUAntonov), Co-Founder of SONM, explains, “Big data is disrupting every industry it integrates with, altering the corporate landscape by determining market trends, behavioral analysis and other insights that only improve business decisions. … Big data comes with massive potential.”[8] That massive potential can only be unlocked by cognitive technologies that can handle both structured and unstructured data. As the technology matures, corporate and governmental expectations will eventually be met. There is no need to be discouraged.

Footnotes
[1] Brandon Vigliarolo, “Big data adoption exploding, but companies struggle to extract meaningful information,” TechRepublic, 19 March 2019.
[2] Ibid.
[3] Kalev Leetaru, “Do We Even Care What Our Data Says Anymore?,” Forbes, 10 March 2019.
[4] Michael C. Mankins and Lori Sherer, “Creating value through advanced analytics,” Bain Brief, 11 February 2015.
[5] Priya Kantaria, “What is Big Data and why does it matter for business?” Verdict, 26 February 2019.
[6] Herbert Chia, “Five stages of big data development,” Ejinsight, 14 February 2019.
[7] Abe Ankumah, “The Big Problem With Big Data,” Forbes, 9 January 2019.
[8] Aleksei Antonov, “Big Data is Kind of a Big Deal: Analyze it to Make Better Business Decisions,” insideBIGDATA, 16 November 2018.