Farmers would go broke if their fields remained fallow year after year. Yet many businesses have fallow fields of data waiting to be plowed and harrowed by advanced analytics. Samantha Ann Schwartz (@SamanthaSchann) reports a survey of 500 IT decisions makers in the U.S. and U.K. by SnapLogic found, “With all the data organizations collect, only 51% actually put it to use. … Less than one-third of respondents said they completely trust their data to contribute to business decisions.”[1] Even among companies putting data to use, challenges exist. Another survey of U.K. decision-makers found, “About half of companies said that their ability to use data to make business decisions was ‘excellent’ or ‘good’, with 13 per cent giving it as ‘poor’ and 0.6 per cent as ‘terrible’. A further 37 per cent said that their approach was merely ‘acceptable’, implying that there is still a lot of work to do to bring data-driven decision making (DDDM) to the forefront.”[2] Both surveys found many decision-makers simply don’t trust their data.

Making data trustworthy

Kayla Matthews (@KaylaEMatthews) observes, “Accurate and reliable data can bring context to research studies, help people understand trends, aid business managers in knowing what’s working well for achieving company goals and much more. However, not all data is as beneficial as it seems at first. Bad data can negate all the positive factors of trustworthy information. … When business leaders blindly trust data — especially when making decisions — they inevitably set the stage for problems.[3] Tom Allen notes, “Poor data hygiene can have a serious impact on DDDM; it slows down the availability of data for decision-making and can mean that analysts need to spend time cleaning the data when they finally get it. Respondents exhibited even lower levels of trust in externally-generated data: only 33 per cent said that their trust in this data is ‘good’ or ‘excellent’, while the vast majority (59 per cent) called it ‘acceptable’. If decision makers cannot trust their data, then DDDM becomes impossible.”[4] In other words, decision-makers have good reason to be wary of untrustworthy data. Making sure the data used to enhance organizational decision-making is of the highest quality is well worth the effort and cost.

One way to help improve your data’s trustworthiness is to collect only the data needed to gain desired results. Too much data can complicate matters. Schwartz notes, “Without effective guidance, big data’s potential remains trapped. But if a company has the ability to reevaluate the who, what and why of data, it can bring new insight to the surface. What can alleviate the pressures of handling data is transitioning big data workloads to the cloud and divesting on-premise platforms.”

Data integration

Another way to improve the effectiveness of collected data is to integrate it. Schwartz notes, “Siloed data is a problem across all sectors and only 2% of respondents said their organization successfully shares data.” Bob Violino (@BobViolino) reports results from another survey agree with that conclusion. “Enterprises are facing a number of difficulties as they attempt to integrate data into their real-time security, operations and business decision making,” Violino writes. “That is among the findings of a new survey by research firm Vanson Bourne commissioned by data operations company Devo Technology. The firm queried 400 business, IT, and security decision makers, and about two thirds (68 percent) said their organization has so much data it struggles to make use of it all. While data is everywhere, creating a unified view of the information is challenging. About three quarters of the organizations (74 percent), said their business is currently using different systems for real-time and historical data storage and analysis, and 95 percent face obstacles when trying to get a single view of data.”[5]

Sometimes the integration challenge involves hardware; sometimes the challenge involves software; and, sometimes the challenge involves people. Schwartz explains, “Legacy technologies [sometimes] stand in the way of pursuing full data-based opportunities. Other barriers include a disconnect between sharing data between different departments and the ‘tedious and manual’ requirements of integration.” Srinivasa Gopal Sugavanam, an associate vice president of Infosys’ Data & Analytics Practice, laments, “Data is often seen as someone else’s problem, rather than everybody’s opportunity. This is unsurprising, given the unimaginable scale of information generated by even modestly-sized businesses, and the perceived difficulty of turning this data into usable insight. The problem, however, is not one of data volumes or the complexity of analytical tools, but the lack of integration of different data sources and the technologies that can turn raw information into actionable insight.”[6] He recommends doing everything possible to eliminate silos and create data lakes. He explains, “If the problem lies with the lack of integration of different data sources and technologies that can turn raw information into actionable insight, then the solution is to build a single ‘data lake’ that can store all the information generated and gathered by a business. With all your data in one place, a reservoir so to speak, information can be channeled into the various analytics engines, business intelligence platforms, and visualization tools that turn raw data into usable insights.” In addition, he states, “It’s imperative to adopt master data management (MDM) practices to create a comprehensive view of data resources, which can manage the complex task of gathering, cleansing, and analyzing the information.” Because data can be either structured or unstructured, companies need a platform that can handle both formats. Most cognitive computing platforms are up to that task.

Summary

Schwartz concludes, “Because the price of storing data is expected to decline and processing speeds to double, there is no excuse not to use all the tools available for optimizing how data is handled.” Companies that successfully integrate and analyze all of their data will be Data Age winners. Those companies, according to Sugavanam, “will turn their internal and external data from a vast, untapped resource, into a source of new revenue opportunities that will repay the technological investment many times over.”

Footnotes
[1] Samantha Ann Schwartz, “Collecting value or dust? Only half of organizations use their data,” CIO Dive, 12 July 2018.
[2] Tom Allen, “Untrusted, low-quality data is hurting decision-making in business,” Computing, 22 August 2018.
[3] Kayla Matthews, “How to spot bad data, and know the limitations when it’s good,” Information Management, 24 July 2018.
[4] Allen, op. cit.
[5] Bob Violino, “Creating a unified view of data still elusive for most organizations,” Information Management, 29 August 2018.
[6] Srinivasa Gopal Sugavanam, “5 best practices to free data from silos and boost the bottom line,” Information Management, 2 March 2018.