“There is a lot of hype and conjecture around the impact of emerging technology,” writes Mark Adams, regional sales director for UK and Ireland at Cohesity, “but one thing is certain: data is — and will continue to be — the key to digital business.”[1] If that key is going to open up opportunities for digital enterprises, data must be available to everyone who needs it. Adams argues, “Companies that really want to make the most of data both now and in the future will need to create a nuanced approach to data collection, analysis and exploitation. While business leaders recognize the value of data, they don’t necessarily appreciate how hard it is to analyze the right information at the right time. There’s a lack of consideration about how information might be used by different people in different circumstances.” As a result, organizations often store data in silos where it is cordoned off from individuals who could make good use of it. This is a widespread problem. Bertrand Moingeon (@bertandmoingeon), a Professor of Strategic Management at HEC Paris, asserts, “Organizational silos are without a doubt the most widespread managerial structure, even though all management textbooks warn against them. This is true for all kinds of organizations, be they businesses, public bodies or non-profit organizations.”[2]

Adams argues that siloed data results in fragmented (or siloed) corporate thinking. He explains, “If data is locked away in silos, then it’s far too difficult to create deeper, cross-organization relationships. That inability to collaborate effectively hampers business growth. As data is the key to creating a competitive advantage, then the proliferation of information silos is a short-cut to failure.” His arguments are not new. Nearly a dozen years ago, financial journalist Gillian Tett (@gilliantett) observed, “The bad news is that the curse of silos will not be easy to beat. For one bizarre paradox of the modern age is that while technology is integrating the world in some senses (say, via the internet), it is simultaneously creating fragmentation too (with people in one mental silo tending to only talk to each other, even on the internet.) And as innovation speeds up, this is creating a plethora of activities that are only understood by ‘experts’ in a silo.”[3] Siloed data and siloed thinking have proven to be intractable challenges.

Why Silos Persist

“The original purpose of a data silo,” writes Keith D. Foote, “was to keep secrets. … A data silo is ostensibly meant to keep private information from the eyes of those who do not need to know. Unfortunately, information that is needed by others may also be stored in the data silo. In the worst-case scenario, a silo becomes a dumping ground for data that ‘might be’ useful sometime in the future, and then sits there, never used.”[4] Data silos are also a natural result of Industrial Age organizational design, where each division within a company decides what processes need to be measured and what data needs to be collected and analyzed. Since knowledge has historically been equated with power, division rivalries ensure data silos are carefully guarded — as a result, internecine competition between division heads is often brutal.

When siloed thinking persists, siloed data proliferates. Tech journalist John Leonard (@_JohnLeonard) notes, “Every few weeks, or so it seems, a new database is added to an ever-growing and increasingly diverse list. In addition to the familiar relational databases there are analytical, transactional, in-memory, NoSQL, NewSQL, time series, event streaming, graph, distributed, wide-column, key-value and document stores. Some databases straddle several of these categories, while others are finely tuned to serve a small set of use cases or a specific domain.”[5] The proliferation of siloed data is now a real challenge for many businesses. Even within organizational areas, like the supply chain, siloed data can be a challenge. Bryan Nella (@bryann123456), a content and thought leader at Infor Nexus, observes, “A major barrier to effective decision making and digital innovation is the prevalence of silos that exist between internal departments, as well as external trading partners. Data and visibility are often locked away in pockets, prohibiting factories, logistics providers, carriers and other trading partners from planning and acting on accurate information.”[6]

Eliminating Siloed Data

Siloed thinking is anathema to digital enterprises. In a digital enterprise, Adams asserts, organizations “need to take data management to the next level of development — one where the cross-business insight you create is drawn from multiple structured and unstructured data sources.” In fact, most experts insist those multiple structured and unstructured data sources need to be drawn into a single source that becomes baseline truth for the entire organization. That is easier said than done. Ed Thompson, CTO and co-founder at Matillion, explains, “Every business needs to be data-driven to be competitive, and the best businesses discipline themselves to bring the best possible data to every decision they make — that is where the data integration problem starts. The answers are all there in those different systems, but it’s simply not ready to be used for decision making. Nowhere near.”[7] He suggests two important steps towards eliminating data silos in businesses. They are:

Step 1: Collect all of the data into one cloudy place. Thompson notes, “The software industry has never been able to standardize its data models and while the past 40 years have been littered with attempts … the dominance of some of the SaaS vendors is at least ensuring a largely open approach across the board. At the same time, it has never been easier or cheaper to store all the sources of data and harmonize them into data sets that anyone can understand. There are plenty of tools to choose from that can vastly simplify this task.”

Step 2: Transform data to get the most meaningful insights for your business — and, make sure everyone can do this. Thompson writes, “It’s critically important to make sure the data and the ability to work with it is accessible across an organization. Sure, some data might be used for critical management reports that the business relies on powered by centrally managed [extract, transform, and load (ETL) processes], but far more of it is most useful for everyday tactical decisions. The ability to transform that data should be a skill that is available right across an organization. Every department should have access to the best visualization and business intelligence made possible by a simple yet powerful data transformation capability.”

Adams concludes, “[Organizations need to create] a single and verifiable version of the truth where users have easy access to the data they need when they need it — and they create deeper and more valuable insight for the business and its customers. It might seem like a stretch for some right now, but in reality, defrag your mindset and your data, and you’ll achieve the data-first vision you seek.”

Footnotes
[1] Mark Adams, “Why the future of data management requires us to remove fragmented thinking,” Data Center Dynamics, 10 May 2021.
[2] Bertrand Moingeon, “Transversal management: how to break out organizational silos,” LinkedIn, 1 April 2017.
[3] Gillian Tett, “The lessons: The dangers of silo thinking,” Financial Times, 14 December 2009.
[4] Keith D. Foote, “A Brief History of Data Silos,” Dataversity, 21 November 2019.
[5] John Leonard, “Towards one database to rule them all,” Computing, 2 June 2020.
[6] Bryan Nella, “How to break free of data silos for a truly intelligence-driven supply chain,” Diginomica, 2 April 2019.
[7] Ed Thompson, “When Data-Driven Meets Data Silos: Let the Fun Really Begin,” insideBIGDATA, 8 September 2019.