We live in the digital age. The world is connected as never before. And data is being generated at an unprecedented rate. Many people, like entrepreneur Richard Branson (@richardbranson), see this as an age of opportunity. He has stated, “In this digital age, it doesn’t really matter if you are in Canary Wharf or the Caribbean; there are opportunities waiting to be grasped by entrepreneurs.”[1] Branson is celebrating the benefits of connectivity in the digital age. Other people view the digital age, connectivity, and big data through a more skeptical eye. Where the optimists see an opportunity to discover new insights, the pessimists see privacy and bias concerns. Add to that a Capgemini study that concluded “the majority of big data projects don’t glean sufficient insights to be labelled profitable” and it could make you question the relevance of big data.[2] On the other hand, in his book entitled The Business Book, organizational theorist Geoffrey Moore (@geoffreyamoore) wrote, “Without big data, you are blind and deaf and in the middle of a freeway.” Although I’m very aware of the legitimacy of the arguments raised by the pessimists, we now have sufficient evidence to conclude Moore was closer to the truth. Of course, it’s not big data that keeps you from being blind and deaf, it’s the analysis of that data.

Reasons to be concerned about big data analytics

In a 2017 TED talk, mathematician Cathy O’Neil (@mathbabedotorg) stated, “Algorithms are opinions embedded in code. It’s really different from what you think most people think of algorithms. They think algorithms are objective and true and scientific. That’s a marketing trick. It’s also a marketing trick to intimidate you with algorithms, to make you trust and fear algorithms because you trust and fear mathematics. A lot can go wrong when we put blind faith in big data.”[3] Algorithm bias can be a problem. Fortunately, O’Neil pointed out later, they can also be checked for bias and fixed if bias is found. She explains:

“The good news is, we can check them for fairness. Algorithms can be interrogated, and they will tell us the truth every time. And we can fix them. We can make them better. I call this an algorithmic audit, and I’ll walk you through it. First, data integrity check. … Second, we should think about the definition of success, audit that. … Next, we have to consider accuracy. … And finally, we have to consider the long-term effects of algorithms.”

If businesses fail to get their algorithms right, they can be assured regulators are coming for them. O’Neil concludes, “This is a political fight. We need to demand accountability for our algorithmic overlords.” Getting the algorithms correct doesn’t end the concerns about big data analysis. Business consultant John Dobson explains, “Every time you search on Google, share on Facebook, buy on Amazon or interact with the internet in any way, these firms are collecting masses of information about you. They act as a humongous surveillance system, scooping up information on a huge scale. This is big data and when I say big data, I really mean BIG data.”[4] He continues:

“Depending on your point of view, much of this activity with big data is benign, simply making businesses more efficient, allowing directors and managers make better, more informed decisions. But how comfortable are you about your lack of privacy brought about by big data? By mining big data (yes, it is called data mining) companies can build up a detailed profile of you. Every aspect of your personality can be examined, whether you are outgoing, whether you are conscious of the environment, where you like to travel and so on. This is hugely valuable information to big data holders as they can sell on this information to those who wish to target us with their product advertising. Make no mistake, there is huge money to be made.”

Privacy is not just an individual concern. Businesses failing to take privacy concerns seriously are likely to find themselves in hot water, facing large fines, and even risk going out of business. With all this negativity surrounding big data collection and analysis, are big data projects worth pursuing?

Reasons to embrace big data analytics

Before discussing the benefits of big data analytics, let’s talk about the data. As I noted in a previous article, “Weighing the potential benefits and drawbacks of collecting and storing all types of big data, not just personal data, is a good idea.”[5] An organization has best chance of leveraging data when they gather and analyze data best-suited to solving a business problem. Jennifer McKevitt (@mckvt) explains, “Too much data, involving the deployment of analytics, is as likely to prove overwhelming as it does helpful. … Assessing the results and deciding what information is actually useful, and then determining appropriate focus and alerts will help in separating the numerical wheat from the chaff. The fact is that analytics only really provide insight when they point the way to resolving an issue. … Assisting in determining the road to positive change and greater efficiency is their true power.”[6]

Like any business decision, big data project success generally rests on being able to make a business case for investing. One big data analyst, who uses the nom de plume cduby, suggests a road map for making a business case for big data. He starts the process by asking a series of questions: “Are data silos preventing the organization from getting a complete view of the customer or logistics? Is the volume of data required to solve the problem too much or too expensive for existing systems to handle? Are the unstructured or semi-structured data required to solve the problem not working effectively in existing systems? If the answer to any of these questions is yes, then Big Data is likely a good fit.”[7] If big data does appear to be a good fit, “Next calculate the return of the solution to the business. Return can come from cost savings from increased efficiency or reduction in loss, increased sales resulting from improved customer satisfaction, or new revenue and growth from new data products. Then estimate the investment required for the solution. What are the costs of the development and infrastructure required for the solution? How much will it cost to operationalize the solution? How much will it cost to maintain the solution in coming years? The value of the solution is the return minus the investment.”

Concluding thoughts

Many companies are learning that making a business case and getting the data right is paying off. Justine Brown insists, ““Business leaders that think digital transformation is simply a trend or a passing fad are simply wrong. To remain competitive in the market, going digital is becoming mandatory.”[8] In other words, the ethical use of big data is not only relevant it’s essential in today’s business environment.

Footnotes
[1] Richard Branson, “Digital Age Quotes,” Brainy Quotes.
[2] Dan Worth, “Most big data projects aren’t profitable, claims Capgemini,” Computing, 2 June 2016.
[3] Cathy O’Neil, “The era of blind faith in big data must end,” TED2017, 20 July 2018.
[4] John Dobson, “Are you worried about Big Data? You should be,” The Sunday Guardian, 13 May 2017.
[5] Stephen DeAngelis, “Can You have too much Data?” Enterra Insights, 30 May 2019.
[6] Jennifer McKevitt, “When the data is overwhelming, separate wheat from chaff,” Supply Chain Dive, 15 February 2017.
[7] cduby, “How to Make a Business Case for Big Data,” Hortonworks Community Connection, 31 March 2017.
[8] Justine Brown, “Want digital transformation? Make sure you’re investing in the right places, McKinsey says,” CIO Dive, 10 February 2017.