Today’s businesses move at computer speed. As a result, analysts at O’Reilly write, “Into a world dominated by discussions of big data, fast data has been born with little fanfare. Yet fast data will be the agent of change in the information-management industry.”[1] And it wasn’t born yesterday. Nearly a decade ago, Alissa Lorentz Vice President of Creative, Marketing and Design at Augify, wrote, “Big data needs to be fast and smart.”[2] As the name implies, fast data is real-time or near-real-time data. O’Reilly analysts explain, “Fast data is data in motion, streaming into applications and computing environments from hundreds of thousands to millions of endpoints — mobile devices, sensor networks, financial transactions, stock tick feeds, logs, retail systems, telco call routing and authorization systems, and more. Real-time applications built on top of fast data are changing the game for businesses that are data dependent: telco, financial services, health/medical, energy, and others.”
The Importance of Fast Data
“You won’t get many arguments from CIOs about the importance of data to the modern enterprise,” writes Ed Anuff (@edanuff), Chief Product Officer at DataStax. “And amid the broad spectrum of data types that power business today, one that keeps rising in importance is fast data. More than three-quarters of modern enterprises use real-time, actionable data for at least some of their applications, according to Forrester Research.”[3] When Lorentz was writing nearly a decade ago, she noted, “From monitoring traffic to tracking epidemic spreads to trading stocks, time is of the essence. A few seconds’ delay in understanding information could cost not only funds, but also lives.” And, despite the pandemic, the speed of business is not slowing down. As Aiden Mathieu, Content Marketing Manager at VANTIQ, reports, “With the amount of big data created reaching 64.2 zettabytes in 2020 (almost 30 zettabytes more than previous projections) and the expectation for it to grow to more than 180 zettabytes by 2025, it shows no signs of stopping.”[4] If you are wondering how big a zettabyte is, the following graphic should help. It’s a lot of data.
Of course, not all that data needs to be analyzed in real-time. Mathieu points out, however, “With 68% of all data collected going un-leveraged, there lies a huge opportunity in turning raw data into something that creates immediate business value.” The challenge is figuring out which fast data contains that business value. According to Mathieu, “Fast data contrasts big data in that instead of collecting as much data as possible, it focuses on turning raw streaming data into immediately actionable events. This method of data analysis is vastly superior to storing everything in a database when working with IoT sensors and devices that are constantly creating new data. Only the important anomalous data points need to be acted on while everything else can be discarded.”
Mark Henderson, Head of R&D at EV Cargo Technology, likens today’s data scientists to placer gold miners who must dig and wash vast amounts of dirt to recover fine flakes of gold. He says, “Fast data is the 21st century gold rush.”[5] He quotes Marissa Mayer, the former President and CEO of Yahoo, who once stated, “With data collection, ‘the sooner the better’ is always the best answer.” Henderson believes the pandemic has made fast data even more valuable. He explains, “The global Covid-19 pandemic has demonstrated that situations change rapidly, a future course cannot reliably be charted based on past experience and you have to adapt to what is happening now.”
Fast Data and Edge Computing
As Mathieu observed, the value of fast data often lies in being able to report only anomalous data (i.e., data is processed at the edge). Anuff notes, “Fast data enables full-circle delivery of data that is ‘in motion.’ In other words, it’s generated and consumed instantly by interactive applications running on large numbers of devices. Fast data enables organizations to act on insights gained from user interactions as these insights are generated at the point of the interaction. And because decisions or actions take place right at the front-end, fast data architectures are, by definition, distributed and real-time.” Mathieu adds, “By utilizing edge computing to perform pre-filtering/processing on streaming data at the source, organizations are able to scale much faster by avoiding moving everything to the cloud. Only data that is useful for future analysis is kept. A fast data approach using edge computing also provides the benefits of distributed processing (by moving compute power closer to or on the edge devices themselves), allowing for massive reductions in latency and bandwidth.”
Concluding Thoughts
With pundits like Mathieu advocating businesses concentrate on “the important anomalous data points” and discarding everything else. You might get the idea that big datasets are dead. Mathieu denies that is the case. He explains, “As with most things in life, the big data versus fast data debate is not completely cut and dry. In order for organizations to be successful in the modern age, a combination of both approaches is necessary. Big data is extremely helpful in finding hidden trends in the data after the fact, while fast data is better suited to responding to events as they happen.” O’Reilly analysts talk about big data as being “data at rest” — something that “can be dealt with some other time.” On the other hand, fast data “demands to be dealt with” in real-time.
Because businesses now move at the speed of computers, some decisions need to be made faster than humans can reasonably respond. That is why Enterra Solutions® is advancing Autonomous Decision Science™ to take advantage of cognitive technologies that can make decisions using logic similar to that used by a company’s best experts. As Anuff concludes, “Fast data makes it possible to offer a user a ‘next best action’ at the point when a user would find it most helpful — in any experience or business process.” Lorentz calls these kinds of actions “in-the-moment decisions.” And Henderson believes fast data is “about seizing opportunities.” Sir Francis Bacon once stated, “A wise man will make more opportunities than he finds.” A wise company will use fast data to make opportunities at a pace that matches a fast world.
Footnotes
[1] Staff, “Chapter 1. What Is Fast Data?” O’Reilly.
[2] Alissa Lorentz, “Big Data, Fast Data, Smart Data,” Wired, April 2013.
[3] Ed Anuff, “Fast Data,” CIO, 12 April 2021.
[4] Aiden Mathieu, “Big Data vs. Fast Data: Breaking the Mold of Database Thinking,” RT Insights, 24 August 2021.
[5] Mark Henderson, “Why The Quest For Fast Data is the 21st Century Gold Rush,” EV Cargo Technology Blog, 30 March 2021.
