There are a number of trends associated with Big Data and cloud computing that are clearly beginning to emerge. In this post, I’d like to discuss a few of them beginning with job growth.

Job Growth

Joe McKendrick writes, “A new study commissioned by SAP and conducted by Sand Hill Group speculates that cloud computing — fueled by mobile computing, social networking and big data — may generate as many or more opportunities in the coming years than the Internet itself did in its early years.” [“Mobile, social and big data drive cloud computing boom: studies,” Service Oriented, 22 March 2012] Although that may sound like a bold statement, companies looking for employees to work with Big Data and cloud computing services know how difficult those employees are to find. McKendrick continues:

“The study’s authors said cloud computing is already generating a sizable number of jobs in the US today, and based on numerous trends and indicators, has the future potential to create very large business opportunities and hundreds of thousands of new jobs. Of course, as anyone who was around during the dot-com craze of the 1990s knows, we’ve been down this road before with over-the-top industry projections. There’s no question that cloud is the hype of the day. Still, the cloud represents a shift in business technology resources that presents both risk and great opportunity for vendors and end-users alike.”

In past posts, I have detailed some of those risks and opportunities. Like most analysts, I conclude that the benefits of Big Data analytics and cloud computing (including software-as-a-service (SaaS) applications) are much greater than the risks involved. One of the reasons that the dot.com era was characterized by irrational exuberance was that people were excited about the potential of increased connectivity. The problem was that too many startups began with no business plan and no real understanding of where all the connectivity was going to lead. I don’t see that happening this time around because most Big Data/Cloud Computing activities are business oriented. Businesses were stung by vendors during the dot.com era that over-promised and under-delivered. Businesses are being much more cautious this time around. McKendrick continues:

“Consider potential job growth, both within IT and the business. For example, the SAP study relates, 11 cloud computing companies added 80,000 jobs in the United States in 2010, and the employment growth rate at these organizations was almost five times than that of the high-tech sector overall. The report cites a previous study out of Bank of America Merrill Lynch Global, which calculated the total number of employees at 11 cloud companies (Amazon, Google, Netflix, OpenTable, Salesforce, Taleo, SuccessFactors, RightNow, Intuit, NetSuite, and Concur) in January 2010 and January 2011. The total number of employees grew 27% during that one-year period, which was an additional 80,000 new jobs. The employee growth at these 11 cloud companies was almost five times the employee growth rate for the high-tech services sector overall, which grew 5.9%, to add about 17,500 jobs during a similar period.”

Those are pretty impressive numbers; especially considering that this job growth took place during economically challenging times. McKendrick reports, “Even more bullish numbers come from new research conducted by IDC and sponsored by Microsoft Corp., which also looked at the economic benefits of cloud computing in the years ahead. Cloud computing will potentially generate at least 14 million new jobs across the globe within the next three years. Moreover, these new jobs may likely be in many areas outside of IT.” The areas may be “outside of IT,” but they are areas that affect any good Sales and Operations Planning (S&OP) process, namely: “areas such as marketing, sales, finance and administration, production, and service.” McKendrick states, “This does not even consider all the new types of jobs that may be created as a result of cloud, perhaps with titles such as ‘virtual resources administrator’ or ‘customer network facilitator.'”

Increasing Revenue and Savings

Another trend highlighted by McKendrick is the increasing revenue that is going to be created in the cloud computing sector. Revenue is going to go up because cloud computing is going to help keeps costs down. He writes:

“IDC’s research also predicts revenues from cloud innovation could reach $1.1 trillion per year within the next 36 months. The analyst firm estimates that last year alone, IT cloud services helped organizations of all sizes and all vertical sectors around the world generate more than $400 billion in revenue. … SAP-Sand Hill’s report also examined the economic impact on consumers — companies buying cloud services. Cloud computing could save US businesses as much as $625 billion over five years, the study’s authors predict.”

McKendrick reports that the IDC study asserts that “three industry megatrends are propelling the growth of cloud services and employment. They are:

• “The boom in mobile computing devices such as smartphones and tablets: ‘Mobile apps will drive massive demand for cloud services on the back end, such as app stores, databases, and storage. The recent success of tablet devices will further expand the demand for cloud services as these mobile devices give users greater access to information.’
• “Social networking: ‘Such massive scalability and elasticity would not be possible without cloud computing technologies to drive these sites.’
• “Big Data: ‘Cloud infrastructure and platforms will play a huge role in accessing, processing, and analyzing such massive amounts of data. This is where cloud-based systems shine.'”

Pattern Recognition

As McKendrick notes, cloud computing shines in the area of Big Data analytics. Big Data can be mined for patterns and insights that provide real value. An interesting trend that is emerging is the search for even larger patterns than those found within the Big Data itself (i.e., patterns that can be applied to sets of data other than the set from which the pattern was detected). As Quentin Hardy writes, “It’s not just about Big Data. For the big players in enterprise technology algorithms, it’s about finding big patterns beyond the data itself.” [“I.B.M.: Big Data, Bigger Patterns,” New York Times, 15 February 2012] Hardy explains:

“The explosion of online life and cheap computer hardware have made it possible to store immense amounts of unstructured information, like e-mails or Internet clickstreams, then search the stored information to find some trend that can be exploited. The real trick is to do this cost-effectively. Companies doing this at a large scale look for similarities between one field and another, hoping for a common means of analysis. When it comes to algorithms, ‘if I can do a power grid, I can do water supply,’ said Steve Mills, I.B.M.’s senior vice president for software and systems. Even traffic, which like water and electricity has value when it flows effectively, can reuse some of the same algorithms.”

Mills calls this reutilization of algorithms, “Leveraging the cost structure of new mathematics.” What I like about this trend is that encourages cross-disciplinary collaboration. Hardy explains:

“That kind of cross-pollination is reminiscent of the way Wall Street, starting in the 1990s, hired astrophysicists and theoretical mathematicians to design arcane financial products. Now the cost of computing has come down so much that it is useful to bring such talent to other industries. I.B.M., Mr. Mills said, is now the largest employer of Ph.D. mathematicians in the world, bringing their talents to things like oil exploration and medicine. ‘On the side we’re doing astrophysics, genomics, proteomics,’ he said. In the last five years, I.B.M. has spent some $14 billion purchasing analytics companies, in the service of its Big Data initiative. ‘We look for adjacencies’ between one business and another, said Mr. Mills. ‘If we can’t get an adjacency, we’ll never get a return.’ The trend of looking for commonalities and overlapping interests is emerging in many parts of both academia and business.”

An exciting side benefit of discovering “adjacencies” is the fact that many of the best innovations occur at the intersections (or along the borders) of disciplines. To put it another way, discovering adjacencies could generate a spike in innovation as well as an increase in profits.

Visualization

Mining data for insights is only the front half of the challenge and, by itself, insufficient to add value to a company. Those insights (or other analytical products) need to be presented to decision makers in a way that is both useful and informative. Insights that aren’t used are worthless. Jeff Kelly puts it this way, “As the infrastructure layer continues to mature, vendors and increasingly enterprises are turning their attention to the real value proposition of Big Data – namely, deriving actionable insight via Big Data Analytics and Visualization.” [“Hadoop, Big Data Focus Shifting To Analytics and Visualization,” Wikibon Blog, 26 October 2011] He continues:

“That’s not to say Big Data infrastructure isn’t important or doesn’t need improving – clearly tasks like writing and managing complex Map Reduce jobs and networking racks of Hadoop nodes still need simplifying – but that it has reached a maturity level where it is now practical for may enterprises to shift at least some of their focus to analyzing and making use of the data in addition to processing and storing it. … To reiterate, there’s still plenty of work to do on the infrastructure layer of Hadoop and other Big Data approaches. … But the focus of the Big Data industry is – and should be – moving to include analytics and visualization. This is especially important for enterprises. Hadoop and other Big Data approaches, while still somewhat novel, should not be treated as some off-to-the-side science project. Enterprises should apply Big Data approaches like Hadoop only when they’ve identified areas where Big Data will help soothe a significant pain-point and/or bring real business value. And this requires analytic/visualization platforms and applications, tools that provide insights from Big Data that facilitate innovation such as identifying new market opportunities or helping create new products.”

Conclusions

As I noted above, the difference between the dot.com era and Big Data/Cloud Computing era is that companies providing hosting, application, and analytic services are going to achieve success because they support traditional business objectives rather than ignore them like some companies did during the dot.com era. It should become clearer each day that Big Data/Cloud Computing era is not a passing fancy; rather it’s the next big thing that is going to change how businesses operate in the decades ahead.