In a song written in 1955 by Pete Seeger and ‎Joe Hickerson, the question is asked, “Where have all the flowers gone?” The 2021 version of that plaintive tune could be: “Where have all the workers gone?” The Economist reports, “As rich countries loosen lockdowns, an economic puzzle is emerging. Businesses are voicing ever-louder concerns about labor shortages, even as millions of people remain out of work. In America a surge of spending is creating job openings, but few people seem willing to fill them. The number of vacancies, at over 8m, has never been so high.”[1] The magazine goes on to note, “Some see news of worker shortages as welcome. If human labor is still in demand, then perhaps predictions of job-killing robots were wrong.” The irony is that the inability to fill job vacancies could force companies to look to automation to fill jobs they would have otherwise filled with humans; thus, making predictions about job-killing robots a self-fulfilling prophecy. The reasons so many jobs are unfilled are myriad — one of which is a lack of skilled workers. Addressing the skills gap is important; however, that’s a subject for a different article. In this article, I want to discuss how cognitive technologies (aka artificial intelligence (AI)) can help organizations ensure essential tasks are being accomplished.

Demand is High for Artificial Intelligence Experts

Journalist John McCormick reports, “Demand for artificial intelligence experts is surging as the economy opens up and companies expand their use of the technology to enhance the customer experience, increase sales and improve operations. There were 37,000 AI job postings in the first quarter of this year, up more than 45% from the fourth quarter of 2020, according to IT trade group CompTIA, which analyzes employer job-posting data. Among the most sought-after AI professionals: data scientists and machine-learning developers and engineers.”[2] Understanding how to leverage machine learning (ML) is, by far, the most important skill AI professionals need in today’s business environment. Eric Siegel (@predictanalytic), a former computer science professor at Columbia University, explains, “A.I. is a big fat lie. Artificial intelligence is a fraudulent hoax — or in the best cases it’s a hyped-up buzzword that confuses and deceives. The much better, precise term would instead usually be machine learning — which is genuinely powerful and everyone oughta be excited about it.”[3] Every job associated with machine learning is inextricably linked to data.

Matthew Mayo, Editor of KDnuggets, believes there is also a lot of confusion about differences in AI professionals skillsets. He explains there are five data career archetypes: data architect; data engineer; data analyst; machine learning engineer; and data scientist.[4]

Data Architect. According to Mayo, “The data architect is concerned with managing data and engineering the infrastructure which stores and supports this data. There is generally little to no data analysis needing to take place in such a role (beyond data store analysis for performance tuning), and the use of languages such as Python and R is likely not necessary. An expert level knowledge of relational and non-relational databases, however, will undoubtedly be necessary for such a role.” He notes that other names for data architects include, data administrator, spark administrator, big data administrator, database engineer, and data manager.

Data Engineer. Mayo writes, “The data engineer focuses on engineering and managing the infrastructure which supports the data and data pipelines.” If you are unsure what data infrastructure is, Mayo explains, “It’s the collection of software and storage solutions that allow for the retrieval of data from a data store, the processing of data in some specified manner (or series of manners), the movement of data between tasks (as well as the tasks themselves), as data is on its way to analysis or modeling, as well as the tasks which come after this analysis or modeling. It’s the pathway that the data takes as it moves along its journey from its home to its ultimate location of usefulness, and beyond.” Other titles for a data engineer include big data engineer and data infrastructure engineer.

Data Analyst. “The data analyst,” Mayo explains, “focuses on the analysis and presentation of data. … This includes the preparation of reporting, dashboards, KPIs, business performance metrics, as well as encompassing anything referred to as ‘business intelligence.’ The role often requires interaction with (or querying of) databases, both relational and non-relational, as well as with other data frameworks.” Related job titles include business analyst and business intelligence (BI) analyst.

Machine Learning Engineer. According to Mayo, “The machine learning engineer develops and optimizes machine learning algorithms, and implements and manages (near) production level machine learning models.” He continues, “Machine learning engineers are those crafting and using the predictive and correlative tools used to leverage data. … Statistics and programming are some of the biggest assets to the machine learning researcher and practitioner. Maths such as linear algebra and intermediate calculus are useful for those employing more complex algorithms and techniques, such as neural networks, or working in computer vision, while an understanding of learning theory is also useful. And, of course, a machine learning engineer must have an understanding of the inner workings of an arsenal of machine learning algorithms (the more algorithms the better, and the deeper the understanding the better!).” Related job titles include machine learning scientist, machine learning practitioner.

Data Scientist. Mayo explains, “The data scientist is concerned primarily with the data, the insights which can be extracted from it, and the stories that it can tell.” While that may sound similar to other jobs discussed above, Mayo writes, “The data architect and data engineer are concerned with the infrastructure which houses and transports the data. The data analyst is concerned with pulling descriptive facts from the data as it exists. The machine learning engineer is concerned with advancing and employing the tools available to leverage data for predictive and correlative capabilities, as well as making the resulting models widely-available. The data scientist is concerned primarily with the data, the insights which can be extracted from it, and the stories that it can tell, regardless of what technologies or tools are needed to carry out that task.” He adds, “Remember that these are simply archetypes of five major data profession roles, and these can vary between organizations.” Mayo provides the following flowchart as a useful tool to help navigate the AI professional landscape.

Because the Digital Age demands big data and AI expertise, these jobs will remain in demand for the foreseeable future — and, because of skilled personnel shortages, available jobs may remain vacant.

Cognitive Technologies can Help Alleviate the AI Expert Shortage

Adam Carrigan (@AdamMCarrigan), Co-founder and COO of MindsDB, writes, “Data scientists and software engineers experienced in ML are in great demand throughout the industry and thus also command a very high price. The vast majority of companies cannot budget for these technical resources.”[5] Because personnel shortages in the analytics field are expected to continue, some organizations are turning to cognitive systems with embedded analytics to help fill the gap. Carrigan explains, “[Organizations] can glean actionable information out of their corporate data stores by merely embedding an AI layer into their databases. Notably, this all gets accomplished without depending on expensive technical personnel.”[5] The traditional approach to analytics has been to assemble a team of three experts:

• A business domain expert — the customer of the analysis who can help explain the drivers behind data anomalies and outliers.
• A statistical expert — to help formulate the correct statistical studies, the business expert knows what they want to study, and what terms to use to help formulate the data in a way that will detect the desired phenomena.
• A data expert — the data expert understands where and how to pull the data from across multiple databases or data feeds.

Having three experts involved dramatically lengthens the time required to analyze, tune, re-analyze, and interpret the results. Cognitive technologies empower the business expert by automating the statistical expert’s and data expert’s knowledge and functions, so the ideation cycle can be dramatically shortened and more insights can be auto-generated. Although this may sound like cognitive technologies are taking over all high-skilled jobs, the fact is they augment the human workforce. Carrigan explains, “As opposed to machine learning replacing employees, the technology actually enhances the human decision-making process. By giving business analysts enhanced capabilities to derive insights from corporate data, smaller companies can make more effective choices no matter the scenario.” Because they democratize analytics through the use of natural language programming, cognitive solutions democratize insights throughout an organization. As Carrigan explains, “This approach lowers the expertise level needed for working with ML models. Actionable insights from a company’s data store are available to the people that need it, without requiring a data scientist or software engineer.” Cognitive technologies can’t replace all the data experts an organization requires; nevertheless, they can help reduce the skills gap.

Footnotes
[1] Staff, “What to do about a labour crunch,” The Economist, 22 May 2021.
[2] John McCormick, “AI Experts Even More in Demand as Economy Opens Up,” The Wall Street Journal, 21 April 2021.
[3] Eric Siegel, “Why A.I. is a big fat lie,” Big Think, 23 January 2019.
[4] Matthew Mayo, “Data Scientist, Data Engineer & Other Data Careers, Explained,” KDNuggets, 1 May 2021.
[5] Adam Carrigan, “Democratizing Machine Learning to Empower the Modern Business,” insideBIGDATA, 17 May 2021.