“We’re entering a new age of artificial intelligence,” insists Daniela Hernandez (@danielaphd). “Drawing on the work of a clever cadre of academic researchers, the biggest names in tech — including Google, Facebook, Microsoft, and Apple — are embracing a more powerful form of AI known as ‘deep learning,’ using it to improve everything from speech recognition and language translation to computer vision, the ability to identify images without human help.” [“Microsoft Challenges Google’s Artificial Brain With ‘Project Adam’,” Wired, 14 July 2014] Deep learning is not a new concept. I first started writing about it a couple of years ago. In my first post on the subject, I cited an article by John Markoff (@markoff), who wrote, “Using an artificial intelligence technique inspired by theories about how the brain recognizes patterns, technology companies are reporting startling gains in fields as diverse as computer vision, speech recognition and the identification of promising new molecules for designing drugs.” [“Scientists See Promise in Deep-Learning Programs,” New York Times, 23 November 2012] He continued:
“The technology, called deep learning, has already been put to use in services like Apple’s Siri virtual personal assistant, which is based on Nuance Communications’ speech recognition service, and in Google’s Street View, which uses machine vision to identify specific addresses. But what is new in recent months is the growing speed and accuracy of deep-learning programs, often called artificial neural networks or just ‘neural nets’ for their resemblance to the neural connections in the brain. ‘There has been a number of stunning new results with deep-learning methods,’ said Yann LeCun, a computer scientist at New York University who did pioneering research in handwriting recognition at Bell Laboratories. ‘The kind of jump we are seeing in the accuracy of these systems is very rare indeed.'”
Hernandez indicates that Microsoft has made another jump in accuracy. She writes, “Microsoft’s research arm says it has achieved new records with a deep learning system it calls Adam. … According to Microsoft, Adam is twice as adept as previous systems at recognizing images — including, say, photos of a particular breed of dog or a type of vegetation — while using 30 times fewer machines (see video below). ‘Adam is an exploration on how you build the biggest brain,’ says Peter Lee, the head of Microsoft Research.”
Hernandez reports, “The trick is that the system better optimizes the way its machines handle data and fine-tunes the communications between them. It’s the brainchild of a Microsoft researcher named Trishul Chilimbi, someone who’s trained not in the very academic world of artificial intelligence, but in the art of massive computing systems.” She continues:
“Like similar deep learning systems, Adam runs across an array of standard computer servers, in this case machines offered up by Microsoft’s Azure cloud computing service. Deep learning aims to more closely mimic the way the brain works by creating neural networks — systems that behave, at least in some respects, like the networks of neurons in your brain — and typically, these neural nets require a large number of servers. The difference is that Adam makes use of a technique called asynchrony. As computing systems get more and more complex, it gets more and more difficult to get their various parts to trade information with each other, but asynchrony can mitigate this problem. Basically, asynchrony is about splitting a system into parts that can pretty much run independently of each other, before sharing their calculations and merging them into a whole. The trouble is that although this can work well with smartphones and laptops — where calculations are spread across many different computer chips — it hasn’t been that successful with systems that run across many different servers, as neural nets do. But various researchers and tech companies — including Google — have been playing around with large asynchronous systems for years now, and inside Adam, Microsoft is taking advantage of this work using a technology developed at the University of Wisconsin called, of all things, ‘HOGWILD!‘”
According to Hernandez, HOGWILD! allows different chips to write to the same memory location, even allowing them to overwrite each other. Although that can create problems, researchers at the University of Wisconsin have demonstrated that, under the right circumstances, it actually speeds up learning in a single machine. “Adam then takes this idea one step further,” reports Hernandez, “applying the asynchrony of HOGWILD! to an entire network of machines.” She continues:
“Although neural nets are extremely dense and the risk of data collision is high, this approach works because the collisions tend to result in the same calculation that would have been reached if the system had carefully avoided any collisions. This is because, when each machine updates the master server, the update tends to be additive. One machine, for instance, will decide to add a ‘1’ to a preexisting value of ‘5,’ while another decides to add a ‘3.’ Rather than carefully controlling which machine updates the value first, the system just lets each of them update it whenever they can. Whichever machine goes first, the end result is still ‘9.’ Microsoft says this setup can actually help its neural networks more quickly and more accurately train themselves to understand things like images. ‘It’s an aggressive strategy, but I do see why this could save a lot of computation,’ says Andrew Ng, a noted deep-learning expert who now works for Chinese search giant Baidu. ‘It’s interesting that this turns out to be a good idea.'”
In another post about deep learning, I quoted Ashlee Vance, who calls deep learning “a funky part of computer science seen as key to building truly intelligent machines.” [“The Race to Buy the Human Brains Behind Deep Learning Machines,” Bloomberg BusinessWeek, 27 January 2014] If Microsoft’s Lee is correct, machine learning won’t seem so funky in the years ahead. Hernandez explains:
“Lee believes Adam could be part of what he calls an ‘ultimate machine intelligence,’ something that could function in ways that are closer to how we humans handle different types of modalities — like speech, vision, and text — all at once. The road to that kind of technology is long — people have been working towards it since the 50s — but we’re certainly getting closer.”
IBM certainly agrees with Lee. The company’s website asserts, “Machine learning will enable cognitive systems to learn, reason and engage with us in a more natural and personalized way. These systems will get smarter and more customized through interactions with data, devices and people.” John Weathington (@johnweathington) writes, “Think of deep learning as cutting-edge AI that generally represents an evolution over primitive neural networks. A key distinction between traditional machine learning and deep learning is the amount of supervision and human intervention the AI system requires. Traditional machine learning techniques, including classic neural networks, need to be supervised by humans so they can learn. Deep learning is an attempt to have the system learn on its own, without human intervention.” You can count me among those who believe that machine learning is going to have a significant impact in the years ahead. When a computer system is capable of learning on its own, it does so 24 hours a day without having to rest. That’s the reason that systems like Adam put us on the eve of a new era of computing.
