If you buy something using links in our stories, we may earn a commission. Learn more.
It's good to be Yann LeCun.
Mark Zuckerberg recently handpicked the longtime NYU professor to run Facebook’s new artificial intelligence lab. The IEEE Computational Intelligence Society just gave him its prestigious Neural Network Pioneer Award, in honor of his work on deep learning, a form of artificial intelligence meant to more closely mimic the human brain. And, perhaps most of all, deep learning has suddenly spread across the commercial tech world, from Google to Microsoft to Baidu to Twitter, just a few years after most AI researchers openly scoffed at it.
All of these tech companies are now exploring a particular type of deep learning called convolutional neural networks, aiming to build web services that can do things like automatically understand natural language and recognize images. At Google, "convnets" power the voice recognition system available on Android phones. At China's Baidu, they drive a new visual search engine. This kind of deep learning has many fathers, but its success should resonate with LeCun more than anyone. "Convolutional neural nets for vision---that's what he pushed more than anybody else," says Microsoft's Leon Bottou, one of LeCun's earliest collaborators.
He pushed it in the face of enormous skepticism. In the '80s, when LeCun first got behind the idea of convnets---an approximation of the networks of neurons in the brain---the powerful computers and enormous data sets needed to make them work just didn't exist. The very notion of a neural network had fallen into disrepute after it failed to deliver on the promises of scientists who first dreamed of artificial intelligence at the dawn of the computer age. It was hard to publish anything related to neural nets in the major academic journals, and this would remain the case in the '90s and on into the aughts.
But LeCun persisted. "He kind of carried the torch through the dark ages," says Geoffrey Hinton, the central figure in the deep learning movement. And eventually, computer power caught up with the remarkable technology.
More than two decades before joining Facebook, LeCun worked at Bell Labs, perhaps the most famous of computer research labs, the birthplace of the transistor, the Unix operating system, and the C programming language. During his stint there, the French researcher developed a system that could recognize written digits. He called it LeNet.
Automatically reading bank checks, it marked the first time convolutional neural nets were applied to practical problems. "Convolutional nets were little toys, and Yann changed them into things that worked on a large scale," says Bottou. And thanks to Larry Jackel, the chief of LeCun's department who recorded a demo in 1993, you can see the technology in action, with a giddy, baby-faced LeCun test-driving the thing (video below).
Some of the concepts baked into LeNet date back to work LeCun did in France at Bottou's college pad and later at a lab run by Hinton, and he was first inspired by two research papers from by Kunihiko Fukushima, another neural-net great who, in the '70s and '80s, had invented what were called the Cognitron and Neocognitron. These early neural nets could learn to pick out patterns in data, on their own, without much human prompting. But they were rather complicated, and researchers couldn't quite figure out how to make them work well. "What was missing was the supervised learning algorithm," says LeCun. "What we call back prop now."
"Back propagation" is a clever way to minimize error. To understand it, you also have to understand how convolutional neural nets work.
Like other flavors of neural nets, convolutional networks are software creations organized into interconnected layers, much like the visual cortex, the part of the brain that process visual information. What makes them different is that they can reuse the same filters at multiple locations in an image. That means that once the network has learned to recognize, say a face, at one location, it can also automatically find faces in others. (The same principle holds for sound waves and written words.)
This allows artificial neural nets to be trained quickly, and because they have a "small memory footprint, you don't need to separately store a filter for each location in the image...[making] them well-suited to building much more scalable deep nets," says Baidu's Andrew Ng. It's also what makes them so adept at recognizing patterns.
When receiving an image---the input---the network translates it into arrays of numbers that represent features, and the "neurons" in each layer of the network are tuned to recognize certain pattens in the numbers. Low-level neurons recognize things like edges or basic shapes, while neurons in higher layers can "see" objects---say, a dog or a person. Each layer communicates with the one above it, and as information travels up the network, some averaging takes place. At the end, the network comes up with an output---a guess at what's in the image.
If the network makes a mistake, engineers can fine-tune the connections between layers to get the right answer. But neural nets work better if they can do this fine tuning on their own. That's where the back propagation algorithm comes in.
Back prop is all about computing the error and using that value to update the strength---or weight---given to each of the layers in a neural net. Hinton, with David Rumelhart and Ronald Williams, came up with a version of back prop that calculated the error for multiple inputs at once and then took the average. That value was then back-propagated through the network, from the output to the input layers. In a paper published in the journal Nature in 1986, they showed that this approach improved learning.
Around the same time, LeCun was busy devising his own recipe for back prop in Paris, based on research that dated back to the 1950s but that others had more or less failed to apply to real-world problems. Instead of averaging, LeCun's version calculated the error for every single example. It was more noisy, but it worked well---and more quickly. "A lot of people didn't believe that you could do that," recalls Bottou.
According to Bottou, the method was the result of the weak machines they were using. "The computers we had in France were less endowed." They had to come up with a hack to calculate error quickly while using as little computational power as possible. But what seemed like a "fudge" at the time---to borrow Hinton's term---turned out to be spot on. Today, it's called stochastic gradient descent, and like convolutional neural nets as a whole, it's a staple of the artificial intelligence toolbox.
LeCun's LeNets would be widely licensed and used in ATMs and banks across the world to read what was written on checks. But skepticism remained. "Somehow, that wasn’t enough to convince the computer vision community that convolutional neural nets were worthy," says LeCun. Part of it was that, although they were powerful, nobody knew why they so powerful. The inner-workings of this technology were a mystery.
There were many critics. Vladimir Vapnik, a mathematician and the father of the support vector machine, one of the most widely used AI models, was among them.
One March afternoon in 1995, Vapnik and Larry Jackel---who'd recruited him and LeCun to Bell Labs---made a bet. Jackel wagered that, by the year 2000, we'd have a have a handle on how deep artificial neural nets worked. Vapnik disagreed. He also thought that by 2005, "no one in his right mind will use neural nets that are essentially like those used in 1995." Fancy dinners were at stake, so they put the bet on paper and signed it---in front of witnesses. LeCun served as the third official signatory, Bottou as an unofficial observer.
Vapnik would win the first half of the wager. In 2000, the inner workings of neural nets were still largely shrouded in mystery, and even now, researchers can’t pinpoint mathematically exactly what makes them work well. But, the second victory belonged to Jackel---and, more importantly, to LeCun. By 2005, deep neural nets were still being used in ATMs and banks, and they were very much rooted in work dating back to LeCun's work in the mid-1980s and early '90s.
"It's rarely the case where a technology that has been around for 20, 25 years---basically unchanged---turns out to be the best," says LeCun. "The speed at which people have embraced it is nothing short of amazing. I've never seen anything like this before."
But this is just a start. The deep learning community---LeCun included---are working to improve the technology. Today's most widely used convolutional neural nets rely almost exclusively on supervised learning. Basically, that means that if you want it to learn how to identify a particular object, you have to label more than a few examples. Yet unsupervised learning---or learning from unlabeled data---is closer to how real brains learn, and some deep learning research is exploring this area.
"How this is done in the brain is pretty much completely unknown. Synapses adjust themselves, but we don't have a clear picture for what the algorithm of the cortex is," says LeCun. "We know the ultimate answer is unsupervised learning, but we don't have the answer yet."
It's also unlikely that back prop---what neural-net expert Yoshua Bengio calls "the workhorse of most deep learning systems"---mirrors the human brain, so researchers are developing alternatives. Plus, the way that convolutional nets "pool," or average, data doesn't sit well with some, so there are efforts to improve this as well. "It loses information," says Hinton.
Say you're looking at faces. The system learns to recognize facial features, like eyes and lips. Based on these, it's good at identifying that there is a face in an image, but much less tuned to picking out differences between faces. If you want to know the precise location of the eyes in a face, for example, it can't do that very well. As tech companies and governments want to build more detailed digital dossiers of their customers and citizens, these kinds of limitations will become, well, limiting.
The general ideas behind LeCun's convnets may not be perfect, but they're the state of the art. "He turned out to be completely right," says Hinton, before adding with a quick laugh, "except for the pooling."
LeCun's work extends well beyond neural nets. In the late '90s, he had a hand in building a seminal image compression system. The idea was to scan documents and then put these up on the web for all to see. The technology never quite panned out for LeCun, but the concepts behind the technology impressed a young Larry Page, the co-founder of Google, who heard a talk LeCun gave at Stanford in 1998 when he was still a graduate student.
LeCun has also worked with robotics and AI hardware. He recently founded NYU's Center for Data Science. And he has mentored a new generation of AI researchers, including Clement Fabaret, whose image-indexing company Madbits was recently acquired by Twitter. In his spare time, he builds model planes.
With that pedigree, it's no surprise that Zuckerberg asked LeCun to help the company make sense of all its data. After all, the social network has been busy acquiring companies---like the virtual-reality company Oculus, solar-powered drone maker Ascenta, and WhatsApp---whose products could benefit from the type of AI LeCun pioneered.
LeCun is actively looking to hire more AI talent at the company---Rob Fergus, with whom he collaborated with at NYU, is already part of his team at Facebook---and he's been tasked with turning the AI lab into a world-class research outfit, a place to compete with Google, Microsoft, IBM and Baidu, to be sure, but also an operation that harkens back to Bell Labs, which served as a breeding ground for innovation and the birthplace of many of the technologies we take for granted today, including deep learning.
A scientist at heart, he also wants to keep developing his own ideas. "I'm not giving up on research, on the contrary. I'm opening up a new venue for new research to take place," he says. "It's a much more exciting situation to be in---when the contribution you make is valued."
*Update 8/15/2014: The story has been updated to clarify that the IEEE Computational Intelligence Society awarded LeCun the Neural Network Pioneer Award. *
