The field of deep learning has been suffering from what you might call a Big Data fallacy, the belief that more and more data is always a good thing.
It may be time to focus on quality rather than just quantity.
“There’s a very fundamental problem that a lot of AI faces,” said Andrew Ng, founder and CEO of Landing AI, a startup working to perfect the technology for industrial uses, in an interview with ZDNet this week.
“A lot of AI is focused on maximizing the number of calories, which works up to a certain point,” he said.
“And sometimes you do have a lot of data, but when you have a small data set, it’s more the quality of the data rather than the sheer volume.”
Ng, who ran development of large-scale AI systems including speech recognition technologies when he was founder of the Google Brain unit at Google a decade ago, now has the mandate to build AI technology to be used by corporate clients for practical purposes, such as factory-floor inspection.
Landing AI has received $57 million in financing to date in one round in November of last year, from McRock Capital Insight Partners, Taiwania Capital, Canada Pension Plan Investment Board, Intel Capital, Samsung Catalyst Fund, Far Eastern Group’s DRIVE Catalyst, Walsin Lihwa, and AI Fund.
Ng is also co-founder of Coursera, the online curriculum company, and an adjunct professor of computer science at Stanford University.
The Big Data fallacy has its origins in the technical need deep learning forms of AI have for large samples of a given domain of inquiry. Deep learning approaches have focused on getting enough data points to avoid what’s called “over-fitting,” where increasingly large neural networks would simply memorize training data.
If deep learning memorizes training data, it fails to generalize about the nature of that data, which is commonly the key to a neural network being useful for making predictions.
But a budget of millions or even billions of data is not feasible in some contexts such as a manufacturing inspection system, where a single defect out of a million identical parts may be the only piece of data about manufacturing flaws.
“I’ve built AI systems of hundreds of millions of images,” says Ng, who was founder of Google’s Google Brain group, and chief scientist at Baidu. “Those techniques don’t really work when you have only 50 images,” he said.
“I’ve built AI systems of hundreds of millions of images” at Google and as chief scientist at Baidu. “Those techniques don’t really work when you have only 50 images,” he said.
Ng said Landing AI has been able to develop useful industrial models for clients with a relative handful of data samples.
“Rather than Big Data, we’ve had to focus on good data,” said Ng.
In contrast to the typical thinking about Big Data, using such a small sample size, of dozens rather than millions of examples, can be workable.
“I’ve been surprised repeatedly at how well we can get a neural network to do with just 50 images if you make sure it has 50 really good images,” said Ng. “The tools we have been innovating at Landing AI are, you only have 50 images, so how do you label it to drive the best possible performance out of on only 50 images.”
“I feel like we’ve cracked the recipe” for using deep learning in manufacturing, he said.
In a sense, the Big Data fallacy could be thought of as the tail wagging the dog. To avoid over-fitting, all attention in deep learning was about how to make the deep learning model, the program’s structure, operate optimally.
Now, Ng is arguing for a greater focus on what data points are most important, and making the model fit that.
“The recipe that people like me and many of my friends have developed for getting AI to work in consumer software companies doesn’t work for manufacturing companies and many other companies,” said Ng.
“The fundamental problem is, if you have a hundred billion or million users, you can build a monolithic AI system; manufacturing is something different.”
“He is a guru,” says Ng of computer vision and industrial automation pioneer David L. Dechow.
The need is many, many architectures, many different neural nets to suit the data.
“The challenge that the field of AI faces is how can we help manufacturing build not one or a dozen AI models, but to help 10,000 different manufacturers to build 10,000 different AI models because every factory, every component needs its own model.”
The challenge for a startup, of course, is “how to do that without me hiring 10,000 engineers.” In Landing AI’s early days — the company was founded in 2017 — Ng said he had been “naive.”
“In the early days, we were doing a lot of consulting work, trying to do a lot of the customization ourselves, and it just didn’t scale.” Ng believes a lot of AI startups run into that problem, becoming consulting firms.
The company pivoted to making tools to make all those customers do the customization. The same, he said, has to be the approach in industries such as healthcare.
“Engineering the data is really challenging,” observed Ng. “I think there are experienced Ai engineers that have been doing intuitively for a long time, but building tools that makes it fast and easy for many people to do it successfully, that was a very deep technical challenge.”
Ng has been spreading the gospel in talks about the new emphasis on quality, and on how to get the data ready for machine learning approaches.
He calls his view “data-centric AI.”
More on Ng’s thinking can be seen in a video interview back in March.
The company on Wednesday announced it has hired David L. Dechow, a pioneer in the field of computer vision technology, who was previously principal vision system architect for startup Integro Technologies.
Much of the machine learning work that has been done in industry for many years has been about putting cameras on the shop floor to observe the manufacturing line, noted Ng. Dechow has been a pioneer in that move of observation into the factory, he said.
“For many decades, even before the recent boom in computer vision, manufacturers have put cameras in factories,” said Ng. “Cameras are very good at measuring the exact length of a part, and detecting certain types of defects if you can code a rule to specify exactly what you’re looking for.”
With modern deep learning systems, the goal is to make much more flexible AI for defect detection without such rigid rules-based approaches.
“In the machine vision world, and industrial automation world, he is a guru,” said Ng of Dechow. “Generations of vision engineers today have been trained by David.
“If you could bring in one person from the machine vision field, it would be him,” said Ng.
This content was originally published here.