Original Source Here
My Top 5 Predictions for AI in 2022
DeepMind, Tesla, OpenAI, and more.
2021 has been an amazing year for artificial intelligence.
Ethics is at the center of AI research more than ever. We have a better understanding of the risks of harm language models entail — companies keep improving language models making them not just bigger but smarter and more efficient, multimodal systems are more common (e.g. Google’s MUM and OpenAI’s DALL·E), and real-world AI is taking leaps forward — and backward. All in all, AI has maintained or even accelerated the pace of progress we’ve seen throughout the last decade.
2022 will continue in the same direction. The AI community will bring new promising developments and impressive breakthroughs, some of which we can foresee. I’ve made a list of the top-5 most influential AI events and developments that will — or won’t — take place in 2022. (Other, more unpredictable milestones will for sure happen but those are for us to be surprised by.)
1. Exploring new ways of building language models
The ‘bigger is better’ trend will die out in 2022.
Language models are The Thing in AI right now. Computer vision was popular in the early years of the past decade but from 2017 onwards, language has become the focus of attention — and the source of profit — for the most advanced AI institutions and organizations. You’ll have a hard time finding a world-class AI company or startup that hasn’t claimed its share of the AI language market.
DeepMind, as arguably the top-1 AI company worldwide, has been surprisingly quiet these years. Not anymore.
A few weeks ago DeepMind published three papers on language models. The news was shyly reported — we’ve developed a tolerance to language models’ revelations as the media has been feeding them uninterruptedly for the last four years. Yet, science doesn’t happen in headlines but in experiments, and DeepMind results are anything but shy.
The first paper features Gopher, a 280-billion-parameter neural network that showed amazing performance, surpassing GPT-3, J1-Jumbo, and MT-NLG (previous top-3 models from which we have performance data) in 100 out of 124 tasks. Gopher is a dense large language model (LLM), built the same way as the others. Despite the improved performance, there’s barely any innovation in the system. It’s been trained the same way, works similarly albeit better, and also engages in toxic behaviors.
The third paper got my attention. RETRO (Retrieval-Enhanced Transformer) is a 7-billion-parameter model with power rivaling that of GPT-3 and J1-Jumbo (which are 25x larger). The model was trained with 10x less computing power than GPT-3 and uses a retrieval mechanic that allows it to access a large database in real-time, removing the need to keep all the knowledge in memory like previous models. (Incoming in-depth article about RETRO very soon!)
RETRO better resembles humans in this sense. We don’t keep all knowledge in our memory; we read books and search the internet to find new information. DeepMind bets on diverging from the trend of ever-larger language models. It’s unsustainable in the long term — AI carbon footprint keeps increasing, the high costs make the tech unaffordable except for a few multi-billion companies, and AI hardware will have a tough time training rapidly-changing models if the rate of growth remains constant in the coming years.
Novel techniques like RETRO’s knowledge retrieval are needed.
Another approach that’s producing promising results is sparse language models. Sparsity consists in using only a fraction of the models’ parameters to make the computations — the different parts or ‘experts’ are activated conditionally. This contrasts with dense models (GPT-3 and Gopher) in which all parameters contribute to processing all inputs. The Switch Transformer (1.7T), Wu Dao 2.0 (1.75T), or M6 (10T) are examples of sparse models.
We’ll also see AI chips take advantage of sparsity to keep up with language model developments. AI hardware startup Tenstorrent is a notable proponent of this approach. They design smarter chips that use their computing power more efficiently instead of making bigger, expensive chips.
We’re witnessing the last days of the 4-year old trend of making the models larger and larger. We may never see a 100 trillion parameter model. Bigger isn’t always better and both software and hardware companies are starting to notice. Novel ways of designing and building language models will be the primary trend in AI in 2022.
2. OpenAI will release GPT-4 — and we’ll be surprised
We’re not expecting what OpenAI has reserved for 2022.
I published an article a few months ago entitled “GPT-4 Will Have 100 Trillion Parameters — 500x the Size of GPT-3.” I no longer stand by that headline. Taking into account what I’ve said about novel approaches to language models, I’m confident OpenAI will also take a new direction and will stop making larger models just for the sake of it. GPT-3 was 100x the size of GPT-2, but GPT-4 will be around the same size as GPT-3 — although it won’t work the same way.
When I wrote the aforementioned article, I thought — as most people did — that OpenAI would continue the scaling trend from GPT, GPT-2, and GPT-3 to GPT-4. Cerebras Systems’ CEO, Andrew Feldman, in promotion of their latest AI chip, WSE-2, hinted that OpenAI would use their chip for the next version of the GPT family — a 100T-parameter model. But Sam Altman, OpenAI’s CEO denied the rumors. In a private Q&A (I won’t give details because Altman asked the assistants to not disclose the info) he said GPT-4 won’t have 100T parameters and will be an only-text model (no multimodality). He also said it’ll include novel features not yet seen in GPT-3 but didn’t specify which.
OpenAI researchers are apparently working on improving efficiency and power while keeping size constant (as DeepMind did with RETRO), shifting away completely from the scaling hypothesis. Retrieval techniques, conditional processing, and more efficient transformer architectures are some of the known alternative approaches to larger models, but there are many other things OpenAI could be experimenting with.
We’ll have to wait for GPT-4 to come out. It’ll be soon and it’ll be surprising.
3. Tesla won’t build an autonomous humanoid robot
Elon Musk is a showman, Tesla is his flagship, we’re his audience, and the world is his playground.
In 2021 Tesla AI Day Elon Musk promised, in his idiosyncratic quirky style, that Tesla will have “sometime next year” a prototype of an autonomous humanoid robot — Optimus, as they call it. They plan to transfer the (still imperfect) self-driving technology into a humanoid form. Musk thinks Tesla is well-suited to work on this project because of its strong focus on autonomy, supercomputing, and real-world AI.
Optimus’ main purpose is to “eliminate dangerous, repetitive, and boring tasks.” Musk said people will be able to command it using natural language. You could say: “Pick up that bolt, and attach it to the car with that wrench,” or “go to the store and get me the following groceries.” However, what Musk doesn’t realize — or maybe he does and simply uses Optimus as another publicity stunt — is that building a humanoid autonomous robot is way, way harder than building a self-driving car — a feat Tesla is still far from achieving.
Humans are multisensory because we’ve evolved in a multimodal world. Autonomous vehicles only have the sense of vision, which ironically blinds them from everything else — sounds, textures, pressure, temperature, odors, flavors…Humans integrate the distinct perceptual channels into a single representation of reality that we then use to navigate the world and make decisions. Optimus would need haptic and tactile sensors, proprioceptive abilities, and a precise inner representation of its body — none of which are present in self-driving technology.
On top of that, Optimus would need a system to select and filter the perceptual information that matters — because it’s useful for the task at hand because it’s life-threatening, or because of any other reason falling on the spectrum in between. Attention mechanisms are easy in language models but when it comes to combining and selecting percepts and deciding which to attend, technology isn’t remotely close to human-level.
Once Optimus has decided what percepts matter, it’d need to process the information, plan the subsequent actions, decide which are urgent and which aren’t, and act taking into account the ever-changing nature of the world and itself. A self-driving car has only two goals: Go from A to B following the traffic rules and avoid everything in the way.
An autonomous robot would also need to understand the world around it as we do. If there are clouds in the sky, Optimus would need to infer possible rain. If it’s raining the floor may be too wet. If the floor is wet, Optimus could slide off and fall. This reasoning chain is immediate to us, but far from obvious to AI.
Multisensory perception, attention, planning, decision-making, and commonsense and causal reasoning have been out of reach for AI since its conception in the 1950s. We haven’t taken but tiny steps to imbuing them into robots.
Elon Musk isn’t going to make it possible in one year.
4. The ongoing fight against bias in language AI
AI ethics has become a hot area in the last few years. In 2022, we’ll see companies getting more conscientious about it, even fostering efforts over profit — although probably because they’ve calculated that in the end, it’s worth it economically.
With language models getting larger and more powerful, they’ve also increased in harm potential. Companies have implemented techniques to reduce model bias and toxicity inherited from the datasets. Some, like curating and filtering the data are applied previous to training, whereas others like fine-tuning the models to improve their behavior, monitoring application releases, or defining and enforcing strict use-case guidelines are better implemented downstream.
But despite the efforts, no language model is free from falling victim to these imperfections.
Ethics experts have repeatedly reported that companies aren’t doing enough to mitigate risky behaviors. They criticize that companies treat profit as the objective to optimize and everything else — including effects over people — is contingent on that. Experts acknowledge that the problems of large language models are intrinsic and hardly removable. That’s why Emily M. Bender and Timnit Gebru, among others, recommend researchers to also consider other directions “beyond ever larger language models.”
DeepMind’s second paper (which I didn’t mention earlier) comprises a rigorous taxonomy of ethical and social risks of harm from language models. They lay out the structure of the risks, analyze their origin, define potential mitigation approaches, and recommend future directions of work. They recognize there’s a lot of work to do to mitigate the risks of harm to the point where openly using these models is safe enough for people of all types and kinds.
Large language models won’t stop developing anytime soon. Bias and toxicity won’t be 100% removed from these models anytime soon. In the years to come, we’ll witness both forces, those whose primary goal is to develop technology and those whose primary goal is to make the world a better place, push and pull towards a compromise solution. I don’t believe we’ll ever completely remove harmful behaviors from AI systems, but I do hope the AI community will eventually realize that LLMs have limited power and that building them larger and larger isn’t the way forward.
5. Self-driving cars still won’t be self-driving
No company in the world has built a self-driving car.
Tesla introduced the concepts of “full self-driving” and “Autopilot” to attract customers into thinking they were close to having cars that could drive themselves, which isn’t true. In 2020 German authorities took action against this behavior and banned Tesla from using this type of misleading language for marketing purposes.
Self-driving cars are far from being a reality. Tesla Autopilot helps cars navigate common scenarios but fails to solve edge cases in which humans would instantly know the best action — which results in accidents, almost-accidents, and bugs.
It’s clear to me that full self-driving won’t be ready soon. But not everyone agrees.
Elon Musk has been promising full autonomy since 2015. In the opening keynote at Nvidia GTC Musk said “I don’t think we have to worry about autonomous cars because it’s a sort of a narrow form of AI. It’s not something I think is very difficult. […] I almost view it like a solved problem.” After that, Musk has promised full self-driving capacity every two years.
In 2016 he thought it’d be ready for 2017: “I feel pretty good about the goal of a demonstration drive of full autonomy all the way from LA to New York.” In 2018: “I think we’ll get to full self-driving next year, as a generalized solution.” In 2020 he said full self-driving would be “feature complete” by the end of the year.
He was wrong each time.
In 2021 he acknowledged the difficulty of the challenge in a Tweet.
Other companies are betting on self-driving. They aren’t as popular as Tesla nor are they following its steps. Waymo and Cruise have fewer cars on the roads in exchange for taking a safer and more technologically heavy approach. Most experts agree that self-driving won’t be possible only with vision — Lidar and other mapping technologies will be necessary to get to the next level. Yet, despite their efforts, they fall way behind Tesla in the race for complete autonomy.
Tesla seems to be going much faster than its competitors for two reasons: First, they’re applying a pure-vision approach; just neural networks doing what they do best, processing tons of data captured with cameras all around the car. However, it’s not clear at all — despite what Musk says— that the autonomy problem can be solved this way. Second, Tesla is better at selling cars than at building them. Not everyone loves Elon Musk and Tesla, but those who do are diehard fans. Tesla cars flood the roads.
Tesla will fail to meet expectations yet another year either because the tech isn’t ready or because its approach is inherently flawed. Other companies won’t build a self-driving car either because safety implies slow progress — although in this case, it’s worth it.
Tesla feels like the leader in self-driving technology but it’s just the louder player. Musk is the master of slipping away from undelivered promises. Self-driving cars are further in the future than most people believe — and Musk isn’t one of those people.
Trending AI/ML Article Identified & Digested via Granola by Ramsey Elbasheer; a Machine-Driven RSS Bot