GPT-4 Will Have 100 Trillion Parameters — 500x the Size of GPT-3*PSxUk3lW6R325xnE

Original Source Here

What can we expect from GPT-4?

100 trillion parameters is a lot. To understand just how big that number is, let’s compare it with our brain. The brain has around 80–100 billion neurons (GPT-3’s order of magnitude) and around 100 trillion synapses.

GPT-4 will have as many parameters as the brain has synapses.

The sheer size of such a neural network could entail qualitative leaps from GPT-3 we can only imagine. We may not be able to even test the full potential of the system with current prompting methods.

However, comparing an artificial neural network with the brain is a tricky business. The comparison seems fair but that’s only because we assume artificial neurons are at least loosely based on biological neurons. A recent study published in Neuron suggests otherwise. They found that at least a 5-layer neural network is needed to simulate the behavior of a single biological neuron. That’s around 1000 artificial neurons for each biological neuron.

But even if GPT-4 isn’t as powerful as our brain, it sure will leave a few surprises. Unlike GPT-3, it probably won’t be just a language model. Ilya Sutskever, the Chief Scientist at OpenAI, hinted about this when he wrote about multimodality in December 2020:

“In 2021, language models will start to become aware of the visual world. Text alone can express a great deal of information about the world, but it is incomplete, because we live in a visual world as well.”

We already saw some of this with DALL·E, a smaller version of GPT-3 (12 billion parameters), trained specifically on text-image pairs. OpenAI said then that “manipulating visual concepts through language is now within reach.”

OpenAI has been working nonstop in exploiting GPT-3’s hidden abilities. DALL·E was a special case of GPT-3, very much like Codex. But they aren’t absolute improvements, more like particular cases. GPT-4 promises more. It promises the depth of specialist systems like DALL·E (text-images) and Codex (coding) combined with the width of generalist systems like GPT-3 (general language).

And what about other human-like features, like reasoning or common sense? In that regard, Sam Altman says they’re not sure but he remains “optimistic.”

There are many questions and very few answers. No one knows if AGI is possible. No one knows how to build it. No one knows if larger neural networks will get increasingly closer to it. But something is undeniable: GPT-4 will be something to keep an eye out for.


Trending AI/ML Article Identified & Digested via Granola by Ramsey Elbasheer; a Machine-Driven RSS Bot

%d bloggers like this: