This Neural Networkfrom OpenAI can Learn from Small Datasets



Original Source Here

This Neural Network from OpenAI can Learn from Small Datasets

Glow is an iconic interesting research about deep neural networks that can generalize with small training sets.

Image Credit: OpenAI

I recently started an AI-focused educational newsletter, that already has over 80,000 subscribers. TheSequence is a no-BS (meaning no hype, no news etc) ML-oriented newsletter that takes 5 minutes to read. The goal is to keep you up to date with machine learning projects, research papers and concepts. Please give it a try by subscribing below:

Since the early days of machine learning, artificial intelligence scenarios have faced with two big challenges in order to experience mainstream adoption. First, we have the data efficiency problem that requires machine or deep learning models to be trained using large and accurate datasets which, as we know, are really expensive to build and maintain. Secondly, we have the generalization problem which AI agents face in order to build new knowledge that is different from the training data. Humans, by contrast, are incredibly efficient learning with minimum supervision and rapidly generalizing knowledge from a few data examples.

Generative models are one of the deep learning disciplines that focuses on addressing the two challenges mentioned above. Conceptually, generative models are focused on observing an initial dataset, like a set of pictures, and try to learn how the data was generated. Using more mathematical terms, generative models try to infer all dependencies within very high-dimensional input data, usually specified in the form of a full joint probability distribution. Entire deep learning areas such as speech synthesis or semi-supervised learning are based on generative models. Recently, generative models such as generative adversarial networks(GANs) have become extremely popular within the deep learning community. Recently, OpenAI experimented with a not-very well-known technique called Flow-Based Generative Models in order to improve over existing methods. The result was the creation of Glow, a new flow-based generative model that is able to generate high quality datasets from a few training examples. The findings were captured in a very famous research paper published by OpenAI in 2018. You can see Glow in action in the following animation:

Video: OpenAI

For some unexplainable reason, flow-based generative models have not been as popular as some of its counterparts like GANs. However, this type of techniques offers some tangible advantages compared to alternatives:

  • Exact latent-variable inference and log-likelihood evaluation: Most generative methods are only able to approximate the value of the latent variables that correspond to a data point. Others like GAN’s have no encoder at all to infer the latents. In reversible generative models, this can be done exactly without approximation. Not only does this lead to accurate inference, it also enables optimization of the exact log-likelihood of the data, instead of a lower bound of it.
  • Efficient inference and efficient synthesis: Flow-based generative models like Glow (and RealNVP) are efficient to parallelize for both inference and synthesis.
  • Significant potential for memory savings: Computing gradients in reversible neural networks requires an amount of memory that is constant instead of linear in their depth as many other generative models like GANs.

Glow Architecture

The architecture of OpenAI Glow builds up on previous work such as NICE and RealNVP but it simplifies those models quite a bit. The Glow model consists of a series of steps combined in a multi-scale architecture. Each step is based on three laeyers: actnorm, followed by an invertible 1×1 convolution followed by a coupling layer.

Image Credit: OpenAI

Given an input dataset, the multi-layer architecture of Glow performs the following tasks.

  1. Permute the inputs by reversing their ordering across the channel dimension.
  2. Split the input into two parts, A and B, down the middle of the feature dimension.
  3. Feed A into a shallow convolutional neural network. Linearly transform B according to the output of the neural network.
  4. Concatenate A and B.

Glow in Action

OpenAI performed a series of quantitive and qualitative experiments to evaluate the capabilities of Glow and some of the results are breathtaking. For instance, Glow was able to generate high resolution images using a predictable and stable performance.

Video: OpenAI

Even more interesting is the fact that Glow learns individual features of images such as hair color or factions of the nose. Glow uses an encoder model to interpolate those individual features and generate new related images.

Video: OpenAI

The research behind Glow shows that flow-based methods have a lot to contribute to the space of generative models. Just like alternatives like GANs, flow-based methods can be a key block when building deep learning models that can learn from small datasets and generalize efficiently.

AI/ML

Trending AI/ML Article Identified & Digested via Granola by Ramsey Elbasheer; a Machine-Driven RSS Bot

%d bloggers like this: