How to Build TensorFlow Models with the Keras Functional API

https://miro.medium.com/max/1200/0*xuydIbzFxELKK1Dz

Original Source Here

How to Build TensorFlow Models with the Keras Functional API

Photo by Marvin Meyer on Unsplash

The Keras Functional API provides a way to build flexible and complex neural networks in TensorFlow. The Functional API is used to design networks that are not linear. In this article, you will discover that the Keras Functional API is used to create networks that:

  • Are non-linear.
  • Share layers.
  • Have multiple inputs and outputs.

Keras Sequential models

We used the Sequential API in the CNN tutorial to build an image classification model with Keras and TensorFlow. The Sequential API involves stacking layers. One layer is followed by another layer until the final dense layer. This makes designing networks with the Sequential API easy and straightforward.

parameters = {"shape":28, "activation": "relu", "classes": 10, "units":12, "optimizer":"adam", "epochs":1,"kernel_size":3,"pool_size":2, "dropout":0.5}
# Setup the layers
model = keras.Sequential(
[
layers.Conv2D(32, kernel_size=(parameters["kernel_size"], parameters["kernel_size"]), input_shape =(parameters["shape"], parameters["shape"], 1),activation=parameters["activation"]),
layers.MaxPooling2D(pool_size=(parameters["pool_size"], parameters["pool_size"])),
layers.Conv2D(64, kernel_size=(parameters["kernel_size"], parameters["kernel_size"]), activation=parameters["activation"]),
layers.MaxPooling2D(pool_size=(parameters["pool_size"], parameters["pool_size"])),
layers.Flatten(),
layers.Dropout(parameters["dropout"]),
layers.Dense(parameters["classes"], activation="softmax"),
]
)

The Sequential API limits you to one input and one output. However, you may want to design neural networks with multiple inputs and outputs in certain scenarios. For example, given an image of a person, you can design a network to predict several attributes such as gender, age, and hair color. This is a network with one input but multiple outputs. To achieve this, the Sequential API is required. Plotting the network shows that the layers are arranged in a linear manner.

keras.utils.plot_model(model, "model.png",show_shapes=True)
Image by author

Keras Functional models

Designing Functional models is a little different from developing Sequential models. Let’s look at those differences.

Defining input

The first difference is the requirement to create an input layer. With the Sequential API, you don’t have to define the input layer. Defining the input shape in the first layer is sufficient.

The inputs layer contains the shape and type of data to be passed to the network.

inputs = keras.Input(shape=(parameters["shape"], parameters["shape"], 1))
inputs.shape
# TensorShape([None, 28, 28, 1])
inputs.dtype
# tf.float32

The input layer is defined without the batch size if the data is one-dimensional.

inputs = keras.Input(shape=(784,))

Connecting layers

The next difference is how layers are connected using the Functional API. To create the connection, we create another layer and pass the inputs layer to it. This is best understood by considering each of the layers as a function. Since the layers are functions, they can be called with parameters. For example, let’s pass the inputs to a Conv2D layer.

conv2D = layers.Conv2D(32)
x = conv2D(inputs)
x
# <KerasTensor: shape=(None, 26, 26, 32) dtype=float32 (created by layer 'conv2d_7')>

In the above example, we create a Conv2D layer, call it as a function and pass the inputs. The resulting output’s shape is different from the initial inputs shape as a result of being passed to the convolution layer.

Functional API Python syntax

The above example shows how to define and connect the networks verbosely. However, the syntax can be simplified. The simplified version looks like this:

conv2D = Conv2d(...) (inputs)

conv2D() is similar to conv2D.__call__(self,....). Python objects implement the __call__() method. Keras layers also implement this method. The method returns the output given an input tensor.

inputs = keras.Input(shape=(parameters["shape"], parameters["shape"], 1))
conv2D = layers.Conv2D(32, kernel_size=(parameters["kernel_size"], parameters["kernel_size"]), input_shape =(parameters["shape"], parameters["shape"], 1),activation=parameters["activation"])(inputs)
conv2D
# <KerasTensor: shape=(None, 26, 26, 32) dtype=float32 (created by layer 'conv2d_8')>

Creating the model

Let’s add a few more layers to the network to demonstrate how to create a Keras model when layers are defined using the Functional API.

parameters = {"shape":28, "activation": "relu", "classes": 10, "units":12, "optimizer":"adam", "epochs":1,"kernel_size":3,"pool_size":2, "dropout":0.5}
inputs = keras.Input(shape=(parameters["shape"], parameters["shape"], 1))
conv2D = layers.Conv2D(32, kernel_size=(parameters["kernel_size"], parameters["kernel_size"]), input_shape =(parameters["shape"], parameters["shape"], 1),activation=parameters["activation"])(inputs)
maxPooling2D = layers.MaxPooling2D(pool_size=(parameters["pool_size"], parameters["pool_size"]))(conv2D)
conv2D_2 =layers.Conv2D(64, kernel_size=(parameters["kernel_size"], parameters["kernel_size"]), activation=parameters["activation"])(maxPooling2D)
maxPooling2D_2 = layers.MaxPooling2D(pool_size=(parameters["pool_size"], parameters["pool_size"]))(conv2D_2)
flatten = layers.Flatten()(maxPooling2D_2)
dropout = layers.Dropout(parameters["dropout"])(flatten)
ouputs = layers.Dense(parameters["classes"], activation="softmax")(dropout)

A Keras model is created using the keras.Model function while passing the inputs and outputs.

model = keras.Model(inputs=inputs, outputs=outputs, name="mnist_model")

We can plot the model to confirm that it’s similar to the one we defined using the Sequential API.

keras.utils.plot_model(model, "model.png",show_shapes=True)
Image by author

Training and evaluation of Functional API models

Training and evaluating models are the same in the Functional API and the Sequential API. keras.Model avails the fit and evaluate methods.

(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()
x_train = x_train.astype("float32") / 255
x_test = x_test.astype("float32") / 255
model.compile(
loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
optimizer=keras.optimizers.RMSprop(),
metrics=["accuracy"],
)
history = model.fit(x_train, y_train, batch_size=64, epochs=2, validation_split=0.2)
test_scores = model.evaluate(x_test, y_test, verbose=2)
print("Test loss:", test_scores[0])
print("Test accuracy:", test_scores[1])
Image by author

Save and serialize Functional API models

Model saving and serialization work the same in the Functional API and the Sequential API. For instance, we can save the entire model using model.save().

model.save("saved_model")
del model
model = keras.models.load_model("saved_model")
model.summary()
Image by author

How to convert a Functional model to a Sequential API model

A Functional model with linear layers can be converted into a Sequential model by creating an instance of Sequential and adding the layers.

seq_model = keras.models.Sequential()
for layer in model.layers:
seq_model.add(layer)
seq_model.summary()
Image by author

How to convert a Sequential model to a Functional API model

Similarly, we can convert Sequential networks to Functional models.

inputs = keras.Input(batch_shape=seq_model.layers[0].input_shape)
x = inputs
for layer in seq_model.layers:
x = layer(x)
outputs = x
func_model = keras.Model(inputs=inputs, outputs=outputs, name="func_mnist_model")
func_model.summary()
Image by author

Standard network models

Let’s look at how to define standard neural networks using the Functional Keras API.

Multilayer perception

We start by defining a neural network with multiple hidden layers and plot the model.

inputs = keras.Input(shape=(parameters["shape"], parameters["shape"], 1))
dense1 = layers.Dense(128)(inputs)
dropout = layers.Dropout(parameters["dropout"])(dense1)
dense2 = layers.Dense(128)(dropout)
dropout1 = layers.Dropout(parameters["dropout"])(dense2)
outputs = layers.Dense(parameters["classes"], activation="softmax")(dropout1)
model = keras.Model(inputs=inputs, outputs=outputs, name="mnist_model")
keras.utils.plot_model(model, "model.png",show_shapes=True)
Image by author

Convolutional Neural Network

Next, we look at how to define Convolutional Neural Networks using the Functional API. The network has convolution, pooling, flatten, and dense layers.

inputs = keras.Input(shape=(parameters["shape"], parameters["shape"], 1))
conv2D = layers.Conv2D(32, kernel_size=(parameters["kernel_size"], parameters["kernel_size"]), input_shape =(parameters["shape"], parameters["shape"], 1),activation=parameters["activation"])(inputs)
maxPooling2D = layers.MaxPooling2D(pool_size=(parameters["pool_size"], parameters["pool_size"]))(conv2D)
conv2D_2 =layers.Conv2D(64, kernel_size=(parameters["kernel_size"], parameters["kernel_size"]), activation=parameters["activation"])(maxPooling2D)
maxPooling2D_2 = layers.MaxPooling2D(pool_size=(parameters["pool_size"], parameters["pool_size"]))(conv2D_2)
flatten = layers.Flatten()(maxPooling2D_2)
dropout = layers.Dropout(parameters["dropout"])(flatten)
outputs = layers.Dense(parameters["classes"], activation="softmax")(dropout)
model = keras.Model(inputs=inputs, outputs=outputs, name="mnist_model")
keras.utils.plot_model(model, "model.png",show_shapes=True)
Image by author

Recurrent Neural Network

Let’s look at the definition of a bidirectional LSTM using the Functional API. The network contains an Embedding layer.

inputs = keras.Input(784,)
embedding = layers.Embedding(512, 64, input_length=1024)(inputs)
bidirectional1 = layers.Bidirectional(layers.LSTM(64, return_sequences=True))(embedding)
bidirectional2 = layers.Bidirectional(layers.LSTM(64,))(bidirectional1)
dense1 = layers.Dense(32, activation='relu')(bidirectional2)
outputs = layers.Dense(1, activation='sigmoid')(dense1)
model = keras.Model(inputs=inputs, outputs=outputs, name="lstm_model")
keras.utils.plot_model(model, "model.png",show_shapes=True)
Image by author

Shared layers model

Defining layers with the Functional API enables the creation of networks that share certain layers. Shared layers are used several times in a network.

Shared input layer

This example defines a CNN with one input layer shared by two convolution blocks. We then join the outputs from these blocks using the concatenate layer. After that, we pass the result to a DropOut layer and finally to fully connected layer.

inputs = keras.Input(shape=(parameters["shape"], parameters["shape"], 1))
conv2D = layers.Conv2D(32, kernel_size=(parameters["kernel_size"], parameters["kernel_size"]), input_shape =(parameters["shape"], parameters["shape"], 1),activation=parameters["activation"])(inputs)
maxPooling2D = layers.MaxPooling2D(pool_size=(parameters["pool_size"], parameters["pool_size"]))(conv2D)
flatten1 = layers.Flatten()(maxPooling2D)
conv2D_2 = layers.Conv2D(64, kernel_size=(parameters["kernel_size"], parameters["kernel_size"]), activation=parameters["activation"])(inputs)
maxPooling2D_2 = layers.MaxPooling2D(pool_size=(parameters["pool_size"], parameters["pool_size"]))(conv2D_2)
flatten2 = layers.Flatten()(maxPooling2D_2)
# merge layers
merged_layers = layers.concatenate([flatten1, flatten2])
dropout = layers.Dropout(parameters["dropout"])(merged_layers)
outputs = layers.Dense(parameters["classes"], activation="softmax")(dropout)
model = keras.Model(inputs=inputs, outputs=outputs, name="mnist_model")
keras.utils.plot_model(model, "model.png",show_shapes=True)

Plotting the network shows the connection between the different layers.

Image by author

Shared feature extraction layer

In this example, we create an embedding layer shared by two bidirectional LSTMs. A shared feature extraction layer enables sharing the same feature extractor multiple times in the network. For example, sharing this information between two inputs can make it possible to train a network with less data.

inputs = keras.Input(784,)
embedding = layers.Embedding(512, 64, input_length=1024)(inputs)
bidirectional1 = layers.Bidirectional(layers.LSTM(64, return_sequences=True))(embedding)
bidirectional2 = layers.Bidirectional(layers.LSTM(64, return_sequences=True))(embedding)
# merge layers
merged_layers = layers.concatenate([bidirectional1, bidirectional2])
dense1 = layers.Dense(32, activation='relu')(merged_layers)
outputs = layers.Dense(1, activation='sigmoid')(dense1)
model = keras.Model(inputs=inputs, outputs=outputs, name="lstm_model")
keras.utils.plot_model(model, "model.png",show_shapes=True)
Image by author

Next, let’s discuss the multiple inputs and outputs scenario.

Multiple input and output models

Networks with multiple inputs and outputs can also be defined using the Functional API. This is not possible with the Sequential API.

Multiple input model

In this example, we define a network that takes two inputs of different lengths. We pass the inputs to dense layers and sum them using the add layer.

input1 = keras.Input(shape=(16,))
x1 =layers.Dense(8, activation='relu')(input1)
input2 = layers.Input(shape=(32,))
x2 = layers.Dense(8, activation='relu')(input2)
# equivalent to `added = tf.keras.layers.add([x1, x2])`
added = layers.Add()([x1, x2])
out = layers.Dense(4)(added)
model = keras.Model(inputs=[input1, input2], outputs=out)
keras.utils.plot_model(model, "model.png",show_shapes=True)
Image by author

Multiple output model

The Functional API enables the definition of models with multiple outputs. The example below defines a convolutional neural network with two output layers. For instance, given an image of a person, this network can predict gender and hair color.

image_input = keras.Input(shape=(parameters["shape"], parameters["shape"], 3), name="images") 
x = layers.Conv2D(filters=32,kernel_size=(3,3),activation='relu')(image_input)
x = layers.MaxPooling2D(pool_size=(2,2))(x)
x = layers.Conv2D(filters=32,kernel_size=(3,3), activation='relu')(x)
x = layers.Dropout(0.25)(x)
x = layers.Conv2D(filters=64,kernel_size=(3,3), activation='relu')(x)
x = layers.MaxPooling2D(pool_size=(2,2))(x)
x = layers.Dropout(0.25)(x)
x = layers.Flatten()(x)
x = layers.Dense(128, activation='relu')(x)
x = layers.Dropout(0.25)(x)
gender_prediction = layers.Dense(3, activation='softmax')(x)
age_prediction = layers.Dense(3, activation='softmax')(x)
model = keras.Model(
inputs=image_input,
outputs=[gender_prediction, age_prediction],
)
keras.utils.plot_model(model, "model.png",show_shapes=True)
Image by author

Use the same graph of layers to define multiple models

The Functional API also enables the definition of multiple models using the same layers. This is possible because creating a model using the Functional API requires only the inputs and outputs. For example, this is applicable in the encode decoder network architecture.

encoder_input = keras.Input(shape=(28, 28, 1), name="img")
x = layers.Conv2D(16, 3, activation="relu")(encoder_input)
x = layers.Conv2D(32, 3, activation="relu")(x)
x = layers.MaxPooling2D(3)(x)
x = layers.Conv2D(32, 3, activation="relu")(x)
x = layers.Conv2D(16, 3, activation="relu")(x)
encoder_output = layers.GlobalMaxPooling2D()(x)
encoder = keras.Model(encoder_input, encoder_output, name="encoder")
encoder.summary()
x = layers.Reshape((4, 4, 1))(encoder_output)
x = layers.Conv2DTranspose(16, 3, activation="relu")(x)
x = layers.Conv2DTranspose(32, 3, activation="relu")(x)
x = layers.UpSampling2D(3)(x)
x = layers.Conv2DTranspose(16, 3, activation="relu")(x)
decoder_output = layers.Conv2DTranspose(1, 3, activation="relu")(x)
autoencoder = keras.Model(encoder_input, decoder_output, name="autoencoder")
autoencoder.summary()

Keras Functional API strengths and weaknesses

Keras Functional API is handy when designing non-linear networks. If you think that you may need to convert a network to a non-linear structure, then you should use the Functional API. Some of the strengths of the Functional API include:

  • It is less verbose compared to subclassing the Model class.
  • The requirement to create an Input ensures that all Functional networks will run because passing the wrong shape leads to an immediate error.
  • A Functional model is easier to plot and inspect.
  • Easy to serialize and save Functional models because they are data structures.

However, one drawback of using the Functional API is that it doesn’t support dynamic architectures such as recursive networks or Tree RNNs.

Functional API best practices

Keep best practices in mind while working with the Keras Functional API:

  • Always print a summary of the network to confirm that the shapes of the various layers are as expected.
  • Plot the network to ensure the layers are connected as you expect them.
  • Name the layers to make it easy to identify them in the network plot and summary. For example Conv2D(...,name="first_conv_layer").
  • Use variable names that are related to the layers, for example, conv1 and conv2 for convolution layers. This will clarify the type of layer when inspecting plots and network summary.
  • Instead of creating a custom training loop, use the keras.Model to create models because it makes it easier to train models via the fit method and evaluate them with the evalaute method.

AI/ML

Trending AI/ML Article Identified & Digested via Granola by Ramsey Elbasheer; a Machine-Driven RSS Bot

%d bloggers like this: