Inherit from, “tensorflow.keras.layers.Layer” to define new Layers for your Deep Learning model

Original Source Here

Inherit from, “tensorflow.keras.layers.Layer” to define new Layers for your Deep Learning model

I write this article to share my knowledge and experience of using the tf.keras.layer.Layer class to build new layers. Using this approach, rather than using predefined layers might deliver a layer that will be a better fit for your data and finally result in an accurate model.


1) Introduction

4) What is a layer ??

2) Implementation procedure

3) Benefits

4) Refference


Before moving on to the structure of the layers. Let’s understand a key concept in OOP (object-oriented programming). This is inheritance.


Inheritance is one of the key concepts in OOP. What this concept describes is you can derive a class from another class. Thus you can generate a hierarchy of classes that share the same set of attributes and methods.

So basically this gives you the ability to declare exceptions or apply custom made logics to an existing framework and furthermore, you can define new methods as well.

figure 01: Basic implementation of inheritance

So this same principle is used here as well. This tf.keras.layers.Layer is the superclass for all the layers in Keras. In another way, they all inherit from tf.keras.layers.Layer. So as shown in figure 01, you can you inheritance to create custom layers for your model.

For example tf.keras.layers.Layer is like mobilePhone class in figure 01 and Dense layer is like androidPhone.

What is a Layer ??

In Tensorflow Keras we can define a layer like this,

  • A layer is an object.
  • A layer is callable.
  • Takes one or more inputs (tensor inputs).
  • Do computations defined under call() method for state variables defined under __init__() constructor or build() method.
  • Output one or more tensors.

Implementation Procedure

Here what we do is, extend the existing tf.keras.layer.Layer and additional you can define your own methods as well. The extension can be done under three methods,

  • __init__() — > where you can do all input-independent initialization
  • build() — > where you know the shapes of the input tensors and can do the rest of the initialization
  • call() — > where you do the forward computation

You can use two methods as your approach during implementation. Following figure 01 and figure 02 are example code segments that explain these two approaches.

figure 02: Method I
figure 03: Method ii

Here the difference is in the first method (figure 01), we are creating our variables in the __init__() constructor itself and in the second method (figure 02) we are using build() method to define our variables after the constructor is defined.

Of course, there must be an advantage behind the method ii. Otherwise, there is no use in defining an additional method.

YES, there is…

The advantage is creating them in build() is that it enables late variable creation based on the shape of the inputs the layer will operate on. On the other hand, creating variables in __init__()would mean that shapes required to create the variables will need to be explicitly specified.

Following is a detailed explanation as in Tensorflow documentation. You can refer to the links provided under the reference section for more information on these.

__init__(): Defines custom layer attributes, and creates layer state variables that do not depend on input shapes, using add_weight().

build(self, input_shape): This method can be used to create weights that depend on the shape(s) of the input(s), using add_weight(). __call__() will automatically build the layer (if it has not been built yet) by calling build().

call(self, inputs, *args, **kwargs): Called in __call__ after making sure build() has been called. call() performs the logic of applying the layer to the input tensors (which should be passed in as an argument). Two reserved keyword arguments you can optionally use in call() are:

get_config(self): Returns a dictionary containing the configuration used to initialize this layer. If the keys differ from the arguments in __init__, then override from_config(self) as well. This method is used when saving the layer or a model that contains this layer.


The main benefit is that the existing predefined layer in Keras might not fit your model, might not fit the data structures that you are giving to the model or computationally these layers might lack some features that you want you to want in your model. So by using this you can adjust changes and do an effective implementation.



Trending AI/ML Article Identified & Digested via Granola by Ramsey Elbasheer; a Machine-Driven RSS Bot

%d bloggers like this: