Recurrent Neural Network with Keras



Original Source Here

Recurrent Neural Network with Keras

Project Implementation and cheatsheet…

Credits : wiki

Recurrent Neural Networks (RNN) initially created in the 1980’s are a powerful and robust type of neural network in which output from the previous step are fed as input to the current step. The most important feature of RNN is Hidden state and they have memory which remembers each and every information through time. In Recurrent neural networks, we use the result obtained through the hidden layers to process future input as shown in the diagram below.

Preferred algorithm for sequential data — financial data, audio, video, time series data, speech, text, etc

Credits : Builtin

Keras is a very powerful open source Python library which is runs on top of top of other open source machine libraries like TensorFlow, Theano etc, used for developing and evaluating deep learning models and leverages various optimization techniques.

Features —

  • Keras fully supports recurrent neural networks and convolution neural networks
  • Keras runs smoothly on both CPU and GPU
  • Keras NN are written in Python which advocates simplicity and great debugging power
  • Keras is known for its incredibly expressive, flexible, minimal structure
  • Keras is consistent, simple and extensible API
  • Keras is also known for its highly computational scalability
  • Extensive support for various platforms and backends

In this project we will train a sequence (math expression) to sequence recurrent neural network to implement/solve simple mathematical operations by creating, training and evaluating a simple recurrent neural network model.

Import all the necessary libraries

import numpy as npfrom tensorflow.keras.models import Sequential
from tensorflow.keras.layers import TimeDistributed, Dense, Dropout, SimpleRNN, RepeatVector
from tensorflow.keras.callbacks import EarlyStopping, LambdaCallback
from termcolor import colored

Generate Data

all_chars='0123456789+'
num_features = len(all_chars)
print('no of features:', num_features)
char_to_index= dict((c,i) for i,c in enumerate(all_chars))
index_to_char= dict((i,c) for i, c in enumerate(all_chars))

Output —

no of features: 11def generate_data():
first = np.random.randint(0,100)
second = np.random.randint(0,100)
example = str(first)+ '+' + str(second)
label = str(first+second)
return example, label
generate_date()

Output —

('52+3', '55')

Create the Model

hidden_units=128
max_time_steps=5
model = Sequential([
SimpleRNN(hidden_units,input_shape=(None,num_features)),
RepeatVector(max_time_steps),
SimpleRNN(hidden_units,return_sequences=True),
TimeDistributed(Dense(num_features,activation='softmax'))
]
)
model.compile(
loss='categorical_crossentropy',
optimizer = 'adam',
metrics=['accuracy']
)
model.summary()

Output —

Layer (type)                 Output Shape              Param #   
=================================================================
simple_rnn_4 (SimpleRNN) (None, 128) 17920
_________________________________________________________________
repeat_vector_2 (RepeatVecto (None, 5, 128) 0
_________________________________________________________________
simple_rnn_5 (SimpleRNN) (None, 5, 128) 32896
_________________________________________________________________
time_distributed_2 (TimeDist (None, 5, 11) 1419
=================================================================
Total params: 52,235
Trainable params: 52,235
Non-trainable params: 0
_________________________________________________________________

Vectorize and Devectorize data

def vectorize_example(example,label):
x=np.zeros((max_time_steps,num_features))
y=np.zeros((max_time_steps,num_features))

diff_x = max_time_steps - len(example)
diff_y = max_time_steps - len(label)

for i,c in enumerate(example):
x[i+diff_x,char_to_index[c]] =1
for i in range(diff_x):
x[i,char_to_index['0']] = 1
for i,c in enumerate(label):
y[i+diff_y,char_to_index[c]] =1
for i in range(diff_y):
y[i,char_to_index['0']] = 1
return x,y
e, l = generate_data()
print(e,l)
x,y= vectorize_example(e,l)
print(x.shape,y.shape)

Output —

26+32 58
(5, 11) (5, 11)
def devectorize_example(example):
result = [index_to_char[np.argmax(vec)] for i,vec in enumerate(example)]
return ''.join(result)
devectorize_example(x)

Output —

'26+32'devectorize_example(y)

Output —

'00058'

Create Dataset

def create_dataset(num_examples=2000):
x=np.zeros((num_examples,max_time_steps,num_features))
y=np.zeros((num_examples,max_time_steps,num_features))
for i in range(num_examples):
e,l = generate_data()
e_v, l_v = vectorize_example(e,l)
x[i] = e_v
y[i] = l_v
return x,y
x,y = create_dataset()
print(x.shape,y.shape)

Output —

(2000, 5, 11) (2000, 5, 11)devectorize_example(x[0])
devectorize_example(y[0])

Output —

'38+68'
'00106'

Training the Model

l_cb=LambdaCallback(
on_epoch_end = lambda e,l: print('{:.2f}'.format(l['val_acc']),end=' _ ')
)
es_cb=EarlyStopping(monitor='val_loss',patience=10)
model.fit(x,y,epochs=500,batch_size=256,validation_split=0.2,
verbose=False,callbacks=[es_cb,l_cb])

Output —

0.58 _ 0.58 _ 0.61 _ 0.61 _ 0.62 _ 0.62 _ 0.63 _ 0.63 _ 0.63 _ 0.64 _ 0.64 _ 0.66 _ 0.64 _ 0.65 _ 0.66 _ 0.65 _ 0.67 _ 0.68 _ 0.67 _ 0.69 _ 0.68 _ 0.69 _ 0.70 _ 0.70 _ 0.71 _ 0.71 _ 0.72 _ 0.72 _ 0.71 _ 0.73 _ 0.73 _ 0.71 _ 0.74 _ 0.75 _ 0.73 _ 0.75 _ 0.75 _ 0.75 _ 0.76 _ 0.76 _ 0.76 _ 0.75 _ 0.77 _ 0.77 _ 0.77 _ 0.77 _ 0.78 _ 0.77 _ 0.78 _ 0.78 _ 0.78 _ 0.79 _ 0.79 _ 0.80 _ 0.80 _ 0.79 _ 0.82 _ 0.82 _ 0.83 _ 0.82 _ 0.82 _ 0.84 _ 0.84 _ 0.84 _ 0.85 _ 0.85 _ 0.85 _ 0.86 _ 0.85 _ 0.86 _ 0.86 _ 0.87 _ 0.88 _ 0.88 _ 0.88 _ 0.88 _ 0.89 _ 0.89 _ 0.90 _ 0.90 _ 0.90 _ 0.90 _ 0.89 _ 0.90 _ 0.90 _ 0.90 _ 0.90 _ 0.92 _ 0.91 _ 0.91 _ 0.92 _ 0.92 _ 0.92 _ 0.91 _ 0.92 _ 0.92 _ 0.92 _ 0.92 _ 0.92 _ 0.92 _ 0.93 _ 0.92 _ 0.93 _ 0.92 _ 0.93 _ 0.94 _ 0.93 _ 0.93 _ 0.93 _ 0.93 _ 0.93 _ 0.94 _ 0.94 _ 0.94 _ 0.94 _ 0.94 _ 0.94 _ 0.94 _ 0.95 _ 0.94 _ 0.95 _ 0.95 _ 0.94 _ 0.94 _ 0.94 _ 0.95 _ 0.94 _ 0.95 _ 0.95 _ 0.95 _ 0.95 _ 0.95 _ 0.95 _ 0.95 _ 0.95 _ 0.95 _ 0.95 _ 0.95 _ 0.95 _ 0.95 _ 0.95 _ 0.95 _ 0.95 _ 0.95 _ 0.95 _ 0.95 _ 0.95 _ 0.95 _ 0.95 _ 0.95 _ 0.95 _ 0.95 _ 0.95 _ 0.95 _ 0.95 _ 0.95 _x_test,y_test = create_dataset(10)
preds = model.predict(x_test)
for i,pred in enumerate(preds):
y=devectorize_example(y_test[i])
y_hat = devectorize_example(pred)
col='green'
if y!= y_hat:
col='red'
out='Input: '+ devectorize_example(x_test[i])+' Out: ' +y+'Pred:' +y_hat
print(colored(out,col))

Output —

Input: 82+54 Out: 00136 Pred:00136
Input: 60+81 Out: 00141 Pred:00141
Input: 15+99 Out: 00114 Pred:00114
Input: 00+10 Out: 00010 Pred:00012
Input: 090+1 Out: 00091 Pred:00090
Input: 20+24 Out: 00044 Pred:00044
Input: 55+29 Out: 00084 Pred:00084
Input: 36+47 Out: 00083 Pred:00083
Input: 10+12 Out: 00022 Pred:00022
Input: 71+56 Out: 00127 Pred:00127

AI/ML

Trending AI/ML Article Identified & Digested via Granola by Ramsey Elbasheer; a Machine-Driven RSS Bot

%d bloggers like this: