Original Source Here
In this blog(which happens to be my very first out of the drafts xD), I am going to implement the same project and will explain in parts, the execution and functioning of the code.
- Define a suitable Architecture
- Data Preprocessing
- Training and Testing
- Hyperparameter Tuning
Importing the required Libraries and Functions
from tensorflow.keras.preprocessing.sequence import pad_sequences
from tensorflow.keras.layers import Embedding, LSTM, Dense, Dropout, Bidirectional
from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.models import Sequential
from tensorflow.keras.optimizers import Adam
from tensorflow.keras import regularizers
import tensorflow.keras.utils as ku
import numpy as np
This is the first section of my code wherein all the required functions, classes, and libraries are imported. I am using the TensorFlow library instead of PyTorch for implementing my sequence model because I do not need a lot of variations in my model and TF helps me achieve that faster. And of course Numpy for mathematical operations on the data. The TF functions that I’ve imported will be explained later as we encounter them in the code.
tokenizer = Tokenizer()!wget --no-check-certificate \
-O /tmp/sonnets.txtdata = open('/tmp/sonnets.txt').read()corpus = data.lower().split("\n")
total_words = len(tokenizer.word_index) + 1
In the above code block, the “wget” command is used to fetch the Shakespeare Sonnet 1 that is uploaded on the given address to a text file sonnets.txt in my tmp folder. This file is then later read and stored in the variable “data” which will be used to train our model after we process it. Tokenizer, as the name suggests tokenizes the data. That is, it breaks down the sentences into words. “corpus” stores the data after the whole text-sentences are converted into lower case and split at new-lines. It now contains individual lines of the sonnet as one element in the list. Tokenizer is then used on the same to further break the lines down to words and assign unique words with unique integers.
# create input sequences using list of tokens
input_sequences = for line in corpus:
token_list = tokenizer.texts_to_sequences([line])
for i in range(1, len(token_list)):
n_gram_sequence = token_list[:i+1]
Now, in most of the applications of NLP, we almost always face a problem of limited data which might not be enough to train our model effectively. so we use the existing text that we have and break it down into multiple parts to generate more information from the same piece of text. This technique is called N-Gram sequencing which has been done in the code by iterating over each of the lines, and the sequences hence generated are added to the “input_sequence” list.
# pad sequences
max_sequence_len = max([len(x) for x in input_sequences])
input_sequences = np.array(pad_sequences(input_sequences, maxlen=max_sequence_len, padding='pre'))
The data, whenever fed to a model show be homogeneous and well processed so that the model doesn’t start creating irrelevant patterns out of the input based on input sizes that do not contribute to the meaning of the text. Therefore to overcome this issue, we pad the “input_sequence” by adding 0’s in front of them.
# create predictors and label
predictors, label = input_sequences[:,:-1],input_sequences[:,-1]
label = ku.to_categorical(label, num_classes=total_words)
Now that our data is clean and homogeneous enough to be fed to our model, we need to label the data as input and outputs. I won’t be explaining why this needs to be done in this blog, you can refer to my workshop recording for a proper explanation along with some other fundamental concepts of Deep Learning and NLP as well.
Architecture of the Model
model = Sequential()
model.add(Embedding(total_words, 100, input_length=max_sequence_len-1))
model.add(Bidirectional(LSTM(150, return_sequences = True)))
model.add(Dense(total_words/2, activation='relu', kernel_regularizer=regularizers.l2(0.01)))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
The model architecture happens to be at the very core of any Machine Learning application, it needs to be well thought of and well structured because it directly affects the output. Any neural network consists of multiple layers that are responsible for contributing and computing their part in the parametric weights. I’ll be explaining the layers I’ve used and their functions concisely below:
- Embedding Layer: A word embedding is a class of approaches for representing words and documents using a dense vector representation. In an embedding, words are represented by a dense vector where a vector represents the projection of the word into a continuous vector space. Simply put, it plots the words of similar meaning together and returns their positions in the form of vectors.
- Long Short Teem Memory(LSTM): Humans don’t start thinking from scratch every second. As you read this sentence, you understand each word based on your understanding of previous words. Your thoughts have persistence. LSTM Architecture basically tries to replicate that persistence of thoughts in your model while training it.
- Bidirectional LSTMs: While training, this layer basically runs through the input, in both forward and backward direction instead of the traditional LSTM forward-only direction. This trains the model in such a way that in the future when it receives input it takes into account the past as well as the future relevance.
- Dropout Layer: This layer shuts off a specified number of neurons for better results. It prevents overfitting and unwanted neuron emphasis. Dropping out neurons is a part of hyperparameter tuning.
- Dense Layer: It is a NN layer that is connected deeply. It receives input from all the neurons in its previous layer. It is also the most commonly used layer in a NN for defining the structure.
- ReLU and Softmax: They are activation functions. They help us in determining the results in the form they are supposed to be. They help in providing non-linearity.
Training the model
history = model.fit(predictors, label, epochs=100, verbose=1)
This code statement fits the data onto my architecture and thereby trains my model with the pre-processed data that we prepared in the initial stage. Epoch is nothing but a hyperparameter that defines the number of times my model will be trained on the data set (in this case 100 times).
With this, the training of my sequence model is complete and it is ready to be used for text prediction.
*Shakespeare picks up the pen and the bass guitar*
seed_text = "Buying a stairway to heaven"
next_words = 100for _ in range(next_words):
token_list = tokenizer.texts_to_sequences([seed_text])
token_list = pad_sequences([token_list], maxlen=max_sequence_len-1, padding='pre')
predicted = model.predict_classes(token_list, verbose=0)
output_word = ""
for word, index in tokenizer.word_index.items():
if index == predicted:
output_word = word
seed_text += " " + output_word
I am using “Buying a stairway to Heaven” as my seed text, i.e., the text on the basis of what my sequence model will predict the next 100 words (as specified by me). The next few statements of my code perform the same data pre-processing on this seed text and then feed it to the model. The words hence predicted are then stored in the “output_word” and then later concatenated with the original seed text. Here is the output of this code:
Buying a stairway to heaven though thou dost more lack more part pride long glory from their woe shall ride impute feast hour her part brought thee by men age much best still grow new pride told from thine need true thine own thy breast in their days will ‘ be told he was so plight chary forsworn her treasure foes about view bright keep confounds room only need eye to hell still go than ground other about prove confounds serving young thee arising lust new ‘ mistaking bow subjects place her substance twain gone so bright confounds she forsworn me alone so bright hence
It does not make absolute sense but yeah it something and the final text predicted definitely matches Shakespeare’s style of writing! You can change the style by changing what the model is trained on (I recommend some long pieces of literature) and by changing the seed text you can bring variations on the output of this Sequence Model.
So there you go, an NLP model that writes songs by poorly imitating Shakespeare’s writing style 😛
Trending AI/ML Article Identified & Digested via Granola by Ramsey Elbasheer; a Machine-Driven RSS Bot