How to create a poet / writer using Deep Learning (Text Generation using Python)?

Introduction

From short stories to writing 50,000 word novels, machines are churning out words like never before. There are tons of examples available on the web where developers have used machine learning to write pieces of text, and the results range from the absurd to delightfully funny.

Thanks to major advancements in the field of Natural Language Processing (NLP), machines are able to understand the context and spin up tales all by themselves.

Examples of text generation include machines writing entire chapters of popular novels like Game of Thrones and Harry Potter, with varying degrees of success.

In this article, we will use python and the concept of text generation to build a machine learning model that can write sonnets in the style of William Shakespeare.

Let’s get into it!

Table of Contents

1. What are text generators?
2. The different steps of Text Generation
   • Importing Dependencies
   • Loading the Data
   • Creating Character/Word mappings
   • Data Preprocessing
   • Modelling
   • Generating text
3. Experimenting with different models
   • A more trained model
   • A deeper model
   • A wider model
   • A gigantic model

What are text generators?

Nowadays, there is a huge amount of data that can be categorized as sequential. It is present in the form of audio, video, text, time series, sensor data, etc. A special thing about this type of data is that if two events are occurring in a particular time frame, the occurrence of event A before event B is an entirely different scenario as compared to the occurrence of event A after event B.

However, in conventional machine learning problems, it hardly matters whether a particular data point was recorded before the other. This consideration gives our sequence prediction problems a different solving approach.

Text, a stream of characters lined up one after another, is a difficult thing to crack. This is because when handling text, a model may be trained to make very accurate predictions using the sequences that have occurred previously, but one wrong prediction has the potential to make the entire sentence meaningless. However, in case of a numerical sequence prediction problem, even if a prediction goes entirely south, it could still be considered a valid prediction (maybe with a high bias). But, it would not strike the eye.

This is what makes text generators tricky!

For a better understanding of the code please go through my previous article, where I have discussed the theory behind LSTMs.

Different Steps of Text Generation

Text generation usually involves the following steps:

1. Importing Dependencies
2. Loading of Data
3. Creating Character/Word mappings
4. Data Preprocessing
5. Modelling
6. Generating text

Let’s look at each one in detail.

Importing Dependencies

import numpy as np
import pandas as pd
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Dropout
from keras.layers import LSTM
from keras.utils import np_utils

This is self-explanatory. We are importing all libraries required for our study.

Loading the Data

text=(open("/Users/pranjal/Desktop/text_generator/sonnets.txt").read())
text=text.lower()

Here, we are loading a combined collection of all Shakespearean sonnets that can be downloaded from here. I cleaned up this file to remove the start and end credits, and it can be downloaded from my git repository.

The text file is opened and saved in text. This content is then converted into lowercase, to reduce the number of possible words (more on this later).

Creating character/word mappings

Mapping is a step in which we assign an arbitrary number to a character/word in the text. In this way, all unique characters/words are mapped to a number. This is important, because machines understand numbers far better than text, and this subsequently makes the training process easier.

characters = sorted(list(set(text)))
n_to_char = {n:char for n, char in enumerate(characters)}
char_to_n = {char:n for n, char in enumerate(characters)}

I have created a dictionary with a number assigned to each unique character present in the text. All unique characters are first stored in characters and are then enumerated.

It must also be noted here that I have used character level mappings and not word mappings. However, when compared with each other, a word-based model shows much higher accuracy as compared to a character-based model. This is because the latter model requires a much larger network to learn long-term dependencies as it not only has to remember the sequences of words, but also has to learn to predict a grammatically correct word. However, in case of a word-based model, the latter has already been taken care of.

But since this is a small dataset (with 17,670 words), and the number of unique words (4,605 in number) constitute around one-fourth of the data, it would not be a wise decision to train on such a mapping. This is because if we assume that all unique words occurred equally in number (which is not true), we would have a word occurring roughly four times in the entire training dataset, which is just not sufficient to build a text generator.

Data preprocessing

This is the most tricky part when it comes to building LSTM models. Transforming the data at hand into a relatable format is a difficult task.

I’ll break down the process into small parts to make it easier for you.

Python Code:

import numpy as np
import pandas as pd
# from keras.models import Sequential
# from keras.layers import Dense
# from keras.layers import Dropout
# from keras.layers import LSTM
# from keras.utils import np_utils

text = '"A natural image usually conveys rich semantic content and can be viewed from different angles. Existing image description methods are largely restricted by small sets of biased visual paragraph annotations"'

text=text.lower()

characters = sorted(list(set(text)))
n_to_char = {n:char for n, char in enumerate(characters)}
char_to_n = {char:n for n, char in enumerate(characters)}

X = []
Y = []
length = len(text)
seq_length = 100
for i in range(0, length-seq_length, 1):
   sequence = text[i:i + seq_length]
   label =text[i + seq_length]
   X.append([char_to_n[char] for char in sequence])
   Y.append(char_to_n[label])

print("Test Data - ",Y)

Here, X is our train array, and Y is our target array.

seq_length is the length of the sequence of characters that we want to consider before predicting a particular character.

The for loop is used to iterate over the entire length of the text and create such sequences (stored in X) and their true values (stored in Y). Now, it’s difficult to visualize the concept of true values here. Let’s understand this with an example:

For a sequence length of 4 and the text “hello india”, we would have our X and Y (not encoded as numbers for ease of understanding) as below:

X	Y
[h, e, l, l]	[o]
[e, l, l, o]	[ ]
[l, l, o,  ]	[i]
[l, o,  , i]	[n]
….	….

Now, LSTMs accept input in the form of (number_of_sequences, length_of_sequence, number_of_features) which is not the current format of the arrays. Also, we need to transform the array Y into a one-hot encoded format.

X_modified = np.reshape(X, (len(X), seq_length, 1))
X_modified = X_modified / float(len(characters))
Y_modified = np_utils.to_categorical(Y)

We first reshape the array X into our required dimensions. Then, we scale the values of our X_modified so that our neural network can train faster and there is a lesser chance of getting stuck in a local minima. Also, our Y_modified is one-hot encoded to remove any ordinal relationship that may have been introduced in the process of mapping the characters. That is, ‘a’ might be assigned a lower number as compared to ‘z’, but that doesn’t signify any relationship between the two.

Our final arrays will look like:

X_modified	Y_modified
[[ 0.44444444], [ 0.33333333], [ 0.66666667], [ 0.66666667]]	[ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  1.]
[[ 0.33333333], [ 0.66666667], [ 0.66666667], [ 0.88888889]]	[ 1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.]
[[ 0.66666667], [ 0.66666667], [ 0.88888889], [ 0.        ]]	[ 0.,  0.,  0.,  0.,  0.,  1.,  0.,  0.,  0.]