(Part 1)
There’s a lot of work on using deep learning for text generation – computer-generating new text that mimics human-written text. Even simple models can generate convincing text, as made famous by Andrej Karpathy. But one place that they struggle is when the text must meet strict structural constraints. How can we improve upon current text generation models to make them better at this task?
Haiku generation is an example of a strictly constrained task. They are very short poems consisting of three lines. Traditionally the first line is 5 syllables, the second is 7 and the third is 5. For example, by Brian del Vecchio:
HaiKu’s inventor must have had seven fingers on his middle hand
Modern haiku preserve the essence of a short, three-line poem but don’t strictly adhere to the 5โ7โ5 syllable structure. Thus most haiku online, from which to derive training data, don’t have this structure, and much prior research into generating haiku doesn’t enforce syllable counts either.
Which brings me to my question: can we generate haiku which are both meaningful and structurally correct?
Prior Work
I’ve done previous work on haiku generation. This generator uses Markov chains trained on a corpus of non-haiku poetry, generates haiku one word at a time, and ensures the 5-7-5 structure by backspacing when all the possible next words would violate the 5โ7โ5 structure. This isn’t unlike what I do when I’m writing a haiku. I try things, count out the syllables, find they don’t work and go back. But it’s unsatisfying. It feels more like brute force than something that actually understands what it means to write a haiku. I’ve also used the same methodology using a non-poetry training corpus.
I’m not the only one who’s attempted this. Sam Ballas and Tanel Kiis & Markus Kรคngsepp both used a standard character-level recurrent neural networks (RNN) to generate haiku. They produced output which resemble haiku at a glance, but aren’t terribly cohesive and certainly don’t follow the structure we’re looking for. Part of the problem is that, as I said above, most modern haiku don’t adhere to that structure, which means that a training corpus won’t reflect it. Another problem is that characters correlate very poorly with syllables, so it’s unlikely that a model given nothing but characters, even with a good corpus, would pick up on the 5โ7โ5 form.
Jack Hopkins and Doewe Kiela produced several models for generating poetry. They focused on sonnets, which not only have a constraint on the number of syllables per line, but must follow a specific rhyme scheme. One model was phoneme-level, which allowed them to constrain the structure at training time. The other, a character-level model, could produce more coherent text, but did not have any constraints on form. They constrained the text at sample time by using a discriminator that would reject text that didn’t conform to the desired meter, similar to my technique with the Markov model.
Data
I used four sources to build my training corpus:
- http://www.tempslibres.org/tl/en/dbhk00.html
- The three-line poems from the "Beyond Narrative Description: Generating Poetry from Images by Multi-Adversarial Training" corpus.
- Sam Ballas’ PoetRNN corpus.
- Herval Freire’s Haikuzao corpus.
This Jupyter Notebook has my data preparation process. It cleans the data and uses The CMU Pronouncing Dictionary (CMUdict) to calculate syllable counts for each row. CMUdict maps words to their phonemes, which in turn can be used to count the syllables for each word. In cases where there are multiple valid pronunciations for a word, each is kept.
The final corpus contains 25,886 poems, of which only 725 (3%!) match the 5โ7โ5 structure, which is not nearly enough to train a model on.
The Problem
This leaves us in a predicament. Our goal is to generate haiku that conform to the traditional 5โ7โ5 structure, but we have very little training data for that. What are our options?
- Get more data. If anyone is volunteering to write thousands of haiku for me, I’d be happy to accept the offer.
- Use a method similar to what Hopkins and Kiela did, where we train a character-level network on the whole corpus unconstrained to "learn English" – learn how to form words, sentence structure, etc., then constrain the form at sample time using a discriminator that can classify poems as haiku or not. One problem with this method is that it will likely struggle with generating cohesive, complete-thought haiku. The generator is, in essence, greedy, where it will keep generating text until the discriminator tells it to stop, at which point it is uncertain whether the generator finished a "complete thought." Given any strictly constrained format, it will be difficult to maintain cohesiveness of the content while enforcing format at sample-time. To demonstrate what I mean by an incomplete thought, here is an example of a poem generated by my previous work using a similar sample-time constraint:
Oh francois see how swift it came and nourishing food till at length slew
Slew what?
- Train on the whole corpus, but tell the network at training time and at sampling time how many syllables the line has. Even if we train on a line that has 3 or 10 syllables, my hope is that when we ask for 5 or 7 syllables at sample time that the model can generalize. This is the technique I went with.
Model
The model is essentially a character-to-character text generation network with a twist. The number of syllables for each line is provided to the network, passed through a dense layer and then added to the LSTM’s internal state. This means that by changing the three numbers provided, we can alter the behavior of the network. My hope is that this will still allow the network to learn "English" from the whole corpus even though most of the samples are not 5โ7โ5 haiku, while still allowing us to generate haiku of that length later. This python file implements the model and generator.
This Jupyter notebook shows the training process. In short, I one-hot-encode the data at the character level, split the data into training and validation sets and then use teacher-forcing to train. I trained the model for about 2 hours on a Paperspace P6000 virutal machine Although it stopped improving much sooner than that, after 4 epochs. Here is a graph of the training and validation loss over time:
Results
Finally we get to the fun part – seeing what nonsense this generates.
To generate each haiku, I first specify the number of syllables I want for each of the three lines (i.e 5, 7, 5). I then randomly pick a character for the first word and feed it into the model which outputs a probability of each character being the next character. I pick one of those characters, proportional to their probabilities and scaled by a "temperature" which affects how likely we are to pick a low-probability character. The higher the temperature, the more random. Once I’ve picked a next-character, I feed it back into the model and repeat till the end of the line. Between lines I input the next line’s syllable count to update the state of the LSTM. This function contains the generation code.
I generated all of the following using a checkpoint after training epoch 4 when the validation loss stopped decreasing.
Here are some 5โ7โ5 haiku generated with a sampling temperature of .1 (less random):
just as i walk the sand and the sun is like a crown of the spring is here
just as i deeper i want to see a bottom of the spring breeze
early morning sun from the carried garden fate stars at the sunset
While the syllable counts aren’t perfect, the cool thing is that we can see that the syllable counts we provide impacts the generated text. For example, inputting 3โ5โ3 yields:
spring sunshine the smell of waters from the stars
deep winter the smell of a crow from the stars
beach to school the smell of water in the sky
While 10โ10โ10 yields much longer lines:
god listening must have overheard the friendliness of the spring are blowing with every beat of my heart i will love you
can you see the parting of the sun i want to see the passing of the sun in the spring is left of the sunset sky
just as the street lamp spake the sun is bright and the soul and the spring are blowing with every beat of my heart i will love you
As you can see, it can be quite repetitive and formulaic. We can turn up the sampling temperature to make the model more "creative." Sampling temperature
5โ7โ5 haiku with a temperature .3:
spring evening breakfast a butterfly and i saw her face to the sea
january dawn the sound of a moth cleaning on the old path tree
zany afternoon in the moon and the contrails of the spring breeze
By .5 it starts having trouble with spelling:
keep off the highway a thousand distant village summer to the sea
every wift of sleep and fall from his orning hand and the same pillow
under the cotton of the personal hours on the compuse light
And by 1.0 it’s mostly nonsense:
zold past last buster? i never yearned mountain the winter taples
This Jupyter notebook has many more generated haiku.
Future work
This is a good start. For all the current shortcomings, we proved that by including the number of syllables as input to the model, we can influence the length of the lines generated.
There’s a lot more work that can be done to improve this. There are a couple of issues that I’d like to address:
- More consistently adhering to the 5โ7โ5 syllable form instead of just being in the right neighborhood.
- Creating more coherent and meaningful haikus
A few ideas with where to start:
- More and better data. I’d like to get more examples of 5โ7โ5 haikus, or a way to train on any text and then produce haikus from it.
- Try out other neural network architectures, such as a simpler model where I input the syllable counts all up front and then generate the whole haiku instead of a line at a time.
- Try a phoneme-level model, similar to what Hopkins and Kiela used in their work. It would make it easier to learn syllables, because only specific phonemes carry syllable weight. But it introduces the complexity of mapping phonemes back to words.
- Use a generative adversarial network (GAN) in which a discriminator can evaluate the content and/or form of the haikus.
Thanks to Abigail Pope-Brooks for editing and feedback.
All the code and data used is available on github. Jeremy Neiman can be found doing equally pointless things on his website: http://jeremyneiman.com/
