shutterstock_401295613

AI for Creative Content Automation: How far have we gone?

How can we make algorithms that generate a new image, video or music? For many researches, a natural extension of artificial intelligence is artificial creativity and there is a titanic ongoing effort on the machine learning community to make algorithms that can generate new information in meaningful and creative ways.

In this post we review the current state-of-the art in Machine Learning for creative content generation, going through the hotest developments from the computer science community on automatic generation of text, images, video and speech.

Generative Algorithms

Generative algorithms allow us to create synthetic data "out of thin air". They are "trained" on a set of examples provided the by the user and may generalize over the provided examples to produce new examples that aren't part of the data. These algorithms can be applied to any form of data such as images, speech, text and video.

However it is incredibly hard to scale this algorithms even to medium sized images and videos. In the last 5 years, due to a rapid increase in computational power and several innovative new algorithms based on Deep Learning, the field has seen an enormous progress.

These generative algorithms are finally getting to a point where they can become useful to generate content. These are very exciting times.

Below we show examples of images, video, speech and text generated by various algorithms developed in the last 5 years.

 

Image and Video Generation

The human vision system is an incredibly sophisticated machinery that, through a combination of evolution and learning, can capture a lot of the rich statistics of the world around us. This sophistication of our own visual systems means that generating believable and interesting images artificially is a very hard problem. This problem has eluded researchers in machine learning and computer vision for many years. In the last few years we have seen a real explosion in the diversity, quality and complexity of image generation algorithms. Some of them can even pass a simple form of visual Turing test which you can verify for yourself here.

Checkout our References List or continue below to see a few remarkable examples of unique images generated by machine learning algorithms.

Google Inception Network

Iterative_Places205-GoogLeNet_buildings
Image from [1]

Facebook DCGAN Generative Model

Google DeepMind DRAW Generative Model

Screen Shot 2016-04-23 at 3.51.10 PM
Image from [5]

Audio and Speech Generation

As with images and video, generating believable speech and music is a very hard task.

Speech has many applications across various industries world-wide, so it is not surprising that companies like Google [16] and Microsoft [14,15] invest a substantial amount of resources in modelling speech.

Music has a long history with machine learning [18], but only recently learnable algorithms can generate meaningful music fragments. Checkout our References List or continue below to see a few interesting examples.

Music Generation (Example from [17])

Speech Synthesis (Examples from [14])

 

 

 

 

Text Generation

Generating  plausible and meaningful text is still an open problem, however machine learning has gone a long way from simple hand-crafted grammar rules and chatbots systems to algorithms that learn grammar and a semantics from scratch by going through thousands of pages of text data (for example, from Wikipedia or Shakespeare).

These systems are being already used for text-translation [12], free-style text-generation [6,7], Obama speeches [9], Donald Trump quotes [13], clickbait titles [10] and even TED Talks [8,11].
Checkout our References List or continue below to see a few interesting examples.

 

Example of Obama speech from [9]

Good afternoon. God bless you.
The United States will step up to the cost of a new challenges of the American people that will share the fact that we created the problem. They were attacked and so that they have to say that all the task of the final days of war that I will not be able to get this done. The promise of the men and women who were still going to take out the fact that the American people have fought to make sure that they have to be able to protect our part. It was a chance to stand together to completely look for the commitment to borrow from the American people. And the fact is the men and women in uniform and the millions of our country with the law system that we should be a strong stretcks of the forces that we can afford to increase our spirit of the American people and the leadership of our country who are on the Internet of American lives.
Thank you very much. God bless you, and God bless the United States of America.

 

Example of Shakespeare text from [7]

PANDARUS:
Alas, I think he shall be come approached and the day
When little srain would be attain'd into being never fed,
And who is but a chain and subjects of his death,
I should not sleep.Second Senator:
They are away this miseries, produced upon my soul,
Breaking and strongly should be buried, when I perish
The earth and thoughts of many states.DUKE VINCENTIO:
Well, your wit is in the care of side and that.Second Lord:
They would be ruled after this chamber, and
my fair nues begun out of the fact, to be conveyed,
Whose noble souls I'll have the heart of the wars.Clown:
Come, sir, I will make did behold your worship.VIOLA:
I'll drink it.

 

Examples of baby names from [7]

Rudi Levette Berice Lussa Hany Mareanne Chrestina Carissy Marylen Hammine Janye Marlise Jacacrie Hendred Romand Charienna Nenotto Ette Dorane Wallen Marly Darine Salina Elvyn Ersia Maralena Minoria Ellia Charmin Antley Nerille Chelon Walmor Evena Jeryly Stachon Charisa Allisa Anatha Cathanie Geetra Alexie Jerin Cassen Herbett Cossie Velen Daurenge Robester Shermond Terisa Licia Roselen Ferine Jayn Lusine Charyanne Sales Sanny Resa Wallon Martine Merus Jelen Candica Wallin Tel Rachene Tarine Ozila Ketia Shanne Arnande Karella Roselina Alessia Chasty Deland Berther Geamar Jackein Mellisand Sagdy Nenc Lessie Rasemy Guen Gavi Milea Anneda Margoris Janin Rodelin Zeanna Elyne Janah Ferzina Susta Pey Castina

 

Example of a whole website generated by Recurrent Neural Networks from [10]

 

Discussion

There is an enormous effort from the machine learning and computer science community to develop better algorithms and methods to generate images, videos, text and speech and they have gone a long way in these directions.

It seems that the different pieces of the "content automation" puzzle are starting to crystallize as powerful independent toolboxes. Now it is time to integrate these different technologies and algorithms into something even more powerful.

Generating high-quality and meaningful automated content is still very hard. Such challenges are now being embraced by a few startups such as the The Grid.

One can easily extrapolate that in the near future such algorithms will generate the vast majority of the textual, visual and auditory content we will experience.  Even converting an entire book into a movie or producing short synthetic video-clips from text is a plausible outcome.

Back to Top

 

Want to try for yourself?

As we hope to have demonstrated so far, generating high-quality and useful content with these amazing machine learning techniques is still an open research question.

To our benefit, most of these research efforts produce open-source code that anyone can try. Checkout our reference list below for the links of all github repositories mentioned in this post.

Note: Running these algorithms requires large scale computing systems, typically involving multiple GPUs and CPUs, please beware of the requirements before getting too excited.

 

References

[1] Inceptionism

[2] Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks

[3] DCGAN github

[4] chainer-DCGAN

[5] DRAW

[6] Generating Text with Recurrent Neural Networks

[7] The Unreasonable Effectiveness of Recurrent Neural Networks

[8] TED-RNN — Machine generated TED-Talks

[9] Obama-RNN — Machine generated political speeches.

[10] Auto-Generating Clickbait With Recurrent Neural Networks

[11] TED-RNN - Machine generated TED-Talks

[12] Neural Machine Translation by LISA

[13] DeepDrumpf

[14] DEEP NEURAL NETWORK (DNN) FOR TTS SYNTHESIS

[15] Deep Learning for Speech Generation and Synthesis

[16] Long Short-Term Memory Recurrent Neural Network Architectures for Large Scale Acoustic Modeling

[17] Asking RNNs+LTSMs: What Would Mozart Write?

[18] Computer Models of Musical Creativity

 

Back to Top

One comment

Leave a Reply

Your email address will not be published. Required fields are marked *