The self-realisation of a tree (2017) is an ongoing collaboration between Matt Maurer and Guillaume Massol.

Coming soon...

I love you for the day (ILYFTD) is an installation by Matthias Maurer and Guillaume Massol situated at the unlikely intersection between mass surveillance and poetry.

The installation is looking around, focusing from time to time on passers-by. Once focused on a person it’s tracking his/her face and breaks it down into fragments, generating a constantly evolving mosaic of eyes, mouthes and noses.

As it watches it is generating lyrics and music based on facial features. While on idle mode it reload bits of previously tracked faces and keeps on generating soundscapes and lyrics.

The face tracking algorithm is built on a Convolutional Neural Network. It uses a model trained on thousands of different faces.

Text and music are created by a multi-layer Recurrent Neural Network. A model trained on love songs generates the text of the installation character by character. While another model – trained on heavy metal – generates the melody of the music which is slowed down and played with electronic instruments.

The name of the installation has also been generated as well as the music of the video.

Machine daydreaming...

The app is watching videos coming from different training datasets. As it watches the videos it generates sentences loosely based on what is happening on the screen. Sometimes creating pearls of wisdom by coincidence.

Visit that page if you want to see the final result, alternatively you can watch a video extract of the project below.

Some of the text are generated using the models created by Ross Goodwin for his NeuralSnap project.

Produce a "smart entity" by harnessing intensive differences of matter (i.e. differences of intensive material properties).

Intensive differences drive processes. For another short project we were asked to produce an entity that can only exist/make sense/function within the field of the poles of intensive material properties (temperature, taste, pain, gravitation, pressure, density, etc.)... The video below is what I've produced, the news feed and video effects are generated in real time while the video was pre-recorded:

Extensive Indifference

Experiments in music creation using RNN.

The "classic" technic to generate music with a RNN is to aggregate songs in ABC notations and train a model on those. Since ABC notation is "just" a succession of letters and symbols we can get a lot of musical data in a text file weighting only a few Mb. This allow us to quickly train some models even on a computer without GPU(s).

I've ran different kind of experiments, starting with simple ABC files with one track and uncomplicated melodies to more complex structures.

When training the model on simple ABC files (one track, basic melodies) the RNN manages to understand the structure very quickly and generate nice tunes without much efforts. For example those are early examples produced by a network trained on religious hymns:

Another example is this set produced by a model trained on national anthems. Check the 2nd track called "Sheetmusic + NatAnthems - Jirnni" where the model made an impro on the US national anthems "à la Jimi Hendrix!"

Using the same logic I've tried training a model on songs with multiple instruments and tracks but the result was very average. When feeding all tracks together the results where very often quite weird and not very usable out of the box. For example the jazz songs below really sounds like they are played by a jazz band with 2 neurones, literally...

There are many abc files available but there are mostly folk songs, so to train some new models I had to get some MIDI files and convert them into ABC notation. Even though not many people are using those anymore you still can find a lot of them online. This one in particular is quite a 💎. Once I had enough files I split the tracks by instrument and train different models for each type of instruments. I wrote a small Python script for this that you can get here. I'll post a longer post on the process later.

After training models on different instruments I got some pretty decent samples and could combine drums + guitars + bass more easily. Here are some of the latest samples trained on heavy metal songs.

While the first models (hymns, national anthems, etc...) produced only maybe 1/3rd of "ready to use" samples (probably because the training sets were small and data were fed "as if" to the RNN) The last models trained separately on different instruments produce samples that are nearly always spot on and ready to be use in a midi-file.

What's next:

• Create a metal machine that can generate metal tracks endlessly 🤘
• Add voice (??)

Bands used for the training

Variation on a theme for a design culture workshop. The verses are generated by a RNN trained on British poetry.

A good neighbour is left out of sight,
and in the next room he steps his face like a ghost’s skin.

A good neighbour is a good song:
The last of my mother is a dream of flesh and all
is a bird that makes me think that the world can make me feel so bad.

A good neighbour is a dog on the street
who says that he is a conscience of the state of the world.

A good neighbour is not relieved to be essential,

A good neighbour is a shout
Of some master in the stormed winds,
In the wild and busy gold and silent sea

A good neighbour is a spirit,
And which she could see him the new stars
Of what a sorrow and red light restore,
The folded of the soul of a sea of head.

A good neighbour is walking on the threshold.

A good neighbour is before the blood,
And the silent shadow of the world and the singing

A good neighbour is the sound of the world.

A good neighbour is who shall see
The stream of the street and fear in the old shadow.

A good neighbour is dead.
The sun is gone.
The light is past the sky.

A good neighbour is before the beauty.
The falling of his face she said.
The world of any house is day.

A good neighbour is the wind,
And the soul in the fire and shadow of the street,

A good neighbour is a family without pets.

Assemble, combine, compile, translate 12 discerned and randomly gathered objects by one of your peers into a single rigorous and comprehensive narrative

This is a brief for a short project we worked on. It was my first attempt to experiment with ML. The project is broken down in 3 'small' experiments:

• 01 - Objects-Viewer
• 02 - Image “Painting”
• 03 - Trained convnet (App)

01 - Objects-Viewer

OFX app based on the ConvnetViewer from ml4a-ofx package. I've added a real time object detection. The objects are detected by the app then the different stages of the process are visible on the left hand side.

02 - Image "Painting"

Those visualisations are rendered using a package called CatsEyes and the Convnet JS framework from Andrej Karpathy

03 - Trained convnet (App)

This part use Tensorflow. I've basically followed this guide and trained a model on top of Google Inception on my 12 objects. I then included the model into an iOS app which can recognise the objects. The results are quite impressive, even an object which is nearly completely masked is still accurately detected. You can view a video of the app in action here

Preview of the dataset used to train the model. (hd version)

Presentation