David M's profile

Ellie Goulding "Lights"

https://adage.com/creativity/work/ellie-goulding-lights/24803

In this case, we believe the best option was WebGL, a new web technology that brings hardware-accelerated 3D graphics to modern browsers without installing additional software.

WebGL is basically a JavaScript API built on top of OpenGL ES 2.0, the standard graphics library in use today for rendering 2D and 3D graphics not only on desktop computers, but also on mobile devices such as tablets and smartphones.

To unleash the power of WebGL we chose the three.js 3D engine, a powerful and easy to use library created by Mr.doob and maintained by a very talented team of developers around the world.

And although you can get started in JavaScript development using a text editor, for a project as ambitious as ours, containing over 12,000 lines of code, you need a more powerful environment. We use WebStorm, a JavaScript IDE with autocompletion, refactoring, on-the-fly code analysis and other advanced features.
As a foundation for the visuals to build up along with the music, we analysed the different stems of the song to capture the structure and energy levels for each part of the experience.
 
By overlaying the waveforms of the different stems we identify the different sections of the song
The stems were provided by Ellie's A&R and gave us a new perspective and a deeper understanding of the music. Some of them were used to sync graphic elements and others were combined to obtain high quality volume and audio spectrum data.
Our creative approach was based around the concept of flying over an infinite landscape which reacts to the sound and evolves along with the music. This allows user interaction but also delivers a great experience if the user chooses not to interact.

We used a variety of tools and techniques to create the different elements of the site, some come from traditional game development, while others are common in graphics programming. We have selected a few of them to share with you.
Terrain
The terrain is made of identical tiles which are placed in the field of view of the camera. We save considerable processing time by reusing them in different places, instead of updating each one individually. On the other hand, we had to make sure to make them seamless so they are fully tileable.

The terrain itself is generated from a 66x66 pixels height map, a greyscale image showing how high the land is. We started by creating it in Photoshop using the Clouds filter, but the resulting terrain was too steep. We obtained a more rounded landscape by applying Gaussian blur, but we knew there was still room for improvement.

We finally realised that the standard colour resolution of 256 levels of grey wasn't high enough for our needs. It required more precision. Our solution involves further softening of the height map values inside JavaScript, to produce gentle hills and rounded valleys.
 
To make it look more interesting, we wanted the ground to display different colours and shapes. To do this, we need to render a dynamic texture in real time and apply it to the landscape. We started doing it in canvas at 512x512 resolution, but the performance suffered when redrawing it on each frame. In the end, we opted for having a separate WebGL scene that's rendered onto the texture that's used for the terrain.

This terrain scene works essentially in 2D with its own orthogonal camera (which doesn't have any perspective). Here we use standard planes to create different types of lines and circles at various parts of the song. At this point, we also render the lights emitted from the other objects in the landscape, and finally we use post processing to make it seamless along the sides.
 
A range of different light emitting objects populate our environment. They are all generated by code and have various shapes, shaders and behaviours.

The spheres are interactive and synced to the beats. For their shading we use a technique called vertex colour, where colours are assigned to each vertex and the graphics hardware smoothly interpolates between them creating lovely gradients.

To fade the spheres to black and white we use a custom shader, with multiply and additive parameters. In graphics cards, RGB colour values range from 0 to 1, instead of 0 to 255. This way, if we multiply by 1 we get the same colour, but if we do it by 0 we obtain black (0,0,0). Adding works as expected, if we add 0 the colour doesn't change, but by adding 1 we get white (1,1,1) as the result is clamped.
[Additive and multiplicative operations on vertex coloured spheres]
Additive and multiplicative operations on vertex coloured spheres
Light beams
The last part of the song is very epic. Starting with a silence, it unleashes a lot of energy at once. Here's where we introduce the light beams.

To create this effect we build a mesh out of three planes rotated 120 degrees apart. Their texture is a simple gradient which is additive blended, so they are rendered on top of each other. The result is a bright stream of light which gives a nice illusion of volume.
Twitter Integration
The name and avatar image of the users who tweet about the site can be seen during the experience. The real-time tweets are powered by Echo's powerful StreamServer platform, which allows us to quickly collect, store and serve the live data with little engineering effort.

A grid of particles is used to display them, with each particle representing a pixel of the original image. To create the text in the Twitter names, we use a bitmap font converted inGlyph Designer, a Mac-only app normally used to create colour bitmap fonts for iOS games. It exports an image with all the characters and a font description file in Cocos2D .fnt format which, being just a text file, is easy to parse in JavaScript.
We believe real-time 3D graphics are particularly suited to enhance music, allowing users to enjoy it in new and exciting ways by creating a richer experience with immersive visuals and interaction.

Although it's important to remember that graphics are meant to serve the experience and no amount of technical wizardry can replace well defined aesthetics, there is little doubt that the advent of hardware accelerated 3D in the web and mobile brings new levels of expressiveness. We have exciting times ahead.
Ellie Goulding "Lights"
Published:

Ellie Goulding "Lights"

A new interactive music experience starring British singer-songwriter Ellie Goulding. Besides creating an exciting, technologically forward visua Read More

Published: