Echoes

3D based - audioreactive - AI generated video

Prompts: storyboard [3D clay renderings – text – audio]

Echoes is an experimental attempt to craft a narration based on specific storyboard guidelines and camera settings. The audio is generated through sample-based methods, wherein an AI model utilizes a library of audio samples to create new sounds. These samples can be combined in diverse ways to produce melodies, rhythms, and harmonies. However, most of the sound design was done in Audition and then adjusted to align with the visual elements.

The process of creating an audioreactive video involves several steps:

Audio analysis: The audio track is examined to extract its key characteristics, such as pitch, tempo, and rhythm.

Parameter mapping: The extracted audio features are mapped to parameters of a video synthesis model. For instance, pitch might be linked to the speed of a particle system, tempo might be associated with the intensity of a lighting effect, and rhythm might be coupled with the movement of objects in the scene.

Video synthesis: The video synthesis model is iteratively executed, producing new frames of video based on the current audio input.

Synchronization: The video frames are synchronized with the audio track to ensure that the visual effects complement the rhythm and melody of the music.

At this stage of experimentation, the AI-aided video has achieved the following:

Mesh deformation: Mesh deformation is utilized to alter the form of 3D objects in response to the audio track.

Optical flow: Optical flow is applied to monitor the movement of objects in the scene and generate a sense of movement.