Google Deepmind recently introduced V2A (Video to Audio) technology to break the monotony of the fast-growing silent video generation system. According to a recent blog post, this new large language model can generate soundtracks and dialogues for videos. It combines video pixels with natural language text prompts to generate rich soundscapes for the on-screen action. Scroll down to read more about the new V2A (Video to Audio) technology, its uses, and the complete working mechanism.
What is V2A (Video to Audio) technology?
V2A is a large language model that makes synchronised audiovisual generation possible. It can be used to add dramatic music, realistic sound effects, and dialogue that matches the video’s tone with natural language text prompts. Google says the new large language model also works with “traditional footage” like silent films and archival material. According to Google Blog, “V2A technology is pairable with video generation models like Veo to create shots with a dramatic score, realistic sound effects, or dialogue that matches the characters and tone of a video.
With enhanced creative control, V2A generates an unlimited number of soundtracks for any video input with a ‘positive prompt’ and a ‘negative prompt’. This flexibility gives users more control over V2A’s audio output, making it possible to rapidly experiment with different audio outputs and choose the best match.
Perplexity VS. Gemini: Which One Is Better? Check Here!
How does V2A work?
Google Deepmind video-to-audio research uses video pixels and text prompts to generate rich soundtracks. The diffusion-based approach for audio generation gave the most realistic and compelling results for synchronizing video and audio information.
The V2A system starts by encoding video input into a compressed representation. Then, the diffusion model iteratively refines the audio from random noise. This process is completely guided by the visual input and natural language prompts given to generate synchronized, realistic audio that closely aligns with the prompt. Finally, the audio output is decoded, turned into an audio waveform, and combined with the video data.
Also, Google aims to improve lip synchronization for videos that involve speech with V2A from the input transcripts.
At present, V2A technology is undergoing rigorous safety assessments and testing. To make sure V2A technology can have a positive impact on the creative community, Google gathered diverse perspectives and insights from leading creators and filmmakers and used this valuable feedback to inform our ongoing research and development. Also, it incorporated our SynthID toolkit to watermark all AI-generated content to help safeguard against the potential for misuse of this technology.
