Alongside a paper published jointly with the The Gebrew University of Jerusalem, Google Research Group recently showcased its newest AI tool, Dreamix on its Github page.
Dreamix, a text-driven video diffusion model, as Google puts it, takes either a video or an image as its input, and works its magic to create a video after you direct it with your text prompts.
What is a “Diffusion Model”?
We have been hearing a lot about diffusion models when it comes to image / video creation and you may be wondering what is it. In simple terms, diffusion models in AI are generative models, and they are trained by first destroying an image they are feed by creating noise. After the destruction, they try to rebuild the images from the noise. And they do it millions of times to improve their creation abilities. Dreamix is a text-conditioned video diffusion model, meaning it uses both text and videos/images to generate outputs. In the published paper, researches note possible future applications as motion interpolation between images and text guided painting.
But what does that mean for us mere mortals. The premise is pretty simple, really. Do you want to make a video of a teddy bear running on the street? No problem, input a photo of a teddy bear into the tool and write the prompt “a bear walking”, and voilà, your walking teddy bear video is ready. Well, not quite yet. Although the tool really makes your favorite teddy bear to walk on a street with a prompt, the tool is not yet publicly available.
The reason is pretty simple (and kind of scary, really): Media generation tools are pretty new and potential malicious use cases are not thoroughly researched. In the research paper, the contributors to this tool raised their concerns:
Malicious parties may try to use edited videos to mis-lead viewers or to engage in targeted harassment. Future research must continue investigating these concerns.
Now the good news! Dreamix has definitely a lot of potential to be helpful to many people a great deal. Video editing has somewhat been an area with a steep learning curve, and it is only becoming mainstream in the last few years due to many easy-to-use video editing apps popping up on iOS / Android App Stores. And now with the ability to generate videos with just 1 picture and a text prompt takes this ease-of-use to next level.
Dreamix accepts videos and images to supplement the text prompt, but here are some creative examples from the demo page on Github:
Creative Uses of Dreamix
1. Enhance a Video Scene
Dreamix can be used to enhance a scenery. No we’re not talking about Facetune type of beauty make-up. Dreamix properly re-creates the scene, and blows life into videos, as shown in the tropical island video below:
2. Animate Still Objects
Do you remember mobile apps that allows you take many videos of your Lego characters and then stitches them to make a short animation. Well, Dreamix can do the video with just a few pictures. And, as the icing on the cake, it can add elements that are not in the original photos. Take a look at this example of a figure doing weight-lifting:
3. Switch the Subject Animal With Another
Did you ever wish your cat to make the suave dancing moves of your neighbour’s dog? Well, it seems that you might just be able to create a video of. See Dreamix at work:
4. Change the Subject in A Video
This one is particularly interesting as it might literally change the food preparation videos. See below how a pot full of onions are switched with onions:
5. Create Timelapse Videos
Another particular video we were impressed by Dreamix is how it can create a timelapse video a plant growing.
We have seen a lot of text generators (ChatGTP), and image creators (Dall-E, Midjourney) in the last couple of years, but this is the first tool that uses text prompts to create and direct videos. While this tech is new, it is still a bit far from mainstream distribution, mainly first because of the computation costs (a short video will need 30 minutes of processing time), and second the ethical considerations as we discussed above. Still we are curious what these types of tools will bring to the table (of Youtube creators mostly?)
