Luma Labs’ new Modify Video tool can reimagine scenes without reshooting erichs211@gmail.com (Eric Hal Schwartz) | usagoldmines.com

• Luma Labs’ new Modify Video tool for Dream Machine uses AI to alter any video footage without reshoots
• Any characters or environments won’t lose their original motion or performances
• Anything from subtle wardrobe tweaks to full magical scene overhauls is feasible

Luma Labs is known for producing AI videos from scratch, but the company has a new feature for its Dream Machine that can utterly transform real video footage in subtle or blatant ways, even if it’s just an old home movie.

The new Modify Video feature does for videos something like the best Photoshop tools do for images. It can change a scene’s setting, style, even whole characters, all without reshooting, reanimating, or even standing up.

The company boasts that the AI video editing preserves everything that matters to you from the original recording, such as actor movement, framing, timing, and other key details, while altering anything else you want.

The outfit you’re wearing, which you’ve decided wasn’t you, is suddenly an entirely different set of clothing. That blanket fort is now a ship sailing a stormy sea, and your friend flailing on the ground is actually an astronaut in space, all without the use of green screens or editing bays.

Luma’s combination of advanced motion and performance capture, AI styling, and what it calls structured presets makes it possible to offer the full range of reimagined videos.

All you need to do is upload a video of up to 10 seconds in length to get started. Then pick from the Adhere, Flex, or Reimagine presets.

Adhere is the most subtle option; it focuses on minimal changes, such as the clothing adjustment below or different textures on furniture. Flex does that but can also adjust the style of the video, the lighting, and other, more obvious details. Reimagine, as the name suggests, can completely remake everything about the video, taking it to another world or remaking people into cartoon animals or sending someone standing on a flat board into a cyberpunk hoverboard race.

Flexible AI video

It all depends on not just prompts, but reference images and frame selections from your video if you choose. As a result, the process is much more user-friendly and flexible.

Although AI video modification is hardly unique to Luma, the company claims it outperforms rivals like Runway and Pika due to its performance fidelity. The altered videos keep an actor’s body language, facial expressions, and lip sync. The final results appear as an organic whole, not just stitched-together bits.

Of course, the Modify Video tools have limitations. These are still capped at 10 seconds per clip for now, which keeps things manageable in terms of wait times. However, if you want a longer film, you need to plan and work out how to artistically incorporate different shots into one film.

Still, features like the ability to isolate elements within a shot are a big deal. Sometimes you have a performance you’re very happy with, but it’s supposed to be a different kind of character in a different setting. Well, you can keep the performance intact and swap a garage for the sea and your actor’s legs for a fish tail.

Dreams to reality

It is genuinely impressive how quickly and thoroughly the AI tools can rework a bit of footage. These tools aren’t just a gimmick; the AI models are aware of performances and timelines in a way that feels closer to human than any I’ve seen. The AI models don’t actually understand pacing, continuity, or structure, but they are very good at mimicking these aspects.

While the technical and ethical limitations will prevent Luma Labs from recreating the entire cinema at this point, these tools will be tempting for many amateur or independent video producers. And while I don’t see it becoming as widely used as common photo filters, there are some fun ideas in Luma’s demos that you might want to try.

You might also like…

• This movie is fully AI-generated and has a fully SAG-AFTRA cast – here’s 3 things you need to know about Echo Hunter
• Sora needs to up its game to match the new Runway AI video model
• I tried adding audio to videos in Dream Machine, and Sora’s silence sounds deafening in comparison