Flow matching optimization yields strong results but sometimes causes duplicated subjects during motion or orientation changes. To mitigate this, we analyze the signals in the frequency domain. By leveraging the fact that spatial displacements cause phase shifts in the Fourier domain, we introduce a phase constraint to improve the spatiotemporal consistency of motion transfer.
A white van cruises down the road, passing by a substantial edifice situated in the distance.A vehicle is observed speeding along a racing circuit, generating thick clouds of smoke that trail behind it as it proceeds.The video shows a duck waddling along a rocky beach at low tide and then soaring above a serene saltwater lake surrounded by misty mountains.
Phase vs Magnitude
We ablate two regularizer for magnitude and phase. Using phase constraints as regularizer yields superior results.
A woman rides a bike down a wooden dock alongside a serene lake at sunrise.A silver motorhome is seen pulling up to a campsite nestled among the towering redwoods of a misty forest, with a dense canopy above it.
Limitations
Demonstration of current shortcomings and broader impacts. Hover over each clip to preview the effect. In the video a boat in the background intermittently appears and disappears, likely due to limitations in the backbone network.
A person is seen water skiing in the ocean, being pulled by a boat.
Comparisons
The following results showcase the comparison of RoPECraft across a wide range of reference motion videos. Hover over each clip to preview the generated motion transfer in action. In each video, the original video is on the top row labeled as "DAVIS" and the generated video is on the other 2 rows labeled as the name of the method. The prompt used for each video is also provided at the bottom.
A silver motorhome is seen pulling up to a campsite nestled among the towering redwoods of a misty forest, with a dense canopy above it.A woman wearing a black dress walks down a worn wooden dock along the edge of a misty lake at dusk.A large group of fire dancers are spinning together in a circle, with one seasoned performer leading the mesmerizing display under the warm glow of string lights on a tropical beach at sunset.The vintage motorcycle drives down a dirt road, passes a truck, and then speeds up as it goes.A sailboat is anchored in a tranquil cove surrounded by lush greenery, as the man walks along the weathered wooden dock, gazing out at the serene ocean waters while the sailboat glides smoothly alongside him.A woman rides a bike down a wooden dock alongside a serene lake at sunrise.
CogVideoX Results
Following results showcase the generalization of RoPECraft to other models. The first video is from the DAVIS dataset, second video is from RoPECraft built on CogVideoX, third video is from Go With The Flow and last one is from DiTFlow. The prompt used for each video is also provided at the bottom. Hover over each clip to preview the generated motion transfer in action.
As we can see, Go With The Flow struggles with object shapes or multiple subjects since it is tied to the original video which the noise was extracted from. DiTFlow on the other hand fails to transfer motion in some scenarios. Our method is able to transfer motion in all scenarios and is able to generalize to other models.
A herd of antelope dashes across the dirt path at twilight against a backdrop of trees.A medieval knight in armor strides along the rocky path, sword at the ready.A herd of llamas moves through a dry, enclosed plateau in the Andes.A black dog strolls calmly down a tree-lined park path.A remote-controlled drone hovers low over the yard, capturing aerial footage.A black taxi drives down a street in a desert outpost as dust swirls around.A classic car cruises through a town square.A moped rider weaves through a narrow path in a tropical mountain village.A public bus drives past a snow-covered rural village.
More Results
Supplementary videos showcase ∼30 more generations across complex trajectories. Hover over each clip to preview the generated motion transfer in action.
Children, one in a duck costume, paddles a small inflatable boat across a pond.A group of remote-controlled submarine models navigates a large aquarium, their periscopes extending and retracting as they move.A flock of black swans swims gracefully through a misty, moonlit swamp, their dark forms contrasting with the eerie glow on the water.A collection of origami swans, enchanted to life, delicately paddles across a reflecting pool in a serene Japanese garden, their paper necks craning with surprising fluidity.
A wooden rowboat drifts slowly across a lily-covered pond.A towboat pushes a barge filled with shipping containers through the harbor.A group of children as captains imagines their rubber ducks are a mighty fleet of galleons sailing across a vast, soapy ocean.
A futuristic maglev pod glides along an elevated track above a bustling metropolis.A herd of sheep marches along the street while a shepherd watches from the sidewalk.A vintage biplane loops gracefully above a sunlit airfield, propeller churning wind.
A police car speeds through a coastal roundabout flanked by palm trees.A miniature car made of cheese, driven by a team of mice, speeds around a tiny roundabout on a kitchen counter, avoiding crumbs.
A vintage muscle car cruises through a foggy pine forest road.The silhouette of a ballerina leaps across a sunlit studio, delicate shadow dancing on polished hardwood floors.
A cyclist pedals hard around a hairpin turn on a sunny hillside road.A sports car speeds through switchbacks with ocean cliffs visible below.A robotic courier zips through a maze of eco-friendly, automated warehouses, packages whizzing past on conveyor belts.
A solar-powered concept vehicle glides silently down the street, its panels gleaming.A vintage steam locomotive rolls past a snowy station, its black iron body steaming in the frosty air.
A horse-drawn sleigh skids dramatically around a frozen bend on an icy track.A rugged off-road buggy powers through a muddy forest trail, throwing up clods of earth.A sleek electric skateboard zooms around the circuit, leaving neon trails in its wake.
A firefly swarm flickers along the path as a silhouette in a flowing dress glides past.
A majestic eagle soars high above a mountain range, searching for prey in the valleys below.
A glider plane soars above the town.A group of wizards on flying carpets, patterned in red and yellow, gently descends towards a hidden magical village nestled in a valley.
A man rides a small scooter in an underground parking lot.A young woman riding a small skateboard through the misty rainforest.
A solitary robot with LED eyes ambles along an abandoned urban plaza, recording its surroundings.A band of elves in shimmering blue tunics walks through an enchanted forest, narrating their magical discoveries to a crystal ball that records everything.
A line of fairytale princesses, disguised as commoners, pushes simple wooden carts (acting as makeshift strollers for royal babies) through a bustling medieval marketplace.
A fearless downhill mountain biker hurtles down a rocky canyon trail, dust plumes.A competitive mountain biker races across the edge of an erupting volcanic crater, molten lava glowing beneath the rugged path.An interstellar rally craft close to ground zooms through, thrusters blazing as it weaves between spinning rock fragments.
A dromedary camel traverses an enclosed pen, walking steadily.A council of mythical creatures, part camel part sphinx, majestically paces within the sacred, walled gardens of a lost desert city.A parade of giant clockwork camels, part of a whimsical festival, mechanically walks across a stage enclosed by velvet ropes.