From Pixels to Humanoid Motion: A Technical Pipeline for Video-to-Robot Learning
I have been exploring computer vision, human-motion reconstruction and simulation because I keep coming back to one technical question:
Can observable human behavior be converted into a representation that another physical body can execute?
The simplified pipeline I am interested in looks like this:
RGB video
→ 3D human reconstruction
→ world-space motion
→ contacts + task constraints
→ motion representation
→ robot retargeting
→ physics-valid reference motion
→ whole-body control
→ simulation
→ sim-to-real
→ physical humanoid
At first this may look like pose estimation followed by animation.
It is much harder than that.
The main reason is simple:
Pixels are measurements, not physical state.
A monocular RGB video does not directly tell us depth, scale, forces, torques, contact state, friction, object mass, task intention or the geometry hidden behind a person.
Almost everything after the pixels has to be estimated, inferred or learned.
1. Start With Pixels
A video is essentially a sequence of image frames:
I₁, I₂, I₃, … Iₜ
where T is the total number of frames in the video.
Each frame is a grid of pixels. For a normal RGB image, its shape can be thought of as:
H × W × 3
where H is the image height, W is the width, and 3 represents the red, green and blue color channels.
For example, a 1920 × 1080 RGB frame contains:
1080 × 1920 × 3 numerical color values.
Before the system understands a person, an arm, an object or an action, this numerical pixel data is essentially where everything starts.
To a human:
pixels → "person picks up a box"
For a computer vision and robotics pipeline:
pixels
→ person detection
→ camera estimation
→ 3D human reconstruction
→ scene reconstruction
→ object tracking
→ contact estimation
→ motion understanding
→ task understanding
→ robot control
That gap is one of the central problems.
2. Recover the Human in 3D
The first useful intermediate representation is an articulated 3D human.
Instead of treating the person as millions of unrelated RGB values, we can represent the body using a skeleton or a parametric body model such as SMPL.
At a simplified level, one reconstructed frame might look like:
human_t = {
"root_position": [x, y, z],
"root_orientation": [qw, qx, qy, qz],
"joint_rotations": [...],
"body_shape": beta,
}
Now we have something more useful than pixels.
We can reason about the pelvis, hands, feet, head, torso, joint rotations and body trajectory.
But another problem appears immediately:
camera motion.
Imagine a person standing still while the camera moves backward.
In image space, that can look similar to a person walking forward while the camera remains stationary.
So ideally we need to estimate both:
human motion
+
camera motion
and recover the movement in a stable world coordinate system.
Without that, the motion may look correct relative to the camera while being completely wrong relative to the physical world.
3. A Skeleton Is Not Enough
A 3D skeleton is useful, but for Physical AI it is probably too weak.
A more useful representation could combine geometry, motion, contact, task and individual style.
For example:
motion_state = {
"root_pose": ...,
"joint_pose": ...,
"joint_velocity": ...,
"left_hand_pose": ...,
"right_hand_pose": ...,
"left_foot_contact": True,
"right_foot_contact": False,
"object_states": ...,
"active_contacts": ...,
"task_phase": ...,
"task": ...,
"style_embedding": ...
}
I think it helps to separate the information into several layers.
Geometry: Where are the human, objects and environment?
Kinematics: How are they moving?
Dynamics: What forces and physical conditions make that movement possible?
Contacts: Which body parts are touching the ground or an object?
Task: What result is the person trying to achieve?
Style: How does this particular person tend to perform the behavior?
These layers are connected, but they are not the same thing.
4. Why Pose Alone Cannot Describe a Skill
Imagine a video of somebody pushing a heavy door.
Suppose we perfectly recover:
hand position
elbow position
shoulder position
door trajectory
We still do not necessarily know how much force was required.
The same visible trajectory could come from a very light door or a very heavy one.
The video gives us the visible result of the physics.
It does not directly tell us all the hidden physical variables that produced that result.
This makes video-to-robot learning partly an inverse problem:
observe the effects
→ infer the hidden causes
Those hidden causes may include:
object mass
friction
contact forces
joint forces
material properties
human intention
Repeated demonstrations, known object models, force sensors, scene priors, robot interaction and learned world models can all help constrain the problem.
5. Human Skeleton != Robot Skeleton
Suppose we reconstruct the human perfectly.
This still does not work:
robot_joint_angles = human_joint_angles
A human body and a humanoid robot have different limb lengths, joint limits, degrees of freedom, mass distribution, actuator characteristics and balance constraints.
A robot such as the Unitree G1 is mechanically very different from a biological human.
So copying joint angles is not enough.
We need motion retargeting.
Instead of copying every human joint directly, we can preserve important task-space targets:
pelvis
head
left hand
right hand
left foot
right foot
Conceptually, the retargeting problem is:
Find robot joint configuration q
that makes important robot body points
match the human targets as closely as possible,
while respecting robot constraints.
A simplified objective might look like:
minimize:
target tracking error
+ motion smoothness error
+ joint-limit penalties
+ collision penalties
In code-like form:
def retarget(human, robot):
targets = {
"pelvis": human.pelvis,
"left_hand": human.left_hand,
"right_hand": human.right_hand,
"left_foot": human.left_foot,
"right_foot": human.right_foot,
}
q = solve_inverse_kinematics(
robot=robot,
targets=targets,
joint_limits=True,
collision_constraints=True,
smoothness=True,
)
return q
But even if this motion looks perfect, the robot may fall immediately once physics is enabled.
6. Kinematic Validity vs Dynamic Validity
This distinction matters a lot.
A movement can be:
kinematically valid
but:
dynamically impossible.
Inverse kinematics may place the robot’s hand and feet exactly where we want them.
Physics asks additional questions.
Can the motors generate enough torque?
Is the robot’s center of mass recoverable?
Does the foot have enough friction?
Is the robot accelerating too aggressively?
Are the required joint velocities possible?
Does the movement violate actuator limits?
Now we have to think about:
joint position limits
joint velocity limits
joint torque limits
center of mass
ground reaction forces
friction
contact timing
momentum
self-collision
actuator dynamics
This is where motion stops being animation and becomes robotics.
7. Contacts Matter More Than They First Appear
Walking gives a simple example.
Pose reconstruction might tell us:
left_ankle_position(t)
But the robot controller also needs to know whether the foot should be touching the ground:
left_foot_contact(t) = 0 or 1
When the foot is planted, its velocity relative to the ground should ideally be close to zero:
foot_velocity ≈ 0
Without contact reasoning, reconstructed human motion often produces foot sliding.
A simple contact representation could look like:
contacts = {
"left_foot": 1,
"right_foot": 0,
"left_hand": 0,
"right_hand": 1,
}
Contact becomes even more important for sitting, lifting, pushing, climbing stairs, opening doors, tool use and getting up from the ground.
Once objects enter the scene, human pose alone is no longer enough.
We need human-object interaction.
8. Reference Motion Is Not a Controller
Suppose retargeting gives us a robot-compatible reference trajectory:
q_ref(t)
That still does not mean the robot can execute it.
A simple joint controller could use proportional-derivative control:
torque =
Kp × position_error
+
Kd × velocity_error
Or more compactly:
τ = Kp(q_ref - q) + Kd(dq_ref - dq)
Here:
τ = joint torque
q = current joint position
q_ref = target joint position
dq = current joint velocity
dq_ref = target joint velocity
For more complicated humanoid behavior, we may instead train a whole-body control policy.
The observation could contain:
observation = {
"joint_positions": q,
"joint_velocities": dq,
"imu": imu,
"root_velocity": root_velocity,
"contacts": contacts,
"reference_motion": reference[t],
}
The policy produces:
action = policy(observation)
Depending on the architecture, the action might represent:
target joint positions
target velocities
joint torques
residual corrections
A simplified control loop looks like:
for t in range(T):
state = robot.observe()
target = reference[t]
action = controller(
state=state,
target=target,
)
robot.step(action)
The important point is that the controller continuously reacts to errors caused by physics.
It is not simply replaying animation frames.
9. The Missing Embodiment-Independent Layer
The representation I find most interesting is not:
human
→ exact robot commands
but something closer to:
human
→ abstract physical behavior
→ embodiment-specific controller
→ robot
Instead of preserving exact human joint angles, we preserve what actually matters for the action:
desired hand trajectory
foot contact sequence
relative object position
body orientation
center-of-mass behavior
task phase
movement rhythm
style
Then each robot solves those constraints using its own mechanical structure.
This could make physical behavior portable across different embodiments.
A Unitree G1, another humanoid, a virtual avatar and a future robot do not need identical skeletons.
They need a common representation of what the movement is trying to accomplish.
10. Separating Motion Content From Style
Another part of this problem interests me even more.
Two people can perform the same action while moving very differently.
So conceptually I think about motion like this:
Motion = f(Content, Style, Embodiment)
Where:
Content = what needs to happen
Style = how this person tends to move
Embodiment = the body executing the action
For example:
Content = walk forward five meters
Style might contain:
cadence
posture
arm swing
stride characteristics
torso movement
acceleration pattern
turning behavior
Embodiment could be:
human
Unitree G1
another humanoid
virtual avatar
Instead of trying to reproduce the human motion exactly, we want:
robot_motion =
f(task_content, person_style, robot_body)
This leads to a more interesting question.
Suppose I record somebody walking, turning and sitting.
Can the model learn enough about their movement style to generate a new behavior, such as walking up stairs, while preserving some characteristic aspects of how that person moves?
If it can only replay existing motion, we have built a motion database.
If the style generalizes to unseen behaviors, then we may have learned something more meaningful.
11. Simulation
Simulation is where I want to test these ideas before touching real hardware.
A simplified pipeline could look like:
video = load_video("human.mp4")
human_motion = reconstruct_human(video)
world_motion = recover_world_coordinates(
human_motion
)
contacts = estimate_contacts(
video,
world_motion
)
robot_reference = retarget(
human_motion=world_motion,
contacts=contacts,
robot="unitree_g1",
)
simulate(robot_reference)
The first measurements I care about are not whether the animation looks impressive.
I would measure:
3D reconstruction error
root trajectory error
end-effector tracking error
foot sliding
contact accuracy
joint-limit violations
torque-limit violations
fall rate
motion completion rate
energy consumption
Then deliberately modify the simulator:
friction ±20%
robot mass ±10%
motor strength ±10%
sensor noise
control latency
ground stiffness
If the controller fails completely after a small physical change, it probably has not learned robust behavior.
12. Why Simulation Is So Useful
This is why simulation keeps pulling me back.
I can reconstruct somebody moving, place a humanoid inside MuJoCo or Isaac-style simulation, and ask it to reproduce the motion.
Then watch it fail.
That failure is useful.
Maybe the feet slide.
Maybe the center of mass moves outside the stable region.
Maybe the motion requires impossible joint velocities.
Maybe the motors saturate.
Maybe it works on one floor and fails when friction changes slightly.
Every failure reveals something missing from the representation, physics model or controller.
Simulation becomes the laboratory between:
observed human behavior
and:
physical machine behavior
13. The Sim-to-Real Gap
Eventually simulation is still not reality.
A real robot contains effects that are difficult to model perfectly:
motor latency
gear friction
backlash
sensor noise
structural compliance
temperature changes
battery variation
floor variation
unexpected contacts
manufacturing differences
A controller that performs perfectly in simulation can therefore still fail on real hardware.
This is the sim-to-real gap.
Common strategies include:
domain randomization
system identification
actuator modeling
residual learning
real-world fine-tuning
online adaptation
One interesting approach is to learn the difference between simulated and physical behavior:
simulated dynamics
+
learned real-world correction
→ improved dynamics model
→ improved control policy
This feedback loop matters.
Physical intelligence probably cannot come only from static internet data.
Eventually the physical world has to become part of the learning process.
14. Video Still Does Not Capture Everything
Even if video reconstruction becomes extremely good, video alone does not contain all human knowledge.
Imagine an expert technician repairing an engine.
Video may reveal:
where the hands moved
which tool was selected
which component was touched
the order of actions
But it may not tell us:
what resistance they felt
what sound caught their attention
what they expected to happen
what alternative they considered
why they changed strategy
what memory they used
A more complete physical-skill representation may eventually combine:
vision
audio
language
proprioception
force
touch
eye gaze
environment state
task outcome
Possibly neural signals become another source one day.
Video may still be the most scalable starting point because humanity has already produced an enormous amount of it.
But video is probably only one part of the full system.
15. Neural Interfaces Are Another Possible Input
This is where brain-computer interfaces become conceptually interesting.
It is important not to confuse today’s systems with mind uploading.
Current neural interfaces are much closer to:
neural activity
→ signal decoding
→ movement intention
→ external device control
than:
brain
→ complete human software
A neural interface may provide information about intention that cannot be directly observed from video.
That makes it interesting as another possible input.
In the future we can imagine systems combining:
video
voice
language
movement
behavior history
personal memory
wearables
neural signals
But reconstructing a complete brain, memories or consciousness remains an entirely different scientific problem.
16. Even a Perfect Digital Brain Would Still Need a Body
There is another problem that science fiction often skips.
Suppose, purely as a thought experiment, we somehow produced a perfect computational copy of a human brain.
That would not automatically give us a humanoid robot that behaves exactly like the person.
A human nervous system developed inside:
human muscles
human bones
human hands
human skin
human eyes
human vestibular system
human sensory delays
Now put the same intelligence inside:
electric motors
rigid links
robot joints
cameras
depth sensors
metal feet
different proportions
different latency
How does it control that body?
Knowing:
"I want to pick up the cup"
is not the same thing as knowing:
motor 1 → torque A
motor 2 → torque B
motor 3 → torque C
...
while maintaining balance, respecting actuator limits and responding to the environment.
This is why embodiment is such an important part of Physical AI.
Intelligence is not simply software that can be dropped into any body without adaptation.
The intelligence and the body have to learn how to operate together.
17. The Prototype I Would Build
Instead of beginning with an enormous question like:
Can we digitize human physical behavior?
I would break it into smaller experiments.
Stage 1: Video → World-Space Human
Input:
ordinary monocular video
Output:
3D human body
camera trajectory
global root trajectory
joint motion
Measure:
3D joint error
trajectory stability
temporal jitter
Stage 2: Human → Robot Kinematics
Retarget important points:
pelvis
hands
feet
head
torso
Measure:
retargeting error
joint-limit violations
motion smoothness
Stage 3: Contact-Aware Retargeting
Estimate:
foot-ground contacts
hand-object contacts
Measure:
foot sliding
contact consistency
contact timing
Stage 4: Physics
Run the motion inside MuJoCo, Isaac Lab or another physics engine.
Add:
mass
gravity
friction
collisions
joint limits
actuator limits
Measure:
falls
torque saturation
center-of-mass stability
tracking error
Stage 5: Whole-Body Tracking
Train a controller:
policy(action | robot_state, reference_motion)
where:
robot_state = current physical state
reference_motion = desired behavior
action = control command
Stage 6: Robustness
Randomize:
mass
friction
latency
motor strength
sensor noise
Then measure how often the policy still completes the behavior.
Stage 7: Motion Style
Collect several motion clips from the same person.
Learn:
style_encoder(motion) → person_style
Then test:
new_motion = generator(
content="walk_and_turn",
style=person_style,
embodiment="unitree_g1",
)
The most interesting test is not reconstruction.
It is generalization.
Can the representation preserve characteristics of a person’s movement in a behavior they were never explicitly recorded performing?
18. A Larger Architecture
Long term, I imagine something like:
HUMAN
│
┌───────────┼───────────┐
│ │ │
video audio language
│ │ │
motion voice knowledge
└───────────┼───────────┘
│
multimodal representation
│
physical behavior
│
task + style
│
world interaction
│
simulation
│
robot controller
│
embodiment
│
real world
│
feedback
│
learning loop
Additional signals could eventually include:
touch
force
wearables
eye tracking
neural signals
But I would not start there.
The scalable starting point is already everywhere:
video.
19. Why This Matters
Humanity has stored an enormous amount of knowledge in text, images, audio and video.
Large language models made text computationally useful in a completely new way.
I think something similar may eventually happen with physical behavior.
A technician with thirty years of experience does not only possess sentences that can be converted into tokens.
They also possess:
perceptual habits
motor patterns
timing
physical heuristics
interaction strategies
failure recovery
Much of that knowledge disappears when the person stops performing the work.
If parts of it can be transformed into structured physical representations, they could become useful for:
robot learning
industrial training
simulation
digital twins
teleoperation
assistive robotics
skill preservation
human-avatar systems
That does not mean we copied the human.
It means we captured another category of information that previously existed mainly inside biological experience.
Conclusion
The problem I am interested in is no longer simply:
video → pose
It is closer to:
pixels
→ geometry
→ motion
→ contact
→ task
→ style
→ physical constraints
→ embodiment
→ control
→ reality
The early parts of this pipeline are becoming increasingly possible.
The later parts remain extremely difficult.
That is exactly why I find the problem interesting.
Maybe human physical knowledge eventually becomes portable between people, simulations, avatars and robots.
Maybe brain interfaces create another path.
Maybe entirely new representations appear.
For now, the experiment can stay much smaller:
one video
→ one reconstructed human motion
→ one structured representation
→ one simulated humanoid
→ one physically valid behavior
Then find the next failure.
Fix it.
And repeat.
Technical References
Some of the research and systems closely related to this direction include:
- SMPL for parametric 3D human-body representation.
- TRAM for recovering human motion and global trajectory from monocular video.
- VideoMimic for learning humanoid skills from ordinary human video.
- ASAP for humanoid motion tracking and sim-to-real adaptation.
- HOVER for whole-body humanoid control using human motion.
- Recent work on human-motion style transfer for humanoid robots.
- MuJoCo for physics-based robotics simulation.
- NVIDIA Isaac Lab for large-scale robot learning and sim-to-real workflows.
- Unitree G1 as one current humanoid platform for motion and control experimentation.