The Inefficient Exploration Problem

Predictive musculoskeletal simulation tries to generate human movement from first principles, without copying recorded motion. In practice, two difficulties show up again and again: it is hard to make the model move the way you intend, and it is hard to make the optimization or training converge within a realistic time budget. Both stem from the same root cause, which I call the inefficient exploration problem.

A simple, single-joint example makes the issue concrete. The joint is driven by two flexors (f1, f2) and two extensors (e1, e2).

The core problem is this: even when the policy actively explores in the muscle-activation space, that exploration does not translate into meaningful exploration in joint space. There are two reasons.

First, muscle-activation profiles are independent across time. Without any temporal structure, the muscles simply flicker on and off, and the limb barely moves.

Second, there is no structural correlation between muscles: a muscle cannot coordinate with its own synergists. As a result, active muscle-space exploration never becomes joint-space exploration, the policy keeps repeating the same behavior throughout training, and it often fails to converge in a realistic time.

The Stretch Reflex System

If the trouble is that the muscles act independently, the cure is to stop commanding each muscle directly and instead let every muscle respond to its own mechanical state. This is what a biological stretch reflex does: a muscle that is being stretched pushes back on its own. We build this reflex into the model, and let the policy tune the reflex itself rather than the raw activations.

Concretely, each muscle's excitation is computed from its current fiber length \(L\) and fiber velocity \(v\):

Here \(\lambda\) is the reflex threshold, \(G_{\text{tonic}}\) and \(G_{\text{phasic}}\) are the reflex gains, and \([x]_0^1 = \mathrm{clip}(x, 0, 1)\) just keeps the excitation between 0 and 1.

The equation has two intuitive parts. The tonic term \(G_{\text{tonic}}(L - \lambda)\) acts like a spring: the further a muscle is stretched beyond its threshold length \(\lambda\), the harder it pulls back, so the limb settles toward an equilibrium posture set by \(\lambda\). This is Feldman's classic \(\lambda\)-model [1], where moving a limb simply means shifting that equilibrium. The phasic term \(G_{\text{phasic}}[v]_+\) acts like a shock absorber: it responds to how fast the muscle is lengthening (\([v]_+\) is active only while it stretches), damping sudden motion and keeping the movement stable.

The important change is what the policy controls. Instead of dictating a separate activation for every muscle at every instant, it only sets three interpretable reflex parameters (\(\lambda\), \(G_{\text{tonic}}\), and \(G_{\text{phasic}}\)), and the actual excitation falls out of the muscle's own physics. Because those excitations are now tied to real mechanical state, the muscles around a joint naturally react to one another's motion. What used to be random, independent flickering becomes temporally correlated and synergistic, and this coordination is a genuine spinal reflex [2], not a hand-crafted trick.

Because the muscles now move together, diverse movements are explored from the earliest stage of training, and the optimization converges faster than prior approaches.

Sprinting Emerges from a Simple Reward

The model is trained purely with reinforcement learning, using a single exponential forward-velocity reward and no other reward terms. From this alone, a human-like sprinting motion emerges within a few hours of training.

Human-like sprinting emerges from a single exponential velocity reward.

• Peak forward speed reaches 6.1 m/s.
• Without the stretch reflex system, training does not converge in a realistic time.
• No imitation (reference data), no dimension reduction (synergies), no symmetry mirroring, and no external exploration tricks are used.

References

1. Feldman AG (1986). Once more on the equilibrium-point hypothesis (λ model) for motor control. Journal of Motor Behavior, 18(1), 17–54.
2. Pierrot-Deseilligny E & Burke D (2005). The Circuitry of the Human Spinal Cord: Its Role in Motor Control and Movement Disorders. Cambridge University Press.