A 3,268-parameter PPO policy, trained in a numpy sandbox, running in your browser at 60 Hz.
by aadi shah·August 2026
The live policy needs a wider screen than this one, so the animation is hidden on mobile — open this page on a laptop to watch the arm run. The write-up below is all here either way.
The arm above is not keyframed. At every frame, the policy processes a simulator observation through a small MLP to generate four control outputs: base velocity, two joint velocities, and a gripper command. This behavior emerged entirely from optimizing reward across 32M training steps.
Reinforcement learning problems are written down as a Markov decision process, which is a formal way of saying: a loop, running one tick at a time. The agent sees the state — a list of numbers describing the world right now. It picks an action. The world advances one tick and hands back the next state and a scalar reward. Then it happens again.
The Markov part is the constraint that makes this tractable: the state has to contain everything relevant about the future, so the agent can decide from the current state alone and never needs to remember how it got there. Learning means finding a policy, a function from states to actions:
The subscript is the thing being learned — here is the weights of a small neural network. It is written as a distribution over actions rather than a single action because during training the policy samples, which is how it explores; at deployment it takes the distribution's mean. The spaces and are both continuous here:
The world itself is the 480×60 pixel strip above: a wheeled base at , a two-link arm with segments of 16 and 14 px, a gripper, and two crate columns 390 px apart.
There is no dataset here. Nobody knows the right action for a given state, so there is nothing to imitate — the only way to find out whether an action was any good is to take it and see what comes back. That makes this online learning: the algorithm generates its own training data by acting, and what it learns changes what it collects next. The 32M steps behind this policy are not 32M examples someone labeled; they are 32M ticks it played out itself.
The quantity being maximized is the expected return , the average total reward the policy collects over a trajectory. It cannot be differentiated directly — the reward comes from a simulator, not from a formula in . The policy gradient theorem gets around that by moving the derivative onto the one part that is a differentiable function of , the policy's own log-probability:
Which reads: push up the probability of actions that turned out better than expected, push down the ones that turned out worse. “Than expected” needs a definition, and it comes from two standard quantities. The value of a state is the return the policy expects to collect from there onward; the action-value is the same thing if you commit to one action first. Their difference is the advantage:
So means the action did better than the policy's own average from that state, and means it did worse. Neither nor is known, so is learned by a second network — the critic, — and is estimated from it and the rewards actually observed. Using the value as a subtracted baseline this way does not bias the gradient; it only reduces its variance.
Estimate the outer expectation by averaging over a batch of collected steps and you have the vanilla policy gradient, REINFORCE [5] with a baseline. It is unbiased, it is about five lines of code, and on its own it is almost unusable: the estimate is so noisy that useful steps are small, and large ones are a coin flip.
The reason you cannot simply take bigger steps is in the subscript. The expectation is over : trajectories drawn from the policy as it currently is. So the gradient is only valid at the parameters that collected the data.
That is what makes the method on-policy. Each batch describes exactly one policy, and the moment you update, it describes a policy you no longer have. Supervised data keeps; this spoils on contact.
Which is why an overshoot is expensive. It leaves you somewhere worse, holding a batch that no longer describes where you are — and the usual remedy, reusing the data to correct course, is the one thing unavailable. In supervised learning a bad step costs an epoch. Here it can cost the run.
TRPO [2] answers this with a hard constraint on the KL divergence between the old and new policy — a measure of how much two distributions differ — enforced with second-order methods, which use curvature as well as slope and are correspondingly expensive. PPO [1] gets most of the same effect from one first-order trick, and it buys something else along the way: it is safe to take several gradient steps on the same batch, which is what makes an on-policy method affordable at all.
Look at how much more or less likely an action has become since the data was collected:
At the moment of collection . As the update proceeds it drifts: above 1 means the action became more likely, below 1 less. This ratio is the thing PPO polices, and it does so by making the objective stop rewarding drift past :
Two cases, and they are worth walking through separately.
Good action (). Raising its probability raises the objective, so the gradient wants to keep pushing. But past the clipped branch is the smaller of the two, the selects it, and it is a constant in — flat, no gradient. That sample stops contributing. Note what this does and does not do: nothing forbids from leaving the interval, and other samples in the batch still push the parameters. The incentive is removed, not the motion.
Bad action (). Now the objective improves by making the action less likely, and the ceiling sits at . Below that the term flattens, so a single bad sample cannot drive its own probability to zero.
The asymmetry that makes this work is that the un-clipped branch is still chosen whenever it is smaller. If an update has already overshot and made things worse, the objective still registers it, so the policy can walk back rather than being pinned at the boundary. Note also what PPO is not doing: there is no constraint, no penalty, and nothing forbids from leaving the interval. Leaving simply stops paying.
That leaves , which has to be estimated from a single trajectory. The one-step version is
Low variance, but it inherits every error in . Summing the actual rewards to the end of the episode instead — the Monte Carlo return — is unbiased, meaning right on average, but noisy, since one lucky episode drags the estimate. GAE [3] interpolates between them with a single knob:
recovers the one-step estimate, the full Monte Carlo return, and values in between blend them geometrically. is what this run uses: enough smoothing to survive 600-step episodes, enough reach to carry the terminal placement bonus back through the drive that earned it.
The loss that actually gets minimized has three parts:
The value term trains the critic by regressing onto observed returns; it is the same critic whose predictions become . The entropy term pays the policy to stay random, which early on is what lets it stumble into a grasp at all. It anneals from to as the behavior forms.
The browser runs the mean action and does not sample, so the Gaussian is training machinery only.
PPO maximizes whatever you write down. Not what you meant — what you wrote. Most of the work here was not tuning the algorithm; it was discovering, repeatedly, that the reward I had written could be satisfied by something other than stacking crates.
The obvious reward — a point per crate, nothing otherwise — never gets off the ground. A random policy would have to drive over, lower onto a crate, close, drive back, and open at the right height entirely by accident before it saw any feedback, and until it sees feedback there is no gradient to follow. So the sparse task reward is replaced with a dense one that pays for progress every step — and the rest of the design is closing the loopholes that substitution opens, since a reward for progress is a reward for looking like progress:
is the distance from the gripper to whatever it currently wants — the crate if empty, the drop slot if carrying — and that term does most of the work. Everything else is a patch over a way the policy found to score without doing the task:
The full task from scratch does not converge in any reasonable budget, so it is split into four stages, each warm-starting the next:
Hyperparameters:
Roughly the order I would take them in: