Stretch New subfield (robot learning) and very broad, but no heavy math: the equations are interface diagrams written in symbols, plus one rigid-transform chain you can do by hand.

In-Context Learning for Robots: Methods and Applications

A robot that already knows how to move learns a new task from a demo, a correction or its own interaction, with its weights frozen, by turning that evidence into one of four intermediates (an action, a motion reference, a predicted future or a skill call), and the lesson survives only if that intermediate keeps what the task needs.

Toolbox4 pages. First question: What does a robot policy take in and put out?

The map10 pages. First question: Why can two robots finish the same tray, yet only one follow the demo?

How solid is the evidence?5 pages. First question: Does more task variety beat more demonstrations?

Your move3 pages. First question: What would a careful reviewer question?

Check yourself3 pages. First question: Which family does MT3's align-then-replay belong to?

Authors: Haojian Huang, Zexi Li, Junhao Guo, Yehang Zhang, Wenxuan Peng, Bohan Zhou, and 33 more
Published: 2026, arXiv preprint
arXiv: 2609.36012 (lesson written from v1)
Code: github.com/JethroJames/awesome-robots-icl (not stated in the paper license; the survey's curated paper list and project page, linked from the paper; not method code)
Paper type: Survey

Toolbox 1 Start here. Our running example: a packing robot.

What does a robot policy take in and put out?¶

It sees a history. \(h_t\) holds every observation and action so far: images, joint angles, force readings (§2.2).
It outputs a block of actions. Not one command but \(A_t\), the next H actions (Eq. 1). Move H.
Only executed actions count. New observations can replace the unexecuted rest of a block (§2.2).
Pretrained policies supply the skills. Robot foundation models, trained on large multi-robot datasets, already grasp and move (§1). [[A_t | the next block of H actions]]

The interface is Eq. 1 and §2.2. The packing history is illustrative.

A policy maps a history to the next block of actions. Next: where new task evidence fits in.

Toolbox 2 Adds task evidence to page 1's policy.

How is in-context learning different from fine-tuning?¶

Fine-tuning changes the weights. Task data update the pretrained \(\theta\) into \(\theta_{\text{task}}\) (Fig. 4a).
In-context learning changes the input. A demonstration enters as context \(C_t\), and the weights stay \(\theta\) (Fig. 4b).
Weights stay frozen while deployed. \(\theta_{t+1} = \theta_t = \theta_0\); only history and context change (Eq. 3).
Where the lesson lives sets its cost. Weights persist until retrained; context can be swapped or reset per task (§2.2). [[θ fixed | deployed weights never change]]
One-shot means amount, not mechanism. One demonstration can drive a gradient update or just condition a frozen policy (§2.2).

The contrast is Fig. 4 and Eq. 3; the boxes are our redraw.

In-context learning teaches through the input, not the weights. Next: moving a demonstrated motion to a new place.

Toolbox 3 A tool for the geometric family on page 9.

How do you move a demonstrated motion to where the object is now?¶

Record the motion relative to the object. The gripper path is stored in the demonstration object's frame (Eq. 8).
Find the object in the new scene. Its new pose replaces the demonstration pose (Eq. 8).
Re-attach the path. \({}^W T^{q}_{E} = {}^W T^{q}_{O}\,({}^W T^{s}_{O})^{-1}\,{}^W T^{s}_{E}\) carries the path along (Eq. 8). Move the object.
It assumes a rigid, still object. Shape changes or moving objects need richer correspondence (§3.2). [[3 | rigid transforms, chained (Eq. 8)]]

Eq. 8 is the paper's rigid object-relative transfer. We draw it in 2-D with an illustrative path.

A rigid transform moves a demonstrated path to the object's new pose. Next: predicting what happens before acting.

Toolbox 4 A tool for the world-model family on page 10.

What is a world model?¶

It predicts what comes next. From context and history it forecasts future observations \(v^+\) (Eq. 9).
Use 1: picture the goal, then act. A decoder turns a predicted future into actions (Eq. 9).
Use 2: try actions in imagination. Predict each candidate's future and keep the best-scoring one (Eq. 11). Toggle the use.
Teaching picks the preferred future. Dynamics says what an action may cause; context says which result is wanted (§3.3). [[v⁺ | predicted future observations]]

Both uses are Eqs. 9 and 11, §3.3. Frames and scores are illustrative.

A world model lets a robot see consequences before moving. Next: the problem that makes robot ICL hard to judge.

Map 1 Page 1's packing robot, taught red before blue and a handle grasp.

Why can two robots finish the same tray, yet only one follow the demo?¶

The demo sets two requirements. Pack red before blue, and grasp the cup by its handle (§1, §3).
Robot A follows both. Red first, handle grasp, everything placed.
Robot B ignores the demo. Blue first, a body grasp, everything placed. Toggle to compare.
The final trays look identical. Placing an object correctly can still violate a required handle grasp (§1). [[2 | requirements the end state can't show]]
So check the path, not just the end. Report goal completion and demo adherence separately (§7.3).

The order and handle-grasp examples follow §1, §3 and §7.3. The scene is illustrative.

A finished task doesn't prove the demo was used. Next: where in-context learning sits among ways robots learn.

Map 2 Places page 2's frozen-weight learning on a longer ladder.

Where does in-context learning sit among ways robots learn?¶

S1: program it. Engineers write the task logic, control laws and plans (§2.1).
S2: learn a policy. Imitation and reinforcement build reusable skills into the weights (§2.1).
S3 and S4: adapt with weights frozen. S3 uses interaction history; S4 uses supplied teaching. Both are robot ICL (§2.1). [[S3 + S4 | the two forms of robot ICL]]
S5: get better at learning. Experience improves how the next task is learned: physical recursive self-improvement (§2.1).
S6: learn as a group. Robots exchange knowledge across bodies, even when motions can't be copied (§2.1).

The six horizons are §2.1 (Figs. 1 and 3). The layout is ours.

Robot ICL is the frozen-weight middle of a longer learning ladder. Next: the four ways it reaches the motors.

Map 3 The survey's main taxonomy, using page 5's demo.

What are the four ways context can reach the motors?¶

Act: infer actions directly. A context-conditioned policy turns evidence into an action distribution (Table 2).
Retarget: reuse a recorded motion. Geometric transfer adapts a motion or contact reference to the new scene (Table 2).
Forecast: predict the outcome first. World-model control uses predicted consequences to choose actions (Table 2).
Script: call skills or tools. An execution specification tells existing skills what to do (Table 2).
Policies and agents dominate. 123 policy, 24 geometric, 22 world-model and 91 skill or agent references (Fig. 9b). [[260 | method references]]

Families and intermediates are Table 2; counts are the paper's own census (Fig. 9b).

The families differ in the intermediate that carries the demo into motion. Next: each family in turn.

Map 4 Family 1 of page 7. Our packing demo has six steps.

How does a policy read a demonstration?¶

Encode the demo, then generate. \(r_t = E_\theta^{\mathrm c}(h_t, C_t)\) summarizes the evidence; a generator turns it into actions (Eq. 4).
Attend to the matching step. Weights \(\alpha_{ti}\) pick demo steps that match the current history (Eq. 6). Move the phase.
The read follows progress. As the robot advances, attention shifts to later demo steps (§3.1).
Or replay what you retrieve. Nearest-neighbour methods return the recorded actions of similar states (Eq. 5).
Retrieval can beat learning. On unseen MuJoCo bodies, REGENT's retrieve-and-play beat its frozen transformer (§3.1). [[123 | policy references (Fig. 9b)]]

Equations are §3.1. The attention weights are illustrative, from a toy similarity.

A policy must keep the taught order in its representation. Next: reusing the recorded motion itself.

Map 5 Family 2 of page 7, using page 3's transform.

How does geometric transfer reuse a recorded motion?¶

Extract motion descriptors. Hand poses, object-relative poses or contacts from each demo step (Eq. 7).
Retarget to the new scene. Map them onto the current objects and the robot's body (Eq. 7).
Track the reference. An executor follows the retargeted motion (Eq. 7).
It scales by adding references. MT3 reached 1,000 single-interaction tasks with under 24 hours of demonstration (§3.2). [[1,000 | tasks, under 24 h of demos (MT3)]]
Its weak spot is applicability. Shape changes, lost contact or moving objects break the match (§3.2).

The pipeline is Eq. 7; MT3's scale is §3.2. Boxes are our redraw.

Geometric transfer makes its assumption checkable: the match must still apply. Next: forecasting outcomes.

Map 6 Family 3 of page 7, using page 4's world model.

How does a world model turn a demonstration into control?¶

Generate the demonstrated future. For packing, \(v^+\) can show red placed before blue in the robot's scene (§3.3).
Decode actions from it. The decoder must realize that particular sequence (Eqs. 9, 10).
Or score candidate actions. Dynamics predicts each candidate's future; teaching says which one is preferred (Eq. 11).
Search can win on new tasks. Demo-JEPA's action head wins familiar real tasks; planning wins unseen ones (Table 16). [[22 | world-model references (Fig. 9b)]]
A forecast can be unreachable. Prediction and decoding are separate sources of error (§3.3).

Equations are §3.3; the Demo-JEPA comparison is Table 16. The frames are illustrative.

Forecasting exposes consequences, but they must be reachable. Next: scripting skills and tools.

Map 7 Family 4 of page 7, on page 5's packing demo.

How do skills and agents use a demonstration?¶

Turn the demo into a specification. \(g = f_\theta(C_t, h_t)\) is a skill order, a program or tool calls (Eq. 13).
An executor realizes it. \(A_t = \kappa(g, o_t)\) uses existing motor skills (Eq. 13).
Programs came with language models. Code as Policies (2022) recombined perception, control and logic from examples (§3.4).
Portability versus physical detail. A compact procedure crosses scenes but can drop a needed grasp (§3.4). [[91 | skill and agent references (Fig. 9b)]]
Feedback revises the script. Outcome errors adjust arguments or code without training weights (§3.4).

Equations are §3.4. The video, outline and arm are illustrative.

A script travels well but must carry every binding detail. Next: binding a demo to new objects.

Map 8 Shared by all four families. Taught task: pour from a jug into a cup.

How does a demonstration carry over to new objects?¶

Swap the held object. Jug → bottle, same cup: adjust the grasp and tilt (Table 6).
Swap the receiver. Same jug, cup → bowl: adjust the pouring position and height (Table 6).
Swap both. Bottle into bowl: grasp, tilt and position all change (Table 6).
Match parts, not whole objects. Part-wise warping raised mug–rack placement from 11/27 to 23/27 (Table 16). [[23/27 | placements with part-wise warping]]
That gain had help. Part correspondences were assisted in that study (Table 16).

The pouring example is Table 6; the mug–rack numbers are Table 16, with assisted parts.

Transfer depends on matching the parts the task actually uses. Next: keeping evidence until it's needed.

Map 9 Shared by all four families. Where page 2's lesson can live.

Where can a robot keep evidence until it needs it?¶

In the supplied context. A demo or instruction sits in the input (Fig. 6a).
In recurrent state. A running summary is carried forward: \(C_{t+1} = U_\theta(C_t, o_t, a_t, o_{t+1})\) (Fig. 6b, Eq. 2).
In retrieved episodes. An external archive returns relevant past experience (Fig. 6c).
Or in the weights. Fast weights change during deployment, or updated weights persist (Fig. 6d,e). [[5 | places a lesson can live (Fig. 6)]]
Each has its own reset test. Remove the context, clear the state, block retrieval, or restore old weights (§2.2).

The carriers and reset tests are Fig. 6 and §2.2.

Knowing where the lesson lives tells you how to test it. Next: how robots learn to read context at all.

Map 10 How page 7's policies learned to use a demo.

What training makes a robot read its context at all?¶

Pair a support with a query. Train on one example and supervise a different execution of the same task (§4.2).
Or train on continuous streams. Earlier observations and actions precede later ones, without fixed roles (§4.2).
Or link attempts. A failed try and its feedback come before the corrected continuation (§4.2).
The context must be needed. If the scene alone fixes the action, the model can ignore the demo (§4.2). [[3 | ways to organize training (§4.2)]]
ICRT shows it. Trained on DROID data alone, it made no progress on the test tasks (Table 16).

The three organizations are §4.2 and Table 11; the ICRT result is Table 16.

A robot reads demos only if training made them useful. Next: which kind of data helps most.

Solid 1 Drawing error in pixels, lower is better, at a fixed 10,000 demonstrations (BPP).

Does more task variety beat more demonstrations?¶

Fix the budget. BPP spends 10,000 drawing demonstrations three different ways (§4.2, Fig. 15a).
500 tasks, 20 demos each. Mean Chamfer error is 9.5 pixels (§4.2).
1,000 tasks, 10 demos each. The error falls to 5.1 pixels (§4.2).
2,000 tasks, 5 demos each. The error falls to 3.4 pixels (§4.2). [[3.4 px | error with the most tasks]]
Variety must change the answer. Useful diversity adds decisions that an example must resolve (§4.2).

Errors are §4.2 and Fig. 15a, averaged over three seeds.

At a fixed budget, more distinct tasks teach context use better. Next: demonstration vs language prompts.

Solid 2 S1 success on seen and unseen tasks, 1k–100k training hours, demo vs language prompts.

Do demonstration prompts beat language prompts on new tasks?¶

Seen tasks: both improve. With more training hours, both prompt types get better on familiar tasks (§4.2).
Unseen tasks: demos pull ahead. Demonstration prompting rises from 1% to 66% (§4.2, Fig. 15c). [[66% vs 9% | unseen tasks, demo vs language]]
Language barely moves. Language prompting rises from 0% to 9% on the same unseen tasks (§4.2).
Otherwise matched. Same data, compute and architecture, apart from the prompt embedding (§4.2).
One caveat on the metric. Its cumulative per-step success counts human recovery, mostly for the language baseline (§4.2).

Values are §4.2 and Fig. 15b,c. Only the 1k and 100k endpoints are drawn; straight lines join them.

Similar seen-task scores can hide very different new-task teaching. Next: the survey's 14 reported comparisons.

Solid 3 Within-study comparisons from Table 16: baseline → variant.

What do the 14 reported comparisons show?¶

More examples, little gain. LMAct: going from 0 to 512 episodes often helps little (Table 16).
The right context helps. NOLO: adding a scene video lifts success from 33.58% to 43.65% (Table 16). [[14 | reported comparisons (Table 16)]]
Missing conventions break agents. Show-Harness with arbitrary API names: 1 success in 20 (Table 16).
Retrieval beats cloning on few demos. MT3: higher success with fewer demonstrations than behaviour cloning (Table 16).
Checking variants pays. RAPID: verifying on scene variants lifts success from 53.2% to 75.9% (Table 16).

Results are Table 16. Each is a comparison inside one study, not across studies.

Gains track the information context supplies, not its volume. Next: does retained experience help?

Solid 4 Table 16's group III: success with vs without retained experience.

Does remembering earlier attempts actually help?¶

Measure keep versus reset. The reuse gain \(\Delta_n = S_n^{\text{keep}} - S_n^{\text{reset}}\) compares the same attempt number (§7.3).
Memory can help a little. RTCF: 86.4% → 88.4%. Zeva: +10 to 20 points (Table 16).
Or help only part of a task. TraceFlow: 21/50 → 39/50 on real packing order; its simulation aggregate is unchanged (Table 16).
Or backfire. FARE's always-search variant scored 91.0, below its 91.5 base (Table 16).
A better learner is the big win. LMPC's trained successor goes from 39.4% to 66.3% on held-out teaching tasks (Table 16). [[39.4 → 66.3 | LMPC successor, new teaching]]

Numbers are Table 16; the reuse-gain formula is §7.3. LMPC trains between sessions. TraceFlow's bars are our percentages of 21/50 and 39/50.

Retaining experience helps unevenly; learning to learn helps most. Next: how to test a robot properly.

Solid 5 Answers page 5: controls that show whether the demo mattered.

How do you test that a robot really used the demonstration?¶

Remove the demo. Success without context shows what the prior alone can do (§7.1).
Swap in a different valid demo. Same scene, another feasible behaviour: the robot should switch (§7.1).
Give a fully specified task. This checks the behaviour is executable at all (§7.1).
Scenes can give the answer away. LIBERO-CF found tested VLAs often continue the scene's usual task (§7.1).
Conventions need evidence too. Without supplied action conventions, Show-Harness succeeded in 1 of 20 episodes (§7.1). [[1 / 20 | episodes without conventions]]

Controls and results are §7.1 and Table 15. Show-Harness also inferred only 23.3% of mappings (§7.1).

Change the demo's meaning and watch whether behaviour changes. Next: what a reviewer would question.

Your move 1 Our critique of the survey itself.

What would a careful reviewer question?¶

🧠 Comparisons are within-study. Each Table 16 row compares variants inside one paper; no shared benchmark links them (Table 16).
🧠 Much of it is very new. 197 of 412 references first appeared in the partial 2026 window (Fig. 9a).
🧠 The tests are proposals. The evaluation controls are suggested, not run by the survey (§7, Table 15).
🧠 Families blur in hybrids. Systems like GLOW or ReCAP span several families; one branch decides the label (§2.3, §3).
🧠 Simulation carries much evidence. NOLO's unseen layouts and LMAct's games are simulated (§7.1, Table 16).
🧠 Steelman: the questions are the contribution. Its controls (swap, remove, reset) apply to any method (§7).

Pointers mark the evidence behind each concern. The pills are our judgment.

Read it as a map and a test plan, not a leaderboard. Next: the verdict for your goal.

Verdict For the goal on the cover. Uses pages 7, 16 and 19.

Which approach should you try for teaching a robot new tasks?¶

New order or relation? Try a policy. Demos beat language on unseen tasks: 66% vs 9% in S1 (§4.2). [[66% | S1 unseen tasks, demo prompts]]
Exact contact matters? Use geometric transfer. It makes the matched interaction explicit and checkable (Table 2, §3.2).
Unknown physical response? Add interaction history. Recent transitions reveal how this body responds (§3.1).
Long procedure? Script the skills. Programs expose order and preconditions, but must carry contact details (§3.4).
🧠 Our verdict: test before trusting. Whatever you pick, run page 19's swap and no-demo controls first.

The need-to-family mapping follows Table 2 and §3; the verdict is ours.

Pick the family by what the task must preserve, then prove the demo matters. Evidence is mostly within-study (page 20).

Your move 2 The survey's four research directions (Fig. 24), plus one of ours.

What comes next?¶

Infer taught relations. Learn an unfamiliar rule from examples, not just pick a known task (§8.1).
Compose task steps. Combine a demonstrated order with other known constraints (§8.2).
Preserve function across physical change. Keep an interaction working when contact geometry changes (§8.3).
Revise retained corrections. Keep useful corrections, and update them when conditions change (§8.4).
🧠 For speech: voice-prompted TTS. A voice prompt conditions a frozen TTS model; the same swap tests apply.

Steps 1–4 are §8 (Fig. 24); step 5 and the map positions are our guesses.

The frontier is robots that get better at being taught. The survey calls it physical recursive self-improvement (§8.5).

Check 1 From pages 7 and 9.

Which family does MT3's align-then-replay belong to?¶

Context-conditioned policy
Geometric demonstration transfer
World-model-based control
Skill- and agent-based execution

Geometric transfer. MT3 retrieves a recorded interaction, aligns to it, then replays the motion (§3.2).

Check 2 From page 18.

What does the reuse gain \(S_n^{\text{keep}} - S_n^{\text{reset}}\) measure?¶

The total successes over all retries
The benefit of retained evidence at the same attempt number
How fast the robot executes a task
The gain from fine-tuning the weights

Retained evidence, attempt for attempt. Cumulative success also rewards plain extra tries (§7.3).

Check 3 Spot the flaw. From pages 18 and 19.

What is wrong with "success rose over five retries, so the robot learned"?¶

Nothing: rising success is learning
Extra tries alone raise cumulative success; compare kept vs reset memory
Five retries are too few to measure anything
Success cannot be measured across retries

Retries alone inflate cumulative success. Hold attempts fixed and compare kept versus cleared memory (§7.3).