the field
The Field is an environment that shows our approach to interaction annotation. We built it for two things: data for the next generation of AI–human interaction, and better steering of the systems we run now.
the problem
High-quality interaction data is scarce. Models learn interaction from whatever data exists, so the interactions they produce come out suboptimal and out of distribution. The problem repeats at runtime: a live system responds to a flow of new and old information without a high-value, high-density summary of the situation.
annotated states
We put an annotation framework on top of interaction surfaces. The framework turns what happens on a surface into annotated states, our field-notes, and those states are the data we need. A state holds three things: the participants and their internal states; the environment, the non-interaction things that relate to the moment and influence it; and the history that led here.
The framework is an MVP, built to be added onto. (States also work as high-density injections for steering current systems; more complexity comes later.) We start on text and video, then move to other surfaces, including 3D.
annotating interactions over the surface of video
Take one video, a scene from a show. The framework starts by untangling it into its information sources: the sound, the video as a whole, the specific gestures it can spot. It analyzes each source on its own, takes the parts that say something about the state of the interaction, and recombines them into one state, a field-note for that moment.
Then it keeps going. States land one after another as the scene plays, and because they sit on one timeline, differences between them can be noticed: a tone shifting, a participant whose internal state has moved since the last note. The video becomes a timeline of states, easier to process and dense with information.
| t | the state · summary |
|---|---|
| 279.5s | Vincent · Jules — Hallway outside an apartment door “That shit ain't right.” · Vincent hovers close |
| 335.5s | Vincent · Jules — Narrow hallway, backlit by window “Take care of her?” · Vincent pauses at the threshold |
| 449.5s | Jules — Mia’s car, then Brett’s breakfast room “You mind if I try one of yours?” · Jules bites into the burger |
| 583.5s | Vincent · Jules · Brett — small apartment dining room “Look,” · Vincent gazes down steadily |
annotating nested interactions (reaction videos)
The framework also applies at meta levels of interaction, where an interaction reacts to another interaction. A reaction video is the plainest case: someone watches a show and reacts to it, and that reacting is an interaction of its own, one we can annotate like any other.
Reaction videos have been hard to actually use as training data, because the reasoning behind a reaction is so layered. Decoding what the person is reacting to makes the reaction much simpler to understand. And we can decode it: the show becomes the environment for the reactor’s states, and we understand that environment because we annotated it ourselves. The interaction is layered, the reactor’s states nesting inside the show’s.
That nesting untangles the reasons behind a reaction. Without the first timeline annotated, a reaction is only a laugh at some moment in some video. With it, the laugh points at the on-screen state it followed, so the reaction carries its reason, and data that carries its reasons is much easier to train on.
untangle and store
The Field untangles, cleans, and stores interaction data across surfaces. The store grows into the foundation for interaction world models — the bet we lay out in AI needs to understand the world of human interactivity.
| t | the state · summary | |
|---|---|---|
| 11.9s | state | Vincent · Jules — inside a moving car “What are you feeding y'all?” · Vincent talking mid-sentence |
| 35.5s | state | Vincent · Jules — inside a moving car “How many up there?” · Vincent looks down toward the camera |
| 47.5s | state | Vincent · Jules — Outside white building “It's possible.” · everyone walk side by side |
| 89.5s | state | Jules · Vincent — dark doorway interior, outside view “Some get chosen and become television programs.” · Jules walks side by side with Vincent |
| 133.5s | state | Vincent · Jules — red-carpeted mansion room “Foot massage?” · Vincent glances slightly upward |
| 159.5s | state | Jules · Vincent — indoors by a beige wall “That's a damn shame.” · Jules walks straight-faced toward the camera |
| 207.5s | state | Jules · Vincent — narrow hallway by staircase “But, you know, touching this wife's feet” · Jules gestures with both hands |
| 239.5s | state | Jules · Vincent — narrow hallway “Fuck you.” · Jules leads down the hallway |
| 279.5s | state | Vincent · Jules — Hallway outside an apartment door “That shit ain't right.” · Vincent hovers close |
| 335.5s | state | Vincent · Jules — Narrow hallway, backlit by window “Take care of her?” · Vincent pauses at the threshold |
| 449.5s | state | Jules — Mia’s car, then Brett’s breakfast room “You mind if I try one of yours?” · Jules bites into the burger |
| 583.5s | state | Vincent · Jules · Brett — small apartment dining room “Look,” · Vincent gazes down steadily |
steering
Current AI systems often underperform in interactions because they don’t feel lucid. That splits in two: they have no clear idea of the current situation, and no way to react logically in character. This approach can solve both.
The situation half comes from the annotation process, run at runtime. It gives the system a clear, value-dense idea of the interaction, parts of our states injected into its input, and with that the situational awareness a stateless model never had: it knows how this moment relates to the ones before it. The annotation is not bolted on, every steered turn logs a record as a side effect of the same process.
The character half is designed. We also give the AI what it needs to estimate some of the internal states we as humans use to select the correct next action: designed aspects, who it is, its role, who it is with, while faster, slightly dumber models propose possible next steps. In essence we mimic some of the world-model approaches, an action-selection model that makes the normal, quality reaction more likely: in character, consistent, designable. None of this is prompt-stuffing: the system already receives the flow, what it lacks is the compressed idea of it, so we inject density, not more text.
And the same nesting extends to steering. On the video from earlier, a steered AI joins alongside the reactor, one more nested interactor, a layer on top of the show and the reaction both. It reads the annotated stack beneath it, the show’s states and the reactor’s, and acts from it, its own states partly designed, a nested situation no prompt could hold.
A system that is lucid about the situation and consistent in character does better, and it shows on benchmarks: more natural, consistency and relevancy across a conversation (VERA), and more effective on collaborative real-world tasks (Sierra). It runs on any model, no finetuning: the improvement is the harness.
The records from those steered turns land in the same store as new data. Graded against outcomes, they improve the steering. And because we annotate live, we see every breakage as it happens, including the ones nobody tracks, and those become benchmarks. The deep dive is on the product page.
Start a co-watching session with our AI system: we record you and annotate the experience you have. Interact with the AI, and have us annotate your interactions.