Frontier Models for Acting in Space and Time#

Pim de Witte (General Intuition, Co-Founder & CEO)

• Why: A different bet on model architecture with a data-moat story an infrastructure person can evaluate: action-labeled gameplay at ~1.5 trillion action tokens/month versus frontier labs' entire robotics training sets, which the speaker claims Medal collects in half a day.
• What: Models that operate in continuous time — frames in, actions out, real time — trained on Medal's game recordings with semantically labeled controller actions. The core insight is the shared action space: humans teleoperate robots with the same controllers (and brains) they use for games, so keyboard/mouse/controller actions are a common interface across embodiments, and the model never has to predict low-level motor values — the existing control stack does that translation, the same way a human player doesn't compute torque.
• How: Action density is the differentiator: games require frequent simultaneous small adjustments (the drone-flying case), so high-APM data puts high-frequency control problems in reach in a way sparse teleoperation data doesn't. World models generate additional environments to train in. Against JEPA (in the Q&A): inferred action labels (his example: you can't tell rudder position from cockpit video) add a separate scaling problem, so observed labels transfer better to action sequences. Deployment model is layered: an orchestrator LLM (Gemini/Astra-class) for long-horizon planning on top of this fast action model; one model claimed to play games and control 10+ robot types with the same primitives. Early access late November/early December; NYC and London hackathons announced.
• Where applicable: Privately deployed with some customers at talk time; claims like "significantly faster than any LLM" and the data-advantage comparison are forecasts and company figures without public benchmarks. An accessibility use case (model supplies the missing keyboard actions for a user who can only operate a mouse, observing as it acts) is the most concrete demonstrated application.