Research
Microsoft Research Project
Acting While Thinking, Self-Evolving Through Play in a living 3D action RPG.
In a real-time 3D world, an agent cannot pause the game while it thinks. JoyTwin pairs a continuous action executive with an asynchronous quest planner in Veloren—dodging, striking, and navigating without standing still, while self-evolving with experience across episodes.
01 / Overview
Deep quest reasoning takes seconds, yet a living 3D world updates every single frame. Acting While Thinking lets JoyTwin deliberate over multi-stage RPG quests without freezing in front of the next attack.
In Veloren, an open-source 3D action RPG, an adventurer must weave together real-time melee combat, town dialogue, merchant bartering, multi-station crafting, skill-tree loadouts, and 2D map navigation. A synchronous “Think, Then Act” agent freezes between reasoning turns and takes fatal hits while waiting for a plan; a purely reactive policy survives brief skirmishes but loses track of long-horizon quests. Operating strictly from player-visible screen frames and mouse-and-keyboard controls, JoyTwin decouples high-level quest planning from continuous real-time execution—and self-evolves through play across episodes.
While the planner reasons over the next move, a blocked controller stands still—taking fatal hits (HP 0) before the plan returns.
On the outdoor grassy hillside, DeepSeek Harness pauses between synchronous reasoning turns while the Green Legoom closes in—taking 16 melee hits while frozen and falling in combat (You Died).
JoyTwin’s reactive controller dodges and strikes in real time while the deliberative planner reasons ahead in parallel—surviving and winning the fight.
Facing the same enemy type on the outdoor hillside, JoyTwin’s continuous action loop tracks, spaces, and strikes in real time while the planner supervises quest progress.
02 / System Architecture
JoyTwin coordinates three loops around the game world: grounded visual perception (See the Game), concurrent planning and execution without a turn barrier (Think & Play), and cross-episode refinement (Self-Evolving with Experience).
Player-visible perception and incremental working memory.
JoyTwin perceives the 3D world and RPG interfaces strictly from screen frames and HUD cues, issuing standard mouse-and-keyboard inputs with zero privileged access to internal engine state.
Independent Plan and Act loops running together.
Continuous planning (Plan) provides high-level stage guidance while continuous action (Act) keeps moving, dodging, and fighting in real time—exchanging immediate feedback without a turn barrier.
Composing runtime routines and carrying lessons forward.
During play, JoyTwin binds and composes multi-step action routines on the fly; across episodes, it distills verified gameplay outcomes into persistent playbook rules and reusable skills—recovering failed quests and executing up to 2× faster.
03 / Experimental Results
We evaluate JoyTwin against five visual-agent baselines—Tool-Calling Agent, DeepSeek Harness, SmolAgents, GamingAgent, and GameScaling—on 25 Veloren RPG Tasks: 11 Basic Action & Survival tasks and 14 Open-World RPG Missions, with each method evaluated over a 3-episode horizon per task. Because synchronous baselines freeze during multi-second reasoning turns and struggle to maintain long-horizon RPG state, three of the five external baselines (Tool-Calling Agent, GamingAgent, GameScaling) solve 0 of 14 (0.0%) RPG missions, and the other two (DeepSeek Harness and SmolAgents) solve only 1 of 14 (7.1%), capping the best overall baseline at 7 of 25 (28.0%).
By Acting While Thinking, JoyTwin solves 14 of 25 (56.0%) tasks zero-shot in the 1st episode (8/11 Basic, 6/14 RPG)—already 2.0× the best baseline. By Self-Evolving with Experience across just 3 episodes, JoyTwin reaches 19 of 25 (76.0%) (10/11 Basic, 9/14 RPG): a +20.0 percentage-point gain (+5 tasks solved) and 2.7× the best overall baseline.
Times are in-game episode durations. Videos marked 4× Replay are accelerated observation replays, not continuous live recordings.
With a novice sword (~0.6–1.5 HP/hit), a 320 HP Ogre is objectively unwinnable in a 1-on-1 duel (requiring 200+ hits). After dying in a head-on fight on its first attempt, JoyTwin learns to tag the Ogre for quest credit and kite it into the high-level town guard to claim the 1,000-coin bounty.
With novice gear, fighting the 320 HP Ogre head-on chips only 44.9 HP (14%) across 71 hits before the player is overwhelmed and killed (1,016.9s).
Recognizing the stat gap, JoyTwin accepts the contract, lands 7 tagging hits for quest credit, draws the 320 HP Ogre into the town guard, and collects the 1,000-coin payout at 100 HP.
Infer a hidden wall-niche branching rule in a practice wing and transfer it across two unseen dungeon wings within a strict 344m travel budget.
Completes all three dungeon wings with zero deaths, but corner checks and backtracking in the application wings exceed the 344m travel limit (2,700.1s).
Decodes the wall-niche branching rule and traverses both unseen application wings in 330.5m with zero deaths, stopping 1.50m from the final chamber center.
Unlock a sword technique, bind the auxiliary hotbar slot, and trigger the configured skill in live combat.
Engages with basic melee swings before completing the skill-tree hotbar binding (300.5s).
Binds the learned Cross Cut sword technique to the auxiliary slot, closes distance, and defeats the opponent.
Barter within a merchant budget, adapt within an episode after native campfire respawn, and streamline gathering, crafting, and equipment binding.
Purchases and equips upgraded armor within coin budget while retaining remaining gold.
Under native campfire respawn rules, after an unarmored initial push and respawn at the stone arch, JoyTwin adapts by crafting a Rawhide Chestpiece and trading with the merchant for 57 coins before solo-killing both 25 HP Green Legooms.
Completes material gathering, recipe crafting, and equipping with exploratory menu navigation.
Streamlines material gathering, crafting confirmation, and equipment slot binding in half the time.
6.32M cached vs. 0.78M fresh input tokens (~12.2 planner calls/episode on the survival suite)—invoking System 2 only at stage transitions while System 1 acts continuously.
1st Attempt (357.4s · 3.10M tok)
With Experience (174.6s · 1.53M tok)
2.0× faster completion and −50.7% total tokens by streamlining material gathering, recipe crafting, and equipment slot binding.
1st Attempt (91.2s · 2.35M tok)
With Experience (52.4s · 1.12M tok)
1.7× faster encounter resolution and −52.5% total tokens via immediate target lock and downhill melee spacing.
Across 25 Veloren RPG Tasks, JoyTwin’s Acting While Thinking pairs responsive combat and survival with deliberate multi-stage quest progress, while Self-Evolving with Experience raises task success from 56.0% in the 1st episode to 76.0% within just 3 episodes (vs. 28.0% for the strongest baseline). Gameplay replays show how learned experience recovers failed missions and cuts crafting and combat time nearly in half, while incremental working memory sustains long-horizon planning with 89.0% prompt-cache reuse. Together, these results point toward RPG agents that keep acting in a world that never waits, while rapidly self-evolving across a handful of episodes.
The Microsoft JoyTwin team consists of Xunlan Zhou, Xianliang Yang, Kaixin Wang, Chang Xu, Ling Zhang, and Li Zhao.
Leadership
Li Zhao
Research Intern
Research SDE
Senior Researcher
Senior Researcher
Senior Researcher
Principal Researcher
