It took a proficient human over 20 minutes and 24,000 precise actions to craft diamond tools in Minecraft. A neural network just matched that, having learned entirely by watching strangers play on the internet.

Teaching AI to work has always required expensive, hand-labeled data

Building agents that can operate software has historically demanded thousands of hours of annotated examples, making general-purpose computer automation prohibitively slow to develop. The bottleneck was always labeled data, and labeled data costs money, time, and human attention.

A model watches humans, then acts like one

Learning to play Minecraft with Video PreTraining ingests a massive unlabeled dataset of human gameplay footage, bootstraps its understanding using a small pool of contractor-labeled examples, and then fine-tunes to complete long-horizon tasks inside the actual game environment. The input is raw video. The output is an agent that presses keys and moves a mouse the same way a person would. What makes this technically striking is that the interface is entirely native, meaning the model is not given special access to the game’s code or state.

Researchers building the next generation of computer agents are watching this closely

• AI safety researchers who need to study how agents acquire and generalize skills without explicit reward shaping
• Enterprise automation architects who are evaluating whether LLM-based agents can replace brittle RPA scripts that break on UI changes
• Academic ML teams who want a reproducible benchmark for long-horizon sequential decision-making beyond standard Atari or MuJoCo tasks

The VPT approach is a direct challenge to the assumption that useful agents require expensive supervised pipelines from day one.

The race to build general computer-using agents just got a concrete proof point

Competing efforts from Google DeepMind and Adept AI have circled the same problem, but few have demonstrated a task this complex, sustained over this many sequential decisions, using only keyboard and mouse. If this scales to standard desktop environments, the current wave of narrow workflow automations starts to look like a temporary phase.

What the research makes possible

• Train agents on existing screen recordings without writing a single label
• Fine-tune a base model toward specific long-horizon goals in a target environment
• Evaluate agent generalization on tasks requiring 20-plus minutes of sustained planning
• Study emergent tool use in a controlled but open-ended simulation

Pricing not listed — check our directory.

This is a research release, not a product, so there is no API or deployment path available to practitioners yet.

Adept AI is building toward similar general computer-use goals with a product layer already in beta. DeepMind’s Gato demonstrated multi-task agents but did not tackle tasks of comparable sequential depth.

Video pretraining is quietly rewriting how AI agents get built

The methods being validated in Minecraft today are the same ones being stress-tested for enterprise desktops tomorrow. We cover tools like this every Friday — subscribe here and we’ll send the best ones straight to you.