Introducing Project Stera: An Open Infrastructure for Embodied AI Research

Today, we are releasing Project Stera, a full-stack platform that democratizes high-fidelity multimodal data capture for embodied AI research.

We're releasing two things today for the broader Embodied AI community:

• Stera-10M comprising 10M frames and 200 hours of diverse, long-horizon multimodal data with persistent state tracking
• Stera Suite: a comprehensive processing pipeline that converts raw mobile captures into high-fidelity training-ready formats for VLA, World Models, and Embodied AI research

This post walks through what we've built, why a democratized platform is required to accelerate progress in embodied AI, and how you can use Stera.

Current state of embodied data

It's very evident that embodied AI is data-bound today. Every foundation model that learns from real-world demonstrations (VLAs, world models, full-body motion estimators, scene-grounded language models, hand-object interaction networks) scales with the volume and quality of the multimodal data it trains on.

Real-world data today is extremely fragmented and expensive to produce. Labs use a wide range of rigs from Meta Aria glasses to Apple Vision Pro, GoPro rigs, custom UMI rigs, Quest 3, Intel RealSense, Zed, Oak Pro, and the long tail of custom hardware, each producing different modalities in different formats with different processing pipelines and having to write custom pipelines to make them training-ready.

A large share of what gets collected is either low-fidelity (raw RGB without depth, missing cross-modal synchronization, no 6-DoF tracking, and stitched-together annotation pipelines with no coherent quality control) or the downstream hardware that generates high-fidelity data is either access-gated or prohibitively expensive, with Project Aria being a canonical example.

We are happy to unveil Stera today, which addresses this gap in the market with commodity infrastructure that turns it into a research-grade capture rig anyone can run without having to put together the harness again.

Stera Suite

Stera is built around a four-stage loop: Capture → Process → Evaluate → Export.

Overall data flow: raw mobile capture (RGB-D, IMU, ARKit pose) is logged in MCAP format, then processed offline into 3D hand trajectories, atomic action labels, and a hierarchical instruction tree.

Stera Capture

Stera Capture is an iOS app that turns an iPhone Pro into a research-grade capture rig.

The app is entirely voice-controlled. “start” and “stop” trigger recording without breaking the natural flow of the activity, which matters when you're capturing hour-plus household sessions and don't want to break to a touchscreen every time you need to pause. The app records synchronized RGB, LiDAR depth, IMU, and ARKit 6-DoF pose into a raw MCAP file, which can be visualized directly or processed through the Stera SDK.

Stera Process

The processing pipeline is an open source SDK that takes raw MCAP recordings and produces the modalities the field trains on. It allows researchers to turn capabilities on and off through a single configuration without modifying the core code.

The pipeline produces:

• 6-DoF trajectory and world-frame anchoring, with ARKit traces aligned into a consistent global frame so hands, bodies, objects, and scenes anchor in shared coordinates and support cross-frame queries at the session scale.
• 21-joint MANO hand poses for both hands, recovered by an ensemble of SOTA 2D keypoint detectors, unprojected to 3D using LiDAR depth, and anchored in the global frame.
• Hierarchical action language, generated by a configurable VLM backend with constrained prompting that forces object modifiers and spatial prepositions (“transfer dough from metal bowl to large plate” rather than “place dough”). Atomic spans organize into a four-level instruction tree (session → sub-goal → episode → atomic span), enabling language conditioning at temporal scales from five-second manipulations to minute-scale sub-goals to full session plans. The launch corpus covers 45,415 distinct action categories.
• Upper-body tracking lifted from camera pose and 3D wrist positions, producing a skeleton (head, shoulders, elbows, wrists) with tunable limb lengths.
• Map and geometry, including SLAM mesh, point cloud, dense depth-derived reconstruction, and vertex texture-mapping from RGB frames.

Over the next few months, we plan to add full-body pose estimation, 3D Gaussian Splat scene reconstruction for data augmentation, and real-to-sim export.

Evaluate

Stera ships with an evaluation layer. The pipeline scores each recording on multiple health metrics like hand tracking, frame drops, depth-RGB synchronization, camera pose-related metrics, and much more.

Export

Stera produces outputs in the formats the field already trains on. The exporter covers MP4, PLY, MCAP, HDF5, RRD, and many more formats on request.

Stera-10M

Stera-10M is the open dataset captured end-to-end through the stack. It demonstrates what the platform produces at scale and stands on its own as a research-grade corpus.

The launch release includes:

• 200 hours of recording across 354 sessions
• 20 contributors across 50+ unique environments
• Mean session length of 21.2 minutes, with the longest at 108 minutes of continuous capture
• 10 million RGB frames, 45,415 atomic action spans, 5,570 episodes, 1,298 sub-goals
• 21-joint MANO hand poses anchored in a globally consistent world frame
• Per-frame LiDAR depth, ARKit 6-DoF pose, IMU, and room mesh
• Faces blurred via EgoBlur; all contributors recorded under informed consent

Compared to existing egocentric datasets, Stera-10M occupies a distinct position:

Stera-10M is the first open dataset that combines hour-plus continuous capture with depth, 6-DoF pose, and dense hand annotations on commodity hardware that a researcher already owns.

What you can build with Stera

Stera is built for a wide spectrum of users to accelerate progress in embodied AI.

Students

• Build hand-conditioned manipulation policies on 21-joint MANO trajectories anchored in a world frame
• Develop hierarchical action models that condition on language across four temporal scales
• Run thesis work on 3D scene understanding using per-frame depth and room mesh, without access to gated hardware

Researchers

• Pretrain video world models on continuous long-horizon footage with per-frame depth, 6-DoF pose, and mesh geometry
• Train hierarchical action models that condition on language from five-second manipulations to seventeen-minute sessions
• Run sim-to-real pipelines by capturing real environments, reconstructing them, exporting to Isaac Lab USD, and training policies in simulation
• Develop 4D scene and motion models that jointly reason about geometry and human activity over hour-long horizons

Labs

• Run multi-iPhone captures for ego-exo perception research
• Operate institutional capture programs at scale: multi-environment, multi-contributor studies on commodity hardware
• Score every recording through the open quality layer before it enters training
• Build new evaluation suites for egocentric perception, action localization, or hand pose recovery on top of Stera-10M's structure

Startups

• Integrate the SDK into your training stack and run captures at whatever scale your product needs
• Ship VLA-based products without rebuilding the data pipeline; the same export format runs from capture to training to production
• Hand the iOS app to data partners, annotators, or in-house operators for distributed capture
• Use LeRobot exports directly in training without format conversion overhead

Over the coming weeks, we will showcase research directions and real-world applications that Project Stera unlocks: manipulation datasets, dense action labels for spatial 4D reasoning, navigation-specific datasets, real-to-sim transfer, and much more.

The data foundation for embodied AI should be open, accessible, and in the hands of every researcher who wants to use it. Stera is open today.