Introducing Ego-OSCAR: An Open-Source Stereo Capture Inertial Stack along with 550 hours of egocentric stereo data

Today we are releasing Ego-OSCAR, an open-source head-mounted stereo-inertial capture device for egocentric data collection in the wild, together with a corresponding dataset of 550 hours captured across 100+ environments, fully open-sourced on Hugging Face.

Egocentric data is still an active research problem, and an open and accessible foundation will empower anyone to participate, experiment, build, and contribute to this space. This continues our past efforts, where we open-sourced Stera, a full-stack capture and processing pipeline for RGB-D data using commodity mobile devices. Ego-OSCAR builds on that lineage and offers a fully open-source stereo-inertial capture stack for anyone working on human-to-robot transfer.

We make three core contributions today:

• An open-hardware capture device with full CAD, wiring and assembly documentation, a complete bill of materials under USD 200, built from commercially available components and 3D-printed parts.
• An open-source capture pipeline - recording daemon, IMU sampler, time-sync tooling and watchdog firmware.
• The Ego-OSCAR-550h dataset: approximately 550 hours of egocentric stereo video per camera with synchronized IMU, released to validate the device at deployment scale. It ships with two corpus-wide annotation layers - 209,315 free-form action segments and per-frame hand detections - rather than as raw sensor streams.

Everything is open-sourced, including hardware designs, capture pipeline, firmware, and the dataset.

Open foundation for human-to-robot transfer

The data demands of vision-language-action models and world action models are growing rapidly. As the field advances toward general-purpose robotic policies, the bottleneck has shifted from model architecture to data: diverse, large-scale, multimodal datasets that capture the richness of real-world interaction.

Each existing collection paradigm carries a distinct cost–quality trade-off:

• Teleoperation yields precise, embodiment-matched demonstrations but scales poorly - every episode requires a physical robot, an operator, and time.
• Simulation offers near-unlimited scale but remains limited by physics fidelity and the sim-to-real gap.
• Robot farms amortize human effort but bias data toward conservative behaviors under hardware safety constraints.

Egocentric data collection offers a compelling middle ground. Human first-person video, captured at scale during everyday activities, naturally covers the diversity of environments, objects, and interactions that embodied AI must ultimately generalize across, and is the best source of cross-embodiment data.

Yet the capture device itself receives very little attention. Existing datasets rely on monocular, rolling-shutter consumer cameras with no hardware-synchronized inertial stream. On the other extreme, closed platforms such as Project Aria offer high sensor fidelity but cannot be freely distributed or reproduced across large contributor networks. We believe this gap can be filled with an open-source stereo device, as modern robotics action prediction pipelines increasingly condition on camera pose in a world frame, and metric-scale pose cannot be recovered from monocular vision alone. Ego-OSCAR is designed to fill this gap: a low-cost, fully open-source stereo-inertial capture device built for deployment at scale and for rapid iteration by the broader research community.

Hardware setup and evaluation

Ego-OSCAR comprises a head-mounted capture device and an accompanying processing pipeline. The system is engineered around four primary technical pillars:

• Affordability - a COTS-only bill of materials, no custom PCBs or proprietary silicon.
• Reproducibility - open-source schematics and a 3D-printable enclosure.
• Sensor fidelity - hardware-synchronized stereo video and a high-frequency IMU for visual-inertial estimation.
• Deployment robustness - real-time fault detection, to maximize data yield across a distributed contributor network.

Rather than executing power-hungry on-device SLAM, we treat pose estimation as an offline batch-processing problem, allowing the hardware to optimize strictly for data ingestion.

The hardware is an all-in-one head-worn device: a hardware-synchronized stereo camera, a single-board computer (SBC) for on-device encoding and storage, a 6-axis IMU, and a real-time microcontroller for timestamping, user interface and watchdog functions. The total assembled mass is ~280 g, mounted via a commodity sports visor.

Every part of the rig, off the cap. Open the 3D viewer to orbit it, isolate a part, or explode it yourself.

The design requirements span three axes: the observation side - sufficient visual context, stereo depth, robustness to rapid head motion; the measurement side - time-aligned inertial data for visual-inertial fusion; and the operational side - autonomous battery operation, real-time fault surfacing and field durability.

Hardware-synchronized global-shutter stereo camera

We use a Dexcin USB stereo camera module: two Omnivision global-shutter sensors, hardware-synchronized through a single ASIC, exposing a single USB 2.0 endpoint to the host. The sensors capture at 30 FPS and 1280×720 px per camera, delivered as one side-by-side stitched MJPEG frame. Each sensor has a 126° field of view, and the pair a 42 mm baseline. Four properties drove the choice:

• Global shutter eliminates the per-row temporal offsets that corrupt visual-inertial estimation under rapid head motion.
• Hardware sync between left and right removes the microsecond-to-millisecond stereo offsets that break depth estimation under motion.
• A single USB endpoint, via the UVC standard, works with existing Linux drivers without dual-cable routing complexity.
• A start-of-exposure trigger - the camera exposes its SoE signal on a STRB pin, which we use to bridge the camera’s internal clock domain to the microcontroller’s.

Unlike monocular wide-FOV approaches that use side mirrors to recover implicit stereo - UMI, for example - Ego-OSCAR captures genuine binocular disparity at the sensor level, enabling standard stereo calibration and rectification pipelines.

Embedded SBC with hardware video acceleration

The recording host is a Radxa ROCK 5C with the Rockchip RK3588 SoC and 2 GB of RAM. The RK3588 provides hardware acceleration for both MJPEG decode and H.264 encode, sufficient to sustain 30 FPS stereo without frame drops on a wearable power budget.

The tradeoff space is narrow: the Raspberry Pi 5 lacks hardware MJPEG decode, and the NVIDIA Jetson Nano and Orin Nano have stronger compute but draw substantially more power and are physically larger. The RK3588 sits at the inflection point where hardware media acceleration meets wearable power and form-factor constraints.

Consumer-grade IMU

The inertial sensor is a TDK InvenSense ICM-20948, from which we record 6 axes - 3-axis accelerometer and 3-axis gyroscope - connected to the Xiao ESP32-S3 over I²C and sampled at 120 Hz. The choice is deliberate: the part sits on a standard I²C bus, so researchers can swap to a higher-grade part such as a BMI088 or ISM330 with no firmware changes beyond the I²C driver.

For most downstream VLA and egocentric understanding tasks the IMU is a secondary signal - assisting visual SLAM in low-feature regimes and providing gravity alignment, rather than serving as the sole pose estimator. A higher-grade IMU adds USD 80–200 to the bill of materials, which is material at deployment scale across hundreds of contributors.

Real-time microcontroller for synchronization, UX and watchdog

A Seeed Studio Xiao ESP32-S3 connects to the Radxa over UART and serves three roles:

• Clock bridging. The camera module and the IMU run on independent clocks with substantial observed offset and drift. The ESP32 taps the camera’s start-of-exposure signal into an interrupt handler: every frame exposure records the ESP32’s current monotonic timestamp. It simultaneously reads the IMU and forwards the merged stream - SoE timestamps plus IMU samples - to the Radxa over UART for real-time logging. An offline pass after the session aligns those frame timestamps to video frame numbers to produce a synchronized trace.
• User feedback. The MCU drives an RGB LED strip to signal device state - booting, ready, recording, error - from a 1 Hz heartbeat sent by the Radxa over UART.
• Watchdog. If the Radxa hangs during recording, the MCU detects the missing heartbeat within two seconds and signals an error state. Without this, a wearer can record for an hour after the SBC has hung without realizing - a failure mode we observed repeatedly in early field deployments.

The capture path: sensors and ISP on the left, encode and storage on the SBC, with the microcontroller bridging clock domains and watching the host.

Battery, enclosure and field durability

The device is powered over USB-PD from a 10,000 mAh power bank, providing 5–6 hours of sustained capture; the bank is hot-swappable for all-day sessions.

For a detailed breakdown of the hardware design choices and trade-offs, read the full paper and the build guide.

We evaluated the device along three tiers corresponding to its design goals.

Sensor fidelity

Stereo geometry. Calibration is per session, not per device model, which matters for a fleet of hand-assembled units whose optical alignment differs slightly and can shift with handling. Each session is calibrated from an 8×6 chessboard with 30 mm squares at the native 1280×720 capture resolution, fitting a pinhole model with radial-tangential distortion (k1, k2, p1, p2, k3) per camera, plus a 3×3 rotation, a translation, and the ~42 mm baseline. The resulting per-camera reprojection error is below 0.03 px, and the calibration ships as a calibration.json alongside every session, so downstream users are never relying on a nominal factory intrinsic. Across all 13 deployed devices, the mean per-pixel epipolar error after rectification is 0.4 px. The 126° per-sensor field of view introduces predictable barrel distortion at the periphery, which the calibration step removes cleanly. Disparity estimation with SGBM and RAFT-Stereo succeeds across the full field of view without specialized tuning.

IMU noise. We characterize the ICM-20948 with an Allan variance protocol over 12 hours of stationary capture. Accelerometer noise density is 3.64 × 10⁻² m/s²/√Hz and gyroscope bias instability is 9.68 × 10⁻⁴ rad/s, placing the part in the consumer-grade range, comparable to phone-grade IMUs. For our target applications - gravity alignment, motion classification, and visual-inertial fusion as a coarse prior - this noise floor is acceptable. For long-horizon inertial integration we recommend the higher-grade IMU swap described above.

Visual-inertial synchronization. After the per-session offset correction, the residual lag between the visual and inertial streams is 700 µs, validated using Kalibr’s Cam-IMU offset test.

Quality control

Data reaching the release passes three successive filters. We state them explicitly because a corpus assembled from a distributed contributor network is only as trustworthy as its rejection criteria.

• At capture. The watchdog ends the illusion of a recording session that is not recording: losing the 1 Hz heartbeat for more than two seconds puts the device into a visible error state, so the wearer stops rather than continuing for an hour against a hung SBC. That is the difference between a failed session and a silently empty one.
• Per batch. Every uploaded batch is validated for decodability, expected clip count and duration, and the presence of the companion IMU and calibration artifacts.
• At selection. Sessions entering the release are screened for hand visibility, which is what makes the corpus consistently rich in hand–object interaction rather than merely long. Sessions that fail calibration or lack a usable synchronized trace are excluded rather than shipped with caveats: the IMU is present in 1,271 of the 1,462 released sessions (86.9%), and the rest are released without an inertial stream rather than with an unverified one.

The end-to-end effect of these filters is the 96% usable-session rate reported below.

Data utility

SGBM and RAFT-Stereo recover dense disparity maps across the full field of view on held-out sequences. The 42 mm baseline and 126° field of view provide reliable depth in the 0.5–4 m range typical of indoor egocentric capture.

We run VINS-Fusion in stereo-inertial mode on 20 held-out sequences of one to three minutes: 12 of 20 produce stable trajectories, while 8 diverge - a combination of brittleness under the dynamic scene content and rapid head motion characteristic of egocentric capture, and IMU bias accumulation over longer segments, consistent with the noise floor above. In the same environments and activities an Intel RealSense reaches 15 of 20, holding an unfair advantage through active stereo depth.

We are deliberate about what this number is and is not. It is a convergence rate, not a trajectory-accuracy result: we have no motion-capture or surveyed ground truth for these sequences, so we report no ATE or RPE, and a reader should not infer metric pose quality from the 12/20 figure. It characterizes the operating envelope of the current device - where off-the-shelf visual-inertial odometry does and does not hold up on this data - which is why we do not ship a camera-trajectory annotation layer. A ground-truthed evaluation requires an instrumented capture campaign we have not yet run; we list it as a limitation rather than approximating it here.

Deployment scale

Across all sessions in the deployment, 96% produced usable data end-to-end. The three dominant failure modes - thermal shutdown during sessions exceeding 90 minutes above 35°C, SD card I/O errors, and cable strain at the camera-to-SBC connector - were each identified in the field and engineered out. The watchdog protocol detected all three failure classes in active deployments, preventing wearers from unknowingly continuing sessions with no data being saved.

The Ego-OSCAR-550h dataset

A capture session with its dense action labels, as annotated by Vidur.

To validate Ego-OSCAR at deployment scale, we deployed it across a contributor network of 25+ operators, who collectively captured ~550+ hours of data across 100+ environments, including residential kitchens, living rooms, bedrooms, and small commercial settings.

Each session ships with:

• Synchronized stereo video (MP4 H.264, 30 FPS)
• Offset-corrected 6-axis IMU data
• Per-session stereo calibration (pinhole + radial-tangential, sub-0.03 px reprojection error) - enabling metric, geometry-aware use of the stereo pair
• Per-session metadata

The dataset also contains dense action annotations performed by Vidur, our internal system for dense action labels, with 209,315 labeled segments covering the entire recorded timeline and a median of 94 segments per session. The captions are free-form and highly descriptive - 460 action verbs, 32,630 object phrases, and 57,104 distinct verb–object combinations. The top 20 task expressions account for only 1.5% of all instances, and no single expression exceeds 0.31%: a genuinely long-tailed distribution combining repeated coverage of foundational manipulation skills with a long tail of rare task expressions.

All data above was collected with informed consent and environment-owner permission, and faces are blurred in the released version. This dataset demonstrates that Ego-OSCAR is not only research-grade hardware but a capture stack ready for fleet-scale deployment - albeit one where the team that takes it to deployment addresses the limitations below.

Limitations and future directions

We do not demonstrate that a policy trained on Ego-OSCAR data outperforms one trained on existing corpora: our evaluation validates the sensor package, not its value for robot learning, and closing that gap is the most important follow-on work.

• IMU: the consumer-grade IMU is the dominant pose-error source. It is swappable on the same I²C bus, but not yet benchmarked.
• Capture only: all SLAM, hand-tracking, and downstream analysis run offline. This is a deliberate power and form-factor trade-off.
• Moisture resistance: the current enclosure is not moisture-resistant.
• Trajectory accuracy: the paper reports visual-inertial odometry convergence on 12 of 20 held-out sequences; this is not a trajectory-accuracy guarantee. More robust evaluation is recommended after changing the IMU.
• Weight: the device currently weighs ~280 grams. This can be reduced further through cleaner CAD for the mounts and smaller, dedicated PCB designs.

Get started

The full dataset is available on Hugging Face, and the CAD models, firmware and capture codebase are on GitHub, alongside a step-by-step DIY guide and the printable enclosure files. The interactive 3D view of the rig is the fastest way to see how the parts go together before you print anything.

Our hope is to provide an open foundation for robot learning that we wish we had. We would love to hear what you do with these resources. Reach out at [email protected][email protected] if you need help, or find us on Discord.