Introducing FPV-SLAM: State of the Art SLAM for egocentric data

Today we are announcing FPV-SLAM, a state-of-the-art SLAM system for egocentric data. FPV-SLAM ranks 1st in the monocular-inertial track and 3rd in the binocular-inertial track of LaMAria, ETH Zurich's city-scale benchmark.

FPV Labs’ goal is to invest in the buildout of the data engine and infrastructure that accelerates deployment in the real world. We believe human-centric learning is the fastest way to achieve that, where robots learn from everyday people doing everyday work and the core of this engine is our SLAM system.

FPV-SLAM is open for early access today.

Why SLAM is the core of our data engine

FPV-SLAM turns first-person videos into 3D environments that robots can learn from. Egocentric video becomes training data only when we know where the camera was at every frame, and every layer of the engine depends on that:

• Hand tracking. Hand labels accurate to the millimeter are useless if the head pose they hang from is off by centimeters.
• 3D reconstruction. Scenes, objects, and their geometry are fused across frames using the camera trajectory.
• Trajectory extraction. The motion a robot learns to imitate is expressed in the world frame, and getting it there requires the pose.
• Transfer to robots. Building real-to-sim environments and transferring skills both need metric, consistent motion.

The research paradigm splits into a few subthreads: Vision-Language-Action Models (VLA), a world model used with an inverse dynamics model, or a World Action Model (WAM). The amount of human action data required differs across these regimes, but the demand for human data seems to be answered resoundingly positively. SLAM is very critical to facilitate the buildout towards this future and is at the center of everything we do

Why existing SLAM falls short

Most existing SLAM systems were built for cars, drones, and handheld rigs, where motion is smooth, and the camera mostly looks ahead. In the wild, egocentric data presents unique problems where existing classical SLAM systems fall short in many ways -

• Fast, aggressive head motion. A quick glance sideways moves features across most of the image in a few frames.
• Motion blur and hard lighting. Frames blur during head turns and go dark or wash out between rooms, under furniture or at night.
• A busy world. People walk through the frame, and reflections and moving vehicles pull the estimate toward motion that isn't the wearer's.
• Little parallax. Someone sitting at a table turns their head constantly but barely translates, which leaves the geometry weakly constrained.
• Hands and arm's-length work. Hands cover the view, and close surfaces have little texture, so features slide off the image.
• Drift over long horizons. With one camera, there is no stereo baseline to fix scale, and no second view to catch a bad track. Small errors add up over minutes and kilometers.

The classical methods show how hard this is. On LaMAria's monocular-inertial track, ORB-SLAM3 averages 10.0 out of 100, and OpenVINS averages 7.8.

High-quality SLAM is locked inside big companies like Meta, Apple, and AR/VR companies who have a decade-plus lead in this field. On Aria’s city-scale benchmark, open-source and academic pipelines are very far behind.

FPV-SLAM proves that high-quality egocentric SLAM is no longer locked inside big AR/VR companies, and that SLAM innovation can also happen in small, leaner startups.

LaMAria benchmark

LaMAria is the hardest public egocentric SLAM benchmark available today. It was recorded with Project Aria glasses, with data ranging from a few hundred meters to several kilometers through a city, day and night, on foot and on trams and buses.

• Test set: 63 walks in five categories: Short, Medium, Long, Low light and Moving platform.
• Ground truth: surveyed control points accurate to centimeters. They are hidden from participants, which leaves no room to tune for the test set.
• Scoring: each walk is aligned with a single similarity transform. The horizontal error at every control point is then converted to points: 5 cm earns 100, 1 m earns 60, 5 m earns 20, and 10 m or more earns nothing.

There are two inertial tracks on this benchmark. The monocular-inertial track allows one of Aria's greyscale tracking cameras plus IMU. The binocular-inertial track allows both cameras. The mono track is harder because the IMU must provide metric scale entirely through motion.

What we built

FPV-SLAM is one pipeline that runs with either monocular + imu or binocular + imu. It has three parts, and each one targets a specific way egocentric tracking fails.

1. An odometry built for egocentric footage

The core of FPV-SLAM is a sliding-window visual-inertial odometry. We adapted every stage of it for head-mounted video:

• A front end that holds on to its features. A patch-based optical-flow tracker with contrast normalization keeps texture in dark and washed-out frames. Feature selection favors corners that survive motion blur.
• Range for fast head turns. Features are tracked across an image pyramid, so a quick sideways glance doesn't lose the track.
• Tight IMU coupling. The IMU is pre-integrated between frames and solved jointly with vision in a fixed-lag window. At start-up, the accelerometer supplies gravity, which gives the estimator a correct vertical from the first frames.
• Outlier rejection after every solve. Bad tracks from moving people, reflections or blur are dropped before they can bend the trajectory. A robust loss (Huber or Cauchy) limits the pull of the tracks that remain.

2. Per-session calibration

Factory calibration is a starting point, not the truth. Calibration can shift in small amounts due to a wide variety of factors ranging from mechanical factors, temperature factors, rigidity of lens fixtures, etc. A focal length off by a fraction of a percent becomes a scale error, and a slightly wrong principal point becomes a slow rotation drift.

FPV-SLAM calibrates every recording per session, from its own footage, with no target and no special motion. It optimizes camera parameters against the recording itself instead of reading them off the factory sheet. This improves the estimate in both tracks, and it is a large part of why we can promise accuracy on commodity devices whose factory calibration is weaker than Aria's.

3. Keyframe visual-inertial bundle adjustment

On top of the odometry, a global refinement solves for all keyframe states and landmarks over the whole walk at once, with long-range feature matching and loop closure. A visual term ties every keyframe to what the camera saw, and an inertial term ties consecutive keyframes to the motion the IMU measured, which keeps the trajectory in metric scale.

When a walk revisits a place, the mismatch between the two estimates becomes a signal that corrects every frame in between. Refinement helps most where drift builds up, which is why it lifts our monocular results the most.

Results

In the mono plus IMU category, FPV-SLAM ranks #1 on the LaMAria benchmark and leads in every one of the five categories. LaMAria orders the board by short score by default, and FPV-SLAM tops it with 61.4, ahead of Mighty Camera at 57.1. It stays ahead of every other submission and every reference baseline, regardless of how the board is sorted.

FPV-SLAM also beats the best score any other submission posted in each category, a combined bar no single method reaches. The biggest margins come exactly where egocentric tracking usually falls apart: +12.2 in low light and +10.1 on long walks.

FPV-SLAM also has the highest R@5m in every category where it is reported. R@5m is the share of control points within 5m, and on long walks FPV-SLAM reaches 86.9% against 76.9% for the next best.

Against the classical baselines, FPV-SLAM scores 4.5x more than ORB-SLAM3 (10.0), 4.8x more than OpenVINS + Maplab (9.4), and 5.8x better than OpenVINS (7.8).

On the binocular-inertial track, FPV-SLAM ranks #3 on the leaderboard with a score of 77.3 on short sequences, which is behind Aria’s own SLAM and Grounded SLAM.

Is LaMAria the right benchmark for robot learning?

We think LaMAria is necessary but not sufficient. A carefully engineered egocentric SLAM system should score well on it. A high score alone, though, doesn't show that a system will work on the in-the-wild data used to train robot policies.

What LaMAria gets right

• Real egocentric motion on a real wearable, in the wild: head turns, passers-by, night, trams.
• Hidden, surveyed ground truth that can't be overfit.
• Long horizons, with walks of several kilometers that expose drift short sequences hide.
• Breadth: five categories and 63 test walks.

Where it differs from data for robot learning

• Motion. LaMAria is mostly city walking: long trajectories, steady forward translation, surfaces meters away. Policy-training footage is short, with fast head motion and sometimes almost no translation. Think of someone sewing at a table, working at arm's length.
• Scoring. The score uses horizontal position at sparse control points, after a Sim(3) alignment that forgives uniform scale error. Real training data has no ground truth to align against, and a robot needs metric motion. The trajectory has to be right in SE(3), at the right scale, straight out of the tracker. If human data is used like robot training data, the field needs more precise pose estimation over time, not less.
• Hands and occlusion. Walking rarely puts a hand in front of the camera. Manipulation does it constantly.

Limitations with FPV-SLAM

• Moving platforms are still hard. We score 25.8 in mono and 29.6 in bino. An estimator tuned for walking can't fully separate a tram's motion from the wearer's. We expect learned dynamic-object masks combined with robust model fitting (RANSAC or MAGSAC++) to push this higher.
• Long open-loop walks drift. This is especially true with one camera. Loop closures in a city are rare and unevenly spread. Without a revisit, drift can be delayed but not eliminated.
• Offline, not real time. The odometry alone runs well above real time. With refinement, the full pipeline runs at about 10 frames per second on a 16-core cloud machine.

We thank the LaMAria team at ETH Zurich for the benchmark, its evaluation server, and the leaderboard they create and maintain.

Get early access

FPV-SLAM is open for early access for researchers and developers. If you are collecting large-scale egocentric data or are training robot policies, book a call with us to get early access to FPV-SLAM.