How accurate is an iPhone tracking, really?

ARKit estimate (green) traced over Vicon ground truth (red), all eight sequences.

The egocentric data collection paradigm is becoming fuel to train general-purpose robot policies. The data pre-processing steps (including hand tracking, depth estimation, and scene reconstruction) all rest on knowing one thing: where the camera was at every frame. This dependency on camera pose makes it a critical bottleneck, and any error in pose propagates into the downstream estimations, drastically affecting the overall quality of the dataset. Existing methods rely on research-grade hardware, which is expensive and not accessible to the public, which quietly caps how much data can be collected, since the barrier to entry is a device most contributors will never own.

Our work, Stera, removes that barrier by using the phone people already carry. But that raises an obvious question: is the phone's tracking good enough?

Shape is not scale

A qualitative answer to this question could be inspecting whether a trajectory looks visually plausible or whether the reconstruction outcomes are coherent. But while this looks right, it overlooks a fundamental issue with monocular visual-inertial systems: scale. Monocular systems recover trajectory shape, but not its absolute metric scale. A scale error can propagate throughout a trajectory, silently stretching or shrinking distances, leading to ill-conditioned pose estimates and, in turn, corrupting downstream data. So a real validation has to test two distinct things:

• Metric accuracy: does a meter on the phone equal a meter in the world, across the motions that dominate real tasks?
• Long-horizon drift: does that accuracy hold over the hour-plus sessions Stera is built to capture, not just over a few seconds?

In this blog, we evaluate the pose estimates provided by commodity hardware (an iPhone 12 Pro) against a high-precision Vicon optical motion-capture system, over a diverse sequence of motions. We measure both the absolute trajectory error and the local drift (relative pose error) under SE(3) alignment, that is, without any scale correction applied.

The hypothesis

Accurate egocentric pose estimation does not require specialized hardware. A commodity, LiDAR-enabled iPhone running ARKit provides metric, drift-bounded 6-DoF pose suitable for downstream tasks such as world-frame hand-pose anchoring, depth unprojection, and dense action annotation over both short, varied motions and long, continuous sessions.

Metric accuracy against Vicon

To establish metric accuracy, we benchmarked ARKit against a 24-camera Vicon optical motion-capture system: sub-millimeter marker accuracy at 200 Hz. An iPhone 12 Pro was rigidly co-mounted with a cluster of 11 retroreflective markers on a head-worn rig. The Vicon system fits the known cluster geometry to the tracked markers every frame, yielding full 6-DoF reference poses that are robust to partial marker occlusion; ARKit's visual-inertial fusion ran concurrently at 60 Hz.

Before computing any error, we time-align the two streams to a common clock and solve the rigid transform between the marker cluster and the iPhone camera frame, so we are comparing the two estimates of the same physical pose.

We then recorded eight sequences spanning the motion regimes that occur during general egocentric tasks:

1. Walking exploration: free walking with head rotations, moving distractors, a seated interaction, and partial occlusions.
2. Walking navigation: free walking with head rotations and moving distractors, focused on continuous navigation.
3. Straight-line traversal: repeated traversals along a fixed straight-line path between reference markers, captured at several walking speeds.
4. Seated egocentric manipulation: opening a bag, handling cameras, and folding clothes.
5. Fast manipulation: egocentric motion at higher speed, with abrupt movements and increased motion blur.
6. In-place rotation: steady in-place rotation on a swivel chair, producing pure-yaw motion.
7. Aggressive rotation: in-place rotation with rapid full revolutions and high angular velocity.
8. Height change: an activity adding height variation, with the participant climbing a wooden box.

The 24-camera Vicon capture volume: infrared cameras tracking reflective markers at 200 Hz with sub-millimetre accuracy.

Across all eight sequences, ARKit maintains high position accuracy, preserving real-world metric scale, and tracks orientation with sub-degree precision. To evaluate this, we report the Absolute Trajectory Error (ATE), which measures how closely the estimated trajectory matches the ground truth, and Relative Pose Error (RPE), which measures short-term drift between consecutive poses. Results are evaluated with the evo toolkit under rigid-body SE(3) alignment (i.e. without scale correction) to verify whether the trajectories are actually metric.

Absolute Trajectory Error (ATE RMSE, SE(3)-aligned) ranged from 6.0 to 12.5 cm. Relative ATE remained below 1% on seven of the eight sequences (0.12–0.35%); the only exception was the in-place rotation sequence at 1.30%, an artifact of the short path length rather than a tracking failure: the sequence covers only 5 m, so a small absolute error (6.5 cm) becomes a large percentage. Its RPE figures are in line with every other sequence. Rotational Relative Pose Error (RPE) stayed at or below ~1° across all sequences (below 0.6° for the walking and manipulation sequences), while translational RPE remained under 5 cm throughout. These results indicate strong local pose consistency even where global drift is slightly elevated. A Sim(3) alignment recovered a scale factor close to unity (0.98–1.01): rescaling the ARKit trajectory by at most 1–2% suffices to align it with the metric ground-truth trajectory, further confirming ARKit's metric accuracy. Overall, ARKit's pose estimation on a consumer iPhone Pro provides trajectory accuracy sufficient for reliable world-frame hand-pose anchoring.

Drift over long horizons

Vicon answers the metric-accuracy question, but it cannot answer the durability question. Stera is built around long-horizon capture sessions averaging 21 minutes and running up to 108 minutes of continuous recording, and a mocap volume can neither follow a contributor through a real home nor run for an hour. ARKit is also closed-source, so we cannot inspect its loop-closure internals directly. So we test drift behaviorally, where it actually has to work: in real environments, over real session lengths. We place an ArUco marker in the scene, observe it during the first minutes of operation, then revisit it twice, roughly at the temporal midpoint and again near the end of a long session. In a system with good long-term tracking and loop closure, the marker should reappear in the same place in the camera's reference frame each time; the gap between sightings is the accumulated drift.

Across all three environments, accumulated drift stayed under 1 cm in most cases and under 0.1% of trajectory length in every case, including a whole-house traversal, the hardest setting for sustained tracking. Drift stays bounded over exactly the hour-scale sessions the framework is designed to collect.

Why this matters downstream

Pose is not a nice-to-have; it's a prerequisite, which makes everything downstream relevant. In the Stera pipeline, each RGB frame is paired with a LiDAR depth map and an ARKit 6-DoF pose. We detect 2D hand keypoints with WiLoR, unproject them into 3D using the depth map, and transform them into a consistent world frame using the recorded camera pose, producing 21-joint MANO hand trajectories anchored in a single global frame across an entire session.

Every step of that chain inherits the pose error. A metric, drift-bounded trajectory is what keeps a reach-and-grasp that happens forty minutes into a session spatially consistent with one from the first minute, the property long-horizon VLA training depends on.

What this proves

For the motions that dominate real-world egocentric data (walking, reaching, looking, turning, changing height), a commodity iPhone delivers high pose accuracy, drift stays bounded across longer runs, and results hold consistently across motion types. It requires no calibration, no capture volume, no device most people will never own.

This is not intended as a critique of research-grade hardware. Rather, the key takeaway is more practical: our results demonstrate that accurate pose estimation is not confined to specialized laboratory setups. High-fidelity trajectory data can be collected using commodity hardware, substantially lowering the barrier to large-scale data acquisition and enabling broader participation in data-driven robotics research.

Acknowledgements. Thanks to the Government of Karnataka (Dept of IT, BT & S&T), ABAI Management, and the AVGC Centre of Excellence, Bengaluru, Lawrence Anthony and team for opening up their motion-capture studio and helping us run these benchmarks. The metric ground-truth comparison would not have been possible without access to their Vicon setup.