Introducing Vidur

Today, we’re releasing Vidur, our production-grade system for fine-grained action labeling of raw robot and human egocentric videos. Given a raw demonstration, Vidur identifies each atomic action, determines when it starts and ends, and writes a self-contained natural-language label - returning a validated, gap-free timeline covering the video from first frame to last.

We evaluated Vidur on two external benchmarks (WGO-Bench and EgoExoLearn), achieving state of the art on both, and highlight use cases where Vidur will be helpful for downstream partners.

Highlights

• Breakthrough performance. Vidur achieves 0.386 semantic end-to-end F1 on the full 100-episode WGO-Bench evaluation - the highest published result, 38% above the next-best system and 2.3× the strongest reported Gemini-based pipeline (0.168, Macrodata WGO + seeded relabeling), with the best GPT-5.6 pipeline at 0.198 and Claude Opus 4.8 at 0.103^[2][3][4]. See Benchmark below for methodology and known scoring limitations.
• The advantage generalizes. On a 10.23-hour evaluation of long, dense, instruction-free first-person video from EgoExoLearn, Vidur reaches 0.239 semantic end-to-end F1 against 0.157 for Gemini 3.5 Flash - a 52% relative improvement. It wins across all eight task groups, at lower cost^[5][6].
• Tunable accuracy - cost operating point. Vidur separates video understanding, temporal segmentation, semantic labeling, and deterministic quality control into distinct stages, letting teams choose the right balance of granularity, accuracy, and cost per use case - from bulk pre-labeling of fleet footage to gold-set creation.
• Full coverage. Vidur returns a validated timeline from first frame to last: sorted, non-overlapping segments with no silently missing labels. Unlabeled spans are marked as unlabeled, not dropped - so you can tell a fully covered video from one the model gave up on.
• Changes the economics of annotation. Manual annotation runs approximately $20-50 per video-hour, with one minute of egocentric footage requiring ten or more minutes of work^[2]. Vidur labels directly at a fraction of that cost.

Overview

Vidur extracts and structures action supervision across all kinds of egocentric videos. Where coarse captioning converts an episode into a single task-level description, Vidur returns a structured representation of the demonstration where each atomic action is localized with start and end timestamps, described in self-contained, open-vocabulary language, and assembled into a timeline that is validated for coverage and consistency. Downstream systems therefore have access not only to what happened in the video, but when each action occurred, in what order, and whether the timeline is complete.

This structure supports several downstream workloads:

• Training samples for VLA policies: each timestamped segment pairs relevant frames with a specific behavior, rather than assigning a single label to the full episode. Prior research has shown that on a corpus of more than one million robot clips and 50K egocentric videos, dense re-annotation improved both VLA and world-action-model performance without collecting any new demonstrations ^[1].
• Dataset curation and retrieval: finding every pouring action in a corpus, measuring how long insertions take, or filtering failed grasps all require footage indexed at the level of individual actions.

Vidur accepts complete robot and first-person episodes - teleoperation, on-policy robot-camera, and egocentric head-camera video - across embodiments and viewpoints. It has been iterated on for 20K+ hours of internal data to serve as a production-grade system and is now being opened up to external partners.

Benchmark

To evaluate Vidur, we compared it against leading published annotation systems, frontier general-purpose models, and pipelines built on them, across two external benchmarks with very different shapes.

A new state of the art on WGO-Bench

WGO-Bench is a public benchmark for automated robot-video subtask annotation. It contains 100 complete episodes, 743 manually annotated action segments, and 62 task instructions, spanning egocentric video from HomER, external robot-camera video from DROID/RoboInter, and robot head-camera video from Galaxea^[2]. It measures three related capabilities: segment F1 - did the system find the correct time windows; label accuracy on temporal matches - once it found a window, did it describe the right action; and semantic end-to-end F1 - did it get both the time window and the action label right.

Semantic end-to-end F1 is the metric that best represents the finished product. A prediction receives semantic credit only after it overlaps a human action at temporal IoU ≥ 0.75 and its label passes the benchmark’s semantic judge - a deliberately strict bar. We ran Vidur on all 100 WGO-Bench episodes and all 743 gold segments, scored with the benchmark’s default LM judge ^[4].

Vidur consumes video natively; GPT-5.6 and Claude Opus 4.8 accept no video input, so any pipeline built on them must first convert each episode into images. We follow Macrodata’s published recipe - sampling a frame every 0.5 seconds and tiling 20 frames per contact sheet with timestamps burned into each tile - which is also how their own model sweep, Gemini included, was run. This is the standard way these models are benchmarked on WGO-Bench, but it means the comparison is between end-to-end pipelines rather than between models on matched inputs: part of the gap reflects input representation, not model quality alone.

Vidur raises semantic end-to-end F1 from the previous high of 0.280^[3] to 0.386, a 38% relative improvement. Compared with Macrodata’s strongest reported seeded-relabeling pipeline, Vidur scores 2.3× as high. It leads not only on the headline number but across every metric, from boundary detection to final labels.

In simpler words, Vidur finds more of the actions humans marked, emits fewer incorrect labels, and more often puts the correct description in the correct time window.

Better timing and understanding

Action labeling systems can make themselves look better by overproducing segments, improving recall while flooding the dataset with false positives. They can also be conservative, emitting only a few easy actions and sacrificing coverage. Vidur’s advantage holds on both sides: semantic precision of 0.433 means more of the labels it emits are fully correct, and semantic recall of 0.349 means it recovers more of the human-annotated actions. This balance is why the end-to-end result matters. The highest-quality timeline is not the one with the most labels or the longest descriptions - it is the one that gets the greatest number of real actions right without manufacturing extra ones.

Seconds-long actions are first-class actions

Robot and first-person demonstrations are dense. A pick, release, insertion, or quick placement may last only a few seconds, and sometimes barely more than one. Vidur correctly localized and labeled WGO actions such as these:

These are real examples from WGO-Bench, not hand-written illustrations. For each row we show the human annotation, Vidur’s prediction, and their temporal IoU - the amount of overlap between the two time windows. High IoU combined with equivalent wording means Vidur identified both the correct action and its boundaries.

Vidur’s strength does not depend only on long, obvious subtasks. On WGO’s 2-5 second actions, Vidur recalls 48.4% of human-annotated actions, against 23.7% reported for Macrodata’s pipeline.

EgoExoLearn Bench

WGO-Bench tests complete robot and human egocentric episodes, but one benchmark cannot establish generalization. We therefore ran a second evaluation in a substantially different operating environment: long, dense, instruction-free first-person video. It has much longer, untrimmed first-person videos; no task name or procedure instruction; no fixed label vocabulary; no audio or transcript; no gaze signal or exocentric demonstration; and human actions with a median duration of only 1.9 seconds.

We completed a fixed, deterministic evaluation from EgoExoLearn^[5]: 48 complete egocentric videos, 10.23 hours, and 7,823 human-annotated fine actions, covering five kitchen task groups and three laboratory task groups.

For this comparison, we tuned Vidur’s inference budget to run at lower cost than Gemini 3.5 Flash. Even at that budget, Vidur achieved 0.239 semantic end-to-end F1 at tIoU 0.50 against Gemini’s 0.157 - a 52% relative improvement - and leads on every metric, with identical inputs and protocol for both systems and wording judged by a third-party LM^[6]:

EgoExoLearn · temporal boundaries

Vidur places temporal boundaries more accurately than Gemini

0.0000.1000.2000.3000.4000.500

0.466

0.405

0.210

0.170

0.253

0.144

0.226

0.173

0.239

0.157

0.115

0.077

Segment F1 @0.50

Segment F1 @0.75

Semantic precision @0.50

Semantic recall @0.50

Semantic E2E F1 @0.50

Semantic E2E F1 @0.75

VidurGemini 3.5 Flash

On the segments both systems located correctly at tIoU 0.50, Vidur’s labels were right 51.3% of the time, against 38.8% for Gemini.

The aggregate result did not come from one favorable activity. Vidur won all eight task groups and both broad domains.

The lead also holds across every dimension of the label itself, compared via multi-aspect criteria - verb (67.2% vs 48.3%), primary object (62.7% vs 49.0%), hand roles, and tool relations.

EgoExoLearn · label accuracy

Vidur leads on label accuracy across every dimension

0%20%40%60%80%100%

67.2%

48.3%

62.7%

49%

67.4%

58%

58%

39.7%

60.8%

46.8%

85.7%

81.3%

64.4%

61.5%

60%

50.9%

Verb accuracy

Primary-object accuracy

Active-hand accuracy

Left-hand role accuracy

Right-hand role accuracy

Tool-presence F1

Tool-identity accuracy

Tool-action relation accuracy

VidurGemini 3.5 Flash

Both benchmarks use untrimmed, open-vocabulary video: Vidur must determine when each action occurs and name it in its own words. Yet they differ sharply in embodiment, viewpoint, environment, video length, action density, and instruction availability.

Together, they show that one approach generalizes from robot manipulation episodes to hours of dense, instruction-free human activity. The second evaluation gives us four kinds of confidence that a single leaderboard cannot:

• Duration - Vidur can turn hours of untrimmed footage into complete timelines.
• Granularity - the action distribution is measured in seconds, not long clips.
• Breadth - the advantage appears in both kitchen and laboratory environments.
• Instruction-free operation - Vidur reconstructs the task from visual evidence alone.

Vidur vs Gemini

Gemini is a useful reference point because Macrodata’s WGO pipeline identified Gemini 3.5 Flash as its strongest model for this task. Vidur’s advantage appears in two different comparisons: on WGO-Bench, Vidur reaches 0.386 against 0.168 for Macrodata’s strongest reported Gemini 3.5 Flash pipeline; on EgoExoLearn, Vidur reaches 0.239 against 0.157 for Gemini 3.5 Flash under the same long-video protocol.

What our labels look like on real video

Aggregate metrics show how often a system is correct, but not what its predictions actually look like. Below, two WGO-Bench episodes each play twice over - once captioned with Vidur’s timeline, once with GPT-5.6 Sol’s - under the identical benchmark protocol, so the wording and the temporal boundaries can be inspected side by side.

Recommended use cases

Vidur supports both high-volume pipelines and interactive annotation workflows, including:

• Dense re-annotation of existing corpora: converting teleoperation, on-policy, and egocentric archives into per-segment training samples for VLA and world-model training.
• Fleet-scale pre-labeling: continuous batch annotation of incoming robot footage, at an operating point tuned for throughput.
• Golden dataset creation with human-in-the-loop verification: maximum-accuracy configuration plus Vidur-assisted review, using a validated first-pass timeline to cut review time by up to 70%.
• Dataset curation, search, and analysis: action-level indexing for filtering, retrieval, and statistics over large video corpora.
• Benchmark and eval construction: timestamped, open-vocabulary labels as the substrate for downstream evaluation suites.

Every Vidur job runs the same underlying harness and always returns a validated, timestamped, open-vocabulary timeline - coverage-checked from first frame to last, with nothing silently dropped. What varies is how much inference you spend per video.

Use the throughput configuration when you want to:

• Convert large archives or continuous fleet footage into training-ready supervision at the lowest cost per video-hour.
• Feed structured timelines directly into VLA training, curation, and retrieval pipelines at scale.
• Pre-label incoming footage continuously, so your labeled dataset grows at the rate your fleet records.

Use the high-accuracy configuration when you want to:

• Produce gold-standard datasets: Vidur applies higher temporal resolution, richer labels, and additional self-verification passes, concentrating extra inference where its own confidence is lowest.
• Enforce a specific annotation convention - granularity and terminology are set per engagement, and Vidur’s output is calibrated to your house style on representative footage before a corpus is processed.
• Meet the strictest quality bars on footage where boundary precision matters most - dense, seconds-long actions.

Limitations

Vidur’s accuracy and cost can be tuned, but not independently. Greater temporal precision, richer labels, and additional verification require more inference, with diminishing returns at the highest accuracy levels. The practical objective is therefore to find the right operating point for each use case rather than to maximize compute spend on every video.

There is also no universal definition of the “correct” action timeline. Granularity and terminology depend on how the labels will be used, and information that is not visually observable - such as intent, applied force, or hidden state - cannot always be inferred from video alone. New domains should therefore be calibrated on representative footage, while applications requiring near-perfect assurance can add targeted human verification to Vidur’s output.

Open for Early Access Partners

FPV Labs is also opening up early access to Vidur for robotics labs, researchers, and data labs. Please write to us at [email protected][email protected] to try Vidur on your data.