Choosing a robot training data pipeline is a vendor-evaluation decision, not an infrastructure purchase.
Every provider sells collection, capture, and a claim about quality. Very few can show statistical evidence that their data was correct before it entered the model.
A gripper closes on air. The arm retracts, resets, and tries again. Sixty-two seconds of perfect trajectory that ended with nothing in the fingers. That failed pick, one recorded episode of robot behavior, is one of the most valuable things a robot training dataset can contain. But the next twenty episodes of the same arm picking up the same cup from the same table at the same angle are expensive noise.
A teleoperator records a pick. A simulator generates ten thousand variations of it. A vendor ships the batch with a quality report attached.
The question is what that report proves, because the same raw episode can be useless, valuable, or harmful depending on whether it was verified.
For many robotics teams, the limiting question has shifted from whether a policy can learn to whether the demonstrations deserve to become a training signal.
Feed a modern policy bad episodes and it fails quietly.
The bottleneck is now checking training data before the model uses it. Choice of pipeline decides whether that bottleneck gets addressed or ignored. This guide gives 10 questions for checking a pipeline’s data quality and its claims about data checks.
TL;DR
- In robotics, training data now matters more than model architecture. The pipeline can determine whether a policy handles new conditions or fails when conditions change.
- Most vendors measure cost-per-collected-episode. The real metric is cost-per-correct-episode, which accounts for the share of episodes that fail quality review in published collection campaigns.
- When evaluating a pipeline, ask how often reviewers agree about delivered episodes, not just how many episodes the vendor rejects. That agreement evidence shows that multiple reviewers checked the data before training begins.
How Does Demonstration Quality Shape Robot Policy Outcomes?
Mandlekar et al. (CoRL 2021) found that policies trained on skilled operators’ demonstrations outperform those trained on mixed-quality pools, even when the mixed pools contain 100 more demonstrations (300 vs 200). Operator skill drives more of the performance gap than raw demonstration count.
Data quality belongs at the center of policy design, because it decides what the robot can learn.
A small amount of unchecked bad data can cause a much larger behavior problem than its share of the dataset suggests.
A pipeline that lets bad episodes into training teaches the model bad behavior.
For a buyer, that changes the evaluation question from ‘how many episodes will this vendor deliver?’ to ‘what evidence shows that the episodes entering training are the ones the policy should learn from?’
The next ten questions examine that evidence. First, check whether the dataset covers the task. Then, check whether the episode judgments and records are trustworthy. Finally, check whether those records remain useful during deployment, regulatory review, and a vendor change.
Question 1: What Sources of Demonstration Diversity Does the Pipeline Cover?
Generalization in robot policies depends on the diversity of environments and objects more than on raw demonstration count (scaling-laws analysis, arXiv:2410.18647). A policy that has seen many different environments and objects generalizes far better than one fed thousands of near-identical demonstrations. Once a policy has seen enough of one environment, extra demonstrations add almost nothing. To make a policy generalize, add new environments and objects instead of repeating the same scene.
The BridgeData V2 authors found the same pattern in a 60,000-demonstration dataset. Training on a greater variety of skills and environments led to improved generalization (BridgeData V2, 2023).
A pipeline that uses one teleoperation station in one fixed lab can produce thousands of trajectories that look different to people but give the policy nearly the same training example.
A vendor’s total trajectory count alone does not show whether the policy will handle new conditions. The relevant evidence is whether its collection changes the environments, objects, and conditions the policy must handle.
A dependable pipeline draws from multiple sources:
- Teleoperation captures human-quality demonstrations directly, but each station produces them slowly. One operator typically collects tens to hundreds of demonstrations per day, and station costs start at $40,000 (Axis Robotics, 2026).
- Simulation can generate demonstrations at a small fraction of teleoperation’s per-trajectory cost, but a policy trained in simulation can fail in the real world unless the simulator deliberately varies conditions such as lighting, object positions, and physics.
- Human first-person video is a third source of training data. The EgoMimic system showed human egocentric video boosts robot task performance by 34 to 228 percent (EgoMimic, 2024). NVIDIA Cosmos includes 20 million hours of video (NVIDIA, 2025).
The best pipelines use simulation to expand real data instead of replacing real data with simulation.
NVIDIA’s GR00T N1 model trained on just 88 hours of real robot teleoperation data, then used that real data as the seed to generate 827 hours of synthetic neural trajectories, roughly a 10 times expansion factor (GR00T N1, 2025). Its DexMimicGen system generated 780,000 simulated trajectories from 11 hours of real demonstrations, an amount equal to about 6,500 hours of human demonstration time (GR00T N1, 2025).
But the multiplier has a ceiling. Simulation is generated from real demonstrations, so expanding does not remove the gaps in that data. It repeats them. That is why every published leading pipeline still trains on real demonstrations, and why edge-case coverage matters especially when a pipeline uses much simulation.
Question 2: How Does the Pipeline Handle Edge Cases?
A robot training dataset that contains only successful demonstrations teaches the policy what to do but not what to avoid.
RoboMIND explicitly includes 5,000 tagged failure demonstrations to address this gap (RoboMIND v1, 2024).
AgiBot World Beta addresses this by including many kinds of objects. Its dataset covers more than 3,000 objects across 217 tasks, including deformable objects whose shape changes during handling, so the policy has to adjust while it works (AgiBot World Beta, 2025).
Using varied environments and objects helps a policy handle conditions that differ from its training examples (scaling-laws analysis, arXiv:2410.18647).
The pipeline should maintain edge cases:
- Perceptual edge cases: lighting changes, object occlusions, reflective surfaces, transparent containers.
- Physical edge cases: fragile objects, deformable materials, tight-clearance insertions, non-rigid assemblies.
- Temporal edge cases: tasks with long-horizon dependencies where an early error propagates.
- Recovery edge cases: failure states that require the robot to retry or re-plan.
A vendor that cannot explain which edge cases it covers is collecting more episodes without showing that they help the policy handle new conditions.
Ask for the edge-case categories, the tasks and environments covered by each category, and example episodes for every failure mode.
Question 3: What Are the Verification Metrics on the Data Itself?
Most pipeline vendors cannot explain how they measure whether delivered episodes are correct.
Every provider measures collection throughput, cost per hour, and rejection rate.
Few publish agreement metrics on the correctness of their delivered episodes. Ask the vendor whether it uses model-led scoring and confidence routing (the system scores the straightforward episodes automatically and flags the uncertain ones for a person) to triage episodes before human review: routine data should not wait on reviewers, and unclear or high-stakes cases should not get zero human judgment.
Researchers already use established statistical methods to check whether individual data items are correct.
Coverage shows which tasks, environments, and objects the dataset includes. Verification checks whether the episodes that passed review meet the same definition of a correct episode.
Tasq, the trust layer for production AI, runs this division of labor as an evaluation layer on top of production-scale pipelines. Models score the bulk of episodes automatically, so routine data never waits on a human. Confidence routing identifies unclear or high-stakes cases and sends them to HERO (Human Expertise & Reasoning Orchestration), which assigns each case to the needed type of reviewer. The review can range from an automated score to a contributor review, a review by a credentialed subject-matter expert, or multiple reviewers assigned to the same task for consensus checks.
Multiple reviewers reduce the effect of one person’s mistake, as shown by jury decisions and Galton’s 1906 ox-weighing experiment.
One review gives only one opinion, so you cannot measure whether other qualified reviewers would reach the same result.
Independent reviews from several experts show how their judgments are spread across possible answers. The individual errors average out against each other. The judgment that reviewers share becomes clear. The spread shows how much the raters disagreed.
Measuring reviewer agreement for every episode shows when to clarify the review instructions or send unclear cases to more experienced reviewers. Measuring agreement across many episodes makes the results less likely to reflect chance.
When needed, Tasq compares independent reviews of the same episode, scoring the level of agreement statistically. That score decides whether the review standard is clear enough for training data.
The score ranges from 0 to 1, where 1 represents all possible reviewer agreement. Ask what annotation granularity the pipeline produces, episode-level verdicts, per-timestep action labels, or frame-level segmentation, and whether it matches what your policy actually consumes. Above 0.8 means reviewers genuinely align on the rubric. Between 0.6 and 0.8 means sharpen definitions and add calibration examples. Below 0.6 means the data does not feed training until the rubric is fixed.
For high-risk episodes, Tasq collects between 15 and 19 independent expert judgments per output, dynamically gathered and stopped early once reviewers reach strong agreement.
They’re then validated, weighted, and aggregated into a single consensus verdict. The impact is that an incorrectly labeled episode gets caught before it reaches training data, because one reviewer’s mistake shows up as a disagreement among the other reviewers instead of passing through as a silent label.
The panel’s agreement is then scored against how often independent experts would match by chance alone, so the verdict that lands in the dataset states what many independent experts concluded.
Buyer verification step: Select episodes from different tasks and operators, have two reviewers judge each one independently, and measure how often their judgments match.
On a scale from 0 to 1, a score above 0.8 means reviewers use the same definition of a correct episode; a lower score means the definition needs clearer instructions. Repeat this weekly so the standard does not drift over a multi-week campaign.
This gives buyers a direct check. The quality report reflects several reviewers using the same definition of a correct episode. Next, examine how many episodes fail review and why they fail.
Question 4: What Is the Demonstrated Rejection Rate and What Causes It?
Some collected episodes should fail quality review; rejecting them is a normal part of the process.
Vendors that reject almost no episodes may be letting poor episodes pass review. Filtering the bottom slice of demonstrations by quality metrics often improves policy performance.
Common rejection causes include:
- Calibration drift: if camera or robot calibration changes by even a few millimeters between collection and deployment, the robot’s 3D view can place objects incorrectly and reduce grasp success. Pointcloud registration means matching the robot’s 3D view to the real object position.
- Operator fatigue: teleoperation quality degrades measurably over long continuous sessions; building in breaks limits that degradation.
- Task ambiguity: grasps and placements that a person might accept but that do not clearly show the policy which outcome counts as correct.
To filter episodes well, identify the failure types that occur most often. The rejection rate should be transparent by operator, by task, and by collection session.
Quality results by operator show how much the best and worst operators differ; without them, a vendor hides where its data quality varies most.
Question 5: How Does the Pipeline Handle Versioning and Reproducibility?
A pipeline used for research or regulatory submission must give every episode a permanent ID, record the schema version, and use repeatable rules to split data into training, validation, and test sets.
The Open X-Embodiment dataset pooled 1 million plus trajectories from 22 robot morphologies, 527 skills, and 60 datasets from 34 laboratories (Open X-Embodiment, 2023). That combined dataset worked because every episode used the same defined set of fields.
Key versioning requirements include:
(1) A unique, immutable identifier for every episode that survives reprocessing.
(2) A schema version recorded in each episode’s metadata. The schema is the definition of what each episode contains, and the version records which definition applied when that episode was collected.
(3) A repeatable, rule-based split method that divides the data into training, validation, and test sets while keeping tasks and environments represented in each. The training set teaches the model, the validation set helps tune it, and the test set measures the finished model using data it has never seen.
(4) A record that links each episode to the operator who collected it, the hardware setup, the collection time, and the calibration settings.
If a supplier cannot show an episode’s full history, a buyer cannot inspect why it entered training or question that decision later.
Question 6: What Is the Real Cost Per Correct Episode?
Cost-per-collected-episode is the industry’s standard metric. It is also misleading.
A vendor that charges $3 per hour and rejects 40 percent of its episodes effectively charges $5.00 per verified episode assuming a collection throughput of one episode per labor-hour, about 1.7 times the advertised hourly rate; the remaining data may still teach the policy worse behavior.
The same arithmetic produces three comparison points:
- Budget labor pool: $3 per hour, 40 percent of episodes rejected, 60 verified episodes per 100 collected episodes, and $5.00 per verified episode.
- Onsite teleoperation team: $15 per hour, 10 percent of episodes rejected, 90 verified episodes per 100 collected episodes, and $16.67 per verified episode.
- Expert operator station: $48 per hour, 5 percent of episodes rejected, 95 verified episodes per 100 collected episodes, and $50.53 per verified episode.
The budget option looks cheapest on paper at $5.00 per verified episode, but it can end up the most expensive if the data it delivers teaches the robot worse behavior. Because skilled operators’ demonstrations can outperform larger mixed-quality datasets (Mandlekar et al., CoRL 2021), compare episodes by how much they improve the policy, not only by their price.
A pipeline that cannot report the cost of each verified episode does not show buyers one of its main costs. Buyers can compare costs only after they know which episodes are correct. Otherwise, a lower collection price may lead to higher GPU costs, more rework, and failed deployments.
Question 7: Can the Pipeline Demonstrate EU AI Act Article 10 Compliance?
Regulation (EU) 2024/1689 (the EU AI Act) requires in Article 10 that training, validation, and testing data sets for high-risk AI systems be relevant, sufficiently representative, and to the best extent possible free of errors and complete. Non-compliance carries fines of up to 15 million euros or 3 percent of global annual turnover, whichever is higher (EU AI Act Article 99). The Annex III high-risk obligations for stand-alone Annex III systems take effect on December 2, 2027 (Regulation (EU) 2026/1744).
Most robot training data pipelines do not include the records needed for compliance. A pipeline cannot show compliance with Article 10 unless it documents why its training episodes fit the intended use, represent the needed conditions, and were screened for errors.
Under Article 11 and Annex IV, providers must keep technical documentation that describes the data, including:
- A general description of the training data sets used, including their provenance, scope, main characteristics, and how the data was obtained and selected.
- A description of how the data’s measurable characteristics fit the system’s intended purpose.
- The procedures used to assign labels, remove unusual data points, and screen for errors.
- A record of data preparation steps, including cleaning, updating, and adding information.
Checking individual episodes is necessary to support compliance.
Reviewer-agreement results, rejection rates, and quality scores for each episode provide evidence that the provider screened the data for errors as Article 10 requires.
Article 16 requires providers to demonstrate conformity on request. Documentation is weak evidence if it says data was screened for errors but does not show measured agreement between reviewers.
Those records help a team check whether the training data still matches the conditions where the robot will operate.
Question 8: What Is the Coverage Gap for the Target Deployment Environment?
Robots fail when their training examples do not resemble the conditions they face where they actually work.
Many pipelines do not measure changes in scenes, object appearance, lighting, or sensors between the selected dataset and the live work environment.
A pipeline that checks data quality measures these changes directly:
- Perceptual drift: are the lighting conditions, camera angles, and background textures in deployment represented in the training set?
- Physical drift: have the robot’s kinematics, grip surfaces, or wear characteristics changed since data collection?
- Task drift: are the deployment tasks the same as the training tasks, or have object positions, tolerances, or success criteria shifted? Continuous evaluation is the cadence that makes drift visible:
When drift is detected, the team should decide whether to collect new data and create a new dataset version.
Without measuring how closely training data matches deployment conditions, a pipeline cannot show whether the policy will handle those conditions.
Question 9: How Do Throughput and Useful Information Per Demonstration Compare?
It is easy to measure and increase the number of episodes a pipeline collects. It is harder to measure and improve how much each demonstration teaches the model.
Two pipelines can deliver the same number of collection hours, but one can give the model much more useful training data.
Figure’s Helix model trained on approximately 500 hours of teleoperation data, less than 5 percent of the data volume used by prior VLA systems (Figure Helix, 2025). How much each hour teaches the model matters more than the total number of hours.
The Gemini Robotics system achieved successful policy adaptation with as few as 100 demonstrations per task, equivalent to 15 minutes to 1 hour of total demonstration collection (Gemini Robotics, 2025).
These systems did not become efficient only because of better models. They also used demonstrations that gave the model more useful information.
Top campaigns invest in quality as well as scale. DROID’s 76,000 trajectories across 564 scenes took 50 collectors on three continents over 12 months, with every episode given language descriptions verified by up to three independent reviewers (DROID, 2024).
AgiBot World combined its 100-robot facility and more than 1 million trajectories with a standard pipeline in which people check data quality during processing (AgiBot World Beta, 2025).
These are not small campaigns. DROID’s trajectories add up to 350 hours of interaction data, and AgiBot World’s to nearly 3,000 hours. These projects used different data-collection goals. DROID set out to build an open, general-purpose dataset whose value comes from breadth of scenes, tasks, and operators. Figure’s Helix trained on roughly 500 hours collected around one system’s target capabilities, and Gemini Robotics studied how few demonstrations a task adaptation takes, reporting as low as 100 per task. Each project collected the number of hours needed for its specific goal.
The right choice depends on whether the team needs to cover many conditions or master one specific task. Additional collection hours are useful only when the pipeline can show what each hour adds. That addition might be coverage of new conditions, clearer evidence that episodes are correct, or evidence that the policy can work in deployment.
Question 10: How Portable Is the Data, and What Are the Lock-In Risks?
The Open X-Embodiment standard, LeRobot V2 format, and RLDS schema let buyers export and reuse datasets with different vendors.
Portability means buyers can move the data to another vendor without losing its collection and review records.
A pipeline that keeps data in a vendor-only format or API can make it harder for buyers to switch vendors.
Portability requirements include:
- Data exportable in at least one standard format (MCAP, Parquet, HDF5, or LeRobot V2).
- Documentation that defines every episode field and the values each field may contain.
- Collection and review records that remain attached to the data after export.
- No contractual restrictions on model training with the data.
Mordor Intelligence forecasts the broader data-collection market growing from $2.67 billion in 2026 to $10.92 billion by 2031. Keep in mind, though, that the figure covers all AI data, and robots are only part of it (Mordor Intelligence, 2026).
In a fast-growing market, new vendors can enter quickly. Choosing one pipeline that prevents data export can keep a buyer from moving to a better option later.
If evidence cannot leave one vendor, the team cannot review it again after the hardware, task, regulatory requirements, or supplier changes.
Choose a Pipeline Based on How It Checks Episode Correctness
Hardware, models, and training methods all depend on one requirement. The data must be correct, and the pipeline must prove it. Cost per collected episode, rejection rates, and throughput are useful measures, but they do not show whether accepted episodes are correct. The missing measure is how often independent reviewers agree that each episode is correct before training uses it.
A pipeline that cannot show measured reviewer agreement is selling episode volume without proving episode correctness. Bad data costs more than collecting replacement data. It is the cost of a policy that fails in deployment, a compliance violation, or a team that cannot trust its own results.
Tasq uses this process to check whether episodes are correct. Models score most episodes so people do not need to review all of them. Confidence routing sends only unclear and high-stakes cases to the right level of human judgment. High-risk cases receive several independent judgments, and Tasq measures how often those reviewers agree. The result is an auditable decision for each episode, based on many independent reviews collected across enough episodes to measure agreement accurately.
Choosing a robot training data pipeline is like choosing a materials testing lab. You pay for evidence that each sample meets the required standard. The highest-throughput lab in the world is useless if it certifies flawed inputs.
Robot data follows the same logic. The best pipeline shows that each episode should be used for training.
Frequently Asked Questions
Teams most often ask the following five questions when they evaluate how pipelines check episode correctness.
How many demonstrations do you need to train a robot?
For single-task policies, requirements vary by task and model; Gemini Robotics reported successful adaptation with at most 100 demonstrations per task, achieving success rates above 70 percent (Gemini Robotics, 2025). Foundation model pre-training requires substantially more data. The Open X-Embodiment project aggregated 1 million plus trajectories across 22 robot morphologies (Open X-Embodiment, 2023), and the Physical Intelligence pi-zero model trained on 10,000 plus hours across 7 robot configurations (Physical Intelligence, 2024).
How much does it cost to collect robot training data?
Teleoperation stations cost $40,000 and up for full setup (Axis Robotics, 2026). Labor costs vary geographically, with Tesla advertising up to $48 per hour for Optimus motion-capture operators (The Verge, 2024). Most pipelines reject some share of collected episodes at their quality-filtering stage, which means the real cost per verified episode can be substantially higher than the cost per collected episode.
How do you know a robot training dataset is actually good before training on it?
Have independent reviewers judge the same episodes and measure how often they agree on the same definition of a correct episode. Then have two people independently review 15 to 20 percent of episodes, and hold weekly sessions to keep that standard consistent.
Is simulation data good enough, or do you still need real demonstrations?
Simulation multiplies real data but does not replace it. Simulated practice helps most when it is created from real robot behavior and used alongside real demonstrations. Every published leading pipeline still uses real teleoperation data as the starting point for simulated or synthetic data. The sim-to-real gap persists in perceptual and physical edge cases that simulation cannot fully model.
What is the sim-to-real gap and how does training data close it?
The sim-to-real gap is the performance degradation that occurs when a policy trained in simulation encounters real-world differences in lighting, physics, sensor noise, and object variation. Domain randomization in simulation helps but does not eliminate the gap. A pipeline that includes real-world demonstrations from varied deployment environments covers real conditions that simulation cannot fully represent.
Further Reading
- Axis Robotics
- AgiBot World
- Figure Helix
- EU AI Act Article 10. Data and governance requirements
- Mordor Intelligence. Robot data services market forecast
- Mandlekar et al
- BridgeData V2. A dataset for large-scale robot learning
- RoboMIND v1
- DROID
- GR00T N1
- Gemini Robotics. Bringing AI into the physical world
- EgoMimic
- Open X-Embodiment
- Data scaling laws for robot learning
- Digital Omnibus on AI (amends Regulation (EU) 2024/1689)
- The Verge
- Physical Intelligence pi-zero
