Data Enhanced Fine-tuning (DEFT)#

Data Enhanced Fine-tuning (DEFT) is an agent-run, disk-backed, iterative model-improvement loop that TAO ships as application workflow skills in the TAO Skill Bank. Instead of tuning a training recipe against a fixed dataset, a DEFT loop measures where the current model is weak, adds data that targets those weaknesses, retrains, and measures again. The agent treats the loop as a disk-backed state machine rather than as a prose recipe, so a run can be interrupted, resumed, and audited.

TAO documents four of these loops:

DEFT for Automated Optical Inspection improves a Visual ChangeNet defect classifier using synthetic generation together with real-image mining. This page refers to it as the automated optical inspection (AOI) loop.
DEFT for Object Detection improves a Grounding DINO detector using mining alone.
DEFT for DINOv3 Self-Supervised Learning improves a DINOv3 backbone by mining similar images from an unlabeled source pool. It is new in TAO 7.3.0.
DEFT for Object Detection in Optical Inspection improves a binary Real-Time DEtection TRansformer (RT-DETR) defect detector by mining real and verified-clean images, with optional synthetic generation. This page refers to it as the object detection for automated optical inspection (OD-AOI) loop. It is new in TAO 7.3.0.

DEFT is a family, and the TAO Skill Bank ships further loops beyond these four. In the DINOv3 loop, a Python controller in TAO Data Services owns the stages and state. Except for Choosing a Workflow, the comparison sections on this page cover the AOI and Object Detection loops only. Each skill’s SKILL.md file and references/ files remain the executable contract; this page summarizes them so that you can compare the loops and choose one. For hyperparameter optimization, refer to AutoML. For the wider set of agent-driven pipelines, refer to Reference Workflows.

Loop Anatomy#

Every DEFT loop follows the same general shape: establish a baseline, analyze where the model fails, enhance the data to close those gaps, retrain, and apply a gate that decides whether to continue. The AOI and Object Detection loops implement that shape with different stage names, so treat the stage sequence as workflow-specific. The analysis step in both loops is backed by the TAO Data Services gap_analysis command.

Workflow	Stage sequence
DEFT for Automated Optical Inspection	`train`, `evaluate`, `rca`, `routing`, `anomalygen`, `data_mining`, `data_merge`, and `loop_stop`. The baseline runs a single time as `train`, `inference`, `evaluate`, and then `rca`, where `inference` is a task name rather than a committed stage.
DEFT for Object Detection	`prep`, `inference`, `kpi_analyze`, `gap_analysis`, `embed`, `mine`, `stage`, `train`, and `loop_stop`. The `prep` stage runs a single time, and the agent skips it when its artifacts already exist. The baseline runs `inference` and then `kpi_analyze`.

The baseline convention differs between the AOI and Object Detection loops, and that difference changes how you read the first result. The AOI loop trains a model at baseline. The Object Detection loop evaluates the supplied zero-shot or pretrained checkpoint as iteration 0 and only trains from iteration 1 onward, so its baseline number is a zero-shot number.

Iteration Budgets#

The max_iterations parameter sets how many enhancement and retraining passes the loop runs after the baseline. The AOI and Object Detection loops resolve it differently, so confirm the value in the Pre-Flight Summary before you approve a run.

The AOI loop has no default: the value is a required input, and the agent refuses to proceed past pre-flight without it. If you supply a time limit instead, the agent converts it into an estimated max_iterations and surfaces that estimate for your confirmation.

The Object Detection loop defaults max_iterations to 1, which is one mine, train, and score pass, the smallest run that yields a comparison against the baseline. An unattended run takes that default rather than stopping to ask you.

Data Enhancement Flavors#

DEFT loops enhance data in two flavors: mining-based selection, which pulls images from an existing pool, and synthetic data generation, which creates new images. A workflow can use one flavor or both. For background on the underlying data services, refer to Data Mining, Embedding Generation, and Targeted Mode Matching (TMM) Mining.

The Object Detection loop uses mining only and performs no synthetic generation. It runs tmm unique-neighbor matching over SigLIP embeddings against a prebuilt source pool whose labels come from Co-DETR pseudo-labeling during a separate one-time preparation run, so every image it adds is a real image that already existed in that pool.

The AOI loop is hybrid and combines both flavors. AnomalyGen synthetic generation contributes defect-type diversity, and the loop keeps all generated images without filtering. Real images from augmentation/mining_pool/ contribute a real-distribution anchor, and the loop keeps only entries whose cosine similarity to the current iteration’s weak-sample embeddings is greater than or equal to state.config.mining_filter.min_similarity, then applies history-aware filtering. Synthetic rows skip the similarity filter, so the cosine threshold applies to real-image mining only.

A routing stage splits the root cause analysis gaps into routing_mining_parquet and routing_anomalygen_parquet, which feed the two branches. The loop skips either branch when its routed parquet file has zero rows. Skipping both is not permitted: if mining yields zero novel rows and AnomalyGen contributed nothing, the loop takes a hard stop.

Data Enhancement Modules#

The AOI and Object Detection workflows share a recognizable structure in how they acquire and label new data. That structure is an observed pattern across the shipped workflows, not a published extension interface. The TAO Skill Bank ships no extension-contract document at 7.3.0, so treat what follows as a description of the existing loops rather than as a supported interface for building your own. The pattern has two parts. The first is a data source, which is either a generator or a source pool paired with a miner. The second is an annotator that emits the target model’s training format, because the retraining stage consumes that format directly.

Workflow	Data source	Annotator	Training format
DEFT for Automated Optical Inspection	AnomalyGen generation plus AMP synthetic data generation, and real images from `augmentation/mining_pool/` selected by SigLIP k-nearest-neighbor search	`scripts/changenet_data_pair_prepare.py`, which maps the NG image to `input_path` and the OK image to `golden_path`	Visual ChangeNet four-column paired CSV file with the columns `input_path`, `golden_path`, `label`, and `object_name`, assembled into `train_combined_iter${N}.csv`
DEFT for Object Detection	An unlabeled raw pool plus `tmm` unique-neighbor mining	Co-DETR pseudo-labels the pool, the agent folds the labels to the target classes, converts KITTI to COCO to Object Detection Visual Grounding (ODVG), and then runs `stage_mined_odvg.py`	ODVG: `tmm_odvg.jsonl` plus `labelmap.json`, appended as one new entry to the `dataset.train_data_sources` list

One invariant holds across the AOI and Object Detection workflows. Every stage commits its outputs through commit_stage.py, which verifies the stage’s required artifacts before it writes state, so a stage that produced nothing cannot record itself as complete.

Metric Contracts#

The metric contract is an AOI concept. The AOI loop optimizes one primary customer metric and can enforce zero or more secondary constraints. The state.metric_contract object, stored once at state["metric_contract"], selects the evaluator and the comparison, and metrics are not restricted to a fixed name or unit. The contract carries the metric name, display_name, operator, target, and unit, an evaluator binding, and a constraints array whose entries repeat the same comparison fields.

The supported operators are <, <=, >, and >=. They determine both success and best-iteration selection. The agent first keeps the candidates that satisfy every secondary constraint, then selects the lowest primary result for < and <= or the highest primary result for > and >=. If no candidate satisfies all constraints, the agent retains the direction-aware primary best and reports it as a clearly non-passing result. The agent never infers direction from the metric’s name.

Canonicalization turns a customer sentence into contract fields. The requirement FAR < 10% at Recall=100%, which reads as a false alarm rate (FAR) below 10 percent at 100 percent recall, becomes a primary metric far_pct with operator <, target 10, unit %, and the bundled far_at_recall evaluator, plus a secondary constraint recall_pct with operator >=, target 100, and unit %. Never compare a fractional FAR value to a percentage target.

The contract binds one of three evaluator types: a builtin evaluator shipped with the skill, a command evaluator that names a customer-owned executable, or an artifact evaluator that consumes a result file an external system produces. A custom metric must have a command or artifact evaluator before the Pre-Flight Summary, because the evaluator is part of the approved run configuration and changing it requires a revised summary. The bundled recorder does not trust passed from the evaluator. It recomputes passed from the approved contract. For the field-by-field contract, the evaluator invocation forms, and the result file schema, refer to DEFT for Automated Optical Inspection.

The contract drives stopping, best-iteration selection, and the report headline. A result can be structurally valid while missing the target, and the loop continues until the metric passes or the loop reaches max_iterations.

The Object Detection loop has no metric contract and does not gate. It reports mean average precision (mAP) rather than gating on it, the loop does not exit early on a metric target, and a regression does not stop the loop.

State and Resume#

The AOI and Object Detection loops use deft_state.json as the single resume record. The agent initializes the file a single time with init_deft_state.py and then mutates it only through commit_stage.py. Never hand-edit it, reinitialize it, or write it with inline Python, jq, or heredocs. The AOI state file also holds the immutable execution_policy, the resolved mining paths, and optional terminal artifacts.

The AOI and Object Detection loops use the same run directory form and resume-detection rule. A fresh run always creates a new timestamped results/run_<YYYYMMDD_HHMMSS>/ directory and preserves prior runs under their own directories. The agent detects a resumable run by checking for results/run_*/deft_state.json, and it does not create the run directory before the user gate.

The AOI and Object Detection loops record the same event field set with different stage enumerations. Each event carries seq, ts, iter, stage, status, summary, skip_reason (only when the agent skips a stage), duration_sec, context_tokens, and tokens. AOI status values are ok, error, and skipped, while Object Detection uses only ok and error. AOI context_tokens starts at 0 and align_token_usage.py backfills it at loop end; the Object Detection field is reserved and always 0.

Where the ordered event stream lives differs between the loops. The Object Detection run directory carries loop_log.jsonl as an append-only event stream with one JSON line per stage, alongside .deft_commit.lock, which enforces one commit_stage.py writer at a time, and .deft_commit.journal, which exists only while a commit is in flight so that the next commit_stage.py run can undo an interrupted one. The AOI loop has no loop_log.jsonl: its ordered events live inside deft_state.json.

Disk is the source of truth, and neither loop infers progress from conversation history. The AOI skill runs deft_context.py on startup, after context compaction, before every stage, and before any completion claim, then resumes from its durable next_stage together with the recorded state. The Object Detection skill runs its audit before every stage and uses the last_committed, next_action, and read_before_action output. Neither skill trusts assistant prose or an artifact that state does not record. On startup or resume each one shows the last five events so that you can see where the run stands.

There is exactly one user gate: pre-flight confirmation. The agent prints the Pre-Flight Summary, then stops and waits for you to type go, yes, looks good, or a similar explicit approval. It launches no side-effecting step before that approval, which rules out docker run, training, synthetic data generation, and mutations under ${RESULTS_DIR}/. Reading specifications, listing files, running docker image inspect, and populating the summary table remain permitted. Autonomous describes the behavior after the gate, not before it.

After the gate the skill runs the entire loop without asking for confirmation, stopping only if a step fails with an unrecoverable error or a hard-stop gate fires. The loop requires auto-mode, because it fires constant side-effecting calls after the gate and stalls on the first prompt without auto-accept or bypass-permissions mode. If any run parameter changes after the agent shows the original summary, the agent re-runs pre-flight and shows an updated summary. The summary preserves every explicit value you supply and labels the source of every run parameter as user, spec, or default.

Outputs by Workflow#

The AOI and Object Detection loops do not produce a uniform artifact set, so read the outputs per workflow rather than as a shared contract. The most visible difference is the report format: the AOI loop renders an HTML report, and the Object Detection loop renders a Markdown report with no HTML template.

Workflow	Report artifact	Distinctive artifacts
DEFT for Automated Optical Inspection	`DEFT_Loop_Report.html`	`deft_state.json`, `best_model.json` for inference handoff metadata, `best_model_inference_spec.yaml` as a ready-to-run TAO inference specification built from the training configuration, `iter${ITER}_summary.md`, a `baseline/` tree, a per-iteration `iter${ITER}/` tree, `dataset/merge_validation.json` as mandatory proof of the merge and leakage checks, `RCA_Report.md` with `rca_images/` per root cause analysis stage, and `allocation.json` under the `anomalygen` stage
DEFT for Object Detection	`DEFT_Loop_Report.md`	`deft_state.json`, `loop_log.jsonl`, `.deft_commit.lock`, `.deft_commit.journal`, `train_grounding_dino.yaml` seeded at baseline from your template, a `prep/` tree, `iter${N}_summary.md`, a `baseline/` tree, and a per-iteration `iter${N}/` tree

The AOI and Object Detection reports are generated, never hand-authored. The AOI loop refreshes its HTML report through a post-commit hook after every successful commit, and the Object Detection loop writes its Markdown report at initialization and re-renders it after every accepted commit_stage.py call. The Object Detection loop produces no best_model.json and no best_model_inference_spec.yaml; those artifacts are AOI only. For the full output trees and the report repair commands, refer to DEFT for Automated Optical Inspection and DEFT for Object Detection.

Execution Platforms#

The AOI skill discovers the execution platforms from the installed platform skills, names Docker, SLURM, Kubernetes, and Brev, and asks which supported platform you intend to use. It accepts an external platform skill in addition to the four it names. The Object Detection skill supports Docker only, so it does not ask which platform to run on. Neither skill lists virtualenv. Both skills require Docker and the NVIDIA Container Toolkit, and the Object Detection skill additionally requires one or more CUDA GPUs.

Air-gapped support is not symmetric. The AOI loop has a first-class air-gapped mode with a dedicated references/air-gap.md file, and its front matter advertises support for air-gapped and offline runs with pre-staged assets. The mode activates when you set AIR_GAPPED=1 in the environment, when you explicitly request it, or when the harness reports restricted networking; the agent never runs a network command to infer the mode. In this mode the agent performs no registry login or pull, converts every fetch instruction into a local presence check, and treats a missing asset as a hard stop. Setting HF_HUB_OFFLINE=1 alone disables Hugging Face access only and does not activate global air-gap mode. Air-gap mode changes asset resolution and network behavior, not user authorization or stage ordering. For the enforcement details and the full pre-staging list, refer to DEFT for Automated Optical Inspection.

For air-gapped hosts, the Object Detection loop ships no references/air-gap.md file and has no --network-mode option. Its accommodations are your own paths for the zero-shot checkpoint (--zero-shot-checkpoint) and for the Co-DETR checkpoint.

Installing the Workflows#

The DEFT workflows ship inside the tao-skills plugin in the TAO Skill Bank, which bundles every model, data, platform, and application skill in one package. Pin the marketplace to the release/7.3.0 branch, which contains the DINOv3 SSL loop. In Claude Code, run these two commands:

/plugin marketplace add NVIDIA-TAO/tao-skill-bank@release/7.3.0
/plugin install tao-skills@tao-skill-bank

For the full installation matrix and a first successful run, refer to Getting Started with NVIDIA TAO Toolkit.

Note

Release branches and tags are stable, while main moves continuously and can reference prerelease container builds. Every link in this documentation therefore points at release/7.3.0 until the 7.3.0 tag exists. Until then, the branch references prerelease images in nvstaging that only NVIDIA accounts can pull, so these commands are for internal use. At release, the links change to the 7.3.0 tag.

Choosing a Workflow#

Choose a workflow by the model you want to improve and the data you can supply. If you have a pool of unlabeled real images and a Grounding DINO detector, choose the Object Detection loop. If you are improving a paired-image defect classifier and want synthetic defects in addition to mined real images, choose the AOI loop. If you are improving a binary RT-DETR defect detector, choose the OD-AOI loop. It needs labeled real defect images and separate benchmark and test COCO sets; verified-clean images are optional. The OD-AOI loop mines real images by default and treats synthetic generation as an option. If you want to improve a DINOv3 backbone with unlabeled images, choose the DINOv3 SSL loop.

Workflow	Domain	Model	Data enhancement flavor	Page
DEFT for Automated Optical Inspection	Automated optical inspection	Visual ChangeNet Classification	Hybrid: AnomalyGen synthetic generation plus real-image k-nearest-neighbor mining	AOI loop
DEFT for Object Detection	Object detection	Grounding DINO	Mining only, using `tmm` unique-neighbor matching over SigLIP embeddings	Object Detection loop
DEFT for DINOv3 Self-Supervised Learning	Self-supervised representation learning	DINOv3	Mining only, using nearest-neighbor search over fixed CLIP or SigLIP embeddings	DINOv3 SSL loop
DEFT for Object Detection in Optical Inspection	Automated optical inspection	RT-DETR	Mining of real defective and verified-clean images over SigLIP embeddings, with optional AnomalyGenNext synthesis	OD-AOI loop

DEFT for Object Detection in Optical Inspection#

The tao-run-deft-od-aoi skill runs a binary industrial-inspection DEFT loop with TAO RT-DETR for one foreground class, defect, and it is not a general multiclass object-detection loop. The loop measures the model on frozen benchmark (internal kpi) and test roles. It mines the real and clean roles over SigLIP embeddings to accumulate the training data it admits. Every iteration retrains from the same frozen base checkpoint, and the loop selects the checkpoint with the highest val_mAP50 on the kpi role while the test role stays report-only. The core loop uses real data only, and AnomalyGenNext synthesis is an optional route with separate preparation, generation, and admission gates. That route uses the tao-prepare-anomalygennext-inputs and tao-generate-od-defects skills and, for missing task weights, the tao-finetune-anomalygennext skill. The launch review includes max_iterations, which is a required input with no default. The SKILL.md file for TAO 7.3.0 is the contract for this loop, and the files in the skill’s references/ directory are part of the contract.

Source of Truth#

The skill files are the executable contract, and this page summarizes them. Read the skill sources on the release/7.3.0 branch when you need the exact behavior: