DEFT for Object Detection#

DEFT for Object Detection is an agent-run loop that grows a Grounding DINO training set by mining a pseudo-labeled source pool against per-class mean average precision (mAP) gaps measured on a key performance indicator (KPI) set, and it runs for a bounded iteration count. The loop evaluates a zero-shot checkpoint first, analyzes which classes fall short on which images, embeds those weak images, mines visually similar images out of a prepared pool, stages them as new training annotations, and retrains. It repeats that sequence for a fixed max_iterations.

This workflow is one of the loops described in Data Enhanced Fine-tuning (DEFT), and it trains the model described in Grounding DINO. It is not the right tool for a single standalone TAO training run, a one-off inference, or a gap analysis on its own. Invoke the relevant single-purpose skill directly for those. The SKILL.md file and its references/ files remain the executable contract that the agent follows. This page summarizes them so that you can decide whether the loop fits your problem and prepare the inputs it needs.

Supported Models#

This loop targets Grounding DINO with open-set object detection with visual grounding (ODVG) training annotations only, namely tmm_odvg.jsonl and labelmap.json, matching the reference pipeline. This workflow does not support DINO or RT-DETR with COCO annotations. Invoke the single-purpose skills directly for those models.

The list shape of dataset.train_data_sources is what makes the growth mechanism work. Each iteration appends one {image_dir, json_file, label_map} entry to that list rather than rewriting a combined file, and earlier sources are never removed. Pre-flight enforces the shape and rejects a mapping there, because a mapping means the specification is COCO-shaped and this workflow cannot append to it. Both the Pre-Flight Summary and the loop report record the model as Grounding DINO with ODVG annotations.

The Two Invocations#

Preparing the source pool and running the loop are two separate runs of the same skill. A separate run prepares the source pool before the loop launches. The two stay separate because you prepare a pool one time and then reuse it across many loop runs, and coupling them would re-label and re-embed the same images on every launch.

Invocation

What it does

What it produces

Prepare the source pool

Co-DETR pseudo-labels raw pool images, folds them to the target classes, converts KITTI to COCO to ODVG, verifies, and embeds

coco.json, odvg/, source_embeddings.parquet, and pool_report.json

Run the DEFT loop

Runs the baseline and then the iterations

Checkpoints, KPI results, and the mAP trend

The loop takes those four paths as inputs. Pre-flight validates them and init_deft_state.py pins them, so a run cannot reach the mining stage with no corpus to search.

Preparing the Source Pool#

The preparation run pseudo-labels the pool with Co-DETR, folds the predictions onto your target classes, converts KITTI to COCO to ODVG, and embeds the pool. The chain runs in the following order:

  1. Refuse a pool that has duplicate basenames, using scripts/verify_pseudo_labels.py --pool-images-dir "$POOL_IMAGES".

  2. Emit the two mappings with scripts/prepare_class_mappings_for_mining_data_prep.py, which writes codetr_category_mapping.yaml, kitti_mapping.yaml, and classmap_target.txt into the preparation directory.

  3. Pseudo-label the pool with Co-DETR, using codetr inference -e "$CODETR_SPEC" inference.checkpoint=... results_dir=${PREP_DIR} inference.num_gpus=... in $TAO_PYT_IMAGE. When the codetr console script is unregistered, the fallback entry point is python3 -m nvidia_tao_pytorch.cv.codetr.entrypoint.codetr.

    Pre-flight resolves the Co-DETR checkpoint. It uses your own path when you supply one; otherwise scripts/fetch_codetr_checkpoint.py downloads the checkpoint.

    The class fold applies at detection time through inference.category_mapping. The overlay assets/overlays/codetr_inference.yaml pins model.num_select to 1000, where the TAO default is 300, and inference.conf_threshold to 0.3, where the TAO default is 0.5. A gate runs afterward: scripts/verify_pseudo_labels.py writes pseudo_label_report.json.

  4. Convert KITTI to COCO with annotations convert -e "${PREP_DIR}/kitti_to_coco.yaml" in $TAO_DS_IMAGE. The overlay assets/overlays/kitti_to_coco.yaml pins kitti.project to coco, which names the output file coco.json. The output lands at <workspace>/source_pool/coco.json and is recorded as config.source_detection_file.

  5. Verify the result with scripts/validate_pool_coco.py, which writes pool_report.json and settles max_labels, one per target class.

  6. Convert COCO to ODVG with annotations convert -e "${PREP_DIR}/coco_to_odvg.yaml" in $TAO_DS_IMAGE, with results_dir set to <workspace>/source_pool/odvg. The output is named after the input, as <basename>_odvg.jsonl and <basename>_odvg_labelmap.json. There is no direct KITTI to ODVG conversion, which is why the pipeline converts to COCO first and then to ODVG.

  7. Embed the pool. scripts/prepare_input_for_image_embeddings.py writes pool_input.parquet, and then embedding image_embeddings -e "$EMBED_SPEC" runs in $TAO_DS_IMAGE with output_parquet set to <workspace>/source_pool/source_embeddings.parquet, model set to $EMBEDDING_MODEL, model_path set to $EMBEDDING_MODEL_PATH, model_config_path set to '""', and batch_size set to 64. For background on this data service, refer to Embedding Generation.

The run finishes with scripts/commit_stage.py --iter-label prep --stage prep, which records the pool ODVG tree, the pool embeddings, and the pool report. The --pool-report argument is what settles rare_class_list. The preparation run writes source_pool/source_embeddings.parquet, source_pool/coco.json, and the source_pool/odvg/ tree, plus pool_report.json, codetr_category_mapping.yaml, kitti_mapping.yaml, classmap_target.txt, pseudo_label_report.json, codetr_spec_report.json, and pool_input.parquet under ${RESULTS_DIR}/prep/.

Preparation is idempotent. The agent skips any step whose output already exists and never re-labels or re-embeds a pool that already has current artifacts. No script performs the skip; the agent checks for each step’s output before it runs the step. Labeling and embedding are proportional to pool size, not to what mining eventually selects, so a large pool costs a large preparation run even when a single loop iteration draws only a few hundred images from it. Preparation must finish before the baseline starts, and the agent reports the pool image count in the Pre-Flight Summary.

Prerequisites and Pre-Flight#

The loop requires Docker, NVIDIA Container Toolkit, and one or more CUDA GPUs. Before it launches anything, the agent runs 14 ordered pre-flight checks and prints a Pre-Flight Summary.

Check

Subject

What it establishes

1

Workspace and run directory

Resolves WORKSPACE and sets RESULTS_DIR to ${WORKSPACE}/results/run_$(date +%Y%m%d_%H%M%S); the directory is not created before the approval gate, and a resume is detected from results/run_*/deft_state.json

2

Host Python

Probes the interpreter with scripts/deft_python.sh

3

Credentials

Records the presence of NGC_KEY and HF_TOKEN without printing them, and defers both verdicts to checks 5 and 10

4

Container images

Resolves TAO_PYT_IMAGE and TAO_DS_IMAGE from the skill bank versions.yaml file using resolve_versions_key.py

5

Image presence

Runs docker image inspect on both images; a missing image is marked WILL_PULL_AFTER_APPROVAL and pulled only after the gate

6

Zero-shot checkpoint

Resolves the Grounding DINO checkpoint, reporting only, with the real download deferred until after approval

7

Co-DETR checkpoint

Runs only when preparation runs; your own path always wins, and the agent stops when that path does not exist

8

Train specification template

Confirms that the template parses and that dataset.train_data_sources is a list; seed sources are optional

9

Source pool

Requires both a non-empty embeddings parquet file carrying the filepath and embedding columns and the ODVG annotation tree; anything else is a hard stop

10

Encoder resolution

Resolves the SigLIP encoder from a local snapshot first and never from an implicit online default

11

KPI inputs

Confirms that the KPI inputs exist and that image_dir does not end in a slash

12

Class thresholds and mining configuration

Records the class thresholds and the mining settings; the agent does not interrogate you about the defaults

13

GPU count

Counts GPUs with nvidia-smi --list-gpus | wc -l, otherwise with a docker run call into $TAO_PYT_IMAGE that reads torch.cuda.device_count()

14

Specification sanity

Confirms that train.checkpoint_interval is less than or equal to train.num_epochs

When you supply a Co-DETR checkpoint, the agent flags a size other than 2,934,763,233 bytes in the Pre-Flight Summary rather than stopping the run. When you supply no path, fetch_codetr_checkpoint.py --plan reports the download plan before the gate. After approval, the download runs and verifies the size and the Secure Hash Algorithm 256-bit (SHA-256) checksum.

At initialization, the agent passes --pool-report with the preparation run’s pool_report.json, and initialization refuses an existing pool without it under every allocation policy. Initialization also stops with an error when the encoder recorded in pool_report.json differs from the run’s encoder.

The host Python probe imports exactly six modules: pandas, numpy, matplotlib, pyarrow, PIL, and yaml. deft_python.sh selects an interpreter only when every one of them imports, so a host missing just matplotlib makes it exit 2 with no interpreter at all. Installing packages is out of scope for the skill, and deft_python.sh never installs. It auto-selects $WORKSPACE/.venv/bin/python when that path exists and WORKSPACE or WORKSPACE_DIR is exported; otherwise you pass DEFT_PYTHON yourself.

Both credentials are conditional. NGC_KEY is needed only when an nvcr.io image has to be pulled or the Grounding DINO checkpoint has to be downloaded. Two images that are already local, together with an ngc command line interface holding its own credentials, make a complete run possible with NGC_KEY unset. HF_TOKEN is needed only when the encoder resolves to a Hugging Face identifier rather than a local snapshot directory. The Co-DETR checkpoint is public on Hugging Face and needs no credential.

Two container images cover every stage, and the deploy image is not used by this workflow.

Variable

Image

Stages that use it

TAO_PYT_IMAGE

nvcr.io/nvidia/tao/tao-toolkit:7.2.0-pyt

train and inference

TAO_DS_IMAGE

nvcr.io/nvidia/tao/tao-toolkit:7.2.0-data-services

gap_analysis, embed, mine, and kpi_analyze

Note

The image tags in the table come from TAO 7.2.0 and are provisional. The skill resolves each image from the versions.yaml file of the installed TAO Skill Bank. The release/7.3.0 branch of the TAO Skill Bank can reference prerelease container builds until NVIDIA publishes the release tag.

A checkpoint path that you supply always wins. Otherwise scripts/fetch_gdino_checkpoint.py resolves nvidia/tao/grounding_dino:grounding_dino_swin_tiny_commercial_trainable_v1.1, which is 1.93 GB, from NVIDIA GPU Cloud (NGC) and requires the ngc command line interface. The script runs with --plan before the gate and reports only. It must be a trainable release, not a deployable one. The specification fields and the checkpoint architecture fields that must agree are model.backbone, num_queries, enc_layers, dec_layers, num_feature_levels, and class_embed_bias. On an air-gapped host, or any time the download cannot run, you supply --zero-shot-checkpoint directly, and the script says so rather than failing obscurely.

Encoder resolution prefers a local snapshot. The agent searches HF_HOME/hub, HUGGINGFACE_HUB_CACHE, $HOME/.cache/huggingface/hub, $WORKSPACE/source_pool/hf_cache/hub, and $WORKSPACE/source_pool/hf_cache for a models--google--siglip-base-patch16-224/snapshots/* directory that holds a config.json file. The bare Hugging Face identifier google/siglip-base-patch16-224 is used only after outbound access is verified, and only then is HF_TOKEN required. The result is recorded as config.embedding_model and config.embedding_model_path.

There is exactly one user gate: pre-flight confirmation. The agent prints the Pre-Flight Summary, stops, and waits for explicit approval before it launches any side-effecting step. After the gate, the skill is fully autonomous, so the Pre-Flight Summary also reminds you to enable auto-mode before you approve.

Defaults#

The agent does not ask you about the values in the following table. Confirm them in the Pre-Flight Summary if you want something else.

Setting

Default

train.num_epochs and train.optim.lr

Taken from the train specification template

multiplier

3

max_iterations

1

allocation_policy

class_stratified when rare classes are given, otherwise global

rare_class_list

Every target class holding a below-mean share of the pool’s annotations, derived at the preparation commit from pool_report.json

distance_metric

euclidean

candidate_expansion_factor

5

embedding_model

SigLIP; embedding_model_path is resolved rather than defaulted

iou_threshold

0.5

kpi.conf_threshold

0.0, frozen as config.kpi_conf_threshold

Workspace root

Taken from your prompt, otherwise ~/workspace

AP50 thresholds

When --ap50-thresholds-json is omitted, the reference intelligent transportation system values apply: car 0.99, bicycle 0.7, person 0.7, and 0.7 for any other target class

Data Layout#

The loop reads and writes a single workspace directory. You author the specifications, the KPI set, and the optional seed training data. The workspace has the following layout:

<workspace>/
    specs/
        train_grounding_dino.yaml
        inference_gdino.yaml
    checkpoints/zero_shot.pth
    kpi/
        images/
        labels/                       KITTI ground truth
        coco.json                     yours; target_detection_file for mining
        mapping.yaml
    base/                             optional seed training set
        images/
        odvg/                         *.jsonl and labelmap.json
    source_pool/
        source_embeddings.parquet     filepath and embedding
        odvg/                         *.jsonl and labelmap.json
        coco.json                     generated by prep; source_detection_file for mining
        hf_cache/                     optional local encoder snapshot
    results/run_<YYYYMMDD_HHMMSS>/    created by this workflow

Note

Do not hand-author source_pool/odvg/ or source_pool/coco.json; the source pool preparation run generates both. Do not hand-author anything under results/run_<YYYYMMDD_HHMMSS>/ either, because the loop creates it.

You must make the following inputs present in the workspace before the run. The preparation run produces the source_pool/ entries and pool_report.json; you author the rest:

Input

What it provides

specs/train_grounding_dino.yaml

The train specification template, and the source of truth for the backbone, the image size, and train.optim

specs/inference_gdino.yaml

dataset.infer_data_sources and inference.conf_threshold

checkpoints/zero_shot.pth

The required checkpoint that the baseline scores without training

kpi/images/

The KPI evaluation images

kpi/labels/

The KPI ground truth, one KITTI .txt file per image

kpi/mapping.yaml

The class mapping that kpi_analyze uses

kpi/coco.json

The mining target detection file, required only for class_stratified mining

source_pool/source_embeddings.parquet

The pool embeddings that mining searches

source_pool/odvg/

The pool annotations that staging looks up

pool_report.json

The report that validate_pool_coco.py writes for the pool, normally ${RESULTS_DIR}/prep/pool_report.json from the preparation run, or <pool>/pool_report.json for a pre-supplied pool. Initialization refuses an existing pool without it under every allocation policy; the report records the encoder only when you generate it with --record embedding_model_path

NGC_KEY and HF_TOKEN

Session credentials, both conditional

Optional Seed Training Data#

Seed training data under base/ is optional, and that is a deliberate difference from the automated optical inspection (AOI) loop. The AOI loop requires a seed training set because Visual ChangeNet must learn the task from scratch. Grounding DINO is zero-shot capable, so this loop can start cold. Two cases follow from that:

  • When the template already lists base sources, pre-flight validates that they resolve, and iteration 1 appends mined data to them.

  • When the template’s train_data_sources is empty, iteration 1 trains on mined data alone. That is not an error, but the first iteration’s dataset is small.

The source pool is mandatory either way. Seed data is what you start from, and the pool is what you draw from. They are not interchangeable, and the loop cannot mine without a pool.

Warning

kpi/mapping.yaml is a YAML list of single-key dictionaries whose values are lists. A bare string value is iterated character by character, so nothing matches and every metric is zero while the run still exits 0. Write - car: [car, automobile], not - car: car. Matching is case-sensitive in both annotations convert and analytics kpi_analyze, so a mapping must enumerate every spelling it expects to see, including Bicycle and bicycle, or AutoMobile and Automobile. The agent narrows the mapping to the target classes before scoring, using scripts/prepare_mapping_for_kpi_analyze.py.

Annotation Formats#

Three annotation formats travel through the loop, and each has its own job. The TAO documentation covers all three in full, so this section covers only where each one lives and what it does here.

Format

Where it lives

Job in this loop

KITTI

kpi/labels/ and all TAO inference output

Carries the KPI ground truth and every set of predictions the loop scores

ODVG

source_pool/odvg/, staged at iter${N}/tmm/annotations/

Supplies the Grounding DINO training annotations that each iteration appends

COCO

source_pool/coco.json and kpi/coco.json

Feeds class_stratified mining only, with detection_format: coco

The two stages that read KITTI spell the format differently on purpose. gap_analysis takes input_format: kitti in lowercase, and kpi_analyze takes input_format: KITTI in uppercase. Both are correct for their own stage, so treat the difference as expected rather than as a typographical error to fix.

Warning

This loop does not verify that mined images are absent from the KPI evaluation set. That is a deliberate decision, and it is documented as one. If the pool does overlap the KPI set, mining pulls evaluation images into training, and every mAP number the loop reports becomes optimistic. The AOI loop takes the opposite stance and treats train and validation leakage as an unrecoverable hard stop.

Loop Stages#

The agent enforces an ordered set of stage transitions, and the audit script checks them. The transitions are the following:

prep:      prep                                   (once; skipped when artifacts exist)
baseline:  inference -> kpi_analyze
iterN:     gap_analysis -> embed -> mine -> stage -> train -> inference -> kpi_analyze

The loop does not train at baseline. It evaluates the supplied zero-shot or pretrained checkpoint as iteration 0 and only trains from iteration 1 onward, after mining has produced data to add. Baseline labels land in ${RESULTS_DIR}/baseline/inference/labels/, and the baseline also seeds ${RESULTS_DIR}/train_grounding_dino.yaml by copying your train specification template. The phase directory and the iteration label are both baseline, not iter_0; the spelling iter_0 appears in prose only.

Stage

What it does

Underlying TAO command

Image

gap_analysis

Finds the per-class gaps and the weak images

gap_analysis object_detection -e "$OD_GAP_SPEC"

$TAO_DS_IMAGE

embed

Embeds the weak images

embedding image_embeddings -e "$EMBED_SPEC"

$TAO_DS_IMAGE

mine

Selects pool images near the weak images

tmm unique_neighbor_matching -e "$MINE_SPEC"

$TAO_DS_IMAGE

stage

Copies images and writes the ODVG annotations

Host scripts only, with no container

Not applicable

train

Fine-tunes on the grown dataset

grounding_dino train -e <spec>

$TAO_PYT_IMAGE

inference

Scores the KPI images with the new checkpoint

grounding_dino inference -e "$INFER_SPEC"

$TAO_PYT_IMAGE

kpi_analyze

Computes the per-class metrics and the aggregate mAP

analytics kpi_analyze -e "$KPI_SPEC"

$TAO_DS_IMAGE

Gap Analysis#

The gap_analysis stage consumes the previous phase’s inference labels, so iteration 1 reads the baseline output and iteration N reads iter${N-1}, together with the KPI ground truth and config.kpi_images_dir. Its key settings are input_format: kitti, iou_threshold: 0.5, conf_threshold: 0.0, min_area: 0, default_ap50_threshold: 0.0, default_recall_threshold: 0.0, default_precision_threshold: 0.0, class_mapping: {}, and every gated class listed explicitly in weak_thresholds. Setting default_ap50_threshold: 0.0 is mandatory: leaving it unset means every class you did not list starts marking images weak at 0.5, which quietly inflates the gap set and the mining budget derived from it. The agent runs emit_default_spec.py --stage gap_analysis, which copies assets/gap_analysis_object_detection.yaml because gap_analysis is absent from the TAO default specification list. The asset leaves weak_thresholds unset. The script scripts/prepare_thresholds_for_gap_analysis.py writes one entry per target class from config.ap50_thresholds. The specification build sets weak_thresholds from that file with --set-from-file and --require-no-mandatory, so skipping the script fails the build. This stage is backed by the TAO Data Services gap_analysis command.

Embedding#

The embed stage takes weak_images.parquet directly, because that file already carries filepath and needs no projection step, and it writes weak_images_embeddings.parquet. The model is SigLIP, resolved from the google/siglip-base-patch16-224 snapshot, with batch_size of 64 and model_config_path set with --set model_config_path='""'. The encoder must match the one used to build the source-pool parquet file. Encoder consistency is the whole point of this stage: mismatched encoders produce vectors that are not comparable, and the failure is silent, because mining succeeds and returns confidently wrong neighbors. The script scripts/verify_embed_encoder.py runs before every embed and refuses a specification whose model or model_path differs from the encoder recorded in deft_state.json. For background, refer to Embedding Generation.

Mining#

The mine stage runs tmm unique_neighbor_matching with source_path set to the source pool embeddings, target_path set to weak_images_embeddings.parquet, and desired_unique_count supplied by prepare_budget_for_mining.py. Its other specification fields are allocation_policy, which is class_stratified or global and to which the reference pipeline’s class_balanced mode maps, distance_metric: euclidean, candidate_expansion_factor: 5, embedding_column: embedding and filepath_column: filepath on both the source and the target, source_detection_file, target_detection_file, rare_class_list, save_embeddings: false, and visualize: false. The value detection_format: coco is never inferred, so set it explicitly. For background on targeted mode matching, refer to Targeted Mode Matching.

Staging#

The stage stage is bundled glue with no leaf skill and no container. It runs three host scripts in order. stage_mined_odvg.py copies the images, looks up the ODVG records by basename, renumbers image_id from 0, remaps instances[].label, and writes tmm_odvg.jsonl and labelmap.json, truncating the JSONL file first, with --min-success-rate defaulting to 0.9. validate_odvg_images.py is a hard-stop gate. prepare_exclude_for_mining.py extends the exclude set and omits --parquet-b on iteration 1 only.

Training, Inference, and KPI Analysis#

This loop searches for the training data mixture that produces the best model, not for a training trajectory. Every iteration therefore fine-tunes the original base checkpoint rather than the previous iteration’s weights. That way the loop evaluates every candidate data mixture from the same starting point, and accumulated weight drift does not confound the comparison across iterations. train.pretrained_model_path stays pointed at the original base checkpoint on every iteration. What grows across iterations is the dataset, not the weights. The loop uses the previous iteration’s checkpoint only for inference and the gap analysis that follows it, and never as a training initialization.

The prepare_spec_for_train.py script builds the train specification and appends one {image_dir, json_file, label_map} entry, preceded by prepare_val_split_for_train.py, which builds a validation COCO file from 10 percent of the pool with the category identifiers rewritten to be zero-based. Inference then runs with inference.checkpoint=${RESULTS_DIR}/iter${N}/train/gdino_model_latest.pth. The agent launches the kpi_analyze stage detached with docker run -d and without --rm, because the mAP exists only on that container’s standard output, and its input_format is the uppercase KITTI.

The KPI overlay assets/overlays/kpi_analyze.yaml scores with 101 recall points and sets ignore_sqwidth to 0. Every reported average precision (AP) value moves with the number of recall points, so a run scored with 11 recall points is not comparable to a run scored with 101.

An ok commit of kpi_analyze requires --map-value, and the value must be finite. When kpi_analyze prints mAP: nan, a target class has no ground truth in the KPI set, and the agent commits the stage with --status error.

Zero weak images ends the loop early. If weak_images.parquet has zero rows, no data can be mined this iteration, so the agent commits gap_analysis with status=ok and --weak-image-count 0 and advances directly to loop_stop, because there is nothing for the remaining six stages to consume. That zero is the only proof of the early stop the audit can read: without it, --require-complete reports a fully successful run as INCOMPLETE. It also means every class met its configured AP50 threshold on every image.

Stopping Behavior#

This loop has no metric contract and no target. The mAP is reported, not gated. The loop does not early-exit on a metric target, a regression does not stop the loop and does not trigger a retry, and kpi_analyze records where the model stands only so the report can show the trend across iterations. If you want target-gated stopping, that is a change to the workflow contract rather than something the agent improvises mid-run.

Three conditions stop the loop:

  • max_iterations is reached, and the agent runs the loop-end sequence and reports the mAP trend across iterations.

  • Zero weak images appear at any gap_analysis, and the agent commits loop_stop and reports that the model met every configured per-class AP50 threshold.

  • An unrecoverable gate failure occurs, and the agent halts and reports the exact missing artifact.

The hard stops are any stage with status=error, a missing or zero-row source-pool embedding parquet file, a zero-row mining result when weak images were present, a missing ODVG annotation source, an image and annotation mismatch after staging, and a train exit that emits no new iteration checkpoint. Pool exhaustion also stops the loop and is recorded with commit_stage.py --stage loop_stop --pool-exhausted --pool-remaining N. Before anyone claims the loop is complete, audit_deft_run.py --results-dir "${RESULTS_DIR}" --require-complete must exit zero.

max_iterations defaults to 1, which is one mine, train, and score pass, the smallest run that yields a comparison against the baseline. The agent confirms the value with you when you have not said how many iterations you want, and an unattended run takes the default rather than stopping to ask.

Behavior Worth Knowing#

The following properties of this loop are easy to get wrong, and assuming otherwise leads to a wrong conclusion about a run:

  • results_dir auto-appends the task name. The TAO update_results_dir helper appends the task name, so passing results_dir=X to train writes X/train/ and passing it to inference writes X/inference/. Never append the subdirectory yourself.

  • exclude_path must be null on iteration 1, because TAO Data Services load_datasets raises FileNotFoundError when exclude_path is set but is not a file. From iteration 2 onward it points at the previous mined_cumulative.parquet.

  • candidate_expansion_factor is not the loop iteration count. It seeds the miner’s internal candidate-pool growth and is unrelated to max_iterations.

  • The agent seeds the mining budget from iteration 1’s weak_images.parquet on every iteration. That is deliberate, so the data added per iteration stays constant as the gap set shrinks. desired_unique_count is the weak image count multiplied by multiplier, which defaults to 3.

  • Grounding DINO caption order is the label map. The model assigns a detection to a class by the position of the caption token it matched, so reordering the list silently relabels every prediction, and max_labels must equal len(captions). verify_class_contract.py checks this before every inference.

  • The target classes are a run-wide contract across four places: classes.yaml, weak_thresholds in gap_analysis, rare_class_list in mine, and kpi/mapping.yaml in kpi_analyze.

Outputs#

Every run writes one timestamped directory under results/. Its layout is the following:

results/run_<YYYYMMDD_HHMMSS>/
    deft_state.json
    loop_log.jsonl
    .deft_commit.lock                  commit_stage.py, one writer at a time
    .deft_commit.journal               present only while a commit is in flight
    DEFT_Loop_Report.md
    train_grounding_dino.yaml          seeded at baseline from your template
    val_coco.json                      written by prepare_val_split_for_train.py
    prep/                              present only when prep ran
        inference/labels/              Co-DETR pseudo-labels, already folded
        codetr_category_mapping.yaml
        kitti_mapping.yaml
        classmap_target.txt
        pool_report.json
    iter${N}_summary.md
    baseline/
        inference/labels/*.txt
        kpi/kpi_calc.csv
    iter${N}/
        gaps/                          weak_images.parquet, box_gaps.parquet,
                                       image_metrics.parquet, gap_report.json
        embeddings/weak_images_embeddings.parquet
        mining/                        final_unique_files.parquet, summary.json
        tmm/
            images/
            annotations/               tmm_odvg.jsonl, labelmap.json
        mined_cumulative.parquet
        train_grounding_dino.yaml
        train/                         gdino_model_latest.pth, status.json
        inference/labels/*.txt
        kpi/kpi_calc.csv

Artifact

What it holds

DEFT_Loop_Report.md

The loop report, in Markdown

iter${N}/train/gdino_model_latest.pth

Iteration N’s checkpoint, which must be a newly emitted file under ${RESULTS_DIR}/iter${N}/train/

<phase>/kpi/kpi_calc.csv

The per-class metrics, with the columns Sequence Name, class_name, TP, FP, FN, TN, Pr, Re, Acc, and AP

<phase>/kpi/kpi_summary.json

The aggregate mAP, written by summarize_kpi.py

iter${N}/mining/summary.json

The mining result, alongside final_unique_files.parquet

deft_state.json

The resume snapshot

loop_log.jsonl

The append-only event stream, one JSON line per stage

Other per-iteration files

iter${N}/tmm/staging_report.json, iter${N}/mining/budget.json, and <phase>/kpi_overrides.json

val_coco.json

The validation COCO file that prepare_val_split_for_train.py writes at the run directory root, not per iteration

The loop report is Markdown at ${RESULTS_DIR}/DEFT_Loop_Report.md. There is no HTML report and no HTML template. The script scripts/render_report.py renders the report from disk state; the agent does not spawn a subagent for it or compose it by hand. The script init_deft_state.py writes the report before any stage runs, and every accepted commit_stage.py call re-renders it, so the report is current after each stage. If rendering fails, the script prints a warning and the commit stands. To refresh the report outside the loop, run scripts/render_report.py --results-dir "${RESULTS_DIR}". Add --require-terminal for an end-of-loop render; the script refuses to render until the agent commits loop_stop.

Each phase also writes a precision-recall curve plot and a kpi_analyze.log file into ${RESULTS_DIR}/<phase>/kpi/. The aggregate mAP is printed to standard output only, and nothing writes it into kpi_calc.csv, which is why the loop captures it into kpi_summary.json. The trend itself lives in the ## 1. KPI Trend section of DEFT_Loop_Report.md, with one row per phase carrying the mAP plus one AP50 column per target class, and in ${RESULTS_DIR}/iter${N}_summary.md, which is at most 300 words. There is no separate trend CSV or JSON file.

Read coverage_pct from iter${N}/mining/summary.json. A retrieved_unique_count of 0 with weak images present is a hard stop, because the source pool has nothing left to give or the exclude set has consumed it. A coverage_pct below 50 percent is a warning rather than a gate: the agent warns in the iteration summary that the pool is running dry and that later iterations add little, and then proceeds. The report also flags any iteration whose mining coverage fell below 50 percent, any iteration in which mined images did not reach training, and any target class for which mining returned no images.

The agent lists in the iteration summary each target class that has no images in resultant_dataset.per_class. Mining backfills that class’s slots from other classes, so retrieved_unique_count and coverage_pct still report normal values. The loop continues.

State lives on disk. init_deft_state.py initializes deft_state.json once, handles fresh runs only, and refuses to overwrite without --force. It also writes the initial loop report, and --force archives the previous report with the previous state and log as *.bak.<UTC stamp> files. commit_stage.py is the only supported state writer. audit_deft_run.py is read-only and prints DEFT_RUN_STATUS, last_committed, next_action, and read_before_action. The loop_log.jsonl schema carries seq, ts, iter, stage, status, summary, duration_sec, context_tokens, which is always 0 and reserved, and tokens, an object added at loop end. Stage identifiers are prep, inference, kpi_analyze, gap_analysis, embed, mine, stage, train, and loop_stop, and the iter field takes baseline, iter1, iter2, and so on. Because disk is the source of truth, the agent runs the audit before every stage and, on startup or resume, prints the last five entries of loop_log.jsonl and proceeds from disk-loaded state.

Execution Platforms#

This workflow supports one execution platform, Docker. The skill does not ask which platform to run on; every stage launches with docker run, and the skill requires Docker and the NVIDIA Container Toolkit. The skill reads the credentials guidance of the tao-run-on-docker skill and checks only whether the required environment variable is set. It never asks for or reads credential values.

When a skill that a stage maps to is unavailable, the loop does not stop. The stage runs through the documented docker run fallback, and the agent commits it with --execution-path direct-container instead of --execution-path skill. On a runtime without the agent’s Skill tool, this fallback is the expected route for every stage. When any stage uses the fallback, audit_deft_run.py prints direct_container_stages.

For air-gapped hosts, this workflow ships no references/air-gap.md file, and init_deft_state.py has no --network-mode option. On an air-gapped host, or any time the download cannot run, you supply --zero-shot-checkpoint directly. When you supply your own Co-DETR checkpoint path, no download runs, and that is how an air-gapped host runs preparation. An offline host still needs both container images already present locally, a local SigLIP snapshot, a Co-DETR checkpoint that you supply on a host that cannot reach Hugging Face, and a fully provisioned host Python. Encoder resolution hard-stops when no local snapshot resolves and the agent cannot verify outbound Hugging Face access. For the loop that takes a different stance on these questions, refer to DEFT for Automated Optical Inspection.

Example Prompts#

The skill lists the following prompts for a loop run:

  • “Run the DEFT OD loop”

  • “Run smart data augmentation for grounding dino”

  • “Mine more training data for my detection model and retrain”

  • “Improve detection mAP with gap analysis and unique-neighbor mining”

The skill lists the following prompts for the source pool preparation, which is a separate run that completes before the loop launches:

  • “Prep the source pool”

  • “Pseudo-label my unlabeled images for mining”

  • “Build the mining pool from these raw images”

Source of Truth#

The skill sources below are the executable contract for this workflow. This page summarizes them, and where the two differ, the skill sources win. For an overview of the workflow family, refer to Reference Workflows, and to install the toolkit, refer to the TAO Quick Start Guide.