Annotations#

The annotation service, which is part of TAO Data Services, offers tools for users to easily manipulate groundtruth labels. Its subtasks include:

  • convert

  • slice

  • merge

  • sparse4d_prepare

Each subtask is explained in the following sections. In TAO 7.0 you drive the convert, slice, and merge subtasks with natural-language prompts to your agent; for DAFT-formatted datasets specifically, the data/tao-convert-dataset-format skill handles conversion between supported DAFT formats. For non-DAFT formats (COCO, KITTI, etc.) and for slicing or merging, the agent scripts the work directly. The sparse4d_prepare subtask is a command, not a prompt; refer to Preparing Sparse4D Co-Training Artifacts.

Supported Data Formats#

The annotation service supports the KITTI, ODVG COCO, and AICity/OVPKL formats.

Configuring a Specification File for Annotation Conversion#

The following is a sample specification file for converting a COCO dataset to KITTI format. It has three key components–data, kitti, and coco–as well as a global parameter, all of which are described below.

Field

Description

Data Type and Constraints

Recommended/Typical Value

results_dir

The directory to save the annotation-conversion log to

string

–

data

The dataset config

Dict

–

kitti

The KITTI config

Dict

–

coco

The COCO config

Dict

–

odvg

The ODVG config

Dict

–

aicity

The AICity config

Dict

–

Data Config#

The data configuration (data) specifies the source and target formats of the label conversion, as well as the output path. Refer to the Data Annotation Format page for more information about the data formats including KITTI, COCO, and ODVG.

  • Note 1: Direct KITTI to ODVG or ODVG to KITTI conversion is not supported, but you can use COCO as an intermediate format to bridge KITTI and ODVG.

  • Note 2: When input_format and output_format are both COCO, a new COCO annotation file is saved with category IDs remapped to contiguous IDs.

Field

Description

Data Type and Constraints

Recommended/Typical Value

input_format

The input data format (“KITTI”, “ODVG”, or “COCO”)

string

output_format

The output data format (“KITTI”, “ODVG”, or “COCO”)

string

output_dir

The path to save the converted annotations

string

KITTI Config#

The KITTI configuration (kitti) specifies the KITTI dataset information.

Field

Description

Data Type and Constraints

Recommended/Typical Value

image_dir

The image directory

string

label_dir

The label directory

string

project

The project name, which is used as the scene_id when converting to COCO format. The default value is the parent directory name of the image_dir.

string

mapping

A YAML file specifying the category mappings. If this value is not not specified, all categories in the label_dir are used).

string

no_skip

If True, do not skip images without any valid annotations.

bool

preserve_hierarchy

If True, preserve the KITTI folder structure.

bool

The following is an example of a category mapping file:

- person:
  - person
  - Person
  - person_group
  - rider
- bag:
  - hand_bag
  - backpack
  - personal_bag
- face:
  - face

COCO Config#

The COCO configuration (coco) specifies the COCO annotation file location.

Field

Description

Data Type and Constraints

Recommended/Typical Value

ann_file

The annotation file

string

refine_box

Whether to refine boxes with segmentation when converting to KITTI

bool

use_all_categories

Whether to use all categories

bool

add_background

Whether to add background categories

bool

ODVG Config#

The ODVG configuration (odvg) specifies the ODVG annotation file location.

Field

Description

Data Type and Constraints

Recommended/Typical Value

ann_file

The annotation file

string

labelmap_file

The label map file

string

AICity Config#

The AICity configuration (aicity) specifies the AICity dataset information.

Field

Description

Data Type and Constraints

Recommended/Typical Value

root

Path to the dataset root directory

string

version

Dataset version identifier

string

“2025”

split

Comma-separated list of dataset split directory names under the dataset root directory.

string

“train”

class_config

Class configuration dictionary that lists classes, mappings, attributes, and valid ID ranges

Dict

recentering

When you enable this field, the converter subtracts the origin of each camera group from the 3D box positions and the camera positions; the shift applies only when camera grouping is enabled. The shipped unlabeled-scene specification sets this field to false.

bool

rgb_format

File format/extension of RGB videos or images

string

“mp4”

depth_format

File format/extension of depth data

string

“h5”

camera_grouping_mode

Comma-separated list of camera grouping strategies. The valid strategies are random and training; write the list without spaces. An empty string turns camera grouping off.

string

“random”

num_frames

Number of frames to load per sequence. If set to -1, all frames will be loaded.

integer

9000

anchor_init_config

Anchor-initialization configuration dictionary (see below)

Dict

load_annotations

Whether to load 3D ground truth and depth. For calibrated scenes without 3D annotations, set this field to false. The converter then writes gt_boxes=None, writes no depth paths, ignores depth_format, and skips anchor initialization.

bool

True

fps

Capture rate in frames per second of the synchronized, contiguous camera frames. The converter derives the frame timestamps and the labeled object velocities from this value.

float; must be finite and greater than 0

30.0

Anchor-Initialization Configuration#

The anchor-initialization configuration (anchor_init_config) specifies the anchor-initialization parameters.

Field

Description

Data Type and Constraints

Recommended/Typical Value

num_anchor

Number of anchor points to generate

integer

900

detection_range

Detection range for anchor points

float

-1

sample_ratio

Sample ratio for anchor points

integer

-1

output_file_name

Name of the output file for anchor points

string

“anchor_init.npy”

Running the Annotation Conversion#

Ask your agent to perform the conversion. For example:

“Convert the COCO annotations at s3://my-bucket/coco/annotations/instances_train2017.json to KITTI format. Images are at s3://my-bucket/coco/train2017/ . Write the converted labels to s3://my-bucket/coco-as-kitti/.”

For DAFT-formatted datasets, the agent invokes the data/tao-convert-dataset-format skill; for other formats it scripts the conversion directly using the spec keys described below.

Configuring a Specification File for the Annotation Slicing Service#

The following is a sample specification file for slicing a COCO annotation file. It has two key components–data and filter–as well as a global parameter (results_dir), all of which are described below.

Field

Description

Data Type and Constraints

Recommended/Typical Value

results_dir

The directory to save the output annotation files and logs to

string

–

data

The dataset configuration

Dict

–

filter

The filter configuration

Dict

–

Data Config#

The dataset configuration (data) specifies the input format and annotation file.

Field

Description

Data Type and Constraints

Recommended/Typical Value

format

The configuration format. Currently, only “COCO” is supported.

string

“COCO”

annotation_file

The input COCO annotation file

string

Filter Config#

The filter configuration (filter) specifies how to slice the annotation data, which can be done in one of four modes:

  1. random: Randomly split the annotation file into N partitions or sample the annotation file by a certain percentage

  2. category: Filter annotation labels by the desired categories

  3. number: Pick N samples in order from the annotations

  4. filename: Filter the annotations by their file names

Field

Description

Data Type and Constraints

Recommended/Typical Value

mode

The filter mode (“random”, “category”, “number”, “filename”)

string

“COCO”

dump_remainder

A flag specifying whether to dump the remainder annotations. This parameter only applies when the mode parameter is set to “random” and the split parameter is a float value.

bool

split

The integer number of splits or the float sampling percentage (in “random” mode)

float or integer

num_samples

The number of annotations to keep (in “number” mode)

integer

included_categories

Categories to keep (in “category” mode)

list

excluded_categories

Categories to exclude (in “category” mode)

list

re_patterns

List of file name patterns to match (in “filename” mode)

list

Running the Annotation Slicer#

Ask your agent to perform the slice. For example:

“Slice the COCO annotation file at s3://my-bucket/coco/instances_train2017.json into an 80/20 random split. Write both outputs to s3://my-bucket/coco-splits/.”

The agent scripts the slice using the spec keys described above.

Configuring a Specification File for Annotation Merge#

The following is a sample specification file for merging COCO annotation files. It has two key components–data and filter–as well as a global parameter (results_dir), all of which are described below.

Field

Description

Data Type and Constraints

Recommended/Typical Value

results_dir

The directory to save the output annotation file and logs

string

–

data

The dataset config

Dict

–

Data Config#

The data configuration (data) specifies the input format and annotation file.

Field

Description

Data Type and Constraints

Recommended/Typical Value

format

The configuration format. Currently, only “COCO” is supported.

string

“COCO”

annotations

A list of COCO annotation files

string

Note

All COCO annotation files must share the same categories.

Running the Annotation Merge#

Ask your agent to perform the merge. For example:

“Merge s3://my-bucket/coco/train_part_a.json and s3://my-bucket/coco/train_part_b.json into a single COCO file at s3://my-bucket/coco/train_merged.json .”

The agent scripts the merge using the spec keys described above.

Preparing Sparse4D Co-Training Artifacts#

The sparse4d_prepare subtask builds the artifacts that Sparse4D reads during 2D and 3D co-training. You run it in the TAO Data Services container, not in the TAO PyTorch container that trains the model. Each run executes one operation, which you select with the operation field. This section describes the lazy_index, ltt_2dgt, and rtdetr_2d operations.

The ltt_2dgt and rtdetr_2d operations write sidecar files. A sidecar file is a separate per-scene file that carries 2D supervision for the training dataset. A scene name must not contain +, which the training dataset reserves as the separator for bird’s-eye view (BEV) groups.

The following table summarizes what each operation reads and writes:

Operation

Input

Output

lazy_index

A directory of annotation pickle (PKL) files, or a .txt file that lists one PKL path on each line.

A lazy index file and, by default, a _pkl_cam_counts.pkl camera-count file.

ltt_2dgt

Raw scene directories with ground-truth JSON files that contain 2D bounding boxes.

One <scene>__ltt2dgt.npz visible 2D ground-truth sidecar file for each scene.

rtdetr_2d

A scene directory with a labels.tar.gz archive of RT-DETR detection labels in each camera subdirectory.

One ltt_rtdetr2d/v1 NumPy archive (NPZ) detection cache.

Refer to Sparse4D Co-Training Artifact Formats for the layout of each output file. Refer to Prepare Artifacts with TAO Data Services for the training workflow that consumes these artifacts.

Running the Sparse4D Preparation#

Run the subtask with the annotations command. The command requires the -e option for this subtask, so you always pass a specification file. Append each field that you want to override as a key=value argument:

annotations sparse4d_prepare -e <specification_file> operation=<operation> results_dir=<results_directory>

Start from the shipped specification:

sparse4d_prepare.yaml

The run reads only the block of fields that operation selects. The following top-level fields apply to every operation:

  • The operation field chooses the operation to run and the block of fields to read. The default is lazy_index.

  • The results_dir field is mandatory. The command stores experiment.yaml, status.json, and sparse4d_prepare_summary.json in this directory. The summary records the artifact paths and counts that the operation returns, and the command also prints it.

  • The overwrite field controls whether the command can replace existing files. The default is false.

  • The class_names field lists the ordered class taxonomy. The default is the four classes person, gr1_t2, agility_digit, and nova_carter. Keep this list identical to dataset.classes in the training specification, including the order.

  • The subclass_map field maps a class in class_names to raw source names (aliases) that resolve to it. Every key must also appear in class_names.

Note

The released loose-to-tight weights require a specific seven-class order. Refer to 2D-to-3D Geometric Distillation for the order and for the training workflow.

Before the command writes any artifact, it checks every output path of the selected operation, including sparse4d_prepare_summary.json in results_dir. If any of these files exists and overwrite is false, the command stops and reports the number of existing files and the first path. To rerun an operation, set overwrite=true, or use a new results_dir and new artifact paths. If a required field is empty, the command stops with an error that names the field. The required fields are lazy_index.annotation_source for lazy_index, ltt_2dgt.output_dir for ltt_2dgt, and rtdetr_2d.input_dir and rtdetr_2d.output_path for rtdetr_2d.

Building the Lazy Index#

The lazy_index operation indexes annotation PKL files so that the training dataset can load frames lazily when dataset.lazy_load is true. Set lazy_index.annotation_source to one of the following:

  • A directory of PKL files

  • A .txt file that lists one PKL path on each line

For a .txt file, the command skips empty lines and lines that start with #, and it rejects a PKL path that appears more than once. It resolves relative paths against the working directory of the process and not against the .txt file, so use absolute paths that are visible in the container.

For example, the following command indexes the PKL files that /data/mixed_train_split.txt lists:

annotations sparse4d_prepare -e /specs/sparse4d_prepare.yaml \
  operation=lazy_index \
  lazy_index.annotation_source=/data/mixed_train_split.txt \
  results_dir=/results/lazy_index

By default, the command writes the lazy index and the camera-count file alongside the annotation source. Refer to Sparse4D Co-Training Artifact Formats for their names and layout. A repeated run reuses the index entries of PKL files whose size and modification time are unchanged. Set lazy_index.force=true to index every PKL file again. To rebuild an existing index, also set overwrite=true. The lazy_index block has the following fields:

Field

value_type

description

default_value

valid_min

valid_max

valid_options

automl_enabled

annotation_source

string

Annotation directory or split text file.

force

bool

Re-index every source PKL instead of reusing unchanged entries.

False

workers

int

Worker count; zero selects the smaller of 32 and the CPU count.

0

0

write_camera_counts

bool

Also write the legacy _pkl_cam_counts.pkl sidecar.

True

camera_counts_path

string

Optional override for the camera-count sidecar path.

Building Visible 2D Ground-Truth Sidecar Files#

The ltt_2dgt operation turns the 2D boxes in raw scene ground truth into one visible 2D ground-truth sidecar file for each scene. Each scene directory contains one of the following ground-truth layouts:

  • ground_truth_final/ground_truth_<frame>.json files, with one JSON list for each frame

  • A single ground_truth.json file

Each annotation provides object type and 2d bounding box, and optionally object id and 2d bounding box visible. Each annotation keys its boxes by camera name. The command skips an annotation that has no 2d bounding box.

When object id is absent, the producer writes instance_id: -1. File generation does not verify that the sidecar can be joined to a Sparse4D training sample. For supervised LTT training on 3D-labeled scenes, provide IDs that match the labeled PKL’s instance_inds, together with matching frame IDs and camera names. The separate 2D-only geometric-distillation route uses rtdetr_2d caches and does not require instance IDs.

Select the scenes with the ltt_2dgt.selection fields. Name scenes with scenes, scenes_file, scene_dirs, or train_split. If you name none, the command processes every subdirectory of selection.data_root that contains ground truth. If the command resolves no scene, it stops with an error.

For example, the following command writes a sidecar file for each scene under /data/labeled:

annotations sparse4d_prepare -e /specs/sparse4d_prepare.yaml \
  operation=ltt_2dgt \
  ltt_2dgt.selection.data_root=/data/labeled \
  ltt_2dgt.output_dir=/data/ltt_2dgt \
  results_dir=/results/ltt_2dgt

The command writes one file for each scene to ltt_2dgt.output_dir and names it after the scene directory. It assigns each object type to a class through class_names and subclass_map. An object type that is not a class name, not an alias in subclass_map, and not listed in ltt_2dgt.drop_names stops the run with an error that names the type. The command skips the types in drop_names, which defaults to pallet, without a message. The ltt_2dgt block has the following fields:

Field

value_type

description

default_value

valid_min

valid_max

valid_options

automl_enabled

selection

collection

Selects the raw scene directories to process.

FALSE

annotation_version

categorical

Annotation version that the operation records in the sidecar metadata; the value does not change which ground truth the operation reads.

v0.1

v0.0,v0.1

output_dir

string

Directory for per-scene ltt_2dgt/v1 NPZ files.

frame_stride

int

Keeps only frames whose numeric ID is divisible by this value.

1

1

max_frames_per_scene

int

Maximum number of frames to write for each scene; 0 keeps all frames.

0

0

drop_names

list

Source object types to omit explicitly.

[‘pallet’]

FALSE

The selection block has the following fields:

Field

value_type

description

default_value

valid_min

valid_max

valid_options

automl_enabled

data_root

string

Root containing raw scene directories.

scenes

list

Optional raw scene names below data_root.

[]

FALSE

scenes_file

string

Optional text file containing raw scene names.

scene_dirs

list

Optional explicit raw scene directories.

[]

FALSE

train_split

string

Optional Sparse4D PKL split used to derive raw scene names.

dedup_regex

string

Regex removing generated prefixes from split scene names.

^CT[w.]+?__

Building RT-DETR 2D Cache Files#

The rtdetr_2d operation normalizes RT-DETR detections that you computed earlier into one cache file for a scene. It does not run the detector. Arrange the detections as one KITTI-format label archive for each camera at <input_dir>/<camera>/labels.tar.gz. Each archive follows these rules:

  • Each text file holds the detections of one frame, and the last integer in the file name is the frame ID.

  • In each row, the first field is the class name, the box corners (x1, y1, x2, y2) are the fifth to eighth fields, and the last field is the confidence score.

  • The command ignores rows with fewer than nine fields and rows whose box or score is not a finite number.

  • The command drops detections with a score below confidence_threshold and boxes whose width or height is at most 1.0 in the units of the box coordinates.

Map the cameras and labels with these rules:

  • Set rtdetr_2d.camera_map in your specification file to map the camera directory names to the camera names in your Sparse4D PKL files. A directory that you do not map keeps its own name.

  • Every detector label must resolve to a class. The command resolves labels through class_names and subclass_map, and rtdetr_2d.class_map adds or overrides aliases. Map a label to null to drop it. A label that does not resolve stops the run with an error.

The command takes the scene name from rtdetr_2d.scene_name; if that field is empty, it uses the name of the directory that contains input_dir. It appends .npz to rtdetr_2d.output_path if the path has no .npz suffix. To let the training dataset find the cache through dataset.rtdetr_2d_cache_dir, name the file <scene>__rtdetr2d.npz. Refer to Sparse4D Co-Training Artifact Formats for how the training dataset resolves the scene name.

For example, the following command converts the labels of the RealWarehouse scene:

annotations sparse4d_prepare -e /specs/sparse4d_prepare.yaml \
  operation=rtdetr_2d \
  rtdetr_2d.input_dir=/data/RealWarehouse/rt-detr \
  rtdetr_2d.output_path=/data/rtdetr/RealWarehouse__rtdetr2d.npz \
  results_dir=/results/rtdetr_2d

The summary in results_dir records the following values:

  • The output path, the scene name, and the camera names

  • The confidence threshold, the frame stride, and the source of the detections

  • The class mapping that the command applied

  • The number of rows with at least nine fields for each raw class label

  • The number of frames for each camera

  • The number of detection rows and the number of processed frame and camera pairs

The rtdetr_2d block has the following fields:

Field

value_type

description

default_value

valid_min

valid_max

valid_options

automl_enabled

input_dir

string

Scene directory containing <camera>/labels.tar.gz.

output_path

string

Output ltt_rtdetr2d/v1 NPZ path.

scene_name

string

Optional scene-name override.

camera_map

collection

Map label archive directory names to calibration camera names.

{}

FALSE

class_map

collection

Map detector labels to class_names; null explicitly drops a source label.

{}

FALSE

confidence_threshold

float

Drops detections with a score below this value.

0.4

0.0

1.0

frame_stride

int

Keeps only label files whose frame ID is divisible by this value.

1

1

max_frames_per_camera

int

Maximum number of distinct frames to keep for each camera; 0 keeps all frames.

0

0