Workspace Schema

VisData workspaces are designed for developers who are used to working directly with JSON files and images. There is no database or query language to learn. You just modify files and observe the changes in the user interface.

Note

Changes in the dataset detail view are instantly viewable, but search or filtering operation will require a re-indexing operation.

Key Terms

Term

Definition

Data Link
A settings file that defines how data is imported (and exported). VisData is designed with
the assumption that external data can frequently change and allows developers to re-import in a convenient way.
The dataset and results can be deleted without impacting the data link.

Dataset
Encompasses all information about a data import including frames, ground truth, and frame metadata.
This excludes algorithm results.

Result
Output from computer vision system / algorithm. A dataset can have many separate results.

Result ROI Layer
A result can be divided into multiple parts called region of interest (ROI) layers. For example each could be a separate layer:
objects detections, motion detections, and tracks.

Dataset Meta

A dataset file contains all of the high-level information about a dataset including dataset-level attributes and frame-level aggregates (but no other frame-level data).

Datasets come in two major varieties: asset (image, video, etc) and package. You can think of an asset dataset as a file and package as a folder that contains references to other (asset) datasets.

Path Structure: $WORKSPACE / repo / dataset_meta / {suite} / {group} / {name}.json

Full Examples

Asset Example

This asset (dataset_type=1) was imported from a video and is referenced in several packages.

Example Asset Path: $WORKSPACE / repo / dataset_meta / san-francisco / embarcadero / 1583908321.json

{
     "suite":        "san-francisco",    // Top-level dataset identifier
     "group":        "embarcadero",      // Mid-level dataset identifier
     "name":         "1583908321",       // Low-level dataset identifier (unix timestamp in this example)
     "dataset_type": 1,                  // 1 / "video"
     "sensor_type":  1,                  // 1 / "rgb" (taken with a visible camera)
     "sensor_mount": 1,                  // 1 / "stationary" (the camera does not move)
     "source":       "video",            // The footage was imported from a video file
     "source_info": {
         "video_path": "$EXTERNAL_DATA/videos/san-francisco-embarcadero.mov"
         "index_end": 105,
         "index_start": 0
     },
     "conditions": {
         "environment": 1,
         "placement": 0,
         "time_of_day": 3,
         "weather": {
             "fog": 0,
             "glare": 0,
             "hail": 0,
             "high_contrast": 0,
             "overcast": 1,
             "rain": 0,
             "snow": 0,
             "snowing": 0,
             "wind": 0
         }
     },
     "comments": "User specified full-text searchable comments",
     "package_refs": [
         // This asset is contained in the following package (ref is a dataset id)
         {"suite": "san-francisco", "group": "embarcadero", "name": "1583908321"},
         {"suite": "san-francisco", "group": "embarcadero", "name": "1583918321"},
         {"suite": "san-francisco", "group": "embarcadero", "name": "1583928321"}
     ],
     "height": 540,
     "width": 962,
     "flag_all_reviewed": 0,
     "flag_cloud": 0,
     "flag_done": 0,
     "flag_flagged": 0,
     "gps": "37.80004598897606, -122.3977882220308",
     "highlights": [ .. ],
     "location": "Pier 9, Embarcadero District, San Francisco, California, United States",
     "nominal_fps": "30.0",
     "preview_filename": "00335.jpg",
     "searchroi_bike": 3836.0376,
     "searchroi_dog": 0.0,
     "searchroi_keywords": "urban morning low_light dawn overcast",
     "searchroi_person": 4473.642,
     "search_thumbnails": { .. },
     "timezone": "Europe/Amsterdam",
     "address": "Pier 9, #9, San Francisco, CA 94111, United States",
     "brightness": 99
}

Package Example

This package (dataset_type=3) was imported from Conservator

Example Package Path: $WORKSPACE / repo / dataset_meta / san-francisco / embarcadero / all.json

{
    "suite":        "san-francisco",
    "group":        "embarcadero",
    "name":         "all",
    "dataset_type": 3,             // 3 / "package", PACKAGE dataset
    "sensor_type":  1,             // 1 / "rgb", ALL children datasets use a visible camera
    "sensor_mount": 3,             // 3 / "various", children have different values
    "source":       "conservator",
    "source_info":  { .. },
    "comments":     "User specified full-text searchable comments",
    // This package Contains the following standard datasets (refs are dataset ids)
    "package_refs": [
      {"suite": "san-francisco", "group": "embarcadero", "name": "1583908321"},
      {"suite": "san-francisco", "group": "embarcadero", "name": "1583918321"},
      {"suite": "san-francisco", "group": "embarcadero", "name": "1583928321"}
    ]
}

Top-Level

Key

Type

Example

Description

“suite”

string

“san-francisco”

Required top-level dataset identifier. Must match {suite} in the path.

“group”

string

“embarcadero”

Required mid-level dataset identifier. Must match {group} in the path.

“name”

string

“1583908321”

Required low-level dataset identifier. Must match {name} in the path.

“dataset_type”

int / enum

2 (“images”)
Required dataset type to help filter and organize datasets.
This determines the type of content that is stored in a dataset.
A video dataset means it’s a continuous set of frames at a specific frame rate.
An image dataset means it consists of discrete images that are not necessarily temporally related.
A package dataset is a collection of other asset datasets (like a folder).
- 0 unknown (assumed asset)
- 1 video (asset)
- 2 images (asset)
- 3 package

“nominal_fps”

string

“30.0”
Required approximate frames per second of the video.
This primarily controls the frequency of the sampling in the activity plot on the search page
and helps balance search relevance between datasets.
Represented as a string to prevent rounding of floating point numbers.

“conditions”

object

{ .. }
Key and object are required, but can be empty.
Conditions that describe the dataset including weather and environment used for filtering.
See Dataset Meta Conditions

“source”

string

“conservator”
Optional descriptor of data source (defaults to unknown).

- unknown undetermined source
- coco COCO Object Detection COCO format
- mot_seq Multi object tracking sequence format
- conservator Conservator dataset format
- nntc The import came from an NNTC (Teledyne FLIR Neural Network Target Classifier)
- images Import from a directory of image files (order will be determined by alphanumerical sort)
- video Import from from a video file

See also:
- Data Link Raw Frame Sources
- Data Link Raw Video Sources

“source_info”

object

{ .. }
Optional object with additional contextual information about the source of data

“comments”

string
Optional free form comments from user. Used for full-text search.

“package_refs”

list

[ {..}, .. ]
A list of dataset IDs (references) that describe package relationships.
If it is a package these will be the children (all of the datasets that are in this package).
If it is an asset then these will be the parents (all of the packages this dataset belongs to)

A Package Ref object is defined as:
{"suite": "{suite}", "group": "{group}", "name": "{name}"}

“sensor_type”

int / enum

1 (“rgb”)
Optional type to describe the sensor to help filter and organize datasets.
- 0 unknown
- 1 rgb (visible)
- 2 thermal (LWIR)
- 3 various

“sensor_mount”

int / enum

2 (“moving”)
Optional type to describe the sensor mount to help filter and organize datasets.
- 0 unknown
- 1 stationary
- 2 moving
- 3 various

“flag_*”

int
-1 (unknown)
0 (no)
1 (yes)
The flags describe high-level attributes about each dataset and they can be set by
the user via the gui. These are top-level fields for technical reasons related to
full-text search indexing.

User-set
flag_flagged - dataset has been flagged by the user

Computed
flag_all_reviewed auto-generated. Specifies all frames have annotations
flag_has_scores dataset has all available scores
flag_trainval the dataset is part of training / validation
flag_is_package the dataset is part of training / validation

“searchroi_{label}”

float

3836.0376
Computed fields for full-text search. Determines full-text search relevance boosting.

“search_thumbnails”

object

{ .. }
Computed object for full-text search. Determines thumbnail to display while searching.

“highlights”

object

{ .. }
Computed object for full-text search. Used to display the mini activity
plot in the datasets view.

Conditions

"conditions": {
    "environment": 5, // Various
    "placement":   0, // Outdoor
    "time_of_day": 1, // Night
    "weather": {
        "rain":    -1, // Unknown
        "hail":     0, // No
        "snow":     1, // Yes
        "fog":      1,
        "glare":    0,
        "wind":     0,
        "overcast": 0
    }
}

  • -1 unknown

  • 0 no

  • 1 yes

Source Info

"source_info": {
  "annotations_path": "$EXTERNAL/conservator/dataset_vid_val/index.json",
  "frames_path": "$EXTERNAL/conservator/dataset_vid_val/data"
}

Frame and Result Meta

A frame meta file describes frame information for a dataset (which includes ground truth & pinned result) and the same format is used for results.

Only asset datasets have frame meta files while package datasets do not have frame information or results.They only include a Dataset Meta file.

There will always be at minimum one file that holds the frames information for a dataset (in repo/frame_meta/…) and there can be many result files for a single dataset (in results/…).

  • Dataset Meta Path structure: $WORKSPACE / repo / frame_meta / {suite} / {group} / {name}.json

  • Result Meta Path structure: $WORKSPACE / results / {suite} / {group} / {name} / {result_title}.json

General Structure

{
    "_description": "Computer vision result...",
    "_roi_schema": {
        // Region of Interest Schema Object
        "roi_{region_of_interest_name}": { .. }
    },
    "frames": [
        // Frame Object
        {
            "roi_{region_of_interest_name}": [
                // Region of Interest Object
                { .. }
            ],
            "frame": 1, // "frame_number" (1 is the first frame)
            "frame_time": 1000,
            "qa_status": "approved",
            "flagged": true,
            "reviewed": true,
            "extra_info": { .. },
            "custom": { .. }
       }
    ]
}
Dataset Meta Example

Example path: $WORKSPACE / repo / frame_meta / san-francisco / embarcadero / 1583908321_meta.json

{
    "_description": "QA dataset",
    "_roi_schema": {
        // Region of Interest Schema Object
        "roi_gt": {
            "name": "Ground truth",
            "color": "green",
            "details": "Human annotations",
            "source_url": "$EXTERNAL_DATA/path",
            "source_created_at": 1601582020,
            "source_updated_at": 1601582020,
            "created_at": 1601582020,
            "updated_at": 1601582020
        },
        // Region of Interest Schema Object
        "roi_s": {
            "name": "Pinned Result",
            "color": "rgba(255,0,0,0.95)",
            "details": "Current best tracking results",
            "source_url": "$EXTERNAL_DATA/path",
            "source_created_at": 1601582020,
            "source_updated_at": 1601582020,
            "created_at": 1601582020,
            "updated_at": 1601582020
        }
    },
    "frames": [
        {
            "roi_gt": [
                // Region of Interest Object
                {
                    "bndbox": {
                        "xmax": 960,
                        "xmin": 212,
                        "ymax": 531,
                        "ymin": 163
                    },
                    "class": "car",
                    "custom": { .. }
                }
            ],
            "roi_s": [
                // Region of Interest Object
                {
                    "bndbox": {
                        "xmax": 960,
                        "xmin": 212,
                        "ymax": 531,
                        "ymin": 163
                    },
                    "class": "car",
                    "custom": { .. }
                }
            ],
            "frame": 1,
            "frame_time": 1000,
            "qa_status": "approved",
            "flagged": true,
            "reviewed": true,
            "custom": { .. }
       }
    ]
}
Result Meta Example

Example path: $WORKSPACE / results / san-francisco / embarcadero / 1583908321 / yolo_v7.json

{
    "_description": "Current best result",
    "_roi_schema": {
        // Region of Interest Schema Object
        "roi_d": {
            "name": "Detections",
            "color": "red",
            "details": "YoloV7",
            "source_url": "$EXTERNAL_DATA/path",
            "source_created_at": 1601582020,
            "source_updated_at": 1601582020,
            "created_at": 1601582020,
            "updated_at": 1601582020
        },
        // Region of Interest Schema Object
        "roi_false_positive": {
            "name": "False Positives",
            "color": "rgba(255,0,0,0.95)",
            "details": "Missed predictions",
            "source_url": "$EXTERNAL_DATA/path",
            "source_created_at": 1601582020,
            "source_updated_at": 1601582020,
            "created_at": 1601582020,
            "updated_at": 1601582020
        },
    },
    "frames": [
        {
            "roi_d": [
                // Region of Interest Object
                {
                    "bndbox": {
                        "xmax": 960,
                        "xmin": 212,
                        "ymax": 531,
                        "ymin": 163
                    },
                    "class": "vehicle",
                    "score": 0.896,
                    "custom": { .. }
                },
                {
                    ..
                }
            ],
            "roi_false_positive": [
                // Region of Interest Object
                {
                    "bndbox": {
                        "xmax": 960,
                        "xmin": 212,
                        "ymax": 531,
                        "ymin": 163
                    },
                    "class": "vehicle",
                    "custom": { .. }
                }
            ],
            "frame": 1,
            "frame_time": 1000,
            "custom": { .. }
        }
    ]
}

Top-Level

Key

Type

Example

Description

“_roi_schema”

list

[ {..}, .. ]
Required list of Region of Interest Schema objects that
describe high-level properties of region of interest
(“roi_{name}” key) objects in the “frames” list.
Starts with a _ so that it readily shows up at the head of the file.

“frames”

list

[ {..}, .. ]
Required list of Frame objects in the dataset.
Dataset meta will only include the following keys:
- roi_gt - Ground truth annotation layer
- roi_s - Pinned result layer

“_description”

string

“QA dataset”
Optional A human-readable description of the
meta file to help the developer/data scientist.
Shows up in UI as comment field. the “frames” list.

Frame

Key

Type

Example

Description

“frame”

int

1

Required frame index that determines the order of the frame (starts from 1).

“frame_time”

int

1000

Required relative time in milliseconds from the first frame.

“qa_status”

string

“approved”

Optional field that determines the QA status of the ground truth in the frame.

“flagged”

boolean

true
Optional field that keeps track of flagged status.
Frames can be flagged for a user-defined reason.

“roi_{name}”

list

[ {..}, .. ]
List of Region of Interest objects.
Allowed characters for {name} are a-z, A-Z, 0-9, and underscore.

“reviewed”

boolean

true
Optional field that determines if the frame has been annotated.
The name “reviewed” is a misnomer and there is a proposed rename.

“extra_info”

object

{ .. }
Dictionary with additional frame information that is automatically adding during import.
For example the original input frame resolution is stored in this dictionary.
The content stored here is often source-specific.

“custom”

object

{ .. }
Optional free-form object for storing custom frame metadata.
Accepts any valid JSON object, including nested objects and arrays.

Region of Interest

A object that describes a region of interest (ROI) for a given frame.

Key

Type

Example

Description

“class”

string

“vehicle”

Optional label name of the region of interest.

“score”

float

0.896
Optional field that determines if the frame has been annotated.
The name “reviewed” is a misnomer and there is a proposed rename.

“bndbox”

object

{ .. }
Determines the bounding box boundaries of the region of interest (ROI).
Example:

“bndbox”: {
“xmax”: 960,
“xmin”: 212,
“ymax”: 531,
“ymin”: 163
}

“custom”

string

{ .. }
Optional free-form object for storing custom frame metadata.
Accepts any valid JSON object, including nested objects and arrays.

Region of Interest Schema

Describes how the region of interest (ROI) layer should be displayed. Also includes hints for the import process.

Key

Type

Example

Description

“name”

string

“Detections”

Required name that shows up for the layer in the user interface

“color”

string

“rgba(255,0,0,0.95)”
Required color description (standard HTML/CSS format).
See Web CSS Color Picker

“details”

string

“YoloV7”
User-defined details.

“source_url”

string

“$EXTERNAL_DATA/path”
Optional source path of the data imported.

“source_created_at”

int

1601582020
Unix timestamp which describes the time the source was created.
This helps when determining if a re-import is necessary.

“source_updated_at”

int

1601582020
Unix timestamp which describes the time the source was updated.
This helps when determining if a re-import is necessary.

“created_at”

int

1601582020
Unix timestamp that describes the time the ROI layer was created.

“updated_at”

int

1601582020
Unix timestamp that describes the time the ROI layer was last updated.

Score Settings

The score settings file determines scoring operations (when running score detections from workspace tools) and display of metrics in the search view. A display score file can by manually created by users.

Path structure: $WORKSPACE / repo / settings / score.json

  • See Remap File definition for details on the "remap" format.

  • See Label Vector definition for details on the "labels" format.

Example

{
  "#description": "Metrics for two algorithms: Algorithm A (CSPNet) & Algorithm B (Yolo V4)",
  "score": [
    {
      "#comment":           "score file at: $WORKSPACE/score/scores_for_algorithm_a/score_map.json",
      "display_score_file": "score_mAP.json",
      "result_name":        "scores_for_algorithm_a",

      "display_name":       "Algorithm A",
      "description":        "overall mAP using CSPnet Resnext50 960x512 model",

      "labels":             "primary_labels.json",
      "remap":              "primary_remap.json"
    },
    {
      "#comment":           "score file at: $WORKSPACE/score/scores_for_algorithm_b/score_map.json",
      "display_score_file": "score_mAP.json",
      "result_name":        "scores_for_algorithm_b",

      "display_name":       "Algorithm B",
      "description":        "overall mAP using Yolo V4 960x512 model",

      "labels":             "primary_labels.json",
      "remap":              "primary_remap.json"
    }
  ]
}

Display Score Map

This is format that maps a dataset to a score. The name of this file is user-defined to accommodate for various different metrics per result. In other words a single result may have mAP scores, framerate values, number of false positives, etc.

Path structure: $WORKSPACE / score / {result_name} / {score_dict_name}.json

Object Structure

{
    "{suite}_{group}_{name1}": 0.9993,
    "{suite}_{group}_{name2}": 0.9226,
    "{suite}_{group}_{name3}": 0.0904
}
Example

Example path: $WORKSPACE / score / scores_for_algorithm_a / score_mAP.json

{
    "san-francisco_embarcadero_1487377316": 0.9993,
    "san-francisco_embarcadero_1487381566": 0.9226,
    "san-francisco_embarcadero_1487386550": 0.0904
}