Skip to content

Releases: mcity/mcity_data_engine

v1.1.0: Depth Estimation, Segmentation, Class Mapping, Dataset SUNRGBD

18 Mar 18:54
@daniel-bogdoll daniel-bogdoll
v1.1.0
8fa9266
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
v1.1.0: Depth Estimation, Segmentation, Class Mapping, Dataset SUNRGBD Latest
Latest

Initial release of the Monocular Depth Estimation Module, adding support for depth estimation, segmentation, and class mapping.
To use this module, configure your workflow and dataset in the config:

SELECTED_WORKFLOW = ["auto_label_mask"]
SELECTED_DATASET = {"name": "SUNRGBD", "n_samples": None}

Each individual workflow can be configured in WORKFLOWS = {...}

Visual Data

As mentioned in V1.0.0, the Mcity Data Engine supports visual data. Leveraging Voxel51, we convert any dataset into their dataset format and perform all operations on it. This way, the Data Engine can be utilized with any dataset as long as it can be converted into the V51 format.

This release expands dataset support by integrating additional dataset

The SUNRGB-D dataset contains 10,335 RGB-D images, each of which has a corresponding RGB image, depth image, and camera intrinsics. It contains images from the NYU depth v2, Berkeley B3DO, and SUN3D datasets.

The integration of SUNRGB-D further strengthens the Mcity Data Engine’s ability to process high-quality depth estimation and segmentation data.

Data Curation

Unlike the other datasets currently available in the Mcity Data Engine, SUNRGB-D requires a manual download.

To download the Dataset, use the following commands:

curl -o sunrgbd.zip https://rgbd.cs.princeton.edu/data/SUNRGBD.zip
unzip sunrgbd.zip

Load the Dataset into Voxel51, using:

python main.py

The dataset should then appear in your Voxel51 environment:

mde_gt_heatmaps

Customize Heatmaps (Optional)

When working with depth maps, adjusting the color scheme and opacity can improve visualization.

To modify these settings:

  • Click the palette icon at the top-left corner of Voxel51.
  • Select value for Color annotations by.
  • Scroll to the bottom to Colorscale and select name with type viridis in the text box for optimal contrast.

mde_color_customization

Intelligent Class Mapping

The class mapping workflow leverages zero-shot classification models from HuggingFace to facilitate the alignment of class labels between a source dataset and a more granular target dataset. This streamlined workflow facilitates working with multiple datasets by easing data preprocessing, to work on subsequent tasks such as sequential/joint finetuing of pretrained models for tasks like object detection. This workflow serves two primary functions. First, it can be used with Voxel51 to visually assess the performance of various zero-shot classifier models on a designated source-to-target mapping—during which only tags are added while the labels remain unchanged. Once the optimal model is identified, the workflow can then be configured to update the labels in the source dataset by setting the change_labels flag to True.

To use this workflow, configure the models, datasets and the required label-mapping in the config.:

SELECTED_WORKFLOW = ["class_mapping"]
#Ensure the SELECTED_DATASET is the same as Source Dataset, which is used for class mapping
SELECTED_DATASET = {"name": "fisheye8k", "n_samples": None}

    "class_mapping": {
            # get the source and target dataset names from datasets.yaml
            "dataset_source": "fisheye8k",
            "dataset_target": "mcity_fisheye_2000",

            # Set to True to change detection labels in the dataset, Set to False to just add tags without changing the dataset
            "change_labels": True,

         # Choose any number of models from the options below hf_models_zeroshot_classification, by removing the comment in front of the model name. 
        "hf_models_zeroshot_classification": [
            "Salesforce/blip2-itm-vit-g",
            #"openai/clip-vit-large-patch14",
            #"google/siglip-so400m-patch14-384",
            #"google/siglip2-so400m-patch14-384",
            #"kakaobrain/align-base",
            #"BAAI/AltCLIP",
            #"CIDAS/clipseg-rd64-refined"
        ],
        "thresholds": {
            "confidence": 0.2
        },
        "candidate_labels": {
            #Target class(Generalized class) : Source classes(specific categories)
            "car": ["car", "van"],
            "truck": ["truck", "pickup"],
            #One_to_one_mapping
            "bike" : ["motorbike/cycler"]
        }
    }

screenshot__1180_

Merged PRs

Full Changelog: https://github.com/mcity/mcity_data_engine/commits/v1.1.0

Contributors

isabelmoore and abeyankargiridharan
Assets 2
Loading

v1.0.0: AWS Integration, Data Curation, Model Training, Model Inference

28 Feb 15:52
@daniel-bogdoll daniel-bogdoll
v1.0.0
58aaac4
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
v1.0.0: AWS Integration, Data Curation, Model Training, Model Inference

Initial release of the Mcity Data Engine with a focus on the task of object detection for visual datasets. To use the Mcity Engine, simply select your workflows and the dataset in the config:

SELECTED_WORKFLOW = ["embedding_selection", "auto_labeling"]
SELECTED_DATASET = {"name": "fisheye8k", "n_samples": None}

Each individual workflow can be configured in WORKFLOWS = {...}. Start the Mcity Data Engine with python main.py.

Visual Data

The Mcity Data Engine supports visual data. Leveraging Voxel51, we convert any dataset into their dataset format and perform all operations on it. This way, the Data Engine can be utilized with any dataset as long as it can be converted into the V51 format.

Currently connected datasets:

image

Data Curation

To select samples of interest, the Data Engine initially provides three workflows:

Selection by Embedding Ensemble

image6

Leveraging the Voxel51 Brain component, the Data Engine computes image embeddings based on an ensemble of models. These are leveraged to select both representative and unique samples.

Selection by Language-Prompted Zero-Shot Ensemble

image4

Leveraging Zero-Shot Object Detection Models from Hugging Face, the Mcity Data Engine identifies images that include n instances of classes of interest. It combines an ensemble of models to reduce both false positives and false negatives.

Selection by Anomaly Detection

image3

Leveraging anomaly detection models from Anomalib, the Data Engine detects frames that contain anomalies. This workflow requires a labeled dataset. During training, a known class is treated as an anomaly and excluded from the training dataset. During inference, samples with a high anomaly score can be selected for inspection.

Data Labeling

The Data Engine is connected to CVAT for manual labeling. Based on the results of the data curation workflows, samples can be filtered, manually inspected, and labeled. Given trained models, auto labeling can be performed through model inference.

Model Training and Inference

The Mcity Data Engine currently supports three model sources for model training and inference:

Trained models are uploaded to Hugging Face and can be used for later inference. Custom models are wrapped in a Container and can be used through Docker or Singularity.

Merged PRs

Full Changelog: https://github.com/mcity/mcity_data_engine/commits/v1.0.0

Contributors

daniel-bogdoll, dependabot, and rajanikant-patnaik
Loading

v0.1

06 Dec 19:30
@daniel-bogdoll daniel-bogdoll
v0.1
8904c21

Choose a tag to compare

View all tags
v0.1 Pre-release
Pre-release

Release Notes

  • Parallelized multi-GPU workflow introduced for zero-shot object detection
Loading