webdataset

A high-performance Python-based I/O system for large (and small) deep learning problems, with strong support for PyTorch.

Minimal sharded dataset loaders, decoders, and utils for multi-modal document, image, and text datasets.

Unofficial DynaDUSt3R reimplementation trained on Stereo4D (research only).

[CVPR 2021: Oral] In this work, we show that high frequency Fourier spectrum decay discrepancies are not inherent characteristics for existing CNN-based generative models.

DALLE-tools provided useful dataset utilities to improve you workflow with WebDatasets.

High-level API for tar-based dataset

This repo is the official released code of FoPro (AAAI-2023)

A sample subset of the NIH Chest X-ray Dataset. At only 2.4% of the size of the original dataset, it allows creating an accurate classifier using the Augmented Chest X-Ray repository.

Efficiently process webdatasets

Scripts to collect data from CARLA and save them as Webdataset

High-Throughput PyTorch Sequential Data Loaders for GPU Starvation Reduction

Index, group, sample, resynchronize, and repack uncompressed WebDataset TAR archives

DataComp-style image-text dataset filtering on Backblaze B2: stream WebDataset shards from object storage, score image-text alignment with CLIP (open_clip), write filtered shards + quality metrics back to B2 — no local staging, no database. Full-stack Next.js + FastAPI sample for VLM-pretraining data curation

Web project, using SparQL on dbpedia and wikidata for mental disorder search feature

Indonesian election campaign-finance filings from KPU: 14,540 pages for document OCR and structured extraction, with 3,333 PII-bearing pages withheld

Stream WebDataset shards straight from Backblaze B2 object storage into PyTorch training — no local staging disk. A FastAPI + Next.js sample app that packs media into .tar shards, writes them to B2, and streams them back as an IterableDataset with live throughput and a distributed worker/node shard split.

Bulk-download image-text datasets into Backblaze B2 as reproducible WebDataset tar shards with img2dataset — no local staging disk. Sample app (FastAPI + Next.js) that streams shards to S3-compatible object storage, validates download yield, and streams them back for PyTorch/JAX training.

Systematic VLA training optimization on 2× RTX 3090. WebDataset + FlashAttention-2 + FSDP → 3.3× throughput, 26% VRAM reduction. Profiler traces and W&B report linked. Reproducible in one command.