Sandchest

The open-source speech-to-text API.

Documentation: sandchest.com/docs · for agents: llms.txt, llms-full.txt, openapi.json.

Sandchest runs real NVIDIA Parakeet models behind an AssemblyAI-compatible API, a typed native TypeScript SDK, and a small, self-hostable SaaS control plane. Responses contain genuine word-level millisecond timestamps, punctuation, confidence scores, language metadata, and the actual model identifier.

There are no placeholder transcripts or mocked production inference paths.

Included

• Next.js 16, React 19, Bun 1.4, Tailwind CSS, and Effect HttpApi.
• PlanetScale-compatible Postgres, Drizzle migrations, and durable jobs.
• Always-warm Parakeet routing: English v2 and multilingual v3 on MLX or CUDA.
• AssemblyAI-compatible upload, transcript creation, and polling endpoints.
• A standalone, fully typed TypeScript SDK.
• Six-digit email-code sign-in with Better Auth and Resend.
• Tenant-isolated uploads, transcript history, usage, and hashed API keys.
• Autumn entitlements, customer billing, and idempotent duration metering.
• Private S3-compatible storage, retention, rate limits, and SSRF protection.
• Docker Compose and validated AWS Terraform/OpenTofu infrastructure.
• Reproducible comparisons against read-only Cap data and AssemblyAI history.

Start on Apple Silicon

Install Bun 1.4, Rust (edition 2024), Xcode command-line tools, and FFmpeg. Run:

bun install
bun run db:migrate
bun run worker:inference

In a second terminal:

bun run dev

If DATABASE_URL is configured, also run the durable queue consumer in a third terminal with the same database, storage, inference URL, and token settings:

bun run worker

External Postgres disables inline processing by default. The web API and native inference process alone will leave submitted jobs queued. With embedded Postgres, leave processing inside the web process; do not start a second database process.

Open http://localhost:3000. Without DATABASE_URL, development uses durable embedded Postgres in the ignored .sandchest/postgres directory. The first worker startup prepares a checksummed native MLX SDK, builds the Rust worker, and downloads pinned v2/v3 weights into the Hugging Face cache. No Python runtime is needed. If Resend is not configured, six-digit codes are logged only during local development.

The English-only primary model is nvidia/parakeet-tdt-0.6b-v2. Automatic language detection and explicit non-English requests route to a separately warmed nvidia/parakeet-tdt-0.6b-v3 model. Detected language and confidence come from the genuine generated transcript; no English language is fabricated for other audio. Locale-aware English/Portuguese number formatting preserves word timestamps and avoids converting ambiguous article words such as "one", "um", and "uma". Native inference is serialized on each worker; scale independent worker processes only after measuring device memory and throughput. The Rust greedy decoder compiles its joint step, synchronizes once per decision, and reuses predictor state across blank tokens. Both platforms use the same acoustic frontend and normalize supported media, including fragmented AAC-in-MP4 files, to bounded 16 kHz mono PCM. The multilingual model accepts its actual 25 supported languages; unknown language codes fail clearly.

The native CUDA deployment replaces NeMo in the image, but CUDA runtime validation is still pending. Apple Silicon results do not establish NVIDIA performance or better accuracy than AssemblyAI. Python remains only for reference experiments.

AssemblyAI compatibility

For requests within Sandchest's supported capabilities, keep the official SDK and change apiKey and baseUrl:

import { AssemblyAI } from "assemblyai";

const client = new AssemblyAI({
  apiKey: process.env.SANDCHEST_API_KEY,
  baseUrl: "http://localhost:3000",
});

const transcript = await client.transcripts.transcribe({
  audio: audioBuffer,
  speech_models: ["universal-3-5-pro", "universal-2"],
  language_detection: true,
  punctuate: true,
  format_text: true,
  disfluencies: true,
});

console.log(transcript.speech_model_used);
console.log(transcript.words);

Supported prerecorded endpoints include upload, transcript creation and polling, transcript listing/deletion, word search, sentences, paragraphs, SRT and VTT. Both raw and Bearer authorization headers work. Idempotency-Key is accepted on creation. Requested provider model names remain compatible, while speech_model_used reports the actual configured model. Webhooks, audio time ranges, custom spelling and punctuation-off are supported. Setting punctuate: false lowercases display text and removes edge punctuation after word alignment, preserving surviving word timestamps and confidence. A matching native worker is required; older workers cannot silently ignore these options. Custom spelling maps case-insensitive spoken words or phrases to one display word, after recognition and alignment. A merged phrase keeps its first start, last end, and mean word confidence. It does not bias recognition or improve acoustic accuracy. New jobs default to disfluencies: false; use true for Cap's verbatim behavior. English filled-pause cleanup runs after alignment and preserves retained words' timestamps and confidence. It protects quoted literals, acronyms and numeric metre units. Other languages retain their model output; multilingual filler behavior has not been established as equivalent to AssemblyAI. Previously accepted jobs retain their original options and idempotent replay behavior. Upgrade the Rust worker before enabling these API defaults: an older worker that does not acknowledge cleanup fails without completing or billing the job.

Options that Sandchest cannot actually perform, including speaker diarization and custom word boosting, are rejected with a clear client error instead of being silently accepted.

Compatibility GET requests return the current transcript state immediately. The official AssemblyAI SDK owns polling, including its configured polling interval. The current implementation remains a subset of the full AssemblyAI prerecorded API; see the replacement gates for open gaps. Unknown options are rejected rather than silently dropped. Deletion erases words, text, source URLs, custom vocabulary and private error details while preserving usage/billing history. An unshared upload is deleted immediately; a shared upload is kept until its last live transcript is deleted. Storage failures remain retryable. Deleted idempotency keys stay reserved and return 409, never a new job. Deploy migration 0004 and the asset-locking creation code before enabling DELETE across a rolling deployment; drain old creation requests before that cutover.

Cap replacement limits

The latest unchanged Rust candidate completed 50 new Cap recordings (two hours) through the local API, with fresh AssemblyAI comparisons and Cap editor/caption consumer checks. General beta remains unapproved: restrictive language hints, no-speech/fallback semantics, multilingual quality and difficult word timing still need work. A separate 50-recording validation sample remains untouched. See the fresh Cap evidence for measured latency, scope and limitations.

Sandchest is not yet a replacement for all Cap transcription traffic. Cap's automatic-language request includes languages that the default Parakeet route does not support; passing that request unchanged to that route fails validation. Do not silently remove those languages from the request. Leave automatic and unverified language routes on AssemblyAI until broader native coverage and routing have been tested.

Deployment alone does not establish readiness. Full-file and fragmented-live requests, captions, editable transcripts, recovery, and sustained load must pass on deployment hardware. An explicitly enabled English canary also needs an AssemblyAI fallback and rollback. Keep Sandchest credentials separate from Cap's existing AssemblyAI credential, which can serve other provider integrations.

See Cap replacement acceptance criteria for the language, accuracy, performance, and rollout gates. Local benchmark results do not establish a deployed latency or general accuracy win.

Native TypeScript SDK

The package source lives in packages/sdk:

import { Sandchest } from "@sandchest/sdk";

const client = new Sandchest({
  apiKey: process.env.SANDCHEST_API_KEY,
  baseUrl: "http://localhost:3000",
});

const transcript = await client.transcripts.transcribe({
  audio: audioBytes,
  disfluencies: true,
  custom_spelling: [{ from: ["sand chest"], to: "Sandchest" }],
});

The native endpoints are POST /api/v1/transcripts and GET /api/v1/transcripts/:id. The SDK propagates cancellation through uploads, requests, and polling, enforces polling deadlines, and bounds its idempotent binary-upload retry cache. Persistent tenant-scoped SHA-256 audio fingerprints keep same-recording retries correct even after an in-memory upload entry expires; genuinely different bytes still conflict.

Architecture

1. Better Auth verifies an email code and provisions a tenant workspace.
2. API keys are stored only as HMAC-SHA256 digests.
3. Audio is written to private local or S3-compatible storage.
4. A single Postgres transaction creates the transcript and durable job.
5. Optional SQS messages wake consumers; Postgres remains authoritative.
6. Consumers claim work with row locking, skip-locked semantics, and leases.
7. A warm Rust worker decodes media and runs genuine NVIDIA Parakeet through MLX.
8. Model subword tokens become punctuation-preserving millisecond-aligned words.
9. Completion and exactly one duration usage event commit atomically.
10. Autumn receives an idempotent event after local usage is durable; queue workers retry unsynchronized events after transient billing failures.

Development can claim durable jobs in the web process; production runs separate web, queue, and inference services.

Configuration

The complete contract is in .env.example:

Variable	Purpose
DATABASE_URL	PlanetScale Postgres or ordinary Postgres
BETTER_AUTH_SECRET	At least 32 random characters in production
SANDCHEST_API_KEY_PEPPER	Independent HMAC key-hashing secret
SANDCHEST_INFERENCE_URL	Private warm-model worker URL
SANDCHEST_INFERENCE_TOKEN	Worker request authentication
SANDCHEST_DEVICE	auto, metal, cuda, or explicit cpu; auto requires the platform GPU
SANDCHEST_MODEL	English-only primary Parakeet model
SANDCHEST_MULTILINGUAL_MODEL	Warm model for automatic/non-English requests
HF_HOME	Cache for the pinned English and multilingual models, both always loaded
SANDCHEST_ENGLISH_REFINEMENT	Selectively improve uncertain detected-English transcripts
SANDCHEST_STORAGE_DRIVER	local or s3
SANDCHEST_S3_BUCKET	Private object-storage bucket
SANDCHEST_AUDIO_RETENTION_HOURS	Uploaded audio retention
SANDCHEST_MAX_UPLOAD_BYTES	Enforced streamed upload and worker body-size limit
SANDCHEST_QUEUE_CONCURRENCY	Independent durable-job processing lanes
SANDCHEST_COMPATIBILITY_POLL_WAIT_MS	Legacy internal wait helper; compatibility GET returns immediately
RESEND_API_KEY	Production email-code delivery
AUTUMN_SECRET_KEY	Hosted production billing and entitlements
AUTUMN_AUDIO_FEATURE_ID	Usage feature measured in audio seconds
AUTUMN_PLAN_ID	Actual Autumn subscription plan identifier
SANDCHEST_SELF_HOSTED	Explicit self-hosted billing bypass
SANDCHEST_SQS_QUEUE_URL	Optional AWS queue wake-up signal

Hosted production fails closed when required database, cryptographic, or billing configuration is absent. /api/health/live verifies that the web process and database can serve requests; /api/health additionally reports actual inference-model readiness.

Verification

bun run typecheck
bun run lint
bun run format:check
bun test
bun run test:python
bun run build
tofu -chdir=infra/aws init -backend=false
tofu -chdir=infra/aws validate

After providing the five independent Docker Compose secrets described in SELFHOST.md, also verify docker compose config --quiet.

With a running inference worker, generate ignored synthetic fixtures:

say -o .sandchest/smoke.aiff "Um, this is a genuine Sandchest transcription test."
ffmpeg -y -i .sandchest/smoke.aiff -c:a aac -movflags +frag_keyframe+empty_moov+default_base_moof -f mp4 .sandchest/smoke-fragmented.mp4
ffmpeg -y -f lavfi -i anullsrc=r=16000:cl=mono -t 2 .sandchest/silence.wav
SANDCHEST_REAL_MODEL_TEST=true bun test tests/integration-real-model.test.ts
bun run verify:local

The local verifier starts a temporary application, completes genuine email-code sign-in, creates a workspace and API key, uploads audio through the dashboard, transcribes again through the unchanged AssemblyAI SDK, and checks the actual API-key, usage, and billing dashboard pages plus exact usage metering. Authentication codes, session cookies, and API keys are never printed.

Honest benchmarks

Use a reader-role PlanetScale tunnel and existing Cap credentials:

pscale connect cap-production main --role reader --port 3327
CAP_BENCHMARK_LIMIT=3 bun --env-file=/absolute/path/to/Cap/.env run benchmark:cap
SANDCHEST_BENCHMARK_LOCAL=true bun run benchmark
bun run verify:cap
ASSEMBLYAI_BASELINE_LIMIT=25 bun --env-file=/absolute/path/to/Cap/.env run benchmark:assemblyai

Customer media and existing encrypted reference transcripts are written only below the Git-ignored .sandchest/benchmarks directory with private permissions. Cap and its production database are never modified.

To evaluate a larger separate holdout without replacing the existing benchmark:

CAP_BENCHMARK_ROOT=.sandchest/benchmarks/cap-expanded \
  CAP_BENCHMARK_LIMIT=10 \
  bun --env-file=/absolute/path/to/Cap/.env run benchmark:cap
SANDCHEST_BENCHMARK_MANIFEST=.sandchest/benchmarks/cap-expanded/manifest.json \
  SANDCHEST_BENCHMARK_LOCAL=true bun run benchmark
SANDCHEST_BENCHMARK_MANIFEST=.sandchest/benchmarks/cap-expanded/manifest.json \
  bun run benchmark:diagnose
SANDCHEST_BENCHMARK_MANIFEST=.sandchest/benchmarks/cap-expanded/manifest.json \
  bun run verify:cap

Experiment with actual decoding, model precision, and acoustic preprocessing without printing customer transcript text:

HF_HUB_OFFLINE=1 uv run --project workers/inference \
  python -m sandchest_worker.experiments --variants original,formatted_numbers

Results include AssemblyAI agreement, aligned timestamp differences, filler preservation, genuine language-detection accuracy, per-language agreement, time to first result, end-to-end latency percentiles, concurrency, and throughput. NVIDIA utilization and VRAM samples are included when nvidia-smi is actually available; they remain null on Apple Silicon. Compute cost is calculated only when SANDCHEST_GPU_HOURLY_USD is supplied. Agreement against AssemblyAI is not word-error rate; WER appears only for explicitly human-verified samples. Historical AssemblyAI created-to-completed timings exclude upload and client polling.

The supplied historical Cap baseline is $0.209 per audio hour; it is not a current provider quote or a claim that Sandchest is cheaper.

See BENCHMARKS.md for the independent LibriSpeech evaluation, fresh AssemblyAI comparison, paired decoder benchmark, and their limitations.

See SELFHOST.md for complete GPU, Docker, AWS, security, and operating-cost guidance.