[Suggestion] Celery distributed audio transcription workers with Redis queue and progress tracking (Python)

[deepgram-robot]

What to build

A Celery-based distributed audio transcription system using Redis as the message broker. The system accepts audio files via an API endpoint, queues them as Celery tasks, processes them through Deepgram's STT and Audio Intelligence APIs across distributed workers, and provides real-time progress tracking via a polling endpoint.

Why this matters

Celery is the most widely used distributed task queue in the Python ecosystem. Organizations processing large volumes of audio files — call centers, media companies, legal transcription services — need distributed processing with progress visibility. This example provides a production-ready pattern that scales horizontally with additional workers and integrates naturally with existing Python backend infrastructure using familiar tools (Celery, Redis, FastAPI).

Suggested scope

• Language: Python (FastAPI + Celery + Redis)
• Deepgram APIs: STT pre-recorded, Audio Intelligence
• Infrastructure: Redis (broker), Celery workers, FastAPI API server
• Key patterns: Task distribution, progress callbacks, result aggregation, error handling
• Docker Compose: Include docker-compose.yml for easy local setup
• Complexity: Moderate

Acceptance criteria

• Runnable with minimal setup (clone, add API key, docker-compose up)
• README explains the pattern clearly
• Uses current SDK version
• Distributed processing across multiple Celery workers
• Real-time progress tracking for batch jobs
• Docker Compose for local development setup
• Handles failures gracefully with retry policies

---

Raised by the DX intelligence system.