A lightweight job processing system: submit work over HTTP, park it in Redis, and let an asyncio worker pool run it with retries and delayed execution. Built as a clear, modular alternative to bolting Celery onto a small service.
Source: github.com/sarweshsoman/Async-Task-engine.
What it does
- REST API for job submit and status (
pending→processing→completed/failed/retrying) - Redis lists for immediate work, sorted sets for delayed jobs, JSON blobs for job state
- Configurable worker concurrency and max retries with exponential backoff
- Docker Compose setup (Redis + API + workers)
- Structured JSON logging
The worker’s process_job() is a placeholder on purpose — swap in your own handlers (email, exports, and so on).
How to run
git clone https://github.com/sarweshsoman/Async-Task-engine.git
cd Async-Task-engine
docker compose up --build
API at http://localhost:8000, docs at /docs. Submit a job:
curl -X POST "http://localhost:8000/api/v1/jobs" \
-H "Content-Type: application/json" \
-d '{"payload": {"task": "process_data"}, "delay_seconds": 0}'
Scale workers with docker compose up --scale worker=5.