🚀 Distributed Job Processing Platform

A backend platform for asynchronous job processing, priority-based task distribution, automatic retries, and job monitoring built with Django REST Framework, Celery, Redis, and SQLite.

📌 Overview

The Distributed Job Processing Platform allows users to submit background jobs through a REST API.

Django REST Framework handles job creation and retrieval, while Celery processes jobs asynchronously using Redis as the message broker. Jobs are assigned to priority-based queues and tracked throughout their lifecycle.

The platform also implements automatic retry handling for failed jobs and provides a web dashboard for monitoring job statistics and recent job activity.

🏗️ Architecture

                    ┌─────────────────────┐
                    │   Client / Browser  │
                    └──────────┬──────────┘
                               │
                               ▼
                    ┌─────────────────────┐
                    │    Django REST      │
                    │        API          │
                    └──────────┬──────────┘
                               │
                               ▼
                    ┌─────────────────────┐
                    │        Redis        │
                    │    Message Broker   │
                    └──────────┬──────────┘
                               │
                ┌──────────────┼──────────────┐
                ▼              ▼              ▼
          HIGH Queue      MEDIUM Queue     LOW Queue
                │              │              │
                └──────────────┼──────────────┘
                               ▼
                    ┌─────────────────────┐
                    │   Celery Worker     │
                    │    process_job()    │
                    └──────────┬──────────┘
                               │
                               ▼
                    ┌─────────────────────┐
                    │       SQLite        │
                    │   Job Data/Status   │
                    └──────────┬──────────┘
                               │
                               ▼
                    ┌─────────────────────┐
                    │     Dashboard       │
                    │   Job Monitoring    │
                    └─────────────────────┘

✨ Features

• Asynchronous background job processing
• REST API for job creation and retrieval
• Celery-based task processing
• Redis message broker
• Priority-based queues
• HIGH, MEDIUM, and LOW job priorities
• Automatic retry mechanism
• Configurable maximum retry count
• Job failure handling
• Job lifecycle tracking
• Error message tracking
• Django monitoring dashboard
• Recent job monitoring
• Docker support

🛠️ Tech Stack

Technology	Purpose
Python	Backend programming
Django	Web framework
Django REST Framework	REST API
Celery	Asynchronous task processing
Redis	Message broker
SQLite	Development database
Django ORM	Database operations
HTML/CSS	Monitoring dashboard
Docker	Containerization
Git & GitHub	Version control

🔄 Job Processing Flow

Client
  ↓
POST /api/jobs/
  ↓
Django creates Job
  ↓
Priority-based queue
  ↓
Redis
  ↓
Celery Worker
  ↓
Process Job
  ↓
Update Database
  ↓
SUCCESS

If processing fails:

Celery Worker
     ↓
   Failure
     ↓
  RETRYING
     ↓
 Retry Attempt
     ↓
 ┌───┴────┐
 ↓        ↓
Success  Failure
 ↓        ↓
SUCCESS  RETRY
          ↓
     Maximum Retries
          ↓
        FAILED

📊 Job Lifecycle

QUEUED
   ↓
RUNNING
   ↓
 ┌───────────────┐
 ↓               ↓
SUCCESS        RETRYING
                  ↓
               RUNNING
                  ↓
               FAILED

⚡ Priority Queues

Jobs are routed to different Celery queues based on their priority.

HIGH
  ↓
high_priority

MEDIUM
  ↓
medium_priority

LOW
  ↓
low_priority

This allows jobs to be distributed according to their configured priority.

🔁 Retry Mechanism

Failed jobs are automatically retried up to the configured maximum retry count.

The current implementation supports 3 retries by default.

Example:

Attempt 1
   ↓
Failure
   ↓
Retry 1
   ↓
Failure
   ↓
Retry 2
   ↓
Failure
   ↓
Retry 3
   ↓
Failure
   ↓
FAILED

The dashboard tracks the retry count for each job.

Example:

3 / 3

🔌 REST API

The API is available under:

/api/jobs/

Create Job

POST /api/jobs/

Example request:

{
    "name": "Generate Sales Report",
    "priority": "HIGH",
    "payload": {
        "task": "generate-report",
        "month": "January",
        "year": 2026
    },
    "max_retries": 3
}

List Jobs

GET /api/jobs/

Retrieve Job

GET /api/jobs/<job_id>/

Example:

GET /api/jobs/JOB-67188FFB/

Example Response

{
    "job_id": "JOB-67188FFB",
    "name": "Final Retry Test",
    "status": "FAILED",
    "priority": "HIGH",
    "retry_count": 3,
    "max_retries": 3,
    "error_message": "Simulated job failure"
}

🖥️ Monitoring Dashboard

The Django dashboard is available at:

http://127.0.0.1:8000/

The dashboard displays:

• Total jobs
• Queued jobs
• Running jobs
• Successful jobs
• Failed jobs
• Retrying jobs
• Recent jobs
• Job IDs
• Job status
• Priority
• Retry count
• Creation time

The dashboard also provides a refresh option for retrieving the latest database state.

🧪 Failure & Retry Testing

The retry mechanism can be tested by submitting a job containing:

{
    "name": "Retry Test",
    "priority": "HIGH",
    "payload": {
        "task": "retry-test",
        "force_fail": true
    },
    "max_retries": 3
}

When force_fail is set to true, the worker intentionally raises a simulated exception.

The job transitions through the retry process and is eventually marked as:

FAILED

after reaching the maximum retry count.

📁 Project Structure

Distributed-Job-Processing-Platform/
│
├── config/
│   ├── settings.py
│   ├── urls.py
│   ├── celery.py
│   └── ...
│
├── jobs/
│   ├── migrations/
│   ├── models.py
│   ├── serializers.py
│   ├── views.py
│   ├── tasks.py
│   ├── urls.py
│   └── ...
│
├── templates/
│   └── jobs/
│       └── dashboard.html
│
├── .dockerignore
├── .gitignore
├── Dockerfile
├── manage.py
├── requirements.txt
└── README.md

⚙️ Installation

1. Clone the Repository

git clone https://github.com/PulaBhavya/Distributed-Job-Processing-Platform.git
cd Distributed-Job-Processing-Platform

2. Create Virtual Environment

python -m venv venv

3. Activate Virtual Environment

Windows

venv\Scripts\activate

Linux/macOS

source venv/bin/activate

4. Install Dependencies

pip install -r requirements.txt

5. Run Database Migrations

python manage.py migrate

6. Start Redis

Make sure Redis is running before starting the Celery worker.

7. Start Celery Worker

Windows

celery -A config worker --loglevel=info --pool=solo

Linux/macOS

celery -A config worker --loglevel=info

8. Start Django

Open another terminal:

python manage.py runserver

Open the dashboard:

http://127.0.0.1:8000/

Open the API:

http://127.0.0.1:8000/api/jobs/

🧪 Validation

Check the Django configuration:

python manage.py check

Run database migrations:

python manage.py migrate

Check the API:

GET http://127.0.0.1:8000/api/jobs/

🐳 Docker

The project includes a Dockerfile for containerizing the Django application.

Docker can be used to create a consistent runtime environment for the application.

Redis is required for Celery message brokering.

🎯 Engineering Work

• Designed a backend architecture for asynchronous job execution
• Developed REST APIs using Django REST Framework
• Integrated Celery for background task processing
• Configured Redis as the message broker
• Implemented priority-based job queues
• Implemented automatic retry handling
• Designed job lifecycle and status tracking
• Implemented failure and error handling
• Built a Django-based job monitoring dashboard
• Added Docker support
• Tested successful, failed, priority, and retry job scenarios

📈 Testing Results

The platform was tested with multiple job scenarios:

Test Scenario	Result
HIGH priority job	SUCCESS
MEDIUM priority job	SUCCESS
LOW priority job	SUCCESS
API job creation	SUCCESS
API job retrieval	SUCCESS
Celery worker processing	SUCCESS
Failed job handling	SUCCESS
Automatic retry mechanism	SUCCESS
Maximum retry handling	SUCCESS

🚀 Future Improvements

• PostgreSQL production database
• Multiple Celery worker nodes
• Job cancellation
• Scheduled jobs using Celery Beat
• API authentication and authorization
• Pagination and filtering
• Real-time updates using WebSockets
• Prometheus and Grafana monitoring
• Distributed tracing
• AWS deployment
• CI/CD using GitHub Actions
• Docker Compose for multi-service deployment

👩‍💻 Author

Bhavya Pula

• GitHub: https://github.com/PulaBhavya
• Portfolio: https://pulabhavya.github.io/

⭐ Repository

GitHub Repository:

https://github.com/PulaBhavya/Distributed-Job-Processing-Platform