================================================================================
                    NATS-BASED MESSAGE QUEUE ARCHITECTURE
                    Comprehensive Technical Documentation
================================================================================

TABLE OF CONTENTS
-----------------
1. Executive Summary
2. Technology Stack Overview
3. Detailed Component Analysis
4. Architecture Diagrams & Flow
5. Deployment Models
6. Integration Points
7. Security & Authentication
8. Monitoring & Observability
9. Scalability & Performance
10. Troubleshooting Guide

================================================================================
1. EXECUTIVE SUMMARY
================================================================================

This system implements a high-performance, scalable message queue architecture
using NATS JetStream as the core messaging backbone. The architecture follows
a producer-consumer pattern where FastAPI endpoints receive HTTP requests,
publish messages to NATS, and worker processes consume and forward these
messages to external APIs.

KEY CHARACTERISTICS:
- Asynchronous message processing with guaranteed delivery
- Horizontal scalability (auto-scaling FastAPI and Worker pods)
- Multi-deployment support (Docker Compose for VPS, Kubernetes for clusters)
- High availability with distributed pod placement
- Real-time monitoring and metrics collection
- Secure authentication and CORS handling

================================================================================
2. TECHNOLOGY STACK OVERVIEW
================================================================================

┌─────────────────────────────────────────────────────────────────────────┐
│                        CORE TECHNOLOGIES                                │
└─────────────────────────────────────────────────────────────────────────┘

1. NATS (NATS Server + JetStream)
   - Purpose: Message broker and streaming platform
   - Version: Latest stable
   - Ports: 4222 (client), 8222 (monitoring), 6222 (clustering)

2. FastAPI (Python Web Framework)
   - Purpose: REST API server that receives HTTP requests
   - Version: Latest (Python 3.11)
   - Port: 8000

3. Python Workers (Async Consumers)
   - Purpose: Background workers that process messages from NATS
   - Port: 9090 (metrics)

4. Nginx (Reverse Proxy)
   - Purpose: CORS handling and proxy for NATS monitoring endpoints
   - Version: Alpine (lightweight)

5. NATS Dashboard (Monitoring UI)
   - Purpose: Web-based monitoring interface for NATS
   - Port: 80 (internal), exposed via Traefik

6. Docker & Docker Compose
   - Purpose: Containerization and local/VPS deployment

7. Kubernetes (k3s/k8s)
   - Purpose: Container orchestration for production clusters

8. Traefik (Ingress Controller)
   - Purpose: TLS termination and routing

9. Prometheus (Metrics)
   - Purpose: Metrics collection and monitoring

================================================================================
3. DETAILED COMPONENT ANALYSIS
================================================================================

┌─────────────────────────────────────────────────────────────────────────┐
│ 3.1 NATS SERVER                                                         │
└─────────────────────────────────────────────────────────────────────────┘

WHAT IT IS:
-----------
NATS is a lightweight, high-performance messaging system designed for cloud-
native applications. JetStream adds persistence and streaming capabilities.

WHY IT'S USED:
-------------
✓ Ultra-low latency (< 1ms message delivery)
✓ High throughput (millions of messages per second)
✓ Built-in persistence with JetStream
✓ At-least-once delivery guarantees
✓ Simple pub/sub model
✓ No external dependencies (single binary)
✓ Perfect for microservices communication

HOW IT'S USED:
--------------
1. Stream Configuration:
   - Stream Name: "EVENTS"
   - Subject Pattern: "api.>"
   - Storage: File-based persistence
   - Retention: Work queue policy

2. Consumer Configuration:
   - Consumer Name: "worker_consumer"
   - Pull-based subscription
   - Acknowledgment required (ACK/NAK)
   - Max pending: 10 messages per worker

3. Message Flow:
   FastAPI → Publishes to subject "api.v1.deliveries.updatedelivery"
   Worker → Pulls messages from stream "EVENTS" via consumer

4. Authentication:
   - Username: admin
   - Password: package@321#
   - Connection URL: nats://admin:package%40321%23@nats:4222

CONFIGURATION FILES:
- docker-compose.yml: NATS service definition
- nats.conf: NATS server configuration (referenced in compose)

┌─────────────────────────────────────────────────────────────────────────┐
│ 3.2 FASTAPI APPLICATION                                                 │
└─────────────────────────────────────────────────────────────────────────┘

WHAT IT IS:
-----------
FastAPI is a modern Python web framework for building REST APIs with automatic
OpenAPI documentation and high performance (comparable to Node.js).

WHY IT'S USED:
-------------
✓ Automatic API documentation (Swagger/OpenAPI)
✓ Type validation with Pydantic
✓ Async/await support (perfect for NATS)
✓ High performance (Starlette + Uvicorn)
✓ Easy CORS configuration
✓ Built-in Prometheus metrics support

HOW IT'S USED:
--------------
1. Endpoints Implemented:
   ├── PUT  /live/api/v1/deliveries/updatedelivery
   ├── POST /live/api/v1/deliveries/createdeliveries
   ├── POST /live/api/v2/partners/createriderlog
   ├── POST /live/api/v2/deliveries/createdeliverylog
   ├── POST /live/api/v2/partners/createbreaklog
   └── POST /live/api/v2/partners/updatebreaklog

2. Request Processing Flow:
   a. Client sends HTTP request to FastAPI endpoint
   b. FastAPI validates request body (Pydantic models)
   c. FastAPI publishes message to NATS JetStream:
      {
        "endpoint": "/live/api/v1/deliveries/updatedelivery",
        "method": "PUT",
        "data": {...request body...},
        "received_at": 1234567890,
        "original_path": "/live/api/v1/deliveries/updatedelivery"
      }
   d. Returns 200 OK with message_id

3. NATS Subject Mapping:
   Endpoint → Subject
   /live/api/v1/deliveries/updatedelivery → api.v1.deliveries.updatedelivery
   /live/api/v2/partners/createriderlog → api.v2.partners.createriderlog

4. Health Checks:
   - GET /health: Basic health check
   - GET /ready: Readiness probe (checks NATS connection)
   - GET /metrics: Prometheus metrics endpoint

5. CORS Configuration:
   - Allowed Origins: queue.workolik.com, localhost:3000, localhost:3001
   - Credentials: Enabled
   - Methods: All (*)

CONFIGURATION:
- File: scripts/app.py
- Environment Variables:
  * NATS_URL: nats://nats-server:4222
  * NATS_USER: admin
  * NATS_PASSWORD: package@321#
  * ALLOWED_ORIGINS: Comma-separated list

┌─────────────────────────────────────────────────────────────────────────┐
│ 3.3 PYTHON WORKER (Async Consumer)                                      │
└─────────────────────────────────────────────────────────────────────────┘

WHAT IT IS:
-----------
Python-based asynchronous worker that consumes messages from NATS JetStream
and forwards them to external APIs with retry logic and error handling.

WHY IT'S USED:
-------------
✓ Decouples API layer from external service calls
✓ Handles retries automatically (5 attempts with exponential backoff)
✓ Processes messages concurrently (configurable concurrency)
✓ Provides metrics for monitoring
✓ Graceful shutdown handling

HOW IT'S USED:
--------------
1. Message Consumption:
   - Subscribes to stream: "EVENTS"
   - Consumer: "worker_consumer"
   - Subject pattern: "api.>"
   - Pull-based (fetches batches of 10 messages)

2. Processing Flow:
   a. Worker pulls message from NATS
   b. Decodes JSON payload
   c. Extracts endpoint and HTTP method
   d. Maps internal endpoint to external URL:
      /live/api/v1/deliveries/updatedelivery → 
      https://jupiter.nearle.app/live/api/v1/deliveries/updatedelivery
   e. Forwards HTTP request with original method (PUT/POST)
   f. On success: ACK message (removes from queue)
   g. On failure: NAK message (requeues for retry)

3. Retry Logic:
   - Max Attempts: 5
   - Retry Delay: Exponential backoff (5s, 10s, 15s, 20s, 25s)
   - Retries on: 5xx errors, timeouts, network errors
   - No retry on: 4xx errors (client errors)

4. Concurrency Control:
   - Semaphore limits concurrent processing
   - Default: 10 concurrent messages
   - Configurable via WORKER_CONCURRENCY env var

5. Metrics Endpoint:
   - Port: 9090
   - Path: /metrics (Prometheus format)
   - Metrics:
     * worker_messages_processed_total (by status, endpoint)
     * worker_message_duration_seconds (by endpoint)
     * worker_messages_in_flight (gauge)
     * worker_queue_depth (gauge)

CONFIGURATION:
- File: scripts/worker.py
- Environment Variables:
  * NATS_URL: nats://nats-server:4222
  * NATS_STREAM: EVENTS
  * NATS_SUBJECT: api.>
  * NATS_CONSUMER: worker_consumer
  * EXTERNAL_BASE_URL: https://jupiter.nearle.app
  * WORKER_CONCURRENCY: 10
  * RETRY_ATTEMPTS: 5

┌─────────────────────────────────────────────────────────────────────────┐
│ 3.4 NGINX (Reverse Proxy)                                               │
└─────────────────────────────────────────────────────────────────────────┘

WHAT IT IS:
-----------
Nginx is a high-performance web server and reverse proxy used here to handle
CORS headers and proxy NATS monitoring endpoints.

WHY IT'S USED:
-------------
✓ CORS header management (required for browser-based dashboard)
✓ Lightweight (Alpine image ~5MB)
✓ High performance proxy
✓ Easy configuration

HOW IT'S USED:
--------------
1. Primary Function: CORS Proxy for NATS Monitoring
   - Proxies requests from NATS Dashboard to NATS monitoring endpoint
   - Adds CORS headers to allow cross-origin requests
   - Origin: https://natsadmin.workolik.com

2. Configuration (nginx-nats.conf):
   - Upstream: nats:8222 (NATS monitoring port)
   - Listens on: Port 80
   - CORS Headers Added:
     * Access-Control-Allow-Origin: https://natsadmin.workolik.com
     * Access-Control-Allow-Methods: GET, OPTIONS, HEAD
     * Access-Control-Allow-Headers: Content-Type, Authorization, Accept
     * Access-Control-Allow-Credentials: true

3. OPTIONS Preflight Handling:
   - Intercepts OPTIONS requests
   - Returns 204 No Content with CORS headers
   - Prevents duplicate headers from backend

4. Proxy Headers:
   - X-Real-IP: Client IP
   - X-Forwarded-For: Forwarded IP chain
   - X-Forwarded-Proto: Original protocol (http/https)

CONFIGURATION:
- File: nginx-nats.conf
- Docker Service: nats-proxy
- Port Mapping: 8082:80

┌─────────────────────────────────────────────────────────────────────────┐
│ 3.5 NATS DASHBOARD                                                      │
└─────────────────────────────────────────────────────────────────────────┘

WHAT IT IS:
-----------
Web-based monitoring interface for NATS server, providing real-time insights
into streams, consumers, and message statistics.

WHY IT'S USED:
-------------
✓ Visual monitoring of NATS streams and consumers
✓ Real-time metrics and statistics
✓ Message inspection capabilities
✓ Consumer lag monitoring
✓ Stream configuration management

HOW IT'S USED:
--------------
1. Configuration:
   - Image: mdawar/nats-dashboard:latest
   - NATS Connection: nats://admin:package%40321%23@nats:4222
   - Monitoring URL: http://148.135.137.174:8082 (via Nginx proxy)

2. Access:
   - Internal Port: 80
   - External: https://natsadmin.workolik.com (via Traefik)
   - Authentication: Basic Auth (Traefik middleware)
     * Username: admin
     * Password: package@321#
     * Hash: $2b$12$RcO6lQBCV9vht1i0GTmYV.l5j09WB0ps0WDGit4MxdtFJIUpvZmEi

3. Features Used:
   - Stream monitoring (EVENTS stream)
   - Consumer monitoring (worker_consumer)
   - Message rate statistics
   - Pending message counts

CONFIGURATION:
- File: docker-compose.yml (nats-dashboard service)
- Config File: dashboard-config.json (mounted as volume)

┌─────────────────────────────────────────────────────────────────────────┐
│ 3.6 DOCKER & DOCKER COMPOSE                                             │
└─────────────────────────────────────────────────────────────────────────┘

WHAT IT IS:
-----------
Docker provides containerization, and Docker Compose orchestrates multiple
containers for local development and VPS deployment.

WHY IT'S USED:
-------------
✓ Consistent environment across dev/staging/production
✓ Easy dependency management
✓ Isolated services
✓ Simple deployment for single-server setups
✓ Network isolation via Docker networks

HOW IT'S USED:
--------------
1. Services Defined:
   ├── nats: NATS server with JetStream
   ├── nats-proxy: Nginx CORS proxy
   └── nats-dashboard: Monitoring UI

2. Network Configuration:
   - Network: coolify (external, shared with Traefik)
   - All services communicate via service names

3. Volume Management:
   - nats-data: Persistent storage for NATS data
   - nats-js: JetStream storage

4. Health Checks:
   - NATS: HTTP health check on port 8222
   - Interval: 30s, Timeout: 10s, Retries: 3

CONFIGURATION:
- File: docker-compose.yml
- Traefik Labels: Configured for HTTPS routing

┌─────────────────────────────────────────────────────────────────────────┐
│ 3.7 KUBERNETES (k3s/k8s)                                               │
└─────────────────────────────────────────────────────────────────────────┘

WHAT IT IS:
-----------
Kubernetes is a container orchestration platform that manages containerized
applications across a cluster of nodes.

WHY IT'S USED:
-------------
✓ High availability (pods distributed across nodes)
✓ Auto-scaling (HPA - Horizontal Pod Autoscaler)
✓ Self-healing (restarts failed pods)
✓ Load balancing
✓ Rolling updates
✓ Resource management

HOW IT'S USED:
--------------
1. Deployments:
   ├── fastapi-backend: 4-20 replicas (HPA)
   └── nats-worker: 2-10 replicas (HPA)

2. Auto-Scaling (HPA):
   FastAPI:
   - Min Replicas: 4
   - Max Replicas: 20
   - CPU Target: 60%
   - Memory Target: 70%

   Worker:
   - Min Replicas: 2
   - Max Replicas: 10
   - CPU Target: 70%
   - Memory Target: 80%

3. Pod Anti-Affinity:
   - Ensures pods spread across different nodes
   - Prevents single point of failure

4. Services:
   - fastapi-backend: ClusterIP (internal)
   - fastapi-lb: LoadBalancer (port 8201)
   - Worker: No external service (internal only)

5. Probes:
   - Liveness: /health (FastAPI), /metrics (Worker)
   - Readiness: /ready (FastAPI), /metrics (Worker)

CONFIGURATION:
- Files: manifests/*.yaml
- Namespace: nats-backend

┌─────────────────────────────────────────────────────────────────────────┐
│ 3.8 TRAEFIK (Ingress Controller)                                       │
└─────────────────────────────────────────────────────────────────────────┘

WHAT IT IS:
-----------
Traefik is a modern reverse proxy and load balancer that automatically
discovers services via Docker labels or Kubernetes ingress.

WHY IT'S USED:
-------------
✓ Automatic TLS certificate management (Let's Encrypt)
✓ Service discovery via labels
✓ Dynamic configuration
✓ HTTP/HTTPS routing
✓ Middleware support (auth, rate limiting)

HOW IT'S USED:
--------------
1. Routing Rules:
   - queue.workolik.com → FastAPI LoadBalancer (port 8201)
   - natsadmin.workolik.com → NATS Dashboard (port 80)
   - nats.workolik.com → NATS Proxy (port 80)

2. TLS Configuration:
   - Entrypoint: https (port 443)
   - Certificate Resolver: letsencrypt
   - Automatic certificate renewal

3. Middleware:
   - Basic Auth for NATS Dashboard
   - CORS (handled by Nginx for NATS monitoring)

4. Network:
   - Uses coolify Docker network
   - Communicates with services via service names

CONFIGURATION:
- Docker Labels in docker-compose.yml
- Automatic discovery from container labels

┌─────────────────────────────────────────────────────────────────────────┐
│ 3.9 PROMETHEUS (Metrics)                                               │
└─────────────────────────────────────────────────────────────────────────┘

WHAT IT IS:
-----------
Prometheus is a monitoring and alerting toolkit that collects metrics via
HTTP endpoints.

WHY IT'S USED:
-------------
✓ Standard metrics format
✓ Time-series database
✓ Integration with Kubernetes HPA
✓ Rich query language (PromQL)
✓ Alerting capabilities

HOW IT'S USED:
--------------
1. FastAPI Metrics (/metrics):
   - http_requests_total: Request count by method, endpoint, status
   - http_request_duration_seconds: Request latency histogram

2. Worker Metrics (port 9090):
   - worker_messages_processed_total: Processed messages by status
   - worker_message_duration_seconds: Processing time
   - worker_messages_in_flight: Current processing count
   - worker_queue_depth: Queue depth estimate

3. Kubernetes Integration:
   - metrics-server scrapes pod metrics
   - HPA uses metrics for auto-scaling decisions

CONFIGURATION:
- FastAPI: Built-in prometheus_client
- Worker: Built-in prometheus_client
- Kubernetes: metrics-server component

================================================================================
4. ARCHITECTURE DIAGRAMS & FLOW
================================================================================

┌─────────────────────────────────────────────────────────────────────────┐
│ 4.1 HIGH-LEVEL ARCHITECTURE                                             │
└─────────────────────────────────────────────────────────────────────────┘

                    ┌─────────────────┐
                    │   Client App    │
                    │ (Mobile/Web)    │
                    └────────┬────────┘
                             │ HTTPS
                             │ queue.workolik.com:443
                             ▼
                    ┌─────────────────┐
                    │    Traefik      │
                    │ (TLS Terminate) │
                    └────────┬────────┘
                             │ HTTP
                             │ Port 8201
                             ▼
                    ┌─────────────────┐
                    │  k3s LoadBalancer│
                    │   (fastapi-lb)  │
                    └────────┬────────┘
                             │ ClusterIP
                             │ Port 8000
                             ▼
        ┌────────────────────┴────────────────────┐
        │                                         │
        ▼                                         ▼
┌───────────────┐                        ┌───────────────┐
│  FastAPI Pod  │                        │  FastAPI Pod  │
│   (Pod 1)     │                        │   (Pod 2)     │
└───────┬───────┘                        └───────┬───────┘
        │                                         │
        └───────────────┬─────────────────────────┘
                        │ NATS Publish
                        │ Subject: api.>
                        ▼
            ┌───────────────────────┐
            │   NATS JetStream      │
            │   Stream: EVENTS      │
            │   Subject: api.>      │
            └───────────┬───────────┘
                        │ Pull Messages
                        │ Consumer: worker_consumer
                        ▼
        ┌───────────────┴───────────────┐
        │                               │
        ▼                               ▼
┌───────────────┐              ┌───────────────┐
│  Worker Pod   │              │  Worker Pod   │
│   (Pod 1)     │              │   (Pod 2)     │
└───────┬───────┘              └───────┬───────┘
        │                               │
        └───────────────┬───────────────┘
                        │ HTTP Request
                        │ (PUT/POST)
                        ▼
            ┌───────────────────────┐
            │   External API        │
            │ jupiter.nearle.app    │
            └───────────────────────┘

┌─────────────────────────────────────────────────────────────────────────┐
│ 4.2 MESSAGE FLOW DETAIL                                                 │
└─────────────────────────────────────────────────────────────────────────┘

Step 1: CLIENT REQUEST
──────────────────────
Client → PUT /live/api/v1/deliveries/updatedelivery
         Body: {"delivery_id": 123, "status": "delivered"}

Step 2: TRAEFIK ROUTING
───────────────────────
Traefik receives HTTPS request
→ Terminates TLS
→ Routes to k3s LoadBalancer (port 8201)
→ LoadBalancer forwards to FastAPI Service (ClusterIP:8000)
→ Service load balances to available FastAPI pod

Step 3: FASTAPI PROCESSING
───────────────────────────
FastAPI receives request:
→ Validates JSON body (Pydantic)
→ Creates message payload:
   {
     "endpoint": "/live/api/v1/deliveries/updatedelivery",
     "method": "PUT",
     "data": {"delivery_id": 123, "status": "delivered"},
     "received_at": 1234567890,
     "original_path": "/live/api/v1/deliveries/updatedelivery"
   }
→ Publishes to NATS subject: "api.v1.deliveries.updatedelivery"
→ Returns 200 OK: {"status": "accepted", "message_id": 12345}

Step 4: NATS STORAGE
────────────────────
NATS JetStream:
→ Receives message on subject "api.v1.deliveries.updatedelivery"
→ Stores in stream "EVENTS"
→ Subject matches pattern "api.>"
→ Message available for consumer "worker_consumer"

Step 5: WORKER CONSUMPTION
──────────────────────────
Worker pod:
→ Pulls batch of messages (up to 10) from stream "EVENTS"
→ Processes message:
   * Decodes JSON payload
   * Extracts endpoint: "/live/api/v1/deliveries/updatedelivery"
   * Extracts method: "PUT"
   * Maps to external URL: "https://jupiter.nearle.app/live/api/v1/deliveries/updatedelivery"
→ Forwards HTTP PUT request to external API
→ Waits for response

Step 6: EXTERNAL API RESPONSE
─────────────────────────────
External API processes request:
→ Returns 200 OK or error

Step 7: WORKER ACKNOWLEDGMENT
─────────────────────────────
If success (200/201):
→ Worker sends ACK to NATS
→ Message removed from queue
→ Metrics updated: messages_processed_total{status="success"}

If failure (5xx/timeout):
→ Worker sends NAK to NATS
→ Message requeued for retry
→ Retry with exponential backoff (5s, 10s, 15s...)
→ Metrics updated: messages_processed_total{status="failed"}

┌─────────────────────────────────────────────────────────────────────────┐
│ 4.3 DOCKER COMPOSE ARCHITECTURE                                         │
└─────────────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│                    coolify Network                               │
│                                                                 │
│  ┌──────────────┐    ┌──────────────┐    ┌──────────────┐    │
│  │     NATS     │    │  nats-proxy  │    │ nats-dashboard│   │
│  │   :4222      │◄───│   (nginx)    │◄───│    :80        │    │
│  │   :8222      │    │    :80       │    │               │    │
│  │   :6222      │    │   :8082      │    │   :8081       │    │
│  └──────────────┘    └──────────────┘    └──────────────┘    │
│         ▲                                                   │
│         │                                                   │
│         └───────────────────────────────────────────────────┘
│                    (FastAPI & Workers connect here)          │
└─────────────────────────────────────────────────────────────────┘
                              │
                              │ Traefik Routes
                              ▼
                    ┌─────────────────────┐
                    │   External Access   │
                    │                     │
                    │ nats.workolik.com   │
                    │ natsadmin.workolik  │
                    └─────────────────────┘

┌─────────────────────────────────────────────────────────────────────────┐
│ 4.4 KUBERNETES ARCHITECTURE                                             │
└─────────────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────────┐
│                    Kubernetes Cluster (3 Nodes)                     │
│                                                                     │
│  Node 1          Node 2          Node 3                           │
│  ┌─────┐         ┌─────┐         ┌─────┐                         │
│  │FAPI │         │FAPI │         │FAPI │                          │
│  │Pod 1│         │Pod 2│         │Pod 3│                          │
│  └──┬──┘         └──┬──┘         └──┬──┘                          │
│     │               │               │                              │
│     └───────────────┼───────────────┘                              │
│                     │                                               │
│              ┌──────▼──────┐                                       │
│              │   Service   │                                       │
│              │ ClusterIP   │                                       │
│              │  :8000      │                                       │
│              └──────┬──────┘                                       │
│                     │                                               │
│              ┌──────▼──────┐                                       │
│              │ LoadBalancer│                                       │
│              │  :8201      │                                       │
│              └──────┬──────┘                                       │
│                     │                                               │
│  ┌──────────────────┼──────────────────┐                          │
│  │                  │                  │                          │
│  ▼                  ▼                  ▼                          │
│ ┌─────┐           ┌─────┐           ┌─────┐                      │
│ │Worker│           │Worker│           │Worker│                    │
│ │Pod 1 │           │Pod 2 │           │Pod 3 │                    │
│ └──┬──┘           └──┬──┘           └──┬──┘                      │
│    │                 │                  │                         │
│    └─────────────────┼──────────────────┘                         │
│                      │                                              │
│              ┌───────▼───────┐                                     │
│              │ External NATS │                                     │
│              │ nats.workolik │                                     │
│              │    .com:4222  │                                     │
│              └───────────────┘                                     │
└─────────────────────────────────────────────────────────────────────┘

================================================================================
5. DEPLOYMENT MODELS
================================================================================

┌─────────────────────────────────────────────────────────────────────────┐
│ 5.1 DOCKER COMPOSE DEPLOYMENT (VPS/Single Server)                       │
└─────────────────────────────────────────────────────────────────────────┘

USE CASE: Local development, small-scale production, single-server deployment

COMPONENTS:
- NATS Server (containerized)
- Nginx Proxy (CORS handling)
- NATS Dashboard (monitoring)

ADVANTAGES:
✓ Simple deployment (docker-compose up)
✓ No Kubernetes complexity
✓ Lower resource overhead
✓ Easy debugging

LIMITATIONS:
✗ No auto-scaling
✗ Single point of failure
✗ Manual scaling required

DEPLOYMENT STEPS:
1. docker-compose up -d
2. Services start automatically
3. Traefik discovers services via labels
4. TLS certificates auto-provisioned

┌─────────────────────────────────────────────────────────────────────────┐
│ 5.2 KUBERNETES DEPLOYMENT (Production Cluster)                         │
└─────────────────────────────────────────────────────────────────────────┘

USE CASE: Production environments, high availability, auto-scaling

COMPONENTS:
- FastAPI Deployment (4-20 pods, HPA)
- Worker Deployment (2-10 pods, HPA)
- Services (ClusterIP, LoadBalancer)
- HPA (Horizontal Pod Autoscaler)
- External NATS (separate server)

ADVANTAGES:
✓ High availability (pods across nodes)
✓ Auto-scaling based on metrics
✓ Self-healing (restarts failed pods)
✓ Rolling updates
✓ Resource limits and requests

DEPLOYMENT STEPS:
1. Build Docker images
2. Push to registry
3. Update manifests/secrets.yaml
4. kubectl apply -f manifests/
5. Verify pods: kubectl get pods -n nats-backend

================================================================================
6. INTEGRATION POINTS
================================================================================

┌─────────────────────────────────────────────────────────────────────────┐
│ 6.1 FASTAPI → NATS INTEGRATION                                          │
└─────────────────────────────────────────────────────────────────────────┘

LIBRARY: nats-py (Python NATS client)

CONNECTION:
- URL: nats://admin:package%40321%23@nats:4222
- Authentication: Username/Password
- Reconnection: Automatic (max 10 attempts)

PUBLISHING:
- JetStream context: nc.jetstream()
- Subject mapping: Endpoint → Subject pattern
- Payload: JSON-encoded message with metadata
- Acknowledgment: Synchronous (waits for ack)

ERROR HANDLING:
- Connection failures: Raises HTTPException 503
- Publish failures: Raises HTTPException 500
- Retry logic: Handled by NATS client

┌─────────────────────────────────────────────────────────────────────────┐
│ 6.2 WORKER → NATS INTEGRATION                                           │
└─────────────────────────────────────────────────────────────────────────┘

CONSUMPTION:
- Pull subscription: js.pull_subscribe()
- Batch fetch: fetch(10, timeout=5)
- Consumer: "worker_consumer"
- Stream: "EVENTS"

MESSAGE PROCESSING:
- Decode: json.loads(msg.data.decode())
- Extract: endpoint, method, data
- Process: Forward to external API
- Acknowledge: msg.ack() or msg.nak()

ERROR HANDLING:
- Invalid JSON: msg.term() (terminate, no retry)
- Processing error: msg.nak() (requeue)
- External API error: Retry logic in worker

┌─────────────────────────────────────────────────────────────────────────┐
│ 6.3 NGINX → NATS MONITORING INTEGRATION                                 │
└─────────────────────────────────────────────────────────────────────────┘

PROXY CONFIGURATION:
- Upstream: nats:8222
- CORS Headers: Added for dashboard access
- OPTIONS Handling: Preflight requests

WHY NEEDED:
- NATS Dashboard runs in browser (JavaScript)
- Browser enforces CORS policy
- NATS monitoring API doesn't include CORS headers
- Nginx adds required headers

┌─────────────────────────────────────────────────────────────────────────┐
│ 6.4 TRAEFIK → SERVICES INTEGRATION                                      │
└─────────────────────────────────────────────────────────────────────────┘

SERVICE DISCOVERY:
- Docker Labels: Automatic discovery
- Routing Rules: Host-based (queue.workolik.com)
- TLS: Automatic Let's Encrypt certificates

MIDDLEWARE:
- Basic Auth: For NATS Dashboard
- TLS Termination: HTTPS → HTTP

================================================================================
7. SECURITY & AUTHENTICATION
================================================================================

┌─────────────────────────────────────────────────────────────────────────┐
│ 7.1 NATS AUTHENTICATION                                                 │
└─────────────────────────────────────────────────────────────────────────┘

METHOD: Username/Password Authentication

CREDENTIALS:
- Username: admin
- Password: package@321#
- URL Encoding: package%40321%23 (@ = %40, # = %23)

CONFIGURATION:
- Set in NATS server config (nats.conf)
- Required for all client connections
- FastAPI and Workers authenticate on startup

┌─────────────────────────────────────────────────────────────────────────┐
│ 7.2 TRAEFIK BASIC AUTH                                                  │
└─────────────────────────────────────────────────────────────────────────┘

PURPOSE: Protect NATS Dashboard from unauthorized access

METHOD: HTTP Basic Authentication

CREDENTIALS:
- Username: admin
- Password: package@321#
- Hash Algorithm: bcrypt ($2b$12$...)

GENERATION:
- Script: setup-auth.sh
- Tool: htpasswd (Apache HTTP Server)
- Command: htpasswd -nbB admin "package@321#"

CONFIGURATION:
- Traefik Middleware: nats-auth
- Applied to: nats-ui router
- Hash Format: admin:$2b$12$RcO6lQBCV9vht1i0GTmYV.l5j09WB0ps0WDGit4MxdtFJIUpvZmEi

┌─────────────────────────────────────────────────────────────────────────┐
│ 7.3 TLS/HTTPS                                                            │
└─────────────────────────────────────────────────────────────────────────┘

PROVIDER: Let's Encrypt (via Traefik)

DOMAINS:
- queue.workolik.com (FastAPI)
- natsadmin.workolik.com (Dashboard)
- nats.workolik.com (Monitoring Proxy)

CERTIFICATE RESOLVER: letsencrypt

AUTOMATIC RENEWAL: Traefik handles certificate renewal

┌─────────────────────────────────────────────────────────────────────────┐
│ 7.4 CORS CONFIGURATION                                                   │
└─────────────────────────────────────────────────────────────────────────┘

PURPOSE: Allow browser-based dashboard to access NATS monitoring API

CONFIGURATION:
- Allowed Origin: https://natsadmin.workolik.com
- Methods: GET, OPTIONS, HEAD
- Credentials: true
- Headers: Content-Type, Authorization, Accept

IMPLEMENTATION:
- Nginx adds CORS headers
- OPTIONS preflight handled separately
- Headers hidden from backend to prevent duplicates

================================================================================
8. MONITORING & OBSERVABILITY
================================================================================

┌─────────────────────────────────────────────────────────────────────────┐
│ 8.1 PROMETHEUS METRICS                                                  │
└─────────────────────────────────────────────────────────────────────────┘

FASTAPI METRICS (/metrics):
- http_requests_total: Counter
  * Labels: method, endpoint, status
  * Tracks: Total requests by type and result

- http_request_duration_seconds: Histogram
  * Labels: method, endpoint
  * Tracks: Request latency distribution

WORKER METRICS (port 9090):
- worker_messages_processed_total: Counter
  * Labels: status (success/failed/invalid/error), endpoint
  * Tracks: Total messages processed

- worker_message_duration_seconds: Histogram
  * Labels: endpoint
  * Tracks: Message processing time

- worker_messages_in_flight: Gauge
  * Tracks: Currently processing messages

- worker_queue_depth: Gauge
  * Tracks: Approximate queue depth

┌─────────────────────────────────────────────────────────────────────────┐
│ 8.2 NATS DASHBOARD                                                      │
└─────────────────────────────────────────────────────────────────────────┘

ACCESS: https://natsadmin.workolik.com

FEATURES:
- Stream Statistics (EVENTS stream)
- Consumer Status (worker_consumer)
- Message Rates (messages/second)
- Pending Messages Count
- Consumer Lag Monitoring

DATA SOURCE: NATS Monitoring API (via Nginx proxy)

┌─────────────────────────────────────────────────────────────────────────┐
│ 8.3 KUBERNETES HEALTH CHECKS                                            │
└─────────────────────────────────────────────────────────────────────────┘

LIVENESS PROBES:
- FastAPI: GET /health (every 30s)
- Worker: GET /metrics (every 30s)

READINESS PROBES:
- FastAPI: GET /ready (checks NATS connection)
- Worker: GET /metrics

PURPOSE:
- Liveness: Restart pod if unhealthy
- Readiness: Remove from service if not ready

================================================================================
9. SCALABILITY & PERFORMANCE
================================================================================

┌─────────────────────────────────────────────────────────────────────────┐
│ 9.1 HORIZONTAL SCALING (Kubernetes HPA)                                 │
└─────────────────────────────────────────────────────────────────────────┘

FASTAPI AUTO-SCALING:
- Min Replicas: 4
- Max Replicas: 20
- CPU Target: 60%
- Memory Target: 70%
- Scale Up: When CPU > 60% or Memory > 70%
- Scale Down: When below thresholds

WORKER AUTO-SCALING:
- Min Replicas: 2
- Max Replicas: 10
- CPU Target: 70%
- Memory Target: 80%
- Scale Up: When CPU > 70% or Memory > 80%
- Scale Down: When below thresholds

METRICS SOURCE: Kubernetes metrics-server

┌─────────────────────────────────────────────────────────────────────────┐
│ 9.2 CONCURRENCY CONTROL                                                  │
└─────────────────────────────────────────────────────────────────────────┘

WORKER CONCURRENCY:
- Semaphore Limit: 10 (configurable)
- Prevents: Overwhelming external API
- Configurable: WORKER_CONCURRENCY env var

MESSAGE BATCHING:
- Batch Size: 10 messages per fetch
- Timeout: 5 seconds
- Benefit: Reduces NATS round trips

┌─────────────────────────────────────────────────────────────────────────┐
│ 9.3 PERFORMANCE CHARACTERISTICS                                         │
└─────────────────────────────────────────────────────────────────────────┘

NATS PERFORMANCE:
- Latency: < 1ms (local network)
- Throughput: Millions of messages/second
- Persistence: File-based (JetStream)

FASTAPI PERFORMANCE:
- Framework: Starlette (async)
- Server: Uvicorn (ASGI)
- Concurrency: Async/await (non-blocking)

WORKER PERFORMANCE:
- Library: aiohttp (async HTTP)
- Concurrency: Configurable semaphore
- Retry Logic: Exponential backoff

┌─────────────────────────────────────────────────────────────────────────┐
│ 9.4 RESOURCE MANAGEMENT                                                 │
└─────────────────────────────────────────────────────────────────────────┘

KUBERNETES RESOURCES:
FastAPI:
- Requests: CPU 100m, Memory 256Mi
- Limits: CPU 500m, Memory 512Mi

Worker:
- Requests: CPU 100m, Memory 256Mi
- Limits: CPU 500m, Memory 512Mi

POD ANTI-AFFINITY:
- Ensures pods spread across nodes
- Prevents resource contention
- Improves availability

================================================================================
10. TROUBLESHOOTING GUIDE
================================================================================

┌─────────────────────────────────────────────────────────────────────────┐
│ 10.1 COMMON ISSUES                                                      │
└─────────────────────────────────────────────────────────────────────────┘

ISSUE: FastAPI cannot connect to NATS
──────────────────────────────────────
SYMPTOMS: 503 errors, "NATS not connected"
SOLUTION:
1. Check NATS service is running: docker ps | grep nats
2. Verify NATS_URL environment variable
3. Check network connectivity: ping nats (from FastAPI container)
4. Verify credentials: NATS_USER, NATS_PASSWORD
5. Check NATS logs: docker logs nats

ISSUE: Worker not processing messages
──────────────────────────────────────
SYMPTOMS: Messages accumulating in queue, no worker logs
SOLUTION:
1. Check worker pods: kubectl get pods -n nats-backend
2. View worker logs: kubectl logs -f deployment/nats-worker
3. Verify NATS connection: Check worker startup logs
4. Check consumer exists: NATS Dashboard → Consumers
5. Verify stream configuration: NATS Dashboard → Streams

ISSUE: CORS errors in browser
──────────────────────────────
SYMPTOMS: Browser console shows CORS errors
SOLUTION:
1. Verify Nginx proxy is running: docker ps | grep nats-proxy
2. Check nginx-nats.conf CORS headers
3. Verify NATS_MONITORING_URL in dashboard config
4. Check browser console for specific CORS error
5. Verify allowed origin matches dashboard URL

ISSUE: Messages not being forwarded to external API
────────────────────────────────────────────────────
SYMPTOMS: Messages processed but external API not receiving
SOLUTION:
1. Check worker logs for HTTP errors
2. Verify EXTERNAL_BASE_URL environment variable
3. Test external API connectivity: curl from worker pod
4. Check API key if required: EXTERNAL_ENDPOINT_API_KEY
5. Verify endpoint mapping in worker.py

ISSUE: Auto-scaling not working
─────────────────────────────────
SYMPTOMS: Pods not scaling despite high load
SOLUTION:
1. Check HPA status: kubectl get hpa -n nats-backend
2. Verify metrics-server: kubectl top pods
3. Check HPA events: kubectl describe hpa -n nats-backend
4. Verify resource requests/limits in deployment
5. Check if max replicas reached

┌─────────────────────────────────────────────────────────────────────────┐
│ 10.2 DEBUGGING COMMANDS                                                 │
└─────────────────────────────────────────────────────────────────────────┘

DOCKER COMPOSE:
- View logs: docker-compose logs -f [service]
- Restart service: docker-compose restart [service]
- Check status: docker-compose ps
- Shell access: docker-compose exec [service] sh

KUBERNETES:
- Pod status: kubectl get pods -n nats-backend -o wide
- Pod logs: kubectl logs -f [pod-name] -n nats-backend
- Describe pod: kubectl describe pod [pod-name] -n nats-backend
- Exec into pod: kubectl exec -it [pod-name] -n nats-backend -- sh
- Service endpoints: kubectl get endpoints -n nats-backend
- HPA status: kubectl get hpa -n nats-backend
- Events: kubectl get events -n nats-backend --sort-by='.lastTimestamp'

NATS:
- Connect via CLI: nats -s nats://admin:package@321#@nats:4222
- Stream info: nats stream info EVENTS
- Consumer info: nats consumer info EVENTS worker_consumer
- Pending messages: nats consumer next EVENTS worker_consumer

┌─────────────────────────────────────────────────────────────────────────┐
│ 10.3 LOG ANALYSIS                                                       │
└─────────────────────────────────────────────────────────────────────────┘

FASTAPI LOGS:
- Look for: "✅ Connected to NATS JetStream"
- Error patterns: "❌ Failed to connect", "❌ Failed to publish"
- Success patterns: "✅ Message published", status 200

WORKER LOGS:
- Look for: "✅ Connected to NATS JetStream", "✅ Subscribed"
- Processing: "📨 Processing message for endpoint"
- Success: "✅ Message processed and forwarded successfully"
- Errors: "❌ Failed to forward", "⚠️ Error forwarding"

NATS LOGS:
- Connection: "[INF] Client connection"
- JetStream: "[INF] JetStream enabled"
- Errors: "[ERR]" prefix

================================================================================
11. CONFIGURATION REFERENCE
================================================================================

┌─────────────────────────────────────────────────────────────────────────┐
│ 11.1 ENVIRONMENT VARIABLES                                              │
└─────────────────────────────────────────────────────────────────────────┘

FASTAPI (app.py):
- NATS_URL: nats://nats-server:4222
- NATS_USER: admin
- NATS_PASSWORD: package@321#
- ALLOWED_ORIGINS: http://localhost:3001,http://localhost:3000,https://queue.workolik.com

WORKER (worker.py):
- NATS_URL: nats://nats-server:4222
- NATS_USER: admin
- NATS_PASSWORD: package@321#
- NATS_STREAM: EVENTS
- NATS_SUBJECT: api.>
- NATS_CONSUMER: worker_consumer
- EXTERNAL_BASE_URL: https://jupiter.nearle.app
- WORKER_CONCURRENCY: 10
- RETRY_ATTEMPTS: 5
- RETRY_DELAY_SECONDS: 5
- EXTERNAL_ENDPOINT_API_KEY: (optional)

NATS DASHBOARD:
- NATS_URL: nats://admin:package%40321%23@nats:4222
- NATS_MONITORING_URL: http://148.135.137.174:8082

┌─────────────────────────────────────────────────────────────────────────┐
│ 11.2 PORT MAPPINGS                                                      │
└─────────────────────────────────────────────────────────────────────────┘

NATS:
- 4222: Client connections
- 8222: HTTP monitoring
- 6222: Cluster routing

FASTAPI:
- 8000: HTTP API (internal)
- 8201: LoadBalancer (external via Traefik)

WORKER:
- 9090: Prometheus metrics

NGINX PROXY:
- 80: Internal (container)
- 8082: External (Docker Compose)

NATS DASHBOARD:
- 80: Internal (container)
- 8081: External (Docker Compose)

KUBERNETES:
- 30826: Dashboard proxy NodePort

┌─────────────────────────────────────────────────────────────────────────┐
│ 11.3 ENDPOINT MAPPINGS                                                  │
└─────────────────────────────────────────────────────────────────────────┘

FASTAPI → NATS SUBJECT:
/live/api/v1/deliveries/createdeliveries → api.v1.deliveries.createdeliveries
/live/api/v1/deliveries/updatedelivery → api.v1.deliveries.updatedelivery
/live/api/v2/partners/createriderlog → api.v2.partners.createriderlog
/live/api/v2/deliveries/createdeliverylog → api.v2.deliveries.createdeliverylog
/live/api/v2/partners/createbreaklog → api.v2.partners.createbreaklog
/live/api/v2/partners/updatebreaklog → api.v2.partners.updatebreaklog

INTERNAL → EXTERNAL API:
/live/api/v1/deliveries/updatedelivery → https://jupiter.nearle.app/live/api/v1/deliveries/updatedelivery
/live/api/v2/partners/createriderlog → https://jupiter.nearle.app/live/api/v2/partners/createriderlog
... (same pattern for all endpoints)

================================================================================
12. FILE STRUCTURE REFERENCE
================================================================================

kubernetes/
├── docker-compose.yml              # Docker Compose deployment
├── Dockerfile                      # Multi-stage build (api + worker)
├── requirements.txt                # Python dependencies
├── nginx-nats.conf                 # Nginx CORS proxy config
├── nginx-k8s-dashboard.conf        # Kubernetes dashboard proxy
├── setup-auth.sh                   # Traefik auth hash generator
├── deploy.sh                       # Kubernetes deployment script
├── build-and-deploy.sh             # Build + deploy script
├── scripts/
│   ├── app.py                      # FastAPI application
│   └── worker.py                   # NATS worker consumer
└── manifests/
    ├── namespace.yaml               # Kubernetes namespace
    ├── fastapi-deployment.yaml     # FastAPI pods
    ├── fastapi-service.yaml        # FastAPI ClusterIP service
    ├── fastapi-loadbalancer.yaml   # FastAPI LoadBalancer
    ├── fastapi-hpa.yaml            # FastAPI auto-scaling
    ├── worker-deployment.yaml       # Worker pods
    ├── worker-hpa.yaml             # Worker auto-scaling
    ├── secrets.yaml                 # Environment variables
    ├── gateway.yaml                 # Gateway API (optional)
    ├── dashboard.yaml               # NATS Dashboard
    ├── dashboard-proxy-deployment.yaml  # Dashboard proxy
    └── dashboard-loadbalancer.yaml # Dashboard LoadBalancer

================================================================================
13. KEY DESIGN DECISIONS
================================================================================

┌─────────────────────────────────────────────────────────────────────────┐
│ 13.1 WHY NATS JETSTREAM?                                                │
└─────────────────────────────────────────────────────────────────────────┘

✓ Lightweight: Single binary, no external dependencies
✓ Performance: Sub-millisecond latency, high throughput
✓ Persistence: JetStream provides message durability
✓ Simplicity: Easy to deploy and configure
✓ Cloud-Native: Designed for microservices
✓ At-Least-Once: Guaranteed delivery with acknowledgments

ALTERNATIVES CONSIDERED:
- RabbitMQ: More complex, heavier resource usage
- Kafka: Overkill for this use case, complex setup
- Redis Streams: Less mature, limited features
- AWS SQS: Vendor lock-in, additional cost

┌─────────────────────────────────────────────────────────────────────────┐
│ 13.2 WHY FASTAPI?                                                       │
└─────────────────────────────────────────────────────────────────────────┘

✓ Async Support: Native async/await for NATS integration
✓ Performance: Comparable to Node.js, faster than Flask
✓ Type Safety: Pydantic validation
✓ Documentation: Automatic OpenAPI/Swagger
✓ Modern: Built on Starlette and Pydantic
✓ Easy: Simple to learn and use

┌─────────────────────────────────────────────────────────────────────────┐
│ 13.3 WHY WORKER PATTERN?                                                │
└─────────────────────────────────────────────────────────────────────────┘

✓ Decoupling: API layer independent of external service availability
✓ Resilience: Retry logic handles transient failures
✓ Scalability: Workers scale independently
✓ Observability: Separate metrics and logs
✓ Flexibility: Easy to add new endpoints

┌─────────────────────────────────────────────────────────────────────────┐
│ 13.4 WHY DUAL DEPLOYMENT (Docker Compose + Kubernetes)?                │
└─────────────────────────────────────────────────────────────────────────┘

✓ Flexibility: Docker Compose for simple deployments, K8s for production
✓ Development: Easy local testing with Docker Compose
✓ Production: Kubernetes for high availability and auto-scaling
✓ Migration Path: Can start simple and scale up

================================================================================
END OF DOCUMENTATION
================================================================================

This document provides a comprehensive overview of the NATS-based message queue
architecture. For specific implementation details, refer to the source code
files listed in Section 12.

For questions or issues, refer to Section 10 (Troubleshooting Guide) or
consult the individual component documentation.

Last Updated: 2025-01-XX
Version: 1.0

