1351 lines
64 KiB
Plaintext
1351 lines
64 KiB
Plaintext
================================================================================
|
|
NATS-BASED MESSAGE QUEUE ARCHITECTURE
|
|
Comprehensive Technical Documentation
|
|
================================================================================
|
|
|
|
TABLE OF CONTENTS
|
|
-----------------
|
|
1. Executive Summary
|
|
2. Technology Stack Overview
|
|
3. Detailed Component Analysis
|
|
4. Architecture Diagrams & Flow
|
|
5. Deployment Models
|
|
6. Integration Points
|
|
7. Security & Authentication
|
|
8. Monitoring & Observability
|
|
9. Scalability & Performance
|
|
10. Troubleshooting Guide
|
|
|
|
================================================================================
|
|
1. EXECUTIVE SUMMARY
|
|
================================================================================
|
|
|
|
This system implements a high-performance, scalable message queue architecture
|
|
using NATS JetStream as the core messaging backbone. The architecture follows
|
|
a producer-consumer pattern where FastAPI endpoints receive HTTP requests,
|
|
publish messages to NATS, and worker processes consume and forward these
|
|
messages to external APIs.
|
|
|
|
KEY CHARACTERISTICS:
|
|
- Asynchronous message processing with guaranteed delivery
|
|
- Horizontal scalability (auto-scaling FastAPI and Worker pods)
|
|
- Multi-deployment support (Docker Compose for VPS, Kubernetes for clusters)
|
|
- High availability with distributed pod placement
|
|
- Real-time monitoring and metrics collection
|
|
- Secure authentication and CORS handling
|
|
|
|
================================================================================
|
|
2. TECHNOLOGY STACK OVERVIEW
|
|
================================================================================
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ CORE TECHNOLOGIES │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
|
|
1. NATS (NATS Server + JetStream)
|
|
- Purpose: Message broker and streaming platform
|
|
- Version: Latest stable
|
|
- Ports: 4222 (client), 8222 (monitoring), 6222 (clustering)
|
|
|
|
2. FastAPI (Python Web Framework)
|
|
- Purpose: REST API server that receives HTTP requests
|
|
- Version: Latest (Python 3.11)
|
|
- Port: 8000
|
|
|
|
3. Python Workers (Async Consumers)
|
|
- Purpose: Background workers that process messages from NATS
|
|
- Port: 9090 (metrics)
|
|
|
|
4. Nginx (Reverse Proxy)
|
|
- Purpose: CORS handling and proxy for NATS monitoring endpoints
|
|
- Version: Alpine (lightweight)
|
|
|
|
5. NATS Dashboard (Monitoring UI)
|
|
- Purpose: Web-based monitoring interface for NATS
|
|
- Port: 80 (internal), exposed via Traefik
|
|
|
|
6. Docker & Docker Compose
|
|
- Purpose: Containerization and local/VPS deployment
|
|
|
|
7. Kubernetes (k3s/k8s)
|
|
- Purpose: Container orchestration for production clusters
|
|
|
|
8. Traefik (Ingress Controller)
|
|
- Purpose: TLS termination and routing
|
|
|
|
9. Prometheus (Metrics)
|
|
- Purpose: Metrics collection and monitoring
|
|
|
|
================================================================================
|
|
3. DETAILED COMPONENT ANALYSIS
|
|
================================================================================
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ 3.1 NATS SERVER │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
|
|
WHAT IT IS:
|
|
-----------
|
|
NATS is a lightweight, high-performance messaging system designed for cloud-
|
|
native applications. JetStream adds persistence and streaming capabilities.
|
|
|
|
WHY IT'S USED:
|
|
-------------
|
|
✓ Ultra-low latency (< 1ms message delivery)
|
|
✓ High throughput (millions of messages per second)
|
|
✓ Built-in persistence with JetStream
|
|
✓ At-least-once delivery guarantees
|
|
✓ Simple pub/sub model
|
|
✓ No external dependencies (single binary)
|
|
✓ Perfect for microservices communication
|
|
|
|
HOW IT'S USED:
|
|
--------------
|
|
1. Stream Configuration:
|
|
- Stream Name: "EVENTS"
|
|
- Subject Pattern: "api.>"
|
|
- Storage: File-based persistence
|
|
- Retention: Work queue policy
|
|
|
|
2. Consumer Configuration:
|
|
- Consumer Name: "worker_consumer"
|
|
- Pull-based subscription
|
|
- Acknowledgment required (ACK/NAK)
|
|
- Max pending: 10 messages per worker
|
|
|
|
3. Message Flow:
|
|
FastAPI → Publishes to subject "api.v1.deliveries.updatedelivery"
|
|
Worker → Pulls messages from stream "EVENTS" via consumer
|
|
|
|
4. Authentication:
|
|
- Username: admin
|
|
- Password: package@321#
|
|
- Connection URL: nats://admin:package%40321%23@nats:4222
|
|
|
|
CONFIGURATION FILES:
|
|
- docker-compose.yml: NATS service definition
|
|
- nats.conf: NATS server configuration (referenced in compose)
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ 3.2 FASTAPI APPLICATION │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
|
|
WHAT IT IS:
|
|
-----------
|
|
FastAPI is a modern Python web framework for building REST APIs with automatic
|
|
OpenAPI documentation and high performance (comparable to Node.js).
|
|
|
|
WHY IT'S USED:
|
|
-------------
|
|
✓ Automatic API documentation (Swagger/OpenAPI)
|
|
✓ Type validation with Pydantic
|
|
✓ Async/await support (perfect for NATS)
|
|
✓ High performance (Starlette + Uvicorn)
|
|
✓ Easy CORS configuration
|
|
✓ Built-in Prometheus metrics support
|
|
|
|
HOW IT'S USED:
|
|
--------------
|
|
1. Endpoints Implemented:
|
|
├── PUT /live/api/v1/deliveries/updatedelivery
|
|
├── POST /live/api/v1/deliveries/createdeliveries
|
|
├── POST /live/api/v2/partners/createriderlog
|
|
├── POST /live/api/v2/deliveries/createdeliverylog
|
|
├── POST /live/api/v2/partners/createbreaklog
|
|
└── POST /live/api/v2/partners/updatebreaklog
|
|
|
|
2. Request Processing Flow:
|
|
a. Client sends HTTP request to FastAPI endpoint
|
|
b. FastAPI validates request body (Pydantic models)
|
|
c. FastAPI publishes message to NATS JetStream:
|
|
{
|
|
"endpoint": "/live/api/v1/deliveries/updatedelivery",
|
|
"method": "PUT",
|
|
"data": {...request body...},
|
|
"received_at": 1234567890,
|
|
"original_path": "/live/api/v1/deliveries/updatedelivery"
|
|
}
|
|
d. Returns 200 OK with message_id
|
|
|
|
3. NATS Subject Mapping:
|
|
Endpoint → Subject
|
|
/live/api/v1/deliveries/updatedelivery → api.v1.deliveries.updatedelivery
|
|
/live/api/v2/partners/createriderlog → api.v2.partners.createriderlog
|
|
|
|
4. Health Checks:
|
|
- GET /health: Basic health check
|
|
- GET /ready: Readiness probe (checks NATS connection)
|
|
- GET /metrics: Prometheus metrics endpoint
|
|
|
|
5. CORS Configuration:
|
|
- Allowed Origins: queue.workolik.com, localhost:3000, localhost:3001
|
|
- Credentials: Enabled
|
|
- Methods: All (*)
|
|
|
|
CONFIGURATION:
|
|
- File: scripts/app.py
|
|
- Environment Variables:
|
|
* NATS_URL: nats://nats-server:4222
|
|
* NATS_USER: admin
|
|
* NATS_PASSWORD: package@321#
|
|
* ALLOWED_ORIGINS: Comma-separated list
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ 3.3 PYTHON WORKER (Async Consumer) │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
|
|
WHAT IT IS:
|
|
-----------
|
|
Python-based asynchronous worker that consumes messages from NATS JetStream
|
|
and forwards them to external APIs with retry logic and error handling.
|
|
|
|
WHY IT'S USED:
|
|
-------------
|
|
✓ Decouples API layer from external service calls
|
|
✓ Handles retries automatically (5 attempts with exponential backoff)
|
|
✓ Processes messages concurrently (configurable concurrency)
|
|
✓ Provides metrics for monitoring
|
|
✓ Graceful shutdown handling
|
|
|
|
HOW IT'S USED:
|
|
--------------
|
|
1. Message Consumption:
|
|
- Subscribes to stream: "EVENTS"
|
|
- Consumer: "worker_consumer"
|
|
- Subject pattern: "api.>"
|
|
- Pull-based (fetches batches of 10 messages)
|
|
|
|
2. Processing Flow:
|
|
a. Worker pulls message from NATS
|
|
b. Decodes JSON payload
|
|
c. Extracts endpoint and HTTP method
|
|
d. Maps internal endpoint to external URL:
|
|
/live/api/v1/deliveries/updatedelivery →
|
|
https://jupiter.nearle.app/live/api/v1/deliveries/updatedelivery
|
|
e. Forwards HTTP request with original method (PUT/POST)
|
|
f. On success: ACK message (removes from queue)
|
|
g. On failure: NAK message (requeues for retry)
|
|
|
|
3. Retry Logic:
|
|
- Max Attempts: 5
|
|
- Retry Delay: Exponential backoff (5s, 10s, 15s, 20s, 25s)
|
|
- Retries on: 5xx errors, timeouts, network errors
|
|
- No retry on: 4xx errors (client errors)
|
|
|
|
4. Concurrency Control:
|
|
- Semaphore limits concurrent processing
|
|
- Default: 10 concurrent messages
|
|
- Configurable via WORKER_CONCURRENCY env var
|
|
|
|
5. Metrics Endpoint:
|
|
- Port: 9090
|
|
- Path: /metrics (Prometheus format)
|
|
- Metrics:
|
|
* worker_messages_processed_total (by status, endpoint)
|
|
* worker_message_duration_seconds (by endpoint)
|
|
* worker_messages_in_flight (gauge)
|
|
* worker_queue_depth (gauge)
|
|
|
|
CONFIGURATION:
|
|
- File: scripts/worker.py
|
|
- Environment Variables:
|
|
* NATS_URL: nats://nats-server:4222
|
|
* NATS_STREAM: EVENTS
|
|
* NATS_SUBJECT: api.>
|
|
* NATS_CONSUMER: worker_consumer
|
|
* EXTERNAL_BASE_URL: https://jupiter.nearle.app
|
|
* WORKER_CONCURRENCY: 10
|
|
* RETRY_ATTEMPTS: 5
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ 3.4 NGINX (Reverse Proxy) │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
|
|
WHAT IT IS:
|
|
-----------
|
|
Nginx is a high-performance web server and reverse proxy used here to handle
|
|
CORS headers and proxy NATS monitoring endpoints.
|
|
|
|
WHY IT'S USED:
|
|
-------------
|
|
✓ CORS header management (required for browser-based dashboard)
|
|
✓ Lightweight (Alpine image ~5MB)
|
|
✓ High performance proxy
|
|
✓ Easy configuration
|
|
|
|
HOW IT'S USED:
|
|
--------------
|
|
1. Primary Function: CORS Proxy for NATS Monitoring
|
|
- Proxies requests from NATS Dashboard to NATS monitoring endpoint
|
|
- Adds CORS headers to allow cross-origin requests
|
|
- Origin: https://natsadmin.workolik.com
|
|
|
|
2. Configuration (nginx-nats.conf):
|
|
- Upstream: nats:8222 (NATS monitoring port)
|
|
- Listens on: Port 80
|
|
- CORS Headers Added:
|
|
* Access-Control-Allow-Origin: https://natsadmin.workolik.com
|
|
* Access-Control-Allow-Methods: GET, OPTIONS, HEAD
|
|
* Access-Control-Allow-Headers: Content-Type, Authorization, Accept
|
|
* Access-Control-Allow-Credentials: true
|
|
|
|
3. OPTIONS Preflight Handling:
|
|
- Intercepts OPTIONS requests
|
|
- Returns 204 No Content with CORS headers
|
|
- Prevents duplicate headers from backend
|
|
|
|
4. Proxy Headers:
|
|
- X-Real-IP: Client IP
|
|
- X-Forwarded-For: Forwarded IP chain
|
|
- X-Forwarded-Proto: Original protocol (http/https)
|
|
|
|
CONFIGURATION:
|
|
- File: nginx-nats.conf
|
|
- Docker Service: nats-proxy
|
|
- Port Mapping: 8082:80
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ 3.5 NATS DASHBOARD │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
|
|
WHAT IT IS:
|
|
-----------
|
|
Web-based monitoring interface for NATS server, providing real-time insights
|
|
into streams, consumers, and message statistics.
|
|
|
|
WHY IT'S USED:
|
|
-------------
|
|
✓ Visual monitoring of NATS streams and consumers
|
|
✓ Real-time metrics and statistics
|
|
✓ Message inspection capabilities
|
|
✓ Consumer lag monitoring
|
|
✓ Stream configuration management
|
|
|
|
HOW IT'S USED:
|
|
--------------
|
|
1. Configuration:
|
|
- Image: mdawar/nats-dashboard:latest
|
|
- NATS Connection: nats://admin:package%40321%23@nats:4222
|
|
- Monitoring URL: http://148.135.137.174:8082 (via Nginx proxy)
|
|
|
|
2. Access:
|
|
- Internal Port: 80
|
|
- External: https://natsadmin.workolik.com (via Traefik)
|
|
- Authentication: Basic Auth (Traefik middleware)
|
|
* Username: admin
|
|
* Password: package@321#
|
|
* Hash: $2b$12$RcO6lQBCV9vht1i0GTmYV.l5j09WB0ps0WDGit4MxdtFJIUpvZmEi
|
|
|
|
3. Features Used:
|
|
- Stream monitoring (EVENTS stream)
|
|
- Consumer monitoring (worker_consumer)
|
|
- Message rate statistics
|
|
- Pending message counts
|
|
|
|
CONFIGURATION:
|
|
- File: docker-compose.yml (nats-dashboard service)
|
|
- Config File: dashboard-config.json (mounted as volume)
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ 3.6 DOCKER & DOCKER COMPOSE │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
|
|
WHAT IT IS:
|
|
-----------
|
|
Docker provides containerization, and Docker Compose orchestrates multiple
|
|
containers for local development and VPS deployment.
|
|
|
|
WHY IT'S USED:
|
|
-------------
|
|
✓ Consistent environment across dev/staging/production
|
|
✓ Easy dependency management
|
|
✓ Isolated services
|
|
✓ Simple deployment for single-server setups
|
|
✓ Network isolation via Docker networks
|
|
|
|
HOW IT'S USED:
|
|
--------------
|
|
1. Services Defined:
|
|
├── nats: NATS server with JetStream
|
|
├── nats-proxy: Nginx CORS proxy
|
|
└── nats-dashboard: Monitoring UI
|
|
|
|
2. Network Configuration:
|
|
- Network: coolify (external, shared with Traefik)
|
|
- All services communicate via service names
|
|
|
|
3. Volume Management:
|
|
- nats-data: Persistent storage for NATS data
|
|
- nats-js: JetStream storage
|
|
|
|
4. Health Checks:
|
|
- NATS: HTTP health check on port 8222
|
|
- Interval: 30s, Timeout: 10s, Retries: 3
|
|
|
|
CONFIGURATION:
|
|
- File: docker-compose.yml
|
|
- Traefik Labels: Configured for HTTPS routing
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ 3.7 KUBERNETES (k3s/k8s) │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
|
|
WHAT IT IS:
|
|
-----------
|
|
Kubernetes is a container orchestration platform that manages containerized
|
|
applications across a cluster of nodes.
|
|
|
|
WHY IT'S USED:
|
|
-------------
|
|
✓ High availability (pods distributed across nodes)
|
|
✓ Auto-scaling (HPA - Horizontal Pod Autoscaler)
|
|
✓ Self-healing (restarts failed pods)
|
|
✓ Load balancing
|
|
✓ Rolling updates
|
|
✓ Resource management
|
|
|
|
HOW IT'S USED:
|
|
--------------
|
|
1. Deployments:
|
|
├── fastapi-backend: 4-20 replicas (HPA)
|
|
└── nats-worker: 2-10 replicas (HPA)
|
|
|
|
2. Auto-Scaling (HPA):
|
|
FastAPI:
|
|
- Min Replicas: 4
|
|
- Max Replicas: 20
|
|
- CPU Target: 60%
|
|
- Memory Target: 70%
|
|
|
|
Worker:
|
|
- Min Replicas: 2
|
|
- Max Replicas: 10
|
|
- CPU Target: 70%
|
|
- Memory Target: 80%
|
|
|
|
3. Pod Anti-Affinity:
|
|
- Ensures pods spread across different nodes
|
|
- Prevents single point of failure
|
|
|
|
4. Services:
|
|
- fastapi-backend: ClusterIP (internal)
|
|
- fastapi-lb: LoadBalancer (port 8201)
|
|
- Worker: No external service (internal only)
|
|
|
|
5. Probes:
|
|
- Liveness: /health (FastAPI), /metrics (Worker)
|
|
- Readiness: /ready (FastAPI), /metrics (Worker)
|
|
|
|
CONFIGURATION:
|
|
- Files: manifests/*.yaml
|
|
- Namespace: nats-backend
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ 3.8 TRAEFIK (Ingress Controller) │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
|
|
WHAT IT IS:
|
|
-----------
|
|
Traefik is a modern reverse proxy and load balancer that automatically
|
|
discovers services via Docker labels or Kubernetes ingress.
|
|
|
|
WHY IT'S USED:
|
|
-------------
|
|
✓ Automatic TLS certificate management (Let's Encrypt)
|
|
✓ Service discovery via labels
|
|
✓ Dynamic configuration
|
|
✓ HTTP/HTTPS routing
|
|
✓ Middleware support (auth, rate limiting)
|
|
|
|
HOW IT'S USED:
|
|
--------------
|
|
1. Routing Rules:
|
|
- queue.workolik.com → FastAPI LoadBalancer (port 8201)
|
|
- natsadmin.workolik.com → NATS Dashboard (port 80)
|
|
- nats.workolik.com → NATS Proxy (port 80)
|
|
|
|
2. TLS Configuration:
|
|
- Entrypoint: https (port 443)
|
|
- Certificate Resolver: letsencrypt
|
|
- Automatic certificate renewal
|
|
|
|
3. Middleware:
|
|
- Basic Auth for NATS Dashboard
|
|
- CORS (handled by Nginx for NATS monitoring)
|
|
|
|
4. Network:
|
|
- Uses coolify Docker network
|
|
- Communicates with services via service names
|
|
|
|
CONFIGURATION:
|
|
- Docker Labels in docker-compose.yml
|
|
- Automatic discovery from container labels
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ 3.9 PROMETHEUS (Metrics) │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
|
|
WHAT IT IS:
|
|
-----------
|
|
Prometheus is a monitoring and alerting toolkit that collects metrics via
|
|
HTTP endpoints.
|
|
|
|
WHY IT'S USED:
|
|
-------------
|
|
✓ Standard metrics format
|
|
✓ Time-series database
|
|
✓ Integration with Kubernetes HPA
|
|
✓ Rich query language (PromQL)
|
|
✓ Alerting capabilities
|
|
|
|
HOW IT'S USED:
|
|
--------------
|
|
1. FastAPI Metrics (/metrics):
|
|
- http_requests_total: Request count by method, endpoint, status
|
|
- http_request_duration_seconds: Request latency histogram
|
|
|
|
2. Worker Metrics (port 9090):
|
|
- worker_messages_processed_total: Processed messages by status
|
|
- worker_message_duration_seconds: Processing time
|
|
- worker_messages_in_flight: Current processing count
|
|
- worker_queue_depth: Queue depth estimate
|
|
|
|
3. Kubernetes Integration:
|
|
- metrics-server scrapes pod metrics
|
|
- HPA uses metrics for auto-scaling decisions
|
|
|
|
CONFIGURATION:
|
|
- FastAPI: Built-in prometheus_client
|
|
- Worker: Built-in prometheus_client
|
|
- Kubernetes: metrics-server component
|
|
|
|
================================================================================
|
|
4. ARCHITECTURE DIAGRAMS & FLOW
|
|
================================================================================
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ 4.1 HIGH-LEVEL ARCHITECTURE │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
|
|
┌─────────────────┐
|
|
│ Client App │
|
|
│ (Mobile/Web) │
|
|
└────────┬────────┘
|
|
│ HTTPS
|
|
│ queue.workolik.com:443
|
|
▼
|
|
┌─────────────────┐
|
|
│ Traefik │
|
|
│ (TLS Terminate) │
|
|
└────────┬────────┘
|
|
│ HTTP
|
|
│ Port 8201
|
|
▼
|
|
┌─────────────────┐
|
|
│ k3s LoadBalancer│
|
|
│ (fastapi-lb) │
|
|
└────────┬────────┘
|
|
│ ClusterIP
|
|
│ Port 8000
|
|
▼
|
|
┌────────────────────┴────────────────────┐
|
|
│ │
|
|
▼ ▼
|
|
┌───────────────┐ ┌───────────────┐
|
|
│ FastAPI Pod │ │ FastAPI Pod │
|
|
│ (Pod 1) │ │ (Pod 2) │
|
|
└───────┬───────┘ └───────┬───────┘
|
|
│ │
|
|
└───────────────┬─────────────────────────┘
|
|
│ NATS Publish
|
|
│ Subject: api.>
|
|
▼
|
|
┌───────────────────────┐
|
|
│ NATS JetStream │
|
|
│ Stream: EVENTS │
|
|
│ Subject: api.> │
|
|
└───────────┬───────────┘
|
|
│ Pull Messages
|
|
│ Consumer: worker_consumer
|
|
▼
|
|
┌───────────────┴───────────────┐
|
|
│ │
|
|
▼ ▼
|
|
┌───────────────┐ ┌───────────────┐
|
|
│ Worker Pod │ │ Worker Pod │
|
|
│ (Pod 1) │ │ (Pod 2) │
|
|
└───────┬───────┘ └───────┬───────┘
|
|
│ │
|
|
└───────────────┬───────────────┘
|
|
│ HTTP Request
|
|
│ (PUT/POST)
|
|
▼
|
|
┌───────────────────────┐
|
|
│ External API │
|
|
│ jupiter.nearle.app │
|
|
└───────────────────────┘
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ 4.2 MESSAGE FLOW DETAIL │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
|
|
Step 1: CLIENT REQUEST
|
|
──────────────────────
|
|
Client → PUT /live/api/v1/deliveries/updatedelivery
|
|
Body: {"delivery_id": 123, "status": "delivered"}
|
|
|
|
Step 2: TRAEFIK ROUTING
|
|
───────────────────────
|
|
Traefik receives HTTPS request
|
|
→ Terminates TLS
|
|
→ Routes to k3s LoadBalancer (port 8201)
|
|
→ LoadBalancer forwards to FastAPI Service (ClusterIP:8000)
|
|
→ Service load balances to available FastAPI pod
|
|
|
|
Step 3: FASTAPI PROCESSING
|
|
───────────────────────────
|
|
FastAPI receives request:
|
|
→ Validates JSON body (Pydantic)
|
|
→ Creates message payload:
|
|
{
|
|
"endpoint": "/live/api/v1/deliveries/updatedelivery",
|
|
"method": "PUT",
|
|
"data": {"delivery_id": 123, "status": "delivered"},
|
|
"received_at": 1234567890,
|
|
"original_path": "/live/api/v1/deliveries/updatedelivery"
|
|
}
|
|
→ Publishes to NATS subject: "api.v1.deliveries.updatedelivery"
|
|
→ Returns 200 OK: {"status": "accepted", "message_id": 12345}
|
|
|
|
Step 4: NATS STORAGE
|
|
────────────────────
|
|
NATS JetStream:
|
|
→ Receives message on subject "api.v1.deliveries.updatedelivery"
|
|
→ Stores in stream "EVENTS"
|
|
→ Subject matches pattern "api.>"
|
|
→ Message available for consumer "worker_consumer"
|
|
|
|
Step 5: WORKER CONSUMPTION
|
|
──────────────────────────
|
|
Worker pod:
|
|
→ Pulls batch of messages (up to 10) from stream "EVENTS"
|
|
→ Processes message:
|
|
* Decodes JSON payload
|
|
* Extracts endpoint: "/live/api/v1/deliveries/updatedelivery"
|
|
* Extracts method: "PUT"
|
|
* Maps to external URL: "https://jupiter.nearle.app/live/api/v1/deliveries/updatedelivery"
|
|
→ Forwards HTTP PUT request to external API
|
|
→ Waits for response
|
|
|
|
Step 6: EXTERNAL API RESPONSE
|
|
─────────────────────────────
|
|
External API processes request:
|
|
→ Returns 200 OK or error
|
|
|
|
Step 7: WORKER ACKNOWLEDGMENT
|
|
─────────────────────────────
|
|
If success (200/201):
|
|
→ Worker sends ACK to NATS
|
|
→ Message removed from queue
|
|
→ Metrics updated: messages_processed_total{status="success"}
|
|
|
|
If failure (5xx/timeout):
|
|
→ Worker sends NAK to NATS
|
|
→ Message requeued for retry
|
|
→ Retry with exponential backoff (5s, 10s, 15s...)
|
|
→ Metrics updated: messages_processed_total{status="failed"}
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ 4.3 DOCKER COMPOSE ARCHITECTURE │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
|
|
┌─────────────────────────────────────────────────────────────────┐
|
|
│ coolify Network │
|
|
│ │
|
|
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
|
|
│ │ NATS │ │ nats-proxy │ │ nats-dashboard│ │
|
|
│ │ :4222 │◄───│ (nginx) │◄───│ :80 │ │
|
|
│ │ :8222 │ │ :80 │ │ │ │
|
|
│ │ :6222 │ │ :8082 │ │ :8081 │ │
|
|
│ └──────────────┘ └──────────────┘ └──────────────┘ │
|
|
│ ▲ │
|
|
│ │ │
|
|
│ └───────────────────────────────────────────────────┘
|
|
│ (FastAPI & Workers connect here) │
|
|
└─────────────────────────────────────────────────────────────────┘
|
|
│
|
|
│ Traefik Routes
|
|
▼
|
|
┌─────────────────────┐
|
|
│ External Access │
|
|
│ │
|
|
│ nats.workolik.com │
|
|
│ natsadmin.workolik │
|
|
└─────────────────────┘
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ 4.4 KUBERNETES ARCHITECTURE │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
|
|
┌─────────────────────────────────────────────────────────────────────┐
|
|
│ Kubernetes Cluster (3 Nodes) │
|
|
│ │
|
|
│ Node 1 Node 2 Node 3 │
|
|
│ ┌─────┐ ┌─────┐ ┌─────┐ │
|
|
│ │FAPI │ │FAPI │ │FAPI │ │
|
|
│ │Pod 1│ │Pod 2│ │Pod 3│ │
|
|
│ └──┬──┘ └──┬──┘ └──┬──┘ │
|
|
│ │ │ │ │
|
|
│ └───────────────┼───────────────┘ │
|
|
│ │ │
|
|
│ ┌──────▼──────┐ │
|
|
│ │ Service │ │
|
|
│ │ ClusterIP │ │
|
|
│ │ :8000 │ │
|
|
│ └──────┬──────┘ │
|
|
│ │ │
|
|
│ ┌──────▼──────┐ │
|
|
│ │ LoadBalancer│ │
|
|
│ │ :8201 │ │
|
|
│ └──────┬──────┘ │
|
|
│ │ │
|
|
│ ┌──────────────────┼──────────────────┐ │
|
|
│ │ │ │ │
|
|
│ ▼ ▼ ▼ │
|
|
│ ┌─────┐ ┌─────┐ ┌─────┐ │
|
|
│ │Worker│ │Worker│ │Worker│ │
|
|
│ │Pod 1 │ │Pod 2 │ │Pod 3 │ │
|
|
│ └──┬──┘ └──┬──┘ └──┬──┘ │
|
|
│ │ │ │ │
|
|
│ └─────────────────┼──────────────────┘ │
|
|
│ │ │
|
|
│ ┌───────▼───────┐ │
|
|
│ │ External NATS │ │
|
|
│ │ nats.workolik │ │
|
|
│ │ .com:4222 │ │
|
|
│ └───────────────┘ │
|
|
└─────────────────────────────────────────────────────────────────────┘
|
|
|
|
================================================================================
|
|
5. DEPLOYMENT MODELS
|
|
================================================================================
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ 5.1 DOCKER COMPOSE DEPLOYMENT (VPS/Single Server) │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
|
|
USE CASE: Local development, small-scale production, single-server deployment
|
|
|
|
COMPONENTS:
|
|
- NATS Server (containerized)
|
|
- Nginx Proxy (CORS handling)
|
|
- NATS Dashboard (monitoring)
|
|
|
|
ADVANTAGES:
|
|
✓ Simple deployment (docker-compose up)
|
|
✓ No Kubernetes complexity
|
|
✓ Lower resource overhead
|
|
✓ Easy debugging
|
|
|
|
LIMITATIONS:
|
|
✗ No auto-scaling
|
|
✗ Single point of failure
|
|
✗ Manual scaling required
|
|
|
|
DEPLOYMENT STEPS:
|
|
1. docker-compose up -d
|
|
2. Services start automatically
|
|
3. Traefik discovers services via labels
|
|
4. TLS certificates auto-provisioned
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ 5.2 KUBERNETES DEPLOYMENT (Production Cluster) │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
|
|
USE CASE: Production environments, high availability, auto-scaling
|
|
|
|
COMPONENTS:
|
|
- FastAPI Deployment (4-20 pods, HPA)
|
|
- Worker Deployment (2-10 pods, HPA)
|
|
- Services (ClusterIP, LoadBalancer)
|
|
- HPA (Horizontal Pod Autoscaler)
|
|
- External NATS (separate server)
|
|
|
|
ADVANTAGES:
|
|
✓ High availability (pods across nodes)
|
|
✓ Auto-scaling based on metrics
|
|
✓ Self-healing (restarts failed pods)
|
|
✓ Rolling updates
|
|
✓ Resource limits and requests
|
|
|
|
DEPLOYMENT STEPS:
|
|
1. Build Docker images
|
|
2. Push to registry
|
|
3. Update manifests/secrets.yaml
|
|
4. kubectl apply -f manifests/
|
|
5. Verify pods: kubectl get pods -n nats-backend
|
|
|
|
================================================================================
|
|
6. INTEGRATION POINTS
|
|
================================================================================
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ 6.1 FASTAPI → NATS INTEGRATION │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
|
|
LIBRARY: nats-py (Python NATS client)
|
|
|
|
CONNECTION:
|
|
- URL: nats://admin:package%40321%23@nats:4222
|
|
- Authentication: Username/Password
|
|
- Reconnection: Automatic (max 10 attempts)
|
|
|
|
PUBLISHING:
|
|
- JetStream context: nc.jetstream()
|
|
- Subject mapping: Endpoint → Subject pattern
|
|
- Payload: JSON-encoded message with metadata
|
|
- Acknowledgment: Synchronous (waits for ack)
|
|
|
|
ERROR HANDLING:
|
|
- Connection failures: Raises HTTPException 503
|
|
- Publish failures: Raises HTTPException 500
|
|
- Retry logic: Handled by NATS client
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ 6.2 WORKER → NATS INTEGRATION │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
|
|
CONSUMPTION:
|
|
- Pull subscription: js.pull_subscribe()
|
|
- Batch fetch: fetch(10, timeout=5)
|
|
- Consumer: "worker_consumer"
|
|
- Stream: "EVENTS"
|
|
|
|
MESSAGE PROCESSING:
|
|
- Decode: json.loads(msg.data.decode())
|
|
- Extract: endpoint, method, data
|
|
- Process: Forward to external API
|
|
- Acknowledge: msg.ack() or msg.nak()
|
|
|
|
ERROR HANDLING:
|
|
- Invalid JSON: msg.term() (terminate, no retry)
|
|
- Processing error: msg.nak() (requeue)
|
|
- External API error: Retry logic in worker
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ 6.3 NGINX → NATS MONITORING INTEGRATION │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
|
|
PROXY CONFIGURATION:
|
|
- Upstream: nats:8222
|
|
- CORS Headers: Added for dashboard access
|
|
- OPTIONS Handling: Preflight requests
|
|
|
|
WHY NEEDED:
|
|
- NATS Dashboard runs in browser (JavaScript)
|
|
- Browser enforces CORS policy
|
|
- NATS monitoring API doesn't include CORS headers
|
|
- Nginx adds required headers
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ 6.4 TRAEFIK → SERVICES INTEGRATION │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
|
|
SERVICE DISCOVERY:
|
|
- Docker Labels: Automatic discovery
|
|
- Routing Rules: Host-based (queue.workolik.com)
|
|
- TLS: Automatic Let's Encrypt certificates
|
|
|
|
MIDDLEWARE:
|
|
- Basic Auth: For NATS Dashboard
|
|
- TLS Termination: HTTPS → HTTP
|
|
|
|
================================================================================
|
|
7. SECURITY & AUTHENTICATION
|
|
================================================================================
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ 7.1 NATS AUTHENTICATION │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
|
|
METHOD: Username/Password Authentication
|
|
|
|
CREDENTIALS:
|
|
- Username: admin
|
|
- Password: package@321#
|
|
- URL Encoding: package%40321%23 (@ = %40, # = %23)
|
|
|
|
CONFIGURATION:
|
|
- Set in NATS server config (nats.conf)
|
|
- Required for all client connections
|
|
- FastAPI and Workers authenticate on startup
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ 7.2 TRAEFIK BASIC AUTH │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
|
|
PURPOSE: Protect NATS Dashboard from unauthorized access
|
|
|
|
METHOD: HTTP Basic Authentication
|
|
|
|
CREDENTIALS:
|
|
- Username: admin
|
|
- Password: package@321#
|
|
- Hash Algorithm: bcrypt ($2b$12$...)
|
|
|
|
GENERATION:
|
|
- Script: setup-auth.sh
|
|
- Tool: htpasswd (Apache HTTP Server)
|
|
- Command: htpasswd -nbB admin "package@321#"
|
|
|
|
CONFIGURATION:
|
|
- Traefik Middleware: nats-auth
|
|
- Applied to: nats-ui router
|
|
- Hash Format: admin:$2b$12$RcO6lQBCV9vht1i0GTmYV.l5j09WB0ps0WDGit4MxdtFJIUpvZmEi
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ 7.3 TLS/HTTPS │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
|
|
PROVIDER: Let's Encrypt (via Traefik)
|
|
|
|
DOMAINS:
|
|
- queue.workolik.com (FastAPI)
|
|
- natsadmin.workolik.com (Dashboard)
|
|
- nats.workolik.com (Monitoring Proxy)
|
|
|
|
CERTIFICATE RESOLVER: letsencrypt
|
|
|
|
AUTOMATIC RENEWAL: Traefik handles certificate renewal
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ 7.4 CORS CONFIGURATION │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
|
|
PURPOSE: Allow browser-based dashboard to access NATS monitoring API
|
|
|
|
CONFIGURATION:
|
|
- Allowed Origin: https://natsadmin.workolik.com
|
|
- Methods: GET, OPTIONS, HEAD
|
|
- Credentials: true
|
|
- Headers: Content-Type, Authorization, Accept
|
|
|
|
IMPLEMENTATION:
|
|
- Nginx adds CORS headers
|
|
- OPTIONS preflight handled separately
|
|
- Headers hidden from backend to prevent duplicates
|
|
|
|
================================================================================
|
|
8. MONITORING & OBSERVABILITY
|
|
================================================================================
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ 8.1 PROMETHEUS METRICS │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
|
|
FASTAPI METRICS (/metrics):
|
|
- http_requests_total: Counter
|
|
* Labels: method, endpoint, status
|
|
* Tracks: Total requests by type and result
|
|
|
|
- http_request_duration_seconds: Histogram
|
|
* Labels: method, endpoint
|
|
* Tracks: Request latency distribution
|
|
|
|
WORKER METRICS (port 9090):
|
|
- worker_messages_processed_total: Counter
|
|
* Labels: status (success/failed/invalid/error), endpoint
|
|
* Tracks: Total messages processed
|
|
|
|
- worker_message_duration_seconds: Histogram
|
|
* Labels: endpoint
|
|
* Tracks: Message processing time
|
|
|
|
- worker_messages_in_flight: Gauge
|
|
* Tracks: Currently processing messages
|
|
|
|
- worker_queue_depth: Gauge
|
|
* Tracks: Approximate queue depth
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ 8.2 NATS DASHBOARD │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
|
|
ACCESS: https://natsadmin.workolik.com
|
|
|
|
FEATURES:
|
|
- Stream Statistics (EVENTS stream)
|
|
- Consumer Status (worker_consumer)
|
|
- Message Rates (messages/second)
|
|
- Pending Messages Count
|
|
- Consumer Lag Monitoring
|
|
|
|
DATA SOURCE: NATS Monitoring API (via Nginx proxy)
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ 8.3 KUBERNETES HEALTH CHECKS │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
|
|
LIVENESS PROBES:
|
|
- FastAPI: GET /health (every 30s)
|
|
- Worker: GET /metrics (every 30s)
|
|
|
|
READINESS PROBES:
|
|
- FastAPI: GET /ready (checks NATS connection)
|
|
- Worker: GET /metrics
|
|
|
|
PURPOSE:
|
|
- Liveness: Restart pod if unhealthy
|
|
- Readiness: Remove from service if not ready
|
|
|
|
================================================================================
|
|
9. SCALABILITY & PERFORMANCE
|
|
================================================================================
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ 9.1 HORIZONTAL SCALING (Kubernetes HPA) │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
|
|
FASTAPI AUTO-SCALING:
|
|
- Min Replicas: 4
|
|
- Max Replicas: 20
|
|
- CPU Target: 60%
|
|
- Memory Target: 70%
|
|
- Scale Up: When CPU > 60% or Memory > 70%
|
|
- Scale Down: When below thresholds
|
|
|
|
WORKER AUTO-SCALING:
|
|
- Min Replicas: 2
|
|
- Max Replicas: 10
|
|
- CPU Target: 70%
|
|
- Memory Target: 80%
|
|
- Scale Up: When CPU > 70% or Memory > 80%
|
|
- Scale Down: When below thresholds
|
|
|
|
METRICS SOURCE: Kubernetes metrics-server
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ 9.2 CONCURRENCY CONTROL │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
|
|
WORKER CONCURRENCY:
|
|
- Semaphore Limit: 10 (configurable)
|
|
- Prevents: Overwhelming external API
|
|
- Configurable: WORKER_CONCURRENCY env var
|
|
|
|
MESSAGE BATCHING:
|
|
- Batch Size: 10 messages per fetch
|
|
- Timeout: 5 seconds
|
|
- Benefit: Reduces NATS round trips
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ 9.3 PERFORMANCE CHARACTERISTICS │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
|
|
NATS PERFORMANCE:
|
|
- Latency: < 1ms (local network)
|
|
- Throughput: Millions of messages/second
|
|
- Persistence: File-based (JetStream)
|
|
|
|
FASTAPI PERFORMANCE:
|
|
- Framework: Starlette (async)
|
|
- Server: Uvicorn (ASGI)
|
|
- Concurrency: Async/await (non-blocking)
|
|
|
|
WORKER PERFORMANCE:
|
|
- Library: aiohttp (async HTTP)
|
|
- Concurrency: Configurable semaphore
|
|
- Retry Logic: Exponential backoff
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ 9.4 RESOURCE MANAGEMENT │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
|
|
KUBERNETES RESOURCES:
|
|
FastAPI:
|
|
- Requests: CPU 100m, Memory 256Mi
|
|
- Limits: CPU 500m, Memory 512Mi
|
|
|
|
Worker:
|
|
- Requests: CPU 100m, Memory 256Mi
|
|
- Limits: CPU 500m, Memory 512Mi
|
|
|
|
POD ANTI-AFFINITY:
|
|
- Ensures pods spread across nodes
|
|
- Prevents resource contention
|
|
- Improves availability
|
|
|
|
================================================================================
|
|
10. TROUBLESHOOTING GUIDE
|
|
================================================================================
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ 10.1 COMMON ISSUES │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
|
|
ISSUE: FastAPI cannot connect to NATS
|
|
──────────────────────────────────────
|
|
SYMPTOMS: 503 errors, "NATS not connected"
|
|
SOLUTION:
|
|
1. Check NATS service is running: docker ps | grep nats
|
|
2. Verify NATS_URL environment variable
|
|
3. Check network connectivity: ping nats (from FastAPI container)
|
|
4. Verify credentials: NATS_USER, NATS_PASSWORD
|
|
5. Check NATS logs: docker logs nats
|
|
|
|
ISSUE: Worker not processing messages
|
|
──────────────────────────────────────
|
|
SYMPTOMS: Messages accumulating in queue, no worker logs
|
|
SOLUTION:
|
|
1. Check worker pods: kubectl get pods -n nats-backend
|
|
2. View worker logs: kubectl logs -f deployment/nats-worker
|
|
3. Verify NATS connection: Check worker startup logs
|
|
4. Check consumer exists: NATS Dashboard → Consumers
|
|
5. Verify stream configuration: NATS Dashboard → Streams
|
|
|
|
ISSUE: CORS errors in browser
|
|
──────────────────────────────
|
|
SYMPTOMS: Browser console shows CORS errors
|
|
SOLUTION:
|
|
1. Verify Nginx proxy is running: docker ps | grep nats-proxy
|
|
2. Check nginx-nats.conf CORS headers
|
|
3. Verify NATS_MONITORING_URL in dashboard config
|
|
4. Check browser console for specific CORS error
|
|
5. Verify allowed origin matches dashboard URL
|
|
|
|
ISSUE: Messages not being forwarded to external API
|
|
────────────────────────────────────────────────────
|
|
SYMPTOMS: Messages processed but external API not receiving
|
|
SOLUTION:
|
|
1. Check worker logs for HTTP errors
|
|
2. Verify EXTERNAL_BASE_URL environment variable
|
|
3. Test external API connectivity: curl from worker pod
|
|
4. Check API key if required: EXTERNAL_ENDPOINT_API_KEY
|
|
5. Verify endpoint mapping in worker.py
|
|
|
|
ISSUE: Auto-scaling not working
|
|
─────────────────────────────────
|
|
SYMPTOMS: Pods not scaling despite high load
|
|
SOLUTION:
|
|
1. Check HPA status: kubectl get hpa -n nats-backend
|
|
2. Verify metrics-server: kubectl top pods
|
|
3. Check HPA events: kubectl describe hpa -n nats-backend
|
|
4. Verify resource requests/limits in deployment
|
|
5. Check if max replicas reached
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ 10.2 DEBUGGING COMMANDS │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
|
|
DOCKER COMPOSE:
|
|
- View logs: docker-compose logs -f [service]
|
|
- Restart service: docker-compose restart [service]
|
|
- Check status: docker-compose ps
|
|
- Shell access: docker-compose exec [service] sh
|
|
|
|
KUBERNETES:
|
|
- Pod status: kubectl get pods -n nats-backend -o wide
|
|
- Pod logs: kubectl logs -f [pod-name] -n nats-backend
|
|
- Describe pod: kubectl describe pod [pod-name] -n nats-backend
|
|
- Exec into pod: kubectl exec -it [pod-name] -n nats-backend -- sh
|
|
- Service endpoints: kubectl get endpoints -n nats-backend
|
|
- HPA status: kubectl get hpa -n nats-backend
|
|
- Events: kubectl get events -n nats-backend --sort-by='.lastTimestamp'
|
|
|
|
NATS:
|
|
- Connect via CLI: nats -s nats://admin:package@321#@nats:4222
|
|
- Stream info: nats stream info EVENTS
|
|
- Consumer info: nats consumer info EVENTS worker_consumer
|
|
- Pending messages: nats consumer next EVENTS worker_consumer
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ 10.3 LOG ANALYSIS │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
|
|
FASTAPI LOGS:
|
|
- Look for: "✅ Connected to NATS JetStream"
|
|
- Error patterns: "❌ Failed to connect", "❌ Failed to publish"
|
|
- Success patterns: "✅ Message published", status 200
|
|
|
|
WORKER LOGS:
|
|
- Look for: "✅ Connected to NATS JetStream", "✅ Subscribed"
|
|
- Processing: "📨 Processing message for endpoint"
|
|
- Success: "✅ Message processed and forwarded successfully"
|
|
- Errors: "❌ Failed to forward", "⚠️ Error forwarding"
|
|
|
|
NATS LOGS:
|
|
- Connection: "[INF] Client connection"
|
|
- JetStream: "[INF] JetStream enabled"
|
|
- Errors: "[ERR]" prefix
|
|
|
|
================================================================================
|
|
11. CONFIGURATION REFERENCE
|
|
================================================================================
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ 11.1 ENVIRONMENT VARIABLES │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
|
|
FASTAPI (app.py):
|
|
- NATS_URL: nats://nats-server:4222
|
|
- NATS_USER: admin
|
|
- NATS_PASSWORD: package@321#
|
|
- ALLOWED_ORIGINS: http://localhost:3001,http://localhost:3000,https://queue.workolik.com
|
|
|
|
WORKER (worker.py):
|
|
- NATS_URL: nats://nats-server:4222
|
|
- NATS_USER: admin
|
|
- NATS_PASSWORD: package@321#
|
|
- NATS_STREAM: EVENTS
|
|
- NATS_SUBJECT: api.>
|
|
- NATS_CONSUMER: worker_consumer
|
|
- EXTERNAL_BASE_URL: https://jupiter.nearle.app
|
|
- WORKER_CONCURRENCY: 10
|
|
- RETRY_ATTEMPTS: 5
|
|
- RETRY_DELAY_SECONDS: 5
|
|
- EXTERNAL_ENDPOINT_API_KEY: (optional)
|
|
|
|
NATS DASHBOARD:
|
|
- NATS_URL: nats://admin:package%40321%23@nats:4222
|
|
- NATS_MONITORING_URL: http://148.135.137.174:8082
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ 11.2 PORT MAPPINGS │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
|
|
NATS:
|
|
- 4222: Client connections
|
|
- 8222: HTTP monitoring
|
|
- 6222: Cluster routing
|
|
|
|
FASTAPI:
|
|
- 8000: HTTP API (internal)
|
|
- 8201: LoadBalancer (external via Traefik)
|
|
|
|
WORKER:
|
|
- 9090: Prometheus metrics
|
|
|
|
NGINX PROXY:
|
|
- 80: Internal (container)
|
|
- 8082: External (Docker Compose)
|
|
|
|
NATS DASHBOARD:
|
|
- 80: Internal (container)
|
|
- 8081: External (Docker Compose)
|
|
|
|
KUBERNETES:
|
|
- 30826: Dashboard proxy NodePort
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ 11.3 ENDPOINT MAPPINGS │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
|
|
FASTAPI → NATS SUBJECT:
|
|
/live/api/v1/deliveries/createdeliveries → api.v1.deliveries.createdeliveries
|
|
/live/api/v1/deliveries/updatedelivery → api.v1.deliveries.updatedelivery
|
|
/live/api/v2/partners/createriderlog → api.v2.partners.createriderlog
|
|
/live/api/v2/deliveries/createdeliverylog → api.v2.deliveries.createdeliverylog
|
|
/live/api/v2/partners/createbreaklog → api.v2.partners.createbreaklog
|
|
/live/api/v2/partners/updatebreaklog → api.v2.partners.updatebreaklog
|
|
|
|
INTERNAL → EXTERNAL API:
|
|
/live/api/v1/deliveries/updatedelivery → https://jupiter.nearle.app/live/api/v1/deliveries/updatedelivery
|
|
/live/api/v2/partners/createriderlog → https://jupiter.nearle.app/live/api/v2/partners/createriderlog
|
|
... (same pattern for all endpoints)
|
|
|
|
================================================================================
|
|
12. FILE STRUCTURE REFERENCE
|
|
================================================================================
|
|
|
|
kubernetes/
|
|
├── docker-compose.yml # Docker Compose deployment
|
|
├── Dockerfile # Multi-stage build (api + worker)
|
|
├── requirements.txt # Python dependencies
|
|
├── nginx-nats.conf # Nginx CORS proxy config
|
|
├── nginx-k8s-dashboard.conf # Kubernetes dashboard proxy
|
|
├── setup-auth.sh # Traefik auth hash generator
|
|
├── deploy.sh # Kubernetes deployment script
|
|
├── build-and-deploy.sh # Build + deploy script
|
|
├── scripts/
|
|
│ ├── app.py # FastAPI application
|
|
│ └── worker.py # NATS worker consumer
|
|
└── manifests/
|
|
├── namespace.yaml # Kubernetes namespace
|
|
├── fastapi-deployment.yaml # FastAPI pods
|
|
├── fastapi-service.yaml # FastAPI ClusterIP service
|
|
├── fastapi-loadbalancer.yaml # FastAPI LoadBalancer
|
|
├── fastapi-hpa.yaml # FastAPI auto-scaling
|
|
├── worker-deployment.yaml # Worker pods
|
|
├── worker-hpa.yaml # Worker auto-scaling
|
|
├── secrets.yaml # Environment variables
|
|
├── gateway.yaml # Gateway API (optional)
|
|
├── dashboard.yaml # NATS Dashboard
|
|
├── dashboard-proxy-deployment.yaml # Dashboard proxy
|
|
└── dashboard-loadbalancer.yaml # Dashboard LoadBalancer
|
|
|
|
================================================================================
|
|
13. KEY DESIGN DECISIONS
|
|
================================================================================
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ 13.1 WHY NATS JETSTREAM? │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
|
|
✓ Lightweight: Single binary, no external dependencies
|
|
✓ Performance: Sub-millisecond latency, high throughput
|
|
✓ Persistence: JetStream provides message durability
|
|
✓ Simplicity: Easy to deploy and configure
|
|
✓ Cloud-Native: Designed for microservices
|
|
✓ At-Least-Once: Guaranteed delivery with acknowledgments
|
|
|
|
ALTERNATIVES CONSIDERED:
|
|
- RabbitMQ: More complex, heavier resource usage
|
|
- Kafka: Overkill for this use case, complex setup
|
|
- Redis Streams: Less mature, limited features
|
|
- AWS SQS: Vendor lock-in, additional cost
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ 13.2 WHY FASTAPI? │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
|
|
✓ Async Support: Native async/await for NATS integration
|
|
✓ Performance: Comparable to Node.js, faster than Flask
|
|
✓ Type Safety: Pydantic validation
|
|
✓ Documentation: Automatic OpenAPI/Swagger
|
|
✓ Modern: Built on Starlette and Pydantic
|
|
✓ Easy: Simple to learn and use
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ 13.3 WHY WORKER PATTERN? │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
|
|
✓ Decoupling: API layer independent of external service availability
|
|
✓ Resilience: Retry logic handles transient failures
|
|
✓ Scalability: Workers scale independently
|
|
✓ Observability: Separate metrics and logs
|
|
✓ Flexibility: Easy to add new endpoints
|
|
|
|
┌─────────────────────────────────────────────────────────────────────────┐
|
|
│ 13.4 WHY DUAL DEPLOYMENT (Docker Compose + Kubernetes)? │
|
|
└─────────────────────────────────────────────────────────────────────────┘
|
|
|
|
✓ Flexibility: Docker Compose for simple deployments, K8s for production
|
|
✓ Development: Easy local testing with Docker Compose
|
|
✓ Production: Kubernetes for high availability and auto-scaling
|
|
✓ Migration Path: Can start simple and scale up
|
|
|
|
================================================================================
|
|
END OF DOCUMENTATION
|
|
================================================================================
|
|
|
|
This document provides a comprehensive overview of the NATS-based message queue
|
|
architecture. For specific implementation details, refer to the source code
|
|
files listed in Section 12.
|
|
|
|
For questions or issues, refer to Section 10 (Troubleshooting Guide) or
|
|
consult the individual component documentation.
|
|
|
|
Last Updated: 2025-01-XX
|
|
Version: 1.0
|
|
|