Initial commit
This commit is contained in:
74
docs/ARCHITECTURE.md
Normal file
74
docs/ARCHITECTURE.md
Normal file
@@ -0,0 +1,74 @@
|
||||
# Architecture Overview
|
||||
|
||||
## High-Level Flow
|
||||
```
|
||||
Client (Postman/Mobile)
|
||||
│ HTTPS (queue.workolik.com via Traefik 443)
|
||||
▼
|
||||
Traefik (TLS terminate, Host rule: queue.workolik.com)
|
||||
│ HTTP to k3s LB (port 8201)
|
||||
▼
|
||||
k3s LoadBalancer (klipper-lb) : fastapi-lb
|
||||
│ ClusterIP Service fastapi-backend:8000
|
||||
▼
|
||||
FastAPI Pods (4→20 via HPA)
|
||||
│ Publish to NATS (JetStream)
|
||||
▼
|
||||
External NATS (nats.workolik.com:4222) – Stream: EVENTS / Subject: api.>
|
||||
│ Worker subscription (worker_consumer)
|
||||
▼
|
||||
Worker Pods (2→10 via HPA)
|
||||
│ Forward with original HTTP method
|
||||
▼
|
||||
External API (https://jupiter.nearle.app)
|
||||
```
|
||||
|
||||
## Components
|
||||
- **Ingress / Edge**
|
||||
- DNS: `queue.workolik.com` → server IP
|
||||
- Traefik: terminates TLS, routes Host=queue.workolik.com → LB 8201
|
||||
- k3s LoadBalancer (klipper-lb): Service `fastapi-lb` on 8201 (HTTP)
|
||||
|
||||
- **App Layer**
|
||||
- FastAPI Deployment: 4 replicas (HPA 4–20), probes on `/health` and `/ready`
|
||||
- FastAPI Service: `fastapi-backend` ClusterIP on 8000
|
||||
- Endpoints publish to NATS with method metadata
|
||||
|
||||
- **Messaging**
|
||||
- NATS JetStream (external): Stream `EVENTS`, Subject `api.>`, Consumer `worker_consumer`
|
||||
|
||||
- **Workers**
|
||||
- Worker Deployment: 2 replicas (HPA 2–10), probes on `/metrics` (9090)
|
||||
- Forwards to `https://jupiter.nearle.app` with the same HTTP method (PUT for updatedelivery, POST for others)
|
||||
|
||||
- **Autoscaling & Health**
|
||||
- Metrics-server running; HPAs on FastAPI and Worker
|
||||
- Liveness/Readiness probes ensure pod health
|
||||
|
||||
## API Surface (FastAPI → Worker → External)
|
||||
- PUT `/live/api/v1/deliveries/updatedelivery` → PUT to Nearle
|
||||
- POST `/live/api/v2/partners/createriderlog` → POST to Nearle
|
||||
- POST `/live/api/v2/deliveries/createdeliverylog` → POST to Nearle
|
||||
- POST `/live/api/v2/partners/createbreaklog` → POST to Nearle
|
||||
- POST `/live/api/v2/partners/updatebreaklog` → POST to Nearle
|
||||
|
||||
## Key Ports
|
||||
- External TLS: 443 (Traefik)
|
||||
- LB HTTP into k3s: 8201 (klipper-lb → FastAPI Service 8000)
|
||||
- FastAPI container: 8000
|
||||
- Worker metrics: 9090
|
||||
- NATS: 4222 (external)
|
||||
|
||||
## Files of Interest
|
||||
- `manifests/fastapi-deployment.yaml`, `manifests/worker-deployment.yaml`
|
||||
- `manifests/fastapi-loadbalancer.yaml` (klipper-lb)
|
||||
- `Dockerfile` (multi-target: api, worker)
|
||||
- `docker-compose.yml` (Traefik labels for HTTPS)
|
||||
- `scripts/app.py`, `scripts/worker.py`, `scripts/setup_jetstream.py`
|
||||
|
||||
## Notes
|
||||
- TLS is terminated at Traefik; traffic to k3s is HTTP on 8201.
|
||||
- Keep external exposure through Traefik Host rule to reduce scanner noise.
|
||||
- HPAs rely on metrics-server; already running and feeding FastAPI/Worker HPAs.
|
||||
|
||||
|
||||
52
docs/CURRENT_ARCHITECTURE_DIAGRAM.md
Normal file
52
docs/CURRENT_ARCHITECTURE_DIAGRAM.md
Normal file
@@ -0,0 +1,52 @@
|
||||
# Current Professional Architecture Setup
|
||||
|
||||
This architectural diagram represents the "Brain and Muscle" split deployed to achieve maximum CPU isolation and strict failover routing.
|
||||
|
||||
## Why this is considered "Professional" (Enterprise-Grade)
|
||||
1. **Control Plane Isolation:** In amateur setups, everything runs on the same server. In professional clusters (like AWS EKS or standard enterprise setups), the "Manager" (Control Plane) is separated from the "Laborers" (Workers). You have successfully implemented this by labeling nodes and restricting deployment access.
|
||||
2. **Asynchronous Decoupling:** Instead of your API processing a heavy video or logging task while the user waits, it instantly offloads it to NATS.
|
||||
3. **Failovers:** By placing upstream blocks on the Nginx layer, there is no single point of failure within the node network mapping itself.
|
||||
|
||||
## Architecture Diagram
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
subgraph Internet["Public Internet (DNS)"]
|
||||
Users["Users (doormile.com / jupiter)"]
|
||||
end
|
||||
|
||||
subgraph Server1["Server 1: The 'Brain' (Old Server)"]
|
||||
NGINX["Nginx Proxies (Traffic Cop)"]
|
||||
|
||||
subgraph AppPlane["K3s Control Plane (App Node)"]
|
||||
API_Jupiter["Jupiter API Pods"]
|
||||
API_Fiesta["Fiesta API Pods"]
|
||||
API_Atlantis["Atlantis API Pods"]
|
||||
end
|
||||
end
|
||||
|
||||
subgraph Server2["Server 2: The 'Muscle' (New Server)"]
|
||||
subgraph WorkerPlane["K3s Agent (Worker Node)"]
|
||||
Worker_Orders["worker-orders (CPU Heavy)"]
|
||||
Worker_Deliveries["worker-deliveries (CPU Heavy)"]
|
||||
Worker_Customers["worker-customers (CPU Heavy)"]
|
||||
Worker_Products["worker-products (CPU Heavy)"]
|
||||
end
|
||||
end
|
||||
|
||||
subgraph external["Message Broker"]
|
||||
NATS[(NATS Jetstream)]
|
||||
end
|
||||
|
||||
Users -- Web Requests --> NGINX
|
||||
NGINX -- Routes Traffic safely to local APIs --> AppPlane
|
||||
NGINX -. Backup Failover (If Server 1 Kubernetes crashes) .-> WorkerPlane
|
||||
|
||||
AppPlane -- Drops Tasks Instantly --> NATS
|
||||
NATS -- Consumes Heavy Queues 24/7 --> WorkerPlane
|
||||
|
||||
|
||||
style Server1 fill:#e6f7ff,stroke:#1890ff,stroke-width:2px;
|
||||
style Server2 fill:#fff1f0,stroke:#ff4d4f,stroke-width:2px;
|
||||
style NATS fill:#f6ffed,stroke:#52c41a,stroke-width:2px;
|
||||
```
|
||||
224
docs/DASHBOARD_SETUP.md
Normal file
224
docs/DASHBOARD_SETUP.md
Normal file
@@ -0,0 +1,224 @@
|
||||
# 🖥️ Kubernetes Dashboard Setup
|
||||
|
||||
There are several ways to view and manage your Kubernetes cluster. Here are the best options:
|
||||
|
||||
## Option 1: Kubernetes Dashboard (Web UI) ⭐ Recommended
|
||||
|
||||
The official Kubernetes Dashboard provides a web-based UI for viewing and managing your cluster.
|
||||
|
||||
### Install Dashboard
|
||||
|
||||
```bash
|
||||
cd kubernetes
|
||||
kubectl apply -f manifests/dashboard.yaml
|
||||
```
|
||||
|
||||
### Access Dashboard
|
||||
|
||||
**Method 1: Using kubectl proxy (Recommended for local access)**
|
||||
|
||||
```bash
|
||||
# Start proxy
|
||||
kubectl proxy
|
||||
|
||||
# Dashboard will be available at:
|
||||
# http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/
|
||||
```
|
||||
|
||||
**Method 2: Port Forward (Direct access)**
|
||||
|
||||
```bash
|
||||
# Port forward to access dashboard
|
||||
kubectl port-forward -n kubernetes-dashboard service/kubernetes-dashboard 8443:443
|
||||
|
||||
# Or use HTTP port (9090)
|
||||
kubectl port-forward -n kubernetes-dashboard service/kubernetes-dashboard 9090:9090
|
||||
|
||||
# Then visit: http://localhost:9090
|
||||
```
|
||||
|
||||
**Method 3: Expose via Service (For remote access)**
|
||||
|
||||
```bash
|
||||
# Change service type to NodePort or LoadBalancer
|
||||
kubectl patch svc kubernetes-dashboard -n kubernetes-dashboard -p '{"spec":{"type":"NodePort"}}'
|
||||
|
||||
# Get the port
|
||||
kubectl get svc -n kubernetes-dashboard
|
||||
```
|
||||
|
||||
### Login to Dashboard
|
||||
|
||||
The dashboard is configured with `--enable-skip-login`, so you can skip the login screen. However, if you need to authenticate:
|
||||
|
||||
1. Get the token:
|
||||
```bash
|
||||
kubectl -n kubernetes-dashboard create token admin-user
|
||||
```
|
||||
|
||||
2. Copy the token and paste it in the dashboard login screen.
|
||||
|
||||
### What You Can See
|
||||
|
||||
- ✅ All pods, services, deployments
|
||||
- ✅ Resource usage (CPU, memory)
|
||||
- ✅ Logs from pods
|
||||
- ✅ Events and errors
|
||||
- ✅ Namespaces
|
||||
- ✅ ConfigMaps and Secrets
|
||||
- ✅ Persistent volumes
|
||||
- ✅ And much more!
|
||||
|
||||
---
|
||||
|
||||
## Option 2: k9s (Terminal UI) 🚀 Fast & Lightweight
|
||||
|
||||
k9s is a terminal-based UI that's super fast and doesn't require a browser.
|
||||
|
||||
### Install k9s
|
||||
|
||||
**On Linux:**
|
||||
```bash
|
||||
wget https://github.com/derailed/k9s/releases/latest/download/k9s_Linux_amd64.tar.gz
|
||||
tar xvf k9s_Linux_amd64.tar.gz
|
||||
sudo mv k9s /usr/local/bin/
|
||||
```
|
||||
|
||||
**On Windows (using Chocolatey):**
|
||||
```bash
|
||||
choco install k9s
|
||||
```
|
||||
|
||||
**On macOS:**
|
||||
```bash
|
||||
brew install k9s
|
||||
```
|
||||
|
||||
### Use k9s
|
||||
|
||||
```bash
|
||||
# Just run k9s - it will connect to your current kubectl context
|
||||
k9s
|
||||
|
||||
# Or specify namespace
|
||||
k9s -n nats-backend
|
||||
```
|
||||
|
||||
### k9s Keyboard Shortcuts
|
||||
|
||||
- `:pods` - View pods
|
||||
- `:svc` - View services
|
||||
- `:deploy` - View deployments
|
||||
- `:ns` - Switch namespace
|
||||
- `d` - Describe resource
|
||||
- `l` - View logs
|
||||
- `e` - Edit resource
|
||||
- `Ctrl+D` - Delete resource
|
||||
- `?` - Help
|
||||
- `q` - Quit
|
||||
|
||||
---
|
||||
|
||||
## Option 3: Lens (Desktop App) 💻
|
||||
|
||||
Lens is a powerful desktop application for Kubernetes management.
|
||||
|
||||
### Install Lens
|
||||
|
||||
Download from: https://k8slens.dev/
|
||||
|
||||
- **Windows:** Download installer from website
|
||||
- **Linux:** Download AppImage or .deb/.rpm
|
||||
- **macOS:** Download .dmg
|
||||
|
||||
### Connect to k3s
|
||||
|
||||
1. Open Lens
|
||||
2. Click "Add Cluster"
|
||||
3. Paste your kubeconfig (from `/etc/rancher/k3s/k3s.yaml` on server)
|
||||
4. Or Lens can auto-detect k3s if running locally
|
||||
|
||||
---
|
||||
|
||||
## Option 4: Rancher UI (For k3s)
|
||||
|
||||
Since you're using k3s (from Rancher), you can also use Rancher UI.
|
||||
|
||||
### Install Rancher
|
||||
|
||||
```bash
|
||||
# Install Rancher (optional - adds overhead)
|
||||
helm repo add rancher-latest https://releases.rancher.com/server-charts/latest
|
||||
helm repo update
|
||||
kubectl create namespace cattle-system
|
||||
helm install rancher rancher-latest/rancher \
|
||||
--namespace cattle-system \
|
||||
--set hostname=rancher.yourdomain.com
|
||||
```
|
||||
|
||||
**Note:** Rancher is heavier and more complex. Only use if you need advanced features.
|
||||
|
||||
---
|
||||
|
||||
## Quick Comparison
|
||||
|
||||
| Tool | Type | Best For | Resource Usage |
|
||||
|------|------|----------|----------------|
|
||||
| **Kubernetes Dashboard** | Web UI | Visual overview, beginners | Medium |
|
||||
| **k9s** | Terminal | Fast operations, CLI lovers | Low |
|
||||
| **Lens** | Desktop | Full-featured, professional | Medium |
|
||||
| **Rancher** | Web UI | Multi-cluster, enterprise | High |
|
||||
|
||||
---
|
||||
|
||||
## Recommended Setup
|
||||
|
||||
For your use case, I recommend:
|
||||
|
||||
1. **Kubernetes Dashboard** - For web-based viewing and monitoring
|
||||
2. **k9s** - For quick terminal-based operations
|
||||
|
||||
Both can be used together!
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Dashboard not loading?
|
||||
|
||||
```bash
|
||||
# Check if dashboard is running
|
||||
kubectl get pods -n kubernetes-dashboard
|
||||
|
||||
# Check logs
|
||||
kubectl logs -n kubernetes-dashboard deployment/kubernetes-dashboard
|
||||
|
||||
# Restart dashboard
|
||||
kubectl rollout restart deployment/kubernetes-dashboard -n kubernetes-dashboard
|
||||
```
|
||||
|
||||
### Can't access dashboard?
|
||||
|
||||
- Ensure `kubectl proxy` is running (for Method 1)
|
||||
- Check firewall rules if accessing remotely
|
||||
- Verify port-forward is working: `kubectl get svc -n kubernetes-dashboard`
|
||||
|
||||
### Permission denied?
|
||||
|
||||
The dashboard has full cluster access via the `admin-user` service account. If you see permission errors, check:
|
||||
|
||||
```bash
|
||||
kubectl get clusterrolebinding admin-user
|
||||
kubectl get serviceaccount admin-user -n kubernetes-dashboard
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. Deploy the dashboard: `kubectl apply -f manifests/dashboard.yaml`
|
||||
2. Access it: `kubectl proxy` then visit the URL
|
||||
3. Explore your `nats-backend` namespace!
|
||||
|
||||
Enjoy your Kubernetes UI! 🎉
|
||||
|
||||
54
docs/DEPLOY.md
Normal file
54
docs/DEPLOY.md
Normal file
@@ -0,0 +1,54 @@
|
||||
# Simple Deployment Guide
|
||||
|
||||
## 🚀 One-Command Deploy
|
||||
|
||||
Just run this from the `kubernetes` folder:
|
||||
|
||||
```bash
|
||||
./build-and-deploy.sh
|
||||
```
|
||||
|
||||
This will:
|
||||
1. ✅ Build both Docker images (FastAPI + Worker)
|
||||
2. ✅ Deploy everything to Kubernetes
|
||||
3. ✅ Set up auto-scaling
|
||||
|
||||
## 📋 What Gets Deployed
|
||||
|
||||
- **FastAPI:** 4 pods initially (scales 4-20 based on load)
|
||||
- **Workers:** 2 pods initially (scales 2-10 based on load)
|
||||
- **Auto-scaling:** Enabled for both
|
||||
- **Pod Distribution:** Spreads across nodes automatically
|
||||
|
||||
## ✅ After Deployment
|
||||
|
||||
Check status:
|
||||
```bash
|
||||
kubectl get pods -n nats-backend
|
||||
kubectl get services -n nats-backend
|
||||
kubectl get hpa -n nats-backend
|
||||
```
|
||||
|
||||
View logs:
|
||||
```bash
|
||||
kubectl logs -f deployment/fastapi-backend -n nats-backend
|
||||
kubectl logs -f deployment/nats-worker -n nats-backend
|
||||
```
|
||||
|
||||
Test FastAPI:
|
||||
```bash
|
||||
kubectl port-forward -n nats-backend service/fastapi-backend 8000:80
|
||||
curl http://localhost:8000/health
|
||||
```
|
||||
|
||||
## 🔧 Configuration
|
||||
|
||||
All settings are in `manifests/`:
|
||||
- `secrets.yaml` - NATS credentials and external API
|
||||
- `fastapi-deployment.yaml` - FastAPI config
|
||||
- `worker-deployment.yaml` - Worker config
|
||||
|
||||
---
|
||||
|
||||
**That's it!** Everything is ready to deploy! 🎉
|
||||
|
||||
151
docs/DEPLOY_CHECKLIST.md
Normal file
151
docs/DEPLOY_CHECKLIST.md
Normal file
@@ -0,0 +1,151 @@
|
||||
# 🚀 Kubernetes Deployment Checklist
|
||||
|
||||
## ✅ Pre-Deployment Checklist
|
||||
|
||||
Before deploying, ensure:
|
||||
|
||||
1. **k3s is running:**
|
||||
```bash
|
||||
sudo systemctl status k3s
|
||||
# If not running:
|
||||
sudo systemctl start k3s
|
||||
```
|
||||
|
||||
2. **Docker images are built and imported:**
|
||||
```bash
|
||||
# Build images
|
||||
docker build -t fastapi-backend:latest --target api -f Dockerfile .
|
||||
docker build -t nats-worker:latest --target worker -f Dockerfile .
|
||||
|
||||
# Import to containerd (k3s uses containerd, not Docker)
|
||||
docker save fastapi-backend:latest | sudo k3s ctr images import -
|
||||
docker save nats-worker:latest | sudo k3s ctr images import -
|
||||
```
|
||||
|
||||
3. **NATS JetStream stream is created:**
|
||||
```bash
|
||||
# Run the setup script
|
||||
./setup-jetstream.sh
|
||||
# Or manually:
|
||||
python3 scripts/setup_jetstream.py
|
||||
```
|
||||
|
||||
4. **kubectl is configured:**
|
||||
```bash
|
||||
# On your local machine, ensure kubectl points to k3s
|
||||
kubectl get nodes
|
||||
```
|
||||
|
||||
## 🚀 Deployment Steps
|
||||
|
||||
### Option 1: Simple Deployment (Recommended)
|
||||
```bash
|
||||
cd kubernetes
|
||||
chmod +x simple-deploy.sh
|
||||
./simple-deploy.sh
|
||||
```
|
||||
|
||||
### Option 2: Manual Deployment
|
||||
```bash
|
||||
cd kubernetes/manifests
|
||||
|
||||
# 1. Create namespace
|
||||
kubectl apply -f namespace.yaml
|
||||
|
||||
# 2. Create secrets
|
||||
kubectl apply -f secrets.yaml
|
||||
|
||||
# 3. Deploy FastAPI
|
||||
kubectl apply -f fastapi-deployment.yaml
|
||||
kubectl apply -f fastapi-service.yaml
|
||||
kubectl apply -f fastapi-hpa.yaml
|
||||
|
||||
# 4. Deploy Workers
|
||||
kubectl apply -f worker-deployment.yaml
|
||||
kubectl apply -f worker-hpa.yaml
|
||||
|
||||
# 5. Deploy Gateway (optional)
|
||||
kubectl apply -f gateway.yaml
|
||||
```
|
||||
|
||||
## 🔍 Verify Deployment
|
||||
|
||||
```bash
|
||||
# Check pods
|
||||
kubectl get pods -n nats-backend
|
||||
|
||||
# Check services
|
||||
kubectl get svc -n nats-backend
|
||||
|
||||
# Check Gateway
|
||||
kubectl get gateway -n nats-backend
|
||||
|
||||
# Watch pods in real-time
|
||||
kubectl get pods -n nats-backend -w
|
||||
|
||||
# Check logs
|
||||
kubectl logs -f deployment/fastapi-backend -n nats-backend
|
||||
kubectl logs -f deployment/nats-worker -n nats-backend
|
||||
```
|
||||
|
||||
## 🌐 Access Your Application
|
||||
|
||||
**Gateway Ports:**
|
||||
- HTTP: Port `8201`
|
||||
- HTTPS: Port `8441`
|
||||
|
||||
**Note:** The Gateway uses non-standard ports (8201/8441) to avoid conflicts with Traefik on port 80/443.
|
||||
|
||||
**To access via Gateway:**
|
||||
```bash
|
||||
# Port forward to test locally
|
||||
kubectl port-forward -n nats-backend svc/fastapi-backend 8000:8000
|
||||
|
||||
# Then test:
|
||||
curl http://localhost:8000/health
|
||||
```
|
||||
|
||||
## ⚠️ Troubleshooting
|
||||
|
||||
### Pods not starting?
|
||||
```bash
|
||||
# Check pod status
|
||||
kubectl describe pod <pod-name> -n nats-backend
|
||||
|
||||
# Check events
|
||||
kubectl get events -n nats-backend --sort-by='.lastTimestamp'
|
||||
```
|
||||
|
||||
### ImagePullBackOff error?
|
||||
- Ensure images are imported to containerd (see step 2 above)
|
||||
- Check `imagePullPolicy: Never` in deployments
|
||||
|
||||
### Worker pods crashing?
|
||||
- Ensure JetStream stream is created (see step 3 above)
|
||||
- Check NATS connection: `kubectl logs deployment/nats-worker -n nats-backend`
|
||||
|
||||
### Gateway not working?
|
||||
- Gateway requires cert-manager for TLS (optional)
|
||||
- HTTP should work without cert-manager
|
||||
- Check Gateway status: `kubectl describe gateway api-gateway -n nats-backend`
|
||||
|
||||
## 📊 Monitoring
|
||||
|
||||
```bash
|
||||
# Check HPA status
|
||||
kubectl get hpa -n nats-backend
|
||||
|
||||
# Check resource usage
|
||||
kubectl top pods -n nats-backend
|
||||
```
|
||||
|
||||
## 🛑 Undeploy
|
||||
|
||||
```bash
|
||||
# Delete all resources
|
||||
kubectl delete namespace nats-backend
|
||||
|
||||
# Or delete individually
|
||||
kubectl delete -f manifests/
|
||||
```
|
||||
|
||||
108
docs/PHASE1_NODE_PLACEMENT_RUNBOOK.md
Normal file
108
docs/PHASE1_NODE_PLACEMENT_RUNBOOK.md
Normal file
@@ -0,0 +1,108 @@
|
||||
# Phase 1 Node Placement Runbook
|
||||
|
||||
This runbook matches the manifest changes in this repository.
|
||||
|
||||
## Goal
|
||||
|
||||
Separate Kubernetes workloads into two planes:
|
||||
|
||||
- app plane for public-facing services
|
||||
- worker plane for async NATS consumers
|
||||
|
||||
## Required node labels
|
||||
|
||||
Apply these labels to your nodes.
|
||||
|
||||
### App nodes
|
||||
|
||||
```powershell
|
||||
kubectl label node <app-node-1> node-role.workolik/app=true
|
||||
kubectl label node <app-node-2> node-role.workolik/app=true
|
||||
```
|
||||
|
||||
### Worker nodes
|
||||
|
||||
```powershell
|
||||
kubectl label node <worker-node-1> node-role.workolik/worker=true
|
||||
kubectl label node <worker-node-2> node-role.workolik/worker=true
|
||||
```
|
||||
|
||||
## Recommended taints
|
||||
|
||||
These taints keep worker jobs away from app nodes and allow only matching workloads onto the correct plane.
|
||||
|
||||
### Worker nodes
|
||||
|
||||
```powershell
|
||||
kubectl taint node <worker-node-1> dedicated=workers:NoSchedule
|
||||
kubectl taint node <worker-node-2> dedicated=workers:NoSchedule
|
||||
```
|
||||
|
||||
### App nodes
|
||||
|
||||
```powershell
|
||||
kubectl taint node <app-node-1> dedicated=apps:NoSchedule
|
||||
kubectl taint node <app-node-2> dedicated=apps:NoSchedule
|
||||
```
|
||||
|
||||
## What the updated manifests now expect
|
||||
|
||||
### Worker plane
|
||||
|
||||
These workloads now require worker-node placement:
|
||||
|
||||
- [`manifests/core/workers.yaml`](E:/Birock/kubernetes/manifests/core/workers.yaml)
|
||||
- [`manifests/core/worker-statefulset.yaml`](E:/Birock/kubernetes/manifests/core/worker-statefulset.yaml)
|
||||
|
||||
They now use:
|
||||
|
||||
- required node affinity for `node-role.workolik/worker=true`
|
||||
- toleration for `dedicated=workers:NoSchedule`
|
||||
- pod anti-affinity
|
||||
- topology spread constraints
|
||||
|
||||
### App plane
|
||||
|
||||
These workloads now require app-node placement:
|
||||
|
||||
- [`manifests/alaska/alaska.yaml`](E:/Birock/kubernetes/manifests/alaska/alaska.yaml)
|
||||
- [`manifests/nearle/nearle-ariane.yaml`](E:/Birock/kubernetes/manifests/nearle/nearle-ariane.yaml)
|
||||
- [`manifests/nearle/nearle-atlantis.yaml`](E:/Birock/kubernetes/manifests/nearle/nearle-atlantis.yaml)
|
||||
- [`manifests/nearle/nearle-fiesta.yaml`](E:/Birock/kubernetes/manifests/nearle/nearle-fiesta.yaml)
|
||||
- [`manifests/nearle/nearle-jupiter.yaml`](E:/Birock/kubernetes/manifests/nearle/nearle-jupiter.yaml)
|
||||
- [`manifests/nearle/nearle-titan.yaml`](E:/Birock/kubernetes/manifests/nearle/nearle-titan.yaml)
|
||||
|
||||
They now use:
|
||||
|
||||
- required node affinity for `node-role.workolik/app=true`
|
||||
- toleration for `dedicated=apps:NoSchedule`
|
||||
- pod anti-affinity
|
||||
- topology spread constraints
|
||||
|
||||
## Safe rollout order
|
||||
|
||||
1. Label nodes.
|
||||
2. Taint nodes.
|
||||
3. Confirm labels and taints:
|
||||
|
||||
```powershell
|
||||
kubectl get nodes --show-labels
|
||||
kubectl describe node <worker-node-1>
|
||||
kubectl describe node <app-node-1>
|
||||
```
|
||||
|
||||
4. Apply manifests.
|
||||
5. Watch rescheduling:
|
||||
|
||||
```powershell
|
||||
kubectl get pods -A -o wide
|
||||
```
|
||||
|
||||
## Important warning
|
||||
|
||||
Do not apply these manifests until your cluster has at least:
|
||||
|
||||
- one labeled app node
|
||||
- one labeled worker node
|
||||
|
||||
If the labels do not exist yet, pods with required node affinity will stay Pending.
|
||||
408
docs/PROFESSIONAL_K3S_TARGET_ARCHITECTURE.md
Normal file
408
docs/PROFESSIONAL_K3S_TARGET_ARCHITECTURE.md
Normal file
@@ -0,0 +1,408 @@
|
||||
# Professional k3s Target Architecture
|
||||
|
||||
This document is a practical upgrade path from the current repo layout to a more professional, resilient platform.
|
||||
|
||||
It is written for the current setup in this repository:
|
||||
|
||||
- k3s is being used as the Kubernetes distribution.
|
||||
- ingress/proxy traffic is still partly handled outside the cluster with Docker Compose and nginx.
|
||||
- app workloads are running in Kubernetes.
|
||||
- workers depend on NATS/JetStream.
|
||||
- some services still use `NodePort`.
|
||||
- the core workloads point to a single external NATS IP.
|
||||
|
||||
## 1. What the current repo already does well
|
||||
|
||||
There is already a good foundation here:
|
||||
|
||||
- API and worker responsibilities are separated.
|
||||
- asynchronous processing with NATS/JetStream is the right pattern.
|
||||
- workers are split by business area in [`manifests/core/workers.yaml`](E:/Birock/kubernetes/manifests/core/workers.yaml).
|
||||
- you already have some resource requests and limits.
|
||||
- you already use PodDisruptionBudget for one worker set.
|
||||
- you already think in namespaces such as `core` and `nearle`.
|
||||
|
||||
That means you are not starting from zero. You are mostly at the stage of cleaning up platform boundaries and removing single points of failure.
|
||||
|
||||
## 2. Main problems in the current architecture
|
||||
|
||||
### Problem A: single point of failure in messaging
|
||||
|
||||
Your core workloads use one external NATS endpoint:
|
||||
|
||||
- [`manifests/core/core-config.yaml#L10`](E:/Birock/kubernetes/manifests/core/core-config.yaml#L10)
|
||||
|
||||
If that node or disk fails, the Kubernetes pods can still be healthy but the platform is still down.
|
||||
|
||||
### Problem B: node failover is not immediate today
|
||||
|
||||
You want "if current node is not working, switch immediately to backup node". Right now that is not really true because:
|
||||
|
||||
- some traffic still depends on host-level Docker Compose proxies
|
||||
- some services are exposed with `NodePort`
|
||||
- some workloads use `hostPath`
|
||||
- there is no clear control-plane and worker-plane separation
|
||||
- there is no shared HA datastore for k3s control-plane state
|
||||
|
||||
For example:
|
||||
|
||||
- [`manifests/nearle/nearle-jupiter.yaml#L58`](E:/Birock/kubernetes/manifests/nearle/nearle-jupiter.yaml#L58) exposes `jupiter` with `NodePort`
|
||||
- [`manifests/nearle/nearle-jupiter.yaml#L43`](E:/Birock/kubernetes/manifests/nearle/nearle-jupiter.yaml#L43) uses `hostPath`
|
||||
|
||||
`NodePort` is not wrong, but it is usually not the best long-term edge pattern for a professional HA setup.
|
||||
|
||||
### Problem C: CPU spikes can still hurt the whole node
|
||||
|
||||
Your workers are separated logically, but they still share the same physical node resources unless you explicitly isolate them with:
|
||||
|
||||
- dedicated worker nodes
|
||||
- taints and tolerations
|
||||
- node labels and node affinity
|
||||
- tighter requests/limits
|
||||
- autoscaling based on the right metrics
|
||||
|
||||
Some workers also have relatively high concurrency values:
|
||||
|
||||
- [`manifests/core/workers.yaml#L42`](E:/Birock/kubernetes/manifests/core/workers.yaml#L42)
|
||||
- [`manifests/core/workers.yaml#L192`](E:/Birock/kubernetes/manifests/core/workers.yaml#L192)
|
||||
|
||||
If the external target slows down or retries increase, these workers can create CPU and network pressure.
|
||||
|
||||
### Problem D: secrets are stored in repo manifests
|
||||
|
||||
There are inline credentials here:
|
||||
|
||||
- [`manifests/core/core-secrets.yaml`](E:/Birock/kubernetes/manifests/core/core-secrets.yaml)
|
||||
|
||||
For a professional setup, secrets should move to a proper secret manager or at least be injected outside git.
|
||||
|
||||
## 3. What "professional architecture" should look like here
|
||||
|
||||
For your use case, the clean target is not "one main node and one cold backup node".
|
||||
|
||||
The cleaner target is:
|
||||
|
||||
1. Highly available k3s control plane
|
||||
2. Separate worker nodes for workloads
|
||||
3. NATS deployed as an HA cluster
|
||||
4. Ingress handled inside Kubernetes
|
||||
5. Backups for both cluster state and NATS data
|
||||
6. Workload placement rules so noisy workers do not affect all services
|
||||
|
||||
## 4. Recommended target layout
|
||||
|
||||
### Minimum professional layout
|
||||
|
||||
- `3` k3s server nodes
|
||||
- `2` workload worker nodes
|
||||
- `3` NATS pods with JetStream replication
|
||||
- `1` in-cluster ingress controller
|
||||
- `1` backup system for cluster resources and persistent volumes
|
||||
|
||||
### Example node roles
|
||||
|
||||
- `cp-1`: k3s server
|
||||
- `cp-2`: k3s server
|
||||
- `cp-3`: k3s server
|
||||
- `app-1`: app services and ingress
|
||||
- `wrk-1`: NATS workers and heavy async jobs
|
||||
|
||||
Better:
|
||||
|
||||
- `app-1`, `app-2`: app/service nodes
|
||||
- `wrk-1`, `wrk-2`: dedicated worker nodes
|
||||
|
||||
### Planes
|
||||
|
||||
Control plane:
|
||||
|
||||
- k3s server nodes only
|
||||
- no application workloads if possible
|
||||
|
||||
Service plane:
|
||||
|
||||
- FastAPI / business services
|
||||
- ingress controller
|
||||
- gateway
|
||||
- observability stack
|
||||
|
||||
Worker plane:
|
||||
|
||||
- NATS worker consumers
|
||||
- batch jobs
|
||||
- CPU-heavy or retry-heavy services
|
||||
|
||||
Messaging plane:
|
||||
|
||||
- NATS cluster with JetStream
|
||||
- dedicated storage
|
||||
|
||||
## 5. Best failover model for your requirement
|
||||
|
||||
You asked for immediate switch to a backup node if one node fails.
|
||||
|
||||
There are two ways to think about that:
|
||||
|
||||
### Option 1: active-passive node failover
|
||||
|
||||
This means one main node and one backup node waiting.
|
||||
|
||||
This is simpler to understand, but it is still not the best Kubernetes design because:
|
||||
|
||||
- one active node is still a bottleneck
|
||||
- failover is not truly instant
|
||||
- stateful components are harder to fail over cleanly
|
||||
- you will still need shared storage or replicated state
|
||||
|
||||
### Option 2: active-active cluster with multiple nodes
|
||||
|
||||
This is the better professional design.
|
||||
|
||||
Instead of "switching to backup", Kubernetes simply reschedules workloads onto healthy nodes because:
|
||||
|
||||
- services already run across more than one node
|
||||
- ingress already points at the cluster, not one host
|
||||
- NATS data is replicated
|
||||
- pods have anti-affinity and replicas across nodes
|
||||
|
||||
This is the model I recommend for you.
|
||||
|
||||
## 6. Recommended production architecture for this repo
|
||||
|
||||
### Edge and ingress
|
||||
|
||||
Move ingress fully into Kubernetes:
|
||||
|
||||
- install Traefik or nginx ingress inside k3s
|
||||
- stop depending on host-level Docker Compose nginx proxies for production routing
|
||||
- expose ingress through a proper load balancer or a floating IP/VIP
|
||||
|
||||
Good choices:
|
||||
|
||||
- `kube-vip` for virtual IP failover on bare metal
|
||||
- `MetalLB` for service load balancers on bare metal
|
||||
|
||||
This is the clean replacement for current host-side proxying in [`docker-compose.yml`](E:/Birock/kubernetes/docker-compose.yml).
|
||||
|
||||
### Kubernetes control plane
|
||||
|
||||
Use HA k3s server nodes:
|
||||
|
||||
- `3` k3s server nodes
|
||||
- external datastore or embedded etcd in HA mode
|
||||
|
||||
For small-to-medium production, HA k3s with embedded etcd is often enough and simpler than trying to maintain a single-node server plus a backup.
|
||||
|
||||
### App services
|
||||
|
||||
Run FastAPI and business services on app nodes:
|
||||
|
||||
- Deployment instead of StatefulSet unless stable pod identity is required
|
||||
- `replicas >= 2`
|
||||
- topology spread constraints
|
||||
- anti-affinity across nodes
|
||||
- proper readiness and liveness probes
|
||||
|
||||
### Worker services
|
||||
|
||||
Run workers only on worker nodes:
|
||||
|
||||
- label worker nodes, for example `workload-type=async`
|
||||
- taint worker nodes
|
||||
- add tolerations to worker pods
|
||||
- add node affinity so worker pods land only there
|
||||
|
||||
This gives you the "separate worker plane" you asked for.
|
||||
|
||||
### NATS
|
||||
|
||||
Deploy NATS inside Kubernetes as an HA cluster, not as a single external endpoint.
|
||||
|
||||
Use:
|
||||
|
||||
- `3` NATS pods
|
||||
- JetStream replication factor `3`
|
||||
- persistent volumes
|
||||
- pod anti-affinity
|
||||
- dedicated node pool if possible
|
||||
|
||||
This is the biggest architecture improvement you can make for service survival.
|
||||
|
||||
### Backups
|
||||
|
||||
Use two backup layers:
|
||||
|
||||
1. Kubernetes resource/state backup
|
||||
2. JetStream or persistent volume backup
|
||||
|
||||
Recommended tools:
|
||||
|
||||
- Velero for cluster resource backup and restore
|
||||
- CSI snapshots or storage-level snapshots for persistent volumes
|
||||
- scheduled export/backup for critical NATS data if needed
|
||||
|
||||
## 7. How to separate service plane and worker plane
|
||||
|
||||
This is a very good idea for your stack.
|
||||
|
||||
### Service plane should host
|
||||
|
||||
- ingress controller
|
||||
- FastAPI/API gateway
|
||||
- frontend-facing services
|
||||
- dashboard/observability tools
|
||||
|
||||
### Worker plane should host
|
||||
|
||||
- NATS consumers
|
||||
- retry-heavy jobs
|
||||
- long-running async processors
|
||||
- any CPU-heavy integration jobs
|
||||
|
||||
### Basic implementation pattern
|
||||
|
||||
On nodes:
|
||||
|
||||
- label app nodes: `node-role.workolik/app=true`
|
||||
- label worker nodes: `node-role.workolik/worker=true`
|
||||
|
||||
Optionally taint worker nodes:
|
||||
|
||||
- `dedicated=workers:NoSchedule`
|
||||
|
||||
Then:
|
||||
|
||||
- app manifests use node affinity for `app=true`
|
||||
- worker manifests use node affinity and toleration for worker nodes
|
||||
|
||||
## 8. CPU spike reduction strategy
|
||||
|
||||
The CPU spike problem is usually not solved by "adding one backup node". It is solved by isolation, limits, and scaling.
|
||||
|
||||
### Do these first
|
||||
|
||||
1. Put workers on separate nodes.
|
||||
2. Tighten worker CPU limits and requests based on real usage.
|
||||
3. Reduce high default concurrency for the noisiest workers.
|
||||
4. Add HPA or KEDA scaling from queue depth, not only CPU.
|
||||
5. Ensure retries do not cause synchronized storms.
|
||||
|
||||
### Very likely spike sources in this repo
|
||||
|
||||
- high worker concurrency
|
||||
- many worker StatefulSets sharing the same node
|
||||
- retries against slow external APIs
|
||||
- host-level proxying plus cluster-level routing mix
|
||||
- single-node k3s carrying ingress, apps, workers, and maybe NATS responsibilities together
|
||||
|
||||
### Better autoscaling choice
|
||||
|
||||
For queue workers, KEDA is often better than plain HPA because it can scale on:
|
||||
|
||||
- NATS lag
|
||||
- queue depth
|
||||
- custom Prometheus metrics
|
||||
|
||||
That is usually more useful than only scaling from CPU percentage.
|
||||
|
||||
## 9. Concrete migration path
|
||||
|
||||
### Phase 1: stabilize current cluster
|
||||
|
||||
Do this before any big redesign:
|
||||
|
||||
- move secrets out of git
|
||||
- standardize on ingress instead of many host nginx proxies
|
||||
- remove unnecessary `NodePort` exposure where possible
|
||||
- add resource dashboards and alerting
|
||||
- capture actual CPU and memory usage for each worker
|
||||
|
||||
### Phase 2: separate worker nodes
|
||||
|
||||
- add a second or third node
|
||||
- label and taint worker nodes
|
||||
- move worker workloads there
|
||||
- keep app services on separate nodes
|
||||
|
||||
This alone will already reduce blast radius from worker CPU spikes.
|
||||
|
||||
### Phase 3: make k3s highly available
|
||||
|
||||
- create `3` k3s server nodes
|
||||
- use HA embedded etcd
|
||||
- put ingress behind `kube-vip` or `MetalLB`
|
||||
|
||||
Now loss of one server does not mean cluster loss.
|
||||
|
||||
### Phase 4: make NATS highly available
|
||||
|
||||
- deploy NATS cluster inside Kubernetes
|
||||
- enable JetStream replication
|
||||
- use persistent volumes
|
||||
- update workloads to connect to in-cluster NATS service
|
||||
|
||||
This replaces the current single external NATS dependency from [`manifests/core/core-config.yaml#L10`](E:/Birock/kubernetes/manifests/core/core-config.yaml#L10).
|
||||
|
||||
### Phase 5: add backup and restore
|
||||
|
||||
- install Velero
|
||||
- schedule cluster backups
|
||||
- schedule volume snapshots
|
||||
- test restore into a fresh environment
|
||||
|
||||
If restore has never been tested, backup is not yet reliable.
|
||||
|
||||
## 10. What I would choose for you
|
||||
|
||||
Because you said you are a beginner, I would not jump straight into a very large platform.
|
||||
|
||||
I would choose this as the practical target:
|
||||
|
||||
- `3` k3s server nodes
|
||||
- `2` worker nodes
|
||||
- in-cluster Traefik
|
||||
- `MetalLB` or `kube-vip`
|
||||
- NATS HA cluster with JetStream replication
|
||||
- Velero backups
|
||||
- node separation for app plane and worker plane
|
||||
|
||||
This is modern, realistic, and still manageable.
|
||||
|
||||
## 11. What I would not recommend
|
||||
|
||||
I would avoid these patterns for your next version:
|
||||
|
||||
- one main Kubernetes node plus one cold backup node as the final design
|
||||
- storing secrets directly in yaml in git
|
||||
- relying on `NodePort` as the main production exposure pattern
|
||||
- mixing host Docker Compose production routing with cluster routing long-term
|
||||
- keeping NATS as a single external IP with no clear HA story
|
||||
|
||||
## 12. Immediate next actions for this repo
|
||||
|
||||
If we continue from this document, the best implementation order is:
|
||||
|
||||
1. Add node placement rules for workers and apps.
|
||||
2. Convert external exposure to a single in-cluster ingress pattern.
|
||||
3. Remove hard dependence on the single external NATS IP.
|
||||
4. Add observability for CPU, memory, restart count, and queue lag.
|
||||
5. Introduce backup tooling and test restore.
|
||||
|
||||
## 13. Summary in simple words
|
||||
|
||||
The clean professional version of your stack is:
|
||||
|
||||
- Kubernetes control plane on multiple HA server nodes
|
||||
- app services on one node group
|
||||
- NATS workers on another node group
|
||||
- NATS itself running as a replicated cluster
|
||||
- ingress and failover handled at cluster level, not manually by switching servers
|
||||
- backups for both Kubernetes state and message data
|
||||
|
||||
That gives you what you want:
|
||||
|
||||
- less CPU blast radius
|
||||
- better reliability
|
||||
- faster failover
|
||||
- cleaner separation of responsibilities
|
||||
- a more modern production setup
|
||||
43
docs/QUICK_START.md
Normal file
43
docs/QUICK_START.md
Normal file
@@ -0,0 +1,43 @@
|
||||
# Quick Start - Kubernetes in 5 Minutes
|
||||
|
||||
## 🎯 Goal
|
||||
Run Kubernetes locally in containers and deploy your apps.
|
||||
|
||||
## ⚡ Fast Track
|
||||
|
||||
### 1. Install k3d (if not using Docker Desktop)
|
||||
```bash
|
||||
# Windows: Download from https://k3d.io/
|
||||
# Or use Docker Desktop Kubernetes (easier!)
|
||||
```
|
||||
|
||||
### 2. Create Cluster
|
||||
```bash
|
||||
k3d cluster create mycluster --servers 1 --agents 2
|
||||
```
|
||||
|
||||
### 3. Build Images
|
||||
```bash
|
||||
cd E:\nats\kubernetes
|
||||
docker build -f Dockerfile --target api -t fastapi-backend:latest .
|
||||
docker build -f Dockerfile --target worker -t nats-worker:latest .
|
||||
```
|
||||
|
||||
### 4. Update Manifests
|
||||
- `manifests/fastapi-deployment.yaml`: Change image to `fastapi-backend:latest`
|
||||
- `manifests/worker-deployment.yaml`: Change image to `nats-worker:latest`
|
||||
|
||||
### 5. Deploy
|
||||
```bash
|
||||
./deploy.sh
|
||||
```
|
||||
|
||||
### 6. Check
|
||||
```bash
|
||||
kubectl get pods -n nats-backend
|
||||
```
|
||||
|
||||
## 🎉 Done!
|
||||
|
||||
Your apps are running in Kubernetes!
|
||||
|
||||
138
docs/READY_TO_DEPLOY.md
Normal file
138
docs/READY_TO_DEPLOY.md
Normal file
@@ -0,0 +1,138 @@
|
||||
# ✅ Kubernetes Setup - Ready to Deploy!
|
||||
|
||||
## 🎯 Everything is Configured and Ready
|
||||
|
||||
### ✅ What's Set Up
|
||||
|
||||
1. **Kubernetes Manifests** (`kubernetes/manifests/`)
|
||||
- ✅ `namespace.yaml` - Namespace for your app
|
||||
- ✅ `secrets.yaml` - NATS credentials and external API config
|
||||
- ✅ `fastapi-deployment.yaml` - FastAPI backend (4 replicas, auto-scaling)
|
||||
- ✅ `fastapi-service.yaml` - Service on port 8000
|
||||
- ✅ `fastapi-hpa.yaml` - Horizontal Pod Autoscaler (4-20 pods)
|
||||
- ✅ `worker-deployment.yaml` - NATS workers (2 replicas, auto-scaling)
|
||||
- ✅ `worker-hpa.yaml` - Worker autoscaler (2-10 pods)
|
||||
- ✅ `gateway.yaml` - Gateway API (ports 8201/8441 - no conflict with Traefik)
|
||||
- ✅ `dashboard.yaml` - Kubernetes Dashboard UI
|
||||
|
||||
2. **Deployment Scripts**
|
||||
- ✅ `simple-deploy.sh` - One-command deployment
|
||||
- ✅ `deploy-dashboard.sh` - Dashboard deployment
|
||||
- ✅ `setup-jetstream.sh` - JetStream setup
|
||||
|
||||
3. **Configuration**
|
||||
- ✅ Port conflicts resolved (8201/8441 instead of 80/443)
|
||||
- ✅ Image pull policy set to `Never` (uses local containerd images)
|
||||
- ✅ NATS connection configured (external NATS server)
|
||||
- ✅ External API endpoints configured
|
||||
- ✅ HTTP methods correctly forwarded (PUT for update delivery)
|
||||
|
||||
4. **Documentation**
|
||||
- ✅ `DEPLOY_CHECKLIST.md` - Step-by-step deployment guide
|
||||
- ✅ `DASHBOARD_SETUP.md` - Dashboard access guide
|
||||
- ✅ `README.md` - Main documentation
|
||||
|
||||
## 🚀 Quick Deployment Steps
|
||||
|
||||
### Step 1: Start k3s (if not running)
|
||||
```bash
|
||||
sudo systemctl start k3s
|
||||
sudo systemctl status k3s
|
||||
```
|
||||
|
||||
### Step 2: Build and Import Docker Images
|
||||
```bash
|
||||
cd ~/kubernetes
|
||||
|
||||
# Build images
|
||||
docker build -t fastapi-backend:latest --target api -f Dockerfile .
|
||||
docker build -t nats-worker:latest --target worker -f Dockerfile .
|
||||
|
||||
# Import to containerd (k3s uses containerd)
|
||||
docker save fastapi-backend:latest | sudo k3s ctr images import -
|
||||
docker save nats-worker:latest | sudo k3s ctr images import -
|
||||
```
|
||||
|
||||
### Step 3: Setup JetStream
|
||||
```bash
|
||||
chmod +x setup-jetstream.sh
|
||||
./setup-jetstream.sh
|
||||
```
|
||||
|
||||
### Step 4: Deploy Everything
|
||||
```bash
|
||||
chmod +x simple-deploy.sh
|
||||
./simple-deploy.sh
|
||||
```
|
||||
|
||||
### Step 5: Deploy Dashboard (Optional)
|
||||
```bash
|
||||
chmod +x deploy-dashboard.sh
|
||||
./deploy-dashboard.sh
|
||||
```
|
||||
|
||||
## 🔍 Verify Deployment
|
||||
|
||||
```bash
|
||||
# Check pods
|
||||
kubectl get pods -n nats-backend
|
||||
|
||||
# Check services
|
||||
kubectl get svc -n nats-backend
|
||||
|
||||
# Check Gateway
|
||||
kubectl get gateway -n nats-backend
|
||||
|
||||
# Check HPA
|
||||
kubectl get hpa -n nats-backend
|
||||
```
|
||||
|
||||
## 🖥️ Access Dashboard
|
||||
|
||||
```bash
|
||||
# Start proxy
|
||||
kubectl proxy
|
||||
|
||||
# Visit in browser:
|
||||
# http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/
|
||||
```
|
||||
|
||||
Or use port-forward:
|
||||
```bash
|
||||
kubectl port-forward -n kubernetes-dashboard service/kubernetes-dashboard 9090:9090
|
||||
# Visit: http://localhost:9090
|
||||
```
|
||||
|
||||
## 📊 What You'll See in Dashboard
|
||||
|
||||
- **Namespaces:** `nats-backend`, `kubernetes-dashboard`
|
||||
- **Pods:** FastAPI pods, Worker pods, Dashboard pod
|
||||
- **Services:** FastAPI service, Dashboard service
|
||||
- **Deployments:** All your deployments
|
||||
- **HPA:** Autoscaling configurations
|
||||
- **Resources:** CPU, Memory usage
|
||||
- **Logs:** View logs from any pod
|
||||
- **Events:** Cluster events and errors
|
||||
|
||||
## ⚙️ Configuration Summary
|
||||
|
||||
| Component | Configuration |
|
||||
|-----------|--------------|
|
||||
| **FastAPI** | 4 initial pods, scales 4-20, port 8000 |
|
||||
| **Workers** | 2 initial pods, scales 2-10 |
|
||||
| **Gateway** | Ports 8201 (HTTP), 8441 (HTTPS) |
|
||||
| **NATS** | External: `nats://nats.workolik.com:4222` |
|
||||
| **Domain** | `queue.workolik.com` |
|
||||
| **External API** | `https://jupiter.nearle.app` |
|
||||
|
||||
## ✅ All Systems Ready!
|
||||
|
||||
Everything is configured and ready to deploy. Just follow the steps above!
|
||||
|
||||
**Need help?** Check:
|
||||
- `DEPLOY_CHECKLIST.md` - Detailed deployment guide
|
||||
- `DASHBOARD_SETUP.md` - Dashboard access options
|
||||
- `README.md` - Full documentation
|
||||
|
||||
🎉 **You're all set!**
|
||||
|
||||
152
docs/SETUP_LOCAL_K8S.md
Normal file
152
docs/SETUP_LOCAL_K8S.md
Normal file
@@ -0,0 +1,152 @@
|
||||
# Simple Kubernetes Setup Guide
|
||||
|
||||
## 🎯 What We're Doing
|
||||
|
||||
We'll run Kubernetes **inside containers** on your computer, then deploy your FastAPI + Worker apps to it.
|
||||
|
||||
Think of it like:
|
||||
- **Kubernetes** = A manager that runs your apps
|
||||
- **k3d** = Tool that runs Kubernetes in Docker containers
|
||||
- **Your Apps** = FastAPI and Worker that we deploy
|
||||
|
||||
## 📦 Step 1: Install k3d (Kubernetes in Docker)
|
||||
|
||||
k3d runs Kubernetes in Docker containers - perfect for learning!
|
||||
|
||||
**Windows (PowerShell):**
|
||||
```powershell
|
||||
# Install via Chocolatey (if you have it)
|
||||
choco install k3d
|
||||
|
||||
# OR download from: https://k3d.io/
|
||||
```
|
||||
|
||||
**Or use Docker Desktop with Kubernetes:**
|
||||
- Open Docker Desktop
|
||||
- Go to Settings → Kubernetes
|
||||
- Enable Kubernetes
|
||||
- Click "Apply & Restart"
|
||||
|
||||
## 🚀 Step 2: Create a 3-Node Kubernetes Cluster
|
||||
|
||||
```bash
|
||||
# Create cluster with 3 nodes (1 master + 2 workers)
|
||||
k3d cluster create mycluster --servers 1 --agents 2
|
||||
|
||||
# OR if using Docker Desktop Kubernetes, skip this step
|
||||
```
|
||||
|
||||
**What this does:**
|
||||
- Creates 3 containers running Kubernetes
|
||||
- 1 master node (controls everything)
|
||||
- 2 worker nodes (run your apps)
|
||||
|
||||
## ✅ Step 3: Verify Cluster is Running
|
||||
|
||||
```bash
|
||||
# Check if cluster is ready
|
||||
kubectl get nodes
|
||||
|
||||
# You should see 3 nodes:
|
||||
# NAME STATUS ROLES
|
||||
# k3d-mycluster-0 Ready control-plane
|
||||
# k3d-mycluster-1 Ready <none>
|
||||
# k3d-mycluster-2 Ready <none>
|
||||
```
|
||||
|
||||
## 🐳 Step 4: Build Your Docker Images
|
||||
|
||||
```bash
|
||||
# Go to your project folder
|
||||
cd E:\nats\kubernetes
|
||||
|
||||
# Build API image
|
||||
docker build -f Dockerfile --target api -t fastapi-backend:latest .
|
||||
|
||||
# Build Worker image
|
||||
docker build -f Dockerfile --target worker -t nats-worker:latest .
|
||||
```
|
||||
|
||||
**What this does:**
|
||||
- Builds your FastAPI app into a Docker image
|
||||
- Builds your Worker app into a Docker image
|
||||
- Stores them locally (k3d can use local images)
|
||||
|
||||
## 📝 Step 5: Update Image Names
|
||||
|
||||
Edit `manifests/fastapi-deployment.yaml`:
|
||||
- Change `your-registry/fastapi-backend:latest` → `fastapi-backend:latest`
|
||||
|
||||
Edit `manifests/worker-deployment.yaml`:
|
||||
- Change `your-registry/nats-worker:latest` → `nats-worker:latest`
|
||||
|
||||
## 🚀 Step 6: Deploy Everything
|
||||
|
||||
```bash
|
||||
# Make script executable (if on Linux/Mac)
|
||||
chmod +x deploy.sh
|
||||
|
||||
# Run deployment
|
||||
./deploy.sh
|
||||
|
||||
# OR manually:
|
||||
kubectl apply -f manifests/namespace.yaml
|
||||
kubectl apply -f manifests/secrets.yaml
|
||||
kubectl apply -f manifests/fastapi-deployment.yaml
|
||||
kubectl apply -f manifests/fastapi-service.yaml
|
||||
kubectl apply -f manifests/fastapi-hpa.yaml
|
||||
kubectl apply -f manifests/worker-deployment.yaml
|
||||
kubectl apply -f manifests/worker-hpa.yaml
|
||||
```
|
||||
|
||||
## ✅ Step 7: Check Everything is Running
|
||||
|
||||
```bash
|
||||
# See all your pods
|
||||
kubectl get pods -n nats-backend
|
||||
|
||||
# See pods spread across nodes
|
||||
kubectl get pods -n nats-backend -o wide
|
||||
|
||||
# Check if pods are running
|
||||
# You should see:
|
||||
# - 4 fastapi-backend pods
|
||||
# - 2 nats-worker pods
|
||||
```
|
||||
|
||||
## 🧪 Step 8: Test Your App
|
||||
|
||||
```bash
|
||||
# Forward port to access FastAPI
|
||||
kubectl port-forward -n nats-backend service/fastapi-backend 8000:80
|
||||
|
||||
# In another terminal, test:
|
||||
curl http://localhost:8000/health
|
||||
```
|
||||
|
||||
## 📊 Useful Commands
|
||||
|
||||
```bash
|
||||
# See everything
|
||||
kubectl get all -n nats-backend
|
||||
|
||||
# See logs
|
||||
kubectl logs -f deployment/fastapi-backend -n nats-backend
|
||||
|
||||
# See which node a pod is on
|
||||
kubectl get pods -n nats-backend -o wide
|
||||
|
||||
# Delete everything (if needed)
|
||||
kubectl delete namespace nats-backend
|
||||
```
|
||||
|
||||
## 🎯 Summary
|
||||
|
||||
1. **Install k3d** → Creates Kubernetes in containers
|
||||
2. **Create cluster** → 3 nodes ready
|
||||
3. **Build images** → Your apps as Docker images
|
||||
4. **Deploy** → Run `./deploy.sh`
|
||||
5. **Test** → Access your app
|
||||
|
||||
**That's it!** Your apps are now running in Kubernetes! 🎉
|
||||
|
||||
70
docs/SIMPLIFIED_FAILOVER_SETUP.md
Normal file
70
docs/SIMPLIFIED_FAILOVER_SETUP.md
Normal file
@@ -0,0 +1,70 @@
|
||||
# Simplified k3s Failover & Performance Setup
|
||||
|
||||
To fix your **CPU Spikes** and achieve **Resilient Failover** without building a massive cluster, follow this simple 2-node or 3-node plan.
|
||||
|
||||
## 1. The Strategy
|
||||
Instead of making one massive node do everything, we split the work:
|
||||
- **App Node:** Only runs your public APIs (`jupiter`, `fiesta`, `atlantis`, etc.).
|
||||
- **Worker Node:** Only runs the heavy background tasks (`worker-orders`, `worker-deliveries`, etc.).
|
||||
|
||||
This ensures that if a background worker spikes to 100% CPU, your **main website remains fast and healthy**.
|
||||
|
||||
---
|
||||
|
||||
## 2. Setting Up Your Planes (Placing Nodes Into Groups)
|
||||
|
||||
You need to "tell" Kubernetes which of your servers is which. Run these commands:
|
||||
|
||||
### Identify your Node names
|
||||
```bash
|
||||
kubectl get nodes
|
||||
```
|
||||
|
||||
### Label your nodes by their role
|
||||
Replace `<node-name>` with your actual server names (e.g., `server-1`, `server-2`).
|
||||
|
||||
#### A. Assign Node 1 as the "App Host"
|
||||
```bash
|
||||
kubectl label node <node-1> node-role.workolik/app=true
|
||||
kubectl taint node <node-1> dedicated=apps:NoSchedule
|
||||
```
|
||||
|
||||
#### B. Assign Node 2 as the "Worker Host"
|
||||
*(This node will take all the CPU spikes)*
|
||||
```bash
|
||||
kubectl label node <node-2> node-role.workolik/worker=true
|
||||
kubectl taint node <node-2> dedicated=workers:NoSchedule
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Simplified Ingress (Replaces Docker Compose)
|
||||
|
||||
You no longer need the complex Nginx proxies in your `docker-compose.yml`. I have created a single "Unified Ingress" that replaces all of them.
|
||||
|
||||
### Why this is better:
|
||||
- **Failover:** If an "App Node" fails, Kubernetes automatically moves your APIs to another node, and k3s's built-in Traefik handles the routing instantly.
|
||||
- **Simplicity:** One file manages all your domains (`queue.workolik.com`, `jupiter.nearle.app`, etc.).
|
||||
|
||||
### How to apply it:
|
||||
1. Apply the Middleware (CORS settings):
|
||||
```bash
|
||||
kubectl apply -f manifests/core/traefik-middlewares.yaml
|
||||
```
|
||||
2. Apply the Unified Ingress:
|
||||
```bash
|
||||
kubectl apply -f manifests/core/ingress-unified.yaml
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. Summary of Improvements
|
||||
1. **CPU Isolation:** Workers in `manifests/core/workers.yaml` now have strict limits and are "pushed" to the Worker Plane.
|
||||
2. **Professional Routing:** Moved from Host-side Nginx (manual failover) to K8s-side Traefik (automatic failover).
|
||||
3. **No NATS Changes:** We are still using your external NATS, but with the new Ingress, your services are much safer.
|
||||
|
||||
## 5. Next Practical Steps (When you are ready)
|
||||
- **High Availability (HA):** When you have 3 control-plane servers, k3s will survive even if a "Master" node reboots.
|
||||
- **Persistence:** Ensure any databases or disks aren't tied to a single machine's filesystem (`hostPath`).
|
||||
|
||||
**You can now stop using the `docker-compose.yml` for traffic routing once you point your DNS/Load Balancer to your k3s cluster IP.**
|
||||
1350
docs/TECH_STACK_ARCHITECTURE.txt
Normal file
1350
docs/TECH_STACK_ARCHITECTURE.txt
Normal file
File diff suppressed because it is too large
Load Diff
985
docs/TECH_STACK_ARCHITECTURE_INTERMEDIATE.txt
Normal file
985
docs/TECH_STACK_ARCHITECTURE_INTERMEDIATE.txt
Normal file
@@ -0,0 +1,985 @@
|
||||
================================================================================
|
||||
NATS MESSAGE QUEUE SYSTEM
|
||||
Intermediate-Level Guide
|
||||
================================================================================
|
||||
|
||||
TABLE OF CONTENTS
|
||||
-----------------
|
||||
1. What This System Does (Simple Explanation)
|
||||
2. The Big Picture - How Everything Works Together
|
||||
3. Each Component Explained Simply
|
||||
4. How Messages Flow Through the System
|
||||
5. Why We Use Each Technology
|
||||
6. How to Deploy and Use
|
||||
7. Common Questions Answered
|
||||
8. Troubleshooting Made Easy
|
||||
|
||||
================================================================================
|
||||
1. WHAT THIS SYSTEM DOES (SIMPLE EXPLANATION)
|
||||
================================================================================
|
||||
|
||||
Imagine you have a restaurant:
|
||||
- Customers place orders (HTTP requests)
|
||||
- Orders go to the kitchen queue (NATS message queue)
|
||||
- Chefs process orders (Workers)
|
||||
- Food gets delivered (Forwarded to external API)
|
||||
|
||||
OUR SYSTEM:
|
||||
-----------
|
||||
1. Your mobile app or website sends a request (like "update delivery status")
|
||||
2. FastAPI receives it and puts it in a message queue (NATS)
|
||||
3. Workers pick up messages from the queue
|
||||
4. Workers forward the request to the actual external API
|
||||
5. Everything is monitored and can scale automatically
|
||||
|
||||
WHY USE A QUEUE?
|
||||
---------------
|
||||
✓ If the external API is slow, your app doesn't wait
|
||||
✓ If the external API is down, messages wait in queue (won't be lost)
|
||||
✓ You can process many requests without overloading anything
|
||||
✓ Easy to add more workers if you have lots of messages
|
||||
|
||||
================================================================================
|
||||
2. THE BIG PICTURE - HOW EVERYTHING WORKS TOGETHER
|
||||
================================================================================
|
||||
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ SIMPLE FLOW DIAGRAM │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
[Your App]
|
||||
│
|
||||
│ Sends HTTP request: "Update delivery #123"
|
||||
▼
|
||||
[FastAPI Server]
|
||||
│
|
||||
│ Puts message in queue
|
||||
▼
|
||||
[NATS Message Queue]
|
||||
│
|
||||
│ Stores message safely
|
||||
│ (like a mailbox)
|
||||
▼
|
||||
[Worker Process]
|
||||
│
|
||||
│ Picks up message
|
||||
│ Forwards to external API
|
||||
▼
|
||||
[External API]
|
||||
│
|
||||
│ Processes the request
|
||||
▼
|
||||
[Done!]
|
||||
|
||||
COMPONENTS BREAKDOWN:
|
||||
---------------------
|
||||
|
||||
1. NATS = The Message Queue (like a post office)
|
||||
- Receives messages from FastAPI
|
||||
- Stores them safely
|
||||
- Gives them to workers when ready
|
||||
|
||||
2. FastAPI = The API Server (like a receptionist)
|
||||
- Receives requests from your app
|
||||
- Quickly puts them in the queue
|
||||
- Returns "OK, got it!" immediately
|
||||
|
||||
3. Workers = The Processors (like workers in a factory)
|
||||
- Take messages from queue
|
||||
- Forward to external API
|
||||
- Handle retries if something fails
|
||||
|
||||
4. Nginx = The Helper (like a translator)
|
||||
- Helps the dashboard talk to NATS
|
||||
- Adds security headers (CORS)
|
||||
|
||||
5. Dashboard = The Monitor (like a control panel)
|
||||
- Shows you what's happening
|
||||
- See how many messages are waiting
|
||||
- Monitor if everything is working
|
||||
|
||||
================================================================================
|
||||
3. EACH COMPONENT EXPLAINED SIMPLY
|
||||
================================================================================
|
||||
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ 3.1 NATS - THE MESSAGE QUEUE │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
WHAT IT IS:
|
||||
-----------
|
||||
Think of NATS like a smart mailbox system:
|
||||
- FastAPI puts messages in
|
||||
- Workers take messages out
|
||||
- Messages are stored safely (won't be lost)
|
||||
- Can handle millions of messages
|
||||
|
||||
WHY WE USE IT:
|
||||
-------------
|
||||
✓ Super fast (messages delivered in less than 1 millisecond)
|
||||
✓ Reliable (messages won't disappear)
|
||||
✓ Simple to use (no complicated setup)
|
||||
✓ Lightweight (doesn't need much resources)
|
||||
|
||||
HOW IT WORKS:
|
||||
-------------
|
||||
1. FastAPI publishes a message: "Here's a delivery update"
|
||||
2. NATS stores it in a "stream" (like a folder)
|
||||
3. Worker asks: "Any messages for me?"
|
||||
4. NATS gives worker the message
|
||||
5. Worker processes it and says "Done!" (ACK)
|
||||
6. NATS removes the message from queue
|
||||
|
||||
REAL EXAMPLE:
|
||||
-------------
|
||||
Message looks like this:
|
||||
{
|
||||
"endpoint": "/live/api/v1/deliveries/updatedelivery",
|
||||
"method": "PUT",
|
||||
"data": {"delivery_id": 123, "status": "delivered"},
|
||||
"received_at": 1234567890
|
||||
}
|
||||
|
||||
This gets stored in NATS and workers pick it up.
|
||||
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ 3.2 FASTAPI - THE API SERVER │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
WHAT IT IS:
|
||||
-----------
|
||||
FastAPI is a Python web server that:
|
||||
- Listens for HTTP requests
|
||||
- Validates the data
|
||||
- Puts messages in NATS queue
|
||||
- Returns response immediately
|
||||
|
||||
WHY WE USE IT:
|
||||
-------------
|
||||
✓ Fast (handles many requests per second)
|
||||
✓ Easy to write (Python is simple)
|
||||
✓ Automatic documentation (shows all endpoints)
|
||||
✓ Built-in validation (catches bad data)
|
||||
|
||||
HOW IT WORKS:
|
||||
-------------
|
||||
1. Your app sends: PUT /live/api/v1/deliveries/updatedelivery
|
||||
2. FastAPI receives it
|
||||
3. Checks if data is valid
|
||||
4. Creates a message with all the info
|
||||
5. Publishes to NATS (super fast, < 1ms)
|
||||
6. Returns: {"status": "accepted", "message_id": 12345}
|
||||
|
||||
ENDPOINTS WE HAVE:
|
||||
------------------
|
||||
- PUT /live/api/v1/deliveries/updatedelivery
|
||||
- POST /live/api/v1/deliveries/createdeliveries
|
||||
- POST /live/api/v2/partners/createriderlog
|
||||
- POST /live/api/v2/deliveries/createdeliverylog
|
||||
- POST /live/api/v2/partners/createbreaklog
|
||||
- POST /live/api/v2/partners/updatebreaklog
|
||||
|
||||
Each endpoint does the same thing: receives request → puts in queue → returns OK
|
||||
|
||||
HEALTH CHECKS:
|
||||
--------------
|
||||
- GET /health → Is the server running? (Yes/No)
|
||||
- GET /ready → Is NATS connected? (Yes/No)
|
||||
- GET /metrics → Statistics (for monitoring)
|
||||
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ 3.3 WORKERS - THE MESSAGE PROCESSORS │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
WHAT IT IS:
|
||||
-----------
|
||||
Workers are background processes that:
|
||||
- Watch the NATS queue for new messages
|
||||
- Take messages out
|
||||
- Forward them to the external API
|
||||
- Handle errors and retries
|
||||
|
||||
WHY WE USE THEM:
|
||||
---------------
|
||||
✓ Keeps your API fast (doesn't wait for external API)
|
||||
✓ Handles failures automatically (retries if API is down)
|
||||
✓ Can scale independently (add more workers if needed)
|
||||
✓ Separates concerns (API vs processing)
|
||||
|
||||
HOW IT WORKS:
|
||||
-------------
|
||||
1. Worker asks NATS: "Any messages?"
|
||||
2. NATS gives worker a batch (up to 10 messages)
|
||||
3. For each message:
|
||||
a. Reads the endpoint and data
|
||||
b. Maps to external API URL
|
||||
c. Sends HTTP request (PUT or POST)
|
||||
d. Waits for response
|
||||
4. If success: Tells NATS "Done!" (ACK) → message removed
|
||||
5. If failure: Tells NATS "Try again" (NAK) → message stays in queue
|
||||
|
||||
RETRY LOGIC:
|
||||
------------
|
||||
If external API fails:
|
||||
- Wait 5 seconds, try again
|
||||
- If fails, wait 10 seconds, try again
|
||||
- If fails, wait 15 seconds, try again
|
||||
- Up to 5 attempts total
|
||||
- After 5 failures, message stays in queue for manual review
|
||||
|
||||
EXAMPLE:
|
||||
--------
|
||||
Message says: "Update delivery #123"
|
||||
Worker:
|
||||
1. Reads message
|
||||
2. Maps to: https://jupiter.nearle.app/live/api/v1/deliveries/updatedelivery
|
||||
3. Sends PUT request with the data
|
||||
4. External API responds: "OK"
|
||||
5. Worker tells NATS: "Done!" → message removed
|
||||
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ 3.4 NGINX - THE HELPER PROXY │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
WHAT IT IS:
|
||||
-----------
|
||||
Nginx is a web server that acts as a middleman:
|
||||
- Sits between Dashboard and NATS monitoring
|
||||
- Adds security headers (CORS)
|
||||
- Handles browser security requirements
|
||||
|
||||
WHY WE NEED IT:
|
||||
---------------
|
||||
The NATS Dashboard runs in your browser (JavaScript).
|
||||
Browsers have security rules (CORS) that prevent websites from
|
||||
talking to other websites unless they have permission.
|
||||
|
||||
Nginx adds the permission headers so the dashboard can work.
|
||||
|
||||
HOW IT WORKS:
|
||||
-------------
|
||||
1. Dashboard (in browser) wants to check NATS status
|
||||
2. Browser blocks it (security rule)
|
||||
3. Nginx intercepts the request
|
||||
4. Adds headers: "Yes, dashboard is allowed to talk to NATS"
|
||||
5. Browser says "OK" and allows the request
|
||||
6. Dashboard gets the data it needs
|
||||
|
||||
SIMPLE ANALOGY:
|
||||
---------------
|
||||
Like a bouncer at a club:
|
||||
- Dashboard wants to enter (access NATS)
|
||||
- Bouncer (Nginx) checks the list
|
||||
- Adds your name to the VIP list (CORS headers)
|
||||
- You get in!
|
||||
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ 3.5 NATS DASHBOARD - THE MONITORING SCREEN │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
WHAT IT IS:
|
||||
-----------
|
||||
A web page that shows you:
|
||||
- How many messages are in the queue
|
||||
- How fast messages are being processed
|
||||
- If workers are running
|
||||
- Statistics and graphs
|
||||
|
||||
WHY WE USE IT:
|
||||
-------------
|
||||
✓ See what's happening in real-time
|
||||
✓ Know if something is wrong
|
||||
✓ Monitor performance
|
||||
✓ Debug issues
|
||||
|
||||
WHAT YOU CAN SEE:
|
||||
-----------------
|
||||
- Stream Statistics: How many messages total
|
||||
- Consumer Status: Are workers processing?
|
||||
- Message Rate: Messages per second
|
||||
- Pending Messages: How many waiting to be processed
|
||||
- Consumer Lag: How far behind are workers?
|
||||
|
||||
HOW TO ACCESS:
|
||||
--------------
|
||||
URL: https://natsadmin.workolik.com
|
||||
Username: admin
|
||||
Password: package@321#
|
||||
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ 3.6 DOCKER - CONTAINERIZATION │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
WHAT IT IS:
|
||||
-----------
|
||||
Docker packages everything into containers:
|
||||
- Like shipping containers for software
|
||||
- Each service runs in its own container
|
||||
- All containers work together
|
||||
|
||||
WHY WE USE IT:
|
||||
-------------
|
||||
✓ Same environment everywhere (dev, staging, production)
|
||||
✓ Easy to deploy (just run docker-compose up)
|
||||
✓ Isolated services (if one crashes, others keep running)
|
||||
✓ Easy to update (just rebuild container)
|
||||
|
||||
HOW IT WORKS:
|
||||
-------------
|
||||
docker-compose.yml defines:
|
||||
- NATS container (the message queue)
|
||||
- Nginx container (the proxy)
|
||||
- Dashboard container (the monitoring)
|
||||
|
||||
All containers talk to each other via a network.
|
||||
|
||||
SIMPLE ANALOGY:
|
||||
---------------
|
||||
Like apartments in a building:
|
||||
- Each apartment (container) is separate
|
||||
- But they share utilities (network)
|
||||
- You can move apartments (containers) easily
|
||||
- If one apartment has issues, others are fine
|
||||
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ 3.7 KUBERNETES - CONTAINER ORCHESTRATION │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
WHAT IT IS:
|
||||
-----------
|
||||
Kubernetes manages containers across multiple servers:
|
||||
- Like a manager for a team of workers
|
||||
- Automatically starts/stops containers
|
||||
- Spreads work across multiple servers
|
||||
- Auto-scales when busy
|
||||
|
||||
WHY WE USE IT:
|
||||
-------------
|
||||
✓ High availability (if one server dies, others take over)
|
||||
✓ Auto-scaling (adds more workers when busy)
|
||||
✓ Self-healing (restarts crashed containers)
|
||||
✓ Load balancing (spreads requests evenly)
|
||||
|
||||
HOW IT WORKS:
|
||||
-------------
|
||||
1. You define: "I want 4 FastAPI servers and 2 workers"
|
||||
2. Kubernetes creates them across different servers
|
||||
3. If one crashes, Kubernetes restarts it
|
||||
4. If load increases, Kubernetes adds more
|
||||
5. If load decreases, Kubernetes removes some
|
||||
|
||||
AUTO-SCALING EXAMPLE:
|
||||
---------------------
|
||||
Normal load: 4 FastAPI pods, 2 worker pods
|
||||
High load: Kubernetes sees CPU at 80%
|
||||
→ Adds more pods automatically
|
||||
→ Now: 10 FastAPI pods, 5 worker pods
|
||||
Low load: Kubernetes sees CPU at 20%
|
||||
→ Removes some pods
|
||||
→ Back to: 4 FastAPI pods, 2 worker pods
|
||||
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ 3.8 TRAEFIK - THE ROUTER │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
WHAT IT IS:
|
||||
-----------
|
||||
Traefik is like a smart receptionist:
|
||||
- Receives all incoming requests
|
||||
- Routes them to the right service
|
||||
- Handles HTTPS certificates automatically
|
||||
- Adds security (like password protection)
|
||||
|
||||
WHY WE USE IT:
|
||||
-------------
|
||||
✓ Automatic HTTPS (free SSL certificates)
|
||||
✓ Easy routing (just add labels to containers)
|
||||
✓ Handles authentication
|
||||
✓ One entry point for everything
|
||||
|
||||
HOW IT WORKS:
|
||||
-------------
|
||||
1. Request comes in: https://queue.workolik.com
|
||||
2. Traefik receives it
|
||||
3. Checks: "This goes to FastAPI"
|
||||
4. Routes to FastAPI service
|
||||
5. Returns response
|
||||
|
||||
HTTPS CERTIFICATES:
|
||||
-------------------
|
||||
Traefik automatically gets free SSL certificates from Let's Encrypt.
|
||||
You don't need to do anything - it just works!
|
||||
|
||||
AUTHENTICATION:
|
||||
---------------
|
||||
For the dashboard (natsadmin.workolik.com):
|
||||
- Traefik adds password protection
|
||||
- Username: admin
|
||||
- Password: package@321#
|
||||
|
||||
================================================================================
|
||||
4. HOW MESSAGES FLOW THROUGH THE SYSTEM
|
||||
================================================================================
|
||||
|
||||
STEP-BY-STEP EXAMPLE:
|
||||
---------------------
|
||||
|
||||
Let's say your app wants to update a delivery status.
|
||||
|
||||
STEP 1: YOUR APP SENDS REQUEST
|
||||
--------------------------------
|
||||
Your mobile app sends:
|
||||
PUT https://queue.workolik.com/live/api/v1/deliveries/updatedelivery
|
||||
Body: {"delivery_id": 123, "status": "delivered"}
|
||||
|
||||
STEP 2: TRAEFIK RECEIVES IT
|
||||
----------------------------
|
||||
Traefik sees: "This is for queue.workolik.com"
|
||||
Routes to: FastAPI service
|
||||
|
||||
STEP 3: FASTAPI PROCESSES IT
|
||||
-----------------------------
|
||||
FastAPI:
|
||||
- Receives the request
|
||||
- Validates the data (checks it's valid JSON)
|
||||
- Creates a message:
|
||||
{
|
||||
"endpoint": "/live/api/v1/deliveries/updatedelivery",
|
||||
"method": "PUT",
|
||||
"data": {"delivery_id": 123, "status": "delivered"},
|
||||
"received_at": 1234567890
|
||||
}
|
||||
- Publishes to NATS (super fast!)
|
||||
- Returns: {"status": "accepted", "message_id": 12345}
|
||||
|
||||
STEP 4: YOUR APP GETS RESPONSE
|
||||
-------------------------------
|
||||
Your app receives: "OK, got it!" (takes < 1 second)
|
||||
Your app doesn't wait for the external API - it's done!
|
||||
|
||||
STEP 5: NATS STORES MESSAGE
|
||||
---------------------------
|
||||
NATS:
|
||||
- Receives message
|
||||
- Stores in stream "EVENTS"
|
||||
- Subject: "api.v1.deliveries.updatedelivery"
|
||||
- Message is safe and won't be lost
|
||||
|
||||
STEP 6: WORKER PICKS IT UP
|
||||
---------------------------
|
||||
Worker:
|
||||
- Asks NATS: "Any messages?"
|
||||
- NATS gives worker the message
|
||||
- Worker reads: "Update delivery #123"
|
||||
|
||||
STEP 7: WORKER FORWARDS TO EXTERNAL API
|
||||
----------------------------------------
|
||||
Worker:
|
||||
- Maps endpoint to: https://jupiter.nearle.app/live/api/v1/deliveries/updatedelivery
|
||||
- Sends PUT request with the data
|
||||
- Waits for response
|
||||
|
||||
STEP 8: EXTERNAL API RESPONDS
|
||||
------------------------------
|
||||
External API:
|
||||
- Processes the request
|
||||
- Updates delivery status
|
||||
- Returns: 200 OK
|
||||
|
||||
STEP 9: WORKER ACKNOWLEDGES
|
||||
----------------------------
|
||||
Worker:
|
||||
- Sees success (200 OK)
|
||||
- Tells NATS: "Done!" (ACK)
|
||||
- NATS removes message from queue
|
||||
- Done!
|
||||
|
||||
IF SOMETHING GOES WRONG:
|
||||
------------------------
|
||||
If external API is down:
|
||||
- Worker gets error
|
||||
- Tells NATS: "Try again later" (NAK)
|
||||
- NATS keeps message in queue
|
||||
- Worker waits 5 seconds
|
||||
- Tries again
|
||||
- Repeats up to 5 times
|
||||
- If still fails, message stays for manual review
|
||||
|
||||
================================================================================
|
||||
5. WHY WE USE EACH TECHNOLOGY
|
||||
================================================================================
|
||||
|
||||
NATS (Message Queue):
|
||||
---------------------
|
||||
✓ Super fast (messages in < 1ms)
|
||||
✓ Reliable (messages won't disappear)
|
||||
✓ Simple (easy to set up and use)
|
||||
✓ Lightweight (doesn't need much resources)
|
||||
✓ Perfect for this use case
|
||||
|
||||
Why not RabbitMQ? Too heavy, more complex
|
||||
Why not Kafka? Overkill, too complex for this
|
||||
Why not Redis? Less mature, fewer features
|
||||
|
||||
FASTAPI (API Server):
|
||||
---------------------
|
||||
✓ Fast (handles many requests)
|
||||
✓ Easy (Python is simple to write)
|
||||
✓ Modern (async/await support)
|
||||
✓ Automatic docs (shows all endpoints)
|
||||
✓ Type validation (catches errors early)
|
||||
|
||||
Why not Flask? Slower, no async support
|
||||
Why not Django? Too heavy, overkill
|
||||
Why not Node.js? Team knows Python better
|
||||
|
||||
WORKERS (Message Processors):
|
||||
------------------------------
|
||||
✓ Decouples API from external service
|
||||
✓ Handles retries automatically
|
||||
✓ Can scale independently
|
||||
✓ Easy to monitor separately
|
||||
|
||||
Why separate workers? Keeps API fast, handles failures better
|
||||
|
||||
NGINX (Proxy):
|
||||
--------------
|
||||
✓ Needed for CORS (browser security)
|
||||
✓ Lightweight (small container)
|
||||
✓ Fast (high performance)
|
||||
✓ Simple config
|
||||
|
||||
Why needed? Browsers block cross-origin requests without CORS headers
|
||||
|
||||
DOCKER (Containers):
|
||||
--------------------
|
||||
✓ Same environment everywhere
|
||||
✓ Easy deployment
|
||||
✓ Isolated services
|
||||
✓ Easy updates
|
||||
|
||||
Why containers? Consistency, easy deployment, isolation
|
||||
|
||||
KUBERNETES (Orchestration):
|
||||
----------------------------
|
||||
✓ High availability (multiple servers)
|
||||
✓ Auto-scaling (adds workers when busy)
|
||||
✓ Self-healing (restarts crashed services)
|
||||
✓ Load balancing
|
||||
|
||||
Why Kubernetes? Production needs reliability and scaling
|
||||
|
||||
TRAEFIK (Router):
|
||||
-----------------
|
||||
✓ Automatic HTTPS (free certificates)
|
||||
✓ Easy routing (just labels)
|
||||
✓ Handles authentication
|
||||
✓ One entry point
|
||||
|
||||
Why Traefik? Simplifies HTTPS and routing
|
||||
|
||||
================================================================================
|
||||
6. HOW TO DEPLOY AND USE
|
||||
================================================================================
|
||||
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ 6.1 DOCKER COMPOSE (SIMPLE DEPLOYMENT) │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
FOR: Local development, small deployments, single server
|
||||
|
||||
STEPS:
|
||||
------
|
||||
1. Make sure Docker is installed
|
||||
2. Go to the kubernetes folder
|
||||
3. Run: docker-compose up -d
|
||||
4. That's it! Everything starts automatically
|
||||
|
||||
WHAT GETS STARTED:
|
||||
-----------------
|
||||
- NATS server (message queue)
|
||||
- Nginx proxy (CORS helper)
|
||||
- NATS Dashboard (monitoring)
|
||||
|
||||
ACCESS:
|
||||
-------
|
||||
- Dashboard: https://natsadmin.workolik.com
|
||||
- NATS Monitoring: https://nats.workolik.com
|
||||
|
||||
CHECK STATUS:
|
||||
-------------
|
||||
docker-compose ps # See what's running
|
||||
docker-compose logs -f # See logs
|
||||
docker-compose restart # Restart everything
|
||||
|
||||
┌─────────────────────────────────────────────────────────────────────┐
|
||||
│ 6.2 KUBERNETES (PRODUCTION DEPLOYMENT) │
|
||||
└─────────────────────────────────────────────────────────────────────┘
|
||||
|
||||
FOR: Production, high availability, auto-scaling
|
||||
|
||||
STEPS:
|
||||
------
|
||||
1. Build Docker images
|
||||
2. Push to registry
|
||||
3. Update secrets.yaml with your credentials
|
||||
4. Run: ./deploy.sh
|
||||
5. Wait for pods to start
|
||||
|
||||
WHAT GETS DEPLOYED:
|
||||
-------------------
|
||||
- FastAPI pods (4-20, auto-scales)
|
||||
- Worker pods (2-10, auto-scales)
|
||||
- Services (load balancers)
|
||||
- Auto-scaling rules
|
||||
|
||||
CHECK STATUS:
|
||||
-------------
|
||||
kubectl get pods -n nats-backend # See pods
|
||||
kubectl logs -f deployment/fastapi-backend # See logs
|
||||
kubectl get hpa -n nats-backend # See auto-scaling
|
||||
|
||||
TEST ENDPOINTS:
|
||||
---------------
|
||||
curl -X PUT https://queue.workolik.com/live/api/v1/deliveries/updatedelivery \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"delivery_id": 123, "status": "delivered"}'
|
||||
|
||||
Expected response:
|
||||
{"status": "accepted", "message_id": 12345, "endpoint": "..."}
|
||||
|
||||
================================================================================
|
||||
7. COMMON QUESTIONS ANSWERED
|
||||
================================================================================
|
||||
|
||||
Q: Why use a message queue instead of calling the API directly?
|
||||
A:
|
||||
- Your app doesn't have to wait for slow external API
|
||||
- If external API is down, messages wait (won't be lost)
|
||||
- Can handle traffic spikes better
|
||||
- Easy to add more workers if needed
|
||||
|
||||
Q: What happens if NATS goes down?
|
||||
A:
|
||||
- Messages are stored on disk (persistent)
|
||||
- When NATS restarts, messages are still there
|
||||
- Workers reconnect automatically
|
||||
- No data loss
|
||||
|
||||
Q: What happens if a worker crashes?
|
||||
A:
|
||||
- Kubernetes restarts it automatically
|
||||
- Messages stay in queue
|
||||
- Other workers keep processing
|
||||
- No data loss
|
||||
|
||||
Q: How fast is it?
|
||||
A:
|
||||
- FastAPI responds in < 1 second (usually < 100ms)
|
||||
- NATS stores message in < 1 millisecond
|
||||
- Worker processes in seconds (depends on external API)
|
||||
- Total: Usually under 2 seconds end-to-end
|
||||
|
||||
Q: Can I add more endpoints?
|
||||
A:
|
||||
- Yes! Add endpoint in FastAPI (app.py)
|
||||
- Add mapping in worker (worker.py)
|
||||
- Deploy and it works
|
||||
|
||||
Q: How do I monitor if everything is working?
|
||||
A:
|
||||
- Check NATS Dashboard: https://natsadmin.workolik.com
|
||||
- Check FastAPI metrics: GET /metrics
|
||||
- Check worker metrics: Port 9090
|
||||
- Check Kubernetes: kubectl get pods
|
||||
|
||||
Q: What if external API is slow?
|
||||
A:
|
||||
- Messages wait in queue
|
||||
- Workers retry automatically
|
||||
- Your API still responds fast (doesn't wait)
|
||||
- No impact on your users
|
||||
|
||||
Q: How many messages can it handle?
|
||||
A:
|
||||
- NATS: Millions per second
|
||||
- FastAPI: Thousands per second (depends on hardware)
|
||||
- Workers: Depends on external API speed
|
||||
- Can scale workers automatically if needed
|
||||
|
||||
Q: Is it secure?
|
||||
A:
|
||||
- HTTPS for all external access (Traefik)
|
||||
- Password protection for dashboard
|
||||
- NATS authentication required
|
||||
- CORS properly configured
|
||||
|
||||
Q: Can I test locally?
|
||||
A:
|
||||
- Yes! Use Docker Compose
|
||||
- Everything runs on your machine
|
||||
- Same as production (just smaller scale)
|
||||
|
||||
================================================================================
|
||||
8. TROUBLESHOOTING MADE EASY
|
||||
================================================================================
|
||||
|
||||
PROBLEM: FastAPI returns 503 "NATS not connected"
|
||||
-----------------------------------------------
|
||||
WHAT IT MEANS: FastAPI can't talk to NATS
|
||||
|
||||
HOW TO FIX:
|
||||
1. Check if NATS is running: docker ps | grep nats
|
||||
2. Check NATS logs: docker logs nats
|
||||
3. Check network: Can FastAPI reach NATS?
|
||||
4. Check credentials: Are username/password correct?
|
||||
|
||||
PROBLEM: Messages not being processed
|
||||
-------------------------------------
|
||||
WHAT IT MEANS: Workers aren't picking up messages
|
||||
|
||||
HOW TO FIX:
|
||||
1. Check if workers are running: kubectl get pods | grep worker
|
||||
2. Check worker logs: kubectl logs -f deployment/nats-worker
|
||||
3. Check NATS Dashboard: Are messages in queue?
|
||||
4. Check consumer: Is worker_consumer active?
|
||||
|
||||
PROBLEM: CORS errors in browser
|
||||
--------------------------------
|
||||
WHAT IT MEANS: Dashboard can't access NATS monitoring
|
||||
|
||||
HOW TO FIX:
|
||||
1. Check Nginx proxy is running: docker ps | grep nats-proxy
|
||||
2. Check nginx-nats.conf: Are CORS headers correct?
|
||||
3. Check NATS_MONITORING_URL in dashboard config
|
||||
4. Restart nginx-proxy: docker-compose restart nats-proxy
|
||||
|
||||
PROBLEM: External API not receiving requests
|
||||
--------------------------------------------
|
||||
WHAT IT MEANS: Workers aren't forwarding messages
|
||||
|
||||
HOW TO FIX:
|
||||
1. Check worker logs: Look for forwarding errors
|
||||
2. Check EXTERNAL_BASE_URL: Is it correct?
|
||||
3. Test connectivity: Can workers reach external API?
|
||||
4. Check endpoint mapping: Is endpoint in worker.py?
|
||||
|
||||
PROBLEM: Auto-scaling not working
|
||||
---------------------------------
|
||||
WHAT IT MEANS: Pods aren't scaling up/down
|
||||
|
||||
HOW TO FIX:
|
||||
1. Check HPA: kubectl get hpa -n nats-backend
|
||||
2. Check metrics-server: kubectl top pods
|
||||
3. Check resource limits: Are they set correctly?
|
||||
4. Check if max replicas reached
|
||||
|
||||
QUICK HEALTH CHECK:
|
||||
-------------------
|
||||
1. Are all pods running? → kubectl get pods
|
||||
2. Are messages processing? → Check dashboard
|
||||
3. Are there errors? → Check logs
|
||||
4. Is external API reachable? → Test from worker pod
|
||||
|
||||
COMMON COMMANDS:
|
||||
----------------
|
||||
# Docker Compose
|
||||
docker-compose ps # Status
|
||||
docker-compose logs -f # Logs
|
||||
docker-compose restart # Restart
|
||||
|
||||
# Kubernetes
|
||||
kubectl get pods # Pod status
|
||||
kubectl logs -f [pod-name] # Pod logs
|
||||
kubectl describe pod [pod] # Pod details
|
||||
kubectl get hpa # Auto-scaling status
|
||||
|
||||
# NATS
|
||||
# Check dashboard: https://natsadmin.workolik.com
|
||||
|
||||
================================================================================
|
||||
9. KEY CONCEPTS EXPLAINED SIMPLY
|
||||
================================================================================
|
||||
|
||||
MESSAGE QUEUE:
|
||||
--------------
|
||||
Like a post office:
|
||||
- You drop off a letter (message)
|
||||
- Post office stores it safely
|
||||
- Mail carrier picks it up
|
||||
- Delivers to destination
|
||||
|
||||
In our system:
|
||||
- FastAPI drops off message
|
||||
- NATS stores it
|
||||
- Worker picks it up
|
||||
- Delivers to external API
|
||||
|
||||
PUBLISH/SUBSCRIBE:
|
||||
------------------
|
||||
Like a radio station:
|
||||
- Radio station broadcasts (publishes)
|
||||
- Radios listen (subscribe)
|
||||
- Many radios can listen to same station
|
||||
|
||||
In our system:
|
||||
- FastAPI publishes messages
|
||||
- Workers subscribe to messages
|
||||
- Many workers can process same queue
|
||||
|
||||
ACKNOWLEDGMENT (ACK/NAK):
|
||||
-------------------------
|
||||
Like a receipt:
|
||||
- You send a package
|
||||
- Recipient signs for it (ACK)
|
||||
- If rejected, you get it back (NAK)
|
||||
|
||||
In our system:
|
||||
- Worker processes message
|
||||
- If success: ACK → message removed
|
||||
- If failure: NAK → message stays for retry
|
||||
|
||||
STREAM:
|
||||
-------
|
||||
Like a folder:
|
||||
- All related messages go in one stream
|
||||
- Easy to organize
|
||||
- Can have multiple consumers
|
||||
|
||||
In our system:
|
||||
- Stream: "EVENTS"
|
||||
- All API messages go here
|
||||
- Workers read from this stream
|
||||
|
||||
CONSUMER:
|
||||
---------
|
||||
Like a reader:
|
||||
- Reads messages from stream
|
||||
- Processes them
|
||||
- Acknowledges when done
|
||||
|
||||
In our system:
|
||||
- Consumer: "worker_consumer"
|
||||
- Workers use this to get messages
|
||||
- Tracks which messages are processed
|
||||
|
||||
AUTO-SCALING:
|
||||
-------------
|
||||
Like a restaurant:
|
||||
- Few customers → Few waiters
|
||||
- Many customers → More waiters
|
||||
- Automatically adjusts
|
||||
|
||||
In our system:
|
||||
- Low load → Few pods
|
||||
- High load → More pods
|
||||
- Kubernetes does it automatically
|
||||
|
||||
LOAD BALANCING:
|
||||
---------------
|
||||
Like distributing work:
|
||||
- Manager gives tasks to available workers
|
||||
- Spreads work evenly
|
||||
- No one gets overloaded
|
||||
|
||||
In our system:
|
||||
- Requests go to available FastAPI pod
|
||||
- Spreads load evenly
|
||||
- Kubernetes handles it
|
||||
|
||||
================================================================================
|
||||
10. REAL-WORLD EXAMPLE
|
||||
================================================================================
|
||||
|
||||
SCENARIO: Your delivery app needs to update delivery status
|
||||
|
||||
WITHOUT MESSAGE QUEUE:
|
||||
----------------------
|
||||
1. App sends request to your API
|
||||
2. Your API calls external API directly
|
||||
3. External API is slow (takes 5 seconds)
|
||||
4. Your app waits 5 seconds
|
||||
5. User sees loading spinner
|
||||
6. If external API is down, your app fails
|
||||
|
||||
WITH MESSAGE QUEUE (OUR SYSTEM):
|
||||
---------------------------------
|
||||
1. App sends request to FastAPI
|
||||
2. FastAPI puts message in queue (< 100ms)
|
||||
3. FastAPI returns "OK" immediately
|
||||
4. User sees success right away
|
||||
5. Worker processes message in background
|
||||
6. If external API is slow/down, message waits
|
||||
7. Worker retries automatically
|
||||
8. No impact on your users
|
||||
|
||||
BENEFITS:
|
||||
---------
|
||||
✓ Users get instant response
|
||||
✓ System handles failures gracefully
|
||||
✓ Can handle traffic spikes
|
||||
✓ Easy to scale
|
||||
✓ Messages never lost
|
||||
|
||||
================================================================================
|
||||
11. QUICK REFERENCE
|
||||
================================================================================
|
||||
|
||||
ENDPOINTS:
|
||||
----------
|
||||
PUT /live/api/v1/deliveries/updatedelivery
|
||||
POST /live/api/v1/deliveries/createdeliveries
|
||||
POST /live/api/v2/partners/createriderlog
|
||||
POST /live/api/v2/deliveries/createdeliverylog
|
||||
POST /live/api/v2/partners/createbreaklog
|
||||
POST /live/api/v2/partners/updatebreaklog
|
||||
|
||||
URLS:
|
||||
-----
|
||||
FastAPI: https://queue.workolik.com
|
||||
Dashboard: https://natsadmin.workolik.com
|
||||
NATS Monitoring: https://nats.workolik.com
|
||||
|
||||
CREDENTIALS:
|
||||
------------
|
||||
NATS:
|
||||
- Username: admin
|
||||
- Password: package@321#
|
||||
|
||||
Dashboard:
|
||||
- Username: admin
|
||||
- Password: package@321#
|
||||
|
||||
PORTS:
|
||||
------
|
||||
NATS: 4222 (client), 8222 (monitoring)
|
||||
FastAPI: 8000 (internal), 8201 (external)
|
||||
Worker: 9090 (metrics)
|
||||
Nginx: 80 (internal), 8082 (external)
|
||||
|
||||
FILES:
|
||||
------
|
||||
docker-compose.yml # Docker deployment
|
||||
scripts/app.py # FastAPI code
|
||||
scripts/worker.py # Worker code
|
||||
nginx-nats.conf # Nginx config
|
||||
manifests/*.yaml # Kubernetes configs
|
||||
|
||||
================================================================================
|
||||
END OF GUIDE
|
||||
================================================================================
|
||||
|
||||
This guide explains the system in simple terms. For more technical details,
|
||||
see TECH_STACK_ARCHITECTURE.txt.
|
||||
|
||||
Remember:
|
||||
- FastAPI receives requests and puts them in queue
|
||||
- NATS stores messages safely
|
||||
- Workers process messages and forward to external API
|
||||
- Everything is monitored and can scale automatically
|
||||
|
||||
Questions? Check Section 7 (Common Questions) or Section 8 (Troubleshooting).
|
||||
|
||||
Last Updated: 2025-01-XX
|
||||
Version: Intermediate Level
|
||||
|
||||
157
docs/test-delivery-logs.ps1
Normal file
157
docs/test-delivery-logs.ps1
Normal file
@@ -0,0 +1,157 @@
|
||||
# PowerShell script to verify delivery logs are working
|
||||
# Usage: .\test-delivery-logs.ps1
|
||||
|
||||
Write-Host "==================================================================" -ForegroundColor Cyan
|
||||
Write-Host " DELIVERY LOG VERIFICATION TEST" -ForegroundColor Cyan
|
||||
Write-Host "==================================================================" -ForegroundColor Cyan
|
||||
Write-Host ""
|
||||
|
||||
# Configuration
|
||||
$FastAPIUrl = if ($env:FASTAPI_URL) { $env:FASTAPI_URL } else { "https://queue.workolik.com" }
|
||||
$Endpoint = "/live/api/v2/deliveries/createdeliverylog"
|
||||
|
||||
# Generate unique test data
|
||||
$Timestamp = [int][double]::Parse((Get-Date -UFormat %s))
|
||||
$OrderId = "TEST-$Timestamp"
|
||||
$CurrentTime = Get-Date -Format "yyyy-MM-dd HH:mm:ss"
|
||||
|
||||
Write-Host "📋 Test Configuration:" -ForegroundColor Yellow
|
||||
Write-Host " FastAPI URL: $FastAPIUrl"
|
||||
Write-Host " Endpoint: $Endpoint"
|
||||
Write-Host " Test Order ID: $OrderId"
|
||||
Write-Host ""
|
||||
|
||||
# Test payload
|
||||
$Payload = @(
|
||||
@{
|
||||
logid = 0
|
||||
tenantid = 1
|
||||
partnerid = 44
|
||||
locationid = 1
|
||||
orderheaderid = $Timestamp
|
||||
deliveryid = $Timestamp + 1000
|
||||
userid = 1111
|
||||
orderid = $OrderId
|
||||
orderstatus = "active"
|
||||
starttime = $CurrentTime
|
||||
logdate = $CurrentTime
|
||||
latitude = "11.0050664"
|
||||
longitude = "76.9508776"
|
||||
}
|
||||
) | ConvertTo-Json
|
||||
|
||||
Write-Host "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" -ForegroundColor Gray
|
||||
Write-Host "STEP 1: Testing FastAPI Endpoint" -ForegroundColor Cyan
|
||||
Write-Host "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" -ForegroundColor Gray
|
||||
Write-Host ""
|
||||
|
||||
Write-Host "Sending POST request to: $FastAPIUrl$Endpoint" -ForegroundColor Yellow
|
||||
Write-Host ""
|
||||
|
||||
try {
|
||||
$Response = Invoke-RestMethod -Uri "$FastAPIUrl$Endpoint" `
|
||||
-Method Post `
|
||||
-ContentType "application/json" `
|
||||
-Body $Payload `
|
||||
-ErrorAction Stop
|
||||
|
||||
Write-Host "HTTP Status Code: 200" -ForegroundColor Green
|
||||
Write-Host "Response Body:" -ForegroundColor Yellow
|
||||
$Response | ConvertTo-Json -Depth 10
|
||||
Write-Host ""
|
||||
|
||||
if ($Response.status -eq "accepted") {
|
||||
Write-Host "✅ SUCCESS: FastAPI accepted the request" -ForegroundColor Green
|
||||
|
||||
if ($Response.message_id) {
|
||||
Write-Host " Message ID: $($Response.message_id)" -ForegroundColor Green
|
||||
}
|
||||
|
||||
Write-Host " Status: Accepted" -ForegroundColor Green
|
||||
} else {
|
||||
Write-Host "⚠️ WARNING: Response status is not 'accepted'" -ForegroundColor Yellow
|
||||
}
|
||||
} catch {
|
||||
$StatusCode = $_.Exception.Response.StatusCode.value__
|
||||
Write-Host "❌ FAILED: FastAPI returned status $StatusCode" -ForegroundColor Red
|
||||
Write-Host ""
|
||||
Write-Host "Error Details:" -ForegroundColor Red
|
||||
Write-Host $_.Exception.Message -ForegroundColor Red
|
||||
Write-Host ""
|
||||
Write-Host "Troubleshooting:" -ForegroundColor Yellow
|
||||
Write-Host "1. Check if FastAPI is running"
|
||||
Write-Host "2. Check FastAPI logs for errors"
|
||||
Write-Host "3. Verify the endpoint URL is correct"
|
||||
exit 1
|
||||
}
|
||||
|
||||
Write-Host ""
|
||||
Write-Host "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" -ForegroundColor Gray
|
||||
Write-Host "STEP 2: Waiting for Processing" -ForegroundColor Cyan
|
||||
Write-Host "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" -ForegroundColor Gray
|
||||
Write-Host ""
|
||||
|
||||
Write-Host "⏳ Waiting 5 seconds for message to be processed by worker..." -ForegroundColor Yellow
|
||||
Start-Sleep -Seconds 5
|
||||
Write-Host ""
|
||||
|
||||
Write-Host "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" -ForegroundColor Gray
|
||||
Write-Host "STEP 3: Next Steps for Verification" -ForegroundColor Cyan
|
||||
Write-Host "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" -ForegroundColor Gray
|
||||
Write-Host ""
|
||||
|
||||
Write-Host "To complete verification, check the following:" -ForegroundColor Yellow
|
||||
Write-Host ""
|
||||
Write-Host "1. 📊 NATS Dashboard:" -ForegroundColor Cyan
|
||||
Write-Host " - URL: https://natsadmin.workolik.com"
|
||||
Write-Host " - Login: admin / package@321#"
|
||||
Write-Host " - Check Stream 'EVENTS' → Look for message count"
|
||||
Write-Host " - Check Consumer 'worker_consumer' → Look for pending messages"
|
||||
Write-Host ""
|
||||
|
||||
Write-Host "2. 📋 Worker Logs:" -ForegroundColor Cyan
|
||||
Write-Host " Docker Compose:" -ForegroundColor Yellow
|
||||
Write-Host " docker-compose logs -f nats-worker | grep deliverylog"
|
||||
Write-Host ""
|
||||
Write-Host " Kubernetes:" -ForegroundColor Yellow
|
||||
Write-Host " kubectl logs -f deployment/nats-worker -n nats-backend | grep deliverylog"
|
||||
Write-Host ""
|
||||
Write-Host " Look for:" -ForegroundColor Yellow
|
||||
Write-Host " ✅ '📨 Processing message for endpoint: /live/api/v2/deliveries/createdeliverylog'"
|
||||
Write-Host " ✅ '➡️ Forwarding createdeliverylog payload: ...'"
|
||||
Write-Host " ✅ '✅ Message processed and forwarded successfully'"
|
||||
Write-Host ""
|
||||
|
||||
Write-Host "3. 📈 FastAPI Metrics:" -ForegroundColor Cyan
|
||||
Write-Host " curl $FastAPIUrl/metrics | grep deliverylog"
|
||||
Write-Host ""
|
||||
|
||||
Write-Host "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" -ForegroundColor Gray
|
||||
Write-Host "QUICK CHECK COMMANDS" -ForegroundColor Cyan
|
||||
Write-Host "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" -ForegroundColor Gray
|
||||
Write-Host ""
|
||||
|
||||
Write-Host "# Check if workers are running (Docker Compose):" -ForegroundColor Yellow
|
||||
Write-Host "docker-compose ps | grep worker"
|
||||
Write-Host ""
|
||||
Write-Host "# Check if workers are running (Kubernetes):" -ForegroundColor Yellow
|
||||
Write-Host "kubectl get pods -n nats-backend | grep worker"
|
||||
Write-Host ""
|
||||
Write-Host "# View recent worker logs (Docker Compose):" -ForegroundColor Yellow
|
||||
Write-Host "docker-compose logs --tail=50 nats-worker"
|
||||
Write-Host ""
|
||||
Write-Host "# View recent worker logs (Kubernetes):" -ForegroundColor Yellow
|
||||
Write-Host "kubectl logs --tail=50 deployment/nats-worker -n nats-backend"
|
||||
Write-Host ""
|
||||
|
||||
Write-Host "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" -ForegroundColor Gray
|
||||
Write-Host ""
|
||||
|
||||
Write-Host "✅ Test request sent successfully!" -ForegroundColor Green
|
||||
Write-Host ""
|
||||
Write-Host "Test Order ID: $OrderId" -ForegroundColor Cyan
|
||||
Write-Host "Use this ID to track the message through the system."
|
||||
Write-Host ""
|
||||
Write-Host "For detailed verification steps, see: HOW_TO_VERIFY_DELIVERY_LOGS.txt" -ForegroundColor Yellow
|
||||
|
||||
|
||||
Reference in New Issue
Block a user