Fix deployment tooling: shell scripts and Terraform
Terraform (validated with the real terraform CLI - was never actually
run against this cluster, no state file existed):
- Delete main.tf: it declared a duplicate kubernetes_namespace.core
(also in namespaces.tf) and a duplicate provider "kubernetes" block
(also in providers.tf), both hard errors that would fail
`terraform plan` immediately.
- Fix workloads.tf references to 6 files deleted in the manifest
cleanup (jupiter-sts/svc, atlantis-sts/svc, fiesta-sts/svc) - now
points at the canonical nearle-jupiter/atlantis/fiesta.yaml.
- Fix every kubernetes_manifest resource: they fed multi-document
YAML (multiple '---'-separated docs per file) straight into
yamldecode(), which only parses a single document. Rewrote using a
split-on-'---' + for_each pattern, confirmed safe first by checking
separator counts exactly match document counts for every affected
file (no embedded '---' inside any script/config content).
- Add the doormile namespace; rename kubernetes_namespace to
kubernetes_namespace_v1 (fixes a deprecation warning).
- `terraform validate` now passes clean.
Shell scripts:
- deploy-nearle-stack.sh only applied 4 of the ~13 files in
manifests/nearle/ - missing the ConfigMap/Secrets fiesta/jupiter/
titan/ariane need via envFrom, the fiesta gateway script ConfigMap,
atlantis entirely, and the Gateway/ReferenceGrant/jupiter-cors-proxy
resources. Now applies every file (verified by diffing the
directory listing against the script).
- Added deploy-doormile.sh and deploy-ingress.sh - nothing previously
applied ingress-unified.yaml or traefik-middlewares.yaml at all.
- Rewrote deploy.sh as an orchestrator calling all of the above in
order (previously referenced a manifests/namespace.yaml layout that
hasn't existed since before this repo's initial commit).
- Rewrote check-k8s-status.sh to check the real namespaces
(core/nearle/alaska/doormile/kubernetes-dashboard) instead of a
'nats-backend' namespace that never existed in this repo.
- Fixed a `cd` bug in setup-jetstream.sh that made it change into
shfiles/ and then look for scripts/setup_jetstream.py there (a
child directory that doesn't exist) - it could never have found its
own target file. Now pulls NATS credentials from the live
nats-credentials Secret instead of a third hardcoded copy.
Python scripts:
- sync_manifests.py had hardcoded Windows paths (e:\nats\kubernetes\...)
- replaced with paths relative to the script's own location so it
actually runs here (or anywhere). Verified by running it.
- setup_jetstream.py created durable consumers under different names
than worker.py computes at runtime ({NATS_CONSUMER}_{subject}), so
its max_deliver/ack_wait settings never actually reached the
consumers workers bind to. Naming now derived with the same logic
worker.py uses - verified all 10 derived names match workers.yaml
exactly.
- purge-old-messages.py had hardcoded NATS credentials with no env
var override at all - fixed to match the pattern used everywhere
else.
This commit is contained in:
Suriya
@@ -1,93 +1,63 @@
|
||||
#!/bin/bash
|
||||
# Status check across every namespace actually in use by this cluster.
|
||||
# Usage: ./shfiles/check-k8s-status.sh
|
||||
|
||||
NAMESPACES=(core nearle alaska doormile kubernetes-dashboard)
|
||||
|
||||
echo "=========================================="
|
||||
echo "🔍 KUBERNETES DEPLOYMENT STATUS"
|
||||
echo "=========================================="
|
||||
echo ""
|
||||
|
||||
echo "📦 1. PODS STATUS (All Namespaces)"
|
||||
echo "📦 1. NODES"
|
||||
echo "-----------------------------------"
|
||||
kubectl get nodes -o wide
|
||||
echo ""
|
||||
|
||||
echo "📦 2. PODS (All Namespaces)"
|
||||
echo "-----------------------------------"
|
||||
kubectl get pods -A -o wide
|
||||
echo ""
|
||||
|
||||
echo "📦 2. NATS-BACKEND NAMESPACE - PODS"
|
||||
echo "-----------------------------------"
|
||||
kubectl get pods -n nats-backend -o wide
|
||||
echo ""
|
||||
for ns in "${NAMESPACES[@]}"; do
|
||||
echo "=========================================="
|
||||
echo "📦 Namespace: ${ns}"
|
||||
echo "=========================================="
|
||||
|
||||
echo "🌐 3. SERVICES & LOAD BALANCER"
|
||||
echo "-----------------------------------"
|
||||
kubectl get svc -n nats-backend -o wide
|
||||
echo ""
|
||||
echo "-- Pods --"
|
||||
kubectl get pods -n "${ns}" -o wide 2>/dev/null || echo " (namespace not found)"
|
||||
echo ""
|
||||
|
||||
echo "⚖️ 4. LOAD BALANCER DETAILS"
|
||||
echo "-----------------------------------"
|
||||
kubectl get svc fastapi-lb -n nats-backend -o yaml | grep -A 10 "spec:"
|
||||
echo ""
|
||||
echo "-- Services --"
|
||||
kubectl get svc -n "${ns}" -o wide 2>/dev/null
|
||||
echo ""
|
||||
|
||||
echo "🚀 5. K3S KUBERNETES LOAD BALANCER PODS (klipper-lb)"
|
||||
echo "-----------------------------------"
|
||||
kubectl get pods -n kube-system -l "svccontroller.k3s.cattle.io/svcname=fastapi-lb" -o wide
|
||||
echo ""
|
||||
echo "-- StatefulSets / Deployments --"
|
||||
kubectl get statefulsets,deployments -n "${ns}" -o wide 2>/dev/null
|
||||
echo ""
|
||||
|
||||
echo "📊 6. DEPLOYMENTS & REPLICAS"
|
||||
echo "-----------------------------------"
|
||||
kubectl get deployments -n nats-backend -o wide
|
||||
echo ""
|
||||
echo "-- HPA / PodDisruptionBudgets --"
|
||||
kubectl get hpa,pdb -n "${ns}" 2>/dev/null
|
||||
echo ""
|
||||
|
||||
echo "📈 7. HORIZONTAL POD AUTOSCALER (HPA)"
|
||||
echo "-----------------------------------"
|
||||
kubectl get hpa -n nats-backend
|
||||
echo ""
|
||||
echo "-- Recent Events (last 10) --"
|
||||
kubectl get events -n "${ns}" --sort-by='.lastTimestamp' 2>/dev/null | tail -10
|
||||
echo ""
|
||||
done
|
||||
|
||||
echo "🔗 8. ENDPOINTS (Service Backends)"
|
||||
echo "-----------------------------------"
|
||||
kubectl get endpoints -n nats-backend
|
||||
echo ""
|
||||
|
||||
echo "🌍 9. INGRESS/GATEWAY STATUS"
|
||||
echo "🌍 GATEWAY / HTTPROUTE / INGRESS (All Namespaces)"
|
||||
echo "-----------------------------------"
|
||||
kubectl get gateway -A 2>/dev/null || echo "No Gateway API resources found"
|
||||
kubectl get httproute -A 2>/dev/null || echo "No HTTPRoute resources found"
|
||||
kubectl get ingress -A 2>/dev/null || echo "No Ingress resources found"
|
||||
echo ""
|
||||
|
||||
echo "📋 10. RECENT EVENTS (Last 20)"
|
||||
echo "💾 RESOURCE USAGE (CPU/Memory)"
|
||||
echo "-----------------------------------"
|
||||
kubectl get events -n nats-backend --sort-by='.lastTimestamp' | tail -20
|
||||
echo ""
|
||||
|
||||
echo "💾 11. RESOURCE USAGE (CPU/Memory)"
|
||||
echo "-----------------------------------"
|
||||
kubectl top pods -n nats-backend 2>/dev/null || echo "Metrics server not available"
|
||||
kubectl top pods -A 2>/dev/null || echo "Metrics server not available"
|
||||
kubectl top nodes 2>/dev/null || echo "Metrics server not available"
|
||||
echo ""
|
||||
|
||||
echo "🔍 12. FASTAPI POD LOGS (Last 30 lines)"
|
||||
echo "-----------------------------------"
|
||||
kubectl logs -n nats-backend -l app=fastapi-backend --tail=30 2>/dev/null || echo "No FastAPI pods found"
|
||||
echo ""
|
||||
|
||||
echo "🔍 13. WORKER POD LOGS (Last 30 lines)"
|
||||
echo "-----------------------------------"
|
||||
kubectl logs -n nats-backend -l app=nats-worker --tail=30 2>/dev/null || echo "No worker pods found"
|
||||
echo ""
|
||||
|
||||
echo "🌐 14. NETWORK FLOW CHECK"
|
||||
echo "-----------------------------------"
|
||||
echo "Load Balancer External IP/Port:"
|
||||
kubectl get svc fastapi-lb -n nats-backend -o jsonpath='{.status.loadBalancer.ingress[0].ip}:{.spec.ports[0].port}' 2>/dev/null || echo "Checking NodePort..."
|
||||
kubectl get svc fastapi-lb -n nats-backend -o jsonpath='NodePort: {.spec.ports[0].nodePort}' 2>/dev/null
|
||||
echo ""
|
||||
echo "FastAPI Service ClusterIP:"
|
||||
kubectl get svc fastapi-backend -n nats-backend -o jsonpath='{.spec.clusterIP}:{.spec.ports[0].port}' 2>/dev/null
|
||||
echo ""
|
||||
|
||||
echo "✅ 15. HEALTH CHECK"
|
||||
echo "-----------------------------------"
|
||||
kubectl run health-check --rm -i --restart=Never --image=curlimages/curl -- curl -s http://fastapi-backend.nats-backend:8000/health 2>/dev/null || echo "Health check failed"
|
||||
echo ""
|
||||
|
||||
echo "=========================================="
|
||||
echo "✅ Status Check Complete!"
|
||||
echo "=========================================="
|
||||
|
||||
|
||||
15
shfiles/deploy-doormile.sh
Normal file
15
shfiles/deploy-doormile.sh
Normal file
@@ -0,0 +1,15 @@
|
||||
#!/bin/bash
|
||||
# Deploy "doormile" stack to Kubernetes
|
||||
# Usage: ./deploy-doormile.sh
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
NAMESPACE="doormile"
|
||||
|
||||
echo "🚀 Deploying Doormile Stack..."
|
||||
kubectl apply -f manifests/doormile/miletruth.yaml
|
||||
|
||||
echo ""
|
||||
echo "✅ Doormile deployment applied."
|
||||
echo "📋 Current status:"
|
||||
kubectl get all -n "${NAMESPACE}" || true
|
||||
15
shfiles/deploy-ingress.sh
Normal file
15
shfiles/deploy-ingress.sh
Normal file
@@ -0,0 +1,15 @@
|
||||
#!/bin/bash
|
||||
# Deploy shared Ingress resources and Traefik CORS middlewares
|
||||
# (queue.workolik.com, jupiter/fiesta/atlantis.nearle.app, alaska + nearle CORS)
|
||||
# Usage: ./deploy-ingress.sh
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
echo "🌐 Applying Ingress resources..."
|
||||
kubectl apply -f manifests/core/ingress-unified.yaml
|
||||
|
||||
echo "🎛️ Applying Traefik CORS middlewares..."
|
||||
kubectl apply -f manifests/core/traefik-middlewares.yaml
|
||||
|
||||
echo ""
|
||||
echo "✅ Ingress & middlewares applied."
|
||||
@@ -9,16 +9,29 @@ NAMESPACE="nearle"
|
||||
echo "🔎 Ensuring namespace '${NAMESPACE}' exists..."
|
||||
kubectl apply -f manifests/nearle/nearle-namespace.yaml
|
||||
|
||||
echo "🔐 Applying config & secrets..."
|
||||
kubectl apply -f manifests/nearle/nearle-config.yaml
|
||||
kubectl apply -f manifests/nearle/nearle-secrets.yaml
|
||||
kubectl apply -f manifests/nearle/nearle-app-secrets.yaml
|
||||
|
||||
echo "📜 Applying fiesta gateway script ConfigMap..."
|
||||
kubectl apply -f manifests/nearle/fiesta-gateway.yaml
|
||||
|
||||
echo "🌐 Applying Gateway API resources..."
|
||||
kubectl apply -f manifests/nearle/nearle-gateway.yaml
|
||||
kubectl apply -f manifests/nearle/nearle-reference-grant.yaml
|
||||
|
||||
echo "🚀 Deploying Services..."
|
||||
kubectl apply -f manifests/nearle/nearle-jupiter.yaml
|
||||
kubectl apply -f manifests/nearle/nearle-titan.yaml
|
||||
kubectl apply -f manifests/nearle/nearle-fiesta.yaml
|
||||
kubectl apply -f manifests/nearle/nearle-ariane.yaml
|
||||
kubectl apply -f manifests/nearle/nearle-atlantis.yaml
|
||||
|
||||
echo "🧭 Deploying Jupiter CORS proxy..."
|
||||
kubectl apply -f manifests/nearle/jupiter-cors-proxy.yaml
|
||||
|
||||
echo ""
|
||||
echo "✅ Nearle stack deployment applied."
|
||||
echo "📋 Current status:"
|
||||
echo " kubectl get all -n ${NAMESPACE}"
|
||||
|
||||
kubectl get all -n "${NAMESPACE}" || true
|
||||
|
||||
@@ -1,10 +1,12 @@
|
||||
#!/bin/bash
|
||||
# Kubernetes Deployment Script for 3-Node Cluster
|
||||
# Usage: ./deploy.sh
|
||||
# Deploy the full stack to Kubernetes (core, nearle, alaska, doormile, ingress)
|
||||
# Usage: ./shfiles/deploy.sh (run from the repo root, or anywhere - it cd's there itself)
|
||||
|
||||
set -e
|
||||
set -euo pipefail
|
||||
|
||||
echo "🚀 Deploying to Kubernetes (3-Node Cluster)..."
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
echo "🚀 Deploying full stack..."
|
||||
|
||||
# Check if kubectl is available
|
||||
if ! command -v kubectl &> /dev/null; then
|
||||
@@ -23,46 +25,29 @@ kubectl cluster-info || {
|
||||
NODE_COUNT=$(kubectl get nodes --no-headers | wc -l)
|
||||
echo "📊 Cluster has $NODE_COUNT node(s)"
|
||||
|
||||
# Apply namespace
|
||||
echo "📦 Creating namespace..."
|
||||
kubectl apply -f manifests/namespace.yaml
|
||||
|
||||
# Apply secrets
|
||||
echo "🔐 Creating secrets..."
|
||||
kubectl apply -f manifests/secrets.yaml
|
||||
|
||||
# Apply FastAPI
|
||||
echo "🐍 Deploying FastAPI backend..."
|
||||
kubectl apply -f manifests/fastapi-deployment.yaml
|
||||
kubectl apply -f manifests/fastapi-service.yaml
|
||||
kubectl apply -f manifests/fastapi-hpa.yaml
|
||||
|
||||
# Apply Workers
|
||||
echo "👷 Deploying NATS workers..."
|
||||
kubectl apply -f manifests/worker-deployment.yaml
|
||||
kubectl apply -f manifests/worker-hpa.yaml
|
||||
|
||||
# Apply Gateway (optional - only if Gateway API is installed)
|
||||
read -p "Deploy Gateway API? (requires Gateway API controller) [y/N] " -n 1 -r
|
||||
echo
|
||||
if [[ $REPLY =~ ^[Yy]$ ]]; then
|
||||
echo "🚪 Deploying Gateway API..."
|
||||
kubectl apply -f manifests/gateway.yaml
|
||||
fi
|
||||
echo ""
|
||||
echo "===== Core stack (NATS workers) ====="
|
||||
bash shfiles/deploy-core-stack.sh
|
||||
|
||||
echo ""
|
||||
echo "✅ Deployment complete!"
|
||||
echo "===== Nearle stack (jupiter, titan, fiesta, ariane, atlantis) ====="
|
||||
bash shfiles/deploy-nearle-stack.sh
|
||||
|
||||
echo ""
|
||||
echo "===== Alaska stack (deliveries gateway + dashboard) ====="
|
||||
bash shfiles/deploy-alaska.sh
|
||||
|
||||
echo ""
|
||||
echo "===== Doormile stack ====="
|
||||
bash shfiles/deploy-doormile.sh
|
||||
|
||||
echo ""
|
||||
echo "===== Ingress & Traefik middlewares ====="
|
||||
bash shfiles/deploy-ingress.sh
|
||||
|
||||
echo ""
|
||||
echo "✅ Full deployment complete!"
|
||||
echo ""
|
||||
echo "📋 Check status:"
|
||||
echo " kubectl get pods -n nats-backend -o wide"
|
||||
echo " kubectl get nodes"
|
||||
echo " kubectl get hpa -n nats-backend"
|
||||
echo " bash shfiles/check-k8s-status.sh"
|
||||
echo ""
|
||||
echo "📊 View pod distribution across nodes:"
|
||||
echo " kubectl get pods -n nats-backend -o wide | grep -E 'NAME|fastapi|worker'"
|
||||
echo ""
|
||||
echo "📝 View logs:"
|
||||
echo " kubectl logs -f deployment/fastapi-backend -n nats-backend"
|
||||
echo " kubectl logs -f deployment/nats-worker -n nats-backend"
|
||||
echo ""
|
||||
|
||||
|
||||
@@ -1,9 +1,13 @@
|
||||
#!/bin/bash
|
||||
# Setup JetStream stream and consumer for NATS
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
echo "🚀 Setting up NATS JetStream..."
|
||||
|
||||
cd "$(dirname "$0")"
|
||||
# scripts/ is a sibling of shfiles/, both under the repo root - cd there so
|
||||
# the relative path below resolves regardless of where this is invoked from.
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
# Check if Python is available
|
||||
if ! command -v python3 &> /dev/null; then
|
||||
@@ -17,11 +21,19 @@ if ! python3 -c "import nats" 2>/dev/null; then
|
||||
pip3 install nats-py
|
||||
fi
|
||||
|
||||
# Set environment variables
|
||||
export NATS_URL="nats://nats.workolik.com:4222"
|
||||
export NATS_USER="admin"
|
||||
export NATS_PASSWORD="package@321#"
|
||||
# Pull live credentials from the cluster's Secret instead of hardcoding them
|
||||
# here - avoids yet another copy of the password to keep in sync if rotated.
|
||||
if command -v kubectl &> /dev/null && kubectl get secret nats-credentials -n core &> /dev/null 2>&1; then
|
||||
export NATS_USER
|
||||
NATS_USER=$(kubectl get secret nats-credentials -n core -o jsonpath='{.data.username}' | base64 -d)
|
||||
export NATS_PASSWORD
|
||||
NATS_PASSWORD=$(kubectl get secret nats-credentials -n core -o jsonpath='{.data.password}' | base64 -d)
|
||||
else
|
||||
echo "⚠️ Could not read nats-credentials Secret from the cluster (kubectl not available or not connected)."
|
||||
echo " Set NATS_USER / NATS_PASSWORD yourself before running this script."
|
||||
fi
|
||||
|
||||
export NATS_URL="${NATS_URL:-nats://66.116.226.161:4222}"
|
||||
|
||||
# Run the setup script
|
||||
python3 scripts/setup_jetstream.py
|
||||
|
||||
|
||||
Reference in New Issue
Block a user