kubernetes/docs/PROFESSIONAL_K3S_TARGET_ARCHITECTURE.md

Professional k3s Target Architecture

This document is a practical upgrade path from the current repo layout to a more professional, resilient platform.

It is written for the current setup in this repository:

• k3s is being used as the Kubernetes distribution.
• ingress/proxy traffic is still partly handled outside the cluster with Docker Compose and nginx.
• app workloads are running in Kubernetes.
• workers depend on NATS/JetStream.
• some services still use NodePort.
• the core workloads point to a single external NATS IP.

1. What the current repo already does well

There is already a good foundation here:

• API and worker responsibilities are separated.
• asynchronous processing with NATS/JetStream is the right pattern.
• workers are split by business area in manifests/core/workers.yaml.
• you already have some resource requests and limits.
• you already use PodDisruptionBudget for one worker set.
• you already think in namespaces such as core and nearle.

That means you are not starting from zero. You are mostly at the stage of cleaning up platform boundaries and removing single points of failure.

2. Main problems in the current architecture

Problem A: single point of failure in messaging

Your core workloads use one external NATS endpoint:

• manifests/core/core-config.yaml#L10

If that node or disk fails, the Kubernetes pods can still be healthy but the platform is still down.

Problem B: node failover is not immediate today

You want "if current node is not working, switch immediately to backup node". Right now that is not really true because:

• some traffic still depends on host-level Docker Compose proxies
• some services are exposed with NodePort
• some workloads use hostPath
• there is no clear control-plane and worker-plane separation
• there is no shared HA datastore for k3s control-plane state

For example:

• manifests/nearle/nearle-jupiter.yaml#L58 exposes jupiter with NodePort
• manifests/nearle/nearle-jupiter.yaml#L43 uses hostPath

NodePort is not wrong, but it is usually not the best long-term edge pattern for a professional HA setup.

Problem C: CPU spikes can still hurt the whole node

Your workers are separated logically, but they still share the same physical node resources unless you explicitly isolate them with:

• dedicated worker nodes
• taints and tolerations
• node labels and node affinity
• tighter requests/limits
• autoscaling based on the right metrics

Some workers also have relatively high concurrency values:

• manifests/core/workers.yaml#L42
• manifests/core/workers.yaml#L192

If the external target slows down or retries increase, these workers can create CPU and network pressure.

Problem D: secrets are stored in repo manifests

There are inline credentials here:

• manifests/core/core-secrets.yaml

For a professional setup, secrets should move to a proper secret manager or at least be injected outside git.

3. What "professional architecture" should look like here

For your use case, the clean target is not "one main node and one cold backup node".

The cleaner target is:

1. Highly available k3s control plane
2. Separate worker nodes for workloads
3. NATS deployed as an HA cluster
4. Ingress handled inside Kubernetes
5. Backups for both cluster state and NATS data
6. Workload placement rules so noisy workers do not affect all services

4. Recommended target layout

Minimum professional layout

• 3 k3s server nodes
• 2 workload worker nodes
• 3 NATS pods with JetStream replication
• 1 in-cluster ingress controller
• 1 backup system for cluster resources and persistent volumes

Example node roles

• cp-1: k3s server
• cp-2: k3s server
• cp-3: k3s server
• app-1: app services and ingress
• wrk-1: NATS workers and heavy async jobs

Better:

• app-1, app-2: app/service nodes
• wrk-1, wrk-2: dedicated worker nodes

Planes

Control plane:

• k3s server nodes only
• no application workloads if possible

Service plane:

• FastAPI / business services
• ingress controller
• gateway
• observability stack

Worker plane:

• NATS worker consumers
• batch jobs
• CPU-heavy or retry-heavy services

Messaging plane:

• NATS cluster with JetStream
• dedicated storage

5. Best failover model for your requirement

You asked for immediate switch to a backup node if one node fails.

There are two ways to think about that:

Option 1: active-passive node failover

This means one main node and one backup node waiting.

This is simpler to understand, but it is still not the best Kubernetes design because:

• one active node is still a bottleneck
• failover is not truly instant
• stateful components are harder to fail over cleanly
• you will still need shared storage or replicated state

Option 2: active-active cluster with multiple nodes

This is the better professional design.

Instead of "switching to backup", Kubernetes simply reschedules workloads onto healthy nodes because:

• services already run across more than one node
• ingress already points at the cluster, not one host
• NATS data is replicated
• pods have anti-affinity and replicas across nodes

This is the model I recommend for you.

6. Recommended production architecture for this repo

Edge and ingress

Move ingress fully into Kubernetes:

• install Traefik or nginx ingress inside k3s
• stop depending on host-level Docker Compose nginx proxies for production routing
• expose ingress through a proper load balancer or a floating IP/VIP

Good choices:

• kube-vip for virtual IP failover on bare metal
• MetalLB for service load balancers on bare metal

This is the clean replacement for current host-side proxying in docker-compose.yml.

Kubernetes control plane

Use HA k3s server nodes:

• 3 k3s server nodes
• external datastore or embedded etcd in HA mode

For small-to-medium production, HA k3s with embedded etcd is often enough and simpler than trying to maintain a single-node server plus a backup.

App services

Run FastAPI and business services on app nodes:

• Deployment instead of StatefulSet unless stable pod identity is required
• replicas >= 2
• topology spread constraints
• anti-affinity across nodes
• proper readiness and liveness probes

Worker services

Run workers only on worker nodes:

• label worker nodes, for example workload-type=async
• taint worker nodes
• add tolerations to worker pods
• add node affinity so worker pods land only there

This gives you the "separate worker plane" you asked for.

NATS

Deploy NATS inside Kubernetes as an HA cluster, not as a single external endpoint.

Use:

• 3 NATS pods
• JetStream replication factor 3
• persistent volumes
• pod anti-affinity
• dedicated node pool if possible

This is the biggest architecture improvement you can make for service survival.

Backups

Use two backup layers:

1. Kubernetes resource/state backup
2. JetStream or persistent volume backup

Recommended tools:

• Velero for cluster resource backup and restore
• CSI snapshots or storage-level snapshots for persistent volumes
• scheduled export/backup for critical NATS data if needed

7. How to separate service plane and worker plane

This is a very good idea for your stack.

Service plane should host

• ingress controller
• FastAPI/API gateway
• frontend-facing services
• dashboard/observability tools

Worker plane should host

• NATS consumers
• retry-heavy jobs
• long-running async processors
• any CPU-heavy integration jobs

Basic implementation pattern

On nodes:

• label app nodes: node-role.workolik/app=true
• label worker nodes: node-role.workolik/worker=true

Optionally taint worker nodes:

• dedicated=workers:NoSchedule

Then:

• app manifests use node affinity for app=true
• worker manifests use node affinity and toleration for worker nodes

8. CPU spike reduction strategy

The CPU spike problem is usually not solved by "adding one backup node". It is solved by isolation, limits, and scaling.

Do these first

1. Put workers on separate nodes.
2. Tighten worker CPU limits and requests based on real usage.
3. Reduce high default concurrency for the noisiest workers.
4. Add HPA or KEDA scaling from queue depth, not only CPU.
5. Ensure retries do not cause synchronized storms.

Very likely spike sources in this repo

• high worker concurrency
• many worker StatefulSets sharing the same node
• retries against slow external APIs
• host-level proxying plus cluster-level routing mix
• single-node k3s carrying ingress, apps, workers, and maybe NATS responsibilities together

Better autoscaling choice

For queue workers, KEDA is often better than plain HPA because it can scale on:

• NATS lag
• queue depth
• custom Prometheus metrics

That is usually more useful than only scaling from CPU percentage.

9. Concrete migration path

Phase 1: stabilize current cluster

Do this before any big redesign:

• move secrets out of git
• standardize on ingress instead of many host nginx proxies
• remove unnecessary NodePort exposure where possible
• add resource dashboards and alerting
• capture actual CPU and memory usage for each worker

Phase 2: separate worker nodes

• add a second or third node
• label and taint worker nodes
• move worker workloads there
• keep app services on separate nodes

This alone will already reduce blast radius from worker CPU spikes.

Phase 3: make k3s highly available

• create 3 k3s server nodes
• use HA embedded etcd
• put ingress behind kube-vip or MetalLB

Now loss of one server does not mean cluster loss.

Phase 4: make NATS highly available

• deploy NATS cluster inside Kubernetes
• enable JetStream replication
• use persistent volumes
• update workloads to connect to in-cluster NATS service

This replaces the current single external NATS dependency from manifests/core/core-config.yaml#L10.

Phase 5: add backup and restore

• install Velero
• schedule cluster backups
• schedule volume snapshots
• test restore into a fresh environment

If restore has never been tested, backup is not yet reliable.

10. What I would choose for you

Because you said you are a beginner, I would not jump straight into a very large platform.

I would choose this as the practical target:

• 3 k3s server nodes
• 2 worker nodes
• in-cluster Traefik
• MetalLB or kube-vip
• NATS HA cluster with JetStream replication
• Velero backups
• node separation for app plane and worker plane

This is modern, realistic, and still manageable.

11. What I would not recommend

I would avoid these patterns for your next version:

• one main Kubernetes node plus one cold backup node as the final design
• storing secrets directly in yaml in git
• relying on NodePort as the main production exposure pattern
• mixing host Docker Compose production routing with cluster routing long-term
• keeping NATS as a single external IP with no clear HA story

12. Immediate next actions for this repo

If we continue from this document, the best implementation order is:

1. Add node placement rules for workers and apps.
2. Convert external exposure to a single in-cluster ingress pattern.
3. Remove hard dependence on the single external NATS IP.
4. Add observability for CPU, memory, restart count, and queue lag.
5. Introduce backup tooling and test restore.

13. Summary in simple words

The clean professional version of your stack is:

• Kubernetes control plane on multiple HA server nodes
• app services on one node group
• NATS workers on another node group
• NATS itself running as a replicated cluster
• ingress and failover handled at cluster level, not manually by switching servers
• backups for both Kubernetes state and message data

That gives you what you want:

• less CPU blast radius
• better reliability
• faster failover
• cleaner separation of responsibilities
• a more modern production setup