Troubleshooting Talos Control-Plane down in a Private Network

1. Diagnose from the mgmt context (read-only)

ctx=<mgmt-context>; cl=<cluster>
kubectl --context "$ctx" get tcp -l cluster.x-k8s.io/cluster-name=$cl -o wide
kubectl --context "$ctx" get machines -l cluster.x-k8s.io/cluster-name=$cl -o wide
kubectl --context "$ctx" get events -A -o json | python3 -c '
import sys,json; d=json.load(sys.stdin)
e=[x for x in d.get("items",[]) if x.get("lastTimestamp") and "'$cl'" in (x.get("involvedObject",{}).get("name") or "")]
e.sort(key=lambda x:x.get("lastTimestamp",""))
[print(x.get("lastTimestamp"),x.get("reason"),"|",(x.get("message") or "")[:150]) for x in e[-40:]]
'

2. Check the cloud LB (API front door)

scw lb lb list zone=fr-par-1 -o json                  # find LB by name tag
scw lb backend list <lb-id> zone=fr-par-1 -o json     # members (key is "pool", not "server")
scw lb backend list-statistics <lb-id> zone=fr-par-1 # per-server health

All stopped/failed → no apiserver listening. LB is victim, not cause — investigate CP nodes, not the LB.

3. Spawn a debug VM in the same PN

pnid=$(kubectl --ctx $ctx get scalewaycluster $cl -oyaml | yq .status.network.privateNetworkID)
VMID=$(scw instance server create image=ubuntu_noble type=GP1-XS name=talos-debug zone=fr-par-2 -o json | jq -r .id)   # create VM and capture its ID
scw instance private-nic create server-id=$VMID private-network-id=$pnid zone=fr-par-2     # attach PN before power-on
scw instance server action $VMID action=poweron zone=fr-par-2
scw instance ip attach <ip-id> server-id=$VMID zone=fr-par-2                              # routed IP flaky
scw instance security-group create-rule security-group-id=<sg-id> protocol=TCP action=accept direction=inbound ip-range=0.0.0.0/0 dest-port-from=22 dest-port-to=22 zone=fr-par-2
scw instance server action $VMID action=reboot zone=fr-par-2   # PN NIC stuck "syncing" until reboot

Gotchas: PN NIC stuck syncing → reboot so it shows available/ens6. scw instance server console panics without a TTY — useless non-interactively.

4. Install talosctl + retrieve talosconfig

kubectl --ctx $ctx get secrets $cl-talosconfig -oyaml | yq -r .data.talosconfig | base64 -d > /tmp/$cl-talosconfig
scp /tmp/$cl-talosconfig root@<vm-pub-ip>:/root/talosconfig
ssh root@<vm-pub-ip> 'curl -fsSL https://github.com/siderolabs/talos/releases/latest/download/talosctl-linux-amd64 -o /usr/local/bin/talosctl && chmod +x /usr/local/bin/talosctl'

5. Probe nodes from the debug VM

export TALOSCONFIG=/root/talosconfig
for ip in <cp-ip-1> <cp-ip-2>; do
  talosctl --endpoints $ip --nodes $ip etcd members   # quorum? dead members?
  talosctl --endpoints $ip --nodes $ip etcd status     # leader + raft index
  talosctl --endpoints $ip --nodes $ip services        # etcd/kubelet/apid
  talosctl --endpoints $ip --nodes $ip containers -k   # static pods
done

Decision tree:

• etcd Running OK + majority members + leader → quorum intact. Do NOT talosctl bootstrap (wipes etcd); remove dead members / fix apiserver.
• New nodes etcd Failed: unhealthy cluster → stale failure from before recovery.
• apid up (50000) but 6443 closed → apiserver static pod crashing.

6. Find the apiserver crash reason

talosctl logs -k <pod> fails for exited containers (GC'd fast). Read the on-disk log:

talosctl --endpoints $ip --nodes $ip get staticpod kube-apiserver --namespace k8s -o yaml   # flags/config
P=$(talosctl --endpoints $ip --nodes $ip list /var/log/pods/ | grep kube-apiserver | awk '{print $2}')
N=$(talosctl --endpoints $ip --nodes $ip list /var/log/pods/$P/kube-apiserver/ | tail -1 | awk '{print $2}')
talosctl --endpoints $ip --nodes $ip read /var/log/pods/$P/kube-apiserver/$N | tail -40

Common root causes seen: a single invalid kube-apiserver flag (Error: unknown flag: ...) → CrashLoopBackOff on every node → LB empty → API dead. The static-pod manifest (get staticpod) shows the flags to compare against kube-apiserver --help of that version.

7. Tear down the debug VM

scw instance server action $VMID action=poweroff zone=fr-par-2
scw instance server delete $VMID with-volumes=all zone=fr-par-2