Known Issues

This page lists the known issues that operators may encounter during installation and validation of the PaletteAI Inference Launchpad appliance, and the workaround for each. For the ordered procedure, refer to Install the Appliance.

Slim ISO Does Not Boot on a GPU Server

Symptom. On a server with GPUs, the slim ISO does not appear in the boot menu, or the screen goes black after you select it and the interactive installer never appears. This applies to any GPU brand and any server brand.

Root cause. The Above 4G Decoding or Re-Size BAR Support BIOS setting is turned off. Both are required to boot the installer on a GPU server.

Workaround. Enable both settings in the server's BIOS, then boot the ISO again. Setting names and menu paths vary by server vendor, so refer to your server vendor's documentation. On HPE Gen11 servers, for example, Re-Size BAR Support is under PCIe Device Configuration > Advanced PCIe Configuration. For the prerequisite, refer to Suggested Hardware: Required BIOS Settings.

A blank screen for several minutes immediately after the GRUB selection is expected while the installer loads. This issue applies only when the installer never appears at all.

GPUs do not enumerate on HPE servers

Symptom. On some HPE servers, for example the DL380a Gen11, the GPUs do not enumerate on the PCI bus. Each GPU reports Region 0/2/4: Memory at ignored when you run lspci with -vv, and the kernel logs NVRM: BAR0 is 0M @ 0x0 with a probe failure for every device.

Workaround. Add pci=realloc=off to the GRUB kernel command line.

1. Boot into GRUB and append pci=realloc=off to GRUB_CMDLINE_LINUX_DEFAULT so that it reads GRUB_CMDLINE_LINUX_DEFAULT="quiet splash pci=realloc=off", then reboot.
2. Verify the GPUs by running lspci with -v and -s <bus:device.function>, then confirm with nvidia-smi.
3. Confirm Re-Size BAR Support is enabled in the BIOS. It is required on any GPU server, as described in Suggested Hardware: Required BIOS Settings, and it also improves performance. On HPE Gen11 servers, go to PCIe Device Configuration > Advanced PCIe Configuration. An RBSU BIOS upgrade may be required.

Side effect: NIC rename. The workaround renames the NICs. Update the interface names in the /etc/systemd/network and /oem directories afterward.

Installation stalls during cluster deployment

Symptom. The cluster does not reach a Running and Healthy state during deployment, and one or more packs stay in a pending or installing state.

Workaround. Confirm that the piraeus-operator and nvidia-gpu-operator-ai packs install correctly. GPU driver installation can take extra time on the first boot, so wait before you treat a slow pack as stalled. If the GPUs do not enumerate as expected, refer to GPUs do not enumerate on HPE servers.

Node stuck at login prompt after first boot

Symptom. After the first boot following the OS install, the console shows a plain login prompt with a UUID-derived hostname (for example edge-<uuid> login:), and no user account exists. Every credential you try fails with Login incorrect. The Palette TUI never paints.

Root cause. At the first-boot GRUB menu, the Registration entry was selected (either explicitly or by countdown timeout). Registration runs Kairos onboarding and marks first-boot complete without launching the Palette TUI, so no interactive account creation happens and the node is left at a login prompt with no account.

Workaround. Boot the state-reset entry to clear the first-boot marker, then boot the primary entry.

1. From the baseboard management controller (BMC), power-cycle the node.
2. When the GRUB menu paints, press an arrow key immediately to freeze the countdown. Move the highlight to Palette eXtended Kubernetes Edge state reset (auto) and press ENTER. The node runs a state cleanup and reboots itself; do not intervene during the mid-cycle reboot.
3. When the GRUB menu paints the second time, select Palette eXtended Kubernetes Edge (the first entry, with no suffix) and press ENTER. The Palette TUI paints, and you can create the initial user account.

If the state-reset entry does not restore first-boot behavior, boot the recovery entry, drop to a shell by editing the GRUB entry (press e, append init=/bin/bash to the linux line, and press Ctrl-X to boot), and run kairos-agent reset from the shell. The node reboots into a fresh first-boot state.

SSH, Local UI, or Kubernetes API unreachable after cluster deploy

Symptom. After the cluster reaches a Running and Healthy state, TCP connections from the jumpbox to the node's Host IP time out on port 22 (SSH), 5080 (Local UI), and 6443 (Kubernetes API), while port 443 on the platform IP (Traefik) continues to serve the appliance console. ICMP ping to the Host IP still succeeds. The failure appears to be partial rather than a complete outage.

Root cause. On a single-node install, the platform IP was set equal to the node's Host IP during the cluster profile wizard. Local UI wires the MetalLB IPAddressPool first-pool to include the Host IP as a /32 and sets the Traefik Service spec.loadBalancerIP to the same address. The MetalLB speaker then announces the Host IP on the node's bond interface through gratuitous ARP, and inbound traffic to the Host IP is delivered to MetalLB rather than to the host's own services. Traefik's ports 80 and 443 remain reachable because they traverse the kube-proxy-replacement DNAT path before MetalLB's L2 handling; host services on ports 22, 5080, and 6443 do not.

Discriminating diagnostic. From a shell on the node (reached through the BMC serial console), inspect the MetalLB pool.

sudo kubectl --namespace metallb-system get ipaddresspool first-pool --output yaml

If spec.addresses contains the node's Host IP, this issue applies.

Workaround. Move the platform IP off the Host IP, and refresh Cilium's load-balancer state.

1. Patch the MetalLB pool to a distinct unused IP from your reserved MetalLB range. Replace <new-platform-ip> with your chosen address.

   sudo kubectl --namespace metallb-system patch ipaddresspool first-pool --type='json' \
     --patch='[{"op":"replace","path":"/spec/addresses/0","value":"<new-platform-ip>/32"}]'

2. Point the Traefik Service at the same new IP. Use a strategic merge patch; a JSON Patch on spec.loadBalancerIP is rejected by the API server.

   sudo kubectl --namespace traefik patch svc traefik \
     --patch='{"spec":{"loadBalancerIP":"<new-platform-ip>"}}'

3. Restart the Cilium daemonset so its BPF load-balancer tables pick up the new mapping. This step is required. Without it, external clients continue to reach the old address.

   sudo kubectl --namespace kube-system rollout restart daemonset/cilium

4. Confirm the node's SSH port is reachable again from the jumpbox. Replace <node-ip> with the node's Host IP.

   curl --output /dev/null --silent --show-error --connect-timeout 5 telnet://<node-ip>:22 \
     && echo reachable

To prevent this on future installs, choose a platform IP that is not the node's Host IP when you complete the cluster profile wizard. Refer to Install the Appliance: Deploy the Cluster for the inline guidance.

Appliance console unreachable from the jumpbox after cluster deploy

Symptom. The Traefik Service reports the expected platform IP as its external IP, and pods are all Running, but https://<platform-ip>/ times out from the jumpbox. Requests from a browser on the same host as the node succeed.

Discriminating diagnostic. Inspect the Traefik Service's source-range allowlist.

sudo kubectl --namespace traefik get svc traefik --output=jsonpath='{.spec.loadBalancerSourceRanges}'

If the output contains a single /32 range that does not include the jumpbox's source IP, this issue applies.

Root cause. Local UI sets spec.loadBalancerSourceRanges on the Traefik Service to a /32 range containing only the platform IP. Because loadBalancerSourceRanges is a load-balancer-level ACL, source IPs outside the range are rejected before reaching Traefik.

Workaround. Remove the allowlist so external clients can reach the appliance console, or replace it with a CIDR that includes your jumpbox network.

sudo kubectl --namespace traefik patch svc traefik --type='json' \
  --patch='[{"op":"remove","path":"/spec/loadBalancerSourceRanges"}]'

"Model artifacts verified" step remains at "not present" after model loads

Symptom. After a model is uploaded, deployed, and serving requests successfully, the appliance console's model boot-sequence view shows Model artifacts verified stuck at No model artifacts present yet — nothing to verify.

Workaround. This is a cosmetic reporting gap and does not affect model serving. Use the vLLM pod status and a successful GET /v1/models response as the ground truth that the model is ready. Replace <token> with an API token from the appliance console and <platform-ip> with the appliance platform IP.

sudo kubectl get pods --all-namespaces | grep vllm
curl --silent --insecure https://<platform-ip>/v1/models --header "Authorization: Bearer <token>"

iLO HTML5 Console drops or duplicates keystrokes

Symptom. On HPE servers with iLO 7, typing longer strings (passwords, hostnames, GRUB edit-line modifications) in the HTML5 Remote Console results in dropped or duplicated characters. Escape sequences such as Ctrl-Alt-F2 or Esc + ( may not register.

Root cause. Browser HTML5 keyboard emulation over iLO's WebSocket transport drops keystrokes under load. HPE's Virtual Serial Port (vsp) over iLO SSH delivers raw scancodes and is not affected.

Workaround. From the jumpbox, SSH into iLO and attach vsp for any text-heavy interaction.

ssh <ipmi-user>@<ilo-ip>

At the iLO prompt, run the vsp command.

vsp

Return to the iLO prompt with Esc then (. Exit iLO with exit.

To skip the hostname field in the Palette TUI, leave the auto-generated edge-<uuid> value and rename the node later from Local UI. This avoids the worst of the HTML5 typing problem during initial configuration.

Stale iLO Virtual Serial Port session on iLO 7

Symptom. On HPE iLO 7, running stop /system1/oemhp_vsp1 at the iLO CLI to clear a stale Virtual Serial Port session returns unknown command. The SMASH CLP path documented for iLO 4 and iLO 5 does not apply.

Workaround. Reset the iLO management processor. This restarts iLO itself, not the server, and takes approximately 30 seconds.

reset /map1

Any HTML5 Remote Console session is dropped by the reset and must be re-launched from the iLO web UI after iLO finishes restarting. The server operating system is not affected.