Troubleshooting VM Launchpad

This page provides troubleshooting guidance for common scenarios you may encounter when using the PaletteAI VM Launchpad.

Scenario - Large VM Stays Pending with Insufficient Memory

A VM with a large memory size, such as a 16 GiB VM on nodes with 32 GiB of memory, does not start, and kubectl get vm reports its status as ErrorUnschedulable. Its virt-launcher pod stays in the Pending state, and the pod events show a message such as 0/3 nodes are available: 3 Insufficient memory.

This occurs because Kubernetes places a VM only on a node that has enough allocatable memory left after the memory requests of the pods that already run on it. Allocatable memory is lower than the total memory of the node, because each node reserves memory for host services. On each control plane node, etcd and the Kubernetes API server also request part of the allocatable memory. Refer to Node Memory Reservation for the amounts.

Make Room for the VM

You need kubectl access to the affected VM Launchpad cluster. From the cluster Overview tab, download the Kubeconfig file, and then set the KUBECONFIG environment variable to point to it.

1. Check how much memory each node can still give to new pods. The loop lists the memory that the pods on each node request, and the last command lists the allocatable memory of each node.

   for node in $(kubectl get nodes --output name); do
     echo "$node"
     kubectl describe "$node" | grep --after-context=6 "Allocated resources"
   done
   kubectl get nodes --output custom-columns=NODE:.metadata.name,ALLOCATABLE:.status.allocatable.memory

2. Free memory for the VM. Stop or migrate other VMs on a node, lower the memory size of the VM, or add nodes to the cluster.

3. Confirm that the VM reaches the Running status.

   kubectl get vm <vm-name> --namespace <vm-namespace>

   Replace <vm-name> with the name of the VM and <vm-namespace> with the namespace of the VM. The STATUS column shows Running when the VM starts.

Scenario - Portworx Storage Cluster Does Not Initialize on VMware vSphere VMs

When you deploy the Portworx variant of VM Launchpad on VMware vSphere VMs, such as for a proof of concept, the Portworx StorageCluster does not finish initializing.

This occurs because vSphere virtual disks report to the guest operating system as rotational disks by default. Portworx requires the disks that it uses to report as non-rotational. When a disk reports as rotational, Portworx classifies it as a MAGNETIC device, and the storage cluster does not initialize.

Use one of the following options before you create the Portworx storage cluster. Apply the option on each node where Portworx is installed. For consistency, you can apply it on every node in the cluster.

• Mark the Virtual Disks as SSD in vSphere. This option marks each virtual disk as a Solid-State Drive (SSD). The setting persists across reboots, so it is the preferred fix when you provision the VMs.
• Override the Rotational Setting on Each Node. This option changes the setting from inside each node, without powering off the VMs. It takes effect immediately, but it does not persist across reboots.

Mark the Virtual Disks as SSD in vSphere

1. Power off the VM.

2. Add an advanced configuration parameter to the VM for each virtual disk that Portworx uses. Set the parameter name to scsi<controller-id>:<disk-id>.virtualSSD and the value to 1.

   Replace <controller-id> with the ID of the Small Computer System Interface (SCSI) controller and <disk-id> with the ID of the virtual disk on that controller. For example, set scsi0:1.virtualSSD to 1 to mark the second disk on the first SCSI controller as an SSD. Refer to How to Emulate an SSD Virtual Disk on a Virtual Machine for details.

3. Power on the VM.

4. Repeat steps 1 - 3 for each VM where Portworx is installed.

5. Create the Portworx storage cluster. Refer to Storage Clusters for instructions.

6. From the VMO left main menu, select Infrastructure > Storage > Portworx Storage Clusters, and confirm that the Phase of the storage cluster is Running.

Override the Rotational Setting on Each Node

This override takes effect immediately, but it does not persist across reboots. To keep the setting, mark the virtual disks as SSD in vSphere.

1. Use SSH to sign in to a cluster node with the account that you created when you installed VM Launchpad.

2. List the disks and their rotational setting. A value of 1 in the ROTA column means that the disk reports as rotational.

   lsblk --nodeps --output NAME,ROTA,SIZE
   Example Output

   NAME ROTA   SIZE
   sda     1   500G
   sdb     1   200G

3. Set each disk that Portworx uses to non-rotational.

   echo 0 | sudo tee /sys/block/<device-name>/queue/rotational

   Replace <device-name> with the disk name, such as sda or sdb. Use the name of the whole disk, even if Portworx uses a partition on it, such as sda6. Run the command once for each disk. Use the disks that you plan to select in the Storage section of the storage cluster configuration.

4. Repeat steps 1 - 3 on each node where Portworx is installed.

5. Create the Portworx storage cluster. Refer to Storage Clusters for instructions.

6. From the VMO left main menu, select Infrastructure > Storage > Portworx Storage Clusters, and confirm that the Phase of the storage cluster is Running.

Scenario - Keycloak, VMO, and Headlamp Consoles Become Inaccessible on FIPS Appliances

On FIPS VM Launchpad appliances, which use Piraeus storage, network disruptions can cause the DRBD replicas that back the Keycloak PostgreSQL database to lose their connection. When this happens, the Keycloak login page stops responding. The VMO and Headlamp consoles authenticate through Keycloak, so they also become inaccessible. The Keycloak and PostgreSQL pods still report a Ready status.

This occurs because the DRBD resource path is missing or uses different network interfaces on its peer nodes, which suspends database I/O. Non-FIPS appliances, including the Non-FIPS Piraeus variant, are not affected, because they store the Keycloak database on node-local storage instead of DRBD replicas. Use the following steps to restore the connection manually.

Prerequisites

• You have kubectl access to the affected VM Launchpad cluster. From the cluster Overview tab, download the Kubeconfig file, and then set the KUBECONFIG environment variable to point to it.

• Run every command in this scenario from the same terminal session. The steps set environment variables, such as NS, RESOURCE, NODE_A, and NODE_B, that later steps reuse.

Identify the LINSTOR Resource

Find the PostgreSQL data PVC, resolve it to its LINSTOR resource, and confirm that at least one replica reports an UpToDate state. If no replica is UpToDate, investigate replica health instead of changing the path.

1. Set the namespace that hosts Keycloak.

   NS=keycloak

2. List the persistent volume claims in the namespace, and confirm that the list includes the pgdata-postgres-0 claim.

   kubectl --namespace "$NS" get pvc

3. Assign the PostgreSQL data PVC, pgdata-postgres-0, to a variable.

   PVC=pgdata-postgres-0

4. Resolve the claim to its persistent volume.

   PV=$(kubectl --namespace "$NS" get pvc "$PVC" --output jsonpath='{.spec.volumeName}')

5. Resolve the persistent volume to its LINSTOR resource.

   RESOURCE=$(kubectl get pv "$PV" --output jsonpath='{.spec.csi.volumeHandle}')

6. List the LINSTOR resource and confirm that at least one replica reports an UpToDate state.

   kubectl --namespace piraeus-system exec deploy/linstor-controller -- \
     linstor resource list --resources "$RESOURCE"

7. List the resource connections, grouped by source and target.

   kubectl --namespace piraeus-system exec deploy/linstor-controller -- \
     linstor resource-connection list "$RESOURCE" --groupby source target properties port

Inspect the DRBD Connection Paths

List every declared connection path and its interface, then inspect the interfaces on each source and target node pair.

1. List every connection path and its interface.

   kubectl get linstornodeconnection \
     --output go-template='{{printf "PATH_NAME\tINTERFACE\n"}}{{range .items}}{{range .spec.paths}}{{printf "%s\t%s\n" .name .interface}}{{end}}{{end}}'

2. Assign the path name, interface, and node names to variables. Replace <path-name> with the connection path, <interface> with the DRBD interface, and <source-node> and <target-node> with the peer node names from the previous step.

   PATH_NAME=<path-name>
   DRBD_IF=<interface>
   NODE_A=<source-node>
   NODE_B=<target-node>

3. Inspect the interfaces on the source node.

   kubectl --namespace piraeus-system exec deploy/linstor-controller -- linstor node interface list "$NODE_A"

4. Inspect the interfaces on the target node.

   kubectl --namespace piraeus-system exec deploy/linstor-controller -- linstor node interface list "$NODE_B"

5. List the existing connection path for the node pair.

   kubectl --namespace piraeus-system exec deploy/linstor-controller -- \
     linstor resource-connection path list "$NODE_A" "$NODE_B" "$RESOURCE"

Restore the Connection Path

If either node is missing the DRBD interface, inspect the piraeus-netiface-builder DaemonSet. To change the storage node interface, update the csi.storageNodeInterface variable in the piraeus-operator pack in your cluster profile, and then redeploy the pack. Otherwise, create or correct each declared path. You can safely rerun the command for the same path name.

kubectl --namespace piraeus-system exec deploy/linstor-controller -- \
  linstor resource-connection path create "$NODE_A" "$NODE_B" "$RESOURCE" "$PATH_NAME" "$DRBD_IF" "$DRBD_IF"

Verify PostgreSQL

After you restore the path, confirm that the connection recovers and that the database serves requests. A pg_isready check only confirms that the database accepts connections, so run an actual query instead.

1. Confirm that every connection for the resource reports an Ok state.

   kubectl --namespace piraeus-system exec deploy/linstor-controller -- \
     linstor resource-connection list "$RESOURCE"

2. List the pods in the namespace, and confirm that the list includes the postgres-0 pod.

   kubectl --namespace "$NS" get pods

3. Set the database pod and credential values.

   DB_POD=postgres-0
   DB_SECRET=keycloak-db-credentials
   DB_SECRET_KEY=POSTGRES_PASSWORD

4. Retrieve the password and run the query. Piping the password into the pod over standard input keeps it out of your shell history and the command arguments.

   kubectl --namespace "$NS" get secret "$DB_SECRET" \
     --output jsonpath="{.data.$DB_SECRET_KEY}" | base64 --decode | \
     kubectl --namespace "$NS" exec --stdin "$DB_POD" --container postgres -- \
     sh -c 'PGCONNECT_TIMEOUT=5 PGPASSWORD="$(cat)" \
       psql --host 127.0.0.1 --username keycloak --dbname keycloak --tuples-only --no-align \
       --command "SET statement_timeout=5000; SELECT 1"'

Scenario - Federated LDAP Users Cannot Access VM Launchpad

Federated LDAP users sign in to Keycloak successfully, but they have no access once they reach VM Launchpad. The following symptoms indicate this scenario.

• The user authenticates against Keycloak and the browser redirects back to VM Launchpad, but no resources are available.

• Assigning a VMO role to the user appears to succeed, but the role grants no permissions.

• Requests to the Kubernetes API made with the user's credentials are rejected as unauthenticated.

• On the Settings > Access Management > Users page, the Email column is empty for the account.

This occurs because the Kubernetes API server in a VM Launchpad cluster runs with --oidc-username-claim=email. Tokens must carry an email claim and email_verified: true. LDAP directories frequently leave the mail attribute empty, so Keycloak imports the account without an address and issues a token that the API server cannot map to a username.

Confirm the Missing Email Claim

1. Log in to the Keycloak admin console as an administrator and select your realm.

2. From the left main menu, select Users, and then select the affected account.

3. Check the Email field and the Email verified toggle.

   • If Email is empty, the LDAP attribute that you mapped to email is not populated for this account.

   • If Email is populated but Email verified is disabled, the LDAP provider does not trust addresses from the directory.

4. Resolve both conditions by mapping an email-formatted LDAP attribute, such as userPrincipalName, and enabling Trust Email on the LDAP provider. Refer to Federate LDAP Users with Keycloak for the full procedure.

5. Run a new synchronization from the LDAP provider so that the mapper applies to accounts that Keycloak already imported. A synchronization does not set Email verified on accounts that already exist, so correct those accounts separately. Refer to Correct Previously Imported Accounts.

6. Ask the affected users to sign out and sign in again. Tokens issued before the change do not carry the new claims.

Scenario - VM Migration Fails During Guest Conversion on Block-Based Storage

When you use the VM Migration Assistant to migrate VMs to a VMO cluster backed by a block-based Container Storage Interface (CSI), such as the LINSTOR/DRBD storage used by the VM Launchpad, migrations can fail during the guest conversion (ConvertGuest) phase. The migration plan reports only a generic message. The following text is an example of the message.

error: { phase: "ConvertGuest", reasons: ["Guest conversion failed. See pod logs for details."] }

This occurs because the destination volumes are provisioned in Block volume mode. During conversion, virt-v2v serves the raw block device through nbdkit, which probes the device's minimum block size. When the underlying storage reports a block size outside the range that nbdkit accepts, the conversion fails. Linux guests (for example, Ubuntu) are most commonly affected, while Windows guests may still succeed.

Use the following steps to confirm the cause and apply a workaround.

Debug Steps

1. Access the VM Migration Assistant service console and confirm that the affected VMs failed at the ConvertGuest phase.

2. Connect to your host cluster using kubectl. Follow the Access Cluster with CLI guide to obtain access.

3. Inspect the logs of the guest conversion pod for the failed migration plan. The conversion pods run in the migration namespace and are named after the plan and VM, for example plan-01-vm-*.

   kubectl logs --namespace <migration-namespace> <conversion-pod-name>

4. Confirm the failure signature. Logs that contain the following lines indicate that nbdkit is operating on a block device whose minimum block size it cannot handle.

   nbdkit: file[1]: debug: extents disabled: lseek: SEEK_HOLE: Invalid argument
   nbdkit: file[1]: error: plugin must set minimum block size between 1 and 64K

5. Route the migration to a Filesystem-mode storage class to work around the issue. Create a storage class that uses the same provisioner and parameters as your default storage class, and patch its StorageProfile to force Filesystem volume mode.

   spec:
     claimPropertySets:
       - accessModes: ["ReadWriteOnce"]
         volumeMode: Filesystem

6. Recreate the migration plan and, in the Storage map step, map the source storage to the new Filesystem-mode storage class. Ensure the storage map specifies accessMode: ReadWriteOnce and volumeMode: Filesystem. Refer to Create Migration Plans for guidance, then start the plan again.

   warning

   This workaround is temporary. On block-based storage such as LINSTOR/DRBD, Filesystem mode is ReadWriteOnce (RWO) only. VMs migrated this way cannot be live-migrated, because KubeVirt live migration requires ReadWriteMany (RWX), which is only available in Block mode on this storage. To restore live migration, apply the permanent fix described in the next section.

Restore Live Migration with Updated Guest Conversion Image

The workaround in the previous section unblocks migrations by using Filesystem volume mode, but it disables live migration for the migrated VMs. To restore live migration, load an updated forklift-virt-v2v content bundle into the cluster's Zot registry. The updated image contains a newer nbdkit that handles the block-size constraints reported by block-based storage such as LINSTOR/DRBD.

1. Contact your Spectro Cloud support representative to obtain the updated forklift-virt-v2v content bundle. The bundle replaces the existing image at the following reference.

   us-docker.pkg.dev/palette-images/third-party/vm-migration-assistant/forklift-virt-v2v:4.9.2

2. Remove the existing forklift-virt-v2v image from the cluster's Zot registry so that the new image can be uploaded in its place. Your Spectro Cloud support representative can guide you through this step for your environment.

3. Upload the updated content bundle to the cluster. Use either the Local UI upload flow or the Palette CLI.

4. Recreate the failed migration plan. In the Storage map step, map the source storage to your default Block-mode storage class and remove the Filesystem/ReadWriteOnce overrides that were added as part of the workaround. Refer to Create Migration Plans for guidance.

5. Start the migration plan. Confirm that guest conversion completes without the nbdkit block-size error, and that the migrated VMs support live migration on the destination storage.