Skip to main content

Virtual Machine Orchestrator

The following are common scenarios that you may encounter when using Virtual Machine Orchestrator (VMO).

Scenario - Hot-Plugged Blank Disk Never Becomes Ready​

On PaletteAI VM Launchpad and VMO pack versions earlier than 4.10.25, when you hot-plug a disk to a running virtual machine and select Create New Blank Disk, the VMO UI creates the DataVolume with the ReadWriteMany access mode and the Block volume mode. If the StorageClass cannot provision Block volumes, for example because its StorageProfile allows only the Filesystem volume mode, the DataVolume never becomes ready. The virtual machine cannot use the disk, even though VMO displays a message that the volume is attached.

To work around this issue, create the DataVolume first, and then attach it as an existing DataVolume.

Debug Steps​

  1. Create a DataVolume in the namespace of the virtual machine with a Blank source. Set the Volume Mode to Filesystem, and select an Access Mode that the StorageClass supports, such as ReadWriteMany.

  2. On the Configuration tab of the virtual machine, select Hot-plug Disk.

  3. Select Existing DataVolume, and then select your DataVolume.

  4. Select Attach Disk.

Scenario - VMO Loading Errors in Self-Hosted Palette​

On self-hosted Palette and Palette VerteX installations, the cluster's Virtual Machines tab or the VMO Graphical UI (GUI) may fail to load. This issue applies to all installation methods but does not apply to self-hosted environments that use an IP address instead of a domain name.

The root cause is a Content-Security-Policy header with frame-ancestors set to https://*.spectrocloud.com, which blocks the VMO GUI from loading when your Palette instance uses a different domain name.

Debug Steps​

  1. Log in to your self-hosted Palette or Palette VerteX system console.

  2. From the left main menu, select Enterprise Cluster.

  3. On the Overview tab, download the Kubernetes Config File.

  4. From your terminal, set the KUBECONFIG variable to the file path of the kubeconfig.

    export KUBECONFIG=<path-to-kubeconfig>

  5. Verify the current Content-Security-Policy value for the following Traefik Middleware resources.

    kubectl get middleware ui-csp-frame-ancestors --namespace ui-system --output yaml
    kubectl get middleware cp-csp-frame-ancestors --namespace cp-system --output yaml
    kubectl get middleware foreq-security-headers --namespace hubble-system --output yaml
    Example output
    apiVersion: traefik.io/v1alpha1
    kind: Middleware
       metadata:
       annotations:
          meta.helm.sh/release-name: hubble
          meta.helm.sh/release-namespace: default
       creationTimestamp: "2026-04-07T15:03:54Z"
       generation: 1
       labels:
          app.kubernetes.io/managed-by: Helm
       name: ui-csp-frame-ancestors
       namespace: ui-system
       resourceVersion: "36023"
       uid: 2f87aada-b8cf-454f-8c32-38be34b060bf
    spec:
       headers:
          customResponseHeaders:
             Content-Security-Policy: frame-ancestors 'self' https://*.spectrocloud.com

  6. Update each applicable Middleware resource by replacing https://*.spectrocloud.com with your root domain. Replace <root-domain> in the following commands with your Palette root domain.

    tip

    Use the following command to identify your root domain.

    kubectl get configmap root-domain-info --namespace hubble-system --output jsonpath='{.data.apiEndpoint}'
    kubectl patch middleware ui-csp-frame-ancestors --namespace ui-system --type merge \
    --patch '{"spec":{"headers":{"customResponseHeaders":{"Content-Security-Policy":"frame-ancestors '\''self'\'' https://*.<root-domain> https://<root-domain>"}}}}'
    kubectl patch middleware cp-csp-frame-ancestors --namespace cp-system --type merge \
    --patch '{"spec":{"headers":{"customResponseHeaders":{"Content-Security-Policy":"frame-ancestors '\''self'\'' https://*.<root-domain> https://<root-domain>"}}}}'
    kubectl patch middleware foreq-security-headers --namespace hubble-system --type merge \
    --patch '{"spec":{"headers":{"customResponseHeaders":{"Content-Security-Policy":"frame-ancestors '\''self'\'' https://*.<root-domain> https://<root-domain>;"}}}}'

  7. Verify the updated values. Each Middleware resource should display a Content-Security-Policy header with frame-ancestors 'self' https://*.<root-domain> https://<root-domain>.

    kubectl get middleware ui-csp-frame-ancestors --namespace ui-system --output yaml
    kubectl get middleware cp-csp-frame-ancestors --namespace cp-system --output yaml
    kubectl get middleware foreq-security-headers --namespace hubble-system --output yaml
    Example output
    apiVersion: traefik.io/v1alpha1
    kind: Middleware
    metadata:
       annotations:
          meta.helm.sh/release-name: hubble
          meta.helm.sh/release-namespace: default
       creationTimestamp: "2026-04-07T15:03:54Z"
       generation: 2
       labels:
          app.kubernetes.io/managed-by: Helm
       name: ui-csp-frame-ancestors
       namespace: ui-system
       resourceVersion: "83150"
       uid: 2f87aada-b8cf-454f-8c32-38be34b060bf
    spec:
       headers:
          customResponseHeaders:
             Content-Security-Policy: frame-ancestors 'self' https://*.docs-test.spectrocloud.com
             https://docs-test.spectrocloud.com

Scenario - Virtual Machine (VM) Migration Plans in Unknown State​

When using the VM Migration Assistant to migrate VMs to your VMO cluster, migration plans can enter an Unknown state if more VMs are selected for migration than the Maximum concurrent VM migrations setting allows. This value determines the maximum number of VMs that can be migrated simultaneously across all active migration plans.

To avoid this scenario, we recommend setting an appropriate value for Maximum concurrent VM migrations based on your workload and expected migration patterns. A higher value helps prevent migration plans from entering an Unknown state due to excessive concurrency.

If migration plans do enter an Unknown state, use the following steps to resolve the issue.

Debug Steps​

  1. Access the VM Migration Assistant service console.

  2. From the left main menu, select Overview.

  3. Select the Settings tab.

  4. Click the Edit option next to Settings.

  5. In the pop-up window, increase the value for Maximum concurrent VM migrations based on the maximum number of VMs you expect to migrate concurrently across all active migration plans.

    For example, if you expect to migrate a maximum of 25 VMs across all active migration plans, set this value to 25 or more.

  6. Click Save after making the change.

You will now need to recreate any migration plans that are in an Unknown state, and start them again. Refer to Create Migration Plans for guidance.

Scenario - Virtual Machines (VM) Stuck in a Migration Loop​

Clusters with the VMO pack may experience VMs getting stuck in a continuous migration loop, as indicated by a Migrating or Migration VM status. Use the following steps to troubleshoot and resolve the issue.

Debug Steps​

  1. Log in to Palette and click Clusters from the left Main Menu.

  2. Select your cluster and click on the Profile tab.

  3. Select the VMO pack layer and click Values under the Pack Details section.

  4. Comment out the following lines under the kubevirtResource section and click Save.

    workloads: {}
    # workloadsUpdateStrategy:
    #     workloadUpdateMethods:
    #     - LiveMigrate

  5. The KubeVirt custom resource may fail to update after changing the VMO pack values in the Palette UI. To ensure the changes take effect, follow the Access Cluster with CLI guide to connect to your host cluster using the kubectl CLI.

  6. In your terminal, issue the following command to edit the KubeVirt custom resource.

    kubectl edit kubevirt --namespace kubevirt

  7. Comment out the following lines under the spec block and save the file.

    # workloadUpdateStrategy:
    #   workloadUpdateMethods:
    #   - LiveMigrate

  8. Within a few minutes, the VMs will stop being stuck in the continuous migration loop.

    warning

    Step 4 affects the CPU Hotplug functionality. If your cluster has CPU Hotplug enabled, you will need to manually restart your VM each time you add a virtual CPU for the changes to take effect.