Can't boot your AlmaLinux 10 VM on Proxmox?

The symptom: a GitLab job that dies in one line

We maintain an unofficial proxmox-backup-client repository (source code on GitHub), which repackages the official binaries for other distributions so that our users can back up to Nimbus Backup from those systems. The guide to backing up a Linux server with proxmox-backup-client covers installation. The repository's GitLab CI tests package installation in 27 containers, including almalinux:10.

The runners use the Docker executor and run in VMs on our Proxmox VE 9.2.5 cluster. The nodes are AMD EPYC 7702 (64 cores, 128 threads). The affected runner's VM uses the x86-64-v2-AES CPU type, the one Proxmox suggests by default when creating a VM. The almalinux:10 container does not even start:

Using Docker executor with image almalinux:10 ...
Executing "step_script" stage of the job script
Fatal glibc error: CPU does not support x86-64-v3
ERROR: Job failed: exit code 127

The hardware is not the problem. The EPYC 7702 (Zen 2, "Rome") supports AVX2, and therefore x86-64-v3. It does not reach x86-64-v4 for lack of AVX-512, as we found out the hard way (see below). This is the textbook case of a recent server whose VM is held back by its CPU type.

When installing directly in a VM, the message does not always show on screen. On the Proxmox forum, users describe an installer that keeps looping back to the boot menu or a VM that reboots right after GRUB.

The cause: x86-64-v3 required, x86-64-v2-AES by default

RHEL 10 is built for x86-64-v3 and drops many older processors. Red Hat documents the failure in its article "RHEL 10 cannot boot on older CPU that does not support x86-64-v3". AlmaLinux 10 follows that choice by default, and so does Rocky Linux 10.

Proxmox suggests x86-64-v2-AES by default for a new VM. That type does not expose AVX2, even when the physical CPU supports it.

Docker does not work around it: a container has no virtual CPU of its own, it uses the one the VM sees. If the VM does not expose x86-64-v3, an AlmaLinux 10, RHEL 10 or Rocky Linux 10 container crashes the same way, whatever the physical host.

A Proxmox LXC container is a different story: it has no CPU type, shares the host kernel and sees the physical processor directly. It only hits this failure if the physical processor itself lacks AVX2.

Check before changing anything

Run the same command on each Proxmox node, then inside the affected VM (the runner's, for instance):

grep -o -w -E 'avx2|bmi2|fma|movbe' /proc/cpuinfo | sort -u

• If avx2 shows up on the node but not in the VM, the VM's CPU type is hiding it.
• If avx2 is missing on the node itself, no CPU type will change anything: see the no-AVX2 hardware case below.

The fixes and their consequences

x86-64-v3: the sensible default in a cluster

If every node in the cluster is Intel Haswell, AMD EPYC or newer, x86-64-v3 gives the VM what AlmaLinux 10 needs while keeping live migration between those nodes.

qm set <vmid> --cpu x86-64-v3

x86-64-v4: no higher than what the host supports

x86-64-v3 would have been enough. We chose to try x86-64-v4, one level up, straight away. That type adds AVX-512 and, according to the Proxmox documentation, needs at least an Intel Skylake or an AMD EPYC Genoa. The EPYC 7702 has no AVX-512: the VM refuses to start.

kvm: warning: host doesn't support requested feature: CPUID[eax=07h,ecx=00h].EBX.avx512f [bit 16]
… avx512dq, avx512cd, avx512bw, avx512vl …
kvm: Host doesn't support requested features
TASK ERROR: start failed: QEMU exited with code 1

AlmaLinux 10 only needs x86-64-v3. Aiming higher brings nothing here, and prevents the VM from starting on a node that lacks those instructions.

host: the fastest, without migration to a different node

With host, the VM gets exactly the host CPU's instructions. It is the fastest type. The trade-off: live migration to a node that lacks those instructions fails (the QEMU process stops), and migration between Intel and AMD is not guaranteed.

qm set <vmid> --cpu host

That is the type we fell back to after x86-64-v4 failed: our gitlab-runner VMs were switched to host, followed by a VM restart. The pipeline is green again. They run in a cluster whose nodes all have the same processor: live migration between them still works. If your VM must be able to migrate between different nodes, prefer x86-64-v3.

qm reboot <vmid>    # shutdown + start, applies the new CPU type

Hardware without AVX2: AlmaLinux's x86_64_v2 variant

AlmaLinux 10 is built for x86-64-v3 like RHEL, but also publishes an x86_64_v2 architecture (separate ISO and repositories) for older hardware.

• For a VM: install from the AlmaLinux 10 x86_64_v2 ISO.
• For Docker: the official almalinux:10 image (Docker Official Library) does not exist for linux/amd64/v2. The v2 variants are only published in the images maintained by AlmaLinux (almalinux/10-base, almalinux/10-minimal, almalinux/10-init…).

docker pull --platform linux/amd64/v2 almalinux/10-minimal

The limit: AlmaLinux reserves it for older hardware, and third-party repositories rarely publish for this variant. It only fits when the base package set is enough, unless you rebuild packages yourself. An EPEL rebuild for v2 has been approved. Rocky Linux 10 and RHEL 10 have no such way out: Rocky Linux 10 does not publish an x86-64-v2 build.

In a cluster, the oldest node sets the limit

Coming from VMware?

This is not specific to Proxmox. The same message shows up under VirtualBox and under VMware vSphere with EVC in Sandy Bridge mode, which hides AVX2. A vSphere cluster set up that way to keep migration between old and new hosts has exactly the same problem as x86-64-v2-AES on Proxmox.

If you are migrating VMs from vSphere, pick the CPU type when creating each Proxmox VM, based on the nodes of the target cluster. Our VMware to Proxmox migration guide covers the other post-migration settings, and the VMware to Proxmox migration page describes how we run these projects.

Frequently asked questions

Sources

Related articles