Failed to create volume from template on ceph storage

[svenknockaert]

We are experiencing an issue when creating a template from an instance’s volume.
In the GUI, the process seems to complete successfully — the status shows as “ready.”
However, when this image is then used to deploy (and start) a new instance, an error appears:

Error
Unable to start a VM [587b6b50-9b00-4604-b0e6-992e7013b110] due to [Unable to create a deployment for VM instance {"id":447,"instanceName":"i-71-447-VM","state":"Starting","type":"User","uuid":"587b6b50-9b00-4604-b0e6-992e7013b110"}].

The root cause of the problem is likely visible in the database:

Here is a working entry from the database:

+-------------+----+----------------------------------------------------------------+------------------------------------------------------------------------------------------------------+-----------------------------------------------------+------+
| template_id | id | install_path                                                   | local_path                                                                                           | error_str                                           | url  |
+-------------+----+----------------------------------------------------------------+------------------------------------------------------------------------------------------------------+-----------------------------------------------------+------+
|         237 | 67 | template/tmpl/2/237/916c05d8-fe0a-3290-a77b-0823c041db3d.qcow2 | /mnt/SecStorage/1609b869-700e-3b63-876b-3c8abfd394d6/template/tmpl/2/237/dnld2323592655182356492tmp_ | Install completed successfully at 9/19/25, 10:30 AM | NULL |
+-------------+----+----------------------------------------------------------------+------------------------------------------------------------------------------------------------------+-----------------------------------------------------+------+

But with my last attempts i get (not working entry):

+-------------+----+----------------------------------------------------------------+------------+-----------+------+
| template_id | id | install_path                                                   | local_path | error_str | url  |
+-------------+----+----------------------------------------------------------------+------------+-----------+------+
|         260 | 87 | template/tmpl/2/260/b12973d6-5c82-4b6d-bfa4-e552b9ece7ff.qcow2 | NULL       | NULL      | NULL |
+-------------+----+----------------------------------------------------------------+------------+-----------+------+
1 row in set (0.00 sec)

You can see that the “local_path” field is no longer populated, and there is also no successful error_str entry.
However, the secondary storage is accessible to the CloudStack Management Server, and the template has indeed been copied to the secondary storage (as shown by the install_path).

What could be the reason why CloudStack no longer defines a local_path, causing instance deployments based on that template to fail?

Using Cloudstack 4.20.1.0

UPDATE:

In the meantime, I know that it has nothing to do with the "local_path" field, because when I convert a volume from an existing Linux instance (e.g. Rocky) to a template, that works fine. I can then deploy new instances from it perfectly.

However, the problem occurs when converting the volume to a template of a Windows 11 instance. The Windows 11 instance was installed via the Windows ISO installation and works fine, and all virtio drivers are installed. The Windows 11 instance is UEFI secure, but that's also the case with the working Rocky Linux template, so it has nothing to do with the UEFI capabilities (I suppose).

In the logs I see the following:

com.cloud.exception.InsufficientServerCapacityException: 
Unable to create a deployment for VM instance ... 
Scope=interface com.cloud.dc.DataCenter; id=1

So I don't really understand the problem since it is definitely possible to run Windows 11 instances with UEFI Secure.
But deploying instances based on the template (on the same KVM hosts) doesn't work.

Does anyone have an idea why deploying instances from a Windows 11 template doesn't work?

[DaanHoogland]

@svenknockaert , are you using ubuntu as hypervisor OS (in which case see #11660)

[svenknockaert]

Well, /mnt/40ce3983-2f4b-322c-b28d-35c2cf39763f is a temporary, dynamically created mount, so it’s already gone.
However, I can still see the QCOW2 image via the CloudStack management server mount:

ikdoeict@pbaictcs:~$ ls -al /var/lib/cloudstack/mnt/206863092750034.502e2277/template/tmpl/2/263/
total 4366338
drwxrwxrwx  2 root root   4471128399 Oct  7 10:06 .
drwxrwxrwx 22 root root 105377352464 Oct  7 15:10 ..
-rw-r--r--  1 root root   4471128064 Oct  7 10:03 510f8711-4f6e-4c8c-82a7-0d4995cba56d.qcow2
-rw-r--r--  1 root root          335 Oct  7 10:06 template.properties

[weizhouapache]

looks good @svenknockaert
can you search for "510f8711-4f6e-4c8c-82a7-0d4995cba56d.qcow2 to RBD image" in agent.log on kvm hosts ?

[svenknockaert]

@weizhouapache
This is from the kvm host:

ikdoeict@pbaictkvm10:~$ grep -i 510f8711-4f6e-4c8c-82a7-0d4995cba56d.qcow2 /var/log/cloudstack/agent/agent.log
2025-10-07 08:34:08,741 WARN  [utils.script.Script] (AgentRequest-Handler-2:[]) (logid:) Execution of process [927472] for command [qemu-img convert -O raw -U --image-opts driver=qcow2,file.filename=/mnt/40ce3983-2f4b-322c-b28d-35c2cf39763f/510f8711-4f6e-4c8c-82a7-0d4995cba56d.qcow2 rbd:pool-ikdoeict-01/e29df9ab-9f74-45ca-8bf1-153f038badfd:mon_host=10.129.97.120\;10.129.97.125\;10.129.97.130:auth_supported=cephx:id=cloudstack:key=xxxxxxxxxx:rbd_default_format=2:client_mount_timeout=30 ] failed.
2025-10-07 08:34:08,741 WARN  [utils.script.Script] (AgentRequest-Handler-2:[]) (logid:) Process [927472] for command [qemu-img convert -O raw -U --image-opts driver=qcow2,file.filename=/mnt/40ce3983-2f4b-322c-b28d-35c2cf39763f/510f8711-4f6e-4c8c-82a7-0d4995cba56d.qcow2 rbd:pool-ikdoeict-01/e29df9ab-9f74-45ca-8bf1-153f038badfd:mon_host=10.129.97.120\;10.129.97.125\;10.129.97.130:auth_supported=cephx:id=cloudstack:key=xxxxxxxxxxxxxxxxxxx:rbd_default_format=2:client_mount_timeout=30 ] encountered the error: [qemu-img: error while reading at byte 3923771392: Input/output errorqemu-img: error while reading at byte 3919577088: Input/output errorqemu-img: error while reading at byte 3917479936: Input/output errorqemu-img: error while reading at byte 3925868544: Input/output errorqemu-img: error while reading at byte 3921674240: Input/output errorqemu-img: error while reading at byte 3913285632: Input/output errorqemu-img: error while reading at byte 3927965696: Input/output errorqemu-img: error while reading at byte 3915382784: Input/output error].
2025-10-07 08:34:08,742 ERROR [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-2:[]) (logid:) Failed to convert from /mnt/40ce3983-2f4b-322c-b28d-35c2cf39763f/510f8711-4f6e-4c8c-82a7-0d4995cba56d.qcow2 to rbd:pool-ikdoeict-01/e29df9ab-9f74-45ca-8bf1-153f038badfd:mon_host=10.129.97.120\;10.129.97.125\;10.129.97.130:auth_supported=cephx:id=cloudstack:key=xxxxxxxxxxxxxxxxxxxxxxxxxxxxx:rbd_default_format=2:client_mount_timeout=30 the error was: qemu-img: error while reading at byte 3923771392: Input/output errorqemu-img: error while reading at byte 3919577088: Input/output errorqemu-img: error while reading at byte 3917479936: Input/output errorqemu-img: error while reading at byte 3925868544: Input/output errorqemu-img: error while reading at byte 3921674240: Input/output errorqemu-img: error while reading at byte 3913285632: Input/output errorqemu-img: error while reading at byte 3927965696: Input/output errorqemu-img: error while reading at byte 3915382784: Input/output error

[weizhouapache]

@svenknockaert
can you run the command manually ?
it looks like the command is correct but stopped at a point.

[pavanaravapalli]

@svenknockaert
Looks like mount issue, it's unable to find mount point for command execution.
Please share the logs from the management server and the KVM agent for the operation.

[harikrishna-patnala]

@svenknockaert please check if title need change "Create instance failed!!!".

were you able to use any other new template to create instance on that specific host ? if not can you please try that.

[svenknockaert]

In the meantime, I’ve made progress.
Following your suggestions, I focused on this line in the logs:
“error while reading at byte 3919577088: Input/output error qemu-img: error while reading at byte 3917479936.”

That made me concentrate on the storage layer and why these read errors were happening.
It turned out that the underlying storage network was dropping quite a few packets.
After resolving that issue and deleting the raw data (because it said "error while converting raw: error rbd create: File exists") from the Ceph RBD storage, the problem seemed to be fixed.

[weizhouapache]

In the meantime, I’ve made progress.
Following your suggestions, I focused on this line in the logs:
“error while reading at byte 3919577088: Input/output error qemu-img: error while reading at byte 3917479936.”

That made me concentrate on the storage layer and why these read errors were happening.
It turned out that the underlying storage network was dropping quite a few packets.
After resolving that issue and deleting the raw data (because it said "error while converting raw: error rbd create: File exists") from the Ceph RBD storage, the problem seemed to be fixed.

good to hear