Best practices for using Ceph with CloudStack (RBD vs NFS integration)

[jeanvetorello]

Hello everyone,

I have a Ceph cluster running as backend storage, and I’m currently using RBD for Primary Storage in CloudStack with KVM. The challenge I’m facing is that when I try to create templates or snapshots, the process fails because the SSVM does not have direct access to Ceph. As a result, it cannot copy the snapshot or template properly.

From my experience, NFS seems to be a more complete integration with CloudStack and KVM, since everything (templates, ISOs, snapshots) works out of the box with the SSVM.

What I would like to achieve is a design where I can:

Take advantage of Ceph’s resiliency and availability, and

Still benefit from NFS integration for templates, snapshots, and SSVM operations.

One idea I had was to mount CephFS on all KVM hosts as a SharedMountPoint. This way, CloudStack would handle it like a shared NFS datastore, but the backend would still be Ceph.

My questions:

Is this a good alternative or are there known pitfalls with this approach?

What are the best practices for combining Ceph (RBD) and NFS in CloudStack to get the best of both worlds?

Should I stick with RBD for Primary and only use NFS for Secondary, or is there a better hybrid model?

Thanks in advance for your insights!

[weizhouapache]

@wido would be the best to answer

[jeanvetorello]

I was able to make it work using Ceph RBD as primary and NFSv3 as secondary.
The issue was that I initially had only NFSv4 enabled, and it seems there is a problem when CloudStack tries to mount NFSv4 pools on KVM hosts.

With NFSv3 enabled, the template creation worked as expected.
I’ll open a separate issue soon to track the NFSv4 mounting behavior

[bhouse-nexthop]

I have no problems using nfs-ganesha using cephfs with cloudstack 4.21 as secondary using NFSv4. I also use RBD as primary. What issue are you having with NFSv4?

I have ansible playbooks and config for my test environment here: https://github.com/bradh352/ansible-scripts/ it may provide some insight.

[wido]

The advise is still to keep the Secondary as NFS and Primary for RBD.

RBD is awesome, but the SSVM wants to perform a lot of posix filesystem operations on the SS, things Ceph can't do. Theoretically CephFS could be used, but this would require the SSVM to have all the tools available.

My advice: Keep a NFS server around for your SS.