How to copy boundary point data from solid solver into AMReX particles with GPU enabled?

[Afraid31]

Hi all,

I am working on a fluid–structure interaction case. I added a solid solver (MBDmodel) into AMReX. Inside MBDmodel, I store boundary points of the solid in matpoints.

I need to copy the positions and velocities of these boundary points into AMReX particles. The relevant code looks like this:

{
    int level = finestLevel();

    for (MyParIter pti(*this, level); pti.isValid(); ++pti) {
        auto& particles = pti.GetArrayOfStructs();

        // Copy solid solver results into AMReX particles
        for (auto& p : particles) {
            auto matpoint = MBDmodel->matpoints[p.idata(Pibm_i::id)];
            const auto& coor = matpoint->Get_Coor();
            const auto& velo = matpoint->Get_Velo();

            p.pos(0) = coor[0];
            p.pos(1) = coor[1];
            p.rdata(Pibm::usx) = velo[0] / v_c;
            p.rdata(Pibm::usy) = velo[1] / v_c;
        }
         ......
         ......
        });
    }
}

This works fine without CUDA, but when I enable CUDA I get a segfault at this line:
auto matpoint = MBDmodel->matpoints[p.idata(Pibm_i::id)];
It seems like accessing p.idata(Pibm_i::id) inside the particle loop is not GPU-safe.

My question:

What is the recommended way to copy external data (from a solid solver) into AMReX particles when using GPU? Or is there a better AMReX-native approach?

[atmyers]

I would try adding the particles to a temporary ParticleTile that lives in host memory:

using HostTile = typename ContainerLike<std::allocator>::ParticleTileType;
HostTile host_tile;

Then, after adding the particles to the host_tile, you can copy them to the permanent one.

auto& particle_tile = DefineAndReturnParticleTile(lev, pti);
particle_tile.resize(new_size);
amrex::copyParticles(particle_tile, host_tile);

[Afraid31]

Thank you for your previous explanation — I understand your point. However, I still have some questions regarding the concrete implementation steps.

In my code, I encountered a runtime error in the

ImmersedBoundaryPoints::Update_Position_and_Velocity function. The issue happens specifically at this line:
auto matpoint = MBDmodel->matpoints[p.idata(Pibm_i::id)];

Here is the full function for reference:

void ImmersedBoundaryPoints::Update_Position_and_Velocity(amrex::Real v_c) {

    int level = finestLevel();

    for (MyParIter pti(*this, level); pti.isValid(); ++pti) {
        const Long np = pti.numParticles();

        using HostTile = typename ImmersedBoundaryPoints::ParticleTileType;
        HostTile host_tile;
        host_tile.resize(np);

        auto& gpu_tile = DefineAndReturnParticleTile(level, pti);

        // Copy particle data from GPU to host
        // Mainly copying particle ids (idata) to host for accessing matpoints
        amrex::copyParticles(host_tile, gpu_tile);
        auto& host_particles = host_tile.GetArrayOfStructs();
        for (int i = 0; i < np; ++i) {
            auto& p = host_particles[i];
            auto matpoint = MBDmodel->matpoints[p.idata(Pibm_i::id)];
            const auto& coor = matpoint->Get_Coor();
            const auto& velo = matpoint->Get_Velo();

            p.pos(0) = coor[0];
            p.pos(1) = coor[1];
            p.rdata(Pibm::usx) = velo[0] / v_c;
            p.rdata(Pibm::usy) = velo[1] / v_c;
#if AMREX_SPACEDIM == 3
            p.pos(2) = coor[2];
            p.rdata(Pibm::usz) = velo[2] / v_c;
#endif
        }
        // Copy updated particle data from host back to GPU
        // Mainly copying positions and velocities (rdata) back to device
        amrex::copyParticles(gpu_tile, host_tile);
    }
    Redistribute();
}

The crash occurs specifically when running with CUDA enabled. I suspect it might be related to how the particle idata values are accessed on the GPU.

Could you advise on the correct way to safely access matpoints when updating particle positions and velocities, or how to ensure idata is correctly initialized before this step?

Thank you in advance!