SIGSEGV in cs_sles_solve_ccc_fv when CS_CUDA_ALLOC_DEVICE_UVM is not set (v9.1.0)

[diskdog]

Description

cs_sles_solve_ccc_fv allocates its extended solver buffers using CS_ALLOC_DEVICE (device-only memory). Those buffers are subsequently passed to the CPU-side convergence check, which dereferences them on the host and causes a SIGSEGV.

Setting CS_CUDA_ALLOC_DEVICE_UVM=1 before launching the solver works around the crash: it remaps cs_alloc_mode_device to CS_ALLOC_HOST_DEVICE_SHARED at initialisation time (see cs_base_cuda.cu:777), making the buffers accessible from both host and device.

Steps to reproduce

1. Build code_saturne v9.1.0 with CUDA support.
2. Run any case that exercises the CUDA sparse linear solver (cs_sles_solve_ccc_fv) without setting CS_CUDA_ALLOC_DEVICE_UVM.
3. Observe SIGSEGV during the convergence check after the first solver call.

Workaround

export CS_CUDA_ALLOC_DEVICE_UVM=1

Set this before invoking the solver. It prevents the crash by ensuring solver buffers are in unified/shared memory.

Expected behaviour

cs_sles_solve_ccc_fv should either allocate convergence-check buffers in host-accessible memory directly, or the convergence check should use a device-to-host copy rather than a direct host dereference.

Environment

• code_saturne version: 9.1.0
• CUDA version: 13.1
• GPU architecture: sm_75
• OS: Ubuntu (x86_64)

[YvanFournier]

Hello,

Do you observe this in the master branch ? Since GPU support is still work in progress, we test mostly on the master branch, not that much on v9.1. A subset of our testsuite runs every night on a few iterations, without forcing CS_CUDA_ALLOC_DEVICE_UVM, and I recall fixing a similar issue a few months (possibly after the v9.1 branching), so I assume the bug concerns v9.1 but not the soon to be released 9.2 (near the end of this month). I am not sure (by the commit messages) which commit actually fixed this.

For your information :

• The initial stages of GPU support involved explicit CPU/GPU memory host synchronization, but obviously, when we started to move more kernels to GPU (to avoid memory transfers as mush as obtain faster non-critical kernels), this was difficult to manage, as a same operator (gradient, solver) can be called in different code paths.
• So we counted on UVM to help us manage this, and this worked quite well, especially after adding hints such as "read_mostly" (in May 2025, this almost doubled performance).
• Since then, we have observed different behavior on different (theoretically similar) machines, where detailed profiling shows that using UVM can add some overhead in some cases. One one cluster, we observed shorter kernel run times, but longer intervals between kernels. Using kernel-only memory when UVM is not needed is faster, because the CUDA runtime does not need to do some mappings on CPU. Using a memory map (another option) is even better, and reduces the performance difference between UVM everywhere and UVM "only where absolutely needed", but this also increases GPU memory use.
• For this reason, and to make sure we do not forget to convert some loops to GPU, our approach is to use device-only allocation as much as possible: if this approach is too aggressive, this leads to a memory access error (as in this issue), which id easy to debug, whereas overlooked loops can lead to memory transfer overheads which require very detailed profiling to spot. This does have the disadvantage of causing issues along the way, but they should be less of an issue in future version now that our automated tests include GPU cases.

So I will look to apply your patch/merge request soon for v9.1 after reviewing it. For the master branch, it should not be necessary (and we are currently working on running more multigrid steps on the GPU only, so later this year, I hope that device-only allocations will be fine for most multigrid paths.

[diskdog]

Hiya!

You'll have to forgive me, I was getting all giddy about my first ever PR response (:tada:)

ahem

Do you observe this in the master branch?

I haven't seen it on master. I only hit it on v9.1.0 (sm_75, CUDA 13.1, channel-flow, CS_MATRIX_NATIVE).

I did have a dig through master to see how it stands now, though. On master, cs_sles_solve_ccc_fv lives in cs_sles_default.cpp, and the extended _vx / _rhs buffers are no longer allocated device-only unconditionally. amode is pulled from the matrix's own alloc mode and only drops to cs_alloc_mode_device inside a guard, with GPU dispatch switched on explicitly:

code_saturne/src/alge/cs_sles_default.cpp

Lines 914 to 921 in 5056ef5

	if (n_cols_ext > m->n_cells_with_ghosts) {
	cs_alloc_mode_t amode = cs_matrix_get_alloc_mode(a);
	cs_dispatch_context ctx;
	if (amode >= CS_ALLOC_HOST_DEVICE_SHARED) {
	amode = cs_alloc_mode_device;
	ctx.set_use_gpu(true);
	}

So in a non-shared build the buffers stay host-readable, which is the case that crashed on v9.1; in a shared build they go device-only deliberately, with GPU dispatch enabled for it. So the v9.1 crash path looks gone on master.

overlooked loops can lead to memory transfer overheads which require very detailed profiling to spot

Gotcha! I hadn't considered the full cost of UVM beyond the transfer itself, the host-side mappings and, from what you saw, the gaps that can open between kernels. Forcing device-only so a stray host read fails loudly, rather than quietly eating that overhead, makes sense as a default. Honestly, you've provided so much detail and I appreciate it.

Note

I've read the master path rather than run it, so treat anything about master as "from the code" rather than confirmed on a run.

Thanku!!
Keith