Barney backend failing on all scenes in AnariCat tool tests with OWL data synchronization errors

[jmb64]

I ran the anariCAT tool (https://www.kitware.com/anari-capability-analysis-tool/)
on the series of scenes available with 4 different libraries (helide, visrtx, tsd, barney).
The first three libraries render every scenes but barney is failing every one of them
with an OWL data synchronization error ...

Buffer.cpp (line 120) --> void DeviceBuffer::upload(const void *hostPtr, size_t offset, int64_t count)
{
assert(deviceData.size() == context->deviceCount());
for (auto dd : deviceData)
dd->asDeviceBuffer::DeviceData().uploadAsync(hostPtr, offset, count);
OWL_CUDA_SYNC_CHECK();
}
UserGeomGroup.cpp (line 283), TrianglesGeomGroup.cpp ( line 270) --> optixAccelBuild()

-----------
Run barney on scene demo:cornell_box 
anariCAT run -l barney   -a -n 3 -s demo:cornell_box 
error (/home/jmb001/data/ppp7/soft/U3/ANARI/nvidia_barney/barney/submodules/owl/owl/TrianglesGeomGroup.cpp: line 270): invalid argument
fatal cuda error
-----------
Run barney on scene demo:gravity_spheres_volume 
anariCAT run -l barney   -a -n 3 -s demo:gravity_spheres_volume 
[barney:anari:Volume: TransferFunction1D::commitParameters] m_valueRange: lower=0, upper=10
error (/home/jmb001/data/ppp7/soft/U3/ANARI/nvidia_barney/barney/submodules/owl/owl/Buffer.cpp: line 122): invalid argument
fatal cuda error
-----------
Run barney on scene file:obj 
anariCAT run -l barney   -a -n 3 -s file:obj 
0/3 frames  0s/60s elapsed[ERROR][0x0x3010b00]app set camera.position to NAN coordinates
[ERROR][0x0x3010b00]app set camera.direction to NAN coordinates
[ERROR][0x0x3010b00]app set camera.up to NAN coordinates
3/3 frames  0.2s/60.0s elapsed

===================
Performance Summary
===================
Total # frames rendered 3
Scene build     0.1ms
Device
        Avg     56.8ms
        Min     2.3ms
        Max     165.4ms
Scene update
        Avg     0.0ms
        Min     0.0ms
        Max     0.0ms
Saved performance metrics to "/fs/homeu3/eccc/mrd/rpnem/jmb001/src/ANARI/PYNARI/FRONTENDS/Qt5/EXPLODED_FE_REMAINDER/metrics.csv"
-----------
Run barney on scene perf:materials 
anariCAT run -l barney   -a -n 3 -s perf:materials 
error (/home/jmb001/data/ppp7/soft/U3/ANARI/nvidia_barney/barney/submodules/owl/owl/UserGeomGroup.cpp: line 283): invalid argument
fatal cuda error
-----------
Run barney on scene perf:particles 
anariCAT run -l barney   -a -n 3 -s perf:particles 
error (/home/jmb001/data/ppp7/soft/U3/ANARI/nvidia_barney/barney/submodules/owl/owl/UserGeomGroup.cpp: line 283): invalid argument
fatal cuda error
-----------
Run barney on scene perf:primitives 
anariCAT run -l barney   -a -n 3 -s perf:primitives 
error (/home/jmb001/data/ppp7/soft/U3/ANARI/nvidia_barney/barney/submodules/owl/owl/UserGeomGroup.cpp: line 283): invalid argument
fatal cuda error
-----------
Run barney on scene perf:spinning_cubes 
anariCAT run -l barney   -a -n 3 -s perf:spinning_cubes 
error (/home/jmb001/data/ppp7/soft/U3/ANARI/nvidia_barney/barney/submodules/owl/owl/TrianglesGeomGroup.cpp: line 270): invalid argument
fatal cuda error
-----------
Run barney on scene perf:surfaces 
anariCAT run -l barney   -a -n 3 -s perf:surfaces 
error (/home/jmb001/data/ppp7/soft/U3/ANARI/nvidia_barney/barney/submodules/owl/owl/TrianglesGeomGroup.cpp: line 270): invalid argument
fatal cuda error
-----------
Run barney on scene test:instanced_cubes 
anariCAT run -l barney   -a -n 3 -s test:instanced_cubes 
error (/home/jmb001/data/ppp7/soft/U3/ANARI/nvidia_barney/barney/submodules/owl/owl/TrianglesGeomGroup.cpp: line 270): invalid argument
fatal cuda error
-----------
Run barney on scene test:pbr_spheres 
anariCAT run -l barney   -a -n 3 -s test:pbr_spheres 
error (/home/jmb001/data/ppp7/soft/U3/ANARI/nvidia_barney/barney/submodules/owl/owl/UserGeomGroup.cpp: line 283): invalid argument
fatal cuda error
-----------
Run barney on scene test:random_cylinders 
anariCAT run -l barney   -a -n 3 -s test:random_cylinders 
error (/home/jmb001/data/ppp7/soft/U3/ANARI/nvidia_barney/barney/submodules/owl/owl/UserGeomGroup.cpp: line 283): invalid argument
fatal cuda error
-----------
Run barney on scene test:random_spheres 
anariCAT run -l barney   -a -n 3 -s test:random_spheres 
error (/home/jmb001/data/ppp7/soft/U3/ANARI/nvidia_barney/barney/submodules/owl/owl/UserGeomGroup.cpp: line 283): invalid argument
fatal cuda error
-----------
Run barney on scene test:textured_cube 
anariCAT run -l barney   -a -n 3 -s test:textured_cube 
error (/home/jmb001/data/ppp7/soft/U3/ANARI/nvidia_barney/barney/submodules/owl/owl/TrianglesGeomGroup.cpp: line 270): invalid argument
fatal cuda error
-----------
Run barney on scene test:triangle_attributes 
anariCAT run -l barney   -a -n 3 -s test:triangle_attributes 
[WARN ][0x0x549c7f0]banari object type 'ANARI_SAMPLER' of subtype 'primitive' not implemented
[WARN ][0x0x549cc00]banari object type 'ANARI_SAMPLER' of subtype 'primitive' not implemented
error (/home/jmb001/data/ppp7/soft/U3/ANARI/nvidia_barney/barney/submodules/owl/owl/TrianglesGeomGroup.cpp: line 270): invalid argument
fatal cuda error

[ingowald]

finally got some time to look into this - it actually looks like "somebody" creates some cuda error state even before the barney anari device gets loaded; this in turn sets an invalid state that barney/owl then later pick up and treat as if it was their own error. haven't found where this happens yet, though.

[ingowald]

Tracked it down (at least this first one): it's an OWL bug that sets an invalid error state on owl tear-down; anariCat tool for some reason creates a temporary device that it immediately tears down, and doing so sets the error state for the second - real - device even before it gets created.

[jmb64]

If it's an OWL bug do you expect other side effects in barney or is this specifically related to anariCat application design ?
Too early to say ?