I am still getting this memory issue despite reducing the image resolution and disabling the warping : -
(SparseGS) forrealnew@sn4622126954:~/repos/gpu-server-dev/gpuserver/SparseGS$ python train.py --source_path data/mousenew --model_path output/mousenew_SparseGS --beta 5.0 --lambda_pearson 0.0 --lambda_local_pearson 0.0 --box_p 128 --p_corr 0.5 --lambda_diffusion 0.0 --SDS_freq 0.0 --step_ratio 0.99 --lambda_reg 0.0 --prune_sched 20000 --prune_perc 0.98 --prune_exp 7.5 --iterations 30000 --checkpoint_iterations 30000 -r 8
Optimizing output/mousenew_SparseGS
Output folder: output/mousenew_SparseGS [09/07 21:16:35]
Tensorboard not available: not logging progress [09/07 21:16:35]
Reading camera 1296/1296Loading Training Cameras [09/07 21:16:39]
Loading Test Cameras [09/07 21:17:10]
Number of points at initialisation : 68282 [09/07 21:17:12]
Training progress: 0%| | 0/30000 [00:00<?, ?it/s][20000] [09/07 21:17:12]
Training progress: 23%|███████████████▏ | 6990/30000 [00:27<01:44, 220.90it/s, EMA Loss=0.0207116, Total Loss=0.0207809]
[ITER 7000] Evaluating train: L1 0.029505799710750583 PSNR 28.724292755126953 [09/07 21:17:40]
[ITER 7000] Saving Gaussians [09/07 21:17:40]
Training progress: 67%|██████████████████████████████████████████▋ | 19990/30000 [01:28<00:42, 234.68it/s, EMA Loss=0.0229467, Total Loss=0.0323046]
Traceback (most recent call last):
File "/home/forrealnew/repos/gpu-server-dev/gpuserver/SparseGS/train.py", line 459, in
training(dataset, op.extract(args), pp.extract(args), args.test_iterations, args.save_iterations, args.checkpoint_iterations, args.start_checkpoint, args.debug_from, args.step, args.max_cameras, args.prune_sched)
File "/home/forrealnew/repos/gpu-server-dev/gpuserver/SparseGS/train.py", line 244, in training
prune_floaters(scene.getTrainCameras().copy(), gaussians, pipe, background, dataset, iteration)
File "/home/forrealnew/repos/gpu-server-dev/gpuserver/SparseGS/train.py", line 297, in prune_floaters
render_pkg = render(view, gaussians, pipe, background, ret_pts=True)
File "/home/forrealnew/repos/gpu-server-dev/gpuserver/SparseGS/gaussian_renderer/init.py", line 71, in render
rendered_image, radii, depth, num_gauss, alpha_depth, mode_id, modes, point_list, means2D, conic_opacity, = rasterizer(
File "/home/forrealnew/miniconda3/envs/SparseGS/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/forrealnew/miniconda3/envs/SparseGS/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
File "/home/forrealnew/miniconda3/envs/SparseGS/lib/python3.10/site-packages/diff_gaussian_rasterization/init.py", line 227, in forward
return rasterize_gaussians(
File "/home/forrealnew/miniconda3/envs/SparseGS/lib/python3.10/site-packages/diff_gaussian_rasterization/init.py", line 33, in rasterize_gaussians
return _RasterizeGaussians.apply(
File "/home/forrealnew/miniconda3/envs/SparseGS/lib/python3.10/site-packages/torch/autograd/function.py", line 575, in apply
return super().apply(*args, **kwargs) # type: ignore[misc]
File "/home/forrealnew/miniconda3/envs/SparseGS/lib/python3.10/site-packages/diff_gaussian_rasterization/init.py", line 99, in forward
num_rendered, color, depth, weight, depth_weight, num_gauss, mode_id, modes, alpha_depth, radii, point_list, means2D, conic_opacity, geomBuffer, binningBuffer, imgBuffer = _C.rasterize_gaussians(*args)
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 96.00 MiB. GPU 0 has a total capacity of 47.51 GiB of which 59.50 MiB is free. Process 201384 has 2.00 GiB memory in use. Process 224096 has 8.15 GiB memory in use. Process 267227 has 1.24 GiB memory in use. Process 268277 has 1.84 GiB memory in use. Process 1851254 has 7.69 GiB memory in use. Process 1851798 has 426.00 MiB memory in use. Process 3337120 has 434.00 MiB memory in use. Including non-PyTorch memory, this process has 25.60 GiB memory in use. Of the allocated memory 25.06 GiB is allocated by PyTorch, and 24.71 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
Training progress: 67%|██████████████████████████████████████████▋ | 20000/30000 [01:32<00:46, 216.93it/s, EMA Loss=0.0229467, Total Loss=0.0323046]
(SparseGS) forrealnew@sn4622126954:~/repos/gpu-server-dev/gpuserver/SparseGS$
I am still getting this memory issue despite reducing the image resolution and disabling the warping : -
(SparseGS) forrealnew@sn4622126954:~/repos/gpu-server-dev/gpuserver/SparseGS$ python train.py --source_path data/mousenew --model_path output/mousenew_SparseGS --beta 5.0 --lambda_pearson 0.0 --lambda_local_pearson 0.0 --box_p 128 --p_corr 0.5 --lambda_diffusion 0.0 --SDS_freq 0.0 --step_ratio 0.99 --lambda_reg 0.0 --prune_sched 20000 --prune_perc 0.98 --prune_exp 7.5 --iterations 30000 --checkpoint_iterations 30000 -r 8
Optimizing output/mousenew_SparseGS
Output folder: output/mousenew_SparseGS [09/07 21:16:35]
Tensorboard not available: not logging progress [09/07 21:16:35]
Reading camera 1296/1296Loading Training Cameras [09/07 21:16:39]
Loading Test Cameras [09/07 21:17:10]
Number of points at initialisation : 68282 [09/07 21:17:12]
Training progress: 0%| | 0/30000 [00:00<?, ?it/s][20000] [09/07 21:17:12]
Training progress: 23%|███████████████▏ | 6990/30000 [00:27<01:44, 220.90it/s, EMA Loss=0.0207116, Total Loss=0.0207809]
[ITER 7000] Evaluating train: L1 0.029505799710750583 PSNR 28.724292755126953 [09/07 21:17:40]
[ITER 7000] Saving Gaussians [09/07 21:17:40]
Training progress: 67%|██████████████████████████████████████████▋ | 19990/30000 [01:28<00:42, 234.68it/s, EMA Loss=0.0229467, Total Loss=0.0323046]
Traceback (most recent call last):
File "/home/forrealnew/repos/gpu-server-dev/gpuserver/SparseGS/train.py", line 459, in
training(dataset, op.extract(args), pp.extract(args), args.test_iterations, args.save_iterations, args.checkpoint_iterations, args.start_checkpoint, args.debug_from, args.step, args.max_cameras, args.prune_sched)
File "/home/forrealnew/repos/gpu-server-dev/gpuserver/SparseGS/train.py", line 244, in training
prune_floaters(scene.getTrainCameras().copy(), gaussians, pipe, background, dataset, iteration)
File "/home/forrealnew/repos/gpu-server-dev/gpuserver/SparseGS/train.py", line 297, in prune_floaters
render_pkg = render(view, gaussians, pipe, background, ret_pts=True)
File "/home/forrealnew/repos/gpu-server-dev/gpuserver/SparseGS/gaussian_renderer/init.py", line 71, in render
rendered_image, radii, depth, num_gauss, alpha_depth, mode_id, modes, point_list, means2D, conic_opacity, = rasterizer(
File "/home/forrealnew/miniconda3/envs/SparseGS/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/forrealnew/miniconda3/envs/SparseGS/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
File "/home/forrealnew/miniconda3/envs/SparseGS/lib/python3.10/site-packages/diff_gaussian_rasterization/init.py", line 227, in forward
return rasterize_gaussians(
File "/home/forrealnew/miniconda3/envs/SparseGS/lib/python3.10/site-packages/diff_gaussian_rasterization/init.py", line 33, in rasterize_gaussians
return _RasterizeGaussians.apply(
File "/home/forrealnew/miniconda3/envs/SparseGS/lib/python3.10/site-packages/torch/autograd/function.py", line 575, in apply
return super().apply(*args, **kwargs) # type: ignore[misc]
File "/home/forrealnew/miniconda3/envs/SparseGS/lib/python3.10/site-packages/diff_gaussian_rasterization/init.py", line 99, in forward
num_rendered, color, depth, weight, depth_weight, num_gauss, mode_id, modes, alpha_depth, radii, point_list, means2D, conic_opacity, geomBuffer, binningBuffer, imgBuffer = _C.rasterize_gaussians(*args)
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 96.00 MiB. GPU 0 has a total capacity of 47.51 GiB of which 59.50 MiB is free. Process 201384 has 2.00 GiB memory in use. Process 224096 has 8.15 GiB memory in use. Process 267227 has 1.24 GiB memory in use. Process 268277 has 1.84 GiB memory in use. Process 1851254 has 7.69 GiB memory in use. Process 1851798 has 426.00 MiB memory in use. Process 3337120 has 434.00 MiB memory in use. Including non-PyTorch memory, this process has 25.60 GiB memory in use. Of the allocated memory 25.06 GiB is allocated by PyTorch, and 24.71 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
Training progress: 67%|██████████████████████████████████████████▋ | 20000/30000 [01:32<00:46, 216.93it/s, EMA Loss=0.0229467, Total Loss=0.0323046]
(SparseGS) forrealnew@sn4622126954:~/repos/gpu-server-dev/gpuserver/SparseGS$