Hi, when I try to train a model on my RTX 3090, it tells me "out of memory." Which variable can I set to reduce the CUDA memory usage?
Output folder: output/raw_hxl [15/01 21:58:51] Tensorboard not available: not logging progress [15/01 21:58:51] Reading camera 238/238Loading Training Cameras [15/01 21:58:55] Loading Test Cameras [15/01 21:59:58] Traceback (most recent call last): File "/home/wangcf/workspace/SparseGS/train.py", line 459, in <module> training(dataset, op.extract(args), pp.extract(args), args.test_iterations, args.save_iterations, args.checkpoint_iterations, args.start_checkpoint, args.debug_from, args.step, args.max_cameras, args.prune_sched) File "/home/wangcf/workspace/SparseGS/train.py", line 42, in training scene = Scene(dataset, gaussians, step=step, max_cameras=max_cameras) File "/home/wangcf/workspace/SparseGS/scene/__init__.py", line 110, in __init__ novel_cam_append_list.append(create_cam_obj(train_cam,degree, File "/home/wangcf/workspace/SparseGS/scene/generate_new_cam.py", line 144, in create_cam_obj return Camera(colmap_id= None, R=R, T=T, FoVx = cam.FoVx, FoVy = cam.FoVy, File "/home/wangcf/workspace/SparseGS/scene/cameras.py", line 52, in __init__ self.original_image = image.clamp(0.0, 1.0).to(self.data_device) torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 58.00 MiB. GPU 0 has a total capacity of 23.67 GiB of which 4.75 MiB is free. Including non-PyTorch memory, this process has 23.66 GiB memory in use. Of the allocated memory 23.32 GiB is allocated by PyTorch, and 13.24 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
Hi, when I try to train a model on my RTX 3090, it tells me "out of memory." Which variable can I set to reduce the CUDA memory usage?
Output folder: output/raw_hxl [15/01 21:58:51] Tensorboard not available: not logging progress [15/01 21:58:51] Reading camera 238/238Loading Training Cameras [15/01 21:58:55] Loading Test Cameras [15/01 21:59:58] Traceback (most recent call last): File "/home/wangcf/workspace/SparseGS/train.py", line 459, in <module> training(dataset, op.extract(args), pp.extract(args), args.test_iterations, args.save_iterations, args.checkpoint_iterations, args.start_checkpoint, args.debug_from, args.step, args.max_cameras, args.prune_sched) File "/home/wangcf/workspace/SparseGS/train.py", line 42, in training scene = Scene(dataset, gaussians, step=step, max_cameras=max_cameras) File "/home/wangcf/workspace/SparseGS/scene/__init__.py", line 110, in __init__ novel_cam_append_list.append(create_cam_obj(train_cam,degree, File "/home/wangcf/workspace/SparseGS/scene/generate_new_cam.py", line 144, in create_cam_obj return Camera(colmap_id= None, R=R, T=T, FoVx = cam.FoVx, FoVy = cam.FoVy, File "/home/wangcf/workspace/SparseGS/scene/cameras.py", line 52, in __init__ self.original_image = image.clamp(0.0, 1.0).to(self.data_device) torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 58.00 MiB. GPU 0 has a total capacity of 23.67 GiB of which 4.75 MiB is free. Including non-PyTorch memory, this process has 23.66 GiB memory in use. Of the allocated memory 23.32 GiB is allocated by PyTorch, and 13.24 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)