Hi! I think this error is due to some crash with GPU, not sure if it is possible to configurate this to be able to continue the training? I'm using a NVIDIA GeForce RTX 3090

return Variable._execution_engine.run_backward(
RuntimeError: CUDA out of memory. Tried to allocate 512.00 MiB (GPU 0; 24.00 GiB total capacity; 20.72 GiB already allocated; 0 bytes free; 21.06 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF