Advertisement
kopyl

can't go over 40 iter

Jul 21st, 2023
814
0
Never
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
Python 0.81 KB | None | 0 0
  1. 2023-07-21 14:54:32,235 [INFO] Saving checkpoint at iters 40 to /workspace/LAVIS/train_output/20230721145/checkpoint_40.pth.
  2. Traceback (most recent call last):
  3.   File "/workspace/LAVIS/train.py", line 112, in <module>
  4.     main()
  5.   File "/workspace/LAVIS/train.py", line 107, in main
  6.     runner.train()
  7.   File "/workspace/LAVIS/lavis/runners/runner_iter.py", line 132, in train
  8.     dist.barrier()
  9.   File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 3327, in barrier
  10.     default_pg = _get_default_group()
  11.   File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 707, in _get_default_group
  12.     raise RuntimeError(
  13. RuntimeError: Default process group has not been initialized, please make sure to call init_process_group.
  14. Selection deleted
  15.  
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement