API tools faq
paste
Advertisement
kopyl

Untitled

kopyl
Jan 27th, 2024
1,085
0
Never
Add comment
Not a member of Pastebin yet? Sign Up, it unlocks many cool features!
Python 11.97 KB | None | 0 0
raw download clone embed print report
  1. root@55cb7f729062:/workspace# ls
  2. HF_HOME  Untitled.ipynb  __pycache__  dataset-cache  test.py  train_notebook_sdxl_mapping_saving.ipynb
  3. root@55cb7f729062:/workspace# python
  4. Python 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] on linux
  5. Type "help", "copyright", "credits" or "license" for more information.
  6. >>> from multiprocess import set_start_method
  7. >>> set_start_method("spawn")
  8. >>> from test import map_train
  9. <class 'datasets.arrow_dataset.Dataset'>
  10. You are using a model of type clip_text_model to instantiate a model of type . This is not supported for all configurations of models and can yield errors.
  11. You are using a model of type clip_text_model to instantiate a model of type . This is not supported for all configurations of models and can yield errors.
  12. >>> map_train()
  13. Map (num_proc=2):   0%|                                                              | 0/833 [00:00<?, ? examples/s]<class 'datasets.arrow_dataset.Dataset'>
  14. You are using a model of type clip_text_model to instantiate a model of type . This is not supported for all configurations of models and can yield errors.
  15. You are using a model of type clip_text_model to instantiate a model of type . This is not supported for all configurations of models and can yield errors.
  16. 0
  17. Map (num_proc=2):   0%|                                                    | 2/833 [00:21<2:29:09, 10.77s/ examples]0
  18. 0
  19. Map (num_proc=2):   1%|▍                                                     | 6/833 [00:21<38:46,  2.81s/ examples]0
  20. 0
  21. 0
  22. Map (num_proc=2):   1%|▊                                                    | 12/833 [00:21<15:05,  1.10s/ examples]0
  23. 0
  24. Map (num_proc=2):   2%|█                                                    | 16/833 [00:21<09:37,  1.41 examples/s]0
  25. 0
  26. Map (num_proc=2):   2%|█▎                                                   | 20/833 [00:22<06:26,  2.10 examples/s]0
  27. 0
  28. Map (num_proc=2):   3%|█▌                                                   | 24/833 [00:22<04:27,  3.03 examples/s]0
  29. 0
  30. <class 'datasets.arrow_dataset.Dataset'>
  31. Map (num_proc=2):   3%|█▊                                                   | 28/833 [00:22<03:09,  4.26 examples/s]0
  32. 0
  33. You are using a model of type clip_text_model to instantiate a model of type . This is not supported for all configurations of models and can yield errors.
  34. Map (num_proc=2):   4%|██                                                   | 32/833 [00:22<02:17,  5.81 examples/s]0
  35. You are using a model of type clip_text_model to instantiate a model of type . This is not supported for all configurations of models and can yield errors.
  36. 0
  37. Map (num_proc=2):   4%|██▎                                                  | 36/833 [00:22<01:43,  7.68 examples/s]0
  38. 0
  39. Map (num_proc=2):   5%|██▌                                                  | 40/833 [00:22<01:20,  9.81 examples/s]0
  40. 0
  41. Map (num_proc=2):   5%|██▊                                                  | 44/833 [00:22<01:04, 12.21 examples/s]0
  42. 0
  43. Map (num_proc=2):   6%|███                                                  | 48/833 [00:23<00:56, 13.88 examples/s]0
  44. 0
  45. Map (num_proc=2):   6%|███▎                                                 | 52/833 [00:23<00:46, 16.93 examples/s]0
  46. 0
  47. Map (num_proc=2):   7%|███▌                                                 | 56/833 [00:23<00:39, 19.65 examples/s]0
  48. 0
  49. Map (num_proc=2):   7%|███▊                                                 | 60/833 [00:23<00:34, 22.67 examples/s]0
  50. 0
  51. Map (num_proc=2):   8%|████                                                 | 64/833 [00:23<00:32, 23.69 examples/s]0
  52. 0
  53. Map (num_proc=2):   8%|████▎                                                | 68/833 [00:23<00:31, 24.54 examples/s]0
  54. 0
  55. Map (num_proc=2):   9%|████▌                                                | 72/833 [00:24<00:52, 14.40 examples/s]0
  56. 0
  57. Map (num_proc=2):   9%|████▊                                                | 76/833 [00:24<01:09, 10.88 examples/s]0
  58. 0
  59. Map (num_proc=2):  10%|█████                                                | 80/833 [00:25<00:56, 13.38 examples/s]0
  60. 0
  61. Map (num_proc=2):  10%|█████▎                                               | 84/833 [00:25<00:46, 16.02 examples/s]0
  62. 0
  63. Map (num_proc=2):  11%|█████▌                                               | 88/833 [00:25<00:40, 18.25 examples/s]0
  64. 0
  65. Map (num_proc=2):  11%|█████▊                                               | 92/833 [00:25<00:34, 21.25 examples/s]0
  66. 0
  67. Map (num_proc=2):  12%|██████                                               | 96/833 [00:25<00:32, 22.81 examples/s]0
  68. 0
  69. Map (num_proc=2):  12%|██████▏                                             | 100/833 [00:25<00:30, 24.12 examples/s]0
  70. 0
  71. Map (num_proc=2):  12%|██████▍                                             | 104/833 [00:25<00:28, 25.68 examples/s]0
  72. 0
  73. Map (num_proc=2):  13%|██████▋                                             | 108/833 [00:26<00:28, 25.85 examples/s]0
  74. 0
  75. Map (num_proc=2):  13%|██████▉                                             | 112/833 [00:26<00:27, 26.50 examples/s]0
  76. 0
  77. Map (num_proc=2):  14%|███████▏                                            | 116/833 [00:26<00:27, 26.38 examples/s]0
  78. 0
  79. Map (num_proc=2):  14%|███████▍                                            | 120/833 [00:26<00:25, 27.80 examples/s]0
  80. 0
  81. Map (num_proc=2):  15%|███████▋                                            | 124/833 [00:26<00:23, 30.05 examples/s]0
  82. 0
  83. Map (num_proc=2):  15%|███████▉                                            | 128/833 [00:26<00:23, 29.90 examples/s]0
  84. 0
  85. Map (num_proc=2):  16%|████████▏                                           | 132/833 [00:26<00:23, 29.77 examples/s]0
  86. 0
  87. Map (num_proc=2):  16%|████████▍                                           | 136/833 [00:26<00:23, 30.13 examples/s]0
  88. 0
  89. Map (num_proc=2):  17%|████████▋                                           | 140/833 [00:27<00:23, 29.88 examples/s]0
  90. 0
  91. Map (num_proc=2):  17%|████████▉                                           | 144/833 [00:27<00:23, 29.59 examples/s]0
  92. 0
  93. Map (num_proc=2):  18%|█████████▏                                          | 148/833 [00:27<00:22, 30.01 examples/s]0
  94. 0
  95. Map (num_proc=2):  18%|█████████▍                                          | 152/833 [00:27<00:22, 30.00 examples/s]0
  96. 0
  97. Map (num_proc=2):  19%|█████████▋                                          | 156/833 [00:27<00:22, 30.37 examples/s]0
  98. 0
  99. Map (num_proc=2):  19%|█████████▉                                          | 160/833 [00:27<00:21, 31.32 examples/s]0
  100. 0
  101. Map (num_proc=2):  20%|██████████▏                                         | 164/833 [00:27<00:21, 31.32 examples/s]0
  102. 0
  103. Map (num_proc=2):  20%|██████████▍                                         | 168/833 [00:27<00:21, 31.43 examples/s]0
  104. Process SpawnPoolWorker-1:
  105. Traceback (most recent call last):
  106.   File "/usr/local/lib/python3.10/dist-packages/multiprocess/process.py", line 314, in _bootstrap
  107.     self.run()
  108.   File "/usr/local/lib/python3.10/dist-packages/multiprocess/process.py", line 108, in run
  109.     self._target(*self._args, **self._kwargs)
  110.   File "/usr/local/lib/python3.10/dist-packages/multiprocess/pool.py", line 114, in worker
  111.     task = get()
  112.   File "/usr/local/lib/python3.10/dist-packages/multiprocess/queues.py", line 370, in get
  113.     return _ForkingPickler.loads(res)
  114.   File "/usr/local/lib/python3.10/dist-packages/dill/_dill.py", line 301, in loads
  115.     return load(file, ignore, **kwds)
  116.   File "/usr/local/lib/python3.10/dist-packages/dill/_dill.py", line 287, in load
  117.     return Unpickler(file, ignore=ignore, **kwds).load()
  118.   File "/usr/local/lib/python3.10/dist-packages/dill/_dill.py", line 442, in load
  119.     obj = StockUnpickler.load(self)
  120.   File "/usr/local/lib/python3.10/dist-packages/torch/storage.py", line 337, in _load_from_bytes
  121.     return torch.load(io.BytesIO(b))
  122.   File "/usr/local/lib/python3.10/dist-packages/torch/serialization.py", line 1028, in load
  123.     return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  124.   File "/usr/local/lib/python3.10/dist-packages/torch/serialization.py", line 1256, in _legacy_load
  125.     result = unpickler.load()
  126.   File "/usr/local/lib/python3.10/dist-packages/torch/serialization.py", line 1193, in persistent_load
  127.     wrap_storage=restore_location(obj, location),
  128.   File "/usr/local/lib/python3.10/dist-packages/torch/serialization.py", line 381, in default_restore_location
  129.     result = fn(storage, location)
  130.   File "/usr/local/lib/python3.10/dist-packages/torch/serialization.py", line 277, in _cuda_deserialize
  131.     return torch.UntypedStorage(obj.nbytes(), device=torch.device(location))
  132. torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB. GPU 0 has a total capacty of 15.73 GiB of which 14.25 MiB is free. Process 195904 has 3.73 GiB memory in use. Process 196147 has 6.99 GiB memory in use. Process 196146 has 4.98 GiB memory in use. Of the allocated memory 4.66 GiB is allocated by PyTorch, and 172.25 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
  133. 0
  134. Map (num_proc=2):  21%|██████████▋                                         | 172/833 [00:28<00:20, 32.04 examples/s]0
  135. 0
  136. Map (num_proc=2):  21%|██████████▉                                         | 176/833 [00:28<00:21, 30.91 examples/s]0
  137. 0
  138. Map (num_proc=2):  22%|███████████▏                                        | 180/833 [00:28<00:21, 30.50 examples/s]0
  139. 0
  140. Map (num_proc=2):  22%|███████████▍                                        | 184/833 [00:28<00:21, 29.98 examples/s]0
  141. 0
  142. Map (num_proc=2):  23%|███████████▋                                        | 188/833 [00:28<00:21, 29.40 examples/s]0
  143. 0
  144. Map (num_proc=2):  23%|███████████▉                                        | 192/833 [00:28<00:21, 29.24 examples/s]0
  145. 0
  146. Map (num_proc=2):  24%|████████████▏                                       | 196/833 [00:28<00:21, 29.00 examples/s]0
  147. 0
  148. Map (num_proc=2):  24%|████████████▍                                       | 200/833 [00:29<00:21, 29.30 examples/s]0
  149. 0
  150. Map (num_proc=2):  24%|████████████▋                                       | 204/833 [00:29<00:21, 29.88 examples/s]0
  151. 0
  152. Map (num_proc=2):  25%|████████████▉                                       | 208/833 [00:29<00:20, 30.95 examples/s]0
  153. 0
  154. Map (num_proc=2):  25%|█████████████                                       | 210/833 [00:29<01:27,  7.13 examples/s]
  155. Traceback (most recent call last):
  156.   File "<stdin>", line 1, in <module>
  157.   File "/workspace/test.py", line 222, in map_train
  158.     return train_dataset.map(compute_embeddings_fn, batched=True, batch_size=2, with_rank=True, num_proc=2)
  159.   File "/usr/local/lib/python3.10/dist-packages/datasets/arrow_dataset.py", line 592, in wrapper
  160.     out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  161.   File "/usr/local/lib/python3.10/dist-packages/datasets/arrow_dataset.py", line 557, in wrapper
  162.     out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  163.   File "/usr/local/lib/python3.10/dist-packages/datasets/arrow_dataset.py", line 3185, in map
  164.     for rank, done, content in iflatmap_unordered(
  165.   File "/usr/local/lib/python3.10/dist-packages/datasets/utils/py_utils.py", line 647, in iflatmap_unordered
  166.     raise RuntimeError(
  167. RuntimeError: One of the subprocesses has abruptly died during map operation.To debug the error, disable multiprocessing.
Advertisement
Add Comment
Please, Sign In to add comment
Advertisement
Public Pastes
Advertisement
We use cookies for various purposes including analytics. By continuing to use Pastebin, you agree to our use of cookies as described in the Cookies Policy.  OK, I Understand
Not a member of Pastebin yet?
Sign Up, it unlocks many cool features!