01/09/2024 17:46:36 - INFO - __main__ -   Total optimization steps = 10000000
Steps:   0%|                                       | 0/10000000 [00:00<?, ?it/s]Traceback (most recent call last):
  File "/workspace/diffusers/examples/text_to_image/train_text_to_image_lora.py", line 950, in <module>
    main()
  File "/workspace/diffusers/examples/text_to_image/train_text_to_image_lora.py", line 777, in main
    accelerator.clip_grad_norm_(params_to_clip, args.max_grad_norm)
  File "/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py", line 2040, in clip_grad_norm_
    self.unscale_gradients()
  File "/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py", line 2003, in unscale_gradients
    self.scaler.unscale_(opt)
  File "/usr/local/lib/python3.10/dist-packages/torch/cuda/amp/grad_scaler.py", line 307, in unscale_
    optimizer_state["found_inf_per_device"] = self._unscale_grads_(
  File "/usr/local/lib/python3.10/dist-packages/torch/cuda/amp/grad_scaler.py", line 229, in _unscale_grads_
    raise ValueError("Attempting to unscale FP16 gradients.")
ValueError: Attempting to unscale FP16 gradients.
wandb: 🚀 View run fiery-bee-255 at: https://wandb.ai/spammmmm1997/text2image-fine-tune/runs/ohv7n9or
wandb: ️⚡ View job at https://wandb.ai/spammmmm1997/text2image-fine-tune/jobs/QXJ0aWZhY3RDb2xsZWN0aW9uOjEyODY2NDE5NA==/version_details/v0
wandb: Synced 5 W&B file(s), 0 media file(s), 2 artifact file(s) and 0 other file(s)
wandb: Find logs at: ./wandb/run-20240109_174636-ohv7n9or/logs
Traceback (most recent call last):
  File "/usr/local/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/dist-packages/accelerate/commands/accelerate_cli.py", line 47, in main
    args.func(args)
  File "/usr/local/lib/python3.10/dist-packages/accelerate/commands/launch.py", line 1017, in launch_command
    simple_launcher(args)
  File "/usr/local/lib/python3.10/dist-packages/accelerate/commands/launch.py", line 637, in simple_launcher
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/usr/bin/python', 'train_text_to_image_lora.py', '--pretrained_model_name_or_path=lambdalabs/miniSD-diffusers', '--dataset_name=kopyl/3M_icons_monochrome_only_no_captioning', '--resolution=256', '--train_batch_size=2', '--max_train_steps=10000000', '--checkpointing_steps=100', '--learning_rate=1e-4', '--lr_scheduler=constant', '--lr_warmup_steps=0', '--output_dir=/workspace/train-checkpoints/lora', '--noise_offset=0.05', '--cache_dir=/workspace/dataset-cache', '--report_to=wandb', '--validation_prompt', 'cat icon', '--seed=1']' returned non-zero exit status 1.