Traceback (most recent call last):
  File "/app/cases/mnist/mnist_classification.py", line 400, in <module>
    build_mnist_cls(path, dataset_cls=dataset_cls, linear_is_kan=False, conv_is_kan=False)
  File "/app/cases/mnist/mnist_classification.py", line 329, in build_mnist_cls
    _optimizer_result = composer.compose_pipeline(dataset_train)
  File "/app/nas/composer/nn_composer.py", line 71, in compose_pipeline
    optimization_result = self.optimizer.optimise(objective_eval.evaluate)
  File "/opt/conda/lib/python3.8/site-packages/golem/core/optimisers/populational_optimizer.py", line 88, in optimise
    self._initial_population(evaluator)
  File "/opt/conda/lib/python3.8/site-packages/golem/core/optimisers/genetic/gp_optimizer.py", line 68, in _initial_population
    self._update_population(evaluator(self.initial_individuals), 'initial_assumptions')
  File "/opt/conda/lib/python3.8/site-packages/golem/core/optimisers/genetic/evaluation.py", line 281, in evaluate_population
    evaluation_results = [self.evaluate_single(ind.graph, ind.uid) for ind in individuals_to_evaluate]
  File "/opt/conda/lib/python3.8/site-packages/golem/core/optimisers/genetic/evaluation.py", line 281, in <listcomp>
    evaluation_results = [self.evaluate_single(ind.graph, ind.uid) for ind in individuals_to_evaluate]
  File "/opt/conda/lib/python3.8/site-packages/golem/core/optimisers/genetic/evaluation.py", line 168, in evaluate_single
    fitness, graph = adapted_evaluate(graph)
  File "/opt/conda/lib/python3.8/site-packages/golem/core/adapter/adapter.py", line 173, in adapted_fun
    result = fun(*adapted_args, **adapted_kwargs)
  File "/opt/conda/lib/python3.8/site-packages/golem/core/optimisers/genetic/evaluation.py", line 181, in _evaluate_graph
    fitness = self._objective_eval(domain_graph)
  File "/app/nas/optimizer/objective/future/nas_objective_evaluate.py", line 52, in evaluate
    fitted_model = self._graph_fit(graph, train_data, log=self._log, debug_test_data=test_data)
  File "/app/nas/optimizer/objective/future/nas_objective_evaluate.py", line 80, in _graph_fit
    trainer.fit_model(train_data=opt_dataset,
  File "/app/nas/model/model_interface.py", line 123, in fit_model
    self.model.fit(train_data,
  File "/app/nas/model/pytorch/base_model.py", line 350, in fit
    train_loss = self._one_epoch_train(train_data, optim, loss, device)
  File "/app/nas/model/pytorch/base_model.py", line 297, in _one_epoch_train
    running_loss += loss.detach().cpu().item()
RuntimeError: CUDA error: uncorrectable ECC error encountered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.