GitHub · Where software is built

Labels Milestones New issue

NCCL timeout while doing multi gpu training

#20832

· aditya-sanas opened

on May 17, 2025

<code>mark_forward_method</code> does not work with <code>ModelParallelStrategy</code>

waiting on author

#20710

· tonyf opened

on Apr 12, 2025

multi-node training runs crash because <code>ddp_weakref</code> is <code>None</code> during backward

#20706

· mishooax opened

on Apr 10, 2025

self.manual_backward() makes all gradients gone

#20685

· samsara-ku opened

on Mar 31, 2025

self.all_gather does not work on on_train_epoch_end

#20683

· yoniaflalo opened

on Mar 28, 2025

<code>ModelCheckpoint</code> not saving best model

callback: model checkpoint

#20657

· sravan953 opened

on Mar 19, 2025

ModelCheckpoint broadcast fails on multiple GPUs

callback: model checkpoint

#20597

· navotoz opened

on Feb 20, 2025

Slurm multi-node work fine but multi-gpu doesn't

environment: slurm

#20438

· atifkhanncl opened

on Nov 22, 2024

Multi-gpu training with slurm times out

environment: slurm

#20434

· nightingal3 opened

on Nov 19, 2024

NCCL backend fails during multi-node, multi-GPU training

environment: slurm

#20306

· raketenolli opened

on Sep 26, 2024

environment variable WORLD_SIZE is incorrectly set to 1 after trainer.fit is done

#20232

· simon-ging opened

on Aug 28, 2024

Sometimes I get Dataset Errors when using the lightning module in a distributed manor

#20088

· asusdisciple opened

on Jul 15, 2024