Loss becomes Inf or NaN - reason

• check the input and output data

• print the data value

• preprocessing: wrong normalization

• torch.autograd.tect_anomaly

• decrease the learning rate

• adjust the loss weight hyper parameters

• whether can be normally backward

• input Tensor has been type transformed to the same type

• a small constant is added to the divisor to ensure the stability of calculation

• check if Nan after batchNorm

• while training and validation set has different distribution, the mean and var learned in the training might not be able to applied to validation

• while encoder and decoder has different architecture
  • encoder: Unet + resnet34
  • decoder: Unet + resnet50

• model.eval():model set to model.train(True)

• Batchnorm: track_running_stats=False

• stride is larger than kernel size

layer {
  name: "faulty_pooling"
  type: "Pooling"
  bottom: "x"
  top: "y"
  pooling_param {
    pool: AVE
    stride: 5
    kernel: 3
  }
}

• batch_norm on data with variance distribution
  • training stage : use shuffle
  • testing stage: no shuffle
  • Distribution might be very different! - causing Nan

• model trained on GPU cannot work on CPU