What are the training best practices of Auto Encoders

 

 Training Best Practices

Achieving stable convergence requires specific strategies for initialization and monitoring.

• Initialization Strategies:

  • Xavier/Glorot: Used to ensure balanced gradient flow during the start of training.

  • He Initialization: Specifically optimized for networks using ReLU activation functions.

  • Symmetry Breaking: Avoiding perfectly symmetric weights is essential to allow the network to learn diverse features.

• Training Monitoring:

  • Loss Tracking: It is vital to monitor reconstruction loss on both training and validation sets to detect overfitting.

  • Gradient Norms: Tracking these helps identify vanishing or exploding signal problems.

  • Qualitative Assessment: Periodically visualizing the reconstructed outputs allows for a human-eye check on the model's progress.