How do you detect and mitigate Overfitting vs Underfitting in deep neural networks?

Overfitting occurs when a model lea s the training data and noise too closely, leading to poor generalization on unseen validation/test data (high variance). Underfitting occurs when a model is too simplistic to capture underlying patte s (high bias).

Diagnostic Signals from Loss Curves

• Overfitting: Training loss decreases steadily towards zero, while validation loss plateaus and begins diverging upwards. High accuracy on train, significantly lower accuracy on validation.
• Underfitting: Both training loss and validation loss remain unacceptably high and plateau early. The model fails to fit even the training distribution.

Mitigation Strategies for Overfitting

1. L1 / L2 Regularization (Weight Decay): Penalizes large weights in the loss function ($Loss_{total} = Loss_{task} + \lambda \sum w^2$), encouraging smoother decision boundaries.
2. Dropout: Randomly zeros out neuron activations during training with probability $p$ (typically $0.2 - 0.5$), preventing co-adaptation of features.
3. Early Stopping: Monitor validation loss and halt training when validation metric stops improving for $n$ consecutive epochs (patience).
4. Data Augmentation: Artificially increase training set diversity through rotations, crops, mixup, or synonym replacement.
5. Batch Normalization: Normalizes layer inputs across mini-batches, regularizing inte al covariate shifts.