Spectral Bias
Definition
The phenomenon that neural networks fit low-frequency components first and struggle to learn high-frequency oscillations. It explains why PINNs converge so slowly on multiscale and high-frequency problems, and motivates adaptive weights, Fourier features and curriculum learning.
What it is
Theory (neural tangent kernel) and experiment agree: gradient descent fits the target function from low frequency to high frequency. For smooth solutions this is fine; for PDE solutions with high-frequency oscillation, steep gradients or multiscale structure, the high-frequency part takes very long to converge and may still be unlearned when training stops.
Key points
- Why PINNs suffer most: PDE residuals contain high-order derivatives, where high frequencies are amplified (
d^n/dx^nbrings ak^nfactor) and the residual landscape becomes rugged. - Main mitigations: random Fourier features / positional encoding (map inputs onto high-frequency bases first), adaptive loss weights, residual-based adaptive refinement (RAR), domain decomposition (XPINN), curriculum learning (low frequency first).
- Connection to FNO: FNO truncates modes in frequency space, i.e. it explicitly gives up high frequencies, so it reconstructs fine detail poorly — the same coin seen from the other side.
- Diagnosis: track the error-decay curve per frequency band, or check whether residuals concentrate in high-gradient regions.
Common pitfalls / boundaries
- Spectral bias is not "insufficient capacity": scaling up the network helps little, because the issue is training dynamics, not expressivity.
- Adding Fourier features is not a cure: if the encoding frequency mismatches the target, it introduces new oscillatory artifacts.