Comparison of LMS and NLMS algorithms for adaptive signal filtering

Adaptive filtering is a key technology in digital signal processing, with applications in noise reduction, echo cancellation, system identification, and many other fields. In this article, we will look at two classic adaptive filtering algorithms - LMS (Least Mean Squares) and its modification NLMS (Normalized LMS), implemented in the Engee language using the EngeeDSP package.
LMS (Least Mean Squares) is a stochastic gradient algorithm that minimizes the RMS error between the desired signal and the output of the adaptive filter. The basic equation of updating weights:

w(n+1) = w(n) + μ·e(n)·x(n)

where:

w(n) is the vector of filter weights at step n
μ is the adaptation step
e(n) is the error (the difference between the desired and actual output)
x(n) is the input vector

NLMS Algorithm

NLMS (Normalized LMS) is a modification of the LMS, where the adaptation step is normalized by the input signal power:

w(n+1) = w(n) + (μ / (xᵀ(n)x(n) + ε))·e(n)·x(n)

where ε is a small constant to prevent division by zero.

Analysis of the first example

The first code compares the LMS and NLMS on a signal with varying amplitude:

x_window = 0.05 * randn(1024*10) .* repeat(0.5 .+ 0.5sin.(2π*(0:1024*10-1)/10240), inner=1)
d_window = filt(FIRFilter([0.5, -0.3, 0.2, 0.1, -0.05]), x_window)

It is being created here:

A noise signal with an amplitude modulated by a sine wave
The "desired" signal obtained by filtering through a FIR filter

The analysis is performed by calculating the frequency response of the adaptive filter on each frame and visualizing using a surface-plot.

using EngeeDSP, DSP, FFTW, Plots

frameSize = 500
Fs = 8192
x_window = 0.05 * randn(1024*10) .* repeat(0.5 .+ 0.5sin.(2π*(0:1024*10-1)/10240), inner=1)
d_window = filt(FIRFilter([0.5, -0.3, 0.2, 0.1, -0.05]), x_window)

HA = EngeeDSP.LMSFilter(
    Algorithm = "LMS",
    FilterLength = 32,
    StepSize = 1.0,
    LeakageFactor = 1.0,
    InitialValueOfFilterWeights = 0,
    AdaptPort = false, 
    ResetPort = "None",
    OutputFilterWeights = true
)

numFrames = length(d_window) ÷ frameSize
response = zeros(512, numFrames)
t_frame = (0:frameSize:length(d_window)-1)/Fs

i = 1
for start_idx in 1:frameSize:length(x_window)-frameSize+1
    end_idx = start_idx + frameSize - 1
    x_frame = x_window[start_idx:end_idx]
    d_frame = d_window[start_idx:end_idx]
    
    if i == 1
        setup!(HA, x_frame, d_frame)
    end
    
    y, e, w = step!(HA, x_frame, d_frame)
    response[:,i] = 20*log10.(abs.(fft([w; zeros(512-length(w))])))
    i += 1
end

p1 = surface(t_frame, Fs*(0:511)/512, response,
    title="LMS Algorithm Response",
    xlabel="Time (s)", ylabel="Frequency (Hz)", zlabel="Power (dB)",
    camera=(-70, 18), zlims=(-80, 2))

HA = EngeeDSP.LMSFilter(
    Algorithm = "Normalized LMS",
    FilterLength = 32,
    StepSize = 0.3,
    LeakageFactor = 1.0,
    InitialValueOfFilterWeights = 0,
    AdaptPort = false,  
    ResetPort = "None",
    OutputFilterWeights = true
)

response = zeros(512, numFrames)

i = 1
for start_idx in 1:frameSize:length(x_window)-frameSize+1
    end_idx = start_idx + frameSize - 1
    x_frame = x_window[start_idx:end_idx]
    d_frame = d_window[start_idx:end_idx]
    
    if i == 1
        setup!(HA, x_frame, d_frame)
    end
    
    y, e, w = step!(HA, x_frame, d_frame)
    response[:,i] = 20*log10.(abs.(fft([w; zeros(512-length(w))])))
    i += 1
end

p2 = surface(t_frame, Fs*(0:511)/512, response,
    title="NLMS Algorithm Response",
    xlabel="Time (s)", ylabel="Frequency (Hz)", zlabel="Power (dB)",
    camera=(-70, 18), zlims=(-80, 2))

plot(p1, p2, layout=(1,2), size=(1000,400))

LMS algorithm:
- Uses a fixed adaptation step (µ = 1.0)

Can be sensitive to changes in the amplitude of the input signal
- Adapts faster, but may be less stable

NLMS algorithm:
- Uses a normalized adaptation step (µ = 0.3)

Automatically adjusts to the input signal level
- More stable operation when the amplitude changes

Analysis of the second example

The second code examines the behavior of the LMS filter on stationary and non-stationary signals:

x = 0.05 * randn(1024*10)
d_stat = filt(FIRFilter([0.5, -0.3, 0.2, 0.1, -0.05]), x)
d_nonstat = filt(FIRFilter([0.8, -0.2, 0.0, 0.1, -0.1]), x)

Here:

Stationary case: immutable system (one FIR filter)
Non-stationary case: the system is changing (another FIR filter)

using EngeeDSP, DSP, FFTW

frameSize = 500
Fs = 44100
x = 0.05 * randn(1024*10)
d_stat = filt(FIRFilter([0.5, -0.3, 0.2, 0.1, -0.05]), x)
d_nonstat = filt(FIRFilter([0.8, -0.2, 0.0, 0.1, -0.1]), x)

HA = EngeeDSP.LMSFilter(
    Algorithm = "LMS",
    FilterLength = 32,
    StepSize = 1.0,
    LeakageFactor = 1.0,
    InitialValueOfFilterWeights = 0,
    AdaptPort = false,
    ResetPort = "None",
    OutputFilterWeights = true
)

t_frame = (0:frameSize:length(x)-1)/Fs
response = zeros(512, length(t_frame))
i = 1
for start_idx in 1:frameSize:length(x)-frameSize+1
    end_idx = start_idx + frameSize - 1
    x_frame = x[start_idx:end_idx]
    d_frame = d_stat[start_idx:end_idx]
    
    if i == 1
        setup!(HA, x_frame, d_frame)
    end
    
    y, e, w = step!(HA, x_frame, d_frame)
    response[:,i] = 20*log10.(abs.(fft([w; zeros(512-length(w))])))
    i += 1
end

p1 = surface(t_frame, Fs*(0:511)/512, response,
    title="Stationary Signal Response",
    xlabel="Time (s)", ylabel="Frequency (Hz)", zlabel="Power (dB)",
    camera=(-70, 18), zlims=(-80, 2))

    response = zeros(512, length(t_frame))
i = 1
for start_idx in 1:frameSize:length(x)-frameSize+1
    end_idx = start_idx + frameSize - 1
    x_frame = x[start_idx:end_idx]
    d_frame = d_nonstat[start_idx:end_idx]
    
    y, e, w = step!(HA, x_frame, d_frame)
    response[:,i] = 20*log10.(abs.(fft([w; zeros(512-length(w))])))
    i += 1
end

p2 = surface(t_frame, Fs*(0:511)/512, response,
    title="Non-Stationary Signal Response",
    xlabel="Time (s)", ylabel="Frequency (Hz)", zlabel="Power (dB)",
    camera=(-70, 18), zlims=(-80, 2))

plot(p1, p2, layout=(1,2), size=(1000,400))

Stationary signal:
- The filter successfully converges to the optimal solution
- The frequency response stabilizes over time
Non-stationary signal:
- The filter is trying to adapt to a changing system

The characteristic continues to change, reflecting non-stationarity

Conclusion

The presented examples demonstrate the key features of adaptive filters. The choice between LMS and NLMS depends on the signal characteristics and system requirements. NLMS provides more stable operation when the signal strength changes, while classic LMS may be preferable in conditions of limited computing resources. The analysis of nonstationary systems requires a special approach and can serve as a topic for further research.
Also, based on their experiences, the following conclusions can be drawn:

NLMS is preferable for signals with varying amplitude.
When working with non-stationary systems, it may be necessary:
- Reduction of the adaptation step
- Using algorithms with variable pitch
- Regular filter reset
Visualization of the frequency response in dynamics is a powerful analysis tool