Engee documentation

EngeeDSP.HampelFilter

Filtering outliers using the Hampel identifier.

Library

EngeeDSP

Description

System object EngeeDSP.HampelFilter detects and removes outliers in the input signal using the Hampel identifier. The Hampel identifier is an outlier-resistant variation of a statistical rule. «Three sigma». For each sample of the input signal, the object calculates the median in a window consisting of the current sample and adjacent counts on each side of it, where — the window length specified in the WindowLength. The object also estimates the standard deviation of the reference relative to the median of the window using the median absolute deviation. If the difference between the sample and the median exceeds the threshold value multiplied by the standard deviation, the filter replaces this sample with the median. For more information, see Algorithms.

To filter the input signal using the Hampel identifier, follow these steps:

  1. Create an object EngeeDSP.HampelFilter and set its properties.

  2. Call the object with arguments as if it were a function.

To learn more about how to work with system objects, see Engee System Objects.

Syntax

Creation

  • hampFilt = EngeeDSP.HampelFilter() — returns the Hampel filter system object hampFilt using default properties.

  • hampFilt = EngeeDSP.HampelFilter(Len) — returns the Hampel filter system object hampFilt, which has for the property WindowLength is set to Len.

  • hampFilt = EngeeDSP.HampelFilter(Len,Lim) — returns the Hampel filter system object hampFilt, which has for the property WindowLength is set to Len, and for the property Threshold — value Lim.

  • hampFilt = EngeeDSP.HampelFilter(Name=Value) — returns the Hampel filter system object hampFilt with the specified properties as a pair Name=Value, where Name — the name of the property, and Value — the appropriate value. You can specify multiple pairs «name-value» the order of the pairs does not matter. Unspecified properties retain their default values.

Using

  • y,isOutlier = hampFilt(x) — Detects and removes outliers in each input channel x by processing them independently in time using a Hampel filter, the result returns in y. It also returns a logical array. isOutlier, in which the value is true indicates that the corresponding element of the input signal is an outlier. Array isOutlier It has the same size as the input and output signals.

Arguments

Input arguments

# x — input signal

+ vector | the matrix

Details

An input signal specified as a vector or matrix. If x — the matrix, each of its columns is considered as an independent channel. The object supports multi-channel input data in size on , where — the number of samples in each frame (or channel), and — the number of channels.

Типы данных

Float32, Float64

Output arguments

# y — filtered output signal

+ vector | the matrix

Details

The filtered output signal returned as a vector or matrix.

Типы данных

Float32, Float64

# isOutlier — definition of emissions

+ vector | the matrix

Details

A logical array whose elements indicate whether the corresponding element of the input array is an outlier. If an array element isOutlier it matters true, then the corresponding element of the input array is an outlier.

Типы данных

Bool

Features

# Threshold — threshold value of emission detection
Real number

Details

The threshold value for detecting outliers, set as a positive real scalar. For more information, see Algorithms.

By default — 3.

# WindowLength — sliding window length
Real number

Details

The length of the sliding window, specified as a positive odd scalar integer. A finite-length window moves through the data, and the object calculates the median and median absolute deviation of the data inside this window.

By default — 11.

Methods

Common to all system objects

step!

Run the system object operation algorithm

release!

Allow changing the value of a system object property

reset!

Resetting the internal states of a system object

Additional Info

The Hampel ID

Details

The Hampel identifier is a kind of statistical rule. «Three sigma», resistant to emissions.

Given the sequence and a sliding length window , we will determine the point estimates of the median and standard deviation using:

  • The local median — ;

  • Standard deviation — , where .

Value It is known as the Median Absolute Deviation (MAD).

If the countdown such that

for a given threshold value , the Hampel ID counts an outlier and replaces it with . If equally 0, then the Hampel filter behaves like a regular median filter.

Algorithms

so hampel filtr 1 en

For a given data count algorithm:

  • Centers an odd-length window relative to the current frame;

  • Calculates the local median and the standard deviation for the current data window;

  • Compares the current countdown with , where — threshold value. If , the filter identifies the current count as an outlier and replaces it with the median value .

Consider the data frame that is passed to the Hampel filter.

so hampel filtr 2

In this example, the Hampel filter slides over the data by a window of length . The threshold value of the filter is . To ensure the completeness of the window at the beginning of the frame, the filtering algorithm adds in front of it zero counts. When calculating the first output sample, the window is centered on the sample with the number in the extended frame — in this case, at the third zero. The filter calculates the median, median absolute deviation, and standard deviation for the data inside the local window.

so hampel filtr 3 en

  • The current countdown: .

  • The Data window: .

  • The local median: .

  • Median absolute deviation: . For this data window .

  • Standard deviation: = , where .

  • The current countdown does not follow the ratio for detecting outliers:

    Thus, the current input sample is formed at the output of the Hampel filter. .

This procedure is repeated for each subsequent count until the algorithm centers the window on the count. , denoted as End. Since it is centered on the latter The window cannot be full of samples, these samples are processed with the next frame of input data.

Here is the first output frame generated by the Hampel filter:

so hampel filtr 4 en

The seventh countdown 23 the added input frame is an outlier. The Hampel filter replaces this count with the median in the local window. [4 9 23 8 12].

using EngeeDSP

hampFilt = EngeeDSP.HampelFilter(WindowLength=5,Threshold=2)

x=[1, 2, 4, 9, 23, 8, 12, 4, 2]
println("x: ",x)

y,isOutlier=hampFilt(x)

println("y: ",y)
x: [1, 2, 4, 9, 23, 8, 12, 4, 2]
y: [0.0, 0.0, 1.0, 2.0, 4.0, 9.0, 9.0, 8.0, 12.0]

Literature

  1. Bodenham, Dean. Adaptive Filtering and Change Detection for Streaming Data. Ph.D. Thesis. Imperial College, London, 2012.

  2. Liu, Hancong, Sirish Shah, and Wei Jiang. On-line outlier detection and data cleaning. Computers and Chemical Engineering. Vol. 28, March 2004, pp. 1635–1647.

  3. Suomela, Jukka. Median Filtering Is Equivalent to Sorting, 2014.