prefilter

The name prefilter stands for pre-processing filtering; some additional filtering might be necessary!

Physiological GSR data measurements often contain quite some noise which makes analyses of those data rather difficult. This function tries to remove part of this noise by applying a derivative-based noise selection and filtering

If the data you have is available in one of the typical GSR formats, the use of the appropriate method is preferred (either ACQ::Filter or NTT::Filter).

Other than that, only the data (column vector) is required.

  prefilter  - spike/shift/resonance filter data
 
  FORMAT:       [fd, w, fcount] = prefilter(d, opts)
 
  Input fields:
 
        d           data to filter
        opts        mandatory struct but with optional fields
         .dim       filter dimension (default: 1)
         .freq      signal frequency (default: 100)
         .fwin      filtering window seconds (default: 2)
         .fwincut   filtering frequency cutoff in seconds (default: 0.5)
         .fwslide   filtering window slide in seconds (default: fwin/16)
         .kern      smoothing kernel (default: 2)
         .post      post-diff smoothing kernel (default: 0)
         .stdt      std(abs(diff)) change threshold (default: 0.02)
 
  Output fields:
 
        fd          filtered data (forced to double datatype)
        w           weights of smoothing requirement
        fcount      number of windows in which filtering occurred

Given that the data is available in variable data, the following syntax could be applied:

<code matlab prefilter_example.m>% data is a Sx1 column vector % - carrying GSR recorded data % - with a 100Hz sampling rate

The options are

• .freq - this is simply the frequency in which the data is assumed to be sampled
• .fwin - this is the size of the sliding window for which noise components are separately examined
• .fwincut' - this is the cut-off frequency for those frequencies which *must* remain in the signal and that are not touched by the noise detection/removal procedure * .fwslide' - by default this is .fwin / 16, that is 0.125s in the fully-default case; this is the amount of time by which the window is slid between each iteration of the noise detection/removal pass
• .kern - preliminary smoothing kernel size (in seconds)
• .post - post-processing smoothing kernel size; this is highly recommended to ameliorate potential small discontinuities after the noise removal, e.g. at very noise passages, the window still is biased by the mean of the noise!
• .stdt - this is the threshold that defines for how long the preliminary smoothing is repeated (until the standard deviation of abs(diff(data)) does not change by more than this factor)

The algorithm can be debunked into four major components:

• preliminary data filtering
• noise detection, based on the observation of abs(diff(data)) and its own derivative
• window-wise noise filtering (for windows with more than 25% noisy samples)
• post-processing (final filtering pass)

An initial estimate of the variability of the first derivative, std(abs(diff(data))), is computed. A gaussian kernel with the specified size is created and applied to the data. This step is repeated until the observed value of the initially estimated term does not change by more than a pre-defined factor (e.g. intially the estimated value is 14.51, after applying the filter once, this drops to 8.91, then to 6.78, then to 6.56, and then to 6.53; given the default of 0.01, the difference is less than this ((6.56 - 6.53) / 6.53)).

After the preliminary filtering, the first derivative is computed. This signal, in itself, has the same intrinsic noise problems and is thus filtered (with exactly the same procedure as the preliminary filtering of the actual signal). Eventually, the absolute value of the second derivative is computed (strong curving of signal). There is a final filtering procedure on this second order derivative, after which parts that show strong noise (e.g. resonance phenomena) are characterized by prolonged periods where this value is high.

The original (preliminary filtered!) signal is now inspected in windowed chunks. For each window where there are more than 25% noisy samples, a robust estimation of desired frequencies is performed (removing single spikes within the noise!) and only those lower frequencies of interest are kept. A sliding sum of all windows is computed and, eventually, the weighted sum of all windowed results replaces the original signal.

To account for the fact that window-wise algorithms do suffer from a window-selection bias (e.g. certain noise components are in frequencies that interact with the window size), a post-processing filtering is applied. For periods where no noise was detected this will not alter the signal significantly.

Note: it is important that this filter is selected in with a cutoff that does not remove any desired frequencies from the signal!