2
\$\begingroup\$

I have a time series measurement of light intensity where the signal is periodically blocked by hardware to provide a moving baseline to correct for drift. This looks like a square wave, where the upper level is the signal being measured. (a small segment of the data shown here).

enter image description here

I want to continuously subtract the lower level from the upper level. I've managed this by quite a bruteforce approach (using derivative to extract areas of near-zero gradient to remove the transitions between levels, separating into upper and lower by thresholding, then subtracting the mean of each lower section from its neighbouring upper section), but I feel there must surely be a better way.

My question is less about any specific problem with the code, and more about the general approach.

Which transform would achieve the same thing?

\$\endgroup\$
1
  • 1
    \$\begingroup\$ Clock the A/D off the chopper frequency and then the zero gradient sample will always be exactly the same and you can simply load it from with no further calculations after the first time you calculate it's position. \$\endgroup\$ Commented Mar 27, 2024 at 15:46

2 Answers 2

2
\$\begingroup\$
  1. Bandpass the signal to get a sinewave out of it.
  2. Measure the amplitude of the sinewave. There are plenty of ways to do it. Choose an approach that uses all points of the sinewave and not the peaks only. That will let you exploit the resolution available from the signal.

In Mathematica, the filtering and amplitude measurement can be done as follows:

enter image description here

A computationally simpler (and also lower resolution!) approach is to take the difference of the quantiles of the signal that are close to the boundary, in this case 1/60th and 59/60th quantile:

enter image description here

Better statistical estimates of the "top" and "bottom" can be had by fitting a mixture of two skew-normal distributions. For guidance, see this answer. See also this answer where an iterative approach easy to implement from scratch in C is demonstrated. As a starting point, one can take a mixture of two normal distributions, but that's not the best model for the data as there will be some interaction between the shape of the transition areas and the mean. The skew-normal distribution will model that better.

Note that a lot of Mathematica is implemented in Wolfram Language (vs. C++ and C), and there's this helpful function called GeneralUtilities`PrintDefinitions that lets you inspect the normalized source code of a built-in function of interest.

So, if you wanted to understand how distribution fitting works, the following code would be a starting point:

Needs["GeneralUtilities`"] GeneralUtilities`PrintDefinitions[SmoothKernelDistribution] GeneralUtilities`PrintDefinitions[Statistics`SmoothKernelDistributionDump`iValidOptionsQ] 

Signal recovery takes each row of the inverted image, converts it to a series of {x,brightness} pairs, builds a polygon on these coordinates, then takes the first (x) coordinate of the polygon's centroid. This takes advantage of the antialiasing of the original plot, and recovers a higher resolution signal than would be obtained by binarizing the image first.

enter image description here

\$\endgroup\$
1
  • \$\begingroup\$ Thanks for the detailed answer. I realised I didn't make it clear in my original question, but I want to preserve the time resolution of signal, as I need to compute the Allan Variance of the signal (light intensity), corrected for measurement drift. The measurement frequency is in the kHz, while the chopper is at 500 Hz. I'm coding this all in python. \$\endgroup\$ Commented Mar 28, 2024 at 10:21
1
\$\begingroup\$

You won't find any simple linear transforms (and no linear transforms) that are simpler than this operation. I've done a lot of data massaging and the method you describe is probably the best and the one I have done in the past with finding minimums.

The other approach to chopping is if you have a known frequency you can fit a square wave or use a PLL to do better detection of the origonal chopping signal. (A square wave would be an on\off, your optical signal might not be on\off but may be trapezoidal or exponential in nature). Once you have a replica of the chopping signal, you can subtract it out and at least get a better idea of when the sensor is on\off. You can also do this with thesholding to some extent but thresholding is not great when you have a noisy signal.

If you can fit a signal with a known model then that would be the other possibility. The approach I've used in the past is non-linear fitting models (if you can find a continuous function for the data). Non-linear models can be hard to fit as they could have many minimums and may be hard to fit to your data. They are also very processor intensive and so you'd need something on the order of a PC to do the post-processing. The problems that arise with this is if you have drift, the non-linear model will try and fit all the drift, so it may not subtract and give you the residual you want.

I'd stick with your first approach, find something that works and use it.

\$\endgroup\$

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.