In a way similar to this implementation.
@mycarta have you tried it? As far as I can tell the code is n-dimensional and would work out of the box:
you are right, it does work out of the box, but it does not do what I was hoping it would to do with the extra-spiky data I sm dealing with (I cranked up both weight and iterations and alpha down).
I may have to find a method where I can explicitly impose blockiness; however that is a separate issue.
I think the Discrete Pulse Transform may do what you want. There are some related morphological filters such as level sets which may yield similar results. I should have some code lying around, if you want me to try it on an example vector.
If you want to give it a try, I saved a copy of one of these arrays here (scaled to 0-100 range floats):
This is a tricky problem because I have several dozens such 1d time series and the proportion of spikes in the +v and -v direction changes wildly, so finding a robust, generic solution that can be applied in a semi-automatic fashion is not trivial (to me at least).
I tried de-trending schemes (with wotan), z-score based de-spiking, spiking with running median schemes with MAD as statistic, quantile regression (statsmodels), LOESS smoother with 20% to 10% of the data, restricted cubic splines. The last two gave the best result on many such time series simultaneously, but they were still quite far from ideal.
The best result I got with a “dumb” Gradient Boosting Regressor (where by dumb I mean with max_depth = 1) to derive a trend and then using a rolling window and MAD to de-spike, but it is of course a veeery slow method.
So I am open to methods I have not tried.
Is this the one you are thinking of, by the way?
That’s the one, yes. There is a simpler implementation here written by a colleague (deceased, sadly), but I’m afraid I’ll have to read their paper again to understand how to use it.
In my initial runs, I wasn’t able to extract the trend effectively (or, should I say, the trend extracted is not the one your eye would follow).