issue_comments

Ah that is a much better compromise - it's still slower for my own much larger dataset but is definitely manageable now. I think that this is what I was trying to find originally when I ended up using |S1.

As the problem was my usage of encoding / netCDF4's slow variable strings and you've given me a good workaround, I'll close this. Thanks for your help!

This would still be very useful to me in future - for the piece of work I was referring to here I came up with a workaround. I filled in the gaps roughly with NaNs, so that I could identify and remove outliers and other bad data. Only then could I use the resample functionality without smearing these artefacts across good data.

However, my solution was quite clunky and slow and was based on the still-mostly-regular resolution of my dataset, rather than any neater general solution in pandas. As I was (and am) also relatively new to Python I did not think this was appropriate to add to xarray myself, but I would like to say that I would definitely use this functionality in future - as would the other colleagues in space physics/meteorology I mentioned this to.

Hi, I did actually just see this - it would solve the unevenly sampled data part but really I need to identify the unphysical values that are not tagged by the quality flags first. Once that has been done then resampling and interpolation would be great - but otherwise I will be spreading the effect of bad data.

For this particular set of data I am looking at, I often get individual points which are close to but clearly outliers from the time series so examining a rolling mean would help find these. That is the example I was hoping to solve with this query, but I have already realised that this extends to other problems I will encounter. For example, sudden jumps in the time series (for which I have been recommended to calculate rolling correlation coefficients between two time series) and multiple points jumping all over the place (for which I will probably compare the variance of groups of points and a rolling gradient)

(I really don't know why these aren't cleaned better first, but unfortunately that is the way things are)

Because I need to clean the data before any analysis, the resampling method would probably allow me to get rid of most but not all the bad data. Then I would have to be extra-cautious and throw out lots of possibly good observations just in case. I will definitely use resampling for the analysis but there are so many ways that this would be helpful at the processing stage.