issue_comments

@jenfly and @shoyer pydap version 3.2.2 (newly released last week) should have fixed this issue. Could you verify?

The reason I ask is that, ideally, coarsen would work exactly the same with dask.array and np.ndarray data. By using both serial and parallel coarsen methods from dask, we are adding a dependency but we are ensuring forward compatibility. @shoyer, what's your preference? (1) replicate serial coarsen into xarray or (2) point to dask coarsen methods?

If it's part of dask then it would be almost trivial to implement in xarray. @mrocklin Can we assume that dask/array/chunk.py::coarsen is part of the public API?

@shoyer @jenfly: Good news, I think I was able to track down the bug in pydap that was preventing compatibility. I'm putting a PR together and we could expect it to be merged pretty soon into the master. I wanted to give you a heads up so that you don't waste more time on this.

I like the idea of passing PydapDataStore objects that include the session object. It seems more likely to be forward compatible, especially if Central Authentication Services multiply (as one would expect) with different authentication mechanisms.

The dask implementation has the following API: dask.array.coarsen(reduction, x, axes, trim_excess=False) so a proposed xarray API could look like: xarray.coarsen(reduction, x, axes, chunks=None, trim_excess=False), resulting in the following implementation: 1. If the underlying data to x is dask.array, yields x.chunks(chunks).array.coarsen(reduction, axes, trim_excess) 2. Else, copy the block_reduce function.

Does that fit with the xarray API?

@shoyer Let me know if you need any more changes.

So I've tried with conventions.py and it works. However, allowing a NaN _FillValue for complex floats raises an exception in h5py: h5py/h5py#805. This means that until this is fixed in h5py we should restrict this default behaviour to real float types.

I know but the alternative would be to scan the data before deciding but that would kind of break the whole dask integration.

The only problem I see is that in the current implementation this fix puts a _FillValue attribute even if there are no NaN in the variable. Usually, cdo would not do that. I guess there are little risks but I think it's important that this be said. Alternative implementations would be way too cumbersome so I think the approach used in this PR is the lesser of two evils.

I'm pretty sure that of over many versions of CDO NaNs are not properly handled. For example, all the setmiss* commands have a very hard time with NaN's.

I have been using the encoding option but I find it not very intuitive and kind of cumbersome when there are many variables.

I think you're right that actually putting _FillValue = NaN in the variable attribute might be enough to solve most issues with third parties. If it still does not work, then it would be reasonable to raise this issue with each one of the third parties.