😄 It's a good community to contribute to.

Isn't that sad that it is my first PR? 😄

I'd support that.

Nice work, @jukent!

Yeah. That's true. I did overlook that. Thanks!

Also, while I was walking through the logic in this problem, I found that in the _reduce_method functions of the DataArrayCoarsen and DatasetCoarsen classes, the kwargs variable is being shadowed:

https://github.com/pydata/xarray/blob/df7b2eae3a26c1e86bd5f1dd7dab9cc8c4e53914/xarray/core/rolling.py#L692-L701

So, you can see that the skipna option is just being ignored. That's another pretty easy fix, but it will change existing behavior. It might be prudent to look to see if there are any known bug reports referencing the skipna parameter being ignored.

@dcherian @jukent: After a little walk through the code, I think the problem is in xarray/core/variable.py. If you look at the first part of the coarsen function in this file:

https://github.com/pydata/xarray/blob/df7b2eae3a26c1e86bd5f1dd7dab9cc8c4e53914/xarray/core/variable.py#L1945-L1953

you will see that **kwargs is not being pass into the _coarsen_reshape function:

https://github.com/pydata/xarray/blob/df7b2eae3a26c1e86bd5f1dd7dab9cc8c4e53914/xarray/core/variable.py#L1961

And down at the bottom of that function:

https://github.com/pydata/xarray/blob/df7b2eae3a26c1e86bd5f1dd7dab9cc8c4e53914/xarray/core/variable.py#L2024-L2025

it retrieves the keep_attrs parameter from global options, and doesn't check for a passed-in argument.

@jukent, why don't you draft a PR to fix this problem?

After some testing, I discovered that:

```python

Your code here

import xarray as xr

xr.set_options(keep_attrs=True) # NOTE GLOBAL OPTION!!!!

ds = xr.tutorial.load_dataset("air_temperature") ds2 = xr.tutorial.load_dataset("air_temperature")

xr.testing.assert_identical(ds, ds2) # passes ds.coarsen(lat=5).mean() xr.testing.assert_identical(ds, ds2) # passes ```

makes your example pass. So, it seems that somewhere along the chain of functions the keep_attrs parameter is being lost or modified.

@rabernat When you say "underlying array store", are you talking about the storage layer? That is, the zarr store or the netcdf file?

It seems to me that the there are lots of "layers" of "chunking", especially when you are talking about chunking an entire dataset, which really confuses the whole issue. On an HPC system, there's filesystem blocksize, NetCDF/HDF5 "internal" chunks, chunking by spreading the data over multiple files, and in-memory chunks (i.e., Dask chunks). I'm not an expert on object store, but my understanding of object store is that (if you are storing NetCDF/HDF5 on object store) there is still "interal" NetCDF/HDF5 "chunking", then chunking over objects/files, and then in-memory chunking.

Well, I'm certainly not trying to argue that the PhysArray implementation is the solution. It's just a solution, and a solution for a much smaller problem.

I understand your concerns about cf_units. I've heard many people make the same argument for pint, and I appreciate it. I went with cf_units because of its dependence on UDUNITS, which is declared as the "standard" in the CF Conventions. I never had the time (nor do I foresee ever having the time) to see if pint was fully compliant with UDUNITS, so I went with something I knew was.

I personally think it's fine to discuss this here, unless other people would like to see this go offline. To address some of your issues:

1) As a container rather than subclass, it does not implement many of the methods of DataArray

Yes. I agree. In fact, my first implementation was a subclass of DataArray, but after reading the Xarray documentation here and the Issue #706, I decided on composition for this implementation.

2) There are a few design choices I don't understand, like why calendar is always a property of a PhysArray even when it isn't storing a time

I personally didn't see much of a benefit to a check like hasattr(obj, 'calendar') versus obj.calendar is None (other than the fact that these two checks are opposite). So, for my code, I felt that obj.calendar is None was a reasonable check to see if the units were "calendared time" units.

why cftime objects aren't used instead of units to manage time

As I mentioned in my post, this is just my personal preference. I think that calendared time units are...well...just units, and that the same mechanism for dealing with all other units should deal with calendared time units. It's just an aesthetic that I prefer. (That said, I'm extremely happy that someone finally dealt with non-standard calendars with cftime.)

why the positive attribute is important enough for PhysArray to manage (I've never seen it in any data I've used, and it's easy to check if a DataArray is all positive or negative with a function call)

The CF Conventions define the positive attribute to indicate that the field represents the vertical component of a vector, and the value of positive indicates what direction the vertical component points (up or down) if the data is positive. So, if you add field X and field Y, and X.positive == 'up' but Y.positive == 'down', then you need to actually subtract the fields. It's an annoying attribute that I've had to deal with when preparing data for CMIP6.

In the end, I'm happy doing whatever the community wants with this code. I can pull it out into it's own repo (along with the unit tests, which are also in the PyConform repo). And then, after that, I have no problem with people taking it in a different direction (e.g., using pint, injecting properties based on the value of the attributes, etc.). I'm also happy with a subclass option, as my C++ experiences in the past have made me very comfortable with inheritance. I take guidance easily. I just don't usually have a lot of time to devote to development any more these days. :-(

Thanks, @jhamman, for the reference.

I have, indeed, been thinking about this a lot. I've hit the same problem that you have @mcgibbon and came to the same conclusion that subclassing was the only way to go. I'm not sure I'm happy with my solution, but I wrote a (partial) implementation of a DataArray subclass to deal with units. @jhamman suggested that I pull this out into a separate repo, and I hope to do that one day... However, if you are curious, now, you can view it here:

https://github.com/NCAR/PyConform/tree/rewrite/source/pyconform/physarrays

Note that this implementation deals with time as just another unit, rather than requiring any special dealing with time. This works if you units package can deal with calendared times, too. I am currently using cf_units to deal with the units, not pint, but the result is similar. However, cf_units does not deal with calendar conversion, and it doesn't always deal with non-standard calendars well. In the end, though, if I could voice my own 2 cents, I believe that calendared times should be dealt with using the same mechanics as any other unit. Just my bias.

Also, note that this implementation also deals with the positive attribute of the data, in addition to the units and calendar attributes. The positive attribute can be tricky, but if the values don't match between two arrays, then you need to convert before doing any math.

Let me know what you think about this. If it's a priority, I can pull this out into its own repository.

Oh, good grief! I looked at that documentation the just yesterday and I completely missed that!

Sorry. I'm getting old.

Sorry for the delay... I saw the reference and then needed to find some time to read back over the issues to get some context.

You are correct. The PyReshaper was designed to address this type of problem, though not exactly the issue with xarray and dask. It's a pretty common problem, and it's the reason that the CESM developers are moving to long-term archival of time-series files ONLY. (In other words, PyReshaper is being incorporated into the automated CESM run-processes.) ...Of course, one could argue that this step shouldn't be necessary with some clever I/O in the models themselves to write time-series directly.

The PyReshaper opens and closes each time-slice file explicitly before and after each read, respectively. And, if fully scaled (i.e., 1 MPI process per output file), you only ever have 2 files open at a time per process. In this particular operation, the overhead associated with open/close on the input files is negligible compared to the total R/W times.

So, anyway, the PyReshaper (https://github.com/NCAR/PyReshaper) can definitely help...though I consider it a stop-gap for the moment. I'm happy to help people figure out how to get it to work for you problems, if that's a path you want to consider.