I think the problem with

cov = _mean(da_a * da_b) - da_a.mean(dim=dim) * da_b.mean(dim=dim)

is that the da_a.mean() and the da_b.mean() calls don't know about each other's missing data.

Shouldn't the following do? python cov = ( (da_a * da_b).mean(dim) - ( da_a.where(da_b.notnull()).mean(dim) * da_b.where(da_a.notnull()).mean(dim) ) ) (See here: https://nbviewer.jupyter.org/gist/willirath/cfaa8fb1b53fcb8dcb05ddde839c794c )

@AndrewWilliams3142 @dcherian Looks like I broke the first Gist. :(

Your Example above does not quite get there, because the xr.DataArray(np...).chunk() just leads to one chunk per data array.

Here's a Gist that explains the idea for the correlations: https://nbviewer.jupyter.org/gist/willirath/c5c5274f31c98e8452548e8571158803

With ```python X = xr.DataArray( darr.random.normal(size=array_size, chunks=chunk_size), dims=("t", "y", "x"), name="X", )

Y = xr.DataArray( darr.random.normal(size=array_size, chunks=chunk_size), dims=("t", "y", "x"), name="Y", ) the "bad" / explicit way of calculating the correlationpython corr_exp = ((X - X.mean("t")) * (Y - Y.mean("t"))).mean("t") ``` leads to a graph like this:

Dask won't release any of the tasks defining X and Y until the marked substraction tasks are done.

The "good" / aggregating way of calculting the correlation python corr_agg = (X * Y).mean("t") - X.mean("t") * Y.mean("t") has the following graph where the marked multiplication and mean_chunk tasks are acting on only pairs of chunks and individual chunks and then release the original chunks of X and Y. This graph can be evaluated with a much smaller memory foot print than the other one. (It's not certain that this is always leading to lower memory use, however. But this is a different issue ...)

I'd also add that https://github.com/pydata/xarray/blob/master/xarray/core/computation.py#L1320_L1330 which is essentially python ((x - x.mean()) * (y - y.mean())).mean() is inferior to python (x * y).mean() - x.mean() * y.mean() because it leads to Dask holding all chunks of x in memory (see, e.g., https://github.com/dask/dask/issues/6674 for details).

Resolved with the latest (1.0.0a2 from conda-forge) netcdftime and with netcdftime.datetime instead of datetime.datetime.

Not really minimal but shows a real use:

```python from pathlib import Path import matplotlib.pyplot as plt import xarray as xr

create data path

daily_data_path = Path("/obs_data/") / "ASCAT" / "v1.x.x" / "data/Daily/Netcdf"

get data files and open mf dataset

data_files_2016 = daily_data_path.glob("2016/??/??/*.nc") raw_data_set = xr.open_mfdataset(data_files_2016).isel(latitude=300, longitude=0)

Resample with filling gaps by NaN's and shift to match original time axis

infilled_data_set = raw_data_set.resample(time="1D").asfreq() infilled_data_set.time.data += (raw_data_set.time.data[0] - infilled_data_set.time.data[0])

extract wind speeds

rws = raw_data_set.wind_speed iws = infilled_data_set.wind_speed.rename("resampled_wind_speed")

and plot (with a slight offset to be able to distinguish the lines)

fig, ax = plt.subplots(2, 1)

(iws+1).plot(ax=ax[0]); rws.plot(ax=ax[0]); ax[0].legend(["resampled", "original"], loc=0, ncol=2);

(iws+1).sel(time=slice("2016-08-01", "2016-10-01")).plot(ax=ax[1]); rws.sel(time=slice("2016-08-01", "2016-10-01")).plot(ax=ax[1]); ```

Note the gaps that are more clearly visible in the lower panel.

Shoud I provide a minimal real-world-example?

Does exactly what I need. Thanks!

Thanks @shoyer!

I like the shorter example. It shows the essence of why pathlib is such a nice thing.

Is there anything more to do in this PR?

Am 31. August 2017 17:48:24 schrieb Stephan Hoyer notifications@github.com:

shoyer approved this pull request.

@@ -21,9 +21,38 @@ v0.9.7 (unreleased) Enhancements ~~~~~~~~~~~~

+- Support for pathlib.Path objects added to

I just pushed a commit to add this

• .. ipython::
• :verbatim:
• In [1]: import xarray as xr +
• In [2]: from pathlib import Path # In Python 2, use pathlib2! +
• In [3]: data_dir = Path("data/") +
• In [4]: one_file = data_dir / "dta_for_month_01.nc" +
• In [5]: print(xr.open_dataset(one_file))
• Out[5]:
• <xarray.Dataset>
• [...] +
• In [6]: all_files = data_dir.glob("dta_for_month_*.nc")

I removed this example from What's New for two reason: 1. The section was getting a little longer than essential 2. It's actually a bit of an anti-pattern, since the order of paths matters to xarray but is arbtirary from glob. The right way to write this is sorted(data_dir.glob(...)).

-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/pydata/xarray/pull/1514#pullrequestreview-59903100

AppVeyor test failed with HTTP time-outs when trying to get repodata.json. Can somebody trigger the build again?

flake8 now fails because of the unused pathlib in xarray/tests/__init__.py.

take a look at what we do for handling the optional dask dependency

Ah, that's nice!

And then I remove the pathlib2 dependency I've introduced in setup.py again?

One more thing:

Can we add pathlib2 as an optional dependency and handle the case where it's not installed?

Currently, I've set pathlib2 as a mandatory requirement for Python 2. How'd we achieve pathlib being optional? Test for pathlib upon import and wrap all the if isinstace(path, Path): path = str(path) in a function just passing unmodified path if pathlib's not present?

• [x] Tests added / passed

Tests are already done on my personal Travis Account (skipped intermediate commits there).

• [x] Passes git diff upstream/master | flake8 --diff

Actually true only for git diff upstream/master xarray/ | flake8 --diff. But I guess that's what should pass the linter?

To me, aae32a8 looks like pathlib support is now present whereever it makes sense.

Thanks for the review @shoyer !

On my machine, I've already stared adapting the other backends (to_netcdf et al) to support pathlib as well. I'll include it here.

Sorry, this was based on an old upstream. I'll close and start from an up-to-date master.

+1 for this one.

Yes, I know that xarray usually doesn't use datetime. For climate data past 2262 with a standard calendar, however, the dtypes of the time axis are falling back to datetime / netCDF4.datetime.

My last commit reflects what I'd consider ready to be pulled. Let me know what you think.

netcdftime.datetime vs. datetime.datetime is discussed in Unidata/netcdftime#8. For now, I only added datetime.datetime to the list of plottable other types (see https://github.com/willirath/xarray/blob/6f5e30f3a2f8ca907f0fd91bcd571e7d98027896/xarray/plot/plot.py#L52).

a85b5c6 and e31aaab add a test plotting against a coordinate with netCDF4.datetime objects.

6f5e30f adds datetime.datetime objects to the list of plottable dtypes and splits the check into numpy dtypes (using numpy's type hierarchy) and all other types.

• [x] Add tests showing that netcdftime.datetime objects can be plotted (line and 2d plots).
• [x] Rewrite xarray.plot._ensure_plottable to only let numeric and date-like types pass.

Does matplotlib actually plot netcdftime datetimes correctly?

Yes it does. They are just treated like datetime.datetime objects.

I was wrong here: netCDF4.datetime is datetime.datetime while (netcdftime.datetime is DatetimeProlepticGregorian.

And matplotlib fails on the latter: https://gist.github.com/willirath/48c04db0481e986fac0e44549e164ab0

(This is related to #1261.)

Does matplotlib actually plot netcdftime datetimes correctly?

Yes it does. They are just treated like datetime.datetime objects.

Failing tests: xarray/tests/test_plot.py::TestPlot1D::test_nonnumeric_index_raises_typeerror FAILED xarray/tests/test_plot.py::TestContourf::test_nonnumeric_index_raises_typeerror FAILED xarray/tests/test_plot.py::TestContour::test_nonnumeric_index_raises_typeerror FAILED xarray/tests/test_plot.py::TestPcolormesh::test_nonnumeric_index_raises_typeerror FAILED xarray/tests/test_plot.py::TestImshow::test_nonnumeric_index_raises_typeerror FAIL

All of these check strings as input to the plot functions and expect a TypeError.

xarray.plot._right_dtype, however, uses numpy.issubdtype to check, if the types of the given data are subtypes of the now included netCDF4.datetime, which is interpreted as numpy.generic. And as numpy.generic is the highest type in Numpy's type hierarchy, xarray.plot._right_dtype now always returns True.

A possible way out would be to check for numeric types and datetime-like types separately.