html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue
https://github.com/pydata/xarray/pull/5390#issuecomment-850820173,https://api.github.com/repos/pydata/xarray/issues/5390,850820173,MDEyOklzc3VlQ29tbWVudDg1MDgyMDE3Mw==,5700886,2021-05-29T11:51:50Z,2021-05-29T11:51:59Z,CONTRIBUTOR,"I think the problem with
> `cov = _mean(da_a * da_b) - da_a.mean(dim=dim) * da_b.mean(dim=dim)`
is that the `da_a.mean()` and the `da_b.mean()` calls don't know about each other's missing data.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,904153867
https://github.com/pydata/xarray/pull/5390#issuecomment-850819741,https://api.github.com/repos/pydata/xarray/issues/5390,850819741,MDEyOklzc3VlQ29tbWVudDg1MDgxOTc0MQ==,5700886,2021-05-29T11:48:02Z,2021-05-29T11:48:02Z,CONTRIBUTOR,"Shouldn't the following do?
```python
cov = (
(da_a * da_b).mean(dim)
- (
da_a.where(da_b.notnull()).mean(dim)
* da_b.where(da_a.notnull()).mean(dim)
)
)
```
(See here: https://nbviewer.jupyter.org/gist/willirath/cfaa8fb1b53fcb8dcb05ddde839c794c )","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,904153867
https://github.com/pydata/xarray/pull/5390#issuecomment-850542572,https://api.github.com/repos/pydata/xarray/issues/5390,850542572,MDEyOklzc3VlQ29tbWVudDg1MDU0MjU3Mg==,5700886,2021-05-28T16:45:55Z,2021-05-28T16:45:55Z,CONTRIBUTOR,"@AndrewWilliams3142 @dcherian Looks like I broke the first Gist. :(
Your Example above does not quite get there, because the `xr.DataArray(np...).chunk()` just leads to one chunk per data array.
Here's a Gist that explains the idea for the correlations: https://nbviewer.jupyter.org/gist/willirath/c5c5274f31c98e8452548e8571158803
With
```python
X = xr.DataArray(
darr.random.normal(size=array_size, chunks=chunk_size),
dims=(""t"", ""y"", ""x""),
name=""X"",
)
Y = xr.DataArray(
darr.random.normal(size=array_size, chunks=chunk_size),
dims=(""t"", ""y"", ""x""),
name=""Y"",
)
```
the ""bad"" / explicit way of calculating the correlation
```python
corr_exp = ((X - X.mean(""t"")) * (Y - Y.mean(""t""))).mean(""t"")
```
leads to a graph like this:
Dask won't release any of the tasks defining `X` and `Y` until the marked `sub`straction tasks are done.
The ""good"" / aggregating way of calculting the correlation
```python
corr_agg = (X * Y).mean(""t"") - X.mean(""t"") * Y.mean(""t"")
```
has the following graph
where the marked `mul`tiplication and `mean_chunk` tasks are acting on only pairs of chunks and individual chunks and then release the original chunks of `X` and `Y`. This graph _can_ be evaluated with a much smaller memory foot print than the other one. (It's not certain that this is always leading to lower memory use, however. But this is a different issue ...)","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 1}",,904153867
https://github.com/pydata/xarray/issues/4804#issuecomment-760114285,https://api.github.com/repos/pydata/xarray/issues/4804,760114285,MDEyOklzc3VlQ29tbWVudDc2MDExNDI4NQ==,5700886,2021-01-14T10:44:19Z,2021-01-14T10:44:19Z,CONTRIBUTOR,"I'd also add that https://github.com/pydata/xarray/blob/master/xarray/core/computation.py#L1320_L1330 which is essentially
```python
((x - x.mean()) * (y - y.mean())).mean()
```
is inferior to
```python
(x * y).mean() - x.mean() * y.mean()
```
because it leads to Dask holding all chunks of `x` in memory (see, e.g., https://github.com/dask/dask/issues/6674 for details).","{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,785329941
https://github.com/pydata/xarray/issues/2043#issuecomment-379713292,https://api.github.com/repos/pydata/xarray/issues/2043,379713292,MDEyOklzc3VlQ29tbWVudDM3OTcxMzI5Mg==,5700886,2018-04-09T10:55:58Z,2018-04-09T10:55:58Z,CONTRIBUTOR,Resolved with the latest (1.0.0a2 from conda-forge) `netcdftime` and with `netcdftime.datetime` instead of `datetime.datetime`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,312449001
https://github.com/pydata/xarray/issues/1596#issuecomment-332624901,https://api.github.com/repos/pydata/xarray/issues/1596,332624901,MDEyOklzc3VlQ29tbWVudDMzMjYyNDkwMQ==,5700886,2017-09-27T19:08:59Z,2017-09-27T19:08:59Z,CONTRIBUTOR,"Not really minimal but shows a real use:
> ```python
> from pathlib import Path
> import matplotlib.pyplot as plt
> import xarray as xr
>
> # create data path
> daily_data_path = Path(""/obs_data/"") / ""ASCAT"" / ""v1.x.x"" / ""data/Daily/Netcdf""
>
> # get data files and open mf dataset
> data_files_2016 = daily_data_path.glob(""2016/??/??/*.nc"")
> raw_data_set = xr.open_mfdataset(data_files_2016).isel(latitude=300,
> longitude=0)
>
> # Resample with filling gaps by NaN's and shift to match original time axis
> infilled_data_set = raw_data_set.resample(time=""1D"").asfreq()
> infilled_data_set.time.data += (raw_data_set.time.data[0] -
> infilled_data_set.time.data[0])
>
> # extract wind speeds
> rws = raw_data_set.wind_speed
> iws = infilled_data_set.wind_speed.rename(""resampled_wind_speed"")
>
> # and plot (with a slight offset to be able to distinguish the lines)
>
> fig, ax = plt.subplots(2, 1)
>
> (iws+1).plot(ax=ax[0]);
> rws.plot(ax=ax[0]);
> ax[0].legend([""resampled"", ""original""], loc=0, ncol=2);
>
> (iws+1).sel(time=slice(""2016-08-01"", ""2016-10-01"")).plot(ax=ax[1]);
> rws.sel(time=slice(""2016-08-01"", ""2016-10-01"")).plot(ax=ax[1]);
> ```
>
> 
>
> Note the gaps that are more clearly visible in the lower panel.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,260912521
https://github.com/pydata/xarray/issues/1596#issuecomment-332617308,https://api.github.com/repos/pydata/xarray/issues/1596,332617308,MDEyOklzc3VlQ29tbWVudDMzMjYxNzMwOA==,5700886,2017-09-27T18:41:10Z,2017-09-27T18:41:10Z,CONTRIBUTOR,Shoud I provide a minimal real-world-example?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,260912521
https://github.com/pydata/xarray/issues/1596#issuecomment-332614980,https://api.github.com/repos/pydata/xarray/issues/1596,332614980,MDEyOklzc3VlQ29tbWVudDMzMjYxNDk4MA==,5700886,2017-09-27T18:32:54Z,2017-09-27T18:32:54Z,CONTRIBUTOR,Does exactly what I need. Thanks!,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,260912521
https://github.com/pydata/xarray/pull/1514#issuecomment-326548654,https://api.github.com/repos/pydata/xarray/issues/1514,326548654,MDEyOklzc3VlQ29tbWVudDMyNjU0ODY1NA==,5700886,2017-09-01T10:35:41Z,2017-09-01T10:35:41Z,CONTRIBUTOR,"Thanks @shoyer!
I like the shorter example. It shows the essence of why pathlib is such a
nice thing.
Is there anything more to do in this PR?
Am 31. August 2017 17:48:24 schrieb Stephan Hoyer :
> shoyer approved this pull request.
>
>
>
>> @@ -21,9 +21,38 @@ v0.9.7 (unreleased)
> Enhancements
> ~~~~~~~~~~~~
>
> +- Support for `pathlib.Path` objects added to
>
> I just pushed a commit to add this
>
>> + .. ipython::
> + :verbatim:
> + In [1]: import xarray as xr
> +
> + In [2]: from pathlib import Path # In Python 2, use pathlib2!
> +
> + In [3]: data_dir = Path(""data/"")
> +
> + In [4]: one_file = data_dir / ""dta_for_month_01.nc""
> +
> + In [5]: print(xr.open_dataset(one_file))
> + Out[5]:
> +
> + [...]
> +
> + In [6]: all_files = data_dir.glob(""dta_for_month_*.nc"")
>
> I removed this example from What's New for two reason:
> 1. The section was getting a little longer than essential
> 2. It's actually a bit of an anti-pattern, since the order of paths matters
> to xarray but is arbtirary from glob. The right way to write this is
> `sorted(data_dir.glob(...))`.
>
> --
> You are receiving this because you were mentioned.
> Reply to this email directly or view it on GitHub:
> https://github.com/pydata/xarray/pull/1514#pullrequestreview-59903100
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,251734482
https://github.com/pydata/xarray/pull/1514#issuecomment-325187598,https://api.github.com/repos/pydata/xarray/issues/1514,325187598,MDEyOklzc3VlQ29tbWVudDMyNTE4NzU5OA==,5700886,2017-08-27T09:35:40Z,2017-08-27T09:35:40Z,CONTRIBUTOR,AppVeyor test failed with HTTP time-outs when trying to get `repodata.json`. Can somebody trigger the build again?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,251734482
https://github.com/pydata/xarray/pull/1514#issuecomment-325158074,https://api.github.com/repos/pydata/xarray/issues/1514,325158074,MDEyOklzc3VlQ29tbWVudDMyNTE1ODA3NA==,5700886,2017-08-26T19:47:40Z,2017-08-26T19:47:40Z,CONTRIBUTOR,`flake8` now fails because of the unused `pathlib` in `xarray/tests/__init__.py`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,251734482
https://github.com/pydata/xarray/pull/1514#issuecomment-324970564,https://api.github.com/repos/pydata/xarray/issues/1514,324970564,MDEyOklzc3VlQ29tbWVudDMyNDk3MDU2NA==,5700886,2017-08-25T16:28:16Z,2017-08-25T16:28:16Z,CONTRIBUTOR,"> take a look at what we do for handling the optional dask dependency
Ah, that's nice!
And then I remove the `pathlib2` dependency I've introduced in [setup.py](https://github.com/willirath/xarray/blob/799-add-pathlib-support-2/setup.py#L40) again?
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,251734482
https://github.com/pydata/xarray/pull/1514#issuecomment-324966586,https://api.github.com/repos/pydata/xarray/issues/1514,324966586,MDEyOklzc3VlQ29tbWVudDMyNDk2NjU4Ng==,5700886,2017-08-25T16:12:23Z,2017-08-25T16:12:23Z,CONTRIBUTOR,"One more thing:
> Can we add pathlib2 as an optional dependency and handle the case where it's not installed?
Currently, I've set `pathlib2` as a mandatory requirement for Python 2. How'd we achieve `pathlib` being optional? Test for `pathlib` upon import and wrap all the `if isinstace(path, Path): path = str(path)` in a function just passing unmodified `path` if `pathlib`'s not present?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,251734482
https://github.com/pydata/xarray/pull/1514#issuecomment-324962241,https://api.github.com/repos/pydata/xarray/issues/1514,324962241,MDEyOklzc3VlQ29tbWVudDMyNDk2MjI0MQ==,5700886,2017-08-25T15:56:20Z,2017-08-25T15:56:20Z,CONTRIBUTOR,"> - [x] Tests added / passed
Tests are already done on my personal Travis Account (skipped intermediate commits there).
> - [x] Passes git diff upstream/master | flake8 --diff
Actually true only for `git diff upstream/master xarray/ | flake8 --diff`. But I guess that's what should pass the linter?
To me, aae32a8 looks like pathlib support is now present whereever it makes sense.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,251734482
https://github.com/pydata/xarray/pull/1514#issuecomment-324898664,https://api.github.com/repos/pydata/xarray/issues/1514,324898664,MDEyOklzc3VlQ29tbWVudDMyNDg5ODY2NA==,5700886,2017-08-25T11:59:27Z,2017-08-25T11:59:27Z,CONTRIBUTOR,"Thanks for the review @shoyer !
On my machine, I've already stared adapting the other backends (`to_netcdf` et al) to support `pathlib` as well. I'll include it here.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,251734482
https://github.com/pydata/xarray/pull/1513#issuecomment-323803983,https://api.github.com/repos/pydata/xarray/issues/1513,323803983,MDEyOklzc3VlQ29tbWVudDMyMzgwMzk4Mw==,5700886,2017-08-21T17:25:45Z,2017-08-21T17:25:57Z,CONTRIBUTOR,"Sorry, this was based on an old upstream. I'll close and start from an up-to-date master.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,251714595
https://github.com/pydata/xarray/issues/799#issuecomment-323785637,https://api.github.com/repos/pydata/xarray/issues/799,323785637,MDEyOklzc3VlQ29tbWVudDMyMzc4NTYzNw==,5700886,2017-08-21T16:04:49Z,2017-08-21T16:04:49Z,CONTRIBUTOR,+1 for this one.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,142675134
https://github.com/pydata/xarray/pull/1261#issuecomment-284397545,https://api.github.com/repos/pydata/xarray/issues/1261,284397545,MDEyOklzc3VlQ29tbWVudDI4NDM5NzU0NQ==,5700886,2017-03-06T13:37:33Z,2017-03-06T13:37:33Z,CONTRIBUTOR,"Yes, I know that xarray usually doesn't use `datetime`. For climate data past 2262 with a standard calendar, however, the dtypes of the time axis are falling back to `datetime` / `netCDF4.datetime`.
My last commit reflects what I'd consider ready to be pulled. Let me know what you think.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,207011524
https://github.com/pydata/xarray/pull/1261#issuecomment-279467664,https://api.github.com/repos/pydata/xarray/issues/1261,279467664,MDEyOklzc3VlQ29tbWVudDI3OTQ2NzY2NA==,5700886,2017-02-13T17:50:13Z,2017-02-13T17:50:13Z,CONTRIBUTOR,"`netcdftime.datetime` vs. `datetime.datetime` is discussed in Unidata/netcdftime#8. For now, I only added `datetime.datetime` to the list of plottable other types (see https://github.com/willirath/xarray/blob/6f5e30f3a2f8ca907f0fd91bcd571e7d98027896/xarray/plot/plot.py#L52).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,207011524
https://github.com/pydata/xarray/pull/1261#issuecomment-279459572,https://api.github.com/repos/pydata/xarray/issues/1261,279459572,MDEyOklzc3VlQ29tbWVudDI3OTQ1OTU3Mg==,5700886,2017-02-13T17:21:37Z,2017-02-13T17:26:12Z,CONTRIBUTOR,"a85b5c6 and e31aaab add a test plotting against a coordinate with `netCDF4.datetime` objects.
6f5e30f adds `datetime.datetime` objects to the list of plottable dtypes and splits the check into numpy dtypes (using numpy's type hierarchy) and all other types.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,207011524
https://github.com/pydata/xarray/pull/1261#issuecomment-279186528,https://api.github.com/repos/pydata/xarray/issues/1261,279186528,MDEyOklzc3VlQ29tbWVudDI3OTE4NjUyOA==,5700886,2017-02-12T00:26:59Z,2017-02-13T17:25:32Z,CONTRIBUTOR,"- [x] Add tests showing that netcdftime.datetime objects can be plotted (line and 2d plots).
- [x] Rewrite `xarray.plot._ensure_plottable` to only let numeric and date-like types pass.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,207011524
https://github.com/pydata/xarray/pull/1261#issuecomment-279340528,https://api.github.com/repos/pydata/xarray/issues/1261,279340528,MDEyOklzc3VlQ29tbWVudDI3OTM0MDUyOA==,5700886,2017-02-13T09:54:30Z,2017-02-13T09:54:30Z,CONTRIBUTOR,"> > Does matplotlib actually plot netcdftime datetimes correctly?
>
> Yes it does. They are just treated like `datetime.datetime` objects.
I was wrong here: [`netCDF4.datetime` _is_ `datetime.datetime`](https://github.com/Unidata/netcdf4-python/blob/master/netCDF4/_netCDF4.pyx#L5084) while ([`netcdftime.datetime` is `DatetimeProlepticGregorian`](https://github.com/Unidata/netcdftime/blob/master/netcdftime/__init__.py#L3).
And matplotlib fails on the latter: https://gist.github.com/willirath/48c04db0481e986fac0e44549e164ab0","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,207011524
https://github.com/pydata/xarray/issues/1263#issuecomment-279223749,https://api.github.com/repos/pydata/xarray/issues/1263,279223749,MDEyOklzc3VlQ29tbWVudDI3OTIyMzc0OQ==,5700886,2017-02-12T14:55:50Z,2017-02-12T14:55:50Z,CONTRIBUTOR,(This is related to #1261.),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,207054921
https://github.com/pydata/xarray/pull/1261#issuecomment-279185373,https://api.github.com/repos/pydata/xarray/issues/1261,279185373,MDEyOklzc3VlQ29tbWVudDI3OTE4NTM3Mw==,5700886,2017-02-12T00:03:26Z,2017-02-12T00:03:26Z,CONTRIBUTOR,"> Does matplotlib actually plot netcdftime datetimes correctly?
Yes it does. They are just treated like `datetime.datetime` objects.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,207011524
https://github.com/pydata/xarray/pull/1261#issuecomment-279185264,https://api.github.com/repos/pydata/xarray/issues/1261,279185264,MDEyOklzc3VlQ29tbWVudDI3OTE4NTI2NA==,5700886,2017-02-12T00:01:21Z,2017-02-12T00:01:21Z,CONTRIBUTOR,"Failing tests:
```
xarray/tests/test_plot.py::TestPlot1D::test_nonnumeric_index_raises_typeerror FAILED
xarray/tests/test_plot.py::TestContourf::test_nonnumeric_index_raises_typeerror FAILED
xarray/tests/test_plot.py::TestContour::test_nonnumeric_index_raises_typeerror FAILED
xarray/tests/test_plot.py::TestPcolormesh::test_nonnumeric_index_raises_typeerror FAILED
xarray/tests/test_plot.py::TestImshow::test_nonnumeric_index_raises_typeerror FAIL
```
All of these check strings as input to the plot functions and expect a `TypeError`.
`xarray.plot._right_dtype`, however, uses `numpy.issubdtype` to check, if the types of the given data are subtypes of the now included `netCDF4.datetime`, which is interpreted as `numpy.generic`. And as `numpy.generic` is the highest type in Numpy's type hierarchy, `xarray.plot._right_dtype` now always returns `True`.
A possible way out would be to check for numeric types and datetime-like types separately.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,207011524