id,node_id,number,title,user,state,locked,assignee,milestone,comments,created_at,updated_at,closed_at,author_association,active_lock_reason,draft,pull_request,body,reactions,performed_via_github_app,state_reason,repo,type
1974350560,I_kwDOAMm_X851rjLg,8402,`where` dtype upcast with numpy 2,1828519,open,0,,,10,2023-11-02T14:12:49Z,2024-04-15T19:18:49Z,,CONTRIBUTOR,,,,"### What happened?
I'm testing my code with numpy 2.0 and current `main` xarray and dask and ran into a change that I guess is expected given the way xarray does things, but want to make sure as it could be unexpected for many users.
Doing `DataArray.where` with an integer array less than 64-bits and an integer as the new value will upcast the array to 64-bit integers (python's `int`). With old versions of numpy this would preserve the dtype of the array. As far as I can tell the relevant xarray code hasn't changed so this seems to be more about numpy making things more consistent.
The main problem seems to come down to:
https://github.com/pydata/xarray/blob/d933578ebdc4105a456bada4864f8ffffd7a2ced/xarray/core/duck_array_ops.py#L218
As this converts my scalar input `int` to a numpy array. If it didn't do this array conversion then numpy works as expected. See the MCVE for the xarray specific example, but here's the numpy equivalent:
```python
import numpy as np
a = np.zeros((2, 2), dtype=np.uint16)
# what I'm intending to do with my xarray `data_arr.where(cond, 2)`
np.where(a != 0, a, 2).dtype
# dtype('uint16')
# equivalent to what xarray does:
np.where(a != 0, a, np.asarray(2)).dtype
# dtype('int64')
# workaround, cast my scalar to a specific numpy type
np.where(a != 0, a, np.asarray(np.uint16(2))).dtype
# dtype('uint16')
```
From a numpy point of view, the second where call makes sense that 2 arrays should be upcast to the same dtype so they can be combined. But from an xarray user point of view, I'm entering a scalar so I expect it to be the same as the first where call above.
### What did you expect to happen?
See above.
### Minimal Complete Verifiable Example
```Python
import xarray as xr
import numpy as np
data_arr = xr.DataArray(np.array([1, 2], dtype=np.uint16))
print(data_arr.where(data_arr == 2, 3).dtype)
# int64
```
### MVCE confirmation
- [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
- [X] Complete example — the example is self-contained, including all data and the text of any traceback.
- [X] Verifiable example — the example copy & pastes into an IPython prompt or [Binder notebook](https://mybinder.org/v2/gh/pydata/xarray/main?urlpath=lab/tree/doc/examples/blank_template.ipynb), returning the result.
- [X] New issue — a search of GitHub Issues suggests this is not a duplicate.
- [X] Recent environment — the issue occurs with the latest version of xarray and its dependencies.
### Relevant log output
_No response_
### Anything else we need to know?
Numpy 1.x preserves the dtype.
```python
In [1]: import numpy as np
In [2]: np.asarray(2).dtype
Out[2]: dtype('int64')
In [3]: a = np.zeros((2, 2), dtype=np.uint16)
In [4]: np.where(a != 0, a, np.asarray(2)).dtype
Out[4]: dtype('uint16')
In [5]: np.where(a != 0, a, np.asarray(np.uint16(2))).dtype
Out[5]: dtype('uint16')
```
### Environment
```
INSTALLED VERSIONS
------------------
commit: None
python: 3.11.4 | packaged by conda-forge | (main, Jun 10 2023, 18:08:17) [GCC 12.2.0]
python-bits: 64
OS: Linux
OS-release: 6.4.6-76060406-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: ('en_US', 'UTF-8')
libhdf5: 1.14.2
libnetcdf: 4.9.2
xarray: 2023.10.2.dev21+gfcdc8102
pandas: 2.2.0.dev0+495.gecf449b503
numpy: 2.0.0.dev0+git20231031.42c33f3
scipy: 1.12.0.dev0+1903.18d0a2f
netCDF4: 1.6.5
pydap: None
h5netcdf: 1.2.0
h5py: 3.10.0
Nio: None
zarr: 2.16.1
cftime: 1.6.3
nc_time_axis: None
PseudoNetCDF: None
iris: None
bottleneck: 1.3.7.post0.dev7
dask: 2023.10.1+4.g91098a63
distributed: 2023.10.1+5.g76dd8003
matplotlib: 3.9.0.dev0
cartopy: None
seaborn: None
numbagg: None
fsspec: 2023.6.0
cupy: None
pint: 0.22
sparse: None
flox: None
numpy_groupies: None
setuptools: 68.0.0
pip: 23.2.1
conda: None
pytest: 7.4.0
mypy: None
IPython: 8.14.0
sphinx: 7.1.2
```
","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/8402/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,issue
1966675016,I_kwDOAMm_X851ORRI,8388,Type annotation compatibility with numpy ufuncs,1828519,closed,0,,,4,2023-10-28T17:25:11Z,2023-11-02T12:44:50Z,2023-11-02T12:44:50Z,CONTRIBUTOR,,,,"### Is your feature request related to a problem?
I'd like mypy to understand that xarray DataArrays passed to numpy ufuncs have a return type of xarray DataArray.
```python
import xarray as xr
import numpy as np
def compute_relative_azimuth(sat_azi: xr.DataArray, sun_azi: xr.DataArray) -> xr.DataArray:
abs_diff = np.absolute(sun_azi - sat_azi)
ssadiff = np.minimum(abs_diff, 360 - abs_diff)
return ssadiff
```
```bash
$ mypy ./xarray_mypy.py
xarray_mypy.py:7: error: Incompatible return value type (got ""ndarray[Any, dtype[Any]]"", expected ""DataArray"") [return-value]
Found 1 error in 1 file (checked 1 source file)
```
### Describe the solution you'd like
I'm not sure if this is possible, if it is something xarray can fix, or something numpy needs to ""fix"". I'd like the above situation to ""just work"" without anything more than maybe some extra type-stub package.
### Describe alternatives you've considered
Cast types or other type coercion or tell mypy to ignore the type issues for these numpy call.
### Additional context
https://stackoverflow.com/questions/77369042/typing-when-passing-xarray-dataarray-objects-to-numpy-ufuncs","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/8388/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed,13221727,issue
1085992113,I_kwDOAMm_X85Auuyx,6092,DeprecationWarning regarding use of distutils Version classes,1828519,closed,0,,,10,2021-12-21T16:11:08Z,2023-09-05T06:39:39Z,2021-12-24T14:50:48Z,CONTRIBUTOR,,,,"
**What happened**:
While working on some tests that catch and check for warnings in my library I found that xarray with new versions of Python (I think this is the trigger) causes a ton of DeprecationWarnings on import:
```python
/home/davidh/miniconda3/envs/satpy_py39/lib/python3.9/site-packages/xarray/core/pycompat.py:22: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
duck_array_version = LooseVersion(duck_array_module.__version__)
```
**What you expected to happen**:
No warnings.
**Minimal Complete Verifiable Example**:
```python
import warnings
warnings.simplefilter(""always"")
import xarray as xr
```
Results in:
```
/home/davidh/miniconda3/envs/satpy_py39/lib/python3.9/site-packages/xarray/core/pycompat.py:22: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
duck_array_version = LooseVersion(duck_array_module.__version__)
/home/davidh/miniconda3/envs/satpy_py39/lib/python3.9/site-packages/xarray/core/pycompat.py:37: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
duck_array_version = LooseVersion(""0.0.0"")
/home/davidh/miniconda3/envs/satpy_py39/lib/python3.9/site-packages/xarray/core/pycompat.py:37: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
duck_array_version = LooseVersion(""0.0.0"")
/home/davidh/miniconda3/envs/satpy_py39/lib/python3.9/site-packages/setuptools/_distutils/version.py:351: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
other = LooseVersion(other)
/home/davidh/miniconda3/envs/satpy_py39/lib/python3.9/site-packages/setuptools/_distutils/version.py:351: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
other = LooseVersion(other)
/home/davidh/miniconda3/envs/satpy_py39/lib/python3.9/site-packages/xarray/core/npcompat.py:82: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
if LooseVersion(np.__version__) >= ""1.20.0"":
/home/davidh/miniconda3/envs/satpy_py39/lib/python3.9/site-packages/setuptools/_distutils/version.py:351: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
other = LooseVersion(other)
/home/davidh/miniconda3/envs/satpy_py39/lib/python3.9/site-packages/xarray/core/pdcompat.py:45: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
if LooseVersion(pd.__version__) < ""0.25.0"":
/home/davidh/miniconda3/envs/satpy_py39/lib/python3.9/site-packages/setuptools/_distutils/version.py:351: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
other = LooseVersion(other)
```
**Anything else we need to know?**:
**Environment**:
Output of xr.show_versions()
```
/home/davidh/miniconda3/envs/satpy_py39/lib/python3.9/site-packages/_distutils_hack/__init__.py:35: UserWarning: Setuptools is replacing distutils.
warnings.warn(""Setuptools is replacing distutils."")
/home/davidh/miniconda3/envs/satpy_py39/lib/python3.9/asyncio/base_events.py:681: ResourceWarning: unclosed event loop <_UnixSelectorEventLoop running=False closed=False debug=False>
_warn(f""unclosed event loop {self!r}"", ResourceWarning, source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback
INSTALLED VERSIONS
------------------
commit: None
python: 3.9.9 | packaged by conda-forge | (main, Dec 20 2021, 02:41:03)
[GCC 9.4.0]
python-bits: 64
OS: Linux
OS-release: 5.15.5-76051505-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: ('en_US', 'UTF-8')
libhdf5: 1.12.1
libnetcdf: 4.8.1
xarray: 0.20.2
pandas: 1.3.5
numpy: 1.20.3
scipy: 1.7.3
netCDF4: 1.5.8
pydap: None
h5netcdf: 0.12.0
h5py: 3.6.0
Nio: None
zarr: 2.10.3
cftime: 1.5.1.1
nc_time_axis: None
PseudoNetCDF: None
rasterio: 1.2.10
cfgrib: None
iris: None
bottleneck: 1.3.2
dask: 2021.12.0
distributed: 2021.12.0
matplotlib: 3.5.1
cartopy: 0.20.1
seaborn: None
numbagg: None
fsspec: 2021.11.1
cupy: None
pint: None
sparse: None
setuptools: 60.0.3
pip: 21.3.1
conda: None
pytest: 6.2.5
IPython: 7.30.1
sphinx: 4.3.2
```
INSTALLED VERSIONS
------------------
commit: None
python: 3.7.6 | packaged by conda-forge | (default, Jan 7 2020, 22:33:48)
[GCC 7.3.0]
python-bits: 64
OS: Linux
OS-release: 5.3.0-7629-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8
libhdf5: 1.10.5
libnetcdf: 4.7.3
xarray: 0.15.1.dev21+g20e6236f
pandas: 1.1.0.dev0+630.gedcf1c8f8
numpy: 1.19.0.dev0+acba244
scipy: 1.5.0.dev0+f614064
netCDF4: 1.5.3
pydap: None
h5netcdf: None
h5py: 2.10.0
Nio: None
zarr: 2.4.0
cftime: 1.0.4.2
nc_time_axis: None
PseudoNetCDF: None
rasterio: 1.1.3
cfgrib: None
iris: None
bottleneck: None
dask: 2.11.0+13.gfcc500c2
distributed: 2.11.0+7.g0d7a31ad
matplotlib: 3.2.0rc3
cartopy: 0.17.0
seaborn: None
numbagg: None
setuptools: 45.2.0.post20200209
pip: 20.0.2
conda: None
pytest: 5.3.5
IPython: 7.12.0
sphinx: 2.4.3
","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/3813/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,issue
1419602897,I_kwDOAMm_X85UnWvR,7197,Unstable pandas causes CF datetime64 issues,1828519,closed,0,,,4,2022-10-23T02:30:39Z,2022-10-26T16:00:35Z,2022-10-26T16:00:35Z,CONTRIBUTOR,,,,"### What happened?
The Satpy project has a CI environment that installs numpy, pandas, and xarray (and a couple other packages) from their unstable sources (nightly builds, github source, etc). In the last week or two this environment has started failing with various datetime64 issues. It all seems to be caused by some recent change in pandas, but I can't place exactly what the problem is nor the commit/PR that started it. It seems there are a couple datetime related PRs.
### What did you expect to happen?
Datetime or datetime64 objects should be allowed to be in whatever units they need to be in (days or minutes or nanoseconds. It seems parts of xarray (or pandas) assume `datetime64[ns]` but a change in pandas is no longer doing this conversion automatically (from `datetime64[X]` to `datetime64[ns]`.
### Minimal Complete Verifiable Example
You should be able to take any environment with modern xarray and install dev pandas with:
```
python -m pip install --index-url https://pypi.anaconda.org/scipy-wheels-nightly/simple/ --trusted-host pypi.anaconda.org --no-deps --pre --upgrade pandas
```
Then run this snippet:
```Python
import xarray as xr
import numpy as np
from xarray.coding.times import CFDatetimeCoder
a = xr.DataArray(np.arange(1.0), dims=(""time"",), coords={""time"": [np.datetime64('2018-05-30T10:05:00')]})
CFDatetimeCoder().encode(a.coords[""time""])
```
I haven't been able to generate a higher-level MVCE yet, but I'm hoping this little snippet will make the issue obvious to someone familiar with xarray internals.
### MVCE confirmation
- [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
- [X] Complete example — the example is self-contained, including all data and the text of any traceback.
- [X] Verifiable example — the example copy & pastes into an IPython prompt or [Binder notebook](https://mybinder.org/v2/gh/pydata/xarray/main?urlpath=lab/tree/doc/examples/blank_template.ipynb), returning the result.
- [X] New issue — a search of GitHub Issues suggests this is not a duplicate.
### Relevant log output
```
----> 1 CFDatetimeCoder().encode(a.coords[""time""])
File ~/miniconda3/envs/satpy_py39_unstable/lib/python3.9/site-packages/xarray/coding/times.py:676, in CFDatetimeCoder.encode(self, variable, name)
672 dims, data, attrs, encoding = unpack_for_encoding(variable)
673 if np.issubdtype(data.dtype, np.datetime64) or contains_cftime_datetimes(
674 variable
675 ):
--> 676 (data, units, calendar) = encode_cf_datetime(
677 data, encoding.pop(""units"", None), encoding.pop(""calendar"", None)
678 )
679 safe_setitem(attrs, ""units"", units, name=name)
680 safe_setitem(attrs, ""calendar"", calendar, name=name)
File ~/miniconda3/envs/satpy_py39_unstable/lib/python3.9/site-packages/xarray/coding/times.py:612, in encode_cf_datetime(dates, units, calendar)
609 dates = np.asarray(dates)
611 if units is None:
--> 612 units = infer_datetime_units(dates)
613 else:
614 units = _cleanup_netcdf_time_units(units)
File ~/miniconda3/envs/satpy_py39_unstable/lib/python3.9/site-packages/xarray/coding/times.py:394, in infer_datetime_units(dates)
392 print(""Formatting datetime object"")
393 reference_date = dates[0] if len(dates) > 0 else ""1970-01-01""
--> 394 reference_date = format_cftime_datetime(reference_date)
395 unique_timedeltas = np.unique(np.diff(dates))
396 units = _infer_time_units_from_diff(unique_timedeltas)
File ~/miniconda3/envs/satpy_py39_unstable/lib/python3.9/site-packages/xarray/coding/times.py:405, in format_cftime_datetime(date)
400 def format_cftime_datetime(date):
401 """"""Converts a cftime.datetime object to a string with the format:
402 YYYY-MM-DD HH:MM:SS.UUUUUU
403 """"""
404 return ""{:04d}-{:02d}-{:02d} {:02d}:{:02d}:{:02d}.{:06d}"".format(
--> 405 date.year,
406 date.month,
407 date.day,
408 date.hour,
409 date.minute,
410 date.second,
411 date.microsecond,
412 )
AttributeError: 'numpy.datetime64' object has no attribute 'year'
```
### Anything else we need to know?
_No response_
### Environment
```
INSTALLED VERSIONS
------------------
commit: None
python: 3.9.12 | packaged by conda-forge | (main, Mar 24 2022, 23:25:59)
[GCC 10.3.0]
python-bits: 64
OS: Linux
OS-release: 5.19.0-76051900-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: ('en_US', 'UTF-8')
libhdf5: 1.12.2
libnetcdf: 4.8.1
xarray: 2022.10.0
pandas: 2.0.0.dev0+422.g6c46013c54
numpy: 1.23.4
scipy: 1.10.0.dev0+1848.f114d8b
netCDF4: 1.6.0
pydap: None
h5netcdf: 1.0.0
h5py: 3.7.0
Nio: None
zarr: 2.13.0a3.dev5
cftime: 1.6.2
nc_time_axis: None
PseudoNetCDF: None
rasterio: 1.4dev
cfgrib: None
iris: None
bottleneck: 1.3.5
dask: 2022.10.0+6.gc8dc3955
distributed: None
matplotlib: 3.7.0.dev473+gc450aa7baf
cartopy: 0.20.3
seaborn: None
numbagg: None
fsspec: 2022.5.0
cupy: None
pint: None
sparse: None
flox: None
numpy_groupies: None
setuptools: 65.3.0
pip: 22.2.2
conda: None
pytest: 7.1.1
IPython: 8.2.0
sphinx: 5.0.0
```
","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/7197/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed,13221727,issue
341331807,MDU6SXNzdWUzNDEzMzE4MDc=,2288,Add CRS/projection information to xarray objects,1828519,open,0,,,45,2018-07-15T16:02:55Z,2022-10-14T20:27:26Z,,CONTRIBUTOR,,,,"#### Problem description
This issue is to start the discussion for a feature that would be helpful to a lot of people. It may not necessarily be best to put it in xarray, but let's figure that out. I'll try to describe things below to the best of my knowledge. I'm typically thinking of raster/image data when it comes to this stuff, but it could probably be used for GIS-like point data.
Geographic data can be projected (uniform grid) or unprojected (nonuniform). Unprojected data typically has longitude and latitude values specified per-pixel. I don't think I've ever seen non-uniform data in a projected space. Projected data can be specified by a CRS (PROJ.4), a number of pixels (shape), and extents/bbox in CRS units (xmin, ymin, xmax, ymax). This could also be specified in different ways like origin (X, Y) and pixel size. Seeing as xarray already computes all `coords` data it makes sense for extents and array shape to be used. With this information provided in an xarray object any library could check for these properties and know where to place the data on a map.
So the question is: Should these properties be standardized in xarray Dataset/DataArray objects and how?
#### Related libraries and developers
* pyresample (me, @mraspaud, @pnuu)
* verde and gmt-python (@leouieda)
* metpy (@dopplershift)
* [geo-xarray](https://github.com/andrewdhicks/geo-xarray/wiki) (@andrewdhicks)
* rasterio
* cartopy
I know @WeatherGod also showed interest on gitter.
#### Complications and things to consider
1. Other related coordinate systems like [ECEF](https://en.wikipedia.org/wiki/ECEF) where coordinates are specified in three dimensions (X, Y, Z). Very useful for calculations like nearest neighbor of lon/lat points or for comparisons between two projected coordinate systems.
2. Specifying what coords arrays are the CRS coordinates or geographic coordinates in general.
3. If xarray should include these properties, where is the line drawn for what functionality xarray supports? Resampling/gridding, etc?
4. How is the CRS object represented? PROJ.4 string, PROJ.4 dict, existing libraries CRS object, new CRS object, `pyproj.Proj` object?
5. Affine versus geotransforms instead of extents: https://github.com/mapbox/rasterio/blob/master/docs/topics/migrating-to-v1.rst#affineaffine-vs-gdal-style-geotransforms
6. Similar to 4, I never mentioned ""rotation"" parameters which some users may want and are specified in the affine/geotransform.
7. Dynamically generated extents/affine objects so that slicing operations don't have to be handled specially.
8. Center of pixel coordinates versus outer edge of pixel coordinates.","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/2288/reactions"", ""total_count"": 14, ""+1"": 14, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,issue
1392878100,I_kwDOAMm_X85TBaIU,7111,New deep copy behavior in 2022.9.0 causes maximum recursion error,1828519,closed,0,,,23,2022-09-30T19:11:38Z,2022-10-06T22:04:02Z,2022-10-06T22:04:02Z,CONTRIBUTOR,,,,"### What happened?
I have a case where a Dataset to be written to a NetCDF file has ""ancillary_variables"" that have a circular dependence. For example, variable A has `.attrs[""ancillary_variables""]` that contains variable B, and B has `.attrs[""ancillary_variables""]` that contains A.
### What did you expect to happen?
Circular dependencies are detected and avoided. No maximum recursion error.
### Minimal Complete Verifiable Example
```Python
In [1]: import xarray as xr
In [2]: a = xr.DataArray(1.0, attrs={})
In [3]: b = xr.DataArray(2.0, attrs={})
In [4]: a.attrs[""other""] = b
In [5]: b.attrs[""other""] = a
In [6]: a_copy = a.copy(deep=True)
---------------------------------------------------------------------------
RecursionError Traceback (most recent call last)
Cell In [6], line 1
----> 1 a_copy = a.copy(deep=True)
File ~/miniconda3/envs/satpy_py310/lib/python3.10/site-packages/xarray/core/dataarray.py:1172, in DataArray.copy(self, deep, data)
1104 def copy(self: T_DataArray, deep: bool = True, data: Any = None) -> T_DataArray:
1105 """"""Returns a copy of this array.
1106
1107 If `deep=True`, a deep copy is made of the data array.
(...)
1170 pandas.DataFrame.copy
1171 """"""
-> 1172 variable = self.variable.copy(deep=deep, data=data)
1173 indexes, index_vars = self.xindexes.copy_indexes(deep=deep)
1175 coords = {}
File ~/miniconda3/envs/satpy_py310/lib/python3.10/site-packages/xarray/core/variable.py:996, in Variable.copy(self, deep, data)
989 if self.shape != ndata.shape:
990 raise ValueError(
991 ""Data shape {} must match shape of object {}"".format(
992 ndata.shape, self.shape
993 )
994 )
--> 996 attrs = copy.deepcopy(self._attrs) if deep else copy.copy(self._attrs)
997 encoding = copy.deepcopy(self._encoding) if deep else copy.copy(self._encoding)
999 # note: dims is already an immutable tuple
File ~/miniconda3/envs/satpy_py310/lib/python3.10/copy.py:146, in deepcopy(x, memo, _nil)
144 copier = _deepcopy_dispatch.get(cls)
145 if copier is not None:
--> 146 y = copier(x, memo)
147 else:
148 if issubclass(cls, type):
File ~/miniconda3/envs/satpy_py310/lib/python3.10/copy.py:231, in _deepcopy_dict(x, memo, deepcopy)
229 memo[id(x)] = y
230 for key, value in x.items():
--> 231 y[deepcopy(key, memo)] = deepcopy(value, memo)
232 return y
File ~/miniconda3/envs/satpy_py310/lib/python3.10/copy.py:153, in deepcopy(x, memo, _nil)
151 copier = getattr(x, ""__deepcopy__"", None)
152 if copier is not None:
--> 153 y = copier(memo)
154 else:
155 reductor = dispatch_table.get(cls)
File ~/miniconda3/envs/satpy_py310/lib/python3.10/site-packages/xarray/core/dataarray.py:1190, in DataArray.__deepcopy__(self, memo)
1187 def __deepcopy__(self: T_DataArray, memo=None) -> T_DataArray:
1188 # memo does nothing but is required for compatibility with
1189 # copy.deepcopy
-> 1190 return self.copy(deep=True)
File ~/miniconda3/envs/satpy_py310/lib/python3.10/site-packages/xarray/core/dataarray.py:1172, in DataArray.copy(self, deep, data)
1104 def copy(self: T_DataArray, deep: bool = True, data: Any = None) -> T_DataArray:
1105 """"""Returns a copy of this array.
1106
1107 If `deep=True`, a deep copy is made of the data array.
(...)
1170 pandas.DataFrame.copy
1171 """"""
-> 1172 variable = self.variable.copy(deep=deep, data=data)
1173 indexes, index_vars = self.xindexes.copy_indexes(deep=deep)
1175 coords = {}
File ~/miniconda3/envs/satpy_py310/lib/python3.10/site-packages/xarray/core/variable.py:996, in Variable.copy(self, deep, data)
989 if self.shape != ndata.shape:
990 raise ValueError(
991 ""Data shape {} must match shape of object {}"".format(
992 ndata.shape, self.shape
993 )
994 )
--> 996 attrs = copy.deepcopy(self._attrs) if deep else copy.copy(self._attrs)
997 encoding = copy.deepcopy(self._encoding) if deep else copy.copy(self._encoding)
999 # note: dims is already an immutable tuple
File ~/miniconda3/envs/satpy_py310/lib/python3.10/copy.py:146, in deepcopy(x, memo, _nil)
144 copier = _deepcopy_dispatch.get(cls)
145 if copier is not None:
--> 146 y = copier(x, memo)
147 else:
148 if issubclass(cls, type):
File ~/miniconda3/envs/satpy_py310/lib/python3.10/copy.py:231, in _deepcopy_dict(x, memo, deepcopy)
229 memo[id(x)] = y
230 for key, value in x.items():
--> 231 y[deepcopy(key, memo)] = deepcopy(value, memo)
232 return y
File ~/miniconda3/envs/satpy_py310/lib/python3.10/copy.py:153, in deepcopy(x, memo, _nil)
151 copier = getattr(x, ""__deepcopy__"", None)
152 if copier is not None:
--> 153 y = copier(memo)
154 else:
155 reductor = dispatch_table.get(cls)
File ~/miniconda3/envs/satpy_py310/lib/python3.10/site-packages/xarray/core/dataarray.py:1190, in DataArray.__deepcopy__(self, memo)
1187 def __deepcopy__(self: T_DataArray, memo=None) -> T_DataArray:
1188 # memo does nothing but is required for compatibility with
1189 # copy.deepcopy
-> 1190 return self.copy(deep=True)
[... skipping similar frames: DataArray.copy at line 1172 (495 times), DataArray.__deepcopy__ at line 1190 (494 times), _deepcopy_dict at line 231 (494 times), Variable.copy at line 996 (494 times), deepcopy at line 146 (494 times), deepcopy at line 153 (494 times)]
File ~/miniconda3/envs/satpy_py310/lib/python3.10/site-packages/xarray/core/variable.py:996, in Variable.copy(self, deep, data)
989 if self.shape != ndata.shape:
990 raise ValueError(
991 ""Data shape {} must match shape of object {}"".format(
992 ndata.shape, self.shape
993 )
994 )
--> 996 attrs = copy.deepcopy(self._attrs) if deep else copy.copy(self._attrs)
997 encoding = copy.deepcopy(self._encoding) if deep else copy.copy(self._encoding)
999 # note: dims is already an immutable tuple
File ~/miniconda3/envs/satpy_py310/lib/python3.10/copy.py:146, in deepcopy(x, memo, _nil)
144 copier = _deepcopy_dispatch.get(cls)
145 if copier is not None:
--> 146 y = copier(x, memo)
147 else:
148 if issubclass(cls, type):
File ~/miniconda3/envs/satpy_py310/lib/python3.10/copy.py:231, in _deepcopy_dict(x, memo, deepcopy)
229 memo[id(x)] = y
230 for key, value in x.items():
--> 231 y[deepcopy(key, memo)] = deepcopy(value, memo)
232 return y
File ~/miniconda3/envs/satpy_py310/lib/python3.10/copy.py:153, in deepcopy(x, memo, _nil)
151 copier = getattr(x, ""__deepcopy__"", None)
152 if copier is not None:
--> 153 y = copier(memo)
154 else:
155 reductor = dispatch_table.get(cls)
File ~/miniconda3/envs/satpy_py310/lib/python3.10/site-packages/xarray/core/dataarray.py:1190, in DataArray.__deepcopy__(self, memo)
1187 def __deepcopy__(self: T_DataArray, memo=None) -> T_DataArray:
1188 # memo does nothing but is required for compatibility with
1189 # copy.deepcopy
-> 1190 return self.copy(deep=True)
File ~/miniconda3/envs/satpy_py310/lib/python3.10/site-packages/xarray/core/dataarray.py:1172, in DataArray.copy(self, deep, data)
1104 def copy(self: T_DataArray, deep: bool = True, data: Any = None) -> T_DataArray:
1105 """"""Returns a copy of this array.
1106
1107 If `deep=True`, a deep copy is made of the data array.
(...)
1170 pandas.DataFrame.copy
1171 """"""
-> 1172 variable = self.variable.copy(deep=deep, data=data)
1173 indexes, index_vars = self.xindexes.copy_indexes(deep=deep)
1175 coords = {}
File ~/miniconda3/envs/satpy_py310/lib/python3.10/site-packages/xarray/core/variable.py:985, in Variable.copy(self, deep, data)
982 ndata = indexing.MemoryCachedArray(ndata.array)
984 if deep:
--> 985 ndata = copy.deepcopy(ndata)
987 else:
988 ndata = as_compatible_data(data)
File ~/miniconda3/envs/satpy_py310/lib/python3.10/copy.py:137, in deepcopy(x, memo, _nil)
134 if memo is None:
135 memo = {}
--> 137 d = id(x)
138 y = memo.get(d, _nil)
139 if y is not _nil:
RecursionError: maximum recursion depth exceeded while calling a Python object
```
### MVCE confirmation
- [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
- [X] Complete example — the example is self-contained, including all data and the text of any traceback.
- [X] Verifiable example — the example copy & pastes into an IPython prompt or [Binder notebook](https://mybinder.org/v2/gh/pydata/xarray/main?urlpath=lab/tree/doc/examples/blank_template.ipynb), returning the result.
- [X] New issue — a search of GitHub Issues suggests this is not a duplicate.
### Relevant log output
_No response_
### Anything else we need to know?
I have at least one other issue related to the new xarray release but I'm still tracking it down. I think it is also related to the deep copy behavior change which was merged a day before the release so our CI didn't have time to test the ""unstable"" version of xarray.
### Environment
```
INSTALLED VERSIONS
------------------
commit: None
python: 3.10.6 | packaged by conda-forge | (main, Aug 22 2022, 20:35:26) [GCC 10.4.0]
python-bits: 64
OS: Linux
OS-release: 5.19.0-76051900-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: ('en_US', 'UTF-8')
libhdf5: 1.12.2
libnetcdf: 4.8.1
xarray: 2022.9.0
pandas: 1.5.0
numpy: 1.23.3
scipy: 1.9.1
netCDF4: 1.6.1
pydap: None
h5netcdf: 1.0.2
h5py: 3.7.0
Nio: None
zarr: 2.13.2
cftime: 1.6.2
nc_time_axis: None
PseudoNetCDF: None
rasterio: 1.3.2
cfgrib: None
iris: None
bottleneck: 1.3.5
dask: 2022.9.1
distributed: 2022.9.1
matplotlib: 3.6.0
cartopy: 0.21.0
seaborn: None
numbagg: None
fsspec: 2022.8.2
cupy: None
pint: None
sparse: None
flox: None
numpy_groupies: None
setuptools: 65.4.0
pip: 22.2.2
conda: None
pytest: 7.1.3
IPython: 8.5.0
sphinx: 5.2.3
```
","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/7111/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed,13221727,issue
812692450,MDU6SXNzdWU4MTI2OTI0NTA=,4934,ImplicitToExplicitIndexingAdapter being returned with dask unstable version,1828519,closed,0,,,4,2021-02-20T19:41:12Z,2021-03-09T22:51:13Z,2021-03-09T22:51:13Z,CONTRIBUTOR,,,,"**What happened**:
I have a couple CI environments that use unstable versions of xarray and dask among other libraries. A couple tests have been failing in different but similar ways. I was able to get one of them down to the below example. This happens with the dask master branch on either the xarray latest release or xarray master. Here are the commits I was using when doing unstable both.
Dask: `ad01acc14088d03127dfec4f881cce959f3ac4d9`
Xarray: `eb7e112d45a9edebd8e5fb4f873e3e6adb18824a`
I don't see this with dask's latest release and this is likely caused by a change in dask, but after seeing PRs like #4884 I'm wondering if this is something similar and requires a change in xarray.
The bottom line is that doing various things like `my_data_arr.data.compute()` or `dask.compute(my_data_arr.data)` produces:
```
ImplicitToExplicitIndexingAdapter(array=CopyOnWriteArray(array=LazilyOuterIndexedArray(array=, key=BasicIndexer((slice(0, 2, 1), slice(0, 100, 1), slice(0, 100, 1))))))
```
**What you expected to happen**:
I would expect to get a numpy array back when computing the underlying dask array.
**Minimal Complete Verifiable Example**:
```python
from PIL import Image
import xarray as xr
import numpy as np
# create a test image
Image.fromarray(np.zeros((5, 5, 3), dtype=np.uint8)).save('test.png')
r = xr.open_rasterio('test.png', chunks='auto')
print(r.data.compute())
# ImplicitToExplicitIndexingAdapter(array=CopyOnWriteArray(array=LazilyOuterIndexedArray(array=, key=BasicIndexer((slice(0, 2, 1), slice(0, 100, 1), slice(0, 100, 1))))))
```
**Anything else we need to know?**:
As mentioned above other weird things are happening when array wrappers seem to be involved but I haven't been able to make a small example of them.
**Environment**:
Output of xr.show_versions()
INSTALLED VERSIONS
------------------
commit: None
python: 3.7.6 | packaged by conda-forge | (default, Jan 7 2020, 22:33:48)
[GCC 7.3.0]
python-bits: 64
OS: Linux
OS-release: 5.8.0-7642-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8
libhdf5: 1.10.5
libnetcdf: 4.7.3
xarray: 0.16.3.dev132+geb7e112d
pandas: 1.2.2
numpy: 1.20.1
scipy: 1.6.1
netCDF4: 1.5.3
pydap: None
h5netcdf: None
h5py: 2.10.0
Nio: None
zarr: 2.6.2.dev42
cftime: 1.4.1
nc_time_axis: None
PseudoNetCDF: None
rasterio: 1.3dev
cfgrib: None
iris: None
bottleneck: 1.4.0.dev0+117.gf2bc792
dask: 2021.02.0+21.gad01acc1
distributed: 2021.02.0+7.g383ea032
matplotlib: 3.4.0rc1
cartopy: 0.17.0
seaborn: None
numbagg: None
pint: None
setuptools: 45.2.0.post20200209
pip: 20.0.2
conda: None
pytest: 5.3.5
IPython: 7.12.0
sphinx: 2.4.3
```
DEBUG:matplotlib:$HOME=/Users/davidh
DEBUG:matplotlib:matplotlib data path /Users/davidh/anaconda/envs/polar2grid_py36/lib/python3.6/site-packages/matplotlib/mpl-data
DEBUG:matplotlib:loaded rc file /Users/davidh/anaconda/envs/polar2grid_py36/lib/python3.6/site-packages/matplotlib/mpl-data/matplotlibrc
DEBUG:matplotlib:matplotlib version 2.2.0
DEBUG:matplotlib:interactive is False
DEBUG:matplotlib:platform is darwin
DEBUG:matplotlib:loaded modules: ['builtins', 'sys', '_frozen_importlib', '_imp', '_warnings', '_thread', '_weakref', '_frozen_importlib_external', '_io', 'marshal', 'posix', 'zipimport', 'encodings', 'codecs', '_codecs', 'encodings.aliases', 'encodings.utf_8', '_signal', '__main__', 'encodings.latin_1', 'io', 'abc', '_weakrefset', 'site', 'os', 'errno', 'stat', '_stat', 'posixpath', 'genericpath', 'os.path', '_collections_abc', '_sitebuiltins', 'sysconfig', '_sysconfigdata_m_darwin_darwin', '_osx_support', 're', 'enum', 'types', 'functools', '_functools', 'collections', 'operator', '_operator', 'keyword', 'heapq', '_heapq', 'itertools', 'reprlib', '_collections', 'weakref', 'collections.abc', 'sre_compile', '_sre', 'sre_parse', 'sre_constants', '_locale', 'copyreg', '_bootlocale', 'importlib', 'importlib._bootstrap', 'importlib._bootstrap_external', 'warnings', 'importlib.util', 'importlib.abc', 'importlib.machinery', 'contextlib', 'mpl_toolkits', 'sphinxcontrib', 'encodings.cp437', 'IPython', 'IPython.core', 'IPython.core.getipython', 'IPython.core.release', 'IPython.core.application', 'atexit', 'copy', 'glob', 'fnmatch', 'logging', 'time', 'traceback', 'linecache', 'tokenize', 'token', 'string', '_string', 'threading', 'shutil', 'zlib', 'bz2', '_compression', '_bz2', 'lzma', '_lzma', 'pwd', 'grp', 'traitlets', 'traitlets.traitlets', 'inspect', 'ast', '_ast', 'dis', 'opcode', '_opcode', 'six', '__future__', 'struct', '_struct', 'traitlets.utils', 'traitlets.utils.getargspec', 'traitlets.utils.importstring', 'ipython_genutils', 'ipython_genutils._version', 'ipython_genutils.py3compat', 'ipython_genutils.encoding', 'locale', 'platform', 'subprocess', 'signal', '_posixsubprocess', 'select', 'selectors', 'math', 'traitlets.utils.sentinel', 'traitlets.utils.bunch', 'traitlets._version', 'traitlets.config', 'traitlets.config.application', 'json', 'json.decoder', 'json.scanner', '_json', 'json.encoder', 'decorator', 'traitlets.config.configurable', 'traitlets.config.loader', 'argparse', 'textwrap', 'gettext', 'ipython_genutils.path', 'random', 'hashlib', '_hashlib', '_blake2', '_sha3', 'bisect', '_bisect', '_random', 'ipython_genutils.text', 'ipython_genutils.importstring', 'IPython.core.crashhandler', 'pprint', 'IPython.core.ultratb', 'pydoc', 'pkgutil', 'urllib', 'urllib.parse', 'IPython.core.debugger', 'bdb', 'IPython.utils', 'IPython.utils.PyColorize', 'IPython.utils.coloransi', 'IPython.utils.ipstruct', 'IPython.utils.colorable', 'pygments', 'pygments.util', 'IPython.utils.py3compat', 'IPython.utils.encoding', 'IPython.core.excolors', 'IPython.testing', 'IPython.testing.skipdoctest', 'pdb', 'cmd', 'code', 'codeop', 'IPython.core.display_trap', 'IPython.utils.openpy', 'IPython.utils.path', 'IPython.utils.process', 'IPython.utils._process_posix', 'pexpect', 'pexpect.exceptions', 'pexpect.utils', 'pexpect.expect', 'pexpect.pty_spawn', 'pty', 'tty', 'termios', 'ptyprocess', 'ptyprocess.ptyprocess', 'fcntl', 'resource', 'ptyprocess.util', 'pexpect.spawnbase', 'pexpect.run', 'IPython.utils._process_common', 'shlex', 'IPython.utils.decorators', 'IPython.utils.data', 'IPython.utils.terminal', 'IPython.utils.sysinfo', 'IPython.utils._sysinfo', 'IPython.core.profiledir', 'IPython.paths', 'tempfile', 'IPython.utils.importstring', 'IPython.terminal', 'IPython.terminal.embed', 'IPython.core.compilerop', 'IPython.core.magic_arguments', 'IPython.core.error', 'IPython.utils.text', 'pathlib', 'ntpath', 'IPython.core.magic', 'getopt', 'IPython.core.oinspect', 'IPython.core.page', 'IPython.core.display', 'base64', 'binascii', 'mimetypes', 'IPython.lib', 'IPython.lib.security', 'getpass', 'IPython.lib.pretty', 'datetime', '_datetime', 'IPython.utils.dir2', 'IPython.utils.wildcard', 'pygments.lexers', 'pygments.lexers._mapping', 'pygments.modeline', 'pygments.plugin', 'pygments.lexers.python', 'pygments.lexer', 'pygments.filter', 'pygments.filters', 'pygments.token', 'pygments.regexopt', 'pygments.unistring', 'pygments.formatters', 'pygments.formatters._mapping', 'pygments.formatters.html', 'pygments.formatter', 'pygments.styles', 'IPython.core.inputsplitter', 'IPython.core.inputtransformer', 'IPython.core.splitinput', 'IPython.utils.tokenize2', 'IPython.core.interactiveshell', 'runpy', 'pickleshare', 'pickle', '_compat_pickle', '_pickle', 'IPython.core.prefilter', 'IPython.core.autocall', 'IPython.core.macro', 'IPython.core.alias', 'IPython.core.builtin_trap', 'IPython.core.events', 'IPython.core.displayhook', 'IPython.core.displaypub', 'IPython.core.extensions', 'IPython.core.formatters', 'IPython.utils.sentinel', 'IPython.core.history', 'sqlite3', 'sqlite3.dbapi2', '_sqlite3', 'IPython.core.logger', 'IPython.core.payload', 'IPython.core.usage', 'IPython.display', 'IPython.lib.display', 'IPython.utils.io', 'IPython.utils.capture', 'IPython.utils.strdispatch', 'IPython.core.hooks', 'IPython.utils.syspathcontext', 'IPython.utils.tempdir', 'IPython.utils.contexts', 'IPython.terminal.interactiveshell', 'prompt_toolkit', 'prompt_toolkit.interface', 'prompt_toolkit.application', 'prompt_toolkit.buffer', 'prompt_toolkit.auto_suggest', 'prompt_toolkit.filters', 'prompt_toolkit.filters.base', 'prompt_toolkit.utils', 'wcwidth', 'wcwidth.wcwidth', 'wcwidth.table_wide', 'wcwidth.table_zero', 'six.moves', 'prompt_toolkit.filters.cli', 'prompt_toolkit.enums', 'prompt_toolkit.key_binding', 'prompt_toolkit.key_binding.vi_state', 'prompt_toolkit.cache', 'prompt_toolkit.filters.types', 'prompt_toolkit.filters.utils', 'prompt_toolkit.clipboard', 'prompt_toolkit.clipboard.base', 'prompt_toolkit.selection', 'prompt_toolkit.clipboard.in_memory', 'prompt_toolkit.completion', 'prompt_toolkit.document', 'prompt_toolkit.history', 'prompt_toolkit.search_state', 'prompt_toolkit.validation', 'prompt_toolkit.buffer_mapping', 'prompt_toolkit.key_binding.bindings', 'prompt_toolkit.key_binding.bindings.basic', 'prompt_toolkit.keys', 'prompt_toolkit.layout', 'prompt_toolkit.layout.containers', 'prompt_toolkit.layout.controls', 'prompt_toolkit.mouse_events', 'prompt_toolkit.token', 'prompt_toolkit.layout.lexers', 'prompt_toolkit.layout.utils', 'prompt_toolkit.layout.processors', 'prompt_toolkit.reactive', 'prompt_toolkit.layout.screen', 'prompt_toolkit.layout.dimension', 'prompt_toolkit.layout.margins', 'prompt_toolkit.renderer', 'prompt_toolkit.layout.mouse_handlers', 'prompt_toolkit.output', 'prompt_toolkit.styles', 'prompt_toolkit.styles.base', 'prompt_toolkit.styles.defaults', 'prompt_toolkit.styles.from_dict', 'prompt_toolkit.styles.utils', 'prompt_toolkit.styles.from_pygments', 'pygments.style', 'pygments.styles.default', 'prompt_toolkit.key_binding.bindings.named_commands', 'prompt_toolkit.key_binding.bindings.completion', 'prompt_toolkit.key_binding.registry', 'prompt_toolkit.key_binding.input_processor', 'prompt_toolkit.key_binding.bindings.emacs', 'prompt_toolkit.key_binding.bindings.scroll', 'prompt_toolkit.key_binding.bindings.vi', 'prompt_toolkit.key_binding.digraphs', 'prompt_toolkit.key_binding.defaults', 'prompt_toolkit.eventloop', 'prompt_toolkit.eventloop.base', 'prompt_toolkit.eventloop.callbacks', 'prompt_toolkit.input', 'prompt_toolkit.terminal', 'prompt_toolkit.terminal.vt100_input', 'prompt_toolkit.shortcuts', 'prompt_toolkit.layout.menus', 'prompt_toolkit.layout.prompt', 'prompt_toolkit.layout.toolbars', 'prompt_toolkit.terminal.vt100_output', 'array', 'prompt_toolkit.key_binding.manager', 'IPython.terminal.debugger', 'IPython.core.completer', 'unicodedata', 'typing', 'typing.io', 'typing.re', 'IPython.core.latex_symbols', 'IPython.utils.generics', 'simplegeneric', 'jedi', 'jedi.api', 'jedi.parser', 'jedi.parser.parser', 'jedi.parser.tree', 'jedi._compatibility', 'imp', 'jedi.parser.pgen2', 'jedi.parser.pgen2.parse', 'jedi.parser.tokenize', 'jedi.parser.token', 'jedi.common', 'jedi.settings', 'jedi.parser.pgen2.pgen', 'jedi.parser.pgen2.grammar', 'jedi.parser.python', 'jedi.parser.python.parser', 'jedi.parser.python.tree', 'jedi.parser.python.diff', 'difflib', 'jedi.debug', 'jedi.parser.cache', 'gc', 'jedi.cache', 'jedi.api.classes', 'jedi.evaluate', 'jedi.evaluate.representation', 'jedi.evaluate.cache', 'jedi.evaluate.compiled', 'jedi.evaluate.helpers', 'jedi.evaluate.filters', 'jedi.evaluate.flow_analysis', 'jedi.evaluate.context', 'jedi.evaluate.compiled.fake', 'jedi.evaluate.recursion', 'jedi.evaluate.iterable', 'jedi.evaluate.analysis', 'jedi.evaluate.pep0484', 'jedi.evaluate.precedence', 'jedi.evaluate.docstrings', 'jedi.evaluate.param', 'jedi.evaluate.imports', 'jedi.evaluate.sys_path', 'jedi.evaluate.site', 'jedi.evaluate.dynamic', 'jedi.evaluate.stdlib', 'jedi.evaluate.instance', 'jedi.evaluate.finder', 'jedi.api.keywords', 'pydoc_data', 'pydoc_data.topics', 'jedi.api.interpreter', 'jedi.evaluate.compiled.mixed', 'jedi.api.usages', 'jedi.api.helpers', 'jedi.api.completion', 'IPython.terminal.ptutils', 'IPython.terminal.shortcuts', 'IPython.terminal.magics', 'IPython.lib.clipboard', 'IPython.terminal.pt_inputhooks', 'IPython.terminal.prompts', 'pkg_resources', 'zipfile', 'plistlib', 'xml', 'xml.parsers', 'xml.parsers.expat', 'pyexpat.errors', 'pyexpat.model', 'pyexpat', 'xml.parsers.expat.model', 'xml.parsers.expat.errors', 'email', 'email.parser', 'email.feedparser', 'email.errors', 'email._policybase', 'email.header', 'email.quoprimime', 'email.base64mime', 'email.charset', 'email.encoders', 'quopri', 'email.utils', 'socket', '_socket', 'email._parseaddr', 'calendar', 'pkg_resources.extern', 'pkg_resources._vendor', 'pkg_resources.extern.six', 'pkg_resources._vendor.six', 'pkg_resources.extern.six.moves', 'pkg_resources._vendor.six.moves', 'pkg_resources.py31compat', 'pkg_resources.extern.appdirs', 'pkg_resources._vendor.packaging.__about__', 'pkg_resources.extern.packaging', 'pkg_resources.extern.packaging.version', 'pkg_resources.extern.packaging._structures', 'pkg_resources.extern.packaging.specifiers', 'pkg_resources.extern.packaging._compat', 'pkg_resources.extern.packaging.requirements', 'pkg_resources.extern.pyparsing', 'pkg_resources.extern.six.moves.urllib', 'pkg_resources.extern.packaging.markers', 'IPython.terminal.ipapp', 'IPython.core.magics', 'IPython.core.magics.auto', 'IPython.core.magics.basic', 'IPython.core.magics.code', 'IPython.core.magics.config', 'IPython.core.magics.display', 'IPython.core.magics.execution', 'timeit', 'cProfile', '_lsprof', 'profile', 'optparse', 'pstats', 'IPython.utils.module_paths', 'IPython.utils.timing', 'IPython.core.magics.extension', 'IPython.core.magics.history', 'IPython.core.magics.logging', 'IPython.core.magics.namespace', 'IPython.core.magics.osm', 'IPython.core.magics.pylab', 'IPython.core.pylabtools', 'IPython.core.magics.script', 'IPython.lib.backgroundjobs', 'IPython.core.shellapp', 'IPython.extensions', 'IPython.extensions.storemagic', 'IPython.utils.frame', 'IPython.core.completerlib', 'pygments.lexers.shell', 'pygments.lexers.html', 'pygments.lexers.javascript', 'pygments.lexers.jvm', 'pygments.lexers.css', 'pygments.lexers.ruby', 'pygments.lexers.perl', 'pygments.lexers.markup', 'prompt_toolkit.eventloop.posix', 'prompt_toolkit.eventloop.inputhook', 'prompt_toolkit.eventloop.select', 'prompt_toolkit.eventloop.posix_utils', 'prompt_toolkit.eventloop.utils', 'storemagic', 'dask', 'dask.core', 'dask.utils_test', 'dask.context', 'dask.local', 'dask.compatibility', 'queue', 'gzip', 'urllib.request', 'http', 'http.client', 'email.message', 'uu', 'email._encoded_words', 'email.iterators', 'ssl', 'ipaddress', '_ssl', 'urllib.error', 'urllib.response', '_scproxy', 'dask.order', 'dask.callbacks', 'dask.optimization', 'dask.delayed', 'uuid', 'ctypes', '_ctypes', 'ctypes._endian', 'ctypes.util', 'ctypes.macholib', 'ctypes.macholib.dyld', 'ctypes.macholib.framework', 'ctypes.macholib.dylib', 'toolz', 'toolz.itertoolz', 'toolz.compatibility', 'toolz.utils', 'toolz.functoolz', 'toolz._signatures', 'toolz.dicttoolz', 'toolz.recipes', 'toolz.sandbox', 'toolz.sandbox.core', 'toolz.sandbox.parallel', 'dask.threaded', 'multiprocessing', 'multiprocessing.context', 'multiprocessing.process', 'multiprocessing.reduction', '__mp_main__', 'multiprocessing.pool', 'multiprocessing.util', 'dask.base', 'dask.hashing', 'dask.utils', 'numbers', 'dask.optimize', 'dask.sharedict', 'cloudpickle', 'cloudpickle.cloudpickle', 'encodings.raw_unicode_escape', 'dask._version', 'dask.array', 'dask.array.core', 'toolz.curried', 'toolz.curried.operator', 'toolz.curried.exceptions', 'numpy', 'numpy._globals', 'numpy.__config__', 'numpy.version', 'numpy._import_tools', 'numpy.add_newdocs', 'numpy.lib', 'numpy.lib.info', 'numpy.lib.type_check', 'numpy.core', 'numpy.core.info', 'numpy.core.multiarray', 'numpy.core.umath', 'numpy.core._internal', 'numpy.compat', 'numpy.compat._inspect', 'numpy.compat.py3k', 'numpy.core.numerictypes', 'numpy.core.numeric', 'numpy.core.arrayprint', 'numpy.core.fromnumeric', 'numpy.core._methods', 'numpy.core.defchararray', 'numpy.core.records', 'numpy.core.memmap', 'numpy.core.function_base', 'numpy.core.machar', 'numpy.core.getlimits', 'numpy.core.shape_base', 'numpy.core.einsumfunc', 'numpy.testing', 'unittest', 'unittest.result', 'unittest.util', 'unittest.case', 'unittest.suite', 'unittest.loader', 'unittest.main', 'unittest.runner', 'unittest.signals', 'numpy.testing.decorators', 'numpy.testing.utils', 'numpy.lib.utils', 'numpy.testing.nosetester', 'numpy.lib.ufunclike', 'numpy.lib.index_tricks', 'numpy.lib.function_base', 'numpy.lib.twodim_base', 'numpy.matrixlib', 'numpy.matrixlib.defmatrix', 'numpy.lib.stride_tricks', 'numpy.lib.mixins', 'numpy.lib.nanfunctions', 'numpy.lib.shape_base', 'numpy.lib.scimath', 'numpy.lib.polynomial', 'numpy.linalg', 'numpy.linalg.info', 'numpy.linalg.linalg', 'numpy.linalg.lapack_lite', 'numpy.linalg._umath_linalg', 'numpy.lib.arraysetops', 'numpy.lib.npyio', 'numpy.lib.format', 'numpy.lib._datasource', 'numpy.lib._iotools', 'numpy.lib.financial', 'numpy.lib.arrayterator', 'numpy.lib.arraypad', 'numpy.lib._version', 'numpy._distributor_init', 'numpy.fft', 'numpy.fft.info', 'numpy.fft.fftpack', 'numpy.fft.fftpack_lite', 'numpy.fft.helper', 'numpy.polynomial', 'numpy.polynomial.polynomial', 'numpy.polynomial.polyutils', 'numpy.polynomial._polybase', 'numpy.polynomial.chebyshev', 'numpy.polynomial.legendre', 'numpy.polynomial.hermite', 'numpy.polynomial.hermite_e', 'numpy.polynomial.laguerre', 'numpy.random', 'numpy.random.info', 'cython_runtime', 'mtrand', 'numpy.random.mtrand', 'numpy.ctypeslib', 'numpy.ma', 'numpy.ma.core', 'numpy.ma.extras', 'dask.array.chunk', 'dask.array.numpy_compat', 'distutils', 'distutils.version', 'dask.array.slicing', 'dask.array.optimization', 'dask.array.routines', 'dask.array.creation', 'dask.array.wrap', 'dask.array.reshape', 'dask.array.ufunc', 'dask.array.reductions', 'dask.array.percentile', 'dask.array.ma', 'dask.array.random', 'dask.array.linalg', 'dask.array.ghost', 'dask.array.learn', 'dask.array.fft', 'scipy', 'scipy._distributor_init', 'scipy.__config__', 'scipy.version', 'scipy._lib', 'scipy._lib._testutils', 'scipy._lib._version', 'scipy._lib.six', 'scipy._lib._ccallback', 'scipy._lib._ccallback_c', 'scipy.fftpack', 'scipy.fftpack.basic', 'scipy.fftpack._fftpack', 'scipy.fftpack.pseudo_diffs', 'scipy.fftpack.convolve', 'scipy.fftpack.helper', 'numpy.dual', 'scipy.fftpack.realtransforms', 'dask.array.rechunk', 'xarray', 'xarray.core', 'xarray.core.alignment', 'xarray.core.utils', 'pandas', 'pytz', 'pytz.exceptions', 'pytz.lazy', 'pytz.tzinfo', 'pytz.tzfile', 'dateutil', 'dateutil._version', 'pandas.compat', 'pandas.compat.chainmap', 'dateutil.parser', 'dateutil.relativedelta', 'dateutil._common', 'dateutil.tz', 'dateutil.tz.tz', 'dateutil.tz._common', 'pandas.compat.numpy', 'pandas._libs', '_cython_0_27_2', 'pandas._libs.tslib', 'pandas._libs.tslibs', 'pandas._libs.tslibs.timedeltas', 'pandas._libs.tslibs.timezones', 'pandas._libs.tslibs.parsing', 'pandas._libs.tslibs.fields', 'pandas._libs.hashtable', 'pandas._libs.lib', 'pandas._libs.interval', 'decimal', '_decimal', 'pandas.core', 'pandas.core.config_init', 'pandas.core.config', 'pandas.io', 'pandas.io.formats', 'pandas.io.formats.printing', 'pandas.core.dtypes', 'pandas.core.dtypes.inference', 'pandas.io.formats.console', 'pandas.io.formats.terminal', 'pandas.core.api', 'pandas.core.algorithms', 'pandas.core.dtypes.cast', 'pandas.core.dtypes.common', 'pandas._libs.algos', 'pandas.core.dtypes.dtypes', 'pandas.core.dtypes.generic', 'pandas.core.dtypes.missing', 'pandas.core.common', 'pandas.api', 'pandas.api.types', 'pandas.core.dtypes.api', 'pandas.core.dtypes.concat', 'pandas.errors', 'pandas.core.categorical', 'pandas.core.accessor', 'pandas.core.base', 'pandas.util', 'pandas.util._decorators', 'pandas._libs.properties', 'pandas.core.util', 'pandas.core.util.hashing', 'pandas._libs.hashing', 'pandas.util._validators', 'pandas.core.nanops', 'bottleneck', 'bottleneck.reduce', 'bottleneck.nonreduce', 'bottleneck.nonreduce_axis', 'bottleneck.move', 'bottleneck.slow', 'bottleneck.slow.reduce', 'bottleneck.slow.nonreduce', 'bottleneck.slow.nonreduce_axis', 'bottleneck.slow.move', 'bottleneck.version', 'bottleneck.benchmark', 'bottleneck.benchmark.bench', 'bottleneck.benchmark.autotimeit', 'bottleneck.benchmark.bench_detailed', 'bottleneck.tests', 'bottleneck.tests.util', 'pandas.compat.numpy.function', 'pandas.core.missing', 'pandas.core.groupby', 'pandas.core.index', 'pandas.core.indexes', 'pandas.core.indexes.api', 'pandas.core.indexes.base', 'pandas._libs.index', 'pandas._libs.join', 'pandas.core.indexes.frozen', 'pandas.core.sorting', 'pandas.core.ops', 'pandas.core.strings', 'pandas.core.indexes.category', 'pandas.core.indexes.multi', 'pandas.core.indexes.interval', 'pandas.core.indexes.datetimes', 'pandas.core.indexes.numeric', 'pandas.tseries', 'pandas.tseries.frequencies', 'pandas.tseries.offsets', 'pandas.core.tools', 'pandas.core.tools.datetimes', 'pandas._libs.tslibs.strptime', 'dateutil.easter', 'pandas._libs.tslibs.frequencies', 'pandas.core.indexes.datetimelike', 'pandas.core.tools.timedeltas', 'pandas._libs.period', 'pandas.core.indexes.timedeltas', 'pandas.core.indexes.range', 'pandas.core.indexes.period', 'pandas.core.frame', 'pandas.core.generic', 'pandas.core.indexing', 'pandas.core.internals', 'pandas.core.sparse', 'pandas.core.sparse.array', 'pandas._libs.sparse', 'pandas.io.formats.format', 'pandas.io.common', 'csv', '_csv', 'mmap', 'pandas.io.formats.common', 'pandas.core.series', 'pandas.core.indexes.accessors', 'pandas.plotting', 'pandas.plotting._misc', 'pandas.plotting._style', 'pandas.plotting._compat', 'pandas.plotting._tools', 'pandas.plotting._core', 'pandas.core.window', 'pandas._libs.window', 'pandas.core.panel', 'pandas.core.reshape', 'pandas.core.reshape.util', 'pandas._libs.groupby', 'pandas.core.panel4d', 'pandas.core.panelnd', 'pandas.core.reshape.reshape', 'pandas.core.sparse.api', 'pandas.core.sparse.list', 'pandas.core.sparse.series', 'pandas.core.sparse.scipy_sparse', 'pandas.core.sparse.frame', 'pandas._libs.reshape', 'pandas.core.tools.numeric', 'pandas.util._depr_module', 'pandas.stats', 'pandas.stats.api', 'pandas.stats.moments', 'pandas.tseries.api', 'pandas.core.computation', 'pandas.core.computation.api', 'pandas.core.computation.eval', 'pandas.core.computation.scope', 'pandas.core.computation.engines', 'pandas.core.computation.align', 'pandas.core.computation.common', 'pandas.core.computation.ops', 'pandas.core.reshape.api', 'pandas.core.reshape.concat', 'pandas.core.reshape.merge', 'pandas.core.reshape.pivot', 'pandas.core.reshape.tile', 'pandas.tools', 'pandas.tools.plotting', 'pandas.util._print_versions', 'pandas.io.api', 'pandas.io.parsers', 'pandas.io.date_converters', 'pandas._libs.parsers', 'pandas.io.clipboards', 'pandas.io.excel', 'pandas._libs.json', 'pandas.compat.openpyxl_compat', 'pandas.io.pytables', 'pandas.core.computation.pytables', 'pandas.core.computation.expr', 'pandas.io.json', 'pandas.io.json.json', 'pandas.io.json.normalize', 'pandas.io.json.table_schema', 'pandas.io.html', 'pandas.io.sql', 'pandas.io.sas', 'pandas.io.sas.sasreader', 'pandas.io.feather_format', 'pandas.io.parquet', 'pandas.io.stata', 'pandas.io.pickle', 'pandas.compat.pickle_compat', 'pandas.io.packers', 'pandas.io.msgpack', 'pandas.io.msgpack.exceptions', 'pandas.io.msgpack._version', 'pandas.io.msgpack._packer', 'pandas.io.msgpack._unpacker', 'pandas.util._move', 'pandas.io.gbq', 'pandas.util._tester', 'pandas.testing', 'pandas.util.testing', 'pandas._libs.testing', 'pandas._version', 'xarray.core.pycompat', 'xarray.core.indexing', 'xarray.core.nputils', 'xarray.core.duck_array_ops', 'xarray.core.npcompat', 'xarray.core.dtypes', 'xarray.core.variable', 'xarray.core.common', 'xarray.core.formatting', 'xarray.core.options', 'xarray.core.ops', 'xarray.core.combine', 'xarray.core.merge', 'xarray.core.computation', 'xarray.core.extensions', 'xarray.core.dataarray', 'xarray.plot', 'xarray.plot.plot', 'xarray.plot.utils', 'xarray.plot.facetgrid', 'xarray.core.groupby', 'xarray.core.resample', 'xarray.core.rolling', 'xarray.core.dask_array_ops', 'xarray.core.accessors', 'xarray.core.coordinates', 'xarray.core.dataset', 'xarray.conventions', 'xarray.coding', 'xarray.coding.times', 'xarray.coding.variables', 'xarray.backends', 'xarray.backends.common', 'xarray.backends.memory', 'xarray.backends.netCDF4_', 'xarray.backends.netcdf3', 'xarray.backends.pydap_', 'xarray.backends.pynio_', 'xarray.backends.scipy_', 'xarray.backends.h5netcdf_', 'xarray.backends.zarr', 'xarray.backends.api', 'xarray.backends.rasterio_', 'xarray.version', 'xarray.util', 'xarray.util.print_versions', 'xarray.tutorial', 'xarray.ufuncs', 'xarray.testing', 'netCDF4', '_cython_0_27_3', 'netCDF4._netCDF4', 'netCDF4.utils', 'netcdftime', 'netcdftime._netcdftime', 'h5netcdf', 'h5netcdf.core', 'h5py', 'h5py._errors', 'h5py._conv', 'h5py.h5r', 'h5py._objects', 'h5py.defs', 'h5py.h5t', 'h5py.utils', 'h5py.h5', 'h5py.h5z', 'h5py.h5a', 'h5py.h5s', 'h5py.h5p', 'h5py.h5ac', 'h5py._proxy', 'h5py.h5d', 'h5py.h5ds', 'h5py.h5f', 'h5py.h5g', 'h5py.h5i', 'h5py.h5fd', 'h5py._hl', 'h5py._hl.filters', 'h5py._hl.base', 'h5py._hl.compat', 'h5py._hl.files', 'h5py._hl.group', 'h5py.h5o', 'h5py.h5l', 'h5py._hl.dataset', 'h5py._hl.selections', 'h5py._hl.selections2', 'h5py._hl.datatype', 'h5py.version', 'h5py._hl.attrs', 'h5py.tests', 'h5py.tests.common', 'h5py.tests.old', 'h5py.tests.old.test_attrs', 'h5py.highlevel', 'h5py.tests.old.test_attrs_data', 'h5py.tests.old.test_base', 'h5py.tests.old.test_dataset', 'h5py.tests.old.test_datatype', 'h5py.tests.old.test_dimension_scales', 'h5py.tests.old.test_file', 'h5py.tests.old.test_file_image', 'h5py.tests.old.test_group', 'h5py.tests.old.test_h5', 'h5py.tests.old.test_h5f', 'h5py.tests.old.test_h5p', 'h5py.tests.old.test_h5t', 'h5py.tests.old.test_objects', 'h5py.tests.old.test_selections', 'h5py.tests.old.test_slicing', 'h5py.tests.hl', 'h5py.tests.hl.test_dataset_getitem', 'h5py.tests.hl.test_dataset_swmr', 'h5py.tests.hl.test_dims_dimensionproxy', 'h5py.tests.hl.test_file', 'h5py.tests.hl.test_attribute_create', 'h5py.tests.hl.test_threads', 'h5py.tests.hl.test_datatype', 'h5netcdf.compat', 'h5netcdf.attrs', 'h5netcdf.dimensions', 'h5netcdf.utils', 'distributed', 'distributed.config', 'logging.config', 'logging.handlers', 'socketserver', 'distributed.compatibility', 'asyncio', 'asyncio.base_events', 'concurrent', 'concurrent.futures', 'concurrent.futures._base', 'concurrent.futures.process', 'multiprocessing.connection', '_multiprocessing', 'concurrent.futures.thread', 'asyncio.compat', 'asyncio.coroutines', 'asyncio.constants', 'asyncio.events', 'asyncio.base_futures', 'asyncio.log', 'asyncio.futures', 'asyncio.base_tasks', '_asyncio', 'asyncio.tasks', 'asyncio.locks', 'asyncio.protocols', 'asyncio.queues', 'asyncio.streams', 'asyncio.subprocess', 'asyncio.transports', 'asyncio.unix_events', 'asyncio.base_subprocess', 'asyncio.selector_events', 'asyncio.sslproto', 'html', 'html.entities', 'yaml', 'yaml.error', 'yaml.tokens', 'yaml.events', 'yaml.nodes', 'yaml.loader', 'yaml.reader', 'yaml.scanner', 'yaml.parser', 'yaml.composer', 'yaml.constructor', 'yaml.resolver', 'yaml.dumper', 'yaml.emitter', 'yaml.serializer', 'yaml.representer', 'yaml.cyaml', '_yaml', 'distributed.core', 'tornado', 'tornado.gen', 'tornado.concurrent', 'tornado.log', 'tornado.escape', 'tornado.util', 'tornado.speedups', 'curses', '_curses', 'tornado.stack_context', 'tornado.ioloop', 'tornado.platform', 'tornado.platform.auto', 'tornado.platform.posix', 'tornado.platform.common', 'tornado.platform.interface', 'tornado.platform.asyncio', 'tornado.locks', 'distributed.comm', 'distributed.comm.addressing', 'distributed.comm.registry', 'distributed.comm.core', 'distributed.metrics', 'psutil', 'psutil._common', 'psutil._compat', 'psutil._exceptions', 'psutil._psosx', 'psutil._psposix', 'psutil._psutil_osx', 'psutil._psutil_posix', 'distributed.utils', 'tblib', 'tblib.cpython', 'tblib.pickling_support', 'multiprocessing.forkserver', 'multiprocessing.semaphore_tracker', 'multiprocessing.spawn', 'distributed.comm.inproc', 'distributed.protocol', 'distributed.protocol.compression', 'distributed.protocol.core', 'msgpack', 'msgpack._version', 'msgpack.exceptions', 'msgpack._packer', 'msgpack._unpacker', 'distributed.protocol.serialize', 'distributed.protocol.pickle', 'distributed.protocol.utils', 'distributed.comm.tcp', 'tornado.netutil', 'certifi', 'certifi.core', 'encodings.idna', 'stringprep', 'tornado.iostream', 'tornado.tcpclient', 'tornado.tcpserver', 'tornado.process', 'distributed.comm.utils', 'distributed.sizeof', 'distributed.system_monitor', 'distributed.deploy', 'distributed.deploy.local', 'distributed.nanny', 'multiprocessing.queues', 'distributed.node', 'distributed.versions', 'distributed.process', 'distributed.proctitle', 'distributed.security', 'distributed.worker', 'distributed.profile', 'bokeh', 'bokeh.util', 'bokeh.util.version', 'bokeh._version', 'bokeh.util.logconfig', 'bokeh.settings', 'bokeh.util.paths', 'bokeh.util.warnings', 'bokeh.sampledata', 'six.moves.urllib', 'six.moves.urllib.request', 'bokeh.palettes', 'distributed.batched', 'distributed.diskutils', 'distributed.locket', 'distributed.preloading', 'filecmp', 'click', 'click.core', 'click.types', 'click._compat', 'click.exceptions', 'click.utils', 'click.globals', 'click.termui', 'click.formatting', 'click.parser', 'click._unicodefun', 'click.decorators', 'distributed.threadpoolexecutor', 'distributed._concurrent_futures_thread', 'distributed.utils_comm', 'distributed.utils_perf', 'distributed.scheduler', 'sortedcontainers', 'sortedcontainers.sortedlist', 'sortedcontainers.sortedset', 'sortedcontainers.sorteddict', 'distributed.publish', 'distributed.queues', 'tornado.queues', 'distributed.client', 'distributed.cfexecutor', 'distributed.recreate_exceptions', 'distributed.lock', 'distributed.stealing', 'distributed.diagnostics', 'distributed.diagnostics.graph_layout', 'distributed.diagnostics.plugin', 'distributed.diagnostics.progressbar', 'distributed.diagnostics.progress', 'distributed.variable', 'distributed.deploy.adaptive', 'distributed.deploy.ssh', 'distributed.worker_client', 'distributed._version', 'matplotlib', 'distutils.sysconfig', 'distutils.errors', 'matplotlib.cbook', 'matplotlib.cbook.deprecation', 'matplotlib.cbook._backports', 'matplotlib.compat', 'matplotlib.compat.subprocess', 'matplotlib.rcsetup', 'matplotlib.testing', 'matplotlib.fontconfig_pattern', 'pyparsing', 'matplotlib.colors', 'matplotlib._color_data', 'cycler', 'matplotlib._version']
DEBUG:shapely.geos:Trying `CDLL(/Users/davidh/anaconda/envs/polar2grid_py36/bin/../lib/libgeos_c.dylib)`
DEBUG:shapely.geos:Library path: '/Users/davidh/anaconda/envs/polar2grid_py36/bin/../lib/libgeos_c.dylib'
DEBUG:shapely.geos:DLL:
DEBUG:shapely.geos:Trying `CDLL(/usr/lib/libc.dylib)`
DEBUG:shapely.geos:Library path: '/usr/lib/libc.dylib'
DEBUG:shapely.geos:DLL:
DEBUG:pip.vcs:Registered VCS backend: git
DEBUG:pip.vcs:Registered VCS backend: hg
DEBUG:pip.vcs:Registered VCS backend: svn
DEBUG:pip.vcs:Registered VCS backend: bzr
INSTALLED VERSIONS
------------------
commit: None
python: 3.6.4.final.0
python-bits: 64
OS: Darwin
OS-release: 17.5.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8
xarray: 0.10.1
pandas: 0.21.0
numpy: 1.13.3
scipy: 1.0.0
netCDF4: 1.3.1
h5netcdf: 0.5.0
h5py: 2.7.1
Nio: None
zarr: None
bottleneck: 1.2.1
cyordereddict: None
dask: 0.17.1
distributed: 1.21.2
matplotlib: 2.2.0
cartopy: 0.16.0
seaborn: None
setuptools: 39.0.1
pip: 9.0.1
conda: None
pytest: 3.4.0
IPython: 6.1.0
sphinx: 1.6.6
```
","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/2060/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed,13221727,issue
462859457,MDU6SXNzdWU0NjI4NTk0NTc=,3068,Multidimensional dask coordinates unexpectedly computed,1828519,closed,0,,,8,2019-07-01T18:52:03Z,2019-11-05T15:41:15Z,2019-11-05T15:41:15Z,CONTRIBUTOR,,,,"#### MCVE Code Sample
```python
from dask.diagnostics import ProgressBar
import xarray as xr
import numpy as np
import dask.array as da
a = xr.DataArray(da.zeros((10, 10), chunks=2), dims=('y', 'x'), coords={'y': np.arange(10), 'x': np.arange(10), 'lons': (('y', 'x'), da.zeros((10, 10), chunks=2))})
b = xr.DataArray(da.zeros((10, 10), chunks=2), dims=('y', 'x'), coords={'y': np.arange(10), 'x': np.arange(10), 'lons': (('y', 'x'), da.zeros((10, 10), chunks=2))})
with ProgressBar():
c = a + b
```
Output:
```
[########################################] | 100% Completed | 0.1s
```
#### Problem Description
Using arrays with 2D dask array coordinates results in the coordinates being computed for any binary operations (anything combining two or more DataArrays). I use `ProgressBar` in the above example to show when coordinates are being computed.
In my own work, when I learned that 2D dask coordinates were possible, I started adding `longitude` and `latitude` coordinates. These are rather large and can take a while to load/compute so I was surprised that simple operations (ex. `a.fillna(b)`) were causing things to be computed and taking a long time.
Is this computation by design or a possible bug?
#### Expected Output
No output from the `ProgressBar`, hoping that no coordinates would be computed/loaded.
#### Output of ``xr.show_versions()``
INSTALLED VERSIONS
------------------
commit: None
python: 3.6.7 | packaged by conda-forge | (default, Feb 28 2019, 02:16:08)
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)]
python-bits: 64
OS: Darwin
OS-release: 18.6.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8
libhdf5: 1.10.4
libnetcdf: 4.6.2
xarray: 0.12.1
pandas: 0.24.2
numpy: 1.14.3
scipy: 1.3.0
netCDF4: 1.5.1.2
pydap: None
h5netcdf: 0.7.4
h5py: 2.9.0
Nio: None
zarr: 2.3.2
cftime: 1.0.3.4
nc_time_axis: None
PseudonetCDF: None
rasterio: 1.0.22
cfgrib: None
iris: None
bottleneck: 1.2.1
dask: 2.0.0
distributed: 2.0.0
matplotlib: 3.1.0
cartopy: 0.17.1.dev147+HEAD.detached.at.5e624fe
seaborn: None
setuptools: 41.0.1
pip: 19.1.1
conda: None
pytest: 4.6.3
IPython: 7.5.0
sphinx: 2.1.2
","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/3068/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed,13221727,issue
479420466,MDU6SXNzdWU0Nzk0MjA0NjY=,3205,Accessors are recreated on every access,1828519,closed,0,,,3,2019-08-11T22:27:14Z,2019-08-14T05:33:48Z,2019-08-14T05:33:48Z,CONTRIBUTOR,,,,"#### MCVE Code Sample
1. Create `test_accessor.py` in current directory with:
```python
import xarray as xr
@xr.register_dataarray_accessor('test')
class TestDataArrayAccessor(object):
def __init__(self, obj):
self._obj = obj
print(""DataArray accessor created"")
@xr.register_dataset_accessor('test')
class TestDatasetAccessor(object):
def __init__(self, obj):
self._obj = obj
print(""Dataset accessor created"")
```
2. Run the following code:
```python
import xarray as xr
import numpy as np
import test_accesor
ds = xr.Dataset({'a': xr.DataArray(np.array([1, 2, 3])), 'b': xr.DataArray(np.array([4, 5, 6]))})
ds.test
# Dataset accessor created
ds.test
#
ds['a'].test
# DataArray accessor created
ds['a'].test
# DataArray accessor created
var = ds['a']
var.test
# DataArray accessor created
var.test
#
```
#### Expected Output
Based on the xarray [accessor documentation](http://xarray.pydata.org/en/stable/internals.html#extending-xarray) I would have assumed that the accessor would stick around on the same DataArray object for the life of the data. My guess is that `Dataset.__getitem__` is recreating the `DataArray` every time from the underlying `Variable` object which means `ds['a'] is not ds['a']`?
#### Problem Description
I'm currently working on an accessor for a new package called `geoxarray` to address the issues talked about in #2288. One of the cases I'm trying to handle is a NetCDF file with CR standard `grid_mapping` variables. This means that the easiest way to set a CRS object for all variables in a Dataset is to have the Dataset accessor use the accessor of every DataArray (to cache a `_crs` property). However, with the way things are working doing something like the below won't work:
```python
ds.geo.apply_grid_mapping()
ds['Rad'].geo.crs # this is current `None` because the DataArray was recreated
```
#### Output of ``xr.show_versions()``
INSTALLED VERSIONS
------------------
commit: None
python: 3.7.3 | packaged by conda-forge | (default, Jul 1 2019, 14:38:56)
[Clang 4.0.1 (tags/RELEASE_401/final)]
python-bits: 64
OS: Darwin
OS-release: 18.6.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8
libhdf5: 1.10.5
libnetcdf: 4.6.2
xarray: 0.12.3
pandas: 0.25.0
numpy: 1.17.0
scipy: None
netCDF4: 1.5.1.2
pydap: None
h5netcdf: None
h5py: 2.9.0
Nio: None
zarr: None
cftime: 1.0.3.4
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
dask: 2.2.0
distributed: 2.2.0
matplotlib: None
cartopy: None
seaborn: None
numbagg: None
setuptools: 41.0.1
pip: 19.2.2
conda: None
pytest: None
IPython: 7.7.0
sphinx: None
","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/3205/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed,13221727,issue
446722089,MDU6SXNzdWU0NDY3MjIwODk=,2976,Confusing handling of NetCDF coordinates,1828519,closed,0,,,6,2019-05-21T16:47:47Z,2019-05-21T23:36:22Z,2019-05-21T23:36:21Z,CONTRIBUTOR,,,,"#### Code Sample, a copy-pastable example if possible
I'm currently trying to figure out why some coordinates of a netcdf file are included in the resulting Dataset object and why some are not. I've tried looking at the `open_dataset` source code and think I'm kind of stuck.
To reproduce:
Download GOES-16 ABI L1b NetCDF file from (click the link to have google redirect you to the actual file):
https://storage.cloud.google.com/gcp-public-data-goes-16/ABI-L1b-RadC/2019/140/18/OR_ABI-L1b-RadC-M6C01_G16_s20191401801336_e20191401804109_c20191401804156.nc?_ga=2.44800740.-1513329882.1547344783
```python
import xarray as xr
a = xr.open_dataset('OR_ABI-L1b-RadC-M6C01_G16_s20191401801336_e20191401804109_c20191401804156.nc')
print(a.coords)
```
Results in:
```
In [5]: a['Rad'].coords
Out[5]:
Coordinates:
t datetime64[ns] ...
* y (y) float32 0.151844 0.151788 0.151732 ... -0.151788 -0.151844
* x (x) float32 -0.151844 -0.151788 -0.151732 ... 0.151788 0.151844
y_image float32 ...
x_image float32 ...
```
Even though `ncdump -h .nc` shows:
```
variables:
short Rad(y, x) ;
...
Rad:coordinates = ""band_id band_wavelength t y x"" ;
...
```
and:
```
a.coords
```
shows:
```
Coordinates:
t datetime64[ns] ...
* y (y) float32 0.151844 0.151788 ... -0.151844
* x (x) float32 -0.151844 -0.151788 ... 0.151844
y_image float32 ...
x_image float32 ...
band_id (band) int8 ...
band_wavelength (band) float32 ...
t_star_look (num_star_looks) datetime64[ns] ...
band_wavelength_star_look (num_star_looks) float32 ...
```
#### Problem description
I would have expected the `'Rad'` variable/DataArray to include the `band_id` and `band_wavelength` coordinate variables but **not** the `x_image` and `y_image` variables since they are not listed in the NetCDF variables `coordinates` attribute.
Can someone summarize the rules on how xarray came up with these coordinates? Or is this a bug?
#### Output of ``xr.show_versions()``
```
INSTALLED VERSIONS
------------------
commit: None
python: 3.6.7 | packaged by conda-forge | (default, Feb 28 2019, 02:16:08)
[GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)]
python-bits: 64
OS: Darwin
OS-release: 18.5.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8
libhdf5: 1.10.4
libnetcdf: 4.6.2
xarray: 0.12.1
pandas: 0.24.2
numpy: 1.14.3
scipy: 1.2.1
netCDF4: 1.5.0.1
pydap: None
h5netcdf: 0.7.1
h5py: 2.9.0
Nio: None
zarr: 2.3.1
cftime: 1.0.3.4
nc_time_axis: None
PseudonetCDF: None
rasterio: 1.0.22
cfgrib: None
iris: None
bottleneck: 1.2.1
dask: 1.1.5
distributed: 1.26.1
matplotlib: 3.0.3
cartopy: 0.17.0
seaborn: None
setuptools: 41.0.0
pip: 19.0.3
conda: None
pytest: 4.4.0
IPython: 7.4.0
sphinx: 2.0.1
```
","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/2976/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed,13221727,issue
384449698,MDU6SXNzdWUzODQ0NDk2OTg=,2576,When is transpose not lazy?,1828519,closed,0,,,1,2018-11-26T18:10:36Z,2019-03-12T15:01:18Z,2019-03-12T15:01:18Z,CONTRIBUTOR,,,,"#### Code Sample, a copy-pastable example if possible
```python
import xarray as xr
import dask.array as da
from dask.diagnostics import ProgressBar
a = xr.DataArray(da.zeros((3, 5, 5), chunks=(1, 2, 2)), dims=('bands', 'y', 'x'))
with ProgressBar():
b = a.transpose('y', 'x', 'bands')
# dask does not show a progress bar due to no computation
with ProgressBar():
b = a.transpose('y', 'x', 'bands')
b.compute()
# dask computes the array (since we told it to) and we see a progress bar
```
#### Question
The [documentation for transpose](http://xarray.pydata.org/en/stable/generated/xarray.DataArray.transpose.html) says that it is not lazy. Is this only in certain situations? By not lazy does it mean that when the data is computed that the transpose task will require all data to be loaded at once (one large chunk) or does it mean that the transpose operation will immediately compute the transposed array? My test above does not seem to compute the data when `transpose` is called.
Or is the documentation just outdated?","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/2576/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed,13221727,issue