id,node_id,number,title,user,state,locked,assignee,milestone,comments,created_at,updated_at,closed_at,author_association,active_lock_reason,draft,pull_request,body,reactions,performed_via_github_app,state_reason,repo,type 1966675016,I_kwDOAMm_X851ORRI,8388,Type annotation compatibility with numpy ufuncs,1828519,closed,0,,,4,2023-10-28T17:25:11Z,2023-11-02T12:44:50Z,2023-11-02T12:44:50Z,CONTRIBUTOR,,,,"### Is your feature request related to a problem? I'd like mypy to understand that xarray DataArrays passed to numpy ufuncs have a return type of xarray DataArray. ```python import xarray as xr import numpy as np def compute_relative_azimuth(sat_azi: xr.DataArray, sun_azi: xr.DataArray) -> xr.DataArray: abs_diff = np.absolute(sun_azi - sat_azi) ssadiff = np.minimum(abs_diff, 360 - abs_diff) return ssadiff ``` ```bash $ mypy ./xarray_mypy.py xarray_mypy.py:7: error: Incompatible return value type (got ""ndarray[Any, dtype[Any]]"", expected ""DataArray"") [return-value] Found 1 error in 1 file (checked 1 source file) ``` ### Describe the solution you'd like I'm not sure if this is possible, if it is something xarray can fix, or something numpy needs to ""fix"". I'd like the above situation to ""just work"" without anything more than maybe some extra type-stub package. ### Describe alternatives you've considered Cast types or other type coercion or tell mypy to ignore the type issues for these numpy call. ### Additional context https://stackoverflow.com/questions/77369042/typing-when-passing-xarray-dataarray-objects-to-numpy-ufuncs","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/8388/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed,13221727,issue 1085992113,I_kwDOAMm_X85Auuyx,6092,DeprecationWarning regarding use of distutils Version classes,1828519,closed,0,,,10,2021-12-21T16:11:08Z,2023-09-05T06:39:39Z,2021-12-24T14:50:48Z,CONTRIBUTOR,,,," **What happened**: While working on some tests that catch and check for warnings in my library I found that xarray with new versions of Python (I think this is the trigger) causes a ton of DeprecationWarnings on import: ```python /home/davidh/miniconda3/envs/satpy_py39/lib/python3.9/site-packages/xarray/core/pycompat.py:22: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. duck_array_version = LooseVersion(duck_array_module.__version__) ``` **What you expected to happen**: No warnings. **Minimal Complete Verifiable Example**: ```python import warnings warnings.simplefilter(""always"") import xarray as xr ``` Results in: ``` /home/davidh/miniconda3/envs/satpy_py39/lib/python3.9/site-packages/xarray/core/pycompat.py:22: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. duck_array_version = LooseVersion(duck_array_module.__version__) /home/davidh/miniconda3/envs/satpy_py39/lib/python3.9/site-packages/xarray/core/pycompat.py:37: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. duck_array_version = LooseVersion(""0.0.0"") /home/davidh/miniconda3/envs/satpy_py39/lib/python3.9/site-packages/xarray/core/pycompat.py:37: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. duck_array_version = LooseVersion(""0.0.0"") /home/davidh/miniconda3/envs/satpy_py39/lib/python3.9/site-packages/setuptools/_distutils/version.py:351: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. other = LooseVersion(other) /home/davidh/miniconda3/envs/satpy_py39/lib/python3.9/site-packages/setuptools/_distutils/version.py:351: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. other = LooseVersion(other) /home/davidh/miniconda3/envs/satpy_py39/lib/python3.9/site-packages/xarray/core/npcompat.py:82: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. if LooseVersion(np.__version__) >= ""1.20.0"": /home/davidh/miniconda3/envs/satpy_py39/lib/python3.9/site-packages/setuptools/_distutils/version.py:351: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. other = LooseVersion(other) /home/davidh/miniconda3/envs/satpy_py39/lib/python3.9/site-packages/xarray/core/pdcompat.py:45: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. if LooseVersion(pd.__version__) < ""0.25.0"": /home/davidh/miniconda3/envs/satpy_py39/lib/python3.9/site-packages/setuptools/_distutils/version.py:351: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. other = LooseVersion(other) ``` **Anything else we need to know?**: **Environment**:
Output of xr.show_versions() ``` /home/davidh/miniconda3/envs/satpy_py39/lib/python3.9/site-packages/_distutils_hack/__init__.py:35: UserWarning: Setuptools is replacing distutils. warnings.warn(""Setuptools is replacing distutils."") /home/davidh/miniconda3/envs/satpy_py39/lib/python3.9/asyncio/base_events.py:681: ResourceWarning: unclosed event loop <_UnixSelectorEventLoop running=False closed=False debug=False> _warn(f""unclosed event loop {self!r}"", ResourceWarning, source=self) ResourceWarning: Enable tracemalloc to get the object allocation traceback INSTALLED VERSIONS ------------------ commit: None python: 3.9.9 | packaged by conda-forge | (main, Dec 20 2021, 02:41:03) [GCC 9.4.0] python-bits: 64 OS: Linux OS-release: 5.15.5-76051505-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: ('en_US', 'UTF-8') libhdf5: 1.12.1 libnetcdf: 4.8.1 xarray: 0.20.2 pandas: 1.3.5 numpy: 1.20.3 scipy: 1.7.3 netCDF4: 1.5.8 pydap: None h5netcdf: 0.12.0 h5py: 3.6.0 Nio: None zarr: 2.10.3 cftime: 1.5.1.1 nc_time_axis: None PseudoNetCDF: None rasterio: 1.2.10 cfgrib: None iris: None bottleneck: 1.3.2 dask: 2021.12.0 distributed: 2021.12.0 matplotlib: 3.5.1 cartopy: 0.20.1 seaborn: None numbagg: None fsspec: 2021.11.1 cupy: None pint: None sparse: None setuptools: 60.0.3 pip: 21.3.1 conda: None pytest: 6.2.5 IPython: 7.30.1 sphinx: 4.3.2 ```
","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/6092/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed,13221727,issue 1419602897,I_kwDOAMm_X85UnWvR,7197,Unstable pandas causes CF datetime64 issues,1828519,closed,0,,,4,2022-10-23T02:30:39Z,2022-10-26T16:00:35Z,2022-10-26T16:00:35Z,CONTRIBUTOR,,,,"### What happened? The Satpy project has a CI environment that installs numpy, pandas, and xarray (and a couple other packages) from their unstable sources (nightly builds, github source, etc). In the last week or two this environment has started failing with various datetime64 issues. It all seems to be caused by some recent change in pandas, but I can't place exactly what the problem is nor the commit/PR that started it. It seems there are a couple datetime related PRs. ### What did you expect to happen? Datetime or datetime64 objects should be allowed to be in whatever units they need to be in (days or minutes or nanoseconds. It seems parts of xarray (or pandas) assume `datetime64[ns]` but a change in pandas is no longer doing this conversion automatically (from `datetime64[X]` to `datetime64[ns]`. ### Minimal Complete Verifiable Example You should be able to take any environment with modern xarray and install dev pandas with: ``` python -m pip install --index-url https://pypi.anaconda.org/scipy-wheels-nightly/simple/ --trusted-host pypi.anaconda.org --no-deps --pre --upgrade pandas ``` Then run this snippet: ```Python import xarray as xr import numpy as np from xarray.coding.times import CFDatetimeCoder a = xr.DataArray(np.arange(1.0), dims=(""time"",), coords={""time"": [np.datetime64('2018-05-30T10:05:00')]}) CFDatetimeCoder().encode(a.coords[""time""]) ``` I haven't been able to generate a higher-level MVCE yet, but I'm hoping this little snippet will make the issue obvious to someone familiar with xarray internals. ### MVCE confirmation - [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray. - [X] Complete example — the example is self-contained, including all data and the text of any traceback. - [X] Verifiable example — the example copy & pastes into an IPython prompt or [Binder notebook](https://mybinder.org/v2/gh/pydata/xarray/main?urlpath=lab/tree/doc/examples/blank_template.ipynb), returning the result. - [X] New issue — a search of GitHub Issues suggests this is not a duplicate. ### Relevant log output ``` ----> 1 CFDatetimeCoder().encode(a.coords[""time""]) File ~/miniconda3/envs/satpy_py39_unstable/lib/python3.9/site-packages/xarray/coding/times.py:676, in CFDatetimeCoder.encode(self, variable, name) 672 dims, data, attrs, encoding = unpack_for_encoding(variable) 673 if np.issubdtype(data.dtype, np.datetime64) or contains_cftime_datetimes( 674 variable 675 ): --> 676 (data, units, calendar) = encode_cf_datetime( 677 data, encoding.pop(""units"", None), encoding.pop(""calendar"", None) 678 ) 679 safe_setitem(attrs, ""units"", units, name=name) 680 safe_setitem(attrs, ""calendar"", calendar, name=name) File ~/miniconda3/envs/satpy_py39_unstable/lib/python3.9/site-packages/xarray/coding/times.py:612, in encode_cf_datetime(dates, units, calendar) 609 dates = np.asarray(dates) 611 if units is None: --> 612 units = infer_datetime_units(dates) 613 else: 614 units = _cleanup_netcdf_time_units(units) File ~/miniconda3/envs/satpy_py39_unstable/lib/python3.9/site-packages/xarray/coding/times.py:394, in infer_datetime_units(dates) 392 print(""Formatting datetime object"") 393 reference_date = dates[0] if len(dates) > 0 else ""1970-01-01"" --> 394 reference_date = format_cftime_datetime(reference_date) 395 unique_timedeltas = np.unique(np.diff(dates)) 396 units = _infer_time_units_from_diff(unique_timedeltas) File ~/miniconda3/envs/satpy_py39_unstable/lib/python3.9/site-packages/xarray/coding/times.py:405, in format_cftime_datetime(date) 400 def format_cftime_datetime(date): 401 """"""Converts a cftime.datetime object to a string with the format: 402 YYYY-MM-DD HH:MM:SS.UUUUUU 403 """""" 404 return ""{:04d}-{:02d}-{:02d} {:02d}:{:02d}:{:02d}.{:06d}"".format( --> 405 date.year, 406 date.month, 407 date.day, 408 date.hour, 409 date.minute, 410 date.second, 411 date.microsecond, 412 ) AttributeError: 'numpy.datetime64' object has no attribute 'year' ``` ### Anything else we need to know? _No response_ ### Environment
``` INSTALLED VERSIONS ------------------ commit: None python: 3.9.12 | packaged by conda-forge | (main, Mar 24 2022, 23:25:59) [GCC 10.3.0] python-bits: 64 OS: Linux OS-release: 5.19.0-76051900-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: ('en_US', 'UTF-8') libhdf5: 1.12.2 libnetcdf: 4.8.1 xarray: 2022.10.0 pandas: 2.0.0.dev0+422.g6c46013c54 numpy: 1.23.4 scipy: 1.10.0.dev0+1848.f114d8b netCDF4: 1.6.0 pydap: None h5netcdf: 1.0.0 h5py: 3.7.0 Nio: None zarr: 2.13.0a3.dev5 cftime: 1.6.2 nc_time_axis: None PseudoNetCDF: None rasterio: 1.4dev cfgrib: None iris: None bottleneck: 1.3.5 dask: 2022.10.0+6.gc8dc3955 distributed: None matplotlib: 3.7.0.dev473+gc450aa7baf cartopy: 0.20.3 seaborn: None numbagg: None fsspec: 2022.5.0 cupy: None pint: None sparse: None flox: None numpy_groupies: None setuptools: 65.3.0 pip: 22.2.2 conda: None pytest: 7.1.1 IPython: 8.2.0 sphinx: 5.0.0 ```
","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/7197/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed,13221727,issue 1392878100,I_kwDOAMm_X85TBaIU,7111,New deep copy behavior in 2022.9.0 causes maximum recursion error,1828519,closed,0,,,23,2022-09-30T19:11:38Z,2022-10-06T22:04:02Z,2022-10-06T22:04:02Z,CONTRIBUTOR,,,,"### What happened? I have a case where a Dataset to be written to a NetCDF file has ""ancillary_variables"" that have a circular dependence. For example, variable A has `.attrs[""ancillary_variables""]` that contains variable B, and B has `.attrs[""ancillary_variables""]` that contains A. ### What did you expect to happen? Circular dependencies are detected and avoided. No maximum recursion error. ### Minimal Complete Verifiable Example ```Python In [1]: import xarray as xr In [2]: a = xr.DataArray(1.0, attrs={}) In [3]: b = xr.DataArray(2.0, attrs={}) In [4]: a.attrs[""other""] = b In [5]: b.attrs[""other""] = a In [6]: a_copy = a.copy(deep=True) --------------------------------------------------------------------------- RecursionError Traceback (most recent call last) Cell In [6], line 1 ----> 1 a_copy = a.copy(deep=True) File ~/miniconda3/envs/satpy_py310/lib/python3.10/site-packages/xarray/core/dataarray.py:1172, in DataArray.copy(self, deep, data) 1104 def copy(self: T_DataArray, deep: bool = True, data: Any = None) -> T_DataArray: 1105 """"""Returns a copy of this array. 1106 1107 If `deep=True`, a deep copy is made of the data array. (...) 1170 pandas.DataFrame.copy 1171 """""" -> 1172 variable = self.variable.copy(deep=deep, data=data) 1173 indexes, index_vars = self.xindexes.copy_indexes(deep=deep) 1175 coords = {} File ~/miniconda3/envs/satpy_py310/lib/python3.10/site-packages/xarray/core/variable.py:996, in Variable.copy(self, deep, data) 989 if self.shape != ndata.shape: 990 raise ValueError( 991 ""Data shape {} must match shape of object {}"".format( 992 ndata.shape, self.shape 993 ) 994 ) --> 996 attrs = copy.deepcopy(self._attrs) if deep else copy.copy(self._attrs) 997 encoding = copy.deepcopy(self._encoding) if deep else copy.copy(self._encoding) 999 # note: dims is already an immutable tuple File ~/miniconda3/envs/satpy_py310/lib/python3.10/copy.py:146, in deepcopy(x, memo, _nil) 144 copier = _deepcopy_dispatch.get(cls) 145 if copier is not None: --> 146 y = copier(x, memo) 147 else: 148 if issubclass(cls, type): File ~/miniconda3/envs/satpy_py310/lib/python3.10/copy.py:231, in _deepcopy_dict(x, memo, deepcopy) 229 memo[id(x)] = y 230 for key, value in x.items(): --> 231 y[deepcopy(key, memo)] = deepcopy(value, memo) 232 return y File ~/miniconda3/envs/satpy_py310/lib/python3.10/copy.py:153, in deepcopy(x, memo, _nil) 151 copier = getattr(x, ""__deepcopy__"", None) 152 if copier is not None: --> 153 y = copier(memo) 154 else: 155 reductor = dispatch_table.get(cls) File ~/miniconda3/envs/satpy_py310/lib/python3.10/site-packages/xarray/core/dataarray.py:1190, in DataArray.__deepcopy__(self, memo) 1187 def __deepcopy__(self: T_DataArray, memo=None) -> T_DataArray: 1188 # memo does nothing but is required for compatibility with 1189 # copy.deepcopy -> 1190 return self.copy(deep=True) File ~/miniconda3/envs/satpy_py310/lib/python3.10/site-packages/xarray/core/dataarray.py:1172, in DataArray.copy(self, deep, data) 1104 def copy(self: T_DataArray, deep: bool = True, data: Any = None) -> T_DataArray: 1105 """"""Returns a copy of this array. 1106 1107 If `deep=True`, a deep copy is made of the data array. (...) 1170 pandas.DataFrame.copy 1171 """""" -> 1172 variable = self.variable.copy(deep=deep, data=data) 1173 indexes, index_vars = self.xindexes.copy_indexes(deep=deep) 1175 coords = {} File ~/miniconda3/envs/satpy_py310/lib/python3.10/site-packages/xarray/core/variable.py:996, in Variable.copy(self, deep, data) 989 if self.shape != ndata.shape: 990 raise ValueError( 991 ""Data shape {} must match shape of object {}"".format( 992 ndata.shape, self.shape 993 ) 994 ) --> 996 attrs = copy.deepcopy(self._attrs) if deep else copy.copy(self._attrs) 997 encoding = copy.deepcopy(self._encoding) if deep else copy.copy(self._encoding) 999 # note: dims is already an immutable tuple File ~/miniconda3/envs/satpy_py310/lib/python3.10/copy.py:146, in deepcopy(x, memo, _nil) 144 copier = _deepcopy_dispatch.get(cls) 145 if copier is not None: --> 146 y = copier(x, memo) 147 else: 148 if issubclass(cls, type): File ~/miniconda3/envs/satpy_py310/lib/python3.10/copy.py:231, in _deepcopy_dict(x, memo, deepcopy) 229 memo[id(x)] = y 230 for key, value in x.items(): --> 231 y[deepcopy(key, memo)] = deepcopy(value, memo) 232 return y File ~/miniconda3/envs/satpy_py310/lib/python3.10/copy.py:153, in deepcopy(x, memo, _nil) 151 copier = getattr(x, ""__deepcopy__"", None) 152 if copier is not None: --> 153 y = copier(memo) 154 else: 155 reductor = dispatch_table.get(cls) File ~/miniconda3/envs/satpy_py310/lib/python3.10/site-packages/xarray/core/dataarray.py:1190, in DataArray.__deepcopy__(self, memo) 1187 def __deepcopy__(self: T_DataArray, memo=None) -> T_DataArray: 1188 # memo does nothing but is required for compatibility with 1189 # copy.deepcopy -> 1190 return self.copy(deep=True) [... skipping similar frames: DataArray.copy at line 1172 (495 times), DataArray.__deepcopy__ at line 1190 (494 times), _deepcopy_dict at line 231 (494 times), Variable.copy at line 996 (494 times), deepcopy at line 146 (494 times), deepcopy at line 153 (494 times)] File ~/miniconda3/envs/satpy_py310/lib/python3.10/site-packages/xarray/core/variable.py:996, in Variable.copy(self, deep, data) 989 if self.shape != ndata.shape: 990 raise ValueError( 991 ""Data shape {} must match shape of object {}"".format( 992 ndata.shape, self.shape 993 ) 994 ) --> 996 attrs = copy.deepcopy(self._attrs) if deep else copy.copy(self._attrs) 997 encoding = copy.deepcopy(self._encoding) if deep else copy.copy(self._encoding) 999 # note: dims is already an immutable tuple File ~/miniconda3/envs/satpy_py310/lib/python3.10/copy.py:146, in deepcopy(x, memo, _nil) 144 copier = _deepcopy_dispatch.get(cls) 145 if copier is not None: --> 146 y = copier(x, memo) 147 else: 148 if issubclass(cls, type): File ~/miniconda3/envs/satpy_py310/lib/python3.10/copy.py:231, in _deepcopy_dict(x, memo, deepcopy) 229 memo[id(x)] = y 230 for key, value in x.items(): --> 231 y[deepcopy(key, memo)] = deepcopy(value, memo) 232 return y File ~/miniconda3/envs/satpy_py310/lib/python3.10/copy.py:153, in deepcopy(x, memo, _nil) 151 copier = getattr(x, ""__deepcopy__"", None) 152 if copier is not None: --> 153 y = copier(memo) 154 else: 155 reductor = dispatch_table.get(cls) File ~/miniconda3/envs/satpy_py310/lib/python3.10/site-packages/xarray/core/dataarray.py:1190, in DataArray.__deepcopy__(self, memo) 1187 def __deepcopy__(self: T_DataArray, memo=None) -> T_DataArray: 1188 # memo does nothing but is required for compatibility with 1189 # copy.deepcopy -> 1190 return self.copy(deep=True) File ~/miniconda3/envs/satpy_py310/lib/python3.10/site-packages/xarray/core/dataarray.py:1172, in DataArray.copy(self, deep, data) 1104 def copy(self: T_DataArray, deep: bool = True, data: Any = None) -> T_DataArray: 1105 """"""Returns a copy of this array. 1106 1107 If `deep=True`, a deep copy is made of the data array. (...) 1170 pandas.DataFrame.copy 1171 """""" -> 1172 variable = self.variable.copy(deep=deep, data=data) 1173 indexes, index_vars = self.xindexes.copy_indexes(deep=deep) 1175 coords = {} File ~/miniconda3/envs/satpy_py310/lib/python3.10/site-packages/xarray/core/variable.py:985, in Variable.copy(self, deep, data) 982 ndata = indexing.MemoryCachedArray(ndata.array) 984 if deep: --> 985 ndata = copy.deepcopy(ndata) 987 else: 988 ndata = as_compatible_data(data) File ~/miniconda3/envs/satpy_py310/lib/python3.10/copy.py:137, in deepcopy(x, memo, _nil) 134 if memo is None: 135 memo = {} --> 137 d = id(x) 138 y = memo.get(d, _nil) 139 if y is not _nil: RecursionError: maximum recursion depth exceeded while calling a Python object ``` ### MVCE confirmation - [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray. - [X] Complete example — the example is self-contained, including all data and the text of any traceback. - [X] Verifiable example — the example copy & pastes into an IPython prompt or [Binder notebook](https://mybinder.org/v2/gh/pydata/xarray/main?urlpath=lab/tree/doc/examples/blank_template.ipynb), returning the result. - [X] New issue — a search of GitHub Issues suggests this is not a duplicate. ### Relevant log output _No response_ ### Anything else we need to know? I have at least one other issue related to the new xarray release but I'm still tracking it down. I think it is also related to the deep copy behavior change which was merged a day before the release so our CI didn't have time to test the ""unstable"" version of xarray. ### Environment
``` INSTALLED VERSIONS ------------------ commit: None python: 3.10.6 | packaged by conda-forge | (main, Aug 22 2022, 20:35:26) [GCC 10.4.0] python-bits: 64 OS: Linux OS-release: 5.19.0-76051900-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: ('en_US', 'UTF-8') libhdf5: 1.12.2 libnetcdf: 4.8.1 xarray: 2022.9.0 pandas: 1.5.0 numpy: 1.23.3 scipy: 1.9.1 netCDF4: 1.6.1 pydap: None h5netcdf: 1.0.2 h5py: 3.7.0 Nio: None zarr: 2.13.2 cftime: 1.6.2 nc_time_axis: None PseudoNetCDF: None rasterio: 1.3.2 cfgrib: None iris: None bottleneck: 1.3.5 dask: 2022.9.1 distributed: 2022.9.1 matplotlib: 3.6.0 cartopy: 0.21.0 seaborn: None numbagg: None fsspec: 2022.8.2 cupy: None pint: None sparse: None flox: None numpy_groupies: None setuptools: 65.4.0 pip: 22.2.2 conda: None pytest: 7.1.3 IPython: 8.5.0 sphinx: 5.2.3 ```
","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/7111/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed,13221727,issue 812692450,MDU6SXNzdWU4MTI2OTI0NTA=,4934,ImplicitToExplicitIndexingAdapter being returned with dask unstable version,1828519,closed,0,,,4,2021-02-20T19:41:12Z,2021-03-09T22:51:13Z,2021-03-09T22:51:13Z,CONTRIBUTOR,,,,"**What happened**: I have a couple CI environments that use unstable versions of xarray and dask among other libraries. A couple tests have been failing in different but similar ways. I was able to get one of them down to the below example. This happens with the dask master branch on either the xarray latest release or xarray master. Here are the commits I was using when doing unstable both. Dask: `ad01acc14088d03127dfec4f881cce959f3ac4d9` Xarray: `eb7e112d45a9edebd8e5fb4f873e3e6adb18824a` I don't see this with dask's latest release and this is likely caused by a change in dask, but after seeing PRs like #4884 I'm wondering if this is something similar and requires a change in xarray. The bottom line is that doing various things like `my_data_arr.data.compute()` or `dask.compute(my_data_arr.data)` produces: ``` ImplicitToExplicitIndexingAdapter(array=CopyOnWriteArray(array=LazilyOuterIndexedArray(array=, key=BasicIndexer((slice(0, 2, 1), slice(0, 100, 1), slice(0, 100, 1)))))) ``` **What you expected to happen**: I would expect to get a numpy array back when computing the underlying dask array. **Minimal Complete Verifiable Example**: ```python from PIL import Image import xarray as xr import numpy as np # create a test image Image.fromarray(np.zeros((5, 5, 3), dtype=np.uint8)).save('test.png') r = xr.open_rasterio('test.png', chunks='auto') print(r.data.compute()) # ImplicitToExplicitIndexingAdapter(array=CopyOnWriteArray(array=LazilyOuterIndexedArray(array=, key=BasicIndexer((slice(0, 2, 1), slice(0, 100, 1), slice(0, 100, 1)))))) ``` **Anything else we need to know?**: As mentioned above other weird things are happening when array wrappers seem to be involved but I haven't been able to make a small example of them. **Environment**:
Output of xr.show_versions() INSTALLED VERSIONS ------------------ commit: None python: 3.7.6 | packaged by conda-forge | (default, Jan 7 2020, 22:33:48) [GCC 7.3.0] python-bits: 64 OS: Linux OS-release: 5.8.0-7642-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: en_US.UTF-8 libhdf5: 1.10.5 libnetcdf: 4.7.3 xarray: 0.16.3.dev132+geb7e112d pandas: 1.2.2 numpy: 1.20.1 scipy: 1.6.1 netCDF4: 1.5.3 pydap: None h5netcdf: None h5py: 2.10.0 Nio: None zarr: 2.6.2.dev42 cftime: 1.4.1 nc_time_axis: None PseudoNetCDF: None rasterio: 1.3dev cfgrib: None iris: None bottleneck: 1.4.0.dev0+117.gf2bc792 dask: 2021.02.0+21.gad01acc1 distributed: 2021.02.0+7.g383ea032 matplotlib: 3.4.0rc1 cartopy: 0.17.0 seaborn: None numbagg: None pint: None setuptools: 45.2.0.post20200209 pip: 20.0.2 conda: None pytest: 5.3.5 IPython: 7.12.0 sphinx: 2.4.3
","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/4934/reactions"", ""total_count"": 3, ""+1"": 3, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed,13221727,issue 425415572,MDU6SXNzdWU0MjU0MTU1NzI=,2853,Future plans for versioneer,1828519,closed,0,,,3,2019-03-26T13:23:51Z,2020-07-26T11:23:44Z,2020-07-26T11:23:44Z,CONTRIBUTOR,,,,"I have projects that use versioneer after finding out that xarray and dask used it. However, it seems the project is no longer actively maintained with the last commit happening almost 2 years ago: https://github.com/warner/python-versioneer There are two issues I've found with versioneer: 1. Wheels do not have a valid version number. They get `0+unknown` unless an `sdist` is built first to populate the egg info. Building sdist is not something that is always done by CI environments like travis. This also happens if someone installs the package from github `pip install git+https://github.com/pydata/xarray.git`. At least these are my findings with my satpy library. 2. `versioneer.py` has some style issues: ``` versioneer.py:941:-13592: W605 invalid escape sequence '\s' versioneer.py:941:-13408: W605 invalid escape sequence '\s' versioneer.py:941:-13228: W605 invalid escape sequence '\s' versioneer.py:941:-11196: W605 invalid escape sequence '\d' versioneer.py:941:-8162: W605 invalid escape sequence '\d' ``` This was worked around in xarray by adding a `noqa` to the top of the module. So my question is, do the xarray devs (or dask devs) see this as a problem? What are your plans for the future? Have you enjoyed using versioneer? I guess I should probably CC @warner on this too in case they are still active on github.","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/2853/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed,13221727,issue 314457748,MDU6SXNzdWUzMTQ0NTc3NDg=,2060,Confusing error message when attribute not equal during concat,1828519,closed,0,,,3,2018-04-15T22:20:12Z,2019-12-24T13:37:04Z,2019-12-24T13:37:04Z,CONTRIBUTOR,,,,"#### Code Sample, a copy-pastable example if possible ```python In [1]: import dask.array as da; import xarray as xr; import numpy as np In [2]: a = xr.DataArray(da.random.random((4, 6), chunks=2), attrs={'test': ['x1', 'y1']}, dims=('y', 'x')) In [3]: b = xr.DataArray(da.random.random((4, 6), chunks=2), attrs={'test': ['x2', 'y2']}, dims=('y', 'x')) In [4]: xr.concat([a, b], 'y') --------------------------------------------------------------------------- ValueError Traceback (most recent call last) in () ----> 1 xr.concat([a, b], 'y') ~/anaconda/envs/polar2grid_py36/lib/python3.6/site-packages/xarray/core/combine.py in concat(objs, dim, data_vars, coords, compat, positions, indexers, mode, concat_over) 119 raise TypeError('can only concatenate xarray Dataset and DataArray ' 120 'objects, got %s' % type(first_obj)) --> 121 return f(objs, dim, data_vars, coords, compat, positions) 122 123 ~/anaconda/envs/polar2grid_py36/lib/python3.6/site-packages/xarray/core/combine.py in _dataarray_concat(arrays, dim, data_vars, coords, compat, positions) 337 338 ds = _dataset_concat(datasets, dim, data_vars, coords, compat, --> 339 positions) 340 return arrays[0]._from_temp_dataset(ds, name) 341 ~/anaconda/envs/polar2grid_py36/lib/python3.6/site-packages/xarray/core/combine.py in _dataset_concat(datasets, dim, data_vars, coords, compat, positions) 303 if k in concat_over: 304 vars = ensure_common_dims([ds.variables[k] for ds in datasets]) --> 305 combined = concat_vars(vars, dim, positions) 306 insert_result_variable(k, combined) 307 ~/anaconda/envs/polar2grid_py36/lib/python3.6/site-packages/xarray/core/variable.py in concat(variables, dim, positions, shortcut) 1772 return IndexVariable.concat(variables, dim, positions, shortcut) 1773 else: -> 1774 return Variable.concat(variables, dim, positions, shortcut) 1775 1776 ~/anaconda/envs/polar2grid_py36/lib/python3.6/site-packages/xarray/core/variable.py in concat(cls, variables, dim, positions, shortcut) 1299 if var.dims != first_var.dims: 1300 raise ValueError('inconsistent dimensions') -> 1301 utils.remove_incompatible_items(attrs, var.attrs) 1302 1303 return cls(dims, data, attrs, encoding) ~/anaconda/envs/polar2grid_py36/lib/python3.6/site-packages/xarray/core/utils.py in remove_incompatible_items(first_dict, second_dict, compat) 157 if (k not in second_dict or 158 (k in second_dict and --> 159 not compat(first_dict[k], second_dict[k]))): 160 del first_dict[k] 161 ~/anaconda/envs/polar2grid_py36/lib/python3.6/site-packages/xarray/core/utils.py in equivalent(first, second) 106 return ((first is second) or 107 (first == second) or --> 108 (pd.isnull(first) and pd.isnull(second))) 109 110 ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() ``` #### Problem description If two or more `DataArray`s are concatentated and they have list attributes that are not equal an exception is raised about arrays not being truth values. #### Expected Output I guess the expected result would be that the list attribute is not included in the resulting DataArray's attributes. #### Output of ``xr.show_versions()``
``` DEBUG:matplotlib:$HOME=/Users/davidh DEBUG:matplotlib:matplotlib data path /Users/davidh/anaconda/envs/polar2grid_py36/lib/python3.6/site-packages/matplotlib/mpl-data DEBUG:matplotlib:loaded rc file /Users/davidh/anaconda/envs/polar2grid_py36/lib/python3.6/site-packages/matplotlib/mpl-data/matplotlibrc DEBUG:matplotlib:matplotlib version 2.2.0 DEBUG:matplotlib:interactive is False DEBUG:matplotlib:platform is darwin DEBUG:matplotlib:loaded modules: ['builtins', 'sys', '_frozen_importlib', '_imp', '_warnings', '_thread', '_weakref', '_frozen_importlib_external', '_io', 'marshal', 'posix', 'zipimport', 'encodings', 'codecs', '_codecs', 'encodings.aliases', 'encodings.utf_8', '_signal', '__main__', 'encodings.latin_1', 'io', 'abc', '_weakrefset', 'site', 'os', 'errno', 'stat', '_stat', 'posixpath', 'genericpath', 'os.path', '_collections_abc', '_sitebuiltins', 'sysconfig', '_sysconfigdata_m_darwin_darwin', '_osx_support', 're', 'enum', 'types', 'functools', '_functools', 'collections', 'operator', '_operator', 'keyword', 'heapq', '_heapq', 'itertools', 'reprlib', '_collections', 'weakref', 'collections.abc', 'sre_compile', '_sre', 'sre_parse', 'sre_constants', '_locale', 'copyreg', '_bootlocale', 'importlib', 'importlib._bootstrap', 'importlib._bootstrap_external', 'warnings', 'importlib.util', 'importlib.abc', 'importlib.machinery', 'contextlib', 'mpl_toolkits', 'sphinxcontrib', 'encodings.cp437', 'IPython', 'IPython.core', 'IPython.core.getipython', 'IPython.core.release', 'IPython.core.application', 'atexit', 'copy', 'glob', 'fnmatch', 'logging', 'time', 'traceback', 'linecache', 'tokenize', 'token', 'string', '_string', 'threading', 'shutil', 'zlib', 'bz2', '_compression', '_bz2', 'lzma', '_lzma', 'pwd', 'grp', 'traitlets', 'traitlets.traitlets', 'inspect', 'ast', '_ast', 'dis', 'opcode', '_opcode', 'six', '__future__', 'struct', '_struct', 'traitlets.utils', 'traitlets.utils.getargspec', 'traitlets.utils.importstring', 'ipython_genutils', 'ipython_genutils._version', 'ipython_genutils.py3compat', 'ipython_genutils.encoding', 'locale', 'platform', 'subprocess', 'signal', '_posixsubprocess', 'select', 'selectors', 'math', 'traitlets.utils.sentinel', 'traitlets.utils.bunch', 'traitlets._version', 'traitlets.config', 'traitlets.config.application', 'json', 'json.decoder', 'json.scanner', '_json', 'json.encoder', 'decorator', 'traitlets.config.configurable', 'traitlets.config.loader', 'argparse', 'textwrap', 'gettext', 'ipython_genutils.path', 'random', 'hashlib', '_hashlib', '_blake2', '_sha3', 'bisect', '_bisect', '_random', 'ipython_genutils.text', 'ipython_genutils.importstring', 'IPython.core.crashhandler', 'pprint', 'IPython.core.ultratb', 'pydoc', 'pkgutil', 'urllib', 'urllib.parse', 'IPython.core.debugger', 'bdb', 'IPython.utils', 'IPython.utils.PyColorize', 'IPython.utils.coloransi', 'IPython.utils.ipstruct', 'IPython.utils.colorable', 'pygments', 'pygments.util', 'IPython.utils.py3compat', 'IPython.utils.encoding', 'IPython.core.excolors', 'IPython.testing', 'IPython.testing.skipdoctest', 'pdb', 'cmd', 'code', 'codeop', 'IPython.core.display_trap', 'IPython.utils.openpy', 'IPython.utils.path', 'IPython.utils.process', 'IPython.utils._process_posix', 'pexpect', 'pexpect.exceptions', 'pexpect.utils', 'pexpect.expect', 'pexpect.pty_spawn', 'pty', 'tty', 'termios', 'ptyprocess', 'ptyprocess.ptyprocess', 'fcntl', 'resource', 'ptyprocess.util', 'pexpect.spawnbase', 'pexpect.run', 'IPython.utils._process_common', 'shlex', 'IPython.utils.decorators', 'IPython.utils.data', 'IPython.utils.terminal', 'IPython.utils.sysinfo', 'IPython.utils._sysinfo', 'IPython.core.profiledir', 'IPython.paths', 'tempfile', 'IPython.utils.importstring', 'IPython.terminal', 'IPython.terminal.embed', 'IPython.core.compilerop', 'IPython.core.magic_arguments', 'IPython.core.error', 'IPython.utils.text', 'pathlib', 'ntpath', 'IPython.core.magic', 'getopt', 'IPython.core.oinspect', 'IPython.core.page', 'IPython.core.display', 'base64', 'binascii', 'mimetypes', 'IPython.lib', 'IPython.lib.security', 'getpass', 'IPython.lib.pretty', 'datetime', '_datetime', 'IPython.utils.dir2', 'IPython.utils.wildcard', 'pygments.lexers', 'pygments.lexers._mapping', 'pygments.modeline', 'pygments.plugin', 'pygments.lexers.python', 'pygments.lexer', 'pygments.filter', 'pygments.filters', 'pygments.token', 'pygments.regexopt', 'pygments.unistring', 'pygments.formatters', 'pygments.formatters._mapping', 'pygments.formatters.html', 'pygments.formatter', 'pygments.styles', 'IPython.core.inputsplitter', 'IPython.core.inputtransformer', 'IPython.core.splitinput', 'IPython.utils.tokenize2', 'IPython.core.interactiveshell', 'runpy', 'pickleshare', 'pickle', '_compat_pickle', '_pickle', 'IPython.core.prefilter', 'IPython.core.autocall', 'IPython.core.macro', 'IPython.core.alias', 'IPython.core.builtin_trap', 'IPython.core.events', 'IPython.core.displayhook', 'IPython.core.displaypub', 'IPython.core.extensions', 'IPython.core.formatters', 'IPython.utils.sentinel', 'IPython.core.history', 'sqlite3', 'sqlite3.dbapi2', '_sqlite3', 'IPython.core.logger', 'IPython.core.payload', 'IPython.core.usage', 'IPython.display', 'IPython.lib.display', 'IPython.utils.io', 'IPython.utils.capture', 'IPython.utils.strdispatch', 'IPython.core.hooks', 'IPython.utils.syspathcontext', 'IPython.utils.tempdir', 'IPython.utils.contexts', 'IPython.terminal.interactiveshell', 'prompt_toolkit', 'prompt_toolkit.interface', 'prompt_toolkit.application', 'prompt_toolkit.buffer', 'prompt_toolkit.auto_suggest', 'prompt_toolkit.filters', 'prompt_toolkit.filters.base', 'prompt_toolkit.utils', 'wcwidth', 'wcwidth.wcwidth', 'wcwidth.table_wide', 'wcwidth.table_zero', 'six.moves', 'prompt_toolkit.filters.cli', 'prompt_toolkit.enums', 'prompt_toolkit.key_binding', 'prompt_toolkit.key_binding.vi_state', 'prompt_toolkit.cache', 'prompt_toolkit.filters.types', 'prompt_toolkit.filters.utils', 'prompt_toolkit.clipboard', 'prompt_toolkit.clipboard.base', 'prompt_toolkit.selection', 'prompt_toolkit.clipboard.in_memory', 'prompt_toolkit.completion', 'prompt_toolkit.document', 'prompt_toolkit.history', 'prompt_toolkit.search_state', 'prompt_toolkit.validation', 'prompt_toolkit.buffer_mapping', 'prompt_toolkit.key_binding.bindings', 'prompt_toolkit.key_binding.bindings.basic', 'prompt_toolkit.keys', 'prompt_toolkit.layout', 'prompt_toolkit.layout.containers', 'prompt_toolkit.layout.controls', 'prompt_toolkit.mouse_events', 'prompt_toolkit.token', 'prompt_toolkit.layout.lexers', 'prompt_toolkit.layout.utils', 'prompt_toolkit.layout.processors', 'prompt_toolkit.reactive', 'prompt_toolkit.layout.screen', 'prompt_toolkit.layout.dimension', 'prompt_toolkit.layout.margins', 'prompt_toolkit.renderer', 'prompt_toolkit.layout.mouse_handlers', 'prompt_toolkit.output', 'prompt_toolkit.styles', 'prompt_toolkit.styles.base', 'prompt_toolkit.styles.defaults', 'prompt_toolkit.styles.from_dict', 'prompt_toolkit.styles.utils', 'prompt_toolkit.styles.from_pygments', 'pygments.style', 'pygments.styles.default', 'prompt_toolkit.key_binding.bindings.named_commands', 'prompt_toolkit.key_binding.bindings.completion', 'prompt_toolkit.key_binding.registry', 'prompt_toolkit.key_binding.input_processor', 'prompt_toolkit.key_binding.bindings.emacs', 'prompt_toolkit.key_binding.bindings.scroll', 'prompt_toolkit.key_binding.bindings.vi', 'prompt_toolkit.key_binding.digraphs', 'prompt_toolkit.key_binding.defaults', 'prompt_toolkit.eventloop', 'prompt_toolkit.eventloop.base', 'prompt_toolkit.eventloop.callbacks', 'prompt_toolkit.input', 'prompt_toolkit.terminal', 'prompt_toolkit.terminal.vt100_input', 'prompt_toolkit.shortcuts', 'prompt_toolkit.layout.menus', 'prompt_toolkit.layout.prompt', 'prompt_toolkit.layout.toolbars', 'prompt_toolkit.terminal.vt100_output', 'array', 'prompt_toolkit.key_binding.manager', 'IPython.terminal.debugger', 'IPython.core.completer', 'unicodedata', 'typing', 'typing.io', 'typing.re', 'IPython.core.latex_symbols', 'IPython.utils.generics', 'simplegeneric', 'jedi', 'jedi.api', 'jedi.parser', 'jedi.parser.parser', 'jedi.parser.tree', 'jedi._compatibility', 'imp', 'jedi.parser.pgen2', 'jedi.parser.pgen2.parse', 'jedi.parser.tokenize', 'jedi.parser.token', 'jedi.common', 'jedi.settings', 'jedi.parser.pgen2.pgen', 'jedi.parser.pgen2.grammar', 'jedi.parser.python', 'jedi.parser.python.parser', 'jedi.parser.python.tree', 'jedi.parser.python.diff', 'difflib', 'jedi.debug', 'jedi.parser.cache', 'gc', 'jedi.cache', 'jedi.api.classes', 'jedi.evaluate', 'jedi.evaluate.representation', 'jedi.evaluate.cache', 'jedi.evaluate.compiled', 'jedi.evaluate.helpers', 'jedi.evaluate.filters', 'jedi.evaluate.flow_analysis', 'jedi.evaluate.context', 'jedi.evaluate.compiled.fake', 'jedi.evaluate.recursion', 'jedi.evaluate.iterable', 'jedi.evaluate.analysis', 'jedi.evaluate.pep0484', 'jedi.evaluate.precedence', 'jedi.evaluate.docstrings', 'jedi.evaluate.param', 'jedi.evaluate.imports', 'jedi.evaluate.sys_path', 'jedi.evaluate.site', 'jedi.evaluate.dynamic', 'jedi.evaluate.stdlib', 'jedi.evaluate.instance', 'jedi.evaluate.finder', 'jedi.api.keywords', 'pydoc_data', 'pydoc_data.topics', 'jedi.api.interpreter', 'jedi.evaluate.compiled.mixed', 'jedi.api.usages', 'jedi.api.helpers', 'jedi.api.completion', 'IPython.terminal.ptutils', 'IPython.terminal.shortcuts', 'IPython.terminal.magics', 'IPython.lib.clipboard', 'IPython.terminal.pt_inputhooks', 'IPython.terminal.prompts', 'pkg_resources', 'zipfile', 'plistlib', 'xml', 'xml.parsers', 'xml.parsers.expat', 'pyexpat.errors', 'pyexpat.model', 'pyexpat', 'xml.parsers.expat.model', 'xml.parsers.expat.errors', 'email', 'email.parser', 'email.feedparser', 'email.errors', 'email._policybase', 'email.header', 'email.quoprimime', 'email.base64mime', 'email.charset', 'email.encoders', 'quopri', 'email.utils', 'socket', '_socket', 'email._parseaddr', 'calendar', 'pkg_resources.extern', 'pkg_resources._vendor', 'pkg_resources.extern.six', 'pkg_resources._vendor.six', 'pkg_resources.extern.six.moves', 'pkg_resources._vendor.six.moves', 'pkg_resources.py31compat', 'pkg_resources.extern.appdirs', 'pkg_resources._vendor.packaging.__about__', 'pkg_resources.extern.packaging', 'pkg_resources.extern.packaging.version', 'pkg_resources.extern.packaging._structures', 'pkg_resources.extern.packaging.specifiers', 'pkg_resources.extern.packaging._compat', 'pkg_resources.extern.packaging.requirements', 'pkg_resources.extern.pyparsing', 'pkg_resources.extern.six.moves.urllib', 'pkg_resources.extern.packaging.markers', 'IPython.terminal.ipapp', 'IPython.core.magics', 'IPython.core.magics.auto', 'IPython.core.magics.basic', 'IPython.core.magics.code', 'IPython.core.magics.config', 'IPython.core.magics.display', 'IPython.core.magics.execution', 'timeit', 'cProfile', '_lsprof', 'profile', 'optparse', 'pstats', 'IPython.utils.module_paths', 'IPython.utils.timing', 'IPython.core.magics.extension', 'IPython.core.magics.history', 'IPython.core.magics.logging', 'IPython.core.magics.namespace', 'IPython.core.magics.osm', 'IPython.core.magics.pylab', 'IPython.core.pylabtools', 'IPython.core.magics.script', 'IPython.lib.backgroundjobs', 'IPython.core.shellapp', 'IPython.extensions', 'IPython.extensions.storemagic', 'IPython.utils.frame', 'IPython.core.completerlib', 'pygments.lexers.shell', 'pygments.lexers.html', 'pygments.lexers.javascript', 'pygments.lexers.jvm', 'pygments.lexers.css', 'pygments.lexers.ruby', 'pygments.lexers.perl', 'pygments.lexers.markup', 'prompt_toolkit.eventloop.posix', 'prompt_toolkit.eventloop.inputhook', 'prompt_toolkit.eventloop.select', 'prompt_toolkit.eventloop.posix_utils', 'prompt_toolkit.eventloop.utils', 'storemagic', 'dask', 'dask.core', 'dask.utils_test', 'dask.context', 'dask.local', 'dask.compatibility', 'queue', 'gzip', 'urllib.request', 'http', 'http.client', 'email.message', 'uu', 'email._encoded_words', 'email.iterators', 'ssl', 'ipaddress', '_ssl', 'urllib.error', 'urllib.response', '_scproxy', 'dask.order', 'dask.callbacks', 'dask.optimization', 'dask.delayed', 'uuid', 'ctypes', '_ctypes', 'ctypes._endian', 'ctypes.util', 'ctypes.macholib', 'ctypes.macholib.dyld', 'ctypes.macholib.framework', 'ctypes.macholib.dylib', 'toolz', 'toolz.itertoolz', 'toolz.compatibility', 'toolz.utils', 'toolz.functoolz', 'toolz._signatures', 'toolz.dicttoolz', 'toolz.recipes', 'toolz.sandbox', 'toolz.sandbox.core', 'toolz.sandbox.parallel', 'dask.threaded', 'multiprocessing', 'multiprocessing.context', 'multiprocessing.process', 'multiprocessing.reduction', '__mp_main__', 'multiprocessing.pool', 'multiprocessing.util', 'dask.base', 'dask.hashing', 'dask.utils', 'numbers', 'dask.optimize', 'dask.sharedict', 'cloudpickle', 'cloudpickle.cloudpickle', 'encodings.raw_unicode_escape', 'dask._version', 'dask.array', 'dask.array.core', 'toolz.curried', 'toolz.curried.operator', 'toolz.curried.exceptions', 'numpy', 'numpy._globals', 'numpy.__config__', 'numpy.version', 'numpy._import_tools', 'numpy.add_newdocs', 'numpy.lib', 'numpy.lib.info', 'numpy.lib.type_check', 'numpy.core', 'numpy.core.info', 'numpy.core.multiarray', 'numpy.core.umath', 'numpy.core._internal', 'numpy.compat', 'numpy.compat._inspect', 'numpy.compat.py3k', 'numpy.core.numerictypes', 'numpy.core.numeric', 'numpy.core.arrayprint', 'numpy.core.fromnumeric', 'numpy.core._methods', 'numpy.core.defchararray', 'numpy.core.records', 'numpy.core.memmap', 'numpy.core.function_base', 'numpy.core.machar', 'numpy.core.getlimits', 'numpy.core.shape_base', 'numpy.core.einsumfunc', 'numpy.testing', 'unittest', 'unittest.result', 'unittest.util', 'unittest.case', 'unittest.suite', 'unittest.loader', 'unittest.main', 'unittest.runner', 'unittest.signals', 'numpy.testing.decorators', 'numpy.testing.utils', 'numpy.lib.utils', 'numpy.testing.nosetester', 'numpy.lib.ufunclike', 'numpy.lib.index_tricks', 'numpy.lib.function_base', 'numpy.lib.twodim_base', 'numpy.matrixlib', 'numpy.matrixlib.defmatrix', 'numpy.lib.stride_tricks', 'numpy.lib.mixins', 'numpy.lib.nanfunctions', 'numpy.lib.shape_base', 'numpy.lib.scimath', 'numpy.lib.polynomial', 'numpy.linalg', 'numpy.linalg.info', 'numpy.linalg.linalg', 'numpy.linalg.lapack_lite', 'numpy.linalg._umath_linalg', 'numpy.lib.arraysetops', 'numpy.lib.npyio', 'numpy.lib.format', 'numpy.lib._datasource', 'numpy.lib._iotools', 'numpy.lib.financial', 'numpy.lib.arrayterator', 'numpy.lib.arraypad', 'numpy.lib._version', 'numpy._distributor_init', 'numpy.fft', 'numpy.fft.info', 'numpy.fft.fftpack', 'numpy.fft.fftpack_lite', 'numpy.fft.helper', 'numpy.polynomial', 'numpy.polynomial.polynomial', 'numpy.polynomial.polyutils', 'numpy.polynomial._polybase', 'numpy.polynomial.chebyshev', 'numpy.polynomial.legendre', 'numpy.polynomial.hermite', 'numpy.polynomial.hermite_e', 'numpy.polynomial.laguerre', 'numpy.random', 'numpy.random.info', 'cython_runtime', 'mtrand', 'numpy.random.mtrand', 'numpy.ctypeslib', 'numpy.ma', 'numpy.ma.core', 'numpy.ma.extras', 'dask.array.chunk', 'dask.array.numpy_compat', 'distutils', 'distutils.version', 'dask.array.slicing', 'dask.array.optimization', 'dask.array.routines', 'dask.array.creation', 'dask.array.wrap', 'dask.array.reshape', 'dask.array.ufunc', 'dask.array.reductions', 'dask.array.percentile', 'dask.array.ma', 'dask.array.random', 'dask.array.linalg', 'dask.array.ghost', 'dask.array.learn', 'dask.array.fft', 'scipy', 'scipy._distributor_init', 'scipy.__config__', 'scipy.version', 'scipy._lib', 'scipy._lib._testutils', 'scipy._lib._version', 'scipy._lib.six', 'scipy._lib._ccallback', 'scipy._lib._ccallback_c', 'scipy.fftpack', 'scipy.fftpack.basic', 'scipy.fftpack._fftpack', 'scipy.fftpack.pseudo_diffs', 'scipy.fftpack.convolve', 'scipy.fftpack.helper', 'numpy.dual', 'scipy.fftpack.realtransforms', 'dask.array.rechunk', 'xarray', 'xarray.core', 'xarray.core.alignment', 'xarray.core.utils', 'pandas', 'pytz', 'pytz.exceptions', 'pytz.lazy', 'pytz.tzinfo', 'pytz.tzfile', 'dateutil', 'dateutil._version', 'pandas.compat', 'pandas.compat.chainmap', 'dateutil.parser', 'dateutil.relativedelta', 'dateutil._common', 'dateutil.tz', 'dateutil.tz.tz', 'dateutil.tz._common', 'pandas.compat.numpy', 'pandas._libs', '_cython_0_27_2', 'pandas._libs.tslib', 'pandas._libs.tslibs', 'pandas._libs.tslibs.timedeltas', 'pandas._libs.tslibs.timezones', 'pandas._libs.tslibs.parsing', 'pandas._libs.tslibs.fields', 'pandas._libs.hashtable', 'pandas._libs.lib', 'pandas._libs.interval', 'decimal', '_decimal', 'pandas.core', 'pandas.core.config_init', 'pandas.core.config', 'pandas.io', 'pandas.io.formats', 'pandas.io.formats.printing', 'pandas.core.dtypes', 'pandas.core.dtypes.inference', 'pandas.io.formats.console', 'pandas.io.formats.terminal', 'pandas.core.api', 'pandas.core.algorithms', 'pandas.core.dtypes.cast', 'pandas.core.dtypes.common', 'pandas._libs.algos', 'pandas.core.dtypes.dtypes', 'pandas.core.dtypes.generic', 'pandas.core.dtypes.missing', 'pandas.core.common', 'pandas.api', 'pandas.api.types', 'pandas.core.dtypes.api', 'pandas.core.dtypes.concat', 'pandas.errors', 'pandas.core.categorical', 'pandas.core.accessor', 'pandas.core.base', 'pandas.util', 'pandas.util._decorators', 'pandas._libs.properties', 'pandas.core.util', 'pandas.core.util.hashing', 'pandas._libs.hashing', 'pandas.util._validators', 'pandas.core.nanops', 'bottleneck', 'bottleneck.reduce', 'bottleneck.nonreduce', 'bottleneck.nonreduce_axis', 'bottleneck.move', 'bottleneck.slow', 'bottleneck.slow.reduce', 'bottleneck.slow.nonreduce', 'bottleneck.slow.nonreduce_axis', 'bottleneck.slow.move', 'bottleneck.version', 'bottleneck.benchmark', 'bottleneck.benchmark.bench', 'bottleneck.benchmark.autotimeit', 'bottleneck.benchmark.bench_detailed', 'bottleneck.tests', 'bottleneck.tests.util', 'pandas.compat.numpy.function', 'pandas.core.missing', 'pandas.core.groupby', 'pandas.core.index', 'pandas.core.indexes', 'pandas.core.indexes.api', 'pandas.core.indexes.base', 'pandas._libs.index', 'pandas._libs.join', 'pandas.core.indexes.frozen', 'pandas.core.sorting', 'pandas.core.ops', 'pandas.core.strings', 'pandas.core.indexes.category', 'pandas.core.indexes.multi', 'pandas.core.indexes.interval', 'pandas.core.indexes.datetimes', 'pandas.core.indexes.numeric', 'pandas.tseries', 'pandas.tseries.frequencies', 'pandas.tseries.offsets', 'pandas.core.tools', 'pandas.core.tools.datetimes', 'pandas._libs.tslibs.strptime', 'dateutil.easter', 'pandas._libs.tslibs.frequencies', 'pandas.core.indexes.datetimelike', 'pandas.core.tools.timedeltas', 'pandas._libs.period', 'pandas.core.indexes.timedeltas', 'pandas.core.indexes.range', 'pandas.core.indexes.period', 'pandas.core.frame', 'pandas.core.generic', 'pandas.core.indexing', 'pandas.core.internals', 'pandas.core.sparse', 'pandas.core.sparse.array', 'pandas._libs.sparse', 'pandas.io.formats.format', 'pandas.io.common', 'csv', '_csv', 'mmap', 'pandas.io.formats.common', 'pandas.core.series', 'pandas.core.indexes.accessors', 'pandas.plotting', 'pandas.plotting._misc', 'pandas.plotting._style', 'pandas.plotting._compat', 'pandas.plotting._tools', 'pandas.plotting._core', 'pandas.core.window', 'pandas._libs.window', 'pandas.core.panel', 'pandas.core.reshape', 'pandas.core.reshape.util', 'pandas._libs.groupby', 'pandas.core.panel4d', 'pandas.core.panelnd', 'pandas.core.reshape.reshape', 'pandas.core.sparse.api', 'pandas.core.sparse.list', 'pandas.core.sparse.series', 'pandas.core.sparse.scipy_sparse', 'pandas.core.sparse.frame', 'pandas._libs.reshape', 'pandas.core.tools.numeric', 'pandas.util._depr_module', 'pandas.stats', 'pandas.stats.api', 'pandas.stats.moments', 'pandas.tseries.api', 'pandas.core.computation', 'pandas.core.computation.api', 'pandas.core.computation.eval', 'pandas.core.computation.scope', 'pandas.core.computation.engines', 'pandas.core.computation.align', 'pandas.core.computation.common', 'pandas.core.computation.ops', 'pandas.core.reshape.api', 'pandas.core.reshape.concat', 'pandas.core.reshape.merge', 'pandas.core.reshape.pivot', 'pandas.core.reshape.tile', 'pandas.tools', 'pandas.tools.plotting', 'pandas.util._print_versions', 'pandas.io.api', 'pandas.io.parsers', 'pandas.io.date_converters', 'pandas._libs.parsers', 'pandas.io.clipboards', 'pandas.io.excel', 'pandas._libs.json', 'pandas.compat.openpyxl_compat', 'pandas.io.pytables', 'pandas.core.computation.pytables', 'pandas.core.computation.expr', 'pandas.io.json', 'pandas.io.json.json', 'pandas.io.json.normalize', 'pandas.io.json.table_schema', 'pandas.io.html', 'pandas.io.sql', 'pandas.io.sas', 'pandas.io.sas.sasreader', 'pandas.io.feather_format', 'pandas.io.parquet', 'pandas.io.stata', 'pandas.io.pickle', 'pandas.compat.pickle_compat', 'pandas.io.packers', 'pandas.io.msgpack', 'pandas.io.msgpack.exceptions', 'pandas.io.msgpack._version', 'pandas.io.msgpack._packer', 'pandas.io.msgpack._unpacker', 'pandas.util._move', 'pandas.io.gbq', 'pandas.util._tester', 'pandas.testing', 'pandas.util.testing', 'pandas._libs.testing', 'pandas._version', 'xarray.core.pycompat', 'xarray.core.indexing', 'xarray.core.nputils', 'xarray.core.duck_array_ops', 'xarray.core.npcompat', 'xarray.core.dtypes', 'xarray.core.variable', 'xarray.core.common', 'xarray.core.formatting', 'xarray.core.options', 'xarray.core.ops', 'xarray.core.combine', 'xarray.core.merge', 'xarray.core.computation', 'xarray.core.extensions', 'xarray.core.dataarray', 'xarray.plot', 'xarray.plot.plot', 'xarray.plot.utils', 'xarray.plot.facetgrid', 'xarray.core.groupby', 'xarray.core.resample', 'xarray.core.rolling', 'xarray.core.dask_array_ops', 'xarray.core.accessors', 'xarray.core.coordinates', 'xarray.core.dataset', 'xarray.conventions', 'xarray.coding', 'xarray.coding.times', 'xarray.coding.variables', 'xarray.backends', 'xarray.backends.common', 'xarray.backends.memory', 'xarray.backends.netCDF4_', 'xarray.backends.netcdf3', 'xarray.backends.pydap_', 'xarray.backends.pynio_', 'xarray.backends.scipy_', 'xarray.backends.h5netcdf_', 'xarray.backends.zarr', 'xarray.backends.api', 'xarray.backends.rasterio_', 'xarray.version', 'xarray.util', 'xarray.util.print_versions', 'xarray.tutorial', 'xarray.ufuncs', 'xarray.testing', 'netCDF4', '_cython_0_27_3', 'netCDF4._netCDF4', 'netCDF4.utils', 'netcdftime', 'netcdftime._netcdftime', 'h5netcdf', 'h5netcdf.core', 'h5py', 'h5py._errors', 'h5py._conv', 'h5py.h5r', 'h5py._objects', 'h5py.defs', 'h5py.h5t', 'h5py.utils', 'h5py.h5', 'h5py.h5z', 'h5py.h5a', 'h5py.h5s', 'h5py.h5p', 'h5py.h5ac', 'h5py._proxy', 'h5py.h5d', 'h5py.h5ds', 'h5py.h5f', 'h5py.h5g', 'h5py.h5i', 'h5py.h5fd', 'h5py._hl', 'h5py._hl.filters', 'h5py._hl.base', 'h5py._hl.compat', 'h5py._hl.files', 'h5py._hl.group', 'h5py.h5o', 'h5py.h5l', 'h5py._hl.dataset', 'h5py._hl.selections', 'h5py._hl.selections2', 'h5py._hl.datatype', 'h5py.version', 'h5py._hl.attrs', 'h5py.tests', 'h5py.tests.common', 'h5py.tests.old', 'h5py.tests.old.test_attrs', 'h5py.highlevel', 'h5py.tests.old.test_attrs_data', 'h5py.tests.old.test_base', 'h5py.tests.old.test_dataset', 'h5py.tests.old.test_datatype', 'h5py.tests.old.test_dimension_scales', 'h5py.tests.old.test_file', 'h5py.tests.old.test_file_image', 'h5py.tests.old.test_group', 'h5py.tests.old.test_h5', 'h5py.tests.old.test_h5f', 'h5py.tests.old.test_h5p', 'h5py.tests.old.test_h5t', 'h5py.tests.old.test_objects', 'h5py.tests.old.test_selections', 'h5py.tests.old.test_slicing', 'h5py.tests.hl', 'h5py.tests.hl.test_dataset_getitem', 'h5py.tests.hl.test_dataset_swmr', 'h5py.tests.hl.test_dims_dimensionproxy', 'h5py.tests.hl.test_file', 'h5py.tests.hl.test_attribute_create', 'h5py.tests.hl.test_threads', 'h5py.tests.hl.test_datatype', 'h5netcdf.compat', 'h5netcdf.attrs', 'h5netcdf.dimensions', 'h5netcdf.utils', 'distributed', 'distributed.config', 'logging.config', 'logging.handlers', 'socketserver', 'distributed.compatibility', 'asyncio', 'asyncio.base_events', 'concurrent', 'concurrent.futures', 'concurrent.futures._base', 'concurrent.futures.process', 'multiprocessing.connection', '_multiprocessing', 'concurrent.futures.thread', 'asyncio.compat', 'asyncio.coroutines', 'asyncio.constants', 'asyncio.events', 'asyncio.base_futures', 'asyncio.log', 'asyncio.futures', 'asyncio.base_tasks', '_asyncio', 'asyncio.tasks', 'asyncio.locks', 'asyncio.protocols', 'asyncio.queues', 'asyncio.streams', 'asyncio.subprocess', 'asyncio.transports', 'asyncio.unix_events', 'asyncio.base_subprocess', 'asyncio.selector_events', 'asyncio.sslproto', 'html', 'html.entities', 'yaml', 'yaml.error', 'yaml.tokens', 'yaml.events', 'yaml.nodes', 'yaml.loader', 'yaml.reader', 'yaml.scanner', 'yaml.parser', 'yaml.composer', 'yaml.constructor', 'yaml.resolver', 'yaml.dumper', 'yaml.emitter', 'yaml.serializer', 'yaml.representer', 'yaml.cyaml', '_yaml', 'distributed.core', 'tornado', 'tornado.gen', 'tornado.concurrent', 'tornado.log', 'tornado.escape', 'tornado.util', 'tornado.speedups', 'curses', '_curses', 'tornado.stack_context', 'tornado.ioloop', 'tornado.platform', 'tornado.platform.auto', 'tornado.platform.posix', 'tornado.platform.common', 'tornado.platform.interface', 'tornado.platform.asyncio', 'tornado.locks', 'distributed.comm', 'distributed.comm.addressing', 'distributed.comm.registry', 'distributed.comm.core', 'distributed.metrics', 'psutil', 'psutil._common', 'psutil._compat', 'psutil._exceptions', 'psutil._psosx', 'psutil._psposix', 'psutil._psutil_osx', 'psutil._psutil_posix', 'distributed.utils', 'tblib', 'tblib.cpython', 'tblib.pickling_support', 'multiprocessing.forkserver', 'multiprocessing.semaphore_tracker', 'multiprocessing.spawn', 'distributed.comm.inproc', 'distributed.protocol', 'distributed.protocol.compression', 'distributed.protocol.core', 'msgpack', 'msgpack._version', 'msgpack.exceptions', 'msgpack._packer', 'msgpack._unpacker', 'distributed.protocol.serialize', 'distributed.protocol.pickle', 'distributed.protocol.utils', 'distributed.comm.tcp', 'tornado.netutil', 'certifi', 'certifi.core', 'encodings.idna', 'stringprep', 'tornado.iostream', 'tornado.tcpclient', 'tornado.tcpserver', 'tornado.process', 'distributed.comm.utils', 'distributed.sizeof', 'distributed.system_monitor', 'distributed.deploy', 'distributed.deploy.local', 'distributed.nanny', 'multiprocessing.queues', 'distributed.node', 'distributed.versions', 'distributed.process', 'distributed.proctitle', 'distributed.security', 'distributed.worker', 'distributed.profile', 'bokeh', 'bokeh.util', 'bokeh.util.version', 'bokeh._version', 'bokeh.util.logconfig', 'bokeh.settings', 'bokeh.util.paths', 'bokeh.util.warnings', 'bokeh.sampledata', 'six.moves.urllib', 'six.moves.urllib.request', 'bokeh.palettes', 'distributed.batched', 'distributed.diskutils', 'distributed.locket', 'distributed.preloading', 'filecmp', 'click', 'click.core', 'click.types', 'click._compat', 'click.exceptions', 'click.utils', 'click.globals', 'click.termui', 'click.formatting', 'click.parser', 'click._unicodefun', 'click.decorators', 'distributed.threadpoolexecutor', 'distributed._concurrent_futures_thread', 'distributed.utils_comm', 'distributed.utils_perf', 'distributed.scheduler', 'sortedcontainers', 'sortedcontainers.sortedlist', 'sortedcontainers.sortedset', 'sortedcontainers.sorteddict', 'distributed.publish', 'distributed.queues', 'tornado.queues', 'distributed.client', 'distributed.cfexecutor', 'distributed.recreate_exceptions', 'distributed.lock', 'distributed.stealing', 'distributed.diagnostics', 'distributed.diagnostics.graph_layout', 'distributed.diagnostics.plugin', 'distributed.diagnostics.progressbar', 'distributed.diagnostics.progress', 'distributed.variable', 'distributed.deploy.adaptive', 'distributed.deploy.ssh', 'distributed.worker_client', 'distributed._version', 'matplotlib', 'distutils.sysconfig', 'distutils.errors', 'matplotlib.cbook', 'matplotlib.cbook.deprecation', 'matplotlib.cbook._backports', 'matplotlib.compat', 'matplotlib.compat.subprocess', 'matplotlib.rcsetup', 'matplotlib.testing', 'matplotlib.fontconfig_pattern', 'pyparsing', 'matplotlib.colors', 'matplotlib._color_data', 'cycler', 'matplotlib._version'] DEBUG:shapely.geos:Trying `CDLL(/Users/davidh/anaconda/envs/polar2grid_py36/bin/../lib/libgeos_c.dylib)` DEBUG:shapely.geos:Library path: '/Users/davidh/anaconda/envs/polar2grid_py36/bin/../lib/libgeos_c.dylib' DEBUG:shapely.geos:DLL: DEBUG:shapely.geos:Trying `CDLL(/usr/lib/libc.dylib)` DEBUG:shapely.geos:Library path: '/usr/lib/libc.dylib' DEBUG:shapely.geos:DLL: DEBUG:pip.vcs:Registered VCS backend: git DEBUG:pip.vcs:Registered VCS backend: hg DEBUG:pip.vcs:Registered VCS backend: svn DEBUG:pip.vcs:Registered VCS backend: bzr INSTALLED VERSIONS ------------------ commit: None python: 3.6.4.final.0 python-bits: 64 OS: Darwin OS-release: 17.5.0 machine: x86_64 processor: i386 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: en_US.UTF-8 xarray: 0.10.1 pandas: 0.21.0 numpy: 1.13.3 scipy: 1.0.0 netCDF4: 1.3.1 h5netcdf: 0.5.0 h5py: 2.7.1 Nio: None zarr: None bottleneck: 1.2.1 cyordereddict: None dask: 0.17.1 distributed: 1.21.2 matplotlib: 2.2.0 cartopy: 0.16.0 seaborn: None setuptools: 39.0.1 pip: 9.0.1 conda: None pytest: 3.4.0 IPython: 6.1.0 sphinx: 1.6.6 ```
","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/2060/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed,13221727,issue 462859457,MDU6SXNzdWU0NjI4NTk0NTc=,3068,Multidimensional dask coordinates unexpectedly computed,1828519,closed,0,,,8,2019-07-01T18:52:03Z,2019-11-05T15:41:15Z,2019-11-05T15:41:15Z,CONTRIBUTOR,,,,"#### MCVE Code Sample ```python from dask.diagnostics import ProgressBar import xarray as xr import numpy as np import dask.array as da a = xr.DataArray(da.zeros((10, 10), chunks=2), dims=('y', 'x'), coords={'y': np.arange(10), 'x': np.arange(10), 'lons': (('y', 'x'), da.zeros((10, 10), chunks=2))}) b = xr.DataArray(da.zeros((10, 10), chunks=2), dims=('y', 'x'), coords={'y': np.arange(10), 'x': np.arange(10), 'lons': (('y', 'x'), da.zeros((10, 10), chunks=2))}) with ProgressBar(): c = a + b ``` Output: ``` [########################################] | 100% Completed | 0.1s ``` #### Problem Description Using arrays with 2D dask array coordinates results in the coordinates being computed for any binary operations (anything combining two or more DataArrays). I use `ProgressBar` in the above example to show when coordinates are being computed. In my own work, when I learned that 2D dask coordinates were possible, I started adding `longitude` and `latitude` coordinates. These are rather large and can take a while to load/compute so I was surprised that simple operations (ex. `a.fillna(b)`) were causing things to be computed and taking a long time. Is this computation by design or a possible bug? #### Expected Output No output from the `ProgressBar`, hoping that no coordinates would be computed/loaded. #### Output of ``xr.show_versions()``
INSTALLED VERSIONS ------------------ commit: None python: 3.6.7 | packaged by conda-forge | (default, Feb 28 2019, 02:16:08) [GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] python-bits: 64 OS: Darwin OS-release: 18.6.0 machine: x86_64 processor: i386 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: en_US.UTF-8 libhdf5: 1.10.4 libnetcdf: 4.6.2 xarray: 0.12.1 pandas: 0.24.2 numpy: 1.14.3 scipy: 1.3.0 netCDF4: 1.5.1.2 pydap: None h5netcdf: 0.7.4 h5py: 2.9.0 Nio: None zarr: 2.3.2 cftime: 1.0.3.4 nc_time_axis: None PseudonetCDF: None rasterio: 1.0.22 cfgrib: None iris: None bottleneck: 1.2.1 dask: 2.0.0 distributed: 2.0.0 matplotlib: 3.1.0 cartopy: 0.17.1.dev147+HEAD.detached.at.5e624fe seaborn: None setuptools: 41.0.1 pip: 19.1.1 conda: None pytest: 4.6.3 IPython: 7.5.0 sphinx: 2.1.2
","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/3068/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed,13221727,issue 479420466,MDU6SXNzdWU0Nzk0MjA0NjY=,3205,Accessors are recreated on every access,1828519,closed,0,,,3,2019-08-11T22:27:14Z,2019-08-14T05:33:48Z,2019-08-14T05:33:48Z,CONTRIBUTOR,,,,"#### MCVE Code Sample 1. Create `test_accessor.py` in current directory with: ```python import xarray as xr @xr.register_dataarray_accessor('test') class TestDataArrayAccessor(object): def __init__(self, obj): self._obj = obj print(""DataArray accessor created"") @xr.register_dataset_accessor('test') class TestDatasetAccessor(object): def __init__(self, obj): self._obj = obj print(""Dataset accessor created"") ``` 2. Run the following code: ```python import xarray as xr import numpy as np import test_accesor ds = xr.Dataset({'a': xr.DataArray(np.array([1, 2, 3])), 'b': xr.DataArray(np.array([4, 5, 6]))}) ds.test # Dataset accessor created ds.test # ds['a'].test # DataArray accessor created ds['a'].test # DataArray accessor created var = ds['a'] var.test # DataArray accessor created var.test # ``` #### Expected Output Based on the xarray [accessor documentation](http://xarray.pydata.org/en/stable/internals.html#extending-xarray) I would have assumed that the accessor would stick around on the same DataArray object for the life of the data. My guess is that `Dataset.__getitem__` is recreating the `DataArray` every time from the underlying `Variable` object which means `ds['a'] is not ds['a']`? #### Problem Description I'm currently working on an accessor for a new package called `geoxarray` to address the issues talked about in #2288. One of the cases I'm trying to handle is a NetCDF file with CR standard `grid_mapping` variables. This means that the easiest way to set a CRS object for all variables in a Dataset is to have the Dataset accessor use the accessor of every DataArray (to cache a `_crs` property). However, with the way things are working doing something like the below won't work: ```python ds.geo.apply_grid_mapping() ds['Rad'].geo.crs # this is current `None` because the DataArray was recreated ``` #### Output of ``xr.show_versions()``
INSTALLED VERSIONS ------------------ commit: None python: 3.7.3 | packaged by conda-forge | (default, Jul 1 2019, 14:38:56) [Clang 4.0.1 (tags/RELEASE_401/final)] python-bits: 64 OS: Darwin OS-release: 18.6.0 machine: x86_64 processor: i386 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: en_US.UTF-8 libhdf5: 1.10.5 libnetcdf: 4.6.2 xarray: 0.12.3 pandas: 0.25.0 numpy: 1.17.0 scipy: None netCDF4: 1.5.1.2 pydap: None h5netcdf: None h5py: 2.9.0 Nio: None zarr: None cftime: 1.0.3.4 nc_time_axis: None PseudoNetCDF: None rasterio: None cfgrib: None iris: None bottleneck: None dask: 2.2.0 distributed: 2.2.0 matplotlib: None cartopy: None seaborn: None numbagg: None setuptools: 41.0.1 pip: 19.2.2 conda: None pytest: None IPython: 7.7.0 sphinx: None
","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/3205/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed,13221727,issue 446722089,MDU6SXNzdWU0NDY3MjIwODk=,2976,Confusing handling of NetCDF coordinates,1828519,closed,0,,,6,2019-05-21T16:47:47Z,2019-05-21T23:36:22Z,2019-05-21T23:36:21Z,CONTRIBUTOR,,,,"#### Code Sample, a copy-pastable example if possible I'm currently trying to figure out why some coordinates of a netcdf file are included in the resulting Dataset object and why some are not. I've tried looking at the `open_dataset` source code and think I'm kind of stuck. To reproduce: Download GOES-16 ABI L1b NetCDF file from (click the link to have google redirect you to the actual file): https://storage.cloud.google.com/gcp-public-data-goes-16/ABI-L1b-RadC/2019/140/18/OR_ABI-L1b-RadC-M6C01_G16_s20191401801336_e20191401804109_c20191401804156.nc?_ga=2.44800740.-1513329882.1547344783 ```python import xarray as xr a = xr.open_dataset('OR_ABI-L1b-RadC-M6C01_G16_s20191401801336_e20191401804109_c20191401804156.nc') print(a.coords) ``` Results in: ``` In [5]: a['Rad'].coords Out[5]: Coordinates: t datetime64[ns] ... * y (y) float32 0.151844 0.151788 0.151732 ... -0.151788 -0.151844 * x (x) float32 -0.151844 -0.151788 -0.151732 ... 0.151788 0.151844 y_image float32 ... x_image float32 ... ``` Even though `ncdump -h .nc` shows: ``` variables: short Rad(y, x) ; ... Rad:coordinates = ""band_id band_wavelength t y x"" ; ... ``` and: ``` a.coords ``` shows: ``` Coordinates: t datetime64[ns] ... * y (y) float32 0.151844 0.151788 ... -0.151844 * x (x) float32 -0.151844 -0.151788 ... 0.151844 y_image float32 ... x_image float32 ... band_id (band) int8 ... band_wavelength (band) float32 ... t_star_look (num_star_looks) datetime64[ns] ... band_wavelength_star_look (num_star_looks) float32 ... ``` #### Problem description I would have expected the `'Rad'` variable/DataArray to include the `band_id` and `band_wavelength` coordinate variables but **not** the `x_image` and `y_image` variables since they are not listed in the NetCDF variables `coordinates` attribute. Can someone summarize the rules on how xarray came up with these coordinates? Or is this a bug? #### Output of ``xr.show_versions()``
``` INSTALLED VERSIONS ------------------ commit: None python: 3.6.7 | packaged by conda-forge | (default, Feb 28 2019, 02:16:08) [GCC 4.2.1 Compatible Clang 4.0.1 (tags/RELEASE_401/final)] python-bits: 64 OS: Darwin OS-release: 18.5.0 machine: x86_64 processor: i386 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: en_US.UTF-8 libhdf5: 1.10.4 libnetcdf: 4.6.2 xarray: 0.12.1 pandas: 0.24.2 numpy: 1.14.3 scipy: 1.2.1 netCDF4: 1.5.0.1 pydap: None h5netcdf: 0.7.1 h5py: 2.9.0 Nio: None zarr: 2.3.1 cftime: 1.0.3.4 nc_time_axis: None PseudonetCDF: None rasterio: 1.0.22 cfgrib: None iris: None bottleneck: 1.2.1 dask: 1.1.5 distributed: 1.26.1 matplotlib: 3.0.3 cartopy: 0.17.0 seaborn: None setuptools: 41.0.0 pip: 19.0.3 conda: None pytest: 4.4.0 IPython: 7.4.0 sphinx: 2.0.1 ```
","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/2976/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed,13221727,issue 384449698,MDU6SXNzdWUzODQ0NDk2OTg=,2576,When is transpose not lazy?,1828519,closed,0,,,1,2018-11-26T18:10:36Z,2019-03-12T15:01:18Z,2019-03-12T15:01:18Z,CONTRIBUTOR,,,,"#### Code Sample, a copy-pastable example if possible ```python import xarray as xr import dask.array as da from dask.diagnostics import ProgressBar a = xr.DataArray(da.zeros((3, 5, 5), chunks=(1, 2, 2)), dims=('bands', 'y', 'x')) with ProgressBar(): b = a.transpose('y', 'x', 'bands') # dask does not show a progress bar due to no computation with ProgressBar(): b = a.transpose('y', 'x', 'bands') b.compute() # dask computes the array (since we told it to) and we see a progress bar ``` #### Question The [documentation for transpose](http://xarray.pydata.org/en/stable/generated/xarray.DataArray.transpose.html) says that it is not lazy. Is this only in certain situations? By not lazy does it mean that when the data is computed that the transpose task will require all data to be loaded at once (one large chunk) or does it mean that the transpose operation will immediately compute the transposed array? My test above does not seem to compute the data when `transpose` is called. Or is the documentation just outdated?","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/2576/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed,13221727,issue