id,node_id,number,title,user,state,locked,assignee,milestone,comments,created_at,updated_at,closed_at,author_association,active_lock_reason,draft,pull_request,body,reactions,performed_via_github_app,state_reason,repo,type 2239835092,PR_kwDOAMm_X85seXWW,8932,FIX: use str dtype without size information,5821660,closed,0,,,11,2024-04-12T10:59:45Z,2024-04-15T19:43:22Z,2024-04-13T12:25:48Z,MEMBER,,0,pydata/xarray/pulls/8932,"Aims to resolve parts of #8844. ```python xarray/tests/test_accessor_str.py::test_case_str: AssertionError: assert dtype(' - [ ] Closes #xxxx - [ ] Tests added - [ ] User visible changes (including notable bug fixes) are documented in `whats-new.rst` ","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/8932/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,pull 2205598061,PR_kwDOAMm_X85qplfG,8874,warn and return bytes undecoded in case of UnicodeDecodeError in h5netcdf-backend,5821660,closed,0,,,0,2024-03-25T12:06:07Z,2024-03-26T15:08:40Z,2024-03-26T15:08:40Z,MEMBER,,0,pydata/xarray/pulls/8874," - [x] Closes #5563 - [x] Tests added - [x] User visible changes (including notable bug fixes) are documented in `whats-new.rst` ","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/8874/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,pull 2202068638,PR_kwDOAMm_X85qdzHk,8865,numpy 2.0 copy-keyword and trapz vs trapezoid,5821660,closed,0,,,2,2024-03-22T09:29:27Z,2024-03-22T15:30:11Z,2024-03-22T15:30:11Z,MEMBER,,0,pydata/xarray/pulls/8865," This fixes two issues with numpy 2.0 (see #8844) - [x] User visible changes (including notable bug fixes) are documented in `whats-new.rst` ","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/8865/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,pull 2194276625,PR_kwDOAMm_X85qDXOy,8851,FIX: adapt handling of copy keyword argument in scipy backend for numpy >= 2.0dev,5821660,closed,0,,,0,2024-03-19T07:56:12Z,2024-03-20T14:54:47Z,2024-03-20T14:47:45Z,MEMBER,,0,pydata/xarray/pulls/8851," - [x] User visible changes (including notable bug fixes) are documented in `whats-new.rst` ","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/8851/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,pull 2194414193,PR_kwDOAMm_X85qD0hP,8852,FIX: do not cast _FillValue/missing_value in CFMaskCoder if _Unsigned is provided,5821660,closed,0,,,0,2024-03-19T09:10:11Z,2024-03-19T14:35:26Z,2024-03-19T14:25:37Z,MEMBER,,0,pydata/xarray/pulls/8852," - [x] User visible changes (including notable bug fixes) are documented in `whats-new.rst` ","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/8852/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,pull 2120261196,PR_kwDOAMm_X85mHZDI,8713,correctly encode/decode _FillValues/missing_values/dtypes for packed data,5821660,closed,0,,,10,2024-02-06T08:51:47Z,2024-03-15T16:31:05Z,2024-03-15T16:31:04Z,MEMBER,,0,pydata/xarray/pulls/8713," - [x] Closes #7691 - [x] Closes #5597 - [x] Closes #2304 - [x] Tests added - [x] User visible changes (including notable bug fixes) are documented in `whats-new.rst` This resurrects some of #7654. It takes special care of correctly handling dtypes when encoding/decoding packed data","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/8713/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,pull 2066854721,PR_kwDOAMm_X85jSUBD,8591,Remove xfails related to SerializeableLock,5821660,closed,0,,,1,2024-01-05T07:18:47Z,2024-01-05T16:12:52Z,2024-01-05T16:12:52Z,MEMBER,,0,pydata/xarray/pulls/8591,That's a follow-up to #8571.,"{""url"": ""https://api.github.com/repos/pydata/xarray/issues/8591/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,pull 2053560589,PR_kwDOAMm_X85iodqD,8571,"ENH: vendor SerializableLock from dask and use as default backend lock, adapt tests",5821660,closed,0,,,4,2023-12-22T08:22:40Z,2024-01-04T06:41:39Z,2024-01-04T06:41:36Z,MEMBER,,0,pydata/xarray/pulls/8571," - [x] Closes #8442 - [x] Tests added - [x] User visible changes (including notable bug fixes) are documented in `whats-new.rst` - [ ] New functions/methods are listed in `api.rst` ","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/8571/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,pull 2043626960,PR_kwDOAMm_X85iGwKH,8552,FIX: reverse index output of bottleneck move_argmax/move_argmin functions,5821660,closed,0,,,0,2023-12-15T12:16:28Z,2023-12-21T06:15:08Z,2023-12-21T02:08:00Z,MEMBER,,0,pydata/xarray/pulls/8552,"- reverse index output of bottleneck move_argmax/move_argmin functions - add move_argmax/move_argmin to bottleneck tests - [x] Closes #8541 - [x] Tests added - [x] User visible changes (including notable bug fixes) are documented in `whats-new.rst` ","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/8552/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,pull 1676309093,PR_kwDOAMm_X85Ovv30,7771,"implement scale_factor/add_offset CF conformance test, add and align tests",5821660,closed,0,,,3,2023-04-20T09:03:06Z,2023-11-17T20:59:38Z,2023-11-17T20:59:38Z,MEMBER,,1,pydata/xarray/pulls/7771," - [ ] Closes #xxxx - [x] Tests added - [ ] User visible changes (including notable bug fixes) are documented in `whats-new.rst` - [ ] New functions/methods are listed in `api.rst` This is adding conformance tests for `scale_factor`/`add_offset`/`dtype` when encoding/decoding [cf packed data](https://cfconventions.org/Data/cf-conventions/cf-conventions-1.10/cf-conventions.html#packed-data) with https://github.com/pydata/xarray/blob/c75ac8b7ab33be95f74d9e6f10b8173c68828751/xarray/coding/variables.py#L217. This doesn't include tests for `_FillValue`/`missing_value`/`valid_range` etc. which need a separate test. The idea behind is two-fold: - be strict when encoding to only write CF conforming data, raise Errors if non-conforming - be liberal when decoding, but issue warnings This is another step to move forward to finally fix issues like #2304 and a bunch more. ","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/7771/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,pull 1633623916,PR_kwDOAMm_X85Mh9TO,7654,cf-coding,5821660,closed,0,,,22,2023-03-21T10:34:44Z,2023-11-17T20:57:44Z,2023-11-17T20:57:44Z,MEMBER,,1,pydata/xarray/pulls/7654," - [x] xref #7652 - [x] Tests added - [ ] User visible changes (including notable bug fixes) are documented in `whats-new.rst` - [ ] New functions/methods are listed in `api.rst` - includes - implement coders - preserve boolean-dtype within encoding, adapt test - determine cf packed data type - ~transform numpy object-dtype strings (vlen) to numpy unicode strings~ This should also fix: - Closes https://github.com/pydata/xarray/issues/7691 - Closes https://github.com/pydata/xarray/issues/2304 and possible other issues which have the float32/float64 issue","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/7654/reactions"", ""total_count"": 3, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 3, ""rocket"": 0, ""eyes"": 0}",,,13221727,pull 1722614601,PR_kwDOAMm_X85RLNla,7869,"preserve vlen string dtypes, allow vlen string fill_values",5821660,closed,0,,,7,2023-05-23T18:53:58Z,2023-11-17T15:26:00Z,2023-11-17T15:26:00Z,MEMBER,,0,pydata/xarray/pulls/7869," - [x] Closes (partly) #1647 - [x] Closes #7652 - [x] Closes #7868 - [x] Tests adapted - [x] User visible changes (including notable bug fixes) are documented in `whats-new.rst` - [ ] New functions/methods are listed in `api.rst` ","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/7869/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,pull 1929620947,PR_kwDOAMm_X85cFMJJ,8277,fix regression in time-like check when decoding masked data,5821660,closed,0,,,10,2023-10-06T07:46:03Z,2023-10-18T16:50:19Z,2023-10-18T16:50:19Z,MEMBER,,0,pydata/xarray/pulls/8277," - [x] Closes #8269 - [x] Tests added - [x] User visible changes (including notable bug fixes) are documented in `whats-new.rst` ","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/8277/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,pull 1913089793,PR_kwDOAMm_X85bNXAT,8232,[skip-ci] dev whats-new,5821660,closed,0,,,3,2023-09-26T09:27:58Z,2023-09-26T15:34:12Z,2023-09-26T15:24:50Z,MEMBER,,0,pydata/xarray/pulls/8232,,"{""url"": ""https://api.github.com/repos/pydata/xarray/issues/8232/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,pull 1910369528,PR_kwDOAMm_X85bEI-s,8229,Release 2023.09.0,5821660,closed,0,,,3,2023-09-24T20:30:43Z,2023-09-26T08:13:02Z,2023-09-26T08:12:45Z,MEMBER,,0,pydata/xarray/pulls/8229,"As requested in #8223 I'm preparing for Xarray release 2023.09.0. ToDo: - [x] Add final date - [x] Approve headline and other changes in `whats-new.rst` - [x] add plan-to-merge PR - [x] Closes #8223 - [x] User visible changes (including notable bug fixes) are documented in `whats-new.rst` ","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/8229/reactions"", ""total_count"": 3, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 3, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,pull 1899505570,PR_kwDOAMm_X85afxBh,8195,decode variable with mismatched coordinate attribute,5821660,closed,0,,,8,2023-09-16T17:11:14Z,2023-09-25T11:30:37Z,2023-09-25T11:30:34Z,MEMBER,,0,pydata/xarray/pulls/8195,"- decode variable with mismatched coordinate attribute - propagate `coordinates` to encoding in any case - [x] Closes #1809 - [x] Closes #5170 - [x] Tests added - [x] User visible changes (including notable bug fixes) are documented in `whats-new.rst` - [ ] New functions/methods are listed in `api.rst` ","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/8195/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,pull 1898420996,PR_kwDOAMm_X85acMrs,8189,test_interpolate_pd_compat with range of fill_value's,5821660,closed,0,,,1,2023-09-15T13:18:11Z,2023-09-24T15:28:34Z,2023-09-24T15:28:31Z,MEMBER,,0,pydata/xarray/pulls/8189,"Update: The enhanced test works well in Upstream-dev tests, Pandas 2.1.1 is available for safe merge. - [x] Closes #8146 - [x] Tests added - [x] User visible changes (including notable bug fixes) are documented in `whats-new.rst` - [ ] New functions/methods are listed in `api.rst` ","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/8189/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,pull 1881254873,I_kwDOAMm_X85wIavZ,8146,Add proper tests with fill_value for test_interpolate_pd_compat-tests,5821660,closed,0,,,2,2023-09-05T06:02:10Z,2023-09-24T15:28:32Z,2023-09-24T15:28:32Z,MEMBER,,,,"### What is your issue? In #8125 breaking tests in Upstream-CI was reported. Root cause is upstream #https://github.com/pandas-dev/pandas/issues/54920 which is tackled in https://github.com/pandas-dev/pandas/pull/54927. To get the tests working, the `fill_value`-kwarg has been removed in #8139. The tests didn't cover the use of `fill_value` for scipy based interpolators (the used np.nan is the default-value) completely. This issue is to keep track of adding proper tests for `fill_value`.","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/8146/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed,13221727,issue 1900923297,PR_kwDOAMm_X85akZg0,8201,override `units` for datetime64/timedelta64 variables to preserve integer dtype,5821660,closed,0,,,17,2023-09-18T13:17:19Z,2023-09-24T15:27:13Z,2023-09-24T15:05:26Z,MEMBER,,0,pydata/xarray/pulls/8201," - [x] Closes #1064 - [x] Tests added - [x] User visible changes (including notable bug fixes) are documented in `whats-new.rst` - [ ] New functions/methods are listed in `api.rst` ","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/8201/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,pull 1899413232,PR_kwDOAMm_X85affPA,8194,remove invalid statement from doc/user-guide/io.rst,5821660,closed,0,,,0,2023-09-16T12:01:42Z,2023-09-19T01:02:23Z,2023-09-19T01:02:23Z,MEMBER,,0,pydata/xarray/pulls/8194," - [x] Closes #6570 ","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/8194/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,pull 1700227455,PR_kwDOAMm_X85P_7oN,7827,Preserve nanosecond resolution when encoding/decoding times,5821660,closed,0,,,17,2023-05-08T13:08:34Z,2023-09-17T16:35:29Z,2023-09-17T08:15:26Z,MEMBER,,0,pydata/xarray/pulls/7827," - [x] Closes #7817, - [x] closes #7942, - [x] closes #7790, - [x] closes #6191 - [x] closes #7096 - [x] closes #1064 - [x] Tests added - [x] User visible changes (including notable bug fixes) are documented in `whats-new.rst` - [ ] New functions/methods are listed in `api.rst` Closes #7098 as not needed, but worked around to preserve the fast int-based timedelta calculation. ","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/7827/reactions"", ""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,pull 1696097756,I_kwDOAMm_X85lGGXc,7817,nanosecond precision lost when reading time data,5821660,closed,0,,,3,2023-05-04T14:06:17Z,2023-09-17T08:15:27Z,2023-09-17T08:15:27Z,MEMBER,,,,"### What happened? When reading nanosecond precision time data from netcdf the precision is lost. This happens because CFMaskCoder will convert the variable to floating point and insert ""NaN"". In CFDatetimeCoder the floating point is cast back to int64 to transform into datetime64. This casting is sometimes undefined, hence #7098. ### What did you expect to happen? Precision should be preserved. The transformation to floating point should be omitted. ### Minimal Complete Verifiable Example ```Python import xarray as xr import numpy as np import netCDF4 as nc import matplotlib.pyplot as plt # create time array and fillvalue min_ns = -9223372036854775808 max_ns = 9223372036854775807 cnt = 2000 time_arr = np.arange(min_ns, min_ns + cnt, dtype=np.int64).astype(""M8[ns]"") fill_value = np.datetime64(""1900-01-01"", ""ns"") # create ncfile with time with attached _FillValue with nc.Dataset(""test.nc"", mode=""w"") as ds: ds.createDimension(""x"", cnt) time = ds.createVariable(""time"", "" array([ 'NaT', 'NaT', 'NaT', ..., '1677-09-21T00:12:43.145226240', '1677-09-21T00:12:43.145226240', '1677-09-21T00:12:43.145226240'], dtype='datetime64[ns]') Dimensions without coordinates: x --- no decoding ---------------------- array([-9223372036854775808, -9223372036854775807, -9223372036854775806, ..., -9223372036854773811, -9223372036854773810, -9223372036854773809]) Dimensions without coordinates: x Attributes: _FillValue: -2208988800000000000 units: nanoseconds since 1970-01-01 --- no time decoding ---------------------- array([-9.22337204e+18, -9.22337204e+18, -9.22337204e+18, ..., -9.22337204e+18, -9.22337204e+18, -9.22337204e+18]) Dimensions without coordinates: x Attributes: units: nanoseconds since 1970-01-01 --- no masking ------------------------------ array([ 'NaT', '1677-09-21T00:12:43.145224193', '1677-09-21T00:12:43.145224194', ..., '1677-09-21T00:12:43.145226189', '1677-09-21T00:12:43.145226190', '1677-09-21T00:12:43.145226191'], dtype='datetime64[ns]') Dimensions without coordinates: x Attributes: _FillValue: -2208988800000000000 ``` ### Anything else we need to know? Plot from above code: ![time-fillval](https://user-images.githubusercontent.com/5821660/236230652-449deced-7bf0-48a2-a366-ec3adcb2772d.png) Xref: #7098, https://github.com/pydata/xarray/issues/7790#issuecomment-1531050846 ### Environment
INSTALLED VERSIONS ------------------ commit: None python: 3.11.0 | packaged by conda-forge | (main, Jan 14 2023, 12:27:40) [GCC 11.3.0] python-bits: 64 OS: Linux OS-release: 5.14.21-150400.24.60-default machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: de_DE.UTF-8 LOCALE: ('de_DE', 'UTF-8') libhdf5: 1.14.0 libnetcdf: 4.9.2 xarray: 2023.4.2 pandas: 2.0.1 numpy: 1.24.2 scipy: 1.10.1 netCDF4: 1.6.3 pydap: None h5netcdf: 1.1.0 h5py: 3.8.0 Nio: None zarr: 2.14.2 cftime: 1.6.2 nc_time_axis: None PseudoNetCDF: None iris: None bottleneck: 1.3.7 dask: 2023.3.1 distributed: 2023.3.1 matplotlib: 3.7.1 cartopy: 0.21.1 seaborn: None numbagg: None fsspec: 2023.3.0 cupy: 11.6.0 pint: 0.20.1 sparse: None flox: None numpy_groupies: None setuptools: 67.6.0 pip: 23.0.1 conda: None pytest: 7.2.2 mypy: None IPython: 8.11.0 sphinx: None
","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/7817/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed,13221727,issue 1897760813,PR_kwDOAMm_X85aZ9iD,8187,"FIX: use ""krogh"" as interpolator method-string instead of ""krog""",5821660,closed,0,,,0,2023-09-15T06:00:37Z,2023-09-16T15:55:37Z,2023-09-16T15:55:37Z,MEMBER,,0,pydata/xarray/pulls/8187,"While working on #8146 this came up. Aligning with upstream naming convention. - [ ] Closes #xxxx - [ ] Tests added - [ ] User visible changes (including notable bug fixes) are documented in `whats-new.rst` - [ ] New functions/methods are listed in `api.rst` ","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/8187/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,pull 1856627163,PR_kwDOAMm_X85YPpRy,8084,FIX: handle NaT values in dt-accessor,5821660,closed,0,,,10,2023-08-18T12:17:58Z,2023-09-14T13:57:40Z,2023-09-14T13:57:37Z,MEMBER,,0,pydata/xarray/pulls/8084," - [x] Closes #7928 - [ ] Tests added - [x] User visible changes (including notable bug fixes) are documented in `whats-new.rst`","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/8084/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,pull 1655569401,I_kwDOAMm_X85irfv5,7723,default fill_value not masked when read from file,5821660,closed,0,,,5,2023-04-05T12:54:53Z,2023-09-13T12:44:54Z,2023-09-13T12:44:54Z,MEMBER,,,,"### What happened? When reading a netcdf file wich has been created with fill_value=None (default) those data is not masked. If one is writing back to disk this manifests. ### What did you expect to happen? Values should be masked. There seems to be a simple solution: On read apply the netcdf default fill_value in the variables attributes before decoding if no `_FillValue` attribute is set. After decoding we could change that to `np.nan` for floating point types. ### Minimal Complete Verifiable Example ```Python import numpy as np import netCDF4 as nc import xarray as xr with nc.Dataset(""test-no-missing-01.nc"", mode=""w"") as ds: x = ds.createDimension(""x"", 5) test = ds.createVariable(""test"", ""f4"", (""x"",), fill_value=None) test[:4] = np.array([0.0, np.nan, 1.0, 8.0], dtype=""f4"") with nc.Dataset(""test-no-missing-01.nc"") as ds: print(ds[""test""]) print(ds[""test""][:]) with xr.open_dataset(""test-no-missing-01.nc"").load() as roundtrip: print(roundtrip) print(roundtrip[""test""].attrs) print(roundtrip[""test""].encoding) roundtrip.to_netcdf(""test-no-missing-02.nc"") with nc.Dataset(""test-no-missing-02.nc"") as ds: print(ds[""test""]) print(ds[""test""][:]) ``` ### MVCE confirmation - [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray. - [X] Complete example — the example is self-contained, including all data and the text of any traceback. - [X] Verifiable example — the example copy & pastes into an IPython prompt or [Binder notebook](https://mybinder.org/v2/gh/pydata/xarray/main?urlpath=lab/tree/doc/examples/blank_template.ipynb), returning the result. - [X] New issue — a search of GitHub Issues suggests this is not a duplicate. ### Relevant log output ```Python float32 test(x) unlimited dimensions: current shape = (5,) filling on, default _FillValue of 9.969209968386869e+36 used [0.0 nan 1.0 8.0 --] Dimensions: (x: 5) Dimensions without coordinates: x Data variables: test (x) float32 0.0 nan 1.0 8.0 9.969e+36 {} {'zlib': False, 'szip': False, 'zstd': False, 'bzip2': False, 'blosc': False, 'shuffle': False, 'complevel': 0, 'fletcher32': False, 'contiguous': True, 'chunksizes': None, 'source': 'test-no-missing-01.nc', 'original_shape': (5,), 'dtype': dtype('float32')} float32 test(x) _FillValue: nan unlimited dimensions: current shape = (5,) filling on [0.0 -- 1.0 8.0 9.969209968386869e+36] ``` ### Anything else we need to know? The issue is similar to #7722 but is more intricate, as now the status of certain data values change from masked to some netcdf specific default value. This is when only parts of the source dataset have been written to. Then the default fill_value get's delivered to the user but it is not backed by an `_FillValue` attribute. ### Environment
INSTALLED VERSIONS ------------------ commit: None python: 3.11.0 | packaged by conda-forge | (main, Jan 14 2023, 12:27:40) [GCC 11.3.0] python-bits: 64 OS: Linux OS-release: 5.14.21-150400.24.55-default machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: de_DE.UTF-8 LOCALE: ('de_DE', 'UTF-8') libhdf5: 1.14.0 libnetcdf: 4.9.2 xarray: 2023.3.0 pandas: 1.5.3 numpy: 1.24.2 scipy: 1.10.1 netCDF4: 1.6.3 pydap: None h5netcdf: 1.1.0 h5py: 3.8.0 Nio: None zarr: 2.14.2 cftime: 1.6.2 nc_time_axis: None PseudoNetCDF: None rasterio: None cfgrib: None iris: None bottleneck: None dask: 2023.3.1 distributed: 2023.3.1 matplotlib: None cartopy: None seaborn: None numbagg: None fsspec: 2023.3.0 cupy: 11.6.0 pint: 0.20.1 sparse: None flox: None numpy_groupies: None setuptools: 67.6.0 pip: 23.0.1 conda: None pytest: 7.2.2 mypy: None IPython: 8.11.0 sphinx: None
","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/7723/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed,13221727,issue 1782037699,PR_kwDOAMm_X85UTjwg,7953,ensure no forward slashes in names for HDF5-based backends,5821660,closed,0,,,0,2023-06-30T07:26:17Z,2023-07-06T15:11:50Z,2023-07-06T15:11:50Z,MEMBER,,0,pydata/xarray/pulls/7953," - [x] Closes #7943 - [x] Tests added - [x] User visible changes (including notable bug fixes) are documented in `whats-new.rst` - [ ] New functions/methods are listed in `api.rst` ","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/7953/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,pull 1655483374,I_kwDOAMm_X85irKvu,7722,Conflicting _FillValue and missing_value on write,5821660,open,0,,,3,2023-04-05T11:56:46Z,2023-06-21T13:23:16Z,,MEMBER,,,,"### What happened? see also #7191 If `missing_value` and `_FillValue` is an attribute of a DataArray it can't be written out to file if these two contradict: ```python ValueError: Variable 'test' has conflicting _FillValue (nan) and missing_value (1.0). Cannot encode data. ``` This happens, if `missing_value` is an attribute of a specific netCDF Dataset of an existing file. On read the `missing_value` will be masked with `np.nan` on the data and it will be preserved within `encoding`. On write, `_FillValue` will be added as attribute by xarray (if not available, at least for floating point types), too. So far so good. The error first manifests if you read back this file and try to write it again. There is no warning on the second read, that the two `_FillValue` and `missing_value` are differing. Only on the second write. ### What did you expect to happen? The file should be written on the second roundtrip. There are at least two solutions to this: 1. Mask `missing_value` on read and purge `missing_value` completely in favor of `_FillValue`. 2. Do not handle `missing_value` at all, but let the user take action. ### Minimal Complete Verifiable Example ```Python import numpy as np import netCDF4 as nc import xarray as xr with nc.Dataset(""test-no-fillval-01.nc"", mode=""w"") as ds: x = ds.createDimension(""x"", 4) test = ds.createVariable(""test"", ""f4"", (""x"",), fill_value=None) test.missing_value = 1. test.valid_min = 2. test.valid_max = 10. test[:] = np.array([0.0, np.nan, 1.0, 8.0], dtype=""f4"") with nc.Dataset(""test-no-fillval-01.nc"") as ds: print(ds[""test""]) print(ds[""test""][:]) with xr.open_dataset(""test-no-fillval-01.nc"").load() as roundtrip: print(roundtrip) print(roundtrip[""test""].attrs) print(roundtrip[""test""].encoding) roundtrip.to_netcdf(""test-no-fillval-02.nc"") with xr.open_dataset(""test-no-fillval-02.nc"").load() as roundtrip: print(roundtrip) print(roundtrip[""test""].attrs) print(roundtrip[""test""].encoding) roundtrip.to_netcdf(""test-no-fillval-03.nc"") ``` ### MVCE confirmation - [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray. - [X] Complete example — the example is self-contained, including all data and the text of any traceback. - [X] Verifiable example — the example copy & pastes into an IPython prompt or [Binder notebook](https://mybinder.org/v2/gh/pydata/xarray/main?urlpath=lab/tree/doc/examples/blank_template.ipynb), returning the result. - [X] New issue — a search of GitHub Issues suggests this is not a duplicate. ### Relevant log output ```Python float32 test(x) missing_value: 1.0 valid_min: 2.0 valid_max: 10.0 unlimited dimensions: current shape = (4,) filling on, default _FillValue of 9.969209968386869e+36 used Dimensions: (x: 4) Dimensions without coordinates: x Data variables: test (x) float32 0.0 nan nan 8.0 {'valid_min': 2.0, 'valid_max': 10.0} {'zlib': False, 'szip': False, 'zstd': False, 'bzip2': False, 'blosc': False, 'shuffle': False, 'complevel': 0, 'fletcher32': False, 'contiguous': True, 'chunksizes': None, 'source': 'test-no-fillval-01.nc', 'original_shape': (4,), 'dtype': dtype('float32'), 'missing_value': 1.0} Dimensions: (x: 4) Dimensions without coordinates: x Data variables: test (x) float32 0.0 nan nan 8.0 {'valid_min': 2.0, 'valid_max': 10.0} {'zlib': False, 'szip': False, 'zstd': False, 'bzip2': False, 'blosc': False, 'shuffle': False, 'complevel': 0, 'fletcher32': False, 'contiguous': True, 'chunksizes': None, 'source': 'test-no-fillval-02.nc', 'original_shape': (4,), 'dtype': dtype('float32'), 'missing_value': 1.0, '_FillValue': nan} File /home/kai/miniconda/envs/xarray_311/lib/python3.11/site-packages/xarray/coding/variables.py:167, in CFMaskCoder.encode(self, variable, name) 160 mv = encoding.get(""missing_value"") 162 if ( 163 fv is not None 164 and mv is not None 165 and not duck_array_ops.allclose_or_equiv(fv, mv) 166 ): --> 167 raise ValueError( 168 f""Variable {name!r} has conflicting _FillValue ({fv}) and missing_value ({mv}). Cannot encode data."" 169 ) 171 if fv is not None: 172 # Ensure _FillValue is cast to same dtype as data's 173 encoding[""_FillValue""] = dtype.type(fv) ValueError: Variable 'test' has conflicting _FillValue (nan) and missing_value (1.0). Cannot encode data. ``` ### Anything else we need to know? The adding of `_FillValue` on write happens here: https://github.com/pydata/xarray/blob/d4db16699f30ad1dc3e6861601247abf4ac96567/xarray/conventions.py#L300 https://github.com/pydata/xarray/blob/d4db16699f30ad1dc3e6861601247abf4ac96567/xarray/conventions.py#L144-L152 ### Environment
INSTALLED VERSIONS ------------------ commit: None python: 3.11.0 | packaged by conda-forge | (main, Jan 14 2023, 12:27:40) [GCC 11.3.0] python-bits: 64 OS: Linux OS-release: 5.14.21-150400.24.55-default machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: de_DE.UTF-8 LOCALE: ('de_DE', 'UTF-8') libhdf5: 1.14.0 libnetcdf: 4.9.2 xarray: 2023.3.0 pandas: 1.5.3 numpy: 1.24.2 scipy: 1.10.1 netCDF4: 1.6.3 pydap: None h5netcdf: 1.1.0 h5py: 3.8.0 Nio: None zarr: 2.14.2 cftime: 1.6.2 nc_time_axis: None PseudoNetCDF: None rasterio: None cfgrib: None iris: None bottleneck: None dask: 2023.3.1 distributed: 2023.3.1 matplotlib: None cartopy: None seaborn: None numbagg: None fsspec: 2023.3.0 cupy: 11.6.0 pint: 0.20.1 sparse: None flox: None numpy_groupies: None setuptools: 67.6.0 pip: 23.0.1 conda: None pytest: 7.2.2 mypy: None IPython: 8.11.0 sphinx: None
","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/7722/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,issue 1039274574,PR_kwDOAMm_X84t1e28,5915,Add wradlib to ecosystem in docs,5821660,closed,0,,,3,2021-10-29T07:54:21Z,2023-05-25T07:24:07Z,2021-10-29T11:29:18Z,MEMBER,,0,pydata/xarray/pulls/5915," - [ ] Closes #xxxx - [ ] Tests added - [ ] Passes `pre-commit run --all-files` - [ ] User visible changes (including notable bug fixes) are documented in `whats-new.rst` - [ ] New functions/methods are listed in `api.rst` ","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/5915/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,pull 741341141,MDExOlB1bGxSZXF1ZXN0NTE5NjcyMTI5,4576,Fix: make copy of dask_gufunc_kwargs before changing content,5821660,closed,0,,,1,2020-11-12T07:07:06Z,2023-05-25T07:22:42Z,2020-11-12T18:30:43Z,MEMBER,,0,pydata/xarray/pulls/4576," - [x] Fixes downstream https://github.com/pangeo-data/xESMF/issues/36 - [x] Tests added - [x] Passes `isort . && black . && mypy . && flake8` - [x] User visible changes (including notable bug fixes) are documented in `whats-new.rst` - [ ] New functions/methods are listed in `api.rst` ","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/4576/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,pull 591101988,MDExOlB1bGxSZXF1ZXN0Mzk2MzI5Mzkx,3922,FIX: correct dask array handling in _calc_idxminmax,5821660,closed,0,,,35,2020-03-31T13:18:35Z,2023-05-25T07:22:40Z,2020-05-13T15:51:44Z,MEMBER,,0,pydata/xarray/pulls/3922,"Fixes dask handling for implementation in #3871. - [ ] Closes #xxxx - [x] Tests added - [x] Passes `isort -rc . && black . && mypy . && flake8` - [x] Fully documented, including `whats-new.rst` for all changes and `api.rst` for new API ","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/3922/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,pull 1508009922,PR_kwDOAMm_X85GDGpY,7400,Fill missing data_vars during concat by reindexing,5821660,closed,0,,,22,2022-12-22T14:41:56Z,2023-05-25T07:21:20Z,2023-01-20T16:04:26Z,MEMBER,,0,pydata/xarray/pulls/7400," - [x] Closes #508, - [x] Closes #3545 - [x] Tests added - [x] User visible changes (including notable bug fixes) are documented in `whats-new.rst` - [ ] New functions/methods are listed in `api.rst` This is another attempt to solve #508. Took inspiration from #3545 by @scottcha. This follows along @dcherian's comment in the same above PR (https://github.com/pydata/xarray/pull/3545#issuecomment-586346637). Update: After review the variable order is estimated by order of appearance in the list of datasets. That keeps full backwards compatibility and is deterministic. Thanks @dcherian and @keewis for the suggestions. ","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/7400/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,pull 1655000231,PR_kwDOAMm_X85NpOIk,7720,preserve boolean dtype in encoding,5821660,closed,0,,,2,2023-04-05T06:11:40Z,2023-05-25T07:20:57Z,2023-04-10T09:26:26Z,MEMBER,,0,pydata/xarray/pulls/7720," - [x] Closes #4826 - [x] Tests added - [x] User visible changes (including notable bug fixes) are documented in `whats-new.rst` Split out of #7654. Follow up to #7719. Needs rebase after #7719 is merged. ","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/7720/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,pull 812655506,MDExOlB1bGxSZXF1ZXN0NTc2OTY5NTU1,4933,FIX: decode/encode `time_reference`,5821660,closed,0,,,3,2021-02-20T16:28:38Z,2023-05-25T07:19:49Z,2021-02-21T08:31:22Z,MEMBER,,0,pydata/xarray/pulls/4933,"This is a fix for #4927. - `time_reference` is overwritten with the actual time string inside `time_reference` before decoding, but only if a variable with the name exists. - `time_reference` name is kept in `encoding` (the name might change, eg. `reference_time`). - `time_reference` is added back to `units` attribute when encoding, to make roundtrips possible. - [x] Closes #4927 - [x] Tests added - [ ] Passes `pre-commit run --all-files` - [ ] User visible changes (including notable bug fixes) are documented in `whats-new.rst` - [ ] New functions/methods are listed in `api.rst` ","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/4933/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,pull 806334969,MDExOlB1bGxSZXF1ZXN0NTcxNzc0MTk3,4893,FIX: h5py>=3 string decoding,5821660,closed,0,,,8,2021-02-11T12:13:46Z,2023-05-25T07:12:39Z,2021-02-17T08:41:20Z,MEMBER,,0,pydata/xarray/pulls/4893,"- set `decode_vlen_strings=True` for h5netcdf backend - unpin h5py This is an attempt to align with `h5py=>3.0.0` string decoding changes. Now all strings are read as `bytes` objects for variable-length strings, or numpy bytes arrays ('S' dtypes) for fixed-length strings [1]. From `h5netcdf=0.10.0` kwarg `decode_vlen_strings` is available. This PR makes use of this to keep backwards compatibility with `h5py=2` and conformance with `netcdf4-python`. [1] https://docs.h5py.org/en/stable/strings.html - [x] Closes #4570 - [ ] Tests added - [ ] Passes `pre-commit run --all-files` - [x] User visible changes (including notable bug fixes) are documented in `whats-new.rst` - [ ] New functions/methods are listed in `api.rst` ","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/4893/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,pull 618081836,MDExOlB1bGxSZXF1ZXN0NDE3ODc1NDEx,4060,ENH: use `dask.array.apply_gufunc` in `xr.apply_ufunc` ,5821660,closed,0,,,63,2020-05-14T09:38:15Z,2023-05-25T07:12:03Z,2020-08-19T06:57:56Z,MEMBER,,0,pydata/xarray/pulls/4060,"use `dask.array.apply_gufunc` in `xr.apply_ufunc` for multiple outputs when `dask='parallelized'`, add/fix tests - [x] Closes #1815, closes #4015 - [x] Tests added - [x] Passes `isort -rc . && black . && mypy . && flake8` - [x] Fully documented, including `whats-new.rst` for all changes and `api.rst` for new API Remaining Issues: - [ ] fitting name for current `dask_gufunc_kwargs` - [ ] rephrase [dask docs](https://github.com/pydata/xarray/blob/master/doc/dask.rst) to fit new behaviour - [ ] combine `output_core_dims` and `output_sizes`, eg. `xr.apply_ufunc(..., output_core_dims=[{""abc"": 2]])`","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/4060/reactions"", ""total_count"": 3, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 3, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,pull 688684690,MDExOlB1bGxSZXF1ZXN0NDc1ODg2MTQ0,4391,move kwarg's `output_sizes` and `meta` to `dask_gufunc_kwargs` for in…,5821660,closed,0,,,2,2020-08-30T09:20:05Z,2023-05-25T07:08:13Z,2020-08-30T16:37:52Z,MEMBER,,0,pydata/xarray/pulls/4391,"…ternal use of `apply_ufunc` (follow-up to #4060, fixes #4385) - [x] Closes #4385 - [ ] Tests added - [x] Passes `isort . && black . && mypy . && flake8` - [ ] User visible changes (including notable bug fixes) are documented in `whats-new.rst` - [ ] New functions/methods are listed in `api.rst` ","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/4391/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,pull 688993462,MDExOlB1bGxSZXF1ZXN0NDc2MTI0OTk4,4392,FIX: handle dask ValueErrors in `apply_ufunc` (set ``allow_rechunk=True``),5821660,closed,0,,,3,2020-08-31T07:40:51Z,2023-05-25T07:07:13Z,2020-09-09T19:00:17Z,MEMBER,,0,pydata/xarray/pulls/4392,"follow-up on #4060 - [x] Closes #4372 - [x] Tests added - [x] Passes `isort . && black . && mypy . && flake8` ","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/4392/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,pull 560219049,MDExOlB1bGxSZXF1ZXN0MzcxMjQ2ODEz,3753,Add new h5netcdf backend phony_dims kwarg,5821660,closed,0,,,4,2020-02-05T08:40:06Z,2023-05-25T07:05:23Z,2020-02-23T19:41:14Z,MEMBER,,0,pydata/xarray/pulls/3753,"This adds support for the new `phony_dims` keyword for the h5netcdf backend which is available from `h5netcdf` version 0.8.0. - [ ] Closes #xxxx - [ ] Tests added - [x] Passes `isort -rc . && black . && mypy . && flake8` - [x] Fully documented, including `whats-new.rst` for all changes and `api.rst` for new API ","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/3753/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,pull 1654988876,PR_kwDOAMm_X85NpL0W,7719,Implement more Variable Coders,5821660,closed,0,,,3,2023-04-05T06:01:09Z,2023-04-07T06:27:34Z,2023-04-07T03:57:50Z,MEMBER,,0,pydata/xarray/pulls/7719," - [x] User visible changes (including notable bug fixes) are documented in `whats-new.rst` This PR implements remaining functionality in `encode_cf_variable`/`decode_cf_variable` as `VariableCoders` (as left as todo by @shoyer). It adapts the needed tests and adds some typing (thanks @Illviljan). Otherwise there are no changes in function to code. It splits out of #7654 which turns out to be hard to follow (at least for me). ","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/7719/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,pull 700839138,MDExOlB1bGxSZXF1ZXN0NDg2Mzk2NjMz,4419,"preserve original dimension, coordinate and variable order in ``concat``",5821660,closed,0,,,3,2020-09-14T07:13:33Z,2022-10-09T13:05:14Z,2020-09-19T03:01:28Z,MEMBER,,0,pydata/xarray/pulls/4419," The problem is that the `result_vars`-dict is filled in two separate loops, handling variables and coordinates (AFAICT). This can change the ordering of dimensions and coordinates depending over which dimensions is concatenated. This fix hooks into the second loop and reinsert the variable in question at the appropriate position by popping it out of the `result_vars`. There might be a more elegant solution to this. - [x] Closes #2811, closes #4072 - [x] Tests added - [x] Passes `isort . && black . && mypy . && flake8` - [x] User visible changes (including notable bug fixes) are documented in `whats-new.rst` - [ ] New functions/methods are listed in `api.rst` ","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/4419/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,pull 1048309254,PR_kwDOAMm_X84uRTWS,5959,"Fix module name retrieval in `backend.plugins.remove_duplicates()`, plugin tests",5821660,closed,0,,,12,2021-11-09T08:14:24Z,2022-02-16T21:31:14Z,2021-11-15T17:33:43Z,MEMBER,,0,pydata/xarray/pulls/5959," - fix regression in `backend.plugins.remove_duplicates()` - switch `pkg_resources` -> `importlib` in `tests/test_plugins.py` - [x] Closes #5944 - [x] Tests fixed (switch pkg_resources -> importlib) - [ ] Tests added - [x] Passes `pre-commit run --all-files` - [x] User visible changes (including notable bug fixes) are documented in `whats-new.rst` - [ ] New functions/methods are listed in `api.rst` ","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/5959/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,pull 1116080396,PR_kwDOAMm_X84xqznc,6200,MNT: prepare h5netcdf backend for (coming) change in dimension handling,5821660,closed,0,,,10,2022-01-27T11:11:57Z,2022-01-29T13:20:01Z,2022-01-29T07:55:01Z,MEMBER,,0,pydata/xarray/pulls/6200,"- [x] Closes #6197 ","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/6200/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,pull 547925778,MDU6SXNzdWU1NDc5MjU3Nzg=,3680,broken output of `find_root_and_group` for h5netcdf,5821660,closed,0,,,4,2020-01-10T08:02:06Z,2022-01-12T08:23:37Z,2020-01-23T06:36:25Z,MEMBER,,,,"#### MCVE Code Sample ```python # Your code here import netCDF4 import h5netcdf import xarray from xarray.backends.common import find_root_and_group root = netCDF4.Dataset(""test.nc"", ""w"", format=""NETCDF4"") test1 = root.createGroup(""test1"") test1.createGroup(""test2"") root.close() h5n = h5netcdf.File('test.nc') print(find_root_and_group(h5n['test1'])[1]) h5n.close() ``` This will output: `//test1//test1/test2` #### Expected Output `/test1/test2` #### Problem Description I stumbled over this while working on #3618. Although the function claims to retrieve root and group name of netCDF4/h5netcdf datasets the output for h5netcdf is broken. #### Output of ``xr.show_versions()``
INSTALLED VERSIONS ------------------ commit: None python: 3.8.0 | packaged by conda-forge | (default, Nov 22 2019, 05:03:59) [GCC 7.3.0] python-bits: 64 OS: Linux OS-release: 4.12.14-lp151.28.36-default machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: de_DE.UTF-8 LOCALE: de_DE.UTF-8 libhdf5: 1.10.5 libnetcdf: 4.7.3 xarray: 0.14.1+43.g3e9d9046.dirty pandas: 0.25.3 numpy: 1.17.3 scipy: 1.3.3 netCDF4: 1.5.3 pydap: None h5netcdf: 0.7.4 h5py: 2.10.0 Nio: None zarr: None cftime: 1.0.4.2 nc_time_axis: None PseudoNetCDF: None rasterio: None cfgrib: None iris: None bottleneck: None dask: 2.8.0 distributed: 2.9.0 matplotlib: 3.1.2 cartopy: 0.17.0 seaborn: None numbagg: None setuptools: 42.0.2.post20191201 pip: 19.3.1 conda: None pytest: 5.3.1 IPython: 7.10.2 sphinx: 2.3.1
","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/3680/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed,13221727,issue 811933067,MDU6SXNzdWU4MTE5MzMwNjc=,4927,Times not decoding due to time_reference being in another variable,5821660,closed,0,,,12,2021-02-19T11:11:47Z,2021-02-21T08:34:15Z,2021-02-21T08:34:15Z,MEMBER,,,," **What happened**: Decoding of times fails for NetCDF4 CfRadial2 style file due to `time_reference`-string is in another variable. **What you expected to happen**: `time_reference` should be extracted from the variable and time should be decoded properly. **Minimal Complete Verifiable Example**: ```python # create file with such feature time = np.arange(10) time_attrs = dict(standard_name=""time"", units=""seconds since time_reference"", calendar=""gregorian"") time_ref_attrs = dict(comments=""UTC reference date. Format follows ISO 8601 standard."") ds = xr.Dataset( data_vars=dict( time=([""time""], time, time_attrs), time_reference=([], '1970-01-01T00:00:00Z', time_ref_attrs), )) ds.to_netcdf(""test.nc"") # breaks with the error below with xr.open_dataset(""test.nc"") as ds: print(ds) # ad-hoc fix with xr.open_dataset(""test.nc"", decode_times=False) as ds: tr = ds.time.attrs[""units""].split("" "") nt = ds[tr[-1]].item() ds.time.attrs[""units""] = "" "".join([*tr[:-1], nt]) ds = xr.decode_cf(ds) print(ds) ``` ```python --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) ~/miniconda3/envs/h5py3/lib/python3.9/site-packages/xarray/coding/times.py in _decode_cf_datetime_dtype(data, units, calendar, use_cftime) 142 try: --> 143 result = decode_cf_datetime(example_value, units, calendar, use_cftime) 144 except Exception: ~/miniconda3/envs/h5py3/lib/python3.9/site-packages/xarray/coding/times.py in decode_cf_datetime(num_dates, units, calendar, use_cftime) 224 try: --> 225 dates = _decode_datetime_with_pandas(flat_num_dates, units, calendar) 226 except (KeyError, OutOfBoundsDatetime, OverflowError): ~/miniconda3/envs/h5py3/lib/python3.9/site-packages/xarray/coding/times.py in _decode_datetime_with_pandas(flat_num_dates, units, calendar) 174 --> 175 delta, ref_date = _unpack_netcdf_time_units(units) 176 delta = _netcdf_to_numpy_timeunit(delta) ~/miniconda3/envs/h5py3/lib/python3.9/site-packages/xarray/coding/times.py in _unpack_netcdf_time_units(units) 127 delta_units, ref_date = [s.strip() for s in matches.groups()] --> 128 ref_date = _ensure_padded_year(ref_date) 129 ~/miniconda3/envs/h5py3/lib/python3.9/site-packages/xarray/coding/times.py in _ensure_padded_year(ref_date) 102 matches_start_digits = re.match(r""(\d+)(.*)"", ref_date) --> 103 ref_year, everything_else = [s for s in matches_start_digits.groups()] 104 ref_date_padded = ""{:04d}{}"".format(int(ref_year), everything_else) AttributeError: 'NoneType' object has no attribute 'groups' During handling of the above exception, another exception occurred: ValueError Traceback (most recent call last) in 13 ds.to_netcdf(""test.nc"") 14 ---> 15 with xr.open_dataset(""test.nc"") as ds: 16 print(ds) 17 ~/miniconda3/envs/h5py3/lib/python3.9/site-packages/xarray/backends/api.py in open_dataset(filename_or_obj, group, decode_cf, mask_and_scale, decode_times, concat_characters, decode_coords, engine, chunks, lock, cache, drop_variables, backend_kwargs, use_cftime, decode_timedelta) 432 from . import apiv2 433 --> 434 return apiv2.open_dataset(**kwargs) 435 436 if mask_and_scale is None: ~/miniconda3/envs/h5py3/lib/python3.9/site-packages/xarray/backends/apiv2.py in open_dataset(filename_or_obj, engine, chunks, cache, decode_cf, mask_and_scale, decode_times, decode_timedelta, use_cftime, concat_characters, decode_coords, drop_variables, backend_kwargs, **kwargs) 266 267 overwrite_encoded_chunks = kwargs.pop(""overwrite_encoded_chunks"", None) --> 268 backend_ds = backend.open_dataset( 269 filename_or_obj, 270 drop_variables=drop_variables, ~/miniconda3/envs/h5py3/lib/python3.9/site-packages/xarray/backends/netCDF4_.py in open_dataset(self, filename_or_obj, mask_and_scale, decode_times, concat_characters, decode_coords, drop_variables, use_cftime, decode_timedelta, group, mode, format, clobber, diskless, persist, lock, autoclose) 557 store_entrypoint = StoreBackendEntrypoint() 558 with close_on_error(store): --> 559 ds = store_entrypoint.open_dataset( 560 store, 561 mask_and_scale=mask_and_scale, ~/miniconda3/envs/h5py3/lib/python3.9/site-packages/xarray/backends/store.py in open_dataset(self, store, mask_and_scale, decode_times, concat_characters, decode_coords, drop_variables, use_cftime, decode_timedelta) 23 encoding = store.get_encoding() 24 ---> 25 vars, attrs, coord_names = conventions.decode_cf_variables( 26 vars, 27 attrs, ~/miniconda3/envs/h5py3/lib/python3.9/site-packages/xarray/conventions.py in decode_cf_variables(variables, attributes, concat_characters, mask_and_scale, decode_times, decode_coords, drop_variables, use_cftime, decode_timedelta) 510 and stackable(v.dims[-1]) 511 ) --> 512 new_vars[k] = decode_cf_variable( 513 k, 514 v, ~/miniconda3/envs/h5py3/lib/python3.9/site-packages/xarray/conventions.py in decode_cf_variable(name, var, concat_characters, mask_and_scale, decode_times, decode_endianness, stack_char_dim, use_cftime, decode_timedelta) 358 var = times.CFTimedeltaCoder().decode(var, name=name) 359 if decode_times: --> 360 var = times.CFDatetimeCoder(use_cftime=use_cftime).decode(var, name=name) 361 362 dimensions, data, attributes, encoding = variables.unpack_for_decoding(var) ~/miniconda3/envs/h5py3/lib/python3.9/site-packages/xarray/coding/times.py in decode(self, variable, name) 515 units = pop_to(attrs, encoding, ""units"") 516 calendar = pop_to(attrs, encoding, ""calendar"") --> 517 dtype = _decode_cf_datetime_dtype(data, units, calendar, self.use_cftime) 518 transform = partial( 519 decode_cf_datetime, ~/miniconda3/envs/h5py3/lib/python3.9/site-packages/xarray/coding/times.py in _decode_cf_datetime_dtype(data, units, calendar, use_cftime) 151 ""if it is not installed."" 152 ) --> 153 raise ValueError(msg) 154 else: 155 dtype = getattr(result, ""dtype"", np.dtype(""object"")) ValueError: unable to decode time units 'seconds since time_reference' with ""calendar 'gregorian'"". Try opening your dataset with decode_times=False or installing cftime if it is not installed. ``` **Anything else we need to know?**: I've searched the issues to no avail, also internet search wasn't successful so far. Any pointers welcome. **Environment**:
Output of xr.show_versions() INSTALLED VERSIONS ------------------ commit: None python: 3.9.0 | packaged by conda-forge | (default, Oct 14 2020, 22:59:50) [GCC 7.5.0] python-bits: 64 OS: Linux OS-release: 5.8.0-43-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: de_DE.UTF-8 LOCALE: de_DE.UTF-8 libhdf5: 1.10.6 libnetcdf: 4.7.4 xarray: 0.16.3.dev124+g0a309e07.d20210218 pandas: 1.2.1 numpy: 1.20.0 scipy: 1.6.0 netCDF4: 1.5.4 pydap: None h5netcdf: 0.10.0 h5py: 3.1.0 Nio: None zarr: None cftime: 1.4.1 nc_time_axis: None PseudoNetCDF: None rasterio: None cfgrib: None iris: None bottleneck: 1.3.2 dask: 2.30.0 distributed: 2.30.1 matplotlib: 3.3.2 cartopy: 0.18.0 seaborn: None numbagg: None pint: None setuptools: 49.6.0.post20201009 pip: 20.2.4 conda: None pytest: 6.1.2 IPython: 7.19.0 sphinx: None
","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/4927/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed,13221727,issue 420930870,MDU6SXNzdWU0MjA5MzA4NzA=,2811,concat changes variable order,5821660,closed,0,,,18,2019-03-14T10:11:28Z,2020-09-19T03:01:28Z,2020-09-19T03:01:28Z,MEMBER,,,,"#### Code Sample, a copy-pastable example if possible A ""Minimal, Complete and Verifiable Example"" will make it much easier for maintainers to help you: http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports - Case 1: Creation of Dataset without Coordinates ```python data = np.zeros((2,3)) ds = xr.Dataset({'test': (['c', 'b'], data)}) print(ds.dims) ds2 = xr.concat([ds, ds], dim='c') print(ds2.dims) ``` yields (assumed correct) output of: ``` Frozen(SortedKeysDict({'c': 2, 'b': 3})) Frozen(SortedKeysDict({'c': 4, 'b': 3})) ``` - Case 2: Creation of Dataset with Coordinates ```python data = np.zeros((2,3)) ds = xr.Dataset({'test': (['c', 'b'], data)}, coords={'c': (['c'], np.arange(data.shape[0])), 'b': (['b'], np.arange(data.shape[1])),}) print(ds.dims) ds2 = xr.concat([ds, ds], dim='c') print(ds2.dims) ``` yields (assumed false) output of: ``` Frozen(SortedKeysDict({'c': 2, 'b': 3})) Frozen(SortedKeysDict({'b': 3, 'c': 4})) ``` #### Problem description `xr.concat` changes the dimension order for `.dims` as well as `.sizes` to an alphanumerically sorted representation. #### Expected Output `xr.concat` should not change the dimension order in any case. ``` Frozen(SortedKeysDict({'c': 2, 'b': 3})) Frozen(SortedKeysDict({'c': 4, 'b': 3})) ``` #### Output of ``xr.show_versions()``
INSTALLED VERSIONS ------------------ commit: None python: 3.7.1 | packaged by conda-forge | (default, Nov 13 2018, 18:33:04) [GCC 7.3.0] python-bits: 64 OS: Linux OS-release: 4.12.14-lp150.12.48-default machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: de_DE.UTF-8 LOCALE: de_DE.UTF-8 libhdf5: 1.10.4 libnetcdf: 4.6.2 xarray: 0.11.3 pandas: 0.24.1 numpy: 1.16.1 scipy: 1.2.0 netCDF4: 1.4.2 pydap: None h5netcdf: 0.6.2 h5py: 2.9.0 Nio: None zarr: None cftime: 1.0.3.4 PseudonetCDF: None rasterio: None cfgrib: None iris: None bottleneck: 1.2.1 cyordereddict: None dask: None distributed: None matplotlib: 3.0.2 cartopy: 0.17.0 seaborn: None setuptools: 40.8.0 pip: 19.0.2 conda: None pytest: 4.2.0 IPython: 7.2.0 sphinx: None
","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/2811/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed,13221727,issue 701764313,MDU6SXNzdWU3MDE3NjQzMTM=,4425,deepcopying variable raises `TypeError: h5py objects cannot be pickled` (Dataset.sortby),5821660,closed,0,,,5,2020-09-15T09:23:20Z,2020-09-18T22:31:09Z,2020-09-18T22:31:09Z,MEMBER,,,," **What happened**: When using `xr.open_dataset` with `H5NetCDFDataStore` and opened `h5py.File` handle `deepcopy` in `Dataset.sortby`/`align` leads to `TypeError: h5py objects cannot be pickled`. **What you expected to happen**: While applying `Dataset.sortby` no error should be raised. **Minimal Complete Verifiable Example**: ```python # create hdf5 file import h5py f = h5py.File('myfile.h5','w') dset = f.create_dataset(""data"", (360, 1000)) f.close() import h5netcdf import xarray as xr import numpy as np f = h5netcdf.File(""myfile.h5"", ""r"", phony_dims=""access"") s0 = xr.backends.H5NetCDFStore(f) ds = xr.open_dataset(s0, engine=""h5netcdf"", chunks=None) ds = ds.assign_coords({""phony_dim_0"": np.arange(ds.dims['phony_dim_0'], 0, -1)}) ds.sortby('phony_dim_0') ds.close() ```
Error Traceback ```python --------------------------------------------------------------------------- TypeError Traceback (most recent call last) in 5 ds = xr.open_dataset(s0, engine=""h5netcdf"", chunks=None) 6 ds = ds.assign_coords({""phony_dim_0"": np.arange(ds.dims['phony_dim_0'], 0, -1)}) ----> 7 ds.sortby('phony_dim_0') 8 ds.close() /home/kai/miniconda/envs/wradlib_devel/lib/python3.8/site-packages/xarray-0.16.1.dev86+g264fdb29-py3.8.egg/xarray/core/dataset.py in sortby(self, variables, ascending) 5293 variables = variables 5294 variables = [v if isinstance(v, DataArray) else self[v] for v in variables] -> 5295 aligned_vars = align(self, *variables, join=""left"") 5296 aligned_self = aligned_vars[0] 5297 aligned_other_vars = aligned_vars[1:] /home/kai/miniconda/envs/wradlib_devel/lib/python3.8/site-packages/xarray-0.16.1.dev86+g264fdb29-py3.8.egg/xarray/core/alignment.py in align(join, copy, indexes, exclude, fill_value, *objects) 336 if not valid_indexers: 337 # fast path for no reindexing necessary --> 338 new_obj = obj.copy(deep=copy) 339 else: 340 new_obj = obj.reindex(copy=copy, fill_value=fill_value, **valid_indexers) /home/kai/miniconda/envs/wradlib_devel/lib/python3.8/site-packages/xarray-0.16.1.dev86+g264fdb29-py3.8.egg/xarray/core/dataset.py in copy(self, deep, data) 1076 """""" 1077 if data is None: -> 1078 variables = {k: v.copy(deep=deep) for k, v in self._variables.items()} 1079 elif not utils.is_dict_like(data): 1080 raise ValueError(""Data must be dict-like"") /home/kai/miniconda/envs/wradlib_devel/lib/python3.8/site-packages/xarray-0.16.1.dev86+g264fdb29-py3.8.egg/xarray/core/dataset.py in (.0) 1076 """""" 1077 if data is None: -> 1078 variables = {k: v.copy(deep=deep) for k, v in self._variables.items()} 1079 elif not utils.is_dict_like(data): 1080 raise ValueError(""Data must be dict-like"") /home/kai/miniconda/envs/wradlib_devel/lib/python3.8/site-packages/xarray-0.16.1.dev86+g264fdb29-py3.8.egg/xarray/core/variable.py in copy(self, deep, data) 938 939 if deep: --> 940 data = copy.deepcopy(data) 941 942 else: /home/kai/miniconda/envs/wradlib_devel/lib/python3.8/copy.py in deepcopy(x, memo, _nil) 170 y = x 171 else: --> 172 y = _reconstruct(x, memo, *rv) 173 174 # If is its own copy, don't memoize. /home/kai/miniconda/envs/wradlib_devel/lib/python3.8/copy.py in _reconstruct(x, memo, func, args, state, listiter, dictiter, deepcopy) 268 if state is not None: 269 if deep: --> 270 state = deepcopy(state, memo) 271 if hasattr(y, '__setstate__'): 272 y.__setstate__(state) /home/kai/miniconda/envs/wradlib_devel/lib/python3.8/copy.py in deepcopy(x, memo, _nil) 144 copier = _deepcopy_dispatch.get(cls) 145 if copier is not None: --> 146 y = copier(x, memo) 147 else: 148 if issubclass(cls, type): /home/kai/miniconda/envs/wradlib_devel/lib/python3.8/copy.py in _deepcopy_tuple(x, memo, deepcopy) 208 209 def _deepcopy_tuple(x, memo, deepcopy=deepcopy): --> 210 y = [deepcopy(a, memo) for a in x] 211 # We're not going to put the tuple in the memo, but it's still important we 212 # check for it, in case the tuple contains recursive mutable structures. /home/kai/miniconda/envs/wradlib_devel/lib/python3.8/copy.py in (.0) 208 209 def _deepcopy_tuple(x, memo, deepcopy=deepcopy): --> 210 y = [deepcopy(a, memo) for a in x] 211 # We're not going to put the tuple in the memo, but it's still important we 212 # check for it, in case the tuple contains recursive mutable structures. /home/kai/miniconda/envs/wradlib_devel/lib/python3.8/copy.py in deepcopy(x, memo, _nil) 144 copier = _deepcopy_dispatch.get(cls) 145 if copier is not None: --> 146 y = copier(x, memo) 147 else: 148 if issubclass(cls, type): /home/kai/miniconda/envs/wradlib_devel/lib/python3.8/copy.py in _deepcopy_dict(x, memo, deepcopy) 228 memo[id(x)] = y 229 for key, value in x.items(): --> 230 y[deepcopy(key, memo)] = deepcopy(value, memo) 231 return y 232 d[dict] = _deepcopy_dict /home/kai/miniconda/envs/wradlib_devel/lib/python3.8/copy.py in deepcopy(x, memo, _nil) 170 y = x 171 else: --> 172 y = _reconstruct(x, memo, *rv) 173 174 # If is its own copy, don't memoize. /home/kai/miniconda/envs/wradlib_devel/lib/python3.8/copy.py in _reconstruct(x, memo, func, args, state, listiter, dictiter, deepcopy) 268 if state is not None: 269 if deep: --> 270 state = deepcopy(state, memo) 271 if hasattr(y, '__setstate__'): 272 y.__setstate__(state) /home/kai/miniconda/envs/wradlib_devel/lib/python3.8/copy.py in deepcopy(x, memo, _nil) 144 copier = _deepcopy_dispatch.get(cls) 145 if copier is not None: --> 146 y = copier(x, memo) 147 else: 148 if issubclass(cls, type): /home/kai/miniconda/envs/wradlib_devel/lib/python3.8/copy.py in _deepcopy_tuple(x, memo, deepcopy) 208 209 def _deepcopy_tuple(x, memo, deepcopy=deepcopy): --> 210 y = [deepcopy(a, memo) for a in x] 211 # We're not going to put the tuple in the memo, but it's still important we 212 # check for it, in case the tuple contains recursive mutable structures. /home/kai/miniconda/envs/wradlib_devel/lib/python3.8/copy.py in (.0) 208 209 def _deepcopy_tuple(x, memo, deepcopy=deepcopy): --> 210 y = [deepcopy(a, memo) for a in x] 211 # We're not going to put the tuple in the memo, but it's still important we 212 # check for it, in case the tuple contains recursive mutable structures. /home/kai/miniconda/envs/wradlib_devel/lib/python3.8/copy.py in deepcopy(x, memo, _nil) 144 copier = _deepcopy_dispatch.get(cls) 145 if copier is not None: --> 146 y = copier(x, memo) 147 else: 148 if issubclass(cls, type): /home/kai/miniconda/envs/wradlib_devel/lib/python3.8/copy.py in _deepcopy_dict(x, memo, deepcopy) 228 memo[id(x)] = y 229 for key, value in x.items(): --> 230 y[deepcopy(key, memo)] = deepcopy(value, memo) 231 return y 232 d[dict] = _deepcopy_dict /home/kai/miniconda/envs/wradlib_devel/lib/python3.8/copy.py in deepcopy(x, memo, _nil) 170 y = x 171 else: --> 172 y = _reconstruct(x, memo, *rv) 173 174 # If is its own copy, don't memoize. /home/kai/miniconda/envs/wradlib_devel/lib/python3.8/copy.py in _reconstruct(x, memo, func, args, state, listiter, dictiter, deepcopy) 268 if state is not None: 269 if deep: --> 270 state = deepcopy(state, memo) 271 if hasattr(y, '__setstate__'): 272 y.__setstate__(state) /home/kai/miniconda/envs/wradlib_devel/lib/python3.8/copy.py in deepcopy(x, memo, _nil) 144 copier = _deepcopy_dispatch.get(cls) 145 if copier is not None: --> 146 y = copier(x, memo) 147 else: 148 if issubclass(cls, type): /home/kai/miniconda/envs/wradlib_devel/lib/python3.8/copy.py in _deepcopy_tuple(x, memo, deepcopy) 208 209 def _deepcopy_tuple(x, memo, deepcopy=deepcopy): --> 210 y = [deepcopy(a, memo) for a in x] 211 # We're not going to put the tuple in the memo, but it's still important we 212 # check for it, in case the tuple contains recursive mutable structures. /home/kai/miniconda/envs/wradlib_devel/lib/python3.8/copy.py in (.0) 208 209 def _deepcopy_tuple(x, memo, deepcopy=deepcopy): --> 210 y = [deepcopy(a, memo) for a in x] 211 # We're not going to put the tuple in the memo, but it's still important we 212 # check for it, in case the tuple contains recursive mutable structures. /home/kai/miniconda/envs/wradlib_devel/lib/python3.8/copy.py in deepcopy(x, memo, _nil) 144 copier = _deepcopy_dispatch.get(cls) 145 if copier is not None: --> 146 y = copier(x, memo) 147 else: 148 if issubclass(cls, type): /home/kai/miniconda/envs/wradlib_devel/lib/python3.8/copy.py in _deepcopy_dict(x, memo, deepcopy) 228 memo[id(x)] = y 229 for key, value in x.items(): --> 230 y[deepcopy(key, memo)] = deepcopy(value, memo) 231 return y 232 d[dict] = _deepcopy_dict /home/kai/miniconda/envs/wradlib_devel/lib/python3.8/copy.py in deepcopy(x, memo, _nil) 170 y = x 171 else: --> 172 y = _reconstruct(x, memo, *rv) 173 174 # If is its own copy, don't memoize. /home/kai/miniconda/envs/wradlib_devel/lib/python3.8/copy.py in _reconstruct(x, memo, func, args, state, listiter, dictiter, deepcopy) 268 if state is not None: 269 if deep: --> 270 state = deepcopy(state, memo) 271 if hasattr(y, '__setstate__'): 272 y.__setstate__(state) /home/kai/miniconda/envs/wradlib_devel/lib/python3.8/copy.py in deepcopy(x, memo, _nil) 144 copier = _deepcopy_dispatch.get(cls) 145 if copier is not None: --> 146 y = copier(x, memo) 147 else: 148 if issubclass(cls, type): /home/kai/miniconda/envs/wradlib_devel/lib/python3.8/copy.py in _deepcopy_tuple(x, memo, deepcopy) 208 209 def _deepcopy_tuple(x, memo, deepcopy=deepcopy): --> 210 y = [deepcopy(a, memo) for a in x] 211 # We're not going to put the tuple in the memo, but it's still important we 212 # check for it, in case the tuple contains recursive mutable structures. /home/kai/miniconda/envs/wradlib_devel/lib/python3.8/copy.py in (.0) 208 209 def _deepcopy_tuple(x, memo, deepcopy=deepcopy): --> 210 y = [deepcopy(a, memo) for a in x] 211 # We're not going to put the tuple in the memo, but it's still important we 212 # check for it, in case the tuple contains recursive mutable structures. /home/kai/miniconda/envs/wradlib_devel/lib/python3.8/copy.py in deepcopy(x, memo, _nil) 144 copier = _deepcopy_dispatch.get(cls) 145 if copier is not None: --> 146 y = copier(x, memo) 147 else: 148 if issubclass(cls, type): /home/kai/miniconda/envs/wradlib_devel/lib/python3.8/copy.py in _deepcopy_dict(x, memo, deepcopy) 228 memo[id(x)] = y 229 for key, value in x.items(): --> 230 y[deepcopy(key, memo)] = deepcopy(value, memo) 231 return y 232 d[dict] = _deepcopy_dict /home/kai/miniconda/envs/wradlib_devel/lib/python3.8/copy.py in deepcopy(x, memo, _nil) 170 y = x 171 else: --> 172 y = _reconstruct(x, memo, *rv) 173 174 # If is its own copy, don't memoize. /home/kai/miniconda/envs/wradlib_devel/lib/python3.8/copy.py in _reconstruct(x, memo, func, args, state, listiter, dictiter, deepcopy) 268 if state is not None: 269 if deep: --> 270 state = deepcopy(state, memo) 271 if hasattr(y, '__setstate__'): 272 y.__setstate__(state) /home/kai/miniconda/envs/wradlib_devel/lib/python3.8/copy.py in deepcopy(x, memo, _nil) 144 copier = _deepcopy_dispatch.get(cls) 145 if copier is not None: --> 146 y = copier(x, memo) 147 else: 148 if issubclass(cls, type): /home/kai/miniconda/envs/wradlib_devel/lib/python3.8/copy.py in _deepcopy_tuple(x, memo, deepcopy) 208 209 def _deepcopy_tuple(x, memo, deepcopy=deepcopy): --> 210 y = [deepcopy(a, memo) for a in x] 211 # We're not going to put the tuple in the memo, but it's still important we 212 # check for it, in case the tuple contains recursive mutable structures. /home/kai/miniconda/envs/wradlib_devel/lib/python3.8/copy.py in (.0) 208 209 def _deepcopy_tuple(x, memo, deepcopy=deepcopy): --> 210 y = [deepcopy(a, memo) for a in x] 211 # We're not going to put the tuple in the memo, but it's still important we 212 # check for it, in case the tuple contains recursive mutable structures. /home/kai/miniconda/envs/wradlib_devel/lib/python3.8/copy.py in deepcopy(x, memo, _nil) 144 copier = _deepcopy_dispatch.get(cls) 145 if copier is not None: --> 146 y = copier(x, memo) 147 else: 148 if issubclass(cls, type): /home/kai/miniconda/envs/wradlib_devel/lib/python3.8/copy.py in _deepcopy_dict(x, memo, deepcopy) 228 memo[id(x)] = y 229 for key, value in x.items(): --> 230 y[deepcopy(key, memo)] = deepcopy(value, memo) 231 return y 232 d[dict] = _deepcopy_dict /home/kai/miniconda/envs/wradlib_devel/lib/python3.8/copy.py in deepcopy(x, memo, _nil) 170 y = x 171 else: --> 172 y = _reconstruct(x, memo, *rv) 173 174 # If is its own copy, don't memoize. /home/kai/miniconda/envs/wradlib_devel/lib/python3.8/copy.py in _reconstruct(x, memo, func, args, state, listiter, dictiter, deepcopy) 268 if state is not None: 269 if deep: --> 270 state = deepcopy(state, memo) 271 if hasattr(y, '__setstate__'): 272 y.__setstate__(state) /home/kai/miniconda/envs/wradlib_devel/lib/python3.8/copy.py in deepcopy(x, memo, _nil) 144 copier = _deepcopy_dispatch.get(cls) 145 if copier is not None: --> 146 y = copier(x, memo) 147 else: 148 if issubclass(cls, type): /home/kai/miniconda/envs/wradlib_devel/lib/python3.8/copy.py in _deepcopy_dict(x, memo, deepcopy) 228 memo[id(x)] = y 229 for key, value in x.items(): --> 230 y[deepcopy(key, memo)] = deepcopy(value, memo) 231 return y 232 d[dict] = _deepcopy_dict /home/kai/miniconda/envs/wradlib_devel/lib/python3.8/copy.py in deepcopy(x, memo, _nil) 170 y = x 171 else: --> 172 y = _reconstruct(x, memo, *rv) 173 174 # If is its own copy, don't memoize. /home/kai/miniconda/envs/wradlib_devel/lib/python3.8/copy.py in _reconstruct(x, memo, func, args, state, listiter, dictiter, deepcopy) 268 if state is not None: 269 if deep: --> 270 state = deepcopy(state, memo) 271 if hasattr(y, '__setstate__'): 272 y.__setstate__(state) /home/kai/miniconda/envs/wradlib_devel/lib/python3.8/copy.py in deepcopy(x, memo, _nil) 144 copier = _deepcopy_dispatch.get(cls) 145 if copier is not None: --> 146 y = copier(x, memo) 147 else: 148 if issubclass(cls, type): /home/kai/miniconda/envs/wradlib_devel/lib/python3.8/copy.py in _deepcopy_dict(x, memo, deepcopy) 228 memo[id(x)] = y 229 for key, value in x.items(): --> 230 y[deepcopy(key, memo)] = deepcopy(value, memo) 231 return y 232 d[dict] = _deepcopy_dict /home/kai/miniconda/envs/wradlib_devel/lib/python3.8/copy.py in deepcopy(x, memo, _nil) 159 reductor = getattr(x, ""__reduce_ex__"", None) 160 if reductor is not None: --> 161 rv = reductor(4) 162 else: 163 reductor = getattr(x, ""__reduce__"", None) /home/kai/miniconda/envs/wradlib_devel/lib/python3.8/site-packages/h5py/_hl/base.py in __getnewargs__(self) 306 limitations, look at the h5pickle project on PyPI. 307 """""" --> 308 raise TypeError(""h5py objects cannot be pickled"") 309 310 def __getstate__(self): TypeError: h5py objects cannot be pickled ```
**Anything else we need to know?**: When invoked with `chunks={}` it works as well as if the following code is used: ```python ds = xr.open_dataset('myfile.h5', group='/', engine='h5netcdf', backend_kwargs=dict(phony_dims='access')) ds = ds.assign_coords({""phony_dim_0"": np.arange(ds.dims['phony_dim_0'], 0, -1)}) ds.sortby('phony_dim_0') ds.close() ``` This was introduced by #4221, see https://github.com/pydata/xarray/blob/66ab0ae4f3aa3c461357a5a895405e81357796b1/xarray/core/variable.py#L939-L941 Before: ```python if deep and ( hasattr(data, ""__array_function__"") or isinstance(data, dask_array_type) or (not IS_NEP18_ACTIVE and isinstance(data, np.ndarray)) ): data = copy.deepcopy(data) ``` All three of the above tests return `False` in my case, so deepcopy should never be used here. **Environment**:
Output of xr.show_versions() INSTALLED VERSIONS ------------------ commit: None python: 3.8.5 | packaged by conda-forge | (default, Aug 29 2020, 01:22:49) [GCC 7.5.0] python-bits: 64 OS: Linux OS-release: 4.12.14-lp151.28.67-default machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: de_DE.UTF-8 LOCALE: de_DE.UTF-8 libhdf5: 1.10.6 libnetcdf: 4.7.4 xarray: 0.16.1.dev86+g264fdb29 pandas: 1.1.1 numpy: 1.19.1 scipy: 1.5.0 netCDF4: 1.5.4 pydap: None h5netcdf: 0.8.0 h5py: 2.10.0 Nio: None zarr: 2.4.0 cftime: 1.2.1 nc_time_axis: None PseudoNetCDF: None rasterio: None cfgrib: 0.9.8.4 iris: None bottleneck: 1.3.2 dask: 2.19.0 distributed: 2.25.0 matplotlib: 3.3.1 cartopy: 0.18.0 seaborn: None numbagg: None pint: None setuptools: 49.6.0.post20200814 pip: 20.2.2 conda: 4.8.3 pytest: 5.4.3 IPython: 7.18.1 sphinx: None
","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/4425/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed,13221727,issue 537576675,MDExOlB1bGxSZXF1ZXN0MzUyOTA2NjA4,3618,ENH: enable `H5NetCDFStore` to work with already open h5netcdf.File a…,5821660,closed,0,,,8,2019-12-13T14:23:10Z,2020-01-23T07:40:19Z,2020-01-20T05:46:14Z,MEMBER,,0,pydata/xarray/pulls/3618,"enable `H5NetCDFStore` to work with open h5netcdf.File and h5netcdf.Group objects, add test - [x] Tests added - [x] Passes `black . && mypy . && flake8` - [x] Fully documented, including `whats-new.rst` for all changes and `api.rst` for new API ","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/3618/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,pull