id,node_id,number,title,user,state,locked,assignee,milestone,comments,created_at,updated_at,closed_at,author_association,active_lock_reason,draft,pull_request,body,reactions,performed_via_github_app,state_reason,repo,type 1933496257,I_kwDOAMm_X85zPs_B,8288,New library for handling complex numbers in netCDF,1486942,closed,0,,,7,2023-10-09T17:21:20Z,2024-03-25T13:16:40Z,2024-03-25T13:16:40Z,CONTRIBUTOR,,,,"### Is your feature request related to a problem? Complex numbers are not natively handled by netCDF, and currently only the `h5netcdf` engine in xarray can handle them. However, it produces files unreadable by current versions of netCDF. There are also [many different conventions](https://github.com/PlasmaFAIR/nc-complex/blob/main/README.md#conventions-for-complex-numbers), and `h5netcdf` can only read a few of them. ### Describe the solution you'd like I've written a [drop-in extension to netCDF](https://github.com/PlasmaFAIR/nc-complex) that seamlessly handles complex numbers, and can read many different conventions. The Python API is built on top of netCDF4-python, so only a very simple change to xarray is needed to use it: ```diff modified xarray/backends/api.py @@ -115,7 +115,10 @@ def _get_default_engine_gz() -> Literal[""scipy""]: def _get_default_engine_netcdf() -> Literal[""netcdf4"", ""scipy""]: engine: Literal[""netcdf4"", ""scipy""] try: - import netCDF4 # noqa: F401 + try: + import nc_complex # noqa: F401 + except ImportError: + import netCDF4 # noqa: F401 engine = ""netcdf4"" except ImportError: # pragma: no cover modified xarray/backends/netCDF4_.py @@ -327,7 +327,10 @@ class NetCDF4DataStore(WritableCFDataStore): def __init__( self, manager, group=None, mode=None, lock=NETCDF4_PYTHON_LOCK, autoclose=False ): - import netCDF4 + try: + import nc_complex as netCDF4 + except ImportError: + import netCDF4 if isinstance(manager, netCDF4.Dataset): if group is None: @@ -364,7 +367,10 @@ class NetCDF4DataStore(WritableCFDataStore): lock_maker=None, autoclose=False, ): - import netCDF4 + try: + import nc_complex as netCDF4 + except ImportError: + import netCDF4 if isinstance(filename, os.PathLike): filename = os.fspath(filename) ``` and then it works out of the box: ```python import numpy as np import xarray as xr complex_array = np.array([0 + 0j, 1 + 0j, 0 + 1j, 1 + 1j, 0.25 + 0.75j], dtype=""c16"") ds = xr.Dataset({""complex_data"": (""x"", complex_array)}) ds.to_netcdf(""test.nc"") with xr.open_dataset(""test.nc"") as df: print(df.complex_data) assert ds == df # # [5 values with dtype=complex128] # Dimensions without coordinates: x ``` If the library is built against the latest `main` of netCDF-C, then it also supports files written by `h5netcdf`. And while the package isn't on PyPI yet, [built wheels with this feature are available from CI jobs](https://github.com/PlasmaFAIR/nc-complex/actions/runs/6459506996). `nc_complex` is not quite ready yet, but I wanted to gauge interest for adding it as an optional replacement for `netCDF4` in xarray. ### Describe alternatives you've considered _No response_ ### Additional context _No response_","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/8288/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed,13221727,issue 1952720792,PR_kwDOAMm_X85dTIJ7,8338,"Enable subclassing the netCDF4 backend, changing the dataset class",1486942,closed,0,,,4,2023-10-19T17:09:22Z,2023-12-04T17:16:50Z,2023-12-04T17:16:49Z,CONTRIBUTOR,,0,pydata/xarray/pulls/8338,"See #8288 - [ ] Closes #xxxx - [ ] Tests added - [ ] User visible changes (including notable bug fixes) are documented in `whats-new.rst` - [ ] New functions/methods are listed in `api.rst` These changes allow me to make a pretty minimal subclass that just swaps out the `netCDF4.Dataset` class for my version: ```python from xarray.backends.netCDF4_ import NetCDF4BackendEntrypoint, NetCDF4DataStore from ._wrapper import Dataset # This is my netCDF extension class NcComplexDataStore(NetCDF4DataStore): @classmethod def DatasetClass(self) -> type: return Dataset class NcComplexBackendEntrypoint(NetCDF4BackendEntrypoint): DataStore = NcComplexDataStore ```","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/8338/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,pull 1949710368,PR_kwDOAMm_X85dI74W,8331,Docs: Add syntax for registering backends in `pyproject.toml`,1486942,closed,0,,,3,2023-10-18T13:06:55Z,2023-10-19T12:47:45Z,2023-10-19T12:47:34Z,CONTRIBUTOR,,0,pydata/xarray/pulls/8331,"Uses `sphinx-inline-tabs` for pretty and compact options. Also reword surrounding text a little for clarity. ","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/8331/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,pull 1807517529,PR_kwDOAMm_X85VqE9h,7995,Use variable name in all exceptions raised in `as_variable`,1486942,closed,0,,,5,2023-07-17T11:03:32Z,2023-07-17T15:33:50Z,2023-07-17T14:59:39Z,CONTRIBUTOR,,0,pydata/xarray/pulls/7995,"This is more consistent with the other exceptions raised in this function, and helps to more quickly diagnose issues when e.g. creating datasets. - [x] User visible changes (including notable bug fixes) are documented in `whats-new.rst` Is this a large enough change to warrant an entry in `whats-new`?","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/7995/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,pull 1793236828,I_kwDOAMm_X85q4p9c,7965,Maximum recursion calling `np.divide` with `DataArray` as `where` argument,1486942,open,0,,,2,2023-07-07T10:11:28Z,2023-07-07T15:16:06Z,,CONTRIBUTOR,,,,"### What happened? Calling `np.divide()` passing a bool expression involving a `DataArray` results in a `RecursionError` starting at `apply_ufunc`. Passing `DataArray.data` works fine. ### What did you expect to happen? _No response_ ### Minimal Complete Verifiable Example ```Python import numpy as np import xarray as xr N = 10 array = np.random.random((N,)) da = xr.DataArray(np.random.random((N,)), dims=(""x"",), coords={""x"": np.arange(N)}) # This works print(np.divide(array, da, where=(da > 0.2).data)) # This gives recursion error print(np.divide(array, da, where=da > 0.2)) ``` ### MVCE confirmation - [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray. - [X] Complete example — the example is self-contained, including all data and the text of any traceback. - [X] Verifiable example — the example copy & pastes into an IPython prompt or [Binder notebook](https://mybinder.org/v2/gh/pydata/xarray/main?urlpath=lab/tree/doc/examples/blank_template.ipynb), returning the result. - [X] New issue — a search of GitHub Issues suggests this is not a duplicate. ### Relevant log output ```Python Traceback (most recent call last): File ""divide_recursion_mvce.py"", line 7, in print(np.divide(array, da, where=da > 0.2)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ""venv/lib64/python3.11/site-packages/xarray/core/arithmetic.py"", line 86, in __array_ufunc__ return apply_ufunc( ^^^^^^^^^^^^ File ""venv/lib64/python3.11/site-packages/xarray/core/computation.py"", line 1197, in apply_ufunc return apply_dataarray_vfunc( ^^^^^^^^^^^^^^^^^^^^^^ File ""venv/lib64/python3.11/site-packages/xarray/core/computation.py"", line 304, in apply_dataarray_vfunc result_var = func(*data_vars) ^^^^^^^^^^^^^^^^ File ""venv/lib64/python3.11/site-packages/xarray/core/computation.py"", line 761, in apply_variable_ufunc result_data = func(*input_data) ^^^^^^^^^^^^^^^^^ File ""venv/lib64/python3.11/site-packages/xarray/core/arithmetic.py"", line 86, in __array_ufunc__ return apply_ufunc( ^^^^^^^^^^^^ File ""venv/lib64/python3.11/site-packages/xarray/core/computation.py"", line 1210, in apply_ufunc return apply_array_ufunc(func, *args, dask=dask) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ... File ""venv/lib64/python3.11/site-packages/xarray/core/computation.py"", line 1210, in apply_ufunc return apply_array_ufunc(func, *args, dask=dask) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ""venv/lib64/python3.11/site-packages/xarray/core/computation.py"", line 815, in apply_array_ufunc if any(is_chunked_array(arg) for arg in args): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ""venv/lib64/python3.11/site-packages/xarray/core/computation.py"", line 815, in if any(is_chunked_array(arg) for arg in args): ^^^^^^^^^^^^^^^^^^^^^ File ""venv/lib64/python3.11/site-packages/xarray/core/pycompat.py"", line 99, in is_chunked_array return is_duck_dask_array(x) or (is_duck_array(x) and hasattr(x, ""chunks"")) ^^^^^^^^^^^^^^^^^^^^^ File ""venv/lib64/python3.11/site-packages/xarray/core/pycompat.py"", line 95, in is_duck_dask_array return is_duck_array(x) and is_dask_collection(x) ^^^^^^^^^^^^^^^^^^^^^ File ""venv/lib64/python3.11/site-packages/xarray/core/pycompat.py"", line 87, in is_dask_collection if module_available(""dask""): ^^^^^^^^^^^^^^^^^^^^^^^^ File ""venv/lib64/python3.11/site-packages/xarray/core/utils.py"", line 1160, in module_available return importlib.util.find_spec(module) is not None ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File """", line 103, in find_spec File """", line 1080, in _find_spec File """", line 924, in find_spec File """", line 241, in _call_with_frames_removed RecursionError: maximum recursion depth exceeded while calling a Python object ``` ### Anything else we need to know? _No response_ ### Environment
INSTALLED VERSIONS ------------------ commit: None python: 3.11.3 (main, Apr 27 2023, 22:08:21) [GCC] python-bits: 64 OS: Linux OS-release: 6.3.9-1-default machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_GB.UTF-8 LOCALE: ('en_GB', 'UTF-8') libhdf5: 1.12.2 libnetcdf: 4.9.3-development xarray: 2023.6.0 pandas: 1.5.3 numpy: 1.25.0 scipy: 1.11.1 netCDF4: 1.6.4 pydap: None h5netcdf: None h5py: None Nio: None zarr: None cftime: 1.6.2 nc_time_axis: None PseudoNetCDF: None iris: None bottleneck: None dask: None distributed: None matplotlib: 3.7.2 cartopy: None seaborn: None numbagg: None fsspec: None cupy: None pint: None sparse: None flox: None numpy_groupies: None setuptools: 65.5.0 pip: 23.1.2 conda: None pytest: 7.4.0 mypy: None IPython: 8.14.0 sphinx: None
","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/7965/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,issue