id,node_id,number,title,user,state,locked,assignee,milestone,comments,created_at,updated_at,closed_at,author_association,active_lock_reason,draft,pull_request,body,reactions,performed_via_github_app,state_reason,repo,type 1933496257,I_kwDOAMm_X85zPs_B,8288,New library for handling complex numbers in netCDF,1486942,closed,0,,,7,2023-10-09T17:21:20Z,2024-03-25T13:16:40Z,2024-03-25T13:16:40Z,CONTRIBUTOR,,,,"### Is your feature request related to a problem? Complex numbers are not natively handled by netCDF, and currently only the `h5netcdf` engine in xarray can handle them. However, it produces files unreadable by current versions of netCDF. There are also [many different conventions](https://github.com/PlasmaFAIR/nc-complex/blob/main/README.md#conventions-for-complex-numbers), and `h5netcdf` can only read a few of them. ### Describe the solution you'd like I've written a [drop-in extension to netCDF](https://github.com/PlasmaFAIR/nc-complex) that seamlessly handles complex numbers, and can read many different conventions. The Python API is built on top of netCDF4-python, so only a very simple change to xarray is needed to use it: ```diff modified xarray/backends/api.py @@ -115,7 +115,10 @@ def _get_default_engine_gz() -> Literal[""scipy""]: def _get_default_engine_netcdf() -> Literal[""netcdf4"", ""scipy""]: engine: Literal[""netcdf4"", ""scipy""] try: - import netCDF4 # noqa: F401 + try: + import nc_complex # noqa: F401 + except ImportError: + import netCDF4 # noqa: F401 engine = ""netcdf4"" except ImportError: # pragma: no cover modified xarray/backends/netCDF4_.py @@ -327,7 +327,10 @@ class NetCDF4DataStore(WritableCFDataStore): def __init__( self, manager, group=None, mode=None, lock=NETCDF4_PYTHON_LOCK, autoclose=False ): - import netCDF4 + try: + import nc_complex as netCDF4 + except ImportError: + import netCDF4 if isinstance(manager, netCDF4.Dataset): if group is None: @@ -364,7 +367,10 @@ class NetCDF4DataStore(WritableCFDataStore): lock_maker=None, autoclose=False, ): - import netCDF4 + try: + import nc_complex as netCDF4 + except ImportError: + import netCDF4 if isinstance(filename, os.PathLike): filename = os.fspath(filename) ``` and then it works out of the box: ```python import numpy as np import xarray as xr complex_array = np.array([0 + 0j, 1 + 0j, 0 + 1j, 1 + 1j, 0.25 + 0.75j], dtype=""c16"") ds = xr.Dataset({""complex_data"": (""x"", complex_array)}) ds.to_netcdf(""test.nc"") with xr.open_dataset(""test.nc"") as df: print(df.complex_data) assert ds == df # # [5 values with dtype=complex128] # Dimensions without coordinates: x ``` If the library is built against the latest `main` of netCDF-C, then it also supports files written by `h5netcdf`. And while the package isn't on PyPI yet, [built wheels with this feature are available from CI jobs](https://github.com/PlasmaFAIR/nc-complex/actions/runs/6459506996). `nc_complex` is not quite ready yet, but I wanted to gauge interest for adding it as an optional replacement for `netCDF4` in xarray. ### Describe alternatives you've considered _No response_ ### Additional context _No response_","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/8288/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed,13221727,issue 1793236828,I_kwDOAMm_X85q4p9c,7965,Maximum recursion calling `np.divide` with `DataArray` as `where` argument,1486942,open,0,,,2,2023-07-07T10:11:28Z,2023-07-07T15:16:06Z,,CONTRIBUTOR,,,,"### What happened? Calling `np.divide()` passing a bool expression involving a `DataArray` results in a `RecursionError` starting at `apply_ufunc`. Passing `DataArray.data` works fine. ### What did you expect to happen? _No response_ ### Minimal Complete Verifiable Example ```Python import numpy as np import xarray as xr N = 10 array = np.random.random((N,)) da = xr.DataArray(np.random.random((N,)), dims=(""x"",), coords={""x"": np.arange(N)}) # This works print(np.divide(array, da, where=(da > 0.2).data)) # This gives recursion error print(np.divide(array, da, where=da > 0.2)) ``` ### MVCE confirmation - [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray. - [X] Complete example — the example is self-contained, including all data and the text of any traceback. - [X] Verifiable example — the example copy & pastes into an IPython prompt or [Binder notebook](https://mybinder.org/v2/gh/pydata/xarray/main?urlpath=lab/tree/doc/examples/blank_template.ipynb), returning the result. - [X] New issue — a search of GitHub Issues suggests this is not a duplicate. ### Relevant log output ```Python Traceback (most recent call last): File ""divide_recursion_mvce.py"", line 7, in print(np.divide(array, da, where=da > 0.2)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ""venv/lib64/python3.11/site-packages/xarray/core/arithmetic.py"", line 86, in __array_ufunc__ return apply_ufunc( ^^^^^^^^^^^^ File ""venv/lib64/python3.11/site-packages/xarray/core/computation.py"", line 1197, in apply_ufunc return apply_dataarray_vfunc( ^^^^^^^^^^^^^^^^^^^^^^ File ""venv/lib64/python3.11/site-packages/xarray/core/computation.py"", line 304, in apply_dataarray_vfunc result_var = func(*data_vars) ^^^^^^^^^^^^^^^^ File ""venv/lib64/python3.11/site-packages/xarray/core/computation.py"", line 761, in apply_variable_ufunc result_data = func(*input_data) ^^^^^^^^^^^^^^^^^ File ""venv/lib64/python3.11/site-packages/xarray/core/arithmetic.py"", line 86, in __array_ufunc__ return apply_ufunc( ^^^^^^^^^^^^ File ""venv/lib64/python3.11/site-packages/xarray/core/computation.py"", line 1210, in apply_ufunc return apply_array_ufunc(func, *args, dask=dask) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ... File ""venv/lib64/python3.11/site-packages/xarray/core/computation.py"", line 1210, in apply_ufunc return apply_array_ufunc(func, *args, dask=dask) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ""venv/lib64/python3.11/site-packages/xarray/core/computation.py"", line 815, in apply_array_ufunc if any(is_chunked_array(arg) for arg in args): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ""venv/lib64/python3.11/site-packages/xarray/core/computation.py"", line 815, in if any(is_chunked_array(arg) for arg in args): ^^^^^^^^^^^^^^^^^^^^^ File ""venv/lib64/python3.11/site-packages/xarray/core/pycompat.py"", line 99, in is_chunked_array return is_duck_dask_array(x) or (is_duck_array(x) and hasattr(x, ""chunks"")) ^^^^^^^^^^^^^^^^^^^^^ File ""venv/lib64/python3.11/site-packages/xarray/core/pycompat.py"", line 95, in is_duck_dask_array return is_duck_array(x) and is_dask_collection(x) ^^^^^^^^^^^^^^^^^^^^^ File ""venv/lib64/python3.11/site-packages/xarray/core/pycompat.py"", line 87, in is_dask_collection if module_available(""dask""): ^^^^^^^^^^^^^^^^^^^^^^^^ File ""venv/lib64/python3.11/site-packages/xarray/core/utils.py"", line 1160, in module_available return importlib.util.find_spec(module) is not None ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File """", line 103, in find_spec File """", line 1080, in _find_spec File """", line 924, in find_spec File """", line 241, in _call_with_frames_removed RecursionError: maximum recursion depth exceeded while calling a Python object ``` ### Anything else we need to know? _No response_ ### Environment
INSTALLED VERSIONS ------------------ commit: None python: 3.11.3 (main, Apr 27 2023, 22:08:21) [GCC] python-bits: 64 OS: Linux OS-release: 6.3.9-1-default machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_GB.UTF-8 LOCALE: ('en_GB', 'UTF-8') libhdf5: 1.12.2 libnetcdf: 4.9.3-development xarray: 2023.6.0 pandas: 1.5.3 numpy: 1.25.0 scipy: 1.11.1 netCDF4: 1.6.4 pydap: None h5netcdf: None h5py: None Nio: None zarr: None cftime: 1.6.2 nc_time_axis: None PseudoNetCDF: None iris: None bottleneck: None dask: None distributed: None matplotlib: 3.7.2 cartopy: None seaborn: None numbagg: None fsspec: None cupy: None pint: None sparse: None flox: None numpy_groupies: None setuptools: 65.5.0 pip: 23.1.2 conda: None pytest: 7.4.0 mypy: None IPython: 8.14.0 sphinx: None
","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/7965/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,issue