home / github

Menu
  • Search all tables
  • GraphQL API

issues

Table actions
  • GraphQL API for issues

3 rows where comments = 4, repo = 13221727 and user = 1828519 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date), closed_at (date)

type 1

  • issue 3

state 1

  • closed 3

repo 1

  • xarray · 3 ✖
id node_id number title user state locked assignee milestone comments created_at updated_at ▲ closed_at author_association active_lock_reason draft pull_request body reactions performed_via_github_app state_reason repo type
1966675016 I_kwDOAMm_X851ORRI 8388 Type annotation compatibility with numpy ufuncs djhoese 1828519 closed 0     4 2023-10-28T17:25:11Z 2023-11-02T12:44:50Z 2023-11-02T12:44:50Z CONTRIBUTOR      

Is your feature request related to a problem?

I'd like mypy to understand that xarray DataArrays passed to numpy ufuncs have a return type of xarray DataArray.

```python import xarray as xr import numpy as np

def compute_relative_azimuth(sat_azi: xr.DataArray, sun_azi: xr.DataArray) -> xr.DataArray: abs_diff = np.absolute(sun_azi - sat_azi) ssadiff = np.minimum(abs_diff, 360 - abs_diff) return ssadiff

```

bash $ mypy ./xarray_mypy.py xarray_mypy.py:7: error: Incompatible return value type (got "ndarray[Any, dtype[Any]]", expected "DataArray") [return-value] Found 1 error in 1 file (checked 1 source file)

Describe the solution you'd like

I'm not sure if this is possible, if it is something xarray can fix, or something numpy needs to "fix". I'd like the above situation to "just work" without anything more than maybe some extra type-stub package.

Describe alternatives you've considered

Cast types or other type coercion or tell mypy to ignore the type issues for these numpy call.

Additional context

https://stackoverflow.com/questions/77369042/typing-when-passing-xarray-dataarray-objects-to-numpy-ufuncs

{
    "url": "https://api.github.com/repos/pydata/xarray/issues/8388/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed xarray 13221727 issue
1419602897 I_kwDOAMm_X85UnWvR 7197 Unstable pandas causes CF datetime64 issues djhoese 1828519 closed 0     4 2022-10-23T02:30:39Z 2022-10-26T16:00:35Z 2022-10-26T16:00:35Z CONTRIBUTOR      

What happened?

The Satpy project has a CI environment that installs numpy, pandas, and xarray (and a couple other packages) from their unstable sources (nightly builds, github source, etc). In the last week or two this environment has started failing with various datetime64 issues. It all seems to be caused by some recent change in pandas, but I can't place exactly what the problem is nor the commit/PR that started it. It seems there are a couple datetime related PRs.

What did you expect to happen?

Datetime or datetime64 objects should be allowed to be in whatever units they need to be in (days or minutes or nanoseconds. It seems parts of xarray (or pandas) assume datetime64[ns] but a change in pandas is no longer doing this conversion automatically (from datetime64[X] to datetime64[ns].

Minimal Complete Verifiable Example

You should be able to take any environment with modern xarray and install dev pandas with:

python -m pip install --index-url https://pypi.anaconda.org/scipy-wheels-nightly/simple/ --trusted-host pypi.anaconda.org --no-deps --pre --upgrade pandas

Then run this snippet:

```Python import xarray as xr import numpy as np from xarray.coding.times import CFDatetimeCoder

a = xr.DataArray(np.arange(1.0), dims=("time",), coords={"time": [np.datetime64('2018-05-30T10:05:00')]})

CFDatetimeCoder().encode(a.coords["time"]) ```

I haven't been able to generate a higher-level MVCE yet, but I'm hoping this little snippet will make the issue obvious to someone familiar with xarray internals.

MVCE confirmation

  • [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
  • [X] Complete example — the example is self-contained, including all data and the text of any traceback.
  • [X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
  • [X] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

```

----> 1 CFDatetimeCoder().encode(a.coords["time"])

File ~/miniconda3/envs/satpy_py39_unstable/lib/python3.9/site-packages/xarray/coding/times.py:676, in CFDatetimeCoder.encode(self, variable, name) 672 dims, data, attrs, encoding = unpack_for_encoding(variable) 673 if np.issubdtype(data.dtype, np.datetime64) or contains_cftime_datetimes( 674 variable 675 ): --> 676 (data, units, calendar) = encode_cf_datetime( 677 data, encoding.pop("units", None), encoding.pop("calendar", None) 678 ) 679 safe_setitem(attrs, "units", units, name=name) 680 safe_setitem(attrs, "calendar", calendar, name=name)

File ~/miniconda3/envs/satpy_py39_unstable/lib/python3.9/site-packages/xarray/coding/times.py:612, in encode_cf_datetime(dates, units, calendar) 609 dates = np.asarray(dates) 611 if units is None: --> 612 units = infer_datetime_units(dates) 613 else: 614 units = _cleanup_netcdf_time_units(units)

File ~/miniconda3/envs/satpy_py39_unstable/lib/python3.9/site-packages/xarray/coding/times.py:394, in infer_datetime_units(dates) 392 print("Formatting datetime object") 393 reference_date = dates[0] if len(dates) > 0 else "1970-01-01" --> 394 reference_date = format_cftime_datetime(reference_date) 395 unique_timedeltas = np.unique(np.diff(dates)) 396 units = _infer_time_units_from_diff(unique_timedeltas)

File ~/miniconda3/envs/satpy_py39_unstable/lib/python3.9/site-packages/xarray/coding/times.py:405, in format_cftime_datetime(date) 400 def format_cftime_datetime(date): 401 """Converts a cftime.datetime object to a string with the format: 402 YYYY-MM-DD HH:MM:SS.UUUUUU 403 """ 404 return "{:04d}-{:02d}-{:02d} {:02d}:{:02d}:{:02d}.{:06d}".format( --> 405 date.year, 406 date.month, 407 date.day, 408 date.hour, 409 date.minute, 410 date.second, 411 date.microsecond, 412 )

AttributeError: 'numpy.datetime64' object has no attribute 'year' ```

Anything else we need to know?

No response

Environment

``` INSTALLED VERSIONS ------------------ commit: None python: 3.9.12 | packaged by conda-forge | (main, Mar 24 2022, 23:25:59) [GCC 10.3.0] python-bits: 64 OS: Linux OS-release: 5.19.0-76051900-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: ('en_US', 'UTF-8') libhdf5: 1.12.2 libnetcdf: 4.8.1 xarray: 2022.10.0 pandas: 2.0.0.dev0+422.g6c46013c54 numpy: 1.23.4 scipy: 1.10.0.dev0+1848.f114d8b netCDF4: 1.6.0 pydap: None h5netcdf: 1.0.0 h5py: 3.7.0 Nio: None zarr: 2.13.0a3.dev5 cftime: 1.6.2 nc_time_axis: None PseudoNetCDF: None rasterio: 1.4dev cfgrib: None iris: None bottleneck: 1.3.5 dask: 2022.10.0+6.gc8dc3955 distributed: None matplotlib: 3.7.0.dev473+gc450aa7baf cartopy: 0.20.3 seaborn: None numbagg: None fsspec: 2022.5.0 cupy: None pint: None sparse: None flox: None numpy_groupies: None setuptools: 65.3.0 pip: 22.2.2 conda: None pytest: 7.1.1 IPython: 8.2.0 sphinx: 5.0.0 ```
{
    "url": "https://api.github.com/repos/pydata/xarray/issues/7197/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed xarray 13221727 issue
812692450 MDU6SXNzdWU4MTI2OTI0NTA= 4934 ImplicitToExplicitIndexingAdapter being returned with dask unstable version djhoese 1828519 closed 0     4 2021-02-20T19:41:12Z 2021-03-09T22:51:13Z 2021-03-09T22:51:13Z CONTRIBUTOR      

What happened:

I have a couple CI environments that use unstable versions of xarray and dask among other libraries. A couple tests have been failing in different but similar ways. I was able to get one of them down to the below example. This happens with the dask master branch on either the xarray latest release or xarray master. Here are the commits I was using when doing unstable both.

Dask: ad01acc14088d03127dfec4f881cce959f3ac4d9 Xarray: eb7e112d45a9edebd8e5fb4f873e3e6adb18824a

I don't see this with dask's latest release and this is likely caused by a change in dask, but after seeing PRs like #4884 I'm wondering if this is something similar and requires a change in xarray.

The bottom line is that doing various things like my_data_arr.data.compute() or dask.compute(my_data_arr.data) produces:

ImplicitToExplicitIndexingAdapter(array=CopyOnWriteArray(array=LazilyOuterIndexedArray(array=<xarray.backends.rasterio_.RasterioArrayWrapper object at 0x7f257b382150>, key=BasicIndexer((slice(0, 2, 1), slice(0, 100, 1), slice(0, 100, 1))))))

What you expected to happen:

I would expect to get a numpy array back when computing the underlying dask array.

Minimal Complete Verifiable Example:

```python from PIL import Image import xarray as xr import numpy as np

create a test image

Image.fromarray(np.zeros((5, 5, 3), dtype=np.uint8)).save('test.png') r = xr.open_rasterio('test.png', chunks='auto') print(r.data.compute())

ImplicitToExplicitIndexingAdapter(array=CopyOnWriteArray(array=LazilyOuterIndexedArray(array=<xarray.backends.rasterio_.RasterioArrayWrapper object at 0x7f25e72bcc10>, key=BasicIndexer((slice(0, 2, 1), slice(0, 100, 1), slice(0, 100, 1))))))

```

Anything else we need to know?:

As mentioned above other weird things are happening when array wrappers seem to be involved but I haven't been able to make a small example of them.

Environment:

Output of <tt>xr.show_versions()</tt> INSTALLED VERSIONS ------------------ commit: None python: 3.7.6 | packaged by conda-forge | (default, Jan 7 2020, 22:33:48) [GCC 7.3.0] python-bits: 64 OS: Linux OS-release: 5.8.0-7642-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: en_US.UTF-8 libhdf5: 1.10.5 libnetcdf: 4.7.3 xarray: 0.16.3.dev132+geb7e112d pandas: 1.2.2 numpy: 1.20.1 scipy: 1.6.1 netCDF4: 1.5.3 pydap: None h5netcdf: None h5py: 2.10.0 Nio: None zarr: 2.6.2.dev42 cftime: 1.4.1 nc_time_axis: None PseudoNetCDF: None rasterio: 1.3dev cfgrib: None iris: None bottleneck: 1.4.0.dev0+117.gf2bc792 dask: 2021.02.0+21.gad01acc1 distributed: 2021.02.0+7.g383ea032 matplotlib: 3.4.0rc1 cartopy: 0.17.0 seaborn: None numbagg: None pint: None setuptools: 45.2.0.post20200209 pip: 20.0.2 conda: None pytest: 5.3.5 IPython: 7.12.0 sphinx: 2.4.3
{
    "url": "https://api.github.com/repos/pydata/xarray/issues/4934/reactions",
    "total_count": 3,
    "+1": 3,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed xarray 13221727 issue

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issues] (
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [number] INTEGER,
   [title] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [state] TEXT,
   [locked] INTEGER,
   [assignee] INTEGER REFERENCES [users]([id]),
   [milestone] INTEGER REFERENCES [milestones]([id]),
   [comments] INTEGER,
   [created_at] TEXT,
   [updated_at] TEXT,
   [closed_at] TEXT,
   [author_association] TEXT,
   [active_lock_reason] TEXT,
   [draft] INTEGER,
   [pull_request] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [state_reason] TEXT,
   [repo] INTEGER REFERENCES [repos]([id]),
   [type] TEXT
);
CREATE INDEX [idx_issues_repo]
    ON [issues] ([repo]);
CREATE INDEX [idx_issues_milestone]
    ON [issues] ([milestone]);
CREATE INDEX [idx_issues_assignee]
    ON [issues] ([assignee]);
CREATE INDEX [idx_issues_user]
    ON [issues] ([user]);
Powered by Datasette · Queries took 75.802ms · About: xarray-datasette