home / github

Menu
  • GraphQL API
  • Search all tables

issues

Table actions
  • GraphQL API for issues

4 rows where type = "issue" and user = 3924836 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date), closed_at (date)

state 2

  • closed 3
  • open 1

type 1

  • issue · 4 ✖

repo 1

  • xarray 4
id node_id number title user state locked assignee milestone comments created_at updated_at ▲ closed_at author_association active_lock_reason draft pull_request body reactions performed_via_github_app state_reason repo type
1646566213 I_kwDOAMm_X85iJJtF 7699 drop_sel returns KeyError for abbreviated dates scottyhq 3924836 open 0     1 2023-03-29T21:31:40Z 2023-03-30T01:05:58Z   MEMBER      

What happened?

for a dataframe with a detailed timestamp coordinate (2022-03-11T13:50:50.551314000)

da.sel(time="2022-03-11") works but da.drop_sel(time="2022-03-11") results in a KeyError

What did you expect to happen?

i expect drop_sel() to work with the same inputs as sel()

Minimal Complete Verifiable Example

```Python import numpy as np import xarray as xr times = np.array(['2022-03-03T13:50:49.548072000', '2022-03-11T13:50:50.551314000', '2022-03-12T13:44:35.547024000', '2022-03-13T12:18:48.015182000'], dtype='datetime64[ns]')

da = xr.DataArray(np.arange(4), dims="time", coords={"time": times}) da.drop_sel(time='2022-03-11') ```

A workaround is to use the exact datestring: da.drop_sel(time='2022-03-11T13:50:50.551314000')

MVCE confirmation

  • [X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
  • [X] Complete example — the example is self-contained, including all data and the text of any traceback.
  • [X] Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
  • [X] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

```Python

KeyError Traceback (most recent call last) Cell In[4], line 1 ----> 1 da.drop_sel(time='2022-03-11')

File ~/.local/envs/xvec/lib/python3.11/site-packages/xarray/core/dataarray.py:3105, in DataArray.drop_sel(self, labels, errors, **labels_kwargs) 3102 if labels_kwargs or isinstance(labels, dict): 3103 labels = either_dict_or_kwargs(labels, labels_kwargs, "drop") -> 3105 ds = self._to_temp_dataset().drop_sel(labels, errors=errors) 3106 return self._from_temp_dataset(ds)

File ~/.local/envs/xvec/lib/python3.11/site-packages/xarray/core/dataset.py:5292, in Dataset.drop_sel(self, labels, errors, **labels_kwargs) 5290 except KeyError: 5291 raise ValueError(f"dimension {dim!r} does not have coordinate labels") -> 5292 new_index = index.drop(labels_for_dim, errors=errors) 5293 ds = ds.loc[{dim: new_index}] 5294 return ds

File ~/.local/envs/xvec/lib/python3.11/site-packages/pandas/core/indexes/base.py:6934, in Index.drop(self, labels, errors) 6932 if mask.any(): 6933 if errors != "ignore": -> 6934 raise KeyError(f"{list(labels[mask])} not found in axis") 6935 indexer = indexer[~mask] 6936 return self.delete(indexer)

KeyError: "['2022-03-11'] not found in axis" ```

Anything else we need to know?

related to https://github.com/pydata/xarray/issues/6605#issuecomment-1126139617

Environment

(xarray: 2023.3.0)

INSTALLED VERSIONS ------------------ commit: None python: 3.11.0 | packaged by conda-forge | (main, Jan 14 2023, 12:27:40) [GCC 11.3.0] python-bits: 64 OS: Linux OS-release: 5.4.231-137.341.amzn2.x86_64 machine: x86_64 processor: x86_64 byteorder: little LC_ALL: en_US.UTF-8 LANG: en_US.UTF-8 LOCALE: ('en_US', 'UTF-8') libhdf5: 1.12.2 libnetcdf: 4.9.1 xarray: 2023.3.0 pandas: 1.5.3 numpy: 1.24.2 scipy: 1.10.1 netCDF4: 1.6.3 pydap: None h5netcdf: 1.1.0 h5py: 3.8.0 Nio: None zarr: None cftime: 1.6.2 nc_time_axis: None PseudoNetCDF: None rasterio: 1.3.6 cfgrib: None iris: None bottleneck: None dask: None distributed: None matplotlib: 3.7.1 cartopy: None seaborn: None numbagg: None fsspec: None cupy: None pint: None sparse: None flox: None numpy_groupies: None setuptools: 67.6.0 pip: 23.0.1 conda: None pytest: None mypy: None IPython: 8.11.0 sphinx: None
{
    "url": "https://api.github.com/repos/pydata/xarray/issues/7699/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    xarray 13221727 issue
811409317 MDU6SXNzdWU4MTE0MDkzMTc= 4925 OpenDAP Documentation Example failing with RunTimeError scottyhq 3924836 closed 0     9 2021-02-18T19:52:24Z 2023-01-17T18:44:00Z 2023-01-17T18:44:00Z MEMBER      

What happened: Tried to follow http://xarray.pydata.org/en/stable/io.html#opendap with @wycheng-uw and ran into a massive traceback ending with RuntimeError: NetCDF: file not found

What you expected to happen: Expecting to successfully plot data as illustrated in documentation

Minimal Complete Verifiable Example:

python remote_data = xr.open_dataset("http://iridl.ldeo.columbia.edu/SOURCES/.OSU/.PRISM/.monthly/dods", decode_times=False) tmax = remote_data["tmax"][:500, ::3, ::3] tmax[0].plot()

Anything else we need to know?:

After digging through unresolved opendap issues (https://github.com/Unidata/netcdf4-python/issues/755, https://github.com/pydata/xarray/issues/3466, https://github.com/pydata/xarray/issues/4353)

https://github.com/pydata/xarray/issues/3580 provided the key workaround that netCDF4>1.5.1 seems to cause issues. Pinning netCDF4==1.5.1 is our current workaround. Not sure if this is specific to http://iridl.ldeo.columbia.edu/ or OpenDAP endpoints more generally...

Environment:

Output of <tt>xr.show_versions() </tt> INSTALLED VERSIONS ------------------ commit: None python: 3.7.8 | packaged by conda-forge | (default, Jul 31 2020, 02:37:09) [Clang 10.0.1 ] python-bits: 64 OS: Darwin OS-release: 20.3.0 machine: x86_64 processor: i386 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: en_US.UTF-8 libhdf5: 1.10.6 libnetcdf: 4.7.4 xarray: 0.16.2 pandas: 1.2.2 numpy: 1.19.1 scipy: 1.5.3 netCDF4: 1.5.6 pydap: None h5netcdf: 0.8.1 h5py: 3.1.0 Nio: None zarr: None cftime: 1.4.1 nc_time_axis: None PseudoNetCDF: None rasterio: 1.1.7 cfgrib: None iris: None bottleneck: None dask: 2.30.0 distributed: 2.30.0 matplotlib: 3.3.4 cartopy: 0.18.0 seaborn: None numbagg: None pint: None setuptools: 49.6.0.post20210108 pip: 20.2.3 conda: installed pytest: 6.1.1 IPython: 7.20.0 sphinx: 3.2.1 None
Full Traceback ```pytb --------------------------------------------------------------------------- RuntimeError Traceback (most recent call last) <ipython-input-4-78116fdadd4b> in <module> ----> 1 tmax[0].plot() ~/miniconda3/envs/intake-stac-gui/lib/python3.7/site-packages/xarray/plot/plot.py in __call__(self, **kwargs) 444 445 def __call__(self, **kwargs): --> 446 return plot(self._da, **kwargs) 447 448 # we can't use functools.wraps here since that also modifies the name / qualname ~/miniconda3/envs/intake-stac-gui/lib/python3.7/site-packages/xarray/plot/plot.py in plot(darray, row, col, col_wrap, ax, hue, rtol, subplot_kws, **kwargs) 161 162 """ --> 163 darray = darray.squeeze().compute() 164 165 plot_dims = set(darray.dims) ~/miniconda3/envs/intake-stac-gui/lib/python3.7/site-packages/xarray/core/dataarray.py in compute(self, **kwargs) 891 """ 892 new = self.copy(deep=False) --> 893 return new.load(**kwargs) 894 895 def persist(self, **kwargs) -> "DataArray": ~/miniconda3/envs/intake-stac-gui/lib/python3.7/site-packages/xarray/core/dataarray.py in load(self, **kwargs) 865 dask.array.compute 866 """ --> 867 ds = self._to_temp_dataset().load(**kwargs) 868 new = self._from_temp_dataset(ds) 869 self._variable = new._variable ~/miniconda3/envs/intake-stac-gui/lib/python3.7/site-packages/xarray/core/dataset.py in load(self, **kwargs) 747 for k, v in self.variables.items(): 748 if k not in lazy_data: --> 749 v.load() 750 751 return self ~/miniconda3/envs/intake-stac-gui/lib/python3.7/site-packages/xarray/core/variable.py in load(self, **kwargs) 437 self._data = as_compatible_data(self._data.compute(**kwargs)) 438 elif not is_duck_array(self._data): --> 439 self._data = np.asarray(self._data) 440 return self 441 ~/miniconda3/envs/intake-stac-gui/lib/python3.7/site-packages/numpy/core/_asarray.py in asarray(a, dtype, order) 81 82 """ ---> 83 return array(a, dtype, copy=False, order=order) 84 85 ~/miniconda3/envs/intake-stac-gui/lib/python3.7/site-packages/xarray/core/indexing.py in __array__(self, dtype) 691 692 def __array__(self, dtype=None): --> 693 self._ensure_cached() 694 return np.asarray(self.array, dtype=dtype) 695 ~/miniconda3/envs/intake-stac-gui/lib/python3.7/site-packages/xarray/core/indexing.py in _ensure_cached(self) 688 def _ensure_cached(self): 689 if not isinstance(self.array, NumpyIndexingAdapter): --> 690 self.array = NumpyIndexingAdapter(np.asarray(self.array)) 691 692 def __array__(self, dtype=None): ~/miniconda3/envs/intake-stac-gui/lib/python3.7/site-packages/numpy/core/_asarray.py in asarray(a, dtype, order) 81 82 """ ---> 83 return array(a, dtype, copy=False, order=order) 84 85 ~/miniconda3/envs/intake-stac-gui/lib/python3.7/site-packages/xarray/core/indexing.py in __array__(self, dtype) 661 662 def __array__(self, dtype=None): --> 663 return np.asarray(self.array, dtype=dtype) 664 665 def __getitem__(self, key): ~/miniconda3/envs/intake-stac-gui/lib/python3.7/site-packages/numpy/core/_asarray.py in asarray(a, dtype, order) 81 82 """ ---> 83 return array(a, dtype, copy=False, order=order) 84 85 ~/miniconda3/envs/intake-stac-gui/lib/python3.7/site-packages/xarray/core/indexing.py in __array__(self, dtype) 566 def __array__(self, dtype=None): 567 array = as_indexable(self.array) --> 568 return np.asarray(array[self.key], dtype=None) 569 570 def transpose(self, order): ~/miniconda3/envs/intake-stac-gui/lib/python3.7/site-packages/numpy/core/_asarray.py in asarray(a, dtype, order) 81 82 """ ---> 83 return array(a, dtype, copy=False, order=order) 84 85 ~/miniconda3/envs/intake-stac-gui/lib/python3.7/site-packages/xarray/coding/variables.py in __array__(self, dtype) 68 69 def __array__(self, dtype=None): ---> 70 return self.func(self.array) 71 72 def __repr__(self): ~/miniconda3/envs/intake-stac-gui/lib/python3.7/site-packages/xarray/coding/variables.py in _scale_offset_decoding(data, scale_factor, add_offset, dtype) 216 217 def _scale_offset_decoding(data, scale_factor, add_offset, dtype): --> 218 data = np.array(data, dtype=dtype, copy=True) 219 if scale_factor is not None: 220 data *= scale_factor ~/miniconda3/envs/intake-stac-gui/lib/python3.7/site-packages/xarray/coding/variables.py in __array__(self, dtype) 68 69 def __array__(self, dtype=None): ---> 70 return self.func(self.array) 71 72 def __repr__(self): ~/miniconda3/envs/intake-stac-gui/lib/python3.7/site-packages/xarray/coding/variables.py in _apply_mask(data, encoded_fill_values, decoded_fill_value, dtype) 136 ) -> np.ndarray: 137 """Mask all matching values in a NumPy arrays.""" --> 138 data = np.asarray(data, dtype=dtype) 139 condition = False 140 for fv in encoded_fill_values: ~/miniconda3/envs/intake-stac-gui/lib/python3.7/site-packages/numpy/core/_asarray.py in asarray(a, dtype, order) 81 82 """ ---> 83 return array(a, dtype, copy=False, order=order) 84 85 ~/miniconda3/envs/intake-stac-gui/lib/python3.7/site-packages/xarray/core/indexing.py in __array__(self, dtype) 566 def __array__(self, dtype=None): 567 array = as_indexable(self.array) --> 568 return np.asarray(array[self.key], dtype=None) 569 570 def transpose(self, order): ~/miniconda3/envs/intake-stac-gui/lib/python3.7/site-packages/xarray/backends/netCDF4_.py in __getitem__(self, key) 71 def __getitem__(self, key): 72 return indexing.explicit_indexing_adapter( ---> 73 key, self.shape, indexing.IndexingSupport.OUTER, self._getitem 74 ) 75 ~/miniconda3/envs/intake-stac-gui/lib/python3.7/site-packages/xarray/core/indexing.py in explicit_indexing_adapter(key, shape, indexing_support, raw_indexing_method) 851 """ 852 raw_key, numpy_indices = decompose_indexer(key, shape, indexing_support) --> 853 result = raw_indexing_method(raw_key.tuple) 854 if numpy_indices.tuple: 855 # index the loaded np.ndarray ~/miniconda3/envs/intake-stac-gui/lib/python3.7/site-packages/xarray/backends/netCDF4_.py in _getitem(self, key) 83 with self.datastore.lock: 84 original_array = self.get_array(needs_lock=False) ---> 85 array = getitem(original_array, key) 86 except IndexError: 87 # Catch IndexError in netCDF4 and return a more informative ~/miniconda3/envs/intake-stac-gui/lib/python3.7/site-packages/xarray/backends/common.py in robust_getitem(array, key, catch, max_retries, initial_delay) 50 for n in range(max_retries + 1): 51 try: ---> 52 return array[key] 53 except catch: 54 if n == max_retries: src/netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Variable.__getitem__() src/netCDF4/_netCDF4.pyx in netCDF4._netCDF4.Variable._get() src/netCDF4/_netCDF4.pyx in netCDF4._netCDF4._ensure_nc_success() RuntimeError: NetCDF: file not found ```
{
    "url": "https://api.github.com/repos/pydata/xarray/issues/4925/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed xarray 13221727 issue
412623833 MDU6SXNzdWU0MTI2MjM4MzM= 2781 enable reading of file-like HDF5 objects scottyhq 3924836 closed 0     2 2019-02-20T20:55:15Z 2019-03-16T00:35:57Z 2019-03-16T00:35:57Z MEMBER      

xarray 11.3 currently won't read HDF5 file-like objects

```python import xarray as xr import gcsfs fs = gcsfs.GCSFileSystem() images = fs.ls('pangeo-data/grfn-v2/137/') fileObj = fs.open('pangeo-data/grfn-v2/137/S1-GUNW-A-R-137-tops-20181129_20181123-020010-43220N_41518N-PP-e2c7-v2_0_0.nc')

but, can we open this w/ xarray anyway? Yes! with modifications to xarray and h5netcdf

da = xr.open_dataset(fileObj, group='/science/grids/data', engine='h5netcdf') da ```

```pytb

ValueError Traceback (most recent call last) <ipython-input-3-22e0010de1f2> in <module>() 1 # but, can we open this w/ xarray anyway? Yes! with modifications to xarray and h5netcdf ----> 2 da = xr.open_dataset(fileObj, group='/science/grids/data', engine='h5netcdf') 3 da

/srv/conda/lib/python3.6/site-packages/xarray/backends/api.py in open_dataset(filename_or_obj, group, decode_cf, mask_and_scale, decode_times, autoclose, concat_characters, decode_coords, engine, chunks, lock, cache, drop_variables, backend_kwargs) 347 else: 348 if engine is not None and engine != 'scipy': --> 349 raise ValueError('can only read file-like objects with ' 350 "default engine or engine='scipy'") 351 # assume filename_or_obj is a file-like object

ValueError: can only read file-like objects with default engine or engine='scipy' ```

Problem description

It is now possible to do this with h5py >2.9.0. see https://github.com/h5py/h5py/pull/1105. This would be a useful feature because there is a lot of NASA data out there in HDF5. This functionality could open up reading without first writing to disk (to translate to Zarr or other formats possibly). There seem to be many issues related to this: https://github.com/dask/s3fs/issues/144 https://github.com/pydata/xarray/issues/2535

I'm guessing adding this functionality doesn't fix many of the performance issues related to HDF5 and Dask https://github.com/dask/dask/issues/2488 https://github.com/dask/distributed/issues/2319

Expected Output

<xarray.Dataset> Dimensions: (latitude: 2045, longitude: 4158) Coordinates: * longitude (longitude) float64 -123.1 -123.1 ... -119.6 -119.6 * latitude (latitude) float64 43.22 43.22 43.22 ... 41.52 41.52 Data variables: crs int32 ... unwrappedPhase (latitude, longitude) float32 ... coherence (latitude, longitude) float32 ... connectedComponents (latitude, longitude) float32 ... amplitude (latitude, longitude) float32 ...

Output of xr.show_versions()

INSTALLED VERSIONS ------------------ commit: None python: 3.6.7 | packaged by conda-forge | (default, Nov 21 2018, 03:09:43) [GCC 7.3.0] python-bits: 64 OS: Linux OS-release: 4.14.65+ machine: x86_64 processor: x86_64 byteorder: little LC_ALL: en_US.UTF-8 LANG: en_US.UTF-8 LOCALE: en_US.UTF-8 libhdf5: 1.10.4 libnetcdf: 4.6.2 xarray: 0.11.3 pandas: 0.24.1 numpy: 1.16.1 scipy: 1.2.0 netCDF4: 1.4.2 pydap: None h5netcdf: 0.6.2 h5py: 2.9.0 Nio: None zarr: 2.2.0 cftime: 1.0.3.4 PseudonetCDF: None rasterio: 1.0.18 cfgrib: None iris: None bottleneck: None cyordereddict: None dask: 1.1.0 distributed: 1.25.2 matplotlib: 3.0.2 cartopy: 0.17.0 seaborn: 0.9.0 setuptools: 40.7.1 pip: 19.0.2 conda: 4.6.3 pytest: None IPython: 7.1.1 sphinx: None
{
    "url": "https://api.github.com/repos/pydata/xarray/issues/2781/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed xarray 13221727 issue
387123433 MDU6SXNzdWUzODcxMjM0MzM= 2588 Enabling rasterio.vrt.WarpedVRT with xr.open_rasterio scottyhq 3924836 closed 0     0 2018-12-04T05:11:12Z 2018-12-23T19:02:53Z 2018-12-23T19:02:53Z MEMBER      

This is not a bug, but rather a feature request and discussion opener for changes to the open_rasterio function

Currently open_rasterio (xarray version 0.11) only accepts filepath strings and does not work with In-memory rasterio.vrt.WarpedVRT objects. I have a solution (see pull request), but it's likely not the best one (Many context managers can feel odd), see example below:

```python

Lazy in-memory warping from UTM to WGS84 lat/lon

with env: with rasterio.open(url) as src: da = xr.open_rasterio(src) print(da.crs, da.sizes) with WarpedVRT(src, crs='epsg:4326') as vrt: with xr.open_rasterio(vrt) as da: print(da.crs, da.sizes)

+init=epsg:32610 (1, 7531, 7751)

+init=epsg:4326 +no_defs (1, 5981, 9183)

``` More detailed gist here: https://gist.github.com/scottyhq/ae90084adaf25e3b361b096d555c45f1

Problem description

In-memory “virtual” reprojection is a key feature of rasterio (see https://gist.github.com/sgillies/7e5cd548110a5b4d45ac1a1d93cb17a3), and it would be fantastic if this worked w/ xarray and dask distributed. Many workflows require warping between WGS84 lat/lon, UTM, Google Mercator, and rasterio can handle most any other projection.

related to: #1575, #2042, #2288 https://github.com/dask/dask/issues/3255 Seems like some synergy here with geoxarray and salem @mrocklin, @fmaussion, @geoxarray

{
    "url": "https://api.github.com/repos/pydata/xarray/issues/2588/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed xarray 13221727 issue

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issues] (
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [number] INTEGER,
   [title] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [state] TEXT,
   [locked] INTEGER,
   [assignee] INTEGER REFERENCES [users]([id]),
   [milestone] INTEGER REFERENCES [milestones]([id]),
   [comments] INTEGER,
   [created_at] TEXT,
   [updated_at] TEXT,
   [closed_at] TEXT,
   [author_association] TEXT,
   [active_lock_reason] TEXT,
   [draft] INTEGER,
   [pull_request] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [state_reason] TEXT,
   [repo] INTEGER REFERENCES [repos]([id]),
   [type] TEXT
);
CREATE INDEX [idx_issues_repo]
    ON [issues] ([repo]);
CREATE INDEX [idx_issues_milestone]
    ON [issues] ([milestone]);
CREATE INDEX [idx_issues_assignee]
    ON [issues] ([assignee]);
CREATE INDEX [idx_issues_user]
    ON [issues] ([user]);
Powered by Datasette · Queries took 218.79ms · About: xarray-datasette