home / github / issues

Menu
  • GraphQL API
  • Search all tables

issues: 514672231

This data as json

id node_id number title user state locked assignee milestone comments created_at updated_at closed_at author_association active_lock_reason draft pull_request body reactions performed_via_github_app state_reason repo type
514672231 MDU6SXNzdWU1MTQ2NzIyMzE= 3466 RuntimeError: NetCDF: DAP failure 47066389 closed 1     4 2019-10-30T13:32:34Z 2023-09-12T16:00:57Z 2023-09-12T16:00:57Z NONE      

Hi all,

I am interested in extracting specific point and variable information from the GEOS-FC product, accessible via OpenDap.

Loading the data seems to work fine, and I can do some processing to my specific needs. Ideally I would like to convert this selection to a dataframe, or if needed store as an intermediate file from which I can read again.

Yet when doing so, I get the following error: RuntimeError: NetCDF: DAP failure

I am not sure what is causing this? Perhaps I chunck the data in the wrong (inefficient) way? Or there is an error with the GEOS netcdf files? Or ...

Below a working code snippet.

``` python import xarray as xr idir_geos = 'https://opendap.nccs.nasa.gov/dods/gmao/geos-cf/assim/chm_tavg_1hr_g1440x721_v1'

def preprocess(ds): ''' Rename variables and select the relevant ones. Remove lev''' ds = ds.rename({'pm25_rh35_gcc': 'PM2.5','no': 'NO','no2': 'NO2','o3': 'O3','so2': 'SO2','co': 'CO'}) ds = ds[['PM2.5','NO','NO2','O3','SO2','CO']] ds = ds.squeeze('lev') return ds

ds = xr.open_mfdataset([idir_geos],preprocess=preprocess,combine='by_coords')

lat = 51.25 lon = 4.25 pol = 'O3' ds_sel = ds.sel(lat=lat,lon=lon,method='nearest')[pol]

df_sel = ds_sel.to_dataframe().drop(['lat','lon'],axis=1)

ds_sel.to_netcdf('test.nc') # Runtime error

```

Traceback error:

Traceback (most recent call last): File "/home/demuzmp4/.local/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3291, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "<ipython-input-2-fccd11da2246>", line 57, in <module> df_sel = ds_sel.to_dataframe().drop(['lat','lon'],axis=1) File "/home/demuzmp4/.local/lib/python3.6/site-packages/xarray/core/dataset.py", line 4285, in to_dataframe return self.to_dataframe(self.dims) File "/home/demuzmp4/.local/lib/python3.6/site-packages/xarray/core/dataset.py", line 4273, in _to_dataframe for k in columns File "/home/demuzmp4/.local/lib/python3.6/site-packages/xarray/core/dataset.py", line 4273, in <listcomp> for k in columns File "/home/demuzmp4/.local/lib/python3.6/site-packages/xarray/core/variable.py", line 437, in values return _as_array_or_item(self._data) File "/home/demuzmp4/.local/lib/python3.6/site-packages/xarray/core/variable.py", line 250, in _as_array_or_item data = np.asarray(data) File "/home/demuzmp4/.local/lib/python3.6/site-packages/numpy/core/_asarray.py", line 85, in asarray return array(a, dtype, copy=False, order=order) File "/usr/lib/python3/dist-packages/dask/array/core.py", line 1138, in __array__ x = self.compute() File "/usr/lib/python3/dist-packages/dask/base.py", line 135, in compute (result,) = compute(self, traverse=False, kwargs) File "/usr/lib/python3/dist-packages/dask/base.py", line 333, in compute results = get(dsk, keys, kwargs) File "/usr/lib/python3/dist-packages/dask/threaded.py", line 75, in get pack_exception=pack_exception, *kwargs) File "/usr/lib/python3/dist-packages/dask/local.py", line 521, in get_async raise_exception(exc, tb) File "/usr/lib/python3/dist-packages/dask/compatibility.py", line 60, in reraise raise exc File "/usr/lib/python3/dist-packages/dask/local.py", line 290, in execute_task result = _execute_task(task, data) File "/usr/lib/python3/dist-packages/dask/local.py", line 271, in _execute_task return func(args2) File "/usr/lib/python3/dist-packages/dask/array/core.py", line 72, in getter c = np.asarray(c) File "/home/demuzmp4/.local/lib/python3.6/site-packages/numpy/core/_asarray.py", line 85, in asarray return array(a, dtype, copy=False, order=order) File "/home/demuzmp4/.local/lib/python3.6/site-packages/xarray/core/indexing.py", line 490, in array return np.asarray(self.array, dtype=dtype) File "/home/demuzmp4/.local/lib/python3.6/site-packages/numpy/core/_asarray.py", line 85, in asarray return array(a, dtype, copy=False, order=order) File "/home/demuzmp4/.local/lib/python3.6/site-packages/xarray/core/indexing.py", line 652, in array return np.asarray(self.array, dtype=dtype) File "/home/demuzmp4/.local/lib/python3.6/site-packages/numpy/core/_asarray.py", line 85, in asarray return array(a, dtype, copy=False, order=order) File "/home/demuzmp4/.local/lib/python3.6/site-packages/xarray/core/indexing.py", line 556, in array return np.asarray(array[self.key], dtype=None) File "/home/demuzmp4/.local/lib/python3.6/site-packages/numpy/core/_asarray.py", line 85, in asarray return array(a, dtype, copy=False, order=order) File "/home/demuzmp4/.local/lib/python3.6/site-packages/xarray/coding/variables.py", line 73, in array return self.func(self.array) File "/home/demuzmp4/.local/lib/python3.6/site-packages/xarray/coding/variables.py", line 142, in _apply_mask data = np.asarray(data, dtype=dtype) File "/home/demuzmp4/.local/lib/python3.6/site-packages/numpy/core/_asarray.py", line 85, in asarray return array(a, dtype, copy=False, order=order) File "/home/demuzmp4/.local/lib/python3.6/site-packages/xarray/core/indexing.py", line 556, in array return np.asarray(array[self.key], dtype=None) File "/home/demuzmp4/.local/lib/python3.6/site-packages/xarray/backends/netCDF4.py", line 72, in getitem key, self.shape, indexing.IndexingSupport.OUTER, self.getitem File "/home/demuzmp4/.local/lib/python3.6/site-packages/xarray/core/indexing.py", line 836, in explicit_indexing_adapter result = raw_indexing_method(raw_key.tuple) File "/home/demuzmp4/.local/lib/python3.6/site-packages/xarray/backends/netCDF4.py", line 84, in _getitem array = getitem(original_array, key) File "/home/demuzmp4/.local/lib/python3.6/site-packages/xarray/backends/common.py", line 54, in robust_getitem return array[key] File "netCDF4/_netCDF4.pyx", line 4408, in netCDF4._netCDF4.Variable.getitem File "netCDF4/_netCDF4.pyx", line 5352, in netCDF4._netCDF4.Variable._get File "netCDF4/_netCDF4.pyx", line 1887, in netCDF4._netCDF4._ensure_nc_success RuntimeError: NetCDF: DAP failure

More info on my xarray installation:

commit: None python: 3.6.9 (default, Jul 3 2019, 07:38:46) [GCC 8.3.0] python-bits: 64 OS: Linux OS-release: 4.15.0-66-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: en_US.UTF-8 LANG: en_GB.UTF-8 LOCALE: en_US.UTF-8 libhdf5: 1.10.4 libnetcdf: 4.6.3 xarray: 0.14.0 pandas: 0.25.2 numpy: 1.17.3 scipy: 1.3.1 netCDF4: 1.5.3 pydap: installed h5netcdf: None h5py: 2.9.0 Nio: None zarr: 2.3.2 cftime: 1.0.4 nc_time_axis: None PseudoNetCDF: None rasterio: 1.0.28 cfgrib: None iris: None bottleneck: 1.2.1 dask: 0.16.0 distributed: None matplotlib: 3.1.1 cartopy: 0.17.0 seaborn: 0.9.0 numbagg: None setuptools: 41.4.0 pip: 9.0.1 conda: None pytest: 5.2.1 IPython: 7.3.0 sphinx: 1.8.4

{
    "url": "https://api.github.com/repos/pydata/xarray/issues/3466/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed 13221727 issue

Links from other tables

  • 1 row from issues_id in issues_labels
  • 4 rows from issue in issue_comments
Powered by Datasette · Queries took 73.011ms · About: xarray-datasette