home / github / issues

Menu
  • Search all tables
  • GraphQL API

issues: 1359181017

This data as json

id node_id number title user state locked assignee milestone comments created_at updated_at closed_at author_association active_lock_reason draft pull_request body reactions performed_via_github_app state_reason repo type
1359181017 I_kwDOAMm_X85RA3TZ 6977 Bad data from opendap server without raising an error 13545615 open 0     1 2022-09-01T17:27:32Z 2023-01-17T18:35:38Z   NONE      

What is your issue?

Hello, sometimes when using xarray to retrieve data from an opendap server I see a warning/error message like

Error:curl error: Failure when receiving data from the peer Error:DAP DATADDS packet is apparently too short

but the error is not raised in python. The load process ends but the data is all bad (all NaN, zeros or very large values). Checking the web I found out that the message above is from the OPeNDAP C API

I tried using a context manager to raise an error if any message was written to stdout/stderr but it didn't work (I think it is because a child process is the one writing the message). Sometimes there is no error message and the download finishs without any problem, I think it depends on the server load.

I am not sure if it is a xarray problem, a dask problem (ds.load calls dask.compute) or a NetCDF C API problem.

Bellow is minimal reproducible example. ``` import sys import io import contextlib import xarray as xr

use the current date or the previous day

url = ('http://nomads.ncep.noaa.gov:80/dods/rtofs/rtofs_global20220831' '/rtofs_glo_3dz_forecast_daily_uvel')

ds = (xr .open_dataset(url, chunks={'time': '100MB'}) .isel(time=slice(1, 9)) .sel(lon=slice(300.0, 340.0), lat=slice(-6.0, 7.0)))

for var in ds.data_vars:

print(var, flush=True)

sout, serr = io.StringIO(), io.StringIO()
with contextlib.redirect_stdout(sout):
    with contextlib.redirect_stderr(serr):
        print('foo')

        ds[var].load()
        # sometimes the message
        # Error:curl error: Failure when receiving data from the peer
        # Error:DAP DATADDS packet is apparently too short
        # appears after the load, but without raising an error.

        print('bar', file=sys.stderr)

out = sout.getvalue() + serr.getvalue()
if out != 'foo\nbar\n':
    raise ValueError(out)

```

Thank you!

{
    "url": "https://api.github.com/repos/pydata/xarray/issues/6977/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
    13221727 issue

Links from other tables

  • 1 row from issues_id in issues_labels
  • 1 row from issue in issue_comments
Powered by Datasette · Queries took 1.427ms · About: xarray-datasette