home / github / issue_comments

Menu
  • Search all tables
  • GraphQL API

issue_comments: 627368616

This data as json

html_url issue_url id node_id user created_at updated_at author_association body reactions performed_via_github_app issue
https://github.com/pydata/xarray/issues/4043#issuecomment-627368616 https://api.github.com/repos/pydata/xarray/issues/4043 627368616 MDEyOklzc3VlQ29tbWVudDYyNzM2ODYxNg== 1197350 2020-05-12T14:07:39Z 2020-05-12T14:07:39Z MEMBER

I have spent plenty of time debugging these sorts of issues. It really helps to take xarray out of the equation. Try making your request with just the netCDF--that's all that xarray uses under the hood. Overall your example is very complicated, which makes it hard to find the core issue.

You generally want to try something like this python import netCDF4 ncds = netCDF4.Dataset(OPENDAP_url) data = ncds[variable_name][:] Try playing around with the slice [:] to see under what circumstances the opendap server fails. Then use chunking in xarray to limit the size of each individual request. That's what's described in pangeo-data/pangeo#767.

A few additional comments about your code: ```python

Select spatial subset [lon,lat]

ds = ds.where((ds.lon >= Lon[0] - dl) & (ds.lon <= Lon[1] + dl) & (ds.lat >= Lat[0] - dl) & (ds.lat <= Lat[1] + dl), drop=True) ```

This is NOT how you do subsetting with xarray. Where is meant for masking. I recommend reviewing the xarray docs on indexing and selecting. Your call should be something like

python ds = ds.sel(lon=slice(...), lat=slice(...))

What's the difference? where downloads all of the data from the opendap server and then fills it with NaNs outside of your selection, while sel lazily limits the size of the request from the opendap server. This could make a big difference in terms of the server's memory usage.

python ds = ds.sortby('lon', 'lat')

Can you do this sorting after loading the data. It's an expensive operation and might not interact well with the opendap server.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  614144170
Powered by Datasette · Queries took 0.828ms · About: xarray-datasette