issue_comments: 497038453

This data as json

html_url issue_url id node_id user created_at updated_at author_association body reactions performed_via_github_app issue
https://github.com/pydata/xarray/issues/2995#issuecomment-497038453 https://api.github.com/repos/pydata/xarray/issues/2995 497038453 MDEyOklzc3VlQ29tbWVudDQ5NzAzODQ1Mw== 1197350 2019-05-29T17:42:45Z 2019-05-29T17:42:45Z MEMBER

Forget about zarr for a minute. Let's stick with the original goal of remote access to netcdf4 files in S3. You can use s3fs (or gcsfs) for this.

python import xarray as xr import s3fs fs_s3 = s3fs.S3FileSystem(anon=True) s3path = 'era5-pds/2008/01/data/air_temperature_at_2_metres.nc' remote_file_obj = fs_s3.open(s3path, mode='rb') ds = xr.open_dataset(remote_file_obj, engine='h5netcdf')

<xarray.Dataset> Dimensions: (lat: 640, lon: 1280, time0: 744) Coordinates: * lon (lon) float32 0.0 0.2812494 ... 359.718 * lat (lat) float32 89.784874 89.5062 ... -89.784874 * time0 (time0) datetime64[ns] 2008-01-01T07:00:00 ... 2008-02-01T06:00:00 Data variables: air_temperature_at_2_metres (time0, lat, lon) float32 ... Attributes: source: Reanalysis institution: ECMWF title: "ERA5 forecasts" history: Wed Jul 4 22:08:50 2018: ncatted /data.e1/wrk/s3_out_in/20...

This takes about a minute to open for me. I have not tried writing, but this is perhaps a starting point.

If you are unsatisfied by the performance of netcdf4 on cloud, I would indeed encourage you to investigate zarr.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   449706080