html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue https://github.com/pydata/xarray/issues/1075#issuecomment-357130204,https://api.github.com/repos/pydata/xarray/issues/1075,357130204,MDEyOklzc3VlQ29tbWVudDM1NzEzMDIwNA==,1217238,2018-01-12T03:03:22Z,2018-01-12T03:03:22Z,MEMBER,"We could potentially add a from_memory() constructor to NetCDF4DataStore to simplify this process. On Thu, Jan 11, 2018 at 6:27 PM Michael Delgado wrote: > yes! Thanks @jhamman and @shoyer > . I hadn't tried it yet, but just did. worked > great! > > In [1]: import xarray as xr > ...: import requests > ...: import netCDF4 > ...: > ...: %matplotlib inline > > In [2]: res = requests.get( > ...: 'http://nasanex.s3.amazonaws.com/NEX-GDDP/BCSD/rcp45/day/atmos/tasmin/' + > ...: 'r1i1p1/v1.0/tasmin_day_BCSD_rcp45_r1i1p1_CESM1-BGC_2073.nc') > > In [3]: res.status_code > Out [3]: 200 > > In [4]: res.headers['content-type'] > Out [4]: 'application/x-netcdf' > > In [5]: nc4_ds = netCDF4.Dataset('tasmin_day_BCSD_rcp45_r1i1p1_CESM1-BGC_2073', memory=res.content) > > In [6]: store = xr.backends.NetCDF4DataStore(nc4_ds) > > In [7]: ds = xr.open_dataset(store) > > In [8]: ds.tasmin.isel(time=0).plot() > /global/home/users/mdelgado/git/public/xarray/xarray/plot/utils.py:51: FutureWarning: 'pandas.tseries.converter.register' has been moved and renamed to 'pandas.plotting.register_matplotlib_converters'. > converter.register() > Out [8]: > > [image: output_7_2] > > > In [9]: ds > Out [9]: > > Dimensions: (lat: 720, lon: 1440, time: 365) > Coordinates: > * time (time) datetime64[ns] 2073-01-01T12:00:00 2073-01-02T12:00:00 ... > * lat (lat) float32 -89.875 -89.625 -89.375 -89.125 -88.875 -88.625 ... > * lon (lon) float32 0.125 0.375 0.625 0.875 1.125 1.375 1.625 1.875 ... > Data variables: > tasmin (time, lat, lon) float64 ... > Attributes: > parent_experiment: historical > parent_experiment_id: historical > parent_experiment_rip: r1i1p1 > Conventions: CF-1.4 > institution: NASA Earth Exchange, NASA Ames Research C... > institute_id: NASA-Ames > realm: atmos > modeling_realm: atmos > version: 1.0 > downscalingModel: BCSD > experiment_id: rcp45 > frequency: day > realization: 1 > initialization_method: 1 > physics_version: 1 > tracking_id: 1865ff49-b20c-4268-852a-a9503efec72c > driving_data_tracking_ids: N/A > driving_model_ensemble_member: r1i1p1 > driving_experiment_name: historical > driving_experiment: historical > model_id: BCSD > references: BCSD method: Thrasher et al., 2012, Hydro... > DOI: http://dx.doi.org/10.7292/W0MW2F2G > experiment: RCP4.5 > title: CESM1-BGC global downscaled NEX CMIP5 Cli... > contact: Dr. Rama Nemani: rama.nemani@nasa.gov, Dr... > disclaimer: This data is considered provisional and s... > resolution_id: 0.25 degree > project_id: NEXGDDP > table_id: Table day (12 November 2010) > source: BCSD 2014 > creation_date: 2015-01-07T19:18:31Z > forcing: N/A > product: output > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > , or mute > the thread > > . > ","{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,186895655 https://github.com/pydata/xarray/issues/1075#issuecomment-357115879,https://api.github.com/repos/pydata/xarray/issues/1075,357115879,MDEyOklzc3VlQ29tbWVudDM1NzExNTg3OQ==,2443309,2018-01-12T01:26:09Z,2018-01-12T01:26:09Z,MEMBER,"@delgadom - did you find a solution here? A few more references, we're exploring ways to do this in the Pangeo project using Fuse (https://github.com/pangeo-data/pangeo/issues/52). There is a s3 equivalent of the gcsfs library used in that issue: https://github.com/dask/s3fs","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,186895655 https://github.com/pydata/xarray/issues/1075#issuecomment-347707575,https://api.github.com/repos/pydata/xarray/issues/1075,347707575,MDEyOklzc3VlQ29tbWVudDM0NzcwNzU3NQ==,1217238,2017-11-29T00:09:09Z,2017-11-29T00:09:31Z,MEMBER,"@delgadom Yes, that should work (I haven't tested it, but yes in principle it should all work now).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,186895655 https://github.com/pydata/xarray/issues/1075#issuecomment-346214120,https://api.github.com/repos/pydata/xarray/issues/1075,346214120,MDEyOklzc3VlQ29tbWVudDM0NjIxNDEyMA==,1217238,2017-11-22T01:23:42Z,2017-11-22T01:23:42Z,MEMBER,"Just to clarify: I wrote about that we use could support initializing a Dataset from a netCDF4 file image. But this wouldn't help yet for streaming access. Initializing a Dataset from a netCDF4 file image should actually work with the latest versions of xarray and netCDF4-python: ```python nc4_ds = netCDF4.Dataset('arbitrary-name', memory=netcdf_bytes) store = xarray.backends.NetCDF4DataStore(nc4_ds) ds = xarray.open_dataset(store) ```","{""total_count"": 3, ""+1"": 3, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,186895655 https://github.com/pydata/xarray/issues/1075#issuecomment-307297489,https://api.github.com/repos/pydata/xarray/issues/1075,307297489,MDEyOklzc3VlQ29tbWVudDMwNzI5NzQ4OQ==,1217238,2017-06-09T05:16:40Z,2017-06-09T05:16:40Z,MEMBER,"Yes, we could support initializing a Dataset from netCDF4 file image in a `bytes` object.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,186895655 https://github.com/pydata/xarray/issues/1075#issuecomment-307277552,https://api.github.com/repos/pydata/xarray/issues/1075,307277552,MDEyOklzc3VlQ29tbWVudDMwNzI3NzU1Mg==,1197350,2017-06-09T02:21:55Z,2017-06-09T02:21:55Z,MEMBER,Is this issue resolvable now that unidata/netcdf4-python#652 has been merged?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,186895655 https://github.com/pydata/xarray/issues/1075#issuecomment-258018067,https://api.github.com/repos/pydata/xarray/issues/1075,258018067,MDEyOklzc3VlQ29tbWVudDI1ODAxODA2Nw==,1217238,2016-11-02T22:24:43Z,2016-11-02T22:25:08Z,MEMBER,"This _does_ work for netCDF3 files, if you provide a file-like object (e.g., wrapped in `BytesIO`) or set `engine='scipy'`. Unfortunately, this is a netCDF4/HDF5 file: ``` >>> data.raw.read(8) '\x89HDF\r\n\x1a\n' ``` And as yet, there is no support for reading from file-like objects in either h5py (https://github.com/h5py/h5py/issues/552) or python-netCDF4 (https://github.com/Unidata/netcdf4-python/issues/295). So we're currently stuck :(. One possibility is to use the new HDF5 library [pyfive](https://github.com/jjhelmus/pyfive) with h5netcdf (https://github.com/shoyer/h5netcdf/issues/25). But pyfive doesn't have enough features yet to read netCDF files. ","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,186895655