issue_comments: 639771646

This data as json

html_url issue_url id node_id user created_at updated_at author_association body reactions performed_via_github_app issue
https://github.com/pydata/xarray/issues/4122#issuecomment-639771646 https://api.github.com/repos/pydata/xarray/issues/4122 639771646 MDEyOklzc3VlQ29tbWVudDYzOTc3MTY0Ng== 1872600 2020-06-05T20:08:37Z 2020-06-05T20:54:36Z NONE

Okay @scottyhq, I tried setting engine='h5netcdf', but still got: OSError: Seek only available in read mode Thinking about this a little more, it's pretty clear why writing NetCDF to S3 would require seek mode.

I asked @martindurant about supporting seek for writing in fsspec and he said that would be pretty hard. And in fact, the performance probably would be pretty terrible as lots of little writes would be required.

So maybe it's best just to write netcdf files locally and then push them to S3.

And to facilitate that, @martindurant merged a PR yesterday to enable simplecache for writing in fsspec, so after doing: pip install git+https://github.com/intake/filesystem_spec.git in my environment, this now works: ```python import xarray as xr import fsspec

ds = xr.open_dataset('http://geoport.usgs.esipfed.org/thredds/dodsC' '/silt/usgs/Projects/stellwagen/CF-1.6/BUZZ_BAY/2651-A.cdf')

outfile = fsspec.open('simplecache::s3://chs-pangeo-data-bucket/rsignell/foo2.nc', mode='wb', s3=dict(profile='default')) with outfile as f: ds.to_netcdf(f) `` (Here I'm tellingfsspec` to use the AWS credentials in my "default" profile)

Thanks Martin!!!

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   631085856