issue_comments

Related issue: https://github.com/pydata/xarray/issues/4122

Hi all,

I'd love to have an effective method to save a netcdf4 Dataset to a bytes object (for the S3 purpose specifically). I'm currently using netcdf3 through scipy as described earlier which works fine, but I'm just missing out on some newer netcdf4 options as a consequence.

Thanks!

It looks like #23 is related. Do we have a plan about this?

That's because it falls back to the 'scipy' engine. Would be nice to have a non-hacky way to write netcdf4 files to byte streams. 😃

I agree, this would be a welcome improvement!

Currently Dataset.to_netcdf() without a path argument always using the SciPy netCDF writer, which only supports netCDF3. This is mostly because support for bytestreams is a relatively new feature in netCDF4-Python and h5py.

I ran into this issue, here's a simple workaround that seems to work:

```python def dataset_to_bytes(ds: xr.Dataset, name: str = "my-dataset") -> bytes: """Converts datset to bytes."""

nc4_ds = netCDF4.Dataset(name, mode="w", diskless=True, memory=ds.nbytes)
nc4_store = NetCDF4DataStore(nc4_ds)
dump_to_store(ds, nc4_store)
res_mem = nc4_ds.close()
res_bytes = res_mem.tobytes()
return res_bytes

```

I tested this using the following:

```python import BytesIO

fname = "REDACTED.nc" ds = xr.load_dataset(fname) ds_bytes = dataset_to_bytes(ds) ds2 = xr.load_dataset(BytesIO(ds_bytes))

assert ds2.equals(ds) and all(ds2.attrs[k]==ds.attrs[k] for k in set(ds2.attrs).union(ds.attrs)) ```

The assertion holds true, however the file size on disk is different. It's possible they were saved using different netCDF4 versions, I haven't had time to test that.

I tried using just ds.to_netcdf() but get the following error:

`ValueError: NetCDF 3 does not support type |S32`

That's because it falls back to the 'scipy' engine. Would be nice to have a non-hacky way to write netcdf4 files to byte streams. :smiley:

Is it possible to read mulitple netcdf files on s3 using open_mfdataset?

Thanks @rabernat I had forgotten about the other netcdf storage engines... do you know if h5netcdf stable enough that I should use in "production" outside of xarray for my netcdf4 reading/writing needs?

This takes about a minute to open for me.

It took me much longer earlier this week when I tried :roll_eyes: Is the bottleneck in the parsing of the coordinates?

Forget about zarr for a minute. Let's stick with the original goal of remote access to netcdf4 files in S3. You can use s3fs (or gcsfs) for this.

python import xarray as xr import s3fs fs_s3 = s3fs.S3FileSystem(anon=True) s3path = 'era5-pds/2008/01/data/air_temperature_at_2_metres.nc' remote_file_obj = fs_s3.open(s3path, mode='rb') ds = xr.open_dataset(remote_file_obj, engine='h5netcdf')

<xarray.Dataset> Dimensions: (lat: 640, lon: 1280, time0: 744) Coordinates: * lon (lon) float32 0.0 0.2812494 ... 359.718 * lat (lat) float32 89.784874 89.5062 ... -89.784874 * time0 (time0) datetime64[ns] 2008-01-01T07:00:00 ... 2008-02-01T06:00:00 Data variables: air_temperature_at_2_metres (time0, lat, lon) float32 ... Attributes: source: Reanalysis institution: ECMWF title: "ERA5 forecasts" history: Wed Jul 4 22:08:50 2018: ncatted /data.e1/wrk/s3_out_in/20...

This takes about a minute to open for me. I have not tried writing, but this is perhaps a starting point.

If you are unsatisfied by the performance of netcdf4 on cloud, I would indeed encourage you to investigate zarr.

Hi @Non-Descript-Individual

I've found that the netcdf4-python library really wants to have direct access to a disk/filesystem to work, it also really wants to do its own file access management. I've always attributed this to the python library being a wrapper for the netcdf C library.

My guess would be that the easiest way to do what you want is to separate the writing of the netcdf file step in xarray from the putting the file into S3. Something like this:

python x.to_netcdf('temp_file.nc') s3.upload_file('temp_file.nc', 'bucketname', 'real_name_for_temp_file.nc')

The netcdf4-python library does seem to provide an interface for the "diskless" flags. In this case, from the examples it looks to give you a bunch of bytes in a memoryview object on calling close(). I'm not sure this is accessible from xarray though.

Alternatively, @rabernat is an advocate of using zarr when putting netcdf compatible data into cloud storage, the zarr docs provide an example using s3fs

Quick edit: Here is the to_zarr docs in xarray