issues

I'm trying to write a netcdf file directly from xarray to S3 object storage. I'm wondering:

1. Why writing NetCDF files requires a "seek"
2. Why the scipy engine is getting used instead of the specified netcdf4 engine.
3. If there are nice workarounds (besides writing the NetCDF file locally, then using the AWS CLI to transfer to S3)

Code sample:

```python import fsspec import xarray as xr

ds = xr.open_dataset('http://geoport.usgs.esipfed.org/thredds/dodsC' '/silt/usgs/Projects/stellwagen/CF-1.6/BUZZ_BAY/2651-A.cdf')

outfile = fsspec.open('s3://chs-pangeo-data-bucket/rsignell/test.nc', mode='wb', profile='default')

with outfile as f: ds.to_netcdf(f, engine='netcdf4') which produces:python-traceback

OSError Traceback (most recent call last) <ipython-input-3-024939f31fe4> in <module> 2 mode='wb', profile='default') 3 with outfile as f: ----> 4 ds.to_netcdf(f, engine='netcdf4')

/srv/conda/envs/pangeo/lib/python3.7/site-packages/xarray/core/dataset.py in to_netcdf(self, path, mode, format, group, engine, encoding, unlimited_dims, compute, invalid_netcdf) 1552 unlimited_dims=unlimited_dims, 1553 compute=compute, -> 1554 invalid_netcdf=invalid_netcdf, 1555 ) 1556

/srv/conda/envs/pangeo/lib/python3.7/site-packages/xarray/backends/api.py in to_netcdf(dataset, path_or_file, mode, format, group, engine, encoding, unlimited_dims, compute, multifile, invalid_netcdf) 1102 finally: 1103 if not multifile and compute: -> 1104 store.close() 1105 1106 if not compute:

/srv/conda/envs/pangeo/lib/python3.7/site-packages/xarray/backends/scipy_.py in close(self) 221 222 def close(self): --> 223 self._manager.close()

/srv/conda/envs/pangeo/lib/python3.7/site-packages/xarray/backends/file_manager.py in close(failed resolving arguments) 331 def close(self, needs_lock=True): 332 del needs_lock # ignored --> 333 self._value.close()

/srv/conda/envs/pangeo/lib/python3.7/site-packages/scipy/io/netcdf.py in close(self) 297 if hasattr(self, 'fp') and not self.fp.closed: 298 try: --> 299 self.flush() 300 finally: 301 self.variables = OrderedDict()

/srv/conda/envs/pangeo/lib/python3.7/site-packages/scipy/io/netcdf.py in flush(self) 407 """ 408 if hasattr(self, 'mode') and self.mode in 'wa': --> 409 self._write() 410 sync = flush 411

/srv/conda/envs/pangeo/lib/python3.7/site-packages/scipy/io/netcdf.py in _write(self) 411 412 def _write(self): --> 413 self.fp.seek(0) 414 self.fp.write(b'CDF') 415 self.fp.write(array(self.version_byte, '>b').tostring())

/srv/conda/envs/pangeo/lib/python3.7/site-packages/fsspec/spec.py in seek(self, loc, whence) 1122 loc = int(loc) 1123 if not self.mode == "rb": -> 1124 raise OSError("Seek only available in read mode") 1125 if whence == 0: 1126 nloc = loc

OSError: Seek only available in read mode ```

What happened?

I have a reproducible notebook where I've loaded a referenceFileSystem dataset into xarray and everything seems fine with time being understood correctly, but when I try to save a subset to zarr or netcdf, I get: numpy.datetime64' object has no attribute 'year' I don't understand this since it seems time is always a datetime64 object in xarray, and I've never had this problem before.

What did you expect to happen?

Expected the file to be written as usual without error.

Minimal Complete Verifiable Example

Python https://nbviewer.org/gist/rsignell-usgs/029b39f0c428b07914f5a6b1129da572

Relevant log output

No response

Anything else we need to know?

I asked the question first over at https://github.com/fsspec/kerchunk/issues/130 and @martindurant thought this looked like an xarray issue, not a kerchunk issue.

Environment

INSTALLED VERSIONS

commit: None python: 3.9.10 | packaged by conda-forge | (main, Feb 1 2022, 21:24:11) [GCC 9.4.0] python-bits: 64 OS: Linux OS-release: 4.12.14-150.17_5.0.85-cray_ari_c machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: None LOCALE: ('en_US', 'UTF-8') libhdf5: 1.12.1 libnetcdf: 4.8.1

xarray: 0.21.0 pandas: 1.4.0 numpy: 1.21.5 scipy: 1.7.3 netCDF4: 1.5.8 pydap: None h5netcdf: 0.13.1 h5py: 3.6.0 Nio: None zarr: 2.10.3 cftime: 1.5.2 nc_time_axis: None PseudoNetCDF: None rasterio: 1.2.10 cfgrib: 0.9.10.0 iris: None bottleneck: 1.3.2 dask: 2021.12.0 distributed: 2021.12.0 matplotlib: 3.5.1 cartopy: 0.20.2 seaborn: None numbagg: None fsspec: 2022.01.0 cupy: None pint: 0.18 sparse: 0.13.0 setuptools: 59.8.0 pip: 22.0.2 conda: 4.11.0 pytest: None IPython: 8.0.1 sphinx: 4.4.0

​