issue_comments

Great to see you are pushing forward on this issue @jhamman and @shoyer. I would really have liked to contribute here, but it seems like there are quite some design choices to make, which are better left in your hands.

@jhamman sorry, I only now saw that you pointed to a previous issue on the same topic (with exactly the same considerations). I did not find that issue when I searched (for "unlimited").

You were against changing to_netcdf then. Are you still?

The above solution would not require much more than that, in set_necessary_dimensions,

def set_necessary_dimensions(self, variable): for d, l in zip(variable.dims, variable.shape): if d not in self.dimensions: self.set_dimension(d, l)

would become

def set_necessary_dimensions(self, variable): for d, l in zip(variable.dims, variable.shape): if d in self._unlimited_dimensions: l = None if d not in self.dimensions: self.set_dimension(d, l)

But maybe the encoding dict is not the way to go after all, since it contains entries per variable, while it is the dimension that must be unlimited.

Currently the dataset variables can be created in any order and their necessary dimensions created whenever needed (in the set_necessary_dimensions function). I would not like to change that logic (e.g. towards creating all dimensions required by all variables first, before adding the data variables).

So how about a new kw argument to to_netcdf, like

ds.to_netcdf(unlimited_dimensions=['time'])

or

ds.to_netcdf(dimension_unlimited={'time': True})

(the second option better for explicitly setting {'time': False})?

I think it makes sense to preserve the UNLIMITED state through read/write. In my case, I subset a netCDF file along lat and lon dimensions, leaving the time dimension untouched and would therefore expect it to pass unchanged through xarray IO (staying UNLIMITED).

However, when the dataset is indexed/subset/resampled along the unlimited dimension, it would make sense that its state is dropped. But that would require a lot of ifs and buts, so I suggest we leave that aside for now.

OK, I'd be up for taking a shot at it.

Since it is per-variable and specific to netCDF, I guess the perfect place to add this is in the encoding dictionary that you can pass to to_netcdf, right? Maybe as key unlimited? E.g.

ds.to_netcdf(encoding={'time': dict(unlimited=True)})

I need to look up whether netCDF allows for defining more than one unlimited dimension, otherwise that must throw an error.

And then it is just about passing None as length to CreateDimension, at least in netCDF4 and scipy.io.netcdf. But I did not look into how xarray handles that under the hood.