issue_comments

I will give xarray.open_mfdataset a shot. Just one question - is this approach memory conservative? My reasoning for chunking in the first place is large file size.

Yes, open_mfdataset uses dask, which allows for streaming computation.

Is it now possible to append to a netCDF file using xarray?

No, it is not. This issue is about appending new variables to an existing netCDF file.

I think what you are looking for is to append along existing dimensions to a netCDF file. This is possible in the netCDF data model, but not yet supported by xarray. See https://github.com/pydata/xarray/issues/1398 for some discussion.

For these types of use cases, I would generally recommend writing a new netCDF file, and then loading everything afterwards using xarray.open_mfdataset.

+1, we probably don't want to read coordinates back from disk On Wed, Oct 4, 2017 at 12:09 PM Fabien Maussion notifications@github.com wrote:

@jhamman https://github.com/jhamman no I haven't looked into this any further (and I also forgot what my workaround at that time actually was).

I also think your example should work, and that we should never check for values on disk: if the dims and coordinates names match, write the variable and assume the coordinates are ok.

If the variable already exists on file, match the behavior of netCDF4 (I actually don't know what netCDF4 does in that case)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/issues/1215#issuecomment-334259359, or mute the thread https://github.com/notifications/unsubscribe-auth/ABKS1u6et-S2R4rynbf2vr8bM1Y4WKdjks5so9gEgaJpZM4LmQme .

OK, I understand what's going on now.

Previously, we had a hack that disabled writing variables along dimensions of the form [0, 1, ..., n-1] to disk, because these corresponded to default coordinates and would get created automatically. We disable this hack as part of #1017, because it was no longer necessary.

So although your example worked in v0.8.2, this small variation did not, because we call netCDF4.Dataset.createVariable twice with a dimension of the name 'dim': ```python ds = xr.Dataset() ds['dim'] = ('dim', [1, 2, 3]) ds['var1'] = ('dim', [10, 11, 12]) ds.to_netcdf(path)

ds = xr.Dataset() ds['dim'] = ('dim', [1, 2, 3]) ds['var2'] = ('dim', [10, 11, 12]) ds.to_netcdf(path, 'a') ```

I find it reassuring that this only worked in limited cases before, so it unlikely that many users are depending on this functionality. It would be nice if mode='a' worked to append new variables to an existing netCDF file in the case of overlapping variables, but perhaps we don't need to fix this for v0.9.

My main concern with squeezing this in is that the proper behavior is not entirely clear and will need to go through some review:

• Do we load existing variable values to check them for equality with the new values, or alternatively always skip or override them?
• How do we handle cases where dims, attrs or encoding differs from the exiting variable? Do we attempt to delete and replace the existing variable, update it inplace or error?

Good catch! Marking this as a bug.