issue_comments

Thanks @keewis !

a. For example, our friends over at Arviz have a InferenceData structure composed of multiple Datasets that is represented on-disk using NetCDF groups: https://arviz-devs.github.io/arviz/notebooks/XarrayforArviZ.html

Just a note that this link has moved to: https://arviz-devs.github.io/arviz/getting_started/XarrayforArviZ.html

Thanks a lot @keewis !

@benbovy it's this comment of yours that convinced me it was a good idea, so thank you!

@shoyer do you think this PR can be merged?

@shikharsg, @jendrikjoe and @rabernat might be interested too.

I guess it's not a bug but just the normal behavior, since da.time is a DataArray, and: python In [4]: da.time - np.timedelta64(1, 'D') Out[4]: <xarray.DataArray 'time' (time: 2)> array(['1999-12-31T00:00:00.000000000', '2000-01-01T00:00:00.000000000'], dtype='datetime64[ns]') Coordinates: * time (time) datetime64[ns] 2000-01-01 2000-01-02

Thanks.

You're right @shikharsg, the chunk_dim argument can be removed. I was not very happy with the complexity it brought as well.

It would be great if you could fix it. @jendrikjoe can give you the permission to push to the branch in his fork.

Thanks @shikharsg for looking into that. This PR and master have diverged quite a bit, I will need to merge the changes, I will let you know.

No problem @rabernat, and thanks a lot for your time in reviewing this PR.

I added a test for chunk_dim. Please let me know if this is clearer, and if I should explain further in the docs.

I added a chunk_dim parameter which allows to rechunk the appended coordinate. I think it is ready for a final review now.

I don't think it's ready yet. I think I should address the chunking issue of the appended dimension, as explained in https://medium.com/pangeo/continuously-extending-zarr-datasets-c54fbad3967d. For instance if we append along a time dimension, the time coordinate (which is a 1-D array) will have very small chunks, instead of maybe only one.

@rabernat you're right, I took your suggestion into account in my last commit. I also rewrote the test.

Thanks @jendrikjoe, I just pushed to your fork: to make sure that the encoding of the appended variables is compatible with the target store, we explicitly put the target store encodings in the appended variable.

Or should I open a new PR?

@jendrikjoe I think you need to give me the permission to push to the branch in your fork.

May I try and take this work over?

Hi @jendrikjoe, do you plan to work on this PR again in the future? I think it would be a great contribution to xarray.

An example involving only zarr (not xarray):

```python import zarr import numpy as np

z1 = zarr.open('a.zarr', mode='w', shape=(1), chunks=(1), dtype='U6') z1[:] = 'foo'

z2 = zarr.open('a.zarr', mode='a') z2.append(np.array(['barbar'], dtype='U6'))

z3 = zarr.open('a.zarr', mode='r') print(z3[:]) ```

It works fine and prints ['foo' 'barbar'], and the encoding is "dtype": "<U6": $ cat a.zarr/.zarray { "chunks": [ 1 ], "compressor": { "blocksize": 0, "clevel": 5, "cname": "lz4", "id": "blosc", "shuffle": 1 }, "dtype": "<U6", "fill_value": "0", "filters": null, "order": "C", "shape": [ 1 ], "zarr_format": 2 }

When we use this feature e.g. to store data that is produced every day, we might start with a data set that has a small size on the time dimension, and thus the chunks will be chosen according to this initial shape. When we append to this data set, will the chunks be kept as in the initial zarr archive? If so, we might end up with a lot of small chunks on the time dimension, where ideally we would have chosen only one chunk.

zarr stores the reference in the .zattrs file: { "_ARRAY_DIMENSIONS": [ "time" ], "calendar": "proleptic_gregorian", "units": "days since 2000-01-01 00:00:00" }

To make it work, time dimensions would have to be treated separately because zarr doesn't encode absolute time values but deltas relative to a reference (see https://github.com/davidbrochart/pangeo_upload/blob/master/py/trmm2pangeo.py#L108).

Hi @jendrikjoe,

Thanks for your PR, I am very interested in it because this is something I was hacking around (see here). In my particular case, I want to append along a time dimension, but it looks like your PR currently doesn't support it. In the following example ds2 should have a time dimension ranging from 2000-01-01 to 2000-01-06: ```python import xarray as xr import pandas as pd

ds0 = xr.Dataset({'temperature': (['time'], [50, 51, 52])}, coords={'time': pd.date_range('2000-01-01', periods=3)}) ds1 = xr.Dataset({'temperature': (['time'], [53, 54, 55])}, coords={'time': pd.date_range('2000-01-04', periods=3)})

ds0.to_zarr('temp') ds1.to_zarr('temp', mode='a', append_dim='time')

ds2 = xr.open_zarr('temp') But it's not the case: ds2.time <xarray.DataArray 'time' (time: 6)> array(['2000-01-01T00:00:00.000000000', '2000-01-02T00:00:00.000000000', '2000-01-03T00:00:00.000000000', '2000-01-01T00:00:00.000000000', '2000-01-02T00:00:00.000000000', '2000-01-03T00:00:00.000000000'], dtype='datetime64[ns]') Coordinates: * time (time) datetime64[ns] 2000-01-01 2000-01-02 ... 2000-01-03 ``` Maybe it's not intended to work with time dimensions yet?

Thanks @shoyer. Unfortunately PR https://github.com/pandas-dev/pandas/pull/22043 has been put on hold.

Thanks @dcherian. For me the issue remains when I create a virtual environment with xarray 0.11.1.