issues: 1433534927

What is your issue?

Hi there,

I've been experiencing a peculiar issue that I hope I could get some further insight on. Some background - I'm processing data from raw NetCDF with Xarray to be written to a Zarr store. The hope is that I can append new pieces of data as they are received. The code is as follows: First I write an empty Zarr store to prepare for incoming data. Each of the dimensions/coordinates are pre-populated with all known values. Dimension w is one million in length. The others vary in length but are generally < 50 ``` empty_ds = xarray.Dataset( coords=dict( a=(a), b=(b), c=(c), d=(d), e=(e), f=(f), g=(g), h=(h), w=(w) ), attrs=dict(description=f"Dataset for {a[0]}") )

ds.to_zarr('./zarr', compute=False, mode="w") ```

I then need to open individual NetCDF files, whose coordinates are a subset of those that exist in the Zarr store. As you can see below, the dataset is ~8MB per file.

I then perform an alignment between the Zarr store, which has all coordinate values, and the individual NetCDF files contents to make sure dimensions and coordinates match:

``` ds1 = xarray.open_mfdataset('data.nc') zarr_ds = xarray.open_zarr(".zarr/")

a, b = xarray.align(zarr_ds, ds1, join="outer", exclude="h") #we exclude h because its the dimension we will append by b.to_zarr("./zarr/", mode="a", append_dim="h") ```

These operations work, however the new aligned dataset b that I want to write to the Zarr store has blown up in size as a result: And after writing back to the Zarr store, results in a large file, whose side should theoretically be ~8MB before compression.

Does anyone have insight as to why this might be happening? I've tried changing chunking settings when both opening and writing data, changing the dtypes of dimensions, etc. My hunch is that it has something to do with the dimension that is one million in length. For context, the data variable contains one million data points that correspond to the one million values of w. Thanks for the help!