html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue
https://github.com/pydata/xarray/issues/7248#issuecomment-1302244548,https://api.github.com/repos/pydata/xarray/issues/7248,1302244548,IC_kwDOAMm_X85NnqzE,2448579,2022-11-03T15:00:47Z,2022-11-03T15:00:47Z,MEMBER,I think perhaps the `region` kwarg is a better solution than appending for your case: https://docs.xarray.dev/en/stable/user-guide/io.html#appending-to-existing-zarr-stores,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1433534927
https://github.com/pydata/xarray/issues/7248#issuecomment-1301171446,https://api.github.com/repos/pydata/xarray/issues/7248,1301171446,IC_kwDOAMm_X85Njkz2,92732695,2022-11-02T20:09:36Z,2022-11-02T20:09:36Z,NONE,"> The size of the data seems consistent with that (i.e. `2*27*4*3/2*48*8MB=125GB`, though not exactly 58GB, looking at the difference in the ""Array Shape"" between the two images)
This would be correct if we added that many 8MB data sets, but we just added one! So I'd expect the entire store to be roughly that size (8MB) before compression since in the code example above we are just adding one data array to the Zarr data store. Would you be able to clarify this calculation `2*27*4*3/2*48*8MB=125GB` ?
>
> Does looking at the values of the dimension that's grown to size 48 help — are those possibly different, even though you don't intend them to be, and causing it to be huge (but sparse!)?
>
> Does that make sense?
So the reason why all the dimension sizes have grown is because we initialize the Zarr with the set of all possibilities that the coordinates might be (read somewhere that this is what you should do - initialize an empty Zarr with the coords/dimensions you will need). Then what I figured I needed to do was align the dimensions of the .nc file with that of the Zarr store, so that when the data array is added the coordinates and dimensions line up - this is why we see the dimensions grow.
Is this the correct approach? I found the limitation/design of `to_zarr()` that allows you to only append to one dimension meant that if you want to append new data to a Zarr store, all dimensions/coords must pre-exist in both the Zarr store and individual DataSet that you'd like to append. This is what drives the decision to perform `xarray.align`.
Without aligning, I would get error messages that complained about dimension sizes being different. Thanks again for your help!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1433534927
https://github.com/pydata/xarray/issues/7248#issuecomment-1301082744,https://api.github.com/repos/pydata/xarray/issues/7248,1301082744,IC_kwDOAMm_X85NjPJ4,5635139,2022-11-02T18:57:48Z,2022-11-02T18:57:48Z,MEMBER,"Even without looking at the size of the data, the dimensions of the array seem to have grown a lot!
The size of the data seems consistent with that (i.e. `2*27*4*3/2*48*8MB=125GB`, though not exactly 58GB, looking at the difference in the ""Array Shape"" between the two images)
Does looking at the values of the dimension that's grown to size 48 help — are those possibly different, even though you don't intend them to be, and causing it to be huge (but sparse!)?
Does that make sense?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1433534927