html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue
https://github.com/pydata/xarray/issues/5878#issuecomment-1315553661,https://api.github.com/repos/pydata/xarray/issues/5878,1315553661,IC_kwDOAMm_X85OacF9,1197350,2022-11-15T16:22:30Z,2022-11-15T16:22:30Z,MEMBER,"Your issue is that the consolidated metadata have not been updated:

```python
import gcsfs
fs = gcsfs.GCSFileSystem()
# the latest array metadata
print(fs.cat('gs://ldeo-glaciology/append_test/test30/temperature/.zarray').decode())
# ->     ""shape"": [ 6 ]
# the consolidated metadata
print(fs.cat(''gs://ldeo-glaciology/append_test/test30/.zmetadata'').decode())
# ->     ""shape"": [ 3 ]
```

There are two ways to fix this.

1. Don't use consolidated metadatda on read. (This will be a bit slower)
   ```python
   ds = xr.open_dataset('gs://ldeo-glaciology/append_test/test30', engine='zarr', consolidated=False)
   ```
1. Reconsolidate your metadata after append. https://zarr.readthedocs.io/en/stable/tutorial.html#consolidating-metadata

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1030811490
https://github.com/pydata/xarray/issues/5878#issuecomment-969021506,https://api.github.com/repos/pydata/xarray/issues/5878,969021506,IC_kwDOAMm_X845whhC,1197350,2021-11-15T15:25:37Z,2021-11-15T15:25:46Z,MEMBER,"So there are two layers here where caching could be happening:
- gcsfs / fsspec (python)
- gcs itself 

I propose we eliminate the python layer entirely for the moment. Whenever you load the dataset, it's shape is completely determined by whatever zarr sees in `gs://ldeo-glaciology/append_test/test5/temperature/.zarray`. So try looking at this file directly. You can figure out its public URL and just do curl, e.g.
```
curl https://storage.googleapis.com/ldeo-glaciology/append_test/test5/temperature/.zarray
{
    ""chunks"": [
        3
    ],
    ""compressor"": {
        ""blocksize"": 0,
        ""clevel"": 5,
        ""cname"": ""lz4"",
        ""id"": ""blosc"",
        ""shuffle"": 1
    },
    ""dtype"": ""<i8"",
    ""fill_value"": null,
    ""filters"": null,
    ""order"": ""C"",
    ""shape"": [
        6
    ],
    ""zarr_format"": 2
}
```

Run this from jupyterhub from the command line. Then try `gcs.cat('ldeo-glaciology/append_test/test5/temperature/.zarray'` and see if you see the same thing. Basically just eliminate as many layers as possible from the problem until you get to the core issue.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1030811490
https://github.com/pydata/xarray/issues/5878#issuecomment-967363845,https://api.github.com/repos/pydata/xarray/issues/5878,967363845,IC_kwDOAMm_X845qM0F,1197350,2021-11-12T19:18:38Z,2021-11-12T19:18:38Z,MEMBER,"Ok I think I may understand what is happening

```python
## load the zarr store
ds_both = xr.open_zarr(mapper)
```

When you do this, zarr reads a file called `gs://ldeo-glaciology/append_test/test5/temperature/.zarray`. Since the data are public, I can look at it right now:

```
$ gsutil cat gs://ldeo-glaciology/append_tet/test5/temperature/.zarray
{
    ""chunks"": [
        3
    ],
    ""compressor"": {
        ""blocksize"": 0,
        ""clevel"": 5,
        ""cname"": ""lz4"",
        ""id"": ""blosc"",
        ""shuffle"": 1
    },
    ""dtype"": ""<i8"",
    ""fill_value"": null,
    ""filters"": null,
    ""order"": ""C"",
    ""shape"": [
        6
    ],
}
```

Right now, it shows the shape is `[6]`, as expected after the appending. However, if you read the file immediately after appending (within the 3600s `max-age`), you will get the cached copy. The cached copy will still be of shape `[3]`--it won't know about the append.

To test this hypothesis, you would need to [disable caching](https://cloud.google.com/storage/docs/metadata) on the bucket. Do you have privileges to do that? ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1030811490
https://github.com/pydata/xarray/issues/5878#issuecomment-967142419,https://api.github.com/repos/pydata/xarray/issues/5878,967142419,IC_kwDOAMm_X845pWwT,1197350,2021-11-12T14:05:36Z,2021-11-12T14:05:36Z,MEMBER,Can you post the full stack trace of the error you get when you try to append?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1030811490
https://github.com/pydata/xarray/issues/5878#issuecomment-966665066,https://api.github.com/repos/pydata/xarray/issues/5878,966665066,IC_kwDOAMm_X845niNq,1197350,2021-11-11T22:17:32Z,2021-11-11T22:17:32Z,MEMBER,"I think that this is not an issue with xarray, zarr, or anything in python world but rather an issue with how caching works on GCS public buckets: https://cloud.google.com/storage/docs/metadata

To test this, forget about xarray and zarr for a minute and just use [gcsfs](https://gcsfs.readthedocs.io/en/latest/) to list the bucket contents before and after your writes. I think you will find that the default cache lifetime of 3600 seconds means that you cannot ""see"" the changes to the bucket or the objects as quickly as needed in order to append.","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1030811490