Your issue is that the consolidated metadata have not been updated:

```python import gcsfs fs = gcsfs.GCSFileSystem()

the latest array metadata

print(fs.cat('gs://ldeo-glaciology/append_test/test30/temperature/.zarray').decode())

-> "shape": [ 6 ]

the consolidated metadata

print(fs.cat(''gs://ldeo-glaciology/append_test/test30/.zmetadata'').decode())

-> "shape": [ 3 ]

```

There are two ways to fix this.

1. Don't use consolidated metadatda on read. (This will be a bit slower) python ds = xr.open_dataset('gs://ldeo-glaciology/append_test/test30', engine='zarr', consolidated=False)
2. Reconsolidate your metadata after append. https://zarr.readthedocs.io/en/stable/tutorial.html#consolidating-metadata

So there are two layers here where caching could be happening: - gcsfs / fsspec (python) - gcs itself

I propose we eliminate the python layer entirely for the moment. Whenever you load the dataset, it's shape is completely determined by whatever zarr sees in gs://ldeo-glaciology/append_test/test5/temperature/.zarray. So try looking at this file directly. You can figure out its public URL and just do curl, e.g. curl https://storage.googleapis.com/ldeo-glaciology/append_test/test5/temperature/.zarray { "chunks": [ 3 ], "compressor": { "blocksize": 0, "clevel": 5, "cname": "lz4", "id": "blosc", "shuffle": 1 }, "dtype": "<i8", "fill_value": null, "filters": null, "order": "C", "shape": [ 6 ], "zarr_format": 2 }

Run this from jupyterhub from the command line. Then try gcs.cat('ldeo-glaciology/append_test/test5/temperature/.zarray' and see if you see the same thing. Basically just eliminate as many layers as possible from the problem until you get to the core issue.

Ok I think I may understand what is happening

```python

load the zarr store

ds_both = xr.open_zarr(mapper) ```

When you do this, zarr reads a file called gs://ldeo-glaciology/append_test/test5/temperature/.zarray. Since the data are public, I can look at it right now:

$ gsutil cat gs://ldeo-glaciology/append_tet/test5/temperature/.zarray { "chunks": [ 3 ], "compressor": { "blocksize": 0, "clevel": 5, "cname": "lz4", "id": "blosc", "shuffle": 1 }, "dtype": "<i8", "fill_value": null, "filters": null, "order": "C", "shape": [ 6 ], }

Right now, it shows the shape is [6], as expected after the appending. However, if you read the file immediately after appending (within the 3600s max-age), you will get the cached copy. The cached copy will still be of shape [3]--it won't know about the append.

To test this hypothesis, you would need to disable caching on the bucket. Do you have privileges to do that?

Can you post the full stack trace of the error you get when you try to append?

I think that this is not an issue with xarray, zarr, or anything in python world but rather an issue with how caching works on GCS public buckets: https://cloud.google.com/storage/docs/metadata

To test this, forget about xarray and zarr for a minute and just use gcsfs to list the bucket contents before and after your writes. I think you will find that the default cache lifetime of 3600 seconds means that you cannot "see" the changes to the bucket or the objects as quickly as needed in order to append.