issue_comments: 1454850970

This data as json

html_url issue_url id node_id user created_at updated_at author_association body reactions performed_via_github_app issue
https://github.com/pydata/xarray/pull/7540#issuecomment-1454850970 https://api.github.com/repos/pydata/xarray/issues/7540 1454850970 IC_kwDOAMm_X85Wt0Oa 7460993 2023-03-04T19:32:19Z 2023-03-04T19:32:19Z NONE

Hi @jhamman,

I have added a test and corrected a problem that I had previously missed. It turns out that a lock must be used to ensure the correct writing of a sharded zarr store. Everything seems to be working as expected now. Here are some comments:

  • Currently, a lock is passed to ArrayWriter even if only one variable is being sharded. While I believe that locks would not be needed to write non-sharded variables, I don't think it is possible to restrict their usage to specific variables, since the zarr store is written as a whole. Any thoughts on this?
  • Related to the previous point, I think the usage of locks could even be made shard-specific, relaxing the writing of different shard files. I also believe this is not currently possible, but let me know if there is a solution.
  • For the test, importing ShardingStorageTransformer is currently a bit awkward, but it can be neated in the future when ShardingStorageTransformer becomes available from the top-level zarr namespace.

Let me know what you think.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   1588516592