issue_comments: 595732104
This data as json
| html_url | issue_url | id | node_id | user | created_at | updated_at | author_association | body | reactions | performed_via_github_app | issue |
|---|---|---|---|---|---|---|---|---|---|---|---|
| https://github.com/pydata/xarray/issues/3831#issuecomment-595732104 | https://api.github.com/repos/pydata/xarray/issues/3831 | 595732104 | MDEyOklzc3VlQ29tbWVudDU5NTczMjEwNA== | 1197350 | 2020-03-06T11:44:02Z | 2020-03-06T11:44:02Z | MEMBER |
This is basically an integration problem. What we are lacking is a comprehensive set of integration tests for this ecosystem (xarray + dask + zarr + fsspec and all its implementations). Pangeo has served as a de facto point for this discussion, since we are using the whole stack. Some similar issues there are: - https://github.com/pangeo-data/pangeo/issues/767 (xarray + opendap) - https://github.com/pangeo-data/pangeo/issues/765 (xarray + zarr + s3fs) - https://github.com/pangeo-data/pangeo/issues/741 (xarray + zarr + gcsfs) - https://github.com/pangeo-data/pangeo/issues/691 etc... All of these libraries understandably want to push the issues somewhere else, since they tend to be complex and hard to reduce to a MCVE. But there are fundamental issues related to integration that have to be addressed somewhere.
Yes and no. The details of how xarray is talking to these stores may matter. Continuing our planned refactor of the backened classes to use entry points, and formalizing the interface for backends, should help surface problems. The way we do consolidated metadata, for example, is pretty ad hoc:
Better diagnostics and more informative errors is always good. But we don't want to do this randomly. I think it should be part of the backend refactor. When is that happening btw? 🙃 We can't hope to test every possible permutation of xarray / zarr store / fsspec implementation within xarray. One idea I have thought about is an "integration bot", who watches all of these libraries and runs its own integration tests. This bot could even be configured to watch the repos and comment on PRs. That would be a cool project! Would appreciate @jhamman's thoughts here. |
{
"total_count": 1,
"+1": 1,
"-1": 0,
"laugh": 0,
"hooray": 0,
"confused": 0,
"heart": 0,
"rocket": 0,
"eyes": 0
} |
576337745 |