html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue
https://github.com/pydata/xarray/issues/7697#issuecomment-1489341690,https://api.github.com/repos/pydata/xarray/issues/7697,1489341690,IC_kwDOAMm_X85YxYz6,2448579,2023-03-29T21:20:59Z,2023-03-29T21:20:59Z,MEMBER,"> I thought the compat='override' option bypassed most of the consistency checking.
we still construct a dataset representation for each file which involves reading all coordinates etc. The consistency checking is bypassed at the ""concatenation"" stage.
You could also speed using dask by setting up a cluster and using `parallel=True`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1646267547
https://github.com/pydata/xarray/issues/7697#issuecomment-1489312337,https://api.github.com/repos/pydata/xarray/issues/7697,1489312337,IC_kwDOAMm_X85YxRpR,10678620,2023-03-29T20:59:24Z,2023-03-29T20:59:24Z,NONE,"@dcherian I'll look at that. I thought the `compat='override'` option bypassed most of the consistency checking. In my case, it is typically safe to assume the set of files are consistent (each file represents one timestep, the structure of each file is otherwise identical).
@headtr1ck I was just informed that the underlying filesystem is actually a networked filesystem. The PR might still be useful, but the latest profile seems more reasonable in light of my new info.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1646267547
https://github.com/pydata/xarray/issues/7697#issuecomment-1489302292,https://api.github.com/repos/pydata/xarray/issues/7697,1489302292,IC_kwDOAMm_X85YxPMU,2448579,2023-03-29T20:53:37Z,2023-03-29T20:53:37Z,MEMBER,"Fundamentally, xarray has to touch every file because there is no guarantee they are consistent with each other.
A number of us now use [kerchunk](https://fsspec.github.io/kerchunk/) to create virtual aggregate datasets that can be read a lot faster.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1646267547
https://github.com/pydata/xarray/issues/7697#issuecomment-1489267595,https://api.github.com/repos/pydata/xarray/issues/7697,1489267595,IC_kwDOAMm_X85YxGuL,10678620,2023-03-29T20:30:49Z,2023-03-29T20:33:28Z,NONE,"> It seems that this problematic code is mostly used to determine the engine that is used to finally open it. Did you try specifying the correct engine directly?
I tried setting the engine to 'netcdf4' and while it did help a little bit, it still seems slow on my system.
Here is my profile with `engine='netcdf4'`
I'm not sure what to make of this profile. I don't see anything in the file_manager that would be especially slow. Perhaps it is a filesystem bottleneck at this point (given that the cpu time is 132s of the total 288s duration).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1646267547
https://github.com/pydata/xarray/issues/7697#issuecomment-1489146483,https://api.github.com/repos/pydata/xarray/issues/7697,1489146483,IC_kwDOAMm_X85YwpJz,43316012,2023-03-29T19:02:39Z,2023-03-29T19:02:39Z,COLLABORATOR,It seems that this problematic code is mostly used to determine the engine that is used to finally open it. Did you try specifying the correct engine directly?,"{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1646267547
https://github.com/pydata/xarray/issues/7697#issuecomment-1489083542,https://api.github.com/repos/pydata/xarray/issues/7697,1489083542,IC_kwDOAMm_X85YwZyW,14371165,2023-03-29T18:17:35Z,2023-03-29T18:17:35Z,MEMBER,Looks like you almost got this figured out! You want to create a PR for this?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,1646267547