id,node_id,number,title,user,state,locked,assignee,milestone,comments,created_at,updated_at,closed_at,author_association,active_lock_reason,draft,pull_request,body,reactions,performed_via_github_app,state_reason,repo,type 416962458,MDU6SXNzdWU0MTY5NjI0NTg=,2799,Performance: numpy indexes small amounts of data 1000 faster than xarray,1386642,open,0,,,42,2019-03-04T19:44:17Z,2024-03-18T17:51:25Z,,CONTRIBUTOR,,,,"Machine learning applications often require iterating over every index along some of the dimensions of a dataset. For instance, iterating over all the `(lat, lon)` pairs in a 4D dataset with dimensions `(time, level, lat, lon)`. Unfortunately, this is very slow with xarray objects compared to numpy (or h5py) arrays. When the Pangeo machine learning working group met [today](https://github.com/pangeo-data/ml-workflow-examples/issues/1), we found that several of us have struggled with this. I made some simplified [benchmarks](https://gist.github.com/nbren12/e781c5a8fe03ee170628194c4b3c3160), which show that xarray is about 1000 times slower than numpy when repeatedly grabbing a small amount of data from an array. This is a problem with both `isel` or `[]` indexing. After doing some profiling, the main culprits seem to be xarray routines like `_validate_indexers` and `_broadcast_indexes`. While python will always be slower than C when iterating over an array in this fashion, I would hope that xarray could be nearly as fast as numpy. I am not sure what the best way to improve this is though. ","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/2799/reactions"", ""total_count"": 9, ""+1"": 9, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,issue 1473152374,I_kwDOAMm_X85XzoV2,7348,Using entry_points to register dataset and dataarray accessors?,1386642,open,0,,,4,2022-12-02T16:48:42Z,2023-09-14T19:53:46Z,,CONTRIBUTOR,,,,"### Is your feature request related to a problem? External libraries often use the dataset/dataarray accessor pattern (e.g. [metpy](https://github.com/Unidata/MetPy/blob/f568aca6325cb23cfccc1006c4965ef7f7b5ad29/src/metpy/xarray.py#L105)). These accessors are not available until importing the external package where the registration occurs. This means scripts using these accessors must include an often-unused import that linters will complain about e.g. ``` import metpy # linter complains here # some data ds: xr.Dataset = ... ds.metpy.... ``` ### Describe the solution you'd like Use importlib entrypoints to register these as entrypoints so that registration is automatically handled. This is currently enabled for the array backend, but not for accessors (e.g. [metpy's setup.cfg](https://github.com/Unidata/MetPy/blob/f568aca6325cb23cfccc1006c4965ef7f7b5ad29/src/metpy/xarray.py#L105)). ### Describe alternatives you've considered _No response_ ### Additional context _No response_","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/7348/reactions"", ""total_count"": 2, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 1}",,,13221727,issue 753852119,MDU6SXNzdWU3NTM4NTIxMTk=,4628,Lazy concatenation of arrays,1386642,open,0,,,5,2020-11-30T22:32:08Z,2022-05-10T17:02:34Z,,CONTRIBUTOR,,,,"**Is your feature request related to a problem? Please describe.** Concatenating xarray objects forces the data to load. I recently learned about this [object](https://github.com/pydata/xarray/blob/235b2e5bcec253ca6a85762323121d28c3b06038/xarray/core/indexing.py#L592) allowing lazy indexing into an DataArrays/sets without using dask. Concatenation along a single dimension is the inverse operation of slicing, so it seems natural to also support it. Also, concatenating along dimensions (e.g. ""run""/""simulation""/""ensemble"") can be a common merging workflow. **Describe the solution you'd like** `xr.concat([a, b], dim=...)` does not load any data in a or b. **Describe alternatives you've considered** One could rename the variables in a and b to allow them to be merged (e.g. `a['air_temperature'] -> ""air_temperature_a""`), but it's more natural to make a new dimension. **Additional context** This is useful when not using dask for performance reasons (e.g. using another parallelism engine like Apache Beam). ","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/4628/reactions"", ""total_count"": 8, ""+1"": 8, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,issue 588112617,MDU6SXNzdWU1ODgxMTI2MTc=,3894,Add public API for Dataset._copy_listed,1386642,open,0,,,15,2020-03-26T02:39:34Z,2022-04-18T16:41:39Z,,CONTRIBUTOR,,,,"In my data pipelines, I have been repeatedly burned using indexing notation to grab a few variables from a dataset in the following way: ``` ds = xr.Dataset(...) vars = ('a' , 'b', 'c') ds[vars] # this errors ds[list(vars)] # this is ok ``` Moreover, because `Dataset__getitem__` is type unstable, it makes it hard to detect this kind of error using mypy, so it often appears 30 minutes into a long data pipeline. It would be great to have a type-stable method that can take any sequence of variable names and return the Dataset consisting of those variables and their coordinates only. In fact, this method already [exists](https://github.com/pydata/xarray/blob/6378a711d50ba7f1ba9b2a451d4d1f5e1fb37353/xarray/core/dataset.py#L1123), but it currently not public API. Could we make it so? Thanks.","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/3894/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,issue 224846826,MDU6SXNzdWUyMjQ4NDY4MjY=,1387,FacetGrid with independent colorbars,1386642,open,0,,,7,2017-04-27T16:47:44Z,2022-04-13T11:07:49Z,,CONTRIBUTOR,,,,"Sometimes the magnitude of a variable can vary dramatically across a given coordinate, which makes 2d plots generated by xr.FacetGrid difficult to interpret. It would be useful if an option to xr.FacetGrid could be specified which allows each subplot to have its own colorbar.","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/1387/reactions"", ""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,issue 1132894350,I_kwDOAMm_X85DhpiO,6269,Adding CDL Parser/`open_cdl`?,1386642,open,0,,,7,2022-02-11T17:31:36Z,2022-02-14T17:18:38Z,,CONTRIBUTOR,,,,"### Is your feature request related to a problem? No. ### Describe the solution you'd like It would be nice to load/generate xarray datasets from Common Data Language ([CDL][1]) descriptions. CDL is a DSL that that defines a netCDF dataset, and is quite nice for testing. We use it to build mock datasets for e.g. integration testing of plotting routines/complex data analysis etc. CDL provides a concise format for storing the schema of this data. This schema can be used for validation or generation (using the CLI `ncgen`). CDL is basically the format produced by `xarray.Dataset.info`. It looks like this: ``` netcdf example { // example of CDL notation dimensions: lon = 3 ; lat = 8 ; variables: float rh(lon, lat) ; rh:units = ""percent"" ; rh:long_name = ""Relative humidity"" ; // global attributes :title = ""Simple example, lacks some conventions"" ; data: /// optional ...ncgen will still build rh = 2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89 ; } ``` I wrote a small pure python [parser](https://github.com/ai2cm/fv3net/blob/5c318e1a594a71baaa502ec4dc6809095b0828d3/external/vcm/vcm/cdl/parser.py#L1) for CDL last night and it seems work! There are [similar projects](https://github.com/rockdoc/cdlparser) on github. Sadly, these projects seem to be abandoned so it would be nice to attach to an effort like xarray. [1]: https://www.unidata.ucar.edu/software/netcdf/workshops/most-recent/nc3model/Cdl.html ### Describe alternatives you've considered Some kind of `schema` object that can be used to validate or generate an xarray Dataset, but does not contain any data. ### Additional context _No response_","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/6269/reactions"", ""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,issue