id,node_id,number,title,user,state,locked,assignee,milestone,comments,created_at,updated_at,closed_at,author_association,active_lock_reason,draft,pull_request,body,reactions,performed_via_github_app,state_reason,repo,type 406971782,MDU6SXNzdWU0MDY5NzE3ODI=,2747,1000x performance regression in transpose from disk between libnetcdf 4.6.1 and 4.6.2,2552981,closed,0,,,3,2019-02-05T20:59:53Z,2020-08-23T18:53:41Z,2020-08-23T18:53:41Z,CONTRIBUTOR,,,,"Having generated `test.nc` as the two-dimensional array from ```python import numpy as np import xarray as xr a = np.random.random((1000, 100)) ds = xr.Dataset({'foo': xr.DataArray(a, [('x', np.arange(1000)), ('y', np.arange(100))])}) ds.to_netcdf('test.nc') ``` I am seeing a huge performance regression from libnetcdf=4.6.1 ```python In [2]: ds = xr.open_dataset('test.nc') ...: %time np.asarray(ds['foo'].transpose('y', 'x')) ...: CPU times: user 2.58 ms, sys: 191 µs, total: 2.77 ms ``` in a conda environment created by ``` conda create -n mwe-fast 'libnetcdf=4.6.1' netcdf4 xarray ipython ``` to the environment created by ``` conda create -n mwe-slow 'libnetcdf=4.6.2' netcdf4 xarray ipython ``` ```python In [5]: ds = xr.open_dataset('test.nc') ...: %time np.asarray(ds['foo'].transpose('y', 'x')) ...: CPU times: user 4.09 s, sys: 12 ms, total: 4.11 s ``` Loading into memory mitigates the regression (on `mwe-slow`): ```python In [2]: ds = xr.open_dataset('test.nc') ...: %time np.asarray(ds['foo'].load().transpose('y', 'x')) ...: CPU times: user 1.12 ms, sys: 80 µs, total: 1.2 ms ``` #### Output of ``xr.show_versions()``
INSTALLED VERSIONS on the working environment `mwe-fast` ------------------ commit: None python: 3.7.1 | packaged by conda-forge | (default, Nov 13 2018, 18:33:04) [GCC 7.3.0] python-bits: 64 OS: Linux OS-release: 4.19.0-2-amd64 machine: x86_64 processor: byteorder: little LC_ALL: None LANG: en_GB.UTF-8 LOCALE: en_GB.UTF-8 libhdf5: 1.10.3 libnetcdf: 4.6.1 xarray: 0.11.3 pandas: 0.24.1 numpy: 1.15.4 scipy: None netCDF4: 1.4.2 pydap: None h5netcdf: None h5py: None Nio: None zarr: None cftime: 1.0.3.4 PseudonetCDF: None rasterio: None cfgrib: None iris: None bottleneck: None cyordereddict: None dask: None distributed: None matplotlib: None cartopy: None seaborn: None setuptools: 40.7.3 pip: 19.0.1 conda: None pytest: None IPython: 7.2.0 sphinx: None INSTALLED VERSIONS on the broken environment `mwe-slow` ------------------ commit: None python: 3.7.1 | packaged by conda-forge | (default, Nov 13 2018, 18:33:04) [GCC 7.3.0] python-bits: 64 OS: Linux OS-release: 4.19.0-2-amd64 machine: x86_64 processor: byteorder: little LC_ALL: None LANG: en_GB.UTF-8 LOCALE: en_GB.UTF-8 libhdf5: 1.10.4 libnetcdf: 4.6.2 xarray: 0.11.3 pandas: 0.24.1 numpy: 1.15.4 scipy: None netCDF4: 1.4.2 pydap: None h5netcdf: None h5py: None Nio: None zarr: None cftime: 1.0.3.4 PseudonetCDF: None rasterio: None cfgrib: None iris: None bottleneck: None cyordereddict: None dask: None distributed: None matplotlib: None cartopy: None seaborn: None setuptools: 40.7.3 pip: 19.0.1 conda: None pytest: None IPython: 7.2.0 sphinx: None
","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/2747/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed,13221727,issue 464152437,MDExOlB1bGxSZXF1ZXN0Mjk0NDg5OTE4,3079,Fix printing summaries of multiindex coords,2552981,closed,0,,,7,2019-07-04T09:05:23Z,2019-07-11T07:48:56Z,2019-07-10T16:48:01Z,CONTRIBUTOR,,0,pydata/xarray/pulls/3079,"Since merging #2293 summarize_variable displays the first and last few entries, so we have to pass all along. MWE: ```python import xarray as xr import numpy as np x = np.arange(0, 90, 0.5) y = np.arange(0, 90, 0.5) da = xr.DataArray(np.random.random((len(x), len(y))), [('x', x), ('y', y)]) da.stack(z=('x', 'y')) ``` gives on master: ``` array([0.497724, 0.793955, 0.140193, ..., 0.05734 , 0.884105, 0.479093]) Coordinates: * z (z) MultiIndex - x (z) float64 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 - y (z) float64 0.0 0.5 1.0 1.5 2.0 2.5 ... 12.5 13.0 13.5 14.0 14.5 ``` and on this branch ``` array([0.497724, 0.793955, 0.140193, ..., 0.05734 , 0.884105, 0.479093]) Coordinates: * z (z) MultiIndex - x (z) float64 0.0 0.0 0.0 0.0 0.0 0.0 ... 89.5 89.5 89.5 89.5 89.5 89.5 - y (z) float64 0.0 0.5 1.0 1.5 2.0 2.5 ... 87.0 87.5 88.0 88.5 89.0 89.5 ``` - [ ] Closes #xxxx - [x] Tests added - [ ] Fully documented, including `whats-new.rst` for all changes and `api.rst` for new API ","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/3079/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,pull