issues

Table actions

2 rows where user = 2552981 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date), closed_at (date)

type 2

state 1

repo 1

id node_id number title user state locked assignee milestone comments created_at updated_at ▲ closed_at author_association active_lock_reason draft pull_request body reactions performed_via_github_app state_reason repo type
406971782 MDU6SXNzdWU0MDY5NzE3ODI= 2747 1000x performance regression in transpose from disk between libnetcdf 4.6.1 and 4.6.2 coroa 2552981 closed 0     3 2019-02-05T20:59:53Z 2020-08-23T18:53:41Z 2020-08-23T18:53:41Z CONTRIBUTOR      

Having generated test.nc as the two-dimensional array from ```python import numpy as np import xarray as xr

a = np.random.random((1000, 100)) ds = xr.Dataset({'foo': xr.DataArray(a, [('x', np.arange(1000)), ('y', np.arange(100))])}) ds.to_netcdf('test.nc') ```

I am seeing a huge performance regression from libnetcdf=4.6.1 python In [2]: ds = xr.open_dataset('test.nc') ...: %time np.asarray(ds['foo'].transpose('y', 'x')) ...: CPU times: user 2.58 ms, sys: 191 µs, total: 2.77 ms in a conda environment created by conda create -n mwe-fast 'libnetcdf=4.6.1' netcdf4 xarray ipython to the environment created by conda create -n mwe-slow 'libnetcdf=4.6.2' netcdf4 xarray ipython python In [5]: ds = xr.open_dataset('test.nc') ...: %time np.asarray(ds['foo'].transpose('y', 'x')) ...: CPU times: user 4.09 s, sys: 12 ms, total: 4.11 s

Loading into memory mitigates the regression (on mwe-slow): python In [2]: ds = xr.open_dataset('test.nc') ...: %time np.asarray(ds['foo'].load().transpose('y', 'x')) ...: CPU times: user 1.12 ms, sys: 80 µs, total: 1.2 ms

Output of xr.show_versions()

INSTALLED VERSIONS on the working environment `mwe-fast` ------------------ commit: None python: 3.7.1 | packaged by conda-forge | (default, Nov 13 2018, 18:33:04) [GCC 7.3.0] python-bits: 64 OS: Linux OS-release: 4.19.0-2-amd64 machine: x86_64 processor: byteorder: little LC_ALL: None LANG: en_GB.UTF-8 LOCALE: en_GB.UTF-8 libhdf5: 1.10.3 libnetcdf: 4.6.1 xarray: 0.11.3 pandas: 0.24.1 numpy: 1.15.4 scipy: None netCDF4: 1.4.2 pydap: None h5netcdf: None h5py: None Nio: None zarr: None cftime: 1.0.3.4 PseudonetCDF: None rasterio: None cfgrib: None iris: None bottleneck: None cyordereddict: None dask: None distributed: None matplotlib: None cartopy: None seaborn: None setuptools: 40.7.3 pip: 19.0.1 conda: None pytest: None IPython: 7.2.0 sphinx: None INSTALLED VERSIONS on the broken environment `mwe-slow` ------------------ commit: None python: 3.7.1 | packaged by conda-forge | (default, Nov 13 2018, 18:33:04) [GCC 7.3.0] python-bits: 64 OS: Linux OS-release: 4.19.0-2-amd64 machine: x86_64 processor: byteorder: little LC_ALL: None LANG: en_GB.UTF-8 LOCALE: en_GB.UTF-8 libhdf5: 1.10.4 libnetcdf: 4.6.2 xarray: 0.11.3 pandas: 0.24.1 numpy: 1.15.4 scipy: None netCDF4: 1.4.2 pydap: None h5netcdf: None h5py: None Nio: None zarr: None cftime: 1.0.3.4 PseudonetCDF: None rasterio: None cfgrib: None iris: None bottleneck: None cyordereddict: None dask: None distributed: None matplotlib: None cartopy: None seaborn: None setuptools: 40.7.3 pip: 19.0.1 conda: None pytest: None IPython: 7.2.0 sphinx: None 
{
    "url": "https://api.github.com/repos/pydata/xarray/issues/2747/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed xarray 13221727 issue
464152437 MDExOlB1bGxSZXF1ZXN0Mjk0NDg5OTE4 3079 Fix printing summaries of multiindex coords coroa 2552981 closed 0     7 2019-07-04T09:05:23Z 2019-07-11T07:48:56Z 2019-07-10T16:48:01Z CONTRIBUTOR   0 pydata/xarray/pulls/3079

Since merging #2293 summarize_variable displays the first and last few entries, so we have to pass all along.

MWE: python import xarray as xr import numpy as np x = np.arange(0, 90, 0.5) y = np.arange(0, 90, 0.5) da = xr.DataArray(np.random.random((len(x), len(y))), [('x', x), ('y', y)]) da.stack(z=('x', 'y')) gives on master: <xarray.DataArray (z: 32400)> array([0.497724, 0.793955, 0.140193, ..., 0.05734 , 0.884105, 0.479093]) Coordinates: * z (z) MultiIndex - x (z) float64 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 - y (z) float64 0.0 0.5 1.0 1.5 2.0 2.5 ... 12.5 13.0 13.5 14.0 14.5 and on this branch <xarray.DataArray (z: 32400)> array([0.497724, 0.793955, 0.140193, ..., 0.05734 , 0.884105, 0.479093]) Coordinates: * z (z) MultiIndex - x (z) float64 0.0 0.0 0.0 0.0 0.0 0.0 ... 89.5 89.5 89.5 89.5 89.5 89.5 - y (z) float64 0.0 0.5 1.0 1.5 2.0 2.5 ... 87.0 87.5 88.0 88.5 89.0 89.5

  • [ ] Closes #xxxx
  • [x] Tests added
  • [ ] Fully documented, including whats-new.rst for all changes and api.rst for new API
 
{
    "url": "https://api.github.com/repos/pydata/xarray/issues/3079/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
     xarray 13221727 pull

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issues] (
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [number] INTEGER,
   [title] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [state] TEXT,
   [locked] INTEGER,
   [assignee] INTEGER REFERENCES [users]([id]),
   [milestone] INTEGER REFERENCES [milestones]([id]),
   [comments] INTEGER,
   [created_at] TEXT,
   [updated_at] TEXT,
   [closed_at] TEXT,
   [author_association] TEXT,
   [active_lock_reason] TEXT,
   [draft] INTEGER,
   [pull_request] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [state_reason] TEXT,
   [repo] INTEGER REFERENCES [repos]([id]),
   [type] TEXT
);
CREATE INDEX [idx_issues_repo]
    ON [issues] ([repo]);
CREATE INDEX [idx_issues_milestone]
    ON [issues] ([milestone]);
CREATE INDEX [idx_issues_assignee]
    ON [issues] ([assignee]);
CREATE INDEX [idx_issues_user]
    ON [issues] ([user]);