home / github / issues

Menu
  • Search all tables
  • GraphQL API

issues: 893692903

This data as json

id node_id number title user state locked assignee milestone comments created_at updated_at closed_at author_association active_lock_reason draft pull_request body reactions performed_via_github_app state_reason repo type
893692903 MDU6SXNzdWU4OTM2OTI5MDM= 5326 map_blocks doesn't handle tranposed arrays 20629530 closed 0     7 2021-05-17T20:34:58Z 2021-05-18T14:14:37Z 2021-05-18T14:14:37Z CONTRIBUTOR      

What happened:

I was using map_blocks for a complex function which returns an array with a different dimension order than the input. Because of the complexity of the wrapped func, I need to generate a template first.

When calling map_blocks and loading the result, it passes all checks in map_blocks but Variable fails when assigning the new data.

What you expected to happen: I expected no failure. Either the result would have transposed dimensions, or it would have been transposed back to fit with template.

Minimal Complete Verifiable Example:

```python import xarray as xr

da = xr.DataArray([[0, 1, 2], [3, 4, 5]], dims=('x', 'y'))

def func(d): return d.transpose()

dac = da.chunk() dac.map_blocks(func, template=dac).load() ```

Traceback: ```python


ValueError Traceback (most recent call last) <ipython-input-1-0da1b18a36a8> in <module> 7 8 dac = da.chunk() ----> 9 dac.map_blocks(func, template=dac).load()

~/.conda/envs/xclim/lib/python3.8/site-packages/xarray/core/dataarray.py in load(self, kwargs) 871 dask.compute 872 """ --> 873 ds = self._to_temp_dataset().load(kwargs) 874 new = self._from_temp_dataset(ds) 875 self._variable = new._variable

~/.conda/envs/xclim/lib/python3.8/site-packages/xarray/core/dataset.py in load(self, **kwargs) 799 800 for k, data in zip(lazy_data, evaluated_data): --> 801 self.variables[k].data = data 802 803 # load everything else sequentially

~/.conda/envs/xclim/lib/python3.8/site-packages/xarray/core/variable.py in data(self, data) 378 data = as_compatible_data(data) 379 if data.shape != self.shape: --> 380 raise ValueError( 381 f"replacement data must match the Variable's shape. " 382 f"replacement data has shape {data.shape}; Variable has shape {self.shape}"

ValueError: replacement data must match the Variable's shape. replacement data has shape (3, 2); Variable has shape (2, 3) `` Iffuncis made to returnd` (no transpose), the code works.

I actually not sure which behaviour would be the best : a result with transposed dimensions to fit with the wrapped func or to tranpose the result to fit with the template. The latter seems much easier to implement by editing core/parallel.py and add the transposition at the end of _wrapper() in map_blocks().

Environment:

Output of <tt>xr.show_versions()</tt> INSTALLED VERSIONS ------------------ commit: None python: 3.8.8 | packaged by conda-forge | (default, Feb 20 2021, 16:22:27) [GCC 9.3.0] python-bits: 64 OS: Linux OS-release: 5.11.16-arch1-1 machine: x86_64 processor: byteorder: little LC_ALL: None LANG: fr_CA.utf8 LOCALE: ('fr_CA', 'UTF-8') libhdf5: 1.10.6 libnetcdf: 4.7.4 xarray: 0.17.1.dev99+gc58e2aeb.d20210430 pandas: 1.2.4 numpy: 1.20.2 scipy: 1.6.3 netCDF4: 1.5.6 pydap: installed h5netcdf: 0.11.0 h5py: 3.2.1 Nio: None zarr: 2.8.1 cftime: 1.4.1 nc_time_axis: 1.2.0 PseudoNetCDF: installed rasterio: 1.2.2 cfgrib: 0.9.9.0 iris: 2.4.0 bottleneck: 1.3.2 dask: 2021.04.0 distributed: 2021.04.1 matplotlib: 3.4.1 cartopy: 0.19.0 seaborn: 0.11.1 numbagg: installed pint: 0.17 setuptools: 49.6.0.post20210108 pip: 21.1 conda: None pytest: 6.2.3 IPython: 7.22.0 sphinx: 3.5.4
{
    "url": "https://api.github.com/repos/pydata/xarray/issues/5326/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  completed 13221727 issue

Links from other tables

  • 0 rows from issues_id in issues_labels
  • 7 rows from issue in issue_comments
Powered by Datasette · Queries took 7.119ms · About: xarray-datasette