id,node_id,number,title,user,state,locked,assignee,milestone,comments,created_at,updated_at,closed_at,author_association,active_lock_reason,draft,pull_request,body,reactions,performed_via_github_app,state_reason,repo,type 684070245,MDExOlB1bGxSZXF1ZXN0NDcyMDQxNjA2,4368,pyupgrade,5635139,closed,0,,,0,2020-08-22T21:29:03Z,2020-08-23T22:29:52Z,2020-08-23T21:09:52Z,MEMBER,,0,pydata/xarray/pulls/4368,,"{""url"": ""https://api.github.com/repos/pydata/xarray/issues/4368/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,pull 684248425,MDU6SXNzdWU2ODQyNDg0MjU=,4370,"Not able to slice dataset using its own coordinate value, after upgrade to pandas 1.1.0",18426375,closed,0,,,4,2020-08-23T20:10:56Z,2020-08-23T20:26:58Z,2020-08-23T20:26:57Z,NONE,,,,"I seem to be having the same issue that was reported here: https://github.com/pydata/xarray/issues/1932, after upgrading pandas to 1.1.0. The error does not arise with pandas 1.0.3. Example: `import xarray as` `ds = xr.tutorial.load_dataset('air_temperature')` `da = ds.air` `da.sel(time=da.time[0])` Result is: `--------------------------------------------------------------------------- KeyError Traceback (most recent call last) in ----> 1 da.sel(time=da.time[0]) 2 #da.time ~/miniconda3/envs/ats641/lib/python3.7/site-packages/xarray/core/dataarray.py in sel(self, indexers, method, tolerance, drop, **indexers_kwargs) 1152 method=method, 1153 tolerance=tolerance, -> 1154 **indexers_kwargs, 1155 ) 1156 return self._from_temp_dataset(ds) ~/miniconda3/envs/ats641/lib/python3.7/site-packages/xarray/core/dataset.py in sel(self, indexers, method, tolerance, drop, **indexers_kwargs) 2100 indexers = either_dict_or_kwargs(indexers, indexers_kwargs, ""sel"") 2101 pos_indexers, new_indexes = remap_label_indexers( -> 2102 self, indexers=indexers, method=method, tolerance=tolerance 2103 ) 2104 result = self.isel(indexers=pos_indexers, drop=drop) ~/miniconda3/envs/ats641/lib/python3.7/site-packages/xarray/core/coordinates.py in remap_label_indexers(obj, indexers, method, tolerance, **indexers_kwargs) 395 396 pos_indexers, new_indexes = indexing.remap_label_indexers( --> 397 obj, v_indexers, method=method, tolerance=tolerance 398 ) 399 # attach indexer's coordinate to pos_indexers ~/miniconda3/envs/ats641/lib/python3.7/site-packages/xarray/core/indexing.py in remap_label_indexers(data_obj, indexers, method, tolerance) 268 coords_dtype = data_obj.coords[dim].dtype 269 label = maybe_cast_to_coords_dtype(label, coords_dtype) --> 270 idxr, new_idx = convert_label_indexer(index, label, dim, method, tolerance) 271 pos_indexers[dim] = idxr 272 if new_idx is not None: ~/miniconda3/envs/ats641/lib/python3.7/site-packages/xarray/core/indexing.py in convert_label_indexer(index, label, index_name, method, tolerance) 188 else: 189 indexer = index.get_loc( --> 190 label.item(), method=method, tolerance=tolerance 191 ) 192 elif label.dtype.kind == ""b"": ~/miniconda3/envs/ats641/lib/python3.7/site-packages/pandas/core/indexes/datetimes.py in get_loc(self, key, method, tolerance) 620 else: 621 # unrecognized type --> 622 raise KeyError(key) 623 624 try: KeyError: 1356998400000000000` Output of xr.show_versions() is: ` INSTALLED VERSIONS ------------------ commit: None python: 3.7.6 | packaged by conda-forge | (default, Jan 7 2020, 22:05:27) [Clang 9.0.1 ] python-bits: 64 OS: Darwin OS-release: 19.6.0 machine: x86_64 processor: i386 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: en_US.UTF-8 libhdf5: 1.10.5 libnetcdf: 4.7.3 xarray: 0.16.0 pandas: 1.1.0 numpy: 1.18.1 scipy: 1.5.2 netCDF4: 1.5.3 pydap: None h5netcdf: None h5py: None Nio: 1.5.5 zarr: None cftime: 1.2.1 nc_time_axis: None PseudoNetCDF: None rasterio: None cfgrib: 0.9.8.3 iris: None bottleneck: None dask: 2.20.0 distributed: None matplotlib: 3.1.3 cartopy: 0.17.0 seaborn: 0.10.1 numbagg: None pint: 0.15 setuptools: 49.6.0.post20200814 pip: 20.0.2 conda: None pytest: None IPython: 7.17.0 sphinx: None ` With pandas 1.0.3, da.sel(time=da.time[0]) works correctly and returns a slice. Thanks for any help you can offer! ","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/4370/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed,13221727,issue 406971782,MDU6SXNzdWU0MDY5NzE3ODI=,2747,1000x performance regression in transpose from disk between libnetcdf 4.6.1 and 4.6.2,2552981,closed,0,,,3,2019-02-05T20:59:53Z,2020-08-23T18:53:41Z,2020-08-23T18:53:41Z,CONTRIBUTOR,,,,"Having generated `test.nc` as the two-dimensional array from ```python import numpy as np import xarray as xr a = np.random.random((1000, 100)) ds = xr.Dataset({'foo': xr.DataArray(a, [('x', np.arange(1000)), ('y', np.arange(100))])}) ds.to_netcdf('test.nc') ``` I am seeing a huge performance regression from libnetcdf=4.6.1 ```python In [2]: ds = xr.open_dataset('test.nc') ...: %time np.asarray(ds['foo'].transpose('y', 'x')) ...: CPU times: user 2.58 ms, sys: 191 µs, total: 2.77 ms ``` in a conda environment created by ``` conda create -n mwe-fast 'libnetcdf=4.6.1' netcdf4 xarray ipython ``` to the environment created by ``` conda create -n mwe-slow 'libnetcdf=4.6.2' netcdf4 xarray ipython ``` ```python In [5]: ds = xr.open_dataset('test.nc') ...: %time np.asarray(ds['foo'].transpose('y', 'x')) ...: CPU times: user 4.09 s, sys: 12 ms, total: 4.11 s ``` Loading into memory mitigates the regression (on `mwe-slow`): ```python In [2]: ds = xr.open_dataset('test.nc') ...: %time np.asarray(ds['foo'].load().transpose('y', 'x')) ...: CPU times: user 1.12 ms, sys: 80 µs, total: 1.2 ms ``` #### Output of ``xr.show_versions()``
INSTALLED VERSIONS on the working environment `mwe-fast` ------------------ commit: None python: 3.7.1 | packaged by conda-forge | (default, Nov 13 2018, 18:33:04) [GCC 7.3.0] python-bits: 64 OS: Linux OS-release: 4.19.0-2-amd64 machine: x86_64 processor: byteorder: little LC_ALL: None LANG: en_GB.UTF-8 LOCALE: en_GB.UTF-8 libhdf5: 1.10.3 libnetcdf: 4.6.1 xarray: 0.11.3 pandas: 0.24.1 numpy: 1.15.4 scipy: None netCDF4: 1.4.2 pydap: None h5netcdf: None h5py: None Nio: None zarr: None cftime: 1.0.3.4 PseudonetCDF: None rasterio: None cfgrib: None iris: None bottleneck: None cyordereddict: None dask: None distributed: None matplotlib: None cartopy: None seaborn: None setuptools: 40.7.3 pip: 19.0.1 conda: None pytest: None IPython: 7.2.0 sphinx: None INSTALLED VERSIONS on the broken environment `mwe-slow` ------------------ commit: None python: 3.7.1 | packaged by conda-forge | (default, Nov 13 2018, 18:33:04) [GCC 7.3.0] python-bits: 64 OS: Linux OS-release: 4.19.0-2-amd64 machine: x86_64 processor: byteorder: little LC_ALL: None LANG: en_GB.UTF-8 LOCALE: en_GB.UTF-8 libhdf5: 1.10.4 libnetcdf: 4.6.2 xarray: 0.11.3 pandas: 0.24.1 numpy: 1.15.4 scipy: None netCDF4: 1.4.2 pydap: None h5netcdf: None h5py: None Nio: None zarr: None cftime: 1.0.3.4 PseudonetCDF: None rasterio: None cfgrib: None iris: None bottleneck: None cyordereddict: None dask: None distributed: None matplotlib: None cartopy: None seaborn: None setuptools: 40.7.3 pip: 19.0.1 conda: None pytest: None IPython: 7.2.0 sphinx: None
","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/2747/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed,13221727,issue 538809911,MDU6SXNzdWU1Mzg4MDk5MTE=,3632,applying ufunc over lon and lat ,44284270,closed,0,,,5,2019-12-17T03:25:44Z,2020-08-23T17:37:29Z,2020-08-23T17:37:29Z,NONE,,,,"I am trying to apply ufunc to my code to speed it up. Currently I am looping over lon and lat and it takes ages. After long reads I came up with this though am still not sure where I a going wrong. Suggestions and Ideas would be very helpful to get me going. thanks ``` python def get_grps(s, thresh=-1, Nmin=3): """""" Nmin : int > 0 Min number of consecutive values below threshold. """""" s = pd.Series(s) m = np.logical_and.reduce([s.shift(-i).le(thresh) for i in range(Nmin)]) if Nmin > 1: m = pd.Series(m, index=s.index).replace({False: np.NaN}).ffill(limit=Nmin-1).fillna(False) else: m = pd.Series(m, index=s.index) # Form consecutive groups gps = m.ne(m.shift(1)).cumsum().where(m) # Return None if no groups, else the aggregations if gps.isnull().all(): return [0] else: d = s.groupby(gps).agg([list, sum, 'size']).reset_index(drop=True) data = np.sum((d['size'])) return data ``` obj is 3d data of shape (324x180x360) (time,lat,lon) ``` python def consec_events(obj): time_nums = xr.DataArray(obj['time'].values.astype(np.float), dims='time', coords={'time': obj['time']}, name='time_nums') trend = xr.apply_ufunc(get_grps,time_nums, obj, vectorize=True, input_core_dims=[['time'], ['time']], output_core_dims=[[]], output_dtypes=[np.float], dask='parallelized',output_sizes={'ntime':time_nums.shape[0]}) return trend results = consec_events(spi) ``` ### ERROR IS HERE ``` ... File ""/Users/mada0007/anaconda3/envs/RESEARCH_QUESTION_TWO/lib/python3.7/site-packages/numpy/lib/function_base.py"", line 2157, in _vectorize_call res = self._vectorize_call_with_signature(func, args) File ""/Users/mada0007/anaconda3/envs/RESEARCH_QUESTION_TWO/lib/python3.7/site-packages/numpy/lib/function_base.py"", line 2221, in _vectorize_call_with_signature output[index] = result ValueError: setting an array element with a sequence. ``` I am relatively new to apply this and I have been reading quite a lot. I would appreciatiate if I can be corrected on how to properly apply this over time dimension of my data and get resulting 2d array for each lon and lat? ","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/3632/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed,13221727,issue