html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue https://github.com/pydata/xarray/issues/873#issuecomment-248409634,https://api.github.com/repos/pydata/xarray/issues/873,248409634,MDEyOklzc3VlQ29tbWVudDI0ODQwOTYzNA==,1328158,2016-09-20T19:37:07Z,2016-09-20T19:37:07Z,NONE,"Thanks for this clarification, Stephan. Apparently I didn't read the API documentation closely enough, as I was assuming that the function is applied to the underlying ndarray rather than to all data variables of a Dataset object. Now that I've taken the approach you suggested I'm cooking with gas, and it's very encouraging. I really appreciate your help. ​--James ​ On Tue, Sep 20, 2016 at 11:54 AM, Stephan Hoyer notifications@github.com wrote: > GroupBy is working as intended here. ds.groupby('something').apply(func) > calls func on objects of the same type as ds. If you group a Dataset, > each time you apply to a Dataset, too. > > You can certainly still use np.convolve, but you'll need to manually > apply it to numpy arrays extracted from a Dataset and then rebuild another > Dataset or DataArray. > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub > https://github.com/pydata/xarray/issues/873#issuecomment-248345053, or mute > the thread > https://github.com/notifications/unsubscribe-auth/ABREHkSB-DvW5OD3DHuYxYzP7l7yfjL2ks5qsAGcgaJpZM4IwE4g > . ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,158958801 https://github.com/pydata/xarray/issues/873#issuecomment-248216388,https://api.github.com/repos/pydata/xarray/issues/873,248216388,MDEyOklzc3VlQ29tbWVudDI0ODIxNjM4OA==,1328158,2016-09-20T06:42:53Z,2016-09-20T06:42:53Z,NONE,"Thanks, Stephan. My code uses numpy.convolve() in several key places, so if that function is a deal breaker for using xarray then I'll hold off until that's fixed. In the meantime if there's anything else I can do to help you work this out then please let me know. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,158958801 https://github.com/pydata/xarray/issues/873#issuecomment-242535724,https://api.github.com/repos/pydata/xarray/issues/873,242535724,MDEyOklzc3VlQ29tbWVudDI0MjUzNTcyNA==,1328158,2016-08-25T20:48:45Z,2016-08-25T20:48:45Z,NONE,"Thanks, Stephan. In general things appear to be working much more as expected now, probably (hopefully) this is just an edge case/nuance that won't be too difficult for you guys to address. If so and if I don't run across any other issues then my code will be dramatically simplified by leveraging xarray rather than writing code to enable shared memory objects for the multiprocessing side of things (my assumption being that you guys have done a better job of that than I can). A gist with example code and a smallish data file attached to the comment is here: https://gist.github.com/monocongo/e8e883c2355f7a92bb0b9d24db5407a8 Please let me know if I can do anything else to help you help me. Godspeed! --James On Tue, Aug 23, 2016 at 12:42 AM, Stephan Hoyer notifications@github.com wrote: > Could you please share a data file and/or code which I can run to > reproduce each of these issues? > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub > https://github.com/pydata/xarray/issues/873#issuecomment-241625354, or mute > the thread > https://github.com/notifications/unsubscribe-auth/ABREHhoGvwv30D2Qk858lHB-U5oWtRQnks5qinpDgaJpZM4IwE4g > . ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,158958801 https://github.com/pydata/xarray/issues/873#issuecomment-241540585,https://api.github.com/repos/pydata/xarray/issues/873,241540585,MDEyOklzc3VlQ29tbWVudDI0MTU0MDU4NQ==,1328158,2016-08-22T20:32:20Z,2016-08-22T20:32:20Z,NONE,"I get the following error now when I try to run the gist code referenced in the original message above: ``` $ python -u xarray_gist.py /dev/shm/nclimgrid_prcp_reduced.nc nclimgrid_prcp_doubled.nc Traceback (most recent call last): File ""xarray_gist.py"", line 45, in encoding = {variable_name: {'_FillValue': np.nan, 'dtype': 'float32'}}) File ""/home/james.adams/anaconda3/lib/python3.5/site-packages/xarray/core/dataset.py"", line 782, in to_netcdf engine=engine, encoding=encoding) File ""/home/james.adams/anaconda3/lib/python3.5/site-packages/xarray/backends/api.py"", line 354, in to_netcdf dataset.dump_to_store(store, sync=sync, encoding=encoding) File ""/home/james.adams/anaconda3/lib/python3.5/site-packages/xarray/core/dataset.py"", line 728, in dump_to_store store.store(variables, attrs, check_encoding) File ""/home/james.adams/anaconda3/lib/python3.5/site-packages/xarray/backends/common.py"", line 234, in store check_encoding_set) File ""/home/james.adams/anaconda3/lib/python3.5/site-packages/xarray/backends/common.py"", line 209, in store self.set_variables(variables, check_encoding_set) File ""/home/james.adams/anaconda3/lib/python3.5/site-packages/xarray/backends/common.py"", line 219, in set_variables target, source = self.prepare_variable(name, v, check) File ""/home/james.adams/anaconda3/lib/python3.5/site-packages/xarray/backends/netCDF4_.py"", line 266, in prepare_variable raise_on_invalid=check_encoding) File ""/home/james.adams/anaconda3/lib/python3.5/site-packages/xarray/backends/netCDF4_.py"", line 167, in _extract_nc4_encoding ' %r' % (backend, invalid)) ValueError: unexpected encoding parameters for 'netCDF4' backend: ['dtype'] ``` Additionally I see the following errors when I run some other code which uses the same dataset.groupby().apply() technique (the trouble appears to show up within numpy.convolve()): ``` Traceback (most recent call last): File ""C:\home\git\indices\src\main\python\indices\spi_gamma_xarray.py"", line 46, in dataset = dataset.groupby('grid_cells').apply(function_to_be_applied) File ""C:\Anaconda3\lib\site-packages\xarray\core\groupby.py"", line 567, in apply combined = self._concat(applied) File ""C:\Anaconda3\lib\site-packages\xarray\core\groupby.py"", line 572, in _concat applied_example, applied = peek_at(applied) File ""C:\Anaconda3\lib\site-packages\xarray\core\utils.py"", line 90, in peek_at peek = next(gen) File ""C:\Anaconda3\lib\site-packages\xarray\core\groupby.py"", line 566, in applied = (func(ds, **kwargs) for ds in self._iter_grouped()) File ""C:\home\git\indices\src\main\python\indices\spi_gamma_xarray.py"", line 27, in function_to_be_applied valid_max) File ""C:\Anaconda3\lib\site-packages\numpy\core\numeric.py"", line 1005, in convolve return multiarray.correlate(a, v[::-1], mode) TypeError: Cannot cast array data from dtype('float64') to dtype(' dataset = dataset.groupby('grid_cells').apply(function_to_be_applied) File ""C:\Anaconda3\lib\site-packages\xarray\core\groupby.py"", line 567, in apply combined = self._concat(applied) File ""C:\Anaconda3\lib\site-packages\xarray\core\groupby.py"", line 572, in _concat applied_example, applied = peek_at(applied) File ""C:\Anaconda3\lib\site-packages\xarray\core\utils.py"", line 90, in peek_at peek = next(gen) File ""C:\Anaconda3\lib\site-packages\xarray\core\groupby.py"", line 566, in applied = (func(ds, **kwargs) for ds in self._iter_grouped()) File ""C:\home\git\indices\src\main\python\indices\spi_gamma_xarray.py"", line 27, in function_to_be_applied valid_max) File ""C:\Anaconda3\lib\site-packages\numpy\core\numeric.py"", line 1005, in convolve return multiarray.correlate(a, v[::-1], mode) TypeError: Cannot cast array data from dtype('float64') to dtype('