issue_comments

Great, glad that worked!

Thanks for this clarification, Stephan. Apparently I didn't read the API documentation closely enough, as I was assuming that the function is applied to the underlying ndarray rather than to all data variables of a Dataset object. Now that I've taken the approach you suggested I'm cooking with gas, and it's very encouraging. I really appreciate your help.

​--James ​

On Tue, Sep 20, 2016 at 11:54 AM, Stephan Hoyer notifications@github.com wrote:

GroupBy is working as intended here. ds.groupby('something').apply(func) calls func on objects of the same type as ds. If you group a Dataset, each time you apply to a Dataset, too.

You can certainly still use np.convolve, but you'll need to manually apply it to numpy arrays extracted from a Dataset and then rebuild another Dataset or DataArray.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/issues/873#issuecomment-248345053, or mute the thread https://github.com/notifications/unsubscribe-auth/ABREHkSB-DvW5OD3DHuYxYzP7l7yfjL2ks5qsAGcgaJpZM4IwE4g .

GroupBy is working as intended here. ds.groupby('something').apply(func) calls func on objects of the same type as ds. If you group a Dataset, each time you apply to a Dataset, too.

You can certainly still use np.convolve, but you'll need to manually apply it to numpy arrays extracted from a Dataset and then rebuild another Dataset or DataArray.

Thanks, Stephan. My code uses numpy.convolve() in several key places, so if that function is a deal breaker for using xarray then I'll hold off until that's fixed. In the meantime if there's anything else I can do to help you work this out then please let me know.

The first issue is an xarray bug. See #989 for a fix.

The work around is not to provide an encoded dtype if the variable already has the dtype you want.

The second issue is that compute_function is being called on an xarray.Dataset, so np.convolve(data, data * 2) is attempting to convolve two xarray Datasets. NumPy gives a pretty unhelpful error message. (If only we had static typing in Python...)

Thanks, Stephan. In general things appear to be working much more as expected now, probably (hopefully) this is just an edge case/nuance that won't be too difficult for you guys to address. If so and if I don't run across any other issues then my code will be dramatically simplified by leveraging xarray rather than writing code to enable shared memory objects for the multiprocessing side of things (my assumption being that you guys have done a better job of that than I can).

A gist with example code and a smallish data file attached to the comment is here: https://gist.github.com/monocongo/e8e883c2355f7a92bb0b9d24db5407a8

Please let me know if I can do anything else to help you help me. Godspeed!

--James

On Tue, Aug 23, 2016 at 12:42 AM, Stephan Hoyer notifications@github.com wrote:

Could you please share a data file and/or code which I can run to reproduce each of these issues?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/issues/873#issuecomment-241625354, or mute the thread https://github.com/notifications/unsubscribe-auth/ABREHhoGvwv30D2Qk858lHB-U5oWtRQnks5qinpDgaJpZM4IwE4g .

Could you please share a data file and/or code which I can run to reproduce each of these issues?

I get the following error now when I try to run the gist code referenced in the original message above:

``` $ python -u xarray_gist.py /dev/shm/nclimgrid_prcp_reduced.nc nclimgrid_prcp_doubled.nc Traceback (most recent call last): File "xarray_gist.py", line 45, in <module> encoding = {variable_name: {'FillValue': np.nan, 'dtype': 'float32'}}) File "/home/james.adams/anaconda3/lib/python3.5/site-packages/xarray/core/dataset.py", line 782, in to_netcdf engine=engine, encoding=encoding) File "/home/james.adams/anaconda3/lib/python3.5/site-packages/xarray/backends/api.py", line 354, in to_netcdf dataset.dump_to_store(store, sync=sync, encoding=encoding) File "/home/james.adams/anaconda3/lib/python3.5/site-packages/xarray/core/dataset.py", line 728, in dump_to_store store.store(variables, attrs, check_encoding) File "/home/james.adams/anaconda3/lib/python3.5/site-packages/xarray/backends/common.py", line 234, in store check_encoding_set) File "/home/james.adams/anaconda3/lib/python3.5/site-packages/xarray/backends/common.py", line 209, in store self.set_variables(variables, check_encoding_set) File "/home/james.adams/anaconda3/lib/python3.5/site-packages/xarray/backends/common.py", line 219, in set_variables target, source = self.prepare_variable(name, v, check) File "/home/james.adams/anaconda3/lib/python3.5/site-packages/xarray/backends/netCDF4.py", line 266, in prepare_variable raise_on_invalid=check_encoding) File "/home/james.adams/anaconda3/lib/python3.5/site-packages/xarray/backends/netCDF4_.py", line 167, in _extract_nc4_encoding ' %r' % (backend, invalid)) ValueError: unexpected encoding parameters for 'netCDF4' backend: ['dtype']

```

Additionally I see the following errors when I run some other code which uses the same dataset.groupby().apply() technique (the trouble appears to show up within numpy.convolve()):

Traceback (most recent call last): File "C:\home\git\indices\src\main\python\indices\spi_gamma_xarray.py", line 46, in <module> dataset = dataset.groupby('grid_cells').apply(function_to_be_applied) File "C:\Anaconda3\lib\site-packages\xarray\core\groupby.py", line 567, in apply combined = self._concat(applied) File "C:\Anaconda3\lib\site-packages\xarray\core\groupby.py", line 572, in _concat applied_example, applied = peek_at(applied) File "C:\Anaconda3\lib\site-packages\xarray\core\utils.py", line 90, in peek_at peek = next(gen) File "C:\Anaconda3\lib\site-packages\xarray\core\groupby.py", line 566, in <genexpr> applied = (func(ds, **kwargs) for ds in self._iter_grouped()) File "C:\home\git\indices\src\main\python\indices\spi_gamma_xarray.py", line 27, in function_to_be_applied valid_max) File "C:\Anaconda3\lib\site-packages\numpy\core\numeric.py", line 1005, in convolve return multiarray.correlate(a, v[::-1], mode) TypeError: Cannot cast array data from dtype('float64') to dtype('<U32') according to the rule 'safe' Traceback (most recent call last): File "C:\home\git\indices\src\main\python\indices\spi_gamma_xarray.py", line 46, in <module> dataset = dataset.groupby('grid_cells').apply(function_to_be_applied) File "C:\Anaconda3\lib\site-packages\xarray\core\groupby.py", line 567, in apply combined = self._concat(applied) File "C:\Anaconda3\lib\site-packages\xarray\core\groupby.py", line 572, in _concat applied_example, applied = peek_at(applied) File "C:\Anaconda3\lib\site-packages\xarray\core\utils.py", line 90, in peek_at peek = next(gen) File "C:\Anaconda3\lib\site-packages\xarray\core\groupby.py", line 566, in <genexpr> applied = (func(ds, **kwargs) for ds in self._iter_grouped()) File "C:\home\git\indices\src\main\python\indices\spi_gamma_xarray.py", line 27, in function_to_be_applied valid_max) File "C:\Anaconda3\lib\site-packages\numpy\core\numeric.py", line 1005, in convolve return multiarray.correlate(a, v[::-1], mode) TypeError: Cannot cast array data from dtype('float64') to dtype('<U32') according to the rule 'safe'

Please advise if I can provide any further information which might help work this out, or if I have made wrong assumptions as to how this feature should be used. Thanks.

Fixed by #867

Thanks for raising this one on GitHub (after I forgot to respond on the mailing list!). I have a partial fix for this in https://github.com/pydata/xarray/pull/867, but we clearly need some more tests to verify that groupby with a multi-index works properly.