issue_comments

@Duane321 While it would be fantastic to have gpu-enabled auto-diff-able xarrays / DataArrays, an interesting development worth looking into are the named tensor in https://pytorch.org/docs/stable/named_tensor.html. This appears to be an attempt to bridge the gap from the that they are making pytorch tensors increasingly dataarray like. I would not be surprised if within the next few iterations they add indexes to the tensors closing the gap even further.

@andersy005 I'm about to start working actively on cupy support in xarray. Would be great to get some of your input.

Cupy requests that instead of calling __array__ you instead call their .get method for explicit conversion to numpy. So we need to add a little compatibility code for this.

Do you have a sense of the overhead / effort of making jax vs cupy as the gpu backend for xarrays ? One advantage of jax would be built in auto-diff functionality that would enable xarray to be plugged directly into deep learning pipelines. Downside is that it is not as numpy compatible as cupy. How much of a non-starter would this be ?

If you have any pointers on how to go about this - I can give it a try.

This might be a good time to revive this thread and see if there is wider interest (and bandwidth) in having xarray use CuPy (https://cupy.chainer.org/ ) as a backend (along with numpy). It appears to be a plug-and-play replacement for numpy - so it might not have all the issues that were brought up regarding pytorch/jax ?

Any thoughts ? cc @mrocklin

Thanks.

That solved that error but introduced another one.

Specifically - this is my dataframe

and this is the error that I get with sparse=True

My numpy version is definitely about 1.16

I also set this os.environ["NUMPY_EXPERIMENTAL_ARRAY_FUNCTION"]='1' just in case

Furthermore, I don't get this error when I don't set sparse=True ( I just get OOM errors but that's another matter) ...

I cloned the master branch and installed it using 'python setup.py develop'.

When I try to use the sparse data loading functionality as per python oo = xa.Dataset.from_dataframe( my_df, sparse=True ) I get the following error:

```

ModuleNotFoundError Traceback (most recent call last) <ipython-input-9-fce0ca6bc4c2> in <module> ----> 1 oo = xa.Dataset.from_dataframe( poly_df.iloc[:10000], sparse=True )

/mnt/local/xarray/xarray/core/dataset.py in from_dataframe(cls, dataframe, sparse) 4040 4041 if sparse: -> 4042 obj._set_sparse_data_from_dataframe(dataframe, dims, shape) 4043 else: 4044 obj._set_numpy_data_from_dataframe(dataframe, dims, shape)

/mnt/local/xarray/xarray/core/dataset.py in _set_sparse_data_from_dataframe(self, dataframe, dims, shape) 3936 self, dataframe: pd.DataFrame, dims: tuple, shape: Tuple[int, ...] 3937 ) -> None: -> 3938 from sparse import COO 3939 3940 idx = dataframe.index

ModuleNotFoundError: No module named 'sparse'

``` Any suggestions on what I need to do ?

Would it be possible that pd.{Series, DataFrame}.to_xarray() automatically creates a sparse dataarray - or we have a flag in to_xarray which allows controlling for this. I have a very sparse dataframe and everytime I try to convert it to xarray I blow out my memory. Keeping it sparse but logically as a DataArray would be fantastic.

@shoyer Is there documentation for using sparse arrays ? Could you point me to some example code ?

While it is pretty straightforward to implement a lot of standard xarray operations with a pytorch / Jax backend (since they just fallback on native functions) - it will be interesting to think about how to implement rolling operations / expanding / exponential window in a way that is both efficient and maintains differentiability.

Expanding and exponential window operations would be easy to do leveraging RNN semantics - but doing rolling using convolutions is going to be very inefficient.

Do you have any thoughts on this?

I haven't used JAX - but was just browsing through its documentation and it looks super cool. Any ideas on how it compares with Pytorch in terms of:

a) Cxecution speed, esp. on GPU b) Memory management on GPUs. Pytorch has the 'Dataloader/Dataset' paradigm which uses background multithreading to shuttle batches of data back and forth - along with a lot of tips and tricks on efficient memory usage. c) support for deep-learning optimization algorithms ?

That looks correct. Let me try and revert back to you

On Wed, Aug 14, 2019, 16:44 Maximilian Roos notifications@github.com wrote:

How is merge_asof different from using reindex with method='pad'?

Yes this is right! Mea culpa. We can already use the pandas reindexing for the 1D case (which should cover your case @fjanoos https://github.com/fjanoos ?)

@fjanoos https://github.com/fjanoos can you confirm this is what you're looking for?

In [4]: da=xr.DataArray(list('abcdefghil'), dims=['x'],coords=dict(x=range(10)))

In [8]: da.reindex(x=[0,2.5,2.6,2.7,5,6.2], method='nearest') Out[8]:<xarray.DataArray (x: 6)> array(['a', 'd', 'd', 'd', 'f', 'g'], dtype='<U1') Coordinates: * x (x) float64 0.0 2.5 2.6 2.7 5.0 6.2

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/issues/3218?email_source=notifications&email_token=AAHBOLQHOB6IQKA4ZU2EHF3QERVBHA5CNFSM4ILW6VK2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4KBVHA#issuecomment-521411228, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHBOLQM7PRFBOFQMPSNZL3QERVBHANCNFSM4ILW6VKQ .

As of now, a simple workaround would be to do these tasks in pandas and switch back and forth.

A couple of years ago - before pandas had pd.merge_asof - I had implemented a version of this logic in numba when working with numpy arrays. It was blazingly fast - and if there is interest I can try to dig it up ? I would need some help making it work for xarrays and publishing it into the master branch.

On Wed, Aug 14, 2019, 14:12 Maximilian Roos notifications@github.com wrote:

I think this would be good. It would need to be implemented outside of python (cython / numba / etc) given the performance requirements. I'm not sure whether we could borrow the pandas functionality and apply it to multi-dimensional arrays.

Assuming we'd need to write our own, xarray doesn't have any cython dependencies, so I think it would be best in a separate and optional package. These could go in numbagg. It's non-trivial work, so someone would have to have a strong need for it.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/issues/3218?email_source=notifications&email_token=AAHBOLSMYMJ5GK3LJLPURBTQERDKFA5CNFSM4ILW6VK2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4JURTY#issuecomment-521357519, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHBOLVZ6VEZTAZV4MFOMBLQERDKFANCNFSM4ILW6VKQ .

Wondering what the status on this is ? Is there a branch with this functionality implemented - would love to give it a spin !