html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue https://github.com/pydata/xarray/issues/1938#issuecomment-510953379,https://api.github.com/repos/pydata/xarray/issues/1938,510953379,MDEyOklzc3VlQ29tbWVudDUxMDk1MzM3OQ==,1217238,2019-07-12T16:40:53Z,2019-07-12T16:40:53Z,MEMBER,"We're at the point where this could be hacked together pretty quickly: 1. We need to remove the explicit casting to NumPy arrays (ala https://github.com/pydata/xarray/pull/2956). Checking for an `__array_function__` attribute is probably a good heuristic for duck arrays (it's what dask is using). 2. Internally, we need to use NumPy functions directly (if `__array_function__` is enabled) instead of our current Dask/NumPy versions. Fortunately, pretty much all this logic lives in one place, in `xarray.core.duck_array_ops`. 3. We'll need to think a little bit about indexing in particular. Right now we have special indexing wrappers for NumPy arrays and Dask arrays; we would need to decide how to handle arbitrary array objects (probably by indexing them like NumPy arrays?). Basic indexing should work either way, but indexing with arrays can be a little tricky since few duck-array types support NumPy's full semantics (which are pretty complex).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-510948162,https://api.github.com/repos/pydata/xarray/issues/1938,510948162,MDEyOklzc3VlQ29tbWVudDUxMDk0ODE2Mg==,2190658,2019-07-12T16:23:41Z,2019-07-12T16:23:41Z,MEMBER,@rabernat I can attend remotely.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-510947988,https://api.github.com/repos/pydata/xarray/issues/1938,510947988,MDEyOklzc3VlQ29tbWVudDUxMDk0Nzk4OA==,306380,2019-07-12T16:23:08Z,2019-07-12T16:23:08Z,MEMBER,"@jacobtomlinson got things sorta-working with NEP-18 and CuPy in an afternoon in Iris (with a strong emphasis on ""kinda""). On the CuPy side you're fine. If you're on NumPy 1.16 you'll need to enable the `__array_function__` interface with the following environment variable: export NUMPY_EXPERIMENTAL_ARRAY_FUNCTION=1 If you're using Numpy 1.17 then this is on by default. I think that most of the work here is on the Xarray side. We'll need to remove things like explicit type checks.","{""total_count"": 2, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 2, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-510947769,https://api.github.com/repos/pydata/xarray/issues/1938,510947769,MDEyOklzc3VlQ29tbWVudDUxMDk0Nzc2OQ==,1197350,2019-07-12T16:22:31Z,2019-07-12T16:22:31Z,MEMBER,@hameerabbasi - are you at SciPy by any chance?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-510944897,https://api.github.com/repos/pydata/xarray/issues/1938,510944897,MDEyOklzc3VlQ29tbWVudDUxMDk0NDg5Nw==,2190658,2019-07-12T16:13:09Z,2019-07-12T16:13:09Z,MEMBER,`uarray`/`unumpy` is shaping up nicely. 😄 ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-510943324,https://api.github.com/repos/pydata/xarray/issues/1938,510943324,MDEyOklzc3VlQ29tbWVudDUxMDk0MzMyNA==,1197350,2019-07-12T16:08:10Z,2019-07-12T16:08:10Z,MEMBER,"I am sitting in the SciPy talk about CuPy. Would be great if someone could give us an update on how this issue stands before tomorrow's xarray sprint. Someone my want to try plugging CuPy arrays into xarray. But this issue doesn't really resolve the best way to do that. As far as I can tell @hameerabbasi's ""arrayish"" project was deprecated in favor of uarray / unumpy. What is the best path forward as of today, July 12, 2019?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-383112044,https://api.github.com/repos/pydata/xarray/issues/1938,383112044,MDEyOklzc3VlQ29tbWVudDM4MzExMjA0NA==,2190658,2018-04-20T14:22:03Z,2018-04-20T14:22:03Z,MEMBER,"Let's move this discussion over to hameerabbasi/arrayish#1. But, in summary, I got the impression that the community in general is unhappy with the name ""duck arrays"".","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-383109977,https://api.github.com/repos/pydata/xarray/issues/1938,383109977,MDEyOklzc3VlQ29tbWVudDM4MzEwOTk3Nw==,306380,2018-04-20T14:15:38Z,2018-04-20T14:15:38Z,MEMBER,"Thanks for taking the initiative here @hameerabbasi ! It's good to see something up already. Here is a link to the discussion that I think @hameerabbasi is referring to: http://numpy-discussion.10968.n7.nabble.com/new-NEP-np-AbstractArray-and-np-asabstractarray-tt45282.html#none I haven't read through that entirely yet, was arrayish decided on by the community or was the term still up for discussion?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-383105722,https://api.github.com/repos/pydata/xarray/issues/1938,383105722,MDEyOklzc3VlQ29tbWVudDM4MzEwNTcyMg==,2190658,2018-04-20T14:01:55Z,2018-04-20T14:01:55Z,MEMBER,"I've written it up and already released version 0.0.1 on PyPI, except `concatenate` and `stack` (which need `TypedSequence`). I can still change the name, but I'd rather not. Also, `import duckarray as da` conflicts with `import dask.array as da`.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-383104966,https://api.github.com/repos/pydata/xarray/issues/1938,383104966,MDEyOklzc3VlQ29tbWVudDM4MzEwNDk2Ng==,306380,2018-04-20T13:59:23Z,2018-04-20T13:59:23Z,MEMBER,"Happy with arrayish too On Fri, Apr 20, 2018 at 9:59 AM, Matthew Rocklin wrote: > What name should we go with? I have a slight preference for duckarray > over arrayish but happy with whatever the group decides. > > On Fri, Apr 20, 2018 at 1:51 AM, Hameer Abbasi > wrote: > >> I've created one, as per your e-mail: https://github.com/hameerabbas >> i/arrayish >> >> — >> You are receiving this because you were mentioned. >> Reply to this email directly, view it on GitHub >> , >> or mute the thread >> >> . >> > > ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-383104907,https://api.github.com/repos/pydata/xarray/issues/1938,383104907,MDEyOklzc3VlQ29tbWVudDM4MzEwNDkwNw==,306380,2018-04-20T13:59:09Z,2018-04-20T13:59:09Z,MEMBER,"What name should we go with? I have a slight preference for duckarray over arrayish but happy with whatever the group decides. On Fri, Apr 20, 2018 at 1:51 AM, Hameer Abbasi wrote: > I've created one, as per your e-mail: https://github.com/ > hameerabbasi/arrayish > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > , or mute > the thread > > . > ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-382985783,https://api.github.com/repos/pydata/xarray/issues/1938,382985783,MDEyOklzc3VlQ29tbWVudDM4Mjk4NTc4Mw==,2190658,2018-04-20T05:51:02Z,2018-04-20T06:02:30Z,MEMBER,"I've created one, as per your e-mail: https://github.com/hameerabbasi/arrayish The name is inspired from a recent discussion about this on the Numpy mailing list.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-382918970,https://api.github.com/repos/pydata/xarray/issues/1938,382918970,MDEyOklzc3VlQ29tbWVudDM4MjkxODk3MA==,1217238,2018-04-20T00:04:43Z,2018-04-20T01:43:28Z,MEMBER,"I like `duckarray` a little better without the underscore. Should we go ahead and start `pydata/duckarray`? Or is it better to incubate in somebody's personal repo?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-382901777,https://api.github.com/repos/pydata/xarray/issues/1938,382901777,MDEyOklzc3VlQ29tbWVudDM4MjkwMTc3Nw==,306380,2018-04-19T22:36:48Z,2018-04-19T22:36:48Z,MEMBER,"Doing this externally sounds sensible to me. Thoughts on a good name? duck_array seems to be free on PyPI On Thu, Apr 19, 2018 at 4:23 PM, Stephan Hoyer wrote: > This library would have hard dependencies only on numpy and > multipledispatch, and would expose a multipledispatch namespace so > extending it doesn't have to happen in the library itself. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > , or mute > the thread > > . > ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-382868997,https://api.github.com/repos/pydata/xarray/issues/1938,382868997,MDEyOklzc3VlQ29tbWVudDM4Mjg2ODk5Nw==,1217238,2018-04-19T20:23:39Z,2018-04-19T20:23:39Z,MEMBER,"This library would have hard dependencies only on numpy and multipledispatch, and would expose a multipledispatch namespace so extending it doesn't have to happen in the library itself.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-382867200,https://api.github.com/repos/pydata/xarray/issues/1938,382867200,MDEyOklzc3VlQ29tbWVudDM4Mjg2NzIwMA==,1217238,2018-04-19T20:17:19Z,2018-04-19T20:17:19Z,MEMBER,"By ""muktipledy"" I mean ""multipledispatch""(on my phone)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-382867083,https://api.github.com/repos/pydata/xarray/issues/1938,382867083,MDEyOklzc3VlQ29tbWVudDM4Mjg2NzA4Mw==,1217238,2018-04-19T20:16:49Z,2018-04-19T20:16:49Z,MEMBER,"Basically, the library would define functions like `concatenate` (everything in the linked sparse issue) using muktipledy with implementations for numpy, dask, sparse, etc.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-382862822,https://api.github.com/repos/pydata/xarray/issues/1938,382862822,MDEyOklzc3VlQ29tbWVudDM4Mjg2MjgyMg==,2190658,2018-04-19T20:01:41Z,2018-04-19T20:01:41Z,MEMBER,"By minimal library, I'm assuming you mean something of the sort discussed about abstract arrays? What functionality would such a library have?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-382859987,https://api.github.com/repos/pydata/xarray/issues/1938,382859987,MDEyOklzc3VlQ29tbWVudDM4Mjg1OTk4Nw==,1217238,2018-04-19T19:51:56Z,2018-04-19T19:51:56Z,MEMBER,"I'm thinking it could make sense to build this minimal library for ""duck typed arrays"" with multipledispatch outside of xarray. That would make it easier for library builders to use and extend it. Anyone interested in getting started o nthat?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-382709490,https://api.github.com/repos/pydata/xarray/issues/1938,382709490,MDEyOklzc3VlQ29tbWVudDM4MjcwOTQ5MA==,306380,2018-04-19T12:05:22Z,2018-04-19T12:05:22Z,MEMBER,In https://github.com/pydata/sparse/issues/1#issuecomment-370248174 @shoyer mentions that some work could likely progress in XArray before deciding on the VarArgs in multipledispatch. If XArray maintainers have time it might be valuable to lay out how that would look so that other devs can try it out.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-368605364,https://api.github.com/repos/pydata/xarray/issues/1938,368605364,MDEyOklzc3VlQ29tbWVudDM2ODYwNTM2NA==,1217238,2018-02-26T18:45:13Z,2018-02-26T18:45:13Z,MEMBER,See https://github.com/mrocklin/multipledispatch/issues/72,"{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 1, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-368602406,https://api.github.com/repos/pydata/xarray/issues/1938,368602406,MDEyOklzc3VlQ29tbWVudDM2ODYwMjQwNg==,2190658,2018-02-26T18:35:21Z,2018-02-26T18:35:21Z,MEMBER,Maybe submit a PR? We could all use this. Does it support variable-length arguments?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-368598394,https://api.github.com/repos/pydata/xarray/issues/1938,368598394,MDEyOklzc3VlQ29tbWVudDM2ODU5ODM5NA==,1217238,2018-02-26T18:22:33Z,2018-02-26T18:22:33Z,MEMBER,"I made a tweaked version of dispatching to list subtypes, which probably suitable for use in xarray: https://drive.google.com/file/d/18zdyUpWLNFzFaz08GUOC5vs1GxE_jHg-/view?usp=sharing Example behavior: ```python @dispatch(List[int]) def f(args): print('integers:', args) @dispatch(List[str]) def f(args): print('strings:', args) @dispatch(List[str, int]) def f(args): print('mixed str-int:', args) f([1, 2]) # integers: [1, 2] f([1, 2, 'foo']) # mixed str-int: [1, 2, 'foo'] f(['foo', 'bar']) # strings: ['foo', 'bar'] f([[1, 2]]) # NotImplementedError: Could not find signature for f: ``` Differences from @llllllllll's `VarArgs`: - I don't actually subclass from `tuple`/`list`. You can't use the `List` constructor directly or do `issubclass` with list objects (this matches `typing.List`) - I added sugar so that you don't need to write the dispatch function for `list`, and implementations actually receive native Python list objects as arguments, not `VarArgs` instances. - Type caching is done based on the *set* of element types, not the sequence of element types. I think this is more performant/correct. It would be straightforward to adapt this to use `typing.List`, but since we'll want to define our own `dispatch` functions anyways for our own xarray-specific multipledispatch namespace, I'm just as happy to use an internal `xarray.dispatching.List` type.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-368399227,https://api.github.com/repos/pydata/xarray/issues/1938,368399227,MDEyOklzc3VlQ29tbWVudDM2ODM5OTIyNw==,2190658,2018-02-26T06:02:22Z,2018-02-26T06:02:22Z,MEMBER,"> Which really is totally fine -- this is all a stop gap measure until NumPy itself supports this sort of duck typing. You're assuming here most users of XArray would be using a recent version of Numpy... Which is a totally fine assumption IMO. We make the same one for sparse. However, consider that some people may be using something like conda, which (because of complex dependencies and all) may end up delaying updates (both for Numpy and XArray). I guess however; if people really wanted the updates they could use pip. > so I'm not sure it's worth enshrining in multipledispatch either I would say a little clean-up with some extra decorators for exactly this purpose may be in order, that way, individual wrapping functions aren't needed.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-368281147,https://api.github.com/repos/pydata/xarray/issues/1938,368281147,MDEyOklzc3VlQ29tbWVudDM2ODI4MTE0Nw==,1217238,2018-02-25T03:56:38Z,2018-02-25T03:56:38Z,MEMBER,"Indeed, typing support for multipledispatch looks it's a ways off. To be honest, the VarArgs solution looks a little ugly to me, so I'm not sure it's with enshrining in multipledispatch either. I guess that leaves putting our own ad-hoc solution on top of multipledispatch in xarray for now. Which really is totally fine -- this is all a stop gap measure until NumPy itself supports this sort of duck typing. On Sat, Feb 24, 2018 at 7:46 PM Joe Jevnik wrote: > Given the issues raised on that PR as well as the profiling results shown > here > > I think that PR will need some serious work before it could be merged. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > , or mute > the thread > > . > ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-368280791,https://api.github.com/repos/pydata/xarray/issues/1938,368280791,MDEyOklzc3VlQ29tbWVudDM2ODI4MDc5MQ==,3064397,2018-02-25T03:47:41Z,2018-02-25T03:47:41Z,NONE,@hameerabbasi This really doesn't work with `*args` due to how multiple dispatch itself works. What we have done in blaze is make top-level functions that accept *args which directly call dispatched functions passing the tuple.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-368280749,https://api.github.com/repos/pydata/xarray/issues/1938,368280749,MDEyOklzc3VlQ29tbWVudDM2ODI4MDc0OQ==,3064397,2018-02-25T03:46:19Z,2018-02-25T03:46:19Z,NONE,Given the issues raised on that PR as well as the profiling results shown [here](https://github.com/mrocklin/multipledispatch/pull/66#issuecomment-362698049) I think that PR will need some serious work before it could be merged.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-368279019,https://api.github.com/repos/pydata/xarray/issues/1938,368279019,MDEyOklzc3VlQ29tbWVudDM2ODI3OTAxOQ==,1217238,2018-02-25T03:02:59Z,2018-02-25T03:02:59Z,MEMBER,"I spent some time thinking about this today. The cleanest answer is probably support for standard typing annotations in multipledispatch, at least for `List`. This is already being pursued for multipledispatch in https://github.com/mrocklin/multipledispatch/pull/69.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-368269360,https://api.github.com/repos/pydata/xarray/issues/1938,368269360,MDEyOklzc3VlQ29tbWVudDM2ODI2OTM2MA==,2190658,2018-02-24T23:41:44Z,2018-02-24T23:43:49Z,MEMBER,"Something like `@starargswrapper` that would just cast to list, and call the VarArgs version. Actually it'd be nice to have something like `@dispatch(int, str, StarArgs[int])`. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-368269205,https://api.github.com/repos/pydata/xarray/issues/1938,368269205,MDEyOklzc3VlQ29tbWVudDM2ODI2OTIwNQ==,2190658,2018-02-24T23:38:33Z,2018-02-24T23:38:33Z,MEMBER,@llllllllll How hard would it be to make this work for star-args? I realize you could just add an extra wrapper but it'd be nice if you didn't have to. ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-368268549,https://api.github.com/repos/pydata/xarray/issues/1938,368268549,MDEyOklzc3VlQ29tbWVudDM2ODI2ODU0OQ==,1217238,2018-02-24T23:25:49Z,2018-02-24T23:25:49Z,MEMBER,"> Is there a way to handle kwargs (not with types, but ignoring them)? Yes, `muiltipledispatch` already ignores all keyword arguments for purposes of dispatching.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-368268456,https://api.github.com/repos/pydata/xarray/issues/1938,368268456,MDEyOklzc3VlQ29tbWVudDM2ODI2ODQ1Ng==,2190658,2018-02-24T23:24:15Z,2018-02-24T23:24:15Z,MEMBER,"Is there a way to handle kwargs (not with types, but ignoring them)? ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-368268266,https://api.github.com/repos/pydata/xarray/issues/1938,368268266,MDEyOklzc3VlQ29tbWVudDM2ODI2ODI2Ng==,2190658,2018-02-24T23:21:01Z,2018-02-24T23:21:01Z,MEMBER,"This might even help us out in Sparse for dispatch with `scipy.sparse.spmatrix`, `numpy.ndarray`, etc. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-368267730,https://api.github.com/repos/pydata/xarray/issues/1938,368267730,MDEyOklzc3VlQ29tbWVudDM2ODI2NzczMA==,306380,2018-02-24T23:11:28Z,2018-02-24T23:11:28Z,MEMBER,"cc @jcrist , who has historically been interested in how we solve this problem within dask.array","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-368207468,https://api.github.com/repos/pydata/xarray/issues/1938,368207468,MDEyOklzc3VlQ29tbWVudDM2ODIwNzQ2OA==,2190658,2018-02-24T07:24:02Z,2018-02-24T07:24:02Z,MEMBER,"Another benefit to this would be that if XArray didn't want to support a particular library in its own code, the library itself could add the hooks.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-368190478,https://api.github.com/repos/pydata/xarray/issues/1938,368190478,MDEyOklzc3VlQ29tbWVudDM2ODE5MDQ3OA==,1217238,2018-02-24T02:25:25Z,2018-02-24T02:25:25Z,MEMBER,"@mrocklin this is roughy what we would want in multipledispatch: https://github.com/blaze/blaze/blob/master/blaze/compute/varargs.py#L20-L90 This involves metaclasses, which frankly do blow my mind a little bit. Probably the magic could be tuned down a little bit, but metaclasses *are* necessary at least for implementing `__getitem__` syntax to create classes (and provide a few other niceties here like custom reprs and subclass checks).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-368159542,https://api.github.com/repos/pydata/xarray/issues/1938,368159542,MDEyOklzc3VlQ29tbWVudDM2ODE1OTU0Mg==,306380,2018-02-23T22:41:54Z,2018-02-23T22:41:54Z,MEMBER,"I would want to see how magical it was. @llllllllll 's calibration of ""mild metaprogramming"" may differ slightly from my own :) Eventually if multipledispatch becomes a dependency of xarray then we should consider changing the decision-making process away from being just me though. Relatedly, SymPy also just adopted it (by vendoring) as a dependency.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-368111050,https://api.github.com/repos/pydata/xarray/issues/1938,368111050,MDEyOklzc3VlQ29tbWVudDM2ODExMTA1MA==,3064397,2018-02-23T19:16:37Z,2018-02-23T19:16:37Z,NONE,"I wouldn't mind submitting this upstream, but I will defer to @mrocklin.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-368110090,https://api.github.com/repos/pydata/xarray/issues/1938,368110090,MDEyOklzc3VlQ29tbWVudDM2ODExMDA5MA==,1217238,2018-02-23T19:13:14Z,2018-02-23T19:13:14Z,MEMBER,"> How about something like checking inside a list if something is top priority, then call a, if second priority, call b, etc. Usually, this is not a good idea. The problem is that it's impossible to know a global priority order across unrelated packages. It's usually better to declare valid type matches explicitly. NumPy tried this with `__array_priority__`, but in practice these priority numbers are basically meaningless for all comparisons other than comparisons to the priority of NumPy arrays.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-368108543,https://api.github.com/repos/pydata/xarray/issues/1938,368108543,MDEyOklzc3VlQ29tbWVudDM2ODEwODU0Mw==,1217238,2018-02-23T19:07:46Z,2018-02-23T19:07:46Z,MEMBER,"As for my last concern, ""Dispatch for the first argument(s) only"" it looks like the simple answer is that multipledispatch already only dispatches based on positional arguments. So as long as we're strict about using keyword arguments for extra parameters like `axis` (which is good style anyways), we only need a single overload per array type for single dispatch functions like `sum()`. It looks like this resolves almost all of my concerns about using multiple dispatch. One thing that would be nice is it `VarArgs` is actually distributed as part of multipledispatch rather than needing to be copied separately into xarray. That would make it easier for third parties to extend our operations, by simply importing `VarArgs` from multipledispatch rather than importing it from somewhere internal in xarray.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-368107036,https://api.github.com/repos/pydata/xarray/issues/1938,368107036,MDEyOklzc3VlQ29tbWVudDM2ODEwNzAzNg==,1217238,2018-02-23T19:02:34Z,2018-02-23T19:02:34Z,MEMBER,"Yes, I just tested out the wrapping dispatch. It works and is quite clean.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-368106885,https://api.github.com/repos/pydata/xarray/issues/1938,368106885,MDEyOklzc3VlQ29tbWVudDM2ODEwNjg4NQ==,2190658,2018-02-23T19:02:02Z,2018-02-23T19:02:02Z,MEMBER,"How about something like checking inside a list if something is top priority, then call `a`, if second priority, call `b`, etc.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-368106529,https://api.github.com/repos/pydata/xarray/issues/1938,368106529,MDEyOklzc3VlQ29tbWVudDM2ODEwNjUyOQ==,3064397,2018-02-23T19:00:39Z,2018-02-23T19:00:39Z,NONE,"The wrapping dispatch would just look like: ```python @dispatch(list) def f(args): return f(VarArgs(args)) ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-368105739,https://api.github.com/repos/pydata/xarray/issues/1938,368105739,MDEyOklzc3VlQ29tbWVudDM2ODEwNTczOQ==,3064397,2018-02-23T18:57:59Z,2018-02-23T18:58:47Z,NONE,"We could make a particular list an _instance_ of a particular `TypedVarArgs`; however, multiple dispatch uses the `type()` of arguments as well as `issubclass` to do dispatching. Multiple dispatch depends on being able to partially order types to make dispatching more efficient. The constructor of `VarArgs` scans for the types of the elements and constructs an instance of a new (but memoized) subclass of `VarArgs` which encodes the element types so that `issubclass` works as expected. The problem is that `type([1.0, 'foo'])` returns just `list` which erases all information about the elements.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-368103344,https://api.github.com/repos/pydata/xarray/issues/1938,368103344,MDEyOklzc3VlQ29tbWVudDM2ODEwMzM0NA==,2190658,2018-02-23T18:49:41Z,2018-02-23T18:49:41Z,MEMBER,"Can't some wild metaprogramming make it so that `[1.0, 'foo']` itself is an instance of `VarArgs[float, str]` (or be converted?)","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-368100305,https://api.github.com/repos/pydata/xarray/issues/1938,368100305,MDEyOklzc3VlQ29tbWVudDM2ODEwMDMwNQ==,3064397,2018-02-23T18:39:50Z,2018-02-23T18:40:46Z,NONE,"`VarArgs` itself is actually a type, so you need to create instances which wrap the list argument, for example: ```python In [1]: from blaze.compute.varargs import VarArgs In [2]: from multipledispatch import dispatch In [3]: @dispatch(VarArgs[float]) ...: def f(args): ...: print('floats') ...: In [4]: @dispatch(VarArgs[str]) ...: def f(args): ...: print('strings') ...: In [5]: @dispatch(VarArgs[str, float]) ...: def f(args): ...: print('mixed') ...: In [6]: f(VarArgs(['foo'])) strings In [7]: f(VarArgs([1.0])) floats In [8]: f(VarArgs([1.0, 'foo'])) mixed In [9]: VarArgs([1.0, 'foo']) Out[9]: VarArgs[float, str]([1.0, 'foo']) ``` You could hide this behind a top-level function that wraps the input for the user, or register a dispatch for list which boxes and recurses into itself.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-368097912,https://api.github.com/repos/pydata/xarray/issues/1938,368097912,MDEyOklzc3VlQ29tbWVudDM2ODA5NzkxMg==,1217238,2018-02-23T18:32:04Z,2018-02-23T18:32:04Z,MEMBER,"@llllllllll very cool! Is there a special trick I need to use this? I tried: ```python # first: pip install https://github.com/blaze/blaze/archive/master.tar.gz import blaze.compute from blaze.compute.varargs import VarArgs from multipledispatch import dispatch @dispatch(VarArgs[float]) def f(args): print('floats') @dispatch(VarArgs[str]) def f(args): print('strings') @dispatch(VarArgs[str, float]) def f(args): print('mixed') ``` This gives me an error when I try to use it: ```python >>> f(['foo']) --------------------------------------------------------------------------- KeyError Traceback (most recent call last) /usr/local/lib/python3.6/dist-packages/multipledispatch/dispatcher.py in __call__(self, *args, **kwargs) 154 try: --> 155 func = self._cache[types] 156 except KeyError: KeyError: (,) During handling of the above exception, another exception occurred: NotImplementedError Traceback (most recent call last) in () ----> 1 f(['foo']) /usr/local/lib/python3.6/dist-packages/multipledispatch/dispatcher.py in __call__(self, *args, **kwargs) 159 raise NotImplementedError( 160 'Could not find signature for %s: <%s>' % --> 161 (self.name, str_signature(types))) 162 self._cache[types] = func 163 try: NotImplementedError: Could not find signature for f: ```","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-368084600,https://api.github.com/repos/pydata/xarray/issues/1938,368084600,MDEyOklzc3VlQ29tbWVudDM2ODA4NDYwMA==,1217238,2018-02-23T17:44:27Z,2018-02-23T18:17:28Z,MEMBER,"Dispatch for stack/concatenate is definitely on the radar for NumPy development, but I don't know when it's actually going to happen. The likely interface is something like `__array_ufunc__`: a special method like `__array_concatenate__` is called on each element in the list, until one does not return NotImplemented. This is a different style of overloads than multipledispatch, one that is slightly simpler to implement but possibly slower and with fewer guarantees of correctness. We only need this for a couple of operations, so in any case we can probably implement our own ad-hoc dispatch system for `np.stack` and `np.concatenate`, either along the of multipledispatch or NumPy/`__array_ufunc__`. On further contemplation, overloading based on union types with a system like multipledispatch does seem tricky. It's not clear to me that there's even a well defined type for inputs to concatenate that should be dispatched to dask vs. numpy, for example. We want to let that dask handle any cases where at least one input is a dask array, but a type like `List[Union[np.ndarray, da.Array]]` actually matches a list of all numpy arrays, too -- unless we require an exact match for the type.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-368091406,https://api.github.com/repos/pydata/xarray/issues/1938,368091406,MDEyOklzc3VlQ29tbWVudDM2ODA5MTQwNg==,3064397,2018-02-23T18:08:30Z,2018-02-23T18:08:30Z,NONE,"In blaze we have variadic sequences for multiple dispatch, and the `List[Union]` case is something we have run into. We have a type called `VarArgs` which takes a variadic sequence of type-arguments and represents a sequence of a unions over the arguments, for example: `VarArgs[pd.Series, pd.DataFrame]` is a sequence of unknown length which is known to hold either series or dataframes. With some mild metaprogramming we made it so that `VarArs[pd.Series]` is a subclass of `VarArgs[pd.Series, pd.DataFrame]`, or in general, more specific sequences are subclasses of more general sequences. This means that you can solve the ambiguity by registering a dispatch for `VarArgs[np.ndarray]` _and_ `VarArgs[np.ndarray, da.Array]` and you know that the second function can only be called if the sequence holds at least one dask array. Here is an example of what that looks like for `merge`, which is concat(axis=1): https://github.com/blaze/blaze/blob/master/blaze/compute/pandas.py#L691 This is the definition of `VarArgs`: https://github.com/blaze/blaze/blob/master/blaze/compute/varargs.py","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-368068500,https://api.github.com/repos/pydata/xarray/issues/1938,368068500,MDEyOklzc3VlQ29tbWVudDM2ODA2ODUwMA==,306380,2018-02-23T16:54:37Z,2018-02-23T16:54:37Z,MEMBER,"Import times on multipledispatch have improved thanks to work by @llllllllll . They could probably be further improved if people wanted to invest modest intellectual effort here. Costs scale with the number of type signatures on each operation. In blaze this was very high, well into the hundreds, in our case it would be, I think, more modest around 2-10. (also, historical note, multipledispatch predates my involvement in Blaze). When possible it would be useful to upstream these concerns to NumPy, even if we have to move faster than NumPy is able to support.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-368066239,https://api.github.com/repos/pydata/xarray/issues/1938,368066239,MDEyOklzc3VlQ29tbWVudDM2ODA2NjIzOQ==,1217238,2018-02-23T16:47:53Z,2018-02-23T16:47:53Z,MEMBER,"> Do we need to be capable of supporting other objects for future extension? If so, we may need to start from (heavy) refactoring. For two array backends, it didn't make sense to write an abstraction layer for this, in part because it wasn't clear what we needed. But for three examples, it probably does -- that's the point where shared use cases become clear. Undoubtedly, there will be other cases in the future where users will want to extend xarray to handle new array types (arrays with units come to mind). For implementing these overloads/functions, there are various possible solutions. Our current ad-hoc system is similar to what @hameerabbasi suggests -- we check the type of the first argument and use that to dispatch to an appropriate function. This has the advantage of being easy to implement for a known set of types, but a single dispatch order is not very extensible -- it's impossible to anticipate every third-party class. Recently, NumPy has moved away from this (e.g., with `__array_ufunc__`). One appealing option is to make use of @mrocklin's [multipledispatch](https://github.com/mrocklin/multipledispatch) library, which was originally developed for Blaze and is still in active use. Possible concerns: 1. **Performance**. Import times need to be fast, and I know this is something that `multipledispatch` can sometimes struggle with. My *guess* is that this wouldn't be a problem for us, since we can rely on other dispatch mechanisms most operations (including `__array_ufunc__` and Python's builtin arithmetic overrides). 2. **Dispatch for `stack`/`concatenate`**: How do we handle dispatching for functions that take a list of arrays? e.g., if a list of arrays has contains any dask arrays, we need to use dask. Ideally, we would resolve the type of an object like `[np.array(...), np.array(...), ..., da.Array(...)]` to a mixed type like `List[Union[np.ndarray, da.Array]]`, for which an override could be implemented. 3. **Dispatch for the first argument(s) only**: This is a minor point, but some functions don't need to be dispatched on all of their arguments, e.g., `sum()` only really needs to dispatch on the array types but can pass other arguments like `axis` directly on. I suppose could simply annotate extra position arguments with `object`, but this will get annoying for multiple optional arguments which would all need separate implementations (if I understand multipledispatch correctly).","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-368016521,https://api.github.com/repos/pydata/xarray/issues/1938,368016521,MDEyOklzc3VlQ29tbWVudDM2ODAxNjUyMQ==,2190658,2018-02-23T14:00:50Z,2018-02-23T14:04:18Z,MEMBER,"Then I would suggest something like the following for hooks (omitting imports): ```python # Registered in order of priority xarray.interfaces.register('DaskArray', lambda ar: isinstance(ar, da.array)) xarray.hooks.register('nansum', 'DaskArray', da.nansum) xarray.interfaces.register('SparseArray', lambda ar: isinstance(ar, sparse.SparseArray)) xarray.hooks.register('nansum', 'SparseArray', sparse.nansum) ``` And then, in code, call the appropriate `nansum` instead of `np.nansum`: ```python nansum = xarray.hooks.get(arr, 'nansum') ``` If you need help, I'd be willing to give it. :-) But I'm not a user of XArray, so I don't really understand the use-cases or codebase.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148 https://github.com/pydata/xarray/issues/1938#issuecomment-368004970,https://api.github.com/repos/pydata/xarray/issues/1938,368004970,MDEyOklzc3VlQ29tbWVudDM2ODAwNDk3MA==,6815844,2018-02-23T13:10:30Z,2018-02-23T13:10:30Z,MEMBER,"Thanks for leading the development of sparse. I'm looking forward to see it in xarray:) Currently, our logic to support `dask.array` and `numpy.ndarray` is hard-coded everywhere. For example, we have many computation paths for `nansum`, `dask.Array`, `np.ndarray` with `bottleneck`, bare `np.ndarray` and we use our in-house implementation for object-type arrays. The easiest way to support `sparse` might be to add a specific path for `sparse` by hard-coding again, but it is less flexible. Do we need to be capable of supporting other objects for future extension? If so, we may need to start from (heavy) refactoring. @shoyer, Could you give any suggestion? I am personally interested in helping this, but I may need to decide the direction first.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,299668148