html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue
https://github.com/pydata/xarray/pull/964#issuecomment-277028203,https://api.github.com/repos/pydata/xarray/issues/964,277028203,MDEyOklzc3VlQ29tbWVudDI3NzAyODIwMw==,1217238,2017-02-02T17:41:48Z,2017-02-02T17:41:48Z,MEMBER,#1245 replaces the unintuitive `signature` argument with separate `input_core_dims` and `output_core_dims`.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,170779798
https://github.com/pydata/xarray/pull/964#issuecomment-270799379,https://api.github.com/repos/pydata/xarray/issues/964,270799379,MDEyOklzc3VlQ29tbWVudDI3MDc5OTM3OQ==,1217238,2017-01-06T00:36:21Z,2017-01-06T00:36:21Z,MEMBER,"OK, in it goes. Once again, there's no public API exposed yet.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,170779798
https://github.com/pydata/xarray/pull/964#issuecomment-270429069,https://api.github.com/repos/pydata/xarray/issues/964,270429069,MDEyOklzc3VlQ29tbWVudDI3MDQyOTA2OQ==,1217238,2017-01-04T17:19:10Z,2017-01-04T17:20:02Z,MEMBER,"I removed the public facing API and renamed the (now private) apply function back to `apply_ufunc`. I also removed the `new_coords` argument, in favor of encouraging using `.coords` or `.assign_coords`.
As discussed above, the current API with `signature` is difficult to use, but this is probably fine for an internal function and we can revisit the public facing API later. Any objections to merging this?","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,170779798
https://github.com/pydata/xarray/pull/964#issuecomment-269878903,https://api.github.com/repos/pydata/xarray/issues/964,269878903,MDEyOklzc3VlQ29tbWVudDI2OTg3ODkwMw==,1217238,2016-12-31T19:29:37Z,2016-12-31T19:30:16Z,MEMBER,"@crusaderky
> any plans to add dask support as suggested above?
Yes, in fact I have a branch with some basic support for this that I was working on a few months ago. I haven't written tests yet but I can potentially push that WIP to another PR after merging this.
There are a couple of recent feature additions to `dask.array.atop` (https://github.com/dask/dask/pull/1612 and https://github.com/dask/dask/pull/1716) that should make this easier and more powerful. I have not built anything on top of these yet, so my prior work is somewhat outdated.
@jhamman
> do we want to get this into 0.9 as a private api function and aim to complete it for the public api by 0.10 or so?
Yes, this seems like a good goal. I'll take another look over this next week when I have the chance, to remove any work-in-progress bits that have snuck in and remove the public facing API.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,170779798
https://github.com/pydata/xarray/pull/964#issuecomment-264020777,https://api.github.com/repos/pydata/xarray/issues/964,264020777,MDEyOklzc3VlQ29tbWVudDI2NDAyMDc3Nw==,1217238,2016-11-30T22:45:07Z,2016-11-30T22:45:07Z,MEMBER,"> Surprisingly, I can't actually find something like this out there. The pandas code is good but highly 1-2 dimension specific.
>
> Let me know if I'm missing (pun intended - long day) something. Is there a library of these sorts of functions over n-dims somewhere else (even R / Julia)? Or are we really the first people in the world to be doing this?
Usually I check `numpy` and `bottleneck`. It actually looks like [`bottleneck.push`](http://berkeleyanalytics.com/bottleneck/reference.html#bottleneck.push) is what you're looking for. I think this is a recent addition to bottleneck, though.","{""total_count"": 1, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 1, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,170779798
https://github.com/pydata/xarray/pull/964#issuecomment-263944098,https://api.github.com/repos/pydata/xarray/issues/964,263944098,MDEyOklzc3VlQ29tbWVudDI2Mzk0NDA5OA==,1217238,2016-11-30T17:51:33Z,2016-11-30T17:51:33Z,MEMBER,"> I'm thinking through how difficult it would be to add back-fill method to DataArray (that could be an argument to fillna or a bfill method - that's a separate discussion).
>
> Would this PR help? I'm trying to wrap my head around the options. Thanks
Yes, quite likely. In the current state, it would depend on if you want to back-fill all variables or just data variables (only the later is currently supported).
Either way, the first step is probably to write a function `backfill(values, axis)` that acts on NumPy arrays.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,170779798
https://github.com/pydata/xarray/pull/964#issuecomment-256814663,https://api.github.com/repos/pydata/xarray/issues/964,256814663,MDEyOklzc3VlQ29tbWVudDI1NjgxNDY2Mw==,1217238,2016-10-28T01:31:02Z,2016-10-28T01:31:02Z,MEMBER,"I'm thinking about making a few tweaks and merging this, but not exposing it to users yet as part of public API. The public API is not quite there yet, but even as it I think it would be a useful building point for internal functionality (e.g., for #1065), and then other people could start to build on this as well.
","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,170779798
https://github.com/pydata/xarray/pull/964#issuecomment-249118534,https://api.github.com/repos/pydata/xarray/issues/964,249118534,MDEyOklzc3VlQ29tbWVudDI0OTExODUzNA==,1217238,2016-09-23T07:07:09Z,2016-09-23T07:07:09Z,MEMBER,"One of the tricky things with `apply` is that there are a lot of similar but distinct use cases to disambiguate. I'll outline a few of these below.
I'd appreciate feedback on which cases are most essential and which can wait until later (this PR is already getting pretty big).
Also, I'd appreciate ideas for how to make the API more easily understood. We will have extensive docs either way, but `xarray.apply` is probably already in the realm of ""too many arguments for one function"". The last thing I want to do is to make a swiss army knife so flexible (like (`numpy.nditer`)[http://docs.scipy.org/doc/numpy/reference/generated/numpy.nditer.html]) that nobody uses it because they don't understand how it works.
## How `func` vectorizes
There are two main cases here:
1. Functions already written to vectorize their arguments:
1. Scalar functions built out of NumPy primitives (e.g., `a + b + c`). These work by default.
2. Functions that use core dimension referred to by `axis` (e.g., `np.mean`). These work if you set `axis=-1` and put the dimension in the signature, but the API is kind of awkward. You're rather that the wrapper just converts argument like `dim='time'` automatically into `axis=2`. Transposing these core dimensions to the end also feels unnecessary, though maybe not a serious concern given that transposing NumPy arrays involves no memory copies.
3. Functions that work mostly like gufuncs, but aren't actually (e.g., `np.svd`). This is pretty common, because NumPy ufuncs have some serious limitations (e.g.., they can't handle non-vectorized arguments). These work about as well as we could hope, modulo possible improvements to the `signature` spec.
4. True gufuncs, most likely written with `numba.guvectorize`. For these functions, we'd like a way to extract/use the signature automatically.
2. Functions for which you only have the inner loop (e.g., `np.polyfit` or `scipy.stats.pearsonr`). Running these is going to entail large Python overhead, but often that's acceptable.
One option for these is to wrap them into something that broadcasts like a gufunc, e.g., via a new function `numpy.guvectorize` (https://github.com/numpy/numpy/pull/8054). But as a user, this is a lot of wrappers to write. You'd rather just add something like `vectorize=True` and let xarray handle all the automatic broadcasting, e.g.,
``` python
def poly_fit(x, y, dim='time', deg=1):
return xr.apply(np.polyfit, x, y,
signature=([(dim,), (dim,)], [('poly_order',)]),
new_coords={'poly_order': range(deg + 1)},
kwargs={'deg': deg}, vectorize=True)
```
## Whether `func` applies to ""data only"" or ""everything""
Most ""computation"" functions/methods in xarray (e.g., arithmetic and reduce methods) follow the rule of merging coordinates, and only applying the core function to data variables. Coordinates that are no longer valid with new dimensions are dropped. This is currently what we do in `apply`.
On the other hand, there are also function/methods that we might refer to as ""organizing"" (e.g., indexing methods, `concat`, `stack`/`unstack`, `transpose`), which generally apply to every variable, including coordinates. It seems like there are definitely use cases for applying these sorts of functions, too, e.g., to wrap Cartopy's `add_cyclic_point` utility (#1005). So, I think we might need another option to toggle what happens to coordinates (e.g., `variables='data'` vs `variables='all'`).
## How to handle mismatched core dimensions
Xarray methods often have fallbacks to handle data with different dimensions. For example, if you write `ds.mean(['x', 'y'])`, it matches on core dimensions to apply four different possible functions to each data variables:
- mean over `('x', 'y')`: for variables with both dimensions
- mean over `'x'`: for variables with only `'x'`
- mean over `'y'`: for variables with only `'y'`
- identity: for variables with neither `'x'` nor `'y'`
Indexing is another example -- it applies to both data and coordinates, but only to matching dimensions for each variable. If you don't have the dimensions, we ignore the variable.
Writing something like `mean` with a single call to `apply` would entail the need for something like a dispatching system to pick which function to use, e.g., instead of a singular `func`/`signature` pair you pass a `dispatcher` function that chooses `func`/`signature` based on the core dimensions of passed variable. This feels like serious over engineering.
Instead, we might support a few pre-canned options for how to deal with mismatched dimensions. For example:
- `missing_core_dims='drop'`: silently drop these variables in the output(s)
- `missing_core_dims='error'`: raise an error. This is the current default behavior, which is probably only be useful with `variables='data' -- otherwise some coordinate variables would always error.
- `missing_core_dims='keep'`: keep these variables unchanged in the output(s) (use `merge_variables` to check for conflicts)
- `missing_core_dims='broadcast'`: broadcast all inputs to have the necessary core dimensions if they don't have them already
Another option would be to consolidate this with the `variables` option to allow only two modes of operation:
- `variables='data'`: For ""computation"" functions. Apply only to data variables, and error if any data variables are missing a core dimension. Merge coordinates on the output(s), silently dropping conflicts for variables that don't label a dimension.
- `variables='matching'`: For ""organizing"" functions. Apply to every variable with matching core dimensions. Merge everything else on the outputs(s), silently dropping conflicts for variables that don't label a dimension.
","{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,170779798
https://github.com/pydata/xarray/pull/964#issuecomment-248798053,https://api.github.com/repos/pydata/xarray/issues/964,248798053,MDEyOklzc3VlQ29tbWVudDI0ODc5ODA1Mw==,1217238,2016-09-22T02:57:29Z,2016-09-22T02:57:29Z,MEMBER,"@chris-b1 thanks for giving this is a try! Using this with numba's guvectorize is exactly what I had in mind.
1) Yes, we need a better error here.
2) Agreed, it's really hard to parse a triply nested list. I don't like encouraging writing signature strings though because that artificially restricts dimensions to use strings (and it's weird to program in strings).
Maybe separate arguments for `input_core_dims` and `output_core_dims` would make more sense, e.g., `xr.apply(std_gufunc, arr, input_core_dims=[('x',)])`?
3) Yes, agreed. The main issue with `xr.apply(std_gufunc, arr, dims=('x',))` is that it's not clear what that means if `std_gufunc` is not a gufunc. For example, one might expect it to generate `axis` arguments.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,170779798
https://github.com/pydata/xarray/pull/964#issuecomment-248349007,https://api.github.com/repos/pydata/xarray/issues/964,248349007,MDEyOklzc3VlQ29tbWVudDI0ODM0OTAwNw==,1217238,2016-09-20T16:06:40Z,2016-09-20T16:06:40Z,MEMBER,"And yes, `py.test` _should_ work on master (not 100% sure for stable, which is only used for pushing doc updates). That's what we test on Travis-CI: https://travis-ci.org/pydata/xarray/builds. I'm happy to help debug, but let's open another issue for that.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,170779798
https://github.com/pydata/xarray/pull/964#issuecomment-248348585,https://api.github.com/repos/pydata/xarray/issues/964,248348585,MDEyOklzc3VlQ29tbWVudDI0ODM0ODU4NQ==,1217238,2016-09-20T16:05:13Z,2016-09-20T16:05:13Z,MEMBER,"@pwolfram If you don't need dask support, then I would suggest simply trying this PR. I think a slight variation of my `mean` example at the top would work.
If you do need dask support, either just use `da.nancumsum` directly with this PR (along with `dask_array='allowed'`) or we can use `core.ops._dask_or_eager_func` (e.g., as in your PR) to make a version of `nancumsum` that works on both dask and numpy arrays, and then use that version along with this PR.
","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,170779798
https://github.com/pydata/xarray/pull/964#issuecomment-246731502,https://api.github.com/repos/pydata/xarray/issues/964,246731502,MDEyOklzc3VlQ29tbWVudDI0NjczMTUwMg==,1217238,2016-09-13T15:59:28Z,2016-09-13T15:59:28Z,MEMBER,"CC @jhamman @rabernat @crusaderky @pwolfram @spencerahill @ajdawson
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,170779798
https://github.com/pydata/xarray/pull/964#issuecomment-246551799,https://api.github.com/repos/pydata/xarray/issues/964,246551799,MDEyOklzc3VlQ29tbWVudDI0NjU1MTc5OQ==,1217238,2016-09-13T02:04:38Z,2016-09-13T02:04:38Z,MEMBER,"This is now tested and ready for review. The API could particularly use feedback -- please take a look at the docstring and examples in the first comment. Long desired operations, like a fill value for `where` (#576) and `cumsum` (#791) should now be writable in only a few lines.
I have not yet hooked this up to the rest of xarray's code base, both because the set of changes we will be able to do with this are quite large, and because I'd like to give other contributors a chance to help/test. Note that the general version of `apply_ufunc` can include some significant overhead for doing the dispatch. For binary operations, we will probably want to use the pre-specialized versions (e.g., `apply_dataset_ufunc`).
Finally, given the generality of this operation, I'm considering renaming it from `xr.apply_ufunc` to simply `xr.apply`.
","{""total_count"": 1, ""+1"": 1, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,170779798
https://github.com/pydata/xarray/pull/964#issuecomment-239506907,https://api.github.com/repos/pydata/xarray/issues/964,239506907,MDEyOklzc3VlQ29tbWVudDIzOTUwNjkwNw==,1217238,2016-08-12T17:21:25Z,2016-08-12T17:22:32Z,MEMBER,"@MaximilianR Two issues come to mind with remapping `da[condition] = other` -> `da = da.where(bool_array, other)`:
1. What should `da[bool_array]` return? For consistency, I think we need to support both. We could alias that to `where`, too (`da[bool_array]` -> `da.where(bool_array)`), but this would be inconsistent with the existing behavior of `da[bool_array]` if `da` and `bool_array` are one-dimensional. This suggests maybe `ds[bool_array]` -> `da.where(bool_array, drop=True)`.
2. What is `other` is not a scalar value, but is a DataArray? Currently, I don't think we align when indexing, but in this case, by necessity we would. For `__getitem__` indexing, it's pretty obvious that alignment should preserve the indexes of the object being indexed. For `__setitem__` indexing, I'm not sure. At the least they would be a little different from the defaults for `where` (which does an inner join like most xarray operations by default). Maybe something like: `left_join(da, inner_join(bool_array, other))`? We need to make sure that every xarray assignment like `obj[key]` or `obj[key] = value` (and also `.loc` and `.sel` and so on) works sanely with these alignment rules.
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,170779798