html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue
https://github.com/pydata/xarray/issues/3349#issuecomment-581584469,https://api.github.com/repos/pydata/xarray/issues/3349,581584469,MDEyOklzc3VlQ29tbWVudDU4MTU4NDQ2OQ==,20629530,2020-02-03T19:43:53Z,2020-02-03T19:43:53Z,CONTRIBUTOR,"I pushed a new PR trying to implement polyfit in xarray, #3733. It is still work in progress, but I would like the opinion on those who participated in this thread.

Considering all options discussed in the thread, I chose an implementation that seemed to give the best performance and generality (skipping NaN values), but it duplicates a lot of code from `numpy.polyfit`.

Main question: 

- Should xarray's implementation really replicate the behaviour of numpy's? 

A lot of extra code could be removed if we'd say we only want to compute and return the residuals and the coefficients. All the other variables are a few lines of code away for the user that really wants them, and they don't need the power of xarray and dask anyway.

I'm guessing @huard  @dcherian @rabernat and @shoyer  might have comments.","{""total_count"": 2, ""+1"": 2, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,499477363
https://github.com/pydata/xarray/issues/3349#issuecomment-566356372,https://api.github.com/repos/pydata/xarray/issues/3349,566356372,MDEyOklzc3VlQ29tbWVudDU2NjM1NjM3Mg==,6777736,2019-12-17T02:51:44Z,2019-12-17T02:51:44Z,NONE,GeoCAT is licensed under Apache 2.0. So if someone wants to incorporate it into Xarray they are welcome to it :-),"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,499477363
https://github.com/pydata/xarray/issues/3349#issuecomment-566220633,https://api.github.com/repos/pydata/xarray/issues/3349,566220633,MDEyOklzc3VlQ29tbWVudDU2NjIyMDYzMw==,81219,2019-12-16T20:04:44Z,2019-12-16T20:04:44Z,CONTRIBUTOR,@clyne Let me rephrase my question: how do you feel about xarray providing a polyfit/polyval implementation essentially duplicating GeoCat's implementation ? ,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,499477363
https://github.com/pydata/xarray/issues/3349#issuecomment-566188037,https://api.github.com/repos/pydata/xarray/issues/3349,566188037,MDEyOklzc3VlQ29tbWVudDU2NjE4ODAzNw==,6777736,2019-12-16T18:40:28Z,2019-12-16T18:40:28Z,NONE,"Currently the plan is to keep GeoCAT as a standalone package that plays well with Xarray.

> On Dec 16, 2019, at 9:21 AM, Mohammad Abouali <notifications@github.com> wrote:
> 
> @maboualidev Is your objective to integrate the GeoCat implementation into xarray or keep it standalone ?
> 
> GeoCAT is the python version of NCL and we are a team at NCAR working on it. I know that the team decision is to make use of Xarray within GeoCAT as much as possible, though.
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or unsubscribe.
> 



","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,499477363
https://github.com/pydata/xarray/issues/3349#issuecomment-566132726,https://api.github.com/repos/pydata/xarray/issues/3349,566132726,MDEyOklzc3VlQ29tbWVudDU2NjEzMjcyNg==,24830983,2019-12-16T16:21:18Z,2019-12-16T16:21:18Z,NONE,"> @maboualidev Is your objective to integrate the GeoCat implementation into xarray or keep it standalone ?

[GeoCAT](https://geocat.ucar.edu) is the python version of [NCL](https://www.ncl.ucar.edu) and we are a team at [NCAR](https://ncar.ucar.edu) working on it. I know that the team decision is to make use of Xarray within GeoCAT as much as possible, though. 

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,499477363
https://github.com/pydata/xarray/issues/3349#issuecomment-566070483,https://api.github.com/repos/pydata/xarray/issues/3349,566070483,MDEyOklzc3VlQ29tbWVudDU2NjA3MDQ4Mw==,81219,2019-12-16T13:53:27Z,2019-12-16T13:53:27Z,CONTRIBUTOR,"@maboualidev Is your objective to integrate the GeoCat implementation into xarray or keep it standalone ? 

On my end, I'll submit a PR to add support for non-standard calendars to `xarray.core.missing.get_clean_interp`, which you'd then be able to use to get x values from coordinates. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,499477363
https://github.com/pydata/xarray/issues/3349#issuecomment-565738511,https://api.github.com/repos/pydata/xarray/issues/3349,565738511,MDEyOklzc3VlQ29tbWVudDU2NTczODUxMQ==,24830983,2019-12-14T17:58:34Z,2019-12-14T18:00:08Z,NONE,"Hi @huard Thanks for the reply.

Regarding:

> There does not seem to be matching polyval implementations for any of those nor support for indexing along a time dimension with a non-standard calendar.

There is a pull request on GeoCAT-comp for [ndpolyval](https://github.com/NCAR/geocat-comp/pull/49). I think `polyval` and `polyfit` go hand-in-hand. If we have `ndpolyfit` there must be a also a `ndpolyval`. 

Regarding:

> I see you're storing the residuals in the DataArray attributes. From my perspective, it would be useful to have those directly as DataArrays. Thoughts ?

I see the point and agree with you. I think it is a good idea to be as similar to `NumPy.polyfit` as possible; even for the style of the output. I will see it through to have that changed in GeoCAT.


attn: @clyne and @khallock","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,499477363
https://github.com/pydata/xarray/issues/3349#issuecomment-565733023,https://api.github.com/repos/pydata/xarray/issues/3349,565733023,MDEyOklzc3VlQ29tbWVudDU2NTczMzAyMw==,81219,2019-12-14T16:43:14Z,2019-12-14T16:43:14Z,CONTRIBUTOR,"@maboualidev Nice ! I see you're storing the residuals in the DataArray attributes. From my perspective, it would be useful to have those directly as DataArrays. Thoughts ?

So it looks like there are multiple inspirations to draw from. Here is what I could gather. 

- `xscale.signal.fitting.polyfit(obj, deg=1, dim=None, coord=None)` supports chunking along the fitting dimension using `dask.array.linalg.lstsq`. No explicit missing data handling.
- `xyzpy.signal.xr_polyfit(obj, dim, ix=None, deg=0.5, poly='hermite')` applies `np.polynomial.polynomial.polyfit` using `xr.apply_ufunc` along dim with the help of `numba`. Also supports other types of polynomial (legendre, chebyshev, ...). Missing values are masked out 1D wise. 
 - `geocat.comp.ndpolyfit(x: Iterable, y: Iterable, deg: int, axis: int = 0, **kwargs) -> (xr.DataArray, da.Array)` reorders the array to apply `np.polyfit` along dim, returns the full outputs (residuals, rank, etc) as DataArray attributes.  Missing values are masked out in bulk if possible, 1D-wise otherwise. 

There does not seem to be matching `polyval` implementations for any of those nor support for indexing along a time dimension with a non-standard calendar. 
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,499477363
https://github.com/pydata/xarray/issues/3349#issuecomment-565675452,https://api.github.com/repos/pydata/xarray/issues/3349,565675452,MDEyOklzc3VlQ29tbWVudDU2NTY3NTQ1Mg==,24830983,2019-12-14T03:01:44Z,2019-12-14T03:03:14Z,NONE,"[geocat.comp.ndpolyfit](https://geocat-comp.readthedocs.io/en/latest/user_api/generated/geocat.comp.ndpolyfit.html#geocat.comp.ndpolyfit) extends `NumPy.polyfit` for multi-dimensional arrays and has support for Xarray and Dask. It does exactly what is requested here.

regards,

@andersy005 @clyne @matt-long @khallock","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,499477363
https://github.com/pydata/xarray/issues/3349#issuecomment-565608876,https://api.github.com/repos/pydata/xarray/issues/3349,565608876,MDEyOklzc3VlQ29tbWVudDU2NTYwODg3Ng==,81219,2019-12-13T21:07:39Z,2019-12-13T21:07:39Z,CONTRIBUTOR,"My current implementation is pretty naive. It's just calling numpy.polyfit using dask.array.apply_along_axis. Happy to put that in a PR as a starting point, but there are a couple of questions I had: 
* How to return the full output (residuals, rank, singular_values, rcond) ? A tuple of dataarrays or a dataset ?
* Do we want to use the dask least square functionality to allow for chunking within the x dimension ? Then it's not just a simple wrapper around polyfit. 
* Should we use np.polyfit or np.polynomial.polynomial.polyfit ?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,499477363
https://github.com/pydata/xarray/issues/3349#issuecomment-565600745,https://api.github.com/repos/pydata/xarray/issues/3349,565600745,MDEyOklzc3VlQ29tbWVudDU2NTYwMDc0NQ==,6628425,2019-12-13T20:39:21Z,2019-12-13T20:39:21Z,MEMBER,"Excellent, looking forward to seeing it in a PR!","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,499477363
https://github.com/pydata/xarray/issues/3349#issuecomment-565504692,https://api.github.com/repos/pydata/xarray/issues/3349,565504692,MDEyOklzc3VlQ29tbWVudDU2NTUwNDY5Mg==,81219,2019-12-13T16:20:19Z,2019-12-13T16:20:19Z,CONTRIBUTOR,"Thanks, it seems to work !","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,499477363
https://github.com/pydata/xarray/issues/3349#issuecomment-565462443,https://api.github.com/repos/pydata/xarray/issues/3349,565462443,MDEyOklzc3VlQ29tbWVudDU2NTQ2MjQ0Mw==,6628425,2019-12-13T14:33:53Z,2019-12-13T14:33:53Z,MEMBER,"If I understand correctly, `pd.to_numeric` (and its inverse) works because it always uses 1970-01-01T00:00:00 as the reference date.  Could we do something similar when working with cftime dates?

Within xarray we typically convert dates to numeric values (e.g. when doing interpolation) using `xarray.core.duck_array_ops.datetime_to_numeric`, which takes an optional `offset` argument to control the reference date.  Would it work to always make sure to pass 1970-01-01T00:00:00 with the appropriate calendar type as the offset when constructing the ordinal x-coordinate for `polyfit`/`polyval`?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,499477363
https://github.com/pydata/xarray/issues/3349#issuecomment-565452240,https://api.github.com/repos/pydata/xarray/issues/3349,565452240,MDEyOklzc3VlQ29tbWVudDU2NTQ1MjI0MA==,81219,2019-12-13T14:04:23Z,2019-12-13T14:04:23Z,CONTRIBUTOR,"Started to work on this and facing some issues with the x-coordinate when its a datetime. For standard calendars, I can use `pd.to_numeric(da.time)`, but for non-standard calendars, it's not clear how to go ahead. If I use `xr.coding.times.encode_cf_datetime(coord)`, the coefficients we'll find will only make sense in the `polyval` function if we use the same time encoding. 

","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,499477363
https://github.com/pydata/xarray/issues/3349#issuecomment-561241142,https://api.github.com/repos/pydata/xarray/issues/3349,561241142,MDEyOklzc3VlQ29tbWVudDU2MTI0MTE0Mg==,2448579,2019-12-03T16:17:00Z,2019-12-03T16:17:00Z,MEMBER,xyzpy has a polyfit too : https://xyzpy.readthedocs.io/en/latest/manipulate.html,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,499477363
https://github.com/pydata/xarray/issues/3349#issuecomment-537622127,https://api.github.com/repos/pydata/xarray/issues/3349,537622127,MDEyOklzc3VlQ29tbWVudDUzNzYyMjEyNw==,1217238,2019-10-02T18:30:25Z,2019-10-02T18:30:25Z,MEMBER,"It seems like these are getting reinvented often enough that it's worth
pulling some of these into xarray proper.

On Wed, Oct 2, 2019 at 9:04 AM Ryan Abernathey <notifications@github.com>
wrote:

> I'm getting deja-vu here... Xscale has a huge and impressive sounding API.
> But no code coverage and no commits since January. Like many of these
> projects, it seems to have bit off more than its maintainers could chew.
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/pydata/xarray/issues/3349?email_source=notifications&email_token=AAJJFVV5XANAGSPKHSF6KZLQMTA7HA5CNFSM4I3HCUUKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAFJCDI#issuecomment-537563405>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAJJFVREMUT5RKJJESG5NVLQMTA7HANCNFSM4I3HCUUA>
> .
>
","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,499477363
https://github.com/pydata/xarray/issues/3349#issuecomment-537563405,https://api.github.com/repos/pydata/xarray/issues/3349,537563405,MDEyOklzc3VlQ29tbWVudDUzNzU2MzQwNQ==,1197350,2019-10-02T16:04:02Z,2019-10-02T16:06:20Z,MEMBER,"I'm getting deja-vu here... Xscale has a huge and impressive sounding API. But no code coverage and no commits since January. Like many of these projects, it seems to have bit off more than its maintainers could chew.

_Edit: I'd love for such a package to really achieve community uptake and become sustainable. I just don't quite know the roadmap for getting there._","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,499477363
https://github.com/pydata/xarray/issues/3349#issuecomment-537552488,https://api.github.com/repos/pydata/xarray/issues/3349,537552488,MDEyOklzc3VlQ29tbWVudDUzNzU1MjQ4OA==,2448579,2019-10-02T15:40:04Z,2019-10-02T15:40:04Z,MEMBER,https://xscale.readthedocs.io/en/latest/generated/xscale.signal.fitting.polyfit.html#xscale.signal.fitting.polyfit,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,499477363
https://github.com/pydata/xarray/issues/3349#issuecomment-536690748,https://api.github.com/repos/pydata/xarray/issues/3349,536690748,MDEyOklzc3VlQ29tbWVudDUzNjY5MDc0OA==,1217238,2019-09-30T18:29:10Z,2019-09-30T18:29:10Z,MEMBER,"From a user perspective, I think people prefer to find stuff in one place.

From a maintainer perspective, as long as it's somewhat domain agnostic (e.g., ""physical sciences"" rather than ""oceanography"") and written to a reasonable level of code quality, I think it's fine to toss it into xarray. ""Already exists in NumPy/SciPy"" is probably a reasonable proxy for the former.

So I say: yes, let's toss in polyfit, along with fast fourier transforms.

If we're concerned about clutter, we can put stuff in a dedicated namespace, e.g., `xarray.wrappers`.","{""total_count"": 3, ""+1"": 3, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,499477363
https://github.com/pydata/xarray/issues/3349#issuecomment-536670564,https://api.github.com/repos/pydata/xarray/issues/3349,536670564,MDEyOklzc3VlQ29tbWVudDUzNjY3MDU2NA==,1197350,2019-09-30T17:42:57Z,2019-09-30T17:42:57Z,MEMBER,"Now that xarray itself has interpolate, gradient, and integrate, it seems like the only thing left in xr-scipy is fourier transforms, which is also what we provide in [xrft](https://github.com/xgcm/xrft)! 😆 ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,499477363
https://github.com/pydata/xarray/issues/3349#issuecomment-536663815,https://api.github.com/repos/pydata/xarray/issues/3349,536663815,MDEyOklzc3VlQ29tbWVudDUzNjY2MzgxNQ==,2448579,2019-09-30T17:25:38Z,2019-09-30T17:25:38Z,MEMBER,The quickest way to close this is probably to extend @fujiisoup's xr-scipy(https://github.com/fujiisoup/xr-scipy) to wrap `scipy.linalg.lstsq` and `dask.array.linalg.lstsq`. It is likely that all the necessary helper functions already exist.,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,499477363
https://github.com/pydata/xarray/issues/3349#issuecomment-536654572,https://api.github.com/repos/pydata/xarray/issues/3349,536654572,MDEyOklzc3VlQ29tbWVudDUzNjY1NDU3Mg==,1197350,2019-09-30T17:01:57Z,2019-09-30T17:01:57Z,MEMBER,"The question of a standalone library has come up many times (see discussion in #1850). Everyone agrees it's a nice idea, but no one seems to have the bandwidth to take on ownership and maintenance of such a project.

Perhaps we need to put this issue on pause and figure out a general strategy here. The current Xarray API is far from a complete feature set, so more development is needed. But we should decide what belongs in xarray and what belongs elsewhere. #1850 is probably the best place to continue that conversation.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,499477363
https://github.com/pydata/xarray/issues/3349#issuecomment-536079602,https://api.github.com/repos/pydata/xarray/issues/3349,536079602,MDEyOklzc3VlQ29tbWVudDUzNjA3OTYwMg==,4992424,2019-09-27T20:07:13Z,2019-09-27T20:07:13Z,NONE,I second @TomNicholas' point... functionality like this would be wonderful to have but where would be the best place for it to live?,"{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,499477363
https://github.com/pydata/xarray/issues/3349#issuecomment-536078107,https://api.github.com/repos/pydata/xarray/issues/3349,536078107,MDEyOklzc3VlQ29tbWVudDUzNjA3ODEwNw==,35968931,2019-09-27T20:02:02Z,2019-09-27T20:02:02Z,MEMBER,"I am in favour of adding this (and other common functionality), but I would comment that perhaps we should move forward with discussion about where to put extra functionality generally (the scipy to xarray's numpy if you will)? If only because otherwise the API could get to an unwieldy size? 

I can't remember where the relevant issue was, but for example this might go under an `xarray.utils` module?","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,499477363
https://github.com/pydata/xarray/issues/3349#issuecomment-535967743,https://api.github.com/repos/pydata/xarray/issues/3349,535967743,MDEyOklzc3VlQ29tbWVudDUzNTk2Nzc0Mw==,2448579,2019-09-27T14:39:02Z,2019-09-27T15:07:33Z,MEMBER,"dask has `lstsq` https://docs.dask.org/en/latest/array-api.html#dask.array.linalg.lstsq . Would that avoid the dimension-must-have-one-chunk issue?

EDIT: I am in favour of adding this. It's a common use case like `differentiate` and `integrate`","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,499477363