home / github

Menu
  • Search all tables
  • GraphQL API

issue_comments

Table actions
  • GraphQL API for issue_comments

4 rows where author_association = "MEMBER", issue = 543825272 and user = 14808389 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date)

user 1

  • keewis · 4 ✖

issue 1

  • apply rolling window on function · 4 ✖

author_association 1

  • MEMBER · 4 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
777878102 https://github.com/pydata/xarray/issues/3656#issuecomment-777878102 https://api.github.com/repos/pydata/xarray/issues/3656 MDEyOklzc3VlQ29tbWVudDc3Nzg3ODEwMg== keewis 14808389 2021-02-12T00:05:51Z 2021-02-12T00:05:51Z MEMBER

I assume you figured this out somehow. If not, feel free to reopen.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  apply rolling window on function 543825272
605525025 https://github.com/pydata/xarray/issues/3656#issuecomment-605525025 https://api.github.com/repos/pydata/xarray/issues/3656 MDEyOklzc3VlQ29tbWVudDYwNTUyNTAyNQ== keewis 14808389 2020-03-28T21:58:40Z 2020-03-28T21:58:40Z MEMBER

is this still an issue, @mada0007? If it is, I tried a bit more (it is still incomplete since I don't really understand what happens, especially with the last groupby-agg) python In [7]: def get_grps(s, axis, thresh=-1, Nmin=2): ...: s = xr.DataArray(s, dims=("time", "lon", "lat", "window")) ...: m = ( ...: xr.concat( ...: [ ...: s.shift(window=-i).pipe(lambda arr: arr <= thresh) ...: for i in range(Nmin) ...: ], ...: dim="shifted", ...: ) ...: .reduce(lambda arr, axis: np.logical_and.reduce(arr, axis=axis), dim="shifted") ...: ) ...: if Nmin > 1: ...: m = ( ...: m.where(lambda arr: arr == False, np.nan) ...: .ffill(limit=Nmin - 1, dim="window") ...: .fillna(False) ...: ) ...: ...: gps = (m != m.shift(window=1)).cumsum(dim="window").where(m) ...: ...: groups = s.groupby(gps.rename("gps_groups")) ...: return xr.where( ...: gps.isnull().all(dim="window"), ...: 0, ...: 1, # use `groups` here once it has the same size as `gps` ...: ) ...: ...: spi = xr.DataArray(dims=("time", "lon", "lat"), data=np.random.randn(118, 18, 36)) ...: spi.rolling(time=59).reduce(get_grps) which mirrors your function except it does the calculation for all points in your space-time-grid at once.

Of course, once again I can't really check for errors since I don't understand what I'm computing :grin:

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  apply rolling window on function 543825272
570056960 https://github.com/pydata/xarray/issues/3656#issuecomment-570056960 https://api.github.com/repos/pydata/xarray/issues/3656 MDEyOklzc3VlQ29tbWVudDU3MDA1Njk2MA== keewis 14808389 2020-01-01T14:19:00Z 2020-01-01T14:19:00Z MEMBER

the reduce function needs to be a function that takes a 4D array (time x lat x lon x window in your case). It should use the elements of the window dimension (1 x 1 x 1 x 59) to calculate a time x lat x lon array.

The only thing wrong with your function is that it ignores the axis parameter: it can only work on 1D arrays (with a shape of 59 in this case).

A simple albeit horribly slow way to use your function without rewriting it would be numpy.apply_along_axis (slow because the function is called time * lat * lon == 20995200 times): python spi.rolling(time=59).reduce(lambda x, axis: np.apply_along_axis(get_grps, axis, x))

I'm not too familiar with writing numpy reduce functions so you might want to wait for the opinion of someone more experienced.

I started to rewrite your function using xarray.DataArray instead of pandas.Series, but I don't have time to finish that right now. Hopefully this points you in the right direction. python In [32]: axis = 3 ...: thresh = -1 ...: Nmin = 2 ...: # function code starts here ...: dim_names = ["time", "lat", "lon"] ...: # fails if axis == -1 but axis from reduce is always a positive number ...: dim_names.insert(axis, "window") ...: # use s = xr.DataArray(s, dims=dim_names) in the function ...: # so we can interactively write and try the function code ...: s = spi.rolling(time=59).construct("window") ...: shifted = xr.concat([s.shift({"window": i}) for i in range(Nmin)], dim="shifted") ...: thresholded = shifted <= thresh ...: # xarray does not support `ufunc.reduce`, so we need to fall back to numpy arrays ...: m = r.copy(data=np.logical_and.reduce(thresholded.values, axis=0)) ...: if Nmin > 1: ...: m = m.where(~m).ffill(limit=Nmin - 1, dim="window").fillna(False) ...: gps = m != m.shift({"window": 1}).cumsum().where(m)

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  apply rolling window on function 543825272
569650603 https://github.com/pydata/xarray/issues/3656#issuecomment-569650603 https://api.github.com/repos/pydata/xarray/issues/3656 MDEyOklzc3VlQ29tbWVudDU2OTY1MDYwMw== keewis 14808389 2019-12-30T11:18:12Z 2019-12-30T13:09:03Z MEMBER

the reason for the exception is that you are trying to convert the input data (a 4D numpy.ndarray of shape 324 x 180 x 360 x 59) to a pandas.Series which only takes 1D data. You will have to rewrite your function to accept a 4D array and reduce it to a 3D array along the given axis (see numpy functions like std or mean).

We should definitely improve the documentation, right now the docstring of DataArrayRolling.reduce and DatasetRolling.reduce is not really clear on the requirements for the reduce function and the examples don't show the use of custom functions.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  apply rolling window on function 543825272

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);
Powered by Datasette · Queries took 109.939ms · About: xarray-datasette