issue_comments

Table actions

15 rows where issue = 295959111 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: reactions, created_at (date), updated_at (date)

author_association 2

issue 1

  • Representing & checking Dataset schemas · 15 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
1184332756 https://github.com/pydata/xarray/issues/1900#issuecomment-1184332756 https://api.github.com/repos/pydata/xarray/issues/1900 IC_kwDOAMm_X85Gl3vU kubaraczkowski 554652 2022-07-14T11:28:37Z 2022-07-14T11:28:37Z NONE

Does this project do (part of?) what's needed? +1 on making xarrays with explicit 'structure' !

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Representing & checking Dataset schemas  295959111
1008488315 https://github.com/pydata/xarray/issues/1900#issuecomment-1008488315 https://api.github.com/repos/pydata/xarray/issues/1900 IC_kwDOAMm_X848HE97 jhamman 2443309 2022-01-10T02:06:59Z 2022-01-10T02:06:59Z MEMBER

Related to the Pandera integration, we are prototyping the xarray schema validation functionality in the xarray-schema project. 

{
    "total_count": 3,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 3,
    "eyes": 0
}
   Representing & checking Dataset schemas  295959111
1008224592 https://github.com/pydata/xarray/issues/1900#issuecomment-1008224592 https://api.github.com/repos/pydata/xarray/issues/1900 IC_kwDOAMm_X848GElQ andersy005 13301940 2022-01-09T03:56:40Z 2022-01-09T03:56:40Z MEMBER 
{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Representing & checking Dataset schemas  295959111
938741037 https://github.com/pydata/xarray/issues/1900#issuecomment-938741037 https://api.github.com/repos/pydata/xarray/issues/1900 IC_kwDOAMm_X8439A0t rabernat 1197350 2021-10-08T15:41:29Z 2021-10-08T15:41:29Z MEMBER

But Pydantic looks promising

Big :+1: to this. 

{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Representing & checking Dataset schemas  295959111
938691112 https://github.com/pydata/xarray/issues/1900#issuecomment-938691112 https://api.github.com/repos/pydata/xarray/issues/1900 IC_kwDOAMm_X84380oo JackKelly 460756 2021-10-08T14:32:44Z 2021-10-08T14:35:46Z NONE

OK, I think pandera isn't the way forwards because it appears very tighly coupled to Pandas (so, for example, I don't think it's possible to use pandera with n-dimensional arrays).

But Pydantic looks promising. Here's a very quick coding experiment showing one way to use pydantic with xarray... it validates a few things; but it's not super-useful as a human-readable specification for what's going on inside a DataArray or Dataset.

 
{
    "total_count": 2,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 1
}
   Representing & checking Dataset schemas  295959111
938409463 https://github.com/pydata/xarray/issues/1900#issuecomment-938409463 https://api.github.com/repos/pydata/xarray/issues/1900 IC_kwDOAMm_X8437v33 shoyer 1217238 2021-10-08T07:25:44Z 2021-10-08T07:25:44Z MEMBER

Pandera provides machine-readable schemas for Pandas and, as of version 0.7, panderas has decoupled pandera and pandas types to make pandera more useful for things like xarray. I haven't tried pandera yet but I plan to do some experiments soon.

Awesome -- would love to hear how this goes!

 
{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Representing & checking Dataset schemas  295959111
938397801 https://github.com/pydata/xarray/issues/1900#issuecomment-938397801 https://api.github.com/repos/pydata/xarray/issues/1900 IC_kwDOAMm_X8437tBp JackKelly 460756 2021-10-08T07:04:51Z 2021-10-08T07:04:51Z NONE

I'm really interested in a machine-readable schema for xarray!

Pandera provides machine-readable schemas for Pandas and, as of version 0.7, panderas has decoupled pandera and pandas types to make pandera more useful for things like xarray. I haven't tried pandera yet but I plan to do some experiments soon.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Representing & checking Dataset schemas  295959111
377012442 https://github.com/pydata/xarray/issues/1900#issuecomment-377012442 https://api.github.com/repos/pydata/xarray/issues/1900 MDEyOklzc3VlQ29tbWVudDM3NzAxMjQ0Mg== max-sixty 5635139 2018-03-28T19:46:33Z 2018-03-28T19:46:33Z MEMBER

The commentary in https://github.com/python/typing/issues/513, and @shoyer 's doc https://docs.google.com/document/d/1vpMse4c6DrWH5rq2tQSx3qwP_m_0lyn-Ij4WHqQqRHY/edit#heading=h.rkj7d39awayl are good & growing

I'll close this as I think riding on those coattails - with the addition of names and Datasets as containers - makes the most sense.

(though reopen if we think there's something we could productively do separately)

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Representing & checking Dataset schemas  295959111
367857689 https://github.com/pydata/xarray/issues/1900#issuecomment-367857689 https://api.github.com/repos/pydata/xarray/issues/1900 MDEyOklzc3VlQ29tbWVudDM2Nzg1NzY4OQ== benbovy 4160723 2018-02-22T23:25:11Z 2018-02-22T23:25:11Z MEMBER

@maxim-lian you're right. In this case xsimlab.Variable is a different concept than xarray.Variable, despite that they both have the same name. The former is tight to the modelling framework while the latter is only used for simulation inputs and outputs in xarray-simlab.

Do we have a list of projects that integrate xarray?

There is an ongoing discussion in #1850 about having something like xarray-contrib (likely a github organization).

 
{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Representing & checking Dataset schemas  295959111
367855371 https://github.com/pydata/xarray/issues/1900#issuecomment-367855371 https://api.github.com/repos/pydata/xarray/issues/1900 MDEyOklzc3VlQ29tbWVudDM2Nzg1NTM3MQ== max-sixty 5635139 2018-02-22T23:13:55Z 2018-02-22T23:13:55Z MEMBER

@benbovy That looks v interesting. I think at the moment it would require a bit of work to validate normal xarray objects, is that right? (I'm looking at the __init__, which doesn't take the traditional args supplied to a Variable - tell me if I'm misreading it)

Separately - I didn't know about the project but looks awesome. Do we have a list of projects that integrate xarray? Let's start one somewhere if not @pydata/xarray ?

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Representing & checking Dataset schemas  295959111
367853004 https://github.com/pydata/xarray/issues/1900#issuecomment-367853004 https://api.github.com/repos/pydata/xarray/issues/1900 MDEyOklzc3VlQ29tbWVudDM2Nzg1MzAwNA== benbovy 4160723 2018-02-22T23:03:27Z 2018-02-22T23:03:27Z MEMBER

Somewhat related to this issue, I have implemented in xarray-simlab some logic to validate xarray.Variable objects (dimensions, dtype, etc.). See this base class and some sub-classes here. I use that in a way which is quite similar to pandas-validator(i.e., using class attributes).

I'm currently in the process of refactoring this using attrs, which supports both validator functions and type annotations. Not sure how to use the latter for xarray objects, though (BTW I wasn't aware of python/typing#513, good to know!!).

I agree that it would be nice to have a more generic way to describe xarray objects that can be reused in many contexts.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Representing & checking Dataset schemas  295959111
364602759 https://github.com/pydata/xarray/issues/1900#issuecomment-364602759 https://api.github.com/repos/pydata/xarray/issues/1900 MDEyOklzc3VlQ29tbWVudDM2NDYwMjc1OQ== shoyer 1217238 2018-02-09T23:53:49Z 2018-02-09T23:53:49Z MEMBER

ncdump -h (xarray.Dataset.info()) is one existing schema of sorts, but it's hardly machine readable.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Representing & checking Dataset schemas  295959111
364599398 https://github.com/pydata/xarray/issues/1900#issuecomment-364599398 https://api.github.com/repos/pydata/xarray/issues/1900 MDEyOklzc3VlQ29tbWVudDM2NDU5OTM5OA== max-sixty 5635139 2018-02-09T23:31:44Z 2018-02-09T23:31:44Z MEMBER

And let me know if there are already textual schema definitions from other libraries that you think are good, before we go and build our own (we don't work with any netCDF-like files so don't have that context)

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Representing & checking Dataset schemas  295959111
364581903 https://github.com/pydata/xarray/issues/1900#issuecomment-364581903 https://api.github.com/repos/pydata/xarray/issues/1900 MDEyOklzc3VlQ29tbWVudDM2NDU4MTkwMw== max-sixty 5635139 2018-02-09T22:01:39Z 2018-02-09T22:01:39Z MEMBER

I think the right word for this may be "schema"

Right! 🤦‍♂️

Just googling around, I came up with pandas-validator

Interesting, thanks.

Do you think this fits into a 'function which validates', rather than a Mypy-like type annotation? I think ideally there would be a representation of the schema that could work with both, so maybe this isn't the important question atm.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Representing & checking Dataset schemas  295959111
364532198 https://github.com/pydata/xarray/issues/1900#issuecomment-364532198 https://api.github.com/repos/pydata/xarray/issues/1900 MDEyOklzc3VlQ29tbWVudDM2NDUzMjE5OA== shoyer 1217238 2018-02-09T19:16:26Z 2018-02-09T19:16:26Z MEMBER

I think the right word for this may be "schema". For applications and models (rather than data analysis), these sort of conventions can be super-valuable. I like the idea of declarative spec that can be validated.

Just googling around, I came up with pandas-validator: https://github.com/c-data/pandas-validator 

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Representing & checking Dataset schemas  295959111

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);