issue_comments
20 rows where issue = 187859705 sorted by updated_at descending
This data as json, CSV (advanced)
Suggested facets: reactions, created_at (date), updated_at (date)
issue 1
- Dataset groups · 20 ✖
id | html_url | issue_url | node_id | user | created_at | updated_at ▲ | author_association | body | reactions | performed_via_github_app | issue |
---|---|---|---|---|---|---|---|---|---|---|---|
873227866 | https://github.com/pydata/xarray/issues/1092#issuecomment-873227866 | https://api.github.com/repos/pydata/xarray/issues/1092 | MDEyOklzc3VlQ29tbWVudDg3MzIyNzg2Ng== | shoyer 1217238 | 2021-07-02T19:56:49Z | 2021-07-02T19:56:49Z | MEMBER | There's a parallel discussion hierarchical storage going on over in https://github.com/pydata/xarray/issues/4118. I'm going to close this issue in favor of the other one just to keep the ongoing discussion in one place. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Dataset groups 187859705 | |
868324949 | https://github.com/pydata/xarray/issues/1092#issuecomment-868324949 | https://api.github.com/repos/pydata/xarray/issues/1092 | MDEyOklzc3VlQ29tbWVudDg2ODMyNDk0OQ== | martinitus 7611856 | 2021-06-25T08:36:03Z | 2021-06-25T08:45:23Z | NONE | Hey Folks,
I stumbled over this discussion having a similar use case as described in some comments above: A However, one point I didn't see in the discussion is the following: Hierarchical structures often force a user to come up with some arbitrary order of hierarchy levels. The classical example is document filing: do you put your health insurance documents under One solution to that is a tagging of documents instead of putting them into a hierarchy. This would give the full flexibility to retrieve any flat Back to the above example, one could think of stuff like: ```python get a flat view (DataSet-like object) on all arrays of tagged that have the 'count' tagds: DataSet(View) = tagged.tag_select("count") bar1 = ds.mean(dim="foo") get a flat view (DataSet-like object) on all arrays of tagged that have the "train and "controlled" tagbar2 = tagged.tag_select("train", "controlled").mean(dim="foo") # order of arguments to Just wanted to add that aspect to the discussion even if it might collide with the hierarchical approach! One side note: If every array in the tagged container has exactly one tag, and tags do not repeat, then the whole thing should be semantically identical to a Regards, Martin |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Dataset groups 187859705 | |
683824965 | https://github.com/pydata/xarray/issues/1092#issuecomment-683824965 | https://api.github.com/repos/pydata/xarray/issues/1092 | MDEyOklzc3VlQ29tbWVudDY4MzgyNDk2NQ== | LunarLanding 4441338 | 2020-08-31T14:45:22Z | 2020-08-31T15:15:20Z | NONE | I did a ctrl-f for zarr in this issue, found nothing, so here's my two cents: it should be possible to write a Datagroup with either zarr or netcdf. I wonder if @emilbiju (posted https://github.com/pydata/xarray/issues/4118 ) has any of that code laying around, could be a starting point. In general, a tree structure to which I can broadcast operations in the same dimensions to different datasets that do not necessary share dimension lengths would solve my use case . This corresponds to bullet point number 3 in https://github.com/pydata/xarray/issues/1092#issuecomment-290159834 . My use case is a set of experiments, that have: the same parameter variables, with different values; the same dimensions with different lengths for their data. The parameters and data would benefit of having a hierarchical naming structure. Currently I build a master dataset containing experiment datasets, with a coordinate for each parameter. Then I map functions over it. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Dataset groups 187859705 | |
571157096 | https://github.com/pydata/xarray/issues/1092#issuecomment-571157096 | https://api.github.com/repos/pydata/xarray/issues/1092 | MDEyOklzc3VlQ29tbWVudDU3MTE1NzA5Ng== | stale[bot] 26384082 | 2020-01-06T14:26:43Z | 2020-01-06T14:26:43Z | NONE | In order to maintain a list of currently relevant issues, we mark issues as stale after a period of inactivity If this issue remains relevant, please comment here or remove the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Dataset groups 187859705 | |
363090775 | https://github.com/pydata/xarray/issues/1092#issuecomment-363090775 | https://api.github.com/repos/pydata/xarray/issues/1092 | MDEyOklzc3VlQ29tbWVudDM2MzA5MDc3NQ== | equaeghe 601177 | 2018-02-05T13:53:55Z | 2018-02-05T13:53:55Z | NONE | I'm late to the discussion and may be repeating some things essentially already said, but I'd still like to add a further voice. @shoyer said on 8 Nov 2016:
If you prepend the paths to all the names (of dimensions, coordinate variables, and variables) and use the resulting strings as names, don't you just get a collection that would fit right in a My use case is data from a single metmast over time. There are various instruments measuring all kinds of variables of which 10-minute statistics are recorded. I use groups to keep an overview. (I use something like @shoyer said on 30 Mar 2017:
I would prefer the former option, as it more clearly shows the hierarchical nature. If also copying the netCDF4-path-separator-convention, then |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Dataset groups 187859705 | |
290256766 | https://github.com/pydata/xarray/issues/1092#issuecomment-290256766 | https://api.github.com/repos/pydata/xarray/issues/1092 | MDEyOklzc3VlQ29tbWVudDI5MDI1Njc2Ng== | benbovy 4160723 | 2017-03-29T23:26:50Z | 2017-03-29T23:26:50Z | MEMBER |
I would be +1 for allowing tuples for data variables names but not for dimensions/coordinates names. It indeed looks like that using tuples for the latter would be a greater source of confusion and would add too much complexity for only little (or no real?) benefit. I'd be fine with raising an error when loading a netCDF4 file which have groups with conflicting dimensions or when assigning an incompatible Dataset as a new group (e.g., For groups that share common dimensions/coordinates with some differences, a data structure built on top of |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Dataset groups 187859705 | |
290250866 | https://github.com/pydata/xarray/issues/1092#issuecomment-290250866 | https://api.github.com/repos/pydata/xarray/issues/1092 | MDEyOklzc3VlQ29tbWVudDI5MDI1MDg2Ng== | shoyer 1217238 | 2017-03-29T22:54:17Z | 2017-03-29T22:54:17Z | MEMBER |
Yes, this is correct. But note that
Yes, this is true. We would possibly want to make another Dataset subclass for the sub-datasets to ensure that their variables are linked to the parent, e.g., But I'm also not convinced this is actually worth the trouble given how easy it is to write NumPy has similar issues, e.g., |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Dataset groups 187859705 | |
290224441 | https://github.com/pydata/xarray/issues/1092#issuecomment-290224441 | https://api.github.com/repos/pydata/xarray/issues/1092 | MDEyOklzc3VlQ29tbWVudDI5MDIyNDQ0MQ== | lamorton 23484003 | 2017-03-29T21:00:42Z | 2017-03-29T21:04:08Z | NONE | @shoyer I see your point about the string manipulation. On the other hand, this is exactly how h5py and netCDF4-python implement the group/subgroup access syntax: just like a filepath. I'm also having thoughts about the attribute access: if For my own understanding, I tried to translate between From netCDF4-python
It appears that the only things special about a A big difference between The As an aside, it seems that ragged arrays are now supported in netCDF4-python:VLen. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Dataset groups 187859705 | |
290165548 | https://github.com/pydata/xarray/issues/1092#issuecomment-290165548 | https://api.github.com/repos/pydata/xarray/issues/1092 | MDEyOklzc3VlQ29tbWVudDI5MDE2NTU0OA== | shoyer 1217238 | 2017-03-29T17:38:03Z | 2017-03-29T17:38:49Z | MEMBER |
Yes, totally agreed, and I've encountered similar cases in my own work. These sort of "ragged" arrays are great use case for groups.
Yes, it's a little confusing because it looks similar to
Yes, it would create a new dataset, which could take ~1 ms. That's slow for inner loops (though we could add caching to help), but plenty fast for interactive use. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Dataset groups 187859705 | |
290159834 | https://github.com/pydata/xarray/issues/1092#issuecomment-290159834 | https://api.github.com/repos/pydata/xarray/issues/1092 | MDEyOklzc3VlQ29tbWVudDI5MDE1OTgzNA== | lamorton 23484003 | 2017-03-29T17:18:23Z | 2017-03-29T17:19:19Z | NONE | @darothen: Hmm, are your coordinate grids identical for each simulation (ie, It might work for my case to convert my 'tags' to indexes for new dimensions (ie, There is still a good reason to have a flexible data model for lumping more heterogeneous collections together under some headings, with the potential for recursion. I suppose my question is, what is the most natural data model & corresponding access syntax? @shoyer: Your approach is quite clever, and 'smells' much better than parsing strings. I do have two quibbles though.
- Accessing via [Edited for formatting] |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Dataset groups 187859705 | |
290142369 | https://github.com/pydata/xarray/issues/1092#issuecomment-290142369 | https://api.github.com/repos/pydata/xarray/issues/1092 | MDEyOklzc3VlQ29tbWVudDI5MDE0MjM2OQ== | shoyer 1217238 | 2017-03-29T16:18:12Z | 2017-03-29T16:18:12Z | MEMBER |
With netCDF4, we could potentially just use groups. Or we could use some sort of naming convention for strings, e.g., joining together together the parts of the tuple with One challenge here is that unless we also let dimensions be group specific, not every netCDF4 file with groups corresponds to a valid xarray Dataset: you can have conflicting sizes on dimensions for netCDF4 files in different groups. In principle, it could be OK to use tuples for dimension names, but we already have lots of logic that distinguishes between single and multiple dimensions by looking for non-strings or tuples. So you would probably have to write How to handle dimensions and coordinate names when assigning groups is clearly one of the important design decisions here. It's obvious that data variables should be grouped but less clear how to handle dimensions/coordinates.
Some sort of further indentation seems natural, possibly with truncation like This is another case where an HTML repr could be powerful, allowing for clearer visual links and potentially interactive expanding/contracting of the tree.
From xarray's perspective, there isn't really a distinction between multiple files and groups in one netCDF file -- it's just a matter of creating a Dataset with data organized in a different way. Presumably we could write helper methods for converting a dimension into a group level (and vice-versa). But it's worth noting that there still limitations to opening large numbers of files in a single dataset, even with groups, because xarray reads all the metadata for every variable into memory at once, and that metadata is copied in every xarray operation. For this reason, you will still probably want a different datastructure (convertible into an xarray.Dataset) when navigating very large datasets like CMIP, which consists of many thousands of files. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Dataset groups 187859705 | |
290133148 | https://github.com/pydata/xarray/issues/1092#issuecomment-290133148 | https://api.github.com/repos/pydata/xarray/issues/1092 | MDEyOklzc3VlQ29tbWVudDI5MDEzMzE0OA== | darothen 4992424 | 2017-03-29T15:47:57Z | 2017-03-29T15:48:17Z | NONE | Ah, thanks for the heads-up @benbovy! I see the difference now, and I agree
both approaches could co-exist. I may play around with building some of
your proposed |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Dataset groups 187859705 | |
290130241 | https://github.com/pydata/xarray/issues/1092#issuecomment-290130241 | https://api.github.com/repos/pydata/xarray/issues/1092 | MDEyOklzc3VlQ29tbWVudDI5MDEzMDI0MQ== | benbovy 4160723 | 2017-03-29T15:38:46Z | 2017-03-29T15:38:46Z | MEMBER | @darothen you might be interested by the discussion we had here, although it doesn't solve anything related to selection across similar Dataset objects. I think that the collection of Both approaches may co-exist, though. I can imagine the case where we have (1) a set of, e.g., grid-search or monte-carlo model runs and (2) for each model run we have diagnostic variables defined in different places on the grid (e.g., nodes, edges...). The tuple-defined groups within a Dataset is useful for 2 and the collection of Dataset objects is useful for 1. As pointed out by @shoyer, such a collection of Dataset objects might be (preferably) implemented outside of xarray. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Dataset groups 187859705 | |
290106782 | https://github.com/pydata/xarray/issues/1092#issuecomment-290106782 | https://api.github.com/repos/pydata/xarray/issues/1092 | MDEyOklzc3VlQ29tbWVudDI5MDEwNjc4Mg== | darothen 4992424 | 2017-03-29T14:26:15Z | 2017-03-29T14:26:15Z | NONE | Would the domain for this just be to simulate the tree-like structure that NetCDF permits, or could it extend to multiple datasets on disk? One of the ideas that we had during the aospy hackathon involved some sort of idiom based on xarray for packing multiple, similar datasets together. For instance, it's very common in climate science to re-run a model multiple times nearly identically, but changing a parameter or boundary condition. So you end up with large archives of data on disk which are identical in shape and metadata, and you want to be able to quickly analyze across them. As an example, I built a helper tool during my dissertation to automate much of this, allowing you to dump your processed output in some sort of directory structure and consistent naming scheme, and then easily ingest what you need for a given analysis. It's actually working great for a much larger, Monte Carlo set of model simulations right now (3 factor levels with 3-5 values at each level, for a total of 1500 years of simulation). My tool works by concatenating each experimental factor as a new dimension, which lets you use xarray's selection tools to perform analyses across the ensemble. You can pre-process things before concatenating too, if the data ends up being too big to fit in memory (e.g. for every simulation in the experiment, compute time-zonal averages before concatenation). Going back to @shoyer's comment, it still seems as though there is room to build some sort of collection of |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Dataset groups 187859705 | |
290065632 | https://github.com/pydata/xarray/issues/1092#issuecomment-290065632 | https://api.github.com/repos/pydata/xarray/issues/1092 | MDEyOklzc3VlQ29tbWVudDI5MDA2NTYzMg== | benbovy 4160723 | 2017-03-29T11:48:12Z | 2017-03-29T11:48:12Z | MEMBER | Just want to say that I'm very enthusiastic about this! Like @lamorton, I also find myself having a lot of variables with names containing the name(s) of their "group(s)". My initial idea was also to keep flat datasets and add some logic to get/set groups, but it wasn't very clear and well explained.
Makes perfect sense! I also find the idea of using tuples very clever! @shoyer do you have an idea on how it would work with serialization to netCDF? We would also have to decide how to display groups in the repr of the flat dataset... @lamorton @shoyer unless you want to open a PR, I'd be willing to start working on this. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Dataset groups 187859705 | |
289923078 | https://github.com/pydata/xarray/issues/1092#issuecomment-289923078 | https://api.github.com/repos/pydata/xarray/issues/1092 | MDEyOklzc3VlQ29tbWVudDI4OTkyMzA3OA== | shoyer 1217238 | 2017-03-28T22:22:31Z | 2017-03-28T22:24:31Z | MEMBER | @lamorton Thanks for explaining the use case here. This makes more sense to me now. I like your idea of groups as syntactic sugar around flat datasets with named keys. With an appropriate naming convention, we might even be able to put this into
|
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Dataset groups 187859705 | |
289916013 | https://github.com/pydata/xarray/issues/1092#issuecomment-289916013 | https://api.github.com/repos/pydata/xarray/issues/1092 | MDEyOklzc3VlQ29tbWVudDI4OTkxNjAxMw== | lamorton 23484003 | 2017-03-28T21:51:30Z | 2017-03-28T21:51:30Z | NONE | One important reason to keep the tree-like structure within a dataset is that it provides some assurance to the recipient of the dataset that all the variables 'belong' in the same coordinate space. Constructing a tree (from a nested dictionary, say) whose leaves are datasets or dataArrays doesn't guarantee that the coordinates/dimensions in all the leaves are compatible, whereas a tree within the dataset does make a guarantee about the leaves. As far as motivation for making trees, I find myself with several dozen variable names such as As far as implementation, the |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Dataset groups 187859705 | |
259390660 | https://github.com/pydata/xarray/issues/1092#issuecomment-259390660 | https://api.github.com/repos/pydata/xarray/issues/1092 | MDEyOklzc3VlQ29tbWVudDI1OTM5MDY2MA== | benbovy 4160723 | 2016-11-09T11:15:01Z | 2016-11-09T11:24:51Z | MEMBER |
Yep once again I haven't thought about all the implications this would have! This would indeed add much complexity at the end. I'll try to follow you suggestion of building another data structure, for example - correct me if it's a wrong approach too - a |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Dataset groups 187859705 | |
259208431 | https://github.com/pydata/xarray/issues/1092#issuecomment-259208431 | https://api.github.com/repos/pydata/xarray/issues/1092 | MDEyOklzc3VlQ29tbWVudDI1OTIwODQzMQ== | rabernat 1197350 | 2016-11-08T17:51:00Z | 2016-11-08T17:51:00Z | MEMBER | This suggestion has some significant overlap with the data store / data discovery discussion from last weekend: https://aospy.hackpad.com/Data-StorageDiscovery-Design-Document-fM6LgfwrJ2K |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Dataset groups 187859705 | |
259206339 | https://github.com/pydata/xarray/issues/1092#issuecomment-259206339 | https://api.github.com/repos/pydata/xarray/issues/1092 | MDEyOklzc3VlQ29tbWVudDI1OTIwNjMzOQ== | shoyer 1217238 | 2016-11-08T17:43:22Z | 2016-11-08T17:43:22Z | MEMBER | I am reluctant to add the additional complexity of groups directly into the I would rather see this living in another data structure built on top of |
{ "total_count": 2, "+1": 2, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
Dataset groups 187859705 |
Advanced export
JSON shape: default, array, newline-delimited, object
CREATE TABLE [issue_comments] ( [html_url] TEXT, [issue_url] TEXT, [id] INTEGER PRIMARY KEY, [node_id] TEXT, [user] INTEGER REFERENCES [users]([id]), [created_at] TEXT, [updated_at] TEXT, [author_association] TEXT, [body] TEXT, [reactions] TEXT, [performed_via_github_app] TEXT, [issue] INTEGER REFERENCES [issues]([id]) ); CREATE INDEX [idx_issue_comments_issue] ON [issue_comments] ([issue]); CREATE INDEX [idx_issue_comments_user] ON [issue_comments] ([user]);
user 9