home / github

Menu
  • Search all tables
  • GraphQL API

issue_comments

Table actions
  • GraphQL API for issue_comments

3 rows where user = 5497186 sorted by updated_at descending

✖
✖

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: issue_url, created_at (date), updated_at (date)

issue 2

  • save/load DataArray to numpy npz functions 2
  • pandas date_range as index causes TypeError on repr 1

user 1

  • jonathanstrong · 3 ✖

author_association 1

  • NONE 3
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
187226020 https://github.com/pydata/xarray/issues/768#issuecomment-187226020 https://api.github.com/repos/pydata/xarray/issues/768 MDEyOklzc3VlQ29tbWVudDE4NzIyNjAyMA== jonathanstrong 5497186 2016-02-22T15:17:29Z 2016-02-22T15:17:29Z NONE

hey,

So - after using netcdf for a few days, definitely not looking back. This is great. By way of background, I am building a way to integrate storage of arbitrary arrays into an otherwise highly-structured schema. After using postgresql arrays flamed out (too slow, even at the raw SQL level), I moved on to saving the file path in my schema. I thought it would be sensible to keep everything in pure ndarray for simplicity. After trying out netcdf, I bit the bullet and wrote constructors for numpy, pandas and xarray types, and it's working great.

Looking back, I actually think the documentation could use some work to help people like me, who haven't used netcdf, realize it's great.

If you look at the docs, it starts with pickle, which for me is kind of a red flag since from my experience pickle is the world's most flaky persistence method (always has dependency issues). Then at netcdf you start with:

"Currently, the only disk based serialization format that xarray directly supports is netCDF."

I read this and think, ok so the IO is not really there yet. It's like an apology there aren't more choices.

"netCDF is a file format for fully self-described datasets that is widely used in the geosciences and supported on almost all platforms."

Hmm...geosciences...who knows what those people are doing? I'm also generally suspicious of academics when it comes to code.

"We use netCDF because xarray was based on the netCDF data model, so netCDF files on disk directly correspond to Dataset objects."

Ok, so it's easy for you. What about me?

I've been a bit over the top but you can see how someone who doesn't use netcdf might read this and think they need to write their own IO functions.

If it were me, I would start off selling how great this format is for xarray. Like, "netcdf is a blazing-fast, binary data format that allows transparent, self-describing persistence with zero of the dependency issues you get with pickle or other formats. It allows xarray Datasets to be saved intact and even used in out-of-core computations for larger-than-memory arrays."

Or something like that.

Finally, regarding DataArray not having it's own method to save: I think this is a deficiency that is easily solved. For me, getting into this library I started with just DataArrays. Now that I am using Datasets I can see how they are pretty dang powerful. But at first the simpler DataArrays were all I was using, and they had no direct IO.

To solve this, you could create a "magic" string for DataArrays. On save, to_netcdf converts the da to a Dataset with the magic key. On load, the load function recognizes the magic string and breaks out that DataArray to return it specifically.

I think that would be a quick, relatively painless way to give DataArrays equal footing with Datasets.

Anyway - my two cents. I am a huge fan of this library and happy to chip in regarding any of the above if desired. Thanks for your hard work on it.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  save/load DataArray to numpy npz functions 134376872
185817180 https://github.com/pydata/xarray/issues/768#issuecomment-185817180 https://api.github.com/repos/pydata/xarray/issues/768 MDEyOklzc3VlQ29tbWVudDE4NTgxNzE4MA== jonathanstrong 5497186 2016-02-18T17:05:45Z 2016-02-18T17:34:53Z NONE

I hadn't, for a number of reasons. First, I've used csv, hdf, sql, json, yaml and other formats but never came across netcdf until using this library as someone who isn't working in the physical sciences. Second, the documentation on netcdf is fairly dense. Third, didn't want to deal with installing the library.

I just did use it and seems like it is great for Datasets. As far as I can tell there is no way to save DataArrays directly, though?

Finally, would note that pandas has io methods for csv, excel, hdf, sql, json, msgpack, html, gbq, stata, "clipboard", and pickle. I think it's a strength to offer more choices.

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  save/load DataArray to numpy npz functions 134376872
170949009 https://github.com/pydata/xarray/issues/716#issuecomment-170949009 https://api.github.com/repos/pydata/xarray/issues/716 MDEyOklzc3VlQ29tbWVudDE3MDk0OTAwOQ== jonathanstrong 5497186 2016-01-12T15:35:18Z 2016-01-12T15:35:18Z NONE

sorry, forgot to mention, calling to_list() on the datetime_index allows it to work.

```

da = xray.DataArray(data=a, dims=['example', 'channel', 'row', 'column'], coords={'example': datetime_index.tolist(), 'channel': np.arange(1), 'row': np.arange(28), 'column': np.arange(28)}) print da <xray.DataArray (example: 32, channel: 1, row: 28, column: 28)> array([[[[ 0.2415703 , 0.49185786, 0.76974749, ..., 0.24266482, 0.51117921, 0.23183344], [ 0.5396343 , 0.7049137 , 0.46284154, ..., 0.14457754, 0.5515094 , 0.06407724], [ 0.6545502 , 0.79442538, 0.61539481, ..., 0.08185602, 0.07505339, 0.71784597], ..., [ 0.74341592, 0.58249646, 0.9264281 , ..., 0.4105471 , 0.18713031, 0.38448372], [ 0.10294858, 0.08118397, 0.03903754, ..., 0.53996466, 0.06814973, 0.37088457], [ 0.79747167, 0.71836927, 0.3760608 , ..., 0.00207118, 0.23272135, 0.39850314]]],

   [[[ 0.54520578,  0.82793812,  0.27934314, ...,  0.47675878,
       0.35750073,  0.81324566],
     [ 0.34234289,  0.26131611,  0.1741479 , ...,  0.23433841,
       0.70296677,  0.60834615],
     [ 0.49224854,  0.35010889,  0.72760039, ...,  0.9655428 ,
       0.28976407,  0.94971166],
     ..., 
     [ 0.52062892,  0.62694481,  0.74069292, ...,  0.14967457,
       0.57194567,  0.20518411],
     [ 0.30441499,  0.29161604,  0.49102084, ...,  0.81442707,
       0.18248721,  0.88117849],
     [ 0.91370959,  0.93829229,  0.79401123, ...,  0.26210045,
       0.00614292,  0.75532125]]],


   [[[ 0.08424663,  0.51539268,  0.15021821, ...,  0.6983625 ,
       0.2479839 ,  0.45416423],
     [ 0.41073492,  0.30244808,  0.23658923, ...,  0.64526482,
       0.39558868,  0.77243522],
     [ 0.91946504,  0.14702082,  0.44866339, ...,  0.88202992,
       0.7439115 ,  0.69984251],
     ..., 
     [ 0.33138424,  0.65155252,  0.512169  , ...,  0.12812792,
       0.11201082,  0.52092288],
     [ 0.40670197,  0.04778641,  0.8280053 , ...,  0.96303147,
       0.00902775,  0.27478836],
     [ 0.63061898,  0.65237575,  0.44703296, ...,  0.26827483,
       0.62652144,  0.33875215]]],


   ..., 
   [[[ 0.02181055,  0.28960575,  0.20066007, ...,  0.96958491,
       0.48909334,  0.53350904],
     [ 0.32687157,  0.87064061,  0.75603015, ...,  0.51215617,
       0.22192232,  0.53660243],
     [ 0.53602119,  0.78761882,  0.61880669, ...,  0.40650488,
       0.21773892,  0.43152135],
     ..., 
     [ 0.6085028 ,  0.88383046,  0.72063395, ...,  0.62093148,
       0.28592564,  0.00169664],
     [ 0.47644684,  0.08029548,  0.2617365 , ...,  0.71717209,
       0.82374822,  0.67171277],
     [ 0.91905357,  0.54944082,  0.83455056, ...,  0.50262155,
       0.20663823,  0.0462887 ]]],


   [[[ 0.38780989,  0.28305091,  0.32381702, ...,  0.55012613,
       0.91371051,  0.18266697],
     [ 0.27822897,  0.19429443,  0.56619312, ...,  0.43792798,
       0.28703397,  0.48266462],
     [ 0.31136054,  0.81075836,  0.63331665, ...,  0.71262749,
       0.2267146 ,  0.81515838],
     ..., 
     [ 0.98302595,  0.85393531,  0.43698721, ...,  0.99933458,
       0.80326649,  0.87700763],
     [ 0.91663997,  0.30024646,  0.5544715 , ...,  0.5316829 ,
       0.2144375 ,  0.90622478],
     [ 0.46303225,  0.47611954,  0.17224521, ...,  0.6319952 ,
       0.22626302,  0.23822273]]],


   [[[ 0.20930888,  0.98717439,  0.07440003, ...,  0.93006893,
       0.87313465,  0.55199699],
     [ 0.56638815,  0.6957562 ,  0.14855453, ...,  0.5018839 ,
       0.88542717,  0.7561218 ],
     [ 0.46551802,  0.96133623,  0.94580378, ...,  0.09468893,
       0.30283214,  0.42860321],
     ..., 
     [ 0.14302203,  0.23090552,  0.84507949, ...,  0.89370003,
       0.23141406,  0.94566799],
     [ 0.75805896,  0.9181345 ,  0.63149841, ...,  0.18180786,
       0.94189951,  0.28630845],
     [ 0.02315291,  0.90541336,  0.86396254, ...,  0.36885415,
       0.57690491,  0.60685422]]]])

Coordinates: * column (column) int64 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 ... * example (example) datetime64[ns] 2016-01-12T15:31:25.870931 ... * channel (channel) int64 0 * row (row) int64 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 ... ```

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
  pandas date_range as index causes TypeError on repr 126205116

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options:

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);
Powered by Datasette · Queries took 13.544ms · About: xarray-datasette
  • Sort ascending
  • Sort descending
  • Facet by this
  • Hide this column
  • Show all columns
  • Show not-blank rows