html_url,issue_url,id,node_id,user,created_at,updated_at,author_association,body,reactions,performed_via_github_app,issue https://github.com/pydata/xarray/issues/1020#issuecomment-832136328,https://api.github.com/repos/pydata/xarray/issues/1020,832136328,MDEyOklzc3VlQ29tbWVudDgzMjEzNjMyOA==,1217238,2021-05-04T18:03:32Z,2021-05-04T18:03:32Z,MEMBER,"@meteoDaniel could you please open thread for discussing your issue? This could be a good use for the GitHub ""Discussions"" tab :) Including a copy of the ""repr"" from printing your dataset would help us give more specific guidance.","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,180080354 https://github.com/pydata/xarray/issues/1020#issuecomment-250615133,https://api.github.com/repos/pydata/xarray/issues/1020,250615133,MDEyOklzc3VlQ29tbWVudDI1MDYxNTEzMw==,1217238,2016-09-29T22:53:39Z,2016-09-29T22:53:39Z,MEMBER,"Looking at your dataset: ``` >>> url ='http://nomads.ncep.noaa.gov:9090/dods/hrrr/hrrr20160801/hrrr_sfc_00z' >>> ds= xarray.open_dataset(url) /Users/shoyer/dev/xarray/xarray/conventions.py:386: RuntimeWarning: Unable to decode time axis into full numpy.datetime64 objects, continuing using dummy netCDF4.datetime objects instead, reason: dates out of range result = decode_cf_datetime(example_value, units, calendar) >>> ds Dimensions: (lat: 1155, lev: 5, lon: 2503, time: 19) Coordinates: * time (time) object 2016-09-28T12:00:00 2016-09-28T13:00:00 ... * lev (lev) float64 1e+03 925.0 850.0 700.0 500.0 * lat (lat) float64 21.14 21.17 21.2 21.22 21.25 21.28 21.3 ... * lon (lon) float64 -134.1 -134.1 -134.0 -134.0 -134.0 ... Data variables: dptprs (time, lev, lat, lon) float64 ... no4lftx180_0mb (time, lat, lon) float64 ... apcpsfc (time, lat, lon) float64 ... asnowsfc (time, lat, lon) float64 ... bgrunsfc (time, lat, lon) float64 ... capesfc (time, lat, lon) float64 ... cape180_0mb (time, lat, lon) float64 ... cape90_0mb (time, lat, lon) float64 ... cape255_0mb (time, lat, lon) float64 ... cfrzrsfc (time, lat, lon) float64 ... cicepsfc (time, lat, lon) float64 ... cinsfc (time, lat, lon) float64 ... cin180_0mb (time, lat, lon) float64 ... cin90_0mb (time, lat, lon) float64 ... cin255_0mb (time, lat, lon) float64 ... cnwatsfc (time, lat, lon) float64 ... cpofpsfc (time, lat, lon) float64 ... crainsfc (time, lat, lon) float64 ... csnowsfc (time, lat, lon) float64 ... dlwrfsfc (time, lat, lon) float64 ... dpt2m (time, lat, lon) float64 ... dswrfsfc (time, lat, lon) float64 ... dzdtsg500_800 (time, lat, lon) float64 ... fricvsfc (time, lat, lon) float64 ... frozrsfc (time, lat, lon) float64 ... gfluxsfc (time, lat, lon) float64 ... gustsfc (time, lat, lon) float64 ... hcdchcll (time, lat, lon) float64 ... hgtsfc (time, lat, lon) float64 ... hgt500mb (time, lat, lon) float64 ... hgt700mb (time, lat, lon) float64 ... hgt850mb (time, lat, lon) float64 ... hgt1000mb (time, lat, lon) float64 ... hgtclb (time, lat, lon) float64 ... hgt263_k (time, lat, lon) float64 ... hgt253_k (time, lat, lon) float64 ... hgttop0c (time, lat, lon) float64 ... hgtceil (time, lat, lon) float64 ... hgteql (time, lat, lon) float64 ... hgtclt (time, lat, lon) float64 ... hgt0c (time, lat, lon) float64 ... hgtl5 (time, lat, lon) float64 ... hlcy3000_0m (time, lat, lon) float64 ... hlcy1000_0m (time, lat, lon) float64 ... hpblsfc (time, lat, lon) float64 ... icecsfc (time, lat, lon) float64 ... landsfc (time, lat, lon) float64 ... lcdclcll (time, lat, lon) float64 ... lftxl100_100 (time, lat, lon) float64 ... lhtflsfc (time, lat, lon) float64 ... ltngclm (time, lat, lon) float64 ... maxdvv400_1000mb (time, lat, lon) float64 ... maxref1000m (time, lat, lon) float64 ... maxuvv400_1000mb (time, lat, lon) float64 ... mcdcmcll (time, lat, lon) float64 ... mslmamsl (time, lat, lon) float64 ... mstav0cm (time, lat, lon) float64 ... mxuphl5000_2000m (time, lat, lon) float64 ... plpl255_0mb (time, lat, lon) float64 ... pot2m (time, lat, lon) float64 ... pratesfc (time, lat, lon) float64 ... pressfc (time, lat, lon) float64 ... presclb (time, lat, lon) float64 ... prestop0c (time, lat, lon) float64 ... presclt (time, lat, lon) float64 ... pres0c (time, lat, lon) float64 ... pwatclm (time, lat, lon) float64 ... refcclm (time, lat, lon) float64 ... refd1000m (time, lat, lon) float64 ... refd4000m (time, lat, lon) float64 ... refd263_k (time, lat, lon) float64 ... retopclt (time, lat, lon) float64 ... rh2m (time, lat, lon) float64 ... rhtop0c (time, lat, lon) float64 ... rh0c (time, lat, lon) float64 ... rhpwclm (time, lat, lon) float64 ... sbt113toa (time, lat, lon) float64 ... sbt114toa (time, lat, lon) float64 ... sbt123toa (time, lat, lon) float64 ... sbt124toa (time, lat, lon) float64 ... sfcrsfc (time, lat, lon) float64 ... shtflsfc (time, lat, lon) float64 ... snodsfc (time, lat, lon) float64 ... snowcsfc (time, lat, lon) float64 ... spfh2m (time, lat, lon) float64 ... ssrunsfc (time, lat, lon) float64 ... tcdcclm (time, lat, lon) float64 ... tcolgclm (time, lat, lon) float64 ... tmpsfc (time, lat, lon) float64 ... tmpprs (time, lev, lat, lon) float64 ... tmp2m (time, lat, lon) float64 ... ugrdprs (time, lev, lat, lon) float64 ... ugrd80m (time, lat, lon) float64 ... ugrd10m (time, lat, lon) float64 ... ulwrfsfc (time, lat, lon) float64 ... ulwrftoa (time, lat, lon) float64 ... ustm0_6000m (time, lat, lon) float64 ... uswrfsfc (time, lat, lon) float64 ... vbdsfsfc (time, lat, lon) float64 ... vddsfsfc (time, lat, lon) float64 ... vgrdprs (time, lev, lat, lon) float64 ... vgrd80m (time, lat, lon) float64 ... vgrd10m (time, lat, lon) float64 ... vgtypsfc (time, lat, lon) float64 ... vilclm (time, lat, lon) float64 ... vissfc (time, lat, lon) float64 ... vstm0_6000m (time, lat, lon) float64 ... vucsh0_1000m (time, lat, lon) float64 ... vucsh0_6000m (time, lat, lon) float64 ... vvcsh0_1000m (time, lat, lon) float64 ... vvcsh0_6000m (time, lat, lon) float64 ... weasdaccsfc (time, lat, lon) float64 ... weasdsfc (time, lat, lon) float64 ... wind10m (time, lat, lon) float64 ... Attributes: title: High Resolution Rapid Refresh 3km 2D Surface forecast from 12Z28sep2016, downloaded Sep 28 13:19 UTC Conventions: COARDS GrADS dataType: Grid history: Thu Sep 29 17:52:17 UTC 2016 : imported by GrADS Data Server 2.0 >>> ds.nbytes / 1e9 57.125497856 ``` So it's at least 57 GB when decoded as float64. This is probably more RAM than you have on your machine. But also, when xarray writes a dataframe every variable first gets expanded to use all dimensions. So this is something like 5 \* 57 GB in memory, and pandas probably needs a memory copy to create the DataFrame, so this probably needs at least 500 GB. You'll have better luck subsetting the dataset first. ","{""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,180080354