issue_comments: 613068333

This data as json

html_url issue_url id node_id user created_at updated_at author_association body reactions performed_via_github_app issue
https://github.com/pydata/xarray/issues/2534#issuecomment-613068333 https://api.github.com/repos/pydata/xarray/issues/2534 613068333 MDEyOklzc3VlQ29tbWVudDYxMzA2ODMzMw== 8419421 2020-04-13T19:57:08Z 2020-04-13T19:57:08Z NONE

Here is another example: import xarray as xr ncdata = xr.open_dataset('https://thredds.ucar.edu/thredds/dodsC/nws/metar/ncdecoded/files/Surface_METAR_20200411_0000.nc') df = ncdata.to_dataframe()

Output is: Traceback (most recent call last): File "bug.py", line 4, in <module> df = ncdata.to_dataframe() File "/home/decker/miniconda3/envs/met212/lib/python3.8/site-packages/xarray/core/dataset.py", line 4399, in to_dataframe return self._to_dataframe(self.dims) File "/home/decker/miniconda3/envs/met212/lib/python3.8/site-packages/xarray/core/dataset.py", line 4385, in _to_dataframe data = [ File "/home/decker/miniconda3/envs/met212/lib/python3.8/site-packages/xarray/core/dataset.py", line 4386, in <listcomp> self._variables[k].set_dims(ordered_dims).values.reshape(-1) MemoryError: Unable to allocate array with shape (117819, 5021) and data type |S64

If I'm doing the math right, xarray is trying to allocate roughly 35 GB even though this NetCDF file is only on the order of 50 MB in size.

Output of xr.show_versions()

Details ``` INSTALLED VERSIONS ------------------ commit: None python: 3.8.1 | packaged by conda-forge | (default, Jan 5 2020, 20:58:18) [GCC 7.3.0] python-bits: 64 OS: Linux OS-release: 5.4.31-1-MANJARO machine: x86_64 processor: byteorder: little LC_ALL: None LANG: en_US.utf8 LOCALE: en_US.UTF-8 libhdf5: 1.10.5 libnetcdf: 4.7.3 xarray: 0.14.1 pandas: 0.25.3 numpy: 1.17.5 scipy: 1.4.1 netCDF4: 1.5.3 pydap: None h5netcdf: None h5py: None Nio: None zarr: None cftime: 1.0.4.2 nc_time_axis: None PseudoNetCDF: None rasterio: None cfgrib: None iris: None bottleneck: None dask: None distributed: None matplotlib: 3.1.2 cartopy: 0.17.0 seaborn: None numbagg: None setuptools: 45.1.0.post20200119 pip: 19.3.1 conda: None pytest: 5.3.4 IPython: 7.11.1 sphinx: None ``` 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   376370028