issues

Table actions

2 rows where state = "closed" and user = 22665917 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: created_at (date), updated_at (date), closed_at (date)

type 1

state 1

  • closed · 2 ✖

repo 1

id node_id number title user state locked assignee milestone comments created_at updated_at ▲ closed_at author_association active_lock_reason draft pull_request body reactions performed_via_github_app state_reason repo type
277538485 MDU6SXNzdWUyNzc1Mzg0ODU= 1745 open_mfdataset() memory error in v0.10 nick-weber 22665917 closed 0     24 2017-11-28T21:08:23Z 2019-01-13T01:51:43Z 2019-01-13T01:51:43Z NONE      

Code Sample

```python import xarray

ncfiles = '/example/path/to/wrf/netcdfs/*' dropvars = ['list', 'of', 'many', 'vars', 'to', 'drop']

dset = xarray.open_mfdataset(ncfiles, drop_variables=dropvars, concat_dim='Time',
autoclose=True, decode_cf=False) ```

Problem description

I am trying to load 73 model (WRF) output files using open_mfdataset(). (Thus, 'Time' is a new dimension). Each netcdf has dimensions {'x' : 405, 'y' : 282, 'z': 37} and roughly 20 variables (excluding the other ~20 in dropvars).

When I run the above code with v0.9.6, it completes in roughly 7 seconds. But with v0.10, it crashes with the following error:

*** Error in `~/anaconda3/bin/python': corrupted size vs. prev_size: 0x0000560e9b6ca7b0 ***

which, as I understand, means I'm exceeding my memory allocation. Any thoughts on what could be the source of this issue?

Output of xr.show_versions()

INSTALLED VERSIONS ------------------ commit: None python: 3.6.3.final.0 python-bits: 64 OS: Linux OS-release: 4.9.0-3-amd64 machine: x86_64 processor: x86_64 byteorder: little LC_ALL: C LANG: C LOCALE: None.None xarray: 0.10.0 pandas: 0.20.3 numpy: 1.13.1 scipy: 0.19.1 netCDF4: 1.2.4 h5netcdf: 0.5.0 Nio: None bottleneck: 1.2.1 cyordereddict: None dask: 0.16.0 matplotlib: 2.0.2 cartopy: None seaborn: 0.8.0 setuptools: 27.2.0 pip: 9.0.1 conda: 4.3.29 pytest: 3.1.3 IPython: 6.1.0 sphinx: 1.6.2 
{
    "url": "https://api.github.com/repos/pydata/xarray/issues/1745/reactions",
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed xarray 13221727 issue
374070147 MDU6SXNzdWUzNzQwNzAxNDc= 2512 to_netcdf() fails because of datetime encoding nick-weber 22665917 closed 0     2 2018-10-25T18:17:47Z 2018-10-27T16:34:54Z 2018-10-27T16:34:54Z NONE      

Simple example:

```python import numpy as np from datetime import datetime, timedelta import xarray

"time" coordinate

dt = datetime(1999, 1, 1) dts = np.array([dt + timedelta(days=x) for x in range(10)]) coords = {'time': dts}

simple float data

data = np.arange(10) vrbls = {'foo': (('time',), data)}

create the Dataset

ds = xarray.Dataset(vrbls, coords)

encode the time coordinate

units = 'days since 1900-01-01' ds.time.encoding['units'] = units

write to netcdf

ds.to_netcdf('test.nc')

```

Problem description

When I run the above, I get the following error when executing the last line: ValueError: unsupported dtype for netCDF4 variable: datetime64[ns]

The documentation indicates that datetime and datetime64 objects are both supported by xarray and should write to netcdf just fine when supplied "units" for encoding (this code fails with or without the encoding lines). Any Idea when is going wrong here?

Output of xr.show_versions()

INSTALLED VERSIONS ------------------ commit: None python: 3.6.2.final.0 python-bits: 64 OS: Linux OS-release: 4.9.0-8-amd64 machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: en_US.UTF-8 xarray: 0.10.9 pandas: 0.20.3 numpy: 1.13.1 scipy: 0.19.1 netCDF4: 1.4.2 h5netcdf: 0.5.0 h5py: 2.8.0 Nio: None zarr: None cftime: 1.0.1 PseudonetCDF: None rasterio: None iris: None bottleneck: 1.2.1 cyordereddict: None dask: 0.16.0 distributed: 1.20.1 matplotlib: 2.1.0 cartopy: None seaborn: 0.8.0 setuptools: 27.2.0 pip: 9.0.1 conda: 4.5.11 pytest: 3.1.3 IPython: 6.1.0 sphinx: 1.6.2 
{
    "url": "https://api.github.com/repos/pydata/xarray/issues/2512/reactions",
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   completed xarray 13221727 issue

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issues] (
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [number] INTEGER,
   [title] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [state] TEXT,
   [locked] INTEGER,
   [assignee] INTEGER REFERENCES [users]([id]),
   [milestone] INTEGER REFERENCES [milestones]([id]),
   [comments] INTEGER,
   [created_at] TEXT,
   [updated_at] TEXT,
   [closed_at] TEXT,
   [author_association] TEXT,
   [active_lock_reason] TEXT,
   [draft] INTEGER,
   [pull_request] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [state_reason] TEXT,
   [repo] INTEGER REFERENCES [repos]([id]),
   [type] TEXT
);
CREATE INDEX [idx_issues_repo]
    ON [issues] ([repo]);
CREATE INDEX [idx_issues_milestone]
    ON [issues] ([milestone]);
CREATE INDEX [idx_issues_assignee]
    ON [issues] ([assignee]);
CREATE INDEX [idx_issues_user]
    ON [issues] ([user]);