issue_comments: 569344638
This data as json
html_url | issue_url | id | node_id | user | created_at | updated_at | author_association | body | reactions | performed_via_github_app | issue |
---|---|---|---|---|---|---|---|---|---|---|---|
https://github.com/pydata/xarray/issues/2404#issuecomment-569344638 | https://api.github.com/repos/pydata/xarray/issues/2404 | 569344638 | MDEyOklzc3VlQ29tbWVudDU2OTM0NDYzOA== | 7776656 | 2019-12-27T20:58:53Z | 2019-12-27T21:01:21Z | NONE | hi, I have hit this (or a similar issue) in my xarray work, and I think I was able to find at least one MWE. In this case coordinates of a dataset have been built with underlying type numpy.str_. In my example below, I construct a pandas DataFrame, then convert it to a Dataset. The dtypes of the offending coordinate is displayed as object, but on writing to netcdf, I get a ValueError. ``` import pandas as pd import numpy as np import xarray as xr # xr.__version --> 0.13.0 import os import itertools make a multi index (where one level is np.str_ type)x = list(np.array([np.str_('idx_%i') %i for i in range(1,11)], dtype=np.str_)) y = list(np.arange(10)) combo = list(itertools.product(x, y)) x,y = zip(*combo) the below is an odd way to construct a DataFrame, but the np.str_ type is preserved if done this waydata_df = np.random.randn(len(x)) df = pd.DataFrame(data=data_df, columns=['test']) df['x'] = x df['y'] = y df = df.set_index(['x', 'y']) ds = xr.Dataset.from_dataframe(df) ``` You can see from the below that the underlying type is np.str_
Writing to netcdf gives the below error: ``` ds.to_netcdf('/tmp/netcdf_repro.cdf', engine='h5netcdf')ValueError Traceback (most recent call last) <ipython-input-19-bca194d02da3> in <module> ----> 1 ds.to_netcdf('/tmp/netcdf_repro.cdf', engine='h5netcdf') ~/.conda/envs/apac/lib/python3.7/site-packages/xarray/core/dataset.py in to_netcdf(self, path, mode, format, group, engine, encoding, unlimited_dims, compute, invalid_netcdf) 1538 unlimited_dims=unlimited_dims, 1539 compute=compute, -> 1540 invalid_netcdf=invalid_netcdf, 1541 ) 1542 ~/.conda/envs/apac/lib/python3.7/site-packages/xarray/backends/api.py in to_netcdf(dataset, path_or_file, mode, format, group, engine, encoding, unlimited_dims, compute, multifile, invalid_netcdf) 1072 # to be parallelized with dask 1073 dump_to_store( -> 1074 dataset, store, writer, encoding=encoding, unlimited_dims=unlimited_dims 1075 ) 1076 if autoclose: ~/.conda/envs/apac/lib/python3.7/site-packages/xarray/backends/api.py in dump_to_store(dataset, store, writer, encoder, encoding, unlimited_dims) 1118 variables, attrs = encoder(variables, attrs) 1119 -> 1120 store.store(variables, attrs, check_encoding, writer, unlimited_dims=unlimited_dims) 1121 1122 ~/.conda/envs/apac/lib/python3.7/site-packages/xarray/backends/common.py in store(self, variables, attributes, check_encoding_set, writer, unlimited_dims) 301 self.set_dimensions(variables, unlimited_dims=unlimited_dims) 302 self.set_variables( --> 303 variables, check_encoding_set, writer, unlimited_dims=unlimited_dims 304 ) 305 ~/.conda/envs/apac/lib/python3.7/site-packages/xarray/backends/common.py in set_variables(self, variables, check_encoding_set, writer, unlimited_dims) 339 check = vn in check_encoding_set 340 target, source = self.prepare_variable( --> 341 name, v, check, unlimited_dims=unlimited_dims 342 ) 343 ~/.conda/envs/apac/lib/python3.7/site-packages/xarray/backends/h5netcdf_.py in prepare_variable(self, name, variable, check_encoding, unlimited_dims) 190 191 attrs = variable.attrs.copy() --> 192 dtype = _get_datatype(variable, raise_on_invalid_encoding=check_encoding) 193 194 fillvalue = attrs.pop("_FillValue", None) ~/.conda/envs/apac/lib/python3.7/site-packages/xarray/backends/netCDF4_.py in _get_datatype(var, nc_format, raise_on_invalid_encoding) 118 def _get_datatype(var, nc_format="NETCDF4", raise_on_invalid_encoding=False): 119 if nc_format == "NETCDF4": --> 120 datatype = _nc4_dtype(var) 121 else: 122 if "dtype" in var.encoding: ~/.conda/envs/apac/lib/python3.7/site-packages/xarray/backends/netCDF4_.py in _nc4_dtype(var) 141 dtype = var.dtype 142 else: --> 143 raise ValueError("unsupported dtype for netCDF4 variable: {}".format(var.dtype)) 144 return dtype 145 ValueError: unsupported dtype for netCDF4 variable: object ``` I believe the issue is in: ``` xarray/conventions.py def _infer_dtype(array, name=None): """Given an object array with no missing values, infer its dtype from its first element """ if array.dtype.kind != "O": raise TypeError("infer_type must be called on a dtype=object array")
``` Because we have a np.str_ which fails the comparison to (bytes, str). It seems that perhaps the failure is in letting coordinates have a np.str_ type in the first place without being coerced to str. I have not contributed to the xarray library yet, but use it frequently and really love it. If, with a bit of guidance, this is something you think I could help fix, I am happy to give it a go. |
{ "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 } |
357729048 |