id,node_id,number,title,user,state,locked,assignee,milestone,comments,created_at,updated_at,closed_at,author_association,active_lock_reason,draft,pull_request,body,reactions,performed_via_github_app,state_reason,repo,type
368833116,MDU6SXNzdWUzNjg4MzMxMTY=,2478,masked_array write/read differences between xarray and netCDF4,16655388,closed,0,,,3,2018-10-10T20:12:19Z,2023-09-13T12:41:03Z,2023-09-13T12:41:02Z,NONE,,,,"Here is code used to read/write a masked_array with the netCDF4 and xarray modules.
As seen if you run the code, for 3 cases the masked_value is read as a np.nan. However, for the netcdf file written by netCDF4 and read by xarray, the masked_value is the default _FillValue of 9.96920997e+36.
I wonder if this is expected or if I am doing something wrong.
```python
import xarray as xr
import netCDF4 as nc
import numpy as np
import os
data = np.ma.array([1.,2.], mask = [True, False])
# create file with netcdf$
nc_file = 'ncfile.nc'
if os.path.exists(nc_file): os.remove(nc_file)
ds = nc.Dataset(nc_file, 'w')
ds.createDimension('dim1', 2)
var = ds.createVariable('data', 'f8', dimensions = ('dim1'))
var[:] = data
ds.close()
# create file with xarray
da = xr.DataArray(data, name = 'data', dims = {'dim1':2})
nc_file = 'xrfile.nc'
if os.path.exists(nc_file): os.remove(nc_file)
da.to_netcdf(nc_file, 'w')
da.close()
print('original data: {}'.format(data))
da = xr.open_dataset('ncfile.nc').data
print('data from nc read by xr: {}'.format(da.values))
da = xr.open_dataset('xrfile.nc').data
print('data from xr read by xr: {}'.format(da.values))
data = nc.Dataset('ncfile.nc').variables['data'][:]
print('data from nc read by nc: {}'.format(da.values))
data = nc.Dataset('xrfile.nc').variables['data'][:]
print('data from xr read by nc: {}'.format(da.values))
print('done')
```
Here is the output I get:
```
original data: [-- 2.0]
data from nc read by xr: [9.96920997e+36 2.00000000e+00]
data from xr read by xr: [nan 2.]
data from nc read by nc: [nan 2.]
data from xr read by nc: [nan 2.]
done
```
#### Output of ``xr.show_versions()``
# Paste the output here xr.show_versions() here
INSTALLED VERSIONS
------------------
commit: None
python: 3.6.6.final.0
python-bits: 64
OS: Darwin
OS-release: 17.7.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: fr_CA.UTF-8
LOCALE: fr_CA.UTF-8
xarray: 0.10.8
pandas: 0.23.4
numpy: 1.15.1
scipy: 1.1.0
netCDF4: 1.4.1
h5netcdf: None
h5py: None
Nio: None
zarr: None
bottleneck: 1.2.1
cyordereddict: None
dask: 0.19.2
distributed: None
matplotlib: None
cartopy: None
seaborn: None
setuptools: 40.2.0
pip: 18.0
conda: None
pytest: 3.8.0
IPython: 7.0.1
sphinx: None
","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/2478/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed,13221727,issue
905476569,MDU6SXNzdWU5MDU0NzY1Njk=,5396,bug with to_netcdf when reading dataset via THREDDS,16655388,open,0,,,1,2021-05-28T14:08:12Z,2022-02-03T11:49:29Z,,NONE,,,,"
I am trying to extract data for a specific station from data observations accessed via THREDDS. At first I got an error but a colleague found a way to make it work. In the code below it works if `APPLY_PATCH=True` but it crashes if `APPLY_PATCH=False`.
I do not know if this behaviour is expected and/or linked to the way the data is 'served' via THREDDS but I would never have found the way around it by myself.
```
import xarray as xr
CLEF_STATION = ""MONTREAL""
NOM_VAR = 'prsn'
APPLY_PATCH = False
url_pavics = ""http://pavics.ouranos.ca:8083/twitcher/ows/proxy/thredds/dodsC""
url_donnees_homo = ""/"".join((url_pavics, ""datasets/station_obs/ECCC_AHCCD_gen2_precipitation.ncml""))
ds = xr.open_dataset(url_donnees_homo)
# data for specific station(s)
dsr = ds.sel(station=ds.station_name.str.contains(CLEF_STATION))
# apply patch
if APPLY_PATCH:
for var in dsr.variables:
if dsr[var].dtype == 'O':
dsr[var].encoding = {}
# save to a netcdf file
dsr.to_netcdf(""out.nc"", format=""NETCDF4_CLASSIC"")
```
**What happened**:
When running this I get the following error
```
runfile('/Users/sbiner/cloud/OneDrive - Ouranos inc/job/stagiaires_etudiants/2021_tempete_neige/code/issue_xaray.py', wdir='/Users/sbiner/cloud/OneDrive - Ouranos inc/job/stagiaires_etudiants/2021_tempete_neige/code')
Traceback (most recent call last):
File ""src/netCDF4/_netCDF4.pyx"", line 4931, in netCDF4._netCDF4.Variable.__setitem__
ValueError: cannot reshape array of size 7 into shape (1,1)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File ""/Users/sbiner/cloud/OneDrive - Ouranos inc/job/stagiaires_etudiants/2021_tempete_neige/code/issue_xaray.py"", line 14, in
dsr.to_netcdf(""out.nc"", format=""NETCDF4_CLASSIC"")
File ""/Users/sbiner/anaconda3/envs/py38/lib/python3.8/site-packages/xarray/core/dataset.py"", line 1689, in to_netcdf
return to_netcdf(
File ""/Users/sbiner/anaconda3/envs/py38/lib/python3.8/site-packages/xarray/backends/api.py"", line 1107, in to_netcdf
dump_to_store(
File ""/Users/sbiner/anaconda3/envs/py38/lib/python3.8/site-packages/xarray/backends/api.py"", line 1154, in dump_to_store
store.store(variables, attrs, check_encoding, writer, unlimited_dims=unlimited_dims)
File ""/Users/sbiner/anaconda3/envs/py38/lib/python3.8/site-packages/xarray/backends/common.py"", line 255, in store
self.set_variables(
File ""/Users/sbiner/anaconda3/envs/py38/lib/python3.8/site-packages/xarray/backends/common.py"", line 297, in set_variables
writer.add(source, target)
File ""/Users/sbiner/anaconda3/envs/py38/lib/python3.8/site-packages/xarray/backends/common.py"", line 146, in add
target[...] = source
File ""/Users/sbiner/anaconda3/envs/py38/lib/python3.8/site-packages/xarray/backends/netCDF4_.py"", line 64, in __setitem__
data[key] = value
File ""src/netCDF4/_netCDF4.pyx"", line 4933, in netCDF4._netCDF4.Variable.__setitem__
File ""<__array_function__ internals>"", line 5, in broadcast_to
File ""/Users/sbiner/anaconda3/envs/py38/lib/python3.8/site-packages/numpy/lib/stride_tricks.py"", line 411, in broadcast_to
return _broadcast_to(array, shape, subok=subok, readonly=True)
File ""/Users/sbiner/anaconda3/envs/py38/lib/python3.8/site-packages/numpy/lib/stride_tricks.py"", line 348, in _broadcast_to
it = np.nditer(
ValueError: operands could not be broadcast together with remapped shapes [original->remapped]: (1,7) and requested shape (1,1)
```
**What you expected to happen**:
I expect a netcdf file to be written
**Environment**:
Output of xr.show_versions()
INSTALLED VERSIONS
------------------
commit: None
python: 3.8.8 (default, Apr 13 2021, 12:59:45)
[Clang 10.0.0 ]
python-bits: 64
OS: Darwin
OS-release: 19.6.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: fr_CA.UTF-8
LANG: fr_CA.UTF-8
LOCALE: fr_CA.UTF-8
libhdf5: 1.10.6
libnetcdf: 4.6.1
xarray: 0.17.0
pandas: 1.2.4
numpy: 1.20.1
scipy: 1.6.2
netCDF4: 1.5.6
pydap: None
h5netcdf: None
h5py: None
Nio: None
zarr: None
cftime: 1.4.1
nc_time_axis: None
PseudoNetCDF: None
rasterio: None
cfgrib: None
iris: None
bottleneck: None
dask: 2021.04.0
distributed: 2021.04.0
matplotlib: 3.3.4
cartopy: 0.18.0
seaborn: 0.11.1
numbagg: None
pint: None
setuptools: 52.0.0.post20210125
pip: 20.3.3
conda: None
pytest: None
IPython: 7.22.0
sphinx: 4.0.1
","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/5396/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,issue
363299007,MDU6SXNzdWUzNjMyOTkwMDc=,2436,"save ""encoding"" when using open_mfdataset",16655388,open,0,,,15,2018-09-24T20:02:49Z,2021-07-28T10:35:42Z,,NONE,,,,"I like the automatic decoding of the time variable when reading netcdf files but I often need to keep the calendar attribute of the time variable.
Could it be possible to keep those attributes in the DataSet/DataArray return by open_dataset?
","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/2436/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,,13221727,issue
363326726,MDU6SXNzdWUzNjMzMjY3MjY=,2437,xarray potential inconstistencies with cftime,16655388,closed,0,,,16,2018-09-24T21:25:46Z,2021-06-22T17:01:35Z,2019-02-08T15:05:38Z,NONE,,,,"I am trying to use xarray with different types of calendars. I made a few tests and wonder if somebody can help me make sense of the results. In my test, I generate a DataArray `da` time series with a 365_day calendar using cftime. I then write that DataArray in a netCDF file and read it in another DataArray `da2`
#### Code Sample, a copy-pastable example if possible
```python
import xarray as xr
import cftime
import numpy as np
# generate data for 365_days calendar
#
units = 'days since 2000-01-01 00:00'
time_365 = cftime.num2date(np.arange(0, 10 * 365), units, '365_day')
da = xr.DataArray(np.arange(time_365.size), coords = [time_365], dims = 'time', name = 'data')
# write dataArray in netcdf and read it in new DataArray
da.to_netcdf('data_365.nc', 'w')
da2 = xr.open_dataset('data_365.nc').data
# try resample da
try:
mean = da.resample(time='Y').mean()
print(mean.values)
except TypeError:
print('got TypeError for da')
# try resample da2
mean = da2.resample(time = 'Y').mean()
print (mean.values)
```
### Problem description
As seen in the code the resampledoes not work for `da` while it does for `da2`. The problem is related to the the type of `da.time` which is `cftime.DatetimeNoLeap` while `da2.time` is a `datetime64`. I thought that xarray is using cftime to make the conversion from time numerical values to dates but it looks to me as if it is not the case.
I wonder if this makes sense or if it is something that should eventually be corrected.
INSTALLED VERSIONS
In [6]: print (cftime.__version__)
1.0.1
------------------
commit: None
python: 3.6.5.final.0
python-bits: 64
OS: Darwin
OS-release: 17.7.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: fr_CA.UTF-8
LOCALE: fr_CA.UTF-8
xarray: 0.10.8
pandas: 0.23.0
numpy: 1.14.3
scipy: 1.1.0
netCDF4: 1.4.1
h5netcdf: None
h5py: 2.7.1
Nio: None
zarr: None
bottleneck: 1.2.1
cyordereddict: None
dask: 0.17.5
distributed: 1.21.8
matplotlib: 2.2.2
cartopy: None
seaborn: 0.8.1
setuptools: 39.1.0
pip: 10.0.1
conda: 4.5.11
pytest: 3.5.1
IPython: 6.4.0
sphinx: 1.7.4
","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/2437/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed,13221727,issue
326553877,MDU6SXNzdWUzMjY1NTM4Nzc=,2187,open_dataset crash with long filenames,16655388,closed,0,,,2,2018-05-25T14:47:31Z,2018-05-29T14:43:50Z,2018-05-29T14:42:35Z,NONE,,,,"#### Code Sample
```python
import xarray as xr
import shutil
import numpy as np
# create netcdf file
data = np.random.rand(4, 3)
foo = xr.DataArray(data)
foo.to_netcdf('test.nc')
f_nc = 'a.nc'
shutil.copy('test.nc', f_nc)
while 1:
print '{:05n} characteres'.format(len(f_nc))
ds1 = xr.open_dataset(f_nc)
ds1.close()
nf_nc = 'a' + f_nc
shutil.move(f_nc, nf_nc)
f_nc = nf_nc
if len(f_nc) == 100:
break
```
#### Problem description
On my linux machine (CentOS) this code crashes (memory corrruption) when the filename length hits 32 characters.
On my OSX machine it is fine until 255 character and stops with an IOError
#### Output of ``xr.show_versions()``
# Paste the output here xr.show_versions() here
INSTALLED VERSIONS
------------------
commit: None
python: 2.7.11.final.0
python-bits: 64
OS: Linux
OS-release: 3.10.0-514.2.2.el7.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_CA.UTF-8
LOCALE: None.None
xarray: 0.10.4
pandas: 0.22.0
numpy: 1.14.2
scipy: 0.16.1
netCDF4: 1.2.2
h5netcdf: None
h5py: 2.5.0
Nio: None
zarr: None
bottleneck: None
cyordereddict: None
dask: 0.17.2
distributed: None
matplotlib: 1.5.0
cartopy: 0.13.1
seaborn: 0.8.1
setuptools: 19.2
pip: 10.0.1
conda: None
pytest: None
IPython: 4.0.1
sphinx: 1.7.2
","{""url"": ""https://api.github.com/repos/pydata/xarray/issues/2187/reactions"", ""total_count"": 0, ""+1"": 0, ""-1"": 0, ""laugh"": 0, ""hooray"": 0, ""confused"": 0, ""heart"": 0, ""rocket"": 0, ""eyes"": 0}",,completed,13221727,issue