issue_comments

As it is currently it is also not possible to write a zarr which follows the GDAL ZARR driver conventions. Writing the _CRS attribute also results in a TypeError:

Can you elaborate? What API are you using to do the write: python, netcdf-c, or what?

dimension names stored by xarray in .zattrs["_ARRAY_DIMENSIONS"] are stored by NCZarr in .zarray["_NCZARR_ARRAY"]["dimrefs"]

I made a recent change to this so that where possible, all NCZarr files contain the xarray _ARRAY_ATTRIBUTE. By "where possible" I mean that the array is in the root group and the dimensions it references are "defined" in the root group (i.e. they have the simple FQN "/XXX" where XXX is the dim name. This means that there is sometimes a duplication of information between _ARRAY_ATTRIBUTE and ".zarray["_NCZARR_ARRAY"]["dimrefs"].

I guess I was not clear. If you are willing to lose netcdf specific metadata, then I believe any xarray or zarr implementation should be able to read nczarr written data with no changes needed.

As the moment, NCzarr format files (as opposed to pure Zarr format files produced by NCZarr) do not include the Xarray _ARRAY_DIMENSIONS attribute. Now that I think about it, there is no reason not to include that attribute where it is meaningful, so I will make that change. After that change, the situation should be as follows:

Xarray can read any nczarr format file subject to the following conditions:
1. xarray attempts to read only the root group and ignores subgroups
    * this is because xarray cannot handle subgroups.
2. the xarray implementation ignores extra dictionary keys in e.g. .zarray and .zattr
   that it does not recognize
    * this should already be the case under the principle of "read broadly, write narrowly".

For Unidata and netcdf, I think the situation is briefly this.

In netcdf-4, dimensions are named objects that can "reside" inside groups. So for example we might have this: netcdf example { dimensions: x=1; y=10; z=20; group g1 { dimensions: a=1; y=10; z=5; variables: float v(/x, /g1/y, /z); } } So base dimension names (e.g. "z") can occur in different groups and can represent different dimension objects (with different sizes).

It is possible to reference any dimension using fully-qualified-names (FQNs) such as "/g1/y". This capability is important so that, for example, related dimensions can be isolated with a group.

NCZarr captures this information by recording fully qualified names as special keys. This differs from XArray where fully qualified names are not supported. From the netcdf point of view, it is as if all dimension objects were declared in the root group.

If XArray is to be extended to support the equivalent of groups and distinct sets of dimensions are going to be supported in different groups, then some equivalent of the netcdf FQN is going to be needed.

One final note. In netcdf, the dimension size is declared once and associated with a name. In zarr/xarray, the size occurs in multiple places (via the "shape" key) and the name-size associated is also declared multlple times via the _ARRAY_DIMENSIONS attribute.

I do not know because I do not understand who is doing the caching. The above archive reference is no longer relevant because the dap2 code now uses an in-memory file rather than something in /tmp. Netcdf-c keeps its curl connections open until nc_close is called. I would assume that each curl connection maintains at least one file descriptor open. But is the cache that shows the problem a python maintained cache or a Windows cache of some sort?

So I tried to duplicate using cygwin with the latest netcdf master and using ncdump. It seems to work ok. But this raises a question? Can someone try this command under windows to see if it fails?

ncdump 'https://www.ncei.noaa.gov/thredds/dodsC/OisstBase/NetCDF/V2.0/AVHRR/201703/avhrr-only-v2.20170322.nc'

If it succeeds then it may mean the problem is with python rather than netcdf.

BTW, what version of the netcdf-c library is being used. I see this in an above comment: netcdf4: 1.5.3 But that cannot possibly be correct.

I agree. To be more precise, NC_EPERM is generally thrown when an attempt is made to modify a read-only file. So it is possible that it isn't the DAP2 code, but somewhere, an attempt is being made to modify the dataset. There are pieces of the netcdf-c library that are conditional on Windows. It might be interesting if anyone can check if this occurs under cygwin.

You would lose your money :-) However, I can offer some info that might help. This message: OSError: [Errno -37] NetCDF: Write to read only is NC_EPERM. It is the signal for opendap that you attempted an operation that is illegal for DAP2. As an aside, it is a lousy message but I cannot find anything that is any more informative.

Anyway, it means that your code somehow called one of the following netcdf-c API functions:

nc_redef, nc__enddef, nc_create, nc_put_vara, nc_put_vars nc_set_fill, nc_def_dim, nc_put_att, nc_def_var

Perhaps with this info, you can figure out which of those above operations you invoked. Perhaps you can set breakpoints in the python wrappers for these functions?

I have a couple of questions about _ARRAY_DIMENSIONS. Let me make sure I understand how it is used. Suppose I am given an array X with shape(10,20,30) and an _ARRAY_DIMENSION attribute on X with the contents _ARRAY_DIMENSION=["time", "lon", "lat"] Then this is equivalent to the following partial netcdf CDL: netcdf ... { dims: time=10; lon=20; lat=30; ...} Correct? I assume that if there are conflicts where two variables end up assigning different sIzes to the same named dimension, then that generates an error. Finally it is unclear where xarray puts these dimensions. In the closest enclosIng Group? or in the root group? =DennIs Heimbigner Unidata

Very possibly. The first thing to look at is what opendap is sending:

http://www.ncei.noaa.gov/thredds/dodsC/avhrr-patmos-x-cloudprops-noaa-asc-fc/files/2003/patmosx_v05r03_NOAA-17_asc_d20030101_c20140314.nc.das

The methods that need to be implemented are (in the C API) as follows:

int nc_def_var_filter(int ncid, int varid, unsigned int id, size_t nparams, const unsigned int* parms); The only tricky part is passing a vector of unsigned integers (the parms argument).

int nc_inq_var_filter(int ncid, int varid, unsigned int idp, size_t nparams, unsigned int* params); This requires passing values out via pointers. Also, this uses the standard netcdf-c trick in which the function is called twice. First with nparams defined, but params has the value NULL. This gives the caller the number of parameters. The function is called a second time after allocating the params vector and with that vector as the final argument.

The API is not yet exposed thru anything but the C api. So the python, fortran, and c++ wrappers do not yet show it. Passing it thru netcdf-python is probably pretty trivian, though.

A bit confusing, but I think the answer is yes. For example we provide a bzip2 compression plugin as an example (see examples/C/hdf5plugins in the netcdf-c distribution).

You may already know, but should note that the filter stuff in netcdf-c is now available in netcdf-c library version 4.6.0. So any filter plugin usable with hdf5 can now be used both for reading and writing thru the netcdf-c api.

The github branch filters.dmh for the netcdf-c library now exposes the HDF5 dynamic filter capability. This is documented here: https://github.com/Unidata/netcdf-c/blob/filters.dmh/docs/filters.md I welcome suggestions for improvements.

I also note that I am extending this branch to now handle szip compression. It turns out there is now a patent-free implementation called libaec (HT Rich Signell) so there is no reason not to make it available.