@shoyer - Thanks for all the help and guidance. I learned a lot. In the development version of pnc, it is now flake8 compliant. I now use pytest to organize my unittests. I have TravisCI implemented. I have a conda-forge recipe.

I’m grateful for the interactions.

The two failures were not pnc:

xarray/tests/test_backends.py::TestRasterio::test_serialization FAILED [ 26%] ... xarray/tests/test_backends.py::TestDataArrayToNetCDF::test_open_dataarray_options FAILED [ 26%]

Neither seems related to PNC at all...

I'm pulling the master and remerging to see if that helps...

Depends on the format and expected size of data in that format. Some formats support lazy; others load immediately into memory. Sill others use memmaps, so that virtual memory is used immediately.

I trust that none of the other formats PNC supports use _FillValue, add_offset or scale_factor attributes?

The challenge here is that if specific formats use a similar functionality, it has already been applied and may or may not use the CF keywords. So, it should be disabled by default.

If it is possible to detect the inferred file format from PNC, then another option (other than requiring the explicit format argument) would be to load the data and raise an error if the detected file format is netCDF.

I was worried this would be hard to implement, but it was actually easier. So that is what I did.

I tried disabling mask and scale, but many other tests fail. At its root this is because I am implicitly supporting netCDF4 and other formats.

I see two ways to solve this. Right now, it is only important to add non-netcdf support to xarray via PseudoNetCDF. I am currently allowing dynamic identification of the file format, which implicitly supports netCDF. I could disable implicit format support, and require the format keyword. In that case, PseudoNetCDF tests no longer should be CFEncodedDataTest. Instead, I can simply test some round tripping with the other formats (uamiv and possibly adding one or two other formats).

What do you think?

Anything else needed?

The conda recipe was approved and merged. Feedstock should be ready soon.

https://github.com/conda-forge/staged-recipes/pull/5449

I've added the latest version to pip. Still waiting on the recipe in conda-forge.

@fujiisoup - I have a recipe in conda-forge that is passing all tests. Do you know how long it typically takes to get added or what I need to do to get it added?

https://github.com/conda-forge/staged-recipes/pull/5449

Help me understand, there are now failures in scipy that seem unrelated to my changes. In fact, I had to switch the writer in my tests to netcdf4 to bypass the scipy problem. Is this going to hold up my branch?

I have added a recipe to conda-forge and it is passing all tests. I don't know when it will be in feedstock.

I only mean to comment, not close and comment.

@bbakernoaa - It may be a bit. I'm

Indexing support. Do you only support basic-indexing like x[0, :5] or is indexing with integer arrays also supported?

My variable objects present a pure numpy array, so they follow numpy indexing precisely with one exception. If the files are actually netCDF4, they have the same limitations of the netCDF4.Variable object.

Serialization/thread-safety. Can we simultaneously read a file with another process or thread using dask?

I have not tested separate processes. In many cases, I use numpy memmap. So that will be the limitation.

API consistency for scalar arrays. Do these require some sort of special API compared to non-scalar arrays?

Same as numpy, but also has support for the netCDF4 style.

Data types support. Are strings and datetimes converted properly into the format xarray expects?

I use relative dates following netcdf time conventions. Within my software, there are special functions for translation, but I have seen this be treated by xarray separately.

Continuous integration testing in PseudoNetCDF, at a minimum on TravicCI,, but Appveyor would be great too.

I added TravisCI, but haven't looked Appveyor.

A conda-forge package to facilitate easy installs of PseudoNetCDF

I added a ci/requirements-py36-netcdf4-dev.yml as a part of my TravisCI integration. I am also working on a recipe (like https://github.com/conda-forge/xarray-feedstock).

I couldn't find a .stickler.yml file in this repository. I can make one for you, or you can create one by following the documentation.

First, I too quickly tried to fast forward and clearly some test was failing. I have updated my code to pass all tests with py.test.

Before closing this request and opening another, I want to make sure I am clear one the extent of what I should add. I can create binary data from within python and then read it, but all those tests are in my software package. Duplicating that seems like a bad idea.

I have added a NetCDF3Only testcase to test_backends.py and it passes. That doesn't stress the multi-format capabilities of pnc, but as I've said all the numerical assertions for the other formats are in my system's test cases. Is the NetCDF3Only test sufficient in this case?

Further, below are some simple applications that download my test data for CAMx and GEOS-Chem and plot it.

Thanks for the input.

``` import xarray as xr from urllib.request import urlretrieve

CAMx test

urlretrieve('https://github.com/barronh/pseudonetcdf/blob/master/src/PseudoNetCDF/testcase/camxfiles/uamiv/test.uamiv?raw=true', 'test.uamiv') xf = xr.open_dataset('test.uamiv', engine = 'pnc') pm = xf.O3.isel(TSTEP = 0, LAY = 0).plot() pm.axes.figure.savefig('test_camx.png')

pm.axes.figure.clf()

GEOS-Chem Test

urlretrieve("https://github.com/barronh/pseudonetcdf/blob/master/src/PseudoNetCDF/testcase/geoschemfiles/test.bpch?raw=true", "test.bpch") urlretrieve("https://github.com/barronh/pseudonetcdf/blob/master/src/PseudoNetCDF/testcase/geoschemfiles/tracerinfo.dat?raw=true", "tracerinfo.dat") urlretrieve("https://github.com/barronh/pseudonetcdf/blob/master/src/PseudoNetCDF/testcase/geoschemfiles/diaginfo.dat?raw=true", "diaginfo.dat") xf = xr.open_dataset('test.bpch', engine = 'pnc') xa = getattr(xf, 'IJ-AVG-$_Ox') xa2d = xa.isel(time = 0, layer3 = 0) pm = xa2d.plot() pm.axes.figure.savefig('test_bpch.png') ```

p.s., I just noticed my edits to whats-new.rst were not pushed until after my pull request.