@barronh Truly great work. I’m very excited about this and think it will do wonders for the climate and AQ data analysis.

@shoyer thanks for being patient and help with getting this going.

@shoyer - Thanks for all the help and guidance. I learned a lot. In the development version of pnc, it is now flake8 compliant. I now use pytest to organize my unittests. I have TravisCI implemented. I have a conda-forge recipe.

I’m grateful for the interactions.

thanks for sticking with this @barronh !

The two failures were not pnc:

xarray/tests/test_backends.py::TestRasterio::test_serialization FAILED [ 26%] ... xarray/tests/test_backends.py::TestDataArrayToNetCDF::test_open_dataarray_options FAILED [ 26%]

Neither seems related to PNC at all...

I'm pulling the master and remerging to see if that helps...

Depends on the format and expected size of data in that format. Some formats support lazy; others load immediately into memory. Sill others use memmaps, so that virtual memory is used immediately.

@barronh clarifying question for you: does PNC support some sort of "lazy loading" of data, where it is only loaded into NumPy arrays when accessed? Or does it eagerly load data into NumPy arrays? (sorry if you already answered this somewhere above!)

I trust that none of the other formats PNC supports use _FillValue, add_offset or scale_factor attributes?

The challenge here is that if specific formats use a similar functionality, it has already been applied and may or may not use the CF keywords. So, it should be disabled by default.

If it is possible to detect the inferred file format from PNC, then another option (other than requiring the explicit format argument) would be to load the data and raise an error if the detected file format is netCDF.

I was worried this would be hard to implement, but it was actually easier. So that is what I did.

I tried disabling mask and scale, but many other tests fail. At its root this is because I am implicitly supporting netCDF4 and other formats.

I also tried this and was surprised to see many other tests fail.

I see two ways to solve this. Right now, it is only important to add non-netcdf support to xarray via PseudoNetCDF. I am currently allowing dynamic identification of the file format, which implicitly supports netCDF. I could disable implicit format support, and require the format keyword. In that case, PseudoNetCDF tests no longer should be CFEncodedDataTest. Instead, I can simply test some round tripping with the other formats (uamiv and possibly adding one or two other formats).

This sounds like a good solution to me. I'll leave it up to your judgment which other tests (if any) are worth adding.

I trust that none of the other formats PNC supports use _FillValue, add_offset or scale_factor attributes?

If it is possible to detect the inferred file format from PNC, then another option (other than requiring the explicit format argument) would be to load the data and raise an error if the detected file format is netCDF.

I tried disabling mask and scale, but many other tests fail. At its root this is because I am implicitly supporting netCDF4 and other formats.

I see two ways to solve this. Right now, it is only important to add non-netcdf support to xarray via PseudoNetCDF. I am currently allowing dynamic identification of the file format, which implicitly supports netCDF. I could disable implicit format support, and require the format keyword. In that case, PseudoNetCDF tests no longer should be CFEncodedDataTest. Instead, I can simply test some round tripping with the other formats (uamiv and possibly adding one or two other formats).

What do you think?

@barronh pleas see my comment above: https://github.com/pydata/xarray/pull/1905#issuecomment-381467470

I would be OK not applying masking/scaling at all. But applying masking/scaling twice by default seems problematic and hard to debug. If there's any chance someone will use psuedonetcdf to access a dataset with these attributes, we need to fix this behavior.

Anything else needed?

OK, I pushed a couple of small changes to your branch. Generally this is looking pretty good. I have a couple of other minor comments that I will post inline.

Your issues with the skipped tests (which I switched to xfail) are part of a large issue in xarray, which is that we don't have good ways to handle backend specific decoding (https://github.com/pydata/xarray/issues/2061).

I agree that we can probably put this most of them for now (especially the string encoding), but I'm somewhat concerned about how decoding could end up with reading incorrect data values, e.g., if a source using scale/offset encoding. Consider this failed test: ``` def test_roundtrip_mask_and_scale(self): decoded = create_masked_and_scaled_data() encoded = create_encoded_masked_and_scaled_data() with self.roundtrip(decoded) as actual:

      assert_allclose(decoded, actual, decode_bytes=False)

E AssertionError: [ nan nan 10. 10.1 10.2] E [ nan nan 11. 11.01 11.02] ```

These sort of bugs can be pretty insidious, so if there's any chance that someone would use PNC to read a netCDF file with this sort of encoding we should try to fix this before merging this in.

One simple approach would be to raise an error for now if mask_and_scale=True in open_dataset(), that is, to force the user to explicitly disable masking and scaling with xr.open_dataset(filename, engine='pseudonetcdf', mask_and_scale=False).

Alternatively, I suppose we could switch the default value to mask_and_scale=None, and pick True or False based on the choice of backend.

The conda recipe was approved and merged. Feedstock should be ready soon.

https://github.com/conda-forge/staged-recipes/pull/5449

I've added the latest version to pip. Still waiting on the recipe in conda-forge.

If you have a pseudonetcdf up on pypi, we can install it for our tests with pip instead of conda. That should work for now until the conda recipe is merged. You could simply add it to the end of the "pip" section in this file: https://github.com/pydata/xarray/blob/master/ci/requirements-py36.yml

Do you know how long it typically takes to get added or what I need to do to get it added?

Sorry, but I do not have experience with conda-forge. @shoyer , could you help?

@fujiisoup - I have a recipe in conda-forge that is passing all tests. Do you know how long it typically takes to get added or what I need to do to get it added?

https://github.com/conda-forge/staged-recipes/pull/5449

The tests are fixed on master, please merge in master to fix this.

I opened a new issue for the scipy 1.0.1 failures: https://github.com/pydata/xarray/issues/2019

(I'll try to take a look at the shortly)

This looks due to scipy 1.0.1 and not related to this change.

I have added a recipe to conda-forge and it is passing all tests. I don't know when it will be in feedstock.

Thanks :) Let's wait until it will be in feedstock.

Help me understand, there are now failures in scipy that seem unrelated to my changes. In fact, I had to switch the writer in my tests to netcdf4 to bypass the scipy problem. Is this going to hold up my branch?

I have added a recipe to conda-forge and it is passing all tests. I don't know when it will be in feedstock.

You guys are great. Thanks @barronh I’m going to be implementing this in MONET really soon.

I only mean to comment, not close and comment.

@bbakernoaa - It may be a bit. I'm

What is the status of this? I would like to start using this ASAP but cannot convince my team to use it until it is an official backend of xarray

My variable objects present a pure numpy array, so they follow numpy indexing precisely with one exception. If the files are actually netCDF4, they have the same limitations of the netCDF4.Variable object.

OK, we will need to surface this information in some way for xarray -- maybe as an attribute of some sort on the psuedonetcdf side? The good news is that we already have handling for indexing like NumPy arrays and netCDF4 variables, but we need to know which behavior to expect to make all of indexing operations work efficiently.

It's also safe to say for now that only basic indexing is supported, but that will result in sub-optimal indexing behavior ( slower and more memory intensive than necessary).

Note that we are currently in the process of refactoring how we handle indexers in backends, so you'll probably need to update things after https://github.com/pydata/xarray/pull/1899 is merged.

Indexing support. Do you only support basic-indexing like x[0, :5] or is indexing with integer arrays also supported?

My variable objects present a pure numpy array, so they follow numpy indexing precisely with one exception. If the files are actually netCDF4, they have the same limitations of the netCDF4.Variable object.

Serialization/thread-safety. Can we simultaneously read a file with another process or thread using dask?

I have not tested separate processes. In many cases, I use numpy memmap. So that will be the limitation.

API consistency for scalar arrays. Do these require some sort of special API compared to non-scalar arrays?

Same as numpy, but also has support for the netCDF4 style.

Data types support. Are strings and datetimes converted properly into the format xarray expects?

I use relative dates following netcdf time conventions. Within my software, there are special functions for translation, but I have seen this be treated by xarray separately.

Continuous integration testing in PseudoNetCDF, at a minimum on TravicCI,, but Appveyor would be great too.

I added TravisCI, but haven't looked Appveyor.

A conda-forge package to facilitate easy installs of PseudoNetCDF

I added a ci/requirements-py36-netcdf4-dev.yml as a part of my TravisCI integration. I am also working on a recipe (like https://github.com/conda-forge/xarray-feedstock).

I couldn't find a .stickler.yml file in this repository. I can make one for you, or you can create one by following the documentation.

I can create binary data from within python and then read it, but all those tests are in my software package. Duplicating that seems like a bad idea.

Right.

The goal isn't to duplicate your tests, but to provide a meaningful integration test for the xarray backend. Can we read data from PNC into xarray and do everything an xarray user would want to do with it? Testing your API with a netCDF3 file would probably be enough, assuming you have good test coverage internally.

We already have a somewhat complete test suite for netCDF data that you could probably hook into, but for reference sorts of issues that tend to come up include: - Indexing support. Do you only support basic-indexing like x[0, :5] or is indexing with integer arrays also supported? - Serialization/thread-safety. Can we simultaneously read a file with another process or thread using dask? - API consistency for scalar arrays. Do these require some sort of special API compared to non-scalar arrays? - Data types support. Are strings and datetimes converted properly into the format xarray expects?

Before closing this request and opening another, I want to make sure I am clear one the extent of what I should add.

First, don't close this PR. You can just keep adding commits to your pnc-backend.

In terms of test coverage, let's start by detailing what functionality we expect the PseudoNetCDF backend to have.

• What file formats are supported by PseudoNetCDF?
• What roundtrip capability is expected when PseudoNetCDF is used (i.e. if I write a netCDF3 file with scipy, can I open it with PNC and get exactly the same dataset)?
• What are PseudoNetCDF's dependencies (Python 3?, netCDF4?), are there optional dependencies?

Finally, two things that would really help grease the skids here would be:

1. Continuous integration testing in PseudoNetCDF, at a minimum on TravicCI, but Appveyor would be great too.
2. A conda-forge package to facilitate easy installs of PseudoNetCDF

Neither of these last two are strictly required but both are satisfied by all the other Xarray supported backends.

First, I too quickly tried to fast forward and clearly some test was failing. I have updated my code to pass all tests with py.test.

Before closing this request and opening another, I want to make sure I am clear one the extent of what I should add. I can create binary data from within python and then read it, but all those tests are in my software package. Duplicating that seems like a bad idea.

I have added a NetCDF3Only testcase to test_backends.py and it passes. That doesn't stress the multi-format capabilities of pnc, but as I've said all the numerical assertions for the other formats are in my system's test cases. Is the NetCDF3Only test sufficient in this case?

Further, below are some simple applications that download my test data for CAMx and GEOS-Chem and plot it.

Thanks for the input.

``` import xarray as xr from urllib.request import urlretrieve

CAMx test

urlretrieve('https://github.com/barronh/pseudonetcdf/blob/master/src/PseudoNetCDF/testcase/camxfiles/uamiv/test.uamiv?raw=true', 'test.uamiv') xf = xr.open_dataset('test.uamiv', engine = 'pnc') pm = xf.O3.isel(TSTEP = 0, LAY = 0).plot() pm.axes.figure.savefig('test_camx.png')

pm.axes.figure.clf()

GEOS-Chem Test

urlretrieve("https://github.com/barronh/pseudonetcdf/blob/master/src/PseudoNetCDF/testcase/geoschemfiles/test.bpch?raw=true", "test.bpch") urlretrieve("https://github.com/barronh/pseudonetcdf/blob/master/src/PseudoNetCDF/testcase/geoschemfiles/tracerinfo.dat?raw=true", "tracerinfo.dat") urlretrieve("https://github.com/barronh/pseudonetcdf/blob/master/src/PseudoNetCDF/testcase/geoschemfiles/diaginfo.dat?raw=true", "diaginfo.dat") xf = xr.open_dataset('test.bpch', engine = 'pnc') xa = getattr(xf, 'IJ-AVG-$_Ox') xa2d = xa.isel(time = 0, layer3 = 0) pm = xa2d.plot() pm.axes.figure.savefig('test_bpch.png') ```

First of all -- this is very cool, thanks for putting this together!

In the long term, I think we would prefer to move more specialized backends out of xarray proper. But at the current time, it's difficult to do this easily, and the backend interface itself is not entirely stable. So it probably makes sense to add this directly into xarray for now.

To merge this into xarray, I'm afraid that an automated test suite of some sort is non-negotiable. Are there example datasets can you can create on the fly from Python?

p.s., I just noticed my edits to whats-new.rst were not pushed until after my pull request.