issue_comments

Yes, I think #636 covers this. Thanks @david-ian-brown for your help getting this working!

Do we want to close this in lieu of #636?

Regarding open source: PyNIO is open source now, although of course it is not as accessible as it could be if it were publicly available on github. We are in the process of moving all our software from SVN to git. In fact PyNIO is on github now, although for now it is in a private repository. I hope we can make it public soon. And it would be great to have some community support for improvements.

On Fri, Oct 16, 2015 at 2:38 PM, David Brown dbrown@ucar.edu wrote:

Sorry I have been away for a few days. Ben: thanks for going through the backend code. I will review your changes, etc. as soon as possible. Joe, as Stephan points out, there is a pynio repository on anaconda.org under my user name: dbrown. To install:

conda install -c dbrown -c asmeurer pynio

The reference to the asmeurer channel is needed to get gcc-4.8.5. For some reason, this version of gcc is still not in anaconda defaults. Previous conda versions of gcc do not seem to work for whatever reason.

The default version is pynio-1.5.0beta. This has support for advanced file features like groups. It still has some kinks to be worked out particularly involving support for advanced variable types like variable length arrays and compound variables. There is also a release I am calling pynio-1.4.2. This has the new features added since the 1.4.1 release to the underlying NIO library (or to NCL if you prefer), but only bug-fix type changes to the Python module.

Based on advice from Jonathan Helmus, I am now building the pynio conda package for Linux using a CentOS 5 virtual box running under vagrant. This is supposed to provide a more universal package because subsequent Linux releases are backwards compatible -- the main issue being the glibc version.

So I have been building with a CentOS 5 box and testing with virtual boxes of other Linux distributions including CentOS 6 and 7, Debian 6 and 7 and Ubuntu 12 and 14. I am not sure whether this is overkill.

The pynio-1.4.2 packages are built using this method. I was thinking I would have the 1.5.0beta versions done yesterday, but I encountered a new issue: within the last week the HDF4-4.2.11 library has been added to anaconda defaults and now trumps the version I built. The anaconda version does not include the optional szip library because of its restrictive license, whereas I had built HDF4 with szip as has been customary here at NCAR. Since I want to play well with the standard conda stack I now need to rebuild without szip support -- not a big deal I guess, but an unexpected obstacle. Anyway, it is now likely that a pynio install using the current conda packages will fail because of a missing szip library error. I am rebuilding the packages now and hope to have new versions if not today then by Monday.

On Tue, Oct 13, 2015 at 3:25 PM, Stephan Hoyer notifications@github.com wrote:

@david-ian-brown https://github.com/david-ian-brown has conda packages for pynio on OS X and Linux on Binstar (which is apparently now "Anaconda Cloud"): https://anaconda.org/dbrown/pynio/files

We could try hooking these up to Travis-CI. When I tested out the OS X

package a few months it did seem to work OK in a simple test.

I think we should reboot this PR by adding readonly support for pynio first. That's a lot simpler to get working and will meet many needs. Once that's working, someone can add write support later if there's a need for that.

— Reply to this email directly or view it on GitHub https://github.com/xray/xray/pull/459#issuecomment-147858588.

Sorry I have been away for a few days. Ben: thanks for going through the backend code. I will review your changes, etc. as soon as possible. Joe, as Stephan points out, there is a pynio repository on anaconda.org under my user name: dbrown. To install:

conda install -c dbrown -c asmeurer pynio

The reference to the asmeurer channel is needed to get gcc-4.8.5. For some reason, this version of gcc is still not in anaconda defaults. Previous conda versions of gcc do not seem to work for whatever reason.

The default version is pynio-1.5.0beta. This has support for advanced file features like groups. It still has some kinks to be worked out particularly involving support for advanced variable types like variable length arrays and compound variables. There is also a release I am calling pynio-1.4.2. This has the new features added since the 1.4.1 release to the underlying NIO library (or to NCL if you prefer), but only bug-fix type changes to the Python module.

Based on advice from Jonathan Helmus, I am now building the pynio conda package for Linux using a CentOS 5 virtual box running under vagrant. This is supposed to provide a more universal package because subsequent Linux releases are backwards compatible -- the main issue being the glibc version.

So I have been building with a CentOS 5 box and testing with virtual boxes of other Linux distributions including CentOS 6 and 7, Debian 6 and 7 and Ubuntu 12 and 14. I am not sure whether this is overkill.

The pynio-1.4.2 packages are built using this method. I was thinking I would have the 1.5.0beta versions done yesterday, but I encountered a new issue: within the last week the HDF4-4.2.11 library has been added to anaconda defaults and now trumps the version I built. The anaconda version does not include the optional szip library because of its restrictive license, whereas I had built HDF4 with szip as has been customary here at NCAR. Since I want to play well with the standard conda stack I now need to rebuild without szip support -- not a big deal I guess, but an unexpected obstacle. Anyway, it is now likely that a pynio install using the current conda packages will fail because of a missing szip library error. I am rebuilding the packages now and hope to have new versions if not today then by Monday.

On Tue, Oct 13, 2015 at 3:25 PM, Stephan Hoyer notifications@github.com wrote:

@david-ian-brown https://github.com/david-ian-brown has conda packages for pynio on OS X and Linux on Binstar (which is apparently now "Anaconda Cloud"): https://anaconda.org/dbrown/pynio/files

We could try hooking these up to Travis-CI. When I tested out the OS X

package a few months it did seem to work OK in a simple test.

I think we should reboot this PR by adding readonly support for pynio first. That's a lot simpler to get working and will meet many needs. Once that's working, someone can add write support later if there's a need for that.

— Reply to this email directly or view it on GitHub https://github.com/xray/xray/pull/459#issuecomment-147858588.

@david-ian-brown has conda packages for pynio on OS X and Linux on Binstar (which is apparently now "Anaconda Cloud"): https://anaconda.org/dbrown/pynio/files

We could try hooking these up to Travis-CI. When I tested out the OS X package a few months it did seem to work OK in a simple test.

I think we should reboot this PR by adding readonly support for pynio first. That's a lot simpler to get working and will meet many needs. Once that's working, someone can add write support later if there's a need for that.

A few general comments that maybe @WeatherGod or @david-ian-brown could answer: - What are the development plans for pynio? According to its website, pynio hasn't had a release since July 28, 2011. I'm a bit concerned about the stated requirement of the package being Python 2.6-2.7 (not 3) and NumPy 1.4-1.6. We require NumPy 1.7 or later. - Would NCAR being willing to open source pynio? I'd bet we could get some community support for bringing the package up to date.

Where do we stand with the conda recipe for pynio?

That's all the time I have at the moment. I do have some more notes from my old, incomplete implementation, though. I'll try to finish the review tomorrow.

The main thing we need to finish this up is automated testing on Travis-CI. For that, we'll either need a conda package for pynio or a bash script that can install it on Ubuntu.

I know you guys are working on the conda package. So, keep me informed on your progress :).