issue_comments

Hi Stephan, I notice that you posted a solution of sorts on stackoverflow. Is this a real fix or a bandaid for the problem? I ran a few tests with pynio by itself where there was no problem opening e.g. a GRIB file for reading and a NetCDF file for writing at the same time. I am wondering if the issue involves using pynio and the netCDF4 modules at the same time. Let me know if still think something in pynio is not working correctly. Thanks. -dave

On Tue, Aug 2, 2016 at 1:35 PM, David Brown dbrown@ucar.edu wrote:

Hi Stephan,

I will look into this issue. -dave

On Tue, Aug 2, 2016 at 11:01 AM, Stephan Hoyer notifications@github.com wrote:

As reported on StackOverflow: http://stackoverflow.com/ questions/38711915/segmentation-fault-writing-xarray-datset-to-netcdf-or- dataframe/

It appears that we can only open a single file at a time with pynio?

Adding a thread lock via lock=True didn't solve the issue.

cc @david-ian-brown https://github.com/david-ian-brown

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/issues/936, or mute the thread https://github.com/notifications/unsubscribe-auth/AB7VbedP7yIJsI6DEz0PLOnVJJBDadaKks5qb3gDgaJpZM4Ja1AT .

Hi Stephan,

I will look into this issue. -dave

On Tue, Aug 2, 2016 at 11:01 AM, Stephan Hoyer notifications@github.com wrote:

As reported on StackOverflow: http://stackoverflow.com/questions/38711915/segmentation-fault-writing-xarray-datset-to-netcdf-or-dataframe/

It appears that we can only open a single file at a time with pynio?

Adding a thread lock via lock=True didn't solve the issue.

cc @david-ian-brown https://github.com/david-ian-brown

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/issues/936, or mute the thread https://github.com/notifications/unsubscribe-auth/AB7VbedP7yIJsI6DEz0PLOnVJJBDadaKks5qb3gDgaJpZM4Ja1AT .

The PyNIO license has been rewritten as a standard 3-clause BSD license. Hopefully this satisfies all concerns.

http://www.pyngl.ucar.edu/Licenses/PyNIO_source_license.shtml

We will put it on the github page when it is made public.

On Thu, Oct 29, 2015 at 2:40 PM, Stephan Hoyer notifications@github.com wrote:

I agree that the license seems complaint with OSI guidelines, but nonetheless it isn't an OSI approved license. These FAQ entries seem relevant: http://opensource.org/faq#approved-licenses-only

— Reply to this email directly or view it on GitHub https://github.com/xray/xray/pull/636#issuecomment-152312224.

OK I'll pass along that information.

On Thu, Oct 29, 2015 at 2:40 PM, Stephan Hoyer notifications@github.com wrote:

I agree that the license seems complaint with OSI guidelines, but nonetheless it isn't an OSI approved license. These FAQ entries seem relevant: http://opensource.org/faq#approved-licenses-only

— Reply to this email directly or view it on GitHub https://github.com/xray/xray/pull/636#issuecomment-152312224.

I was just informed that "our lawyers looked it over and said we're OSI compliant."

Specifically "He compared the wording of both ours and the BSD one that you referenced, and said they are about 98% similar. The other 2%, which is something that's in the BSD that's not in ours, doesn't cause our license to be non-compliant."

Do you think this works? I do think we should fix the bad link to GRIBEX and point to the current ECMWF license. -dave

On Thu, Oct 29, 2015 at 2:16 PM, David Brown dbrown@ucar.edu wrote:

We are looking into our options on this. I know there were NCAR legal people involved in producing the current license. Hopefully given the current environment we will be to put it into a standard form that clearly meets the OSI requirement.

On Thu, Oct 29, 2015 at 11:14 AM, Filipe notifications@github.com wrote:

but it would be nice to be able to say that you use a standard license

[image: :+1:]

— Reply to this email directly or view it on GitHub https://github.com/xray/xray/pull/636#issuecomment-152253168.

We are looking into our options on this. I know there were NCAR legal people involved in producing the current license. Hopefully given the current environment we will be to put it into a standard form that clearly meets the OSI requirement.

On Thu, Oct 29, 2015 at 11:14 AM, Filipe notifications@github.com wrote:

but it would be nice to be able to say that you use a standard license

[image: :+1:]

— Reply to this email directly or view it on GitHub https://github.com/xray/xray/pull/636#issuecomment-152253168.

OK thanks for the info and the doc suggestions.

On Thu, Oct 29, 2015 at 10:37 AM, Stephan Hoyer notifications@github.com wrote:

Aside from the notice at the top "PLEASE READ THIS SOFTWARE LICENSE CAREFULLY" the license is direct copy of 3 clause BSD. It would be great to get this in the readme (once you're on github) to make this entirely clear.

The link for the license to GRIBX no longer works on the ECMWF website. But some googling turns up source files with the Apache 2.0 license attached, so I think we're in the clear (Apache 2.0 is BSD compatible). In any case, it should be in the PyNIO source code. For easier reference, it would be better to copy in the text of any referenced licenses in the PyNIO license itself, though.

— Reply to this email directly or view it on GitHub https://github.com/xray/xray/pull/636#issuecomment-152240832.

Good question. I don't know. I do know it was intended to be as open as BSD, but I will need to research further what the actual situation is.

On Thu, Oct 29, 2015 at 10:03 AM, Filipe notifications@github.com wrote:

The license here: http://www.pyngl.ucar.edu/Licenses/PyNIO_source_license.shtml

@david-ian-brown https://github.com/david-ian-brown sorry for my ignorance but is that OSI approved http://opensource.org/licenses?

— Reply to this email directly or view it on GitHub https://github.com/xray/xray/pull/636#issuecomment-152226596.

The license here:

http://www.pyngl.ucar.edu/Licenses/PyNIO_source_license.shtml

On Wed, Oct 28, 2015 at 3:51 PM, Stephan Hoyer notifications@github.com wrote:

@david-ian-brown https://github.com/david-ian-brown What license are you using for PyNIO? It looks like https://www.pyngl.ucar.edu/Licenses/PyNIO_source_license.shtml a standard 3 clause BSD? This would be helpful information to highlight on the new GitHub site -- I could only finding it by searching google, not by navigating on the website.

— Reply to this email directly or view it on GitHub https://github.com/xray/xray/pull/636#issuecomment-152005733.

Hi Filipe, We will be making the pynio github site public soon, and I will update the PyPI information when that happens.

On Wed, Oct 28, 2015 at 3:51 AM, Filipe notifications@github.com wrote:

Maybe we could get ioos to also keep a pynio conda build up to date (cc @ocefpaf https://github.com/ocefpaf)

@david-ian-brown https://github.com/david-ian-brown is pynio open source now? If so I can add it to the ioos channel, but at PyPI it is still missing the source.

— Reply to this email directly or view it on GitHub https://github.com/xray/xray/pull/636#issuecomment-151783551.

Great! Just FYI, the conda channel name will almost certainly change soon to something more representative of NCAR. I will let you know when that happens. Thanks for working with us on this.

Hi Stephan, So in this message: RuntimeError: module compiled against API version a but this version of numpy is 9 the "a" actually means 10 in hex I suppose.

I see that I need to do more thorough testing. My latest builds have all pulled in 1.10.x by default. I did put some NumPy specific versions up on the conda site before 1.10 was released. Maybe I need to make all the builds I do numpy (and python) version specific. I'll get a new 1.9 version up today.

Regarding open source: PyNIO is open source now, although of course it is not as accessible as it could be if it were publicly available on github. We are in the process of moving all our software from SVN to git. In fact PyNIO is on github now, although for now it is in a private repository. I hope we can make it public soon. And it would be great to have some community support for improvements.

On Fri, Oct 16, 2015 at 2:38 PM, David Brown dbrown@ucar.edu wrote:

Sorry I have been away for a few days. Ben: thanks for going through the backend code. I will review your changes, etc. as soon as possible. Joe, as Stephan points out, there is a pynio repository on anaconda.org under my user name: dbrown. To install:

conda install -c dbrown -c asmeurer pynio

The reference to the asmeurer channel is needed to get gcc-4.8.5. For some reason, this version of gcc is still not in anaconda defaults. Previous conda versions of gcc do not seem to work for whatever reason.

The default version is pynio-1.5.0beta. This has support for advanced file features like groups. It still has some kinks to be worked out particularly involving support for advanced variable types like variable length arrays and compound variables. There is also a release I am calling pynio-1.4.2. This has the new features added since the 1.4.1 release to the underlying NIO library (or to NCL if you prefer), but only bug-fix type changes to the Python module.

Based on advice from Jonathan Helmus, I am now building the pynio conda package for Linux using a CentOS 5 virtual box running under vagrant. This is supposed to provide a more universal package because subsequent Linux releases are backwards compatible -- the main issue being the glibc version.

So I have been building with a CentOS 5 box and testing with virtual boxes of other Linux distributions including CentOS 6 and 7, Debian 6 and 7 and Ubuntu 12 and 14. I am not sure whether this is overkill.

The pynio-1.4.2 packages are built using this method. I was thinking I would have the 1.5.0beta versions done yesterday, but I encountered a new issue: within the last week the HDF4-4.2.11 library has been added to anaconda defaults and now trumps the version I built. The anaconda version does not include the optional szip library because of its restrictive license, whereas I had built HDF4 with szip as has been customary here at NCAR. Since I want to play well with the standard conda stack I now need to rebuild without szip support -- not a big deal I guess, but an unexpected obstacle. Anyway, it is now likely that a pynio install using the current conda packages will fail because of a missing szip library error. I am rebuilding the packages now and hope to have new versions if not today then by Monday.

On Tue, Oct 13, 2015 at 3:25 PM, Stephan Hoyer notifications@github.com wrote:

@david-ian-brown https://github.com/david-ian-brown has conda packages for pynio on OS X and Linux on Binstar (which is apparently now "Anaconda Cloud"): https://anaconda.org/dbrown/pynio/files

We could try hooking these up to Travis-CI. When I tested out the OS X

package a few months it did seem to work OK in a simple test.

I think we should reboot this PR by adding readonly support for pynio first. That's a lot simpler to get working and will meet many needs. Once that's working, someone can add write support later if there's a need for that.

— Reply to this email directly or view it on GitHub https://github.com/xray/xray/pull/459#issuecomment-147858588.

Sorry I have been away for a few days. Ben: thanks for going through the backend code. I will review your changes, etc. as soon as possible. Joe, as Stephan points out, there is a pynio repository on anaconda.org under my user name: dbrown. To install:

conda install -c dbrown -c asmeurer pynio

The reference to the asmeurer channel is needed to get gcc-4.8.5. For some reason, this version of gcc is still not in anaconda defaults. Previous conda versions of gcc do not seem to work for whatever reason.

The default version is pynio-1.5.0beta. This has support for advanced file features like groups. It still has some kinks to be worked out particularly involving support for advanced variable types like variable length arrays and compound variables. There is also a release I am calling pynio-1.4.2. This has the new features added since the 1.4.1 release to the underlying NIO library (or to NCL if you prefer), but only bug-fix type changes to the Python module.

Based on advice from Jonathan Helmus, I am now building the pynio conda package for Linux using a CentOS 5 virtual box running under vagrant. This is supposed to provide a more universal package because subsequent Linux releases are backwards compatible -- the main issue being the glibc version.

So I have been building with a CentOS 5 box and testing with virtual boxes of other Linux distributions including CentOS 6 and 7, Debian 6 and 7 and Ubuntu 12 and 14. I am not sure whether this is overkill.

The pynio-1.4.2 packages are built using this method. I was thinking I would have the 1.5.0beta versions done yesterday, but I encountered a new issue: within the last week the HDF4-4.2.11 library has been added to anaconda defaults and now trumps the version I built. The anaconda version does not include the optional szip library because of its restrictive license, whereas I had built HDF4 with szip as has been customary here at NCAR. Since I want to play well with the standard conda stack I now need to rebuild without szip support -- not a big deal I guess, but an unexpected obstacle. Anyway, it is now likely that a pynio install using the current conda packages will fail because of a missing szip library error. I am rebuilding the packages now and hope to have new versions if not today then by Monday.

On Tue, Oct 13, 2015 at 3:25 PM, Stephan Hoyer notifications@github.com wrote:

@david-ian-brown https://github.com/david-ian-brown has conda packages for pynio on OS X and Linux on Binstar (which is apparently now "Anaconda Cloud"): https://anaconda.org/dbrown/pynio/files

We could try hooking these up to Travis-CI. When I tested out the OS X

package a few months it did seem to work OK in a simple test.

I think we should reboot this PR by adding readonly support for pynio first. That's a lot simpler to get working and will meet many needs. Once that's working, someone can add write support later if there's a need for that.

— Reply to this email directly or view it on GitHub https://github.com/xray/xray/pull/459#issuecomment-147858588.