issue_comments

Hi @jonathanstrong,

Just thought it would be useful to point out that the people who maintain NetCDF is Unidata, a branch of the University Corporation for Atmospheric Research. In fact, netCDF-4 is essentially built on top of HDF5 - a much more widely-known file format, with first-class support including an I/O layer in pandas. While it would certainly be great to "sell" netCDF as a format in the documentation, those of us who still have to write netCDF-based I/O modules for our Fortran models might have to throw up a little in our mouths when we do so...

hey,

So - after using netcdf for a few days, definitely not looking back. This is great. By way of background, I am building a way to integrate storage of arbitrary arrays into an otherwise highly-structured schema. After using postgresql arrays flamed out (too slow, even at the raw SQL level), I moved on to saving the file path in my schema. I thought it would be sensible to keep everything in pure ndarray for simplicity. After trying out netcdf, I bit the bullet and wrote constructors for numpy, pandas and xarray types, and it's working great.

Looking back, I actually think the documentation could use some work to help people like me, who haven't used netcdf, realize it's great.

If you look at the docs, it starts with pickle, which for me is kind of a red flag since from my experience pickle is the world's most flaky persistence method (always has dependency issues). Then at netcdf you start with:

"Currently, the only disk based serialization format that xarray directly supports is netCDF."

I read this and think, ok so the IO is not really there yet. It's like an apology there aren't more choices.

"netCDF is a file format for fully self-described datasets that is widely used in the geosciences and supported on almost all platforms."

Hmm...geosciences...who knows what those people are doing? I'm also generally suspicious of academics when it comes to code.

"We use netCDF because xarray was based on the netCDF data model, so netCDF files on disk directly correspond to Dataset objects."

Ok, so it's easy for you. What about me?

I've been a bit over the top but you can see how someone who doesn't use netcdf might read this and think they need to write their own IO functions.

If it were me, I would start off selling how great this format is for xarray. Like, "netcdf is a blazing-fast, binary data format that allows transparent, self-describing persistence with zero of the dependency issues you get with pickle or other formats. It allows xarray Datasets to be saved intact and even used in out-of-core computations for larger-than-memory arrays."

Or something like that.

Finally, regarding DataArray not having it's own method to save: I think this is a deficiency that is easily solved. For me, getting into this library I started with just DataArrays. Now that I am using Datasets I can see how they are pretty dang powerful. But at first the simpler DataArrays were all I was using, and they had no direct IO.

To solve this, you could create a "magic" string for DataArrays. On save, to_netcdf converts the da to a Dataset with the magic key. On load, the load function recognizes the magic string and breaks out that DataArray to return it specifically.

I think that would be a quick, relatively painless way to give DataArrays equal footing with Datasets.

Anyway - my two cents. I am a huge fan of this library and happy to chip in regarding any of the above if desired. Thanks for your hard work on it.

I hadn't, for a number of reasons. First, I've used csv, hdf, sql, json, yaml and other formats but never came across netcdf until using this library as someone who isn't working in the physical sciences. Second, the documentation on netcdf is fairly dense. Third, didn't want to deal with installing the library.

I just did use it and seems like it is great for Datasets. As far as I can tell there is no way to save DataArrays directly, though?

Finally, would note that pandas has io methods for csv, excel, hdf, sql, json, msgpack, html, gbq, stata, "clipboard", and pickle. I think it's a strength to offer more choices.