issue_comments

You can view the rendered docs on readthedocs, even for the dev version: http://xray.readthedocs.org/en/latest/io.html#combining-multiple-files

open_mfdataset is not quite ready for prime-time -- it needs better documentation and the library we use to power it (dask) has a few annoying bugs that will hopefully be fixed soon. I can't offer any guarantees, but if you want to give it a try (you'll need to install the development version of dask), let me know how it goes. I'll be releasing a new version of xray once those dask fixes go in, probably within the next week or two.

I literally merged this into the dev version of the docs a few hours ago :).

Lazy loading goodness is next on my to-do list. Hopefully I'll have more to share soon.

@aidanheerdegen Not directly (yet), but there are some straightforward recipes. In fact, this has been a popular question, so I wrote a new doc section on this the other day: http://xray.readthedocs.org/en/latest/io.html#combining-multiple-files

For now, this is only the development version docs but everything is equally valid for the latest released version.

Slicing the data you need before concatenating is definitely a good strategy here.

Eventually, I'm optimistic that we'll be able to make concat not require loading everything into memory (https://github.com/xray/xray/issues/328)

Oh, OK. In that case, you do want to use concat.

Something like this should work:

python ds = xray.concat([xray.open_dataset(f) for f in my_files], dim='time')

xray doesn't use or set unlimited dimensions. (It's pretty irrelevant for us, given that NumPy arrays can be stored in either row-major or column-major order.)

To clarify -- you have different files for different variables? For example, one file has temperature, another has dewpoint, etc? I think you want to use the Dataset.merge method for this.

On Mon, Mar 2, 2015 at 3:09 AM, JoyMonteiro notifications@github.com wrote:

Hello, I have multiple nc files, and I want to pick one variable from all of them to write to a separate file, and if possible pick one vertical level. The issue is that it has no aggregation dimension, so MFDataset does not work. The idea is to get all data about one variable from one vertical level into a single file. When I use the example in the netCDF4-python website, concat merges all variables along all dimensions, making the in-memory size really large. I'm new to xray, and I was hoping something of this sort can be done. In fact, I don't really need to write it to a new file. Even if I can get one "descriptor" (instead of an array of Dataset objects) to access my data, I will be quite happy! TIA,

Joy

Reply to this email directly or view it on GitHub: https://github.com/xray/xray/issues/349