issue_comments

Sure. I'd be happy to make a PR once this gets merged. On Sat, Sep 16, 2017 at 10:39 PM Stephan Hoyer notifications@github.com wrote:

Alternatively apply_ufunc could see if the func object has a pre_dask_atop method, and apply it if it does.

This seems like a reasonable option to me. Once we get this merged, want to make a PR?

@jhamman https://github.com/jhamman could you give this a review? I have not included extensive documentation yet, but I am also reluctant to squeeze that into this PR before we make it public API. (Which I'd like to save for another one.)

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/pull/1517#issuecomment-330022743, or mute the thread https://github.com/notifications/unsubscribe-auth/ABUoksuo9P3AJIzemncQQJZ3D5Ga2Opsks5sjLCdgaJpZM4PAViG .

I guess the key issue here is that some computations (E.g. finite differences) cannot be boiled down to passing one numpy function to atop. Instead, these calculations consist of three steps: 1) some dask calls, 2) a call to atop, map_blocks, etc, and 3) some more dask calls. In spencer's example, step 1 would be a the call to da.ghost.ghost and step 3 would be the call to da.ghost.trim_internal. While many dask functions have analogies in XArray, there are others that do not, such as da.ghost.ghost, so it won't always be possible to support this three step class of calculations by replacing the dask calls before and after atop with xarray calls.

@shoyer Would it unreasonably complicated to add some sort of pre_dask_atop and post_dask_atop arguments to apply_ufunc. Alternatively apply_ufunc could see if the func object has a pre_dask_atop method, and apply it if it does.

Hey Spencer! Thanks. That makes much more sense. I have written nearly identical code for centered differencing, but did not know about apply_ufunc, so I had a couple manual calls to transpose and xr.DataArray. The thing I was hoping for an official way to pass centerd_diff_numpy directly to apply_ufunc, which would avoid some of the boiler-plate in your centered_diff function, which basically rewrites much of the code in this PRs version apply_ufunc.

Ok thanks. just so I understand you correctly, are you recommending something like this:

xarr_ghosted  =  apply_ufunc(partial(da.ghost.ghost, ...), xarr, dask='allowed',...)
fd_ghosted = apply_ufunc(finite_difference_func, xarr_ghosted, dask='parallelized',...)
fd   = apply_ufunc(partial(da.ghost.trim_internal, ...), fd_ghosted, dask='allowed')

Wouldn't xarray complain because the ghosted axes data would have different size than the corresponding coordinates?

This looks great!

I am not sure if this is the right place to bring this up, but is there any way to add ghost cell functionality to apply_ufunc or perhaps to create an apply_ufunc_ghosted function. This would be very handy for performing finite differences and filters. I have successfully used dask.ghost.map_blocksas well as dask.array.ghost.ghost+dask.array.core.atop for this, so I doubtt this would be too hard.