issue_comments

We heard over the weekend that the xarray tutorial was not selected for Scipy 2018. From reading the reviewer comments, it sounds like we (mostly I) did not provide a sufficient outline of topics to fully describe what the tutorial would cover. This seems mostly like a misunderstanding on my part as to the expected level of detail in the abstract.

In the hopes that we'll be able to get a slot for one of these in the next few years, I'll post both the abstract and review comments here.

Abstract:

Xarray provides data structures for N-dimensional labeled arrays and a toolkit for scalable data analysis on large, complex datasets with many related variables, such as those that occur in the disciplines of earth science, astronomy, and finance. Xarray combines the power of labeled data structures from Pandas, with the N-dimensional arrays from Numpy and parallel out-of-core computation from Dask, to provide an intuitive and powerful platform for scientific analysis of large multi-dimensional datasets.

This tutorial introduces data scientists who may already be familiar with Numpy or Pandas to the Xarray data model and tool kit. Following an introduction to Xarray, we will introduce tools for scaling real-world scientific data analysis workflows using Xarray and Dask. Students will leave this tutorial with 1) a comprehensive understanding of the Xarray data model, 2) the ability to apply the Xarray tool kit to analysis workflows that fit in memory, and 3) the ability to scale those same workflows to datasets that are much too large to fit into memory (GBs to TBs). Participants are expected to have some familiarity with Jupyter, Numpy, and Pandas.

Links: http://xarray.pydata.org, http://dask.pydata.org

Short Description of the Tutorial:

Xarray provides data structures for N-dimensional labeled arrays and a toolkit for scalable data analysis on large, complex datasets with many related variables. Xarray combines the power of labeled data structures from Pandas, with the N-dimensional arrays from Numpy and parallel out-of-core computation from Dask, to provide an intuitive and powerful platform for scientific analysis of large multi-dimensional datasets. This tutorial introduces data scientists who may already be familiar with Numpy or Pandas to the Xarray package. We will guide participants through the process of scaling Xarray computations from small to big data science workflows.

Review Comments:

Dear Joseph, We didn't select your tutorial, "Xarray for Scalable Scientific Data Analysis", for SciPy 2018, but we would like to wholeheartedly thank you for your submission. The proposals were exceptionally good this year. We received 55 applications and had only 24 spots. We made our selection based on the reviewers' feedback and the likely popularity of the tutorial. We also made a few tough calls to ensure a good diversity of topics and presenters.

Below is the raw feedback from the reviewers who looked at your application. We hope it'll prove useful and we look forward to receiving another proposal from you next year.

Best regards, Alex, Ben & Mike. SciPy 2018 Tutorials Committee

----------------------- REVIEW 1 --------------------- PAPER: 70 TITLE: Xarray for Scalable Scientific Data Analysis AUTHORS: Joseph Hamman

I do not have a conflict of interest.: yes Overall evaluation: 2 (accept) What level of interest do you think this tutorial will generate?: 3 (Widespread appeal)

----------- Overall evaluation ----------- This seems like a very promising presentation quite apt for attendees of the SciPy conference about an underappreciated tool for scientific Python. It also seems like the proposer is well positioned to be the instructor for such a tutorial. However, I am concerned about the lack of specific information about what topics will be presented in what order and with what coding exercises and duration. I worry the proposer has not yet developed details about what would be presented. Right now this sounds more like an hour long tutorial than a 3 or 4 hour long tutorial. It would be good if the presented could develop a more detailed outline, also so that potential attendees have a better idea of what to expect.

----------------------- REVIEW 2 --------------------- PAPER: 70 TITLE: Xarray for Scalable Scientific Data Analysis AUTHORS: Joseph Hamman

I do not have a conflict of interest.: yes Overall evaluation: 1 (weak accept) What level of interest do you think this tutorial will generate?: 3 (Widespread appeal)

----------- Overall evaluation ----------- I don't have experience with xarray, but I believe that it is revolutionary software that will change how scientists use Python to analyze data. I spent way too much time doing nasty tricks with numpy to analyze hyperspectral data in my olden days as a scientist, and I'm convinced that xarray will save people that time today. I would really want past me to see this tutorial.

That said, this proposal needs much more detail. What's your time schedule? What example data will you use? What specific functionality of xarray will you be showing (and when), and how do these examples build on each other? Based on the author's background, I trust that this tutorial will turn out fine if it is accepted, but this proposal would be stronger with more detail. I'm recommending that this tutorial be accepted on the solid foundation of the xarray project, the value that I think that project presents, and some trust based on Joseph's background.

The setup instructions are good - I am excited that cloud-based JupyterHub will remove need of local installation. I do think for your local installation guide, you'd be better off specifying version ranges for each dependency, or a link to some environment specification file, in case APIs change between now and the conference. Not a big deal - people can update or downgrade as necessary - just saves some confusion.

----------------------- REVIEW 3 --------------------- PAPER: 70 TITLE: Xarray for Scalable Scientific Data Analysis AUTHORS: Joseph Hamman

I do not have a conflict of interest.: yes Overall evaluation: 2 (accept) What level of interest do you think this tutorial will generate?: 3 (Widespread appeal)

----------- Overall evaluation ----------- This seems like a very strong tutorial, hopefully with broad appeal to both traditional and data scientists.

----------------------- REVIEW 4 --------------------- PAPER: 70 TITLE: Xarray for Scalable Scientific Data Analysis AUTHORS: Joseph Hamman

I do not have a conflict of interest.: yes Overall evaluation: 1 (weak accept) What level of interest do you think this tutorial will generate?: 3 (Widespread appeal)

----------- Overall evaluation ----------- I know there is interesting xarray and that it is playing playing a role in solving hard problems. I would have like to see a more detailed outline, with plans for exercises and timing information.

----------------------- REVIEW 5 --------------------- PAPER: 70 TITLE: Xarray for Scalable Scientific Data Analysis AUTHORS: Joseph Hamman

I do not have a conflict of interest.: yes Overall evaluation: 1 (weak accept) What level of interest do you think this tutorial will generate?: 3 (Widespread appeal)

----------- Overall evaluation ----------- XArray is a fantastic project, and I have been very interested in seeing it grow and gain wider acceptance. The tool can be tricky to use right, and the documentation can be sparse in some places. So, a tutorial would be extremely valuable here. However, the proposal in very lacking in details, which would make it difficult to judge whether or not there is a 4-hour tutorial (I am sure there is), but also makes it hard for a potential participant to judge whether this tutorial is for them or not.

Note: while I don't have a conflict of interest with the author, I am acknowledged in a paper about XArray as an early contributor.

@GiorgioBalestrieri - Yes. Both tutorials will be made available. Stay tuned!

@jakevdp - do you know of any astronomy applications of xarray?

I submitted an abstract for an xarray tutorial today. More information to come as we get closer to the conference but for now the title: Xarray for Scalable Scientific Data Analysis. Stay tuned!

If I can get some help putting together the materials, I can lead tutorial. @kmpaul and I are already preparing an Xarray tutorial for April (link) so really we just need to adapt what we come up with for that to be useful for Scipy.