issue_comments

I guess that by your definition it would be suggested?

Correct.

Is rasterio going to make it to debian as an apt package?

That's already the case, see: https://packages.debian.org/source/sid/rasterio

What about toolz used by auto_combine?

Well, I am considering (perhaps wrongly) what's in xarray/__init__.py as the public API of xarray. With that logic in mind, any unconditional import should be treated as an install dependency. Otherwise, users will indeed get an ImportError which is not very good.

Back to the original topic, should toolz and rasterio be strict dependencies, recommended or suggested then?

And open_rasterio is also exposed by xarray/__init__.py and imports rasterio unconditionally. So again, that makes rasterio an install_requires too.

Indeed. xarray/__init__.py does import auto_combine from xarray/core/combine.py, so toolz should be listed as install_depends I guess.

And what about toolz which is used in xarray/core/combine.py?

So, just to confirm, the recommended dependencies are just dask, netCDF4 and bottleneck, right? The rest should be suggested.

What about cyordereddict? It is listed in the docs but its description says it is obsolete.

And it looks like most projects in the Debian archive using pytest_addoption have it declared either as a plugin or inside the tests folder, as the note in the pytest documentation says.

I can get the tests to work at build time

By adding conftest.py manually if that was not clear.

I can get the tests to work at build time, but the CI tests using --pyargs (to test against the installed library under site-packages) no longer do, i.e. I get a bunch of ValueError: no option named '--run-flaky'

There is a special note for pytest_addoption which might explain the problem.

Another alternative is to make the test-suite robust to its absence, and not bother to packaging it, given that it is mostly useful for development purposes.

If by packaging you mean distributing here, then I am very disappointed by your statement. From a package maintainer's perspective, I would consider the lack of a test suite a severe regression, should the latter be removed from source distribution in the future.

I would also remind you that it is that very same test suite that lead to the triage of the regressions with bottleneck whilst I was bootstrapping the Debian packaging effort.

Please consider fixing the current test suite or reverting to the old design. The fact that conftest.py does not work reliably (see the bug linked above) should be a sign that the decision to jump to this design happened too soon.

Unfortunately, even if I patch the code base to add the missing file, the tests cannot be run due to the following bug: https://github.com/pytest-dev/pytest/issues/1889

Too bad you guys decided to change the design of the test suite for a point release :-(

I really fail to understand how much of a bug deal it is to introduce a NOTICE file with the right content and keep the LICENSE file and the rest of the code base unchanged. It cannot get any simpler than this.

Anyway, your commit provided the missing copyright information I was seeking. For the rest, I gave you a piece of advice for easy compliance (the NOTICE file) and you are free to follow or disregard it. Moving on.

FYI, xarray is now available in Debian.

You meant MANIFEST.in right?

The complex bit of code embedding is that both MIT and BSD have a clause saying that redistribution from source (even partially) should keep the original copyright notice (which includes authors, dates and license summary). It is straightforward to do when the original file is untouched, however things get a bit more difficult when portions of code gets included in larger modules, or so-called "compatibility" modules, where there is an aggregate of code with mixed copyright.

That's why the NOTICE file is useful. You just have to aggregate all copyright information in this one file and you are guaranteed to comply. Besides, commercial entities or packagers like myself, who need to survey the compliance of each software, have an easier time doing their work.

Did my explanation make sense?

Most Apache-licensed software follow these guidelines, nonetheless. And for the case of xarray, where there are instances of embedded code from other copyrighted projects, I believe it should be beneficial to follow these conventions too. The final decision is yours at the end.

Regardless of your decision, I need a comprehensive list of all embedded code which falls under a different copyright than xarray.

Thanks,

Probably we should include text of each license at the appropriate locations, or at least a direct reference to the fact hat it has a different license.

That is indeed the purpose of the NOTICE file. Since the Apache license state that all source code should start with the Apache copyright header, then individual copyrights from embedded portions of code should be listed in an additional NOTICE file to comply with their respective redistribution clauses.

I'm not opposed to adding a NOTICE file

You do know you are supposed to provide one as per the Apache Software Policy, right? Details about its content and formatting are provided here.

I am also providing an example from the ITK project, since you seem to be having doubts about my statement. I can give you many more if your are still not convinced.

but I haven't seen it in other projects in the scientific Python space

Because most projects from the scientific Python stack (including Numpy and Pandas that you mentioned) use the simpler BSD-3-Clause.

Maybe we're all just being careless, though :)

Nope, just yourself in this case.

Re-opening. Debian now has a version of Numpy with the fix which broke bottleneck. However, the tests still do not pass with the following log: ``` =================================== FAILURES =================================== ___ TestDataArray.test_groupby_apply_center ______

self = <xarray.tests.test_dataarray.TestDataArray testMethod=test_groupby_apply_center>

def test_groupby_apply_center(self):
    def center(x):
        return x - np.mean(x)

    array = self.make_groupby_example_array()
    grouped = array.groupby('abc')

    expected_ds = array.to_dataset()
    exp_data = np.hstack([center(self.x[:, :9]),
                          center(self.x[:, 9:10]),
                          center(self.x[:, 10:])])
    expected_ds['foo'] = (['x', 'y'], exp_data)
    expected_centered = expected_ds['foo']

  self.assertDataArrayAllClose(expected_centered, grouped.apply(center))

xarray/tests/test_dataarray.py:1495:

xarray/tests/init.py:169: in assertDataArrayAllClose assert_allclose(ar1, ar2, rtol=rtol, atol=atol) xarray/testing.py:125: in assert_allclose assert_allclose(a.variable, b.variable)

a = <xarray.Variable (x: 10, y: 20)> array([[ 1.086411e-01, -4.940766e-01, -3.4...e-01, 3.025169e-01, -4.970776e-03, -2.526321e-02, 4.464467e-01]]) b = <xarray.Variable (x: 10, y: 20)> array([[ 1.086411e-01, -4.940766e-01, -3.4...e-01, 3.025169e-01, -4.970776e-03, -2.526321e-02, 4.464467e-01]]) rtol = 1e-05, atol = 1e-08, decode_bytes = True

def assert_allclose(a, b, rtol=1e-05, atol=1e-08, decode_bytes=True):
    """Like :py:func:`numpy.testing.assert_allclose`, but for xarray objects.

    Raises an AssertionError if two objects are not equal up to desired
    tolerance.

    Parameters
    ----------
    a : xarray.Dataset, xarray.DataArray or xarray.Variable
        The first object to compare.
    b : xarray.Dataset, xarray.DataArray or xarray.Variable
        The second object to compare.
    rtol : float, optional
        Relative tolerance.
    atol : float, optional
        Absolute tolerance.
    decode_bytes : bool, optional
        Whether byte dtypes should be decoded to strings as UTF-8 or not.
        This is useful for testing serialization methods on Python 3 that
        return saved strings as bytes.

    See also
    --------
    assert_identical, assert_equal, numpy.testing.assert_allclose
    """
    import xarray as xr
    ___tracebackhide__ = True  # noqa: F841
    assert type(a) == type(b)
    if isinstance(a, xr.Variable):
        assert a.dims == b.dims
        allclose = _data_allclose_or_equiv(a.values, b.values,
                                           rtol=rtol, atol=atol,
                                           decode_bytes=decode_bytes)

      assert allclose, '{}\n{}'.format(a.values, b.values)

E AssertionError: [[ 1.08641053e-01 -4.94076627e-01 -3.45099073e-01 2.39968246e-01 E 1.70028797e-04 -3.54496330e-01 2.22851030e-01 3.72358514e-01 E -1.43769756e-01 -2.74687658e-01 -2.45407871e-01 2.15556579e-01 E -4.03911607e-01 3.87184795e-01 -4.18200302e-01 -4.06039565e-01 E -3.83684191e-01 -2.96458046e-01 4.74874038e-02 1.26236739e-01] E [ 3.61580852e-01 -4.79206943e-02 -2.66600721e-01 1.55428994e-01 E 2.61949104e-01 -2.14716556e-01 -3.06136650e-01 1.22278069e-01 E -6.66015156e-02 -1.69577684e-01 3.87460083e-01 1.76306072e-01 E -4.44331607e-01 -5.25418189e-01 1.02862502e-01 -2.85689326e-02 E -2.02277044e-02 9.84154078e-02 2.13794060e-01 3.01328151e-01] E [ 2.95736742e-01 4.25678417e-02 1.09718836e-01 -4.88189636e-01 E 4.03312049e-02 -2.29350832e-01 4.14023579e-01 2.97767319e-02 E -2.67128794e-01 3.21459151e-01 3.17043658e-01 -2.23716814e-01 E -2.66104828e-01 -3.28529266e-01 8.57802322e-02 -3.77281481e-01 E 3.19405419e-01 -2.14206353e-01 1.44674807e-01 -3.10715376e-01] E [ 2.81673580e-01 2.73754200e-03 -3.38780740e-01 -3.34309419e-01 E -3.53325764e-01 3.60925598e-01 -1.82147600e-01 4.78926180e-01 E -5.15467328e-01 -3.67560456e-02 -4.19994747e-01 -1.07446917e-01 E 2.66933947e-01 -1.08693646e-01 4.91030682e-03 2.77558294e-01 E -1.86235207e-01 2.48140256e-01 4.43512756e-02 1.26844065e-01] E [ 3.01363974e-01 -1.09877347e-01 1.22309126e-01 4.31480566e-01 E 4.76386916e-01 2.85348322e-01 2.76157706e-02 4.35295440e-01 E 1.70634220e-02 -1.45109715e-01 3.74386086e-01 -3.66322521e-01 E -3.46253595e-02 -2.91631220e-01 -2.43084987e-01 2.23311739e-01 E -4.16467323e-01 2.14347435e-02 -3.60138536e-01 -9.76142770e-02] E [ -2.41033567e-01 -3.46475587e-01 -1.15366138e-01 4.41278309e-01 E -1.18403292e-01 2.82655028e-01 4.28244101e-01 -1.79257638e-01 E -1.03689353e-01 -2.18645467e-02 4.46969306e-02 5.81598169e-02 E 2.65938587e-01 1.92085853e-01 -4.32495899e-01 -2.24346584e-01 E 5.34276468e-02 5.48206231e-02 -5.89989991e-03 -3.15437391e-01] E [ -1.62051319e-01 -9.45765818e-02 -5.03109630e-01 4.17684367e-01 E -3.06186685e-01 -1.46334025e-01 -9.56430059e-02 -4.25805199e-01 E 1.03351228e-01 4.13053193e-01 -1.06769941e-01 3.80051241e-01 E -2.82681942e-01 -5.40183672e-02 -1.20729983e-01 2.51349129e-01 E 3.98594071e-01 -1.25509013e-02 -2.20384829e-01 3.53787835e-01] E [ 8.63994358e-02 3.30289251e-01 3.34951245e-01 4.74269877e-01 E -3.76742368e-01 -1.06150098e-01 2.75894947e-01 2.58433673e-01 E 1.72780099e-01 -1.23719179e-01 1.87148326e-01 2.30802594e-02 E -1.91651687e-02 -1.24840109e-01 -3.47666075e-02 1.94775931e-01 E 1.57854118e-01 2.36980122e-01 4.19620283e-01 3.92725774e-01] E [ -1.20034155e-01 2.87247357e-01 3.46581678e-02 -2.91391233e-01 E -1.41828417e-01 -1.02849154e-01 -2.22434220e-01 5.88958162e-02 E 5.10373849e-02 8.00973720e-02 -4.20339680e-01 3.77197946e-02 E -1.19250042e-01 1.89674423e-01 -3.08228692e-02 2.09467152e-01 E -2.95852048e-01 3.88376081e-02 -7.35402315e-02 2.08679192e-01] E [ -5.81460294e-02 -4.49176762e-01 1.06408912e-01 4.43941698e-01 E -4.43932980e-01 8.99339636e-02 -1.99292909e-01 -4.60542036e-01 E 1.91585641e-01 -4.28948878e-02 -4.56597716e-01 1.85430017e-01 E 3.29358074e-01 1.29811129e-01 2.80559136e-01 3.70777734e-01 E 3.02516908e-01 -4.97077643e-03 -2.52632109e-02 4.46446732e-01]] E [[ 1.08641053e-01 -4.94076627e-01 -3.45099073e-01 2.39968246e-01 E 1.70028797e-04 -3.54496330e-01 2.22851030e-01 3.72358514e-01 E -1.43769756e-01 1.03282389e-01 -2.45407871e-01 2.15556579e-01 E -4.03911607e-01 3.87184795e-01 -4.18200302e-01 -4.06039565e-01 E -3.83684191e-01 -2.96458046e-01 4.74874038e-02 1.26236739e-01] E [ 3.61580852e-01 -4.79206943e-02 -2.66600721e-01 1.55428994e-01 E 2.61949104e-01 -2.14716556e-01 -3.06136650e-01 1.22278069e-01 E -6.66015156e-02 2.08392364e-01 3.87460083e-01 1.76306072e-01 E -4.44331607e-01 -5.25418189e-01 1.02862502e-01 -2.85689326e-02 E -2.02277044e-02 9.84154078e-02 2.13794060e-01 3.01328151e-01] E [ 2.95736742e-01 4.25678417e-02 1.09718836e-01 -4.88189636e-01 E 4.03312049e-02 -2.29350832e-01 4.14023579e-01 2.97767319e-02 E -2.67128794e-01 6.99429199e-01 3.17043658e-01 -2.23716814e-01 E -2.66104828e-01 -3.28529266e-01 8.57802322e-02 -3.77281481e-01 E 3.19405419e-01 -2.14206353e-01 1.44674807e-01 -3.10715376e-01] E [ 2.81673580e-01 2.73754200e-03 -3.38780740e-01 -3.34309419e-01 E -3.53325764e-01 3.60925598e-01 -1.82147600e-01 4.78926180e-01 E -5.15467328e-01 3.41214002e-01 -4.19994747e-01 -1.07446917e-01 E 2.66933947e-01 -1.08693646e-01 4.91030682e-03 2.77558294e-01 E -1.86235207e-01 2.48140256e-01 4.43512756e-02 1.26844065e-01] E [ 3.01363974e-01 -1.09877347e-01 1.22309126e-01 4.31480566e-01 E 4.76386916e-01 2.85348322e-01 2.76157706e-02 4.35295440e-01 E 1.70634220e-02 2.32860332e-01 3.74386086e-01 -3.66322521e-01 E -3.46253595e-02 -2.91631220e-01 -2.43084987e-01 2.23311739e-01 E -4.16467323e-01 2.14347435e-02 -3.60138536e-01 -9.76142770e-02] E [ -2.41033567e-01 -3.46475587e-01 -1.15366138e-01 4.41278309e-01 E -1.18403292e-01 2.82655028e-01 4.28244101e-01 -1.79257638e-01 E -1.03689353e-01 3.56105501e-01 4.46969306e-02 5.81598169e-02 E 2.65938587e-01 1.92085853e-01 -4.32495899e-01 -2.24346584e-01 E 5.34276468e-02 5.48206231e-02 -5.89989991e-03 -3.15437391e-01] E [ -1.62051319e-01 -9.45765818e-02 -5.03109630e-01 4.17684367e-01 E -3.06186685e-01 -1.46334025e-01 -9.56430059e-02 -4.25805199e-01 E 1.03351228e-01 7.91023240e-01 -1.06769941e-01 3.80051241e-01 E -2.82681942e-01 -5.40183672e-02 -1.20729983e-01 2.51349129e-01 E 3.98594071e-01 -1.25509013e-02 -2.20384829e-01 3.53787835e-01] E [ 8.63994358e-02 3.30289251e-01 3.34951245e-01 4.74269877e-01 E -3.76742368e-01 -1.06150098e-01 2.75894947e-01 2.58433673e-01 E 1.72780099e-01 2.54250868e-01 1.87148326e-01 2.30802594e-02 E -1.91651687e-02 -1.24840109e-01 -3.47666075e-02 1.94775931e-01 E 1.57854118e-01 2.36980122e-01 4.19620283e-01 3.92725774e-01] E [ -1.20034155e-01 2.87247357e-01 3.46581678e-02 -2.91391233e-01 E -1.41828417e-01 -1.02849154e-01 -2.22434220e-01 5.88958162e-02 E 5.10373849e-02 4.58067419e-01 -4.20339680e-01 3.77197946e-02 E -1.19250042e-01 1.89674423e-01 -3.08228692e-02 2.09467152e-01 E -2.95852048e-01 3.88376081e-02 -7.35402315e-02 2.08679192e-01] E [ -5.81460294e-02 -4.49176762e-01 1.06408912e-01 4.43941698e-01 E -4.43932980e-01 8.99339636e-02 -1.99292909e-01 -4.60542036e-01 E 1.91585641e-01 3.35075160e-01 -4.56597716e-01 1.85430017e-01 E 3.29358074e-01 1.29811129e-01 2.80559136e-01 3.70777734e-01 E 3.02516908e-01 -4.97077643e-03 -2.52632109e-02 4.46446732e-01]]

xarray/testing.py:123: AssertionError _____ TestDataArray.test_groupby_math ______

self = <xarray.tests.test_dataarray.TestDataArray testMethod=test_groupby_math>

def test_groupby_math(self):
    array = self.make_groupby_example_array()
    for squeeze in [True, False]:
        grouped = array.groupby('x', squeeze=squeeze)

        expected = array + array.coords['x']
        actual = grouped + array.coords['x']
        self.assertDataArrayIdentical(expected, actual)

        actual = array.coords['x'] + grouped
        self.assertDataArrayIdentical(expected, actual)

        ds = array.coords['x'].to_dataset('X')
        expected = array + ds
        actual = grouped + ds
        self.assertDatasetIdentical(expected, actual)

        actual = ds + grouped
        self.assertDatasetIdentical(expected, actual)

    grouped = array.groupby('abc')
    expected_agg = (grouped.mean() - np.arange(3)).rename(None)
    actual = grouped - DataArray(range(3), [('abc', ['a', 'b', 'c'])])
    actual_agg = actual.groupby('abc').mean()

  self.assertDataArrayAllClose(expected_agg, actual_agg)

xarray/tests/test_dataarray.py:1541:

xarray/tests/init.py:169: in assertDataArrayAllClose assert_allclose(ar1, ar2, rtol=rtol, atol=atol) xarray/testing.py:125: in assert_allclose assert_allclose(a.variable, b.variable)

a = <xarray.Variable (abc: 3)> array([ 0.504303, -0.53899 , -1.97593 ]) b = <xarray.Variable (abc: 3)> array([ 0.504303, -0.53899 , -0.175924]) rtol = 1e-05, atol = 1e-08, decode_bytes = True

def assert_allclose(a, b, rtol=1e-05, atol=1e-08, decode_bytes=True):
    """Like :py:func:`numpy.testing.assert_allclose`, but for xarray objects.

    Raises an AssertionError if two objects are not equal up to desired
    tolerance.

    Parameters
    ----------
    a : xarray.Dataset, xarray.DataArray or xarray.Variable
        The first object to compare.
    b : xarray.Dataset, xarray.DataArray or xarray.Variable
        The second object to compare.
    rtol : float, optional
        Relative tolerance.
    atol : float, optional
        Absolute tolerance.
    decode_bytes : bool, optional
        Whether byte dtypes should be decoded to strings as UTF-8 or not.
        This is useful for testing serialization methods on Python 3 that
        return saved strings as bytes.

    See also
    --------
    assert_identical, assert_equal, numpy.testing.assert_allclose
    """
    import xarray as xr
    ___tracebackhide__ = True  # noqa: F841
    assert type(a) == type(b)
    if isinstance(a, xr.Variable):
        assert a.dims == b.dims
        allclose = _data_allclose_or_equiv(a.values, b.values,
                                           rtol=rtol, atol=atol,
                                           decode_bytes=decode_bytes)

      assert allclose, '{}\n{}'.format(a.values, b.values)

E AssertionError: [ 0.5043027 -0.53899037 -1.97592983] E [ 0.5043027 -0.53899037 -0.17592373]

xarray/testing.py:123: AssertionError ----------------------------- Captured stderr call ----------------------------- /<<PKGBUILDDIR>>/.pybuild/pythonX.Y_2.7/build/xarray/tests/test_dataarray.py:1529: FutureWarning: the order of the arguments on DataArray.to_dataset has changed; you now need to supply name as a keyword argument ds = array.coords['x'].to_dataset('X') ___ TestDataArray.testgroupby_sum ____

self = <xarray.tests.test_dataarray.TestDataArray testMethod=test_groupby_sum>

def test_groupby_sum(self):
    array = self.make_groupby_example_array()
    grouped = array.groupby('abc')

    expected_sum_all = Dataset(
        {'foo': Variable(['abc'], np.array([self.x[:, :9].sum(),
                                            self.x[:, 10:].sum(),
                                            self.x[:, 9:10].sum()]).T),
         'abc': Variable(['abc'], np.array(['a', 'b', 'c']))})['foo']
    self.assertDataArrayAllClose(expected_sum_all, grouped.reduce(np.sum))

  self.assertDataArrayAllClose(expected_sum_all, grouped.sum())

xarray/tests/test_dataarray.py:1440:

xarray/tests/init.py:169: in assertDataArrayAllClose assert_allclose(ar1, ar2, rtol=rtol, atol=atol) xarray/testing.py:125: in assert_allclose assert_allclose(a.variable, b.variable)

a = <xarray.Variable (abc: 3)> array([ 45.861725, 46.894773, 4.057272]) b = <xarray.Variable (abc: 3)> array([ 45.861725, 46.894773, 0.877923]) rtol = 1e-05, atol = 1e-08, decode_bytes = True

def assert_allclose(a, b, rtol=1e-05, atol=1e-08, decode_bytes=True):
    """Like :py:func:`numpy.testing.assert_allclose`, but for xarray objects.

    Raises an AssertionError if two objects are not equal up to desired
    tolerance.

    Parameters
    ----------
    a : xarray.Dataset, xarray.DataArray or xarray.Variable
        The first object to compare.
    b : xarray.Dataset, xarray.DataArray or xarray.Variable
        The second object to compare.
    rtol : float, optional
        Relative tolerance.
    atol : float, optional
        Absolute tolerance.
    decode_bytes : bool, optional
        Whether byte dtypes should be decoded to strings as UTF-8 or not.
        This is useful for testing serialization methods on Python 3 that
        return saved strings as bytes.

    See also
    --------
    assert_identical, assert_equal, numpy.testing.assert_allclose
    """
    import xarray as xr
    ___tracebackhide__ = True  # noqa: F841
    assert type(a) == type(b)
    if isinstance(a, xr.Variable):
        assert a.dims == b.dims
        allclose = _data_allclose_or_equiv(a.values, b.values,
                                           rtol=rtol, atol=atol,
                                           decode_bytes=decode_bytes)

      assert allclose, '{}\n{}'.format(a.values, b.values)

E AssertionError: [ 45.86172541 46.89477337 4.05727211] E [ 45.86172541 46.89477337 0.87792268]

xarray/testing.py:123: AssertionError ============= 3 failed, 1161 passed, 341 skipped in 40.75 seconds ============== ```

We'd need to wait for numpy-1.12.1 to be absolutely sure. I don't have time to deploy a dev version of numpy to test.

It turned out to be a bug in numpy 1.12.0, fixed in 1.12.1, which made bottleneck fail. Closing.

Thanks, I'll iterate with the Debian maintainer of bottleneck.

The build process does run Sphinx so only the source suffices. You don't need to ship a build of the documentation.

Indeed wheels should be the preferred mean of installation.

The final release should be some time this week, up to you if you want to wait.

Thanks @shoyer, I used the rc to try out the packaging and give you some initial feedback. See #1208 and #1209.

@shoyer Do you plan to push a release tarball for 0.9.0rc1 on PyPI? I'd prefer to use this version over 0.8.2 as the basis for the initial Debian packaging. I tested both and found 0.9.0rc1 to lead to less errors with the test suite than 0.8.2. I could also wait for 0.9.0 final if its release is planned soon.