issue_comments

I can see that the conda-forge-feedstock has been updated; however requirements.txt still contains the pin

All blockers to pandas 2 linked in this issue have been merged; is there anything outstanding?

It just occurred to me that xarray dropped Python 3.8 3 months before what NEP-29 recommends. I think this is a problem. Let's continue this discussion on #7777.

This also breaks xarray on ubuntu 20.04 which ships with Python 3.8 and is supported until April 2025. Python 3.8 is also supported at least until October 2024.

Not investing effort to support 5 years old dependencies was a very conscious decision. This is not something unique we do; we simply adhere to NEP29: https://numpy.org/neps/nep-0029-deprecation_policy.html

If for whatever reason you want to use python shipped by ubuntu 20.04, as opposed to conda/venv/poetry/whatever, you should also be prepared to stick to older versions of the python packages. Note that 5 years is the duration of security support. I'm not personally aware of security issues in xarray since Python 3.8 was abandoned (I've been a bit out of the loop and I could be proven wrong), but in the unlikely event that one should arise in xarray, we would consider a backport to Python 3.8.

I assume you have given this a lot of thought, but imho the minimum dependency versions should be decided according to features needed, not timing.

It's not based on timing. The policy is there so that, when a developer finds that they have to do extra labour to support an old version of a dependency, they can instead drop the support for the old version without needing to seek approval from the maintainers.

python new_data_future = xr.apply_ufunc( _copy_test, data, a_x, ... ) instead of using kwargs.

I've opened https://github.com/dask/distributed/issues/7140 to simplify this. With it implemented, my snippet python test = np.full((20,), 30) a = da.from_array(test) dsk = client.scatter(dict(a.dask), broadcast=True) a = da.Array(dsk, name=a.name, chunks=a.chunks, dtype=a.dtype, meta=a._meta, shape=a.shape) a_x = xarray.DataArray(a, dims=["new_z"]) would become python test = np.full((20,), 30) a_x = xarray.DataArray(test, dims=["new_z"]).chunk() a_x = client.scatter(a_x)

Having said the above, your design is... contrived.

There isn't, as of today, a straightforward way to scatter a local dask collection (persist() will push the whole thing through the scheduler and likely send it out of memory).

Workaround: python test = np.full((20,), 30) a = da.from_array(test) dsk = client.scatter(dict(a.dask), broadcast=True) a = da.Array(dsk, name=a.name, chunks=a.chunks, dtype=a.dtype, meta=a._meta, shape=a.shape) a_x = xarray.DataArray(a, dims=["new_z"]) Once you have a_x, you just pass it to the args (not kwargs) of apply_ufunc.

This is still an issue. I noticed that the documentation of map_blocks states: kwargs (mapping) – Passed verbatim to func after unpacking. xarray objects, if any, will not be subset to blocks. Passing dask collections in kwargs is not allowed.

Is this the case for apply_ufunc as well?

test_future is not a dask collection. It's a distributed.Future, which points to an arbitrary, opaque data blob that xarray has no means to know about.

FWIW, I could reproduce the issue, where the future in the kwargs is not resolved to the data it points to as one would expect. Minimal reproducer:

```python import distributed import xarray

client = distributed.Client(processes=False) x = xarray.DataArray([1, 2]).chunk() test_future = client.scatter("Hello World")

def f(d, test): print(test) return d

y = xarray.apply_ufunc( f, x, dask='parallelized', output_dtypes="float64", kwargs={'test':test_future}, ) y.compute() `` Expected print output:Hello WorldActual print output: <Future: finished, type: str, key: str-b012273bcde56eadf364cd3ce9b4ca26>`

Lets skip windows for now.

@crusaderky this looks weird:

For some reason counting the number of tasks in the dask graph via len(ds.dask_graph()) raises an Error on Windows.

I think that's the context manager teardown, not the task counting

More than a one-off — I got them multiple times. Though they look like tests that are liable to be flaky...

They are both tests that fail on the cleanup of @gen_cluster, and specifically on check_process_leak. However, in both cases, the test itself doesn't spawn any processes. What I think is happening is that something unrelated, at some point before the failing tests, spawned a subprocess that has become unresponsive to SIGTERM. I'm updating check_process_leak to be more aggressive in the cleanup before the test.

I'd like to attend too

Third and final issue is when numpy.broadcast_to is applied to the output of zeros_like: ```

import sparse s = sparse.COO.from_numpy([0, 0, 1, 2]) np.broadcast_to(np.zeros_like(s.todense(), shape=()), (3, )) array([0, 0, 0]) np.broadcast_to(np.zeros_like(s, shape=()), (3, )) ValueError: The data length does not match the coordinates given. len(data) = 0, but 3 coords specified. ```

Ah, shape= was very recently added in 0.12.0. It wasn't there in 0.11.2.

[EDIT] It is not the only problem. Investigating further.

The second issue is that sparse.zeros_like doesn't accept the order= parameter, which is required by the same dask code linked above (it's in the kwargs in dask/wrap.py:133). This in turn triggers an unfortunate handling of TypeError on behalf of @curry, which obfuscates the exception.

da.zeros_like(a) internally invokes da.zeros(a.shape, meta=a._meta) which internally invokes np.broadcast(np.zeros_like(a._meta, shape=1), a.shape)

The problem is that sparse.zeros_like does not accept the shape= optional parameter, which is new in numpy 1.17.

This is where it gets triggered: https://github.com/dask/dask/blob/85f0b14bd36a5135ce51aeee067b6207374b00c4/dask/array/wrap.py#L128-L168

I don't think dask should write a workaround to this, and it should be just fixed upstream? CC @jrbourbeau for opinion.

Narrowed it down.

```python

import dask.array as da import sparse s = sparse.COO.from_numpy([0, 0, 1, 2]) a = da.from_array(s) z = da.zeros_like(a) z dask.array<zeros_like, shape=(4,), dtype=int64, chunksize=(4,), chunktype=sparse.COO> z.compute() <function _broadcast_trick_inner at 0x7fb8a3874e50> ``` numpy-1.20.1 dask-2021.3.0 sparse-0.11.2

I see now. I got really confused by the matplotlib issue which is what the opening post of this ticket is about. Would it be possible to have the two issues tracked by separate tickets?

I'm a bit lost. The failure in xarray/tests/test_sparse.py::test_chunk doesn't appear anywhere in recent CI runs and I can't reproduce it locally.

The ongoing failures in upstream-dev: FAILED xarray/tests/test_plot.py::TestFacetGrid::test_can_set_norm FAILED xarray/tests/test_plot.py::TestCFDatetimePlot::test_cfdatetime_line_plot FAILED xarray/tests/test_plot.py::TestCFDatetimePlot::test_cfdatetime_pcolormesh_plot FAILED xarray/tests/test_plot.py::TestCFDatetimePlot::test_cfdatetime_contour_plot Aren't in any way related to either sparse or dask; they appear when I upgrade matplotlib from 3.4.3 to 3.5.0b1.

Unless you want to add a "internals" whats-new.rst entry?

I think it may be a bit overkill?

LGTM. Note that the function doesn't align indices. e.g. if you have:

python a = DataArray([0,1,2,3], dims=["x"], coords={"x": [0,10,20,30]}).chunk(3) b = DataArray([0,1,2,3], dims=["x"], coords={"x": [10,30,40,50]}).chunk(2) a, b = unify_chunks(a, b) You'll end up with aligned chunks, but not aligned coords (e.g. both outputs have still values=[0,1,2,3]), which doesn't make much sense. I think this is OK to leave it as it is in this specific case the issue should not do much harm anyway and it would just be a slowdown in most cases; I'd like to hear @dcherian's or @jhamman's opinions though.

Reproduced and reopened at https://github.com/dask/dask/issues/7583. This impacts the threading and sync scheduler. processes and distributed are unaffacted. Closing here, as this issue has nothing to do with xarray.

This is ready for review and merge

@keewis looks like the xarray code is making assumptions on the dask internal implementation, instead of relying on the public interface alone. I'm on it; expect a fix very shortly.

Indeed we can. Strong +1 for having a note in the main xarray documentation too!

@max-sixty definitely unrelated

To confirm, this is still "the oldest version released within the time period", rather than "the version that existed at the start of the time period"

Yes, this hasn't changed.

pandas is really unstable and its API breaks every other version. Extending its support window from 1 to 2 years would be extremely expensive and frustrating to maintain.

Discussion seems to have died down here. Can we get to a consensus and wrap this up? My vote is to simply require setuptools >= 38.4 at runtime (for which PR https://github.com/pydata/xarray/pull/4296 is ready to go).

It's not clear from the OP how they were installing -- i.e. from wheels or source, but if wheels, then pushing teh run time dependency back would fix it.

I don't think we should be discussing a solution that works on wheels but breaks on sources...

Ubuntu 18.04 ships Python 3.6.5 and setuptools 39.0. Ubuntu 16.04 ships Python 3.5 so it's not to be taken into consideration anyway.

My preference would be to say that we support setuptools 30.3 and newer, even if we can't test it

I have tested that setuptools < 36.7 breaks setuptools-scm; the installed version becomes 0.0.0 which in turns breaks any other package that contains a minimum version check (namely, pandas).

Also, I think we agreed when we implemented NEP29 that we should not support Python 3.6.0, but only the latest patch version for any given minor version of a package. Python 3.6.11 (released 1 month ago) is shipped with setuptools 40.6.

Any pip or conda-based environment can trivially upgrade from Python 3.6.0 to 3.6.11. The only users that have problems with getting setuptools >=38.4 (2.5 years old!!!) are those that use /usr/bin/python3 from a very old Linux distribution, which for some reason never got the patch updates of Python, AND expect everything to be compatible with the very latest python packages freshly downloaded from the internet. I mean, seriously?

@max-sixty afraid it is

importlib.resources (available since 3.7) and importlib.metadata (available since 3.8). Both also have backports (importlib-resources and importlib-metadata), so we should be able to get rid of the install-dependency on setuptools.

-1 from me, because dependencies that are only required on a specific Python version are incompatible with noarch conda recipes. This would force us to change conda to build one package for each OS x python version.

I think that xarray should offer a "compatibility test toolkit" to any numpy-like, NEP18-compatible library that wants to integrate with it. Instead of having a module full of tests specifically for pint, one for sparse, one for cupy, one for awkward, etc. etc. etc. those projects could just write a minimal test module like this:

```python import xarray import sparse

xarray.testing.test_nep18_module( sparse, # TODO: lambda to create an array # TODO: list of xfails ) ``` which would automatically expand into a comprehensive suite of tests thanks to pytest parameterize/fixture magic. this would allow developers of numpy-like libraries to just test their package vs what's expected from a generic NEP-18 compliant package.

PR ready for review

I am getting the same error plus another: FileNotFoundError: [Errno 2] No such file or directory: '/Users/crusaderky/PycharmProjects/tmp/dist/t1/distributed/distributed.yaml' [5

Both xarray and distributed work fine with pip install, python setup.py sdist, and python setup.py bdist. Please open a ticket on the pyinstaller board. I tried re-adding the static files in setup.py but it doesn't fix the issue.

I'm opening a PR to stop loading the resource files if you're not running on jupyter notebook. This will allow pure xarray (without dask) to work with pyinstaller as long as you don't display an xarray object in jupyter.

Temporarily pinning pandas=1.0 in https://github.com/pydata/xarray/pull/4296

The key problem in "as-old-as-they-can-be" is that you end up with dependencies that depend on each other and are 1 year apart in release date. Since very frequently other projects are a lot less rigorous with testing vs old dependencies (if they test at all!) that has caused an endless amount of breakages in the past. Testing with all packages as of 1 year ago is a lot less bug-prone and time-wasting.

setuptools-scm doesn't work with setuptools < 36.7 (Nov 2017). The conda metadata is malformed for setuptools < 38.4 (Jan 2018) - it's missing a timestamp which prevents the minimum versions tool from working.

Is everybody happy with >= 38.4?

then you should be testing with-as-old-as-they-can-be versions

We used to do that and we abandoned that policy in favour of the current rolling window, because it made developers (particularly the less experienced ones) waste a considerable amount of effort retaining backwards compatibility with obsolete versions of the dependencies that nobody cared about.

I was surprised to see this in our setup.cfg file, added by @crusaderky in #3628. The version requirement is not documented in our docs.

It is documented: https://xarray.pydata.org/en/stable/installing.html#minimum-dependency-versions

xarray adopts a rolling policy regarding the minimum supported version of its dependencies: [...] all other libraries: 6 months

The requirement is explicitly set in setup.cfg because don't ship what you don't test.

I see no problem in explicitly adding a special case to the policy for setuptools - I guess 24 months should be fine for all? I do not recommend just going back to "whatever the very first version that works" as we were doing before the introduction of the rolling policy.

I'm preparing a PR...

Is it acceptable for a Pint Quantity to always have the Dask collection interface defined (i.e., be a duck Dask array), even when its magnitude (what it wraps) is not a Dask Array?

I think there are already enough headaches with __iter__ being always defined and confusing libraries such as pandas (https://github.com/hgrecco/pint/issues/1128). I don't see why pint should be explicitly aware of dask (except in unit tests)? It should only deal with generic NEP18-compatible libraries (numpy, dask, sparse, cupy, etc.).

How should xarray check for a duck Dask Array?

We should ask the dask team to formalize what defines a "dask-array-like", like they already did with dask collections, and implement their definition in xarray. I'd personally make it "whatever defines a numpy-array-like AND has a chunks method AND the chunks method returns a tuple".

Actually, I'm just bad at reading. Our policy is "the minor version (X.Y) initially published no more than N months ago", which is correctly implemented by the code.

I think we might want to change the policy, but that's a different matter....

Yes, the policy works well in NEP-29 for numpy and python, but can be problematic for seldom-updated packages

Still relevant

@nbren12 it seems to me that mypy is being overly aggressive when parsing the hinted code (hence why I had to put # type: ignore on it) but it is being more lax when the same code is invoked somewhere else like in my test script. Overall I suspect it may be fragile and break in future mypy versions...

I took the liberty to rework it, please have a look Test script:

```python from typing import Hashable, Mapping import xarray ds: xarray.Dataset

class D(Hashable, Mapping): def hash(self): ... def getitem(self, item): ... def iter(self): ... def len(self): ...

reveal_type(ds["foo"]) reveal_type(ds[["foo", "bar"]]) reveal_type(ds[{}]) reveal_type(ds[D()]) mypy output: t1.py:12: note: Revealed type is 'xarray.core.dataarray.DataArray' t1.py:13: note: Revealed type is 'xarray.core.dataset.Dataset' t1.py:14: note: Revealed type is 'xarray.core.dataset.Dataset' t1.py:15: note: Revealed type is 'xarray.core.dataset.Dataset' ```

LGTM, ready to merge whenever

I went through everything and it seems all fine. I'm happy to merge as soon as you add a line to the What's New

LGTM - can't wait for the tool to mature!

LGTM. I'm the author of the upstream change - apologies, I did not expect the gen_cluster decorator to be used in downstream projects.

What you're asking for has two huge blocker dependencies:

• Type annotations for numpy: https://github.com/numpy/numpy/issues/7370
• The ability to define TypedDict-like annotations for a generic MutableMapping subclass. Such a feature should be suggested to the python core devs.

still relevant

@shoyer to me this it would make the most sense to do a union of the inputs: - if a key is present only in one input, it goes to the output - if a key is present in multiple inputs, always take the leftmost

Note how this would be different from how scalar coords are treated; scalar coords are discarded when they arrive from multiple inputs and are mismatched. The reason I don't think it's wise to do the same with attrs is that it could be uncontrollably expensive to compute equality, depending on what people loaded in them. I've personally seen them used as back-references to the whole application framework. Also there's no guarantee that they implement __eq__ or that it returns a bool; e.g. you can't compare two data structures that somewhere inside contain numpy arrays.

Why would you want a .drop_attrs() method? .attrs.clear() will do just fine. I fully agree we should keep attrs by default.

This has nothing to do with to_netcdf or slicing. Your upstream data is broken for the variable specific_humidity_ml. When you invoke to_netcdf(), the remote resources actually holding the variable(s) are requested for the first time, and the server is responding 404.

Just load the variable into memory: ```python

fnx.specific_humidity_ml.compute() RuntimeError: NetCDF: file not found ``` Please reopen if you have reason to believe the issue is in the xarray pydap driver and not in the source data.

P.S. all your lines invoking to_netcdf could be replaced with a single command: python fnx[ [ "forecast_reference_time", "p0", "ap", "b", "projection_lambert", "ozone_profile_c", "specific_humidity_ml", ] ].isel(time = slice(0,48)).to_netcdf('meps_out.nc', 'w', format = 'NETCDF4')

@keewis I'll have a look

xr.apply_ufunc(sparse.COO, ds, dask='parallelized')

Still relevant

+1

This also means that either the new array is no longer C-contiguous, or the .unstack() operation has had to copy all the data to rearrange it.

The former. As a core design principle, xarray does not care about dimensions order, and any user code that implicitly relies on it should be considered a bad design. The .transpose() method mostly only exists for when people need to access the numpy .data object directly with a numpy function.

you just need to

1. load up your NetCDF files with xarray.open_mfdataset. This will give you
2. an xarray.Dataset,
3. that wraps around one dask.array.Array per variable,

4. that wrap around one numpy.ndarray (DENSE array) per dask chunk.

5. convert to sparse with xarray.apply_ufunc(sparse.COO, ds). This will give you

6. an xarray.Dataset,
7. that wraps around one dask.array.Array per variable,

8. that wrap around one sparse.COO (SPARSE array) per dask chunk.

9. use xarray.merge or whatever to align and merge

10. you may want to rechunk at this point to obtain less, larger chunks. You can estimate your chunk size in bytes if you know your data density (read my previous email).

11. Do whatever other calculations you want. All operations will produce in output the same data type as point 2.

12. To go back to dense, invoke xarray.apply_ufunc(lambda x: x.todense(), ds) to go back to the format as in (1). This step is only necessary if you have something that won't accept/recognize sparse arrays directly in input; namely, writing to a NetCDF dataset. If your data has not been reduced enough, you may need to rechunk into smaller chunks first in order to fit into your RAM constraints.

Regards

On Tue, 18 Feb 2020 at 13:56, fmfreeze notifications@github.com wrote:

Thank you @crusaderky https://github.com/crusaderky for your input.

I understand and agree with your statements for sparse data files. My approach is different, because within my (hdf5) data files on disc, I have no sparse datasets at all.

But as I combine two differently sampled xarray dataset (initialized by h5py > dask > xarray) with xarrays built-in top-level function "xarray.merge()" (resp. xarray.combine_by_coords()), the resulting dataset is sparse.

Generally that is nice behaviour, because two differently sampled datasets get aligned along a coordinate/dimension, and the gaps are filled by NaNs.

Nevertheless, thos NaN "gaps" seem to need memory for every single NaN. That is what should be avoided. Maybe by implementing a redundant pointer to the same memory adress for each NaN?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/issues/3213?email_source=notifications&email_token=ABPM4MFWF22BFFYDHV6BS2DRDPSHXA5CNFSM4ILGYGP2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMCBWHQ#issuecomment-587471646, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABPM4MHIUWDYX6ZFKRRBIJLRDPSHXANCNFSM4ILGYGPQ .

@tqfjo unrelated. You're comparing the creation of a dataset with 2 variables with the creation of one with 3000. Unsurprisingly, the latter will take 1500x. If your dataset doesn't functionally contain 3000 variables but just a single two-dimensional variable, use xarray.DataArray(ds).

Hi fmfreeze,

> Dask integration enables xarray to scale to big data, only as long as the data has no sparse character. Do you agree on that formulation or am I missing something fundamental?

I don't agree. To my understanding xarray->dask->sparse works very well (save bugs), as long as your data density (the percentage of non-default points) is roughly constant across dask chunks. If it isn't, then you'll have some chunks that consume substantially more RAM and CPU to compute than others. This can be mitigated, if you know in advance where you are going to have more samples, by setting uneven dask chunk sizes. For example, if you have a one-dimensional array of 100k points and you know in advance that the density of non-default samples follows a gaussian or triangular distribution, then it may be wise to have very large chunks at the tails and then get them progressively smaller towards the center, e.g. (30k, 12k, 5k, 2k, 1k, 1k, 2k, 5k, 10k, 30k). Of course, there are use cases where you're going to have unpredictable hotspots; I'm afraid that in those the only thing you can do is size your chunks for the worst case and end up oversplitting everywhere else.

Regards Guido

On Thu, 13 Feb 2020 at 10:55, fmfreeze notifications@github.com wrote:

Thank you all for making xarray and its tight development with dask so great!

As @shoyer https://github.com/shoyer mentioned

Yes, it would be useful (eventually) to have lazy loading of sparse arrays from disk, like we want we currently do for dense arrays. This would indeed require knowing that the indices are sorted.

I am wondering, if creating a lazy & sparse xarray Dataset/DataArray is already possible? Especially when creating the sparse part at runtime, and loading only the data part: Assume two differently sampled - and lazy dask - DataArrays are merged/combined along a coordinate axis into a Dataset. Then the smaller (= less dense) DataVariable is filled with NaNs. As far as I experienced the current behaviour is, that each NaN value requires memory.

That issue might be formulated this way: Dask integration enables xarray to scale to big data, only as long as the data has no sparse character. Do you agree on that formulation or am I missing something fundamental?

A code example reproducing that issue is described here: https://stackoverflow.com/q/60117268/9657367

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/issues/3213?email_source=notifications&email_token=ABPM4MBFBIH7EK4PPAWHRH3RCURJLA5CNFSM4ILGYGP2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELUJNRQ#issuecomment-585668294, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABPM4MCMRVUZXQSDYCAIP3LRCURJLANCNFSM4ILGYGPQ .

Hi Scott,

I can't think of a generic situation where text labels have a numerical weight that is hardcoded to their position on the alphabet, e.g. mean("A", "C") = "B". What one typically does is map the labels (any string) to their (arbitrary) weights, interpolate the weights, and then do a nearest-neighbour interpolation (or floor or ceil, depending on the preference) back to the label. Which is what you described but with the special caveat that your weights are the ASCII codes for your labels.

On Sat, 8 Feb 2020 at 20:43, scottcanoe notifications@github.com wrote:

I'd like to suggest an improvement to enable a repeat-based interpolation mechanism for non-numerical data. In my use case, I have time series data (dim='t'), where each timepoint is associated with a measured variable (e.g., fluorescence) as well as a label indicating the stimulus being presented (e.g., "A"). However, if and when I need to upsample my data, the string-valued stimulus information is lost, and its imperative that the stimulus information is still present when working on the resampled data.

My solution to this problem has been to map the labels to integers, use nearest-neighbor interpolation on the integer-valued representation, and finally map the integers back to labels. (I'm willing to bet there's a name for this technique, but I wasn't able to find it by googling around for it.)

I'm new to xarray, but so far as I can tell this functionality is not provided. More specifically, calling DataArray.interp on a string-valued array results in a type error (<builtins.TypeError: interp only works for a numeric type array. Given <U1.>).

Finally, I'd like to applaud you for your work on xarray. I only wish I had found it sooner!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/pydata/xarray/issues/3763?email_source=notifications&email_token=ABPM4MER3APWULR2QQVFE23RB4KOTA5CNFSM4KR43K22YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IMAS3NA, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABPM4MEFUTJISHNCHFOYEXLRB4KOTANCNFSM4KR43K2Q .

ERROR: Could not find a version that satisfies the requirement setuptools_scm (from versions: none) ERROR: No matching distribution found for setuptools_scm

Should get fixed by changing recipe/meta.yaml: yaml requirements: build: - python >=3.6 - pip - setuptools - setuptools_scm

Ready for review and merge. TODO After merging in master: - check rtd build - check binder

@dcherian I've heavily changed this, please give it a second read and merge if you're happy with it

I have no clue whatsoever what's going on with docs?!? I didn't change it in any way and it's just... freezing?!?

Ok, I've played with it a bit and it's a hard -1 from me.

• I could not find a way to run it locally (correct me if I'm wrong); meaning all PRs will need to go through a million of trial-and-error commits and pushes
• Very heavily overlapping with what we already have (flake8, black, isort, mypy)
• Will trigger on issues that are explictly (made) ok for the other linter tools

If you merge from master you should get all green on the upstream-dev test now

if you merge from master you should get all green on the upstream-dev test now

if you merge from master you should get all green on the upstream-dev test now

I think a unit test for the use case is in order?

setuptools is not a runtime dependency since https://github.com/pydata/xarray/pull/3720 Also, I'm confused - both conda and cpython always come with setuptools preinstalled. How did you manage to get an environment without it?

agree

It works https://github.com/pydata/xarray/network/dependencies

We could increase the support window for seldom-updated packages (read: not dask) to 1 year.

I have no problems with supporting versions older than those mandated by the policy as long as there isn't any major benefit in dropping them...

@max-sixty that's what PULL_REQUEST_TEMPLATE.md is for...

@max-sixty I wouldn't 100% advise using such level of automation for isort. Have a look at the # isort:skip tags in our codebase; they're all cases where isort would otherwise break the code.

Demo isort CI in action: https://github.com/pydata/xarray/pull/3721/checks?check_run_id=406904833

@keewis interesting question. setuptools is technically not part of the Python standard library, but it is part of pip and is always included with the cpython binaries, conda create python=3.6, and pypy. So... I'm not sure?

Because NEP29 states support for all versions in the latest X months, not the most recent version released more than X months ago.

Awesome, everything looks in order. This is now ready for review and merge.

The obvious downside is that anybody with a link to one of the internal pages of our documentation will have the link broken. Also I'm unsure how straightforward it will be to rebuild all of our historical versions.

Very glad to upvote anything that rids us of the RTD CI!

@keewis done

I seem to have no way to test RTD >_<

https://readthedocs.org/projects/crusaderky-xarray/builds/10305870/

I need to retest the RTD CI before merging

Related: #3714

I've been using setuptools-scm in multiple other projects and it's great!

I think we're very close to 0.15...

fixed

pickle should never be used, with any library, as a means of long-term storage because it intrinsically relies on implementation details to remain the same across versions. Please use one of the several tools that are made just for that purpose (NetCDF et al.). xarray already guarantees stability on the public API - we typically have a deprecation cycle lasting at least 1 major version. It will never offer stability on the implementation details.

Let us know if you find evidence of xarray-specific problematic behaviour on the matter

I feel this should be opened on the pandas board? xarray should be better off not hacking around the deficiencies of its dependencies...

Thank you!

Looks straightforward - could you open a PR?

In it goes *ducks for cover*

Implemented @shoyer 's suggestions and aligned to master. Merging into master tomorrow!