issue_comments

@rhkleijn thanks for the huge effort! We just discussed this on our team call and were universally impressed.

FYI I resolved conflicts in GitHub; I may have made a mistake, tests should tell us though

I'll merge this in the next couple of days unless I hear anything! Thanks everyone

Thanks for having a glance @shoyer . Let us know when it's OK to merge!

@rhkleijn I missed your last comment, apologies.

When at some point in the future these tools indeed do catch up it is straightforward to adapt the script and regenerate.

That makes sense, and it sounds like you've checked these cases in lots of detail, thank you for all your hard work. If you want, add a comment to the code that references your comments here, so there's some record of the research you've done. (Optional ofc)

Let's answer @mathause questions, and then you should feel free to merge unless there are any objections — we may learn more from others using it — and very easy to iterate more from here.

Thanks a lot for the full response!

I forgot to reply to one question of yours — let's use this PR, if you want to replace the current code with the gen_ops branch.

It takes precedence only during type checking as the source for static type information.

Ofc, I see, there's no type information in the .py file. Out of interest, was there a reason we don't make a single .py file with the typing info too? (tbc, no need to redo!)

Unfortunately we have to define them both ways for typing purposes.

Thanks for the comments — even better to have them in the code. I still have one question — if we remove a line like this, does mypy not recognize it can use the method on a Dataset?

python class DataArrayOpsMixin: def __add__(self, other: T_Dataset) -> T_Dataset: ...

With NoReturn mypy correctly errors in that case. Without NoReturn mypy incorrectly infers the result to be Variable.

That's surprising — I wonder where it's getting that information from. I would have expected that it wouldn't be able to find any compatible implementation of a GroupBy + Variable.

There seem to be quite some open issues here with no indication they currently are actively working on fixing them. I am not using PyCharm myself, but as it is widely used I would also expect quite some xarray users to be using it.

I would vote not to prioritize a tool if it's not compatible — this code will last for a long time, and I expect PyCharm will catch up to the language specs soon.

For these unary ops with simple type hints if would probably work but for most other methods with multiple overloads the typing ecosystem is not yet mature enough. In this issue I have commented on it.

Great, thank you

The gen-ops branch looks really cool @rhkleijn .

I have a few questions, but they don't need to hold back merging — if others agree — in particular on the .py and .pyi design — we could merge now and then iterate.

• Above you mentioned:

  The mypy documentation states that if a directory contains both a .py and a .pyi file for the same module, the .pyi file takes precedence. It is my understanding that stub files work on a file by file basis and you cannot mix them for a single file.

And here we have both a xarray/core/_typed_ops.pyi and xarray/core/_typed_ops.py. Does that mean the .pyi file is obscuring the .py file? - Do you know the extent to which PyCharm not picking up the @overloads is because PyCharm hasn't implemented a PEP yet, vs. an issue with the proposed code? If PyCharm is going to catch up soon, I don't think we need to engineer around it. - Python typing has run ahead of me recently, so forgive the uninformed question, but do you know whether it's possible to define a parent class to be generic over a type, and avoid repeating methods like this for each Mixin class? python def __neg__(self: T_Dataset) -> T_Dataset: ... def __pos__(self: T_DataArray) -> T_DataArray: ... def __neg__(self: T_Variable) -> T_Variable: ... Or are there only a few of these that are common across the classes, such that it's not worth having a parent class for the Mixins? - Do we need to define methods for the DataArray's ops with a Dataset — e.g. here — or can we rely on Datasets implementation? I think that's how the runtime handles it. - Do we need to define these NoReturn methods? I would have thought that if there weren't a valid type signature we'd get an error anyway.

Using python to template the methods works pretty well! Better than I would have expected relative to a templating library like jinja.

@rhkleijn I'm a bit late to respond here, but that could be really excellent IMO, I'd be really interested to see that working. Let me know if there's anything from my end that would be helpful.

Thanks a lot @rhkleijn , this is very impressive! The results look great. The __init_subclass__ approach seems dominant over our existing approach.

Did you ever consider replacing the injections with this macro-like approach to writing out the python methods, rather than using the macro to generate the stub files? (I don't bring this up with a view that it's better, but the injections have always seemed awkward)

Is there anything we should be aware of around the interactions between stub files and inline definitions?

Others will have useful thoughts too, so would be interested to hear from them too.