issue_comments

Turns out it is a pyodide bug with how they hook in to importlib

I don't think this is a bug in importlib but rather pyodide. They seem to add a hook to importlib (which is allowed, but probably causing this issue?)

Edit: The error message from pyodide seems more like this is intentional? Is scipy really installed correctly?

FYI I'm getting this error when trying to update the version in pyodide: https://app.circleci.com/pipelines/github/pyodide/pyodide/5016/workflows/e2cc0865-a91d-4cdb-8dbc-dfaeeb522c5f/jobs/60731

```python JavascriptException message: Traceback (most recent call last): File "/lib/python3.10/asyncio/futures.py", line 201, in result raise self.exception File "/lib/python3.10/asyncio/tasks.py", line 232, in __step result = coro.send(None) File "/lib/python3.10/_pyodide/_base.py", line 531, in eval_code_async await CodeRunner( File "/lib/python3.10/_pyodide/_base.py", line 357, in run_async coroutine = eval(self.code, globals, locals) File "<exec>", line 1, in <module> File "/lib/python3.10/site-packages/xarray/__init__.py", line 1, in <module> from . import testing, tutorial File "/lib/python3.10/site-packages/xarray/tutorial.py", line 17, in <module> from .backends.api import open_dataset as _open_dataset File "/lib/python3.10/site-packages/xarray/backends/__init__.py", line 16, in <module> from .scipy import ScipyBackendEntrypoint, ScipyDataStore File "/lib/python3.10/site-packages/xarray/backends/scipy_.py", line 239, in <module> class ScipyBackendEntrypoint(BackendEntrypoint): File "/lib/python3.10/site-packages/xarray/backends/scipy_.py", line 260, in ScipyBackendEntrypoint available = module_available("scipy") File "/lib/python3.10/site-packages/xarray/core/utils.py", line 1009, in module_available return importlib.util.find_spec(module) is not None File "/lib/python3.10/importlib/util.py", line 103, in find_spec return _find_spec(fullname, parent_path) File "/lib/python3.10/_pyodide/_importhook.py", line 226, in find_spec raise ModuleNotFoundError( ModuleNotFoundError: The module 'scipy' is included in the Pyodide distribution, but it is not installed. You can install it by calling: await micropip.install("scipy") in Python or await pyodide.loadPackage("scipy") in JavaScript. See https://pyodide.org/en/stable/usage/loading-packages.html for more details.

````

this seems like a bug in importlib

Exciting improvements on usability for the next version!

Thanks @headtr1ck great PR!

It seems that cfgrib has its own xarray_plugin now. Should the internal version be deprecated?

I managed to get dask.array lazy as well, but dask.utils is still imported by backends.locks.

Someone else has to tackle this, I am not too familiar with this...

This got the import time quite low. I am happy with this now :)

Seems like it is but a good idea to mess with sys.modules in pytest. Anyone got an idea how to solve that?

Edit: solved :)

Now I have made every module lazy that is not too much work, this includes: matplotlib, flox and all backends.

The only remaining large import is dask.array, here I have to check if that is possible to make lazy? This alone is 500ms import time, since it also imports numba, scipy and sparse... Maybe we could even open a ticket at dask about this topic as well?

The simplest solution would be to add a simple unit test that checks that blacklisted packets are not in sys.modules.

For import timings we have asv, but that is not run so often.

Do you think there's a risk something gets added back by mistake?

A galaxy brain idea is to have a lint in pre-commit which looks for ^import (foo|bar) for the libraries we should be lazy loading. Not fool proof. Or something to measure import time so we can see if it rises again.

Not as much of an improvement as I thought, but shaving off another 20ms.

I can see if I can bring it down even further...

Very nice @headtr1ck !