issue_comments

Table actions

9 rows where issue = 1284475176 sorted by updated_at descending

✎ View and edit SQL

This data as json, CSV (advanced)

Suggested facets: reactions, created_at (date), updated_at (date)

author_association 3

issue 1

  • Long import time · 9 ✖
id html_url issue_url node_id user created_at updated_at ▲ author_association body reactions performed_via_github_app issue
1280072309 https://github.com/pydata/xarray/issues/6726#issuecomment-1280072309 https://api.github.com/repos/pydata/xarray/issues/6726 IC_kwDOAMm_X85MTFp1 hmaarrfk 90008 2022-10-16T22:33:17Z 2022-10-16T22:33:17Z CONTRIBUTOR

In developing https://github.com/pydata/xarray/pull/7172, there are also some places where class types are used to check for features: https://github.com/pydata/xarray/blob/main/xarray/core/pycompat.py#L35

Dask and sparse and big contributors due to their need to resolve the class name in question.

Ultimately. I think it is important to maybe constrain the problem.

Are we ok with 100 ms over numpy + pandas? 20 ms?

On my machines, the 0.5 s that xarray is close to seems long... but everytime I look at it, it seems to "just be a python problem".

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Long import time 1284475176
1265721162 https://github.com/pydata/xarray/issues/6726#issuecomment-1265721162 https://api.github.com/repos/pydata/xarray/issues/6726 IC_kwDOAMm_X85LcV9K headtr1ck 43316012 2022-10-03T16:30:25Z 2022-10-03T16:30:25Z COLLABORATOR

Nice. Does it work on python 3.8?

according to the docu it exists since 3.4.

 
{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Long import time 1284475176
1265717585 https://github.com/pydata/xarray/issues/6726#issuecomment-1265717585 https://api.github.com/repos/pydata/xarray/issues/6726 IC_kwDOAMm_X85LcVFR dcherian 2448579 2022-10-03T16:27:42Z 2022-10-03T16:27:42Z MEMBER

his could be replaced by importlib.util.find_spec.

Nice. Does it work on python 3.8?

However, many backends also check for ImportErrors (not ModuleNotFoundError) that occur when a library is not correctly installed. I am not sure if in this case the backend should simply be disabled like it is now (At least cfgrib is raising a warning instead)?

Would it be a problem if this error is only appearing when actually trying to open a file

Sounds OK to error when trying to use the backend.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Long import time 1284475176
1261491296 https://github.com/pydata/xarray/issues/6726#issuecomment-1261491296 https://api.github.com/repos/pydata/xarray/issues/6726 IC_kwDOAMm_X85LMNRg headtr1ck 43316012 2022-09-28T21:37:30Z 2022-09-28T21:38:37Z COLLABORATOR

I just checked, many backends are importing their external dependencies at module level with a try-except block. This could be replaced by importlib.util.find_spec.

However, many backends also check for ImportErrors (not ModuleNotFoundError) that occur when a library is not correctly installed. I am not sure if in this case the backend should simply be disabled like it is now (At least cfgrib is raising a warning instead)? Would it be a problem if this error is only appearing when actually trying to open a file? If that is the case, we could move to lazy external lib loading for the backends.

Not sure how much it actually saves, but should be ~0.2s (at least on my machine, but depends on the number of intalled backends, the fewer are installed the faster the import should be).

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Long import time 1284475176
1257279640 https://github.com/pydata/xarray/issues/6726#issuecomment-1257279640 https://api.github.com/repos/pydata/xarray/issues/6726 IC_kwDOAMm_X85K8JCY headtr1ck 43316012 2022-09-25T21:05:38Z 2022-09-25T21:05:38Z COLLABORATOR

I think we could rework our backend solution to do the imports lazy: To check if a file might be openable via some backend we usually do not need to import its dependency module.

 
{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Long import time 1284475176
1223863010 https://github.com/pydata/xarray/issues/6726#issuecomment-1223863010 https://api.github.com/repos/pydata/xarray/issues/6726 IC_kwDOAMm_X85I8qri eendebakpt 883786 2022-08-23T10:17:46Z 2022-08-23T10:17:46Z CONTRIBUTOR

Some other projects are considering lazy imports as well: https://scientific-python.org/specs/spec-0001/

 
{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Long import time 1284475176
1200279262 https://github.com/pydata/xarray/issues/6726#issuecomment-1200279262 https://api.github.com/repos/pydata/xarray/issues/6726 IC_kwDOAMm_X85His7e mathause 10194086 2022-07-30T19:12:06Z 2022-07-30T19:12:20Z MEMBER

I just had another look at this using bash python -X importtime -c "import llvmlite" 2> import.log and tuna for the visualization.

  • pseudoNETCDF adds quite some overhead, but I think only few people have this installed (could be made faster, but not sure if worth it)
  • llmvlite (required by numba) seems the last dependency relying on pkg_resources but this is fixed in the new version which should be out soonish
  • dask recently merged a PR that avoids a slow import dask/dask/pull/9230 (from which we should profit)

This should bring it down a bit by another 0.25 s, but I agree it would be nice to have it even lower.

 
{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Long import time 1284475176
1166335744 https://github.com/pydata/xarray/issues/6726#issuecomment-1166335744 https://api.github.com/repos/pydata/xarray/issues/6726 IC_kwDOAMm_X85FhN8A headtr1ck 43316012 2022-06-25T18:04:43Z 2022-06-25T18:04:43Z COLLABORATOR

Useful for debugging: python -X importtime -c "import xarray"

 
{
    "total_count": 1,
    "+1": 1,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Long import time 1284475176
1166307431 https://github.com/pydata/xarray/issues/6726#issuecomment-1166307431 https://api.github.com/repos/pydata/xarray/issues/6726 IC_kwDOAMm_X85FhHBn mathause 10194086 2022-06-25T15:10:03Z 2022-06-25T15:10:03Z MEMBER

Thanks for the report. I think one resaon is that we import all the io libraries non-lazy (I think since the backend refactor). And many of the dependecies still use pkg_resources instead of importlib.metadata (which is considetably slower).

We'd need to take a look at the lazy loader. 

{
    "total_count": 0,
    "+1": 0,
    "-1": 0,
    "laugh": 0,
    "hooray": 0,
    "confused": 0,
    "heart": 0,
    "rocket": 0,
    "eyes": 0
}
   Long import time 1284475176

Advanced export

JSON shape: default, array, newline-delimited, object

CSV options: 

CREATE TABLE [issue_comments] (
   [html_url] TEXT,
   [issue_url] TEXT,
   [id] INTEGER PRIMARY KEY,
   [node_id] TEXT,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [updated_at] TEXT,
   [author_association] TEXT,
   [body] TEXT,
   [reactions] TEXT,
   [performed_via_github_app] TEXT,
   [issue] INTEGER REFERENCES [issues]([id])
);
CREATE INDEX [idx_issue_comments_issue]
    ON [issue_comments] ([issue]);
CREATE INDEX [idx_issue_comments_user]
    ON [issue_comments] ([user]);