issue_comments

I was able to reproduce this issue in a Docker container using the s390x Debian image. After a little experimentation I narrowed it down to the following minimal example: ```

import numpy as np; import pandas as pd np.version '1.23.3' pd.version '1.4.4' pd.Series(np.array([1]).astype("<M8[h]")) Traceback (most recent call last): File "\<stdin>", line 1, in \<module> File "/usr/local/lib/python3.9/dist-packages/pandas/core/series.py", line 451, in __init__ data = sanitize_array(data, index, dtype, copy) File "/usr/local/lib/python3.9/dist-packages/pandas/core/construction.py", line 570, in sanitize_array subarr = _try_cast(data, dtype, copy, raise_cast_failure) File "/usr/local/lib/python3.9/dist-packages/pandas/core/construction.py", line 729, in _try_cast return sanitize_to_nanoseconds(arr, copy=copy) File "/usr/local/lib/python3.9/dist-packages/pandas/core/dtypes/cast.py", line 1717, in sanitize_to_nanoseconds values = conversion.ensure_datetime64ns(values) File "pandas/_libs/tslibs/conversion.pyx", line 257, in pandas._libs.tslibs.conversion.ensure_datetime64ns File "pandas/_libs/tslibs/np_datetime.pyx", line 120, in pandas._libs.tslibs.np_datetime.check_dts_bounds pandas._libs.tslibs.np_datetime.OutOfBoundsDatetime: Out of bounds nanosecond timestamp: 8220291319602-05-05 16:00:00 This confirms it is an upstream issue. Interestingly if we use the native byte order (big-endian on this architecture) for the dtype, this example works: pd.Series(np.array([1]).astype("M8[h]")) 0 1970-01-01 01:00:00 dtype: datetime64[ns] or more explicitly pd.Series(np.array([1]).astype(">M8[h]")) 0 1970-01-01 01:00:00 dtype: datetime64[ns] ``` It appears the inverse of this issue (big-endian dtype leading to a failure on a little-endian system) came up once in pandas: https://github.com/pandas-dev/pandas/issues/29684. @amckinstry I'm not sure what it will take to fix this issue in pandas, but you are welcome to open an issue there. They may also have a difficult time reproducing and testing this, however (https://github.com/pandas-dev/pandas/pull/30976#issuecomment-573989082).

Interesting. Thanks for checking that #6988 indeed solves this. I went ahead and merged it, but when I get a chance I’ll keep trying to track down the root cause of this issue.

Both code fragments work on s390x as described. This is for pandas 1.4.3 and xarray 2022.06.0. The commit in #6988 solves the issue.

Thanks @amckinstry. I guess my last try to produce a pandas minimal example might be: ```

import numpy as np; import pandas as pd pd.Series(np.array([np.int64(1000000).astype("<M8[h]")])) 0 2084-01-29 16:00:00 dtype: datetime64[ns] or potentially more simply: import numpy as np; import pandas as pd pd.Series(np.int64(1000000).astype("<M8[h]")) 0 2084-01-29 16:00:00 dtype: datetime64[ns] `` Somewhere something is going wrong in converting a non-nanosecond-precision datetime value to a nanosecond-precision one (maybe the cast to apd.Timestamp` in my earlier example was short-circuiting this).

I think #6988 should likely work around this issue at least on the xarray side, since it passes datetime64[ns] values into the DataArray constructor immediately. It also seems like the function where the error occurs (ensure_datetime64ns) was recently eliminated in favor of an updated implementation in pandas, so I wonder if this will be an issue there going forward.

@spencerkclark That works on s390x too (same output).

we have CI: https://ci.debian.net/packages/p/python-xarray/unstable/s390x/ https://ci.debian.net/packages/p/python-xarray/unstable/i386/

Its working ok on Arm, but failing i386 (32-bi) and s390x (64-bit).

Is it not possible to have a CI run with ARM and S390x architectures? Maybe that would be a good improvement.

Thanks for trying that. Maybe it has to do with casting to a pd.Series. Could you maybe also try: ```

import numpy as np; import pandas as pd pd.Series(pd.Timestamp(np.int64(1000000).astype("<M8[h]"))) 0 2084-01-29 16:00:00 dtype: datetime64[ns] ```

The code snippet succeeds on s390x in the environment.

Apologies for taking a while to look into this. I have not been able to set up an environment to reproduce these test failures, which makes it tricky. It seems like the tests are failing in the setup step, where a DataArray of some random times is generated: data = xr.DataArray(np.random.randint(1, 1000000, size=(4, 5)).astype("<M8[h]"), dims=("x", "y")) In principle the NumPy code should not generate any times larger than 1,000,000 hours since 1970-01-01, i.e. 2084-01-29T16:00:00, which in theory should be should be representable with a nanosecond-precision pandas Timestamp.

Trying to narrow things down, I guess my first question would be: does the following fail in this environment? Is this maybe a pandas issue? ```

import numpy as np; import pandas as pd pd.Timestamp(np.int64(1000000).astype("<M8[h]")) Timestamp('2084-01-29 16:00:00') ``` I think these tests could be simplified some to remove the randomness, but that's probably a separate issue.

INSTALLED VERSIONS

commit: None python: 3.10.5 (main, Jun 8 2022, 09:26:22) [GCC 11.3.0] python-bits: 64 OS: Linux OS-release: 5.10.0-16-s390x machine: s390x processor: byteorder: big LC_ALL: None LANG: None LOCALE: ('en_US', 'UTF-8') libhdf5: 1.10.7 libnetcdf: 4.9.0

xarray: 0.16.1 pandas: 1.3.5 numpy: 1.21.5 scipy: 1.8.1 netCDF4: 1.6.0 pydap: None h5netcdf: 1.0.2 h5py: 3.7.0 Nio: None zarr: 2.12.0 cftime: 1.6.1 nc_time_axis: None PseudoNetCDF: None rasterio: 1.3.0 cfgrib: 0.9.10.1 iris: None bottleneck: 1.3.2 dask: 2022.02.0+dfsg distributed: None matplotlib: 3.5.2 cartopy: 0.20.3 seaborn: 0.11.2 numbagg: None fsspec: 2022.5.0 cupy: None pint: None sparse: None flox: None numpy_groupies: None setuptools: 59.6.0 pip3: None conda: None pytest: 7.1.2 IPython: 7.31.1 sphinx: 4.5.0