okay thank you, started issue at: https://github.com/Unidata/netcdf-c/issues/2459

Or maybe I should add to this issue https://github.com/Unidata/netcdf4-python/issues/812 rather than starting a new one? Guidance welcome thanks.

Thanks for these suggestions, very helpful. See below, for details, but as far as I can tell it looks like: * my conda env ("EQ") has the same curl, libcurl, ca-certificates, and certifi as your system. * the ncdump commands gives same error (as netcdf4 and xarray).

I should post an issue at netcdf4, correct?

``` (EQ) PS C:\Users\Codiga_D> conda list curl

packages in environment at C:\Users\Codiga_D\AppData\Local\Continuum\miniconda3\envs\EQ:

Name Version Build Channel

curl 7.83.1 h789b8ee_0 conda-forge libcurl 7.83.1 h789b8ee_0 conda-forge

(EQ) PS C:\Users\Codiga_D> conda list certifi

packages in environment at C:\Users\Codiga_D\AppData\Local\Continuum\miniconda3\envs\EQ:

Name Version Build Channel

ca-certificates 2022.6.15 h5b45459_0 conda-forge certifi 2022.6.15 py37h03978a9_0 conda-forge

(EQ) PS C:\Users\Codiga_D> ncdump -h http://psl.noaa.gov/thredds/dodsC/Datasets/NARR/monolevel/uwnd.10m.2000.nc Error:curl error: SSL connect error curl error details: Warning:oc_open: Could not read url C:\Users\Codiga_D\AppData\Local\Continuum\miniconda3\envs\EQ\Library\bin\ncdump.exe: http://psl.noaa.gov/thredds/dodsC/Datasets/NARR/monolevel/uwnd.10m.2000.nc: NetCDF: I/O failure ```

Thanks for the quick response. Result from xr.show_versions()is below. I tried netCDF4.Dataset(url)and it gave the same error.

Just a thought: I still wonder if this could be related to certification... which is something that did change on my system recently. I looked for information as to where netCDF4 would check for its certificate chain, but wasn't able to find something useful so far.

INSTALLED VERSIONS

commit: None python: 3.7.12 | packaged by conda-forge | (default, Oct 26 2021, 05:37:49) [MSC v.1916 64 bit (AMD64)] python-bits: 64 OS: Windows OS-release: 10 machine: AMD64 processor: Intel64 Family 6 Model 158 Stepping 10, GenuineIntel byteorder: little LC_ALL: None LANG: en LOCALE: (None, None) libhdf5: 1.12.1 libnetcdf: 4.8.1

xarray: 0.20.2 pandas: 1.3.5 numpy: 1.21.6 scipy: 1.7.3 netCDF4: 1.6.0 pydap: None h5netcdf: None h5py: None Nio: None zarr: None cftime: 1.6.1 nc_time_axis: None PseudoNetCDF: None rasterio: 1.2.10 cfgrib: 0.9.10.1 iris: None bottleneck: 1.3.4 dask: None distributed: None matplotlib: 3.5.2 cartopy: 0.20.2 seaborn: 0.11.2 numbagg: None fsspec: None cupy: None pint: None sparse: None setuptools: 59.8.0 pip: 22.1.2 conda: None pytest: None IPython: 7.33.0 sphinx: 4.3.2

@dcherian, @spencerkclark , and @mada0007 Could u plz tell me that how to join that data after selecting Oct-March. Basically, I want to say whenever I am plotting a time series of this selected monthly data. My time time series is not continuous. Kindly plz let me know. I am attaching a plot for the reference. Example.pdf

Sorry for late response. I was trying to read a big geotif file as follows. import xarray as xr xds = xr.open_rasterio(geotif_file)

My task was to array indexing and to save output into disk.

columns = [8,9,7,100,1050,......, 9000] rows = [18,19,17,1100,1105,......, 9100]

data = xds.isel(x=xr.DataArray(columns), y=xr.DataArray(rows))

np.save('output.npy', data)

Unfortunately, the performance in terms of time requirement seems quite unsatisfactory.

When I saw docs on xr.open_rasterio(), it mentions it is an experimental now. So, I'm curious if it could be much faster when it becomes stable.

I look forward to see it as stable version. Thank you so much.

Each individual dataset opens successfully.

```python

rslt = xr.open_dataset('D:\dasktest\data_dir\EM2040\converted\test\test4.nc') rslt <xarray.Dataset> Dimensions: (beams: 250, sectors: 3, time: 1000) Coordinates: * time (time) int32 4000 4001 4002 4003 4004 ... 4996 4997 4998 4999 * sectors (sectors) object '40107_0_260000' ... '40107_2_290000' * beams (beams) int32 0 1 2 3 4 5 6 7 ... 243 244 245 246 247 248 249 Data variables: soundspeed (time, sectors, beams) float64 ... rslt.soundspeed.compute() <xarray.DataArray 'soundspeed' (time: 1000, sectors: 3, beams: 250)> array([[[ 1.44506928e+00, -1.51871078e+00, -3.23406027e+00, ..., -3.23209458e+00, 1.51651459e+00, 6.42639524e-01], [ 4.50549891e-01, -1.65891239e-01, 5.03328065e-01, ..., 2.90603391e-02, -2.03399058e+00, 1.07384206e+00], [ 1.10102985e+00, -1.04792461e-01, 5.01006754e-01, ..., -6.85958163e-01, -6.03574819e-01, 3.82282291e-01]],

   [[ 1.84926818e-01,  9.15722054e-02, -1.21527347e+00, ...,
     -5.01625941e-01, -1.81885721e+00,  5.06066650e-01],
    [ 1.15374575e+00, -7.46658574e-01, -6.06853717e-01, ...,
     -1.91083642e+00,  1.82612868e-01, -1.75664446e+00],
    [ 5.15672540e-01, -6.77122781e-01, -1.52097291e+00, ...,
      1.20734955e+00, -4.89001010e-02,  1.62015033e+00]],

   [[ 1.21801104e-01, -4.34477805e-01, -1.34871232e+00, ...,
     -1.24627225e+00,  5.74698640e-01,  1.14271464e+00],
    [ 4.67490848e-01,  1.16150690e+00, -9.00648837e-01, ...,
     -1.48880232e+00, -1.41956820e-02, -8.61839051e-01],
    [-1.04143082e-01, -3.48620768e-01,  2.55312206e-01, ...,
      4.37058964e-01,  4.46445888e-01, -4.10217694e-01]],

   ...,

   [[ 1.61535162e+00,  1.74216005e-01, -3.09371662e+00, ...,
      1.26629794e+00, -9.18653539e-01, -2.37142082e+00],
    [-4.38608161e-02,  1.32472408e+00,  3.92494217e-01, ...,
     -1.03977601e+00, -1.87358228e+00,  1.71303085e+00],
    [ 1.40801908e-01, -4.85799815e-01, -4.28494910e-01, ...,
      5.88979031e-01, -3.16616983e-01, -4.03942439e-01]],

   [[-1.25221965e+00,  1.40617329e+00,  3.81911399e+00, ...,
     -1.41840680e+00, -8.81194686e-01,  2.24217335e+00],
    [-1.04704250e+00,  6.58982192e-01,  4.30801840e-01, ...,
      1.36966939e+00,  1.51575602e+00, -6.29969352e-01],
    [ 2.11058905e-01,  1.05525622e+00, -1.29686045e-01, ...,
     -1.33221475e+00,  1.74849717e-01,  2.15998814e+00]],

   [[-7.33979313e-02,  2.61711631e-01,  1.02465836e+00, ...,
      3.64389987e-01, -1.20189741e+00,  3.39280076e-01],
    [-4.03723405e-01, -2.21631345e+00, -9.94534629e-01, ...,
     -5.07692651e-01, -1.81561305e+00, -1.23491815e+00],
    [-6.36790749e-01, -8.00208607e-02,  5.46987721e-01, ...,
     -2.25807078e-03,  1.59638267e-01, -1.29816760e+00]]])

Coordinates: * time (time) int32 4000 4001 4002 4003 4004 ... 4995 4996 4997 4998 4999 * sectors (sectors) object '40107_0_260000' '40107_1_320000' '40107_2_290000' * beams (beams) int32 0 1 2 3 4 5 6 7 8 ... 242 243 244 245 246 247 248 249

cl <Client: 'tcp://127.0.0.1:59016' processes=4 threads=16, memory=34.27 GB> ```

I know what "NaN" means. I was hoping that by transforming the dataset into a dataframe and then returning back, the dataset variables would recover its original shape.

It seems that this is not a problem with xarray but only with rasterio and netCDF4. Also this fails: ```python import rasterio import netCDF4

with netCDF4.Dataset('test.nc', mode='w') as ds: ds.createDimension('x') ds.createVariable('foo', float, dimensions=('x')) print(ds) `` Commenting outimport rasterio` removes the HDF error. I’ll report this to rasterio.

I have similar problem, when importing rasterio in the same script (not even using it for anything). This fails with HDF error:

import xarray as xa
import numpy as np
#import netCDF4
import rasterio

ds = xa.Dataset()
ds['z'] = (('y', 'x'), np.zeros((100, 100), np.float32))
print(ds)
ds.to_netcdf('test.nc')
ds.close()

with xa.open_dataset('test.nc') as ds:
    print(ds)

If I import netCDF4 before rasterio it works fine (uncomment line 3). This is probably an issue with rasterio somehow.

I installed everything with pip:

$ pip install Cython
$ pip install netCDF4 xarray rasterio numpy

From pip freeze:

affine==2.2.1
attrs==18.2.0
cftime==1.0.3
Click==7.0
click-plugins==1.0.4
cligj==0.5.0
Cython==0.29.1
netCDF4==1.4.2
numpy==1.15.4
pandas==0.23.4
pyparsing==2.3.0
python-dateutil==2.7.5
pytz==2018.7
rasterio==1.0.11
six==1.11.0
snuggs==1.4.2
xarray==0.11.0

Does that sound like it will play well with GeoViews if I want widgets for the categorical vars?

Ok. Looks like the way forward is a netCDF file for each level of my categorical variables. Will give it a shot.

@shoyer Thank you. Does metacsv look likely to work to you? It has attracted almost no attention so I wonder if it will exhaust memory. I'm kind of surprised this path (csv -> xarray) isn't better fleshed out as I would have expected it to be very common, perhaps the most common esp. for "found data."

This looks potentially helpful http://metacsv.readthedocs.io/en/latest/

PS: I started with Dask but haven't found a way to go from Dask to xarray.

Hi @jhamman The original data is literally just a flat csv file with ie: lat,lon,epoch,cat1,cat2,var1,var2,...,var50 with 1 billion rows.

I'm looking to xarray for GeoViews, which I think would benefit from having the data properly grouped/indexed by its categories

Closing. Superseded by #1682.

This was fixed through https://github.com/pydap/pydap/pull/159! Thank you.

The method pydap.model.BaseType._get_data_index(self, index=Ellipsis) gets value 'tlml' for index. It tries to return self._data[index], but self._data is a numpy.ndarray. Hence the IndexError.

For some reason, the name of the variable at some point becomes 'tlml.tlml'. Method pydap.model.StructureType._getitem_string would work fine if the name was just 'tlml'. Maybe this is related to https://github.com/pydap/pydap/issues/121?

Thanks for the suggestion! Installing both latest master of xarray (0092911) and latest master of pydap (4ae73e3) fixed this issue, and now I can open the dataset.

After pulling (Git says ‘Already up-to-date.’), my xarray version (xarray.__version__) is still '0.10.0+dev44.g0a0593d'. What version are you using?

Same thing: ``` Traceback (most recent call last): File "C:\Anaconda3\envs\xa_test\lib\site-packages\pydap\model.py", line 295, in getattr return self[attr] File "C:\Anaconda3\envs\xa_test\lib\site-packages\pydap\model.py", line 556, in getitem return StructureType.getitem(self, key) File "C:\Anaconda3\envs\xa_test\lib\site-packages\pydap\model.py", line 326, in getitem return self._dict[key] KeyError: 'shape'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\Anaconda3\envs\xa_test\lib\site-packages\pydap\model.py", line 180, in getattr return self.attributes[attr] KeyError: 'shape'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "<stdin>", line 1, in <module> File "c:\src\xarray\xarray\backends\api.py", line 305, in open_dataset return maybe_decode_store(store, lock) File "c:\src\xarray\xarray\backends\api.py", line 225, in maybe_decode_store drop_variables=drop_variables) File "c:\src\xarray\xarray\conventions.py", line 598, in decode_cf vars, attrs = obj.load() File "c:\src\xarray\xarray\backends\common.py", line 133, in load for k, v in self.get_variables().items()) File "c:\src\xarray\xarray\backends\pydap_.py", line 85, in get_variables for k in self.ds.keys()) File "c:\src\xarray\xarray\core\utils.py", line 309, in FrozenOrderedDict return Frozen(OrderedDict(args, *kwargs)) File "c:\src\xarray\xarray\backends\pydap_.py", line 85, in <genexpr> for k in self.ds.keys()) File "c:\src\xarray\xarray\backends\pydap_.py", line 79, in open_store_variable data = indexing.LazilyIndexedArray(PydapArrayWrapper(var)) File "c:\src\xarray\xarray\core\indexing.py", line 482, in init key = BasicIndexer((slice(None),) * array.ndim) File "c:\src\xarray\xarray\core\utils.py", line 428, in ndim return len(self.shape) File "c:\src\xarray\xarray\backends\pydap_.py", line 20, in shape return self.array.shape File "C:\Anaconda3\envs\xa_test\lib\site-packages\pydap\model.py", line 297, in getattr return DapType.getattr(self, attr) File "C:\Anaconda3\envs\xa_test\lib\site-packages\pydap\model.py", line 184, in getattr % (self.class, attr)) AttributeError: '<class 'pydap.model.GridType'>' object has no attribute 'shape' ```

Now that the tests are passing again, is there anything else left to change?

I've refactored setting the I/O engine option as per our discussion. Hopefully, it captures now all the requested functionality.

Let's see if we can get this PR over the line... 😄

A list of engines would need some way of declaring their I/O capabilities: only file-based, only HTTP-based, or both. Something like:

```python io_engines = [ {'engine': 'netcdf4', 'capabilities': ['file', 'http']},

{'engine': 'pydap', 
 'capabilities': ['http']},

{'engine': 'scipy', 
 'capabilities': ['file']},

{'engine': 'h5netcdf',
 'capabilities': ['file']},

] ```

On xarray import or any time this option would change, the list of engines would be checked to remove unavailable engines.

Yes, there could be more I/O engine options. How about file_engine and web_engine? Keeping the naming more generic as there may be other file-based formats or data web services in the future.

On the other hand, setting this global option should indicate a willingness to accept the consequences. If automatic selection of the optional I/O engine is preferred, this global option should not be set.

How about io_engine for the option's name? The data can come from an OPeNDAP server as well.

I have reverted to the original engine=None in the functions/methods and use python engine = engine or OPTIONS['io_engine'] to assign the correct engine value.

Just saw xr.DataArray.dot(). PROBLEM SOLVED.

How do I make dot product (np.dot or pandas.dataframe.dot) between two DataArrays? X has dimensions [dim_0, dim_1, dim_2], Y has dimensions [dim0, dim3]. result should have dimensions [dim_1, dim2, dim3]. result = np.dot(X,Y) OR, result = pd.DataFrame.dot(X,Y) In both cases, error "shapes are not aligned" occurred.

I agree that it's the point with np.asarray, but given the implementation you'd think np.asanyarray would work. My initial takeaway (until examining the source) was that this was an ndarray with additional attributes and properties. Perhaps, I'm leaning too far towards numpy and too far away from pandas.

As background: my usage involves RF pattern data which typically involves a lot of independent variables to lug around as well as the measured data.

I'll look into your other suggestions. Thank you for your reply.