Skip to content

Feature: Add support for pandas 3 - #443

Merged
akeeste merged 13 commits into
MHKiT-Software:developfrom
simmsa:feat_add_pandas_3.0_support
Jun 9, 2026
Merged

akeeste merged 13 commits into
MHKiT-Software:developfrom
simmsa:feat_add_pandas_3.0_support

Conversation

@simmsa

@simmsa simmsa commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

Pandas 3 Changes

  • Updated pandas dependency to allow versions >=2.2.2 without an upper limit in both environment-dev.yml and pyproject.toml, enabling support for pandas 3.x.
  • Added a new function replace_pandas_missing_values_with_nan in mhkit/wave/io/ndbc.py to replace missing values with NaN in a way that is compatible with both pandas 2.x and 3.x.

Test Updates

https://pandas.pydata.org/docs/user_guide/timeseries.html#period-aliases

@simmsa
simmsa marked this pull request as draft June 8, 2026 14:36
@akeeste akeeste linked an issue Jun 8, 2026 that may be closed by this pull request
pyarrow provides native string dtype support for mhkit.

While technically optional, mhkit benefits from one standardized backend
for representing columnar data. At this time pyarrow is that standard.
simmsa added 6 commits June 8, 2026 17:22
The narrow hdf5 range causes the solve when pyarrow is included to be
slow. Increasing the allowed range improves the pip solver speed.
These should no longer be necessary with the changes to dependencies
This removes any explicit coveralls intalls in favor of the
coverallsapp/github-action@v2 which is self contained and contains all
binaries necessary for coveralls to run.
Agg is a non interactive matplotlib backend and lets the tests runh
"headless". Without this specification tests can choose their backend
which can sometimes lead to failing tests on headless machines (GH
        Actions).

@akeeste akeeste left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @simmsa! This looks great. All the tests are passing. I think we can just clean up the edit to main.yml re the test matrix on this PR and merge

Comment thread .github/workflows/main.yml
@simmsa
simmsa marked this pull request as ready for review June 9, 2026 18:13
@akeeste

akeeste commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

All tests passed on the last significant commit. Approved and merging.

@akeeste
akeeste merged commit df4e431 into MHKiT-Software:develop Jun 9, 2026
80 of 113 checks passed
@simmsa

simmsa commented Jun 9, 2026

Copy link
Copy Markdown
Contributor Author

@akeeste, thank you!

@simmsa simmsa mentioned this pull request Jun 29, 2026
3 tasks
simmsa added a commit that referenced this pull request Jul 8, 2026
# MHKiT-Python v1.1.0

## Additions

* Acoustics: Add millicdecade and WISPR instrument support
  * Added millidecade spectral conversion
  * Added a WISPR hydrophone reader
  * Added a voltage-based `export_audio` resampling option
  * Refactored band-averaging to avoid losing information at frequency-band boundaries
  * #447
  * Author: @jmcvey3
  * Reviewer: @simmsa
* DOLfYN: Add [Nortek Aquadopp](https://www.nortekgroup.com/oceanography/aquadopp-series) ADCP support
  * Added DOLfYN support for reading Aquadopp instruments
  * Cleaned Nortek parsing code
  * Simplified handling of the non-cabled ADV orientation flag
  * #434
  * Author: @jmcvey3
  * Reviewer: @akeeste
* Examples: Add ADCP waves example: `example/adcp_waves_example.ipynb`
  * Added an example notebook showing how to ingest and analyze wave measurements from a dual-profile Nortek Signature 250 deployment at PacWave.
  * #430
  * Author: @jmcvey3
  * Reviewer: @akeeste

## Improvements

### Acoustics

- Improved flexibility and robustness of the Acoustics module
  - Added a configurable FFT length for sound pressure PSDs
  - Renamed bin/windowing attributes for clarity
  - Fixed an incorrectly signed gain correction
  - #433
  - Author: @jmcvey3
  - Reviewer: @akeeste

### DOLfYN

- Refactored PSD calculations to use `scipy.signal.welch`
  - Replaced DOLfYN's custom Welch-like PSD implementation (built each segment from bin slices, could not overlap FFT segments) with `scipy.signal.welch`
  - Removed the bin-based segment structure and added a `step` argument to control overlap
  - Dropped float32 casts in some functions in favor of float64
  - Removed `fft.py` and renamed `tools/misc.py` to `tools.py`
  - Breaking Changes:
    - NaNs are no longer tolerated in PSD calculations
    - DOLfYN defaults to 0% overlap to preserve existing array shapes
    - Acoustics defaults to 50% overlap
    - PSD output time dimension renamed to `time_psd`
  - #452
  - Author: @jmcvey3
  - Reviewer: @akeeste, @simmsa
- Improved handling of "averaged" profiles
  - Fixed handling of Nortek Signature dual-profile ADCP data by defaulting to "_avg" velocity variables when untagged ones are absent.
  - #430
  - Author: @jmcvey3
  - Reviewer: @akeeste
- Critical PSD bugfix
  - Fixed a bug where individual FFTs received a 50% overlap twice, which corrupted the first and last spectrum of a timeseries.
  - #430
  - Author: @jmcvey3
  - Reviewer: @akeeste

### Examples

- Added histograms to ADCP example
  - #448
  - Author: @browniea
  - Reviewer: @akeeste


### River/IO

- Fixed Qhull interpolation and D3D coordinate-system errors
  - #448
  - Author: @browniea
  - Reviewer: @akeeste
  - Fixes: #442, #444
- Delft3D module updates
  - Added new Delft3D coordinate names
  - Added a new grid-convergence-index calculation function
  - Added support for xarray/netCDF4 input in the D3D module
  - #428
  - Author: @browniea
  - Reviewer: @akeeste

### Wave

- NDBC Directional Wave Units
  - Fixed NDBC directional wave spectrum output to return degrees instead of radians
  - Updated polar plots so 0 deg is at the top and increases clockwise
  - #437
  - Author: @jmcvey3
  - Reviewer: @akeeste
  - Fixes: #427

### Wave/Hindcast

- Added a `hindcast_guard` exception-handling decorator (`hindcast_exceptions.py`) that surfaces a clear error on HSDS request failures, distinguishing the known NLR HSDS outage (#450) from other failures
  - #449
  - Author: @simmsa
  - Reviewer: @akeeste

## Maintenance

- Added Python 3.13 support
  - #445
  - Author: @simmsa
  - Reviewer: @akeeste
  - Fixes: #441
- Added [pandas 3](https://pandas.pydata.org/community/blog/pandas-3.0.html) Support
  - Updated the pandas dependency to allow pandas 3.x
  - Added a compatibility shim for NDBC missing-value handling related to pandas 3 object to String dtype api changes
  - Fixed a deprecated period alias in tests
  - #443
  - Author: @simmsa
  - Reviewer: @akeeste
  - Fixes: #440
- Updated GitHub Actions CI, expanded installation/developer documentation, refreshed dev environment, and trimmed dependencies
  - Refactored optional dependencies
  - Standardized conda/conda-forge environment builds
  - Scoped black linting to changed files
  - #436
  - Author: @simmsa
  - Reviewer: @akeeste
- Update `rex` dependency to target pypi package to [`NLR-rex[hsds]>=0.5.0`](https://pypi.org/project/NLR-rex/)
  - #449
  - Author: @simmsa
  - Reviewer: @akeeste
@simmsa simmsa mentioned this pull request Jul 8, 2026
simmsa added a commit to simmsa/MHKiT-Python that referenced this pull request Jul 10, 2026
## Pandas 3 Changes

* Updated pandas dependency to allow versions >=2.2.2 without an upper
limit in both `environment-dev.yml` and `pyproject.toml`, enabling
support for pandas 3.x.
* Added a new function `replace_pandas_missing_values_with_nan` in
`mhkit/wave/io/ndbc.py` to replace missing values with `NaN` in a way
that is compatible with both pandas 2.x and 3.x.


## Test Updates

* Increased `test_get_buoy_metadata` latitude and longitude assertions
delta tolerance to allow for a small amount real world buoy drift.
* Fix deprecation in `test_request_parse_workflow_multiyear`
(`mhkit/tests/wave/io/test_cdip.py`) by using `"D"` instead of `"d"` in
the `floor` method. Pandas deprecated `d` in 3.0+
pandas-dev/pandas#58998. period aliases are
here:

https://pandas.pydata.org/docs/user_guide/timeseries.html#period-aliases
simmsa added a commit to simmsa/MHKiT-Python that referenced this pull request Jul 10, 2026
# MHKiT-Python v1.1.0

## Additions

* Acoustics: Add millicdecade and WISPR instrument support
  * Added millidecade spectral conversion
  * Added a WISPR hydrophone reader
  * Added a voltage-based `export_audio` resampling option
  * Refactored band-averaging to avoid losing information at frequency-band boundaries
  * MHKiT-Software#447
  * Author: @jmcvey3
  * Reviewer: @simmsa
* DOLfYN: Add [Nortek Aquadopp](https://www.nortekgroup.com/oceanography/aquadopp-series) ADCP support
  * Added DOLfYN support for reading Aquadopp instruments
  * Cleaned Nortek parsing code
  * Simplified handling of the non-cabled ADV orientation flag
  * MHKiT-Software#434
  * Author: @jmcvey3
  * Reviewer: @akeeste
* Examples: Add ADCP waves example: `example/adcp_waves_example.ipynb`
  * Added an example notebook showing how to ingest and analyze wave measurements from a dual-profile Nortek Signature 250 deployment at PacWave.
  * MHKiT-Software#430
  * Author: @jmcvey3
  * Reviewer: @akeeste

## Improvements

### Acoustics

- Improved flexibility and robustness of the Acoustics module
  - Added a configurable FFT length for sound pressure PSDs
  - Renamed bin/windowing attributes for clarity
  - Fixed an incorrectly signed gain correction
  - MHKiT-Software#433
  - Author: @jmcvey3
  - Reviewer: @akeeste

### DOLfYN

- Refactored PSD calculations to use `scipy.signal.welch`
  - Replaced DOLfYN's custom Welch-like PSD implementation (built each segment from bin slices, could not overlap FFT segments) with `scipy.signal.welch`
  - Removed the bin-based segment structure and added a `step` argument to control overlap
  - Dropped float32 casts in some functions in favor of float64
  - Removed `fft.py` and renamed `tools/misc.py` to `tools.py`
  - Breaking Changes:
    - NaNs are no longer tolerated in PSD calculations
    - DOLfYN defaults to 0% overlap to preserve existing array shapes
    - Acoustics defaults to 50% overlap
    - PSD output time dimension renamed to `time_psd`
  - MHKiT-Software#452
  - Author: @jmcvey3
  - Reviewer: @akeeste, @simmsa
- Improved handling of "averaged" profiles
  - Fixed handling of Nortek Signature dual-profile ADCP data by defaulting to "_avg" velocity variables when untagged ones are absent.
  - MHKiT-Software#430
  - Author: @jmcvey3
  - Reviewer: @akeeste
- Critical PSD bugfix
  - Fixed a bug where individual FFTs received a 50% overlap twice, which corrupted the first and last spectrum of a timeseries.
  - MHKiT-Software#430
  - Author: @jmcvey3
  - Reviewer: @akeeste

### Examples

- Added histograms to ADCP example
  - MHKiT-Software#448
  - Author: @browniea
  - Reviewer: @akeeste


### River/IO

- Fixed Qhull interpolation and D3D coordinate-system errors
  - MHKiT-Software#448
  - Author: @browniea
  - Reviewer: @akeeste
  - Fixes: MHKiT-Software#442, MHKiT-Software#444
- Delft3D module updates
  - Added new Delft3D coordinate names
  - Added a new grid-convergence-index calculation function
  - Added support for xarray/netCDF4 input in the D3D module
  - MHKiT-Software#428
  - Author: @browniea
  - Reviewer: @akeeste

### Wave

- NDBC Directional Wave Units
  - Fixed NDBC directional wave spectrum output to return degrees instead of radians
  - Updated polar plots so 0 deg is at the top and increases clockwise
  - MHKiT-Software#437
  - Author: @jmcvey3
  - Reviewer: @akeeste
  - Fixes: MHKiT-Software#427

### Wave/Hindcast

- Added a `hindcast_guard` exception-handling decorator (`hindcast_exceptions.py`) that surfaces a clear error on HSDS request failures, distinguishing the known NLR HSDS outage (MHKiT-Software#450) from other failures
  - MHKiT-Software#449
  - Author: @simmsa
  - Reviewer: @akeeste

## Maintenance

- Added Python 3.13 support
  - MHKiT-Software#445
  - Author: @simmsa
  - Reviewer: @akeeste
  - Fixes: MHKiT-Software#441
- Added [pandas 3](https://pandas.pydata.org/community/blog/pandas-3.0.html) Support
  - Updated the pandas dependency to allow pandas 3.x
  - Added a compatibility shim for NDBC missing-value handling related to pandas 3 object to String dtype api changes
  - Fixed a deprecated period alias in tests
  - MHKiT-Software#443
  - Author: @simmsa
  - Reviewer: @akeeste
  - Fixes: MHKiT-Software#440
- Updated GitHub Actions CI, expanded installation/developer documentation, refreshed dev environment, and trimmed dependencies
  - Refactored optional dependencies
  - Standardized conda/conda-forge environment builds
  - Scoped black linting to changed files
  - MHKiT-Software#436
  - Author: @simmsa
  - Reviewer: @akeeste
- Update `rex` dependency to target pypi package to [`NLR-rex[hsds]>=0.5.0`](https://pypi.org/project/NLR-rex/)
  - MHKiT-Software#449
  - Author: @simmsa
  - Reviewer: @akeeste
simmsa added a commit that referenced this pull request Jul 10, 2026
## Pandas 3 Changes

* Updated pandas dependency to allow versions >=2.2.2 without an upper
limit in both `environment-dev.yml` and `pyproject.toml`, enabling
support for pandas 3.x.
* Added a new function `replace_pandas_missing_values_with_nan` in
`mhkit/wave/io/ndbc.py` to replace missing values with `NaN` in a way
that is compatible with both pandas 2.x and 3.x.


## Test Updates

* Increased `test_get_buoy_metadata` latitude and longitude assertions
delta tolerance to allow for a small amount real world buoy drift.
* Fix deprecation in `test_request_parse_workflow_multiyear`
(`mhkit/tests/wave/io/test_cdip.py`) by using `"D"` instead of `"d"` in
the `floor` method. Pandas deprecated `d` in 3.0+
pandas-dev/pandas#58998. period aliases are
here:

https://pandas.pydata.org/docs/user_guide/timeseries.html#period-aliases
simmsa added a commit that referenced this pull request Jul 10, 2026
# MHKiT-Python v1.1.0

## Additions

* Acoustics: Add millicdecade and WISPR instrument support
  * Added millidecade spectral conversion
  * Added a WISPR hydrophone reader
  * Added a voltage-based `export_audio` resampling option
  * Refactored band-averaging to avoid losing information at frequency-band boundaries
  * #447
  * Author: @jmcvey3
  * Reviewer: @simmsa
* DOLfYN: Add [Nortek Aquadopp](https://www.nortekgroup.com/oceanography/aquadopp-series) ADCP support
  * Added DOLfYN support for reading Aquadopp instruments
  * Cleaned Nortek parsing code
  * Simplified handling of the non-cabled ADV orientation flag
  * #434
  * Author: @jmcvey3
  * Reviewer: @akeeste
* Examples: Add ADCP waves example: `example/adcp_waves_example.ipynb`
  * Added an example notebook showing how to ingest and analyze wave measurements from a dual-profile Nortek Signature 250 deployment at PacWave.
  * #430
  * Author: @jmcvey3
  * Reviewer: @akeeste

## Improvements

### Acoustics

- Improved flexibility and robustness of the Acoustics module
  - Added a configurable FFT length for sound pressure PSDs
  - Renamed bin/windowing attributes for clarity
  - Fixed an incorrectly signed gain correction
  - #433
  - Author: @jmcvey3
  - Reviewer: @akeeste

### DOLfYN

- Refactored PSD calculations to use `scipy.signal.welch`
  - Replaced DOLfYN's custom Welch-like PSD implementation (built each segment from bin slices, could not overlap FFT segments) with `scipy.signal.welch`
  - Removed the bin-based segment structure and added a `step` argument to control overlap
  - Dropped float32 casts in some functions in favor of float64
  - Removed `fft.py` and renamed `tools/misc.py` to `tools.py`
  - Breaking Changes:
    - NaNs are no longer tolerated in PSD calculations
    - DOLfYN defaults to 0% overlap to preserve existing array shapes
    - Acoustics defaults to 50% overlap
    - PSD output time dimension renamed to `time_psd`
  - #452
  - Author: @jmcvey3
  - Reviewer: @akeeste, @simmsa
- Improved handling of "averaged" profiles
  - Fixed handling of Nortek Signature dual-profile ADCP data by defaulting to "_avg" velocity variables when untagged ones are absent.
  - #430
  - Author: @jmcvey3
  - Reviewer: @akeeste
- Critical PSD bugfix
  - Fixed a bug where individual FFTs received a 50% overlap twice, which corrupted the first and last spectrum of a timeseries.
  - #430
  - Author: @jmcvey3
  - Reviewer: @akeeste

### Examples

- Added histograms to ADCP example
  - #448
  - Author: @browniea
  - Reviewer: @akeeste


### River/IO

- Fixed Qhull interpolation and D3D coordinate-system errors
  - #448
  - Author: @browniea
  - Reviewer: @akeeste
  - Fixes: #442, #444
- Delft3D module updates
  - Added new Delft3D coordinate names
  - Added a new grid-convergence-index calculation function
  - Added support for xarray/netCDF4 input in the D3D module
  - #428
  - Author: @browniea
  - Reviewer: @akeeste

### Wave

- NDBC Directional Wave Units
  - Fixed NDBC directional wave spectrum output to return degrees instead of radians
  - Updated polar plots so 0 deg is at the top and increases clockwise
  - #437
  - Author: @jmcvey3
  - Reviewer: @akeeste
  - Fixes: #427

### Wave/Hindcast

- Added a `hindcast_guard` exception-handling decorator (`hindcast_exceptions.py`) that surfaces a clear error on HSDS request failures, distinguishing the known NLR HSDS outage (#450) from other failures
  - #449
  - Author: @simmsa
  - Reviewer: @akeeste

## Maintenance

- Added Python 3.13 support
  - #445
  - Author: @simmsa
  - Reviewer: @akeeste
  - Fixes: #441
- Added [pandas 3](https://pandas.pydata.org/community/blog/pandas-3.0.html) Support
  - Updated the pandas dependency to allow pandas 3.x
  - Added a compatibility shim for NDBC missing-value handling related to pandas 3 object to String dtype api changes
  - Fixed a deprecated period alias in tests
  - #443
  - Author: @simmsa
  - Reviewer: @akeeste
  - Fixes: #440
- Updated GitHub Actions CI, expanded installation/developer documentation, refreshed dev environment, and trimmed dependencies
  - Refactored optional dependencies
  - Standardized conda/conda-forge environment builds
  - Scoped black linting to changed files
  - #436
  - Author: @simmsa
  - Reviewer: @akeeste
- Update `rex` dependency to target pypi package to [`NLR-rex[hsds]>=0.5.0`](https://pypi.org/project/NLR-rex/)
  - #449
  - Author: @simmsa
  - Reviewer: @akeeste
simmsa added a commit that referenced this pull request Jul 10, 2026
# MHKiT-Python v1.1.0

## Additions

* Acoustics: Add millicdecade and WISPR instrument support
  * Added millidecade spectral conversion
  * Added a WISPR hydrophone reader
  * Added a voltage-based `export_audio` resampling option
  * Refactored band-averaging to avoid losing information at frequency-band boundaries
  * #447
  * Author: @jmcvey3
  * Reviewer: @simmsa
* DOLfYN: Add [Nortek Aquadopp](https://www.nortekgroup.com/oceanography/aquadopp-series) ADCP support
  * Added DOLfYN support for reading Aquadopp instruments
  * Cleaned Nortek parsing code
  * Simplified handling of the non-cabled ADV orientation flag
  * #434
  * Author: @jmcvey3
  * Reviewer: @akeeste
* Examples: Add ADCP waves example: `example/adcp_waves_example.ipynb`
  * Added an example notebook showing how to ingest and analyze wave measurements from a dual-profile Nortek Signature 250 deployment at PacWave.
  * #430
  * Author: @jmcvey3
  * Reviewer: @akeeste

## Improvements

### Acoustics

- Improved flexibility and robustness of the Acoustics module
  - Added a configurable FFT length for sound pressure PSDs
  - Renamed bin/windowing attributes for clarity
  - Fixed an incorrectly signed gain correction
  - #433
  - Author: @jmcvey3
  - Reviewer: @akeeste

### DOLfYN

- Refactored PSD calculations to use `scipy.signal.welch`
  - Replaced DOLfYN's custom Welch-like PSD implementation (built each segment from bin slices, could not overlap FFT segments) with `scipy.signal.welch`
  - Removed the bin-based segment structure and added a `step` argument to control overlap
  - Dropped float32 casts in some functions in favor of float64
  - Removed `fft.py` and renamed `tools/misc.py` to `tools.py`
  - Breaking Changes:
    - NaNs are no longer tolerated in PSD calculations
    - DOLfYN defaults to 0% overlap to preserve existing array shapes
    - Acoustics defaults to 50% overlap
    - PSD output time dimension renamed to `time_psd`
  - #452
  - Author: @jmcvey3
  - Reviewer: @akeeste, @simmsa
- Improved handling of "averaged" profiles
  - Fixed handling of Nortek Signature dual-profile ADCP data by defaulting to "_avg" velocity variables when untagged ones are absent.
  - #430
  - Author: @jmcvey3
  - Reviewer: @akeeste
- Critical PSD bugfix
  - Fixed a bug where individual FFTs received a 50% overlap twice, which corrupted the first and last spectrum of a timeseries.
  - #430
  - Author: @jmcvey3
  - Reviewer: @akeeste

### Examples

- Added histograms to ADCP example
  - #448
  - Author: @browniea
  - Reviewer: @akeeste

### River/IO

- Fixed Qhull interpolation and D3D coordinate-system errors
  - #448
  - Author: @browniea
  - Reviewer: @akeeste
  - Fixes: #442, #444
- Delft3D module updates
  - Added new Delft3D coordinate names
  - Added a new grid-convergence-index calculation function
  - Added support for xarray/netCDF4 input in the D3D module
  - #428
  - Author: @browniea
  - Reviewer: @akeeste

### Wave

- NDBC Directional Wave Units
  - Fixed NDBC directional wave spectrum output to return degrees instead of radians
  - Updated polar plots so 0 deg is at the top and increases clockwise
  - #437
  - Author: @jmcvey3
  - Reviewer: @akeeste
  - Fixes: #427

### Wave/Hindcast

- Added a `hindcast_guard` exception-handling decorator (`hindcast_exceptions.py`) that surfaces a clear error on HSDS request failures, distinguishing the known NLR HSDS outage (#450) from other failures
  - #449
  - Author: @simmsa
  - Reviewer: @akeeste

## Maintenance

- Added Python 3.13 support
  - #445
  - Author: @simmsa
  - Reviewer: @akeeste
  - Fixes: #441
- Added [pandas 3](https://pandas.pydata.org/community/blog/pandas-3.0.html) Support
  - Updated the pandas dependency to allow pandas 3.x
  - Added a compatibility shim for NDBC missing-value handling related to pandas 3 object to String dtype api changes
  - Fixed a deprecated period alias in tests
  - #443
  - Author: @simmsa
  - Reviewer: @akeeste
  - Fixes: #440
- Updated GitHub Actions CI, expanded installation/developer documentation, refreshed dev environment, and trimmed dependencies
  - Refactored optional dependencies
  - Standardized conda/conda-forge environment builds
  - Scoped black linting to changed files
  - #436
  - Author: @simmsa
  - Reviewer: @akeeste
- Update `rex` dependency to target pypi package to [`NLR-rex[hsds]>=0.5.0`](https://pypi.org/project/NLR-rex/)
  - #449
  - Author: @simmsa
  - Reviewer: @akeeste
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Function failing test: test_get_buoy_metadata

2 participants