ModuleNotFoundError: No module named 'scipy.fft._pocketfft'

[seanlaw]

According to these recently failing tests:

https://github.com/stumpy-dev/automate/actions/runs/27837317419/job/82405841925
https://github.com/stumpy-dev/automate/actions/runs/27837448667/job/82388725360
https://github.com/stumpy-dev/automate/actions/runs/27837448733/job/82406007263

which are erroring out with:

ModuleNotFoundError: No module named 'scipy.fft._pocketfft'

It appears that _pocketfft cannot be found in https://github.com/scipy/scipy/tree/main/scipy/fft

because pocketfft has been succeeded by ducc0.fft (both of which are maintained by the same creator - mreineck):

scipy/scipy#23436

They mention:

The most significant of those from SciPy's perspective is probably that storage requirements for internally cached plans have been significantly reduced for most long 1D transforms. Plans that require more storage than 1MB will no longer be cached; this mainly affects huge 1D transforms of prime and near-prime sizes.

[seanlaw]

@NimaSarajpoor I believe that this directly affects #1132

Would you mind taking a look?

[seanlaw]

Also, in a "patting ourselves on the back" kind of way (or, maybe, "future us thanks past us"), our automated tests are doing its job in ensuring that we find bugs in a timely fashion! ❤️

[NimaSarajpoor]

@seanlaw

It appears that _pocketfft cannot be found in https://github.com/scipy/scipy/tree/main/scipy/fft
because pocketfft has been succeeded by ducc0.fft (both of which are maintained by the same creator - mreineck):
scipy/scipy#23436

@NimaSarajpoor I believe that this directly affects #1132

Would you mind taking a look?

Right. I will fix it in the main branch first, and then get into #1132

Also, in a "patting ourselves on the back" kind of way (or, maybe, "future us thanks past us"), our automated tests are doing its job in ensuring that we find bugs in a timely fashion! ❤️

Indeed!! Better to catch them sooner than later!

---

IIRC, you mentioned we should keep scipy's convolve / oaconvolve anyway as they are public APIs and the private ones may break. Well... that just happened!

[NimaSarajpoor]

Okay, the issue is resolved when I upgraded scipy to its latest version in my env and then:

from scipy.fft._duccfft.basic import c2r, r2c

The tests are passing. We need to:

• set minimum version of numpy to 2.0
• set minimum version of scipy to 1.18

@seanlaw
If we bump the minimum versions of numpy and scipy just for this, it affects everyone, not just users of this feature. So, anyone upgrading to the upcoming STUMPY release will need numpy>=2.0 and scipy>=1.18. The current minimum versions are:

stumpy/pyproject.toml

Lines 39 to 43 in f7af69a

	dependencies = [
	"numpy >= 1.24",
	"scipy >= 1.10",
	"numba >= 0.61.2"
	]

That's a big jump and could cause dependency conflicts in users' environments. Thoughts?

[seanlaw]

Okay, the issue is resolved when I upgraded scipy to its latest version in my env and then:

That's great (but see comment below)!

We need to:

set minimum version of numpy to 2.0
set minimum version of scipy to 1.18

Can you please confirm that you determined this by running versions.py? Can you provide the full output from versions.py?

That's a big jump and could cause dependency conflicts in users' environments. Thoughts?

I think "it is what it is" (i.e., this is the reality that we are faced with) and, since we follow the NumPy deprecation policy, according to their timeline, then we should be okay.

Perhaps, we can consider doing:

try:
    from scipy.fft._pocketfft.basic import c2r, r2c
except ModuleNotFoundError:
    from scipy.fft._duccfft.basic import c2r, r2c

[NimaSarajpoor]

Can you please confirm that you determined this by running versions.py? Can you provide the full output from versions.py?

No, I did not determine this by running versions.py. I think you want to see the minimum version of dependencies if python version is set to 3.12, the minimum version that supports scipy 1.18. Is that correct? python version.py -mode min 3.12 gives error unfortunately (Created a separate issue: see #1143).

Or, did you mean python version.py? IIUC, this returns the min versions of dependencies when the python's not-at-EOL version is set to its minimum, which is 3.10 at this time. I believe you are not looking for this.

I think "it is what it is" (i.e., this is the reality that we are faced with) and, since we follow the NumPy deprecation policy, according to their timeline, then we should be okay.

Thanks for pointing that out! Okay then 👍

Perhaps, we can consider doing:

I did think about that... and do not know if that is okay or not. Tbh, I have slightly bad feeling about it. We are trying to handle different versions of a dependency, which is a bit dirty IMO (??)

[seanlaw]

We are trying to handle different versions of a dependency, which is a bit dirty IMO (??)

I think it's reasonable. Eventually, we would remove the from scipy.fft._pocketfft.basic import c2r, r2c line. Perhaps, since _pocketfft will be phased out, we should flip the order and do:

try:
    from scipy.fft._duccfft.basic import c2r, r2c
except ModuleNotFoundError:
    from scipy.fft._pocketfft.basic import c2r, r2c

To me, this is the exact use case for why there exists a ModuleNotFoundError and how we should be handling it. If we do this, I don't think we need to bump any dependency versions

The tests are passing. We need to:

set minimum version of numpy to 2.0
set minimum version of scipy to 1.18

Can you explain why we would need to bump the minimum numpy and scipy version? I don't think we do if we add the try/except and follow the natural end-of-life for those packages

[NimaSarajpoor]

we should flip the order ...
I don't think we do if we add the try/except

Yes and Yes.

To me, this is the exact use case for why there exists a ModuleNotFoundError and how we should be handling it.

Cool 👍 Will submit a PR.