Skip to content

BUG: Order find results by score inside the search region. - #67

Open
breken-ai wants to merge 1 commit into
botcity-dev:mainfrom
breken-ai:fix/main/find-region-offset-crash
Open

breken-ai wants to merge 1 commit into
botcity-dev:mainfrom
breken-ai:fix/main/find-region-offset-crash

Conversation

@breken-ai

@breken-ai breken-ai commented Sep 25, 2026 •

Copy link
Copy Markdown

Fixes #70

Description

find / find_until / get_element_coords crash with IndexError (or rank matches by the wrong score) whenever the search region does not start at the top-left corner of the screen, for example bot.find("label", x=100).

locate_all_opencv sorts its matches with key=lambda p: result[p[1]][p[0]], where p is the returned screen position. That position already includes the region offset (matches[1] * step + region[0]), but result (the cv2.matchTemplate output) is indexed from the region's own origin. So the key reads the score of a different pixel, and once x + region[0] passes the end of result it raises:

IndexError: index 110 is out of bounds for axis 0 with size 91

The sort was added in #25 ("Order find results by best match."); any x or y offset has hit this path since then.

Fix

Take the scores at the match positions inside result (result[matches]), sort by them, and keep adding the offset only for the returned Box. Behaviour with no offset is unchanged.

Tests

New botcity/core/tests/test_cv2find.py, pure numpy images, no real screen:

  • locate_all_opencv(..., region=(100, 0, 100, 100)) returns the match at (150, 60)
  • with two matches in an offset region, the exact copy comes before the damaged copy
  • DesktopBot.find("needle", x=100) with get_screenshot / _fix_display_size stubbed returns the element

On main all three fail with the IndexError above; with this change python run_tests.py passes (4 passed) and flake8 botcity is clean.


This PR was prepared by an AI coding agent (Breken) running as the breken-ai account. I checked the change and the tests before opening it and am happy to change anything.

locate_all_opencv sorted its matches by reading the matchTemplate result
at the screen position (x, y). That position already includes the
region offset, but the result array is indexed from the region's own
origin. A search that starts away from the top-left corner, such as
find(label, x=100), therefore read the wrong scores or raised
IndexError as soon as it found a match.

Sort by the score taken at the match's position inside the result
array, then add the offset for the returned box.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

find in offset screen region raises IndexError or ranks matches incorrectly

1 participant