Fix task rescheduling on worker shutdown - #2
Merged
fangpenlin merged 1 commit intoDec 20, 2024
Merged
Conversation
Contributor
|
thanks for fixing the problem 🙇♀️ |
Fazel94
pushed a commit
to Fazel94/bq
that referenced
this pull request
Nov 23, 2025
Identified 8 bugs in threading implementation: CRITICAL (3): - Bug LaunchPlatform#1: Database session never closed in worker heartbeat thread - Bug LaunchPlatform#2: AttributeError when worker is None in HTTP health check - Bug LaunchPlatform#3: Session leak in HTTP request handler HIGH (2): - Bug LaunchPlatform#4: No error handling in worker heartbeat thread - Bug LaunchPlatform#5: Stale worker object in heartbeat thread MODERATE (3): - Bug LaunchPlatform#6: Race condition in metrics server shutdown - Bug LaunchPlatform#7: Potential duplicate NOTIFY when transaction is None - Bug LaunchPlatform#8: Thread join timeout may leave zombie threads Report includes detailed descriptions, reproduction steps, recommended fixes, and priority recommendations for each bug.
Fazel94
pushed a commit
to Fazel94/bq
that referenced
this pull request
Dec 30, 2025
CRITICAL FIXES (3): - Bug LaunchPlatform#1: Fixed session management in update_workers thread * Create fresh session each iteration instead of reusing * Always close session in finally block * Prevents connection pool exhaustion and stale connections - Bug LaunchPlatform#2: Fixed AttributeError in HTTP health check * Check if worker is None before accessing worker.state * Return proper error state ("NOT_FOUND") when worker is missing * Prevents HTTP server crashes - Bug LaunchPlatform#3: Fixed session leak in HTTP request handler * Close database session in finally block for all requests * Prevents connection pool exhaustion from health checks HIGH SEVERITY FIXES (2): - Bug LaunchPlatform#4: Added error handling to update_workers thread * Wrap all database operations in try/except * Log errors with full traceback * Rollback failed transactions * Continue heartbeat loop after errors (no silent death) - Bug LaunchPlatform#5: Fixed stale worker object in heartbeat thread * Refresh worker object from database each iteration * Check if worker exists (handle deletion case) * Prevents updates to detached/stale objects MODERATE FIXES (3): - Bug LaunchPlatform#6: Fixed metrics server shutdown race condition * Use threading.Event instead of callback assignment * Store server instance in thread-safe manner * Prevents race condition during early shutdown - Bug LaunchPlatform#7: Added logging for duplicate NOTIFY * Log debug message when transaction is None * Explains why deduplication is not possible * Minor performance impact, not critical - Bug LaunchPlatform#8: Fixed thread join timeout handling * Check if threads stopped after join timeout * Log errors if threads still alive * Only mark worker as SHUTDOWN if heartbeat thread stopped * Prevents zombie threads and state inconsistencies All fixes tested with syntax check and module import. Module instantiation successful.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hey there! Loving bq so far. I noticed when I gracefully shutdown
process_tasksI was getting a TypeError. This fixes it and passes all tests locally.