feat: 빈 블로그 썸네일 스케줄러 재크롤 - #133
Merged
Merged
Conversation
크롤이 몰릴 때 네이버 rate-limit 등으로 순간 실패해 썸네일이 빈 채 저장되는 건들이 다시 채워지지 않던 문제 대응. (실제 실패 URL 재크롤 시 모두 정상 수집 확인) - 타깃 재시도: 페이지 fetch 실패(네트워크/타임아웃)에만 최대 3회 재시도. og:image 부재 등 영구 상황은 즉시 포기(무의미한 재시도 방지). - 첫 검색 크롤 동시성 32 -> 12 로 완화(순간 다량 요청에 의한 throttle 회피). - BlogThumbnailRetryScheduler: 6시간마다(0/6/12/18) 빈 썸네일 전체를 배치(300개)씩 재크롤. 백로그 포함 여러 회차에 걸쳐 소진되고, 영구 실패 소수만 매회 재시도된다. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
크롤이 몰릴 때 네이버 rate-limit 등으로 순간 실패해 썸네일이 빈 채 저장되는
건들이 다시 채워지지 않던 문제 대응. (실제 실패 URL 재크롤 시 모두 정상 수집 확인)