feat(apify): delete the scrape-notification email type (both senders) - #829
Conversation
Scraping a social profile emailed every owner of every artist scraped, one
email per artist. On a real label roster that is a flood: 109 emails to a
single recipient over 7 days, peaking at 50 in one day, which is what prompted
the customer complaint behind chat#1955.
There were TWO live senders and they had to go together, or main would keep
emailing customers from the other one:
- digest/maybeSendScrapeDigest, the batched "N new posts across N platforms"
email, fired from apifyWebhookHandler for every scrape batch
- sendApifyWebhookEmail, the legacy single-platform Instagram alert, fired
from handleInstagramProfileScraperResults for non-batch runs
They were also coupled: the legacy sender imported the digest module's renderer
and extractors, so deleting the digest alone would have broken that path rather
than removed it.
Deletes lib/apify/digest (20 source + 5 tests), the legacy sender and its test,
and the two apify_scraper_runs select helpers that only the digest read. The
extractors were email-only; nothing in the post-persistence path used them.
Keeps the apify_scraper_runs table and its write path. Registration plus
completion is the only record of which scrapes ran and what was new, which is
useful for debugging independently of email and cost real time to reconstruct
without during this incident. No migration.
Scraping, post persistence, Instagram follow-up comment scrapes, and the
/api/socials/{id}/scrape contract are unchanged. The webhook response body
loses its now-meaningless sentEmails field.
Tests assert the invariant at the Resend boundary rather than at the deleted
modules, so it keeps holding through future refactors: a completed scrape
webhook, and an Instagram scrape that finds new posts, both send nothing.
4496 tests pass. grep for sendEmailWithResend under lib/apify returns nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
📝 WalkthroughWalkthroughThe PR removes Apify scrape-digest generation and email notification modules. Webhook handlers retain scraper-run completion and new-post bookkeeping. Instagram result handling continues follow-up scraping and returns account email data without sending emails. ChangesApify notification removal
Estimated code review effort: 3 (Moderate) | ~20 minutes Possibly related issues
Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (2)
lib/apify/apifyWebhookHandler.ts (1)
32-44: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winExtract scrape-run bookkeeping from
apifyWebhookHandler.
apifyWebhookHandleris longer than 20 lines and performs request validation, actor dispatch, audit persistence, failure policy, and response construction. Move the bookkeeping block into a focused helper. This keeps the route handler focused and makes the audit policy easier to test.As per coding guidelines, functions longer than 20 lines must be flagged and functions must stay small and focused. As per path instructions,
lib/**/*.tsfunctions must have a single responsibility.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@lib/apify/apifyWebhookHandler.ts` around lines 32 - 44, Extract the scrape-run bookkeeping block from apifyWebhookHandler into a focused helper responsible only for recording the run ID and new post URLs, including its try/catch and warning behavior. Call this helper from apifyWebhookHandler after actor processing, while preserving the never-fail webhook policy and existing updateApifyScraperRun inputs.Sources: Coding guidelines, Path instructions
lib/apify/instagram/handleInstagramProfileScraperResults.ts (1)
98-107: 📐 Maintainability & Code Quality | 🔵 Trivial | 🏗️ Heavy liftSplit
handleInstagramProfileScraperResultsinto focused helpers.The function spans about 77 lines. It loads the dataset, computes new URLs, persists posts and social records, resolves account data, starts follow-up runs, and assembles the result. Extract the persistence and account/follow-up responsibilities. If a helper is exported, place it in a file whose name matches the function.
As per coding guidelines, functions longer than 20 lines must be flagged and functions must stay small and focused. As per path instructions,
lib/**/*.tsfunctions must have one responsibility and stay under 50 lines.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@lib/apify/instagram/handleInstagramProfileScraperResults.ts` around lines 98 - 107, Split handleInstagramProfileScraperResults into focused helpers, extracting the post/social persistence work and the account resolution plus handleInstagramProfileFollowUpRuns flow while preserving the returned { posts, social, accountSocials, accountEmails, newPostUrls } contract. Keep each helper under 50 lines with one responsibility; if any helper is exported, place it in a file matching that helper’s function name, and have the main function only coordinate the focused operations.Sources: Coding guidelines, Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@lib/apify/apifyWebhookHandler.ts`:
- Around line 40-42: Update handleInstagramProfileScraperResults so its
no-social-record early return includes the already computed newPostUrls,
preserving those URLs on every return path. Add a regression test covering the
missing social record and asserting the returned newPostUrls are passed through
to the handler’s updateApifyScraperRun flow.
- Around line 43-44: Make the bookkeeping flow around updateApifyScraperRun
recoverable before the webhook returns its intentional HTTP 200: replace the
current swallowed-error-only handling with an idempotent retry, durable outbox,
or reconciliation mechanism that persists and later applies failed completed_at
and new_post_urls updates. Keep the response status 200 while ensuring failed
bookkeeping is not lost.
In `@lib/apify/instagram/handleInstagramProfileScraperResults.ts`:
- Around line 98-100: Remove the unused accountEmails lookup and eliminate
accountEmails from the handler result returned by the Instagram profile scraper
flow. Update the relevant handler function and response construction while
preserving all other scraping behavior; only retain the field if an explicit
external API contract requires it.
---
Nitpick comments:
In `@lib/apify/apifyWebhookHandler.ts`:
- Around line 32-44: Extract the scrape-run bookkeeping block from
apifyWebhookHandler into a focused helper responsible only for recording the run
ID and new post URLs, including its try/catch and warning behavior. Call this
helper from apifyWebhookHandler after actor processing, while preserving the
never-fail webhook policy and existing updateApifyScraperRun inputs.
In `@lib/apify/instagram/handleInstagramProfileScraperResults.ts`:
- Around line 98-107: Split handleInstagramProfileScraperResults into focused
helpers, extracting the post/social persistence work and the account resolution
plus handleInstagramProfileFollowUpRuns flow while preserving the returned {
posts, social, accountSocials, accountEmails, newPostUrls } contract. Keep each
helper under 50 lines with one responsibility; if any helper is exported, place
it in a file matching that helper’s function name, and have the main function
only coordinate the focused operations.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 59d9df92-594a-4e35-83fd-0bf8e9a55025
⛔ Files ignored due to path filters (8)
lib/apify/__tests__/apifyWebhookHandler.test.tsis excluded by!**/*.test.*,!**/__tests__/**and included bylib/**lib/apify/__tests__/sendApifyWebhookEmail.test.tsis excluded by!**/*.test.*,!**/__tests__/**and included bylib/**lib/apify/digest/__tests__/extractArtistNameFromDatasetItems.test.tsis excluded by!**/*.test.*,!**/__tests__/**and included bylib/**lib/apify/digest/__tests__/extractPostsFromDatasetItems.test.tsis excluded by!**/*.test.*,!**/__tests__/**and included bylib/**lib/apify/digest/__tests__/maybeSendScrapeDigest.test.tsis excluded by!**/*.test.*,!**/__tests__/**and included bylib/**lib/apify/digest/__tests__/renderScrapeDigestHtml.test.tsis excluded by!**/*.test.*,!**/__tests__/**and included bylib/**lib/apify/digest/__tests__/sendScrapeDigestEmail.test.tsis excluded by!**/*.test.*,!**/__tests__/**and included bylib/**lib/apify/instagram/__tests__/handleInstagramProfileScraperResults.test.tsis excluded by!**/*.test.*,!**/__tests__/**and included bylib/**
📒 Files selected for processing (25)
lib/apify/apifyWebhookHandler.tslib/apify/digest/asNumberOrNull.tslib/apify/digest/asRecord.tslib/apify/digest/asStringOrNull.tslib/apify/digest/buildPostStats.tslib/apify/digest/extractArtistNameFromDatasetItems.tslib/apify/digest/extractInstagramPosts.tslib/apify/digest/extractPostsFromDatasetItems.tslib/apify/digest/extractTiktokPosts.tslib/apify/digest/extractTwitterPosts.tslib/apify/digest/formatCompactCount.tslib/apify/digest/formatPostStats.tslib/apify/digest/formatUtcDateLabel.tslib/apify/digest/getPlatformLabel.tslib/apify/digest/getRunDigestSection.tslib/apify/digest/getScrapeDigestRecipients.tslib/apify/digest/maybeSendScrapeDigest.tslib/apify/digest/parseNewPostUrls.tslib/apify/digest/renderScrapeDigestHtml.tslib/apify/digest/sendScrapeDigestEmail.tslib/apify/digest/truncateText.tslib/apify/instagram/handleInstagramProfileScraperResults.tslib/apify/sendApifyWebhookEmail.tslib/supabase/apify_scraper_runs/selectApifyScraperRun.tslib/supabase/apify_scraper_runs/selectApifyScraperRuns.ts
💤 Files with no reviewable changes (23)
- lib/apify/digest/getPlatformLabel.ts
- lib/apify/digest/formatPostStats.ts
- lib/apify/digest/truncateText.ts
- lib/apify/digest/asStringOrNull.ts
- lib/apify/digest/asRecord.ts
- lib/apify/digest/formatUtcDateLabel.ts
- lib/apify/digest/asNumberOrNull.ts
- lib/apify/digest/getRunDigestSection.ts
- lib/apify/digest/extractArtistNameFromDatasetItems.ts
- lib/supabase/apify_scraper_runs/selectApifyScraperRuns.ts
- lib/apify/digest/buildPostStats.ts
- lib/apify/sendApifyWebhookEmail.ts
- lib/apify/digest/getScrapeDigestRecipients.ts
- lib/apify/digest/maybeSendScrapeDigest.ts
- lib/apify/digest/sendScrapeDigestEmail.ts
- lib/apify/digest/parseNewPostUrls.ts
- lib/apify/digest/extractPostsFromDatasetItems.ts
- lib/apify/digest/formatCompactCount.ts
- lib/apify/digest/extractTwitterPosts.ts
- lib/apify/digest/renderScrapeDigestHtml.ts
- lib/supabase/apify_scraper_runs/selectApifyScraperRun.ts
- lib/apify/digest/extractTiktokPosts.ts
- lib/apify/digest/extractInstagramPosts.ts
| const newPostUrls = | ||
| (result as { newPostUrls?: string[] } | null | undefined)?.newPostUrls ?? []; | ||
| const run = await updateApifyScraperRun(runId, newPostUrls); | ||
| await maybeSendScrapeDigest(run?.batch_id); | ||
| } catch (digestError) { | ||
| console.error("[WARN] scrape digest bookkeeping failed:", digestError); | ||
| await updateApifyScraperRun(runId, newPostUrls); |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Preserve newPostUrls on every handler return path.
lib/apify/instagram/handleInstagramProfileScraperResults.ts computes newPostUrls at Line 45 but returns without it at Line 75 when no social record is found. The ?? [] fallback here then records no new URLs for that scrape. Return newPostUrls from that early path and add a no-social regression test.
Suggested fix
- if (!social) return { posts, social: null };
+ if (!social) return { posts, social: null, newPostUrls };🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@lib/apify/apifyWebhookHandler.ts` around lines 40 - 42, Update
handleInstagramProfileScraperResults so its no-social-record early return
includes the already computed newPostUrls, preserving those URLs on every return
path. Add a regression test covering the missing social record and asserting the
returned newPostUrls are passed through to the handler’s updateApifyScraperRun
flow.
| } catch (bookkeepingError) { | ||
| console.error("[WARN] scrape run bookkeeping failed:", bookkeepingError); |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -eu
printf '%s\n' '--- candidate files ---'
fd -i 'apifyWebhookHandler|apify|webhook' . | head -200
printf '%s\n' '--- handler outline ---'
handler="$(fd -i -t f 'apifyWebhookHandler.ts' . | head -1)"
if [ -n "${handler}" ]; then
ast-grep outline "$handler" || true
printf '%s\n' '--- handler contents ---'
cat -n "$handler"
fi
printf '%s\n' '--- related symbols ---'
rg -n --hidden -g '!node_modules' -g '!dist' -g '!build' \
'updateApifyScraperRun|apifyWebhookHandler|webhook.*retry|retry.*webhook|APIFY' .
printf '%s\n' '--- package and configuration files ---'
git ls-files | rg '(^|/)(package\.json|.*config.*|.*workflow.*|README.*|.*\.md$|.*\.env.*)$' | head -200Repository: recoupable/api
Length of output: 9667
🏁 Script executed:
#!/bin/bash
set -eu
python3 - <<'PY'
from pathlib import Path
import re
paths = list(Path(".").rglob("apifyWebhookHandler.ts"))
if not paths:
print("HANDLER_NOT_FOUND")
raise SystemExit(0)
path = paths[0]
text = path.read_text()
print(f"HANDLER={path}")
print(f"LINES={len(text.splitlines())}")
print("RESPONSE_CONSTRUCTORS:")
for i, line in enumerate(text.splitlines(), 1):
if re.search(r'NextResponse|Response|status\s*:\s*200|status\s*:\s*5\d\d|throw|catch', line):
print(f"{i}: {line.strip()}")
print("PERSISTENCE_CALLS:")
for i, line in enumerate(text.splitlines(), 1):
if "updateApifyScraperRun" in line:
print(f"{i}: {line.strip()}")
print("CATCH_BLOCKS:")
for match in re.finditer(r'catch\s*\(([^)]*)\)\s*\{', text):
start = match.start()
line = text.count("\n", 0, start) + 1
depth = 0
end = None
for pos in range(match.end(), len(text)):
if text[pos] == "{":
depth += 1
elif text[pos] == "}":
if depth == 0:
end = pos
break
depth -= 1
block = text[match.start():end + 1 if end else len(text)]
print(f"catch_at_line={line}:")
print(block[:1000])
PYRepository: recoupable/api
Length of output: 1217
🏁 Script executed:
#!/bin/bash
set -eu
printf '%s\n' '--- persistence implementation ---'
cat -n lib/supabase/apify_scraper_runs/updateApifyScraperRun.ts
printf '%s\n' '--- route ---'
cat -n app/api/apify/route.ts
printf '%s\n' '--- focused handler tests ---'
cat -n lib/apify/__tests__/apifyWebhookHandler.test.ts
printf '%s\n' '--- scraper-run files ---'
find lib/supabase/apify_scraper_runs -maxdepth 2 -type f -print -exec sh -c 'echo "--- $1"; cat -n "$1"' sh {} \;
printf '%s\n' '--- recovery mechanisms ---'
rg -n --hidden -g '!node_modules' -g '!dist' -g '!build' \
'outbox|dead.?letter|retry|retries|queue|scraper.?run|apify_scraper_runs' \
app lib supabase db migrations 2>/dev/null | head -300Repository: recoupable/api
Length of output: 45331
🏁 Script executed:
#!/bin/bash
set -eu
python3 - <<'PY'
from pathlib import Path
import re
handler = Path("lib/apify/apifyWebhookHandler.ts").read_text()
persistence = Path("lib/supabase/apify_scraper_runs/updateApifyScraperRun.ts").read_text()
print("HANDLER_STATUS_LITERALS")
for line_no, line in enumerate(handler.splitlines(), 1):
if re.search(r'status:\s*200|status:\s*5\d\d|NextResponse\.json', line):
print(f"{line_no}: {line.strip()}")
print("PERSISTENCE_ERROR_FLOW")
for line_no, line in enumerate(persistence.splitlines(), 1):
if re.search(r'try|catch|throw|return|error|update|upsert', line, re.I):
print(f"{line_no}: {line.strip()}")
# A small control-flow probe for the specific claim: persistence rejection is
# caught locally, and the success response remains 200.
async def probe():
calls = []
async def update():
calls.append("update")
raise RuntimeError("transient database failure")
try:
try:
await update()
except Exception as error:
calls.append(f"logged:{error}")
response = {"status": 200}
except Exception:
response = {"status": 200, "outer": True}
return calls, response
import asyncio
calls, response = asyncio.run(probe())
print("PROBE_CALLS=", calls)
print("PROBE_RESPONSE=", response)
assert calls == ["update", "logged:transient database failure"]
assert response["status"] == 200
PYRepository: recoupable/api
Length of output: 873
🏁 Script executed:
#!/bin/bash
set -eu
python3 - <<'PY'
from pathlib import Path
import re
handler = Path("lib/apify/apifyWebhookHandler.ts").read_text()
persistence = Path("lib/supabase/apify_scraper_runs/updateApifyScraperRun.ts").read_text()
print("HANDLER_STATUS_LITERALS")
for line_no, line in enumerate(handler.splitlines(), 1):
if re.search(r'status:\s*200|status:\s*5\d\d|NextResponse\.json', line):
print(f"{line_no}: {line.strip()}")
print("PERSISTENCE_ERROR_FLOW")
for line_no, line in enumerate(persistence.splitlines(), 1):
if re.search(r'try|catch|throw|return|error|update|upsert', line, re.I):
print(f"{line_no}: {line.strip()}")
async def probe():
calls = []
async def update():
calls.append("update")
raise RuntimeError("transient database failure")
try:
try:
await update()
except Exception as error:
calls.append(f"logged:{error}")
response = {"status": 200}
except Exception:
response = {"status": 200, "outer": True}
return calls, response
import asyncio
calls, response = asyncio.run(probe())
print("PROBE_CALLS=", calls)
print("PROBE_RESPONSE=", response)
assert calls == ["update", "logged:transient database failure"]
assert response["status"] == 200
PYRepository: recoupable/api
Length of output: 873
🌐 Web query:
Apify webhook retry policy HTTP 2xx response webhook retries official documentation
💡 Result:
Apify requires that webhook endpoints respond with an HTTP status code in the 2xx range to confirm successful delivery [1][2]. Any response outside of the 2xx range, or a request timeout, is treated as an error and triggers an automatic retry mechanism [1][3]. The retry policy utilizes an exponential back-off strategy [1][2]: - The system attempts up to 11 retries [1][2]. - The intervals between retries increase progressively, with the eleventh retry occurring after approximately 32 hours [1][2]. - After 11 failed attempts, the system ceases retries for that dispatch [1][2]. Important considerations for implementing your endpoint: - Timeout: Webhook requests have a strict timeout (official documentation specifies 30 seconds [3], while some external resources note 2 minutes [1][2]). It is highly recommended to respond with a 2xx status code immediately upon receiving the payload and process any time-consuming operations asynchronously (e.g., using a message queue) [1][4][5]. - Idempotency: Because network issues or other factors can occasionally lead to duplicate webhook deliveries even after a successful 2xx response, you should design your endpoint to be idempotent [5][3]. Using a unique identifier, such as the actorRunId, to track processed events is recommended [5][6]. - Disabling Retries: You can configure a webhook to disable retries by setting the doNotRetry option to true [7][6].
Citations:
- 1: https://docs.apify.com/platform/integrations/webhooks/actions
- 2: https://docs.apify.com/integrations/webhooks/actions
- 3: https://docs.apify.com/platform/integrations/webhooks/actions.md
- 4: https://use-apify.com/blog/apify-webhooks-complete-guide
- 5: https://apifyhub.com/guides/send-results-to-webhook
- 6: https://docs.apify.com/sdk/js/reference/interface/WebhookOptions
- 7: https://docs.apify.com/api/v2/webhooks-post
Make scrape-run bookkeeping recoverable before acknowledging the webhook. updateApifyScraperRun swallows Supabase errors, and the handler also logs bookkeeping failures before returning HTTP 200. This can lose completed_at and new_post_urls; Apify retries only non-2xx responses or timeouts. Add an idempotent retry, durable outbox, or reconciliation path that preserves the intentional 200 response.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@lib/apify/apifyWebhookHandler.ts` around lines 43 - 44, Make the bookkeeping
flow around updateApifyScraperRun recoverable before the webhook returns its
intentional HTTP 200: replace the current swallowed-error-only handling with an
idempotent retry, durable outbox, or reconciliation mechanism that persists and
later applies failed completed_at and new_post_urls updates. Keep the response
status 200 while ensuring failed bookkeeping is not lost.
| // Scraping no longer notifies anyone: the per-platform alert and the batched | ||
| // digest were both removed (chat#1955). `accountEmails` is still resolved and | ||
| // returned because callers use it, but nothing is sent. |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n --glob '*.{ts,tsx,js,jsx}' '\b(accountEmails|selectAccountEmails)\b' .Repository: recoupable/api
Length of output: 15299
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- function file outline ---'
ast-grep outline lib/apify/instagram/handleInstagramProfileScraperResults.ts --view expanded
printf '%s\n' '--- function implementation ---'
cat -n lib/apify/instagram/handleInstagramProfileScraperResults.ts
printf '%s\n' '--- webhook handler ---'
cat -n lib/apify/apifyWebhookHandler.ts
printf '%s\n' '--- direct call sites and result-field consumers ---'
rg -n -C 4 --glob '*.{ts,tsx,js,jsx}' \
'\bhandleInstagramProfileScraperResults\b|\baccountEmails\b|\bnewPostUrls\b|\baccountSocials\b' \
lib appRepository: recoupable/api
Length of output: 48831
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- all repository references ---'
rg -n -C 3 '\b(accountEmails|selectAccountEmails)\b' . \
-g '!node_modules' -g '!dist' -g '!build'
printf '%s\n' '--- account email selector ---'
cat -n lib/supabase/account_emails/selectAccountEmails.ts
printf '%s\n' '--- Instagram handler tests ---'
cat -n lib/apify/instagram/__tests__/handleInstagramProfileScraperResults.test.ts
printf '%s\n' '--- webhook response tests ---'
sed -n '55,185p' lib/apify/__tests__/apifyWebhookHandler.test.ts
printf '%s\n' '--- API and contract references ---'
rg -n -i -C 3 'apify|instagram.*scrap|scraper.*result|accountEmails' \
README.md docs app lib .github 2>/dev/null || trueRepository: recoupable/api
Length of output: 50371
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- tracked production references ---'
git grep -n -E '\b(accountEmails|selectAccountEmails)\b' -- \
'*.ts' '*.tsx' '*.js' '*.jsx' \
':(exclude)**/__tests__/**' ':(exclude)**/*.test.*' ':(exclude)**/*.spec.*' \
| sed -n '1,220p'
printf '%s\n' '--- selector implementation ---'
cat -n lib/supabase/account_emails/selectAccountEmails.ts
printf '%s\n' '--- Instagram handler test assertions ---'
sed -n '55,170p' lib/apify/instagram/__tests__/handleInstagramProfileScraperResults.test.ts
printf '%s\n' '--- webhook response assertions ---'
sed -n '55,185p' lib/apify/__tests__/apifyWebhookHandler.test.ts
printf '%s\n' '--- route references ---'
git grep -n -E 'apifyWebhookHandler|/api/apify' -- \
'*.ts' '*.tsx' '*.js' '*.jsx' '*.md' \
| sed -n '1,180p'Repository: recoupable/api
Length of output: 20803
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- webhook route and validator ---'
cat -n app/api/apify/route.ts
cat -n lib/apify/validateApifyWebhookRequest.ts
printf '%s\n' '--- static production-consumer verifier ---'
python3 - <<'PY'
from pathlib import Path
import re
handler = Path("lib/apify/instagram/handleInstagramProfileScraperResults.ts").read_text()
tracked = []
for path in Path(".").rglob("*"):
if not path.is_file() or any(part in {"node_modules", ".git", "dist", "build"} for part in path.parts):
continue
if path.suffix not in {".ts", ".tsx", ".js", ".jsx"}:
continue
if "__tests__" in path.parts or path.name.endswith((".test.ts", ".test.tsx", ".spec.ts", ".spec.tsx")):
continue
text = path.read_text(errors="ignore")
if re.search(r"\baccountEmails\b", text):
tracked.append(str(path))
print("production files containing accountEmails:")
for path in sorted(tracked):
print(path)
print("handler return includes accountEmails:", bool(re.search(r"return\s*\{[^}]*\baccountEmails\b", handler, re.S)))
print("handler performs selectAccountEmails:", "selectAccountEmails({" in handler)
PYRepository: recoupable/api
Length of output: 4161
Remove the unused accountEmails lookup and response field
POST /api/apify returns the handler result directly. The selector retrieves all account-email columns, including email addresses. Remove the lookup and returned field unless an external API contract requires them.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@lib/apify/instagram/handleInstagramProfileScraperResults.ts` around lines 98
- 100, Remove the unused accountEmails lookup and eliminate accountEmails from
the handler result returned by the Instagram profile scraper flow. Update the
relevant handler function and response construction while preserving all other
scraping behavior; only retain the field if an explicit external API contract
requires it.
There was a problem hiding this comment.
3 issues found across 33 files
Confidence score: 4/5
- In
lib/apify/__tests__/apifyWebhookHandler.test.ts, the webhook test uses an unregistered actorId, so it never reaches the scrape handler path it claims to protect; this leaves a real regression gap where scrape handling could break unnoticed—register a valid actorId (or mockHANDLERS_BY_ACTOR_ID) so the intended branch is actually exercised. - In
lib/apify/instagram/handleInstagramProfileScraperResults.ts, the comment about retainingaccountEmailsconflicts with current flow (apifyWebhookHandleronly consumesnewPostUrls), which can mislead future changes and preserve dead fields—either removeaccountEmailsfrom the selected output or update the comment/usage to match reality. - In
lib/apify/instagram/__tests__/handleInstagramProfileScraperResults.test.ts, the test name still promises email behavior while assertingsendEmailWithResendis not called; this mismatch increases maintenance risk and reviewer confusion—rename the test to reflect current behavior (posts/linking without email).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="lib/apify/instagram/__tests__/handleInstagramProfileScraperResults.test.ts">
<violation number="1" location="lib/apify/instagram/__tests__/handleInstagramProfileScraperResults.test.ts:95">
P3: This test title still says it "fires follow-up runs + email", but the change makes it assert `sendEmailWithResend` is never called and the implementation no longer sends email. Rename it (e.g., "persists posts, links social_posts, and fires follow-up runs, sending no email") so it documents the new invariant.</violation>
</file>
<file name="lib/apify/__tests__/apifyWebhookHandler.test.ts">
<violation number="1" location="lib/apify/__tests__/apifyWebhookHandler.test.ts:174">
P2: This test does not exercise the scrape path it claims to guard. The actorId "shu8hvrXbJbY3Eb9W" is unregistered in getApifyResultHandler's HANDLERS_BY_ACTOR_ID, so apifyWebhookHandler returns the "Unhandled actorId" 200 error before any handler runs and sendEmailWithResend is trivially never called. The test passes for the wrong reason and gives false confidence that the no-email invariant holds at the boundary. Use a registered actorId such as "dSCLg0C3YEZ83HzYX" so a real scrape handler actually runs.</violation>
</file>
<file name="lib/apify/instagram/handleInstagramProfileScraperResults.ts">
<violation number="1" location="lib/apify/instagram/handleInstagramProfileScraperResults.ts:100">
P3: The added comment claims `accountEmails` is retained "because callers use it," but no caller reads it: getApifyResultHandler passes the result to apifyWebhookHandler, which only reads `newPostUrls`. That leaves `selectAccountEmails` (and the `accountArtistIds`/`uniqueAccountIds` chain feeding it) as dead DB work on every Instagram scrape, and the comment is misleading. Drop the `selectAccountEmails` call, the intermediate chain, the `accountEmails` return field, and reword the comment.</violation>
</file>
Architecture diagram
sequenceDiagram
participant Webhook as Apify Webhook<br/>POST /api/apify
participant Handler as apifyWebhookHandler
participant Validator as validateApifyWebhookRequest
participant Results as handleInstagramProfileScraperResults
participant Persist as Supabase<br/>Posts/Socials
participant Audit as apify_scraper_runs
participant FollowUp as Instagram Follow-Up<br/>Scraper
participant Client as Apify Client
Note over Webhook,Client: Scrape Webhook Flow (no notification emails sent)
Webhook->>Handler: Incoming scrape webhook
Handler->>Validator: Validate request signature + payload
alt Valid webhook payload
Validator-->>Handler: Validated payload
Handler->>Results: Route to platform handler (Instagram profile)
Results->>Results: Extract dataset items
Results->>Persist: Upsert posts + social profile
Results->>Persist: Link social-posts associations
Results->>Persist: Query existing posts
Results->>Results: Filter genuinely new post URLs
Note over Results,Client: Email send path removed -<br/>no sendEmailWithResend call
Results->>Audit: Resolve account emails (for return value only)
Results->>FollowUp: Queue comments scraper for new posts
FollowUp-->>Results: Follow-up runs dispatched
Results-->>Handler: Summary { posts, social, newPostUrls }
Handler->>Audit: Update scrape run (runId, newPostUrls)
Note over Handler,Audit: Audit trail kept, no digest triggered
Audit-->>Handler: Run record updated
Handler-->>Webhook: 200 OK
else Invalid payload
Validator-->>Handler: Validation error
Handler-->>Webhook: Error response
end
Note over Webhook,Client: Follow-up Instagram Comments Scrape
Webhook->>Handler: Comments scrape webhook
Handler->>Validator: Validate
Validator-->>Handler: Validated payload
Handler->>Client: Dispatch comments scraper handler
Client-->>Handler: Comments processed
Handler-->>Webhook: 200 OK
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
| const res = await apifyWebhookHandler( | ||
| makeRequest({ | ||
| ...baseBody, | ||
| eventData: { actorId: "shu8hvrXbJbY3Eb9W" }, |
There was a problem hiding this comment.
P2: This test does not exercise the scrape path it claims to guard. The actorId "shu8hvrXbJbY3Eb9W" is unregistered in getApifyResultHandler's HANDLERS_BY_ACTOR_ID, so apifyWebhookHandler returns the "Unhandled actorId" 200 error before any handler runs and sendEmailWithResend is trivially never called. The test passes for the wrong reason and gives false confidence that the no-email invariant holds at the boundary. Use a registered actorId such as "dSCLg0C3YEZ83HzYX" so a real scrape handler actually runs.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At lib/apify/__tests__/apifyWebhookHandler.test.ts, line 174:
<comment>This test does not exercise the scrape path it claims to guard. The actorId "shu8hvrXbJbY3Eb9W" is unregistered in getApifyResultHandler's HANDLERS_BY_ACTOR_ID, so apifyWebhookHandler returns the "Unhandled actorId" 200 error before any handler runs and sendEmailWithResend is trivially never called. The test passes for the wrong reason and gives false confidence that the no-email invariant holds at the boundary. Use a registered actorId such as "dSCLg0C3YEZ83HzYX" so a real scrape handler actually runs.</comment>
<file context>
@@ -160,4 +158,25 @@ describe("apifyWebhookHandler", () => {
+ const res = await apifyWebhookHandler(
+ makeRequest({
+ ...baseBody,
+ eventData: { actorId: "shu8hvrXbJbY3Eb9W" },
+ resource: { ...baseBody.resource, id: "run_1" },
+ }),
</file context>
| expect(upsertPosts).toHaveBeenCalledOnce(); | ||
| expect(upsertSocialPosts).toHaveBeenCalledOnce(); | ||
| expect(sendApifyWebhookEmail).toHaveBeenCalledWith(expect.any(Object), ["x@y.com"], ["u1"]); | ||
| // chat#1955: scraping must persist posts and send nothing. |
There was a problem hiding this comment.
P3: This test title still says it "fires follow-up runs + email", but the change makes it assert sendEmailWithResend is never called and the implementation no longer sends email. Rename it (e.g., "persists posts, links social_posts, and fires follow-up runs, sending no email") so it documents the new invariant.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At lib/apify/instagram/__tests__/handleInstagramProfileScraperResults.test.ts, line 95:
<comment>This test title still says it "fires follow-up runs + email", but the change makes it assert `sendEmailWithResend` is never called and the implementation no longer sends email. Rename it (e.g., "persists posts, links social_posts, and fires follow-up runs, sending no email") so it documents the new invariant.</comment>
<file context>
@@ -91,13 +87,13 @@ describe("handleInstagramProfileScraperResults", () => {
expect(upsertPosts).toHaveBeenCalledOnce();
expect(upsertSocialPosts).toHaveBeenCalledOnce();
- expect(sendApifyWebhookEmail).toHaveBeenCalledWith(expect.any(Object), ["x@y.com"], ["u1"]);
+ // chat#1955: scraping must persist posts and send nothing.
+ expect(sendEmailWithResend).not.toHaveBeenCalled();
expect(handleInstagramProfileFollowUpRuns).toHaveBeenCalledOnce();
</file context>
|
|
||
| // Scraping no longer notifies anyone: the per-platform alert and the batched | ||
| // digest were both removed (chat#1955). `accountEmails` is still resolved and | ||
| // returned because callers use it, but nothing is sent. |
There was a problem hiding this comment.
P3: The added comment claims accountEmails is retained "because callers use it," but no caller reads it: getApifyResultHandler passes the result to apifyWebhookHandler, which only reads newPostUrls. That leaves selectAccountEmails (and the accountArtistIds/uniqueAccountIds chain feeding it) as dead DB work on every Instagram scrape, and the comment is misleading. Drop the selectAccountEmails call, the intermediate chain, the accountEmails return field, and reword the comment.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At lib/apify/instagram/handleInstagramProfileScraperResults.ts, line 100:
<comment>The added comment claims `accountEmails` is retained "because callers use it," but no caller reads it: getApifyResultHandler passes the result to apifyWebhookHandler, which only reads `newPostUrls`. That leaves `selectAccountEmails` (and the `accountArtistIds`/`uniqueAccountIds` chain feeding it) as dead DB work on every Instagram scrape, and the comment is misleading. Drop the `selectAccountEmails` call, the intermediate chain, the `accountEmails` return field, and reword the comment.</comment>
<file context>
@@ -97,33 +95,14 @@ export async function handleInstagramProfileScraperResults(parsed: ApifyWebhookP
-
+ // Scraping no longer notifies anyone: the per-platform alert and the batched
+ // digest were both removed (chat#1955). `accountEmails` is still resolved and
+ // returned because callers use it, but nothing is sent.
try {
await handleInstagramProfileFollowUpRuns(dataset, firstResult);
</file context>
Preview verification: a real scrape now sends nothingPreview Why a preview test is valid here
Safety: which artist was scraped, and whyThe digest resolved recipients by artist ownership, not by who triggered the scrape. So a wrong change here would email whoever owns the scraped artist, regardless of whose credentials ran the test. I ran the exact No artist on the account satisfied "sole owner and has a scrapeable social", so I created a throwaway artist, attached a public Instagram that was not already in the The first attempt was a false pass, and I threw it outAttempt 1 used an Instagram account with Attempt 2, with real data
The social was brand new, so all 12 posts counted as genuinely new — precisely the condition that triggered the legacy Instagram alert before this PR. The scrape completed fully and sent nothing. Checked One thing this surfaced, not a regression
Suite4496 tests pass across 815 files. |
Deletes the automated "new posts" notification entirely. Scraping continues; it just stops emailing.
Tracking issue: recoupable/chat#1955
Why
Scraping a social profile emailed every owner of every artist scraped, one email per artist. On a real label roster that is a flood — measured before this PR: 109 emails to a single recipient over 7 days, peaking at 50 in one day, which is what produced the customer complaint. There was no opt-out, no feature flag, and no per-account preference anywhere in the path.
The feature was built for an account holding a handful of artists. It does not survive a 67-artist roster, and the roster in question is planning 3,000+.
Two senders, deleted together
digest/maybeSendScrapeDigestapifyWebhookHandler.ts:42sendApifyWebhookEmailhandleInstagramProfileScraperResults.ts:112They had to go in one commit. Removing only the batched digest would have left customers receiving the legacy single-platform alert, which looks like a fix and isn't. They were also coupled: the legacy sender imported
renderScrapeDigestHtmlandextractPostsFromDatasetItemsfrom the digest module, so deleting the digest first would have broken that path rather than removed it.What this does not touch
Scraping, post persistence, Instagram follow-up comment scrapes, and the
POST /api/socials/{id}/scrapecontract are all unchanged. Verified the digest extractors (extractPostsFromDatasetItems,extractInstagramPosts,extractTiktokPosts,extractTwitterPosts,buildPostStats) were email-only — nothing in the persistence path imported them.No docs change. The notification behaviour was documented nowhere:
POST /api/socials/{id}/scrapedescribes only run metadata and polling, and no reference page mentions notifications. There is no contract to retract, which is worth noting on its own — this was an undocumented side effect that emailed customers.Kept deliberately: the scrape-run audit trail
apify_scraper_runsregistration (upsertApifyScraperRuns) and completion (updateApifyScraperRun) stay. Together they record which scrapes ran, when they finished, and what was genuinely new — useful for debugging independently of email, and reconstructing this incident without leaning on it cost real time. Only the twoselect*helpers, read exclusively by the digest, are deleted. No migration; dropping the table would be a schema change with no upside today.Tests assert the invariant, not the implementation
The new assertions mock
@/lib/emails/sendEmailand checksendEmailWithResendis never called, rather than asserting the deleted modules aren't invoked. That survives future refactors: the property being protected is "a scrape sends no email", not "this particular function isn't called."The Instagram test was a genuine RED → GREEN: with the real sender still wired up it failed with
expected "spy" to not be called at all, but actually been called 1 times.grep -rn "sendEmailWithResend" lib/apifyeslinton touched pathstsc --noEmiterrors in touched filesAccepted tradeoff
This removes the only proactive "something happened" notification in the product. After this, nothing tells a customer a new post appeared; they see it by opening the app or in a scheduled report. That is a capability going away, not only a bug being fixed, taken deliberately on the evidence that the sole recorded feedback is a complaint. If proactive notifications are wanted later they should be designed for roster scale from the start (one digest per scrape session, not per artist) rather than revived from this code.
Preview verification to follow on this PR.
🤖 Generated with Claude Code
Summary by cubic
Stops all scrape-notification emails; scrapes still persist posts and record run history. Previously we sent per-artist alerts and a batched digest; now we send no email to prevent roster-scale floods (chat#1955).
lib/apify/digest/*(batched digest) andlib/apify/sendApifyWebhookEmail.ts(legacy IG alert), plusselectApifyScraperRun(s)helpers used only by the digest.apifyWebhookHandlerstill validates, routes, and callsupdateApifyScraperRun, but no longer callsmaybeSendScrapeDigest.instagram/handleInstagramProfileScraperResultsstill persists posts/socials and queues follow-ups; it resolves account emails but sends none and no longer returnssentEmails.sendEmailWithResendis never called for scrape webhooks or IG runs that find new posts.Rollout
apify_scraper_runstable and write path are kept for audit.POST /api/socials/{id}/scrape).Written for commit 3768d39. Summary will update on new commits.
Summary by CodeRabbit