Skip to content

/api/image/generate - arweave upload - #3

Merged
sweetmantech merged 7 commits into
mainfrom
sweetmantech/myc-3549-apiimagegenerate-arweave-upload
Nov 28, 2025
Merged

/api/image/generate - arweave upload#3
sweetmantech merged 7 commits into
mainfrom
sweetmantech/myc-3549-apiimagegenerate-arweave-upload

Conversation

@sweetmantech

@sweetmantech sweetmantech commented Nov 28, 2025

Copy link
Copy Markdown
Contributor

Summary by CodeRabbit

  • New Features
    • Generated images are now automatically uploaded and exposed via a public fetchable URL; API responses include imageUrl and Arweave transaction details.
    • Added utilities to detect, normalize, and convert Arweave/IPFS URLs into fetchable gateway URLs.
  • Refactor
    • Image output schema flattened and now surfaces usage, imageUrl, and arweaveResult at the top level.

✏️ Tip: You can customize this high-level summary in your review settings.

@vercel

vercel Bot commented Nov 28, 2025

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Preview Updated (UTC)
recoup-api Ready Ready Preview Nov 28, 2025 4:11pm

@coderabbitai

coderabbitai Bot commented Nov 28, 2025

Copy link
Copy Markdown

Warning

Rate limit exceeded

@sweetmantech has exceeded the limit for the number of commits or files that can be reviewed per hour. Please wait 2 minutes and 12 seconds before requesting another review.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

📥 Commits

Reviewing files that changed from the base of the PR and between 09e817f and 849d889.

📒 Files selected for processing (3)
  • lib/arweave/isGatewayIPFSUrl.ts (1 hunks)
  • lib/arweave/isNormalizeableIPFSUrl.ts (1 hunks)
  • lib/arweave/normalizeIPFSUrl.ts (1 hunks)

Note

Other AI code review bot(s) detected

CodeRabbit has detected other AI code review bot(s) in this pull request and will avoid duplicating their findings in the review comments. This may lead to a less comprehensive review.

Walkthrough

Adds Arweave upload of the first generated image in the image-generation GET flow, new Arweave/IPFS URL utilities, Arweave client upload implementation, and package dependencies; response schema extended with imageUrl, arweaveResult, and flattened image fields.

Changes

Cohort / File(s) Summary
Image API + Schema
app/api/image/generate/route.ts, middleware.ts
After generating images, uploads the first image to Arweave and awaits the transaction. Response now includes top-level imageUrl, arweaveResult, and flattened image entries plus existing usage.
Arweave upload implementation
lib/arweave/uploadToArweave.ts
New default-exported uploadToArweave(imageData, getProgress?) that reads ARWEAVE_KEY, initializes Arweave client, creates/signs a transaction with Content-Type tag, uploads chunks with progress, and returns the completed Transaction.
Arweave core & gateway helpers
lib/arweave/arweave.ts, lib/arweave/arweaveGatewayUrl.ts, lib/arweave/getFetchableUrl.ts
Added ArweaveURL type and isArweaveURL; arweaveGatewayUrl converts ar:// to https://arweave.net/<id>; getFetchableUrl normalizes/validates Arweave, IPFS, https/data/blob URLs and rejects insecure http.
IPFS utilities
lib/arweave/isCID.ts, lib/arweave/isGatewayIPFSUrl.ts, lib/arweave/isIPFSUrl.ts, lib/arweave/isNormalizeableIPFSUrl.ts, lib/arweave/isNormalizedIPFSURL.ts, lib/arweave/normalizeIPFSUrl.ts, lib/arweave/ipfsGatewayUrl.ts
New helpers for detecting, normalizing, and converting IPFS/CID/gateway URLs (isCID, isGatewayIPFSUrl, isIPFSUrl, isNormalizeableIPFSUrl, isNormalizedIPFSURL, normalizeIPFSUrl, ipfsGatewayUrl).
Dependencies
package.json
Added arweave (^1.15.7) and multiformats (^13.4.1).

Sequence Diagram(s)

sequenceDiagram
    actor Client
    participant API as Image Generation API
    participant ImageGen as Image Generator
    participant Uploader as uploadToArweave
    participant ArweaveNet as arweave.net

    Client->>API: GET /api/image/generate
    API->>ImageGen: generate images
    ImageGen-->>API: base64 image + mimeType (images[])
    API->>Uploader: uploadToArweave({base64Data, mimeType})
    Uploader->>Uploader: read ARWEAVE_KEY, init client
    Uploader->>ArweaveNet: upload chunks (signed transaction)
    ArweaveNet-->>Uploader: upload confirmation (tx id)
    Uploader-->>API: Transaction (arweaveResult)
    API->>API: ar://id -> fetchable URL (getFetchableUrl/arweaveGatewayUrl)
    API-->>Client: 200 { images, imageUrl, arweaveResult, usage }
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

  • Pay attention to: environment key decoding and error paths in uploadToArweave.ts.
  • Verify transaction tagging (Content-Type) and chunked uploader loop correctness.
  • Validate URL normalization rules and security checks in getFetchableUrl.ts.
  • Confirm response construction and error handling in app/api/image/generate/route.ts.

Possibly related PRs

Poem

🐰 I hopped a trail of base64 light,

Carried one image into Arweave night.
Chunk by chunk, the ledger sings,
Now fetchable dreams on immutable wings.
🥕✨

Pre-merge checks and finishing touches

✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title directly describes the main change: adding Arweave upload functionality to the /api/image/generate endpoint.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread app/api/image/generate/route.ts
Comment thread lib/arweave/uploadToArweave.ts

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
app/api/image/generate/route.ts (1)

40-54: Critical: Null check on result comes after accessing result.images[0].

The code accesses result.images[0] on lines 42-43 before checking if result is null/falsy on line 46. If generateImage returns null, this will throw a runtime error. Additionally, there's no check that result.images exists and has at least one element.

     const result = await generateImage(prompt);
+
+    if (!result || !result.images?.length) {
+      return NextResponse.json(
+        { error: "Failed to generate image" },
+        {
+          status: 500,
+          headers: getCorsHeaders(),
+        },
+      );
+    }
+
     const arweaveResult = await uploadToArweave({
       base64Data: result.images[0].base64,
       mimeType: result.images[0].mediaType,
     });

-    if (!result) {
-      return NextResponse.json(
-        { error: "Failed to generate image" },
-        {
-          status: 500,
-          headers: getCorsHeaders(),
-        },
-      );
-    }
-
     return NextResponse.json(
🧹 Nitpick comments (6)
lib/arweave/uploadFile.tsx (2)

1-32: File extension should be .ts instead of .tsx.

This file contains no JSX—rename to uploadFile.ts for consistency.


13-22: Check HTTP status before parsing JSON to handle server errors gracefully.

If the server returns a non-2xx status with a non-JSON body, res.json() will throw before reaching your json.success check, producing an unclear error message.

     const res = await fetch("/api/upload", {
       method: "POST",
       body: data,
     });

+    if (!res.ok) {
+      throw new Error(`Upload request failed with status ${res.status}`);
+    }
+
     const json = await res.json();

     if (!json.success) {
       throw new Error(json.error || "Upload failed");
     }
lib/arweave/arweave.ts (1)

9-11: Consider using a type predicate for better type narrowing.

Returning url is ArweaveURL instead of boolean would allow TypeScript to narrow the type when used in conditionals.

-export function isArweaveURL(url: string | null | undefined): boolean {
-  return url && typeof url === "string" ? url.startsWith("ar://") : false;
+export function isArweaveURL(url: string | null | undefined): url is ArweaveURL {
+  return typeof url === "string" && url.startsWith("ar://");
 }
lib/arweave/uploadToArweave.ts (2)

34-46: Consider validating input parameters.

The function doesn't validate that base64Data is non-empty or that mimeType is a valid MIME type. Invalid inputs would result in an Arweave transaction with empty or malformed data.

 const uploadToArweave = async (
   imageData: { base64Data: string; mimeType: string },
   getProgress: (progress: number) => void = () => {},
 ): Promise<Transaction> => {
+  if (!imageData.base64Data) {
+    throw new Error("base64Data is required");
+  }
+  if (!imageData.mimeType) {
+    throw new Error("mimeType is required");
+  }
+
   const buffer = Buffer.from(imageData.base64Data, "base64");

50-56: No retry logic for chunk uploads.

Network interruptions during chunked uploads could cause the entire upload to fail. Consider adding retry logic with exponential backoff for transient failures.

lib/arweave/gateway.ts (1)

14-17: Add validation before URL replacement.

The function performs string replacement without verifying the input is actually an Arweave URL. If a non-Arweave URL is passed, it returns the input unchanged (or with an unintended mid-string replacement).

Consider adding validation:

 export function arweaveGatewayUrl(normalizedArweaveUrl: string | null) {
   if (!normalizedArweaveUrl || typeof normalizedArweaveUrl !== "string") return null;
+  if (!isArweaveURL(normalizedArweaveUrl)) return null;
   return normalizedArweaveUrl.replace("ar://", `${ARWEAVE_GATEWAY}/`);
 }

This ensures the function only processes valid Arweave URLs and returns null for invalid inputs, making the behavior more predictable.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 23eeef7 and 236592c.

⛔ Files ignored due to path filters (1)
  • pnpm-lock.yaml is excluded by !**/pnpm-lock.yaml
📒 Files selected for processing (7)
  • app/api/image/generate/route.ts (3 hunks)
  • lib/arweave/arweave.ts (1 hunks)
  • lib/arweave/gateway.ts (1 hunks)
  • lib/arweave/ipfs.ts (1 hunks)
  • lib/arweave/uploadFile.tsx (1 hunks)
  • lib/arweave/uploadToArweave.ts (1 hunks)
  • package.json (1 hunks)
🧰 Additional context used
🧬 Code graph analysis (2)
lib/arweave/gateway.ts (2)
lib/arweave/ipfs.ts (2)
  • normalizeIPFSUrl (30-60)
  • isNormalizeableIPFSUrl (107-109)
lib/arweave/arweave.ts (1)
  • isArweaveURL (9-11)
app/api/image/generate/route.ts (1)
lib/arweave/gateway.ts (1)
  • getFetchableUrl (40-65)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
  • GitHub Check: Vercel Agent Review
🔇 Additional comments (6)
app/api/image/generate/route.ts (1)

41-44: Arweave upload failure will propagate as a generic error.

If uploadToArweave fails, the error is caught by the outer try-catch and returns a 500 with the error message. Consider whether a partial success response (image generated but upload failed) would be more appropriate, or add explicit handling to distinguish Arweave failures.

lib/arweave/uploadToArweave.ts (1)

5-11: Module-level throw will crash the app at startup if ARWEAVE_KEY is missing.

This validation runs when the module is first imported, which means the entire application will fail to start if ARWEAVE_KEY is not set—even for endpoints that don't use Arweave. Consider lazy initialization or handling this more gracefully if Arweave is optional.

If Arweave integration is mandatory for this service, this approach is acceptable. Otherwise, consider deferring the check:

-const rawArweaveKey = process.env.ARWEAVE_KEY;
-
-if (!rawArweaveKey) {
-  throw new Error(
-    "ARWEAVE_KEY environment variable is missing. Please set it to a base64-encoded JSON key.",
-  );
-}
+const getArweaveKey = (): JWKInterface => {
+  const rawArweaveKey = process.env.ARWEAVE_KEY;
+  if (!rawArweaveKey) {
+    throw new Error(
+      "ARWEAVE_KEY environment variable is missing. Please set it to a base64-encoded JSON key.",
+    );
+  }
+  try {
+    const decodedKey = Buffer.from(rawArweaveKey, "base64").toString();
+    return JSON.parse(decodedKey) as JWKInterface;
+  } catch (error) {
+    throw new Error(
+      `Failed to decode ARWEAVE_KEY. Ensure it is base64-encoded JSON. ${
+        error instanceof Error ? error.message : error
+      }`,
+    );
+  }
+};
package.json (1)

21-22: Dependency versions are current and secure.

The arweave@1.15.7 and multiformats@13.4.1 packages specified in the PR are both confirmed as the latest stable versions with no known security vulnerabilities. The caret version constraints (^1.15.7 and ^13.4.1) follow standard npm practices and appropriately allow minor and patch updates while protecting against breaking changes.

lib/arweave/gateway.ts (2)

25-32: LGTM!

The function correctly leverages normalizeIPFSUrl for input validation and normalization before converting to a gateway URL. The null-handling is appropriate.


40-65: LGTM!

The function provides a well-structured security-conscious URL resolver:

  • Blocks insecure http:// URLs (Line 44)
  • Properly routes IPFS and Arweave URLs to their respective gateway converters
  • Handles already-fetchable URLs (https/data/blob) appropriately
  • Returns null for unrecognized formats

The ordering of checks is correct, ensuring protocol-specific handling occurs before the generic https fallthrough.

lib/arweave/ipfs.ts (1)

30-60: I'll help you verify the review comment about the CID extraction robustness in the normalizeIPFSUrl function. Let me start by examining the actual implementation and looking for test cases.
<function_calls>

#!/bin/bash

First, let's locate and read the IPFS file to see the actual implementation

fd ipfs.ts --type f


</function_calls>

Now let me read the file to see the complete function:
<function_calls>

#!/bin/bash

Read the IPFS file to see the full context

cat -n lib/arweave/ipfs.ts | head -100


</function_calls>

Let me search for test files related to this function:
<function_calls>

#!/bin/bash

Search for test files that might contain tests for normalizeIPFSUrl

fd -e test.ts -e spec.ts -e test.js -e spec.js | head -20


</function_calls>

Let me search more broadly for any test coverage:
<function_calls>

#!/bin/bash

Search for normalizeIPFSUrl mentions in the codebase

rg -n "normalizeIPFSUrl" --type ts


</function_calls>

<function_calls>

#!/bin/bash

Look for test directories

find . -type d -name "test" -o -type d -name "spec" | grep -v node_modules | head -20


</function_calls>

<function_calls>

#!/bin/bash

Check the full ipfs.ts file to understand the functions better

wc -l lib/arweave/ipfs.ts


</function_calls>

<function_calls>

#!/bin/bash

Read the complete ipfs.ts file

cat lib/arweave/ipfs.ts


</function_calls>

Comment thread lib/arweave/ipfs.ts Outdated
Comment thread lib/arweave/ipfs.ts Outdated
Comment thread lib/arweave/uploadFile.tsx Outdated
Comment thread app/api/image/generate/route.ts
Comment thread lib/arweave/isGatewayIPFSUrl.ts Outdated

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🧹 Nitpick comments (1)
lib/arweave/isNormalizedIPFSURL.ts (1)

7-9: Normalized IPFS URL check looks good; can be simplified

Behavior (non‑empty string starting with ipfs://) is correct and matches the intended “normalized IPFS URL” concept. For readability you could collapse the ternary to a simple boolean expression:

export function isNormalizedIPFSURL(url: string | null | undefined): boolean {
  return typeof url === "string" && url.startsWith("ipfs://");
}
📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between db730d1 and 3ee5e58.

📒 Files selected for processing (8)
  • lib/arweave/getFetchableUrl.ts (1 hunks)
  • lib/arweave/ipfsGatewayUrl.ts (1 hunks)
  • lib/arweave/isCID.ts (1 hunks)
  • lib/arweave/isGatewayIPFSUrl.ts (1 hunks)
  • lib/arweave/isIPFSUrl.ts (1 hunks)
  • lib/arweave/isNormalizeableIPFSUrl.ts (1 hunks)
  • lib/arweave/isNormalizedIPFSURL.ts (1 hunks)
  • lib/arweave/normalizeIPFSUrl.ts (1 hunks)
🚧 Files skipped from review as they are similar to previous changes (2)
  • lib/arweave/getFetchableUrl.ts
  • lib/arweave/ipfsGatewayUrl.ts
🧰 Additional context used
🧬 Code graph analysis (4)
lib/arweave/isNormalizeableIPFSUrl.ts (2)
lib/arweave/isIPFSUrl.ts (1)
  • isIPFSUrl (10-12)
lib/arweave/isCID.ts (1)
  • isCID (9-20)
lib/arweave/isIPFSUrl.ts (2)
lib/arweave/isNormalizedIPFSURL.ts (1)
  • isNormalizedIPFSURL (7-9)
lib/arweave/isGatewayIPFSUrl.ts (1)
  • isGatewayIPFSUrl (9-20)
lib/arweave/normalizeIPFSUrl.ts (4)
lib/arweave/isNormalizedIPFSURL.ts (1)
  • isNormalizedIPFSURL (7-9)
lib/arweave/isCID.ts (1)
  • isCID (9-20)
lib/arweave/isIPFSUrl.ts (1)
  • isIPFSUrl (10-12)
lib/arweave/isGatewayIPFSUrl.ts (1)
  • isGatewayIPFSUrl (9-20)
lib/arweave/isGatewayIPFSUrl.ts (1)
lib/arweave/isNormalizedIPFSURL.ts (1)
  • isNormalizedIPFSURL (7-9)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
  • GitHub Check: Vercel Agent Review
🔇 Additional comments (1)
lib/arweave/isIPFSUrl.ts (1)

10-12: IPFS URL composition via helpers looks correct

isIPFSUrl cleanly delegates to isNormalizedIPFSURL and isGatewayIPFSUrl and correctly treats falsy/empty values as non‑IPFS. No issues from my side here.

Comment thread lib/arweave/isCID.ts
Comment thread lib/arweave/isGatewayIPFSUrl.ts
Comment thread lib/arweave/isNormalizeableIPFSUrl.ts Outdated
Comment thread lib/arweave/normalizeIPFSUrl.ts
sweetmantech added a commit that referenced this pull request Jul 9, 2026
… posts first (#759)

The Instagram scrape alert fired whenever a scrape returned posts, with no
comparison against posts already stored — every scrape re-announced the
profile's recent feed as new (observed: 6 of 7 alerts in one day announced
posts up to 10 days old). New reusable filterNewPostUrls diffs candidate
URLs against posts BEFORE upsert; the alert is gated on a non-empty result.
Persistence unchanged (recoupable/chat#1855, PR #3 of 6).

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
sweetmantech added a commit that referenced this pull request Jul 9, 2026
* feat(apify): notify only on genuinely new posts — diff against stored posts first

The Instagram scrape alert fired whenever a scrape returned posts, with no
comparison against posts already stored — every scrape re-announced the
profile's recent feed as new (observed: 6 of 7 alerts in one day announced
posts up to 10 days old). New reusable filterNewPostUrls diffs candidate
URLs against posts BEFORE upsert; the alert is gated on a non-empty result.
Persistence unchanged (recoupable/chat#1855, PR #3 of 6).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(apify): one consolidated new-posts digest per scrape batch

A roster scrape starts one Apify run per platform, each completing
independently — extending per-platform alerts would mean 4+ emails per
scrape. This registers every run under a batch_id at scrape start
(apify_scraper_runs, columns from recoupable/database#41), records each
webhook completion with its genuinely-new post URLs, and when the batch's
last run completes sends ONE digest (per-platform sections, BCC-only)
via the new digest module. Platforms with nothing new are omitted; a
batch with nothing new sends nothing. Instagram's solo alert is
suppressed for batch runs; legacy/non-batch runs keep today's behavior
(recoupable/chat#1855, PR #4 of 6).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor(supabase): align apify_scraper_runs helpers with naming convention

Review feedback on #760: completeApifyScraperRun -> updateApifyScraperRun,
insertApifyScraperRuns -> upsertApifyScraperRuns (it upserts on run_id),
selectApifyScraperRunsByBatch -> selectApifyScraperRuns with an optional
{batchId} filter object, matching the select* convention. Also repoints
the stale types.ts reference from database#41 to #47.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor(supabase): use generated types for apify_scraper_runs + zod-parse the JSONB column

Regenerates database.types.ts (table landed in database#47), deletes the
hand-rolled lib/supabase/apify_scraper_runs/types.ts shim, and drops every
'as never' cast — the helpers now use Tables/TablesInsert like every sibling
lib. new_post_urls is the one column codegen can't narrow past Json, so a
zod boundary (parseNewPostUrls) validates it at read time; malformed JSONB
degrades to 'no new posts' instead of crashing the digest assembler.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor(apify): move parseNewPostUrls out of lib/supabase into lib/apify/digest

Review feedback: lib/supabase is for direct queries only; this is a pure
JSONB parser consumed by the digest assembler.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
sweetmantech added a commit that referenced this pull request Jul 9, 2026
… post, no LLM body (#761)

* feat(apify): notify only on genuinely new posts — diff against stored posts first

The Instagram scrape alert fired whenever a scrape returned posts, with no
comparison against posts already stored — every scrape re-announced the
profile's recent feed as new (observed: 6 of 7 alerts in one day announced
posts up to 10 days old). New reusable filterNewPostUrls diffs candidate
URLs against posts BEFORE upsert; the alert is gated on a non-empty result.
Persistence unchanged (recoupable/chat#1855, PR #3 of 6).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(apify): one consolidated new-posts digest per scrape batch

A roster scrape starts one Apify run per platform, each completing
independently — extending per-platform alerts would mean 4+ emails per
scrape. This registers every run under a batch_id at scrape start
(apify_scraper_runs, columns from recoupable/database#41), records each
webhook completion with its genuinely-new post URLs, and when the batch's
last run completes sends ONE digest (per-platform sections, BCC-only)
via the new digest module. Platforms with nothing new are omitted; a
batch with nothing new sends nothing. Instagram's solo alert is
suppressed for batch runs; legacy/non-batch runs keep today's behavior
(recoupable/chat#1855, PR #4 of 6).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(apify): deterministic digest template with direct links to each new post

Replaces the per-send LLM email body (nondeterministic branding, vendor
jargon reaching customers, no reliable post links) with a shared
deterministic renderer: stable subject/branding, one section per
platform, a direct link to every genuinely-new post, chat CTA secondary.
Used by both the batch digest and the legacy solo Instagram alert, which
now also requires new-post URLs and is BCC-only (recoupable/chat#1855,
PR #5 of 6; supersedes the interim body from PR #4 and, for this file,
the standalone BCC fix in api#758 — same invariant, tests included).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* test(apify): guard sendScrapeDigestEmail's renderer wiring + BCC invariant

The branch imported renderScrapeDigestHtml in sendScrapeDigestEmail but
never called it — the digest still rendered the old inline body (found
during the main-sync conflict resolution; wired in that merge commit).
This test fails against the unwired version: it asserts the send payload
is byte-identical to the renderer's output, plus the BCC-only invariant
and the empty-input no-send.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(apify): house-style digest template with post media, captions, and dates

The v1 deterministic template regressed visual quality vs the old LLM
emails (bare h3/ul list). This upgrades the renderer to the DESIGN.md
house style — achromatic chrome, card-per-post with 72px thumbnail,
escaped caption excerpt, date, and a direct link; black CTA button —
while staying deterministic and email-safe (tables + inline styles).

Media plumbing: extractPostsFromDatasetItems maps platform dataset items
(IG latestPosts, TikTok items) to {url, caption, thumbnailUrl, timestamp},
limited to the genuinely-new URLs. The solo alert enriches from the
dataset already in memory; the digest enriches via getRunDigestSection,
which re-reads each run's dataset from Apify (source of truth) and
degrades to URL-only links on any failure — enrichment never blocks a
send. Captions are HTML-escaped so scraped content can't inject markup.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* style: prettier pass on the digest template files

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(apify): digest addressed by artist name + per-post engagement stats + fixed chat CTA

Three feedback items on the template:
- The digest header/subject said 'Your artist' — the assembler never passed
  a name. getRunDigestSection now also extracts the profile display name
  from the dataset (IG fullName, TikTok authorMeta.nickName) and
  maybeSendScrapeDigest addresses the email with the first platform's name.
- Post cards now carry compact engagement stats (12.3K likes · 678
  comments · 1.2M views · shares) mapped from platform counts (IG
  likesCount/commentsCount/videoViewCount, TikTok diggCount/commentCount/
  playCount/shareCount); omitted entirely when the scraper returns none.
- The CTA now always points at https://chat.recoupable.dev — previously it
  derived from the deployment base URL, which on previews is the API
  deployment itself. Funnel-tracking landing page tracked as follow-up.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(apify): Recoup logo in the email header + artist-named roster footer

Header is now a two-cell row with the brand icon top-right (hosted PNG —
email clients don't render SVG) linking to recoupable.com. Footer names
the artist: 'because {artist} is in your roster on Recoup', falling back
to 'this artist' when no profile name resolved.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor(apify): strict SRP/DRY pass on the digest renderer + delete orphaned pre-rename supabase files

Review feedback:
- Every helper gets its own lib file: escapeHtml (lib/emails — repo had
  none), formatUtcDateLabel, truncateText, formatCompactCount,
  formatPostStats, getPlatformLabel, extractInstagramPosts,
  extractTiktokPosts, buildPostStats, asRecord/asStringOrNull/
  asNumberOrNull coercers. renderScrapeDigestHtml and
  extractPostsFromDatasetItems now contain only their eponymous functions.
- CHAT_APP_URL / WEBSITE_URL / RECOUP_LOGO_URL move to lib/const.ts as
  shared constants (no existing shared home found for platform labels —
  getPlatformLabel is the new one).
- Deletes 4 orphaned pre-rename files under lib/supabase/apify_scraper_runs
  (completeApifyScraperRun, insertApifyScraperRuns,
  selectApifyScraperRunsByBatch, types): this branch predates #760's
  renames, so the merge of main added the renamed set alongside the
  branch's inherited old set — both landed with zero conflicts and zero
  references to the old files. Also confirms no legacy LLM email
  generation remains in lib/apify (generateText usage gone with the
  template rewrite).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat(apify): X/Twitter digest extractor + LinkedIn platform label

Review feedback: Twitter's handler already feeds the digest (filterNewPostUrls
-> newPostUrls) but had no extractor, so its sections degraded to URL-only
links. extractTwitterPosts maps apidojo tweet items (fullText, extendedEntities
media, createdAt via toIsoDate, like/reply/view/retweet counts), registered
under both 'twitter' and 'x'. Artist-name extraction gains the same aliases
(author.name ?? userName). LinkedIn added to platform labels.

YouTube and LinkedIn deliberately have no extractors: their results handlers
persist only the social row (no posts, no newPostUrls), so they never
contribute digest sections today — extractors would be dead code until post
persistence ships for them (chat#1833 territory).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(apify): only persist and report the artist's own tweets — no retweets or replies

Feedback from a real-account digest test: an all-retweet X section reported
hundreds of likes the artist never earned (retweet items carry the ORIGINAL
author's stats). isOriginalTweet keeps originals and quote tweets (the
artist's own words and metrics), drops retweets and replies, applied at the
persistence layer in handleTwitterProfileScraperResults so both stored posts
and digest sections stay accurate. Items without flags pass (defensive
default on schema drift).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(apify): fetch 10 X timeline items by default — depth 1 rarely survives the retweet filter

Real-account evidence (2026-07-09): the user's timeline had 7 retweets above
their 2 originals and a quote tweet; a depth-1 (or 3) fetch returned only
retweets, which isOriginalTweet now drops — so no authored post could ever
reach the digest. Fetch deeper, filter after.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant