Skip to content

Make project archive downloads streaming-safe for large Overleaf projects #1

Description

@dkritarth

Problem

Project downloads currently fetch the entire Overleaf zip into a Buffer and then unpack it in memory. That works for small projects, but it becomes brittle for larger projects because the CLI must hold the whole archive at once before extracting anything.

Current code paths

  • src/client.ts downloadProject() returns a full Buffer
  • src/cli.ts pull does new AdmZip(zipBuffer)
  • src/cli.ts zip writes the full Buffer to disk
  • src/mcp.ts pull_project does the same AdmZip(zipBuf) flow
  • src/remote-helper.ts clone/fetch handling also depends on the same full-buffer archive path
  • src/client.ts downloadByPath() falls back to archive extraction when entity lookup fails

Expected behavior

  • Large projects should be downloadable and extractable without buffering the whole archive in memory first.
  • The archive path should be safe for both direct CLI use and the MCP/git remote integrations.
  • Existing commands should keep working, but the implementation should be able to handle much larger projects reliably.

Proposed direction

  • Add a shared streaming-safe or file-backed archive download/extract path in OverleafClient.
  • Prefer lazy entry iteration or stream extraction over AdmZip(Buffer).
  • Keep the current downloadProject() API only as compatibility, not as the primary path for large-project workflows.

Acceptance criteria

  • olcli pull works on large nested projects without requiring the entire zip in memory.
  • olcli zip still downloads the archive successfully, but without unnecessary extra copies.
  • downloadByPath() fallback still works.
  • MCP pull_project and git-remote-overleaf use the same archive handling path.
  • Existing single-file download/upload commands remain unchanged.

Notes

This issue is about the archive transport/extraction layer only. It should not change Overleaf auth, project resolution, or sync semantics.


Generated by GPT-5.4 mini.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions