dav DeleteRecursive walks the entire blobstore instead of the target prefix

[wayneeseguin]

Summary

The dav backend's recursive delete lists the whole blobstore before every delete instead of scoping the listing to the prefix it was given. On a large WebDAV blobstore this turns a single-app delete into thousands of PROPFIND requests, and the operation times out. The s3 backend does not have this problem because it pushes the prefix down to the server.

Version

• storage-cli v0.0.9 (current tip), as bundled in capi-release 1.240.0.
• Present since the dav List/DeleteRecursive implementation in Implement Copy, List, DeleteRecursive, Properties for DAV #103; unchanged in v0.0.9.

Impact

Cloud Controller calls delete-recursive through its storage-cli dav client whenever it clears an app's buildpack cache, which happens on every cf delete -r. Against a production-sized WebDAV blobstore the walk does not finish in time. In our case Cloud Controller's buildpack-cache-delete job never completes, the cc-worker queue backs up behind it, and app deletes time out. Migrating a foundation from the native WebDAV blobstore client to storage-cli is enough to trigger it.

Root cause

DeleteRecursive delegates to List, and List starts its walk at the blobstore endpoint root rather than at the prefix:

// dav/client/storage_client.go
func (c *storageClient) List(prefix string) ([]string, error) {
	rootURL, err := url.Parse(c.config.Endpoint)   // endpoint ROOT, not the prefix
	...
	return c.listRecursive(rootURL.String(), rootURL.Path, prefix)
}

func (c *storageClient) DeleteRecursive(prefix string) error {
	blobs, err := c.List(prefix)                   // lists the whole store, filters at the leaves
	...
	for _, blob := range blobs {
		if err := c.Delete(blob); err != nil { ... }
	}
}

listRecursive issues a Depth: 1 PROPFIND per directory and recurses the full tree, applying the prefix only when it reaches a leaf (strings.HasPrefix(blobID, prefix)). So deleting one prefix walks every directory in the blobstore. WebDAV blobstores partition keys into two-character directories, which multiplies the directory count, and each PROPFIND is a network round trip.

By comparison, the s3 backend scopes the same operation server-side:

// s3/client/aws_s3_blobstore.go
func (b *awsS3Client) DeleteRecursive(prefix string) error {
	input := &s3.ListObjectsV2Input{ Bucket: aws.String(b.s3cliConfig.BucketName) }
	if prefix != "" {
		input.Prefix = b.key(prefix)               // prefix pushed to the server
	}
	...
}

Steps to reproduce

1. Point storage-cli at a WebDAV blobstore that holds many blobs across many partition directories.
2. Run storage-cli -s dav -c <config> -log-level debug delete-recursive <some-prefix>.
3. Observe a PROPFIND request per directory across the entire store, not just under <some-prefix>. Wall-clock time scales with the size of the whole blobstore rather than the size of the prefix.

Expected vs actual

• Expected: listing and deleting under a prefix touches only that prefix's subtree, as the s3 backend does.
• Actual: the dav backend walks the entire blobstore on every recursive delete.

Suggested fix

Root the walk at the prefix instead of the endpoint. List/listRecursive should start the PROPFIND at the prefix's collection URL (endpoint joined with the prefix), so the traversal is bounded by the subtree the caller asked for. buildBlobURL already joins the endpoint and a key, so the same join can produce the starting URL for the walk.

A single collection DELETE on the prefix URL, which is what the old native Cloud Controller WebDAV client used, would be even cheaper where the server supports it, but scoping the walk is the minimal change that matches the s3 behavior.

[wayneeseguin]

FYI I am working on a PR not just filing the issue.