This is a potential performance enhancement / laziness enhancement -- we could stop crawling a registry when a particular URI of desire is found -- perhaps relatedly we could store crawled URIs in a trie or something so we go directly to the most likely location to find the resource (e.g. look in http://example.com/foo first if we're looking for http://example.com/foo/bar).
crawl(until=None) (the default) would crawl the entire set of uncrawled resources, i.e. the current behavior.
Doing this should first come with some benchmarking.
This is a potential performance enhancement / laziness enhancement -- we could stop crawling a registry when a particular URI of desire is found -- perhaps relatedly we could store crawled URIs in a trie or something so we go directly to the most likely location to find the resource (e.g. look in
http://example.com/foofirst if we're looking forhttp://example.com/foo/bar).crawl(until=None)(the default) would crawl the entire set of uncrawled resources, i.e. the current behavior.Doing this should first come with some benchmarking.