Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 20 additions & 2 deletions content/docs/configuration/dotenv.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -807,7 +807,7 @@ Uncomment `ENDPOINTS` to customize the available endpoints in LibreChat.

The web search feature enables internet search capabilities within LibreChat.

**Important**: The exact environment variable names shown below are default references and can be customized through the `librechat.yaml` configuration file to use any variable names you prefer.
**Important**: The exact environment variable names shown below are default references. Credential and URL fields represented in `librechat.yaml` can use custom variable names; `KEENABLE_FETCH_URL` is an environment-only override and keeps this exact name.

For detailed configuration and customization options, see: [Web Search Configuration](/docs/configuration/librechat_yaml/object_structure/web_search)

Expand Down Expand Up @@ -837,6 +837,24 @@ For detailed configuration and customization options, see: [Web Search Configura
'Custom Tavily Extract API URL (optional). Only needed for custom or proxy Tavily-compatible extract endpoints.',
'# TAVILY_EXTRACT_URL=',
],
[
'KEENABLE_API_KEY',
'string',
'Optional API key shared by the Keenable search and scraper providers. Keenable works keylessly; a key raises the public rate limits.',
'# KEENABLE_API_KEY=',
],
[
'KEENABLE_API_URL',
'string',
'Custom Keenable Search API URL (optional). The default changes between the public and keyed endpoint according to whether KEENABLE_API_KEY is set.',
'# KEENABLE_API_URL=',
],
[
'KEENABLE_FETCH_URL',
'string',
'Custom Keenable Fetch API URL (optional). This environment-only override is used when scraperProvider is keenable.',
'# KEENABLE_FETCH_URL=',
],
[
'FIRECRAWL_API_KEY',
'string',
Expand Down Expand Up @@ -871,7 +889,7 @@ For detailed configuration and customization options, see: [Web Search Configura
]}
/>

**Note**: These variable names can be customized in your `librechat.yaml` configuration file. For example, you could use `CUSTOM_SERPER_KEY` instead of `SERPER_API_KEY` by configuring it in the web search settings. See the [Web Search Configuration](/docs/configuration/librechat_yaml/object_structure/web_search) documentation for details on customizing variable names.
**Note**: Fields represented in `librechat.yaml` can use custom variable names. For example, you could use `CUSTOM_SERPER_KEY` instead of `SERPER_API_KEY` in the web search settings. `KEENABLE_FETCH_URL` is read directly by the Agents package and cannot be renamed through YAML. See [Web Search Configuration](/docs/configuration/librechat_yaml/object_structure/web_search) for details.

### Anthropic

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ webSearch:
serperApiKey: "${SERPER_API_KEY}"
searxngInstanceUrl: "${SEARXNG_INSTANCE_URL}"
searxngApiKey: "${SEARXNG_API_KEY}"
searchProvider: "serper" # Options: "serper", "searxng", "tavily"
searchProvider: "serper" # Options: "serper", "searxng", "tavily", "keenable"

# Optional: query options sent to a SearXNG instance
searxngSearchOptions:
Expand All @@ -34,11 +34,23 @@ webSearch:
tavilySearchUrl: "${TAVILY_SEARCH_URL}"
tavilyExtractUrl: "${TAVILY_EXTRACT_URL}"

# Keenable Configuration (search and/or scraper; keyless by default)
keenableApiKey: "${KEENABLE_API_KEY}" # Optional; raises rate limits
keenableApiUrl: "${KEENABLE_API_URL}" # Optional search endpoint override
keenableSearchOptions:
maxResults: 5
site: "example.com"
attributionTitle: "LibreChat"
timeout: 15000

# Scraper Configuration
firecrawlApiKey: "${FIRECRAWL_API_KEY}"
firecrawlApiUrl: "${FIRECRAWL_API_URL}"
firecrawlVersion: "${FIRECRAWL_VERSION}"
scraperProvider: "firecrawl" # Options: "firecrawl", "serper", "tavily"
scraperProvider: "firecrawl" # Options: "firecrawl", "serper", "tavily", "keenable"
keenableScraperOptions:
attributionTitle: "LibreChat"
timeout: 15000

# Reranker Configuration
jinaApiKey: "${JINA_API_KEY}"
Expand Down Expand Up @@ -79,7 +91,7 @@ LibreChat automatically exempts a configured HTTP(S) forward-proxy endpoint so i

<OptionTable
options={[
['searchProvider', 'String', 'Specifies which search provider to use.', 'Options: "serper", "searxng", "tavily"'],
['searchProvider', 'String', 'Specifies which search provider to use.', 'Options: "serper", "searxng", "tavily", "keenable"'],
]}
/>

Expand Down Expand Up @@ -250,6 +262,65 @@ Adding more engines widens coverage but also slows every search down, since Sear
]}
/>

### keenableApiKey

<OptionTable
options={[
['keenableApiKey', 'String', 'Environment variable name for the optional Keenable API key. Keenable search and scraping work without it; a key raises the public rate limits and is shared by both services.', '${KEENABLE_API_KEY}'],
]}
/>

### keenableApiUrl

<OptionTable
options={[
['keenableApiUrl', 'String', 'Environment variable name for a custom Keenable Search API URL. When unset, LibreChat chooses the public keyless endpoint or keyed endpoint according to whether an API key is present.', '${KEENABLE_API_URL}'],
]}
/>

Keenable's fetch endpoint is configured separately with [`KEENABLE_FETCH_URL`](/docs/configuration/dotenv#web-search), because it is an environment-only override rather than a `librechat.yaml` field.

### keenableSearchOptions

<OptionTable
options={[
['keenableSearchOptions', 'Object', 'Configuration options for Keenable search.', ''],
]}
/>

**Subkeys:**

<OptionTable
options={[
['maxResults', 'Number', 'Maximum number of Keenable results returned to the search pipeline. Keenable returns its result set first and LibreChat applies this limit locally.', 'Range: 1-20. Default: 8'],
['site', 'String', 'Restricts the search to one site or domain.', ''],
['attributionTitle', 'String', 'Value sent in the X-Keenable-Title attribution header.', 'Default: "LibreChat"'],
['timeout', 'Number', 'HTTP request timeout in milliseconds.', 'Range: 0-120000. Default: 15000'],
]}
/>

### Fully keyless Keenable stack

Keenable can provide both search and page extraction without a credential. Disable reranking to make all three web-search categories keyless:

```yaml filename="webSearch"
webSearch:
searchProvider: "keenable"
scraperProvider: "keenable"
rerankerType: "none"
# Optional; raises the public rate limits for search and fetch
# keenableApiKey: "${KEENABLE_API_KEY}"
keenableSearchOptions:
maxResults: 5
attributionTitle: "LibreChat"
timeout: 15000
keenableScraperOptions:
attributionTitle: "LibreChat"
timeout: 15000
```

If an admin leaves the three provider fields unpinned, each user can save this same provider, scraper, and no-reranker combination through the API-key dialog without entering a Keenable key.

## Scrapers

### firecrawlApiKey
Expand Down Expand Up @@ -284,7 +355,7 @@ Adding more engines widens coverage but also slows every search down, since Sear

<OptionTable
options={[
['scraperProvider', 'String', 'Specifies which scraper service to use.', 'Options: "firecrawl", "serper", "tavily"'],
['scraperProvider', 'String', 'Specifies which scraper service to use.', 'Options: "firecrawl", "serper", "tavily", "keenable"'],
]}
/>

Expand Down Expand Up @@ -495,6 +566,25 @@ webSearch:

**Note:** For detailed information about Tavily API options, see the [Tavily API Documentation](https://docs.tavily.com).

### keenableScraperOptions

<OptionTable
options={[
['keenableScraperOptions', 'Object', 'Configuration options for Keenable page extraction.', ''],
]}
/>

**Subkeys:**

<OptionTable
options={[
['attributionTitle', 'String', 'Value sent in the X-Keenable-Title attribution header.', 'Default: "LibreChat"'],
['timeout', 'Number', 'HTTP request timeout in milliseconds. The top-level scraperTimeout takes precedence when both are set.', 'Range: 0-120000. Default: 7500'],
]}
/>

The scraper uses the optional `keenableApiKey` configured above. Set [`KEENABLE_FETCH_URL`](/docs/configuration/dotenv#web-search) only when an admin needs to override Keenable's default fetch endpoint.

## Rerankers

### jinaApiKey
Expand Down
58 changes: 49 additions & 9 deletions content/docs/features/web_search.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ LibreChat's web search feature allows you to search the internet and retrieve re

## Quick Start

To get started with web search, you'll need to configure API keys for a search provider and scraper. Reranking can use Jina or Cohere, or be disabled with `rerankerType: "none"`. You can do this in two ways:
To get started with web search, choose a search provider and scraper. Most services require API keys, but Keenable can provide both search and scraping without a key; adding a Keenable key only raises its public rate limits. Reranking can use Jina or Cohere, or be disabled with `rerankerType: "none"`. You can configure the stack in two ways:

<Callout type="warning" title="Private self-hosted endpoints">
Web search, scrape, and rerank connections block private, loopback, link-local, and cloud-metadata destinations by default. If SearXNG, Firecrawl, Jina, or another configured provider endpoint is private, add its exact host and port to [`webSearch.allowedAddresses`](/docs/configuration/librechat_yaml/object_structure/web_search#ssrf-protection-and-private-providers).
Expand All @@ -22,6 +22,11 @@ To get started with web search, you'll need to configure API keys for a search p
SEARXNG_API_KEY=your_searxng_api_key # Optional
# or
TAVILY_API_KEY=your_tavily_api_key
# or use Keenable without a key; this optional key raises rate limits
# KEENABLE_API_KEY=your_keenable_api_key
# Optional: custom Keenable search and fetch endpoints
# KEENABLE_API_URL=your_keenable_search_url
# KEENABLE_FETCH_URL=your_keenable_fetch_url

# Scraper (Required - choose one)
FIRECRAWL_API_KEY=your_firecrawl_api_key
Expand All @@ -31,6 +36,8 @@ To get started with web search, you'll need to configure API keys for a search p
# FIRECRAWL_VERSION=v1
# or
TAVILY_API_KEY=your_tavily_api_key
# or use Keenable without a key (the same optional key covers both services)
# KEENABLE_API_KEY=your_keenable_api_key

# Reranker (Optional - choose one, or set rerankerType: "none")
JINA_API_KEY=your_jina_api_key
Expand All @@ -42,7 +49,8 @@ To get started with web search, you'll need to configure API keys for a search p

2. **User Interface** (If environment variables are not set):
- Users will be prompted to enter the required API keys when they first use the web search feature
- They can choose which search provider (Serper, SearXNG, or Tavily) and which reranker service to use (Jina, Cohere, or none)
- They can choose the search provider and scraper independently, including keyless Keenable, and choose Jina, Cohere, or no reranker
- LibreChat saves those choices even when the selected Keenable services have no credential to store

## Obtaining API Keys

Expand Down Expand Up @@ -71,7 +79,15 @@ Each enabled external service requires its own API key. Here's how to obtain the
4. Set `TAVILY_API_KEY` in your environment variables or provide it through the UI
5. Tavily can be used as both a search provider and a scraper provider

### Scraper: Firecrawl
#### Keenable
1. Select Keenable as the search provider, scraper, or both
2. Leave the API key empty to use the public keyless endpoints
3. Optionally get a key from [Keenable](https://keenable.ai) to raise the public rate limits and set `KEENABLE_API_KEY` or provide it through the UI
4. Set `rerankerType: "none"` for a fully keyless stack, or configure Jina or Cohere separately

### Scrapers

#### Firecrawl

1. Visit [Firecrawl.dev](https://docs.firecrawl.dev/introduction#api-key)
2. Sign up for an account
Expand All @@ -80,6 +96,10 @@ Each enabled external service requires its own API key. Here's how to obtain the
5. Set it in your environment variables or provide it through the UI
6. (Optional) If you're using a custom Firecrawl instance, you'll also need to set the API URL

#### Keenable

Keenable's page fetch is keyless. Select it as the scraper without entering a credential, or use the same optional `KEENABLE_API_KEY` used for Keenable search to raise rate limits. `KEENABLE_FETCH_URL` overrides the fetch endpoint for an admin-managed deployment.

### Rerankers

#### Jina
Expand Down Expand Up @@ -113,6 +133,9 @@ Search providers are responsible for performing the initial web search and retur
- Get your API key from [Tavily](https://app.tavily.com/home)
- Supports configurable search depth, topic filtering, domain filtering, and more
- Can also serve as a scraper provider
- **Keenable**: Keyless web search with an optional API key for higher rate limits
- Supports a site restriction, result limit, attribution title, and request timeout
- Can also serve as a keyless scraper provider

### 2. Scrapers

Expand All @@ -126,6 +149,9 @@ Scrapers extract the actual content from web pages returned by the search provid
- **Tavily**: Batch URL extraction via Tavily Extract API
- Uses the same `TAVILY_API_KEY` as the search provider
- Supports configurable extract depth, image extraction, and favicon extraction
- **Keenable**: Keyless page-to-markdown extraction
- Uses the same optional API key as Keenable search
- Supports a custom attribution title and request timeout

**Planned Scrapers:**
- **Local Firecrawl**: Self-hosted version of Firecrawl
Expand Down Expand Up @@ -174,6 +200,18 @@ webSearch:
tavilySearchUrl: "${CUSTOM_TAVILY_SEARCH_URL}"
tavilyExtractUrl: "${CUSTOM_TAVILY_EXTRACT_URL}"

# Keenable Configuration (search and/or scraper; keyless by default)
keenableApiKey: "${CUSTOM_KEENABLE_API_KEY}" # Optional; raises rate limits
keenableApiUrl: "${CUSTOM_KEENABLE_API_URL}" # Optional search endpoint override
keenableSearchOptions:
maxResults: 5
site: "example.com"
attributionTitle: "LibreChat"
timeout: 15000
keenableScraperOptions:
attributionTitle: "LibreChat"
timeout: 15000

# Scraper Configuration
firecrawlApiKey: "${CUSTOM_FIRECRAWL_API_KEY}"
firecrawlApiUrl: "${CUSTOM_FIRECRAWL_API_URL}"
Expand Down Expand Up @@ -205,20 +243,21 @@ webSearch:
searchProvider: "serper" # Only use Serper for search
# searchProvider: "searxng" # Only use SearXNG for search
# searchProvider: "tavily" # Only use Tavily for search
# searchProvider: "keenable" # Use keyless Keenable search
scraperProvider: "firecrawl" # Only use Firecrawl for scraping
# scraperProvider: "tavily" # Only use Tavily for scraping
# scraperProvider: "keenable" # Use keyless Keenable scraping
rerankerType: "jina" # Options: "jina", "cohere", "none"
```

### User Configuration

If the admin hasn't configured all the necessary API keys, users will be prompted to provide them through the UI. The interface allows users to:

1. Choose their preferred reranker (Jina, Cohere, or none)
2. Enter API keys for the required services
3. Configure the Firecrawl API URL if needed (optional)
4. Configure the Jina API URL if needed (optional)
5. Configure Tavily Search or Extract API URLs if needed (optional)
1. Choose their preferred search provider, scraper, and reranker
2. Save a fully keyless Keenable + Keenable + none stack without entering credentials
3. Enter API keys for services that require them, or an optional Keenable key for higher limits
4. Configure supported provider URLs if needed

## Usage

Expand All @@ -229,10 +268,11 @@ Once configured, you can use web search in two ways:

## Notes

- Search provider and scraper configuration are required; reranking can be disabled with `rerankerType: "none"`
- A search provider and scraper must be selected; Keenable can fill both roles without an API key, and reranking can be disabled with `rerankerType: "none"`
- The Firecrawl API URL is optional and defaults to their hosted service
- The Jina API URL is optional and defaults to their hosted service
- The Tavily Search and Extract API URLs are optional and default to Tavily's hosted services
- The Keenable Search and Fetch API URLs are optional and default to Keenable's public or keyed endpoints according to whether a key is present
- Safe search provides three levels of content filtering: OFF (0), MODERATE (1 - default), and STRICT (2)
- Tavily does not inherit the global `safeSearch` setting by default; use `tavilySearchOptions.safeSearch` only if your Tavily account supports `safe_search`
- The scraper timeout is set to 7.5 seconds (7500ms) by default
Expand Down
Loading