Cambodia's open-source search engine β built to index, search, and surface Khmer & English content from across the web.
Made with love by Ing Muyleang Β· KhmerStack
AngkorSearch is a fully self-hosted, open-source search engine built from scratch for Cambodia. It crawls websites across the entire web (with priority on Cambodian and Khmer-language sites), indexes content using PostgreSQL full-text search + trigram fuzzy matching, and serves results through a C++ REST API β with a modern Next.js frontend.
No external search APIs. No Google. No Bing. 100% self-hosted.
- 9-strategy fuzzy search β FTS + trigram + URL + prefix/suffix + per-word + description + domain matching
- Google-style sitelinks β top result shows sub-page links from the same domain
- Auto Web Discovery β when no results found, auto-discovers related URLs (GitHub, personal sites, npm, LinkedIn, Twitter/X, Substack) and crawls them live
- Direct Force Crawl β instantly fetch and index any URL; automatically saves og:image, twitter:image, and GitHub profile avatars to the images table
- Multiple content tabs β All, News, Images, Videos, Dev & Tech, AI Tools, Saved, History
- Full-width media grid β masonry image grid (2β6 columns), video grid (1β4 columns) with platform filter pills
- Image side panel β click any image to open a fixed right panel with preview, details, prev/next navigation (arrow keys + ESC), full-size / source / copy-URL actions
- Right-click image context menu β View details, Open full size, Open source page, Copy URL
- Video platform badges β per-result color badge (YouTube, TikTok, Twitter, Vimeo, etc.) with platform filter pills
- YouTube thumbnails β auto-constructed from video ID (
maxresdefault.jpg) - oEmbed thumbnails β TikTok, Twitter, Spotify thumbnails fetched via official oEmbed APIs
- Social links extraction β crawler saves Facebook, YouTube, TikTok, Telegram, Twitter, Instagram, LinkedIn links found on pages; exposed via
GET /social?domain= - AI Answer overview β powered by local Ollama LLM (no cloud, no API keys)
- AI Tools tab β quick-launch grid for Claude, ChatGPT, Perplexity, Gemini, HuggingFace, Ollama, arXiv, Discord
- Knowledge Panel β right-side info card for top results
- Data Manager (Admin) β storage overview (DB size, Redis, disk free), per-table size bars, delete all pages for a domain by name
- Dark / Light mode β persisted via localStorage
- Autocomplete suggestions β as you type
- Bookmarks & Search History β saved per user
- Admin Dashboard β seed domains, crawl queue, system monitoring, top searches, data management
- Multi-worker crawler β 4 concurrent C++ crawlers, up to 100,000 pages each
- Smart content cap β Cambodian/news/shallow pages get 80 KB of content indexed; all other pages get 500 chars summary-only to save DB space
- Fast search β no ILIKE on large content column (FTS GIN index covers it); Redis cache TTL 300s
- Open-domain crawling β crawls any public website (blocks walled gardens: Facebook, Instagram, TikTok, Twitter)
- Responsive UI β works on mobile, tablet, and desktop
ββββββββββββββββββββββββββββββββββββββββββββ
β User's Browser β
ββββββββββββββββββββ¬ββββββββββββββββββββββββ
β HTTP :80
ββββββββββββββββββββΌββββββββββββββββββββββββ
β nginx β
β Reverse Proxy :80 β
ββββββββ¬βββββββββββββββ¬βββββββββββββββ¬ββββββ
β β β
/auth/* /admin/ β /api/* β /* β
(users,roles...) β C++ API β Next.js β
β β Frontend β
βββββββββββββββΌβββ ββββββββββΌβββ βββββββββΌβββββββ
β auth service β β C++ API β β Next.js 14 β
β Go :8081 β β :8080 β β :3000 β
ββββββββββββββββββ βββββββ¬ββββββ ββββββββ¬ββββββββ
β β
ββββββββββββββΌββββββββββββββββ
β β
ββββββββββββββββΌβββ ββββββββΌββββββββββββββββ
β PostgreSQL 16 β β Redis 7 β
β :5432 β β :6379 β
β β β β
β pages β β visited URL set β
β crawl_queue β β search result cache β
β seeds β β (TTL 300s) β
β users β β domain cooldown β
β bookmarks β βββββββββββββββββββββββββ
β search_history β
β images β ββββββββββββββββββ
β videos β β Ollama LLM β
β news β β :11434 β
β github_repos β β qwen2.5:3b β
β social_links β ββββββββββββββββββ
β crawler_live β
ββββββββββββββββββββ
βββββββββββββββββββββββββββββββββββββββββββββββββ
β C++ Crawlers Γ 4 β
β crawler_1 crawler_2 crawler_3 crawler_4 β
β β
β 1. Pull URL from crawl_queue (priority ASC) β
β 2. Fetch with libcurl (12s timeout) β
β 3. Parse HTML (Gumbo parser) β
β Β· title, meta description, body text β
β Β· og:image, twitter:image (social meta) β
β Β· srcset / data-lazy-src images β
β Β· social link hrefs (FB/YT/TG/IG/TW/LI) β
β Β· YouTube embed β thumbnail URL β
β Β· TikTok/Twitter oEmbed thumbnails β
β 4. Smart content cap: β
β Β· Cambodian / news / depthβ€2 β 80 KB β
β Β· Everything else β 500 chars (summary) β
β 5. Save to pages table (PostgreSQL FTS) β
β 6. Save images/videos/social_links tables β
β 7. Enqueue outbound links β
β 8. Track visited URLs in Redis SET β
βββββββββββββββββββββββββββββββββββββββββββββββββ
User types query
β
βΌ
Next.js frontend
useSearch hook β GET /api/search?q=muyleang
β
βΌ
nginx β C++ API :8080 /search endpoint
β
βΌ
Query Expansion (9 strategies)
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β 1. FTS β tsvector @@ plainto_tsquery('simple', q) β
β 2. Trigram β title % q (pg_trgm fuzzy match) β
β 3. URL β url ILIKE '%q%' β
β 4. Title exact β title ILIKE '%q%' β
β 5. Title prefix β title ILIKE '%first65%' β
β 6. Title suffix β title ILIKE '%last60%' β
β 7. Per-word β title ILIKE '%word2%' β
β 8. Description β description ILIKE '%q%' β
β 9. Domain β domain ILIKE '%q%' β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β
βΌ
Ranking Score = FTS*3.0 + URL_match*1.5 + trigram*1.2
+ title_match*0.8 + description*0.2
β
βΌ
Check Redis cache β return if hit
β
βΌ
JSON response β Next.js β SearchResults component
β
βΌ (if 0 results)
WebDiscovery component (SSE)
β /api/auto-discover?q=muyleang
β guesses URLs: github.com/muyleang, muyleang.com,
muyleang.github.io, muyleang.dev, ...
β calls /admin/crawl-now for each candidate
β streams live terminal output to user
β auto-refreshes search when pages found
Admin enters URL in dashboard
β
βΌ
POST /api/crawl-stream?url=https://example.com
β
βΌ
nginx β Next.js API route (SSE stream)
β POST /admin/crawl-now (C++ API)
β
βββ fetch URL with libcurl (timeout 12s)
βββ parse HTML: title, meta description, body text
βββ detect language (Khmer Unicode range U+1780βU+17FF)
βββ INSERT INTO pages ... ON CONFLICT DO UPDATE
βββ mark crawled in crawl_queue
βββ SADD visited in Redis
β
βΌ
SSE stream β shows progress live in admin UI
β "Fetching page content..."
β "Indexed: <title> (N words)"
graph TB
Browser["Browser"] --> Nginx["nginx :80\nReverse Proxy"]
Nginx -->|"/auth/* /admin/users..."| Auth["auth service\nGo :8081"]
Nginx -->|"/api/*"| API["C++ API Server\n:8080"]
Nginx -->|"/api/crawl-stream\n/api/auto-discover"| Frontend["Next.js 14\n:3000"]
Nginx -->|"/*"| Frontend
API --> PG[("PostgreSQL 16\n:5432")]
API --> Redis[("Redis 7\n:6379")]
API --> Ollama["Ollama LLM\n:11434\nqwen2.5:3b"]
Frontend --> API
Crawlers["C++ Crawlers Γ 4"] --> PG
Crawlers --> Redis
subgraph Search Algorithm
FTS["1. FTS tsvector"]
Trigram["2. pg_trgm fuzzy"]
URLMatch["3. URL ILIKE"]
TitleExact["4. Title exact"]
TitlePrefix["5. Title prefix"]
TitleSuffix["6. Title suffix"]
PerWord["7. Per-word"]
DescMatch["8. Description"]
DomainMatch["9. Domain"]
end
API --> FTS
| Layer | Technology |
|---|---|
| Frontend | Next.js 14 (App Router), TypeScript, Tailwind CSS, Framer Motion |
| API Server | C++20, libpq, hiredis, libcurl, nlohmann/json |
| Crawler | C++20, libcurl, Gumbo HTML parser, libpq, hiredis |
| Database | PostgreSQL 16 with pg_trgm, unaccent, full-text search |
| Cache / Queue | Redis 7 |
| AI Answers | Ollama (local LLM β qwen2.5:3b by default) |
| Auth | Go service with JWT + session cookies |
| Proxy | nginx Alpine |
| Container | Docker + Docker Compose |
- Docker Desktop (Mac/Windows) or Docker + Docker Compose v2 (Linux)
- 4 GB RAM minimum (8 GB recommended for Ollama LLM)
# Clone
git clone https://github.com/MuyleangIng/angkorsearch
cd angkorsearch
# Start everything (builds all images, ~3-5 min first time)
docker compose up -d --build
# Open in browser
open http://localhostFirst boot takes 2β3 minutes. The crawler starts indexing seed domains automatically.
# Stop services (keeps data)
docker compose down
# Stop and wipe all data (fresh start)
docker compose down -v| Service | Description | Port |
|---|---|---|
nginx |
Reverse proxy β routes all traffic | 80 |
frontend |
Next.js 14 UI (standalone build) | 3000 |
api |
C++ REST API server | 8080 |
auth |
Go authentication service | 8081 |
crawler_1β4 |
4 parallel C++ web crawlers | β |
postgres |
PostgreSQL 16 database | 5432 |
redis |
Redis 7 cache + queue | 6379 |
ollama |
Local LLM inference server | 11434 |
ollama-init |
One-shot model downloader | β |
| Method | Endpoint | Description |
|---|---|---|
GET |
/search?q=angkor&type=web&page=1&lang=km |
Full-text + fuzzy search |
GET |
/suggest?q=cambo |
Autocomplete suggestions |
GET |
/ai/answer?q=what+is+angkor+wat |
AI-generated answer (Ollama) |
GET |
/live?since=10 |
Recently crawled pages |
GET |
/stats |
Index statistics |
GET |
/sitelinks?domain=example.com&exclude=https://example.com |
Sub-page links for Google-style sitelinks |
GET |
/social?domain=example.com |
Social media links extracted from a domain |
GET |
/health |
Health check |
Search types: web, news, image, video, github
Lang filter: km (Khmer), en (English), or omit for all
| Method | Endpoint | Description |
|---|---|---|
POST |
/bookmark |
Save a bookmark |
GET |
/bookmarks?user_id=1 |
Get saved bookmarks |
GET |
/history?user_id=1 |
Get search history |
DELETE |
/history?user_id=1 |
Clear search history |
| Method | Endpoint | Description |
|---|---|---|
GET |
/admin/stats |
Full index + crawl statistics |
GET |
/admin/seeds |
List seed domains |
POST |
/admin/seeds |
Add new seed domain |
PATCH |
/admin/seeds |
Update seed priority or status |
DELETE |
/admin/seeds?id=1 |
Delete a seed |
POST |
/admin/queue |
Force-add URL to crawl queue (P1) |
POST |
/admin/crawl-now |
Directly fetch + index a URL; saves og:image + GitHub avatar |
DELETE |
/admin/domain?domain=example.com |
Delete all pages/images/videos/news/social_links for a domain |
GET |
/admin/system |
System resource metrics |
| Method | Endpoint | Description |
|---|---|---|
GET |
/api/crawl-stream?url=... |
Force-crawl a URL, stream progress via SSE |
GET |
/api/auto-discover?q=... |
Auto-discover related URLs for a query, stream results via SSE |
AngkorSearch uses 9 parallel strategies to find results, then combines them with a ranking score:
Score = FTS_rank Γ 3.0 (full-text search β most important)
+ URL_match Γ 1.5 (query appears in URL)
+ trigram_similarity Γ 1.2 (fuzzy match via pg_trgm)
+ title_match Γ 0.8 (query in title)
+ description_match Γ 0.2 (query in description)
Example β searching "muyleang":
| Strategy | Match example |
|---|---|
| FTS | documents with "muyleang" in indexed tsvector |
| Trigram | "muyleanging.com" has ~33% trigram overlap with "muyleang" |
| URL ILIKE | url LIKE '%muyleang%' β catches muyleanging.com, github.com/muyleanging |
| Title prefix | searches %muylea% (65% of first word) |
| Title suffix | searches %eang% (last 60%) |
| Per-word | if multi-word query, searches each word separately |
| Description | description LIKE '%muyleang%' |
| Domain | domain LIKE '%muyleang%' |
This means a search for "leang" can still find "muyleanging.com" because the URL contains "leang".
When a search returns 0 results, the WebDiscovery component activates automatically:
- Sends the query to
/api/auto-discover?q=... - Generates candidate URLs from the query words (ordered by likelihood):
github.com/{slug}+{slug}.github.ioβ GitHub profile / pagesgitlab.com/{slug}β GitLab profile{slug}.com,{slug}.io,{slug}.dev,{slug}.me, β¦ (20 TLDs) β personal/project sites{slug}.vercel.app,{slug}.netlify.app,{slug}.pages.devβ hosted deploymentsdev.to/{slug},medium.com/@{slug},{slug}.hashnode.devβ dev blogsnpmjs.com/package/{slug},pypi.org/user/{slug}β package registrieslinkedin.com/in/{slug}β LinkedIn profiletwitter.com/{slug},x.com/{slug}β Twitter/X public profiles{slug}.substack.comβ Substack newsletteryoutube.com/@{slug}β YouTube channelhuggingface.co/{slug}β HuggingFace profile
- Calls
/admin/crawl-nowfor each candidate in parallel (8 at a time, 8s timeout each) - Streams live progress in a terminal-style UI
- Auto-refreshes search results when new pages are indexed
Note: Facebook, Instagram, TikTok, and other walled gardens block all bots. Even Google does not index private social media content. Public Twitter/X profile pages can sometimes be crawled.
Access at http://localhost/admin
| Tab | Features |
|---|---|
| Overview | Index stats, top domains, content breakdown, crawl progress, recently crawled table |
| Seed Domains | Add/remove seeds, set priority, block/allow toggle |
| Crawl Queue | Force-add any URL at Priority 1, domain progress bars, queue stats |
| System | RAM, Disk, Redis memory gauges, pages/hour, API uptime, DB table sizes |
| Searches | Top search queries bar chart |
| Data Manager | DB + Redis + disk storage overview, per-table size bars, delete all data for a domain |
angkorsearch/
βββ docker-compose.yml orchestrates all services
β
βββ angkorsearch-web/ Next.js 14 frontend (TypeScript)
β βββ app/
β β βββ page.tsx Homepage β search + discover feed
β β βββ search/page.tsx Search results + Knowledge Panel
β β βββ admin/page.tsx Admin dashboard (5 tabs)
β β βββ about/page.tsx About page + contributors
β β βββ api/
β β βββ crawl-stream/ SSE: force-crawl any URL live
β β β βββ route.ts
β β βββ auto-discover/ SSE: auto-discover + crawl related URLs
β β βββ route.ts
β βββ components/
β β βββ layout/ Header, Footer, Sidebar
β β βββ search/ SearchBox, SearchTabs, SearchResults
β β β βββ WebDiscovery (auto-discovery panel)
β β βββ results/ WebResult, NewsResult, ImageResult,
β β β VideoResult, GithubResult
β β βββ widgets/ AIOverview, KnowledgePanel, TopResult,
β β PeopleAlsoAsk, StatsBar, DiscoverFeed
β βββ hooks/ useSearch, useSuggest, useBookmark
β βββ lib/ api.ts, constants.ts, utils.ts, theme.tsx
β βββ Dockerfile Multi-stage Node 20 Alpine -> standalone
β
βββ api/
β βββ api_server.cpp C++ HTTP API server
β β Endpoints: search, suggest, ai/answer,
β β live, stats, bookmarks, history,
β β admin/stats, admin/seeds, admin/queue,
β β admin/crawl-now, admin/system
β βββ Dockerfile
β
βββ crawler/
β βββ crawler.cpp C++ multi-worker web crawler
β β libcurl + Gumbo HTML parser + libpq
β β Crawls all public domains (blocks walled gardens)
β β Priority: Cambodian=3, GitHub=4, others=7
β βββ Dockerfile
β
βββ auth/
β βββ ... Go auth service (JWT + sessions)
β
βββ postgres/
β βββ init.sql Database schema + indexes + views
β Tables: pages, crawl_queue, seeds, users,
β bookmarks, search_history, images, videos,
β news, github_repos, social_links,
β crawler_live
β
βββ nginx/
β βββ nginx.conf Reverse proxy config
β SSE routes: /api/crawl-stream,
β /api/auto-discover -> Next.js
β /api/* -> C++ API
β /auth/* -> Go auth
β
βββ data/
βββ dict/
βββ khmer_dict.txt Khmer word segmentation dictionary
-- Indexed web pages
pages (
id SERIAL PRIMARY KEY,
url TEXT UNIQUE,
domain TEXT,
title TEXT,
description TEXT,
content TEXT,
type TEXT, -- 'web' | 'news' | 'image' | 'video' | 'github'
lang TEXT, -- 'km' | 'en'
score FLOAT,
indexed_at TIMESTAMP,
tsv TSVECTOR -- FTS index
)
-- Crawl queue with priority
crawl_queue (
id SERIAL PRIMARY KEY,
url TEXT UNIQUE,
domain TEXT,
priority INT, -- 1=force, 2=high, 5=normal, 10=low
crawled BOOLEAN DEFAULT FALSE,
added_at TIMESTAMP
)
-- Seed domains
seeds (
id SERIAL PRIMARY KEY,
url TEXT,
domain TEXT,
type TEXT,
priority INT,
active BOOLEAN
)
-- Social media links extracted from crawled pages
social_links (
id SERIAL PRIMARY KEY,
domain TEXT,
platform TEXT, -- 'facebook' | 'youtube' | 'tiktok' | 'telegram' | 'twitter' | 'instagram' | 'linkedin'
url TEXT,
source_page TEXT,
found_at TIMESTAMP,
UNIQUE(domain, platform, url)
)
-- Images indexed from crawled pages
images (
id SERIAL PRIMARY KEY,
url TEXT UNIQUE,
src TEXT,
alt TEXT,
domain TEXT,
page_url TEXT,
width INT,
height INT,
indexed_at TIMESTAMP
)
-- Videos indexed from crawled pages
videos (
id SERIAL PRIMARY KEY,
url TEXT UNIQUE,
title TEXT,
thumb_url TEXT,
domain TEXT,
channel TEXT,
page_url TEXT,
indexed_at TIMESTAMP
)# Connect to PostgreSQL
docker compose exec postgres psql -U angkor -d angkorsearch-- Total pages indexed
SELECT COUNT(*) FROM pages;
-- By content type
SELECT type, COUNT(*) FROM pages GROUP BY type ORDER BY COUNT(*) DESC;
-- Crawl status by domain
SELECT * FROM v_crawl_status;
-- Top searches
SELECT query, COUNT(*) FROM search_history GROUP BY query ORDER BY COUNT(*) DESC LIMIT 20;
-- Queue status
SELECT
COUNT(*) FILTER (WHERE NOT crawled) AS pending,
COUNT(*) FILTER (WHERE crawled) AS done
FROM crawl_queue;
-- Search for a specific page
SELECT url, title, lang, score FROM pages WHERE url ILIKE '%muyleang%';Via Admin UI at /admin β Seed Domains tab, or via API:
curl -X POST http://localhost/api/admin/seeds \
-d "url=https://phnompenhpost.com&type=news&priority=2"Priority levels:
1β Force (crawled immediately)2β High5β Normal (default)10β Low
Set in docker-compose.yml or a .env file:
| Variable | Default | Description |
|---|---|---|
POSTGRES_DB |
angkorsearch |
Database name |
POSTGRES_USER |
angkor |
Database user |
POSTGRES_PASSWORD |
angkor123 |
Database password |
NEXT_PUBLIC_API_URL |
http://localhost |
Public API URL |
API_INTERNAL_URL |
http://api:8080 |
Internal API URL (Next.js β C++ API) |
OLLAMA_MODEL |
qwen2.5:3b |
Ollama model for AI answers |
MAX_PAGES |
100000 |
Max pages per crawler worker |
Run more crawler workers:
docker compose up -d --scale crawler=8Run multiple API instances behind nginx:
docker compose up -d --scale api=3# On Ubuntu / Debian VPS
curl -fsSL https://get.docker.com | sh
git clone https://github.com/MuyleangIng/angkorsearch
cd angkorsearch
# Set your domain in nginx/nginx.conf:
# server_name yourdomain.com www.yourdomain.com;
docker compose up -d --buildFor HTTPS, add Certbot + nginx SSL config, or put Cloudflare in front.
- Google-style sitelinks on top search result (
/sitelinksendpoint) - Social links extraction β crawler detects and stores FB/YT/TT/TG/TW/IG/LI links (
social_linkstable,/socialendpoint) - YouTube thumbnails auto-constructed from embed video IDs
- oEmbed thumbnails for TikTok, Twitter, Spotify via official APIs
crawlNownow saves og:image, twitter:image, and GitHub avatar toimagestable- Full-width media layout β masonry image grid, video grid with platform filter pills
- Image side panel with keyboard nav (arrows + ESC), context menu, copy/open actions
- Video platform badges (YouTube, TikTok, Vimeo, Twitter, etc.)
- AI Tools tab β quick-launch panel for popular AI services
- Data Manager admin tab β storage stats + delete-by-domain
DELETE /admin/domainendpoint β wipes all rows for a domain across 6 tables- Auto-discovery expanded: Twitter/X profiles, Substack newsletters
- Smart content cap: 80 KB for Cambodian/news/shallow pages, 500 chars for others
- Search speed: removed
content ILIKEfrom WHERE (FTS covers it); Redis TTL 300s
- FTS switched to
'simple'dictionary + ILIKE fallback (fixed Khmer/short-word search) - 4 parallel C++ crawler workers
- AI Answer via local Ollama (qwen2.5:3b)
- Admin dashboard with domain chart, type/lang breakdown, live feed
- Knowledge Panel, PeopleAlsoAsk, WebDiscovery SSE stream
- Complete rewrite: C++ API + Next.js 14 frontend
- PostgreSQL FTS + pg_trgm 9-strategy search
- Docker Compose multi-service setup
| Avatar | Username | Role |
|---|---|---|
| MuyleangIng | Creator & Lead Engineer | |
| ingdavann | Contributor | |
| Jessiebrownleo | Contributor | |
| MengseuThoeng | Contributor | |
| prox-dex | Contributor | |
| YithSopheaktra8 | Contributor |
Part of the KhmerStack organization β building modern tech for Cambodia.
MIT License β free to use, modify, and deploy.