Skip to content

Latest commit

Β 

History

8 Commits

Folders and files

NameName
Last commit message
Last commit date
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

AngkorSearch v2.3

Cambodia's open-source search engine β€” built to index, search, and surface Khmer & English content from across the web.

Made with love by Ing Muyleang Β· KhmerStack


What is AngkorSearch?

AngkorSearch is a fully self-hosted, open-source search engine built from scratch for Cambodia. It crawls websites across the entire web (with priority on Cambodian and Khmer-language sites), indexes content using PostgreSQL full-text search + trigram fuzzy matching, and serves results through a C++ REST API β€” with a modern Next.js frontend.

No external search APIs. No Google. No Bing. 100% self-hosted.


Features

  • 9-strategy fuzzy search β€” FTS + trigram + URL + prefix/suffix + per-word + description + domain matching
  • Google-style sitelinks β€” top result shows sub-page links from the same domain
  • Auto Web Discovery β€” when no results found, auto-discovers related URLs (GitHub, personal sites, npm, LinkedIn, Twitter/X, Substack) and crawls them live
  • Direct Force Crawl β€” instantly fetch and index any URL; automatically saves og:image, twitter:image, and GitHub profile avatars to the images table
  • Multiple content tabs β€” All, News, Images, Videos, Dev & Tech, AI Tools, Saved, History
  • Full-width media grid β€” masonry image grid (2–6 columns), video grid (1–4 columns) with platform filter pills
  • Image side panel β€” click any image to open a fixed right panel with preview, details, prev/next navigation (arrow keys + ESC), full-size / source / copy-URL actions
  • Right-click image context menu β€” View details, Open full size, Open source page, Copy URL
  • Video platform badges β€” per-result color badge (YouTube, TikTok, Twitter, Vimeo, etc.) with platform filter pills
  • YouTube thumbnails β€” auto-constructed from video ID (maxresdefault.jpg)
  • oEmbed thumbnails β€” TikTok, Twitter, Spotify thumbnails fetched via official oEmbed APIs
  • Social links extraction β€” crawler saves Facebook, YouTube, TikTok, Telegram, Twitter, Instagram, LinkedIn links found on pages; exposed via GET /social?domain=
  • AI Answer overview β€” powered by local Ollama LLM (no cloud, no API keys)
  • AI Tools tab β€” quick-launch grid for Claude, ChatGPT, Perplexity, Gemini, HuggingFace, Ollama, arXiv, Discord
  • Knowledge Panel β€” right-side info card for top results
  • Data Manager (Admin) β€” storage overview (DB size, Redis, disk free), per-table size bars, delete all pages for a domain by name
  • Dark / Light mode β€” persisted via localStorage
  • Autocomplete suggestions β€” as you type
  • Bookmarks & Search History β€” saved per user
  • Admin Dashboard β€” seed domains, crawl queue, system monitoring, top searches, data management
  • Multi-worker crawler β€” 4 concurrent C++ crawlers, up to 100,000 pages each
  • Smart content cap β€” Cambodian/news/shallow pages get 80 KB of content indexed; all other pages get 500 chars summary-only to save DB space
  • Fast search β€” no ILIKE on large content column (FTS GIN index covers it); Redis cache TTL 300s
  • Open-domain crawling β€” crawls any public website (blocks walled gardens: Facebook, Instagram, TikTok, Twitter)
  • Responsive UI β€” works on mobile, tablet, and desktop

Architecture

High-Level Overview

                        β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
                        β”‚              User's Browser               β”‚
                        β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                           β”‚ HTTP :80
                        β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
                        β”‚                nginx                      β”‚
                        β”‚          Reverse Proxy :80                β”‚
                        β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”˜
                               β”‚              β”‚              β”‚
              /auth/* /admin/  β”‚   /api/*     β”‚     /*       β”‚
              (users,roles...) β”‚  C++ API     β”‚  Next.js     β”‚
                               β”‚              β”‚  Frontend    β”‚
                 β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”  β”Œβ”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”
                 β”‚  auth service  β”‚  β”‚ C++ API   β”‚  β”‚  Next.js 14  β”‚
                 β”‚  Go :8081      β”‚  β”‚ :8080     β”‚  β”‚  :3000       β”‚
                 β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜  β””β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”˜  β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜
                                           β”‚               β”‚
                              β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                              β”‚            β”‚
               β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”  β”Œβ”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
               β”‚   PostgreSQL 16  β”‚  β”‚      Redis 7          β”‚
               β”‚   :5432          β”‚  β”‚      :6379            β”‚
               β”‚                  β”‚  β”‚                       β”‚
               β”‚  pages           β”‚  β”‚  visited URL set      β”‚
               β”‚  crawl_queue     β”‚  β”‚  search result cache  β”‚
               β”‚  seeds           β”‚  β”‚  (TTL 300s)           β”‚
               β”‚  users           β”‚  β”‚  domain cooldown      β”‚
               β”‚  bookmarks       β”‚  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
               β”‚  search_history  β”‚
               β”‚  images          β”‚         β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
               β”‚  videos          β”‚         β”‚   Ollama LLM   β”‚
               β”‚  news            β”‚         β”‚   :11434       β”‚
               β”‚  github_repos    β”‚         β”‚   qwen2.5:3b   β”‚
               β”‚  social_links    β”‚         β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
               β”‚  crawler_live    β”‚
               β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

               β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
               β”‚              C++ Crawlers Γ— 4                 β”‚
               β”‚  crawler_1  crawler_2  crawler_3  crawler_4   β”‚
               β”‚                                               β”‚
               β”‚  1. Pull URL from crawl_queue (priority ASC)  β”‚
               β”‚  2. Fetch with libcurl (12s timeout)          β”‚
               β”‚  3. Parse HTML (Gumbo parser)                 β”‚
               β”‚     Β· title, meta description, body text      β”‚
               β”‚     Β· og:image, twitter:image (social meta)   β”‚
               β”‚     Β· srcset / data-lazy-src images           β”‚
               β”‚     Β· social link hrefs (FB/YT/TG/IG/TW/LI)  β”‚
               β”‚     Β· YouTube embed β†’ thumbnail URL           β”‚
               β”‚     Β· TikTok/Twitter oEmbed thumbnails        β”‚
               β”‚  4. Smart content cap:                        β”‚
               β”‚     Β· Cambodian / news / depth≀2 β†’ 80 KB     β”‚
               β”‚     Β· Everything else β†’ 500 chars (summary)  β”‚
               β”‚  5. Save to pages table (PostgreSQL FTS)      β”‚
               β”‚  6. Save images/videos/social_links tables    β”‚
               β”‚  7. Enqueue outbound links                    β”‚
               β”‚  8. Track visited URLs in Redis SET           β”‚
               β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Data Flow: Search Request

User types query
      β”‚
      β–Ό
Next.js frontend
  useSearch hook β†’ GET /api/search?q=muyleang
      β”‚
      β–Ό
nginx β†’ C++ API :8080  /search endpoint
      β”‚
      β–Ό
  Query Expansion (9 strategies)
  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
  β”‚  1. FTS  β€” tsvector @@ plainto_tsquery('simple', q)     β”‚
  β”‚  2. Trigram β€” title % q  (pg_trgm fuzzy match)          β”‚
  β”‚  3. URL β€” url ILIKE '%q%'                               β”‚
  β”‚  4. Title exact β€” title ILIKE '%q%'                     β”‚
  β”‚  5. Title prefix β€” title ILIKE '%first65%'              β”‚
  β”‚  6. Title suffix β€” title ILIKE '%last60%'               β”‚
  β”‚  7. Per-word β€” title ILIKE '%word2%'                    β”‚
  β”‚  8. Description β€” description ILIKE '%q%'               β”‚
  β”‚  9. Domain β€” domain ILIKE '%q%'                         β”‚
  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
      β”‚
      β–Ό
  Ranking Score = FTS*3.0 + URL_match*1.5 + trigram*1.2
                + title_match*0.8 + description*0.2
      β”‚
      β–Ό
  Check Redis cache β†’ return if hit
      β”‚
      β–Ό
JSON response β†’ Next.js β†’ SearchResults component
      β”‚
      β–Ό (if 0 results)
WebDiscovery component (SSE)
  β†’ /api/auto-discover?q=muyleang
  β†’ guesses URLs: github.com/muyleang, muyleang.com,
                  muyleang.github.io, muyleang.dev, ...
  β†’ calls /admin/crawl-now for each candidate
  β†’ streams live terminal output to user
  β†’ auto-refreshes search when pages found

Data Flow: Force Crawl (Admin)

Admin enters URL in dashboard
      β”‚
      β–Ό
POST /api/crawl-stream?url=https://example.com
      β”‚
      β–Ό
nginx β†’ Next.js API route (SSE stream)
  β†’ POST /admin/crawl-now  (C++ API)
        β”‚
        β”œβ”€β”€ fetch URL with libcurl (timeout 12s)
        β”œβ”€β”€ parse HTML: title, meta description, body text
        β”œβ”€β”€ detect language (Khmer Unicode range U+1780–U+17FF)
        β”œβ”€β”€ INSERT INTO pages ... ON CONFLICT DO UPDATE
        β”œβ”€β”€ mark crawled in crawl_queue
        └── SADD visited in Redis
      β”‚
      β–Ό
SSE stream β†’ shows progress live in admin UI
  β†’ "Fetching page content..."
  β†’ "Indexed: <title> (N words)"

Mermaid Architecture Diagram

graph TB
    Browser["Browser"] --> Nginx["nginx :80\nReverse Proxy"]

    Nginx -->|"/auth/* /admin/users..."| Auth["auth service\nGo :8081"]
    Nginx -->|"/api/*"| API["C++ API Server\n:8080"]
    Nginx -->|"/api/crawl-stream\n/api/auto-discover"| Frontend["Next.js 14\n:3000"]
    Nginx -->|"/*"| Frontend

    API --> PG[("PostgreSQL 16\n:5432")]
    API --> Redis[("Redis 7\n:6379")]
    API --> Ollama["Ollama LLM\n:11434\nqwen2.5:3b"]

    Frontend --> API

    Crawlers["C++ Crawlers Γ— 4"] --> PG
    Crawlers --> Redis

    subgraph Search Algorithm
        FTS["1. FTS tsvector"]
        Trigram["2. pg_trgm fuzzy"]
        URLMatch["3. URL ILIKE"]
        TitleExact["4. Title exact"]
        TitlePrefix["5. Title prefix"]
        TitleSuffix["6. Title suffix"]
        PerWord["7. Per-word"]
        DescMatch["8. Description"]
        DomainMatch["9. Domain"]
    end

    API --> FTS
Loading

Tech Stack

Layer Technology
Frontend Next.js 14 (App Router), TypeScript, Tailwind CSS, Framer Motion
API Server C++20, libpq, hiredis, libcurl, nlohmann/json
Crawler C++20, libcurl, Gumbo HTML parser, libpq, hiredis
Database PostgreSQL 16 with pg_trgm, unaccent, full-text search
Cache / Queue Redis 7
AI Answers Ollama (local LLM β€” qwen2.5:3b by default)
Auth Go service with JWT + session cookies
Proxy nginx Alpine
Container Docker + Docker Compose

Quick Start

Requirements

  • Docker Desktop (Mac/Windows) or Docker + Docker Compose v2 (Linux)
  • 4 GB RAM minimum (8 GB recommended for Ollama LLM)

Run

# Clone
git clone https://github.com/MuyleangIng/angkorsearch
cd angkorsearch

# Start everything (builds all images, ~3-5 min first time)
docker compose up -d --build

# Open in browser
open http://localhost

First boot takes 2–3 minutes. The crawler starts indexing seed domains automatically.

Stop

# Stop services (keeps data)
docker compose down

# Stop and wipe all data (fresh start)
docker compose down -v

Docker Services

Service Description Port
nginx Reverse proxy β€” routes all traffic 80
frontend Next.js 14 UI (standalone build) 3000
api C++ REST API server 8080
auth Go authentication service 8081
crawler_1–4 4 parallel C++ web crawlers β€”
postgres PostgreSQL 16 database 5432
redis Redis 7 cache + queue 6379
ollama Local LLM inference server 11434
ollama-init One-shot model downloader β€”

API Endpoints

Search & Discovery

Method Endpoint Description
GET /search?q=angkor&type=web&page=1&lang=km Full-text + fuzzy search
GET /suggest?q=cambo Autocomplete suggestions
GET /ai/answer?q=what+is+angkor+wat AI-generated answer (Ollama)
GET /live?since=10 Recently crawled pages
GET /stats Index statistics
GET /sitelinks?domain=example.com&exclude=https://example.com Sub-page links for Google-style sitelinks
GET /social?domain=example.com Social media links extracted from a domain
GET /health Health check

Search types: web, news, image, video, github Lang filter: km (Khmer), en (English), or omit for all

Bookmarks & History

Method Endpoint Description
POST /bookmark Save a bookmark
GET /bookmarks?user_id=1 Get saved bookmarks
GET /history?user_id=1 Get search history
DELETE /history?user_id=1 Clear search history

Admin

Method Endpoint Description
GET /admin/stats Full index + crawl statistics
GET /admin/seeds List seed domains
POST /admin/seeds Add new seed domain
PATCH /admin/seeds Update seed priority or status
DELETE /admin/seeds?id=1 Delete a seed
POST /admin/queue Force-add URL to crawl queue (P1)
POST /admin/crawl-now Directly fetch + index a URL; saves og:image + GitHub avatar
DELETE /admin/domain?domain=example.com Delete all pages/images/videos/news/social_links for a domain
GET /admin/system System resource metrics

Next.js API Routes (SSE)

Method Endpoint Description
GET /api/crawl-stream?url=... Force-crawl a URL, stream progress via SSE
GET /api/auto-discover?q=... Auto-discover related URLs for a query, stream results via SSE

Search Algorithm

AngkorSearch uses 9 parallel strategies to find results, then combines them with a ranking score:

Score = FTS_rank Γ— 3.0          (full-text search β€” most important)
      + URL_match Γ— 1.5          (query appears in URL)
      + trigram_similarity Γ— 1.2 (fuzzy match via pg_trgm)
      + title_match Γ— 0.8        (query in title)
      + description_match Γ— 0.2  (query in description)

Example β€” searching "muyleang":

Strategy Match example
FTS documents with "muyleang" in indexed tsvector
Trigram "muyleanging.com" has ~33% trigram overlap with "muyleang"
URL ILIKE url LIKE '%muyleang%' β€” catches muyleanging.com, github.com/muyleanging
Title prefix searches %muylea% (65% of first word)
Title suffix searches %eang% (last 60%)
Per-word if multi-word query, searches each word separately
Description description LIKE '%muyleang%'
Domain domain LIKE '%muyleang%'

This means a search for "leang" can still find "muyleanging.com" because the URL contains "leang".


Auto Web Discovery

When a search returns 0 results, the WebDiscovery component activates automatically:

  1. Sends the query to /api/auto-discover?q=...
  2. Generates candidate URLs from the query words (ordered by likelihood):
    • github.com/{slug} + {slug}.github.io β€” GitHub profile / pages
    • gitlab.com/{slug} β€” GitLab profile
    • {slug}.com, {slug}.io, {slug}.dev, {slug}.me, … (20 TLDs) β€” personal/project sites
    • {slug}.vercel.app, {slug}.netlify.app, {slug}.pages.dev β€” hosted deployments
    • dev.to/{slug}, medium.com/@{slug}, {slug}.hashnode.dev β€” dev blogs
    • npmjs.com/package/{slug}, pypi.org/user/{slug} β€” package registries
    • linkedin.com/in/{slug} β€” LinkedIn profile
    • twitter.com/{slug}, x.com/{slug} β€” Twitter/X public profiles
    • {slug}.substack.com β€” Substack newsletter
    • youtube.com/@{slug} β€” YouTube channel
    • huggingface.co/{slug} β€” HuggingFace profile
  3. Calls /admin/crawl-now for each candidate in parallel (8 at a time, 8s timeout each)
  4. Streams live progress in a terminal-style UI
  5. Auto-refreshes search results when new pages are indexed

Note: Facebook, Instagram, TikTok, and other walled gardens block all bots. Even Google does not index private social media content. Public Twitter/X profile pages can sometimes be crawled.


Admin Dashboard

Access at http://localhost/admin

Tab Features
Overview Index stats, top domains, content breakdown, crawl progress, recently crawled table
Seed Domains Add/remove seeds, set priority, block/allow toggle
Crawl Queue Force-add any URL at Priority 1, domain progress bars, queue stats
System RAM, Disk, Redis memory gauges, pages/hour, API uptime, DB table sizes
Searches Top search queries bar chart
Data Manager DB + Redis + disk storage overview, per-table size bars, delete all data for a domain

Project Structure

angkorsearch/
β”œβ”€β”€ docker-compose.yml              orchestrates all services
β”‚
β”œβ”€β”€ angkorsearch-web/               Next.js 14 frontend (TypeScript)
β”‚   β”œβ”€β”€ app/
β”‚   β”‚   β”œβ”€β”€ page.tsx                Homepage β€” search + discover feed
β”‚   β”‚   β”œβ”€β”€ search/page.tsx         Search results + Knowledge Panel
β”‚   β”‚   β”œβ”€β”€ admin/page.tsx          Admin dashboard (5 tabs)
β”‚   β”‚   β”œβ”€β”€ about/page.tsx          About page + contributors
β”‚   β”‚   └── api/
β”‚   β”‚       β”œβ”€β”€ crawl-stream/       SSE: force-crawl any URL live
β”‚   β”‚       β”‚   └── route.ts
β”‚   β”‚       └── auto-discover/      SSE: auto-discover + crawl related URLs
β”‚   β”‚           └── route.ts
β”‚   β”œβ”€β”€ components/
β”‚   β”‚   β”œβ”€β”€ layout/                 Header, Footer, Sidebar
β”‚   β”‚   β”œβ”€β”€ search/                 SearchBox, SearchTabs, SearchResults
β”‚   β”‚   β”‚                           └── WebDiscovery (auto-discovery panel)
β”‚   β”‚   β”œβ”€β”€ results/                WebResult, NewsResult, ImageResult,
β”‚   β”‚   β”‚                               VideoResult, GithubResult
β”‚   β”‚   └── widgets/                AIOverview, KnowledgePanel, TopResult,
β”‚   β”‚                                   PeopleAlsoAsk, StatsBar, DiscoverFeed
β”‚   β”œβ”€β”€ hooks/                      useSearch, useSuggest, useBookmark
β”‚   β”œβ”€β”€ lib/                        api.ts, constants.ts, utils.ts, theme.tsx
β”‚   └── Dockerfile                  Multi-stage Node 20 Alpine -> standalone
β”‚
β”œβ”€β”€ api/
β”‚   β”œβ”€β”€ api_server.cpp              C++ HTTP API server
β”‚   β”‚                               Endpoints: search, suggest, ai/answer,
β”‚   β”‚                               live, stats, bookmarks, history,
β”‚   β”‚                               admin/stats, admin/seeds, admin/queue,
β”‚   β”‚                               admin/crawl-now, admin/system
β”‚   └── Dockerfile
β”‚
β”œβ”€β”€ crawler/
β”‚   β”œβ”€β”€ crawler.cpp                 C++ multi-worker web crawler
β”‚   β”‚                               libcurl + Gumbo HTML parser + libpq
β”‚   β”‚                               Crawls all public domains (blocks walled gardens)
β”‚   β”‚                               Priority: Cambodian=3, GitHub=4, others=7
β”‚   └── Dockerfile
β”‚
β”œβ”€β”€ auth/
β”‚   └── ...                         Go auth service (JWT + sessions)
β”‚
β”œβ”€β”€ postgres/
β”‚   └── init.sql                    Database schema + indexes + views
β”‚                                   Tables: pages, crawl_queue, seeds, users,
β”‚                                   bookmarks, search_history, images, videos,
β”‚                                   news, github_repos, social_links,
β”‚                                   crawler_live
β”‚
β”œβ”€β”€ nginx/
β”‚   └── nginx.conf                  Reverse proxy config
β”‚                                   SSE routes: /api/crawl-stream,
β”‚                                   /api/auto-discover -> Next.js
β”‚                                   /api/* -> C++ API
β”‚                                   /auth/* -> Go auth
β”‚
└── data/
    └── dict/
        └── khmer_dict.txt          Khmer word segmentation dictionary

Database Schema

Key Tables

-- Indexed web pages
pages (
    id          SERIAL PRIMARY KEY,
    url         TEXT UNIQUE,
    domain      TEXT,
    title       TEXT,
    description TEXT,
    content     TEXT,
    type        TEXT,   -- 'web' | 'news' | 'image' | 'video' | 'github'
    lang        TEXT,   -- 'km' | 'en'
    score       FLOAT,
    indexed_at  TIMESTAMP,
    tsv         TSVECTOR  -- FTS index
)

-- Crawl queue with priority
crawl_queue (
    id        SERIAL PRIMARY KEY,
    url       TEXT UNIQUE,
    domain    TEXT,
    priority  INT,   -- 1=force, 2=high, 5=normal, 10=low
    crawled   BOOLEAN DEFAULT FALSE,
    added_at  TIMESTAMP
)

-- Seed domains
seeds (
    id       SERIAL PRIMARY KEY,
    url      TEXT,
    domain   TEXT,
    type     TEXT,
    priority INT,
    active   BOOLEAN
)

-- Social media links extracted from crawled pages
social_links (
    id          SERIAL PRIMARY KEY,
    domain      TEXT,
    platform    TEXT,   -- 'facebook' | 'youtube' | 'tiktok' | 'telegram' | 'twitter' | 'instagram' | 'linkedin'
    url         TEXT,
    source_page TEXT,
    found_at    TIMESTAMP,
    UNIQUE(domain, platform, url)
)

-- Images indexed from crawled pages
images (
    id         SERIAL PRIMARY KEY,
    url        TEXT UNIQUE,
    src        TEXT,
    alt        TEXT,
    domain     TEXT,
    page_url   TEXT,
    width      INT,
    height     INT,
    indexed_at TIMESTAMP
)

-- Videos indexed from crawled pages
videos (
    id         SERIAL PRIMARY KEY,
    url        TEXT UNIQUE,
    title      TEXT,
    thumb_url  TEXT,
    domain     TEXT,
    channel    TEXT,
    page_url   TEXT,
    indexed_at TIMESTAMP
)

Useful Queries

# Connect to PostgreSQL
docker compose exec postgres psql -U angkor -d angkorsearch
-- Total pages indexed
SELECT COUNT(*) FROM pages;

-- By content type
SELECT type, COUNT(*) FROM pages GROUP BY type ORDER BY COUNT(*) DESC;

-- Crawl status by domain
SELECT * FROM v_crawl_status;

-- Top searches
SELECT query, COUNT(*) FROM search_history GROUP BY query ORDER BY COUNT(*) DESC LIMIT 20;

-- Queue status
SELECT
  COUNT(*) FILTER (WHERE NOT crawled) AS pending,
  COUNT(*) FILTER (WHERE crawled)     AS done
FROM crawl_queue;

-- Search for a specific page
SELECT url, title, lang, score FROM pages WHERE url ILIKE '%muyleang%';

Adding Seed Domains

Via Admin UI at /admin β†’ Seed Domains tab, or via API:

curl -X POST http://localhost/api/admin/seeds \
  -d "url=https://phnompenhpost.com&type=news&priority=2"

Priority levels:

  • 1 β€” Force (crawled immediately)
  • 2 β€” High
  • 5 β€” Normal (default)
  • 10 β€” Low

Environment Variables

Set in docker-compose.yml or a .env file:

Variable Default Description
POSTGRES_DB angkorsearch Database name
POSTGRES_USER angkor Database user
POSTGRES_PASSWORD angkor123 Database password
NEXT_PUBLIC_API_URL http://localhost Public API URL
API_INTERNAL_URL http://api:8080 Internal API URL (Next.js β†’ C++ API)
OLLAMA_MODEL qwen2.5:3b Ollama model for AI answers
MAX_PAGES 100000 Max pages per crawler worker

Scaling

Run more crawler workers:

docker compose up -d --scale crawler=8

Run multiple API instances behind nginx:

docker compose up -d --scale api=3

Production Deployment

# On Ubuntu / Debian VPS
curl -fsSL https://get.docker.com | sh

git clone https://github.com/MuyleangIng/angkorsearch
cd angkorsearch

# Set your domain in nginx/nginx.conf:
# server_name yourdomain.com www.yourdomain.com;

docker compose up -d --build

For HTTPS, add Certbot + nginx SSL config, or put Cloudflare in front.


Changelog

v2.3 (2025)

  • Google-style sitelinks on top search result (/sitelinks endpoint)
  • Social links extraction β€” crawler detects and stores FB/YT/TT/TG/TW/IG/LI links (social_links table, /social endpoint)
  • YouTube thumbnails auto-constructed from embed video IDs
  • oEmbed thumbnails for TikTok, Twitter, Spotify via official APIs
  • crawlNow now saves og:image, twitter:image, and GitHub avatar to images table
  • Full-width media layout β€” masonry image grid, video grid with platform filter pills
  • Image side panel with keyboard nav (arrows + ESC), context menu, copy/open actions
  • Video platform badges (YouTube, TikTok, Vimeo, Twitter, etc.)
  • AI Tools tab β€” quick-launch panel for popular AI services
  • Data Manager admin tab β€” storage stats + delete-by-domain
  • DELETE /admin/domain endpoint β€” wipes all rows for a domain across 6 tables
  • Auto-discovery expanded: Twitter/X profiles, Substack newsletters
  • Smart content cap: 80 KB for Cambodian/news/shallow pages, 500 chars for others
  • Search speed: removed content ILIKE from WHERE (FTS covers it); Redis TTL 300s

v2.2 (2024)

  • FTS switched to 'simple' dictionary + ILIKE fallback (fixed Khmer/short-word search)
  • 4 parallel C++ crawler workers
  • AI Answer via local Ollama (qwen2.5:3b)
  • Admin dashboard with domain chart, type/lang breakdown, live feed
  • Knowledge Panel, PeopleAlsoAsk, WebDiscovery SSE stream

v2.0 (2024)

  • Complete rewrite: C++ API + Next.js 14 frontend
  • PostgreSQL FTS + pg_trgm 9-strategy search
  • Docker Compose multi-service setup

Contributors

Avatar Username Role
MuyleangIng MuyleangIng Creator & Lead Engineer
ingdavann ingdavann Contributor
Jessiebrownleo Jessiebrownleo Contributor
MengseuThoeng MengseuThoeng Contributor
prox-dex prox-dex Contributor
YithSopheaktra8 YithSopheaktra8 Contributor

Part of the KhmerStack organization β€” building modern tech for Cambodia.


License

MIT License β€” free to use, modify, and deploy.


Built for Cambodia Β· by Cambodians
muyleanging.com Β· KhmerStack

About

πŸ‡°πŸ‡­ Open source search engine for Cambodia β€” Web, News, Images, Videos & GitHub. Built with C++, PostgreSQL, Redis & Next.js. Supporting Khmer and English.

Resources

Contributing

Stars

2 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages