data(website): seed the new website category with 40,084 Wikidata records - #138
Conversation
Seeds the new `website` category (TechEngine §6.12) with the Q35127 subtree:
40,084 records, all verified:false, each citing its Wikidata item.
Fields are written only where the source had them — homepage_url on 85%,
plus launch_date, languages and owners where present. Nothing is inferred.
Also mirrors the engine's WEBSITE_REQUIRED rule into the bundled validator
{slug, name, source_urls, verified} with slug, launch_date and source-url checks.
Import notes: 47,631 QIDs in the subtree yielded 40,273 English-labelled items;
189 were dropped for an unusable name, and 243 slug collisions took a
-<qid> suffix. Windows-reserved stems (CON, PRN, LPT1 …) are skipped by the
importer — a `con.json` from the game import previously made the tree
unindexable by git.
Refs #1
Bumps the TechEngine submodule to the commit that carries the website category (§6.12) plus the seed date-coercion fix, and publishes site/public/v1/websites for all 40,084 records. The manifest keeps its existing twelve collections and gains websites; the partial dump writes a manifest describing only what it generated, so the entry was merged into the real one rather than replacing it. Refs #1
🔎 Data verification — Tier 0 (offline existence/trust)Scored by Changed records in this PRNo records scored. Full-dataset baseline105742 record(s) scored. %%{init: {"theme":"base","themeVariables":{"pie1":"#3fb950","pie2":"#d29922","pie3":"#f85149","pieStrokeWidth":"0px","pieOpacity":"1"}}}%%
pie showData
title Verification bands — all records
"Green" : 14778
"Yellow" : 89175
"Red" : 1789
Hard violations (forced red):
green = authoritative source + complete + consistent · yellow = plausible, needs confirmation · red = sparse/weak source or a hard contradiction. Promotion to |
TechEngine change review: PASS
Changed data
Changed record examples
Heuristic review
|
TechEngine validation stats: PASS
Data summary
Warning Tracked verified coverage is below 50% for watch 2.5% (11/433), brand 4.8% (10/207), tablet 5.0% (174/3455), soc 5.8% (123/2104), smartphone 9.1% (8453/93396), all 9.5% (10028/105742), gpu 12.0% (244/2030), pda 19.3% (27/140), and 1 more. Validation notes
Key output:
|
What
Seeds the
websitecategory added in TechEngine #48 with 40,084 records from Wikidata's Q35127 subtree, bumps the submodule to the engine that serves them, and publishes the dump.Dataset: 1,113,452 → 1,153,536 records, and the API gains a 13th collection at
/v1/websites.Why this source
The dataset's own device categories are exhausted, so the question was what adds genuinely new entities rather than more of what we already hold. Two candidates were measured before committing to either:
game— rejected. 90% of its titles already exist, and the apparent 9% remainder is mostly name variants of records we have (PUBG: BATTLEGROUNDSvs ourplayerunknowns-battlegrounds,Grand Theft Auto V Legacyvsgrand-theft-auto-v). Importing it would have inflated the count with duplicate entities.What was imported
47,631 QIDs → 40,273 English-labelled items → 40,084 records. 189 dropped for an unusable name, 243 slug collisions took a
-<qid>suffix, and Windows-reserved stems (CON,PRN,LPT1…) are skipped — acon.jsonfrom the game import previously made the tree unindexable by git.Fields are written only where the source had them:
homepage_url(85% of imported rows),launch_date,languages,owners. Everything isverified: falseand cites its Wikidata item.Notes on the dump
SQLite Date type only accepts Python date objects as input) because the loader coerced dates by the field namerelease_dateand this category's column islaunch_date.websitesentry was merged into the realsite/public/v1/index.jsonrather than replacing it — the other twelve collections are intact (verified in the diff).python -m app.validate→ Data validation passed (the bundled validator gained the matchingWEBSITE_REQUIREDrule).Closes #1