Skip to content

data(website): seed the new website category with 40,084 Wikidata records - #138

Merged
Seungpyo1007 merged 2 commits into
mainfrom
data/website-category
Jul 30, 2026
Merged

data(website): seed the new website category with 40,084 Wikidata records#138
Seungpyo1007 merged 2 commits into
mainfrom
data/website-category

Conversation

@Seungpyo1007

@Seungpyo1007 Seungpyo1007 commented Jul 30, 2026

Copy link
Copy Markdown
Member

What

Seeds the website category added in TechEngine #48 with 40,084 records from Wikidata's Q35127 subtree, bumps the submodule to the engine that serves them, and publishes the dump.

Dataset: 1,113,452 → 1,153,536 records, and the API gains a 13th collection at /v1/websites.

Why this source

The dataset's own device categories are exhausted, so the question was what adds genuinely new entities rather than more of what we already hold. Two candidates were measured before committing to either:

  • Steam / SteamSpy top-up of game — rejected. 90% of its titles already exist, and the apparent 9% remainder is mostly name variants of records we have (PUBG: BATTLEGROUNDS vs our playerunknowns-battlegrounds, Grand Theft Auto V Legacy vs grand-theft-auto-v). Importing it would have inflated the count with duplicate entities.
  • Wikidata websites — 47,631 items in the subtree, no overlap with any existing category, and 89% carry an official URL. That is far better populated than the electronics categories probed earlier this session (digital cameras: 34 items total).

What was imported

47,631 QIDs → 40,273 English-labelled items → 40,084 records. 189 dropped for an unusable name, 243 slug collisions took a -<qid> suffix, and Windows-reserved stems (CON, PRN, LPT1…) are skipped — a con.json from the game import previously made the tree unindexable by git.

Fields are written only where the source had them: homepage_url (85% of imported rows), launch_date, languages, owners. Everything is verified: false and cites its Wikidata item.

Notes on the dump

  • The submodule moves to the engine commit carrying both the category and the seed fix from TechEngine data(soc): GPU enrichment round 2 (13 older chips) #49 — without that fix, seeding this tree failed on the first insert (SQLite Date type only accepts Python date objects as input) because the loader coerced dates by the field name release_date and this category's column is launch_date.
  • A partial dump writes a manifest describing only the collection it generated, so the websites entry was merged into the real site/public/v1/index.json rather than replacing it — the other twelve collections are intact (verified in the diff).

python -m app.validateData validation passed (the bundled validator gained the matching WEBSITE_REQUIRED rule).

Closes #1

Seeds the new `website` category (TechEngine §6.12) with the Q35127 subtree:
40,084 records, all verified:false, each citing its Wikidata item.

Fields are written only where the source had them — homepage_url on 85%,
plus launch_date, languages and owners where present. Nothing is inferred.

Also mirrors the engine's WEBSITE_REQUIRED rule into the bundled validator
{slug, name, source_urls, verified} with slug, launch_date and source-url checks.

Import notes: 47,631 QIDs in the subtree yielded 40,273 English-labelled items;
189 were dropped for an unusable name, and 243 slug collisions took a
-<qid> suffix. Windows-reserved stems (CON, PRN, LPT1 …) are skipped by the
importer — a `con.json` from the game import previously made the tree
unindexable by git.

Refs #1
Bumps the TechEngine submodule to the commit that carries the website category
(§6.12) plus the seed date-coercion fix, and publishes site/public/v1/websites
for all 40,084 records.

The manifest keeps its existing twelve collections and gains websites; the
partial dump writes a manifest describing only what it generated, so the entry
was merged into the real one rather than replacing it.

Refs #1
@TechEngineBot

Copy link
Copy Markdown
Member

🔎 Data verification — Tier 0 (offline existence/trust)

Scored by app.verify; posted by TechEngineBot. Informational only — the structural gate (app.validate) is separate and authoritative for merge.

Changed records in this PR

No records scored.

Full-dataset baseline

105742 record(s) scored.

%%{init: {"theme":"base","themeVariables":{"pie1":"#3fb950","pie2":"#d29922","pie3":"#f85149","pieStrokeWidth":"0px","pieOpacity":"1"}}}%%
pie showData
    title Verification bands — all records
    "Green" : 14778
    "Yellow" : 89175
    "Red" : 1789
Loading
Category Distribution Total 🟢 🟡 🔴 🟢 %
brand 🟩🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨 207 24 183 0 11.6%
soc 🟩🟨🟨🟨🟨🟨🟥🟥🟥🟥🟥🟥 2104 123 804 1177 5.8%
smartphone 🟩🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟥 93396 11602 81206 588 12.4%
tablet 🟩🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟥 3455 174 3266 15 5.0%
watch 🟩🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟥 433 11 415 7 2.5%
pda 🟩🟩🟨🟨🟨🟨🟨🟨🟨🟨🟨🟥 140 27 112 1 19.3%
gpu 🟩🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨 2030 245 1785 0 12.1%
cpu 🟩🟩🟩🟩🟩🟩🟩🟨🟨🟨🟨🟥 3977 2572 1404 1 64.7%
All 🟩🟩🟨🟨🟨🟨🟨🟨🟨🟨🟨🟥 105742 14778 89175 1789 14.0%

Hard violations (forced red):

Count Check
1 !boost_ge_base

green = authoritative source + complete + consistent · yellow = plausible, needs confirmation · red = sparse/weak source or a hard contradiction. Promotion to verified runs in the scheduled verify-network workflow.

@Seungpyo1007
Seungpyo1007 merged commit 13b566c into main Jul 30, 2026
5 checks passed
@github-project-automation github-project-automation Bot moved this from In Progress to Done in TechAPI-Project Jul 30, 2026
@Seungpyo1007
Seungpyo1007 deleted the data/website-category branch July 30, 2026 09:16
@TechEngineBot

Copy link
Copy Markdown
Member

TechEngine change review: PASS

Check Result
python -m app.validate PASS
python integrity_check.py TechAPI/data --strict PASS

Changed data

Category Added Modified Deleted Added verified Added unverified Added Kaggle-sourced
brand 0 0 0 0 0 0
soc 0 0 0 0 0 0
smartphone 0 0 0 0 0 0
tablet 0 0 0 0 0 0
watch 0 0 0 0 0 0
pda 0 0 0 0 0 0
gpu 0 0 0 0 0 0
cpu 0 0 0 0 0 0

Changed record examples

  • No data file changes detected.

Heuristic review

  • Heuristic warnings: none found.

@TechEngineBot

Copy link
Copy Markdown
Member

TechEngine validation stats: PASS

Data summary

Category Total Verified Unverified Missing verified Tracked Verified % of tracked
brand 207 10 197 0 207 4.8%
soc 2104 123 1981 0 2104 5.8%
smartphone 93396 8453 84943 0 93396 9.1%
tablet 3455 174 3281 0 3455 5.0%
watch 433 11 422 0 433 2.5%
pda 140 27 113 0 140 19.3%
gpu 2030 244 1786 0 2030 12.0%
cpu 3977 986 2991 0 3977 24.8%
all 105742 10028 95714 0 105742 9.5%

Warning

Tracked verified coverage is below 50% for watch 2.5% (11/433), brand 4.8% (10/207), tablet 5.0% (174/3455), soc 5.8% (123/2104), smartphone 9.1% (8453/93396), all 9.5% (10028/105742), gpu 12.0% (244/2030), pda 19.3% (27/140), and 1 more.
Tracked coverage excludes records missing the verified field; see the Missing verified column for those records.
This does not fail validation. Keep imported records verified: false until manual audit, but treat this as follow-up verification work before relying on the affected categories as curated data.

Validation notes

  • Full advisory outlier listings are suppressed on successful runs because they are dataset-wide and mostly stable between PRs.
  • Failure runs still include a detailed log excerpt for debugging.

Key output:

## app.validate
## integrity_check.py --strict
loaded CPU=3977 GPU=2030
✅ integrity gate: no hard anomalies.
Integrity section Flagged lines
structural 0
CPU name/tier consistency (desktop mainstream only) 0
CPU single>multi (cinebench/geekbench — should be multi>=single) 0
CPU era-vs-score outliers 8
CPU cross-source ratio outliers (possible wrong-variant) 152
GPU cross-source ratio outliers + sanity 18

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

app Validator or application code changes data Dataset changes enhancement New feature or request

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

Massive dataset rebuild: CPU + brand + GPU + smartphone + SoC (1989-2026)

2 participants