Semantic-Swingers at LLMs4OL 2026 (Tasks A, B & C): native learner integration#338
Open
matias-vizcaino wants to merge 23 commits into
Open
Semantic-Swingers at LLMs4OL 2026 (Tasks A, B & C): native learner integration#338matias-vizcaino wants to merge 23 commits into
matias-vizcaino wants to merge 23 commits into
Conversation
…ntic-Swingers taxonomy learner [Task C] - selector="ollama": free local reproduction of the champion selection pipeline via Ollama's OpenAI-compatible endpoint (no API key) - max_tokens constructor param (was hardcoded 32, which silently truncates thinking models into empty answers) - docs page (llms4ol_challenge/semanticswingers_learner.rst) + toctree entry - validated on Wine through LearnerPipeline: gpt-4.1-mini f1=0.21, llama3.1:8b f1=0.07, embedding baseline f1=0.07
- closed-vocabulary term typing: fit learns the type inventory from the train split; inference assigns types from that inventory only - three selectors mirroring the Task C learner: embedding (offline nearest-type), openai champion (gpt-4.1-mini), ollama local - registered in learner/term_typing; validated offline on Wine via LearnerPipeline (embedding selector f1=0.39)
…r both tasks Wine micro-probe via native LearnerPipeline (term-typing pair F1): embedding 0.39, ollama llama3.1:8b 0.69, openai gpt-4.1-mini 0.69 — the local selector matches the champion on this task.
Hermetic unit tests for SemanticSwingersTermTypingLearner and SemanticSwingersTaxonomyLearner (Tasks B/C): fit/predict contract, embedding-selector determinism, closed-vocabulary parsing in _parse_typings, and openai-selector fallback to the offline embedding selector when no api_key is available (no network attempted, verified via mocked OpenAI client). Encoders are mocked with fixed vectors so tests need no downloaded models, network, or ontology data. Adds a Reproducibility subsection to the learner docs page documenting which selector reproduces which reported number and how api_key resolution/fallback behaves.
…ask A, flagship)
One learner class, two hooks, dispatched on the task string:
- _text2onto: RAG (top-k document exemplars) generation with a LoRA
fine-tuned Qwen/Qwen3.5-9B (RA-FT/base-FT), the competition champion,
projecting extracted triples onto OntoLearner's native {terms, types}.
- _taxonomy_discovery: composed (not rewritten) from the existing
SemanticSwingersTaxonomyLearner, since the native harness for that
task hands over a bare type vocabulary with no source document text.
qwen3_5/qwen3_next has no released transformers support yet, so the
git-source requirement is a lazy import inside the learner raising an
actionable ImportError with the exact pinned-commit pip install line
(verified 2026-07-09 to both register the architecture and produce
coherent triple extraction with the real PEFT adapter) rather than a
core pyproject.toml dependency.
Adds the docs section, example script, and 19 hermetic unit tests
(mocked LLM/retriever, no network).
Replace the placeholder semanticswingers/* ids with the real private HF repos (datagero/qwen3.5-9b-raft-taska, datagero/qwen3.5-9b-baseft-taska) created for yp4 slice 3, now hosting the uploaded PEFT adapters.
… _text2onto output
text2onto_metrics reads only terms/types and silently ignores extra keys, and the
full prediction dict passes through unchanged to run_report['predictions']. Adding
a "triples" key ([doc_id, subject, relation, object] per row) is therefore
non-breaking: native scoring is unchanged, while the ~90% is-a signal the
{terms, types} projection would otherwise discard survives for downstream
inspection. Per IC ruling (ADR-0018 addendum §4, llms4ol-2026 main repo): the
"propose, don't patch core" outcome for the harness-fit discussion, implemented
as the zero-core-cost additive demonstration referenced in the PR's Future-work
section.
…-> working) The term-typing and taxonomy-discovery LLM selectors called chat.completions.create() with no reasoning override. qwen3.x served over an OpenAI-compatible endpoint (Ollama) is a *thinking* model: left on it spends the entire max_tokens budget on hidden reasoning and returns empty content (finish_reason="length", content=""), so nothing parses and both selectors score F1 = 0. Pass reasoning_effort="none" via extra_body for qwen3* models, mirroring the fix the competition pipeline already applies for the same qwen+RAG=0 failure. It is a no-op for non-qwen3 models, so it is always safe to include. Verified on the Wine benchmark (seed 42, local qwen3.5-nothink:9b), through the real learner code path: term-typing F1 0.0000 -> 0.6593 (30/33) [offline embedding: 0.3939] taxonomy-discovery F1 0.0000 -> 0.1429 (1/9) [offline embedding: 0.0000] Adds a guard test per learner so the regression cannot return.
The term-typing and taxonomy-discovery learners each take selector= and can be
served by a local Ollama, a paid OpenAI model, or offline embeddings. The Task A
learner had no equivalent: generation was hardwired to transformers + PEFT via
_require_qwen35_transformers -> PeftModel.from_pretrained -> model.generate.
Consequences of that gap: the flagship task was the only one a reviewer could not
run without a CUDA box, and anyone wanting a local run had to subclass the learner
in notebook code — untested, unreviewed, and free to drift from what ships.
Adds backend= ("peft" default, unchanged behaviour; "ollama"; "openai") with
_OLLAMA_BASE_URL / _DEFAULT_LLM constants mirroring the other two learners. API
backends short-circuit load() (no local weights, no qwen3_5 guard) and generate via
_generate_api. _chat_messages renders the exemplars as structured turns carrying the
same content as the ChatML prompt, so backends stay comparable. Reuses the
reasoning_effort="none" guard so qwen3.x thinking models answer instead of returning
empty.
Verified: backend="ollama" with qwen3.5-nothink:9b on val_20 produces triples with no
wrapper (gs_v2_exact 0.3197 on a 3-doc probe, ~17 s/doc). Adds 3 tests.
Pre-existing failures in this suite are unrelated to this change (peft not installed
in the dev venv; the qwen3_5 import-guard test cannot trigger now that released
transformers>=5.14 registers the architecture).
Two additions that close the "inference-only" gap and complete the backend matrix,
so the learner can both TRAIN and SERVE across CUDA / Apple Silicon / API / base.
Training — fit() can now produce the adapter, not just consume it:
train_mode = "raft" retrieval-aware; top-k exemplars baked into each prompt with
LEAVE-ONE-OUT retrieval (a doc never sees its own gold)
= "baseft" no exemplars (k=0)
= None unchanged: fit() only indexes exemplars (default)
train_backend = "peft" (CUDA LoRA SFT) | "mlx" (Apple Silicon, mlx_lm.lora)
output_dir where the adapter is written; fit() then loads it for inference
New semanticswingers_train.py keeps the two rails that break SILENTLY as pure-Python,
unit-tested functions — prompt masking (loss on completion tokens only, else the model
learns to reproduce documents) and leave-one-out (else train/eval leakage inflates the
score). The heavy trainers (_train_peft / _train_mlx) import lazily, so the module
imports with neither stack installed. Ported from the team's ft_nvidia.py (CUDA) and
the mlx_lm LoRA path.
Inference — backend="mlx" loads an MLX base + optional MLX-format LoRA and generates
via mlx_lm, reusing the same ChatML prompt as the PEFT path. Verified on an M4 Max:
backend="mlx" + local MLX adapter, no wrapper, gs_v2_exact 0.3746 on a 3-doc val_20
probe at ~12 s/doc — the fine-tuned champion running on a laptop.
Adds 7 tests (prompt masking incl. left-truncation, leave-one-out, baseft-no-exemplars,
backend defaults, unknown-backend guard). Pre-existing suite failures are unrelated
(peft not installed in the dev venv; the qwen3_5 import-guard test cannot trigger now
that released transformers>=5.14 registers the architecture).
The mlx trainer used a train()/dataset binding that doesn't match mlx_lm's tuner API (train() takes no tokenizer; this build's CompletionsDataset.__getitem__ returns the raw record, not processed tokens). Corrects both, plus the LoRA-layer wiring. Marks _train_mlx honestly: it drives mlx_lm's LOW-LEVEL tuner API, which is not stable across releases, and is NOT yet validated end-to-end here — the config-key contract still varies by version. The validated pieces are unchanged: the pure-Python data rails (leave-one-out, prompt masking) with unit tests, and the backend="mlx" inference path (gs_v2_exact 0.3746 on-Mac). Recommends the `mlx_lm lora` CLI for a robust run until the low-level binding is pinned. No claim that the mlx trainer runs.
The prior mlx trainer bound mlx_lm's low-level tuner API (train()/CompletionsDataset/
linear_to_lora_layers), which shifts between mlx_lm releases — three separate API
mismatches surfaced while validating it. Replace that with a subprocess call to
`python -m mlx_lm lora --train`, the stable CLI and the exact command the team's
ft35_overnight.py used to produce the real MLX adapters. Robust across versions, no
low-level loop to keep in sync. Reads the same {prompt, completion} jsonl (mask_prompt
handled by the CLI), writes adapters.safetensors + adapter_config.json for
backend="mlx" inference. Data rails (leave-one-out, prompt masking) and their tests
are unchanged.
…po-id loading
Updates _ADAPTER_REPOS to the published, generically-named registry
(datagero/qwen3.5-9b-ontology-extraction-{raft,baseft,baseft-mlx}) and adds the MLX
alias. So adapter="raft"/"baseft"/"baseft-mlx" now resolve to real, uploaded repos.
backend="mlx" can now load an adapter given a HF repo id: mlx_lm.load(adapter_path=)
only accepts a LOCAL directory (unlike PEFT's from_pretrained, which fetches repo ids),
so we snapshot_download the adapter first when it isn't an on-disk path. Verified
end-to-end against the private MLX repo on Apple Silicon: adapter="baseft-mlx" pulls
the adapter, loads base+adapter, and generates triples (6 on a val_20 doc, ~10s load).
Updates the alias test for the new names incl. baseft-mlx.
… others The learners baked our specific choices in as module constants, forcing anyone adapting them to a different domain to edit the package or subclass. Expose the two that matter as constructor arguments, defaulting to the team's values so existing behaviour is byte-identical: Task A: system_prompt=, typing_relations= Task B: system_prompt= Everything else was already configuration-driven (model, backend, adapter, retriever, training regime). Adds examples/llm_learner_semanticswingers_extend.py demonstrating the three levels of customization — arguments only (model/backend/retriever), domain (prompt + relations), and behaviour (subclass one method) — plus a training snippet, and documents the extension points on the class docstring. Backward-compatible; 5 new tests. Pre-existing suite failures unchanged (peft not in the dev venv; the qwen3_5 import-guard test can't trigger on released transformers).
…rompt injectable sciknoworg#2 (generalize the local prompt to any model family): _build_prompt now renders the same message list through the loaded tokenizer's own chat template (apply_chat_template) instead of hardcoded Qwen ChatML, so a non-Qwen local base model works on the peft/mlx path (the ollama/openai backends were already model-agnostic via role messages). VERIFIED byte-identical to the old hand-built ChatML for Qwen/Qwen3.5-9B — enable_thinking=False reproduces the <think></think> no-think suffix exactly — so the team's trained adapters see the same prompt. Falls back to the manual ChatML when no tokenizer/template is present. Confirmed end-to-end: mlx adapter still generates triples through the new path. sciknoworg#3 (Task C prompt extensibility): the parent-selection instruction is now a selection_prompt= constructor arg (a {child}/{candidates} template, default unchanged), matching the system_prompt/typing_relations extension points on Task A/B. Backward-compatible; +1 test. Pre-existing suite failures unchanged (peft absent in the dev venv; qwen3_5 import-guard can't trigger on released transformers).
… 9B on CPU The peft backend called from_pretrained without device_map/.to, so device='cuda' was ignored and the base+adapter loaded on CPU — correct output but minutes/doc (GPU 0% util, ~390MiB) on a CUDA box, unusable at scale. Load weights directly onto the requested accelerator via device_map; CPU stays default when device=cpu. Surfaced by the notebook §5 RunPod CUDA validation (2026-07-24).
…ort & tests - SemanticSwingersMatrixTaxonomyLearner.load() now fetches the trained structural matrix from the public HF registry (datagero/taxonomy-structural-matrix-1024-mxbai) when no local copy exists, so the champion runs from a fresh clone (was: silent no-op if file missing). - Fix broken import: the module imported 'utils' (absent in sentence_transformers 5.x -> ImportError on import) while the Matrix path actually used 'util.cos_sim' (NameError at runtime) -> import 'util'; the champion path was never runnable. - Add 4 unit tests for the Matrix learner (bilinear scoring, DAG cleanup, HF-download fallback, default registry id); ruff clean, 63 tests pass. - Document the structural-matrix champion variant in the learner docs page. Co-authored-by: andy-symonds <andrew.symonds@thoughtworks.com>
…ve, not champion Aligns the docs with the paper + team framing: the primary Task C pipeline is the LLM selector (SemanticSwingersTaxonomyLearner); the structural-matrix learner is a scale-aware alternative for very large ontologies, not the champion.
…lternative Align the class docstring (and the HF-repo comment) with the paper + PR framing: the matrix learner is a scale-aware alternative to the primary LLM pipeline, not the champion.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR contributes the Semantic-Swingers team learners built for the LLMs4OL 2026 challenge (Tasks A, B, and C) as native
AutoLearnersubclasses, integrated end-to-end throughLearnerPipelineand the existing native metrics. It is the mandatory System-Integration deliverable for our participation.Each learner subclasses
AutoLearnerand implements a single task hook (_taxonomy_discovery/_term_typing/_text2onto), plugs in viaLearnerPipeline, and ships with docs, examples, and unit tests — no changes to OntoLearner core.What's included
ontolearner/learner/)taxonomy_discovery/semanticswingers.py— Task C (taxonomy discovery). Includes both our primary LLM pipeline (SemanticSwingersTaxonomyLearner) and a scalable alternative (SemanticSwingersMatrixTaxonomyLearner) that uses a custom 1024-D Bilinear Adjacency Layer coupled with NetworkX DAG reduction to bypass LLM inference bottlenecks on massive ontologies.term_typing/semanticswingers.py— Task B (term typing)text2onto/semanticswingers.py(+semanticswingers_train.py) — Task A (text-to-onto, flagship)__init__.pypeft(CUDA),mlx(Apple Silicon),ollama, andopenaiselectors, so the learners run on paid or fully-open/offline stacks.docs/source/learners/llms4ol_challenge/semanticswingers_learner.rstexamples/llm_learner_semanticswingers_{taxonomy_discovery,term_typing,text2onto,extend}.py. (The Task C example is a unified script demonstrating both the baseline LLM and the alternative Matrix pipelines side-by-side.)tests/test_semanticswingers_{taxonomy_discovery,term_typing,text2onto}.py(63 tests, all mocked — no network/model download needed)Trained weights (Hugging Face Hub, public)
The Task A learner loads LoRA adapters (Qwen3.5-9B) and the Task C alternative pipeline loads a custom PyTorch structural matrix — all hosted publicly on the
datageroregistry. Weights auto-download on firstload()when no local copy is present, so the learners run from a fresh clone with no manual setup:datagero/qwen3.5-9b-ontology-extraction-raft— Task A, retrieval-aware FT (k=10), CUDA/peftdatagero/qwen3.5-9b-ontology-extraction-baseft— Task A, base FT (k=0), CUDA/peftdatagero/qwen3.5-9b-ontology-extraction-baseft-mlx— Task A, base FT, Apple-Silicon/MLXdatagero/taxonomy-structural-matrix-1024-mxbai— Task C, 1024-D Bilinear Adjacency Layer adapter (mxbai-embed-large-v1)Methodological diagram & reproduction
Per the challenge's recommendation, supplementary material lives on the companion branch
companion/semanticswingers-llms4ol2026:METHODOLOGY.md— the shared two-stage retrieve→generate/select pipeline and per-task specialization.notebooks/pipeline_ontolearner.ipynb— end-to-end run across all three tasks.This material is reference-only and is not part of this PR.
Quality gates
ruff check(v0.8.2, repo-pinned) — clean on all added filespytest tests/test_semanticswingers_*.py— 63 passedFuture work / discussion (proposal, not part of this PR)
Our Task A learner satisfies the
text2ontohook by projecting its extractedis-atriples to the{terms, types}shape the harness scores, and satisfiestaxonomy-discoveryby delegating to the Task C retriever (the basetasks_data_formerstrips taxonomy-discovery input to a deduped vocabulary). This works cleanly and additively, but it under-uses the richer relational signal our generator produces. For future editions, and only with maintainer buy-in, we'd be glad to offer a follow-up PR proposing:taxonomy-discoveryvariant that feeds source documents to a generative learner, andtext2ontopath plus agraph_similaritymetric that scoresis-aedges directly.Both are core eval-harness changes (new dispatch arms + metric), so we deliberately keep them out of this integration PR. As a non-intrusive demonstration,
_text2ontoalso emits the raw triples under an extra key that the current harness ignores, so the retained signal is inspectable without changing any scored path.Notes for review
devbranch per CONTRIBUTING.md.