Conversation
jdsika
force-pushed
the
feat/envited-x-pipeline
branch
9 times, most recently
from
May 12, 2026 16:35
a705c35 to
3d3a52a
Compare
jdsika
force-pushed
the
feat/envited-x-pipeline
branch
3 times, most recently
from
June 10, 2026 19:42
cab84ad to
b2b3dba
Compare
rmessaou
force-pushed
the
feat/envited-x-pipeline
branch
from
July 8, 2026 09:02
97e73d0 to
ef7ad85
Compare
3 tasks done
jdsika
force-pushed
the
feat/envited-x-pipeline
branch
from
September 11, 2026 14:23
6aa7702 to
233adc3
Compare
jdsika
force-pushed
the
feat/envited-x-pipeline
branch
from
September 11, 2026 14:44
233adc3 to
b8a388e
Compare
This was referenced Sep 14, 2026
RDFC-1.0 canonicalization already makes RDF output deterministic:
isomorphic graphs always serialize identically. It does not make output
diffable. Blank nodes are numbered `c14nN` in a single global order, so
inserting one class can renumber every blank node after it and rewrite
most of the file. A one-line semantic change lands as a whole-file diff,
which makes generated OWL/SHACL hard to review and noisy to keep under
version control.
Add a `diff_stable` argument to `canonicalize_rdf_graph()` and a
`--diff-stable/--no-diff-stable` flag to the four RDF generators. When
enabled, blank-node labels are derived from each node's own neighbourhood
via Weisfeiler-Lehman refinement, so an edit relabels only the blank
nodes it actually touches.
Measured churn on a real schema (add one class, count changed lines):
generator default --diff-stable
owlgen 2091 17
shexgen 796 50
shaclgen 291 13
rdfgen 115 25
Output stays deterministic and isomorphic either way; only the choice of
label changes. Off by default, because enabling it relabels existing
output.
The refinement itself lives in `diffable-rdf`, whose only dependencies
(rdflib, pyoxigraph) are already linkml-runtime dependencies at higher
versions, so this adds no new transitive dependencies.
…-opping Bump the floor to diffable-rdf 0.3.0 and add the missing uv.lock entry: the dependency was declared in pyproject.toml but never locked, so "uv lock --check" and the "uv sync --frozen" anti-malware gate would both have failed CI. 0.3.0 also fixes two defects in the Weisfeiler-Lehman labelling this feature relies on. Disconnected blank-node components now converge independently, so an edit in one region no longer relabels an unrelated one. And the suffix used to tell structurally indistinguishable nodes apart was assigned in c14nN *text* order, so c14n10 sorted between c14n1 and c14n2 -- adding a tenth tied blank node relabelled eight of the nine already there, the exact opposite of what this labelling is for. Separately, diff_stable=True was silently ignored whenever pyoxigraph refused the graph and canonicalize_rdf_graph degraded to rdflib. Weisfeiler-Lehman refinement consumes canonical pyoxigraph quads, and that path exists precisely because there are none, so the argument could not be honoured -- but the caller was never told. "shaclgen --include-annotations --diff-stable" reaches it, via the literal predicate an annotation tag without a ':' produces, and returned output byte-identical to --no-diff-stable. It now warns, with a regression test asserting the warning and the byte-identical output that makes silence misleading.
0.4.0 carries graph.base through the library's rdflib fallback, verifying that every absolute IRI of the source survives a re-read rather than dropping the directive outright, and adds a diff_stable parameter to canonicalize_rdf_graph. The lock entry is written by hand because the workspace sets exclude-newer = "7 days", which filters any release younger than that from resolution; 0.3.0 was pinned the same way for the same reason, and both become resolvable normally on 2026-09-18. uv lock --check and uv sync --all-groups both accept the entry. https://github.com/ASCS-eV/diffable-rdf/releases/tag/v0.4.0
Registers a diffable_rdf pytest marker and applies it to the six diff_stable tests (including the parametrized per-generator test), so an external CI job can run 'pytest -m diffable_rdf' against a local diffable-rdf checkout without pulling in the rest of the suite. Follow-up to a review comment on this PR requesting canary tests from linkml running in diffable-rdf's own CI, modeled on the numpydantic project's tests-linkml.yml workflow.
Expose runtime and LinkML extras with a lazy import and an installation hint. Move usage guidance into the collaboration guide and shorten generator help. Assert designed RDF edits, source preservation, generator output and CLI flags. Check ordinary and extra wheel installs in the existing package build job.
Addresses the second review round. Remove the installed-wheel checks and the CI step that drove them. They were a one-off apparatus for something the test suite already has a mechanism for, and part of it only asserted that uv builds wheel metadata correctly from pyproject.toml rather than testing anything about linkml. Test the missing extra with a new mock_missing_import fixture built on the existing MockImportErrorFinder, so the next optional dependency needs no fixture of its own. Guard the import the way bigquerygen already does: find_spec raises under that fixture, so the install hint never surfaced. Drop the default-value test, restore the pre-existing sort test, and cut the CLI matrix to the single case that shows the flag reaching the serializer. Record the convention in AGENTS.md, point MockImportErrorFinder at the general fixture, and link the guide from the diff_stable parameter.
gen-owl canonicalizes its output with RDFC-1.0 before serializing, and the determinism work adds a --diff-stable option on top of it, but neither is mentioned anywhere in the generator documentation. Describe what is guaranteed without passing any flag, why RDFC-1.0's sequential blank-node numbering can still produce noisy diffs across schema edits, and what --diff-stable changes. Also record the behaviour users meet in practice but cannot discover from --help: that the same option exists on gen-rdf, gen-shacl and gen-shex, and that graphs which are not standard RDF -- literal predicates from SHACL annotation mode, relative IRIs such as the metamodel's bibo:status <testing> -- take an rdflib fallback that stays reproducible across processes, warns, and deliberately does not honour --diff-stable.
The rules-to-SHACL-SPARQL converter added in linkml#3451 recognised a single named pattern. This adds the presence-implies-value pattern: a precondition asserting `value_presence: PRESENT` on one slot, and a postcondition constraining another slot with `equals_string` or `equals_string_in`. It reads as "if the guard slot is present, the target slot must be present and hold one of the allowed values", and generalises the existing boolean guard to arbitrary enum values. The boolean guard is now gated on the target slot's range actually being `boolean`. Without that gate a slot of range `string` carrying `equals_string: "true"` was translated as a boolean comparison, which does not match the string `"true"` in the data and so flagged conforming instances as violations. String-ranged slots now fall through to the presence-implies-value pattern and compare as strings. Pattern matching is exact: each converter requires its conditions to set precisely the operators it translates. A rule whose conditions carry anything further -- extra scalar operators, or expression-level any_of / all_of / none_of / exactly_one_of -- is skipped rather than partially translated, since dropping a term would either widen the precondition (false positives) or weaken the postcondition (false negatives). Slot resolution goes through induced slots so that `slot_usage` overrides, `slot_uri` overrides and alias-form keys resolve to the same IRI that `sh:path` emits. Co-authored-by: jdsika <carlo.van-driesten@vdl.digital>
…ersion The named-pattern converters only recognise whole-rule shapes, so a rule one operator away from a known pattern produced no constraint at all. This adds a compositional fallback, tried only after every named pattern has declined, that builds the query from the operators present rather than from a fixed template. Preconditions become a conjunction of graph patterns and FILTERs; the single postcondition becomes its negation. Together they select focus nodes satisfying every precondition while violating the postcondition, which is exactly the SHACL-SPARQL violation contract of SHACL 5.3.1, with $this pre-bound to the focus node. Operator support is declared per operator, and the builder returns None -- skipping the rule -- as soon as it meets one it does not handle, so an unsupported combination is never partially translated. This covers conditional-required and conditional-absent postconditions, numeric threshold preconditions, nested-object preconditions and has_member list membership, in any combination the operators allow. Numeric bounds are validated before interpolation. minimum_value and maximum_value have metamodel range Anything, so YAML strings, dates and .nan / .inf reach the generator unchanged; interpolating them raw either produced unparsable SPARQL that poisons the whole shapes graph at validation time, or -- for a date such as 2020-01-01 -- parsed as an arithmetic expression that silently never fires. Only int and finite float are rendered; anything else skips the rule. String literals are escaped rather than interpolated, and a slot carrying both minimum_value and maximum_value now yields both bounds instead of only the first. Nested and member slots resolve against the range class of their container, so an inner slot is no longer shadowed by a same-named slot on the outer class, and a range narrowed through slot_usage resolves its enum permissible values from the narrowed range. Co-authored-by: jdsika <carlo.van-driesten@vdl.digital>
The SHACL generator translates LinkML rules into sh:sparql constraints, but the generator documentation did not mention it, so the feature was undiscoverable and its limits undocumented. Describe the recognised named patterns and the compositional fallback, the --emit-rules flag, the skip-never-mis-translate contract and which rule attributes warn, with a worked YAML-to-Turtle example. Note that SPARQL-based constraints need a processor with SHACL-SPARQL support.
The SHACL generator translated any_of branches by dispatching
solely on `any.range` (class, type, enum, or simple datatype).
If a branch specified `pattern:` — either alone or combined
with a range — the constraint was silently dropped, producing
an empty blank node `[ ]` (trivially satisfied) instead of the
intended `[ sh:pattern "..." ]`.
This is a problem for schemas that use pattern alternatives in
`any_of`, such as the SPDX license field where valid values are
either members of a fixed enum (SPDX identifiers), IRIs, or
custom identifiers matching the LicenseRef- pattern defined in
SPDX Specification v2.3 Annex D (ABNF: license-ref =
["DocumentRef-"(idstring)":"]"LicenseRef-"(idstring)).
The fix adds a single check after the range dispatch:
if any.pattern:
g.add((range_list[-1], SH.pattern, Literal(any.pattern)))
This correctly handles:
- Pattern-only branches (no range): node gets only sh:pattern
- Range + pattern branches: node gets both sh:datatype and sh:pattern
- Range-only branches (no pattern): unchanged behaviour
The test suite now includes a dedicated schema exercising all
three cases, with assertions on both the generated RDF triples
and pyshacl validation of conforming/non-conforming data.
Signed-off-by: Carlo van Driesten <carlo.van-driesten@bmw.de>
…ogical constraints gen-shacl dropped class-level any_of, all_of, exactly_one_of and none_of without a trace, so a class stating "a code or a name is required" or "one of two complete profiles" generated shapes that accepted everything. The metamodel maps these operators to sh:or, sh:and, sh:xone and sh:not (their exact_mappings), and SHACL Core defines them with the same semantics (SHACL 4.6). They are now emitted on the class's NodeShape: - any_of / all_of / exactly_one_of: sh:or / sh:and / sh:xone over a list of anonymous member shapes - none_of: one sh:not per member; the values of sh:not are separate constraints that all apply (SHACL 2.1.1) - a member's is_a gives sh:class, nested expressions recurse, and each slot condition gives an sh:property on the path of the slot as induced for the class, so slot_usage and attributes resolve as in the slot loop - conditions translate required, value_presence, the cardinalities, minimum/maximum_value, pattern, equals_string(_in) (as the enum renders its permissible values on an enum slot), equals_number (as an inclusive bound on both sides, so 5 matches 5.0) and range - a parameter SHACL allows once per shape (sh:minInclusive, sh:in, sh:pattern, ...) that one condition needs twice keeps its first value and moves the second into an sh:and member, so the shapes graph stays well-formed - a condition holds vacuously for an absent slot unless required, value_presence or a minimum cardinality of at least 1 says otherwise; inside none_of, at any depth, a condition that constrains values requires the slot, so that the negation does not reject absent slots, unless it decides presence itself (required, value_presence, or a maximum or exact cardinality of 0). The JSON Schema generator requires the slot in a class's own none_of for every condition that sets neither required nor value_presence - an operator whose members use anything else (has_member, slot-level boolean expressions inside a condition, a name that is not a slot, the identifier slot, equals_string on a non-string range, ...) is skipped as a whole with a warning, because dropping a member would change what the operator admits - the slot loop's range dispatch and sh:path computation move into _add_range and _slot_path so that slot conditions reuse them; the output for schemas without class expressions is unchanged The compliance tests test_class_any_of and test_class_any_of_with_required now run for SHACL through the validator's SHACL plugin instead of being skipped as incomplete. Rows stay incomplete only where an integer in a string slot is the sole violation: instances reach the shapes through python dataclasses, which coerce the value. Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
…nstraints For an inlined-as-dict slot whose range class has an identifier/key slot, render the key slot's string-applicable constraints onto JSON Schema propertyNames (draft-06+) instead of dropping them. In the inlined-dict form the mapping key is the identifier value, so the key slot's constraints constrain the keys. JSON object keys are always strings, so only pattern, enum (equals_string_in) and a string const (equals_string) are emitted; numeric minimum/maximum, numeric const (equals_number) and allOf are excluded -- a numeric const would otherwise reject every key. structured_pattern is honored when materialize_patterns is enabled, consistent with value patterns. Backward compatible: emitted only when a string-applicable key constraint applies. Signed-off-by: Carlo van Driesten <carlo.van-driesten@bmw.de> Includes the documentation for the emitted propertyNames constraint.
… names Add an opt-in --normalize-prefixes flag to OWL, SHACL, and JSON-LD Context generators that normalises non-standard prefix aliases to well-known names from a static prefix map (derived from rdflib 7.x defaults, cross-checked against prefix.cc consensus). Key design decisions: - Static frozen map (MappingProxyType) instead of runtime Graph().namespaces() lookup eliminates rdflib version dependency - Both http://schema.org/ and https://schema.org/ map to 'schema' - Shared normalize_graph_prefixes() helper used by OWL and SHACL - Two-phase graph normalisation: Phase 1 normalises schema-declared prefixes, Phase 2 cleans up runtime-injected bindings - Collision detection: skip with warning when standard prefix name is already user-declared for a different namespace - Phase 2 guard prevents overwriting HTTPS bindings with HTTP variants The flag defaults to off, preserving existing behaviour. Includes the documentation for the flag in docs/generators/owl.rst.
jdsika
force-pushed
the
feat/envited-x-pipeline
branch
from
October 2, 2026 12:10
3026d81 to
abd2032
Compare
A permissible value can be an instance of a class defined in another schema:
an open vocabulary declares the class, and an extension contributes named
individuals of it. The metamodel states this with `instantiates` on the
permissible value ("an element in another schema which this element
instantiates"), but gen-owl ignored it - its slot URI is linkml:instantiates,
which add_metadata skips - so the individual came out typed only with the
enum's own class. Validators checking `sh:class` against the open vocabulary
then rejected it.
Each `instantiates` value of a permissible value with an IRI now becomes an
rdf:type of that IRI. The enum still closes its own class with owl:oneOf; the
instantiated class stays open. On a permissible value rendered as a literal
there is nothing to type, so `instantiates` is ignored with a warning, as
`meaning` already is.
Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
gen-owl renders every annotation and every string-ranged metadata slot as a literal. For properties whose value is a resource by their own specification that is wrong: an ontology's owl:priorVersion, owl:versionIRI, prov:wasDerivedFrom, dcterms:license or dcterms:conformsTo comes out as "https://..." instead of an IRI, so it can be neither dereferenced nor followed by tools that read the ontology header. There was no way to express these header terms at all. A value of one of those properties is now emitted as an IRI when it is an absolute IRI or a CURIE with a declared prefix; every other value, and every other property, keeps its literal. The properties are the OWL 2 ontology properties and owl:versionIRI, rdfs:seeAlso and rdfs:isDefinedBy, the DCMI terms relating a resource to another resource, and prov:wasDerivedFrom, prov:wasRevisionOf and prov:hadPrimarySource. dcterms:identifier is left out: DCMI gives it the range rdfs:Literal. Refs: linkml#3546 Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
A schema can hold a URI as a link to a resource or as data. A link is an IRI node. A URI reference held as data - a file path such as ./data/file.png, which may be relative - has to stay a literal: as an IRI, JSON-LD resolves it against the document base and it becomes a machine-specific absolute IRI. The way to declare the second is a type derived from string with uri: xsd:anyURI. The OWL generator already decides by type ancestry (is_xsd_anyuri_range): only uri, uriorcurie and the types derived from them are promoted to IRIs by --xsd-anyuri-as-iri. The other three RDF components keyed on the xsd:anyURI datatype instead, so such a type was an IRI in SHACL, in the JSON-LD context (with the flag) and in the RDF dumper, and a literal in OWL: - gen-shacl emits sh:nodeKind sh:Literal and sh:datatype xsd:anyURI for it - gen-jsonld-context keeps "@type": "xsd:anyURI" with --xsd-anyuri-as-iri, through the same predicate, for single ranges and any_of branches - the RDF dumper emits a typed literal for it, and resolves the type with inheritance, so a type derived from uri without a datatype of its own is an IRI node instead of a plain literal uri, uriorcurie and their derived types behave exactly as before. Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
…inlined-as-node A class-ranged value is either inlined - its content is part of the instance - or a reference to an object described elsewhere. LinkML reads an inlined value as an instance of the slot's range class whether or not it states a type, and the JSON Schema generator validates it structurally through $ref. gen-shacl always emitted sh:class <class_uri> in class-URI naming mode, so an inlined value was only validated once it carried that rdf:type, and an untyped one was rejected for its type instead of being checked for its content. --inlined-as-node (default off) emits sh:node <range class shape> for inlined values, and keeps sh:class for references, whose type is all that is visible from the referring shape. Inlined is decided by SchemaView.is_inlined: the slot declares inlined or inlined_as_list, directly or through its ancestors, or the range class has no identifier. Each any_of member is decided on its own range, and the range of a class-expression slot condition for the slot it names; the is_a of a class expression stays sh:class, as it tests a type. The referenced shape is named after the class_uri plus --suffix, as emitted. A value reached through sh:node is validated by its range class's shape alone, without the targets of the ancestors' shapes. The metamodel states that a class's rules and expressions apply to all its members, so under the option each class shape also requires its is_a parent's and mixins' shapes. Native names mode already emits sh:node and is unaffected. Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
has_member states that a slot has at least one value satisfying an expression. gen-shacl ignored it without a warning, so "a manifest has at least one artifact of category data" generated a shape that accepted every manifest (linkml#2465). It now becomes sh:qualifiedValueShape with sh:qualifiedMinCount 1 on the slot's property shape, the SHACL component with exactly that meaning. The member shape takes the expression's range, numeric bounds, pattern and equals_string / equals_string_in / equals_number, translated as in a slot condition, and its range_expression as a nested class expression over the slots of the value's class; a member's class range is decided with --inlined-as-node as the slot's values are. As in the JSON Schema generator's `contains`, a condition inside the expression constrains the values that are present; required: true is what makes a member need the slot. has_member is accepted on a slot and inside a class-level slot condition. In the latter each member of the operator is its own shape, so all_of can require several different members of one slot. A has_member using anything else is not emitted and is logged as not translated to SHACL, as an untranslatable class expression is. Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
7 tasks done
DCMI Metadata Terms give dcterms:creator, dcterms:contributor, dcterms:publisher and dcterms:rightsHolder the range dcterms:Agent, so their value is a resource, like the properties gen-owl already emits as IRIs. As an annotation they still came out as literals: an ontology header could name its publisher only as "https://..." text, and an ORCID or organisation IRI given as dcterms:creator was not followable. They join the IRI-valued metadata properties: a value that is an absolute IRI or a CURIE with a declared prefix is emitted as an IRI; an agent named in plain text keeps its literal. Refs: linkml#3546 Signed-off-by: jdsika <carlo.van-driesten@vdl.digital>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Feature integration branch. This branch is consumed directly by downstream
ENVITED-X projects, so it carries the features developed in this fork, applied
on top of current upstream
main. It is not an upstream submission — eachfeature is submitted individually (see the table below).
Based on upstream
mainat62d30bf16. Head:e3081ef8b.Contents
Twenty commits, in dependency order.
diffable-rdf0.4.0docs(owl): document deterministic serialization and--diff-stablesh:patternfor pattern constraints insideany_ofany_of/all_of/exactly_one_of/none_of→sh:or/sh:and/sh:xone/sh:notpropertyNamesfrom inlined-dict key constraints--normalize-prefixesfor well-known prefix namesgen-owl: a permissible value'sinstantiatesbecomes anrdf:typeof its IRIgen-owl: IRI-valued metadata (owl:versionIRI,owl:priorVersion,prov:wasDerivedFrom,dcterms:license, …) emitted as IRIsxsd:anyURItypes stay typed literals ingen-shacl,gen-jsonld-contextand the RDF dumpergen-shacl --inlined-as-node: inlined values validated by their range class's shape, class shapes require their parents' shapesgen-shacl:has_member→sh:qualifiedValueShape+sh:qualifiedMinCount 1gen-owl: the DCMI agent terms (dcterms:creator,contributor,publisher,rightsHolder) join the IRI-valued metadataRows 8–13 are cherry-picks of their PR commits; their source changes are identical to the PRs (compared line by line, context aside). The one textual conflict,
docs/generators/owl.rstandtest_owlgen.pybetween rows 8 and 9, was two independent insertions at the same place and keeps both.Not on this branch
diffable-rdf).Already merged upstream — dropped from this branch
feat(gen-shacl): generate sh:sparql constraints from LinkML rules→ feat(gen-shacl): generate sh:sparql constraints from LinkML rules linkml/linkml#3451fix(shaclgen): emit sh:minCount/maxCount 0 for zero cardinality values→ fix(shaclgen): emit sh:minCount/maxCount 0 for zero cardinality values linkml/linkml#3473feat(jsonschemagen): add --include-null/--no-include-null→ mergedQuality gates
Every commit is signed and carries its original authorship (the two SHACL rule
features remain authored by Rayene Messaoud).
On the head, run as CI does:
pytest tests/linkml/ --ignore=tests/linkml/test_notebooks -m "not kroki" -n 8— 12514 passed,2029 skipped, 12 xfailed, 0 failed
pytest tests/linkml_runtime -m "not kroki"— 1838 passed, 323 skipped, 2 xfailedruff checkandruff format --check(ruff 0.11.13) clean on every file rows 8–13 changeKnown caveat
The
--normalize-prefixescommit pinsprefixmapsto a git revision, becausethe fix it needs (linkml/prefixmaps#82) is merged but unreleased — PyPI is
still on
0.2.6. This is fine for consumers installing from git, but it breaksthe Docker image build, since
pip installof the built wheel has to fetch thedependency from GitHub and the base image has no
git.