diff --git a/docs/generators/json-schema.rst b/docs/generators/json-schema.rst index 2f8f1d8d91..bfc23e6558 100644 --- a/docs/generators/json-schema.rst +++ b/docs/generators/json-schema.rst @@ -130,6 +130,13 @@ LinkML supports analogous elements: Use of these elements will be translated into the appropriate JSON-Schema construct. +At the class level, a class carries the expressions of its ancestors and mixins. +A slot condition is unknown for an absent slot unless it decides whether the slot +may be absent, and an instance is invalid only when an expression is definitely +false, as described under "Class-level expressions and absent slots" in +:doc:`Advanced features `. +The SHACL generator reads them the same way. + Inlining ^^^^^^^^ diff --git a/docs/generators/shacl.rst b/docs/generators/shacl.rst index 3e88f0090f..93ef1dd9af 100644 --- a/docs/generators/shacl.rst +++ b/docs/generators/shacl.rst @@ -84,6 +84,92 @@ Example Output: shacl:targetClass . +Class Expressions +^^^^^^^^^^^^^^^^^ + +Class-level boolean expressions become the SHACL logical constraint components +their metamodel definitions map to (`SHACL §4.6 +`__): + +================== ===================================================== +LinkML SHACL, on the class's ``sh:NodeShape`` +================== ===================================================== +``any_of`` ``sh:or`` over the member shapes +``all_of`` ``sh:and`` over the member shapes +``exactly_one_of`` ``sh:xone`` over the member shapes +``none_of`` one ``sh:not`` per member +================== ===================================================== + +Each member becomes an anonymous node shape. ``is_a`` gives ``sh:class``, and +nested expressions recurse. Each entry of ``slot_conditions`` gives an +``sh:property`` whose path is that of the slot as induced for the class, so +``slot_usage`` applies: + +* ``required``, ``value_presence`` and the cardinalities give ``sh:minCount`` / + ``sh:maxCount``; +* ``minimum_value`` / ``maximum_value`` give ``sh:minInclusive`` / + ``sh:maxInclusive``, and ``equals_number`` gives both, so that ``5`` also + matches ``5.0``; +* ``pattern`` gives ``sh:pattern``; +* ``equals_string`` and ``equals_string_in`` give ``sh:in``; on an enum slot the + values are the permissible values as the enum renders them, the IRI of their + ``meaning`` where they have one; +* ``range`` gives the same class, type or enum constraint as a slot's range. + +A shape may have at most one value of ``sh:minInclusive``, ``sh:maxInclusive`` +or ``sh:in``, and of ``sh:pattern``, whose component also takes ``sh:flags`` +(`SHACL §2.1.1 `__). Where one condition +needs one of them twice, for example ``minimum_value`` next to +``equals_number``, the second value goes into an ``sh:and`` member of the +property shape, where it applies to the same values. + +A slot condition is unknown for an absent slot unless it decides whether the slot +may be absent, and an instance violates an expression only when the expression is +definitely false, as described under "Class-level expressions and absent slots" in +:doc:`Advanced features `. +The JSON Schema generator reads them the same way. In SHACL, each expression gets +its "not false" form, and, under ``sh:not``, its "definitely true" form, where +such a condition also requires its slot (``sh:minCount 1``). ``exactly_one_of`` +becomes ``sh:xone`` over the "definitely true" forms of its members. + +.. code-block:: yaml + + GeodeticReferenceSystem: + slots: [code, name] + any_of: + - slot_conditions: + code: + required: true + - slot_conditions: + name: + required: true + +.. code-block:: turtle + + ex:GeodeticReferenceSystem a sh:NodeShape ; + sh:or ( [ sh:property [ sh:path ex:code ; sh:minCount 1 ] ] + [ sh:property [ sh:path ex:name ; sh:minCount 1 ] ] ) ; + ... + +A class expression constrains every instance of its class, so the shape of a +class carries the expressions of its ancestors and mixins as well, as it carries +their slots. Each is translated in the context of that class, where +``slot_usage`` applies. A node typed only with a subclass, as ``rdflib_dumper`` +and the JSON-LD context produce it, is therefore checked without an +``rdfs:subClassOf`` triple in the data. The ``sh:class`` that ``is_a`` gives +recognises instances of subclasses only where the data graph states the +``rdfs:subClassOf`` (`SHACL §4.1.1 +`__), as it does for a +slot's range. + +An operator whose members use anything else is skipped as a whole and logged as +a warning, because leaving out one member would change what the operator +admits. That covers, for example, ``has_member`` or a slot-level ``any_of`` +inside a slot condition, a condition on a name that is not a slot, a condition +on the identifier slot (the node's IRI rather than a property), and +``equals_string`` on a slot whose range does not hold strings. + + Command Line ^^^^^^^^^^^^ diff --git a/docs/schemas/advanced.md b/docs/schemas/advanced.md index c56a808379..2f173f2443 100644 --- a/docs/schemas/advanced.md +++ b/docs/schemas/advanced.md @@ -64,6 +64,36 @@ The following LinkML constructs can be used to express boolean constraints: These can be applied at the class or slot level. The range of each of these is an array of *expressions*. +### Class-level expressions and absent slots + +At the class level, each member of `any_of`, `all_of`, `exactly_one_of` and `none_of` is a class expression whose `slot_conditions` constrain the instance's slots. The expressions of a class also constrain the instances of its subclasses, and of the classes that use it as a mixin. + +A slot condition needs a meaning when its slot is absent. The JSON Schema and SHACL generators read it as an SQL `CHECK` constraint reads a condition on a null value: an instance is invalid only when an expression is definitely false. + +- A condition that decides whether its slot may be absent is true or false as usual. Such a condition sets `value_presence: PRESENT` or `ABSENT`, `required: true`, a minimum or exact cardinality of at least 1, or a maximum or exact cardinality of 0. +- Any other condition is *unknown* when its slot is absent. +- `any_of`, `all_of` and `none_of` combine these as "or", "and" and "not". An unknown member doesn't make `any_of` true, nor `none_of` false. +- `exactly_one_of` holds when exactly one member is definitely true. + +```yaml +classes: + Sample: + none_of: + - slot_conditions: + status: + equals_string: retracted +``` + +A `Sample` without `status` is valid: the condition is unknown, so `none_of` isn't false. To require the slot, state it with `status: {required: true}`. + +| Expression | `{}` | `{label: A}` | `{label: B}` | +|---|---|---|---| +| `any_of: [label = A]` | valid | valid | invalid | +| `none_of: [label = A]` | valid | invalid | valid | +| `exactly_one_of: [label = A, note = B]` | invalid | valid | invalid | + +[Rules](#rules) are read differently: their preconditions require their slots, and so do their postconditions unless the rule is `open_world`. + ### Unions as ranges [any_of](https://w3id.org/linkml/any_of) can be used to express that a range must satisfy any of a set of ranges. diff --git a/packages/linkml/src/linkml/generators/common/class_expression.py b/packages/linkml/src/linkml/generators/common/class_expression.py new file mode 100644 index 0000000000..21ff99af4f --- /dev/null +++ b/packages/linkml/src/linkml/generators/common/class_expression.py @@ -0,0 +1,96 @@ +"""Shared semantics of slot conditions in class-level boolean expressions. + +The class-level operators ``any_of``, ``all_of``, ``exactly_one_of`` and +``none_of`` combine class expressions whose slot conditions constrain slots of +an instance. The specification does not say what a condition means for an +absent slot. Generators that translate class expressions read it the same +way, as an SQL CHECK constraint (ISO/IEC 9075) reads a condition on a null +value: a condition that doesn't state presence is *unknown* for an absent +slot, and an expression is violated only when it is definitely false. + +Each expression therefore has two forms: + +* its **"not false" form**, which the instance must satisfy, and in which a + condition holds for an absent slot unless it states presence; +* its **"definitely true" form**, used under a negation, in which a condition + that doesn't state presence also requires its slot. + +``none_of`` takes its members in the opposite form, ``exactly_one_of`` counts +the members that are definitely true, and ``any_of`` / ``all_of`` keep the +form. + +>>> from linkml_runtime.linkml_model.meta import SlotDefinition +>>> value_bounds(SlotDefinition("label", equals_string="A"), definite=False) +(0, None) +>>> value_bounds(SlotDefinition("label", equals_string="A"), definite=True) +(1, None) +>>> value_bounds(SlotDefinition("label", required=True, value_presence="ABSENT"), definite=True) +(0, 0) +>>> value_bounds(SlotDefinition("tags", minimum_cardinality=2, maximum_cardinality=3), definite=False) +(2, 3) +""" + +from linkml_runtime.linkml_model.meta import PresenceEnum, SlotDefinition + +_PRESENT = PresenceEnum(PresenceEnum.PRESENT) +_ABSENT = PresenceEnum(PresenceEnum.ABSENT) + + +def states_presence(condition: SlotDefinition) -> bool: + """Whether *condition* decides whether its slot may be absent. + + Such a condition is definitely true or false for an absent slot: + ``value_presence: PRESENT`` or ``ABSENT``; ``required: true``, unless + ``value_presence`` overrides it; a minimum or exact cardinality of at least + 1, which an absent slot fails; and a maximum or exact cardinality of 0, + which it satisfies. Other bounds, ``required: false`` and ``UNCOMMITTED`` + leave absence open, so adding one never changes the verdict on an absent + slot. + + >>> states_presence(SlotDefinition("label", equals_string="A")) + False + >>> states_presence(SlotDefinition("label", required=False, equals_string="A")) + False + >>> states_presence(SlotDefinition("tags", maximum_cardinality=5)) + False + >>> states_presence(SlotDefinition("tags", maximum_cardinality=0)) + True + >>> states_presence(SlotDefinition("tags", minimum_cardinality=1)) + True + """ + if condition.value_presence is not None: + if condition.value_presence in (_PRESENT, _ABSENT): + return True + elif condition.required: + return True + lower = (condition.minimum_cardinality, condition.exact_cardinality) + upper = (condition.maximum_cardinality, condition.exact_cardinality) + return any(bound is not None and bound >= 1 for bound in lower) or 0 in upper + + +def value_bounds(condition: SlotDefinition, definite: bool) -> tuple[int, int | None]: + """The least and the greatest number of values *condition* allows its slot. + + ``value_presence`` takes precedence over ``required``. In the "definitely + true" form (*definite*), a condition that doesn't state presence also + requires the slot. The greatest number is ``None`` when unbounded. + """ + lower, upper = [0], [] + if condition.value_presence is not None: + if condition.value_presence == _PRESENT: + lower.append(1) + elif condition.value_presence == _ABSENT: + upper.append(0) + elif condition.required: + lower.append(1) + if definite and not states_presence(condition): + lower.append(1) + for bound, target in ( + (condition.minimum_cardinality, lower), + (condition.exact_cardinality, lower), + (condition.maximum_cardinality, upper), + (condition.exact_cardinality, upper), + ): + if bound is not None: + target.append(int(bound)) + return max(lower), min(upper) if upper else None diff --git a/packages/linkml/src/linkml/generators/jsonschemagen.py b/packages/linkml/src/linkml/generators/jsonschemagen.py index 586b21dc4b..e09780817f 100644 --- a/packages/linkml/src/linkml/generators/jsonschemagen.py +++ b/packages/linkml/src/linkml/generators/jsonschemagen.py @@ -14,6 +14,7 @@ from linkml.generators.common import build from linkml.generators.common.array import ArrayRangeGenerator, ArrayRepresentation from linkml.generators.common.build import RangeResult +from linkml.generators.common.class_expression import value_bounds from linkml.generators.common.lifecycle import LifecycleMixin from linkml.generators.common.subproperty import get_subproperty_values from linkml.generators.common.type_designators import ( @@ -275,24 +276,12 @@ def add_property( self["required"].append(canonical_name) # JSON Schema does not have a very natural way to express that a property cannot be present. - # The apparent best way to do it is to use: - # { - # properties: { - # foo: ... - # }, - # not: { - # required: ['foo'] - # } - # } - # The {required: [foo]} subschema evaluates to true if the foo property is present with any - # value. Wrapping that in a `not` keyword inverts that condition. + # The apparent best way to do it is `not: {required: [foo]}`: the {required: [foo]} + # subschema evaluates to true if the foo property is present with any value, and `not` + # inverts that. Each absent property gets its own `not`, in `allOf`, since + # `not: {required: [foo, bar]}` would only forbid foo and bar together. if value_disallowed: - if "not" not in self: - self["not"] = {} - if "required" not in self["not"]: - self["not"]["required"] = [] - - self["not"]["required"].append(canonical_name) + self.setdefault("allOf", []).append({"not": {"required": [canonical_name]}}) def add_keyword(self, keyword: str, value: Any): if value is None: @@ -623,29 +612,20 @@ def handle_class(self, cls: ClassDefinition) -> None: class_subschema["allOf"] = [] class_subschema["allOf"].extend(rule_subschemas) - if cls.any_of is not None and len(cls.any_of) > 0: - class_subschema["anyOf"] = [self.get_subschema_for_anonymous_class(c, False) for c in cls.any_of] - - if cls.all_of is not None and len(cls.all_of) > 0: - if "allOf" not in class_subschema: - class_subschema["allOf"] = [] - class_subschema["allOf"].extend([self.get_subschema_for_anonymous_class(c, False) for c in cls.all_of]) - - if cls.exactly_one_of is not None and len(cls.exactly_one_of) > 0: - class_subschema["oneOf"] = [self.get_subschema_for_anonymous_class(c, False) for c in cls.exactly_one_of] - - if cls.none_of is not None and len(cls.none_of) > 0: - # properties_required=True so absent slots make their branch fail; otherwise - # properties is vacuously true and `not(anyOf)` rejects instances missing the slot. - new_not = {"anyOf": [self.get_subschema_for_anonymous_class(c, True) for c in cls.none_of]} - if "not" in class_subschema: - existing_not = class_subschema.pop("not") - if "allOf" not in class_subschema: - class_subschema["allOf"] = [] - class_subschema["allOf"].append({"not": existing_not}) - class_subschema["allOf"].append({"not": new_not}) - else: - class_subschema["not"] = new_not + # A class expression constrains every instance of its class, so a class carries the + # expressions of its ancestors and mixins, translated in its own context. + for owner in self.schemaview.class_ancestors(cls.name): + for operator in self.CLASS_EXPRESSION_OPERATORS: + members = getattr(self.schemaview.get_class(owner), operator) or [] + if members: + expression = self.get_subschema_for_class_operator(cls, operator, members, definite=False) + for keyword, value in expression.items(): + if keyword == "allOf": + class_subschema.setdefault("allOf", []).extend(value) + elif keyword not in class_subschema: + class_subschema[keyword] = value + else: + class_subschema.setdefault("allOf", []).append({keyword: value}) class_subschema = self.after_generate_class( ClassResult.model_construct(schema_=class_subschema, source=cls), self.schemaview @@ -671,6 +651,93 @@ def handle_class(self, cls: ClassDefinition) -> None: if key not in self.top_level_schema: self.top_level_schema[key] = value + CLASS_EXPRESSION_OPERATORS = ("any_of", "all_of", "exactly_one_of", "none_of") + _CLASS_OPERATOR_KEYWORDS = {"any_of": "anyOf", "all_of": "allOf", "exactly_one_of": "oneOf"} + + def get_subschema_for_class_operator( + self, cls: ClassDefinition, operator: str, members: list[AnonymousClassExpression], definite: bool + ) -> JsonSchema: + """The subschema of the class-level boolean expression *operator* over *members*, for *cls*. + + A condition that doesn't state presence is unknown for an absent slot, + and an expression is violated only when it is definitely false (see + :mod:`linkml.generators.common.class_expression`). With *definite* the + subschema accepts the instances for which the expression is definitely + true, otherwise those for which it is not false. ``none_of`` takes its + members in the opposite form, ``exactly_one_of`` counts the members + that are definitely true, and ``any_of`` / ``all_of`` keep the form. + """ + if operator == "none_of": + return JsonSchema( + {"not": {"anyOf": [self.get_subschema_for_class_expression(cls, m, not definite) for m in members]}} + ) + form = True if operator == "exactly_one_of" else definite + return JsonSchema( + { + self._CLASS_OPERATOR_KEYWORDS[operator]: [ + self.get_subschema_for_class_expression(cls, m, form) for m in members + ] + } + ) + + def get_subschema_for_class_expression( + self, cls: ClassDefinition, expr: AnonymousClassExpression, definite: bool + ) -> JsonSchema: + """The subschema of one member *expr* of a class-level boolean expression of *cls*. + + Each slot condition constrains the slot as induced for *cls*, so + ``slot_usage`` applies. Its values must satisfy its value operators + and its ``range``; the number of values must lie within + :func:`~linkml.generators.common.class_expression.value_bounds`, where + an empty list counts as no value. *definite* selects the form, as in + :meth:`get_subschema_for_class_operator`. + """ + subschema = JsonSchema() + conjuncts: list[JsonSchema] = [] + for slot_name, condition in expr.slot_conditions.items(): + slot = self._class_expression_slot(cls, slot_name) or condition + if self.use_curies: + prop_name = self._curie(slot) + else: + prop_name = self.aliased_slot_name(slot) + values = self.get_subschema_for_slot(condition, omit_type=True, include_null=False) + if condition.range is not None: + typed = self.get_subschema_for_slot( + SlotDefinition( + slot.name, range=condition.range, inlined=slot.inlined, inlined_as_list=slot.inlined_as_list + ), + include_null=False, + ) + values = JsonSchema({"allOf": [values, typed]}) if values else typed + lower, upper = value_bounds(condition, definite) + if slot.multivalued: + prop = JsonSchema.array_of(values, include_null=False, required=False) + prop.add_keyword("minItems", lower or None) + prop.add_keyword("maxItems", upper) + subschema.add_property(prop_name, prop, value_required=lower > 0) + else: + subschema.add_property(prop_name, values, value_required=lower > 0, value_disallowed=upper == 0) + if lower > 1: + # a single-valued slot holds at most one value + conjuncts.append(JsonSchema({"not": {}})) + for operator in self.CLASS_EXPRESSION_OPERATORS: + members = getattr(expr, operator) or [] + if members: + conjuncts.append(self.get_subschema_for_class_operator(cls, operator, members, definite)) + if expr.is_a is not None: + # `is_a: ` in a class expression requires instances of the expression to be instances of . + conjuncts.append(self.get_subschema_for_slot(AnonymousSlotExpression(range=expr.is_a))) + if conjuncts: + subschema.setdefault("allOf", []).extend(conjuncts) + return subschema + + def _class_expression_slot(self, cls: ClassDefinition, slot_name: str) -> SlotDefinition | None: + """The slot a condition of a class expression of *cls* names, as induced for *cls*, if any.""" + sv = self.schemaview + if slot_name in sv.class_slots(cls.name): + return sv.induced_slot(slot_name, cls.name) + return sv.get_slot(slot_name) + def get_subschema_for_anonymous_class( self, cls: AnonymousClassExpression, properties_required: bool = False ) -> None | JsonSchema: diff --git a/packages/linkml/src/linkml/generators/shaclgen.py b/packages/linkml/src/linkml/generators/shaclgen.py index 4731b9f0b8..8b61fa3fe6 100644 --- a/packages/linkml/src/linkml/generators/shaclgen.py +++ b/packages/linkml/src/linkml/generators/shaclgen.py @@ -2,7 +2,7 @@ import os import string from collections.abc import Callable -from dataclasses import dataclass +from dataclasses import dataclass, fields import click from jsonasobj2 import JsonObj, as_dict @@ -11,12 +11,23 @@ from rdflib.namespace import RDF, RDFS, SH, XSD from linkml._version import __version__ +from linkml.generators.common.class_expression import value_bounds from linkml.generators.common.subproperty import get_subproperty_values, is_uri_range from linkml.generators.shacl.shacl_data_type import ShaclDataType from linkml.generators.shacl.shacl_ifabsent_processor import ShaclIfAbsentProcessor from linkml.utils.generator import Generator, shared_arguments from linkml.utils.language_tags import LanguageTagResolver -from linkml_runtime.linkml_model.meta import ClassDefinition, ElementName, PresenceEnum +from linkml_runtime.linkml_model.meta import ( + AnonymousClassExpression, + AnonymousSlotExpression, + ClassDefinition, + ClassExpression, + Element, + ElementName, + PresenceEnum, + SlotDefinition, + SlotExpression, +) from linkml_runtime.utils.formatutils import underscore from linkml_runtime.utils.rdf_canonicalize import canonicalize_rdf_graph from linkml_runtime.utils.yamlutils import TypedNode, extended_float, extended_int, extended_str @@ -208,6 +219,8 @@ def as_graph(self) -> Graph: for pfx in self.schema.prefixes.values(): g.bind(str(pfx.prefix_prefix), pfx.prefix_reference) + self._class_expressions_added: set[tuple[URIRef, str, str]] = set() + self._class_expression_problems: dict[tuple[str, str, str], list[str]] = {} for c in sv.all_classes(imports=not self.exclude_imports).values(): def shape_pv(p, v): @@ -248,12 +261,7 @@ def shape_pv(p, v): self._add_annotations(shape_pv, c) order = 0 for s in sv.class_induced_slots(c.name): - # fixed in linkml-runtime 1.1.3 - if s.name in sv.element_by_schema_map(): - slot_uri = URIRef(sv.get_uri(s, expand=True)) - else: - pfx = sv.schema.default_prefix - slot_uri = URIRef(sv.expand_curie(f"{pfx}:{underscore(s.name)}")) + slot_uri = URIRef(self._slot_iri(s)) pnode = BNode() shape_pv(SH.property, pnode) @@ -371,21 +379,7 @@ def st_node_pv(p, v): f" require range 'string' and not '{r}'" ) - if r in all_classes: - cls_def = sv.get_class(r) - is_any = cls_def and getattr(cls_def, "class_uri", None) == "linkml:Any" - self._add_class(prop_pv, r) - if not is_any: - if sv.get_identifier_slot(r) is not None: - prop_pv(SH.nodeKind, SH.IRI) - else: - prop_pv(SH.nodeKind, SH.BlankNodeOrIRI) - elif r in sv.all_types(): - self._add_type(prop_pv, r) - elif r in sv.all_enums(): - self._add_enum(g, prop_pv, r) - else: - add_simple_data_type(prop_pv, r) + self._add_range(g, prop_pv, r) if s.pattern: prop_pv(SH.pattern, Literal(s.pattern)) if s.equals_string: @@ -405,13 +399,362 @@ def st_node_pv(p, v): if default_value: prop_pv(SH.defaultValue, default_value) + self._add_class_expressions(g, class_uri_with_suffix, c) + if self.emit_rules: self._add_rules(g, class_uri_with_suffix, c) + self._report_class_expression_problems() return g LINKML_ANY_URI = "https://w3id.org/linkml/Any" + # ------------------------------------------------------------------- + # Class expressions → sh:or / sh:and / sh:xone / sh:not + # ------------------------------------------------------------------- + + # The SHACL logical constraint component, taking a list, of each list operator. + _LIST_OPERATORS = {"any_of": SH["or"], "all_of": SH["and"], "exactly_one_of": SH.xone} + _CLASS_EXPRESSION_OPERATORS = ("any_of", "all_of", "exactly_one_of", "none_of") + + # The fields of an anonymous class expression that carry meaning, derived from + # the metamodel, so that a new semantic field is reported as untranslatable + # instead of being ignored. Every other field is metadata. + _CLASS_EXPRESSION_FIELDS = frozenset(f.name for f in fields(ClassExpression)) | {"is_a"} + + # Slot-condition fields translated by _slot_condition_shape: those deciding + # whether the slot is present, and those constraining its values. + _SLOT_CONDITION_PRESENCE_FIELDS = frozenset( + {"required", "value_presence", "minimum_cardinality", "maximum_cardinality", "exact_cardinality"} + ) + _SLOT_CONDITION_VALUE_FIELDS = frozenset( + {"minimum_value", "maximum_value", "pattern", "equals_string", "equals_string_in", "equals_number", "range"} + ) + _SLOT_CONDITION_FIELDS = _SLOT_CONDITION_PRESENCE_FIELDS | _SLOT_CONDITION_VALUE_FIELDS + + # Parameters a shape may have at most one value of: sh:minInclusive and + # sh:maxInclusive (SHACL §4.3), sh:in (§4.8.3), sh:datatype and sh:nodeKind + # (§4.1), and sh:pattern, as a parameter of a component with more than one + # parameter (§2.1.1). A condition that needs one of them twice gets the + # second value in an sh:and member. + _SINGLE_VALUE_PARAMETERS = frozenset( + {SH.minInclusive, SH.maxInclusive, SH["in"], SH.pattern, SH.datatype, SH.nodeKind} + ) + + # Fields on a slot condition / class expression that carry no constraint + # semantics: they never change which instances satisfy the condition, so + # they are ignored by the operator accounting below. Anything set on a + # condition that is neither here nor explicitly translated by a converter + # makes the rule untranslatable — the converters must SKIP such a rule + # rather than emit a query that silently drops a conjunct (which would + # widen the trigger or narrow the check: a mis-translation, not a skip). + # Derived from the metamodel: the metadata every ``element`` carries, minus + # anything that is a ``slot_expression`` operator. + _NON_OPERATOR_FIELDS = frozenset(f.name for f in fields(Element)) - frozenset( + f.name for f in fields(SlotExpression) + ) + + @classmethod + def _set_operator_fields( + cls, condition: SlotDefinition | AnonymousSlotExpression | AnonymousClassExpression + ) -> set[str]: + """Return the names of the constraint-bearing fields actually set on a + rule condition or class expression. + + A field counts as *set* when it is not ``None`` and not an empty + collection (SchemaView materialises unset multivalued fields as empty + lists / dicts). Scalars are never judged by truthiness, so legitimate + falsy constraints such as ``minimum_value: 0`` or + ``equals_string: ""`` still count as set. Metadata fields + (:data:`_NON_OPERATOR_FIELDS`) are excluded. + + The converters compare this set against the exact operator set they + translate and skip the rule on any mismatch, so an unrecognised (or + future-metamodel) operator can never be silently dropped. + """ + return { + name + for name, value in vars(condition).items() + if not name.startswith("_") + and name not in cls._NON_OPERATOR_FIELDS + and value is not None + and not (isinstance(value, list | dict) and not value) + } + + def _add_class_expressions(self, g: Graph, shape_uri: URIRef, cls: ClassDefinition) -> None: + """Emit the class-level boolean expressions of *cls* as SHACL logical constraints. + + Each operator is mapped to the SHACL logical constraint component with the + same semantics (`SHACL §4.6 `_): + + * ``any_of`` → ``sh:or``, ``all_of`` → ``sh:and``, ``exactly_one_of`` → + ``sh:xone``, each over a list of the member shapes; + * ``none_of`` → one ``sh:not`` per member. A shape's values of ``sh:not`` + are separate constraints that all apply (SHACL §2.1.1), so the node must + conform to none of the members. + + Every member becomes an anonymous node shape: ``is_a`` gives ``sh:class``, + each slot condition gives an ``sh:property`` on the path of the slot as + induced for *cls* (so ``slot_usage`` applies), and nested expressions + recurse. + + The specification doesn't say what a slot condition means for an absent + slot. As with an SQL CHECK constraint (ISO/IEC 9075), which is + satisfied unless its condition is false, a class expression is violated + only when it is definitely false, and a condition that doesn't state + presence (:func:`~linkml.generators.common.class_expression.states_presence`) is unknown + for an absent slot. + Each operator is therefore translated in its "not false" form, and, + under a negation, in its "definitely true" form, where such a condition + requires its slot. ``exactly_one_of`` holds when exactly one member is + definitely true. + + A class expression constrains every instance of its class, so the shape + of *cls* carries the expressions of its ancestors and mixins as well, as + it carries their slots. Each is translated in the context of *cls*, + where ``slot_usage`` may refine a slot it names. Classes that share a + ``class_uri`` share one shape, which carries each expression once. + + An operator whose members use anything that cannot be translated is + skipped as a whole, with a warning: dropping one member would change what + the operator admits. + """ + sv = self.schemaview + for owner in sv.class_ancestors(cls.name): + for operator in self._CLASS_EXPRESSION_OPERATORS: + members = getattr(sv.get_class(owner), operator, None) or [] + if not members or (shape_uri, owner, operator) in self._class_expressions_added: + continue + self._class_expressions_added.add((shape_uri, owner, operator)) + reason = next(filter(None, (self._untranslatable(cls, m) for m in members)), None) + if reason is not None: + classes = self._class_expression_problems.setdefault((owner, operator, reason), []) + if cls.name not in classes: + classes.append(cls.name) + continue + self._add_logical_constraint(g, shape_uri, cls, operator, members, definite=False) + + def _report_class_expression_problems(self) -> None: + """Warn once about each class expression that is not translated, naming the + class shapes it is missing from unless that is only the declaring class.""" + for (owner, operator, reason), classes in self._class_expression_problems.items(): + shapes = "" if classes == [owner] else f" (in the shapes of {', '.join(map(repr, classes))})" + logger.warning( + "Class %r: %s is not translated to SHACL, because it uses %s%s.", owner, operator, reason, shapes + ) + + def _add_logical_constraint( + self, + g: Graph, + subject: URIRef | BNode, + cls: ClassDefinition, + operator: str, + members: list[AnonymousClassExpression], + definite: bool, + ) -> None: + """Add the logical constraint for *operator* over *members* to *subject*. + + With *definite* the constraint holds when the expression is definitely + true, otherwise when it is not false. ``none_of`` gives one ``sh:not`` + per member, in the opposite form, since an expression is not false + exactly when its negation is not definitely true. ``exactly_one_of`` + counts the members that are definitely true, in either form. The + other list operators keep the form. + """ + if operator == "none_of": + for member in members: + g.add((subject, SH["not"], self._class_expression_shape(g, cls, member, not definite))) + return + predicate = self._LIST_OPERATORS[operator] + member_form = True if operator == "exactly_one_of" else definite + shapes = [self._class_expression_shape(g, cls, m, member_form) for m in members] + list_node = BNode() + Collection(g, list_node, shapes) + g.add((subject, predicate, list_node)) + + def _class_expression_shape( + self, g: Graph, cls: ClassDefinition, expr: AnonymousClassExpression, definite: bool + ) -> BNode: + """Build the anonymous node shape for one class expression *expr*, in its + "definitely true" form with *definite*, otherwise its "not false" form.""" + node = BNode() + + def node_pv(p, v): + if v is not None: + g.add((node, p, v)) + + if expr.title is not None: + node_pv(RDFS.label, Literal(expr.title, lang=self._resolve_language(expr))) + if expr.description is not None: + node_pv(RDFS.comment, Literal(expr.description, lang=self._resolve_language(expr))) + if expr.is_a is not None: + self._add_class(node_pv, expr.is_a) + for slot_name, condition in expr.slot_conditions.items(): + node_pv(SH.property, self._slot_condition_shape(g, cls, slot_name, condition, definite)) + for operator in self._CLASS_EXPRESSION_OPERATORS: + members = getattr(expr, operator) or [] + if members: + self._add_logical_constraint(g, node, cls, operator, members, definite) + return node + + def _slot_condition_shape( + self, g: Graph, cls: ClassDefinition, slot_name: str, condition: SlotDefinition, definite: bool + ) -> BNode: + """Build the property shape for the condition on *slot_name*, in its + "definitely true" form with *definite*, otherwise its "not false" form.""" + slot = self._condition_slot(cls, slot_name) + pnode = BNode() + repeated = [] + + def prop_pv(p, v): + if v is None: + return + if p in self._SINGLE_VALUE_PARAMETERS and (pnode, p, None) in g: + repeated.append((p, v)) + else: + g.add((pnode, p, v)) + + prop_pv(SH.path, URIRef(self._slot_iri(slot))) + if condition.title is not None: + prop_pv(SH.name, Literal(condition.title, lang=self._resolve_language(condition))) + if condition.description is not None: + prop_pv(SH.description, Literal(condition.description, lang=self._resolve_language(condition))) + + lower, upper = value_bounds(condition, definite) + if lower: + prop_pv(SH.minCount, Literal(lower)) + if upper is not None: + prop_pv(SH.maxCount, Literal(upper)) + + if condition.minimum_value is not None: + prop_pv(SH.minInclusive, Literal(condition.minimum_value)) + if condition.maximum_value is not None: + prop_pv(SH.maxInclusive, Literal(condition.maximum_value)) + if condition.pattern is not None: + prop_pv(SH.pattern, Literal(condition.pattern)) + value_range = condition.range or slot.range + for values in ( + [condition.equals_string] if condition.equals_string is not None else [], + condition.equals_string_in, + ): + if values: + in_node = BNode() + Collection(g, in_node, self._string_value_terms(value_range, values)) + prop_pv(SH["in"], in_node) + if condition.equals_number is not None: + # A value comparison, unlike the slot loop's sh:hasValue: 5 matches 5.0, + # and like every other value constraint in a condition it holds when the + # slot is absent. + prop_pv(SH.minInclusive, Literal(condition.equals_number)) + prop_pv(SH.maxInclusive, Literal(condition.equals_number)) + if condition.range is not None: + self._add_range(g, prop_pv, condition.range) + if repeated: + # Each repeated parameter in a member shape of its own: all of them hold + # for every value, as they would on the property shape itself. + members = [] + for p, v in repeated: + member = BNode() + g.add((member, p, v)) + members.append(member) + and_node = BNode() + Collection(g, and_node, members) + g.add((pnode, SH["and"], and_node)) + return pnode + + def _string_value_terms(self, r: ElementName | None, values: list[str]) -> list[URIRef | Literal]: + """The RDF terms of the ``equals_string`` / ``equals_string_in`` *values* of a slot with range *r*. + + Permissible values of an enum are rendered as :meth:`_add_enum` renders them, as + the IRI of their ``meaning`` where they have one; anything else is a plain literal. + """ + sv = self.schemaview + if r in sv.all_enums(): + pvs = sv.get_enum(r).permissible_values + return [ + URIRef(sv.expand_curie(pvs[v].meaning)) if v in pvs and pvs[v].meaning else Literal(v) for v in values + ] + return [Literal(v) for v in values] + + def _condition_slot(self, cls: ClassDefinition, slot_name: str) -> SlotDefinition | None: + """The slot a condition of *cls* names, as induced for *cls*, or ``None`` if there is none.""" + try: + return self.schemaview.induced_slot(slot_name, cls.name) + except ValueError: + return None + + def _type_uri(self, r: ElementName | None) -> str | None: + """The expanded datatype IRI of type range *r*, or ``None`` when *r* is not a type. + + Resolved through the induced type, so a type derived with ``typeof`` + inherits the ``uri`` of its ancestor. A built-in type name in a schema + that does not import ``linkml:types`` resolves as the main slot loop + resolves it (:class:`ShaclDataType`). + """ + sv = self.schemaview + if r in sv.all_types(): + return sv.get_uri(sv.induced_type(r), expand=True) + builtin = next((t for t in ShaclDataType if t.linkml_type == r), None) + return str(builtin.uri_ref) if builtin is not None else None + + def _is_string_range(self, r: ElementName | None) -> bool: + """Whether a slot with range *r* holds strings, which ``equals_string`` compares against. + + True for an enum, whose permissible values are rendered as their + ``meaning`` IRI or as a plain literal (as :meth:`_add_enum` renders + them); for a type whose datatype is ``xsd:string``, whose values are + plain literals; and for no range at all, whose values are untyped and + compared as strings, as the JSON Schema generator compares them. A + type with any other datatype, including one derived from ``string`` + (``xsd:anyURI``, ``xsd:token``, ...), holds typed literals or IRIs + that a string literal does not match. + """ + if r is None or r in self.schemaview.all_enums(): + return True + return self._type_uri(r) == str(XSD.string) + + def _untranslatable(self, cls: ClassDefinition, expr: AnonymousClassExpression) -> str | None: + """Return what in class expression *expr* of *cls* cannot be translated, or ``None``.""" + sv = self.schemaview + unknown = self._set_operator_fields(expr) - self._CLASS_EXPRESSION_FIELDS + if unknown: + return f"'{sorted(unknown)[0]}'" + if expr.is_a is not None and expr.is_a not in sv.all_classes(): + return f"is_a '{expr.is_a}', which is not a class" + for slot_name, condition in expr.slot_conditions.items(): + unknown = self._set_operator_fields(condition) - self._SLOT_CONDITION_FIELDS + if unknown: + return f"'{sorted(unknown)[0]}' in the condition on slot '{slot_name}'" + slot = self._condition_slot(cls, slot_name) + if slot is None: + return f"a condition on '{slot_name}', which is not a slot" + if slot.identifier: + # An identifier is the node's IRI, not a property arc. + return f"a condition on the identifier slot '{slot_name}'" + if condition.range is not None and not self._is_known_range(condition.range): + return f"the unknown range '{condition.range}' in the condition on slot '{slot_name}'" + value_range = condition.range or slot.range + if (condition.equals_string is not None or condition.equals_string_in) and not self._is_string_range( + value_range + ): + return f"equals_string on slot '{slot_name}', whose range '{value_range}' does not hold strings" + for operator in self._CLASS_EXPRESSION_OPERATORS: + for member in getattr(expr, operator) or []: + reason = self._untranslatable(cls, member) + if reason is not None: + return reason + return None + + def _is_known_range(self, r: ElementName) -> bool: + """Whether *r* names a class, type or enum of the schema, or a built-in type.""" + sv = self.schemaview + return ( + r in sv.all_classes() + or r in sv.all_types() + or r in sv.all_enums() + or any(datatype.linkml_type == r for datatype in ShaclDataType) + ) + # ------------------------------------------------------------------- # Rules → sh:sparql # ------------------------------------------------------------------- @@ -671,6 +1014,37 @@ def _add_class(self, func: Callable, r: ElementName) -> None: range_ref += self.suffix func(SH["node"], URIRef(range_ref)) + def _slot_iri(self, slot: SlotDefinition) -> str: + """The full IRI of *slot*, exactly as ``sh:path`` in the main slot loop renders it. + + An induced slot carries its ``slot_usage`` overrides, so an overridden + ``slot_uri`` yields the same IRI as ``sh:path``; otherwise the query + would use a property the data never uses and never fire. + """ + sv = self.schemaview + if slot.name in sv.element_by_schema_map(): + return sv.get_uri(slot, expand=True) + return sv.expand_curie(f"{sv.schema.default_prefix}:{underscore(slot.name)}") + + def _add_range(self, g: Graph, func: Callable, r: ElementName) -> None: + """Add the value-type constraint for range *r*: a class, type, enum or built-in datatype.""" + sv = self.schemaview + if r in sv.all_classes(): + cls_def = sv.get_class(r) + is_any = cls_def and getattr(cls_def, "class_uri", None) == "linkml:Any" + self._add_class(func, r) + if not is_any: + if sv.get_identifier_slot(r) is not None: + func(SH.nodeKind, SH.IRI) + else: + func(SH.nodeKind, SH.BlankNodeOrIRI) + elif r in sv.all_types(): + self._add_type(func, r) + elif r in sv.all_enums(): + self._add_enum(g, func, r) + else: + add_simple_data_type(func, r) + def _add_enum(self, g: Graph, func: Callable, r: ElementName) -> None: sv = self.schemaview enum = sv.get_enum(r) diff --git a/tests/linkml/test_compliance/test_boolean_slot_compliance.py b/tests/linkml/test_compliance/test_boolean_slot_compliance.py index fd900bf938..e73ff46851 100644 --- a/tests/linkml/test_compliance/test_boolean_slot_compliance.py +++ b/tests/linkml/test_compliance/test_boolean_slot_compliance.py @@ -505,7 +505,7 @@ def test_class_any_of(framework, data_name, s1value, s2value, is_valid): core_elements=["any_of", "ClassDefinition"], ) expected_behavior = ValidationBehavior.IMPLEMENTS - if framework not in [OWL]: + if framework not in [OWL, SHACL]: # TODO: rdflib transformer has issues around ranges expected_behavior = ValidationBehavior.INCOMPLETE # TODO: rdflib transformer has issues around ranges @@ -632,8 +632,22 @@ def test_class_any_of_with_required(framework, nest, op, name, family_name, give core_elements=[op, "ClassDefinition"], ) expected_behavior = ValidationBehavior.IMPLEMENTS - if framework not in [JSON_SCHEMA]: + if framework not in [JSON_SCHEMA, SHACL]: expected_behavior = ValidationBehavior.INCOMPLETE + elif framework == SHACL and 5 in (name, family_name, given_name): + # SHACL validation makes its instances through python dataclasses, which coerce + # the integer to a string, so the range violation never reaches the shapes. A + # row can then only be detected through the operator itself. + present = [value is not None for value in (name, family_name, given_name)] + members = [present[0], present[1] and present[2]] + operator_holds = { + "any_of": any(members), + "all_of": all(members), + "exactly_one_of": sum(members) == 1, + "none_of": not any(members), + }[op] + if operator_holds: + expected_behavior = ValidationBehavior.INCOMPLETE data = {SLOT_S1: name, SLOT_S2: family_name, SLOT_S3: given_name} if nest: diff --git a/tests/linkml/test_generators/test_jsonschemagen.py b/tests/linkml/test_generators/test_jsonschemagen.py index b22dcc7ae3..abdbc2e266 100644 --- a/tests/linkml/test_generators/test_jsonschemagen.py +++ b/tests/linkml/test_generators/test_jsonschemagen.py @@ -1785,3 +1785,133 @@ def test_top_class_matches_regardless_of_case(tmp_path): assert schema["additionalProperties"] is False assert "name" in schema["properties"] + + +_CLASS_EXPRESSION_INHERITANCE_SCHEMA = """ +id: https://example.org/class-expressions +name: class_expressions +prefixes: + linkml: https://w3id.org/linkml/ + ex: https://example.org/class-expressions/ +imports: + - linkml:types +default_prefix: ex +default_range: string +slots: + a: {} + b: {} +classes: + Marker: + mixin: true + none_of: + - slot_conditions: + a: + equals_string: forbidden + Parent: + slots: [a, b] + any_of: + - slot_conditions: + a: + required: true + - slot_conditions: + b: + required: true + Child: + is_a: Parent + mixins: [Marker] + GrandChild: + is_a: Child +""" + + +@pytest.mark.parametrize("target_class", ["Parent", "Child", "GrandChild"]) +@pytest.mark.parametrize( + "instance,valid_for_parent,valid_for_marker_users", + [ + pytest.param({"a": "1"}, True, True, id="a"), + pytest.param({"b": "2"}, True, True, id="b"), + pytest.param({}, False, False, id="neither"), + pytest.param({"a": "forbidden"}, True, False, id="forbidden-by-mixin"), + ], +) +def test_class_expressions_apply_to_subclasses(target_class, instance, valid_for_parent, valid_for_marker_users): + """A class expression constrains every instance of its class, so the + definitions of its subclasses, and of the classes using a mixin, carry it.""" + json_schema = json.loads( + JsonSchemaGenerator(_CLASS_EXPRESSION_INHERITANCE_SCHEMA, top_class=target_class).serialize() + ) + expected = valid_for_parent if target_class == "Parent" else valid_for_marker_users + assert jsonschema.Draft7Validator(json_schema).is_valid(instance) == expected + + +_TWO_ABSENT_RULE_SCHEMA = """ +id: https://example.org/absent +name: absent +prefixes: + linkml: https://w3id.org/linkml/ +imports: + - linkml:types +default_range: string +classes: + Thing: + attributes: + trigger: {} + a: {} + b: {} + rules: + - preconditions: + slot_conditions: + trigger: + value_presence: PRESENT + postconditions: + slot_conditions: + a: + value_presence: ABSENT + b: + value_presence: ABSENT +""" + + +@pytest.mark.parametrize( + "instance,valid", + [ + pytest.param({"trigger": "t"}, True, id="neither"), + pytest.param({"trigger": "t", "a": "x"}, False, id="one"), + pytest.param({"trigger": "t", "a": "x", "b": "y"}, False, id="both"), + pytest.param({"a": "x", "b": "y"}, True, id="not-triggered"), + ], +) +def test_several_absent_slots_each_must_be_absent(instance, valid): + """value_presence: ABSENT on several slots of one expression requires each + to be absent, not merely that they aren't all present.""" + json_schema = json.loads(JsonSchemaGenerator(_TWO_ABSENT_RULE_SCHEMA, top_class="Thing").serialize()) + assert jsonschema.Draft7Validator(json_schema).is_valid(instance) == valid + + +@pytest.mark.parametrize("tags,valid", [(["A"], True), (["A", "B"], False), ([], True)]) +def test_class_expression_condition_uses_the_induced_slot(tags, valid): + """A condition constrains the slot as induced for the class, so a + slot_usage that makes it multivalued applies the condition to every value.""" + schema = """ +id: https://example.org/induced +name: induced +prefixes: + linkml: https://w3id.org/linkml/ +imports: + - linkml:types +default_range: string +slots: + tags: {} +classes: + Thing: + slots: [tags] + slot_usage: + tags: + multivalued: true + all_of: + - slot_conditions: + tags: + equals_string: A +""" + json_schema = json.loads(JsonSchemaGenerator(schema, top_class="Thing").serialize()) + assert jsonschema.Draft7Validator(json_schema).is_valid({"tags": tags}) == valid diff --git a/tests/linkml/test_generators/test_shaclgen.py b/tests/linkml/test_generators/test_shaclgen.py index f367de97e7..3813992f2d 100644 --- a/tests/linkml/test_generators/test_shaclgen.py +++ b/tests/linkml/test_generators/test_shaclgen.py @@ -1,9 +1,12 @@ +import json +import logging from collections import Counter from typing import Any import pytest import rdflib -from rdflib import RDF, RDFS, SH, Literal, URIRef +import yaml +from rdflib import RDF, RDFS, SH, XSD, Literal, URIRef from rdflib.collection import Collection from linkml.generators.shacl.shacl_data_type import ShaclDataType @@ -2833,3 +2836,1463 @@ def test_shacl_modular_schema_with_reused_attribute_name(tmp_path) -> None: graph.parse(data=ShaclGenerator(str(domain)).serialize(), format="turtle") shapes = set(graph.subjects(RDF.type, SH.NodeShape)) assert URIRef("https://example.org/domain/Pedido") in shapes + + +# --------------------------------------------------------------------------- +# Class expressions → sh:or / sh:and / sh:xone / sh:not +# --------------------------------------------------------------------------- + +EX_CE = rdflib.Namespace("https://example.org/class-expressions/") + +_CLASS_EXPRESSION_HEADER = """ +id: https://example.org/class-expressions +name: class_expressions +prefixes: + linkml: https://w3id.org/linkml/ + ex: https://example.org/class-expressions/ +imports: + - linkml:types +default_prefix: ex +default_range: string +""" + +_REFERENCE_SYSTEM_SCHEMA = ( + _CLASS_EXPRESSION_HEADER + + """ +slots: + codeEPSG: + range: integer + coordinateSystemName: {{}} +classes: + ReferenceSystem: + class_uri: ex:ReferenceSystem + slots: [codeEPSG, coordinateSystemName] + {operator}: + - slot_conditions: + codeEPSG: + required: true + - slot_conditions: + coordinateSystemName: + required: true +""" +) + +_LOGICAL_PREDICATES = (SH["or"], SH["and"], SH.xone, SH["not"]) + + +def _list_members(g, shape, predicate): + """Members of the single SHACL list that *shape* has for *predicate*.""" + lists = list(g.objects(shape, predicate)) + assert len(lists) == 1, f"expected one {predicate} list on {shape}, got {len(lists)}" + return list(Collection(g, lists[0])) + + +def _condition(g, member, path): + """The property shape for *path* inside the member shape *member*.""" + shapes = [p for p in g.objects(member, SH.property) if (p, SH.path, path) in g] + assert len(shapes) == 1, f"expected one condition on {path}, got {len(shapes)}" + return shapes[0] + + +def _conforms(shacl_ttl: str, data_ttl: str) -> bool: + """Validate *data_ttl*; meta_shacl makes pyshacl fail on an ill-formed shapes graph.""" + import pyshacl + + conforms, _, _ = pyshacl.validate( + data_graph=data_ttl, + shacl_graph=shacl_ttl, + data_graph_format="turtle", + shacl_graph_format="turtle", + advanced=True, + meta_shacl=True, + ) + return conforms + + +@pytest.mark.parametrize( + "operator,predicate", + [("any_of", SH["or"]), ("all_of", SH["and"]), ("exactly_one_of", SH.xone)], +) +def test_class_expression_list_operator_generates_logical_constraint(operator, predicate): + """any_of, all_of and exactly_one_of become sh:or, sh:and and sh:xone over the member shapes.""" + g = _parse_shacl(_REFERENCE_SYSTEM_SCHEMA.format(operator=operator)) + + members = _list_members(g, EX_CE.ReferenceSystem, predicate) + assert len(members) == 2 + for member, path in zip(members, (EX_CE.codeEPSG, EX_CE.coordinateSystemName)): + condition = _condition(g, member, path) + assert (condition, SH.minCount, Literal(1)) in g + assert (member, SH.targetClass, None) not in g + others = set(_LOGICAL_PREDICATES) - {predicate} + assert not any((EX_CE.ReferenceSystem, p, None) in g for p in others) + + +def test_class_expression_none_of_generates_one_sh_not_per_member(): + """none_of becomes one sh:not per member; the negated constraints all apply (SHACL §2.1.1).""" + g = _parse_shacl(_REFERENCE_SYSTEM_SCHEMA.format(operator="none_of")) + + negated = list(g.objects(EX_CE.ReferenceSystem, SH["not"])) + assert len(negated) == 2 + paths = {path for member in negated for p in g.objects(member, SH.property) for path in g.objects(p, SH.path)} + assert paths == {EX_CE.codeEPSG, EX_CE.coordinateSystemName} + + +@pytest.mark.parametrize( + "operator,properties,expected", + [ + ("any_of", 'ex:codeEPSG 4326 ; ex:coordinateSystemName "WGS 84"', True), + ("any_of", "ex:codeEPSG 4326", True), + ("any_of", "", False), + ("exactly_one_of", "ex:codeEPSG 4326", True), + ("exactly_one_of", 'ex:codeEPSG 4326 ; ex:coordinateSystemName "WGS 84"', False), + ("exactly_one_of", "", False), + ("all_of", 'ex:codeEPSG 4326 ; ex:coordinateSystemName "WGS 84"', True), + ("all_of", "ex:codeEPSG 4326", False), + ("none_of", "", True), + ("none_of", "ex:codeEPSG 4326", False), + ], +) +def test_class_expression_pyshacl_end_to_end(operator, properties, expected): + """End-to-end: each operator admits exactly the instances the metamodel says it holds for.""" + shacl_ttl = ShaclGenerator(_REFERENCE_SYSTEM_SCHEMA.format(operator=operator), mergeimports=False).serialize() + data = f""" + @prefix ex: . + ex:rs a ex:ReferenceSystem {";" if properties else ""} {properties} . + """ + assert _conforms(shacl_ttl, data) is expected + + +_FORMAT_PROFILES_SCHEMA = ( + _CLASS_EXPRESSION_HEADER + + """ +slots: + fileFormat: {} + formatType: {} + version: {} + hasChannel: + range: Channel + multivalued: true + inlined: true +classes: + Channel: + class_uri: ex:Channel + Format: + class_uri: ex:Format + slots: [fileFormat, formatType, version, hasChannel] + any_of: + - description: A single-channel trace. + slot_conditions: + fileFormat: + equals_string_in: [OSI, TXTH] + formatType: + required: true + version: + required: true + - description: A multi-channel container. + slot_conditions: + fileFormat: + required: true + equals_string: MCAP + hasChannel: + required: true +""" +) + + +@pytest.mark.parametrize( + "properties,expected", + [ + ('ex:fileFormat "OSI" ; ex:formatType "SensorView" ; ex:version "3.7.0"', True), + ('ex:fileFormat "MCAP" ; ex:hasChannel ex:ch', True), + ('ex:fileFormat "MCAP" ; ex:formatType "SensorView" ; ex:version "3.7.0"', False), + ('ex:fileFormat "OSI" ; ex:formatType "SensorView" ; ex:hasChannel ex:ch', False), + ], +) +def test_class_expression_any_of_profiles_pyshacl_end_to_end(properties, expected): + """End-to-end: a class-level any_of selects between two complete profiles of a class.""" + shacl_ttl = ShaclGenerator(_FORMAT_PROFILES_SCHEMA, mergeimports=False).serialize() + data = f""" + @prefix ex: . + ex:ch a ex:Channel . + ex:f a ex:Format ; {properties} . + """ + assert _conforms(shacl_ttl, data) is expected + + +def test_class_expression_member_metadata(): + """A member's title and description annotate its shape, as they do for a class's NodeShape.""" + g = _parse_shacl(_FORMAT_PROFILES_SCHEMA) + + comments = {str(c) for m in _list_members(g, EX_CE.Format, SH["or"]) for c in g.objects(m, RDFS.comment)} + assert comments == {"A single-channel trace.", "A multi-channel container."} + + +_CONDITIONS_SCHEMA = ( + _CLASS_EXPRESSION_HEADER + + """ +enums: + ColourEnum: + permissible_values: + red: {} + blue: {} +slots: + label: {} + size: + range: integer + count: + multivalued: true + exact: + multivalued: true + colour: {} + target: {} + note: {} +classes: + Target: + class_uri: ex:Target + Thing: + class_uri: ex:Thing + slots: [label, size, count, exact, colour, target, note] + all_of: + - title: every condition + slot_conditions: + label: + description: Starts upper case. + required: true + pattern: "^[A-Z]" + equals_string_in: [Alpha, Beta] + size: + minimum_value: 1 + maximum_value: 10 + equals_number: 5 + count: + required: true + minimum_cardinality: 2 + maximum_cardinality: 4 + exact: + exact_cardinality: 3 + colour: + range: ColourEnum + target: + value_presence: PRESENT + range: Target + note: + value_presence: ABSENT +""" +) + + +def test_class_expression_slot_condition_fields(): + """Each supported slot-condition field maps to the SHACL constraint the slot loop uses for it.""" + g = _parse_shacl(_CONDITIONS_SCHEMA) + (member,) = _list_members(g, EX_CE.Thing, SH["and"]) + assert (member, RDFS.label, Literal("every condition")) in g + + def values(path, predicate): + return set(g.objects(_condition(g, member, path), predicate)) + + def in_list(path): + (node,) = values(path, SH["in"]) + return list(Collection(g, node)) + + assert values(EX_CE.label, SH.minCount) == {Literal(1)} + assert values(EX_CE.label, SH.pattern) == {Literal("^[A-Z]")} + assert values(EX_CE.label, SH.description) == {Literal("Starts upper case.")} + assert in_list(EX_CE.label) == [Literal("Alpha"), Literal("Beta")] + + # equals_number is a value comparison; SHACL allows one sh:minInclusive and one + # sh:maxInclusive per shape, so next to the bounds it moves into an sh:and member + assert values(EX_CE.size, SH.minInclusive) == {Literal(1)} + assert values(EX_CE.size, SH.maxInclusive) == {Literal(10)} + (and_node,) = values(EX_CE.size, SH["and"]) + repeated = {(p, o) for m in Collection(g, and_node) for p, o in g.predicate_objects(m)} + assert repeated == {(SH.minInclusive, Literal(5)), (SH.maxInclusive, Literal(5))} + assert values(EX_CE.size, SH["in"]) == set() + assert values(EX_CE.size, SH.minCount) == set() + assert values(EX_CE.size, SH.hasValue) == set() + + # required and a cardinality give one sh:minCount, the stricter of the two + assert values(EX_CE["count"], SH.minCount) == {Literal(2)} + assert values(EX_CE["count"], SH.maxCount) == {Literal(4)} + assert values(EX_CE.exact, SH.minCount) == {Literal(3)} + assert values(EX_CE.exact, SH.maxCount) == {Literal(3)} + + assert in_list(EX_CE.colour) == [Literal("red"), Literal("blue")] + + assert values(EX_CE.target, SH.minCount) == {Literal(1)} + assert values(EX_CE.target, SH["class"]) == {EX_CE.Target} + assert values(EX_CE.target, SH.nodeKind) == {SH.BlankNodeOrIRI} + + assert values(EX_CE.note, SH.maxCount) == {Literal(0)} + assert values(EX_CE.note, SH.minCount) == set() + + +@pytest.mark.parametrize( + "condition,properties,expected", + [ + # outside none_of a value constraint says nothing about presence + ("any_of", "", True), + ("any_of", 'ex:label "b"', False), + # inside none_of it requires the slot, so an absent slot is not rejected + ("none_of", "", True), + ("none_of", 'ex:label "A"', False), + ("none_of", 'ex:label "B"', True), + ], +) +def test_class_expression_presence_semantics(condition, properties, expected): + """A condition holds vacuously for an absent slot, except under none_of, as in the JSON Schema generator.""" + schema = ( + _CLASS_EXPRESSION_HEADER + + f""" +slots: + label: {{}} +classes: + Thing: + class_uri: ex:Thing + slots: [label] + {condition}: + - slot_conditions: + label: + {"pattern: '^[A-Z]'" if condition == "any_of" else "equals_string: A"} +""" + ) + shacl_ttl = ShaclGenerator(schema, mergeimports=False).serialize() + data = f""" + @prefix ex: . + ex:t a ex:Thing {";" if properties else ""} {properties} . + """ + assert _conforms(shacl_ttl, data) is expected + + +def test_class_expression_nested_and_is_a(): + """Nested expressions recurse into the member shape; is_a gives sh:class.""" + schema = ( + _CLASS_EXPRESSION_HEADER + + """ +slots: + a: {} + b: {} + c: {} +classes: + Marker: + class_uri: ex:Marker + Thing: + class_uri: ex:Thing + slots: [a, b, c] + any_of: + - all_of: + - slot_conditions: + a: + required: true + - slot_conditions: + b: + required: true + - is_a: Marker + slot_conditions: + c: + required: true +""" + ) + # open shapes: the instance is also a Marker, whose own shape declares no slots + shacl_ttl = ShaclGenerator(schema, mergeimports=False, closed=False).serialize() + g = rdflib.Graph().parse(data=shacl_ttl) + first, second = _list_members(g, EX_CE.Thing, SH["or"]) + assert len(_list_members(g, first, SH["and"])) == 2 + assert (second, SH["class"], EX_CE.Marker) in g + + prefix = "@prefix ex: ." + assert _conforms(shacl_ttl, f'{prefix} ex:t a ex:Thing ; ex:a "1" ; ex:b "2" .') + assert not _conforms(shacl_ttl, f'{prefix} ex:t a ex:Thing ; ex:a "1" .') + assert _conforms(shacl_ttl, f'{prefix} ex:t a ex:Thing, ex:Marker ; ex:c "3" .') + assert not _conforms(shacl_ttl, f'{prefix} ex:t a ex:Thing ; ex:c "3" .') + + +_INHERITED_CLASS_EXPRESSION_SCHEMA = ( + _CLASS_EXPRESSION_HEADER + + """ +slots: + a: {} + b: {} +classes: + Marker: + class_uri: ex:Marker + mixin: true + none_of: + - slot_conditions: + a: + equals_string: forbidden + Parent: + class_uri: ex:Parent + slots: [a, b] + any_of: + - slot_conditions: + a: + required: true + - slot_conditions: + b: + required: true + Child: + class_uri: ex:Child + is_a: Parent + mixins: [Marker] + GrandChild: + class_uri: ex:GrandChild + is_a: Child +""" +) +_CE_PREFIXES = ( + "@prefix ex: .\n@prefix rdfs: .\n" +) + + +@pytest.mark.parametrize("subclass_axioms", [False, True], ids=["typed-only", "with-rdfs-subClassOf"]) +@pytest.mark.parametrize("class_name", ["Parent", "Child", "GrandChild"]) +@pytest.mark.parametrize( + "values,conforms", + [ + pytest.param(' ; ex:a "1"', True, id="a"), + pytest.param(' ; ex:b "2"', True, id="b"), + pytest.param("", False, id="neither"), + ], +) +def test_class_expression_applies_to_subclass_instances(subclass_axioms, class_name, values, conforms): + """A class expression constrains every instance of its class, so the shapes + of its subclasses carry it too. A node typed only with a subclass, as + rdflib_dumper and the JSON-LD context produce it, is checked without an + rdfs:subClassOf triple in the data; with one, the verdict is the same.""" + shacl_ttl = ShaclGenerator(_INHERITED_CLASS_EXPRESSION_SCHEMA, mergeimports=False, closed=False).serialize() + axioms = ( + "ex:Child rdfs:subClassOf ex:Parent . ex:GrandChild rdfs:subClassOf ex:Child .\n" if subclass_axioms else "" + ) + assert _conforms(shacl_ttl, f"{_CE_PREFIXES}{axioms}ex:x a ex:{class_name}{values} .") == conforms + + +@pytest.mark.parametrize("class_name,conforms", [("Parent", True), ("Child", False), ("GrandChild", False)]) +def test_class_expression_of_a_mixin_applies_to_the_classes_using_it(class_name, conforms): + """A mixin's class expression constrains the classes that use the mixin and + their subclasses, and no other class.""" + shacl_ttl = ShaclGenerator(_INHERITED_CLASS_EXPRESSION_SCHEMA, mergeimports=False, closed=False).serialize() + assert _conforms(shacl_ttl, f'{_CE_PREFIXES}ex:x a ex:{class_name} ; ex:a "forbidden" .') == conforms + + +def test_inherited_class_expression_follows_the_subclass_slot_usage(): + """An inherited class expression is translated in the subclass's context, so + its conditions use the subclass's slot_usage, as the subclass's own + property shapes do.""" + schema = yaml.safe_load(_INHERITED_CLASS_EXPRESSION_SCHEMA) + schema["classes"]["Child"]["slot_usage"] = {"a": {"slot_uri": "ex:childA"}} + g = rdflib.Graph().parse(data=ShaclGenerator(json.dumps(schema), mergeimports=False).serialize()) + for shape, path in ((EX_CE.Parent, EX_CE.a), (EX_CE.Child, EX_CE.childA), (EX_CE.GrandChild, EX_CE.childA)): + (or_list,) = g.objects(shape, SH["or"]) + paths = { + p + for member in Collection(g, or_list) + for prop in g.objects(member, SH.property) + for p in g.objects(prop, SH.path) + } + assert paths == {path, EX_CE.b}, (shape, paths) + + +def test_classes_sharing_a_class_uri_carry_an_inherited_expression_once(): + """Classes with the same class_uri share one shape, which carries an + expression they inherit once.""" + schema = yaml.safe_load(_INHERITED_CLASS_EXPRESSION_SCHEMA) + schema["classes"]["Child"]["class_uri"] = "ex:Shared" + schema["classes"]["GrandChild"]["class_uri"] = "ex:Shared" + g = rdflib.Graph().parse(data=ShaclGenerator(json.dumps(schema), mergeimports=False).serialize()) + assert len(list(g.objects(EX_CE.Shared, SH["or"]))) == 1 + assert len(list(g.objects(EX_CE.Shared, SH["not"]))) == 1 + + +@pytest.mark.parametrize( + "child_usage,expected_shapes,warning", + [ + pytest.param( + None, + {EX_CE.Parent: False, EX_CE.Child: False, EX_CE.GrandChild: False}, + "Class 'Parent': any_of is not translated to SHACL, because it uses 'has_member' in the condition on " + "slot 'a' (in the shapes of 'Parent', 'Child', 'GrandChild').", + id="untranslatable-everywhere", + ), + pytest.param( + {"a": {"range": "integer"}}, + {EX_CE.Parent: True, EX_CE.Child: False, EX_CE.GrandChild: False}, + "Class 'Parent': any_of is not translated to SHACL, because it uses equals_string on slot 'a', whose " + "range 'integer' does not hold strings (in the shapes of 'Child', 'GrandChild').", + id="untranslatable-in-subclasses", + ), + ], +) +def test_untranslatable_inherited_class_expression_warned_once(caplog, child_usage, expected_shapes, warning): + """An inherited expression that cannot be translated is skipped in the + shapes where it cannot, and reported once, naming them.""" + schema = yaml.safe_load(_INHERITED_CLASS_EXPRESSION_SCHEMA) + schema["classes"]["Parent"]["any_of"] = ( + [{"slot_conditions": {"a": {"has_member": {"equals_string": "x"}}}}] + if child_usage is None + else [{"slot_conditions": {"a": {"equals_string": "x"}}}] + ) + if child_usage is not None: + schema["classes"]["Child"]["slot_usage"] = child_usage + with caplog.at_level(logging.WARNING, logger="linkml.generators.shaclgen"): + g = rdflib.Graph().parse(data=ShaclGenerator(json.dumps(schema), mergeimports=False).serialize()) + for shape, translated in expected_shapes.items(): + assert ((shape, SH["or"], None) in g) == translated, shape + messages = [rec.message for rec in caplog.records if "any_of is not translated" in rec.message] + assert messages == [warning] + + +def test_class_expression_untranslatable_operator_skipped_with_warning(caplog): + """An operator with an untranslatable member is skipped whole, with a warning; the others are kept.""" + import logging + + schema = ( + _CLASS_EXPRESSION_HEADER + + """ +slots: + tags: + multivalued: true + a: {} +classes: + Thing: + class_uri: ex:Thing + slots: [tags, a] + any_of: + - slot_conditions: + tags: + has_member: + equals_string: x + - slot_conditions: + a: + required: true + none_of: + - slot_conditions: + a: + equals_string: forbidden +""" + ) + with caplog.at_level(logging.WARNING, logger="linkml.generators.shaclgen"): + g = _parse_shacl(schema) + + assert (EX_CE.Thing, SH["or"], None) not in g + assert any( + "any_of" in rec.message and "has_member" in rec.message and "tags" in rec.message for rec in caplog.records + ) + # the translatable none_of is kept, and enforced + shacl_ttl = g.serialize(format="turtle") + assert not _conforms(shacl_ttl, f'{_CE_PREFIXES}ex:t a ex:Thing ; ex:a "forbidden" .') + assert _conforms(shacl_ttl, f'{_CE_PREFIXES}ex:t a ex:Thing ; ex:a "ok" .') + + +def test_class_expression_condition_on_identifier_skipped_with_warning(caplog): + """An identifier is the node's IRI, not a property arc, so a condition on it is not translated.""" + import logging + + schema = ( + _CLASS_EXPRESSION_HEADER + + """ +slots: + id: + identifier: true + a: {} +classes: + Thing: + class_uri: ex:Thing + slots: [id, a] + exactly_one_of: + - slot_conditions: + id: + pattern: "^ex:" + - slot_conditions: + a: + required: true +""" + ) + with caplog.at_level(logging.WARNING, logger="linkml.generators.shaclgen"): + g = _parse_shacl(schema) + + assert (EX_CE.Thing, SH.xone, None) not in g + assert any("exactly_one_of" in rec.message and "identifier" in rec.message for rec in caplog.records) + + +def test_class_expression_absent_leaves_node_shapes_unchanged(): + """Without class-level expressions no node shape gets a logical constraint; slot-level any_of is unaffected.""" + schema = ( + _CLASS_EXPRESSION_HEADER + + """ +slots: + value: + any_of: + - range: integer + - range: string +classes: + Thing: + class_uri: ex:Thing + slots: [value] +""" + ) + g = _parse_shacl(schema) + + for shape in g.subjects(SH.targetClass, None): + assert not any((shape, p, None) in g for p in _LOGICAL_PREDICATES) + (value_shape,) = [p for p in g.objects(EX_CE.Thing, SH.property) if (p, SH.path, EX_CE.value) in g] + assert (value_shape, SH["or"], None) in g + + +def test_class_expression_condition_path_is_the_slot_induced_for_the_class(): + """A condition's sh:path is the one the class's own property shape uses, slot_usage and attributes included.""" + schema = ( + _CLASS_EXPRESSION_HEADER + + """ +slots: + code: + slot_uri: ex:baseCode + exact mappings: + slot_uri: ex:exactMatch +classes: + Thing: + class_uri: ex:Thing + slots: [code, exact mappings] + slot_usage: + code: + slot_uri: ex:thingCode + attributes: + loc: + slot_uri: ex:thingLoc + any_of: + - slot_conditions: + code: + required: true + - slot_conditions: + exact_mappings: + required: true + - slot_conditions: + loc: + required: true + Other: + class_uri: ex:Other + attributes: + loc: + slot_uri: ex:otherLoc +""" + ) + g = _parse_shacl(schema) + + class_paths = {path for p in g.objects(EX_CE.Thing, SH.property) for path in g.objects(p, SH.path)} + condition_paths = [ + path + for member in _list_members(g, EX_CE.Thing, SH["or"]) + for p in g.objects(member, SH.property) + for path in g.objects(p, SH.path) + ] + assert condition_paths and set(condition_paths) <= class_paths + assert set(condition_paths) == {EX_CE.thingCode, EX_CE.exactMatch, EX_CE.thingLoc} + + +_PRESENCE_SCHEMA = ( + _CLASS_EXPRESSION_HEADER + + """ +enums: + Color: + permissible_values: + red: {} + green: {} +slots: + label: {} + note: {} + tags: + multivalued: true +classes: + Thing: + class_uri: ex:Thing + tree_root: true + slots: [label, note, tags] +""" +) + + +def _class_expression_verdicts(expression: dict, obj: dict) -> tuple[bool, bool]: + """Whether the JSON Schema and the SHACL shapes generated for ``Thing`` with + the class-level *expression* accept *obj*. For SHACL, *obj* is loaded + through the generated JSON-LD context.""" + import jsonschema + import pyshacl + + from linkml.generators.jsonldcontextgen import ContextGenerator + from linkml.generators.jsonschemagen import JsonSchemaGenerator + + schema = yaml.safe_load(_PRESENCE_SCHEMA) + schema["classes"]["Thing"].update(expression) + schema = yaml.safe_dump(schema) + json_schema = json.loads(JsonSchemaGenerator(schema, top_class="Thing").serialize()) + context = json.loads(ContextGenerator(schema).serialize())["@context"] + data = rdflib.Graph().parse(data=json.dumps({"@context": context, "@type": "Thing", **obj}), format="json-ld") + shapes = rdflib.Graph().parse(data=ShaclGenerator(schema, mergeimports=False, closed=False).serialize()) + conforms, _, _ = pyshacl.validate(data, shacl_graph=shapes, meta_shacl=True) + return jsonschema.Draft7Validator(json_schema).is_valid(obj), conforms + + +def _is_a(value: str) -> dict: + """A class expression requiring ``label`` to equal *value*.""" + return {"slot_conditions": {"label": {"equals_string": value}}} + + +def _tags(**condition) -> dict: + """A class expression with *condition* on the multivalued ``tags``.""" + return {"slot_conditions": {"tags": condition}} + + +_LABEL_A_OR_NOTE_B = [_is_a("A"), {"slot_conditions": {"note": {"equals_string": "B"}}}] + + +@pytest.mark.parametrize( + "expression,obj,valid", + [ + # a condition is unknown for an absent slot, and only a definitely false expression is a violation + pytest.param({"any_of": [_is_a("A")]}, {}, True, id="any_of-absent"), + pytest.param({"any_of": [_is_a("A")]}, {"label": "Z"}, False, id="any_of-other"), + pytest.param({"any_of": _LABEL_A_OR_NOTE_B}, {"label": "Z"}, True, id="any_of-one-false-one-unknown"), + pytest.param({"none_of": [_is_a("A")]}, {}, True, id="none_of-absent"), + pytest.param({"none_of": [_is_a("A")]}, {"label": "A"}, False, id="none_of-match"), + pytest.param({"none_of": [_is_a("A")]}, {"label": "B"}, True, id="none_of-other"), + pytest.param({"none_of": [{"any_of": [_is_a("A"), _is_a("B")]}]}, {}, True, id="nested-in-none_of-absent"), + pytest.param( + {"none_of": [{"any_of": [_is_a("A"), _is_a("B")]}]}, {"label": "B"}, False, id="nested-in-none_of" + ), + pytest.param({"none_of": [{"any_of": [_is_a("A"), _is_a("B")]}]}, {"label": "C"}, True, id="nested-other"), + # the reading composes: wrapping in a one-member all_of, or negating twice, changes nothing + pytest.param({"all_of": [{"none_of": [_is_a("A")]}]}, {}, True, id="none_of-in-all_of-absent"), + pytest.param({"all_of": [{"none_of": [_is_a("A")]}]}, {"label": "A"}, False, id="none_of-in-all_of-match"), + pytest.param({"none_of": [{"none_of": [_is_a("A")]}]}, {}, True, id="double-negation-absent"), + pytest.param({"none_of": [{"none_of": [_is_a("A")]}]}, {"label": "A"}, True, id="double-negation-match"), + pytest.param({"none_of": [{"none_of": [_is_a("A")]}]}, {"label": "B"}, False, id="double-negation-other"), + # stated presence is definite; required: false only restates the default + pytest.param( + {"none_of": [{"slot_conditions": {"label": {"required": False, "equals_string": "A"}}}]}, + {}, + True, + id="restated-default", + ), + pytest.param( + {"none_of": [{"slot_conditions": {"label": {"value_presence": "ABSENT", "equals_string": "A"}}}]}, + {}, + False, + id="stated-absent", + ), + pytest.param( + {"none_of": [{"slot_conditions": {"label": {"value_presence": "ABSENT", "equals_string": "A"}}}]}, + {"label": "A"}, + True, + id="stated-absent-present", + ), + pytest.param( + {"all_of": [{"slot_conditions": {"note": {"required": True, "value_presence": "ABSENT"}}}]}, + {}, + True, + id="value_presence-over-required", + ), + pytest.param( + {"all_of": [{"slot_conditions": {"note": {"required": True, "value_presence": "ABSENT"}}}]}, + {"note": "x"}, + False, + id="value_presence-over-required-present", + ), + # an empty condition is unknown for an absent slot, and definitely true for a present one + pytest.param({"none_of": [{"slot_conditions": {"label": {}}}]}, {}, True, id="empty-condition-absent"), + pytest.param({"none_of": [{"slot_conditions": {"label": {}}}]}, {"label": "x"}, False, id="empty-condition"), + # exactly_one_of holds when exactly one member is definitely true + pytest.param({"exactly_one_of": _LABEL_A_OR_NOTE_B}, {}, False, id="exactly_one_of-none-known"), + pytest.param({"exactly_one_of": _LABEL_A_OR_NOTE_B}, {"label": "A"}, True, id="exactly_one_of-first"), + pytest.param({"exactly_one_of": _LABEL_A_OR_NOTE_B}, {"label": "Z"}, False, id="exactly_one_of-none-true"), + pytest.param( + {"exactly_one_of": _LABEL_A_OR_NOTE_B}, {"label": "A", "note": "X"}, True, id="exactly_one_of-one-of-two" + ), + pytest.param( + {"exactly_one_of": _LABEL_A_OR_NOTE_B}, {"label": "A", "note": "B"}, False, id="exactly_one_of-both" + ), + # cardinalities count the values of a list, of which an absent slot has none + pytest.param({"all_of": [_tags(minimum_cardinality=2)]}, {"tags": ["a"]}, False, id="minimum-cardinality"), + pytest.param({"all_of": [_tags(minimum_cardinality=2)]}, {"tags": ["a", "b"]}, True, id="minimum-met"), + pytest.param({"all_of": [_tags(minimum_cardinality=2)]}, {}, False, id="minimum-absent"), + pytest.param({"all_of": [_tags(maximum_cardinality=1)]}, {"tags": ["a", "b"]}, False, id="maximum-cardinality"), + pytest.param({"none_of": [_tags(maximum_cardinality=1)]}, {}, True, id="none_of-maximum-absent"), + pytest.param({"none_of": [_tags(maximum_cardinality=1)]}, {"tags": ["a"]}, False, id="none_of-maximum"), + pytest.param( + {"none_of": [_tags(maximum_cardinality=1)]}, {"tags": ["a", "b"]}, True, id="none_of-maximum-over" + ), + pytest.param({"any_of": [_tags(maximum_cardinality=0), _is_a("A")]}, {}, True, id="maximum-zero-absent"), + # a single-valued slot holds at most one value + pytest.param( + {"all_of": [{"slot_conditions": {"label": {"minimum_cardinality": 2}}}]}, + {"label": "x"}, + False, + id="single-valued-minimum-two", + ), + # a range in a condition narrows the slot's values + pytest.param( + {"all_of": [{"slot_conditions": {"label": {"range": "Color"}}}]}, {"label": "red"}, True, id="range" + ), + pytest.param( + {"all_of": [{"slot_conditions": {"label": {"range": "Color"}}}]}, {"label": "blue"}, False, id="range-other" + ), + pytest.param({"all_of": [{"slot_conditions": {"label": {"range": "Color"}}}]}, {}, True, id="range-absent"), + pytest.param( + {"none_of": [{"slot_conditions": {"label": {"range": "Color"}}}]}, + {"label": "red"}, + False, + id="none_of-range", + ), + ], +) +def test_class_expression_absent_slot_semantics(expression, obj, valid): + """The specification doesn't say what a slot condition means for an absent + slot. As with an SQL CHECK constraint, an expression is violated only when + it is definitely false: a condition that doesn't state presence is unknown + for an absent slot. exactly_one_of holds when exactly one member is + definitely true. The JSON Schema and SHACL generators decide alike.""" + assert _class_expression_verdicts(expression, obj) == (valid, valid) + + +@pytest.mark.parametrize( + "condition,tags,conforms", + [ + # a bound an absent slot satisfies and a present one may too leaves absence open + pytest.param({"maximum_cardinality": 1}, [], True, id="at-most-one-absent"), + pytest.param({"maximum_cardinality": 1}, ["a"], False, id="at-most-one"), + pytest.param({"maximum_cardinality": 1}, ["a", "b"], True, id="more-than-one"), + # a maximum of 0 decides that the slot is absent + pytest.param({"maximum_cardinality": 0, "pattern": "^A"}, [], False, id="at-most-zero-absent"), + pytest.param({"maximum_cardinality": 0, "pattern": "^A"}, ["A"], True, id="at-most-zero-present"), + ], +) +def test_class_expression_none_of_cardinality(condition, tags, conforms): + """Within none_of, a condition is definitely true or false for an absent + slot only when it decides presence, such as a maximum cardinality of 0; + otherwise it requires the slot in its "definitely true" form.""" + schema = yaml.safe_load(_PRESENCE_SCHEMA) + schema["classes"]["Thing"]["none_of"] = [{"slot_conditions": {"tags": condition}}] + shacl_ttl = ShaclGenerator(yaml.safe_dump(schema), mergeimports=False, closed=False).serialize() + values = "" if not tags else " ; ex:tags " + ", ".join(f'"{t}"' for t in tags) + assert _conforms(shacl_ttl, f"{_CE_PREFIXES}ex:x a ex:Thing{values} .") == conforms + + +@pytest.mark.parametrize( + "condition,properties,expected", + [ + # a cardinality is taken literally inside none_of: "not at most zero values" means present + ("maximum_cardinality: 0", "", False), + ("maximum_cardinality: 0", 'ex:tag "x"', True), + ("maximum_cardinality: 1", 'ex:tag "x"', False), + ("maximum_cardinality: 1", 'ex:tag "x", "y"', True), + ], +) +def test_class_expression_none_of_takes_cardinality_literally(condition, properties, expected): + """Inside none_of only a condition silent on presence and cardinality is made to require the slot.""" + schema = ( + _CLASS_EXPRESSION_HEADER + + f""" +slots: + tag: + multivalued: true +classes: + Thing: + class_uri: ex:Thing + slots: [tag] + none_of: + - slot_conditions: + tag: + {condition} +""" + ) + shacl_ttl = ShaclGenerator(schema, mergeimports=False).serialize() + data = f""" + @prefix ex: . + ex:t a ex:Thing {";" if properties else ""} {properties} . + """ + assert _conforms(shacl_ttl, data) is expected + + +def test_class_expression_equals_string_on_enum_uses_the_meaning(): + """equals_string on an enum slot compares against the permissible value as _add_enum renders it.""" + schema = ( + _CLASS_EXPRESSION_HEADER + + """ +enums: + FormatEnum: + permissible_values: + OSI: + meaning: ex:OSI + MCAP: {} +slots: + fileFormat: + range: FormatEnum + hasChannel: {} +classes: + Format: + class_uri: ex:Format + slots: [fileFormat, hasChannel] + any_of: + - slot_conditions: + fileFormat: + equals_string: OSI + - slot_conditions: + fileFormat: + equals_string: MCAP + hasChannel: + required: true +""" + ) + shacl_ttl = ShaclGenerator(schema, mergeimports=False).serialize() + g = rdflib.Graph().parse(data=shacl_ttl) + first, second = _list_members(g, EX_CE.Format, SH["or"]) + (osi_in,) = g.objects(_condition(g, first, EX_CE.fileFormat), SH["in"]) + assert list(Collection(g, osi_in)) == [EX_CE.OSI] + (mcap_in,) = g.objects(_condition(g, second, EX_CE.fileFormat), SH["in"]) + assert list(Collection(g, mcap_in)) == [Literal("MCAP")] + + prefix = "@prefix ex: ." + assert _conforms(shacl_ttl, f"{prefix} ex:f a ex:Format ; ex:fileFormat ex:OSI .") + assert not _conforms(shacl_ttl, f'{prefix} ex:f a ex:Format ; ex:fileFormat "MCAP" .') + + +@pytest.mark.parametrize("value,expected", [("5.0", True), ("5", True), ("6.0", False)]) +def test_class_expression_equals_number_compares_values(value, expected): + """equals_number matches the value, whatever numeric datatype carries it.""" + schema = ( + _CLASS_EXPRESSION_HEADER + + """ +slots: + size: + range: float + label: {} +classes: + Thing: + class_uri: ex:Thing + slots: [size, label] + any_of: + - slot_conditions: + size: + equals_number: 5 + - slot_conditions: + label: + required: true +""" + ) + shacl_ttl = ShaclGenerator(schema, mergeimports=False, closed=False).serialize() + # sh:datatype xsd:float on the class's own property shape would reject an xsd:decimal, + # so every value is given as xsd:float + data = f""" + @prefix ex: . + @prefix xsd: . + ex:t a ex:Thing ; ex:size "{value}"^^xsd:float . + """ + assert _conforms(shacl_ttl, data) is expected + + +_UNTRANSLATABLE_CONDITIONS = { + # an identifier is the node's IRI; here it becomes one only through slot_usage + "identifier": """ +slots: + id: {} + a: {} +classes: + Thing: + class_uri: ex:Thing + slots: [id, a] + slot_usage: + id: + identifier: true + any_of: + - slot_conditions: + id: + required: true + - slot_conditions: + a: + required: true +""", + "'undefined', which is not a slot": """ +slots: + a: {} +classes: + Thing: + class_uri: ex:Thing + slots: [a] + any_of: + - slot_conditions: + undefined: + required: true + - slot_conditions: + a: + required: true +""", + "does not hold strings": """ +slots: + count: + range: integer + a: {} +classes: + Thing: + class_uri: ex:Thing + slots: [count, a] + any_of: + - slot_conditions: + count: + equals_string: "5" + - slot_conditions: + a: + required: true +""", +} + + +@pytest.mark.parametrize("reason", list(_UNTRANSLATABLE_CONDITIONS)) +def test_class_expression_untranslatable_condition_skipped_with_warning(caplog, reason): + """A condition on an identifier, on no slot, or equals_string on a non-string range is not translated.""" + import logging + + with caplog.at_level(logging.WARNING, logger="linkml.generators.shaclgen"): + g = _parse_shacl(_CLASS_EXPRESSION_HEADER + _UNTRANSLATABLE_CONDITIONS[reason]) + + assert (EX_CE.Thing, SH["or"], None) not in g + assert any("any_of" in rec.message and reason in rec.message for rec in caplog.records) + + +def test_class_expression_is_a_with_native_names_uses_sh_node(): + """With native names, is_a references the member class's shape, suffix included, as a range does.""" + schema = ( + _CLASS_EXPRESSION_HEADER + + """ +slots: + a: {} +classes: + Marker: + class_uri: ex:Marker + Thing: + class_uri: ex:Thing + slots: [a] + any_of: + - is_a: Marker + - slot_conditions: + a: + required: true +""" + ) + g = _parse_shacl(schema, use_class_uri_names=False, suffix="Shape") + + (first, _) = _list_members(g, EX_CE.ThingShape, SH["or"]) + assert set(g.objects(first, SH.node)) == {EX_CE.MarkerShape} + assert (first, SH["class"], None) not in g + assert (EX_CE.MarkerShape, RDF.type, SH.NodeShape) in g + + +def _condition_property_shapes(g: rdflib.Graph, shape: URIRef, path: URIRef) -> list: + """The property shapes on *path* that the logical constraints of *shape* reach.""" + own = set(g.objects(shape, SH.property)) + return [node for node in g.transitive_objects(shape, None) if (node, SH.path, path) in g and node not in own] + + +def _class_expression_shacl(classes: str, **kwargs) -> rdflib.Graph: + """The SHACL shapes graph of a class-expression schema with the given ``slots:`` / ``classes:`` YAML.""" + return _parse_shacl(_CLASS_EXPRESSION_HEADER + classes, **kwargs) + + +def test_class_expression_condition_title_names_the_property_shape(): + """A condition's title becomes the sh:name of its property shape.""" + g = _class_expression_shacl( + """ +slots: + a: {} + b: {} +classes: + Thing: + class_uri: ex:Thing + slots: [a, b] + any_of: + - slot_conditions: + a: + title: the a condition + required: true + - slot_conditions: + b: + required: true +""" + ) + (condition,) = _condition_property_shapes(g, EX_CE.Thing, EX_CE.a) + assert set(g.objects(condition, SH.name)) == {Literal("the a condition")} + + +@pytest.mark.parametrize("values,conforms", [('"a", "b"', True), ('"a", "b", "c"', False)]) +def test_class_expression_strictest_maximum_applies(values, conforms): + """With an exact and a maximum cardinality, the stricter bound applies.""" + shacl_ttl = _class_expression_shacl( + """ +slots: + tag: + multivalued: true +classes: + Thing: + class_uri: ex:Thing + slots: [tag] + all_of: + - slot_conditions: + tag: + exact_cardinality: 2 + maximum_cardinality: 3 +""" + ).serialize(format="turtle") + assert _conforms(shacl_ttl, f"{_CE_PREFIXES}ex:t a ex:Thing ; ex:tag {values} .") == conforms + + +@pytest.mark.parametrize("ref,conforms", [(None, True), ("ex:s", False), ("ex:o", True)]) +def test_class_expression_none_of_range(ref, conforms): + """A range in a condition is a value constraint: within none_of it requires + the slot in its "definitely true" form, so only a Special value violates.""" + shacl_ttl = _class_expression_shacl( + """ +slots: + ref: + range: Target +classes: + Target: + class_uri: ex:Target + Special: + class_uri: ex:Special + Thing: + class_uri: ex:Thing + slots: [ref] + none_of: + - slot_conditions: + ref: + range: Special +""", + closed=False, + ).serialize(format="turtle") + value = f" ; ex:ref {ref}" if ref else "" + data = f"{_CE_PREFIXES}ex:s a ex:Special, ex:Target . ex:o a ex:Target . ex:t a ex:Thing{value} ." + assert _conforms(shacl_ttl, data) == conforms + + +def test_class_expression_equals_string_uses_the_condition_range(): + """equals_string is rendered for the condition's own range: an enum value + with a meaning becomes that IRI.""" + g = _class_expression_shacl( + """ +enums: + FormatEnum: + permissible_values: + OSI: + meaning: ex:OSI +slots: + fileFormat: {} + other: {} +classes: + Format: + class_uri: ex:Format + slots: [fileFormat, other] + any_of: + - slot_conditions: + fileFormat: + range: FormatEnum + equals_string: OSI + - slot_conditions: + other: + required: true +""" + ) + (condition,) = _condition_property_shapes(g, EX_CE.Format, EX_CE.fileFormat) + in_lists = [list(Collection(g, node)) for node in g.objects(condition, SH["in"])] + in_lists += [ + list(Collection(g, node)) + for and_list in g.objects(condition, SH["and"]) + for member in Collection(g, and_list) + for node in g.objects(member, SH["in"]) + ] + assert in_lists and all(values == [EX_CE.OSI] for values in in_lists) + + +def test_class_expression_equals_string_on_a_custom_string_type(): + """A type derived from string holds strings, so equals_string on it is translated.""" + g = _class_expression_shacl( + """ +types: + Code: + typeof: string +slots: + code: + range: Code + a: {} +classes: + Thing: + class_uri: ex:Thing + slots: [code, a] + any_of: + - slot_conditions: + code: + equals_string: AB + - slot_conditions: + a: + required: true +""" + ) + assert (EX_CE.Thing, SH["or"], None) in g + + +def test_class_expression_builtin_range_without_types_import(): + """A built-in range name in a schema that doesn't import linkml:types gives its datatype.""" + schema = """ +id: https://example.org/class-expressions +name: class_expressions +prefixes: + ex: https://example.org/class-expressions/ +default_prefix: ex +slots: + size: {} + a: {} +classes: + Thing: + class_uri: ex:Thing + slots: [size, a] + any_of: + - slot_conditions: + size: + range: integer + - slot_conditions: + a: + required: true +""" + g = _parse_shacl(schema) + (condition,) = _condition_property_shapes(g, EX_CE.Thing, EX_CE.size) + assert (condition, SH.datatype, XSD.integer) in g + + +def test_class_expression_range_with_identifier_is_an_iri(): + """A condition ranged on a class with an identifier expects IRIs, as a slot's range does.""" + g = _class_expression_shacl( + """ +slots: + id: + identifier: true + ref: {} + a: {} +classes: + Target: + class_uri: ex:Target + slots: [id] + Thing: + class_uri: ex:Thing + slots: [ref, a] + any_of: + - slot_conditions: + ref: + range: Target + - slot_conditions: + a: + required: true +""" + ) + (condition,) = _condition_property_shapes(g, EX_CE.Thing, EX_CE.ref) + assert set(g.objects(condition, SH.nodeKind)) == {SH.IRI} + + +_UNTRANSLATABLE_MEMBERS = { + "untranslatable-second-member": ( + "has_member", + """ + - slot_conditions: + a: + required: true + - slot_conditions: + tags: + has_member: + equals_string: x""", + ), + "unknown-condition-range": ( + "unknown range", + """ + - slot_conditions: + a: + range: Undefined + - slot_conditions: + b: + required: true""", + ), + "untranslatable-nested-member": ( + "has_member", + """ + - all_of: + - slot_conditions: + tags: + has_member: + equals_string: x + - slot_conditions: + a: + required: true""", + ), + "is_a-not-a-class": ( + "not a class", + """ + - is_a: Undefined + - slot_conditions: + a: + required: true""", + ), + "equals_string-on-class-range": ( + "does not hold strings", + """ + - slot_conditions: + ref: + equals_string: x + - slot_conditions: + a: + required: true""", + ), + "equals_string_in-on-integer": ( + "does not hold strings", + """ + - slot_conditions: + count: + equals_string_in: ["5"] + - slot_conditions: + a: + required: true""", + ), +} + + +@pytest.mark.parametrize("case", list(_UNTRANSLATABLE_MEMBERS)) +def test_class_expression_untranslatable_member_skips_the_operator(caplog, case): + """An operator with an untranslatable member anywhere, at any depth or + position, is skipped as a whole, with a warning naming the reason.""" + reason, members = _UNTRANSLATABLE_MEMBERS[case] + schema = ( + _CLASS_EXPRESSION_HEADER + + """ +slots: + tags: + multivalued: true + ref: + range: Target + count: + range: integer + a: {} + b: {} +classes: + Target: + class_uri: ex:Target + Thing: + class_uri: ex:Thing + slots: [tags, ref, count, a, b] + any_of:""" + + members + + "\n" + ) + with caplog.at_level(logging.WARNING, logger="linkml.generators.shaclgen"): + g = _parse_shacl(schema) + assert (EX_CE.Thing, SH["or"], None) not in g + assert any("any_of" in rec.message and reason in rec.message for rec in caplog.records), caplog.text + + +_REPEATED_PARAMETERS_SCHEMA = ( + _CLASS_EXPRESSION_HEADER + + """ +types: + Code: + typeof: string + pattern: "^[A-Z]+$" +enums: + LetterEnum: + permissible_values: + A: {} + B: {} + C: {} +slots: + size: + range: integer + label: {} + letter: {} + code: {} + other: {} +classes: + Thing: + class_uri: ex:Thing + slots: [size, label, letter, code, other] + any_of: + - slot_conditions: + size: + minimum_value: 3 + equals_number: 5 + label: + equals_string: A + equals_string_in: [A, B] + letter: + range: LetterEnum + equals_string: B + code: + range: Code + pattern: "^AB" + - slot_conditions: + other: + required: true +""" +) + + +@pytest.mark.parametrize( + "properties,expected", + [ + ('ex:size 5 ; ex:label "A" ; ex:letter "B" ; ex:code "ABC"', True), + ("ex:size 4", False), + ('ex:label "B"', False), + ('ex:letter "A"', False), + ('ex:code "ABc"', False), + ('ex:code "XY"', False), + ('ex:size 4 ; ex:other "x"', True), + ], +) +def test_class_expression_repeated_parameters_stay_well_formed(properties, expected): + """A parameter SHACL allows once per shape, needed twice by one condition, goes into sh:and. + + _conforms runs pyshacl with meta_shacl, which fails on an ill-formed shapes graph. + """ + shacl_ttl = ShaclGenerator(_REPEATED_PARAMETERS_SCHEMA, mergeimports=False).serialize() + data = f""" + @prefix ex: . + ex:t a ex:Thing ; {properties} . + """ + assert _conforms(shacl_ttl, data) is expected + + +@pytest.mark.parametrize( + "extra,properties,expected", + [ + ("", "", True), + ("maximum_cardinality: 5", "", True), + ("minimum_cardinality: 0", "", True), + ("maximum_cardinality: 5", 'ex:tag "A"', False), + ("maximum_cardinality: 5", 'ex:tag "B"', True), + ], +) +def test_class_expression_none_of_presence_is_monotonic(extra, properties, expected): + """Inside none_of, a cardinality that an absent slot satisfies does not flip an absent slot to rejected.""" + schema = ( + _CLASS_EXPRESSION_HEADER + + f""" +slots: + tag: + multivalued: true +classes: + Thing: + class_uri: ex:Thing + slots: [tag] + none_of: + - slot_conditions: + tag: + equals_string: A + {extra} +""" + ) + shacl_ttl = ShaclGenerator(schema, mergeimports=False).serialize() + data = f""" + @prefix ex: . + ex:t a ex:Thing {";" if properties else ""} {properties} . + """ + assert _conforms(shacl_ttl, data) is expected diff --git a/tests/linkml/test_validator/test_validation_context.py b/tests/linkml/test_validator/test_validation_context.py index a674048952..e19c1bda48 100644 --- a/tests/linkml/test_validator/test_validation_context.py +++ b/tests/linkml/test_validator/test_validation_context.py @@ -647,9 +647,9 @@ def _schema_with_value_disallowed() -> SchemaDefinition: def test_cache_hit_preserves_value_disallowed_not_keyword(): - """A slot with value_presence=ABSENT produces a root-level `not` keyword. - Cache-hit must carry it through from defs_class so the validator rejects - instances that include the forbidden field.""" + """A slot with value_presence=ABSENT produces a root-level `not: {required}` + in `allOf`. Cache-hit must carry it through from defs_class so the + validator rejects instances that include the forbidden field.""" schema = _schema_with_value_disallowed() cold = ValidationContext(schema, "ForbidsX").json_schema_validator( @@ -661,8 +661,9 @@ def test_cache_hit_preserves_value_disallowed_not_keyword(): closed=False, include_range_class_descendants=False ) - assert "not" in cold.schema, cold.schema - assert "not" in warm.schema, "value_disallowed `not` keyword must survive cache-hit" + forbids = {"not": {"required": ["forbidden_field"]}} + assert forbids in cold.schema.get("allOf", []), cold.schema + assert forbids in warm.schema.get("allOf", []), "value_disallowed `not` keyword must survive cache-hit" # Instance includes the forbidden field -> both validators reject bad = {"forbidden_field": "anything", "other_field": "ok"} @@ -677,18 +678,19 @@ def test_cache_hit_preserves_value_disallowed_not_keyword(): def test_cache_hit_does_not_leak_value_disallowed_not_to_target_without(): - """Warming with ForbidsX places `not` at root in cached. Hitting cache for - Plain (which has no value_disallowed slots) must NOT inherit `not`.""" + """Warming with ForbidsX places `not: {required}` at root in cached. Hitting + cache for Plain (which has no value_disallowed slots) must NOT inherit it.""" schema = _schema_with_value_disallowed() cache_key = _make_cache_key(schema, include_range_class_descendants=False) ValidationContext(schema, "ForbidsX").json_schema_validator(closed=False, include_range_class_descendants=False) - assert "not" in _json_schema_cache[cache_key] + forbids = {"not": {"required": ["forbidden_field"]}} + assert forbids in _json_schema_cache[cache_key].get("allOf", []) validator = ValidationContext(schema, "Plain").json_schema_validator( closed=False, include_range_class_descendants=False ) - assert "not" not in validator.schema, validator.schema.get("not") + assert forbids not in validator.schema.get("allOf", []), validator.schema.get("allOf") # Plain has no constraint against `forbidden_field` — the leak would falsely # reject this. Confirm it doesn't.