json-semantic-diff

Semantic similarity scoring for JSON structures — not just whether they differ, but how similar they are.

Overview

json-semantic-diff compares two JSON documents and returns a normalised similarity score in [0.0, 1.0], along with an audit trail of which keys matched, which were renamed, and which are missing. It handles naming convention differences (camelCase, snake_case, PascalCase, kebab-case) transparently.

Traditional JSON diff tools are binary: match or no match. json-semantic-diff tells you how similar two documents are, which is what you actually need when testing LLM outputs, validating API migrations, or measuring generator stability.

Core algorithm: STED (Semantic Tree Edit Distance) — tree edit distance extended with semantic key matching via the Hungarian algorithm and per-level normalisation.

Key Features

Similarity scoring — Normalised float in [0.0, 1.0] instead of binary match/no-match
Naming convention tolerance — user_name, userName, UserName, user-name all score 1.0 against each other
Explainable scores — Per-path breakdown of what drove the score down: value changes (30 -> 31), key renames, keys only on one side
Rich audit trail — Key mappings, matched pairs, unmatched paths, computation time
Generator consistency — Single-number stability metric for LLM outputs
CLI included — json-semantic-diff left.json right.json, with --explain, --json, and CI-friendly --threshold exit codes
Three backends — Zero-dependency Levenshtein, local ONNX embeddings, or OpenAI cloud
Evaluation platform adapters — pytest, LangSmith, Braintrust, W&B Weave
Configurable — Structural/content weights, array comparison modes, type coercion, null handling
Zero global state — Each comparator instance owns its own cache; thread-safe by design

Installation

pip install json-semantic-diff

With optional backends:

pip install json-semantic-diff[fastembed]   # Local ONNX embeddings (384-dim)
pip install json-semantic-diff[openai]      # OpenAI cloud embeddings (1536-dim)

With evaluation platform adapters:

pip install json-semantic-diff[langsmith]
pip install json-semantic-diff[braintrust]
pip install json-semantic-diff[weave]

Quick Start

Compare two JSON documents

from json_semantic_diff import compare

# Note: `mailAddress` vs `email_address` is a fuzzy (not exact) key match,
# so the score is close to but below 1.0.
result = compare(
    {"user_name": "Alice", "email_address": "alice@corp.com"},
    {"userName": "Alice", "mailAddress": "alice@corp.com"},
)

print(result.similarity_score)   # ~0.99
print(result.key_mappings)       # {"user_name": "userName", "email_address": "mailAddress"}
print(result.unmatched_left)     # ()
print(result.unmatched_right)    # ()
print(result.computation_time_ms)  # sub-millisecond on small docs

# Naming-convention-only differences score a perfect 1.0:
compare(
    {"user_name": "Alice"},
    {"userName": "Alice"},
).similarity_score  # 1.0

Quick similarity score

from json_semantic_diff import similarity_score

# Pure naming-convention difference -> identical after normalisation.
score = similarity_score(
    {"first_name": "Bob", "last_name": "Smith"},
    {"firstName": "Bob", "lastName": "Smith"},
)
print(score)  # 1.0

Boolean equivalence check

from json_semantic_diff import is_equivalent

# Passes — naming convention difference only
is_equivalent({"user_name": "Alice"}, {"userName": "Alice"})  # True

# Fails — different structure entirely
is_equivalent({"name": "Alice"}, {"product": "Widget"})  # False

# Custom threshold
is_equivalent({"user_name": "Alice"}, {"userName": "Alice"}, threshold=0.99)

Explain why a score isn't 1.0

from json_semantic_diff import STEDConfig, compare, format_diff

config = STEDConfig(collect_explanation=True)
result = compare(
    {"user_name": "Alice", "age": 30, "debug": True},
    {"userName": "Alice", "age": 31},
    config=config,
)

# Programmatic access: per-path contributions, highest impact first
for c in result.explanation:
    print(c.path, c.kind, c.detail)
# /debug unmatched_left debug
# /age   value_mismatch 30 -> 31

# Or render the whole result as a report
print(format_diff(result))
# Similarity: 0.58  (computed in 0.2 ms)
#
# Matched (2):
#   age <-> age
#   user_name <-> userName
#
# Unmatched in left (1):
#   /debug
#
# Differences (2):
#   /debug: only in left
#   /age: 30 -> 31

Command line

The json-semantic-diff console script (also python -m json_semantic_diff) compares two files, or stdin via -:

json-semantic-diff left.json right.json          # prints the score
json-semantic-diff --explain left.json right.json  # human-readable report of what differs
json-semantic-diff --json left.json right.json   # full audit trail as JSON
cat left.json | json-semantic-diff - right.json

# CI gate: exit 0 when similarity >= threshold, 1 otherwise.
# Combine with --explain to see why a gate failed.
json-semantic-diff --threshold 0.95 --explain expected.json actual.json

Flags: --explain, --json, --threshold FLOAT, --verbose, --structural-weight, --content-weight, --array-mode {ordered,unordered,auto} (default ordered, matching the library default), --version.

Measure generator consistency

from json_semantic_diff import consistency_score

# Stable generator → 1.0
docs = [
    {"name": "Alice", "age": 30},
    {"name": "Alice", "age": 30},
    {"name": "Alice", "age": 30},
]
print(consistency_score(docs))  # 1.0

# Erratic generator → low score
erratic = [
    {"name": "Alice", "age": 30},
    {"fullName": "Alice", "years": 30},
    {"person": "Alice"},
]
print(consistency_score(erratic))  # ~0.32

API Reference

Public Functions

Function	Signature	Returns
`compare(left, right, config=None)`	Two JSON values + optional config	`ComparisonResult`
`similarity_score(left, right, config=None)`	Two JSON values + optional config	`float` in [0.0, 1.0]
`is_equivalent(left, right, threshold=0.85, config=None)`	Two JSON values + threshold + config	`bool`
`consistency_score(docs, config=None)`	List of JSON values + optional config	`float` in [0.0, 1.0]
`compare_batch(lefts, right, config=None)`	Many left values vs one right	`list[ComparisonResult]`
`compare_batch_pairs(pairs, config=None)`	List of (left, right) tuples	`list[ComparisonResult]`
`format_diff(result, indent=2)`	A `ComparisonResult`	Human-readable report `str`

ComparisonResult

Returned by compare(). Frozen dataclass with seven fields:

Field	Type	Description
`similarity_score`	`float`	Normalised similarity in [0.0, 1.0]. 1.0 = identical.
`matched_pairs`	`tuple[tuple[str, str], ...]`	JSON Pointer pairs for matched KEY nodes
`key_mappings`	`dict[str, str]`	Raw left key name → raw right key name
`unmatched_left`	`tuple[str, ...]`	JSON Pointer paths with no match in right document
`unmatched_right`	`tuple[str, ...]`	JSON Pointer paths with no match in left document
`computation_time_ms`	`float`	Wall-clock duration in milliseconds
`explanation`	`tuple[NodeContribution, ...]`	Per-path contributions, highest impact first. Populated when `STEDConfig(collect_explanation=True)`; empty tuple otherwise.

Each NodeContribution has path (JSON Pointer), contribution (raw distance — higher means it moved the score more), kind (value_mismatch, key_rename, unmatched_left, unmatched_right, or matched for a container whose subtree differs), and detail (e.g. 30 -> 31 for a value change, email_address -> mailAddress for a rename).

STEDConfig

Immutable configuration for the algorithm. All parameters have sensible defaults.

from json_semantic_diff import STEDConfig, ArrayComparisonMode

config = STEDConfig(
    w_s=0.5,                                          # Structural weight [0, 1]
    w_c=0.5,                                          # Content weight [0, 1] (must sum to 1.0)
    lambda_unmatched=0.5,                              # Penalty for unmatched children (>= 0)
    array_comparison_mode=ArrayComparisonMode.ORDERED,  # ORDERED | UNORDERED | AUTO
    type_coercion=False,                               # "42" == 42?
    null_equals_missing=False,                         # {x: null} == {}?
    ignore_paths=("/timestamp", "/users/*/id"),        # Drop volatile keys before comparing
    numeric_tolerance=0.0,                             # |a - b| <= tol counts as equal
    aliases=(("uid", "user_id"),),                     # Domain synonyms match at 1.0
    max_depth=None,                                    # Cap traversal depth (None = full)
    collect_explanation=False,                         # Populate result.explanation
)

result = compare(doc1, doc2, config=config)

Parameter	Type	Default	Description
`w_s`	`float`	`0.5`	Structural weight. Higher → structure matters more.
`w_c`	`float`	`0.5`	Content weight. Higher → values matter more. Must sum to 1.0 with `w_s`.
`lambda_unmatched`	`float`	`0.5`	Penalty per unmatched child. 0.0 = ignore extras. 1.0 = full penalty.
`array_comparison_mode`	`ArrayComparisonMode`	`ORDERED`	How arrays are compared.
`type_coercion`	`bool`	`False`	When True, `"42"` and `42` compare as equal.
`null_equals_missing`	`bool`	`False`	When True, `{"x": null}` and `{}` compare as equal.
`ignore_paths`	`tuple[str, ...]`	`()`	JSON Pointer patterns stripped from both inputs before comparison (timestamps, generated ids). `` matches one path component: `/users//id`.
`numeric_tolerance`	`float`	`0.0`	Absolute tolerance for numeric comparison. `abs(a - b) <= tol` counts as equal (bools excluded).
`aliases`	`tuple[tuple[str, str], ...]`	`()`	Key pairs forced to similarity 1.0, symmetric. Applied to raw and normalised labels.
`max_depth`	`int \| None`	`None`	Cap traversal depth. Subtrees past the cap contribute full cost unless identical. Faster on deep documents, less resolution past the cap.
`collect_explanation`	`bool`	`False`	When True, `result.explanation` carries the per-path breakdown of what drove the score. Off by default — no overhead on the hot path.

Array Comparison Modes

Mode	Strategy	Use case
`ORDERED`	Positional DP alignment	Sequences, logs, ordered lists
`UNORDERED`	Hungarian matching	Tags, feature flags, sets
`AUTO`	Infer from content (scalars → unordered, objects → ordered)	When array semantics vary

Backends

Three embedding backends for key matching, each optimising for different constraints:

Backend	Dependencies	Latency	Key matching quality	Cost
StaticBackend (default)	None	~0.1ms	Naming conventions	Free
FastEmbedBackend	`fastembed`	~1-2s cold start	Semantic understanding	Free
OpenAIBackend	`openai`, `tenacity`	~100-500ms	Best quality	API costs

StaticBackend (default)

Uses Levenshtein edit distance on normalised keys. No ML model, no API calls.

from json_semantic_diff.backends import StaticBackend

backend = StaticBackend()
backend.similarity("user_name", "userName")   # 1.0
backend.similarity("user_name", "address")    # < 0.5

FastEmbedBackend

Local ONNX embeddings via sentence-transformers/all-MiniLM-L6-v2 (384-dim).

from json_semantic_diff.backends.fastembed import FastEmbedBackend
from json_semantic_diff.comparator import STEDComparator

backend = FastEmbedBackend()  # ~1-2s cold start for model loading
cmp = STEDComparator(backend=backend)
result = cmp.compare(doc1, doc2)

OpenAIBackend

Cloud embeddings via text-embedding-3-small (1536-dim). API key from OPENAI_API_KEY environment variable.

from json_semantic_diff.backends.openai import OpenAIBackend
from json_semantic_diff.comparator import STEDComparator

backend = OpenAIBackend()  # Reads OPENAI_API_KEY from env
cmp = STEDComparator(backend=backend)
result = cmp.compare(doc1, doc2)

Auto-retries rate-limited requests with jittered exponential backoff (up to 6 attempts via tenacity).

Using STEDComparator directly

For batch comparisons where you want cache reuse across calls:

from json_semantic_diff.comparator import STEDComparator

cmp = STEDComparator()  # Or with backend= and config=
for doc in documents:
    result = cmp.compare(doc, reference)
    # Embedding cache reused across all calls

Integrations

pytest

Auto-discovered fixture — install the package and it's available in every test:

def test_api_response(assert_json_equivalent):
    actual = get_user_from_api()
    expected = {"user_name": "Alice", "age": 30}

    # Passes even if API returns {"userName": "Alice", "age": 30}
    assert_json_equivalent(actual, expected)

Custom threshold and config:

from json_semantic_diff import STEDConfig

def test_strict(assert_json_equivalent):
    config = STEDConfig(type_coercion=True)
    assert_json_equivalent(actual, expected, threshold=0.95, config=config)

Failure messages include full context:

AssertionError: JSON documents not equivalent:
  similarity=0.42 < threshold=0.85
  actual:   {"product": "Widget", "price": 19.99}
  expected: {"user_name": "Alice", "age": 30}
  key_mappings: {"product": "user_name"}
  unmatched_left: ["/price"]
  unmatched_right: ["/age"]

LangSmith

from json_semantic_diff.comparator import STEDComparator
from json_semantic_diff.integrations import LangSmithEvaluator

comparator = STEDComparator()
evaluator = LangSmithEvaluator(comparator, output_key="response")
# Pass directly to langsmith.evaluate()

Braintrust

from json_semantic_diff.comparator import STEDComparator
from json_semantic_diff.integrations import BraintrustScorer

comparator = STEDComparator()
scorer = BraintrustScorer(comparator)
# Returns float in [0.0, 1.0] or None when no expected value

W&B Weave

from json_semantic_diff.comparator import STEDComparator
from json_semantic_diff.integrations import WeaveScorer

comparator = STEDComparator()
scorer = WeaveScorer(comparator)
# Returns {"semantic_similarity": float}

How It Works

The STED Algorithm

JSON → Typed Tree. Each JSON value becomes a tree node with a type (OBJECT, KEY, ARRAY, ELEMENT, SCALAR). KEY nodes store both raw and normalised labels.
Key Normalisation. A five-pass regex pipeline converts all naming conventions to a canonical form: userName → "user name", user_name → "user name", UserName → "user name", user-name → "user name".
Hungarian Key Matching. For each pair of OBJECT nodes, a cost matrix is built using key similarity scores. The Hungarian algorithm finds the optimal key-to-key assignment.
Blended Cost. Each node comparison blends structural and content distance: cost = w_s * structural_distance + w_c * content_distance.
Per-Level Normalisation (Zhang-Shasha). At each structural tree level: similarity = 1.0 - min(1.0, [d_matched + λ·|n_left - n_right|] / max(n_left, n_right, 1)), where n_left / n_right are the sum of children subtree sizes on each side (not just the child count). This keeps scores in [0.0, 1.0] regardless of document depth and prevents deep value-subtree edits from binary-collapsing the score when the parent has few direct children.

Consistency Score Formula

pairwise_scores = [compare(docs[i], docs[j]).similarity_score for all unique (i, j) pairs]
consistency = max(0, mean(pairwise_scores) - std(pairwise_scores))

Penalises both low average similarity (different outputs) and high variance (erratic outputs).

Embedding Cache

All unique KEY labels are collected before the algorithm runs and embedded in a single batch call. The algorithm then runs entirely from cache — zero additional backend calls per comparison.

Each STEDComparator instance owns its own cache (no global state). The public API functions create a fresh comparator per call for isolation. Use STEDComparator directly for batch comparisons where cache reuse matters.

Development

git clone https://github.com/mokhld/json-semantic-diff.git
cd json-semantic-diff
poetry install --with dev

Running tests

poetry run pytest                    # All 661 tests
poetry run pytest -x                 # Stop on first failure
poetry run pytest tests/unit/        # Unit tests only
poetry run pytest tests/algorithm/   # Algorithm tests only

Type checking

poetry run mypy src/          # Strict mode

Linting

poetry run ruff check src/ tests/
poetry run ruff format src/ tests/

Project structure

src/json_semantic_diff/
├── __init__.py              # Public API exports
├── api.py                   # 4 public functions
├── comparator.py            # STEDComparator orchestrator
├── scorer.py                # ConsistencyScorer
├── result.py                # ComparisonResult dataclass
├── protocols.py             # EmbeddingBackend Protocol
├── cache.py                 # EmbeddingCache (LRU)
├── algorithm/
│   ├── config.py            # STEDConfig + ArrayComparisonMode
│   ├── sted.py              # STEDAlgorithm (recursive core)
│   ├── costs.py             # Insert/delete/update cost functions
│   ├── matcher.py           # Hungarian algorithm wrapper
│   └── normalizer.py        # Per-level similarity normalisation
├── backends/
│   ├── static.py            # Levenshtein (zero dependencies)
│   ├── fastembed.py         # ONNX (local embeddings)
│   └── openai.py            # OpenAI (cloud embeddings)
├── tree/
│   ├── nodes.py             # TreeNode + NodeType
│   ├── builder.py           # JSON → Tree conversion
│   └── normalizer.py        # Key normalisation pipeline
└── integrations/
    ├── _pytest_plugin.py    # pytest fixture (auto-discovered)
    ├── _langsmith.py        # LangSmith evaluator adapter
    ├── _braintrust.py       # Braintrust scorer adapter
    └── _weave.py            # W&B Weave scorer adapter

License

MIT

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
src/json_semantic_diff		src/json_semantic_diff
tests		tests
.gitignore		.gitignore
.pre-commit-config.yaml		.pre-commit-config.yaml
CHANGELOG.md		CHANGELOG.md
CLAUDE.md		CLAUDE.md
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
README.md		README.md
SECURITY.md		SECURITY.md
poetry.lock		poetry.lock
pyproject.toml		pyproject.toml

Latest commit

History

Folders and files

Repository files navigation