An application that allows SPARQL-based queries over biodiversity datasets using Darwin Core Conceptual Model semantics over Parquet-backed DuckDB views.
Biodiversity data is commonly published as Darwin Core Archives distributed across institutional repositories. The newly proposed Darwin Core Data Package (DwC-DP) format introduces additional semantics and flexibility, but also increased complexity in data integration and querying. Querying across multiple such datasets typically requires either centralising the data or negotiating heterogeneous APIs.
The Darwin Core Conceptual Model (DwC-CM), which is the basis for DwC-DP, is a highly interconnected data model. In this regard, it is well suited to graph representations, making the Resource Description Framework (RDF) a clean, intuitive and semantically-rich data model. However, transforming tabular datasets into RDF represents a considerable Extract, Transform, Load (ETL) process and raises deduplication concerns, as the data must be maintained in both a relational database and a triplestore.
This project takes a different approach: data tables contained in each DwC-DP are hosted as Parquet files on object storage. On demand, a materialised DuckDB database of views is assembled from the relevant files and exposed through a SPARQL interface via a Virtual Knowledge Graph (VKG). Datasets can then be queried using a lightweight Web Ontology Language (OWL) ontology based primarily on Darwin Core terms, without any ETL step or permanent data duplication.
- No data duplication. Datasets stay on object storage as Parquet files. The application builds DuckDB views directly over them rather than downloading or copying rows into a local database, so the underlying data exists in exactly one place.
- No ETL pipeline. Datasets are queryable as soon as they're registered, no extract-transform-load step or intermediate format conversion is required, just pointing the application at the relevant Parquet assets.
- Schema heterogeneity accommodation. Datasets with differing numbers of tables and columns can be queried uniformly, without having to pad the Parquet data with empty columns or tables.
- Entity- and relationship-based querying. SPARQL lets users think in terms of entities (e.g. occurrences, events, agents, etc.) and how they relate to one another, rather than reasoning about foreign keys and join conditions.
- Language-agnostic access. Queries are submitted over plain HTTP, so any language or tool capable of making HTTP requests (e.g. Python, JavaScript, R, curl, etc.) can interact with the application.
- Context-scoped, on-demand containers. Spatial, temporal, and license filters resolve the relevant datasets before a query runs, so each context spins up only the database views and container it actually needs.
The application brings the semantic expressivity of the RDF data model and the SPARQL Query Language to users, letting them declare exactly the data they need across related entities. For example, the following SELECT query retrieves occurrences of Antarctic lanternfish (Electrona antarctica) and their life stage, linked to material entities as evidence, along with the material entity's disposition, preparations, the event date, and the recording agent:
PREFIX dcterms: <http://purl.org/dc/terms/>
PREFIX dwc: <http://rs.tdwg.org/dwc/terms/>
PREFIX dwcdp: <http://rs.tdwg.org/dwcdp/terms/>
SELECT ?lifeStage ?eventDate ?eventType ?disposition ?preparations ?preferredAgentName ?agentType
WHERE {
?occ a dwc:Occurrence ;
dwc:scientificName "Electrona antarctica" ;
dwc:lifeStage ?lifeStage ;
dwcdp:happenedDuring ?evt ;
dwcdp:recordedBy ?agt .
?evt a dwc:Event ;
dwc:eventDate ?eventDate ;
dwc:eventType ?eventType .
?mat a dwc:MaterialEntity ;
dwc:disposition ?disposition ;
dwc:preparations ?preparations ;
dwcdp:evidenceFor ?occ ;
dwcdp:collectedDuring ?evt .
?agt a dcterms:Agent ;
dcterms:title ?preferredAgentName ;
dwc:agentType ?agentType .
}
LIMIT 10One solution to this query is shown below (data taken from the BROKE-West fish dataset):
graph LR
Occ1((dwc:Occurrence))
Evt1((dwc:Event))
Ment1((dwc:MaterialEntity))
Agt1((dcterms:Agent))
Lit1["Electrona antarctica"]
Lit2["Larvae"]
Lit3["2006-01-20"]
Lit4["Survey - device subunit"]
Lit5["pending accession into Institue of Natural Sciences' collection"]
Lit6["Ethanol"]
Lit7["Anton Van de Putte"]
Lit8["person"]
Occ1 -- dwc:scientificName --> Lit1
Occ1 -- dwc:lifeStage --> Lit2
Occ1 -- dwcdp:happenedDuring --> Evt1
Occ1 -- dwcdp:recordedBy --> Agt1
Evt1 -- dwc:eventDate --> Lit3
Evt1 -- dwc:eventType --> Lit4
Ment1 -- dwc:disposition --> Lit5
Ment1 -- dwc:preparations --> Lit6
Ment1 -- dwcdp:collectedDuring --> Evt1
Ment1 -- dwcdp:evidenceFor --> Occ1
Agt1 -- dcterms:title --> Lit7
Agt1 -- dwc:agentType --> Lit8
classDef Occurrence fill:#2e7d32,stroke:#1b1b1b,color:#ffffff;
classDef Event fill:#6a1b9a,stroke:#1b1b1b,color:#ffffff;
classDef MaterialEntity fill:#1565c0,stroke:#1b1b1b,color:#ffffff;
classDef Agent fill:#c62828,stroke:#1b1b1b,color:#ffffff;
class Occ1 Occurrence
class Evt1 Event
class Ment1 MaterialEntity
class Agt1 Agent
As this example illustrates, biodiversity data is inherently graph-structured, with rich relationships between occurrences, events, material entities, and agents that are difficult to represent in flat tables.
The full record for an occurrence (i.e. not just the variables declared in a SELECT clause) can be obtained in a machine-readable format with a DESCRIBE query using the resource's URI/IRI (see RFC 3986 and RFC 3987), such as:
PREFIX dwc: <http://rs.tdwg.org/dwc/terms/>
PREFIX dwcdp: <http://rs.tdwg.org/dwcdp/terms/>
DESCRIBE <https://biobang.org/occurrence/BROKE_WEST_RMT_031_RMT8_217697_Larvae>Queries are submitted as a JSON payload over HTTP to the /sparql endpoint. The SPARQL query should be contained in the query field of the JSON payload. The application supports all four SPARQL query forms (SELECT, ASK, CONSTRUCT and DESCRIBE). SELECT and ASK queries return results in SPARQL 1.1 Query Results JSON, whereas CONSTRUCT and DESCRIBE queries return an RDF graph in RDF 1.1 Turtle format. See the API reference for the full request/response specification.
The request body can also include bbox, temporal, and licenses fields to narrow which datasets are loaded before the query runs, restricting the result, for instance, to only datasets that consider South American records from 2000 to 2015 and published under CC-BY-NC-4.0. See the API reference for the full request/response specification.
Each generated context also produces a citations file listing the source datasets used, in line with the GBIF data user agreement and GBIF's citation guidelines.
A live instance of the application is running, covering around 50 datasets across various domains, downloaded from GBIF and converted to the exploded Darwin Core Data Package format described in the starter guide documentation. See the dataset information document for more information about the datasets available. It is fully queryable and requires no installation or deployment.
The live instance exposes the following services/endpoints:
- A SPARQL endpoint at: https://data.qcbs.ca/sparql
- A metadata catalog at: https://data.qcbs.ca/metadata-api
- An MCP server at: https://data.qcbs.ca/mcp
For example, the above query can be sent to the endpoint using curl with the following code:
curl -X POST https://data.qcbs.ca/sparql \
-H "Content-Type: application/json" \
-d '{"query": "PREFIX dcterms: <http://purl.org/dc/terms/> PREFIX dwc: <http://rs.tdwg.org/dwc/terms/> PREFIX dwcdp: <http://rs.tdwg.org/dwcdp/terms/> SELECT ?lifeStage ?eventDate ?eventType ?disposition ?preparations ?preferredAgentName ?agentType WHERE { ?occ a dwc:Occurrence ; dwc:scientificName \"Electrona antarctica\" ; dwc:lifeStage ?lifeStage ; dwcdp:happenedDuring ?evt ; dwcdp:recordedBy ?agt . ?evt a dwc:Event ; dwc:eventDate ?eventDate ; dwc:eventType ?eventType . ?mat a dwc:MaterialEntity ; dwc:disposition ?disposition ; dwc:preparations ?preparations ; dwcdp:evidenceFor ?occ ; dwcdp:collectedDuring ?evt . ?agt a dcterms:Agent ; dcterms:title ?preferredAgentName ; dwc:agentType ?agentType . } LIMIT 10"}'The returned results will follow the standard SPARQL 1.1 Query Results JSON format. The above command can be adapted to your favorite language and HTTP request library (e.g. Requests in Python) or command-line tool (e.g. Wget). The same request pattern applies to other query forms, only the query field's value needs to be changed.
More information about the considered datasets can be obtained by considering the metadata catalog, where full dataset EML data can be requested in JSON-LD format. For example, metadata about the BROKE-West dataset can be obtained by visiting https://data.qcbs.ca/metadata-api/dataset/broke-west-fish.
Additionally, the deployment also offers a SPARQL-LLM chatbot, which consists of an open-source LLM with access to the SPARQL endpoint through the built-in Model Context Protocol (MCP) server. Users may interact with it through the chat interface, which accepts natural language questions, builds and run the underlying SPARQL query for you and will answer back.
The application is fully containerized using Docker. As long as Docker and Docker Compose are installed, running the application is as simple as cloning the repository and starting the stack:
git clone https://github.com/QCBS/semantic-data-cloud
cd semantic-data-cloud
docker compose up --buildThe application exposes three services:
- The SPARQL proxy at: http://localhost:8000
- The EML metadata catalog at: http://localhost:7788
- The MCP server at: http://localhost:9000/mcp
To host your own datasets rather than connect to an existing bucket, see the starter guide for how to prepare and upload Darwin Core Data Packages.
Additional, more detailed documentation can be found in the docs/ directory:
- Architecture, describing the overall architecture, components, design of the application.
- API reference, describing the endpoint specification and request/response formats.
- Ontology and mappings, describing the Darwin Core OWL ontology and OBDA mapping conventions.
- MCP server, describing a natural language interface via the Model Context Protocol.
- Starter guide, for help regarding how to prepare and host Darwin Core Data Packages for use with the application.
- Datasets info, for a brief summary of the datasets contained in the live instance of the application.