Introduce STAMPED as operational companion to FAIR

book/HPC.md

@@ -421,6 +421,8 @@ Python 3.14.2
 
 We could also set this as a default using `module save`, and the next time we log in we would see that this module is loaded.  If there is a package that you need that isn't in the current list of packages, a friendly email to the help desk is usually sufficient to get it installed as a module.
 
+Loading explicit module versions with `module load python/3.14.2`, rather than relying on whatever default happens to be installed, is one form of what the STAMPED framework [@Macdonald:2026aa] calls **T**racking: what the job runs against is spelled out, not implicit.
+
 ### Virtual environments
 
 Throughout the book I have talked about the utility of virtual environments, and they are commonly used on HPC systems to can access to packages or package versions that are not available as modules on the system. There is, however, one issue that should be kept in mind when using virtual environments in the HPC context.  When we install a virtual environment, the environment folder contains all of the dependencies that are installed in the environment. For some projects this can end up being quite large, to the degree that one can run into disk quota issues if they are stored in the home directory. For example, the full Anaconda installation is almost 10GB, which would largely fill the 15 GB quota for my home directory on the local HPC system; for this reason, I always recommend using miniconda which is a more minimal installation.  `uv` does a better job of caching but its local cache directory can also get very large over many projects.  For this reason, I we generally install Conda-based environments outside of the home directory, on a filesystem that has a larger quota.  When using `uv`, we generally set the `$UV_CACHE_DIR` environment variable to a location with a larger quota as well.  
@@ -429,7 +431,9 @@ Throughout the book I have talked about the utility of virtual environments, and
 
 Containers solve at least two important problems for HPC users.  First, as I noted in the earlier discussion, they provide a strong platform for reproducible computing; given that the HPC user has no control over operating system upgrades, containers help ensure that software dependencies will remain consistent over time.  Second, they allow users to install software that may not be runnable on the HPC system; for example, the user may require a version of a package that is too old to run on the HPC's current operating system.  For these reasons, containers have become very popular on HPC systems. 
 
-However, there is a rub: Because Docker requires that the user have root access to the system, it can't be run by users on HPC systems (which restrict root access to sysadmnins).  Instead, most HPC systems support Apptainer (formerly called Singularity) as a platform for running containers.  However, Apptainer can't actually build the container; it can only use existing containers, defined through an image file or downloaded from an image registry such as Dockerhub.  In many cases, there are existing containers on Dockerhub that have the appropriate software installed, which can be called directly from Apptainer. However, if no suitable pre-existing container image is available, the user will need to first generate a container using Docker on a system where they have administrative access (usually their own personal computer), and then convert the Docker image into an Apptainer image for use on the HPC system.  This can lead to long debugging cycles if there are problems with the container that require rebuilding, but once it's working it should continue working in the long term.  
+However, there is a rub: Because Docker requires that the user have root access to the system, it can't be run by users on HPC systems (which restrict root access to sysadmnins).  Instead, most HPC systems support Apptainer (formerly called Singularity) as a platform for running containers.  However, Apptainer can't actually build the container; it can only use existing containers, defined through an image file or downloaded from an image registry such as Dockerhub.  In many cases, there are existing containers on Dockerhub that have the appropriate software installed, which can be called directly from Apptainer. However, if no suitable pre-existing container image is available, the user will need to first generate a container using Docker on a system where they have administrative access (usually their own personal computer), and then convert the Docker image into an Apptainer image for use on the HPC system.  This can lead to long debugging cycles if there are problems with the container that require rebuilding, but once it's working it should continue working in the long term.
+
+Running each HPC job from a container image, and against a scratch directory that will be cleared after the job, also exercises the STAMPED **E**phemerality principle: the environment is built from an explicit specification, used, and then discarded, so a successful run confirms the specification was complete rather than depending on hidden state accumulated on a long-lived machine.
 
 ### Interactive access to HPC systems
 
@@ -469,4 +473,4 @@ In addition to the HPC resources available at many research institutions, anothe
 
 Benefits of commercial cloud computing are that the resources are generally available immediately, and that the user has complete control and root access to the resources.  The downside of commercial cloud computing is that it can become very expensive very quickly.  Every component costs money: computing time, data storage, and data egress, as well as other features (e.g. database hosting).  There are ways to reduce costs, such as using pre-emptible resources (which may be interrupted), but these systems can still be very costly for large data/compute needs.  
 
-We have used cloud computing for various projects, and it can be very useful to obtain specific kinds of compute resources quickly.  However, I generally feel that pay-to-play is a bad model for scientific computing, because it inhibits the kind of creativity and exploration that I think scientists need to be able to engage in.  When every computing cycle costs money, researchers are less likely to play and possible discover something new.  That said, they are an important tool in the computational toolbox, and can often help solve problems that would be intractable using on-prem HPC resources.
+We have used cloud computing for various projects, and it can be very useful to obtain specific kinds of compute resources quickly.  However, I generally feel that pay-to-play is a bad model for scientific computing, because it inhibits the kind of creativity and exploration that I think scientists need to be able to engage in.  When every computing cycle costs money, researchers are less likely to play and possible discover something new.  That said, they are an important tool in the computational toolbox, and can often help solve problems that would be intractable using on-prem HPC resources.

book/data_management.md

@@ -47,6 +47,23 @@ Data are reusable if the requirements for reuse are clearly specified.  Making d
 The FAIR principles are relatively abstract, in the sense that they don't provide specific guidance about what FAIR means in any particular domain.  However, there are numerous resources that can help implement these principles, such as [RDMKit](https://rdmkit.elixir-europe.org/) and the [FAIR Cookbook](https://faircookbook.elixir-europe.org/content/home.html), both generated by the European ELIXIR organization.
 
 
+### STAMPED: an operational companion to FAIR
+
+Where FAIR focuses on the discovery and governance of digital objects, the STAMPED principles {cite:p}`Macdonald:2026aa` address a complementary layer: the day-to-day practices that determine whether a research object can actually be sensibly shared, re-executed, and potentially extended by others.  Seven principles are defined:
+
+- **S**elf-containment: a research object is a complete retrieval unit; every component needed to reproduce its computational results is reachable from a single top-level object (the "don't look up" rule — no reliance on implicit external state).
+- **T**racking: the state and provenance of every component are recorded, typically via versioning manifests, content-addressed identifiers (checksums), and code-driven provenance records.
+- **A**ctionability: the object contains machine-actionable instructions to reproduce its results — executable specifications, not just documentation (albeit with AI, good documentation becomes actionable as well).
+- **M**odularity: components are organized as independent, composable modules, each carrying its own license, so they can be reused, replaced, or evolved on independent schedules (code often has a different life cycle than the data it processes).
+- **P**ortability: computational environments are explicitly specified and version-controlled; procedures do not depend on undocumented host state.
+- **E**phemerality: computational results are produced in throwaway environments — staged in a temporary location, run there, then discarded — so that a successful run confirms the environment was fully specified.
+- **D**istributability: all referenced modules are persistently retrievable by others, with resolvable license identifiers, across repositories and portals used in scientific practice.
+
+Each principle is expressed as a graduated set of requirements (MUST for the practical minimum, SHOULD and MAY for progressively more aspirational practices), and the framework is accompanied by an [interactive checklist](https://checklist.stamped-principles.org/) that a project can use to self-assess and communicate its operational maturity.  Much of the rest of this book puts STAMPED into practice: the *DataLad*-based tracking discussed later in this chapter, the containerized environments and structured project layouts in later chapters, and the workflow and CI patterns for reproducible re-execution.
+
+Full disclosure: STAMPED was co-developed by Yaroslav O. Halchenko, who also contributed to this book.
+
+
 ## The data lifecycle
 
 An important concept in research data management is the *data lifecycle*, which describes the role of data management in each of the different stages of a research project.  [](#lifecycle-fig) shows an example of how the [RDMkit project](https://rdmkit.elixir-europe.org/data_life_cycle) outlines the stages of the data lifecycle. This figure highlights the fact that data management should be part of the discussion at each stage in a project.  In this chapter I will discuss several of the stages in the data lifecycle in detail, though I leave in-depth discussion of data processing and analysis workflows to a later chapter.
@@ -953,7 +970,7 @@ However,*git* does not work well for version control on larger datasets using bi
 
 ### Using *DataLad* for version control on larger datasets
 
-A solution to this problem is to use a version control tool that is specifically designed for large data.  There are several tools that address this problem; we will focus on [Dat*aLad](https://www.datalad.org/), which is a data management system that functions very similarly to git.  It is based on a tool called [*git-annex*](https://git-annex.branchable.com/), but provides much greater ease of use for researchers.  (Full disclosure: Our group collaborates with the *DataLad* group and our grants have supported some of their development work.)
+A solution to this problem is to use a version control tool that is specifically designed for large data.  There are several tools that address this problem; we will focus on [*DataLad*](https://www.datalad.org/) [@Halchenko:2021aa], which is a data management system that functions very similarly to git.  It is based on a tool called [*git-annex*](https://git-annex.branchable.com/), but provides much greater ease of use for researchers.  (Full disclosure: Our group collaborates with the *DataLad* group and our grants have supported some of their development work.)  What *DataLad* provides — content-addressed identification of every component, plus code-driven provenance of every modification — is what the STAMPED framework calls the **T**racking principle in practice: two datasets can only be called "identical" if their content hashes match, not merely their labels.
 
 An important note: *DataLad* is quite powerful but has a significant learning curve, and takes a bit of time to get accustomed to.  In particular, its use of symbolic links can sometimes confuse new users. Having said that, let's look at some simple examples.
 

book/extras.md

@@ -573,4 +573,21 @@ It’s a perfect example:
 Serial: Too slow for 10^ 6 iterations.
 Vectorized: Might run out of memory.
 Parallel: The "Goldilocks" solution where you send chunks of iterations to different cores.
-This provides a narrative arc: You start with the math, realize the serial version is too slow, try vectorizing but hit a memory limit, and finally arrive at parallelization as the engineering solution.
+This provides a narrative arc: You start with the math, realize the serial version is too slow, try vectorizing but hit a memory limit, and finally arrive at parallelization as the engineering solution.
+
+
+## STAMPED principles ↔ book chapters mapping
+
+For readers who want to use the STAMPED framework [@Macdonald:2026aa] to self-assess their own projects, the table below indicates where each of the seven principles is put into practice in this book, together with representative items from the [interactive checklist](https://checklist.stamped-principles.org/).  Checklist item IDs (`must/NNN`, `should/NNN`, `may/NNN`) come from `stamped-checklist` v0.1.0.
+
+| Principle | Where in this book | Representative checklist items |
+|---|---|---|
+| **S**elf-containment | Project structure chapter (§ "Should code and data live alongside one another?"); Data management chapter (§ *DataLad*) | `must/001`–`must/003` (all files under a common root; datasets and external software reachable from that root) |
+| **T**racking | Data management chapter (§ *DataLad* for version control on larger datasets); Software engineering chapter (Git workflow) | `must/004`, `must/005` (Git for code and text; `git-annex` / *DataLad* / Git-LFS for large data); `must/006` (env specs in provenance records); `should/001` (one VCS across components) |
+| **A**ctionability | Workflow chapter (§ Using a workflow engine); Sharing chapter (§ Software metadata — `CITATION.cff`, `codemeta.json`) | `must/007`, `must/008` (`README.md` or `Makefile` with usage; a clear starting point); `should/002` (workflow tested regularly) |
+| **M**odularity | Project structure chapter (§ project structure, BIDS folder-naming); data-vs-code separation in the data management and sharing chapters | `should/003` (raw data / processed data / code / env separated into distinct modules); `may/001`–`may/005` (submodules, modular boundaries, composition mechanisms) |
+| **P**ortability | Project structure chapter (§ "It needs to be portable", § Containers); HPC chapter (§ Modules, § Containers); Sharing chapter (env-sharing sections) | `must/009`–`must/011` (relative paths; deps in env spec; host assumptions documented); `must/012` (system requirements in `README`); `must/013`, `must/014` (env specs in VCS, updated as deps change) |
+| **E**phemerality | Workflow chapter (§ Reproducible environments with containers); HPC chapter (§ Containers, per-job scratch directories) | `should/004`, `should/005` (pipeline tested in a fresh container / disposable env per execution) |
+| **D**istributability | Sharing chapter (§ Persistent identifiers for code; § Choosing a data repository); Data management chapter (*DataLad* remotes and portals) | `must/015`–`must/017` (env specs persistent, shareable, documented); `should/006`–`should/008` (env artifacts regularly re-tested and archived) |
+
+Item numbers reflect the checklist at time of writing; verify current numbering at <https://checklist.stamped-principles.org/> or <https://github.com/stamped-principles/stamped-checklist-schema> before printing or citing.

book/introduction.md

@@ -176,6 +176,11 @@ It is increasingly common for researchers to share both code and data from their
 However, in our experience, it can be very difficult to actually run the shared code, due to various problems that limit the portability of the code.
 Throughout this book I will discuss the tools and techniques that can help improve the portability of shared code and thus increase the reproducibility of published results.
 
+Two frameworks give shape to this discussion.
+The FAIR principles (Findable, Accessible, Interoperable, Reusable) [@Wilkinson:2016aa] structure the discovery and governance layer of research-object sharing; we return to them in the data management chapter.
+STAMPED [@Macdonald:2026aa] is a recent operational companion: research objects should be **S**elf-contained, **T**racked, **A**ctionable, **M**odular, **P**ortable, **E**phemeral, and **D**istributable, with an [interactive checklist](https://checklist.stamped-principles.org/) for self-assessment.
+Much of what this book teaches — version control, containers, structured project layouts, and reproducible workflows — facilitate FAIR and STAMPED practices.
+
 ### Bug-hacking
 
 A particular concern is that not all software errors are created equal.

book/project_organization.md

@@ -26,7 +26,7 @@ As I discussed in the section in Chapter 3 on clean coding, one of the most impo
 
 ### It needs to be portable
 
-It's rare for one to perform analyses that are only meant to run on one specific computer system.  Coding portably (as discussed in Chapter 3) makes it easy to run the code on other machines.  This can be useful, for example, when one replaces one's laptop, or when one needs to scale their code to run on a high-performance computing system.  It also helps ensure that the code can be tested using automated testing tools, like those discussed in Chapter 4.  
+It's rare for one to perform analyses that are only meant to run on one specific computer system.  Coding portably (as discussed in Chapter 3) makes it easy to run the code on other machines.  This can be useful, for example, when one replaces one's laptop, or when one needs to scale their code to run on a high-performance computing system.  It also helps ensure that the code can be tested using automated testing tools, like those discussed in Chapter 4.  Portability, expressed as explicitly specifying environments and avoiding hidden dependencies on the host, is what the STAMPED framework [@Macdonald:2026aa] calls the **P**ortability principle; the container-based approach discussed later in this chapter is one of the strongest ways to achieve it.
 
 
 ## Project structure
@@ -45,6 +45,8 @@ For my projects with datasets larger than a few gigabytes, I tend to keep data s
 
 In general, for portability it's also nice to have the data location parameterized in the code (e.g., via a .env file or local config file) rather than hardcoded through the use of a local directory name.  Thus, even if you decide to put the data within the code directory, it's good to write the code in a way that can allow the data to live in an arbitrary location.
 
+Keeping code and data as separately versioned modules is an instance of STAMPED-**M**odularity: code often changes on a different schedule than the curated data it processes, so pinning them to independent identifiers pays off later.  At the same time, ensuring that everything needed to reproduce the project is reachable from a single top-level object — even when the data physically live elsewhere on the filesystem — is what STAMPED calls **S**elf-containment (the "don't look up" rule: no reliance on implicit external state).  Tools such as *DataLad* (discussed in the data management chapter) make it practical to hold code and large data together in one logical project while keeping their storage separate.
+
 ### Folder structure
 
 A consistent and rational folder structure is key to good project organization.  For a simple Python project, I recommend starting using the package organization provided by `uv`.