Documented db corruption analysis and recovery procedures by daravep · Pull Request #1539 · canton-network/cf-docs

daravep · 2026-09-03T15:22:43Z

Before this pr, we didn't explain how to detect and recover broken databases. Now, we have a summary from past experience.

mintlify · 2026-09-03T15:22:46Z

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
cantonfoundation	🟢 Ready	View Preview	Sep 3, 2026, 3:25 PM

💡 Tip: Enable Automations to automatically generate PRs for you.

mziolekda · 2026-09-03T15:37:52Z

+
+# Use a Professional Cloud Operator
+
+While there are procedures to recover a node's data from the network, they are rather complex (there is an incoming improvement with GetRemoteAcs in 3.7), time-consuming, and only allow you to recover the current state, not history. The simpler approach is therefore to avoid performing these procedures by not corrupting your data.


Suggested change

While there are procedures to recover a node's data from the network, they are rather complex (there is an incoming improvement with GetRemoteAcs in 3.7), time-consuming, and only allow you to recover the current state, not history. The simpler approach is therefore to avoid performing these procedures by not corrupting your data.

While there are procedures to recover a node's data from the network, they are rather complex, time-consuming, and only allow you to recover the current state, not history. The simpler approach is therefore to avoid performing these procedures by not corrupting your data.

We will not remember to change it back

mziolekda · 2026-09-03T15:48:27Z

+
+While there are procedures to recover a node's data from the network, they are rather complex (there is an incoming improvement with GetRemoteAcs in 3.7), time-consuming, and only allow you to recover the current state, not history. The simpler approach is therefore to avoid performing these procedures by not corrupting your data.
+
+Operating a database is not trivial, especially when dealing with faults and issues. The simplest way is to rely on cloud-based SQL services offered by all cloud providers, and additionally export the database and store it securely in a second location (ideally encrypted).


Suggested change

Operating a database is not trivial, especially when dealing with faults and issues. The simplest way is to rely on cloud-based SQL services offered by all cloud providers, and additionally export the database and store it securely in a second location (ideally encrypted).

Operating a database is not trivial, especially when handling faults and operational issues. The simplest approach leverages cloud-native SQL services offered by all cloud providers, complemented by exporting the database and storing a secure and encrypted backup in a secondary location.

mziolekda · 2026-09-03T15:50:32Z

+The following action items may help you to prevent data corruption.
+
+- **Prevent Duplicate Volume Access:** If two Postgres instances access the same data volume, they can corrupt the data directory. Postgres does not auto-detect this situation in containerized deployments (as each process runs in its own namespace with pid=1). Therefore, you must ensure that this doesn’t happen at the orchestration and operations layers!
+In Kubernetes, use StatefulSets and use at least ReadWriteOnce (or even better ReadWriteOncePod) to protect the volume.


Why not say use ReadWriteOncePod full stop?

Because we don't use ReadWriteOncePod either.

mziolekda · 2026-09-03T15:54:11Z

+In Kubernetes, use StatefulSets and use at least ReadWriteOnce (or even better ReadWriteOncePod) to protect the volume.
+If you are using Docker Compose, be very careful when manipulating compose.yaml or changing volume and Postgres service names. Always check what is running before operating the cluster, and manually shut down any existing ones before starting new containers. Never run with `--scale=2` and similar.
+
+- **Ensure Postgres Shuts Down Cleanly:** Postgres is designed to recover from crashes and sudden system shutdowns. However, recovering from a crash delays startup and therefore the restart time, and increases the risk of data corruption by triggering bugs in either the containerization, the file system, or the disk driver. It may also be a sign of duplicate access to the volume. Therefore, monitor the postgres.log of your database, and check whether Postgres is invoking crash recovery, which is indicated by the following log line:


Suggested change

- **Ensure Postgres Shuts Down Cleanly:** Postgres is designed to recover from crashes and sudden system shutdowns. However, recovering from a crash delays startup and therefore the restart time, and increases the risk of data corruption by triggering bugs in either the containerization, the file system, or the disk driver. It may also be a sign of duplicate access to the volume. Therefore, monitor the postgres.log of your database, and check whether Postgres is invoking crash recovery, which is indicated by the following log line:

- **Ensure Postgres Shuts Down Cleanly:** Postgres is designed to recover from crashes and sudden system shutdowns. However, recovering from a crash delays startup and lengthens the overall restart time. That in turn increases the risk of data corruption by triggering bugs in either the containerization, the file system, or the disk driver. It may also trigger the edge cases leading to duplicate access to the volume. Therefore, monitor the postgres.log of your database, and check whether Postgres is invoking crash recovery, which is indicated by the following log line:

No. It's not that the overall restart time increases risk of data corruption.

Also, if you manually happen to start a second container due to mismanipulation, that second container would report performing crash recovery (as the other node is still writing and has not cleanly shut down the storage).

mziolekda · 2026-09-03T15:57:45Z

+
+The only case where a database might be repaired is when some indexes on auxiliary tables are broken. In this case, a simple REINDEX TABLE <tablename> will rebuild the database index. This might be an option in case the issue affects a section of the database where errors cannot propagate. Note that broken indexes may result from hardware failure; as such, further investigation is necessary.
+
+# Final Summary


We do not summarize the whole article here. I would rephrase it to

Suggested change

# Final Summary

# Final Takeaway

mziolekda · 2026-09-03T15:59:17Z

+
+# Final Summary
+
+There has been no reported case of database corruption in Canton operated on Cloud SQL. The same cannot be said for databases run in containers, whether on K8S or Docker Compose setups.


Suggested change

There has been no reported case of database corruption in Canton operated on Cloud SQL. The same cannot be said for databases run in containers, whether on K8S or Docker Compose setups.

There has been no reported case of database corruption in Canton operated on cloud SQL. The same cannot be said for databases run in containers, whether on K8S or Docker Compose setups. Therefore, if you want to avoid troubles use a cloud-native SQL solution.

brianweir-da · 2026-09-04T12:36:48Z

+
+The following action items may help you to prevent data corruption.
+
+- **Prevent Duplicate Volume Access:** If two Postgres instances access the same data volume, they can corrupt the data directory. Postgres does not auto-detect this situation in containerized deployments (as each process runs in its own namespace with pid=1). Therefore, you must ensure that this doesn’t happen at the orchestration and operations layers!


I believe you mean pod, not namespace in this regard?

Suggested change

- **Prevent Duplicate Volume Access:** If two Postgres instances access the same data volume, they can corrupt the data directory. Postgres does not auto-detect this situation in containerized deployments (as each process runs in its own namespace with pid=1). Therefore, you must ensure that this doesn’t happen at the orchestration and operations layers!

- **Prevent Duplicate Volume Access:** If two Postgres instances access the same data volume, they can corrupt the data directory. Postgres does not auto-detect this situation in containerized deployments (as each process runs in its own pod with pid=1). Therefore, you must ensure that this doesn’t happen at the orchestration and operations layers!

brianweir-da · 2026-09-04T12:41:47Z

+
+# Repairing Data Corruption
+
+There are several ways to recover from disasters, documented already here. https://docs.canton.network/global-synchronizer/production-operations/validator-disaster-recovery and global-synchronizer/production-operations/disaster-recovery


Suggested change

There are several ways to recover from disasters, documented already here. https://docs.canton.network/global-synchronizer/production-operations/validator-disaster-recovery and global-synchronizer/production-operations/disaster-recovery

There are several ways to recover from disasters. Please review the documentation https://docs.canton.network/global-synchronizer/production-operations/validator-disaster-recovery and https://docs.canton.network/global-synchronizer/production-operations/disaster-recovery.

Before this pr, we didn't explain how to detect and recover broken databases. Now, we have a summary from past experience. Signed-off-by: Ratko Veprek <ratko@digitalasset.com>

Signed-off-by: Ratko Veprek <ratko@digitalasset.com>

martinflorian-da

LGTM, thank you!

martinflorian-da · 2026-09-09T11:43:40Z

+
+# Use a Professional Cloud Operator
+
+While there are procedures to recover a node's data from the network, they are rather complex, time-consuming, and only allow you to recover the current state, not history. The simpler approach is therefore to avoid performing these procedures by not corrupting your data.


Suggested change

While there are procedures to recover a node's data from the network, they are rather complex, time-consuming, and only allow you to recover the current state, not history. The simpler approach is therefore to avoid performing these procedures by not corrupting your data.

While there are [procedures](/global-synchronizer/production-operations/validator-disaster-recovery) to recover a node's data from the network, they are rather complex, time-consuming, and only allow you to recover the current state, not history. The simpler approach is therefore to avoid performing these procedures by not corrupting your data.

nit as I see you also link it below

martinflorian-da · 2026-09-09T11:45:53Z

+
+While there are procedures to recover a node's data from the network, they are rather complex, time-consuming, and only allow you to recover the current state, not history. The simpler approach is therefore to avoid performing these procedures by not corrupting your data.
+
+Operating a database is not trivial, especially when handling faults and operational issues. The simplest approach leverages cloud-native SQL services offered by all cloud providers, complemented by exporting the database and storing an encrypted backup in a secure secondary location.


Suggested change

Operating a database is not trivial, especially when handling faults and operational issues. The simplest approach leverages cloud-native SQL services offered by all cloud providers, complemented by exporting the database and storing an encrypted backup in a secure secondary location.

Operating a database is not trivial, especially when handling faults and operational issues. The simplest approach leverages cloud-native SQL services offered by all cloud providers, complemented by exporting the database and storing an encrypted [backup](/global-synchronizer/production-operations/validator-backups) in a secure secondary location.

nit as I see you also link it below

martinflorian-da · 2026-09-09T11:47:21Z

+- **Ensure Postgres Shuts Down Cleanly:** Postgres is designed to recover from crashes and sudden system shutdowns. However, recovering from a crash delays startup and therefore the restart time, and increases the risk of data corruption by triggering bugs in either the containerization, the file system, or the disk driver. It may also be an indication of duplicate access to the volume. Therefore, monitor the postgres.log of your database, and check whether Postgres is invoking crash recovery, which is indicated by the following log line:
+`database system was not properly shut down; automatic recovery in progress`
+
+- **Checksums:** Make sure that Postgres checksums are enabled. Use PSQL to check `SHOW data_checksums`. Alternatively, use `SELECT name, setting, source FROM pg_settings WHERE name IN ('fsync','full_page_writes','synchronous_commit', 'wal_sync_method','data_checksums');` to inspect various consistency-related settings of your Postgres instance.


FYI we plan to make this hard to miss: canton-network/splice#6855

martinflorian-da · 2026-09-09T11:54:12Z

+
+- **Use Professional Hardware and Software:** Invest in your hardware and software setup, ensuring you use ECC memory, file systems, and hard drives resilient to data degradation.
+
+- **Back up your Database:** Use `pg_dump` to back up your database regularly, rather than relying on file-system backups. A file-system backup is only possible if you can take a snapshot of the filesystem. A copy of the files will not be synchronized and therefore will be corrupted. Furthermore, Postgres offers many different ways to perform backups, even at scale (e.g., [WAL copying for point-in-time recovery](https://www.postgresql.org/docs/current/continuous-archiving.html)).


Suggested change

- **Back up your Database:** Use `pg_dump` to back up your database regularly, rather than relying on file-system backups. A file-system backup is only possible if you can take a snapshot of the filesystem. A copy of the files will not be synchronized and therefore will be corrupted. Furthermore, Postgres offers many different ways to perform backups, even at scale (e.g., [WAL copying for point-in-time recovery](https://www.postgresql.org/docs/current/continuous-archiving.html)).

- **Back up your Database:** Use `pg_dump` to back up your database regularly, rather than relying on file-system backups. A file-system backup is only possible if you can take a snapshot of the filesystem. A copy of the files will not be synchronized and therefore will be corrupted. Furthermore, Postgres offers many different ways to perform backups, even at scale (e.g., [WAL copying for point-in-time recovery](https://www.postgresql.org/docs/current/continuous-archiving.html)). See [Validator Backups](/global-synchronizer/production-operations/validator-backups) for more details, and specifically note the ordering requirement between participant and app database backups (within the same backup set, the app database backup must be taken at a strictly earlier point in time).

martinflorian-da · 2026-09-09T11:56:26Z

+- **Prune Your System:** Recovering a database of a few GBs is faster and simpler than recovering a 1TB database. Therefore, ensure that [pruning of your node](/appdev/faq#how-do-i-enable-pruning-on-my-validator)  is enabled and keep your active contract set small (e.g., UTXO management).
+Monitor the metric `daml.pruning.max-event-age` ([see reference](/global-synchronizer/reference/canton-metrics#daml-pruning-max-event-age)) to verify that your system is getting pruned. You will also save on storage costs.
+
+- **Verify your Backup:** Postgres offers a tool (pg_verifybackup https://www.postgresql.org/docs/current/app-pgverifybackup.html ) to verify your backups. There is a reason that this tool exists. Therefore, you should use it.


Suggested change

- **Verify your Backup:** Postgres offers a tool (pg_verifybackup https://www.postgresql.org/docs/current/app-pgverifybackup.html ) to verify your backups. There is a reason that this tool exists. Therefore, you should use it.

- **Verify your Backup:** Postgres offers a tool (pg_verifybackup https://www.postgresql.org/docs/current/app-pgverifybackup.html ) to verify your backups. There is a reason that this tool exists - you should use it.

We might want to mention this and the DR point also under /global-synchronizer/production-operations/validator-backups#backups-of-postgres-instances...

martinflorian-da · 2026-09-09T11:59:36Z

+
+# Final Takeaway
+
+There has been no reported case of database corruption in Canton operated on Cloud SQL. The same cannot be said for databases run in containers, whether on K8S or Docker Compose setups. Therefore, if you want to avoid troubles use a cloud-native SQL solution.


That reads a bit like "Canton doesn't work well on hand-rolled postgres" where really what we want to say is "hand-rolling posgres is a bad idea unless you really know what you are doing". I'd honestly just skip this last section.

remyhaemmerle-da · 2026-09-10T07:13:49Z

+
+Operating a database is not trivial, especially when handling faults and operational issues. The simplest approach leverages cloud-native SQL services offered by all cloud providers, complemented by exporting the database and storing an encrypted backup in a secure secondary location.
+
+If you choose to run your own database yourself, you need to ensure that your data is correct and that your recovery procedures are in place. The present guide summarizes a few recipes in this regard. Note that some recipes require expert knowledge and are listed here solely to guide experts through the recovery process. Manipulating databases without care may result in data corruption and loss.


[nitpicking] The expression "without care" can sound a bit accusatory or condescending.

Suggested change

If you choose to run your own database yourself, you need to ensure that your data is correct and that your recovery procedures are in place. The present guide summarizes a few recipes in this regard. Note that some recipes require expert knowledge and are listed here solely to guide experts through the recovery process. Manipulating databases without care may result in data corruption and loss.

If you choose to run your own database yourself, you need to ensure that your data is correct and that your recovery procedures are in place. The present guide summarizes a few recipes in this regard. Note that some recipes require expert knowledge and are listed here solely to guide experts through the recovery process. Manipulating databases without great care may result in data corruption and loss.

mintlify Bot deployed to staging - docs-main September 3, 2026 15:25 View deployment

mziolekda reviewed Sep 3, 2026

View reviewed changes

brianweir-da reviewed Sep 4, 2026

View reviewed changes

Documented db corruption analysis and recovery procedures

f8525ae

Before this pr, we didn't explain how to detect and recover broken databases. Now, we have a summary from past experience. Signed-off-by: Ratko Veprek <ratko@digitalasset.com>

daravep force-pushed the document_amm branch from a59ca9e to 1618978 Compare September 9, 2026 10:51

mintlify Bot deployed to staging - docs-main September 9, 2026 10:53 View deployment

daravep requested a review from martinflorian-da September 9, 2026 11:04

Addressed reviewer comments

c08ec91

Signed-off-by: Ratko Veprek <ratko@digitalasset.com>

daravep force-pushed the document_amm branch from 1618978 to c08ec91 Compare September 9, 2026 11:30

mintlify Bot deployed to staging - docs-main September 9, 2026 11:33 View deployment

martinflorian-da approved these changes Sep 9, 2026

View reviewed changes

remyhaemmerle-da reviewed Sep 10, 2026

View reviewed changes


		# Use a Professional Cloud Operator

		While there are procedures to recover a node's data from the network, they are rather complex (there is an incoming improvement with GetRemoteAcs in 3.7), time-consuming, and only allow you to recover the current state, not history. The simpler approach is therefore to avoid performing these procedures by not corrupting your data.


		While there are procedures to recover a node's data from the network, they are rather complex (there is an incoming improvement with GetRemoteAcs in 3.7), time-consuming, and only allow you to recover the current state, not history. The simpler approach is therefore to avoid performing these procedures by not corrupting your data.

		Operating a database is not trivial, especially when dealing with faults and issues. The simplest way is to rely on cloud-based SQL services offered by all cloud providers, and additionally export the database and store it securely in a second location (ideally encrypted).


		The only case where a database might be repaired is when some indexes on auxiliary tables are broken. In this case, a simple REINDEX TABLE <tablename> will rebuild the database index. This might be an option in case the issue affects a section of the database where errors cannot propagate. Note that broken indexes may result from hardware failure; as such, further investigation is necessary.

		# Final Summary


		# Final Summary

		There has been no reported case of database corruption in Canton operated on Cloud SQL. The same cannot be said for databases run in containers, whether on K8S or Docker Compose setups.


		The following action items may help you to prevent data corruption.

		- Prevent Duplicate Volume Access: If two Postgres instances access the same data volume, they can corrupt the data directory. Postgres does not auto-detect this situation in containerized deployments (as each process runs in its own namespace with pid=1). Therefore, you must ensure that this doesn’t happen at the orchestration and operations layers!


		# Repairing Data Corruption

		There are several ways to recover from disasters, documented already here. https://docs.canton.network/global-synchronizer/production-operations/validator-disaster-recovery and global-synchronizer/production-operations/disaster-recovery

	While there are procedures to recover a node's data from the network, they are rather complex, time-consuming, and only allow you to recover the current state, not history. The simpler approach is therefore to avoid performing these procedures by not corrupting your data.
	While there are [procedures](/global-synchronizer/production-operations/validator-disaster-recovery) to recover a node's data from the network, they are rather complex, time-consuming, and only allow you to recover the current state, not history. The simpler approach is therefore to avoid performing these procedures by not corrupting your data.


		- Use Professional Hardware and Software: Invest in your hardware and software setup, ensuring you use ECC memory, file systems, and hard drives resilient to data degradation.

		- Back up your Database: Use `pg_dump` to back up your database regularly, rather than relying on file-system backups. A file-system backup is only possible if you can take a snapshot of the filesystem. A copy of the files will not be synchronized and therefore will be corrupted. Furthermore, Postgres offers many different ways to perform backups, even at scale (e.g., [WAL copying for point-in-time recovery](https://www.postgresql.org/docs/current/continuous-archiving.html)).

	- Verify your Backup: Postgres offers a tool (pg_verifybackup https://www.postgresql.org/docs/current/app-pgverifybackup.html ) to verify your backups. There is a reason that this tool exists. Therefore, you should use it.
	- Verify your Backup: Postgres offers a tool (pg_verifybackup https://www.postgresql.org/docs/current/app-pgverifybackup.html ) to verify your backups. There is a reason that this tool exists - you should use it.


		# Final Takeaway

		There has been no reported case of database corruption in Canton operated on Cloud SQL. The same cannot be said for databases run in containers, whether on K8S or Docker Compose setups. Therefore, if you want to avoid troubles use a cloud-native SQL solution. No newline at end of file


		Operating a database is not trivial, especially when handling faults and operational issues. The simplest approach leverages cloud-native SQL services offered by all cloud providers, complemented by exporting the database and storing an encrypted backup in a secure secondary location.

		If you choose to run your own database yourself, you need to ensure that your data is correct and that your recovery procedures are in place. The present guide summarizes a few recipes in this regard. Note that some recipes require expert knowledge and are listed here solely to guide experts through the recovery process. Manipulating databases without care may result in data corruption and loss.

Conversation

daravep commented Sep 3, 2026

Uh oh!

mintlify Bot commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mziolekda Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

martinflorian-da left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

mintlify Bot commented Sep 3, 2026 •

edited

Loading

mziolekda Sep 3, 2026 •

edited

Loading