Skip to content

CS-10933: PR diff comment workflow + scripts/diff.sh#4544

Merged
lukemelia merged 4 commits into
mainfrom
cs-10933-pr-diff-workflow
Apr 28, 2026
Merged

CS-10933: PR diff comment workflow + scripts/diff.sh#4544
lukemelia merged 4 commits into
mainfrom
cs-10933-pr-diff-workflow

Conversation

@lukemelia

Copy link
Copy Markdown
Contributor

Stacked on PR #4542 (cs-10936-production-apply-workflow) → which is stacked on the rest of the chain. Review/merge those first.

Draft until staging apply (#4538) is verified working post-merge — the diff workflow pulls from staging Grafana, so it needs the staging side ready first.

Summary

When a PR touches `packages/observability/**`, post a sticky comment showing the diff between the committed `grafanactl/resources/` state and the live state of staging Grafana. Mirrors the existing terraform-plan-comment pattern that the repo uses for infra changes.

This was Path B in CS-10933's design — grafanactl has no native `resources diff` subcommand (verified via `grafanactl resources --help`), so we implement the equivalent: pull live state to a tempdir, `git diff --no-index` against the committed tree.

Files added

  • `packages/observability/scripts/diff.sh` — pulls live grafanactl resources to a tempdir, diffs against committed `./grafanactl/resources/`. Empty output (and exit 0) = committed equals live.
  • `.github/workflows/observability-diff.yml` — PR trigger, paths filter on `packages/observability/**`, reuses the staging `boxel-observability-apply` role, posts a sticky comment with the diff (or "no changes" if empty), truncates at 60KB.

Scope of the diff

The diff is limited to grafanactl-managed kinds: dashboards and folders. The `provisioning/` tree (data sources, alert rules) is delivered to Grafana via file mount at startup, not API push — so there's no "live state" to diff against from the outside. PR review of those YAMLs is the only check for that tree.

This is documented inline in `diff.sh` so future maintainers don't try to extend the diff to provisioning/ without realizing the architecture mismatch.

Sticky comment

  • Marker: `` at the top of the comment body.
  • On each push: list comments, find one starting with marker, update if exists else create.
  • Concurrency group is per-PR with `cancel-in-progress: true` — older runs cancel when a new push lands, so the comment always reflects the latest commit.
  • Diff is also printed to the run log inside a collapsible `::group::` so reviewers can read it without expanding the comment.

Truncation

GitHub PR comments cap at ~64KB. We truncate at 60KB and append a link to the run details for the full diff. Most observability PRs will be well under this — the synapse dashboard is the only resource large enough to risk overflow, and it changes rarely (vendored from upstream Synapse).

IAM

Reuses `arn:aws:iam::680542703984:role/boxel-observability-apply` (the staging apply role from cardstack/infra#665). The role has `ssm:GetParameter` on `/staging/grafana/grafanactl_token` only — same scope a diff needs.

Note: the staging Grafana service-account token has Admin role, so technically the diff workflow could push instead of pull (it's just a question of which subcommand we invoke). The boundary between "diff" and "apply" is enforced by the workflow YAML, not by the token's permissions. A separate read-only Grafana service account would be tighter; deferred as a possible follow-up if it becomes valuable.

Test plan

After staging-apply (#4538) is verified working post-merge:

  • PR touching `packages/observability/grafanactl/resources/dashboards/`: comment appears showing the dashboard diff (or "No changes" if the committed state matches what was last applied to staging).
  • Subsequent push to the same PR: existing comment updates in place (no new comment).
  • PR touching only `packages/observability/scripts/`: workflow runs but the diff is empty (no resource changes) — comment says "No changes detected".
  • PR touching only `provisioning/`: workflow runs, diff is empty (provisioning tree isn't grafanactl-managed). Comment notes the limitation… actually the comment just says "no dashboard/folder changes" which is true. The reviewer is responsible for diffing provisioning/ via the regular GitHub file diff.
  • Forks: TBD — `pull-requests: write` permission has limited scope on fork PRs. cardstack/boxel rarely takes fork contributions; if it becomes an issue we add a fork-detection branch.
  • Long diff (Synapse-dashboard scale): truncates at 60KB with a link to the run.

Linear

  • Closes CS-10933
  • Will benefit from CS-10959 (lint step before diff) once that ticket lands.

🤖 Generated with Claude Code

@lukemelia
lukemelia force-pushed the cs-10933-pr-diff-workflow branch from 0fe4f1c to 3005a78 Compare April 27, 2026 22:16
lukemelia added a commit that referenced this pull request Apr 27, 2026
…ite action

CI failure on PR #4544: `curl: (22) The requested URL returned error: 404`
on the grafanactl tarball download. Two bugs in the install snippet:

1. Version pinned to 0.7.0 — that release doesn't exist. Latest published
   release as of 2026-04-27 is v0.1.10 (grafanactl is still pre-1.0).
   Bumped to "0.1.10".

2. Asset name pattern was wrong:
   - was: grafanactl_${VERSION}_linux_amd64.tar.gz
   - actual: grafanactl_Linux_x86_64.tar.gz (capitalized OS, x86_64
     instead of amd64, no version embedded in filename)
   - checksums file: grafanactl_checksums.txt, not checksums.txt

Verified asset names by hitting the GitHub Releases API for
grafana/grafanactl.

Three workflows duplicate this install step. Extracted into a composite
action at .github/actions/install-grafanactl/action.yml so the fix lands
once and bumps stay in one place. This commit converts the staging
workflow; cs-10936 and cs-10933 (stacked) follow with their own use-the-
composite-action edits when rebased.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
lukemelia added a commit that referenced this pull request Apr 27, 2026
…ite action

CI failure on PR #4544: `curl: (22) The requested URL returned error: 404`
on the grafanactl tarball download. Two bugs in the install snippet:

1. Version pinned to 0.7.0 — that release doesn't exist. Latest published
   release as of 2026-04-27 is v0.1.10 (grafanactl is still pre-1.0).
   Bumped to "0.1.10".

2. Asset name pattern was wrong:
   - was: grafanactl_${VERSION}_linux_amd64.tar.gz
   - actual: grafanactl_Linux_x86_64.tar.gz (capitalized OS, x86_64
     instead of amd64, no version embedded in filename)
   - checksums file: grafanactl_checksums.txt, not checksums.txt

Verified asset names by hitting the GitHub Releases API for
grafana/grafanactl.

Three workflows duplicate this install step. Extracted into a composite
action at .github/actions/install-grafanactl/action.yml so the fix lands
once and bumps stay in one place. This commit converts the staging
workflow; cs-10936 and cs-10933 (stacked) follow with their own use-the-
composite-action edits when rebased.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@lukemelia
lukemelia force-pushed the cs-10936-production-apply-workflow branch from c25410c to e56b80e Compare April 27, 2026 23:37
lukemelia and others added 2 commits April 27, 2026 19:37
When a PR touches packages/observability/**, post a sticky comment
showing the diff between the committed grafanactl/resources/ state
and the live state of staging Grafana. Mirrors the existing
terraform-plan-comment pattern.

Two pieces:

1. packages/observability/scripts/diff.sh
   - Pulls the live grafanactl resources to a tempdir via
     `grafanactl resources pull`, diffs against the committed
     ./grafanactl/resources/ tree using `git diff --no-index`.
   - This is "Path B" from the CS-10933 design exploration —
     grafanactl has no native `resources diff`, so we do it ourselves.
   - Scope is limited to grafanactl-managed kinds (dashboards, folders).
     The provisioning/ tree (data sources, alert rules) is delivered
     to Grafana via file mount at startup, not API push, so there's
     no live state to diff against from the outside. Documented
     inline.
   - Empty output (and exit 0) means committed == live.

2. .github/workflows/observability-diff.yml
   - PR-trigger workflow with paths filter on packages/observability/**.
   - Reuses the staging boxel-observability-apply role (the SSM
     permission is the same for diff as for apply — read the token).
   - Same SHA-256 checksum-verified install of grafanactl as the
     apply workflows.
   - Sticky PR comment: marker `<!-- observability-diff -->`, find
     existing → update, else create. Truncates at 60KB with a link
     to the run details for full diff.
   - Concurrency group per-PR with cancel-in-progress: true (cheap
     operation; want the comment to reflect latest commit).
   - Diff also printed to the run log inside a collapsible group so
     it's visible without expanding the comment.

Stacks on cs-10936 (production-apply); together with #4538
(staging-apply), this completes the Phase 4 CI workflows.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Same fix as cs-10932 / cs-10936 (CI failure on PR #4544 led to this
chain of fixes). Switches to the shared composite action, bumps version
to 0.1.10.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@lukemelia
lukemelia force-pushed the cs-10933-pr-diff-workflow branch from 3005a78 to f3da858 Compare April 27, 2026 23:37
CI failure on PR #4544 after the install fix:

  WARN Could not pull resources err="404 NotFound" cmd=GetAll:noop.v0alpha1.features.grafana.app
  WARN Could not pull resources err="403 Forbidden: plugins.plugins.grafana.app" ...
  WARN Could not pull resources err="403 Forbidden: metas.plugins.grafana.app" ...
  Error: Unexpected error
  3 resource(s) failed to pull

`grafanactl resources pull` with no args enumerates every API kind it
knows about. Three of them aren't usable by our service-account token:
- features.grafana.app/noop: 404 (doesn't exist on this Grafana version)
- plugins.grafana.app/{plugins,metas}: 403 (service-account doesn't
  have plugins permission — and shouldn't, that'd be over-broad)

We only care about dashboards + folders for the diff (the App Platform
resource kinds grafanactl manages — provisioning/ kinds aren't via API).
Passing them as positional args narrows the pull to exactly those.

Comment block above the call explains why we're explicit, so the next
person doesn't try to "simplify" by removing the args.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@github-actions

github-actions Bot commented Apr 28, 2026

Copy link
Copy Markdown
Contributor

Observability diff (vs staging)

Diff truncated (565909 bytes; limit 60000). Full diff: https://github.com/cardstack/boxel/actions/runs/25029118136

diff --git a/tmp/grafanactl-pull.VRgQuS/Dashboard/a2195b5f-b293-4e12-93d4-4536592de634.json b/tmp/grafanactl-pull.VRgQuS/Dashboard/a2195b5f-b293-4e12-93d4-4536592de634.json
deleted file mode 100644
index d7b1b7b..0000000
--- a/tmp/grafanactl-pull.VRgQuS/Dashboard/a2195b5f-b293-4e12-93d4-4536592de634.json
+++ /dev/null
@@ -1,1023 +0,0 @@
-{
-  "apiVersion": "dashboard.grafana.app/v0alpha1",
-  "kind": "Dashboard",
-  "metadata": {
-    "annotations": {},
-    "labels": {},
-    "name": "a2195b5f-b293-4e12-93d4-4536592de634",
-    "namespace": "default"
-  },
-  "spec": {
-    "annotations": {
-      "list": [
-        {
-          "builtIn": 1,
-          "datasource": {
-            "type": "grafana",
-            "uid": "-- Grafana --"
-          },
-          "enable": true,
-          "hide": true,
-          "iconColor": "rgba(0, 211, 255, 1)",
-          "name": "Annotations \u0026 Alerts",
-          "type": "dashboard"
-        }
-      ]
-    },
-    "editable": true,
-    "fiscalYearStartMonth": 0,
-    "graphTooltip": 0,
-    "links": [],
-    "liveNow": false,
-    "panels": [
-      {
-        "datasource": {
-          "type": "cloudwatch",
-          "uid": "b66e3d56-fb4e-49d6-9625-5d06ee69010b"
-        },
-        "fieldConfig": {
-          "defaults": {
-            "color": {
-              "mode": "palette-classic"
-            },
-            "custom": {
-              "axisCenteredZero": false,
-              "axisColorMode": "text",
-              "axisLabel": "",
-              "axisPlacement": "auto",
-              "barAlignment": 0,
-              "drawStyle": "line",
-              "fillOpacity": 0,
-              "gradientMode": "none",
-              "hideFrom": {
-                "legend": false,
-                "tooltip": false,
-                "viz": false
-              },
-              "lineInterpolation": "linear",
-              "lineWidth": 1,
-              "pointSize": 5,
-              "scaleDistribution": {
-                "type": "linear"
-              },
-              "showPoints": "auto",
-              "spanNulls": false,
-              "stacking": {
-                "group": "A",
-                "mode": "none"
-              },
-              "thresholdsStyle": {
-                "mode": "off"
-              }
-            },
-            "mappings": [],
-            "max": 100,
-            "min": 0,
-            "thresholds": {
-              "mode": "absolute",
-              "steps": [
-                {
-                  "color": "green",
-                  "value": null
-                },
-                {
-                  "color": "red",
-                  "value": 80
-                }
-              ]
-            },
-            "unit": "percent"
-          },
-          "overrides": []
-        },
-        "gridPos": {
-          "h": 7,
-          "w": 12,
-          "x": 0,
-          "y": 0
-        },
-        "id": 1,
-        "options": {
-          "legend": {
-            "calcs": [],
-            "displayMode": "list",
-            "placement": "bottom",
-            "showLegend": true
-          },
-          "tooltip": {
-            "mode": "single",
-            "sort": "none"
-          }
-        },
-        "targets": [
-          {
-            "datasource": {
-              "type": "cloudwatch",
-              "uid": "b66e3d56-fb4e-49d6-9625-5d06ee69010b"
-            },
-            "dimensions": {
-              "ClusterName": "$cluster"
-            },
-            "expression": "",
-            "hide": false,
-            "id": "",
-            "label": "",
-            "logGroups": [],
-            "matchExact": false,
-            "metricEditorMode": 0,
-            "metricName": "CPUUtilization",
-            "metricQueryType": 0,
-            "namespace": "AWS/ECS",
-            "period": "",
-            "queryMode": "Metrics",
-            "refId": "B",
-            "region": "default",
-            "sqlExpression": "",
-            "statistic": "Maximum"
-          }
-        ],
-        "title": "CPU Utilization",
-        "type": "timeseries"
-      },
-      {
-        "datasource": {
-          "type": "cloudwatch",
-          "uid": "b66e3d56-fb4e-49d6-9625-5d06ee69010b"
-        },
-        "fieldConfig": {
-          "defaults": {
-            "color": {
-              "mode": "palette-classic"
-            },
-            "custom": {
-              "axisCenteredZero": false,
-              "axisColorMode": "text",
-              "axisLabel": "",
-              "axisPlacement": "auto",
-              "barAlignment": 0,
-              "drawStyle": "line",
-              "fillOpacity": 0,
-              "gradientMode": "none",
-              "hideFrom": {
-                "legend": false,
-                "tooltip": false,
-                "viz": false
-              },
-              "lineInterpolation": "linear",
-              "lineWidth": 1,
-              "pointSize": 5,
-              "scaleDistribution": {
-                "type": "linear"
-              },
-              "showPoints": "auto",
-              "spanNulls": false,
-              "stacking": {
-                "group": "A",
-                "mode": "none"
-              },
-              "thresholdsStyle": {
-                "mode": "off"
-              }
-            },
-            "mappings": [],
-            "max": 100,
-            "min": 0,
-            "thresholds": {
-              "mode": "absolute",
-              "steps": [
-                {
-                  "color": "green",
-                  "value": null
-                },
-                {
-                  "color": "red",
-                  "value": 80
-                }
-              ]
-            },
-            "unit": "percent"
-          },
-          "overrides": []
-        },
-        "gridPos": {
-          "h": 7,
-          "w": 12,
-          "x": 12,
-          "y": 0
-        },
-        "id": 2,
-        "options": {
-          "legend": {
-            "calcs": [],
-            "displayMode": "list",
-            "placement": "bottom",
-            "showLegend": true
-          },
-          "tooltip": {
-            "mode": "single",
-            "sort": "none"
-          }
-        },
-        "targets": [
-          {
-            "datasource": {
-              "type": "cloudwatch",
-              "uid": "b66e3d56-fb4e-49d6-9625-5d06ee69010b"
-            },
-            "dimensions": {
-              "ClusterName": "$cluster"
-            },
-            "expression": "",
-            "hide": false,
-            "id": "",
-            "label": "",
-            "logGroups": [],
-            "matchExact": false,
-            "metricEditorMode": 0,
-            "metricName": "MemoryUtilization",
-            "metricQueryType": 0,
-            "namespace": "AWS/ECS",
-            "period": "",
-            "queryMode": "Metrics",
-            "refId": "B",
-            "region": "default",
-            "sqlExpression": "",
-            "statistic": "Maximum"
-          }
-        ],
-        "title": "Memory Utilization",
-        "type": "timeseries"
-      },
-      {
-        "datasource": {
-          "type": "cloudwatch",
-          "uid": "b66e3d56-fb4e-49d6-9625-5d06ee69010b"
-        },
-        "gridPos": {
-          "h": 7,
-          "w": 24,
-          "x": 0,
-          "y": 7
-        },
-        "id": 3,
-        "options": {
-          "dedupStrategy": "none",
-          "enableLogDetails": true,
-          "prettifyLogMessage": false,
-          "showCommonLabels": false,
-          "showLabels": false,
-          "showTime": true,
-          "sortOrder": "Descending",
-          "wrapLogMessage": false
-        },
-        "targets": [
-          {
-            "datasource": {
-              "type": "cloudwatch",
-              "uid": "b66e3d56-fb4e-49d6-9625-5d06ee69010b"
-            },
-            "dimensions": {},
-            "expression": "fields @timestamp, @message |\n filter @logStream like \"/$service/\" |\n sort @timestamp desc |\n limit $logslimit",
-            "id": "",
-            "label": "",
-            "logGroups": [
-              {
-                "arn": "$loggroup",
-                "name": "$loggroup"
-              }
-            ],
-            "matchExact": true,
-            "metricEditorMode": 0,
-            "metricName": "",
-            "metricQueryType": 0,
-            "namespace": "",
-            "period": "",
-            "queryMode": "Logs",
-            "refId": "A",
-            "region": "default",
-            "sqlExpression": "",
-            "statistic": "Average",
-            "statsGroups": []
-          }
-        ],
-        "title": "Logs",
-        "type": "logs"
-      },
-      {
-        "datasource": {
-          "type": "cloudwatch",
-          "uid": "b66e3d56-fb4e-49d6-9625-5d06ee69010b"
-        },
-        "fieldConfig": {
-          "defaults": {
-            "color": {
-              "mode": "palette-classic"
-            },
-            "custom": {
-              "axisCenteredZero": false,
-              "axisColorMode": "text",
-              "axisLabel": "",
-              "axisPlacement": "auto",
-              "barAlignment": 0,
-              "drawStyle": "line",
-              "fillOpacity": 0,
-              "gradientMode": "none",
-              "hideFrom": {
-                "legend": false,
-                "tooltip": false,
-                "viz": false
-              },
-              "lineInterpolation": "linear",
-              "lineWidth": 1,
-              "pointSize": 5,
-              "scaleDistribution": {
-                "type": "linear"
-              },
-              "showPoints": "auto",
-              "spanNulls": false,
-              "stacking": {
-                "group": "A",
-                "mode": "none"
-              },
-              "thresholdsStyle": {
-                "mode": "off"
-              }
-            },
-            "mappings": [],
-            "thresholds": {
-              "mode": "absolute",
-              "steps": [
-                {
-                  "color": "green",
-                  "value": null
-                },
-                {
-                  "color": "red",
-                  "value": 80
-                }
-              ]
-            },
-            "unit": "s"
-          },
-          "overrides": []
-        },
-        "gridPos": {
-          "h": 7,
-          "w": 6,
-          "x": 0,
-          "y": 14
-        },
-        "id": 4,
-        "options": {
-          "legend": {
-            "calcs": [],
-            "displayMode": "list",
-            "placement": "bottom",
-            "showLegend": true
-          },
-          "tooltip": {
-            "mode": "single",
-            "sort": "none"
-          }
-        },
-        "targets": [
-          {
-            "datasource": {
-              "type": "cloudwatch",
-              "uid": "b66e3d56-fb4e-49d6-9625-5d06ee69010b"
-            },
-            "dimensions": {
-              "LoadBalancer": "$lb"
-            },
-            "expression": "",
-            "id": "",
-            "label": "Maximum",
-            "logGroups": [],
-            "matchExact": true,
-            "metricEditorMode": 0,
-            "metricName": "TargetResponseTime",
-            "metricQueryType": 0,
-            "namespace": "AWS/ApplicationELB",
-            "period": "",
-            "queryMode": "Metrics",
-            "refId": "A",
-            "region": "default",
-            "sqlExpression": "",
-            "statistic": "Maximum"
-          },
-          {
-            "datasource": {
-              "type": "cloudwatch",
-              "uid": "b66e3d56-fb4e-49d6-9625-5d06ee69010b"
-            },
-            "dimensions": {
-              "LoadBalancer": "$lb"
-            },
-            "expression": "",
-            "hide": false,
-            "id": "",
-            "label": "Average",
-            "logGroups": [],
-            "matchExact": true,
-            "metricEditorMode": 0,
-            "metricName": "TargetResponseTime",
-            "metricQueryType": 0,
-            "namespace": "AWS/ApplicationELB",
-            "period": "",
-            "queryMode": "Metrics",
-            "refId": "B",
-            "region": "default",
-            "sqlExpression": "",
-            "statistic": "Average"
-          }
-        ],
-        "title": "Response Time",
-        "type": "timeseries"
-      },
-      {
-        "datasource": {
-          "type": "cloudwatch",
-          "uid": "b66e3d56-fb4e-49d6-9625-5d06ee69010b"
-        },
-        "fieldConfig": {
-          "defaults": {
-            "color": {
-              "mode": "palette-classic"
-            },
-            "custom": {
-              "axisCenteredZero": false,
-              "axisColorMode": "text",
-              "axisLabel": "",
-              "axisPlacement": "auto",
-              "barAlignment": 0,
-              "drawStyle": "line",
-              "fillOpacity": 0,
-              "gradientMode": "none",
-              "hideFrom": {
-                "legend": false,
-                "tooltip": false,
-                "viz": false
-              },
-              "lineInterpolation": "linear",
-              "lineWidth": 1,
-              "pointSize": 5,
-              "scaleDistribution": {
-                "type": "linear"
-              },
-              "showPoints": "auto",
-              "spanNulls": false,
-              "stacking": {
-                "group": "A",
-                "mode": "none"
-              },
-              "thresholdsStyle": {
-                "mode": "off"
-              }
-            },
-            "mappings": [],
-            "thresholds": {
-              "mode": "absolute",
-              "steps": [
-                {
-                  "color": "green",
-                  "value": null
-                },
-                {
-                  "color": "red",
-                  "value": 80
-                }
-              ]
-            },
-            "unit": "none"
-          },
-          "overrides": []
-        },
-        "gridPos": {
-          "h": 7,
-          "w": 6,
-          "x": 6,
-          "y": 14
-        },
-        "id": 5,
-        "options": {
-          "legend": {
-            "calcs": [],
-            "displayMode": "list",
-            "placement": "bottom",
-            "showLegend": false
-          },
-          "tooltip": {
-            "mode": "single",
-            "sort": "none"
-          }
-        },
-        "targets": [
-          {
-            "datasource": {
-              "type": "cloudwatch",
-              "uid": "b66e3d56-fb4e-49d6-9625-5d06ee69010b"
-            },
-            "dimensions": {
-              "LoadBalancer": "$lb"
-            },
-            "expression": "",
-            "id": "",
-            "label": "",
-            "logGroups": [],
-            "matchExact": true,
-            "metricEditorMode": 0,
-            "metricName": "RequestCount",
-            "metricQueryType": 0,
-            "namespace": "AWS/ApplicationELB",
-            "period": "",
-            "queryMode": "Metrics",
-            "refId": "A",
-            "region": "default",
-            "sqlExpression": "",
-            "statistic": "Sum"
-          }
-        ],
-        "title": "Request Count",
-        "type": "timeseries"
-      },
-      {
-        "datasource": {
-          "type": "cloudwatch",
-          "uid": "b66e3d56-fb4e-49d6-9625-5d06ee69010b"
-        },
-        "fieldConfig": {
-          "defaults": {
-            "color": {
-              "mode": "palette-classic"
-            },
-            "custom": {
-              "axisCenteredZero": false,
-              "axisColorMode": "text",
-              "axisLabel": "",
-              "axisPlacement": "auto",
-              "barAlignment": 0,
-              "drawStyle": "line",
-              "fillOpacity": 0,
-              "gradientMode": "none",
-              "hideFrom": {
-                "legend": false,
-                "tooltip": false,
-                "viz": false
-              },
-              "lineInterpolation": "linear",
-              "lineWidth": 1,
-              "pointSize": 5,
-              "scaleDistribution": {
-                "type": "linear"
-              },
-              "showPoints": "auto",
-              "spanNulls": false,
-              "stacking": {
-                "group": "A",
-                "mode": "none"
-              },
-              "thresholdsStyle": {
-                "mode": "off"
-              }
-            },
-            "mappings": [],
-            "thresholds": {
-              "mode": "absolute",
-              "steps": [
-                {
-                  "color": "green",
-                  "value": null
-                },
-                {
-                  "color": "red",
-                  "value": 80
-                }
-              ]
-            }
-          },
-          "overrides": [
-            {
-              "matcher": {
-                "id": "byName",
-                "options": "5XX"
-              },
-              "properties": [
-                {
-                  "id": "color",
-                  "value": {
-                    "fixedColor": "red",
-                    "mode": "fixed"
-                  }
-                }
-              ]
-            },
-            {
-              "matcher": {
-                "id": "byName",
-                "options": "2XX"
-              },
-              "properties": [
-                {
-                  "id": "color",
-                  "value": {
-                    "fixedColor": "green",
-                    "mode": "fixed"
-                  }
-                }
-              ]
-            }
-          ]
-        },
-        "gridPos": {
-          "h": 7,
-          "w": 6,
-          "x": 12,
-          "y": 14
-        },
-        "id": 6,
-        "options": {
-          "legend": {
-            "calcs": [],
-            "displayMode": "list",
-            "placement": "bottom",
-            "showLegend": true
-          },
-          "tooltip": {
-            "mode": "single",
-            "sort": "none"
-          }
-        },
-        "targets": [
-          {
-            "datasource": {
-              "type": "cloudwatch",
-              "uid": "b66e3d56-fb4e-49d6-9625-5d06ee69010b"
-            },
-            "dimensions": {
-              "LoadBalancer": "$lb"
-            },
-            "expression": "",
-            "hide": false,
-            "id": "",
-            "label": "2XX",
-            "logGroups": [],
-            "matchExact": true,
-            "metricEditorMode": 0,
-            "metricName": "HTTPCode_Target_2XX_Count",
-            "metricQueryType": 0,
-            "namespace": "AWS/ApplicationELB",
-            "period": "",
-            "queryMode": "Metrics",
-            "refId": "C",
-            "region": "default",
-            "sqlExpression": "",
-            "statistic": "Sum"
-          },
-          {
-            "datasource": {
-              "type": "cloudwatch",
-              "uid": "b66e3d56-fb4e-49d6-9625-5d06ee69010b"
-            },
-            "dimensions": {
-              "LoadBalancer": "$lb"
-            },
-            "expression": "",
-            "hide": false,
-            "id": "",
-            "label": "4XX",
-            "logGroups": [],
-            "matchExact": true,
-            "metricEditorMode": 0,
-            "metricName": "HTTPCode_Target_4XX_Count",
-            "metricQueryType": 0,
-            "namespace": "AWS/ApplicationELB",
-            "period": "",
-            "queryMode": "Metrics",
-            "refId": "B",
-            "region": "default",
-            "sqlExpression": "",
-            "statistic": "Sum"
-          },
-          {
-            "datasource": {
-              "type": "cloudwatch",
-              "uid": "b66e3d56-fb4e-49d6-9625-5d06ee69010b"
-            },
-            "dimensions": {
-              "LoadBalancer": "$lb"
-            },
-            "expression": "",
-            "hide": false,
-            "id": "",
-            "label": "5XX",
-            "logGroups": [],
-            "matchExact": true,
-            "metricEditorMode": 0,
-            "metricName": "HTTPCode_Target_5XX_Count",
-            "metricQueryType": 0,
-            "namespace": "AWS/ApplicationELB",
-            "period": "",
-            "queryMode": "Metrics",
-            "refId": "A",
-            "region": "default",
-            "sqlExpression": "",
-            "statistic": "Sum"
-          }
-        ],
-        "title": "Target Response Code",
-        "type": "timeseries"
-      },
-      {
-        "datasource": {
-          "type": "cloudwatch",
-          "uid": "b66e3d56-fb4e-49d6-9625-5d06ee69010b"
-        },
-        "fieldConfig": {
-          "defaults": {
-            "color": {
-              "mode": "palette-classic"
-            },
-            "custom": {
-              "axisCenteredZero": false,
-              "axisColorMode": "text",
-              "axisLabel": "",
-              "axisPlacement": "auto",
-              "barAlignment": 0,
-              "drawStyle": "line",
-              "fillOpacity": 0,
-              "gradientMode": "none",
-              "hideFrom": {
-                "legend": false,
-                "tooltip": false,
-                "viz": false
-              },
-              "lineInterpolation": "linear",
-              "lineWidth": 1,
-              "pointSize": 5,
-              "scaleDistribution": {
-                "type": "linear"
-              },
-              "showPoints": "auto",
-              "spanNulls": false,
-              "stacking": {
-                "group": "A",
-                "mode": "none"
-              },
-              "thresholdsStyle": {
-                "mode": "off"
-              }
-            },
-            "mappings": [],
-            "thresholds": {
-              "mode": "absolute",
-              "steps": [
-                {
-                  "color": "green",
-                  "value": null
-                },
-                {
-                  "color": "red",
-                  "value": 80
-                }
-              ]
-            }
-          },
-          "overrides": [
-            {
-              "matcher": {
-                "id": "byName",
-                "options": "5XX"
-              },
-              "properties": [
-                {
-                  "id": "color",
-                  "value": {
-                    "fixedColor": "red",
-                    "mode": "fixed"
-                  }
-                }
-              ]
-            }
-          ]
-        },
-        "gridPos": {
-          "h": 7,
-          "w": 6,
-          "x": 18,
-          "y": 14
-        },
-        "id": 7,
-        "options": {
-          "legend": {
-            "calcs": [],
-            "displayMode": "list",
-            "placement": "bottom",
-            "showLegend": true
-          },
-          "tooltip": {
-            "mode": "single",
-            "sort": "none"
-          }
-        },
-        "targets": [
-          {
-            "datasource": {
-              "type": "cloudwatch",
-              "uid": "b66e3d56-fb4e-49d6-9625-5d06ee69010b"
-            },
-            "dimensions": {
-              "LoadBalancer": "$lb"
-            },
-            "expression": "",
-            "hide": false,
-            "id": "",
-            "label": "5XX",
-            "logGroups": [],
-            "matchExact": true,
-            "metricEditorMode": 0,
-            "metricName": "HTTPCode_ELB_5XX_Count",
-            "metricQueryType": 0,
-            "namespace": "AWS/ApplicationELB",
-            "period": "",
-            "queryMode": "Metrics",
-            "refId": "A",
-            "region": "default",
-            "sqlExpression": "",
-            "statistic": "Sum"
-          },
-          {
-            "datasource": {
-              "type": "cloudwatch",
-              "uid": "b66e3d56-fb4e-49d6-9625-5d06ee69010b"
-            },
-            "dimensions": {
-              "LoadBalancer": "$lb"
-            },
-            "expression": "",
-            "hide": false,
-            "id": "",
-            "label": "4XX",
-            "logGroups": [],
-            "matchExact": true,
-            "metricEditorMode": 0,
-            "metricName": "HTTPCode_ELB_4XX_Count",
-            "metricQueryType": 0,
-            "namespace": "AWS/ApplicationELB",
-            "period": "",
-            "queryMode": "Metrics",
-            "refId": "B",
-            "region": "default",
-            "sqlExpression": "",
-            "statistic": "Sum"
-          }
-        ],
-        "title": "ALB Response Code",
-        "type": "timeseries"
-      }
-    ],
-    "refresh": "30s",
-    "schemaVersion": 38,
-    "style": "dark",
-    "tags": [],
-    "templating": {
-      "list": [
-        {
-          "current": {
-            "selected": true,
-            "text": "10",
-            "value": "10"
-          },
-          "hide": 0,
-          "includeAll": false,
-          "label": "Logs Limit",
-          "multi": false,
-          "name": "logslimit",
-          "options": [
-            {
-              "selected": true,
-              "text": "10",
-              "value": "10"
-            },
-            {
-              "selected": false,
-              "text": "100",
-              "value": "100"
-            },
-            {
-              "selected": false,
-              "text": "1000",
-              "value": "1000"
-            }
-          ],
-          "query": "10, 100, 1000",
-          "queryValue": "",
-          "skipUrlSync": false,
-          "type": "custom"
-        },
-        {
-          "current": {
-            "selected": false,
-            "text": "realm-demo-staging",
-            "value": "realm-demo-staging"
-          },
-          "datasource": {
-            "type": "cloudwatch",
-            "uid": "b66e3d56-fb4e-49d6-9625-5d06ee69010b"
-          },
-          "definition": "",
-          "hide": 2,
-          "includeAll": false,
-          "multi": false,
-          "name": "cluster",
-          "options": [],
-          "query": {
-            "dimensionKey": "ClusterName",
-            "metricName": "CPUUtilization",
-            "namespace": "AWS/ECS",
-            "queryType": "dimensionValues",
-            "refId": "CloudWatchVariableQueryEditor-VariableQuery",
-            "region": "default"
-          },
-          "refresh": 1,
-          "regex": "hub-staging",
-          "skipUrlSync": false,
-          "sort": 0,
-          "type": "query"
-        },
-        {
-          "current": {
-            "selected": false,
-            "text": "app/waypoint-ecs-realm-demo/7e699a3b9ff13ebc",
-            "value": "app/waypoint-ecs-realm-demo/7e699a3b9ff13ebc"
-          },
-          "datasource": {
-            "type": "cloudwatch",
-            "uid": "b66e3d56-fb4e-49d6-9625-5d06ee69010b"
-          },
-          "definition": "",
-          "hide": 2,
-          "includeAll": false,
-          "multi": false,
-          "name": "lb",
-          "options": [],
-          "query": {
-            "dimensionKey": "LoadBalancer",
-            "metricName": "RequestCount",
-            "namespace": "AWS/ApplicationELB",
-            "queryType": "dimensionValues",
-            "refId": "CloudWatchVariableQueryEditor-VariableQuery",
-            "region": "default"
-          },
-          "refresh": 1,
-          "regex": "app/waypoint-ecs-hub/.*",
-          "skipUrlSync": false,
-          "sort": 0,
-          "type": "query"
-        },
-        {
-          "current": {
-            "selected": false,
-            "text": "waypoint-logs",
-            "value": "arn:aws:logs:us-east-1:680542703984:log-group:waypoint-logs:*"
-          },
-          "datasource": {
-            "type": "cloudwatch",
-            "uid": "b66e3d56-fb4e-49d6-9625-5d06ee69010b"
-          },
-          "definition": "",
-          "hide": 2,
-          "includeAll": false,
-          "multi": false,
-          "name": "loggroup",
-          "options": [],
-          "query": {
-            "logGroupPrefix": "waypoint-logs",
-            "queryType": "logGroups",
-            "refId": "CloudWatchVariableQueryEditor-VariableQuery",
-            "region": "default"
-          },
-          "refresh": 1,
-          "regex": "",
-          "skipUrlSync": false,
-          "sort": 0,
-          "type": "query"
-        },
-        {
-          "hide": 2,
-          "name": "service",
-          "query": "hub",
-          "skipUrlSync": false,
-          "type": "constant"
-        }
-      ]
-    },
-    "time": {
-      "from": "now-15m",
-      "to": "now"
-    },
-    "timepicker": {},
-    "timezone": "",
-    "title": "Hub Server",
-    "weekStart": ""
-  }
-}
diff --git a/tmp/grafanactl-pull.VRgQuS/Dashboard/b5a43a39-bb43-42b5-917b-ac54a0bdeff5.json b/tmp/grafanactl-pull.VRgQuS/Dashboard/b5a43a39-bb43-42b5-917b-ac54a0bdeff5.json
deleted file mode 100644
index e95e707..0000000
--- a/tmp/grafanactl-pull.VRgQuS/Dashboard/b5a43a39-bb43-42b5-917b-ac54a0bdeff5.json
+++ /dev/null
@@ -1,406 +0,0 @@
-{
-  "apiVersion": "dashboard.grafana.app/v0alpha1",
-  "kind": "Dashboard",
-  "metadata": {
-    "annotations": {},
-    "labels": {},
-    "name": "b5a43a39-bb43-42b5-917b-ac54a0bdeff5",
-    "namespace": "default"
-  },
-  "spec": {
-    "annotations": {
-      "list": [
-        {
-          "builtIn": 1,
-          "datasource": {
-            "type": "grafana",
-            "uid": "-- Grafana --"
-          },
-          "enable": true,
-          "hide": true,
-          "iconColor": "rgba(0, 211, 255, 1)",
-          "name": "Annotations \u0026 Alerts",
-          "type": "dashboard"
-        }
-      ]
-    },
-    "editable": true,
-    "fiscalYearStartMonth": 0,
-    "graphTooltip": 0,
-    "links": [],
-    "liveNow": false,
-    "panels": [
-      {
-        "datasource": {
-          "type": "cloudwatch",
-          "uid": "b66e3d56-fb4e-49d6-9625-5d06ee69010b"
-        },
-        "fieldConfig": {
-          "defaults": {
-            "color": {
-              "mode": "palette-classic"
-            },
-            "custom": {
-              "axisCenteredZero": false,
-              "axisColorMode": "text",
-              "axisLabel": "",
-              "axisPlacement": "auto",
-              "barAlignment": 0,
-              "drawStyle": "line",
-              "fillOpacity": 0,
-              "gradientMode": "none",
-              "hideFrom": {
-                "legend": false,
-                "tooltip": false,
-                "viz": false
-              },
-              "lineInterpolation": "linear",
-              "lineWidth": 1,
-              "pointSize": 5,
-              "scaleDistribution": {
-                "type": "linear"
-              },
-              "showPoints": "auto",
-              "spanNulls": false,
-              "stacking": {
-                "group": "A",
-                "mode": "none"
-              },
-              "thresholdsStyle": {
-                "mode": "off"
-              }
-            },
-            "mappings": [],
-            "thresholds": {
-              "mode": "absolute",
-              "steps": [
-                {
-                  "color": "green",
-                  "value": null
-                },
-                {
-                  "color": "red",
-                  "value": 80
-                }
-              ]
-            },
-            "unit": "percent"
-          },
-          "overrides": []
-        },
-        "gridPos": {
-          "h": 8,
-          "w": 12,
-          "x": 0,
-          "y": 0
-        },
-        "id": 1,
-        "options": {
-          "legend": {
-            "calcs": [],
-            "displayMode": "list",
-            "placement": "bottom",
-            "showLegend": true
-          },
-          "tooltip": {
-            "mode": "single",
-            "sort": "none"
-          }
-        },
-        "targets": [
-          {
-            "datasource": {
-              "type": "cloudwatch",
-              "uid": "b66e3d56-fb4e-49d6-9625-5d06ee69010b"
-            },
-            "dimensions": {
-              "ClusterName": "cardie-staging-Cluster-XVKsuc5732Dj",
-              "ServiceName": "cardie-staging-bot-Service-DoUGKX86rb2z"
-            },
-            "expression": "",
-            "id": "",
-            "label": "Average",
-            "logGroups": [],
-            "matchExact": true,
-            "metricEditorMode": 0,
-            "metricName": "CPUUtilization",
-            "metricQueryType": 0,
-            "namespace": "AWS/ECS",
-            "period": "",
-            "queryMode": "Metrics",
-            "refId": "A",
-            "region": "default",
-            "sqlExpression": "",
-            "statistic": "Average"
-          },
-          {
-            "datasource": {
-              "type": "cloudwatch",
-              "uid": "b66e3d56-fb4e-49d6-9625-5d06ee69010b"
-            },
-            "dimensions": {
-              "ClusterName": "cardie-staging-Cluster-XVKsuc5732Dj",
-              "ServiceName": "cardie-staging-bot-Service-DoUGKX86rb2z"
-            },
-            "expression": "",
-            "hide": false,
-            "id": "",
-            "label": "Maximum",
-            "logGroups": [],
-            "matchExact": false,
-            "metricEditorMode": 0,
-            "metricName": "CPUUtilization",
-            "metricQueryType": 0,
-            "namespace": "AWS/ECS",
-            "period": "",
-            "queryMode": "Metrics",
-            "refId": "B",
-            "region": "default",
-            "sqlExpression": "",
-            "statistic": "Maximum"
-          }
-        ],
-        "title": "CPU Utilization",
-        "type": "timeseries"
-      },
-      {
-        "datasource": {
-          "type": "cloudwatch",
-          "uid": "b66e3d56-fb4e-49d6-9625-5d06ee69010b"
-        },
-        "fieldConfig": {
-          "defaults": {
-            "color": {
-              "mode": "palette-classic"
-            },
-            "custom": {
-              "axisCenteredZero": false,
-              "axisColorMode": "text",
-              "axisLabel": "",
-              "axisPlacement": "auto",
-              "barAlignment": 0,
-              "drawStyle": "line",
-              "fillOpacity": 0,
-              "gradientMode": "none",
-              "hideFrom": {
-                "legend": false,
-                "tooltip": false,
-                "viz": false
-              },
-              "lineInterpolation": "linear",
-              "lineWidth": 1,
-              "pointSize": 5,
-              "scaleDistribution": {
-                "type": "linear"
-              },
-              "showPoints": "auto",
-              "spanNulls": false,
-              "stacking": {
-                "group": "A",
-                "mode": "none"
-              },
-              "thresholdsStyle": {
-                "mode": "off"
-              }
-            },
-            "mappings": [],
-            "thresholds": {
-              "mode": "absolute",
-              "steps": [
-                {
-                  "color": "green",
-                  "value": null
-                },
-                {
-                  "color": "red",
-                  "value": 80
-                }
-              ]
-            },
-            "unit": "percent"
-          },
-          "overrides": []
-        },
-        "gridPos": {
-          "h": 8,
-          "w": 12,
-          "x": 12,
-          "y": 0
-        },
-        "id": 2,
-        "options": {
-          "legend": {
-            "calcs": [],
-            "displayMode": "list",
-            "placement": "bottom",
-            "showLegend": true
-          },
-          "tooltip": {
-            "mode": "single",
-            "sort": "none"
-          }
-        },
-        "targets": [
-          {
-            "datasource": {
-              "type": "cloudwatch",
-              "uid": "b66e3d56-fb4e-49d6-9625-5d06ee69010b"
-            },
-            "dimensions": {
-              "ClusterName": "cardie-staging-Cluster-XVKsuc5732Dj",
-              "ServiceName": "cardie-staging-bot-Service-DoUGKX86rb2z"
-            },
-            "expression": "",
-            "id": "",
-            "label": "Average",
-            "logGroups": [],
-            "matchExact": true,
-            "metricEditorMode": 0,
-            "metricName": "MemoryUtilization",
-            "metricQueryType": 0,
-            "namespace": "AWS/ECS",
-            "period": "",
-            "queryMode": "Metrics",
-            "refId": "A",
-            "region": "default",
-            "sqlExpression": "",
-            "statistic": "Average"
-          },
-          {
-            "datasource": {
-              "type": "cloudwatch",
-              "uid": "b66e3d56-fb4e-49d6-9625-5d06ee69010b"
-            },
-            "dimensions": {
-              "ClusterName": "cardie-staging-Cluster-XVKsuc5732Dj",
-              "ServiceName": "cardie-staging-bot-Service-DoUGKX86rb2z"
-            },
-            "expression": "",
-            "hide": false,
-            "id": "",
-            "label": "Maximum",
-            "logGroups": [],
-            "matchExact": true,
-            "metricEditorMode": 0,
-            "metricName": "MemoryUtilization",
-            "metricQueryType": 0,
-            "namespace": "AWS/ECS",
-            "period": "",
-            "queryMode": "Metrics",
-            "refId": "B",
-            "region": "default",
-            "sqlExpression": "",
-            "statistic": "Maximum"
-          }
-        ],
-        "title": "Memory Utilization",
-        "type": "timeseries"
-      },
-      {
-        "datasource": {
-          "type": "cloudwatch",
-          "uid": "b66e3d56-fb4e-49d6-9625-5d06ee69010b"
-        },
-        "gridPos": {
-          "h": 6,
-          "w": 24,
-          "x": 0,
-          "y": 8
-        },
-        "id": 3,
-        "options": {
-          "dedupStrategy": "none",
-          "enableLogDetails": true,
-          "prettifyLogMessage": false,
-          "showCommonLabels": false,
-          "showLabels": false,
-          "showTime": true,
-          "sortOrder": "Descending",
-          "wrapLogMessage": false
-        },
-        "targets": [
-          {
-            "datasource": {
-              "type": "cloudwatch",
-              "uid": "b66e3d56-fb4e-49d6-9625-5d06ee69010b"
-            },
-            "dimensions": {},
-            "expression": "fields @timestamp, @message |\n sort @timestamp desc |\n limit $logslimit",
-            "id": "",
-            "label": "",
-            "logGroups": [
-              {
-                "accountId": "680542703984",
-                "arn": "arn:aws:logs:us-east-1:680542703984:log-group:/copilot/cardie-staging-bot:*",
-                "name": "/copilot/cardie-staging-bot"
-              }
-            ],
-            "matchExact": true,
-            "metricEditorMode": 0,
-            "metricName": "",
-            "metricQueryType": 0,
-            "namespace": "",
-            "period": "",
-            "queryMode": "Logs",
-            "refId": "A",
-            "region": "default",
-            "sqlExpression": "",
-            "statistic": "Average",
-            "statsGroups": []
-          }
-        ],
-        "title": "Logs",
-        "type": "logs"
-      }
-    ],
-    "refresh": "5s",
-    "schemaVersion": 38,
-    "style": "dark",
-    "tags": [],
-    "templating": {
-      "list": [
-        {
-          "current": {
-            "selected": true,
-            "text": "10",
-            "value": "10"
-          },
-          "hide": 0,
-          "includeAll": false,
-          "label": "Logs Limit",
-          "multi": false,
-          "name": "logslimit",
-          "options": [
-            {
-              "selected": true,
-              "text": "10",
-              "value": "10"
-            },
-            {
-              "selected": false,
-              "text": "100",
-              "value": "100"
-            },
-            {
-              "selected": false,
-              "text": "1000",
-              "value": "1000"
-            }
-          ],
-          "query": "10, 100, 1000",
-          "queryValue": "",
-          "skipUrlSync": false,
-          "type": "custom"
-        }
-      ]
-    },
-    "time": {
-      "from": "now-15m",
-      "to": "now"
-    },
-    "timepicker": {},
-    "timezone": "",
-    "title": "Cardie",
-    "weekStart": ""
-  }
-}
diff --git a/tmp/grafanactl-pull.VRgQuS/Dashboard/b5e70a32-aa79-4a80-9ebc-00a4db975798.json b/tmp/grafanactl-pull.VRgQuS/Dashboard/b5e70a32-aa79-4a80-9ebc-00a4db975798.json
deleted file mode 100644
index fb98022..0000000
--- a/tmp/grafanactl-pull.VRgQuS/Dashboard/b5e70a32-aa79-4a80-9ebc-00a4db975798.json
+++ /dev/null
@@ -1,1016 +0,0 @@
-{
-  "apiVersion": "dashboard.grafana.app/v0alpha1",
-  "kind": "Dashboard",
-  "metadata": {
-    "annotations": {},
-    "labels": {},
-    "name": "b5e70a32-aa79-4a80-9ebc-00a4db975798",
-    "namespace": "default"
-  },
-  "spec": {
-    "annotations": {
-      "list": [
-        {
-          "builtIn": 1,
-          "datasource": {
-            "type": "grafana",
-            "uid": "-- Grafana --"
-          },
-          "enable": true,
-          "hide": true,
-          "iconColor": "rgba(0, 211, 255, 1)",
-          "name": "Annotations \u0026 Alerts",
-          "type": "dashboard"
-        }
-      ]
-    },
-    "editable": true,
-    "fiscalYearStartMonth": 0,
-    "graphTooltip": 0,
-    "links": [],
-    "liveNow": false,
-    "panels": [
-      {
-        "datasource": {
-          "type": "cloudwatch",
-          "uid": "b66e3d56-fb4e-49d6-9625-5d06ee69010b"
-        },
-        "fieldConfig": {
-          "defaults": {
-            "color": {
-              "mode": "palette-classic"
-            },
-            "custom": {
-              "axisCenteredZero": false,
-              "axisColorMode": "text",
-              "axisLabel": "",
-              "axisPlacement": "auto",
-              "barAlignment": 0,
-              "drawStyle": "line",
-              "fillOpacity": 0,
-              "gradientMode": "none",
-              "hideFrom": {
-                "legend": false,
-                "tooltip": false,
-                "viz": false
-              },
-              "lineInterpolation": "linear",
-              "lineWidth": 1,
-              "pointSize": 5,
-              "scaleDistribution": {
-                "type": "linear"
-              },
-              "showPoints": "auto",
-              "spanNulls": false,
-              "stacking": {
-                "group": "A",
-                "mode": "none"
-              },
-              "thresholdsStyle": {
-                "mode": "off"
-              }
-            },
-            "mappings": [],
-            "max": 100,
-            "min": 0,
-            "thresholds": {
-              "mode": "absolute",
-              "steps": [
-                {
-                  "color": "green",
-                  "value": null
-                },
-                {
-                  "color": "red",
-                  "value": 80
-                }
-              ]
-            },
-            "unit": "percent"
-          },
-          "overrides": []
-        },
-        "gridPos": {
-          "h": 7,
-          "w": 12,
-          "x": 0,
-          "y": 0
-        },
-        "id": 1,
-        "options": {
-          "legend": {
-            "calcs": [],
-            "displayMode": "list",
-            "placement": "bottom",
-            "showLegend": true
-          },
-          "tooltip": {
-            "mode": "single",
-            "sort": "none"
-          }
-        },
-        "targets": [
-          {
-            "datasource": {
-              "type": "cloudwatch",
-              "uid": "b66e3d56-fb4e-49d6-9625-5d06ee69010b"
-            },
-            "dimensions": {
-              "ClusterName": "$cluster"
-            },
-            "expression": "",
-            "hide": false,
-            "id": "",
-            "label": "",
-            "logGroups": [],
-            "matchExact": false,
-            "metricEditorMode": 0,
-            "metricName": "CPUUtilization",
-            "metricQueryType": 0,
-            "namespace": "AWS/ECS",
-            "period": "",
-            "queryMode": "Metrics",
-            "refId": "B",
-            "region": "default",
-            "sqlExpression": "",
-            "statistic": "Maximum"
-          }
-        ],
-        "title": "CPU Utilization",
-        "type": "timeseries"
-      },
-      {
-        "datasource": {
-          "type": "cloudwatch",
-          "uid": "b66e3d56-fb4e-49d6-9625-5d06ee69010b"
-        },
-        "fieldConfig": {
-          "defaults": {
-            "color": {
-              "mode": "palette-classic"
-            },
-            "custom": {
-              "axisCenteredZero": false,
-              "axisColorMode": "text",
-              "axisLabel": "",
-              "axisPlacement": "auto",
-              "barAlignment": 0,
-              "drawStyle": "line",
-              "fillOpacity": 0,
-              "gradientMode": "none",
-              "hideFrom": {
-                "legend": false,
-                "tooltip": false,
-                "viz": false
-              },
-              "lineInterpolation": "linear",
-              "lineWidth": 1,
-              "pointSize": 5,
-              "scaleDistribution": {
-                "type": "linear"
-              },
-              "showPoints": "auto",
-              "spanNulls": false,
-              "stacking": {
-                "group": "A",
-                "mode": "none"
-              },
-              "thresholdsStyle": {
-                "mode": "off"
-              }
-            },
-            "mappings": [],
-            "max": 100,
-            "min": 0,
-            "thresholds": {
-              "mode": "absolute",
-              "steps": [
-                {
-                  "color": "green",
-                  "value": null
-                },
-                {
-                  "color": "red",
-                  "value": 80
-                }
-              ]
-            },
-            "unit": "percent"
-          },
-          "overrides": []
-        },
-        "gridPos": {
-          "h": 7,
-          "w": 12,
-          "x": 12,
-          "y": 0
-        },
-        "id": 2,
-        "options": {
-          "legend": {
-            "calcs": [],
-            "displayMode": "list",
-            "placement": "bottom",
-            "showLegend": true
-          },
-          "tooltip": {
-            "mode": "single",
-            "sort": "none"
-          }
-        },
-        "targets": [
-          {
-            "datasource": {
-              "type": "cloudwatch",
-              "uid": "b66e3d56-fb4e-49d6-9625-5d06ee69010b"
-            },
-            "dimensions": {
-              "ClusterName": "$cluster"
-            },
-            "expression": "",
-            "hide": false,
-            "id": "",
-            "label": "",
-            "logGroups": [],
-            "matchExact": false,
-            "metricEditorMode": 0,
-            "metricName": "MemoryUtilization",
-            "metricQueryType": 0,
-            "namespace": "AWS/ECS",
-            "period": "",
-            "queryMode": "Metrics",
-            "refId": "B",
-            "region": "default",
-            "sqlExpression": "",
-            "statistic": "Maximum"
-          }
-        ],
-        "title": "Memory Utilization",
-        "type": "timeseries"
-      },
-      {
-        "datasource": {
-          "type": "cloudwatch",
-          "uid": "b66e3d56-fb4e-49d6-9625-5d06ee69010b"
-        },
-        "gridPos": {
-          "h": 7,
-          "w": 24,
-          "x": 0,
-          "y": 7
-        },
-        "id": 3,
-        "options": {
-          "dedupStrategy": "none",
-          "enableLogDetails": true,
-          "prettifyLogMessage": false,
-          "showCommonLabels": false,
-          "showLabels": false,
-          "showTime": true,
-          "sortOrder": "Descending",
-          "wrapLogMessage": false
-        },
-        "targets": [
-          {
-            "datasource": {
-              "type": "cloudwatch",
-              "uid": "b66e3d56-fb4e-49d6-9625-5d06ee69010b"
-            },
-            "dimensions": {},
-            "expression": "fields @timestamp, @message |\n filter @logStream like \"/realm-demo/\" |\n sort @timestamp desc |\n limit $logslimit",
-            "id": "",
-            "label": "",
-            "logGroups": [
-              {
-                "arn": "$loggroup",
-                "name": "$loggroup"
-              }
-            ],
-            "matchExact": true,
-            "metricEditorMode": 0,
-            "metricName": "",
-            "metricQueryType": 0,
-            "namespace": "",
-            "period": "",
-            "queryMode": "Logs",
-            "refId": "A",
-            "region": "default",
-            "sqlExpression": "",
-            "statistic": "Average",
-            "statsGroups": []
-          }
-        ],
-        "title": "Logs",
-        "type": "logs"
-      },
-      {
-        "datasource": {
-          "type": "cloudwatch",
-          "uid": "b66e3d56-fb4e-49d6-9625-5d06ee69010b"
-        },
-        "fieldConfig": {
-          "defaults": {
-            "color": {
-              "mode": "palette-classic"
-            },
-            "custom": {
-              "axisCenteredZero": false,
-              "axisColorMode": "text",
-              "axisLabel": "",
-              "axisPlacement": "auto",
-              "barAlignment": 0,
-              "drawStyle": "line",
-              "fillOpacity": 0,
-              "gradientMode": "none",
-              "hideFrom": {
-                "legend": false,
-                "tooltip": false,
-                "viz": false
-              },
-              "lineInterpolation": "linear",
-              "lineWidth": 1,
-              "pointSize": 5,
-              "scaleDistribution": {
-                "type": "linear"
-              },
-              "showPoints": "auto",
-              "spanNulls": false,
-              "stacking": {
-                "group": "A",
-                "mode": "none"
-              },
-              "thresholdsStyle": {
-                "mode": "off"
-              }
-            },
-            "mappings": [],
-            "thresholds": {
-              "mode": "absolute",
-              "steps": [
-                {
-                  "color": "green",
-                  "value": null
-                },
-                {
-                  "color": "red",
-                  "value": 80
-                }
-              ]
-            },
-            "unit": "s"
-          },
-          "overrides": []
-        },
-        "gridPos": {
-          "h": 7,
-          "w": 6,
-          "x": 0,
-          "y": 14
-        },
-        "id": 4,
-        "options": {
-          "legend": {
-            "calcs": [],
-            "displayMode": "list",
-            "placement": "bottom",
-            "showLegend": true
-          },
-          "tooltip": {
-            "mode": "single",
-            "sort": "none"
-          }
-        },
-        "targets": [
-          {
-            "datasource": {
-              "type": "cloudwatch",
-              "uid": "b66e3d56-fb4e-49d6-9625-5d06ee69010b"
-            },
-            "dimensions": {
-              "LoadBalancer": "$lb"
-            },
-            "expression": "",
-            "id": "",
-            "label": "Maximum",
-            "logGroups": [],
-            "matchExact": true,
-            "metricEditorMode": 0,
-            "metricName": "TargetResponseTime",
-            "metricQueryType": 0,
-            "namespace": "AWS/ApplicationELB",
-            "period": "",
-            "queryMode": "Metrics",
-            "refId": "A",
-            "region": "default",
-            "sqlExpression": "",
-            "statistic": "Maximum"
-          },
-          {
-            "datasource": {
-              "type": "cloudwatch",
-              "uid": "b66e3d56-fb4e-49d6-9625-5d06ee69010b"
-            },
-            "dimensions": {
-              "LoadBalancer": "$lb"
-            },
-            "expression": "",
-            "hide": false,
-            "id": "",
-            "label": "Average",
-            "logGroups": [],
-            "matchExact": true,
-            "metricEditorMode": 0,
-            "metricName": "TargetResponseTime",
-            "metricQueryType": 0,
-            "namespace": "AWS/ApplicationELB",
-            "period": "",
-            "queryMode": "Metrics",
-            "refId": "B",
-            "region": "default",
-            "sqlExpression": "",
-            "statistic": "Average"
-          }
-        ],
-        "title": "Response Time",
-        "type": "timeseries"
-      },
-      {
-        "datasource": {
-          "type": "cloudwatch",
-          "uid": "b66e3d56-fb4e-49d6-9625-5d06ee69010b"
-        },
-        "fieldConfig": {
-          "defaults": {
-            "color": {
-              "mode": "palette-classic"
-            },
-            "custom": {
-              "axisCenteredZero": false,
-              "axisColorMode": "text",
-              "axisLabel": "",
-              "axisPlacement": "auto",
-              "barAlignment": 0,
-              "drawStyle": "line",
-              "fillOpacity": 0,
-              "gradientMode": "none",
-              "hideFrom": {
-                "legend": false,
-                "tooltip": false,
-                "viz": false
-              },
-              "lineInterpolation": "linear",
-              "lineWidth": 1,
-              "pointSize": 5,
-              "scaleDistribution": {
-                "type": "linear"
-              },
-              "showPoints": "auto",
-              "spanNulls": false,
-              "stacking": {
-                "group": "A",
-                "mode": "none"
-              },
-              "thresholdsStyle": {
-                "mode": "off"
-              }
-            },
-            "mappings": [],
-            "thresholds": {
-              "mode": "absolute",
-              "steps": [
-                {
-                  "color": "green",
-                  "value": null
-                },
-                {
-                  "color": "red",
-                  "value": 80
-                }
-              ]
-            },
-            "unit": "none"
-          },
-          "overrides": []
-        },
-        "gridPos": {
-          "h": 7,
-          "w": 6,
-          "x": 6,
-          "y": 14
-        },
-        "id": 5,
-        "options": {
-          "legend": {
-            "calcs": [],
-            "displayMode": "list",
-            "placement": "bottom",
-            "showLegend": false
-          },
-          "tooltip": {
-            "mode": "single",
-            "sort": "none"
-          }
-        },
-        "targets": [
-          {
-            "datasource": {
-              "type": "cloudwatch",
-              "uid": "b66e3d56-fb4e-49d6-9625-5d06ee69010b"
-            },
-            "dimensions": {
-              "LoadBalancer": "$lb"
-            },
-            "expression": "",
-            "id": "",
-            "label": "",
-            "logGroups": [],
-            "matchExact": true,
-            "metricEditorMode": 0,
-            "metricName": "RequestCount",
-            "metricQueryType": 0,
-            "namespace": "AWS/ApplicationELB",
-            "period": "",
-            "queryMode": "Metrics",
-            "refId": "A",
-            "region": "default",
-            "sqlExpression": "",
-            "statistic": "Sum"
-          }
-        ],
-        "title": "Request Count",
-        "type": "timeseries"
-      },
-      {
-        "datasource": {
-          "type": "cloudwatch",
-          "uid": "b66e3d56-fb4e-49d6-9625-5d06ee69010b"
-        },
-        "fieldConfig": {
-          "defaults": {
-            "color": {
-              "mode": "palette-classic"
-            },
-            "custom": {
-              "axisCenteredZero": false,
-              "axisColorMode": "text",
-              "axisLabel": "",
-              "axisPlacement": "auto",
-              "barAlignment": 0,
-              "drawStyle": "line",
-              "fillOpacity": 0,
-              "gradientMode": "none",
-              "hideFrom": {
-                "legend": false,
-                "tooltip": false,
-                "viz": false
-              },
-              "lineInterpolation": "linear",
-              "lineWidth": 1,
-              "pointSize": 5,
-              "scaleDistribution": {
-                "type": "linear"
-              },
-              "showPoints": "auto",
-              "spanNulls": false,
-              "stacking": {
-                "group": "A",
-                "mode": "none"
-              },
-              "thresholdsStyle": {
-                "mode": "off"
-              }
-            },
-            "mappings": [],
-            "thresholds": {
-              "mode": "absolute",
-              "steps": [
-                {
-                  "color": "green",
-                  "value": null
-                },
-                {
-                  "color": "red",
-                  "value": 80
-                }
-              ]
-            }
-          },
-          "overrides": [
-            {
-              "matcher": {
-                "id": "byName",
-                "options": "5XX"
-              },
-              "properties": [
-                {
-                  "id": "color",
-                  "value": {
-                    "fixedColor": "red",
-                    "mode": "fixed"
-                  }
-                }
-              ]
-            },
-            {
-              "matcher": {
-                "id": "byName",
-                "options": "2XX"
-              },
-              "properties": [
-                {
-                  "id": "color",
-                  "value": {
-                    "fixedColor": "green",
-                    "mode": "fixed"
-                  }
-                }
-              ]
-            }
-          ]
-        },
-        "gridPos": {
-          "h": 7,
-          "w": 6,
-          "x": 12,
-          "y": 14
-        },
-        "id": 6,
-        "options": {
-          "legend": {
-            "calcs": [],
-            "displayMode": "list",
-            "placement": "bottom",
-            "showLegend": true
-          },
-          "tooltip": {
-            "mode": "single",
-            "sort": "none"
-          }
-        },
-        "targets": [
-          {
-            "datasource": {
-              "type": "cloudwatch",
-              "uid": "b66e3d56-fb4e-49d6-9625-5d06ee69010b"
-            },
-            "dimensions": {
-              "LoadBalancer": "$lb"
-            },
-            "expression": "",
-            "hide": false,
-            "id": "",
-            "label": "2XX",
-            "logGroups": [],
-            "matchExact": true,
-            "metricEditorMode": 0,
-            "metricName": "HTTPCode_Target_2XX_Count",
-            "metricQueryType": 0,
-            "namespace": "AWS/ApplicationELB",
-            "period": "",
-            "queryMode": "Metrics",
-            "refId": "C",
-            "region": "default",
-            "sqlExpressio

@lukemelia
lukemelia marked this pull request as ready for review April 28, 2026 01:10
@lukemelia
lukemelia requested a review from Copilot April 28, 2026 01:12

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an automated “live vs committed” Grafana resources diff for observability changes, posting the result as a sticky PR comment to help reviewers validate dashboards/folders before merge and staging apply.

Changes:

  • Added packages/observability/scripts/diff.sh to pull live Grafana dashboards/folders into a temp dir and git diff --no-index against committed grafanactl/resources/.
  • Added .github/workflows/observability-diff.yml to run on PRs touching packages/observability/**, compute the diff against staging, and post/update a sticky PR comment (with truncation handling).

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 5 comments.

File Description
packages/observability/scripts/diff.sh Implements pull-to-tempdir + git diff output for dashboards/folders vs live Grafana.
.github/workflows/observability-diff.yml PR workflow to assume staging role, fetch Grafana token, compute diff, and post/update a sticky PR comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread .github/workflows/observability-diff.yml Outdated
Comment thread .github/workflows/observability-diff.yml Outdated
Comment thread packages/observability/scripts/diff.sh Outdated
Comment thread .github/workflows/observability-diff.yml
Comment thread .github/workflows/observability-diff.yml Outdated
1. Fork-PR guard. Workflow assumes an AWS role and fetches a Grafana
   service-account token; both should not run on PRs from forks (where
   the workflow code is contributor-controlled). Added
   `if: github.event.pull_request.head.repo.full_name == github.repository`
   at the job level. Same pattern other privileged workflows in the repo
   already use.

2. Pin actions/github-script to a SHA (60a0d83 = v7.0.1). The step has
   pull-requests: write and runs after assuming a cloud role —
   defense-in-depth pinning matters more here than for unprivileged
   actions. Other actions in this workflow are already SHA-pinned.

3. Truncate by bytes, not UTF-16 code units. `diff.length` undercounts
   when the diff contains non-ASCII characters (dashboard titles,
   descriptions). Switched to Buffer.byteLength(diff, 'utf8') for the
   threshold check, and walk back to a UTF-8 character boundary when
   slicing so we don't produce mid-codepoint garbage.

4. Paginate listComments. The marker-finding lookup previously stopped
   at 100 comments — on PRs with more than that, the existing sticky
   comment would be missed and a duplicate created on every push.
   Switched to github.paginate(github.rest.issues.listComments, ...).

5. diff.sh: distinguish git diff exit 1 (diff present, expected) from
   exit 2+ (real error, e.g., unreadable paths). Previously `|| true`
   swallowed all non-zero exits, masking real failures. Now wraps the
   diff invocation with `set +e`, captures the exit, and re-raises
   when > 1.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@lukemelia

Copy link
Copy Markdown
Contributor Author

Addressed all 5 Copilot suggestions in commit 30cc041:

  1. Fork-PR guard. Added `if: github.event.pull_request.head.repo.full_name == github.repository` at the job level. Workflow no longer runs on fork PRs (would have failed at OIDC anyway, but explicit guard is the right pattern; matches other privileged workflows in the repo).

  2. Pin `actions/github-script` to SHA (60a0d83 = v7.0.1). The step has `pull-requests: write` and runs after assuming a cloud role — defense-in-depth pinning matters more here than for unprivileged actions.

  3. Truncate by bytes, not UTF-16 code units. Switched to `Buffer.byteLength(diff, 'utf8')` for the threshold check, and walk back to a UTF-8 character boundary when slicing so the truncated comment never lands mid-codepoint.

  4. Paginate `listComments`. Switched to `github.paginate(github.rest.issues.listComments, ...)`. Marker-comment lookup is now reliable on PRs with >100 comments.

  5. `diff.sh` exit-code handling. Wrapped `git diff --no-index` with `set +e` and explicit exit-code check: 0/1 = success (no diff / diff found, expected), 2+ = real error and propagate. Previous `|| true` would have masked unreadable-paths or git internal errors as success.

@lukemelia
lukemelia changed the base branch from cs-10936-production-apply-workflow to main April 28, 2026 02:25
@lukemelia
lukemelia merged commit c2ea7ca into main Apr 28, 2026
24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants