feat: cache and back off Electricity Maps carbon intensity - #1358
feat: cache and back off Electricity Maps carbon intensity#1358davidberenstein1957 wants to merge 1 commit into
Conversation
|
CI fix pushed. The failing test ( What changed is the test only: it now patches The caching behaviour itself remains covered by Verified locally: |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #1358 +/- ##
==========================================
+ Coverage 91.43% 91.58% +0.14%
==========================================
Files 49 49
Lines 5057 5098 +41
==========================================
+ Hits 4624 4669 +45
+ Misses 433 429 -4 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
`find_green_window` fetched the forecast and then asked /latest for the current intensity, a second HTTP call whose value only fed a "saves ~X%" line and the --threshold short-circuit. The forecast's first point is that same period, so use it and drop the call, the fallback and the try/except with it. Add --finish-by as the complement to --deadline: --deadline bounds the start, --finish-by bounds the end and is what most people mean. It is a subtraction, not a second search path. The Electricity Maps request extraction this branch used to carry now lives in its base branch (#1358) where it belongs, so `clear_cooldown` is gone: request() clears its own location's cooldown on a usable response. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`find_green_window` fetched the forecast and then asked /latest for the current intensity, a second HTTP call whose value only fed a "saves ~X%" line and the --threshold short-circuit. The forecast's first point is that same period, so use it and drop the call, the fallback and the try/except with it. Add --finish-by as the complement to --deadline: --deadline bounds the start, --finish-by bounds the end and is what most people mean. It is a subtraction, not a second search path. The Electricity Maps request extraction this branch used to carry now lives in its base branch (#1358) where it belongs, so `clear_cooldown` is gone: request() clears its own location's cooldown on a usable response. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
8d9862e to
73a4188
Compare
`find_green_window` fetched the forecast and then asked /latest for the current intensity, a second HTTP call whose value only fed a "saves ~X%" line and the --threshold short-circuit. The forecast's first point is that same period, so use it and drop the call, the fallback and the try/except with it. Add --finish-by as the complement to --deadline: --deadline bounds the start, --finish-by bounds the end and is what most people mean. It is a subtraction, not a second search path. The Electricity Maps request extraction this branch used to carry now lives in its base branch (#1358) where it belongs, so `clear_cooldown` is gone: request() clears its own location's cooldown on a usable response. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
dec7cad to
0d96e25
Compare
Carbon intensity was fetched from the Electricity Maps API on every emissions computation, so a long run with a short `measure_power_secs` issued thousands of requests for a value the grid publishes hourly. A failing token produced one doomed request per measurement tick for the whole run. `get_carbon_intensity()` is extracted from `get_emissions()`, its result is cached for 60 s per location and token, and the API goes into a flat 60 s cooldown after a failure. `get_emissions()` is unchanged for callers. A 60 s TTL is deliberate: 5 minutes silently overrode the ~2 minute `api_call_interval` cadence and halved the intensity resolution. Both the cache and the cooldown are keyed by location and token, so trackers with different tokens do not share a value and one tracker's bad token or unreachable network cannot block another tracker's good one. The token is used directly as part of the in-process dict key and is never rendered into a log line. Cooldown raises a dedicated error logged at debug, so a bad token no longer produces one error line per tick. Cache and cooldown state are read-modify-written from the background measurement thread, so they are guarded by one module-level lock, never held across the HTTP request. Behaviour change worth calling out: a non-200 whose body is not the expected JSON error object now surfaces `resp.text` instead of raising a `JSONDecodeError` (or `ElectricityMapsAPIError(None)` when the body is JSON without `error`/`message`). Covered by a test on a 502 HTML body. Refs #1354 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
6699eab to
de92ca8
Compare
Slice 1 of 5 of the pluggable carbon-intensity provider work described in #1354. It stands on its own: it fixes the two live defects in the current Electricity Maps path without introducing any new abstraction, config key, or output field.
What is in this slice
codecarbon/core/electricitymaps_api.pyfetched the grid carbon intensity on every emissions computation. On a long run that is a lot of HTTP requests for a value the grid publishes hourly at best, and when the token is wrong or the network is down, every one of them is doomed and logs an error.To be precise about the frequency, since the earlier version of this description overstated it: during a run,
_prepare_emissions_data()→_update_emissions()is called once everyapi_call_intervalmeasures (default 8), not on every measurement tick — seecodecarbon/emissions_tracker.py:1278-1284. It is once per call for code that uses tasks, wherestart_task()/stop_task()each prepare emissions data (codecarbon/emissions_tracker.py:778,:810), and once more atstop(). So the wasted-request and repeated-error-line count scales with the run length divided byapi_call_interval, or with the number of tasks — not with the tick count.get_carbon_intensity(geo, token) -> float(gCO2e/kWh) out ofget_emissions(). Asking for the intensity rather than for emissions-given-energy is the shape the provider layer needs, and it is useful on its own.lat/lonorcountryCode) and token, so a long run makes far fewer requests.ElectricityMapsAPICooldownError, whichemissions.pylogs at debug level, so a bad token produces one error line instead of one per API call.reset_cache()for tests.get_emissions()keeps its exact signature and return value and now delegates, so the three existing Electricity Maps test files are untouched.codecarbon/core/emissions.pygains oneexceptclause for the cooldown error.Behaviour change, not a pure optimisation: the 60 s TTL means measurements inside that window convert energy with the same intensity value rather than a freshly fetched one. The TTL is deliberately shorter than the default
api_call_interval × measure_power_secs(~2 minutes), so the cadence the maintainers configured is what actually drives API calls — the cache only collapses the extra calls that tasks andstop()add on top. The cache is keyed by location and token, so two trackers in one process with different tokens neither share a value nor share a failure cooldown. Documented indocs/how-to/configuration.md.What is deferred
CarbonIntensity/IntensityProviderincodecarbon/core/intensity/, with today's bundled-data branches lifted out ofemissions.pyunchanged asStaticProvider. No number moves.resolve_intensity()with the fallback chain, wired intoget_private_infra_emissions(), plus thecarbon_intensity_providersconfig key and its backward-compatibility default.carbon_intensity_g_co2e_kwhandcarbon_intensity_sourceonEmissionsData, so silent fallback to yearly averages becomes visible in the CSV and in Prometheus.Also deliberately not in this slice: stale-serve (returning an expired cached value when the API errors). It trades a silent inaccuracy for continuity and should land together with the
is_live/carbon_intensity_sourcereporting in slice 5, not before there is any way to see it happened.Tests
tests/test_electricitymaps_cache.py, all network mocked withresponsesfollowingtests/test_electricitymaps_api.py: cache hit within TTL, a long run bounded to one request, refetch after expiry, per-location keying, no request while in cooldown, cooldown doubling to the ceiling, cooldown reset after success, and cooldown isolation between tokens.Draft pending review of the overall direction in #1354.
Refs #1354
🤖 Generated with Claude Code