diff --git a/di/heartbeat/deps.q b/di/heartbeat/deps.q new file mode 100644 index 00000000..42185407 --- /dev/null +++ b/di/heartbeat/deps.q @@ -0,0 +1,4 @@ +/ hard module dependencies and their minimum versions, validated by di.depcheck +/ di.heartbeat has no hard dependencies - all runtime dependencies (log, timer, +/ handlers, pubsub, servers) are injected via init as dictionaries of functions +deps:(`$())!(); diff --git a/di/heartbeat/heartbeat.md b/di/heartbeat/heartbeat.md new file mode 100644 index 00000000..f578f826 --- /dev/null +++ b/di/heartbeat/heartbeat.md @@ -0,0 +1,189 @@ +# di.heartbeat + +Periodic process-liveness signalling over pub/sub for kdb-x. Every process can publish a regular heartbeat so that downstream monitors detect when a process has stopped beating - i.e. it is stalled or blocked - even when the underlying connection is still valid. The module covers both sides: publishing heartbeats, and on the monitoring side storing received beats and raising a warning then an error when a process stops within configurable grace periods. + +--- + +## Features + +- Publish a periodic heartbeat row over pub/sub so downstream monitors can detect a stalled or blocked process even while its connection is still open +- Monitor side: subscribe to other processes' heartbeats and store the latest beat per process as inspectable module state +- Escalate from healthy to *warning* to *error* when a process stops heartbeating, using per-process-type grace periods (`warningtolerance`/`errortolerance` x `publishinterval`) +- Fire user-supplied `onwarning`/`onerror` callbacks with the rows entering each state +- Seed expected processes with `addprocs` so a never-seen process is flagged immediately +- Owns its own clock (`cp`, default `.z.p`), overridable via `setcp` for deterministic tests or simulation +- Idempotent `init` - re-running clears and re-registers its timer jobs safely +- No hard module dependencies - log, timer, pubsub (and servers/handlers when monitoring) are all injected via `init` + +--- + +## Dependencies + +| Dependency | Key | Required | Description | +|---|---|---|---| +| logger | `` `log `` | yes | `info`/`warn`/`error` - binary `{[c;m]}` where `c` is a symbol context and `m` is a string. Heartbeat uses all three. A `kx.log` instance is accepted directly | +| timer | `` `timer `` | yes | `addjob`, `deletejobs` - schedules the publish / check / subscribe jobs (`deletejobs` lets `init` be re-run safely). The full `di.timer` dict may be passed | +| pubsub | `` `pubsub `` | yes | `publish` (`{[table;data]}`), `subscribe` (`{[handle]}`) - publishing heartbeats / subscribing to publishers | +| servers | `` `servers `` | when `subenabled` | `getservers` (`{[proctype]}` returning handles) - discovering heartbeat publishers by process type | +| handlers | `` `handlers `` | when `subenabled` | `register` - wiring the connection-close (`.z.pc`) cleanup | + +**Hard dependency:** none - all runtime dependencies are injected via `init`, so any module exporting the contracted signatures can be supplied. + +The dependencies are passed to `init` inside the single `deps` dict alongside any configuration. They are **required**: `init` throws immediately if `deps` is not a dictionary or a required dependency is missing or malformed. `servers` and `handlers` are only required when `subenabled` is set (i.e. this process monitors others); a pure publisher needs only `log`, `timer` and `pubsub`. + +**Logging contract.** Internally the module calls the logger as binary `.z.m.log[\`info][\`heartbeat;"msg"]` (`{[c;m]}` - context symbol + message). `info`/`warn`/`error` are all mandated because heartbeat uses all three. You may pass either a `kx.log` instance (`(use\`kx.log).createLog[]`) - its monadic `{[msg]}` functions are detected and auto-wrapped to the binary contract by the internal `normlog`, folding the context tag into the message (`"heartbeat: ..."`) - or a custom `` `info`warn`error `` dict of `{[c;m]}` functions, used as-is. + +The module keeps its **own** current-time function rather than taking it from the timer dependency. It defaults to `.z.p`; override it with `setcp` for deterministic tests or simulation, e.g. `heartbeat.setcp[{2025.01.01D00:00:00.000}]`. + +--- + +## Initialisation + +`init[deps]` takes a single dictionary combining the injected dependencies (above) with any configuration overrides. All config keys are optional - omit any and the module falls back to the default; unrecognised keys are ignored. + +| Key | Default | Description | +|---|---|---| +| `` `enabled `` | `1b` | publish and check heartbeats | +| `` `subenabled `` | `0b` | act as a monitor: subscribe to other heartbeats and register the disconnect handler | +| `` `debug `` | `1b` | log warning / error transitions (callbacks still fire when off) | +| `` `publishinterval `` | `0D00:00:30` | how often heartbeats are published | +| `` `checkinterval `` | `0D00:00:10` | how often received heartbeats are checked | +| `` `warningtolerance `` | `1.5` | warning after `warningtolerance*publishinterval` without a beat | +| `` `errortolerance `` | `2f` | error after `errortolerance*publishinterval` without a beat | +| `` `proctype `` | `` `unknown `` | this process's type (published as `sym`) | +| `` `procname `` | `.z.h` | this process's name | +| `` `pid `` `` `host `` `` `port `` | from `.z` | this process's identity | +| `` `connections `` | `` `$() `` | process types this monitor subscribes to (used by the subscribe job) | +| `` `onwarning `` | no-op | callback invoked with the rows entering warning state | +| `` `onerror `` | no-op | callback invoked with the rows entering error state | + +`init` must be called before any other function. It applies config, wires dependencies, and schedules the timer jobs. Re-running `init` resets unset config keys to their defaults and re-registers the timer jobs. + +--- + +## Exported Functions + +### `init[deps]` +Wire config + dependencies (one dict) and schedule the timer jobs. Must be called before anything else. +```q +heartbeat.init[`proctype`procname`log`timer`pubsub!(`rdb;`rdb1;kxlog;timerdep;psdep)] +/ or as a monitor: +heartbeat.init[`subenabled`connections`log`timer`pubsub`servers`handlers!(1b;`rdb`hdb;kxlog;timerdep;psdep;serversdep;handlersdep)] +``` + +### `publishheartbeat[]` +Publish a single heartbeat row over pub/sub and increment the counter. Normally driven by the timer; a no-op when `enabled` is `0b`. +```q +heartbeat.publishheartbeat[] +``` + +### `checkheartbeat[]` +Flag processes that have not heartbeated within the warning / error grace periods, firing `onwarning`/`onerror` on transitions. Driven by the timer on the monitor. +```q +heartbeat.checkheartbeat[] +``` + +### `storeheartbeat[batch]` +Store one or more incoming heartbeats, keeping the latest per process and clearing warning / error state. Call from `upd` on the monitor. +```q +upd:{[t;x] if[t~`heartbeat; heartbeat.storeheartbeat[x]]; } +``` + +### `addprocs[proctypes;procnames]` +Seed the store with expected processes so a never-seen process is flagged. Real heartbeats arriving later override the seeded rows. +```q +heartbeat.addprocs[`rdb`hdb; `rdb1`hdb1] +``` + +### `subscribe[handles]` +Subscribe to heartbeats on the given remote handle(s), tracking successful subscriptions and skipping any that fail. +```q +heartbeat.subscribe hopen `:remotehost:5050 +``` + +### `gethb[]` +Return the current heartbeat store (keyed on `sym`,`procname`) for inspection. +```q +heartbeat.gethb[] +``` + +### `setcp[f]` +Replace the current-time function (for deterministic tests / simulation). Defaults to `.z.p`. +```q +heartbeat.setcp[{2025.01.01D00:00:00.000}] +``` + +--- + +## Heartbeat store schema + +`gethb[]` returns the store, keyed on `sym`,`procname`: + +| Column | Type | Description | +|---|---|---| +| sym | `symbol` | process type | +| procname | `symbol` | process name | +| time | `timestamp` | time of the last received heartbeat | +| counter | `long` | counter from the last heartbeat | +| pid | `int` | publisher process id | +| host | `symbol` | publisher host | +| port | `int` | publisher port | +| warning | `boolean` | process is in warning state | +| error | `boolean` | process is in error state | + +--- + +## Usage Example + +```q +/ --- publisher --- +kxlog:use`kx.log + +heartbeat:use`di.heartbeat + +timer:use`di.timer +timer.init[()!()] +/ heartbeat needs addjob and deletejobs - it keeps its own clock (see setcp) +/ note: di.timer's addjob is a namespace; addjob.custom has the [id;func;params;period;mode;opts] signature heartbeat calls +timerdep:`addjob`deletejobs!(timer.addjob.custom; timer.deletejobs) + +/ pubsub must provide publish[table;data] and subscribe[handle] +pubsub:use`di.pubsub +psdep:`publish`subscribe!(pubsub.publish; {[h] h(`.m.di.0pubsub.subscribe;`heartbeat;`)}) + +/ initialise from a single dict - config keys + the (required) log/timer/pubsub dependencies +/ the kx.log instance is passed straight through; normlog wraps it to the binary contract +heartbeat.init[`proctype`procname`log`timer`pubsub!(`rdb;`rdb1;kxlog.createLog[];timerdep;psdep)] + +/ publish a heartbeat immediately (normally the timer does this) +heartbeat.publishheartbeat[] +``` + +On the monitoring side, route received heartbeats into the store from `upd` and let the scheduled `checkheartbeat` raise warnings / errors: + +```q +upd:{[t;x] if[t~`heartbeat; heartbeat.storeheartbeat[x]]; } +heartbeat.gethb[] +``` + +--- + +## Running Tests + +```q +k4unit:use`di.k4unit +k4unit.moduletest`di.heartbeat +``` + +The test suite injects no-op binary mock loggers and a capturing logger that records messages for assertion. It covers: dependency validation (non-dict deps throws; missing/non-dict `log` throws; `log` missing a level throws; missing `timer`/`pubsub` throws; missing monitor `servers`/`handlers` throws); the capturing-logger test confirming `init` logs `"di.heartbeat initialised"`; `normlog` wrapping a fake `kx.log` instance end-to-end (and extra log levels passing through untouched); idempotency of re-running `init`; and a real-process integration test driving a background publisher over a live handle. + +--- + +## Notes + +- The module keeps its own clock (`cp`, default `.z.p`) rather than taking one from the timer dependency - override it with `setcp` for deterministic tests or simulation +- Timer jobs use mode 2 (period after the previous actual start) so missed beats are not replayed as a catch-up storm +- `init` is idempotent: it deletes its jobs (`hbpublish`/`hbcheck`/`hbsubscribe`) before re-registering, so it is safe to call again +- `servers` and `handlers` are only required when `subenabled` is set; a pure publisher needs only `log`, `timer` and `pubsub` +- A `kx.log` instance is accepted directly - `normlog` wraps its monadic functions to the binary `{[c;m]}` contract and folds the context tag into the message +- `debug` toggles whether warning / error transitions are logged; the `onwarning`/`onerror` callbacks still fire either way diff --git a/di/heartbeat/heartbeat.q b/di/heartbeat/heartbeat.q new file mode 100644 index 00000000..201e9cb5 --- /dev/null +++ b/di/heartbeat/heartbeat.q @@ -0,0 +1,271 @@ +/ heartbeat module for kdb-x +/ every process can publish a periodic heartbeat over pub/sub so that downstream +/ monitors can detect when a process has stopped beating - i.e. it is stalled or +/ blocked - even when the underlying connection is still valid +/ the module handles both publishing heartbeats and, on the monitoring side, +/ storing received heartbeats and raising warnings / errors when they stop +/ config and dependencies are passed to init in a single dictionary: config keys (see heartbeat.md) +/ are optional with defaults; log/timer/pubsub are required (servers/handlers when monitoring) and +/ init errors immediately if a required dependency is missing - see heartbeat.md +/ module-local state convention: read config bare, mutate via .z.m, and access +/ injected dependencies via .z.m at every call site + +/ ============================================================ +/ module state and defaults +/ ============================================================ + +/ table used to publish heartbeats - sym holds the publishing process type +heartbeat:([] time:`timestamp$(); sym:`symbol$(); procname:`symbol$(); counter:`long$(); pid:`int$(); host:`symbol$(); port:`int$()); + +/ keyed store of the latest received heartbeat per process, with warning / error state +hb:update warning:0b,error:0b from `sym`procname xkey heartbeat; + +/ remote handles we have already subscribed to for heartbeats +subscribedhandles:`int$(); + +/ heartbeat counter - bumped on each publish +hbcounter:0; + +/ current-time function - heartbeat owns its clock; override via setcp for testing / simulation +cp:{.z.p}; + +/ configuration defaults - overridden by the config dictionary passed to init +enabled:1b; / whether heartbeat publishing / checking is enabled +subenabled:0b; / whether this process monitors (subscribes to) other heartbeats +debug:1b; / whether to log warning / error transitions +publishinterval:0D00:00:30; / how often heartbeats are published +checkinterval:0D00:00:10; / how often received heartbeats are checked +warningtolerance:1.5; / warning after warningtolerance*publishinterval without a beat +errortolerance:2f; / error after errortolerance*publishinterval without a beat +proctype:`unknown; / this process's type (published as sym) +procname:.z.h; / this process's name +pid:.z.i; / this process's pid +host:.z.h; / this process's host +port:`int$system"p"; / this process's port +connections:`$(); / process types this monitor should subscribe to +onwarning:{[procs]}; / callback fired with the rows entering warning state +onerror:{[procs]}; / callback fired with the rows entering error state + +/ ============================================================ +/ internal helpers +/ ============================================================ + +normlog:{[logdict] + / detect kx.log instance by presence of kx.log-specific keys (getlvl, sinks, fmts) + / kx.log functions are monadic - wrap each into binary {[c;m]} and embed context in the message + / plain {[c;m]} log dicts (info`warn`error only) pass through unchanged + $[any `getlvl`sinks`fmts in key logdict; + `info`warn`error!( + {[fn;c;m] fn[string[c],": ",m]}[logdict`info;]; + {[fn;c;m] fn[string[c],": ",m]}[logdict`warn;]; + {[fn;c;m] fn[string[c],": ",m]}[logdict`error;]); + logdict] + }; + +/ extract a required dependency dictionary, erroring immediately if absent or null +requiredep:{[deps;name] + d:$[99h=type deps;$[(name in key deps) and not (::)~deps name;deps name;()!()];()!()]; + if[not count d; + '"di.heartbeat: ",(string name)," dependency is required; pass it via init deps - see di.",string name]; + d + }; + +/ warning / error grace periods - vary by process type if required +warningperiod:{[processtype] `timespan$warningtolerance*publishinterval}; +errorperiod:{[processtype] `timespan$errortolerance*publishinterval}; + +/ convert a timespan into whole seconds for the timer period +tosecs:{[span] `int$span%0D00:00:01}; + +/ wire the injected dependencies from the single deps dict (which also carries config keys) +setdeps:{[deps] + / log, timer and pubsub are required; servers and handlers only when monitoring + / init errors immediately if deps is not a dictionary or a required dependency is missing/malformed + / nested if guards (not and) - and evaluates both sides eagerly and key would throw on a non-dict + if[99h<>type deps; + '"di.heartbeat: deps must be a dictionary of config and injected dependencies - see heartbeat.md"]; + / log - required: info/warn/error (heartbeat uses all three); a kx.log instance is auto-wrapped + / to the binary {[c;m]} contract by normlog, so it can be passed directly + if[not `log in key deps; + '"di.heartbeat: log dependency is required; pass `info`warn`error (or a kx.log logger) keyed on `log"]; + if[99h<>type deps`log; + '"di.heartbeat: log must be a dict of info/warn/error functions (or a kx.log logger)"]; + lg:normlog deps`log; + if[not all `info`warn`error in key lg; + '"di.heartbeat: log must provide info/warn/error; got: ",", " sv string key lg]; + .z.m.log:lg; + timerdict:requiredep[deps;`timer]; + .z.m.timeraddjob:timerdict`addjob; + .z.m.timerdeletejobs:timerdict`deletejobs; + pubsubdict:requiredep[deps;`pubsub]; + .z.m.pubsubpublish:pubsubdict`publish; + .z.m.pubsubsubscribe:pubsubdict`subscribe; + / monitor-only deps are wired via a separate function so the conditional stays a + / single statement - the style guide says avoid block statements within conditionals + if[subenabled;setmonitordeps deps]; + }; + +/ wire the monitor-only dependencies, required only when subenabled (this process monitors others) +setmonitordeps:{[deps] + / split out of setdeps to keep that conditional a single statement per the coding standards + serversdict:requiredep[deps;`servers]; + .z.m.serversgetservers:serversdict`getservers; + handlersdict:requiredep[deps;`handlers]; + .z.m.handlersregister:handlersdict`register; + }; + +/ apply recognised config overrides from the deps dict, defaulting where a key is absent +setconfig:{[deps] + / explicit per-key .z.m assignment (like di.eodtime) - no .z.M, and reinit resets to defaults + cfg:$[99h=type deps;deps;()!()]; + .z.m.enabled:$[`enabled in key cfg;cfg`enabled;1b]; + .z.m.subenabled:$[`subenabled in key cfg;cfg`subenabled;0b]; + .z.m.debug:$[`debug in key cfg;cfg`debug;1b]; + .z.m.publishinterval:$[`publishinterval in key cfg;cfg`publishinterval;0D00:00:30]; + .z.m.checkinterval:$[`checkinterval in key cfg;cfg`checkinterval;0D00:00:10]; + .z.m.warningtolerance:$[`warningtolerance in key cfg;cfg`warningtolerance;1.5]; + .z.m.errortolerance:$[`errortolerance in key cfg;cfg`errortolerance;2f]; + .z.m.proctype:$[`proctype in key cfg;cfg`proctype;`unknown]; + .z.m.procname:$[`procname in key cfg;cfg`procname;.z.h]; + .z.m.pid:$[`pid in key cfg;cfg`pid;.z.i]; + .z.m.host:$[`host in key cfg;cfg`host;.z.h]; + .z.m.port:$[`port in key cfg;cfg`port;`int$system"p"]; + .z.m.connections:$[`connections in key cfg;cfg`connections;`$()]; + .z.m.onwarning:$[`onwarning in key cfg;cfg`onwarning;{[procs]}]; + .z.m.onerror:$[`onerror in key cfg;cfg`onerror;{[procs]}]; + }; + +/ schedule the periodic heartbeat jobs via the injected timer +registertimers:{ + / mode 2 = period after previous actual start - a heartbeat says "alive now", so + / missed beats must not be replayed as a catch-up storm (which mode 1 would do) + / clear any previously-registered jobs first so init is safe to call again + .z.m.timerdeletejobs[`hbpublish`hbcheck`hbsubscribe]; + if[enabled; + .z.m.timeraddjob[`hbpublish;publishheartbeat;();tosecs publishinterval;2;()!()]; + .z.m.timeraddjob[`hbcheck;checkheartbeat;();tosecs checkinterval;2;()!()]]; + if[subenabled; + .z.m.timeraddjob[`hbsubscribe;hbsubscriptions;();60;2;()!()]]; + }; + +/ wire the connection-close cleanup through the injected handler manager +registerhandlers:{ + if[subenabled; + .z.m.handlersregister[`.z.pc;`heartbeat;closeconnection]]; + }; + +/ log a single process moving into warning state +logwarnproc:{[r] + .z.m.log[`warn][`heartbeat;"process ",(string r`procname)," (type ",(string r`sym),") has not heartbeated since ",string r`time]; + }; + +/ log a single process moving into error state +logerrproc:{[r] + .z.m.log[`error][`heartbeat;"process ",(string r`procname)," (type ",(string r`sym),") has not heartbeated since ",string r`time]; + }; + +/ move processes into warning state, log and fire the warning callback +warn:{[procs] + if[debug;logwarnproc each 0!procs]; + .z.m.hb:hb upsert select sym,procname,warning:1b from procs; + onwarning procs; + }; + +/ move processes into error state, log and fire the error callback +err:{[procs] + if[debug;logerrproc each 0!procs]; + .z.m.hb:hb upsert select sym,procname,error:1b from procs; + onerror procs; + }; + +/ subscribe to a single remote heartbeat publisher, logging and skipping on failure +subscribeone:{[h] + ok:@[{.z.m.pubsubsubscribe x;1b};h;{[h;e] .z.m.log[`error][`heartbeat;"failed to subscribe to heartbeats on handle ",(string h),": ",e];0b}[h]]; + if[ok;.z.m.subscribedhandles:distinct subscribedhandles,h]; + }; + +/ subscribe to publishers of the given process type(s) that are not yet subscribed +getheartbeats:{[proctype] + handles:(.z.m.serversgetservers proctype) except subscribedhandles; + if[count handles; + .z.m.log[`info][`heartbeat;"subscribing to new heartbeat handle(s) ",", " sv string handles]; + subscribe handles]; + }; + +/ subscribe to all configured heartbeat publishers (by configured process type) - timer job +hbsubscriptions:{ + getheartbeats connections; + }; + +/ drop a closed handle from the tracked subscriptions - registered against .z.pc +closeconnection:{[h] + .z.m.subscribedhandles:subscribedhandles except h; + }; + +/ ============================================================ +/ public api +/ ============================================================ + +/ publish a single heartbeat row over pub/sub and bump the counter +publishheartbeat:{ + if[not enabled;:()]; + .z.m.pubsubpublish[`heartbeat;enlist `time`sym`procname`counter`pid`host`port!(cp[];proctype;procname;hbcounter;pid;host;port)]; + .z.m.hbcounter:hbcounter+1; + }; + +/ flag processes that have not heartbeated within the warning / error grace periods - timer job +checkheartbeat:{ + / status: 0 healthy, 1 warning, 2+ error + / grace periods are computed as locals first - module functions do not resolve inside qsql + now:cp[]; + t:0!hb; + wp:warningperiod each t`sym; + ep:errorperiod each t`sym; + stats:update status:(`short$now>time+wp)+`short$2*now>time+ep from t; + newwarn:select sym,procname,time from stats where status=1,not warning; + newerr:select sym,procname,time from stats where status>1,not error; + if[count newwarn;warn newwarn]; + if[count newerr;err newerr]; + }; + +/ store one or more incoming heartbeats, keeping the latest per process and clearing warning / error state +storeheartbeat:{[batch] + / call this from upd when a heartbeat arrives + .z.m.hb:hb upsert update warning:0b,error:0b from select by sym,procname from batch; + }; + +/ seed the store with expected processes so a never-seen process is flagged +addprocs:{[proctypes;procnames] + / real heartbeats arriving later override these seeded rows + seed:2!([]sym:proctypes,();procname:procnames,();time:cp[];counter:0N;pid:0Ni;host:`;port:0Ni;warning:0b;error:0b); + .z.m.hb:seed,hb; + }; + +/ subscribe to heartbeats on the given remote handle(s), tracking successful subscriptions +subscribe:{[handles] + subscribeone each (),handles; + }; + +/ return the current heartbeat store for inspection +gethb:{hb}; + +/ replace the current-time function (used by tests and simulation) +setcp:{[f] .z.m.cp:f}; + +init:{[deps] + / initialise from a single dictionary holding config overrides and injected dependencies - see heartbeat.md + / config keys (see heartbeat.md) are optional and fall back to defaults; dependencies are required: + / `log - a logger - required; must provide info/warn/error (a kx.log instance is auto-wrapped) + / `timer - `addjob`deletejobs (full di.timer dict may be passed) - required + / `pubsub - `publish`subscribe - required + / `servers - `getservers - required when subenabled + / `handlers - `register`remove`list - required when subenabled + / note: the module keeps its own clock (cp, default .z.p) - override via setcp + / example: + / heartbeat.init[`proctype`procname`log`timer`pubsub!(`rdb;`rdb1;kxlog;timerdep;psdep)] + setconfig deps; + setdeps deps; + registertimers[]; + registerhandlers[]; + .z.m.log[`info][`heartbeat;"di.heartbeat initialised"]; + }; diff --git a/di/heartbeat/init.q b/di/heartbeat/init.q new file mode 100644 index 00000000..a004ce22 --- /dev/null +++ b/di/heartbeat/init.q @@ -0,0 +1,8 @@ +/ load core functionality into the module +\l ::heartbeat.q + +/ module version - compared against dependants' minimum requirements by di.depcheck. Removed for now until PR merged +/version:"0.1.0"; + +/ public api - only the functions intended to be called externally are exported +export:([init;publishheartbeat;checkheartbeat;storeheartbeat;addprocs;subscribe;gethb;setcp]) diff --git a/di/heartbeat/test.csv b/di/heartbeat/test.csv new file mode 100644 index 00000000..27396a11 --- /dev/null +++ b/di/heartbeat/test.csv @@ -0,0 +1,123 @@ +action,ms,bytes,lang,code,repeat,minver,comment +comment,,,,,,,Setup - load module and inject mock dependencies (single deps dict holds config + deps) +before,0,0,q,heartbeat:use`di.heartbeat,1,1,load the heartbeat module +before,0,0,q,logdep:`info`warn`error!({[c;m]};{[c;m]};{[c;m]}),1,1,silent binary log mock - {[c;m]} is the internal contract +before,0,0,q,.test.now:2025.01.01D00:00:00.000,1,1,controllable current time for the clock +before,0,0,q,timerdep:`addjob`deletejobs!({[id;func;params;period;mode;opts]};{[ids]}),1,1,timer mock - heartbeat requires addjob and deletejobs +before,0,0,q,psdep:`publish`subscribe!({[t;x] .test.pubt:t;.test.pubx:x;.test.pubcount:.test.pubcount+1};{[h] .test.subh:h}),1,1,pubsub mock capturing publish and subscribe +before,0,0,q,.test.pubcount:0,1,1,initialise publish counter +before,0,0,q,deps:`log`timer`pubsub!(logdep;timerdep;psdep),1,1,dependency-only dict (reused by the required-dependency fail tests) +before,0,0,q,heartbeat.init[`proctype`procname`log`timer`pubsub!(`rdb;`rdb1;logdep;timerdep;psdep)],1,1,initialise from a single dict of config + deps +before,0,0,q,heartbeat.setcp[{[].test.now}],1,1,point heartbeat clock at the controllable test time + +comment,,,,,,,publishheartbeat publishes a row and bumps the counter +run,0,0,q,.test.pubcount:0,1,1,reset publish counter +run,0,0,q,heartbeat.publishheartbeat[],1,1,publish a heartbeat +true,0,0,q,1=.test.pubcount,1,1,publish dependency was called once +true,0,0,q,`heartbeat~.test.pubt,1,1,published to the heartbeat table +true,0,0,q,`rdb~(first .test.pubx)`sym,1,1,published row carries the process type +true,0,0,q,1=.m.di.0heartbeat.hbcounter,1,1,counter incremented + +comment,,,,,,,disabled publishing is a no-op +run,0,0,q,.m.di.0heartbeat.enabled:0b,1,1,disable heartbeating +run,0,0,q,.test.pubcount:0,1,1,reset publish counter +run,0,0,q,heartbeat.publishheartbeat[],1,1,attempt to publish while disabled +true,0,0,q,0=.test.pubcount,1,1,nothing published when disabled +run,0,0,q,.m.di.0heartbeat.enabled:1b,1,1,re-enable heartbeating + +comment,,,,,,,storeheartbeat records incoming beats with cleared state +run,0,0,q,.m.di.0heartbeat.hb:0#.m.di.0heartbeat.hb,1,1,clear the store +run,0,0,q,.test.batch:([]time:enlist .test.now;sym:enlist`rdb;procname:enlist`rdb1;counter:enlist 1;pid:enlist 1i;host:enlist`h;port:enlist 5000i),1,1,build an incoming heartbeat +run,0,0,q,heartbeat.storeheartbeat[.test.batch],1,1,store the heartbeat +true,0,0,q,1=count heartbeat.gethb[],1,1,one process recorded +true,0,0,q,not first exec warning from heartbeat.gethb[],1,1,stored beat has warning cleared + +comment,,,,,,,addprocs seeds an expected process +run,0,0,q,heartbeat.addprocs[`rdb;`rdb2],1,1,seed an expected process +true,0,0,q,`rdb2 in exec procname from heartbeat.gethb[],1,1,seeded process present + +comment,,,,,,,checkheartbeat flags warning then error as time passes +run,0,0,q,.m.di.0heartbeat.hb:0#.m.di.0heartbeat.hb,1,1,clear the store +run,0,0,q,.test.now:2025.01.01D00:00:00.000,1,1,reset time to t0 +run,0,0,q,heartbeat.addprocs[`rdb;`stale],1,1,seed a process that will go stale +run,0,0,q,.test.now:2025.01.01D00:00:50.000,1,1,advance 50s past the 45s warning period +run,0,0,q,heartbeat.checkheartbeat[],1,1,run the check +true,0,0,q,first exec warning from heartbeat.gethb[] where procname=`stale,1,1,process moved to warning +true,0,0,q,not first exec error from heartbeat.gethb[] where procname=`stale,1,1,not yet in error +run,0,0,q,.test.now:2025.01.01D00:02:00.000,1,1,advance past the 60s error period +run,0,0,q,heartbeat.checkheartbeat[],1,1,run the check again +true,0,0,q,first exec error from heartbeat.gethb[] where procname=`stale,1,1,process moved to error + +comment,,,,,,,subscribe tracks handles and closeconnection removes them +run,0,0,q,.m.di.0heartbeat.subscribedhandles:`int$(),1,1,clear tracked handles +run,0,0,q,heartbeat.subscribe[5i],1,1,subscribe to a remote handle +true,0,0,q,5i in .m.di.0heartbeat.subscribedhandles,1,1,handle tracked after subscribe +true,0,0,q,5i~.test.subh,1,1,subscribe dependency was invoked with the handle +run,0,0,q,.m.di.0heartbeat.closeconnection[5i],1,1,simulate connection close +true,0,0,q,not 5i in .m.di.0heartbeat.subscribedhandles,1,1,handle removed on close + +comment,,,,,,,log is required - init errors clearly when deps or log is missing or malformed +fail,0,0,q,heartbeat.init[(::)],1,1,errors when deps is not a dictionary +fail,0,0,q,heartbeat.init[()!()],1,1,errors when log dependency missing +fail,0,0,q,heartbeat.init[enlist[`log]!enlist 42],1,1,errors when log value is not a dictionary +fail,0,0,q,heartbeat.init[enlist[`log]!enlist `info`warn!(logdep`info;logdep`warn)],1,1,errors when log dict is missing the error key + +comment,,,,,,,timer and pubsub are required - init errors when they are missing +fail,0,0,q,heartbeat.init[enlist[`log]!enlist logdep],1,1,errors when timer dependency missing +fail,0,0,q,heartbeat.init[`log`timer!(logdep;timerdep)],1,1,errors when pubsub dependency missing +fail,0,0,q,heartbeat.init[`subenabled`log`timer`pubsub!(1b;logdep;timerdep;psdep)],1,1,monitor errors when servers dependency missing +run,0,0,q,heartbeat.init[`proctype`procname`subenabled`log`timer`pubsub!(`rdb;`rdb1;0b;logdep;timerdep;psdep)],1,1,re-init with full deps and subenabled 0b to restore clean module state + +comment,,,,,,,an injected (binary) logger is actually invoked +run,0,0,q,.test.loginfo:"",1,1,reset the capture +run,0,0,q,caplog:`info`warn`error!({[c;m] .test.loginfo:m};{[c;m] .test.logwarn:m};{[c;m] .test.logerr:m}),1,1,capturing binary logger - records the last message at each level +run,0,0,q,heartbeat.init[`proctype`procname`subenabled`log`timer`pubsub!(`rdb;`rdb1;0b;caplog;timerdep;psdep)],1,1,init with the capturing logger +true,0,0,q,.test.loginfo~"di.heartbeat initialised",1,1,init message routed through the injected logger + +comment,,,,,,,a kx.log instance (getlvl/sinks/fmts + unary fns) is auto-wrapped to the binary contract by normlog +run,0,0,q,.test.kxmsg:"",1,1,reset the capture +run,0,0,q,fakekx:`info`warn`error`getlvl`sinks`fmts!({[m] .test.kxmsg:m};{[m]};{[m]};`info;()!();()!()),1,1,fake kx.log instance - unary level fns plus the marker keys +run,0,0,q,heartbeat.init[`proctype`procname`subenabled`log`timer`pubsub!(`rdb;`rdb1;0b;fakekx;timerdep;psdep)],1,1,init with the kx.log-style instance +true,0,0,q,.test.kxmsg~"heartbeat: di.heartbeat initialised",1,1,unary fn wrapped to binary - context folded into the message +true,0,0,q,`info`warn`error~key .m.di.0heartbeat.log,1,1,wrapped logger exposes exactly the binary info/warn/error + +comment,,,,,,,a binary logger with extra levels is accepted and passed through unchanged +run,0,0,q,caplog2:`info`warn`error`debug!({[c;m]};{[c;m]};{[c;m]};{[c;m]}),1,1,binary logger providing an extra debug level +run,0,0,q,heartbeat.init[`proctype`procname`subenabled`log`timer`pubsub!(`rdb;`rdb1;0b;caplog2;timerdep;psdep)],1,1,init with the richer binary logger +true,0,0,q,`debug in key .m.di.0heartbeat.log,1,1,extra level retained - a non-kx.log dict is passed through unchanged +true,0,0,q,all `info`warn`error in key .m.di.0heartbeat.log,1,1,mandatory levels present + +comment,,,,,,,init is idempotent - re-running clears jobs first instead of colliding on ids +run,0,0,q,.test.jobids:(`$())!`boolean$(),1,1,track registered job ids +run,0,0,q,timerdep2:`addjob`deletejobs!({[id;func;params;period;mode;opts] if[id in key .test.jobids;'"duplicate job id"]; .test.jobids[id]:1b};{[ids] .test.jobids:(key[.test.jobids] except ids)#.test.jobids}),1,1,stateful timer mock that rejects duplicate job ids like di.timer +run,0,0,q,heartbeat.init[`proctype`procname`subenabled`log`timer`pubsub!(`rdb;`rdb1;0b;logdep;timerdep2;psdep)],1,1,first init registers the jobs +run,0,0,q,heartbeat.init[`proctype`procname`subenabled`log`timer`pubsub!(`rdb;`rdb1;0b;logdep;timerdep2;psdep)],1,1,second init must not collide - clears then re-adds +true,0,0,q,`hbcheck`hbpublish~asc key .test.jobids,1,1,exactly the publish and check jobs are registered after re-init + +comment,,,,,,,real-process integration - a background publisher beats over real di.pubsub into this process +run,0,0,q,system"q -p 5050 -q &",1,,start a background heartbeat publisher +run,0,0,q,system"sleep 1",1,,wait for the publisher to come up +run,0,0,q,h:hopen`::5050,1,,connect to the publisher +true,0,0,q,h>0,1,,connected to the publisher +run,0,0,q,h"hbmod:use`di.heartbeat",1,,load heartbeat on the publisher +run,0,0,q,h"pubsub:use`di.pubsub",1,,load pubsub on the publisher +run,0,0,q,h"heartbeat:.m.di.0heartbeat.heartbeat",1,,expose the heartbeat schema at root for pubsub +run,0,0,q,h"pubsub.init[]",1,,initialise the pubsub registry on the publisher +run,0,0,q,h"logdep:`info`warn`error!({[c;m]};{[c;m]};{[c;m]})",1,,silent binary log dep on the publisher +run,0,0,q,h"timerdep:`addjob`deletejobs!({[id;func;params;period;mode;opts]};{[ids]})",1,,no-op timer dep - publish is triggered manually for determinism +run,0,0,q,h"psdep:`publish`subscribe!(pubsub.publish;{[x]})",1,,inject the real pubsub publish into heartbeat +run,0,0,q,h"hbmod.init[`proctype`procname`port`log`timer`pubsub!(`rdb;`pub1;5050i;logdep;timerdep;psdep)]",1,,init the publisher as an rdb named pub1 from a single dict +run,0,0,q,upd:{[t;x] if[`heartbeat~t; heartbeat.storeheartbeat x]},1,,route incoming beats into the local store +run,0,0,q,.m.di.0heartbeat.hb:0#.m.di.0heartbeat.hb,1,,clear the local store before subscribing +run,0,0,q,h(`pubsub.subscribe;`heartbeat;`),1,,subscribe this process to the publisher's heartbeat feed +run,0,0,q,h".m.di.0heartbeat.publishheartbeat[]",1,,publisher emits one heartbeat over real ipc +true,0,0,q,1=count heartbeat.gethb[],1,,beat received and stored via real pubsub +true,0,0,q,`pub1 in exec procname from heartbeat.gethb[],1,,stored beat carries the remote publisher identity +run,0,0,q,heartbeat.setcp[{.z.p}],1,,use real time - the just-received beat is fresh +run,0,0,q,heartbeat.checkheartbeat[],1,,check staleness while the beat is fresh +true,0,0,q,not first exec error from heartbeat.gethb[] where procname=`pub1,1,,fresh beat is not in error +run,0,0,q,heartbeat.setcp[{.z.p+0D01:00:00}],1,,skew the clock 1h ahead so the beat looks stale +run,0,0,q,heartbeat.checkheartbeat[],1,,re-check after the publisher has effectively gone silent +true,0,0,q,first exec error from heartbeat.gethb[] where procname=`pub1,1,,stale publisher escalated to error over real ipc +run,0,0,q,neg[h](exit;0);neg[h](::),1,,shut down the background publisher +run,0,0,q,hclose h,1,,close the handle