Rebuild ping++ engine: correctness, hygiene, and R1–R18 classification pipeline - #1
Open
0sm0s1z wants to merge 64 commits into
Open
Rebuild ping++ engine: correctness, hygiene, and R1–R18 classification pipeline#10sm0s1z wants to merge 64 commits into
0sm0s1z wants to merge 64 commits into
Conversation
Remove ICMP-alive override when TCP ports are closed, enumerate all configured TCP ports, stop using local IP_TTL as remote TTL, strip hard-coded debug paths, fix numeric formatting, and use a single OS aggregation path in appscanner. Add regression tests for each fix. Co-authored-by: mtoussain <mtoussain@gmail.com>
Add pkg/model with Asset, Address, Endpoint (multi-state), ObservationRecord envelope with typed payloads, Claim kinds, confidence tiers, Artifact and Reachability types. Protocol-specific fields stay out of Asset. Include JSON round-trip and tier mapping tests. Co-authored-by: mtoussain <mtoussain@gmail.com>
Add pkg/engine Collector interface, Registry (Register/Create/Metadata), ScanState/Budget stubs, and BuildDefaultRegistry wiring via package Register funcs. Add discovery.icmp, discovery.tcp, and enumerate.tcp collectors that emit ObservationRecords without product claims. Co-authored-by: mtoussain <mtoussain@gmail.com>
Add staged Engine.ScanTarget pipeline, Planner with port-prior ordering (not identity), Profile quick/default/deep, Budget controls, RateLimiter, Scheduler, ProbeOutcome types, and IPv4/IPv6 hostname Target resolution that retains hostname for SNI/Host. Co-authored-by: mtoussain <mtoussain@gmail.com>
Prove enumerate.tcp emits one observation and EndpointState per requested port (open vs closed/RST). Add curated port prior helpers. Ports remain priors only; discovery skip and profile port sets come from the engine. Co-authored-by: mtoussain <mtoussain@gmail.com>
Add pkg/artifact MemoryStore with strict size limits and content addressing so protocol transcripts can be retained for offline fingerprint replay. Co-authored-by: mtoussain <mtoussain@gmail.com>
Add YAML fingerprint rules, observation field matching, and Fuse() that keeps only the strongest score per correlation group while combining independent groups via 1-∏(1-s). Include nginx/OpenSSH starter rules. Co-authored-by: mtoussain <mtoussain@gmail.com>
Add collect.tls as a first-class protocol layer (not embedded in HTTP). Capture TLS version, cipher, ALPN, certificate chain identity fields, SHA-256 and SPKI hashes. Successful TLS does not imply HTTPS. Co-authored-by: mtoussain <mtoussain@gmail.com>
Collect status, headers, cookies, redirects context, title, meta generator, auth/CSP/Server fields, bounded body, raw and normalized SHA-256, SimHash, and favicon URL hints. No product logic in collector. Co-authored-by: mtoussain <mtoussain@gmail.com>
Add adapter interfaces so third-party fingerprint corpora never leak into canonical model types. Include NativeWebTech (Wappalyzer-style) and NativeRecog YAML loaders with offline unit tests and starter Recog rules. Co-authored-by: mtoussain <mtoussain@gmail.com>
Wire collect.ssh and collect.smb via pkg/scan.NewRegistry without import cycles. Extend scan profiles and add Windows SMB / Cisco SSH fingerprint packs that claim from observations rather than collectors. Co-authored-by: mtoussain <mtoussain@gmail.com>
Add banner-oriented text protocol collectors with EHLO feature capture for SMTP. Observations only — no product claims in collectors. Co-authored-by: mtoussain <mtoussain@gmail.com>
…ched Small unauthenticated handshake/info collectors that emit typed observations suitable for later fingerprint packs. Co-authored-by: mtoussain <mtoussain@gmail.com>
Add negotiation/identification collectors. SNMP uses an explicitly configured community only (default public) with no bruteforce. Co-authored-by: mtoussain <mtoussain@gmail.com>
Register the remaining breadth collectors in pkg/scan.NewRegistry so planner priors can schedule them without a central protocol switch. Co-authored-by: mtoussain <mtoussain@gmail.com>
Add YAML fingerprint corpus covering major web servers and applications (nginx/Apache/IIS/Caddy/Grafana/Jenkins/GitLab/WordPress/etc.) with pack loader helpers and offline fixture/regression tests. Co-authored-by: mtoussain <mtoussain@gmail.com>
Add device fingerprint rules for Cisco/Fortinet/Palo Alto/VMware/ Synology/QNAP/iDRAC/iLO/pfSense/OPNsense and related appliances via HTTP/SNMP observations plus negative lookalike coverage. Co-authored-by: mtoussain <mtoussain@gmail.com>
Downgrade generic OpenSSH/nginx/TTL-derived Linux claims to hint-level while preserving strong SMB/NTLM and SNMP-backed OS evidence. Co-authored-by: mtoussain <mtoussain@gmail.com>
Record connect-level TCP features as observations with an explicit platform_limited flag. Full MSS/option-order capture is deferred to raw-socket/pcap support without inventing OS labels in the collector. Co-authored-by: mtoussain <mtoussain@gmail.com>
Add pkg/output legacy adapter and ToSiriusHost that preserve IP/hostname/OS/confidence while exposing endpoints and claims. Appscanner strategy prefers the new engine path with fingerprint pack matching. Co-authored-by: mtoussain <mtoussain@gmail.com>
Add precision-at-tier counters (exact/strong/probable), unknown rate, and probes/bytes per resolved endpoint instrumentation. Full suite remains green as the release gate. Co-authored-by: mtoussain <mtoussain@gmail.com>
Co-authored-by: mtoussain <mtoussain@gmail.com>
Classification tasks dropped the resolved Target, so HTTP Host and TLS SNI used the raw IP. Attach the target, add DNS/SOCKS port priors, and add a full-pipeline example scanner for hostname scans. Co-authored-by: mtoussain <mtoussain@gmail.com>
Treat a hostname argument as a seed so www.example.com also tries example.com (and the reverse) before scanning. Co-authored-by: mtoussain <mtoussain@gmail.com>
Accept URLs and hostnames, expand apex/www only for registrable domains, and report protocol-confirmed services separately from TCP-connect-only ports so accept-all middleboxes are not inventory. Co-authored-by: mtoussain <mtoussain@gmail.com>
Accept-all middleboxes SYN-ACK every port. Enumeration now records connect success as responsive; open is reserved for a protocol handshake. Empty FTP/SMTP/Redis reads no longer count as confirmation. Co-authored-by: mtoussain <mtoussain@gmail.com>
Co-authored-by: mtoussain <mtoussain@gmail.com>
Default output is the complete asset JSON (every endpoint, observation payload, DNS answer, and claim). -text dumps the same records line-oriented. Nothing is truncated or summarized. Co-authored-by: mtoussain <mtoussain@gmail.com>
Testing needs a complete human dump (every port, payload, and DNS answer). -json still writes the same document for other tools. Co-authored-by: mtoussain <mtoussain@gmail.com>
Record the alignment contract without changing planner, fusion, or Engine ownership. Make the freeze CI gate explicit: go vet, race tests, and pinned golangci-lint v2.12.2. Fix errcheck findings that blocked that gate. Co-authored-by: mtoussain <mtoussain@gmail.com>
Scheduler now runs collector tasks; transport dials increment a scan meter so budgets track connections, not just collector invocations. TCP discovery/enumeration uses a per-host worker pool and DialTCP. Co-authored-by: mtoussain <mtoussain@gmail.com>
Drop the hardcoded port-to-collector table. Planner priors now come from CollectorMetadata.DefaultPorts, with an unknown-port sequence starting at collect.banner. Banner records first bytes only and never treats them as protocol identity. Unregister collect.tcpstack from the production registry. Co-authored-by: mtoussain <mtoussain@gmail.com>
TLS, HTTP, and SSH now report Success vs NoMatch from handshake behavior. Engine attaches endpoint-scoped ClaimProtocol on Success and preserves those claims across fingerprint fusion. HTTP scheme follows observed TLS (Extra/state), not ports 443/8443. Co-authored-by: mtoussain <mtoussain@gmail.com>
Add enumerate.udp with concurrent datagram probes and schedule it from default/deep profiles. Rebuild collect.dns on miekg/dns so identification comes from a DNS response at the endpoint, not LookupHost through the OS resolver. Co-authored-by: mtoussain <mtoussain@gmail.com>
Replace the hand-rolled BER scanner with gosnmp GETs for sysDescr, sysObjectID, sysName, and ENTITY-MIB entPhysicalDescr. Community is used on the wire only and is never written into observation JSON. Co-authored-by: mtoussain <mtoussain@gmail.com>
Follow redirects only on the same host; record cross-host Location and stop. Persist truncated bodies in the artifact store, hash fetched favicons (SHA-256 + Shodan-style MMH3), and add collect.http.enrich for up to three extra same-host URLs. Co-authored-by: mtoussain <mtoussain@gmail.com>
Parse Rapid7 Recog XML and Wappalyzer technologies.json behind the existing adapter interfaces. Fingerprint matching hydrates HTTP bodies from the artifact store. NativeWebTech remains a test fixture only. Co-authored-by: mtoussain <mtoussain@gmail.com>
Text protocols require their greeting prefixes (FTP/SMTP 220, POP3 +OK, IMAP OK). Postgres only accepts SSLRequest S/N, MySQL uses framed protocol-10 handshakes, Redis requires PONG, LDAP RootDSE must be BER, RDP requires TPKT, and SSH parses KEXINIT via packet length rather than hunting for byte 20. Lookalike payloads return NoMatch. Co-authored-by: mtoussain <mtoussain@gmail.com>
Network budgets now count HTTP/DNS/SNMP/ICMP operations and connection Read/Write bytes, not only DialTCP/UDP. After port-associated collectors return NoMatch, classification continues into the general sequence so services on unusual ports remain discoverable. UDP silence is unknown and still scheduled; ICMP port-unreachable may mark closed. Co-authored-by: mtoussain <mtoussain@gmail.com>
Replace the freeze-era snapshot with R1–R18 status, commit, acceptance tests, and known debt so completed planner/scheduler/metering/UDP work is not still described as missing. Co-authored-by: mtoussain <mtoussain@gmail.com>
…dress scans Stop treating R11–R14 as done at framework-only. Collectors now have positive/negative tests, LDAP/RDP/SSH/SMB meet the stated parse bars, built-in fingerprints ship via go:embed, and ScanTarget walks every A/AAAA. Co-authored-by: mtoussain <mtoussain@gmail.com>
Co-authored-by: mtoussain <mtoussain@gmail.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0d631bee43
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+173
to
+175
| for i := range claims { | ||
| claims[i].EvidenceIDs = []string{obs.ID} | ||
| claims[i].CorrelationGroup = "recog:" + obs.ObservationType |
There was a problem hiding this comment.
Set each Recog claim's Subject from the observation's endpoint (or AssetID when there is no endpoint), as the other adapters do. Currently every HTTP, SMB, and SSH Recog claim has an empty subject, so fingerprint.Fuse groups identical products from different endpoints together and can combine evidence or version conflicts across unrelated services; even a single-endpoint result loses where the claim applies.
Useful? React with 👍 / 👎.
Comment on lines
+150
to
+155
| for _, id := range p.classificationSequence(ep) { | ||
| if state != nil && exclusiveProtocol(strings.TrimPrefix(id, "collect.")) && state.HasProtocol(key, "http") && id != "collect.http" && id != "collect.tls" && id != "collect.banner" { | ||
| continue | ||
| } | ||
| if task, ok := p.taskIfAvailable(asset, ep, state, id, StageClassify); ok { | ||
| return task, true |
There was a problem hiding this comment.
Stop scheduling collectors once an endpoint reaches Budget.MaxRequestsPerEndpoint. For an unidentified responsive TCP endpoint, this loop can traverse the entire unknownTCPSequence—roughly two dozen active protocol handshakes—even though the default budget promises at most 8 requests per endpoint (16 for deep scans); the host-wide probe cap does not enforce that per-endpoint safety limit.
Useful? React with 👍 / 👎.
Comment on lines
+143
to
+145
| // Remaining reports whether collector or network budget remains. | ||
| func (b *Budget) Remaining() bool { | ||
| return b.RemainingProbes() && b.RemainingNetworkOps() |
There was a problem hiding this comment.
Include MaxBytesPerHost in this remaining-budget check. syncMeterLocked continuously populates BytesUsed, but Remaining ignores it, so multiple large HTTP responses can exceed the configured 2 MiB host limit while the engine continues issuing collectors until a probe or network-operation limit happens to stop it.
Useful? React with 👍 / 👎.
Comment on lines
+27
to
+30
| return claimIdentity{ | ||
| subject: c.Subject, | ||
| kind: c.Kind, | ||
| product: product, |
There was a problem hiding this comment.
Populate the existing vendor field in claimIdentity. The bundled device pack contains many vendor-only device claims, so when conflicting vendor signals match the same endpoint they currently share the same empty product/family/value identity and are fused into one claim, silently discarding one vendor and incorrectly boosting the surviving claim with the other's evidence.
Useful? React with 👍 / 👎.
Comment on lines
+113
to
+114
| if !sameHTTPHost(prev.URL, req.URL) { | ||
| return http.ErrUseLastResponse |
There was a problem hiding this comment.
Treat the retained target hostname as equivalent to the endpoint IP when validating redirects. The initial request URL always contains the resolved IP while Host contains the original hostname, so a common absolute redirect such as Location: https://example.com/login fails this comparison against https://192.0.2.1/; the client returns the redirect response instead of collecting the destination page, causing application fingerprints on that page to be missed.
Useful? React with 👍 / 👎.
Default stdout lists every positive match (open/responsive ports, protocol claims, fingerprint claims, and successful observation fields). The old full dump is -verbose; JSON is unchanged. Recog claims now attach to the observation endpoint so they show on the right port. Co-authored-by: mtoussain <mtoussain@gmail.com>
Enumeration still classifies TCP accept/RST as responsive or closed. The default examples/scan report now prints only protocol-confirmed findings; use -verbose to see the rest. Co-authored-by: mtoussain <mtoussain@gmail.com>
Recog now keeps service, OS, and hardware params separate, fills versions from pos=N captures (including CPE substitutions), and sets endpoint Subject so :22 and :2222 never fuse. FTP/SMTP identify from FEAT 211 and EHLO 250 rather than greeting keywords. Implicit TLS covers 465/993/995; TLS success replans the port-associated application collector. MQTT/AMQP/VNC/SOCKS implement ResultCollector with lookalike tests. Multi-address scans share one network budget and retain Matched, RuledOut, and reachability reasons. SMB dials through the scan meter. Byte and per-endpoint request budgets are enforced. Runtime protocol matches count ClaimProtocol, not collector success; collector panics become internal_error. Sirius DisableICMP no longer skips TCP discovery, and ProbeTypes filter the Engine path. R18 stays PARTIAL: unmatched dump is still in-memory, and performance coverage is a microbenchmark rather than a live regression suite. Co-authored-by: mtoussain <mtoussain@gmail.com>
Lock CLI → runner → Session → engine layering, scan.Config, target/output rules, and the C1–C16 sequence before any production CLI work starts.
Lazy CIDR iteration, max-targets safety, excludes, and file diagnostics so Runner V2 can ingest large inventories without materializing them.
C6 needs on-disk SHA-256 blobs with atomic rename, digest dedup, and path-safe lookup so artifact IDs remain valid across process restarts.
Canonical scan.Config compiles port tri-state onto the engine, full stays a non-default generated TCP range, Session loads fingerprints once, and transport paces dials for the run.
The C3–C7 parent evaluation and program tracker now match HEAD so the C8 push loop can stop.
SMTP/IMAP/POP3 now upgrade after a protocol match without counting a second dial. Unmatched banners append to a JSONL file, and the builtin positive corpus is timed so claim counts cannot collapse unnoticed. Co-authored-by: mtoussain <mtoussain@gmail.com>
Ship a thin scan/version CLI over Runner V2 and scan.Session. Live targets showed HTTP using the 3s probe timeout, findings URLs as raw IPs, serialized Session.Scan, and probe counts past MaxProbesPerHost. Co-authored-by: mtoussain <mtoussain@gmail.com>
Preserve Sirius ProbeTypes through ConfigFromScanOptions as a private compatibility field so icmp,tcp no longer silently expands to the full Quick collector set. ScanResolved now runs discovery and enumeration across every A/AAAA before round-robin classification/enrichment, keeps the hostname-level State.AssetID, and records endpoint execution (attempted / not_attempted_budget / cancelled / timed_out). C13: typed RunError, synchronized sink/event/run-error paths, run-timeout exit 1, input/config exit 2, scan --help exit 0. Default text shows responsive unclassified endpoints. Banner sink uses 0700/0600 and a buffered writer. Co-authored-by: mtoussain <mtoussain@gmail.com>
Co-authored-by: mtoussain <mtoussain@gmail.com>
ResolveTarget and ScanResolved de-duplicate A/AAAA answers, including IPv4-mapped forms, and order IPv4 before IPv6 so later stages are not resolver-order dependent. A dead first address no longer starves HTTP evidence from a live sibling. HTTP DialContext is pinned to Endpoint.Address while the request URL, Host header, and TLS SNI use the logical hostname. Observation payloads record logical_host, transport_ip, and tls_server_name so evidence is not attributed to a silently re-resolved address. Co-authored-by: mtoussain <mtoussain@gmail.com>
Co-authored-by: mtoussain <mtoussain@gmail.com>
Close the remaining HTTP pinning edge (apex vs www), guarantee a terminal disposition for every scheduler task, bound real host-wide transport I/O, record IPv6/multi-address outcomes without starving siblings, and keep Endpoint.Execution monotonic. Add fingerprint normalization without collapsing contradictions, keep the Sirius adapter Engine-path only (legacy runner is rollback), land Runner C14–C16 introspection/README/stress, meter SNMP via DialUDP, estimate ICMP echo bytes, and document architecture plus third-party NOTICE. Co-authored-by: mtoussain <mtoussain@gmail.com>
Co-authored-by: mtoussain <mtoussain@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Rebuilds ping++ from a probe-and-aggregate scanner into an evidence-driven classification engine, starting from discovery-semantics hygiene (PR0) through the R1–R18 rebuild contract.
IP_TTLis not treated as remote TTL; ports are priors, never identity; unknown is a valid state.resolve → discover → enumerate → plan → collect → fingerprint → re-plan → enrich → Asset, with Engine owning fingerprinting, artifacts, and fused claims.ScanTarget, and a living rebuild contract indocs/prototype-known-gaps.md.47 commits, 175 files (
+22840 / −371) oncursor/ping-correctness-and-hygiene-d7fc. Head:0d631be(docs pointer to contract head9403a4d).Test plan
go test ./...(including-race) on this branchgo vet ./...and golangci-lint v2.12.2 (CI workflow already gatescursor/**and PRs intomaster)