RFPD-77178: log ingestion api (v3.0) - #28
Closed
aommm wants to merge 74 commits into
Closed
Conversation
…ategory taxonomy Refresh all CSV outputs after merging master. Replace ~430 blanket is_published=true overrides with marketplace lookup-redirects (double-prefix fix, offerId filter-query fallback, pre-check lookup-key redirects). Adds category_primary taxonomy, ARM-expression table-name filter, and override-driven discovered-connector corrections. Net data: +7 solutions, +11 connectors.
…x SlashNext index drop) Mapper now always emits an empty-table placeholder row when a connector resolves to zero tables (table_detection_failed, parser_tables_only, partial_parser_tables, no_table_definitions), instead of silently dropping the connector and its solution from the index. Restores SlashNext, which a v9.9 reported_table_exclusions override had reduced to zero tables. Both doc generators (generate_connector_docs.py, generate_interactive_docs.py) now union-seed by_solution from solutions.csv so every solution is indexed even with no mapping rows (also covers deprecated solutions). Updated README v9.9 changelog and script-docs; regenerated CSVs.
…onnectors - Extract table names from connectorUiConfig.dataTypes for standalone *_ConnectorDefinition.json files - Establishes dataTypes as Priority 0 (authoritative) source above companion files and query analysis - Fixes 24 CCF v3 connectors that previously reported zero tables despite having 90+ defined tables - Primary impact: AlibabaCloudNetworkingConnector now correctly exports 3 tables (VPCFlowLogs, WAFLogs, APIGatewayLogs) - Updated README.md v9.9 changelog with priority hierarchy details - Regenerated sentinelninja connector docs with extracted table corrections
…'outputStream' fields - Priority 2 DCR extraction now checks both standard 'streams' array and legacy 'outputStream' field - Fixes missed tables in connectors using Azure DCR standard format (e.g., AlibabaCloudNetworking_DCR.json) - Enables Priority 2 to properly extract 'Microsoft-Sentinel*' prefixed stream names from dataFlows - Improves robustness for connectors that may only have DCR files without ConnectorDefinition.json - AlibabaCloudNetworking now correctly extracts all 3 tables from dataTypes (Priority 0) regardless
- DCR table extraction now strips leading Sentinel token after Microsoft-/Custom- prefix normalization - Supports stream forms like Microsoft-Sentinel<TableName> and maps to canonical table names - Aligns DCR-derived tables with connector dataTypes for CCF connectors (Alibaba Cloud Networking now matches) - Update mapper script-docs to document outputStream/streams handling and normalization behavior
- Add TABLE_SCHEMA_REFERENCES mapping for SecurityAlert, DnsEvents, DnsInventory, AMA_DNS - Implement get_schema_references() function with table-specific and fallback logic - Add 'Schema References' section to generated table documentation pages - Include schema references in Table of Contents generation - Update script documentation with configuration details - Update version history to v9.10
Adds the HoneyLabs solution: honeypot-derived threat intelligence ingested through the built-in Threat Intelligence - TAXII data connector against the HoneyLabs TAXII 2.1 server. Includes a data connector page with copyable configuration values, three TI-map analytic rules (IP to CommonSecurityLog, IP to SigninLogs, URL to CommonSecurityLog), and an incident enrichment playbook that comments the full HoneyLabs verdict on IP entities. Package generated with the V3 packaging tool.
DetectionTemplateSchemaValidation rejected the custom connectorId HoneyLabsThreatIntelligenceTAXII in the analytic rules' requiredDataConnectors (not in ValidConnectorIds.json). Indicators arrive through the built-in ThreatIntelligenceTaxii connector, which the rules already require, so the custom id is removed from all three. SolutionValidations flagged bare 'Sentinel'; the connector description now says 'Microsoft Sentinel'. Package regenerated.
…XII setup, and remove a false-positive source in the CommonSecurityLog rule - Data connector: one server entry using the new combined 'all indicators' collection instead of two, dropping setup from 5 steps to 4. - Workbook: feed health, confidence-band tuning, top source networks and countries, and matches in the customer's own logs ordered by evidence. - CommonSecurityLog rule no longer matches IPs extracted from the free-text Message field, which produced false positives on incidental mentions. - Indicators now carry graduated confidence and asn/country/type tags, so a SOC can pick an alerting threshold instead of muting the feed.
…aden SOC coverage Microsoft's current guidance is that threat indicators live in the ThreatIntelIndicators table, which is the basis for Analytics, Hunting and Workbooks; the legacy ThreatIntelligenceIndicator table is on its way out. All queries now target the new table, following the idiom used by shipped solutions. - Four analytic rules (CommonSecurityLog, SigninLogs, URL, and a new ASIM _Im_NetworkSession rule so workspaces without CEF are covered). - Three hunting queries: first contact with a high-evidence indicator, outbound contact with a loader/C2 URL, and one indicator spraying several hosts. - Workbook: feed health, confidence-band tuning, top source networks and countries, and matches in the customer's own logs ordered by evidence. - Indicators carry graduated Confidence (90 = 100+ observed attacks) and asn/country/type labels, so a SOC can set a threshold rather than mute the feed. - Restores the analytic rules to the generated package: the previous regeneration silently dropped them because the packaging container lacked the powershell-yaml module, so every rule failed to deserialize.
- Connector now points at the attacker-IP collection: source addresses observed attacking the sensors, which is what the 30M-probe corpus is actually about. Malware infrastructure moves to an optional second entry; those URLs are a VirusTotal-enriched extraction from payloads, a smaller and different signal that was previously diluting the primary feed. - Paid plans receive CVE probers through the same collection, each indicator labelled with the CVE ids that source went after. Workbook gains a most-probed-CVEs panel and a fourth hunting query cross-references those sources against your own logs, so patch prioritisation gets an in-the-wild signal rather than a vendor severity score. - Playbook now only comments when HoneyLabs actually knows the address, tags the incident, and raises severity for a malicious verdict (the real verdict value; an earlier draft compared against a value the API never returns).
run-arm-ttk failed on CreateUIDefinition Must Not Have Blanks. The generated workbook1 section carried label: null and options.text: null, because the packaging tool had no title or description to render: the workbook was listed in Solution_HoneyLabs.json but never registered in Workbooks/WorkbooksMetadata.json, which is where the tool reads them from. GreyNoiseThreatIntelligence is registered there, which is why the same section is populated in its package. Adds the WorkbooksMetadata entry (logoFileName points at Logos/HoneyLabs.svg, which already exists), gives the solution a WorkbookDescription the way other solutions do, and fills the two null values in the generated createUiDefinition.json to match. Verified: no blank or null values remain anywhere in createUiDefinition.json, and every mainTemplate parameter without a defaultValue is present in the UI outputs, which is the second rule that failed.
… to ThreatIntelIndicators, and regenerate the V3 package bundle
…et/scope_name last-entry-wins bug in authorize events (v3.0.6)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: f8374209-6203-4a25-ad41-5cd2c8c4f2e6
# Conflicts: # Workbooks/WorkbooksMetadata.json
Deploying the package into a live Sentinel workspace showed the playbook content template registering as v1.0, not v1.0.0. The playbook's azuredeploy.json carries no version field, so the packaging tool falls back to 1.0. Match the release notes to what actually installs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ErikMangstenRecFut
force-pushed
the
feat/RFPD-77178-log-ingestion-api
branch
from
August 14, 2026 08:41
405ad33 to
2d32fc6
Compare
… drop Git Trees API + index file collect_table_info.py now flags categoryless Azure Monitor tables (e.g. ApiManagementGatewayLlmLog) via source_feature_support instead of the rate-limited GitHub Git Trees API, correcting the flag/link in-memory before writing tables_reference.csv. Removes fetch_azure_monitor_table_index(), the two Git Trees API constants, and the azure_monitor_tables_index.txt companion file. map_solutions_connectors_tables.py consumes the corrected flags directly. Proven equivalent (identical 919 Azure Monitor tables). CSVs/docs regenerated via full --force-refresh=all run.
…into tools/solutions-analyzer/v9.11
…g-ingestion-api Recorded Future Identity: Migrate from Data Collector API
New solution: HoneyLabs threat intelligence (TAXII)
Parameterize Recorded Future alert ID
* Update Cisco Umbrella solution to version 3.3.0 - Bump version number in Solution_CiscoUmbrella.json and mainTemplate.json to 3.3.0. - Add new package 3.3.0.zip for the updated solution. - Update data connector versions and descriptions to reflect changes in schema version 16. - Enable Gzip decompression for all log streams. - Introduce new columns: DestinationSgtOriginId and FwBlockReason in CiscoUmbrellaCloudFirewall, and IsolationProfileId in CiscoUmbrellaWebTraffic. - Add connector instruction notes regarding optional headers and support for Cisco-managed/customer-managed Amazon S3 buckets. - Update ReleaseNotes.md to document changes for version 3.3.0. * Update Cisco Umbrella parser to version 1.0.4 - Incremented version to 1.0.4 and updated last modified date. - Added 'IsolationProfileId' to the output of the Cisco Umbrella threat query. - Enhanced Cisco Umbrella Cloud Firewall view by including 'DestinationSgtOriginId' and 'FwBlockReason' fields.
…-audit-connector Add combined Datadog CCF connector
…etadata, not first connector
Co-authored-by: Srikar Shastry <Srikar.Sistla@microsoft.com>
…history publisher handling
Solutions Analyzer v9.11: parser, table discovery, and mapping accuracy
…ogleWS Gighub Issue Fix Azure#14387 - Google Workspace Reports
Co-authored-by: Dhanunjaya Bedudooru <v-dhbedu@microsoft.com>
…ata-file Cisco ASA: Update package version in data input file
They are anyway hardcoded within DCE/DCR, no need for params
ErikMangstenRecFut
force-pushed
the
feat/RFPD-77178-log-ingestion-api-2
branch
from
August 19, 2026 11:36
28417f2 to
1f9f172
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
15 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TODO: Once Azure#14714 is merged, close this PR and open towards Microsoft instead