happy-technologies-llc/vulnerability-deduplication

Vulnerability Deduplication

Overview

This skill enables vulnerability management teams to identify and consolidate duplicate vulnerability records that arise when multiple scanners report the same CVE against the same asset. Deduplication reduces noise, improves remediation efficiency, and provides an accurate count of the organization's true vulnerability exposure.

Key capabilities:

• Identify duplicate vulnerable items by matching CVE ID and configuration item
• Detect near-duplicates from different scanner sources reporting the same issue
• Consolidate redundant remediation tasks into a single actionable item
• Prioritize deduplicated vulnerabilities using CVSS scores and business criticality
• Generate reports showing duplicate reduction and true exposure counts
• Maintain audit trail of deduplication decisions

When to use: After vulnerability scan imports, during periodic vulnerability hygiene reviews, when onboarding a new scanner, or when vulnerability counts seem inflated due to multi-scanner overlap.

Prerequisites

• Roles: sn_vul.manager, sn_vul.admin
• Access: Read/write access to sn_vul_vulnerable_item, sn_vul_entry, sn_vul_third_party_entry, sn_vul_remediation_task tables
• Plugins: Vulnerability Response (sn_vul)
• Data: Vulnerability scan data from one or more scanners imported into ServiceNow

Procedure

Step 1: Identify Potential Duplicates by CVE and CI

Query for vulnerable items that share the same CVE and configuration item, which indicates a duplicate from multiple scanner sources.

Using MCP:

Tool: SN-Execute-Background-Script
Parameters:
  script: |
    var ga = new GlideAggregate('sn_vul_vulnerable_item');
    ga.addQuery('state', 'IN', 'Open,Under Investigation');
    ga.addAggregate('COUNT');
    ga.groupBy('vulnerability');
    ga.groupBy('cmdb_ci');
    ga.addHaving('COUNT', '>', 1);
    ga.query();
    var dupCount = 0;
    while (ga.next()) {
      dupCount++;
      gs.info('Duplicate: Vuln=' + ga.vulnerability.getDisplayValue() +
              ' CI=' + ga.cmdb_ci.getDisplayValue() +
              ' Count=' + ga.getAggregate('COUNT'));
    }
    gs.info('Total duplicate groups found: ' + dupCount);

Using REST:

GET /api/now/stats/sn_vul_vulnerable_item?sysparm_query=stateINOpen,Under Investigation&sysparm_count=true&sysparm_group_by=vulnerability,cmdb_ci&sysparm_having=COUNT>1

Step 2: Retrieve Duplicate Records for a Specific CVE

For each duplicate group, pull the full records to compare scanner sources and detection details.

Using MCP:

Tool: SN-Query-Table
Parameters:
  table_name: sn_vul_vulnerable_item
  query: vulnerability=[VULNERABILITY_SYS_ID]^cmdb_ci=[CI_SYS_ID]^stateINOpen,Under Investigation
  fields: sys_id,vulnerability,cmdb_ci,state,risk_score,cvss_score,first_found,last_found,source,scanner,assignment_group,assigned_to,remediation_task
  limit: 20

Using REST:

GET /api/now/table/sn_vul_vulnerable_item?sysparm_query=vulnerability=[VULNERABILITY_SYS_ID]^cmdb_ci=[CI_SYS_ID]^stateINOpen,Under Investigation&sysparm_fields=sys_id,vulnerability,cmdb_ci,state,risk_score,cvss_score,first_found,last_found,source,scanner,assignment_group,assigned_to,remediation_task&sysparm_limit=20&sysparm_display_value=true

Step 3: Retrieve CVE Details from Vulnerability Entry

Look up the canonical vulnerability entry to get authoritative CVE information.

Using MCP:

Tool: SN-Query-Table
Parameters:
  table_name: sn_vul_entry
  query: sys_id=[VULNERABILITY_SYS_ID]
  fields: sys_id,cve_id,description,cvss_score,severity,exploit_available,vendor_advisory,published_date,solution
  limit: 1

Using REST:

GET /api/now/table/sn_vul_entry?sysparm_query=sys_id=[VULNERABILITY_SYS_ID]&sysparm_fields=sys_id,cve_id,description,cvss_score,severity,exploit_available,vendor_advisory,published_date,solution&sysparm_limit=1

Step 4: Check Third-Party Scanner Entries

Review scanner-specific entries to understand which scanners reported the same vulnerability.

Using MCP:

Tool: SN-Query-Table
Parameters:
  table_name: sn_vul_third_party_entry
  query: vulnerability_entry=[VULNERABILITY_SYS_ID]
  fields: sys_id,source,scanner_id,vulnerability_entry,severity,first_found,last_found,status,plugin_id
  limit: 20

Using REST:

GET /api/now/table/sn_vul_third_party_entry?sysparm_query=vulnerability_entry=[VULNERABILITY_SYS_ID]&sysparm_fields=sys_id,source,scanner_id,vulnerability_entry,severity,first_found,last_found,status,plugin_id&sysparm_limit=20&sysparm_display_value=true

Step 5: Get Business Context for the Affected CI

Retrieve asset details to factor business criticality into prioritization.

Using MCP:

Tool: SN-Query-Table
Parameters:
  table_name: cmdb_ci
  query: sys_id=[CI_SYS_ID]
  fields: sys_id,name,sys_class_name,ip_address,os,environment,business_criticality,support_group,department,operational_status
  limit: 1

Using REST:

GET /api/now/table/cmdb_ci?sysparm_query=sys_id=[CI_SYS_ID]&sysparm_fields=sys_id,name,sys_class_name,ip_address,os,environment,business_criticality,support_group,department,operational_status&sysparm_limit=1&sysparm_display_value=true

Step 6: Consolidate Duplicate Records

Keep the record with the earliest first_found date and most complete data. Close duplicates with a reference to the primary record.

Using MCP:

Tool: SN-Update-Record
Parameters:
  table_name: sn_vul_vulnerable_item
  sys_id: [DUPLICATE_SYS_ID]
  fields:
    state: Closed
    close_notes: "Duplicate of [PRIMARY_SYS_ID]. Same CVE ([CVE_ID]) on same CI ([CI_NAME]) reported by different scanner ([SCANNER_NAME]). Consolidated into primary record."

Tool: SN-Add-Work-Notes
Parameters:
  table_name: sn_vul_vulnerable_item
  sys_id: [PRIMARY_SYS_ID]
  work_notes: "Deduplication: Consolidated [N] duplicate records from scanners [SCANNER_LIST]. This record retained as primary. Earliest first_found: [DATE]. Highest risk_score: [SCORE]."

Using REST:

PATCH /api/now/table/sn_vul_vulnerable_item/[DUPLICATE_SYS_ID]
Content-Type: application/json

{
  "state": "Closed",
  "close_notes": "Duplicate of [PRIMARY_SYS_ID]. Same CVE on same CI from different scanner. Consolidated."
}

Step 7: Consolidate Remediation Tasks

Identify redundant remediation tasks created for duplicate vulnerable items and merge them.

Using MCP:

Tool: SN-Query-Table
Parameters:
  table_name: sn_vul_remediation_task
  query: vulnerable_item=[PRIMARY_SYS_ID]^ORvulnerable_item=[DUPLICATE_SYS_ID]^stateINOpen,In Progress
  fields: sys_id,number,short_description,state,assigned_to,assignment_group,vulnerable_item,opened_at,due_date
  limit: 20

Close duplicate remediation tasks:

Tool: SN-Update-Record
Parameters:
  table_name: sn_vul_remediation_task
  sys_id: [DUPLICATE_TASK_SYS_ID]
  fields:
    state: Closed
    close_notes: "Consolidated with [PRIMARY_TASK_NUMBER]. Duplicate vulnerability record closed during deduplication."

Using REST:

GET /api/now/table/sn_vul_remediation_task?sysparm_query=vulnerable_itemIN[PRIMARY_SYS_ID],[DUPLICATE_SYS_ID]^stateINOpen,In Progress&sysparm_fields=sys_id,number,short_description,state,assigned_to,assignment_group,vulnerable_item,opened_at,due_date&sysparm_limit=20&sysparm_display_value=true

PATCH /api/now/table/sn_vul_remediation_task/[DUPLICATE_TASK_SYS_ID]
Content-Type: application/json

{
  "state": "Closed",
  "close_notes": "Consolidated with primary remediation task during deduplication."
}

Step 8: Generate Deduplication Report and Prioritized Exposure List

After deduplication, generate a clean view of unique vulnerabilities prioritized by risk.

Using MCP:

Tool: SN-Execute-Background-Script
Parameters:
  script: |
    var gr = new GlideRecord('sn_vul_vulnerable_item');
    gr.addQuery('state', 'IN', 'Open,Under Investigation');
    gr.orderByDesc('risk_score');
    gr.query();
    var results = [];
    while (gr.next()) {
      results.push({
        vuln: gr.vulnerability.getDisplayValue(),
        ci: gr.cmdb_ci.getDisplayValue(),
        cvss: gr.cvss_score + '',
        risk: gr.risk_score + '',
        first_found: gr.first_found + '',
        source: gr.source + ''
      });
    }
    gs.info('Unique open vulnerabilities after dedup: ' + results.length);
    // Output top 20 by risk score
    for (var i = 0; i < Math.min(20, results.length); i++) {
      gs.info((i+1) + '. ' + results[i].vuln + ' on ' + results[i].ci +
              ' (CVSS: ' + results[i].cvss + ', Risk: ' + results[i].risk + ')');
    }

Deduplication Summary Template:

=== VULNERABILITY DEDUPLICATION REPORT ===
Date: 2026-03-19 | Analyst: J. Smith

BEFORE DEDUPLICATION:
Total Open Vulnerable Items: 2,847
Unique CVE+CI Combinations: 1,923

DUPLICATES IDENTIFIED: 924 (32.5% duplication rate)

BY SCANNER SOURCE:
| Scanner | Records | Unique | Duplicates | Dup Rate |
|---------|---------|--------|------------|----------|
| Qualys | 1,450 | 1,200 | 250 | 17.2% |
| Tenable | 1,100 | 950 | 150 | 13.6% |
| Rapid7 | 297 | 273 | 24 | 8.1% |
| Cross-scanner overlap | - | - | 500 | - |

ACTIONS TAKEN:
- Closed 924 duplicate vulnerable item records
- Consolidated 312 redundant remediation tasks
- Updated 1,923 primary records with consolidated scanner data

TOP 10 CRITICAL UNIQUE VULNERABILITIES:
| Rank | CVE | CVSS | CI | Environment | Exploit |
|------|-----|------|----|-------------|---------|
| 1 | CVE-2025-44123 | 9.8 | WEBSVR-01 | Production | Yes |
| 2 | CVE-2025-33456 | 9.1 | DB-PROD-02 | Production | Yes |
| 3 | CVE-2025-22789 | 8.8 | APPSVR-05 | Production | No |
| 4 | CVE-2025-11234 | 8.5 | FILESVR-03 | Production | Yes |
| 5 | CVE-2025-99876 | 8.1 | MAIL-01 | Production | No |
...

RECOMMENDATIONS:
1. Schedule recurring deduplication job (weekly recommended)
2. Configure scanner import rules to prevent duplicate creation
3. Prioritize patching for top 10 critical vulnerabilities with exploits
4. Review Qualys/Tenable overlap rules to reduce cross-scanner duplicates

Tool Usage

| Tool | Purpose | |------|---------| | SN-Query-Table | Retrieve vulnerable items, CVE entries, scanner data, and CIs | | SN-Execute-Background-Script | Run aggregate queries and bulk deduplication scripts | | SN-Update-Record | Close duplicate records and update primary records | | SN-Add-Work-Notes | Document deduplication decisions on vulnerability records |

Best Practices

• Always preserve the earliest first_found date when consolidating duplicates to maintain accurate exposure duration
• Keep the highest risk_score and cvss_score on the primary record to avoid underestimating risk
• Document consolidation decisions with work notes referencing the closed duplicates for audit trail
• Run deduplication after each scan import to prevent duplicate accumulation
• Consider scanner-specific plugin IDs as some scanners report the same CVE with different plugin identifiers
• Validate before bulk closure by reviewing a sample of identified duplicates to ensure accuracy
• Automate recurring deduplication using scheduled scripts or business rules on scan import
• Maintain scanner source metadata on the primary record so remediation teams know which scanners confirmed the finding

Troubleshooting

Aggregate Query Returns No Duplicates

Cause: The vulnerability field may reference different sn_vul_entry records for the same CVE if imported by different scanners. Solution: Join on cve_id from sn_vul_entry instead of the vulnerability sys_id. Query sn_vul_entry for duplicate CVE IDs first.

Scanner Source Field Is Empty

Cause: The scanner integration may not populate the source or scanner field on vulnerable items. Solution: Check the sn_vul_third_party_entry table for scanner source data. Review the scanner integration configuration to ensure source mapping is configured.

Remediation Tasks Orphaned After Deduplication

Cause: Closing a duplicate vulnerable item does not automatically close its remediation task. Solution: Always query and close associated remediation tasks as part of the deduplication process (Step 7). Consider creating a business rule to automate this.

Deduplication Closes a Record That Was Actually Unique

Cause: Two different vulnerable items on the same CI may appear as duplicates if they share a CVE but affect different software components (e.g., same CVE on two different installed applications). Solution: Add port, protocol, or software fields to the matching criteria. Review scanner plugin details in sn_vul_third_party_entry to confirm true duplicates.

Examples

Example 1: Post-Scan Import Deduplication

1. Tenable scan imports 500 new vulnerable items
2. Run duplicate detection: 78 match existing Qualys findings (same CVE + CI)
3. Review sample of 10 duplicates to confirm match accuracy
4. Bulk close 78 duplicates, retaining Qualys records (earlier first_found)
5. Consolidate 23 redundant remediation tasks
6. Generate report showing true new findings: 422 (not 500)

Example 2: Quarterly Vulnerability Hygiene Review

1. Query all open vulnerable items: 4,200 records
2. Run full deduplication analysis across 3 scanner sources
3. Identify 1,100 duplicates (26.2% duplication rate)
4. Find 45 CVEs that appear in all 3 scanners on the same assets
5. Close duplicates, consolidate tasks, update primary records
6. Generate prioritized exposure list: 3,100 unique vulnerabilities
7. Present cleaned data to CISO with accurate exposure metrics

Related Skills

• secops/correlation-insights - Correlate deduplicated vulnerabilities with active security incidents
• secops/metrics-analysis - Track deduplication rates and vulnerability exposure trends
• secops/incident-summarization - Reference deduplicated vulnerability data in incident summaries
• secops/post-incident-analysis - Review vulnerability deduplication effectiveness during PIR
• cmdb/data-quality - Ensure CI records are accurate for proper vulnerability matching