Auditing the policy you own

Lesson 7 · New Relic alert conditions · what the Carrier Capabilities (CCAP) policy actually contains

Prerequisites: lessons 1 to 4. This lesson applies them to one real policy and reports what it finds, without softening.

Every number below comes from reading policy 4525023, Carrier Capabilities (CCAP), on 2026-08-20. Re-run the queries before you act on any of it.

The inventory

Exactly half the policy is switched off. Of the half still running, under forty percent can wake anyone up. Nineteen conditions out of ninety-eight carry the entire paging load for carrier capabilities.

Twenty-three descriptions that prove the plan was bypassed

The house Rego policy rejects any Terraform plan whose NRQL condition has a description shorter than five words. Twenty-three enabled conditions in this policy have exactly that. A plan could not have produced them, so they were created in the New Relic UI, where no policy check runs.

Seven enabled conditions have no runbook_url, which the same Rego policy also requires. When one of those fires, the person on the other end gets a threshold breach and no instructions.

This is worth stating plainly: the repository looks like it enforces a standard, and roughly half of what is actually running never went through it. Reading the Terraform will therefore tell you what the disciplined subset looks like, not what the policy contains.

The naming is not a naming problem

Ninety-eight conditions carry 25 distinct bracket prefixes, and 12 have no prefix at all.

Two names are used twice, by two different conditions each: [CFG][USPS] Access Token Cache Miss Rate Too High and [CFG][USPS] Error Rate For USPS V3 Refresh Tokens Timeout. If either fires, the name in the notification does not tell you which condition produced it.

The reason this matters more than tidiness: the name is the only durable handle a condition has. Between 13 and 20 August the four [CCAP] FedEx REST Rating conditions were deleted and recreated, and their ids all changed. The names carried over. Anything that cited the number now points at nothing.

The policy has drifted past carriers

Sitting inside Carrier Capabilities (CCAP), all enabled:

• K8s Pods Image Pull Error Alert
• K8s Container Error Alert
• Optimizely Audience Condition Type Mismatch - Carrier Capabilities (CCAP)

The first two are cluster health. They fire on image pulls and container restarts, which is honest infrastructure monitoring and has nothing to do with a carrier integration. Their presence here means the policy's issue grouping and its notification workflow now cover two unrelated concerns. Anyone who mutes the policy during a carrier incident also mutes Kubernetes.

The bug that survived a rewrite

Four enabled conditions filter on status_code >=, and all four are the [CCAP] FedEx REST Rating set. Three of them compare status_code >= 500.

status_code on CARRIER_INTEGRATION_RESPONSE is reported by keyset() as both string and numeric, and some rows carry a literal '5XX'. A numeric comparison matches none of those rows.

SELECT count(*) FROM Log
WHERE event = 'CARRIER_INTEGRATION_RESPONSE'
  AND environment = 'prod'
  AND status_code IN ('5XX', '4XX')
FACET status_code, integration
SINCE 24 hours ago

Every row that comes back is 5XX. UPS leads with 33,519 in a day, then FedEx with 4,629, then twenty-odd other carriers. Of the FedEx rows, 1,393 land inside the exact filter of the three 5xx conditions and are counted as zero. Because all three alert on a percentage, those events leave the numerator and stay in the denominator, so the reported failure rate is pushed down from both sides.

An audit you can repeat

None of the above required reading a condition by hand. Pull the policy and compute over it.

mcp__shippo-new-relic-mcp__get_nrql_alert_conditions
  { "policy_id": "4525023", "limit": 300 }

Then, over the saved result:

jq -r '[.conditions[]|select(.enabled)] as $e |
  "enabled \($e|length)",
  "can page \([$e[]|select((.terms//[])|map(.priority)|index("CRITICAL"))]|length)",
  "no runbook \([$e[]|select(.runbookUrl==null or .runbookUrl=="")]|length)",
  "desc under 5 words \([$e[]|select(((.description//"")|split(" ")
     |map(select(length>0))|length)<5)]|length)",
  "numeric status_code \([$e[]|select(.nrql.query|test("status_code *>="))]|length)"
' conditions.json

Four numbers and a list of names. That is the whole audit, and it takes about a minute.

What to fix, in the order the evidence supports

1. The three 5xx comparisons. There is a measured, ongoing under-count and a one-line fix: (status_code >= 500 OR status_code LIKE '5%').
2. The two descriptions that contradict their settings. They are the text a human reads at 3 a.m.
3. The seven enabled conditions with no runbook link.
4. The 30 warning-only conditions. Decide, per condition, whether it should be able to page. Some should not. Right now that has not been decided so much as defaulted.
5. The 49 disabled conditions. Delete or document. A policy that is half decoys is harder to read every single time.
6. The two Kubernetes conditions. Move them to a policy about Kubernetes.

Check yourself

Read this next

For the naming guidance the vendor actually recommends, see New Relic docs, Alert conditions, section "Name your condition". It proposes priority, signal, and entity in the name, for example P2 | High Avg Latency | WebPortal App. Compare that against the 25 prefixes above.