The four blocks of an alert condition

Lesson 1 · New Relic alert conditions · read one completely before you trust it

Where a condition sits

A condition is one stage in a chain. Each stage has a name, and mixing the names up is how alerting discussions go wrong.

StageWhat it isWhat decides it
SignalA stream of values over time, produced by a query.The nrql block and the aggregation settings.
Threshold breachThe signal crossed a line and stayed there.The terms block.
Alert eventThe record that a breach opened. Sometimes called an incident.The condition, once a term is satisfied.
IssueA grouping of alert events. What a human actually looks at.The policy's issue creation preference.
NotificationThe message that reaches Slack or on-call.A workflow attached to the policy.

Depending on a policy's issue creation preference, and any workflows you may have configured, an alert event may result in the creation of an issue, or notifications being sent.

New Relic docs, Set thresholds for an alert condition.

The condition controls everything up to the alert event, and nothing after it. If you are getting too many messages rather than too many breaches, the fix is in the policy or the workflow, not in your threshold.

Issue creation preference: the setting that is not on the condition

PreferenceGroupingNotification volume
One issue per policy (default)Every alert event from every condition in the policy joins one issue.Fewest
One issue per conditionOne open issue per condition. All signals on that condition join it.More
One issue per condition and signalOne issue per condition per facet value.Most

Source: New Relic docs, Decide when issues are created. In Terraform and the API the third value is named PER_CONDITION_AND_TARGET.

In Shippo's account

Policy 4525023 "Carrier Capabilities (CCAP)" uses PER_CONDITION. Policy 4446701 "Carrier Monitoring" uses PER_CONDITION_AND_TARGET, so a faceted condition there opens a separate issue per carrier. The orders-py policies use PER_POLICY.

This matters for a faceted condition. A per-carrier facet under PER_CONDITION_AND_TARGET can open dozens of issues from one underlying fault.

The four blocks of a condition

Every NRQL alert condition is four blocks. Each block answers exactly one question. If you read only one block, you cannot predict the alert.

BlockThe one question it answersTerraform variables
nrql What number do we measure? query
terms Which numbers are bad, and for how long? critical_term, warning_term
signal How do we cut time into windows, and what do we do with gaps? aggregation_window, aggregation_method, aggregation_delay, fill_option, fill_value, slide_by
expiration What happens when the signal stops arriving, and when do we force-close? expiration_duration, open_violation_on_expiration, close_violations_on_expiration, violation_time_limit_seconds

Source: New Relic docs, alert conditions, mapped to Shippo variable names in shippo-tf-services/modules/newrelic-alert/README.md.

The two words that cause most misreadings

threshold_duration is per term, not per condition. A condition with a critical term at 600 s and a warning term at 360 s does not alert after 360 s at critical priority. Read every entry in terms[] and note the priority field on each one.

threshold_occurrences decides how the duration is consumed. ALL requires every aggregation window inside the duration to breach. AT_LEAST_ONCE requires a single window. Same duration, very different sensitivity.

The worked example: a live condition

This is [CCAP] Carrier MCA Registration Count | Anomaly Detection (id 63897179, policy Carrier Capabilities (CCAP)), read live with mcp__shippo-new-relic-mcp__get_nrql_alert_conditions. Do not read a condition from Terraform or from a runbook. Both drift.

Cite conditions by name, not by id

Ids are not stable. Between 13 and 20 August the four [CCAP] FedEx REST Rating conditions were deleted and recreated. The names survived. The ids did not: 61106123, 61061369, 57969138 and 57968861 became 66242857, 66242858, 66242859 and 66242860. Any note, runbook, or lesson that cited only the number now points at nothing.

Name first, id second. The name is what a person recognises in an alert and what survives a rebuild.

name        [CCAP] Carrier MCA Registration Count | Anomaly Detection
type        BASELINE
enabled     false
policyId    4525023

nrql        FROM Metric
            SELECT sum(carrier_account_registration.count)
            WHERE success IS TRUE
              AND carrier != 'Australia Post'
              AND environment = 'prod'
            FACET carrier

terms       [ { priority: WARNING, operator: ABOVE, threshold: 5,
                thresholdDuration: 7200, thresholdOccurrences: ALL } ]

signal      aggregationWindow: 7200   aggregationMethod: EVENT_FLOW
            aggregationDelay:  120    fillOption: NONE

expiration  expirationDuration: 86400
            openViolationOnExpiration:   true
            closeViolationsOnExpiration: false
            violationTimeLimitSeconds:   259200

Now read it block by block.

nrql

Successful carrier registrations per carrier, production only, Australia Post excluded. FACET carrier means one independent signal per carrier. Each carrier opens and closes its own incident.

terms

One term only, and it is WARNING. There is no critical term. On a BASELINE condition the threshold 5 is not a count of registrations. It is a number of standard deviations away from the predicted baseline. The 7200 s duration with ALL occurrences means: the deviation must hold for one full 7200 s window before an incident opens.

signal

fillOption: NONE is the important line. A carrier with no traffic produces no data point. New Relic does not substitute a zero, so the baseline model simply receives nothing for that carrier.

expiration

openViolationOnExpiration: true combined with expirationDuration: 86400 is a signal-loss alert. If a carrier sends no registrations for 24 hours, an incident opens even though no threshold was crossed. A carrier that is simply idle over a weekend looks identical to a carrier that is broken.

The lesson underneath the example

This condition previously produced weekend false positives. The nrql block was never the cause. The cause was in expiration: signal loss on a low-traffic, per-carrier facet. When you triage a noisy alert, name the block before you edit the query.

Live state is the only state

This condition is currently enabled: false and its query now carries environment = 'prod'. Earlier notes recorded it as enabled and unfiltered. About half of Shippo's carrier alerting is edited in the New Relic UI, so Terraform and Confluence both go stale. Re-query before you cite a threshold.

Check yourself

1. [CCAP] Carrier MCA Registration Count | Anomaly Detection opened incidents for idle carriers over a weekend. No threshold was crossed. Which block caused it?

2. The condition is type BASELINE with operator ABOVE and threshold 5. What is the 5?

3. thresholdOccurrences is ALL over a 7200 s duration. What must happen for the incident to open?

4. One fault opens 40 separate issues from a single condition faceted per carrier. Your threshold is correct. Where is the setting that caused this?

Read this next

Primary source, 10 minutes: New Relic docs, Alert conditions. Read the section on condition types and the section on thresholds. Skip the UI walkthrough; you write these in Terraform.