Lesson 1 · New Relic alert conditions · read one completely before you trust it
A condition is one stage in a chain. Each stage has a name, and mixing the names up is how alerting discussions go wrong.
| Stage | What it is | What decides it |
|---|---|---|
| Signal | A stream of values over time, produced by a query. | The nrql block and the aggregation settings. |
| Threshold breach | The signal crossed a line and stayed there. | The terms block. |
| Alert event | The record that a breach opened. Sometimes called an incident. | The condition, once a term is satisfied. |
| Issue | A grouping of alert events. What a human actually looks at. | The policy's issue creation preference. |
| Notification | The message that reaches Slack or on-call. | A workflow attached to the policy. |
Depending on a policy's issue creation preference, and any workflows you may have configured, an alert event may result in the creation of an issue, or notifications being sent.
The condition controls everything up to the alert event, and nothing after it. If you are getting too many messages rather than too many breaches, the fix is in the policy or the workflow, not in your threshold.
| Preference | Grouping | Notification volume |
|---|---|---|
| One issue per policy (default) | Every alert event from every condition in the policy joins one issue. | Fewest |
| One issue per condition | One open issue per condition. All signals on that condition join it. | More |
| One issue per condition and signal | One issue per condition per facet value. | Most |
Source: New Relic docs, Decide when issues are created. In Terraform and the API the third value is named PER_CONDITION_AND_TARGET.
Policy 4525023 "Carrier Capabilities (CCAP)" uses
PER_CONDITION. Policy 4446701 "Carrier Monitoring"
uses PER_CONDITION_AND_TARGET, so a faceted condition there opens
a separate issue per carrier. The orders-py policies use
PER_POLICY.
This matters for a faceted condition. A per-carrier facet under
PER_CONDITION_AND_TARGET can open dozens of issues from one
underlying fault.
Every NRQL alert condition is four blocks. Each block answers exactly one question. If you read only one block, you cannot predict the alert.
| Block | The one question it answers | Terraform variables |
|---|---|---|
nrql |
What number do we measure? | query |
terms |
Which numbers are bad, and for how long? | critical_term, warning_term |
signal |
How do we cut time into windows, and what do we do with gaps? | aggregation_window, aggregation_method, aggregation_delay, fill_option, fill_value, slide_by |
expiration |
What happens when the signal stops arriving, and when do we force-close? | expiration_duration, open_violation_on_expiration, close_violations_on_expiration, violation_time_limit_seconds |
Source: New Relic docs, alert conditions,
mapped to Shippo variable names in shippo-tf-services/modules/newrelic-alert/README.md.
threshold_duration is per term, not per
condition. A condition with a critical term at 600 s and a warning term at 360 s
does not alert after 360 s at critical priority. Read every entry in
terms[] and note the priority field on each one.
threshold_occurrences decides how the duration is
consumed. ALL requires every aggregation window inside the duration
to breach. AT_LEAST_ONCE requires a single window. Same duration,
very different sensitivity.
This is [CCAP] Carrier MCA Registration Count | Anomaly Detection
(id 63897179, policy Carrier Capabilities (CCAP)), read live with
mcp__shippo-new-relic-mcp__get_nrql_alert_conditions. Do not read a
condition from Terraform or from a runbook. Both drift.
Ids are not stable. Between 13 and 20 August the four
[CCAP] FedEx REST Rating conditions were deleted and recreated.
The names survived. The ids did not: 61106123, 61061369, 57969138 and 57968861
became 66242857, 66242858, 66242859 and 66242860. Any note, runbook, or lesson
that cited only the number now points at nothing.
Name first, id second. The name is what a person recognises in an alert and what survives a rebuild.
name [CCAP] Carrier MCA Registration Count | Anomaly Detection
type BASELINE
enabled false
policyId 4525023
nrql FROM Metric
SELECT sum(carrier_account_registration.count)
WHERE success IS TRUE
AND carrier != 'Australia Post'
AND environment = 'prod'
FACET carrier
terms [ { priority: WARNING, operator: ABOVE, threshold: 5,
thresholdDuration: 7200, thresholdOccurrences: ALL } ]
signal aggregationWindow: 7200 aggregationMethod: EVENT_FLOW
aggregationDelay: 120 fillOption: NONE
expiration expirationDuration: 86400
openViolationOnExpiration: true
closeViolationsOnExpiration: false
violationTimeLimitSeconds: 259200
Now read it block by block.
Successful carrier registrations per carrier, production only, Australia Post
excluded. FACET carrier means one independent signal per carrier.
Each carrier opens and closes its own incident.
One term only, and it is WARNING. There is no critical term. On a
BASELINE condition the threshold 5 is not a count of
registrations. It is a number of standard deviations away from the predicted
baseline. The 7200 s duration with ALL occurrences means: the
deviation must hold for one full 7200 s window before an incident opens.
fillOption: NONE is the important line. A carrier with no traffic
produces no data point. New Relic does not substitute a zero, so the baseline
model simply receives nothing for that carrier.
openViolationOnExpiration: true combined with
expirationDuration: 86400 is a signal-loss alert. If a carrier sends
no registrations for 24 hours, an incident opens even though no threshold was
crossed. A carrier that is simply idle over a weekend looks identical to a
carrier that is broken.
This condition previously produced weekend false positives. The
nrql block was never the cause. The cause was in
expiration: signal loss on a low-traffic, per-carrier facet.
When you triage a noisy alert, name the block before you edit the query.
This condition is currently enabled: false and its query now
carries environment = 'prod'. Earlier notes recorded it as enabled
and unfiltered. About half of Shippo's carrier alerting is edited in the New
Relic UI, so Terraform and Confluence both go stale. Re-query before you cite a
threshold.
1. [CCAP] Carrier MCA Registration Count | Anomaly Detection opened incidents for idle carriers over a weekend. No threshold was crossed. Which block caused it?
2. The condition is type BASELINE with operator ABOVE and threshold 5. What is the 5?
3. thresholdOccurrences is ALL over a 7200 s duration. What must happen for the incident to open?
4. One fault opens 40 separate issues from a single condition faceted per carrier. Your threshold is correct. Where is the setting that caused this?
Primary source, 10 minutes: New Relic docs, Alert conditions. Read the section on condition types and the section on thresholds. Skip the UI walkthrough; you write these in Terraform.
Next: Lesson 2, thresholds under diurnal volume. Shippo carrier traffic swings 1.84× between its quietest and busiest hour. That breaks every static count threshold, and the fix is a pattern you already have in production.
Ask your teacher. Anything unclear in this lesson is a question for the agent, not a gap you should live with. Ask for a deeper pass on any block, or ask it to pull a different live condition and quiz you on it.