Terms, durations, and paired conditions

Lesson 4 · New Relic alert conditions · why one signal often needs more than one condition

Prerequisites: lessons 1, 2, and 3.

A term is a complete rule, not a number

Each entry in terms[] carries four fields and one priority. Read all five or you will attribute a behaviour to the wrong term.

Field	Meaning
`priority`	`CRITICAL` or `WARNING`. Decides which notification path fires. A condition may carry one of each, never two of the same.
`operator`	`ABOVE`, `ABOVE_OR_EQUALS`, `BELOW`, `BELOW_OR_EQUALS`, `EQUALS`.
`threshold`	The comparison value. On a `BASELINE` condition this is a deviation count, not a raw value.
`threshold_duration`	How long the breach must persist, in seconds. Per term.
`threshold_occurrences`	`ALL` or `AT_LEAST_ONCE`. How the duration is consumed.

The CCAP-2753 mistake, stated plainly

An alert appeared to lag by six minutes. The investigation looked for a 360-second setting and found one. It belonged to the warning term. Both critical terms were at 600 seconds. The 360 was never on the paging path.

The habit that prevents it: when you read a condition, print every term with its priority attached. Never quote a duration without its priority.

`ALL` against `AT_LEAST_ONCE`

Both consume threshold_duration. They disagree about what counts.

Setting	Opens an incident when	Behaviour	Terms in 1000 sampled conditions
`ALL`	Every aggregation window inside the duration breaches.	Slow, certain. One clean window resets the clock.	966
`AT_LEAST_ONCE`	Any single window inside the duration breaches.	Fast, noisy. Catches spikes that `ALL` would smooth away.	264

The interaction with aggregation_window decides everything. If the window equals the duration, there is exactly one window inside it and the two settings are identical. [SHIPMENT] Canada Post Carrier Integration Error Rate (tf) is a live example: window 1200 s, duration 1200 s, occurrences AT_LEAST_ONCE. The occurrences setting there is decorative, except that slideBy: 60 creates overlapping windows, which brings it back to life.

The paired-condition pattern

CCAP runs four separate conditions over the same FedEx rating endpoint. This is deliberate. One condition cannot express four different responses. All four live in the Carrier Capabilities (CCAP) policy and were read live on 2026-08-20.

Condition	Splits on	Terms	Window
`[CCAP] FedEx REST Rating - 5xx burst (Shippo origin)` id 66242858	Error message is not a FedEx outage code	WARNING ABOVE 15%, 300 s, ALL	300 s
`[CCAP] FedEx REST Rating - 5xx errors (Shippo origin)` id 66242859	Same filter, sustained	CRITICAL ABOVE 10% and WARNING ABOVE 5%, both 900 s, ALL	900 s
`[CCAP] FedEx REST Rating - 5xx errors (carrier upstream)` id 66242857	Error message is `SERVICE.UNAVAILABLE.ERROR` or `SYSTEM.UNAVAILABLE.EXCEPTION`	WARNING ABOVE 25%, 1800 s, ALL	1800 s
`[CCAP] FedEx REST Rating - 4xx errors` id 66242860	`status_code` between 400 and 499	CRITICAL ABOVE 10% and WARNING ABOVE 5%, both 600 s, ALL	300 s

Two of these four disagree with their own descriptions

[CCAP] FedEx REST Rating - 5xx errors (Shippo origin) says "Evaluated over a 15-minute sliding window". Its slideBy is null. There is no sliding window. The 900-second figure is right; the word "sliding" is not.

[CCAP] FedEx REST Rating - 4xx errors says "Evaluated over a 10-minute window". Its aggregationWindow is 300 seconds. The ten minutes is the threshold duration, which is a different setting doing a different job.

These descriptions are the text that lands in Slack when the condition fires. Both were written on 20 August, by someone who had the config open. The four-block reading habit from lesson 1 is not a beginner exercise.

Only two of the four can page. The burst condition and the carrier-upstream condition carry a WARNING term and nothing else, so a sustained FedEx outage posts to a channel and never wakes anyone. That may well be the right call for a carrier-side problem. It is worth knowing that it is the call, rather than discovering it during an incident.

Two independent axes are in play.

Axis one: whose fault is it. The same HTTP 5xx is split by error message into a FedEx-side outage and a Shippo-side defect. They get different thresholds because they need different responses. A FedEx outage is acknowledged and monitored. A Shippo defect is a deploy rollback.

Axis two: burst against sustained. A short window with a low threshold catches a spike and channel-posts. A long window with a higher threshold pages only when the problem persists. Putting both on one condition is impossible: a condition has at most one critical and one warning term, and those share a single aggregation window.

The same split appears in services/carrier-monitoring/newrelic.tf, where two module blocks divide traffic on shippo.integration.response.error.expected into "Unexpected errors" and "Expected errors".

The design question to ask first

Not "what threshold?" but "how many distinct responses does this signal need?" Each distinct response is a separate condition. Then each condition gets the threshold that fits its response.

Reference: the same ideas expressed in code

Reference section

Everything above is platform knowledge and applies to any New Relic account. What follows is only how Shippo happens to write it down. Read it for the two conceptual details marked below; skim the rest.

A live module call from shippo-tf-services/services/carrier-monitoring/newrelic.tf, trimmed for length.

module "largeacctrans_carriersmonitoringerrortransactiontrackingv3" {
  source = "../../modules/newrelic-alert"

  count     = var.env_name == "prod" ? 1 : 0
  policy_id = data.newrelic_alert_policy.carrier_monitoring[0].id
  env_name  = var.env_name
  type      = "static"
  name      = "[LARGE-ACC-TRANS] Carriers_Monitoring_Error_Transaction_v3 - Unexpected errors (tf)"

  description = <<-EOT
  Unexpected integration failures (normalization marks the error as not
  expected). Treat as real outages or defects requiring immediate investigation.

  error_limit_pct: {{threshold}}
  EOT

  runbook_url = "https://shippo.atlassian.net/wiki/x/DgASMgE"

  enabled                        = true
  violation_time_limit_seconds   = 2592000
  fill_option                    = "static"
  fill_value                     = 0
  aggregation_window             = 300
  aggregation_method             = "event_flow"
  aggregation_delay              = 60
  expiration_duration            = 600
  open_violation_on_expiration   = false
  close_violations_on_expiration = true

  query = <<-EOT
    SELECT (floor(count(*) / clamp_min(count(*), 5)) * percentage(count(*),
      WHERE status != 'success'
      AND environment='prod'
      AND test!=true
      AND service IN ('transaction')
      AND shippo.integration.response.error.expected != true
      ...
      )) as failure_rate
    FROM Log
    WHERE service IN ('transaction')
    AND environment='prod'
    ...
    FACET integration, service
    EOT

  critical_term = {
    operator              = "above"
    threshold             = 5
    threshold_duration    = 900
    threshold_occurrences = "at_least_once"
  }

  tags = merge(module.defaults.default_tags, module.defaults.noc_tags, {
    Team = "noc"
  })
}

Two conceptual details worth taking away

The volume floor is sized per signal, not copied. This condition uses clamp_min(count(*), 5), not 50. A per-carrier facet on a transaction service carries far less traffic than the FedEx rating endpoint. Floor, window, and facet cardinality are one decision. See lesson 2.
{{threshold}} is substituted at notification time. Putting it in the description means the message that reaches a human carries the actual number that was breached. {{entity.name}} behaves the same way. This is a platform feature, not a Shippo convention, and it is the cheapest way to make an alert self-explaining.

Mechanical notes, for when you actually edit the file: Terraform takes lowercase values ("static", "event_flow", "above", "at_least_once") while the API returns them uppercase. Same value, two spellings. opa-policies/newrelic.rego rejects a plan whose NRQL condition has a description under 5 words, a blank runbook_url, or an overridden account_id.

Evidence that OPA is not the whole story

The policy requires runbook_url on every NRQL alert condition. Of the 1000 live conditions returned by the API, 278 have no runbook_url at all.

A Terraform plan could not have produced those. They were created in the New Relic UI, where no policy runs. This is the click-ops problem stated as a number: about 28% of live conditions never passed the gate that the repository appears to enforce.

Check yourself

1. A condition has a CRITICAL term at 600 s and a WARNING term at 360 s. The on-call page arrived late. Which duration explains the paging delay?

2. A condition uses a 300-second aggregation window and a 300-second threshold duration. What difference does switching ALL to AT_LEAST_ONCE make?

3. You need a fast channel post on a FedEx spike and a slow page on a sustained FedEx outage. Why can this not be one condition?

4. A condition uses ALL over a 900-second duration with 300-second windows. Two of the three windows breach, the middle one does not. What happens?

Read this next

Primary source: New Relic docs, Create NRQL alert conditions, section "Threshold duration and occurrences".

Reference, when you need the argument names: shippo-tf-services/modules/newrelic-alert/README.md lists every input with its type and default.

That completes the core track. Lessons 5 and 6 are the extension track: New Relic capabilities that exist and that Shippo alerting mostly does not use.

Next: Lesson 5, baseline conditions and seasonality.

Ask your teacher. Bring any condition and ask how many distinct responses its signal really needs. That answer decides how many conditions it should be.