Gaps, fills, and signal loss

Lesson 3 · New Relic alert conditions · what happens when the data simply stops

Prerequisites: Lesson 1 and Lesson 2. This lesson covers the signal and expiration blocks, the two that never appear in a runbook and cause most false positives.

The knowledge: absence is not zero

An NRQL condition evaluates a time series. Every aggregation window produces one data point per facet. When no events arrive, there is no data point. That is a gap, and a gap is not the number zero.

New Relic gives you two independent controls, in two different blocks:

Block	Setting	Question it answers
`signal`	`fill_option`, `fill_value`	Should a short gap be replaced by a value before the threshold sees it?
`expiration`	`expiration_duration`, `open_violation_on_expiration`, `close_violations_on_expiration`	After a long silence, should we open an incident, close open ones, or do nothing?

They are ordered. Filling happens first. Expiration only applies when the silence outlasts expiration_duration.

The three fill options

Value	Effect on a gap	Use when	Count in 1000 sampled conditions
`NONE`	Leave the gap. The series has a hole.	Absence is meaningful, or you want expiration to handle it.	667
`STATIC`	Insert `fill_value`, normally `0`.	Absence genuinely means zero, because no errors happened.	319
`LAST_VALUE`	Repeat the previous data point.	A gauge that persists: queue depth, a cached ratio.	14

The reasoning trap

fill_option: STATIC with fill_value: 0 is safe on an error signal and dangerous on a success signal. Filling zero errors during an outage hides the outage. Filling zero successes during a quiet night invents one.

Ask: if my pipeline broke and sent nothing, does a zero here read as healthy or as broken? If it reads as healthy, do not fill with zero.

Three live Shippo conditions, three different answers

All three are real, all three were read live today. Compare them column by column.

Setting	`[CCAP] Carrier MCA Registration Count` id 63897179, disabled	`[SHIPMENT] Canada Post Carrier Integration Error Rate (tf)` id 65685767	`[CCAP] FedEx REST Rating 5xx errors (carrier upstream)` id 66242857
`fillOption`	NONE	NONE	STATIC, value 0
`aggregationWindow`	7200 s	1200 s	1800 s
`slideBy`	n/a	60 s	n/a
`expirationDuration`	86400 s	900 s	null
`openViolationOnExpiration`	true	false	false
`closeViolationsOnExpiration`	false	true	false
Behaviour on silence	Opens an incident after 24 h.	Closes open incidents after 15 min.	Never reacts to silence at all.

MCA Registration Count: silence means broken

This condition faceted on carrier and treated a missing carrier as a fault. With fillOption: NONE there is nothing to evaluate, so after 24 hours openViolationOnExpiration fires. A carrier that is merely idle over a weekend is indistinguishable from a carrier that is broken. This produced repeated weekend false positives. The condition is now enabled: false.

Canada Post Error Rate: silence means unknown

The opposite choice. If Canada Post traffic stops for 15 minutes, any open incident closes. The reasoning: this condition measures a failure rate, and with no calls there is no failure rate to report. Holding an incident open on no evidence would be a lie.

Note also slideBy: 60 on a 1200 s window. The window slides every 60 seconds instead of stepping every 1200. Detection is faster and the series is smoother. Only 62 of the 1000 sampled conditions use this. It is covered in lesson 6.

FedEx Rating 5xx upstream: silence means zero

fillOption: STATIC with fillValue: 0 means gaps never reach the expiration logic. Every empty window becomes a 0% error rate. expirationDuration is null, so signal loss is not monitored at all.

Recall from lesson 2 that this condition's IF(count(*) >= 50, ...) guard emits nothing below 50 events. The STATIC fill is what converts that into a harmless 0. The guard and the fill setting are one design, split across two blocks. Change either alone and the condition changes meaning.

What this means for triage

When a condition fires and the NRQL looks innocent, check whether the incident is a threshold breach at all. An incident opened by openViolationOnExpiration has no breaching value behind it. Only 33 of the 1000 sampled conditions can do this, but when one of them fires it reports as an ordinary incident and reads like a real breach.

The other two expiration controls

violation_time_limit_seconds force-closes an incident that has stayed open too long, whatever the signal says. [CCAP] Carrier MCA Registration Count | Anomaly Detection sets 259200 s, which is 72 hours. It is a safety valve against an incident that can never resolve, not a detection setting.

close_violations_on_expiration and open_violation_on_expiration are independent booleans. All four combinations are legal and all four mean something different.

open	close	Meaning
false	false	Ignore silence entirely. Incidents persist until the signal returns and clears them.
false	true	Silence is uninformative. Stop asserting a problem. (Canada Post Error Rate)
true	false	Silence is itself the fault. Page on it. (MCA Registration Count)
true	true	Replace any existing incident with a signal-loss incident.

How to choose, in order

Decide what silence means for this specific signal. Healthy, broken, or unknown. Write it in the description.
Healthy → fill_option: STATIC, fill_value: 0, and leave expiration off.
Broken → fill_option: NONE, open_violation_on_expiration: true, and set expiration_duration longer than the longest legitimate quiet period. Check that period against real data, per facet.
Unknown → fill_option: NONE, close_violations_on_expiration: true.
Verify the quiet period you assumed:

SELECT count(*) FROM Log
WHERE event = 'CARRIER_INTEGRATION_RESPONSE'
  AND environment = 'prod'
  AND integration = 'canada_post'
FACET hourOf(timestamp)
SINCE 7 days ago LIMIT 24

If the quietest hour is not comfortably above zero, a facet-level gap is normal traffic and open_violation_on_expiration will page you for it.

Check yourself

1. A condition measures carrier error rate and alerts ABOVE 10%. The log pipeline breaks and delivers nothing during a genuine carrier outage. fill_option is STATIC with fill_value 0. What happens?

2. Canada Post traffic stops for 20 minutes while an incident is open on [SHIPMENT] Canada Post Carrier Integration Error Rate (tf). What happens to that incident?

3. You inherit a paging condition with fillOption NONE and openViolationOnExpiration true, faceted per carrier. Which check must you run before trusting it?

4. The FedEx Rating 5xx upstream condition pairs an IF-guarded ratio with fillOption STATIC and fillValue 0. Someone changes fillOption to NONE and nothing else. What breaks?

Read this next

Primary source: New Relic docs, Set thresholds, sections "Gap filling strategies" and "Loss of signal".

Provider reference for the exact argument names: Terraform newrelic_nrql_alert_condition.

Next: Lesson 4, terms, durations, and the paired-condition pattern. Why CCAP runs two conditions on the same FedEx endpoint, and how to write one that passes OPA on the first plan.

Ask your teacher. Bring any condition ID and ask for the silence analysis: what its gaps mean, and whether its expiration settings agree.