Back to docs
Security Runbook

Breach Response Runbook

Step-by-step procedures for detecting, containing, investigating, and recovering from a security incident. Keep this tab open if an alert fires.

If you are in an active incident: go directly to Phase 1 → Detect & declare. Do not read the whole document first.
1

Detect & declare

Owner: On-call engineer

0 – 15 min
  1. 1Confirm the alert is a genuine breach (not a test, scanner, or misconfigured webhook).
  2. 2Declare an incident in Slack #incidents: "P0 BREACH — [brief description]". Pin the message.
  3. 3Page the incident lead (CEO / CTO) immediately. Do not wait.
  4. 4Do NOT delete logs, rotate keys, or restart services until the incident lead says so — it destroys evidence.

Completion checklist

  • Incident declared in Slack
  • Incident lead paged
  • No evidence destroyed
2

Contain

Owner: Incident lead + on-call

15 min – 2 hr
  1. 1Identify the blast radius: which identities, platforms, and API keys were in scope.
  2. 2Revoke all API keys in the affected environment via dashboard → API Keys → Revoke all. Document which keys were revoked and when.
  3. 3If the breach involves the signing key (RS256 private key for VIT): rotate it immediately in the secrets manager, then republish the JWKS endpoint. All existing VITs become invalid — communicate this to affected platforms.
  4. 4Block the attacker IP / CIDR at the WAF / load balancer level. Do not terminate the connection cleanly (TCP RST is fine — it generates a log entry).
  5. 5If OrbitShield signals a spike in aml_flagged or blacklisted lookups on a single key, auto-suspend that key (or suspend it manually) and record the identity IDs queried.
  6. 6Snapshot the relevant DB tables (platform_connections, verified_identities, api_keys, audit_events) to a write-once store before any cleanup.

Completion checklist

  • Blast radius identified
  • Affected API keys revoked
  • Signing key rotated (if applicable)
  • Attacker IP blocked
  • DB snapshot taken
3

Investigate

Owner: Security lead

2 – 24 hr
  1. 1Pull full audit_events for affected identity IDs. Look for unusual event_type values (status_check storms, bulk vit_issued, unexpected platform_connection events).
  2. 2Correlate API key usage with IP addresses in the access logs. One key from many IPs = key compromise. Many keys from one IP = credential stuffing.
  3. 3Check OrbitShield fraud graph for patterns: was this a single actor or a coordinated ring?
  4. 4Determine root cause: leaked key, compromised signing secret, misconfigured CORS, injection, or insider.
  5. 5Preserve all evidence with timestamps. Use SHA-256 hashes to establish chain of custody if law enforcement may be involved.
  6. 6Draft a timeline: first evidence of the breach → detection → containment → now.

Completion checklist

  • Audit log reviewed
  • Root cause identified
  • Timeline drafted
  • Evidence preserved with hashes
4

Notify

Owner: CEO + Legal

< 72 hr from discovery
  1. 1Affected platforms: email the platform's registered address with (a) what happened, (b) which of their users are affected, (c) what data was exposed, (d) what we have done, (e) what they should do. Use plain language — no weasel words.
  2. 2Affected identities: if PII was exposed, notify affected users directly if you have their contact (phone or email). Be specific — do not send vague "security incident" emails.
  3. 3Regulators: Nigeria's NDPC requires breach notification within 72 hours of discovery under the NDPA 2023. Other countries' data authorities as applicable.
  4. 4Do NOT post on social media until regulatory notifications are filed or legal has cleared it.
  5. 5Log every notification: to whom, when sent, message ID or email thread.

Completion checklist

  • Affected platforms notified
  • Affected users notified (if contact available)
  • NDPC/regulators notified within 72h
  • Notification log completed
5

Recover

Owner: Incident lead

24 – 72 hr
  1. 1Issue new API keys to affected platforms. Coordinate the rotation — do not leave platforms broken.
  2. 2If the VIT signing key was rotated: work with each platform to re-verify affected users or accept a grace-period re-issuance.
  3. 3Re-enable services in staging first, validate with sandbox credentials, then cut over production.
  4. 4Verify OrbitShield fraud graph is clean — no residual fraud signals from the incident period.
  5. 5Remove WAF blocks once the threat is neutralized (keep the block rule, disable it).
  6. 6Monitor for 48 hours post-recovery: watch for recurrence, unusual lookup patterns, or new attacker pivots.

Completion checklist

  • New API keys issued to all affected platforms
  • VIT re-issuance coordinated (if signing key rotated)
  • Production restored from staging
  • Post-recovery monitoring active
6

Post-mortem

Owner: All responders

< 7 days post-recovery
  1. 1Write a blameless post-mortem: timeline, root cause, impact, what worked, what didn't, action items.
  2. 2Assign every action item an owner and a due date. Put them in the sprint immediately — not a backlog.
  3. 3Typical action items: add a detection test for the attack vector, improve alerting thresholds, tighten CORS/network policies, add rate limiting to newly discovered exposure, improve runbook steps that were unclear.
  4. 4Share the post-mortem with all connected platforms. Transparency builds trust.
  5. 5Store the post-mortem in a versioned location (GitHub, Notion). Link it from this runbook.

Completion checklist

  • Post-mortem written
  • Action items assigned and dated
  • Post-mortem shared with platforms
  • Runbook updated if steps were wrong

Emergency contacts

RoleContactNote
Incident lead (CTO)Page via on-call rotationAlways first call for P0
CEODirect callNotify within 30 min of P0 declaration
Legal / Compliancelegal@verifyafrica.comRequired for regulatory notifications
NDPC (Nigeria)info@ndpc.gov.ngMandatory within 72h under NDPA 2023
Platform supportsupport@verifyafrica.comExternal escalations from platforms

Severity guide

P0

Active breach, signing key exposure, bulk identity data exfiltration. Page incident lead immediately.

P1

Suspected compromise, abnormal lookup volumes, single compromised API key. Contain within 1h.

P2

Potential misconfiguration, low-volume anomalous activity. Investigate within 24h.

P3

Security improvement or near-miss. Schedule remediation in next sprint.

Keep this runbook accurate. Update it after every incident post-mortem.← Back to docs