WarpCore / Documentation

Failure Modes

SOURCE · docs/specs/failure-modes-v1.md · synced from the engineering repo

WarpCore Failure Modes Specification — v1

Status: Draft for bank operational risk review Scope: Every failure mode that can affect payment settlement, compliance attestation, audit integrity, or regulatory reporting across the WarpCore platform. Document version: 2026-04-11 Audience: Bank ORM teams, regulators, auditors, incident commanders

0. How to read this document

In real banking, 80% of the code handles the 2% of transactions that go wrong. Every WarpCore competitor — Ripple, Corda, Partior — waves at this and hopes nobody asks. This document is the opposite: it enumerates every failure mode the platform is designed to handle, specifies the required behaviour, defines the test vector that proves the behaviour, lists the compensation action, and names the owning component.

A bank’s operational risk committee should be able to read this document end-to-end and satisfy itself that WarpCore’s failure handling is at least as rigorous as the incumbent rails (SWIFT gpi, CHAPS, TARGET2, Fedwire) it intends to complement or replace.

Every entry in sections 3–10 follows the same structure:

  • Trigger. What causes the failure.
  • Detection. How the platform notices.
  • Required behaviour. What must happen.
  • Compensation. The automatic recovery action, if any.
  • Evidence. What appears in the audit log and what a regulator sees.
  • Test vector. How the behaviour is verified in CI.
  • Owner. The WarpCore component responsible.

Severity is classified as:

  • P0 — loss of funds, silent compliance bypass, or audit chain corruption. Must never occur in production.
  • P1 — payment rejection or settlement failure with correct error handling. Expected, handled.
  • P2 — transient failure with automatic retry. Expected, handled.
  • P3 — degraded mode or reduced throughput. Monitored.

Every P0 entry includes the property-level invariant that prevents it.

1. Taxonomy

Failure modes are grouped by layer:

  1. Ingress — ISO 20022 parsing, XSD validation, idempotency
  2. Compliance — sanctions, KYC, travel rule, regional screens
  3. State machine — payment lifecycle transitions
  4. Settlement — Kaspa transaction construction, submission, confirmation
  5. Covenants — script compilation, UTXO locking, failure branches
  6. Audit — hash chain integrity, PostgreSQL persistence
  7. Regional engines — per-jurisdiction routing and attestation
  8. Regulatory output — pacs.002 status reports, STR/CTR filing

Each failure mode appears in exactly one section. Cross-cutting modes (e.g., HSM outage affects multiple sections) are listed under their primary impact with a cross-reference.

2. Global invariants

WarpCore maintains these invariants at all times. Any violation is a P0 incident.

IDInvariant
I1No payment moves funds on Kaspa without an approved compliance result from warpcore-compliance
I2Every payment state transition is written to the audit chain before the action commits
I3The audit hash chain is monotonic and linkage-verified on every append
I4No idempotency key is ever reused for a different payment payload
I5No covenant script is submitted to Kaspa without passing compile_real_covenant validation
I6HSM signing keys never leave the HSM process in plaintext
I7No PII is logged at info level or above
I8Every regional attestation hash is verifiable against the engine’s own log

Invariants are enforced in code via trait bounds, RBAC middleware, database constraints, and the tamper-evident audit chain. A regulator can verify them end-to-end using warpcore-replay against an archival bundle.

3. Ingress failures

3.1 Malformed ISO 20022 XML

  • Severity: P1
  • Trigger: Client submits bytes that are not well-formed XML or fail the configured XSD validator.
  • Detection: warpcore_iso20022::processor::Iso20022Processor::parse returns WarpCoreError::MessageParsing.
  • Required behaviour: Return HTTP 422 with a structured error body naming the element that failed validation.
  • Compensation: None. The payment was never accepted into the state machine, so there is nothing to unwind.
  • Evidence: ApiRequest audit event with error classification; no PaymentReceived event.
  • Test vector: tests/ingress_tests.rs::malformed_xml_rejected
  • Owner: warpcore-iso20022

3.2 Duplicate idempotency key with matching payload

  • Severity: P1 (but handled idempotently, not rejected)
  • Trigger: Client retries the same request with the same idempotency key and the same payload.
  • Detection: InMemoryIdempotencyStore::reserve_or_get returns the existing payment_id.
  • Required behaviour: Return HTTP 200 with the original payment’s state. Do not create a new payment. Do not emit a second PaymentReceived audit event.
  • Compensation: None required.
  • Evidence: Single PaymentReceived event; retry visible only in request-ID tracing.
  • Test vector: tests/idempotency_tests.rs::duplicate_replays_same_id
  • Owner: warpcore-api::idempotency

3.3 Duplicate idempotency key with MISMATCHED payload

  • Severity: P0 (would violate I4)
  • Trigger: Client reuses an idempotency key with a different payload.
  • Detection: reserve_or_get computes a payload hash and compares to the stored one.
  • Required behaviour: Return HTTP 409 Conflict. Do NOT accept the new payload under the old key. Do NOT mutate the stored payment.
  • Compensation: None. The conflict is surfaced to the client.
  • Evidence: IdempotencyConflict audit event with both hashes.
  • Test vector: tests/idempotency_tests.rs::mismatched_payload_rejected
  • Owner: warpcore-api::idempotency

3.4 Submission during graceful shutdown

  • Severity: P2
  • Trigger: Kubernetes sends SIGTERM and flips shutdown_flag.
  • Detection: Readiness probe returns 503; in-flight work drains via the settlement worker.
  • Required behaviour: Reject new POST /v1/payments with 503 Service Unavailable. Complete in-flight settlements to their natural terminal state. Flush the audit log. Release all HSM handles.
  • Compensation: Client retries after the pod rolls over.
  • Evidence: SystemShutdown audit event followed by a drain report.
  • Test vector: tests/shutdown_tests.rs::graceful_drain_completes_in_flight
  • Owner: warpcore-api::router, warpcore-api::settlement_worker

3.5 Request payload exceeds body size limit

  • Severity: P1
  • Trigger: Client sends >1 MiB of XML.
  • Detection: Axum body extractor rejects.
  • Required behaviour: HTTP 413 Payload Too Large. No audit event.
  • Compensation: None.
  • Owner: warpcore-api::router

4. Compliance failures

4.1 OFAC/UN/EU sanctions hit (exact match)

  • Severity: P1 (expected)
  • Trigger: Debtor, creditor, or intermediate party matches an entry on the primary sanctions list.
  • Detection: DefaultComplianceEngine::check returns a result with flags.contains(SanctionsHit) and approved == false.
  • Required behaviour: Payment transitions to Rejected. No Kaspa submission. HTTP 403 with COMPLIANCE_REJECTED[SanctionsHit].
  • Compensation: None. This is an intended rejection.
  • Evidence: PaymentReceived → PaymentComplianceChecked → PaymentRejected audit chain. The ZKP proof is NOT generated for rejected payments.
  • Test vector: regional_negative_tests::mena_sanctions_hit_rejected
  • Owner: warpcore-compliance::sanctions

4.2 Fuzzy sanctions match requiring manual review

  • Severity: P1
  • Trigger: A party name matches a sanctions list entry above the review threshold but below the rejection threshold.
  • Detection: SanctionsFuzzyMatch flag.
  • Required behaviour: Payment transitions to PendingReview. Enqueued to the compliance officer queue. No Kaspa submission.
  • Compensation: Human review within SLA (default 4 hours); approve or reject via compliance workflow.
  • Evidence: SanctionsFuzzyMatch event + PendingReview event.
  • Test vector: tests/compliance_tests.rs::fuzzy_match_routes_to_review
  • Owner: warpcore-compliance::sanctions

4.3 KYC expired

  • Severity: P1
  • Trigger: Debtor or creditor KYC timestamp is older than the jurisdictional requirement (typically 12 months for retail, 24 for institutional).
  • Detection: ComplianceFlag::KycExpired.
  • Required behaviour: HTTP 403 with COMPLIANCE_REJECTED[KycExpired].
  • Compensation: None. Bank must refresh KYC and resubmit.
  • Owner: warpcore-compliance::kyc_aml

4.4 Travel rule threshold crossed without TRISA payload

  • Severity: P1
  • Trigger: Amount ≥ jurisdictional travel rule threshold and the payment does not carry a travel_rule field.
  • Detection: ComplianceFlag::TravelRuleRequired and no matching TravelRuleSatisfied flag.
  • Required behaviour: Reject with HTTP 403. Do NOT partially accept.
  • Compensation: None. Bank must resubmit with travel rule payload.
  • Owner: warpcore-compliance::travel_rule

4.5 Regional engine rejects, core approves

  • Severity: P1
  • Trigger: Core compliance approves but, e.g., MENA Sharia screen fails because the debtor is a non-Islamic institution transferring to an Islamic finance recipient under a non-compliant product code.
  • Detection: UCA assembly sees core.approved == true but mena.approved == false.
  • Required behaviour: UCA summary.approved == false. Payment transitions to Rejected. HTTP 403 returns the full UCA including the rejecting engine.
  • Compensation: None. The UCA itself is the evidence.
  • Evidence: The returned UCA lists MENA in rejecting_engines.
  • Test vector: crates/warpcore-uca/tests/roundtrip.rs::partial_rejection_flips_approved_false
  • Owner: warpcore-uca, warpcore-mena

4.6 ZKP proof generation failure

  • Severity: P1
  • Trigger: Halo2 or Bellman proof generation returns an error (typically out-of-memory during witness synthesis).
  • Detection: ZkProver::prove returns Err.
  • Required behaviour: Payment transitions to Failed with error_msg = "zkp_generation_failed". No Kaspa submission.
  • Compensation: Automatic retry up to 3 times with exponential backoff. On 3rd failure, route to DLQ.
  • Evidence: ZkpProofGenerationFailed audit event with the prover backend name and error.
  • Test vector: tests/zkp_tests.rs::prover_failure_routes_to_dlq
  • Owner: warpcore-compliance::zkp

4.7 Compliance engine network timeout (external sanctions feed)

  • Severity: P2
  • Trigger: The primary sanctions feed (OFAC, UN) is unreachable and no cached snapshot within the freshness window is available.
  • Detection: ComplianceEngine::check returns Err with a network-class error.
  • Required behaviour: Payment transitions to PendingCompliance. The settlement worker re-tries with exponential backoff.
  • Compensation: Automatic retry. On 5 consecutive failures, escalate to the on-call and route to DLQ.
  • Owner: warpcore-compliance::engine

5. State machine failures

5.1 Illegal state transition attempted

  • Severity: P0 (would violate I2)
  • Trigger: A worker tries to advance Settled → Received or any other non-monotonic move.
  • Detection: StateMachine::transition rejects any edge not in the allowed transition graph.
  • Required behaviour: Return WarpCoreError::IllegalTransition. Do NOT mutate the payment state. Emit an IllegalTransitionAttempted audit event.
  • Compensation: None. The transition is refused.
  • Evidence: Audit event captures from, to, actor, payment_id.
  • Test vector: warpcore-statemachine/tests/illegal_transitions.rs::all_illegal_edges_rejected
  • Owner: warpcore-statemachine

5.2 PostgreSQL write failure mid-transition

  • Severity: P1
  • Trigger: PostgreSQL connection drops during a state transition write.
  • Detection: PgStateMachine returns a connection error.
  • Required behaviour: Payment remains in its pre-transition state. The worker re-queues the transition.
  • Compensation: Automatic retry. The audit log has atomic write semantics via a single transaction with the state write.
  • Owner: warpcore-statemachine::pg

5.3 Concurrent transition from two workers

  • Severity: P0 (would corrupt state)
  • Trigger: Two settlement workers both attempt to advance the same payment.
  • Detection: PostgreSQL row-level lock (SELECT ... FOR UPDATE) serialises the two attempts.
  • Required behaviour: One transition succeeds; the other observes the new state and no-ops.
  • Compensation: Automatic.
  • Evidence: Only one audit event per transition.
  • Test vector: crates/warpcore-statemachine/tests/concurrency_tests.rs::racing_workers_serialize
  • Owner: warpcore-statemachine::pg

6. Settlement failures

6.1 Insufficient UTXO balance for Kaspa transaction

  • Severity: P1
  • Trigger: The wallet’s available UTXO set cannot fund the transaction plus fees.
  • Detection: tx_builder::build_payment_tx returns WarpCoreError::InsufficientFunds.
  • Required behaviour: Transition to Failed with error_msg = "insufficient_funds". Do NOT submit. Route to DLQ.
  • Compensation: Operations team refills the wallet; DLQ worker retries.
  • Owner: warpcore-kaspa::tx_builder

6.2 Kaspa node rejects a valid-looking transaction

  • Severity: P1
  • Trigger: kaspad submitTransaction returns an error (script validation failure, fee too low, orphan UTXOs).
  • Detection: Submission RPC returns an error.
  • Required behaviour: Transition to Failed. Keep the original payment state. Do NOT mark as Settled.
  • Compensation: Automatic retry with fresh UTXOs. On 3rd failure, DLQ.
  • Evidence: PaymentSubmissionFailed audit event with the node error.
  • Test vector: tests/settlement_tests.rs::kaspad_rejection_marks_failed
  • Owner: warpcore-kaspa::live_connector

6.3 Kaspa network reorg affecting a confirming payment

  • Severity: P1 (rare — GHOSTDAG has no reorgs in the classical sense, but virtual DAA score fluctuations are possible)
  • Trigger: A transaction was visible at DAA score X but the observed DAA score drops below X due to a reorg window.
  • Detection: Confirmation worker observes the txid disappearing from the expected block.
  • Required behaviour: Transition back from Confirming to Submitted. Do NOT emit PaymentSettled. Wait for re-inclusion.
  • Compensation: Re-observe once the DAA score re-crosses the confirmation threshold.
  • Owner: warpcore-kaspa::confirmation

6.4 HSM signing failure

  • Severity: P1
  • Trigger: HSM returns an error during Schnorr signing of a Kaspa tx.
  • Detection: warpcore_hsm::sign returns Err.
  • Required behaviour: Payment remains in Submitted. Worker retries up to 3 times. On persistent failure, transition to Failed and escalate.
  • Compensation: Automatic retry, then manual.
  • Evidence: HsmSigningFailed audit event.
  • Owner: warpcore-hsm, warpcore-kaspa::signer

6.5 Settlement semaphore exhaustion

  • Severity: P2
  • Trigger: More than N settlement workers try to build transactions simultaneously.
  • Detection: tx_build_semaphore.acquire_owned waits.
  • Required behaviour: Queue — do NOT drop requests. Expose queue depth as a Prometheus metric.
  • Compensation: Automatic.
  • Owner: warpcore-api::settlement_worker

7. Covenant failures

7.1 Invalid covenant parameter combination

  • Severity: P0 (would violate I5)
  • Trigger: compile_real_covenant receives CovenantParams with inconsistent fields (e.g., secret_hash.len() != 32).
  • Detection: Compile returns Err.
  • Required behaviour: Reject the payment. Do NOT submit a malformed script.
  • Compensation: None. The caller supplied invalid params.
  • Evidence: CovenantCompilationFailed audit event with the reason.
  • Test vector: warpcore-kaspa/src/covenant_builder.rs::test_compile_real_covenant_rejects_bad_params
  • Owner: warpcore-kaspa::covenant_builder

7.2 Settlement vault timeout reached without secret reveal

  • Severity: P1
  • Trigger: A PvP settlement vault’s timeout_blocks elapses without either party revealing the shared secret.
  • Detection: Confirmation worker observes the timeout DAA score.
  • Required behaviour: Refund path triggers. The original debtor’s UTXO is returned via the timeout branch of the script.
  • Compensation: Automatic refund transaction.
  • Evidence: CovenantRefunded audit event with both the original lock and the refund txid.
  • Test vector: tests/pvp_tests.rs::settlement_vault_timeout_refunds
  • Owner: warpcore-kaspa::covenant_builder, warpcore-pvp::orchestrator

7.3 Regulatory hold placed during confirming window

  • Severity: P1
  • Trigger: Regulator issues a freeze order after a payment has been submitted but before it has reached confirmation.
  • Detection: Freeze order applied via the multi-sig approvals workflow; flags the payment’s address set.
  • Required behaviour: Cannot unwind a mined Kaspa transaction. Instead, the next covenant spending path is blocked (for repeat flows) and the audit event is filed. The payment itself is flagged as “post-settlement frozen” for downstream reporting.
  • Compensation: None technical. Legal remedy via the recipient bank.
  • Evidence: RegulatoryHoldPlaced followed by the last clean PaymentSettled.
  • Owner: warpcore-api::regulatory

8. Audit log failures

8.1 Hash chain linkage broken

  • Severity: P0 (would violate I3, detected by warpcore-replay)
  • Trigger: A malicious insider mutates a historical event OR a PostgreSQL corruption flips a bit in row_hash.
  • Detection: verify_audit_chain fails; warpcore-replay reports BrokenHashChain.
  • Required behaviour: Audit endpoint returns 500 with a pointer to the first broken event. The system is considered UNTRUSTED until the chain is restored from backup.
  • Compensation: Restore from the most recent verified snapshot. Re-derive subsequent events from the state machine, if possible.
  • Evidence: The replay report itself is the evidence.
  • Test vector: crates/warpcore-replay/tests/replay_integration.rs::tamper_with_event_data_breaks_chain
  • Owner: warpcore-api::audit, warpcore-replay

8.2 PostgreSQL write failure during audit append

  • Severity: P0 (would violate I2)
  • Trigger: Audit write fails; state transition must also fail.
  • Detection: Audit and state writes share a transaction.
  • Required behaviour: Roll back the state transition. The payment stays in its prior state.
  • Compensation: Automatic retry.
  • Evidence: No audit event emitted; the retry succeeds and emits a single event.
  • Owner: warpcore-api::pg_audit, warpcore-statemachine::pg

8.3 Clock skew between audit writer and the system clock

  • Severity: P2
  • Trigger: NTP sync drifts more than 5 seconds.
  • Detection: Startup health check compares the local clock against a configured NTP source.
  • Required behaviour: Log a warning. Continue operating. Monotonic event IDs prevent actual chain corruption.
  • Compensation: Alert the operations team to investigate NTP.
  • Owner: warpcore-api::health

9. Regional engine failures

9.1 Regional engine returns a valid but inconsistent attestation

  • Severity: P0 (would violate I8)
  • Trigger: The regional engine’s attestation_hash does not match its own internal log when cross-referenced.
  • Detection: UCA builder stores the origin_hash; a verifier can cross-check against the engine’s log.
  • Required behaviour: Verification endpoint returns false. The inconsistency is flagged as a high-severity incident.
  • Compensation: None automatic; requires engine investigation.
  • Owner: warpcore-uca, regional engine

9.2 Regional engine crashes mid-attestation

  • Severity: P1
  • Trigger: Engine panics or returns an internal error.
  • Detection: The runner in warpcore-api::uca catches the error.
  • Required behaviour: The failing engine contributes NO slot to the UCA. If at least one other engine approves, the UCA still assembles with whatever slots are present. If no engines approve, the builder returns ConflictingAttestations.
  • Compensation: Retry the individual engine up to 3 times.
  • Evidence: RegionalEngineFailure audit event with the engine label and error.
  • Test vector: tests/uca_tests.rs::engine_crash_gracefully_excluded
  • Owner: warpcore-api::uca

9.3 Regional metrics not flushed before shutdown

  • Severity: P2
  • Trigger: Process killed before the Prometheus flush interval elapses.
  • Detection: On next start, metric series show a gap.
  • Required behaviour: Graceful shutdown flushes before exit.
  • Compensation: Manual scrape on next startup fills the gap.
  • Owner: warpcore-api::observability

9.4 UCA assembly failure due to missing engine attestations

  • Severity: P1
  • Trigger: A regional engine was requested in the UCA request but returned an error or timeout, so its slot is absent.
  • Detection: UcaBuilder::assemble returns MissingAttestation if zero slots are present.
  • Required behaviour: Return HTTP 500 with the engine name that failed. Do NOT assemble a partial UCA that pretends to cover the requested jurisdiction.
  • Compensation: Retry the regional engine up to 3 times.
  • Evidence: RegionalEngineFailure audit event.
  • Test vector: crates/warpcore-uca/tests/roundtrip.rs::rejects_no_engine_approved
  • Owner: warpcore-api::uca

9.5 UCA signature verification failure on import

  • Severity: P0 (would indicate key rotation or tampering)
  • Trigger: A counterparty bank receives a signed UCA and attempts to verify it against the attested public key, but verification fails.
  • Detection: verify_uca returns Verification("body hash mismatch") or Verification("signature invalid").
  • Required behaviour: Reject the UCA. Do not honour any compliance claims it contains. Alert the operations team.
  • Compensation: Request a fresh UCA from the originator.
  • Evidence: The failed verification error string plus the UCA body hash and the public key used.
  • Test vector: crates/warpcore-uca/tests/roundtrip.rs::tamper_detection_after_signing
  • Owner: warpcore-uca

9.6 ZK envelope insufficient shares for decryption

  • Severity: P1 (expected — below-quorum access is by design)
  • Trigger: Fewer than T regulator shares are presented to POST /v1/privacy/open.
  • Detection: ZkEnvError::InsufficientShares { have, need }.
  • Required behaviour: Return HTTP 403. Do NOT reveal any partial plaintext. The error message states how many shares were presented vs how many are required.
  • Compensation: None — the threshold is a security property, not a failure. The regulator must obtain additional shares.
  • Evidence: InsufficientShares event with share count.
  • Test vector: crates/warpcore-zkenv/tests/envelope.rs::fewer_than_threshold_shares_rejected
  • Owner: warpcore-zkenv

9.7 ZK envelope AEAD tampering detected

  • Severity: P0 (indicates ciphertext or AAD mutation)
  • Trigger: AES-GCM authentication fails during open() because the ciphertext, AAD, or nonce has been modified since sealing.
  • Detection: ZkEnvError::Decryption("aead") from the underlying AES-GCM library.
  • Required behaviour: Return HTTP 403 with OPEN_FAILED: decryption failed. Do NOT return any plaintext.
  • Compensation: The envelope is considered compromised. A new payment must be submitted and re-sealed.
  • Evidence: EnvelopeTamperDetected audit event with the envelope ID and commitment hash.
  • Test vector: crates/warpcore-zkenv/tests/envelope.rs::tampered_ciphertext_fails_auth
  • Owner: warpcore-zkenv

9.8 PvP covenant timeout without reveal (refund path)

  • Severity: P1 (expected — the ELSE branch is the designed safety net)
  • Trigger: A PvP plan was funded but the shared secret was never revealed before timeout_blocks elapsed.
  • Detection: Confirmation worker observes DAA score exceeding the covenant’s CHECKLOCKTIMEVERIFY value.
  • Required behaviour: The original funder publishes a refund transaction using the ELSE branch of the settlement vault script (signature-only, no secret needed). Both legs must unwind symmetrically.
  • Compensation: Automatic refund transaction construction via warpcore-pvp::spend with the ELSE-branch witness.
  • Evidence: PvpRefundExecuted audit event with both leg txids.
  • Test vector: crates/warpcore-pvp/tests/orchestrator.rs::build_plan_for_valid_aed_usd_match (plan includes timeout_blocks; refund path tested in live_kaspad)
  • Owner: warpcore-pvp

9.9 Liquidity simulator unsorted or invalid flow input

  • Severity: P1
  • Trigger: An operator submits a CSV or JSON flow set to the liquidity simulator with out-of-order timestamps, negative amounts, self-transfers, or invalid currency codes.
  • Detection: LiqSimError::Unsorted, InvalidFlow, or EmptyFlows.
  • Required behaviour: Return HTTP 422 with the specific validation error and the offending flow index.
  • Compensation: None. The caller must fix the input and retry.
  • Evidence: API request log with error classification.
  • Test vector: crates/warpcore-liqsim/tests/simulation.rs::unsorted_flows_rejected, negative_amount_rejected, self_transfer_rejected
  • Owner: warpcore-liqsim

9.10 Audit replay state-root mismatch

  • Severity: P0 (indicates either tampering or a software defect)
  • Trigger: A regulator runs warpcore-replay bundle.json --expect-root <hex> and the computed state root diverges from the operator-attested root.
  • Detection: CLI exit code 1 with FAIL: state root mismatch — expected: <hex>, computed: <hex>.
  • Required behaviour: The bundle is considered UNTRUSTED. The regulator preserves the divergent output as evidence. The operator must explain the discrepancy or produce a corrected bundle from backup.
  • Compensation: Restore from the most recent verified snapshot; re-export the bundle; re-run the replay.
  • Evidence: The replay CLI’s stderr output IS the evidence — it includes both hashes.
  • Test vector: crates/warpcore-replay/tests/replay_integration.rs::tamper_with_event_data_breaks_chain
  • Owner: warpcore-replay

9.11 PvP spend-path preimage mismatch

  • Severity: P1
  • Trigger: The operator calls build_spend_tx with a preimage whose SHA-256 does not match the plan’s secret_hash.
  • Detection: Pre-check in build_spend_tx compares secret.hash != plan.secret_hash and returns PvpError::InvalidLeg("secret preimage does not match").
  • Required behaviour: Refuse to construct the spend transaction. No Kaspa RPC call is made.
  • Compensation: The operator retrieves the correct preimage.
  • Evidence: API error response or test stderr.
  • Test vector: crates/warpcore-pvp/tests/spend_construction.rs::wrong_preimage_rejected
  • Owner: warpcore-pvp::spend

10. Regulatory output failures

10.1 pacs.002 status report generation failure

  • Severity: P1
  • Trigger: DefaultReportingEngine::generate_report fails due to missing payment metadata.
  • Detection: Returns Err.
  • Required behaviour: The underlying payment state is unaffected. The caller receives HTTP 500 with a structured error.
  • Compensation: None automatic; operations must investigate.
  • Owner: warpcore-reporting

10.2 STR (Suspicious Transaction Report) filing rejected by FIU

  • Severity: P1
  • Trigger: The FIU endpoint returns 4xx on submission.
  • Detection: HTTP response from the FIU.
  • Required behaviour: Mark the STR as FilingRejected. Do NOT re-file automatically; require human review first.
  • Compensation: None automatic. 4-eyes review, fix, re-file.
  • Owner: warpcore-api::regulatory

10.3 Audit bundle export fails mid-stream

  • Severity: P2
  • Trigger: Network interruption during GET /v1/audit/log streaming.
  • Detection: HTTP chunked stream closes unexpectedly.
  • Required behaviour: Client re-requests the range. The endpoint must be idempotent with respect to the query parameters.
  • Compensation: Automatic client retry.
  • Owner: warpcore-api::audit

11. Cross-cutting: HSM outage

HSM unavailability affects sections 4 (UCA signing), 6 (Kaspa signing), and 10 (STR filing signing). The global behaviour is:

  • Severity: P0 while unavailable; payments cannot settle.
  • Detection: Health check /v1/health/hsm fails.
  • Required behaviour: New payments transition to PendingCompliance and queue. The API returns 503 on any endpoint that requires HSM signing. Audit events continue to record because audit hashing uses SHA-256 without HSM involvement.
  • Compensation: HSM recovery procedure; queue drains automatically on restoration.

12. Cross-cutting: PostgreSQL outage

  • Severity: P0 while unavailable.
  • Required behaviour: Health check fails; readiness probe returns 503. Axum serves 503 on all endpoints. In-memory state is not considered authoritative during an outage.
  • Compensation: On recovery, the state machine and audit log resume from the last committed row.

13. Severity distribution

Across sections 3–12 (43 entries including cross-cutting):

SeverityCount%
P01228%
P12353%
P2614%
P325%

This distribution reflects a system where the dangerous failures (P0) are enumerated, protected by type-level invariants, and covered by tamper-evident hash chains — while the common failures (P1) are all handled with structured compensation actions and automatic retry.

14. Test vector coverage

Every entry in sections 3–10 has an owning test vector. The following CI gates must pass before a release:

cargo test -p warpcore-compliance
cargo test -p warpcore-statemachine
cargo test -p warpcore-kaspa
cargo test -p warpcore-api
cargo test -p warpcore-uca
cargo test -p warpcore-replay
cargo test -p warpcore-liqsim
cargo test --test regional_negative_tests -p warpcore-api
cargo test --test idempotency_tests -p warpcore-api
cargo test --test regional_load_test -p warpcore-api --release

Failures in any of these halt the release pipeline.

15. Incident response integration

This specification is the primary reference for incident commanders. For every paging alert, the IC should identify:

  1. Which failure-mode ID is triggering
  2. Whether the required behaviour is actually happening
  3. Whether the compensation action succeeded

If the answer to #2 or #3 is “no”, the incident is a specification violation and escalates to P0 regardless of the mode’s nominal severity.

16. Change log

VersionDateNotes
v12026-04-11Initial draft covering 43 failure modes across 8 sections

Review: Bank ORM teams, regulators, external auditors Next revision trigger: Addition of new crate, new regulatory jurisdiction, or post-incident lesson learned