Marsen n8n approval webhook recovery lab v1.0.0
Recorded run: 5 October 2026. n8n 2.35.4, Node 24.18.1.

Purpose
Exercise authenticated callbacks, exact-payload approval, duplicate delivery,
durable decisions and destination receipt reconciliation across process restarts.
The destination and proposals are fictional. This is a reproducible teaching lab,
not a deployed approval service, human review portal or production CRM template.

Reproduce on Linux with Python 3 and Docker
1. Download and unzip recovery-lab.zip into a fresh directory.
2. Review workflow.json, fixture-server.mjs, test-lab.mjs and run-lab.py.
3. Run: sudo python3 run-lab.py
   Run without sudo only if your Linux UID is 1000 and you can use Docker.
4. Open test-results.json. Expect total=18 and passed=18. Inspect each case,
   the separate n8n execution records and preRestartEvidence.
5. Inspect cleanup.json for productionContainersStillRunning=true and
   labContainersRemoved=true. Only names observed before the run are checked.

Docker downloads docker.n8n.io/n8nio/n8n:2.35.4 if it isn't cached. The runner
creates a uniquely named internal Docker network and two limited containers.
It publishes no host port and mounts no live n8n state. Credentials are random,
temporary lab credentials generated for this run. The runner kills and restarts
only those two named containers, exports evidence and removes the containers
and network in its cleanup block. Temporary credential files and disposable
n8n state are removed. Synthetic fixture state remains at the printed path;
review it, then remove that exact temporary directory when no longer needed.
Runtime logging is saved locally, not included in the published evidence bundle.
The recorded run used ARM64 Linux; other host platforms were not evaluated.

Workflow and reviewer identity
The import file uses three built-in nodes:
Authenticated callback -> Record exact decision -> Reconcile then deliver.
The callback is POST /webhook/approval-recovery-lab in the disposable instance.
Header Auth authenticates one fixed lab reviewer-a credential. The HTTP Request
node sets reviewer-a itself; incoming actor/reviewer fields are ignored. This
maps one credential to one reviewer, and does not implement multi-user identity.
The two credential references in workflow.json contain names/IDs, never tokens.
Importing the JSON alone is insufficient: it needs the private fixture service,
seeded proposals and matching credentials. The runner configures these together.
Do not activate this workflow on a live n8n instance.

Durable state and destination
fixture-server.mjs uses separate SQLite databases for approvals and destination
receipts. BEGIN IMMEDIATE and a unique proposal decision prevent two concurrent
callbacks from making conflicting decisions in this single-process fixture.
An approval records a pending delivery before the destination is called. The
writer checks the stable action ID for an existing receipt before creating a
synthetic destination entry. A receipt and target version update are committed
in one destination transaction. This atomic behavior belongs to the fixture;
it does not demonstrate that a real CRM supplies the same capabilities.
Decisions bind proposal ID, version, target and note. Expiry uses Date.now() on
the fixture server and treats now >= expires_at as expired. A rejected decision
is terminal. A replay must still pass reviewer, version and payload checks.

18 recorded scenarios
Authenticated approval; missing credential; incorrect credential; wrong owner;
spoofed body identity; changed note; changed target; old version; rejection
followed by approval; expired server-clock deadline; invalid decision; ten
concurrent approvals; concurrent approve/reject; target-version conflict;
changed replay after a write; pending proposal across restart; failure before
write followed by recovery; committed write with lost response followed by
receipt reconciliation after restart.

The initial two delivery faults are expected failed n8n executions. Passing the
test means the ledger and final destination match the scenario's expectation,
not that every execution has status=success. For approve/reject concurrency,
whichever valid decision commits first is terminal; the test accepts either
winner and verifies its matching destination effect.

Fault and restart boundaries
The lost-response case deliberately returns HTTP 504 after a committed fixture
write. The before-write case returns HTTP 503 without a write. These are injected
HTTP responses, not measured provider network timeouts. Both n8n and the fixture
receive SIGKILL after the initial callbacks have completed, then start with the
same isolated database mounts. The harness resubmits the same exact callback
once, and verifies one receipt. Recovery is explicitly initiated by the test;
no autonomous recovery worker, scheduler or retry policy is included.

Not measured
Real CRM integration, real reviewer sessions, Wait-node resumption, queue mode,
multiple workers, multiple fixture processes, database-level crash during a
transaction, heavy load, retention, backup recovery or customer outcomes.
Use the separate production acceptance worksheet for checks in your real stack.

Files
workflow.json: credential-free import graph.
fixture-server.mjs: isolated durable decision and synthetic destination service.
test-lab.mjs: assertions and HTTP callback driver, run in the private container.
run-lab.py: reproducible container setup, restart, evidence export and cleanup.
build-workflow.mjs: rebuilds workflow.json; doesn't change an imported workflow.
test-results.json: original dated evidence; reproduction overwrites the local copy.
manifest.json: SHA-256 of published source and evidence files.
LICENSE.txt: MIT license for this example kit.

Primary documentation
https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.webhook/
https://docs.n8n.io/integrations/builtin/core-nodes/n8n-nodes-base.wait/
https://docs.n8n.io/deploy/host-n8n/configure-n8n/use-the-command-line

Guide: https://www.marsentech.com/guides/n8n-approval-webhook-recovery
Earlier in-memory lab: https://www.marsentech.com/guides/n8n-human-approval-workflow