API and Webhook Reliability on VPS

Webhook reliability is not the same as keeping one HTTP endpoint online. A delivery can arrive twice, arrive late, time out after the receiver committed work, or succeed at the receiver while a downstream dependency fails. Reliable handling comes from defining those states and making each one recoverable.

This guide focuses on the receiver and worker design that a team controls. Before implementation, read the webhook provider's current documentation for its signature format, retry schedule, response deadline, event identifier, ordering behavior, and replay tools. Those contracts differ between providers and must not be guessed.

The useful way to frame the problem

Map one event from the sender to its final business effect. Name the sender, public endpoint, authentication method, durable receipt store or queue, worker, downstream dependencies, final state, and operator. Then model these failure modes separately:

• Duplicate delivery: the same event is delivered or replayed more than once.
• Timeout ambiguity: the receiver completes or records work, but the sender times out before seeing the response.
• Lost callback: the sender never delivers the event, or the receiver fails before durably recording it.
• Downstream outage: receipt succeeds but a database, API, email provider, payment system, or internal service is unavailable.
• Invalid signature: authentication fails because the request is forged, altered, stale, or verified against the wrong bytes or secret.
• Retry storm: many failed events retry together and consume receiver, queue, worker, or dependency capacity.
• Slow processing: business work exceeds the sender's response deadline or blocks unrelated events.

For each failure, define what the sender will observe, what the receiver will persist, whether another attempt is safe, where the event waits, how an operator finds it, and how it reaches a terminal outcome. A larger server does not resolve an undefined delivery state.

Design for retries

Assume a sender may deliver an event more than once. Use the provider's stable event ID when available; otherwise derive an idempotency key from fields whose uniqueness and stability are part of the integration contract. Store that key with the processing state and claim it atomically so two workers cannot both perform the same side effect.

Idempotency must protect the business action, not only the HTTP request. If an event creates an invoice, grants access, sends a notification, or updates an external system, record enough state to determine whether that effect already happened. A duplicate can then return the recorded result or safely do nothing.

Classify failures before retrying:

• retry transient transport, throttling, and dependency-availability failures only when another attempt is safe;
• treat malformed payloads, failed authentication, unsupported event types, and rejected business rules as permanent until data, code, or policy changes;
• use bounded attempts and backoff with jitter so one dependency outage does not create synchronized retries;
• cap concurrency separately for each dependency whose limits or failure behavior differ; and
• move exhausted or manually held events to a visible failed-event or dead-letter state rather than retrying forever.

Provide an operator-controlled replay path that uses the same validation, idempotency, logging, and state transitions as normal delivery. Record who initiated a replay and why. Do not make “delete the idempotency record and try again” the default recovery method; that can repeat an external side effect.

Separate receipt from processing

The receiver should do only the work required to authenticate, validate, durably record, and acknowledge the event within the sender's documented deadline. Slow business work belongs in a worker when the provider contract permits asynchronous handling.

A defensible receipt path is:

1. Read the request under an explicit body-size limit.
2. Verify the signature using the exact raw bytes and algorithm required by the sender.
3. Check any documented timestamp or replay constraint before accepting the event.
4. Validate the minimum envelope fields, event type, and stable event identifier.
5. Persist the event or enqueue it durably with an initial state.
6. Return the provider-defined success response only after durable receipt succeeds.

Do not acknowledge first and enqueue later: a crash between those actions creates a lost event. Do not perform an unbounded dependency call before acknowledgement: the sender may time out and redeliver while the first attempt continues.

The worker should claim one event, record its attempt, perform the business transition, and write the resulting state. Define explicit states such as received, processing, succeeded, retryable failure, permanent failure, and held for review. Recover events left in processing after a worker crash through a documented lease or reconciliation mechanism rather than assuming they completed.

Log the right context

One event must be traceable across receipt, queueing, processing, dependencies, and replay without storing unnecessary secrets or payload data. Include:

• provider event or request ID;
• internal correlation ID;
• source/provider and event type;
• receiver endpoint and receive time;
• authentication outcome without logging the signature or secret;
• HTTP response status returned to the sender;
• queue or receipt record ID;
• attempt count and worker identity;
• processing start, duration, and queue wait time;
• dependency name and bounded error category; and
• final outcome, terminal reason, and replay actor when applicable.

Redact authorization headers, signing secrets, API credentials, session material, personal data, payment data, and sensitive payload fields. Prefer a payload hash or selected non-sensitive identifiers when operators need to correlate a request without retaining its full contents.

Set retention from incident, audit, privacy, and recovery needs. Restrict log access and test redaction with representative failed requests; error paths often expose more data than success paths.

Protect external dependencies

Give every outbound dependency an explicit connection and response timeout based on its contract and the application's latency budget. A missing timeout allows stuck work to occupy workers indefinitely. Distinguish timeouts from rejected requests, throttling, authentication failures, invalid data, and server errors because they do not share one safe recovery action.

Apply bounded concurrency and rate controls per dependency. If one service slows down, isolate its queue or worker capacity so it does not block unrelated event types. Pause or open the failure path after repeated dependency failures when continuing would only add load, and define the evidence required to resume. This behavior can be implemented with a circuit breaker or an equivalent explicit state; the exact trigger must come from measured traffic and dependency behavior, not a universal number.

Make retry ownership unambiguous. If a client library retries, the worker must account for those attempts before adding its own outer retry loop. If the downstream API supports idempotency keys, keep the same key across safe retries of one business action.

Plan for backpressure. When arrival rate exceeds processing rate, preserve accepted events, limit admission or concurrency deliberately, expose queue age, and protect the database and dependencies. Adding workers helps only if the queue, database, network, and downstream services can accept the additional concurrency.

Measure the integration path

Measure the delivery path end to end rather than alerting only when the web process stops. Track:

• accepted-event count and authentication rejection count;
• success rate by provider and event type;
• retry rate and attempts per event;
• queue depth and oldest queued event age;
• receipt latency, queue wait, processing duration, and end-to-end completion latency;
• events stuck in processing;
• permanent-failure and dead-letter counts;
• manual replay count and outcome; and
• error and throttling rates for each downstream dependency.

Define an expected traffic pattern and service objective for each integration. Alert on evidence of customer impact or loss risk: sustained queue-age growth, no events when events are expected, a rising failed-event count, authentication anomalies, processing states that do not terminate, or a dependency error rate that prevents completion.

Test the complete path with controlled events. Exercise a duplicate, invalid signature, slow dependency, transient failure, permanent failure, worker restart, queue backlog, and manual replay. Confirm that each produces the expected response, persisted state, metric, log context, alert, and final outcome.

Checklist

• Document the sender's signature, response deadline, retry, ordering, event-ID, and replay contracts from its current documentation.
• Inventory every event type and map it to one owner and one intended business effect.
• Verify signatures against the correct request bytes before trusting payload fields.
• Apply body-size, method, content-type, and event-type validation at receipt.
• Persist or durably enqueue the event before returning success.
• Use a stable idempotency key and atomic processing claim.
• Make every side effect duplicate-safe or record why it cannot be repeated.
• Separate retryable, permanent, and operator-held failures.
• Bound attempts, backoff, concurrency, and dependency timeouts without relying on arbitrary universal thresholds.
• Provide a failed-event view and an audited replay path.
• Log event ID, correlation ID, timing, attempts, dependencies, and final state while redacting secrets and sensitive payload data.
• Measure success, retries, oldest queue age, processing latency, dead letters, stuck events, and dependency errors.
• Send alerts to a destination that remains available when the VPS or application is down.
• Test duplicate, timeout, outage, restart, backlog, and replay scenarios before launch.
• Record the security, monitoring, queue, worker, dependency, and incident owner.

When a VPS is the right fit

A self-managed VPS can fit a webhook receiver when the team needs a continuously running worker, direct control of its runtime and queue process, access to detailed logs, and control over host firewall or network configuration. Those controls can support the design, but they do not create webhook reliability automatically.

The customer owns the operating system, application, queue, database, secrets, access, firewall policy, patching, monitoring, recovery, and incident response. One VPS is also one infrastructure failure domain unless the team explicitly designs independent instances, data durability, health checks, traffic routing, and recovery.

Use a managed queue, serverless receiver, integration platform, or other managed service when its delivery contract and operational model fit better than a team-owned worker stack. Choose by testing duplicate handling, durable receipt, backlog recovery, dependency failure, operating effort, and total cost against the same acceptance criteria.

FAQs

Why did the same webhook arrive twice?

Duplicate delivery is normal in many at-least-once delivery systems and can also follow timeout ambiguity: the sender did not observe a success response even though the receiver committed work. Use the stable event ID and a duplicate-safe business transition. Do not treat a second request as proof that the first one failed.

What if processing takes longer than the sender allows?

Authenticate, validate, and durably record the event within the documented response window, acknowledge only after durable receipt, and process it asynchronously. If the provider requires synchronous processing, design within that exact contract or choose a different integration pattern; returning success before durable receipt risks loss.

When should webhook signatures be validated?

Validate before trusting or acting on the payload. Follow the sender's current instructions for raw-body handling, algorithm, secret selection, timestamp tolerance, and secret rotation. Reject failed verification and log only a safe reason, never the signature or signing secret.

Which webhook metrics should trigger alerts?

Alert on conditions tied to loss or delayed business effects: increasing oldest-event age, sustained queue growth, permanent failures, stuck processing, unexpected authentication failures, missing expected traffic, or dependency errors that block completion. Derive thresholds from the integration's measured traffic and service objective rather than copying a universal number.