Hosting Incident Notes Template

Start incident notes as soon as an event needs coordinated investigation or could affect users, data, security, or a critical business process. The notes are a shared operational record: capture what was known at the time, what the team did, who made each decision, and which evidence supports it.

Use one time zone throughout—preferably UTC—and write facts before interpretations. Keep secrets, credentials, private keys, and unnecessary personal data out of the record. Link to access-controlled evidence instead of pasting sensitive log or payload content.

The useful way to frame the problem

Copy this header when opening an incident:

• Incident ID/title:
• Severity:
• Status: Investigating / Identified / Monitoring / Resolved
• Start time:
• Detection time:
• Recovery time:
• End time:
• Services affected:
• User/business impact:
• Current owner/incident lead:
• Trigger/detection source:
• Immediate mitigation:
• Links to logs/dashboards/tickets/runbooks:

Update status and ownership when they change. Leave unknown times marked Unknown rather than estimating them as fact; they can be reconstructed later from reliable evidence.

Capture the timeline

Add an entry whenever the observed state changes, somebody takes an action, the team makes a decision, or new evidence changes the working theory. Use precise timestamps and preserve the order in which information became available.

Record both successful and unsuccessful actions. An unsuccessful mitigation can explain elapsed time and prevent another responder from repeating it. Do not paste authentication tokens or full sensitive payloads into the evidence column.

Separate impact from cause

During active response, write impact as an observed fact and cause as a labelled hypothesis. Do not turn an unverified explanation into a root-cause statement because it sounds plausible.

• Observed user/business impact: Which services, journeys, customers, regions, data, or internal processes were affected? What remained functional?
• Scope evidence: Which external checks, application errors, request samples, support reports, or business metrics establish that scope?
• Working hypothesis: What might explain the observations?
• Evidence for or against the hypothesis: Which result would confirm or reject it?
• Root cause (only after confirmed): State the causal mechanism and cite the confirming evidence. If confirmation is incomplete, say so.
• Contributing factors: Record conditions that increased impact, delayed detection, complicated recovery, or allowed recurrence without presenting them as the initiating cause.

Recovery does not prove cause. A restart may restore a service without establishing why it failed, and a rollback may reduce errors without proving that every changed line contributed. Finalise the cause only after the evidence supports it.

Record decisions

Record consequential decisions in the timeline and summarise them here:

• Decision: What did the team decide?
• Time and decision owner: Who had authority and when was the decision made?
• Evidence available: Which observations, risks, and unknowns informed it?
• Alternatives considered: What other response was considered?
• Expected result and risk: What should change, and what could the action make worse?
• Verification: How will the team know the action worked?
• Result: What actually happened?

Include decisions to deploy, stop a deployment, roll back, restore data, change DNS, rotate credentials, disable functionality, take a service offline, or communicate externally. For a restore, record the recovery point and any newer writes that may be lost. The VPS backup guide explains why a backup's existence is different from a verified recovery path.

List follow-up owners

Every follow-up needs a named owner, due date, and verification method. “Engineering will improve monitoring” is not an action item.

Classify actions where useful: immediate risk reduction, permanent corrective work, detection improvement, recovery improvement, documentation, access, or customer follow-up. Track them in the team's normal work system and link the tickets from the incident record.

Turn notes into prevention

Complete this review after service is stable:

• What worked: Detection, ownership, communication, tooling, procedure, or recovery steps that reduced impact.
• What failed: Missing alerts, unclear ownership, unavailable access, unsafe changes, incomplete rollback, failed recovery, or misleading documentation.
• Detection gap: Could the team have detected the user-visible problem earlier or with a clearer signal?
• Recovery gap: Which dependency, access path, decision, or procedure delayed recovery?
• Recurrence control: Which change prevents the causal mechanism or limits its impact?
• Verification owner and date: Who proves each control works and when?
• Review participants: Who reviewed the final timeline, impact, cause, and actions?
• Finalised date: When was the record accepted as complete?

Keep the original timeline intact. Correct factual errors with an explicit amendment rather than silently rewriting the history responders relied on.

A simple checklist to keep the article actionable

• The incident header identifies severity, status, all four times, affected services, impact, detection source, mitigation, and current lead.
• Timeline entries use one time zone and link observations or actions to owners and evidence.
• Impact is described independently from the working hypothesis.
• Root cause is marked confirmed only when cited evidence supports the causal mechanism.
• Decisions record owner, available evidence, expected result, risk, and verification.
• Follow-ups have individual owners, due dates, and objective verification.
• Logs, dashboards, tickets, changes, communications, and runbooks are linked without exposing secrets.
• What worked, what failed, contributing factors, detection gaps, and recovery gaps are recorded.
• The final record names reviewers and a finalised date.

When a VPS is the right fit

A VPS gives a team direct access to operating-system, application, database, service, and network evidence that can make incident notes more useful. It does not create monitoring, response ownership, backups, recovery, or root-cause certainty automatically.

Virtarix VPS service is self-managed. The customer is responsible for operating-system and application administration, access, security, monitoring, backups, incident response, and recovery. One VPS is also one infrastructure failure domain; continuity and independent evidence storage must be designed separately where the incident plan requires them.

FAQs

When should we start incident notes?

Start when an event needs coordinated investigation or could materially affect users, data, security, or a critical process. Early notes preserve detection time, initial symptoms, and decisions that are difficult to reconstruct later.

Who owns the notes?

Name a note owner or scribe while the incident lead coordinates response. The incident lead remains accountable for the operational record, but another responder should maintain it when combining both roles would slow recovery.

What should we record during active response?

Record timestamped observations, actions, decisions, owners, evidence, status changes, impact changes, communications, and the current hypothesis. Keep entries short and factual; analysis and prose polish can wait until service is stable.

When should we finalise root cause?

Only after the causal mechanism is supported by evidence and reviewed by the relevant owners. If evidence cannot confirm a single root cause, document the remaining uncertainty and verified contributing factors instead of inventing certainty.