An IT incident management process organizes the response to an unplanned service interruption or degradation. Its immediate purpose is to restore usable service while coordinating investigation, decisions and communication. Cause analysis and prevention can continue after recovery.
This guide follows an illustrative internal IT incident: staff at one office cannot release jobs from shared printers after a configuration change. The scenario, records and messages are fictional. They show how to apply the process to an infrastructure service, rather than report results from a real organization.
Define the process before an incident starts
A usable process names the supported services, response coverage, declaring authority, specialist contacts and tools available during a disruption. Check that responders can reach the runbook and communication route when the affected service is unavailable.
Decide what qualifies as a coordinated incident. An isolated device issue might remain with ordinary support; the same symptom across several departments may require a service owner and additional responders. A routine request for another printer is different from an unexpected loss of printing.
Document the separate routes for suspected security events and for disruptions that require business-continuity arrangements. Do not apply breach-detection statistics or a security containment sequence as if they were timing benchmarks for every IT service incident.
| Phase | Required decision | Record to retain |
|---|---|---|
| Detect and assess | Is there a service incident, and what is its impact? | Observed symptoms, affected service, scope and uncertain facts |
| Coordinate | Who owns response, technical work and communication? | Accepted responsibilities and next update |
| Investigate and restore | Which authorized action is justified by the evidence? | Hypothesis, change decision, result and recovery option |
| Verify recovery | Does the service meet the agreed recovery criteria? | User-facing checks, monitoring and remaining limitations |
| Review and improve | What contributed, and what follow-up is warranted? | Timeline, analysis, owned actions and completion evidence |
For a general stage-by-stage map, see our incident management workflow. Here, the focus is how an IT team makes and records the decisions within those stages.
1. Capture the symptom and establish the affected scope
In the printing example, the first report says a job appears in the queue but does not release at the device. Record that observation before naming a cause. “The print server is down” would be premature if the server is reachable and another office can print.
Create a shared incident reference and link matching tickets. Keep the original requesters and their specific impact; duplicate reports can show different locations, devices or conditions. An alert is evidence to assess, not automatic proof of a declared incident.
Illustrative initial assessment Service: Shared printing, Office A Observed: Queued jobs do not release at two tested shared devices. Reports: Several staff reports linked to the incident record. Unaffected comparison: One approved check succeeds in Office B. First report time: Record timestamp and timezone. Earliest confirmed impact: Unknown at initial assessment. Business impact: Staff cannot produce required paper documents through the affected route. Cause: Unconfirmed. Next check: Confirm affected device groups and available alternatives.
Ask which tasks are blocked, whether deadlines matter, and whether an approved alternative works. A service used by fewer people can still have urgent impact. Reassess scope as evidence arrives; the number of tickets alone is not a severity definition.
2. Assign coordination and accept technical work
Google’s incident management guide separates incident coordination, communications and operations. Those responsibilities apply beyond online production services, although a small internal team may combine roles.
In the example, the service owner coordinates the incident, an endpoint specialist investigates the printing configuration, and a support representative maintains employee updates. A recorder can retain decisions and timestamps; that responsibility may sit with the coordinator if the workload allows it.
Confirm who has accepted each task. Sending a notification to a specialist group is not acceptance. If the assigned specialist is unavailable, use the documented backup route. The current owner should keep communication moving until a handoff is accepted.
Keep one shared decision record. Specialists can investigate separate components, but proposed changes should return to the response lead rather than happen independently. Google’s incident guidance describes the risks of uncoordinated operational changes during response.
At a shift change, record the current service state, actions taken, pending decisions, next update and incoming owner’s acceptance. A chat transcript alone can leave the next responder guessing.
3. Investigate without delaying a justified restoration
Compare the failing and working conditions: device group, user permissions, queue configuration, recent changes and relevant logs. Each check should help distinguish a hypothesis, not merely add another screenshot to the record.
Suppose the illustrative team finds that failures began after a print-routing configuration change affecting Office A. That timing supports investigating the change. It does not establish the precise faulty setting or prove that unrelated devices share the same problem.
Evaluate a known workaround or reversal under the existing change authority. A recovery action can be justified before the full cause is understood, provided its risk, scope and verification are explicit.
| Decision field | Illustrative entry |
|---|---|
| Hypothesis | The latest routing configuration contributes to Office A release failures |
| Proposed action | Restore the previously approved configuration for the affected scope |
| Authority and owner | Record emergency-change authorization and the person executing it |
| Expected evidence | Representative authorized users can release jobs at affected devices |
| Risk and fallback | Review routing side effects and the next approved option if recovery fails |
| Outcome | Record actual results and limitations; do not fill this in before execution |
Follow the service’s specific runbook. Restarting everything can introduce another failure or remove evidence without answering the diagnosis question. If a vendor is needed, link the vendor case and clarify the next response expected; retain ownership of updates to your employees.
4. Give each audience a useful update
Employees need the affected task, available alternative and next update time. Technical responders need evidence and decisions. Business stakeholders need the operational effect and any continuity decision. Keep these views consistent without distributing unnecessary diagnostic detail.
The following is an illustrative update for the printing example:
Shared printing in Office A is currently affected. Jobs may remain queued and fail to release at shared devices. The team is checking the routing configuration and testing recovery options. Use the approved alternative described at [internal reference], if applicable. Do not resend the same job repeatedly while the queue is being assessed. Next update by: [date, time and timezone]. Incident reference: [reference].
Only offer an alternative that has actually been approved and checked for the user’s task. Avoid a restoration estimate unless the responsible team has a defensible basis. A next-update commitment can still be honored when the cause or recovery time is uncertain.
| Audience | Update content | Choose cadence by |
|---|---|---|
| Affected staff | Task impact, safe alternative, next update | Impact and how quickly useful information changes |
| Responders | Checks, findings, decisions and action owners | Active work and material changes in evidence |
| Business stakeholders | Blocked operations, dependencies and decisions needed | Operational deadlines and agreed reporting expectations |
Use a fallback channel if email, the portal or identity service is affected. Frequency should follow the incident’s needs; a generic “every 30 minutes” rule is not appropriate for all services or audiences.
5. Verify the service, backlog and remaining limitations
A reachable server or an empty alert list is insufficient recovery evidence. In the illustrative case, test the complete authorized print-release task at representative affected devices and with relevant employee roles. Confirm output reaches the intended device and that access restrictions still behave correctly.
Inspect queued work as well. Decide how to handle stale, repeated or abandoned jobs under the service policy. Recovery can reveal a backlog or side effect that was not visible during the outage.
Record the configuration in use, checks performed, results and coverage limits. Set an observation period suited to the failure pattern and name its monitoring owner. If intermittent failures continue, communicate degraded service rather than declaring complete restoration.
A temporary workaround can restore service while a permanent correction remains open. Make that limitation clear, with an owner and review condition. Closing the incident and completing prevention work are separate decisions.
6. Review causes and improve the relevant controls
A review should explain what happened, what helped restoration and where uncertainty remains. Google’s postmortem guidance emphasizes contributing causes and preventive actions rather than blaming individuals.
For the fictional printing incident, investigate whether the change’s test scope covered Office A’s device group, whether configuration dependencies were documented and whether monitoring exercised the release task. These are review questions, not assumed findings.
Link cause investigation to problem management when it needs further work. Track a confirmed defect, runbook correction or monitoring change with a named owner and completion evidence. “Improve change management” is too vague to verify.
For example, a justified action might be to add an approved representative device-group check to the printing change checklist. Verify it can reveal the relevant failure before treating the action as complete.
Prevent repeat incidents without promising prediction
Review trends across related incidents, noisy alerts, missed detection and recent changes. Google’s monitoring guidance distinguishes user-visible symptoms from diagnostic causes and favors understandable alerts. A capacity trend or anomaly score can inform investigation; it does not guarantee a future crash is predictable.
Integrate support evidence with change planning. Document service dependencies, applicable checks, communication, rollback considerations and who will observe the result. Automate record creation, linking and notifications where useful, but confirm duplicate handling and delivery failures. Automated recovery actions need the same clear authorization and outcome checks as manual actions.
Measure the process with explicit event definitions
A label such as MTTR can mean recovery, restoration or resolution in different reports. Define your events first:
- Impact start: earliest confirmed service disruption, or explicitly unknown.
- Detection: when monitoring or a person first identified the relevant symptom.
- Acceptance: when the responsible responder accepted the incident.
- Mitigation: when an action reduced impact under the stated criteria.
- Verified recovery: when the agreed service checks passed.
- Closure: when final incident communication and required records were completed.
Report detection-to-acceptance separately from impact-to-recovery. If impact start is unknown, do not silently substitute detection and compare it with incidents that have confirmed onset. Segment by service and impact, inspect long cases and state sample sizes.
Pair durations with recurrence, missed updates, unaccepted handoffs and overdue improvement actions. Readiness measures such as exercised runbooks describe preparation, not proof that future incidents will be avoided. Any productivity or cost estimate needs an explicit calculation and assumptions rather than an invented savings figure.
Practice the process before expanding it
Run a bounded tabletop exercise using a fictional service disruption. Check whether participants can find the contact route, accept roles, make a documented recovery decision and explain verification. Record gaps and rehearse the corrected handoff.
Screendesk recordings can supplement a visible symptom or demonstrate the restored task within an existing support workflow. Keep written context and approved diagnostic evidence alongside them. A video alone does not identify the incident cause or prevent future disruptions.
