Post-Implementation Review

Post-Implementation Review

Deploying a new system is not the end of the analyst's job — it is a checkpoint. A few weeks or months after go-live, when the dust has settled and real users have been working with the system in earnest, the project team gathers for a Post-Implementation Review (PIR), sometimes called a post-deployment review or benefits realisation review. Its purpose is disciplined reflection: did we actually deliver the benefits we promised, and what can we learn so the next project goes better?

Without a PIR, organisations repeat the same planning mistakes cycle after cycle, promised benefits evaporate without accountability, and lessons learned stay in people's heads until those people leave. With a rigorous PIR, the organisation accumulates institutional knowledge, sponsors gain confidence that IT investments are tracked, and analysts develop a feedback loop between what they estimated and what actually happened.

What the PIR Covers

A thorough PIR examines four dimensions:

1. Delivery performance — Did we finish on time, on budget, and within scope? What caused any variance?
2. Benefit realisation — Are the promised benefits materialising against the original business case?
3. Quality and technical health — How stable is the system? What does support data say?
4. Lessons learned — What should we start, stop, or continue on the next project?

Benefit Realisation: Did We Deliver the Value?

The most strategically important question in a PIR is whether the investment delivered the value that justified it. This requires returning to the business case and feasibility study written before the project began and comparing promises to reality.

Consider a logistics firm that replaced its manual route-planning process with an automated dispatch system. The business case stated three quantified benefits:

• Reduce average delivery time by 15% within three months of go-live.
• Cut fuel costs by 10% through optimised routing.
• Reduce dispatcher headcount from eight to five through automation (redeployment, not redundancy).

Six months after go-live, the PIR examines actual KPI data from operations, compares it to the pre-system baseline, and rates each benefit as achieved, partially achieved, or not achieved. It also identifies unplanned benefits (e.g., customer satisfaction scores improved because ETAs became more accurate — a side-effect nobody measured in the business case) and unplanned costs (e.g., additional training required because drivers were unfamiliar with the mobile app).

Delivery Performance Analysis

Before examining benefits, the PIR records how the project was delivered. Compare actuals against the approved project baseline:

• Schedule variance — How many weeks over or under the planned end date? What caused the variance (scope additions, underestimated complexity, resource gaps)?
• Budget variance — Final spend versus approved budget, broken down by category (hardware, software licences, consultancy, internal labour). Explain variances above a defined threshold (commonly 5–10%).
• Scope variance — What was descoped, deferred, or added via change control? Each change-control decision should be referenced by its change request ID.

For example: the clinic booking system was delivered two weeks late because the HL7 integration with the laboratory system was more complex than estimated during feasibility. The PIR records this as a risk materialised, links it to the original risk register entry, and notes that the risk had been rated "medium" but should have been "high" — a lesson for the next integration project.

Quality and Technical Health

A PIR examines post-go-live system quality using operational data rather than test data:

• Helpdesk ticket volume — How many incidents were raised in the first four to eight weeks? Is the trend improving?
• Defect backlog — How many known defects remain open? What are their severities?
• System availability — Uptime percentage versus the SLA commitment (e.g., 99.5% target).
• Performance against NFRs — Are response times, concurrency limits, and data volumes behaving as specified?
• User adoption rate — What percentage of intended users are actively using the system? Low adoption often signals unresolved usability issues or inadequate training.

Lessons Learned: The Most Durable Output

The lessons-learned section is frequently the most valuable part of the PIR because it directly improves future projects. A good lessons-learned process is blameless, specific, and actionable — not a list of vague complaints.

Structure each lesson as three parts:

1. What happened? — A factual description of the event or pattern.
2. Why did it happen? — The root cause (use the "5 Whys" or a simple cause-and-effect analysis).
3. What should we do differently? — A concrete, ownable recommendation for the next project.

Who Participates in the PIR?

The PIR is not a project-team-only event. Broad participation produces richer findings:

• Project sponsor and key stakeholders — Assess whether the business objectives were met and sign off on the benefit realisation findings.
• Business analyst(s) — Lead the benefit realisation analysis; own the lessons-learned facilitation and documentation.
• Project manager — Present delivery performance data (schedule, budget, scope variance).
• Development / technical lead — Report on system quality, known defects, and technical debt.
• End-user representatives — Provide ground-truth feedback on usability, training adequacy, and real-world fit.
• IT operations / support — Report helpdesk volumes, incident patterns, and maintenance observations.

The PIR Report

The PIR produces a formal report that becomes part of the project's permanent record. A standard PIR report contains:

1. Executive summary — One page: overall verdict on benefit delivery and three headline lessons.
2. Project overview — Objectives, scope, go-live date, and team composition for reference.
3. Delivery performance — Schedule, budget, and scope variance with root-cause explanations.
4. Benefit realisation scorecard — Each planned benefit rated achieved/partial/not achieved with evidence, plus unplanned benefits and costs.
5. Quality and technical health summary — Support data, open defects, availability statistics.
6. Lessons learned — Structured table (what/why/recommendation) with an owner and a target date for each recommendation.
7. Actions and follow-up — Any remaining actions from the PIR itself (e.g., a second benefits check at 12 months, a change request for a deferred enhancement).

Benefits Tracking Beyond the First PIR

For large systems, a single PIR at six months is not enough. Some benefits — particularly cultural ones like improved data quality or reduced process workarounds — take a year or more to fully materialise. Best practice is to set a benefits tracking schedule at project closure: a formal check at six months (first PIR), a second check at twelve months, and an ongoing annual review for systems that carry strategic KPI targets.

The online store example makes this clear. A new recommendation engine was expected to increase average order value by 8% within six months. At the six-month PIR, the figure was only 3% — partially achieved. But the team noted that the personalisation model was still learning from limited data. The twelve-month check showed 9% improvement — ultimately above target. Without the second check, the organisation would have incorrectly classified a successful investment as a partial failure.

Summary

• The Post-Implementation Review is a structured evaluation held after go-live to assess delivery performance, benefit realisation, system quality, and lessons learned.
• Use a two-stage approach: a technical PIR at four to six weeks and a benefits PIR at three to six months.
• Benefit realisation compares actual KPI movements to the promises made in the business case; record achieved, partial, not achieved, unplanned benefits, and unplanned costs.
• Lessons learned are most valuable when structured as: what happened, root cause, and a concrete actionable recommendation with an owner.
• Facilitate lessons-learned sessions as blameless process reviews — focus on decisions and systems, not individuals.
• Store PIR reports accessibly and schedule follow-up benefits checks for large investments; some benefits take 12+ months to materialise fully.