Most software does not fail because the code was wrong. It fails because something nobody owned was missing on the day it went live: an alert that pointed at a dead inbox, a database with no tested restore, a dependency that expired on a Sunday. A production readiness review is the meeting where you find those things while they are still cheap.

Blocks the launchRed

A red item stops the release until it is resolved.

Needs an ownerAmber

Amber items need a named owner and a date against them.

DemonstratedGreen

Green means someone has demonstrated it, not that someone believes it.

This is the checklist we work through before putting a system in front of real users. Use it as a gate, not a formality. Anything you cannot answer with evidence is an open risk, not a green tick.

This is the Sthenos seven step production readiness playbook. The seven areas are reliability and failure behaviour, data and recovery, observability, security, performance and capacity, release and rollback, and ownership and operations. There are 29 checks across them, and this page scores every check from 0 to 4 on evidence rather than opinion, for a maximum of 116. The delivery playbook that gets an AI built prototype to this standard is the seven step playbook in from vibe coding to production. Those seven steps are the work. These seven areas are the measurement.

29checks in the scorecardCounted from the seven area tables below.
7areas they sit inEach area is scored on its own, and each carries its own maximum.
0 to 4score per check0 absent, 2 unproven, 4 demonstrated with evidence.
116maximum totalThe seven area maxima below sum to it.
  1. Reliability and failure behaviour5 checks, out of 20
  2. Data and recovery4 checks, out of 16
  3. Observability4 checks, out of 16
  4. Security5 checks, out of 20
  5. Performance and capacity3 checks, out of 12
  6. Release and rollback4 checks, out of 16
  7. Ownership and operations4 checks, out of 16

Figure 1. What each area is worth, out of 116

A horizontal bar per area, the bar length proportional to that area’s maximum score, from twenty for reliability down to twelve for performance and capacityReliability and failure behaviour20Data and recovery16Observability16Security20Performance and capacity12Release and rollback16Ownership and operations16Maximum points per area

Bar length is proportional to the area maximum. Source: the footer row of each of the seven tables on this page, which sum to 116. The weights are equal by design, so an area is large here only because it holds more checks.

1. Reliability and failure behaviour

  • Every external dependency has an explicit timeout. No unbounded waits anywhere in the request path.
  • Retries use exponential backoff with jitter, and are only applied to operations that are safe to repeat.
  • The system degrades rather than collapses when a downstream service is unavailable. You have decided, in writing, what each failure looks like to the user.
  • There is a documented single point of failure list. Each entry has either a mitigation or an accepted risk with a named owner.
  • Health checks test real dependencies, not just that the process is alive.
Check Score it 0, 2 or 4 Evidence a reviewer accepts
1.1 Explicit timeouts on every dependency 0 No timeouts are set, so a slow dependency can hold a request open indefinitely.
2 Timeouts exist on some dependencies, and nobody has walked the request path to find the unbounded ones.
4 Every dependency in the request path has an explicit timeout, and the path has been walked to prove none is unbounded.		 The timeout value per dependency in code or configuration, plus the request path walkthrough that found no unbounded wait.
1.2 Backoff with jitter, retries only where repeating is safe 0 Retries are immediate, unlimited, or applied to anything that fails.
2 Backoff is configured somewhere, and which operations are safe to repeat has never been decided.
4 Exponential backoff with jitter is in place, and the operations it retries are listed and are safe to repeat.		 The retry policy in code or configuration, and the list of operations classified as safe to repeat.
1.3 Degrades rather than collapses, decided in writing 0 A downstream outage takes the whole system down, and nobody has decided what the user sees.
2 Some fallbacks exist, and the user facing behaviour for each failure is not written down.
4 Degraded behaviour is written down per dependency and has been exercised with that dependency switched off.		 The written degradation behaviour per dependency, plus a test or game day record showing it degrade with the dependency disabled.
1.4 A single point of failure list, every entry owned 0 No single point of failure list exists.
2 A list exists, and entries carry neither a mitigation nor a named owner.
4 Every entry carries either a mitigation or an accepted risk with a named owner against it.		 The single point of failure list with a mitigation or an owner name on every row.
1.5 Health checks that test real dependencies 0 There is no health check, or one that returns success whenever the process is running.
2 A health check exists and it does not touch the database, the queue or the downstream APIs.
4 The health check exercises real dependencies and has been seen to fail while one of them was broken.		 The health check implementation, and a log of it failing during a real or simulated dependency outage.
Your score for this area (5 checks at 4 points each) out of 20

2. Data and recovery

  • Backups run on a schedule, and a restore has actually been performed into a clean environment. An untested backup is not a backup.
  • Recovery point and recovery time objectives are written down and agreed with the business, not assumed by engineering.
  • Database migrations are reversible, or the forward-only path has been rehearsed.
  • Personal data is inventoried, and deletion requests can be honoured across every store that holds it, including logs, caches and analytics.
Check Score it 0, 2 or 4 Evidence a reviewer accepts
2.1 Backups with a restore actually performed 0 No scheduled backup, or nobody can say whether one runs.
2 Backups run on a schedule and no restore has ever been performed from them.
4 A restore into a clean environment has been performed and the restored data checked against production.		 The backup schedule and a dated restore record naming who ran it and what was verified afterwards.
2.2 Recovery objectives agreed with the business 0 No recovery point or recovery time objective exists.
2 Engineering has numbers in mind and the business has never agreed them.
4 Both objectives are written down, agreed with the business, and the tested restore meets them.		 The signed off recovery point and recovery time objectives, plus the restore timing that shows they are achievable.
2.3 Reversible migrations, or a rehearsed forward only path 0 Migrations go straight to production with no reverse and no rehearsal.
2 Migrations are reversible on paper, and neither the reversal nor the forward only path has been rehearsed.
4 Either the down migration or the forward only recovery has been rehearsed on a production shaped copy.		 The migration rehearsal record on a production shaped copy, showing the reverse or forward only path completing.
2.4 Personal data inventoried, deletion honoured everywhere 0 There is no inventory of where personal data lives.
2 An inventory exists and covers the primary database only, not logs, caches or analytics.
4 The inventory covers every store, and a deletion has been executed end to end across all of them.		 The data inventory listing every store, and a completed deletion request showing removal from logs, caches and analytics too.
Your score for this area (4 checks at 4 points each) out of 16

3. Observability

  • Logs are structured and centrally searchable, and they do not contain credentials, tokens or personal data.
  • Dashboards exist for the handful of metrics that actually indicate user pain: error rate, latency at the tail, queue depth, saturation.
  • Every alert routes to a person who is awake and empowered to act. Alerts that nobody can action are deleted, not muted.
  • You can answer “what changed?” within minutes, because deploys are correlated with the metrics.
Check Score it 0, 2 or 4 Evidence a reviewer accepts
3.1 Structured, searchable logs with no secrets in them 0 Logs are unstructured, local to each host, or absent.
2 Logs are centralised, and nobody has checked them for credentials, tokens or personal data.
4 Logs are structured and centrally searchable, and a search for secrets and personal data came back clean.		 A sample structured log line, the central search tool, and the dated scan of log content for secrets and personal data.
3.2 Dashboards for the metrics that show user pain 0 There is no dashboard, or one that shows only host CPU and memory.
2 A dashboard exists and is missing tail latency, queue depth or saturation.
4 Error rate, latency at the tail, queue depth and saturation are all on a dashboard somebody actually looks at.		 The dashboard itself, showing those four metric families.
3.3 Alerts that reach someone awake and able to act 0 Alerts go to a shared inbox nobody reads, or there are no alerts.
2 Alerts route somewhere, and some fire routinely and are ignored or muted.
4 Every alert has a recipient who can act, and unactionable alerts have been deleted rather than muted.		 The alert routing table with a named recipient per alert, and the list of alerts deleted for being unactionable.
3.4 Deploys correlated with the metrics 0 Deploy times are not recorded anywhere near the metrics.
2 Deploys are logged, and correlating one with a metric spike is a manual archaeology exercise.
4 Deploy markers appear on the dashboards, and a recent change was identified from them within minutes.		 A dashboard carrying deploy markers, and an incident note showing the change identified from it.
Your score for this area (4 checks at 4 points each) out of 16

4. Security

  • No secrets in source control, in environment files committed to the repo, or in client-side code. Rotation has been tested at least once.
  • Authentication and authorisation are enforced server-side on every route, including the ones nobody links to.
  • Dependencies are scanned, and there is a named owner for acting on findings within an agreed window.
  • Access to production follows least privilege, and there is a leavers process that revokes standing credentials, including API tokens and application passwords, not just logins.
  • TLS certificates and domain registrations have expiry monitoring. A surprising number of outages are simply an expiry nobody was watching.
Check Score it 0, 2 or 4 Evidence a reviewer accepts
4.1 No secrets in source control, rotation tested 0 Secrets sit in the repository or ship in client side code.
2 Secrets have been moved to a manager, and rotation has never been tested.
4 No secret is in source control or client code, and a rotation has been performed without an outage.		 A secret scan across the full repository history returning clean, plus the dated rotation record.
4.2 Server side authentication and authorisation on every route 0 Access control is enforced in the interface only, or not at all.
2 Most routes are guarded server side, and nobody has enumerated the routes to prove it.
4 Every route is enumerated, and each one is proven to enforce authentication and authorisation server side.		 The enumerated route list naming the guard on each route, and a test that calls the unlinked routes without a session and is refused.
4.3 Dependency scanning with a named owner and a window 0 Dependencies are never scanned.
2 A scanner runs and findings accumulate with no owner and no deadline.
4 Scanning runs on a schedule, a named owner acts on findings, and the agreed window has been met.		 The scanner output, the named owner, and the record of findings closed inside the agreed window.
4.4 Least privilege, and a leavers process that revokes tokens 0 Everyone has production access and there is no leavers process.
2 Access is roughly scoped, and the leavers process revokes logins only.
4 Access is least privilege, and a leaver has been offboarded with API tokens and application passwords revoked as well.		 The production access list by role, and a completed leaver record showing API tokens and application passwords revoked, not just the login.
4.5 Expiry monitoring on certificates and domains 0 Nothing monitors certificate or domain expiry.
2 Renewal is automatic for some of them, and nothing alerts when a renewal fails.
4 Every certificate and domain has an expiry alert that reaches a person, and it has been seen to fire.		 The expiry monitor configuration listing each certificate and domain, and one alert it has actually sent.
Your score for this area (5 checks at 4 points each) out of 20

5. Performance and capacity

  • The system has been load tested at a realistic multiple of expected peak, with realistic data volumes rather than an empty database.
  • Slow queries have been profiled against production-sized data. An index that is unnecessary at a thousand rows is mandatory at ten million.
  • Autoscaling limits and cost ceilings are set deliberately, so a traffic spike cannot become an unbounded invoice.
Check Score it 0, 2 or 4 Evidence a reviewer accepts
5.1 Load tested above expected peak with realistic data 0 The system has never been load tested.
2 A load test has been run against an empty or toy dataset, or only at expected peak.
4 A load test has been run at a realistic multiple of expected peak against production sized data, and the results are recorded.		 The load test report naming the multiple of peak, the data volume used, and the latency and error rate observed.
5.2 Slow queries profiled at production size 0 No query profiling has been done.
2 Queries have been profiled on a development sized dataset only.
4 The slow queries have been profiled against production sized data and the resulting indexes are in place.		 The slow query report against production sized data, with the query plans before and after the indexes were added.
5.3 Deliberate autoscaling limits and cost ceilings 0 There is no scaling ceiling and no spend alert.
2 Limits sit at the platform default and nobody chose them.
4 Scaling limits and a spend ceiling were chosen deliberately, and an alert fires before the ceiling is reached.		 The autoscaling configuration and the budget alert, with a note of who set the numbers and on what basis.
Your score for this area (3 checks at 4 points each) out of 12

6. Release and rollback

  • Deployment is automated and repeatable. If it only works when one particular person runs it, it is not ready.
  • Rollback has been performed successfully in a rehearsal, not just documented.
  • Feature flags allow risky changes to be disabled without a redeploy.
  • There is a defined change window and a named release owner for the first period after launch.
Check Score it 0, 2 or 4 Evidence a reviewer accepts
6.1 Automated, repeatable deployment 0 Deployment is manual and undocumented.
2 A deployment script exists and only one person has ever run it successfully.
4 Deployment is automated and has been run to completion by someone other than the person who wrote it.		 The pipeline definition and a deploy log showing a successful run by a second person.
6.2 Rollback performed in a rehearsal 0 There is no rollback path.
2 A rollback procedure is documented and has never been executed.
4 A rollback has been executed in a rehearsal and the system returned to a known good state.		 The dated rollback rehearsal record, with start and finish times and the state check afterwards.
6.3 Feature flags that disable risk without a redeploy 0 Every change ships permanently, and reversing one requires a redeploy.
2 Flags exist for some features, and switching one off has never been tried in production.
4 Risky changes ship behind a flag, and a flag has been switched off in production without a redeploy.		 The flag configuration, and a timestamped record of a flag being switched off in production.
6.4 A defined change window and a named release owner 0 Anyone deploys at any time and nobody owns the release.
2 A change window is discussed informally and no release owner is named.
4 The change window and the named release owner for the period after launch are written down and communicated.		 The release plan naming the change window, the release owner, and the period they cover.
Your score for this area (4 checks at 4 points each) out of 16

7. Ownership and operations

  • The service has a named owning team. “Everyone” means nobody.
  • An on-call rotation exists, with an escalation path and a runbook for the three most likely failures.
  • Third-party support contracts and their response times are known before you need them.
  • A post-incident review process exists and is blameless, so people report problems early.
Check Score it 0, 2 or 4 Evidence a reviewer accepts
7.1 A named owning team 0 No team owns the service.
2 One team is assumed to own it and that is recorded nowhere.
4 A named owning team is recorded in a service catalogue or equivalent, and that team agrees it owns the service.		 The service catalogue entry, or an equivalent record naming the owning team.
7.2 On-call rotation, escalation path and runbooks 0 There is no on-call rotation.
2 A rotation exists with no escalation path, or with no runbook behind it.
4 The rotation, the escalation path and runbooks for the three most likely failures all exist and have been used or rehearsed.		 The rotation schedule, the escalation path, and the three runbooks.
7.3 Third party support contracts and response times known 0 Nobody knows what support is bought or what response time it carries.
2 Contracts exist and their response times have not been read.
4 Every third party contract, its support tier and its response time are recorded where the on-call engineer can find them.		 The vendor list with the support tier, response time and contact route for each.
7.4 A blameless post-incident review process 0 Incidents are not reviewed.
2 Reviews happen occasionally and are informal, or they attribute fault to individuals.
4 A blameless review has been run on a real incident and the actions from it were closed.		 A completed post-incident review with its actions and their closure dates.
Your score for this area (4 checks at 4 points each) out of 16

How to read your score

Score each of the 29 checks from 0 to 4. A 0 means the control is absent. A 2 means it exists but nobody has proved it works. A 4 means it has been demonstrated and there is evidence a reviewer can open. A 1 sits between absent and unproven, and a 3 between unproven and demonstrated, for the cases where part of the estate is covered and part is not.

Every check counts the same. Four points per check means an area is worth four times its number of checks: 20, 16, 16, 20, 12, 16 and 16. The whole scorecard is 116. The weights are equal by design. We have not measured that a security check is worth more than an observability check, so we do not publish a weighting that says it is.

The scale maps onto the red, amber and green gate below, because both are defined by evidence. A 4 is green, since green means someone has demonstrated it. A 2 or a 3 is amber, since the thing exists but nobody has proved it, so it needs a named owner and a date. A 0 or a 1 is red.

There is no total that means ready to launch, and we do not publish one. We have not measured a threshold, so printing one would be a number with nothing behind it. A single red check blocks the release on its own, which means a high total carrying one zero is not a pass. The total is useful for one thing: comparing the same system against itself before and after the fixes, and seeing which of the seven areas is carrying the risk.

0. AbsentThe control is not there at all.
1. Barely thereBetween absent and unproven, where part of the estate is covered and part is not.
2. UnprovenIt exists, and nobody has proved it works.
3. Partly provenBetween unproven and demonstrated, on the same part-covered basis.
4. DemonstratedThere is evidence a reviewer can open.

Using this as a gate

Red blocks, amber needs an owner, green is demonstrated

Score each section red, amber or green with evidence attached. Red items block the launch. Amber items need a named owner and a date. Green means someone has demonstrated it, not that someone believes it. The value of the review is entirely in refusing to accept assertions without evidence.

A single red check blocks the release on its own, which means a high total carrying one zero is not a pass.

What you receive when the review is done

Whoever runs it, a finished review leaves the same record behind, because every item below is something this page already asks for.

  • A score on each of the 29 checks, from 0 to 4, with evidence a reviewer can open behind every 4.
  • A red, amber or green on each area, with the evidence attached.
  • The red items, each of which blocks the release on its own.
  • A named owner and a date against every amber item.
  • The total out of 116, for comparing the same system against itself before and after the fixes, and for seeing which of the seven areas is carrying the risk.

When the review is commissioned as a production readiness audit, you receive a prioritised findings report: what blocks launch, what needs an owner and a date, and what is acceptable risk to carry knowingly. Each finding carries the evidence behind it, so your team can act without re-deriving the problem.

Scoring it yourself versus an independent review

The scorecard is the same either way. What changes is who holds the pen, and the production readiness audit page is plain about why that matters: the people who built a system are the worst placed to audit it, and that is not a criticism of their skill.

Scoring it yourself

Start here

  • Every check, table and score on this page is published in full, so the team that owns the system can run it.
  • They know how it is supposed to work, so they read the code that way.
  • They have normalised the manual step in the deploy, the alert that always fires, the one server nobody reboots.

An independent review

When you want an independent view

  • Someone outside the delivery pressure looks at the system without the assumptions the team has been living inside for months.
  • It reads the code and the infrastructure definitions, interviews the engineers who built it, and exercises the system rather than taking documentation at face value.
  • Where a claim cannot be demonstrated, it is recorded as unverified rather than assumed to hold.
  • It is a separate engagement, and it needs your code, your infrastructure definitions and time with the engineers who built the system.

Questions to ask any team that says a system is ready

These are written to be asked of any team, ours included. Each one comes straight from a check above, and each has an answer you can verify: evidence a reviewer can open.

  • Has a restore been performed into a clean environment? An untested backup is not a backup.
  • Has rollback been performed successfully in a rehearsal, not just documented? A 2 means it exists but nobody has proved it works.
  • Where does each alert route? Every alert should reach a person who is awake and empowered to act.
  • What data was the load test run against? A load test on a toy dataset scores 2, not 4.
  • Can we see the secret scan across the full repository history, and the route list naming the server side guard on each route? Those are what count as evidence for a security item.
  • Who is the named owning team, and who is the release owner for the first period after launch? “Everyone” means nobody.
  • Which checks are amber, and what owner and date sits against each one? Amber items need both.
The answer that should worry you is a total with no gate. A single red check blocks the release on its own, so a high total carrying one zero is not a pass, and anything you cannot answer with evidence is an open risk, not a green tick.

Checklists. The AI-Built Prototype Security Checklist (25 Points) breaks area 4 of this page into the specific questions to ask about an AI built prototype: secrets and credentials, authentication and access control, data protection, input handling, dependencies and code, and operations and monitoring.

Method. From vibe coding to production is the seven step playbook for taking a vibe coded or AI built prototype to the standard this page measures.

Service. Production readiness audit, for when you want the scoring done by someone outside the delivery pressure.

Security testing. Area 4 of this page covers secrets, authorisation on every route, dependency scanning, production access and certificate expiry; it is not a vulnerability assessment checklist. The difference between a vulnerability assessment and a penetration test is set out on our penetration testing services page.

Frequently asked questions

What does the scorecard measure?

It measures evidence, not intent. There are 29 checks across 7 areas: reliability and failure behaviour, data and recovery, observability, security, performance and capacity, release and rollback, and ownership and operations. Each check scores 0 to 4 on whether the control exists and whether anyone has demonstrated it, for a maximum of 116.

How is load tested before launch?

Area 5 asks for a load test at a realistic multiple of expected peak, run against production sized data rather than an empty database, alongside slow query profiling at that same data volume and autoscaling limits and cost ceilings that somebody chose deliberately. A load test on a toy dataset scores 2, not 4.

What counts as evidence for a security item?

Something a reviewer can open. A secret scan across the full repository history. An enumerated route list naming the server side guard on each route. A dependency scan with the findings closed inside the agreed window. A completed leaver record showing API tokens and application passwords revoked, not just the login.

Is there a score that means we are ready to launch?

No, and we do not publish one. A single red check blocks the release on its own, so a high total carrying one zero is not a pass. The score shows where the evidence is missing. The red, amber and green gate decides the launch.

How does this relate to the seven step playbook for AI built apps?

The playbook for taking a vibe coded or AI built prototype to production is seven steps of work: code review and hardening, tests, architecture, infrastructure, monitoring, compliance and handover. This page is the other half, the seven areas you score once that work is claimed to be done.

If you would like a second pair of eyes on a system before it goes live, talk to our engineers. We run this review against your architecture, your data and your team’s actual operational reality, and hand you a prioritised list of what to fix first.

Contact us
Talk to an engineer

We’re happy to answer any questions you may have and help you determine which of our services best fit your needs.

Rates and delivery
  • Our rates are $150 to $250 per hour, depending on the seniority and mix of the team.
  • Where the scope is clear we quote a fixed price for a defined outcome, with a costed roadmap before any build starts.
  • We build with AI-assisted engineering, and we deliver projects up to five times faster than a traditional IT services company.
Call us at: +1 (703) 945-5300
What happens next?
1

We schedule a call at your convenience

2

We run a short, bounded discovery, scoped per engagement

3

We give you a costed roadmap before committing to a build

Request a Free Consultation
Book a 30-minute call →Prefer to talk first? Skip the form and grab a time directly.

We respond within one business day