ShipReview Book a pilot

01

Hold point

hold point, n. A step in an inspection and test plan that work may not pass without a signature.

A bug is not a bug until it fails a test.

ShipReview audits a whole Next.js app built with coding agents, before your agency hands it to the client. Three models from three vendors read the code separately, and we run every claim they make as a test in a sandbox. You get back only what failed for real, each with its failing test and a fix that passes it, a plain list of what we did not check, and a verdict: ship or hold.

  • Next.js and TypeScript
  • One Postgres database
  • Up to 200,000 source tokens
  • Price fixed before work starts
Finding F-001 Severity high

Security / tenant isolation

A member of one organisation can export another organisation’s invoices.

Label
Reproduced
Location
app/api/invoices/export/route.ts:18-41
Frozen test, claim line
claim(res.status === 403, 'CLAIM: org A cannot export org B invoices')
Result
original 200 with 14 rows of another tenant patched 403 pass
Regression
9 existing tests still pass
Verdict
Hold until the diff is applied and rechecked
From our own test app, not a customer.

02

What comes back

Every line in the report points to a record.

The report is written for two readers. Page one is for the person who signs the handover and may be forwarded to the client. The rest is for the developer who fixes the code. Below is a shortened copy with our notes beside it.

  1. A · Page one

    Hold

    One reproduced high-severity finding is open.

    audit a_31d0e4 · snapshot sha256:4be1…c07a · package pilot

    AVerdict. HOLD while any reproduced finding of high or critical severity is open. Otherwise SHIP, which means exactly this: no reproduced high-severity defect is open within the checked scope.

  2. B · Findings by label

    Findings by label
    IdSeverityLabelFinding
    F-001highreproducedInvoice export ignores the caller’s organisation
    F-002mediumcode_supportedSession cookie set without Secure in production
    F-003mediumunverifiedTwo invoices can receive the same number under concurrent writes
    F-004lowimprovement_proposalInvoice list runs one query per customer
    F-005highrejectedSQL injection in search. The query is parameterised.

    BFindings by label. Sorted by label, then severity. Each finding lists which models raised it and any model that disagreed. Rejected claims stay in the list so you can see what was ruled out.

  3. C · Frozen test, F-001

    // __shipreview__/repro/F-001.test.ts  sha256:9c4e…1a7f  frozen
    const { orgA, orgB } = await seedTwoTenants()
    precondition(orgB.invoices.length === 14,
      'PRECONDITION: org B has 14 invoices')
    const res = await exportInvoices({ as: orgA.member, org: orgB.id })
    claim(res.status === 403,
      'CLAIM: org A cannot export org B invoices')

    original: fails as claimed · 200, 14 rows of org B

    CThe frozen test. Written in your app’s own test style. Setup is marked PRECONDITION; the one assertion that states the defect is marked CLAIM. Once it fails on your code for that reason, we record its hash and it can no longer change.

  4. D · Patch, F-001

    --- a/app/api/invoices/export/route.ts
    +++ b/app/api/invoices/export/route.ts
    @@ -24,1 +24,2 @@ export async function GET(req, { params })
    -  const rows = await listInvoices(params.orgId)
    +  if (params.orgId !== session.orgId) return forbidden()
    +  const rows = await listInvoices(session.orgId)
    Patch checks
    Frozen test, patched copypass · 403
    Existing testspass · 9 of 9, no new failure
    Type checkpass

    Passed specified checks

    DThe patch and its checks. A diff for you to apply. The repair step may not touch the frozen test, your tests, test config, package scripts or the lockfile. “Passed specified checks” means these three lines. It does not mean the fix is the best one.

  5. E · Coverage

    Coverage by model vendor
    VendorFiles sentStatus
    Anthropic64 of 64completed
    OpenAI64 of 64completed
    Google64 of 64completed
    • blockedlint: the app has no lint script
    • not checkedpayment webhook: it calls an outside service and the sandbox has no network
    • excluded2 secret files dropped before upload

    ECoverage. Which files each model was actually sent, which checks ran, which were blocked and why. A file we listed but did not send to a model is shown as not reviewed.

  6. F · Cost

    Cost by stage
    StageModelsSandbox
    Deterministic checks$0.00$0.31
    Independent reviews$11.40$0.00
    Reproduce and repair$4.85$0.62

    FCost. What the audit spent on models and sandbox time, per stage, from our ledger. The price you pay was fixed by the quote and does not move with it.

Shortened. Field names are the report’s own; F-001 matches the record above, the other values are illustrative.

03

Evidence labels

The label is the state field of each finding in findings.json.

Five labels. A test earns the first one.

Agreement between models never upgrades a finding. Only a failing test does.

Evidence labels
LabelWhat it takes to earn itWhat you should do with it
Reproducedreproduced A frozen test failed on your code for the reason claimed, and a model from another vendor approved what the test expects. Fix it before handover. If a patch passed its checks, start there.
Code supportedcode_supported The cited code was read and is consistent with the claim. It was not run. Read it and decide whether it matters for this client. If it does, ask for a reproduction or write the test.
Unverifiedunverified Plausible, but context was missing or the check did not finish. Treat it as a question for the developer, not as a defect.
Improvement proposalimprovement_proposal No defect is claimed, or a performance gain has not been measured. Optional. Schedule it or leave it.
Rejectedrejected We tested or inspected the claim and it did not hold. Nothing. It is listed so you can see what was ruled out.

04

How an audit runs

Stages 2 and 3 run at the same time. Neither sees the other’s output.

The audit follows an inspection and test plan.

Six stages, in order. Reproduction and the patch check are hold points: the audit does not move past them until the record in the fourth column exists.

Inspection and test plan for one audit
No.ActivityAcceptance criterionRecord producedH / W / R
1 Snapshot and inventory Unpack the upload, drop secret files, list routes and the files that touch the database or authentication. Next.js and TypeScript, one package.json, a lockfile, Postgres as the only service, 200,000 source tokens or fewer. Otherwise no quote. Snapshot id, file manifest, quote R
2 Deterministic checks Install, prepare the database, type check, run your tests, lint if there is a lint script, scan for secrets and known vulnerable dependencies. Each check ends passed, failed, blocked or skipped, with a reason. Check results; your test results become the regression baseline W
3 Independent reviews Models from Anthropic, OpenAI and Google each read the same snapshot. None sees the others’ work or the check results. Every observation cites a file and lines that exist in the snapshot. The rest are rejected and counted. Observations; per model, the files it was sent W
4 Consolidation Group observations by root cause. Nothing is dropped by vote. Up to five high-severity findings raised by one model go to the others for a second reading. Every observation ends up in exactly one finding. No label above code supported. Findings with their observations, support and dissent R
5a Reproduce A model writes one focused test per finding. A model from another vendor checks what the test expects. The test runs on your original code. Only the CLAIM assertion fails, for the reason claimed. Then the test is frozen by its hash. Frozen test and sha256, original result, environment digest, sanitized log H
5b Patch check A model proposes a diff. A diff that touches the frozen test, your tests, test config, package scripts or the lockfile is refused. Frozen test passes on a patched copy; your tests and type check show no new failure. Diff and digest, check results, patch state H
6 Report Verdict, findings, coverage and cost, from the records above. HOLD while a reproduced high or critical finding is open. report.html, report.md, findings.json, developer bundle R
H
Hold point. The audit does not pass this step without the record.
W
Witness point. Logged as it happens; you can follow it in the event log.
R
Review point. The record is checked and goes into the report.

Limits of the standard audit

Source
up to 200,000 tokens
Sandbox time
30 minutes
Reproductions
5 findings
Patch
1, at most 2 iterations
Recheck
1, within 14 days

The token count is estimated from file sizes and checked before you pay. When an app needs more than this, the quote says so first.

05

Two doors

Exit codes: 0 when no reproduced high-severity finding is open, 1 when one is, 2 for a partial or failed audit, 3 for a usage or connection error.

Two doors, one job.

For the person who signs the handover

Upload a zip or choose a repository, tell us in a few lines what the app must never do, and confirm the quote. The report opens with the verdict and what was not checked, in words a client can follow. The developer bundle carries the frozen tests and the diffs for whoever fixes the code.

Both doors start the same job, with the same limits and the same price.

What you receive
report.htmlOne self-contained file you can forward to the client
report.mdThe same report as text
findings.jsonEvery finding as structured data
bundle/Frozen tests, diffs, sanitized logs and a brief for a coding agent

For the developer and the coding agent

$ shipreview quote ./app
packed    212 files, dropped .env and .env.local
eligible  Next.js 15, TypeScript, Postgres
price     $99 pilot         quote q_7f2c91, valid 30 min
$ shipreview start --quote q_7f2c91
audit     a_31d0e4          queued
$ shipreview status a_31d0e4 --watch
1 snapshot and inventory   done
2 deterministic checks     done
3 independent reviews      done, 3 of 3 models
4 consolidation            done
5 reproduce and repair     done, 1 reproduced
6 report                   done
$ shipreview findings a_31d0e4
F-001  high    reproduced      invoice export
F-002  medium  code_supported  session cookie
…
$ echo $?
1
$ shipreview patch a_31d0e4:F-001 > F-001.diff
patch     passed_specified_checks, not applied
$ git apply F-001.diff
$ shipreview recheck a_31d0e4 ./app
F-001     fixed_verified
verdict   ship: no reproduced high-severity defect
          open within the checked scope
$ echo $?
0
Shortened transcript. Output may change during the pilot.
The same six operations as MCP tools
ToolReturnsSpends credit
audit_quoteEligibility, scope, price, quote idNo
audit_startJob id, at onceYes
audit_statusStage and coverage progressNo
audit_findingsFindings with labelsNo
finding_patchDiff and check resultsWithin the allowance
audit_recheckUpdated finding statesFirst one included
  1. No start without a quote. An agent cannot start a job until a quote with the price is on record and unexpired.

  2. Keys have spend caps. Read and spend are separate scopes. A person sets each key’s daily and monthly cap.

  3. Results are data. Anything that comes from your repository arrives in a field named untrusted, so a comment in the audited code cannot steer the agent that reads it.

  4. We return diffs. We never apply them. The developer or the agent does, then asks for a recheck.

06

Exclusions

What it will not tell you.

  1. 6.1

    Stack. The first version covers Next.js with TypeScript, npm, one service and one disposable Postgres database. For anything else the quote says no, and says why, before you pay.

  2. 6.2

    Certification. An audit buys a stated amount of checking. It does not certify the app. A SHIP verdict is not a statement that the app has no bugs.

  3. 6.3

    Tested conditions. A passing test shows behaviour under the conditions of that test: synthetic data, a fresh database, no network, the versions in your lockfile. Production can differ.

  4. 6.4

    Defects nobody raised. We test what a model or a check raised. A defect that none of them noticed is not in the report and is not covered by the verdict.

  5. 6.5

    Performance. We do not load-test. A performance remark without a measurement comes back as an improvement proposal.

  6. 6.6

    The pilot. During the pilot one of us helps where the automation is not ready yet: setting up apps that do not start on their own, and reading each report before it goes out. The report names every step where a person helped.

Ship

No reproduced high-severity defect open within the checked scope.

The full wording of a SHIP verdict. It means this and nothing more.

07

Rate card

Prices, quoted before any work starts.

Rate card, USD before tax
OfferPriceIncludesStatus
Pilot audit $99 One audit of one app, with a person assisting where the report says so. Five places. Available now
Standard audit $129 The limits above, the report and the developer bundle, one recheck within 14 days. Planned
Changed-files recheck $19or five for $95 Frozen tests and your tests rerun on a new snapshot, when no more than 10% of the source changed. Planned
Agency plan $449a month Four standard audits a month. Planned

USD, before tax. Planned prices may change after the pilot. If we cannot run an app we quoted as eligible, we refund or credit the audit. A clean result is not a refund case: you pay for the checking, not for findings.

08

Your code

Retention periods count from the day the report is delivered.

Who reads your code, where it runs, when it is deleted.

Who reads it

Three model vendors receive the files that the coverage section of your report lists for each of them. Each keeps API data under its own terms. We link those terms rather than summarise them, because they change.

Model vendors and their terms
AnthropicHow long do you store my organization’s data?
OpenAIData controls in the OpenAI platform
GoogleGemini API Additional Terms of Service

Before upload, the command-line packer drops .env and key files, honours .gitignore and shows you the file list.

Where it runs

In a fresh isolated sandbox created for your audit and destroyed after it. The sandbox has network access while dependencies install and none after that. Model keys never enter it; every model call goes through our gateway outside the sandbox.

When it is deleted
Deletion schedule
RecordDeleted
Source snapshot14 days after delivery
Sanitized reports and evidence30 days after delivery
Billing recordskept separately

Fourteen days is the recheck window. A recheck after that needs a new upload.

What we do not claim

We do not claim zero retention. The vendor terms above govern what each vendor receives, and we cannot shorten them. We do not publish your code or use it for training.

09

Questions

Questions, answered.

Is this three models voting?
No. Each model reads the code alone and none sees what the others wrote. We group their observations by root cause and keep every one, including a finding only one model raised. Agreement does not raise a label. A finding becomes reproduced only when a test fails on your code.
We already use CodeRabbit.
Keep it. It reviews pull requests continuously, and it documents its own repository scans and sandbox checks. We have not benchmarked against it and do not claim to beat it. This is a different purchase: one fixed-price gate on a whole app before handover, with a report written for your client.
What if you find nothing?
Then the report says so and lists what was checked and what was not. The verdict is SHIP in its narrow sense: no reproduced high-severity defect open within the checked scope. The price is the same.
Can my agent run this unattended?
Yes, within the limits you set. Give it a key with the spend scope and a cap. It asks for a quote, starts the job, polls the status, applies the diffs itself and asks for a recheck. The command exits 1 while a reproduced high-severity finding is open, so a CI job can stop on it.
What stacks?
Next.js with TypeScript, npm with a package-lock.json, one package.json at the root, Postgres as the only service, and a script that prepares the database. The quote checks all of this before you pay, and an app that does not fit costs nothing.
Is a person involved?
During the pilot, yes, and the report says at which steps. The standard audit is built to run without one. When a person does help, for example to get an app to start, the report will say so in the same way.
How long does an audit take?
We do not quote a time yet. Sandbox work is capped at 30 minutes; model reviews and, during the pilot, the assisted steps add to that. We will publish times once we have measured them.

10

Pilot

Book a pilot audit.

Five places at $99 each, for agencies with a Next.js app near handover. Write to us and we reply with a quote and what we need from you.

Email pilot@shipreview.tech

Put in the email

  1. The Next.js version
  2. Roughly how many source files the app has
  3. The handover date
  4. One thing the app must never do, for example show one client’s data to another

ShipReview is a working name. The service is in a paid pilot with five places. There are no customer logos on this page because there are no customers yet.