An AI Reliability Checklist for Law Firms (One Page — Print It)

I wrote a longer piece about what happens when a frontier model gets recalled and your firm’s tools go dark. This is the short version — the one-pager you can print, pin to a wall, and actually use. No theory, just the checks.

Before you adopt an AI tool

• Where does our data go? On-device, your cloud, or the vendor’s? Is testimony leaving the building, and does it need to?
• What model is under the hood, and can it be swapped? If the vendor’s model is recalled or deprecated, does the tool keep working?
• Is the method portable, or model-specific? Prompts and processes should survive a model change. (This is why the vendor-neutral approach behind CasePrompts matters.)
• Does it cite the record? Page:line on every claim, “not in the record” when it can’t. No exceptions for convenience.

For everything already in your workflow

• Human in the loop on anything accountable. A professional reviews before it leaves the firm. Always.
• A tested fallback exists. A second model — even a smaller or local one — you’ve actually verified works.
• You can run the critical path by hand. Slower is acceptable. Impossible is not.
• Continuity plan written down. If your primary AI vanishes tomorrow, the next steps are on paper, not in someone’s head.

A worked review example

This miniature record and the proposed output are synthetic, written for this example. They contain no client material and are not a model benchmark.

Demo transcript, 12:3 — Q: When did the package arrive?

12:4 — A: I saw the delivery email on Tuesday. I don’t know when the package arrived.

Proposed summary: “The package arrived on Tuesday (12:4).”

Review decision: reject. The citation exists, but the answer dates an email observation, not the delivery. A citation alone does not establish that it supports the claim.

Checked revision: “The witness saw the delivery email on Tuesday; they did not know the package’s arrival date (12:4).” Both parts follow the supplied answer, and the unknown remains explicit.

Save the rejected sentence, source location, reason, corrected wording, and reviewer with the review record. If the AI service is unavailable, the same comparison can be performed manually. This demonstrates one record-checking step; it does not prove a firm’s fallback or continuity plan has been tested.

The one-line version

Treat AI like any other dependency a professional relies on: document the process, test the fallback, and keep a human responsible for the work.

Open the printable AI Reliability Guide or read the longer continuity note.