Vendor-neutral checklist
Legal AI Vendor Evaluation: 12 Questions to Ask Before You Sign
Most legal AI evaluations stall on demos and feature lists. The questions that actually protect your firm are about where your data goes, what survives the contract, and who is accountable when something breaks. Ask these twelve of every vendor, in writing, and compare the answers side by side.
Data governance
1Where does our firm's data physically live while the system is running, and who can access it?
Listen for: a specific answer. Your servers, their cloud, or a third party's cloud. Vague answers here are the answer.
2Is our data, or anything derived from it, used to train or improve models that serve other customers?
Listen for: a flat no, in writing. "Anonymized" or "aggregated" carve-outs are where firm knowledge leaks.
3When the contract ends, what happens to our documents, the search index built from them, and every copy? How fast, and how do we verify?
Listen for: a deletion timeline with certification, including backups. "Within a reasonable period" is not a timeline.
4Which third parties touch our data: subprocessors, model providers, hosting vendors? Under what agreements?
Listen for: a complete current list, plus notice rights when it changes. Each name on the list is another breach surface.
Deployment model
5Can the system run entirely on infrastructure we control, whether our own servers or our private cloud? If not, what is the closest available option?
Listen for: whether on-premise is a real offering or a roadmap item.
6What access does the vendor need to install, update, and support the system? Standing access, or supervised and time-boxed?
Listen for: who holds admin credentials and whether remote access can be revoked by the firm.
7How does the system connect to our existing document management and email, and what happens to those connections and the index if we leave?
Listen for: whether the work product of indexing twenty years of iManage, NetDocuments, or Outlook files is portable or held hostage.
Privilege and confidentiality
8If client files and work product pass through the vendor's servers, what is the vendor's position on whether privilege and work-product protection survive? Will they stand behind that in writing?
Listen for: awareness of recent federal decisions on third-party AI vendors and privilege waiver. A vendor who hasn't thought about this is making you the test case.
9If the vendor is subpoenaed or compelled to produce data, what is their obligation to notify us and resist? Where is our data legally discoverable?
Listen for: a contractual notice-and-challenge commitment, not a "we comply with valid legal process" shrug.
Model behavior
10When the system gives an answer, does it cite the specific source documents so an attorney can verify? What does it do when it doesn't know: say so, or generate something anyway?
Listen for: document-level citations and an explicit "no answer found" behavior. This is the difference between a research tool and a liability.
11Is the system searching our own documents, public legal databases, or the open internet? Can we control which sources it uses for a given matter?
Listen for: clarity on what corpus actually gets searched. Many tools search published law and call it firm knowledge.
Commercial
12What does this cost over three years: per-seat fees, usage fees, implementation, and renewal increases? At the end, do we own anything, or have we been renting?
Listen for: total cost of ownership, not the year-one quote. Per-seat pricing compounds as the firm grows.