Steno General Counsel Carly Savar joined Jared Correia on his Adventures in Legal Tech podcast for an episode titled "Inside In-House: What It's Really Like to Be In-House Counsel for a Legal Tech Vendor" to talk through what it takes to vet an AI vendor. She's negotiated AI and data agreements from both sides of the table, as counsel to a legal tech vendor and as a buyer evaluating other vendors herself.

For firms weighing whether their next AI vendor can be trusted with the record, her answer is consistent from start to finish: don't take the feature list's word for it, take the contract's.

Make sure that they're not going to train on your data. That would probably be the biggest thing that law firms should think about. You've got your transcripts, you've got confidential data, you've got your attorney-client privileged documents, and you need to make sure that your vendors are keeping that secure. Don't assume that they're not gonna try to train on your data.

That scrutiny can't stop at the vendor itself. Whatever large language model the vendor is built on has to meet the same bar, since the security promise only holds if it holds all the way downstream.

Carly described a contract where opting out was the only thing standing between her client's data and the training pipeline:

We had a vendor that said, 'We're gonna train on your data unless you opt out.' And that was very scary. Obviously, we opted out, and they're not going to train on the data. But had I not read that agreement, or had somebody not shared that with me, the default was train. And that's scary.

Not every kind of training deserves the same suspicion, she's careful to add. Aggregated, anonymized usage data, the kind that tracks button clicks rather than case content, doesn't concern her. She's watching for anything that touches client-specific material. On certifications, she treats SOC 2 Type 1 as a courtesy mention at best, since it only proves a vendor's controls looked good on one particular day. Type 2 is the one that means something, because it tracks those controls over a full year.

The conversation closed on the billing shift that AI is forcing. Several states now require attorneys to bill for the time they actually spent, not the time a tool saved them, which Carly sees as the right outcome for clients even as it complicates old billing habits.

Listen to the full episode on Adventures in Legal Tech.