Data privacy in AI systems: the questions to settle before you build
AI features move personal data through more systems than the architecture diagram usually admits. Mapping that flow early is far cheaper than discovering it during a data protection review.
By Quality AboveAll · · 9 min read
- Map every place personal data flows, including prompts, retrieval indexes, logs and provider infrastructure.
- Read the actual provider terms on training and retention. Defaults differ between consumer and enterprise tiers.
- Deletion is harder than it looks: data may persist in embeddings, caches and logs after the source record is removed.
Map the flow honestly
Personal data in an AI feature typically appears in more places than expected: the prompt sent to a provider, the retrieval index, application logs, monitoring traces, evaluation datasets and any cached responses. Each is a location subject to your obligations.
Logs are the most commonly overlooked. Teams log full prompts for debugging and inadvertently create an unmanaged store of personal data with different retention than the system of record, which is exactly the kind of finding that turns a routine audit into a project.
What provider terms actually say
Terms differ substantially between consumer and business tiers of the same product, and defaults are not always what teams assume. Confirm in writing whether inputs are used for training, how long they are retained, where processing happens geographically, and what subprocessors are involved.
For anything covered by GDPR or similar regimes, you also need the right contractual instruments in place before data flows, not after. This is procurement work that should start in parallel with development rather than at the end.
The tier you signed up with in an afternoon and the tier your legal team assumed you were on are frequently not the same tier.
Minimise before you send
The most robust control is not sending the data at all. Redact identifiers that the task does not require, aggregate where possible, and pass internal references rather than names where the model does not need to know who someone is.
Where the task genuinely requires personal data, consider whether processing can happen within your own boundary using a self-hosted model, which is often the deciding factor for regulated clients. Options are covered in small language models versus LLMs.
Deletion, and why embeddings complicate it
When a person exercises a deletion right, the source record is the easy part. The same content may exist as chunks in a retrieval index, as vectors, in caches, in evaluation sets and in logs, and each needs a deletion path designed in advance.
Maintain a mapping from source records to every derived artefact so deletion is executable rather than archaeological. Retrofitting this after launch is one of the more painful pieces of remediation work we get asked to do.
Transparency and defensibility
Tell users when they are interacting with AI, what data it uses and what it is used for. Beyond being a legal requirement in many contexts, it is what makes the feature defensible when someone asks, and it materially affects how users calibrate trust.
Keep records: what data flows where, what assessments were done, what decisions were taken and why. Governance frameworks like the NIST AI RMF provide structure, and the practical version is covered in AI model governance.
Frequently asked questions
Can we send personal data to a hosted model provider?
It depends on your legal basis, the provider's terms and your contractual instruments. It is a resolvable question, but it needs answering before the architecture is settled, not after.
How do we handle deletion requests for embedded content?
Maintain a mapping from source records to derived artefacts, including chunks, vectors, caches and logs, so each can be removed. Design this before launch.
Does on-premise deployment solve compliance?
It removes the third-party transfer question, which is often the hardest one. Your own obligations around retention, access control and transparency still apply.
Handling personal or regulated data in an AI feature? A free 30-minute consultation will map the data flows and the decisions that need making first.
