What is a membership inference attack?

An attack that tries to tell whether a particular person's record was used to train a model. It works because models tend to be more confident about records they have seen. Against the normal model in our tests it was right 72% of the time, where guessing scores about 50%.

Does differential privacy make a model less accurate?

It can. More noise gives stronger privacy and, past a point, lower accuracy: accuracy on new patients fell from 83% at ε ≈ 8 to 75% at ε ≈ 2. With light noise the private model was more accurate than the normal one, 83% against 79%, because the normal model had memorised its training records.

Was any real patient data used?

No. Every record is synthetic: 3,000 made-up patients with 50 measurements and a yes or no diagnosis each. No personal information was used.

Yes, on request. The code is not published on the site. Email contact@sovereignsystemslabs.com and tell us what you want to build, and we will share it with you directly, with the licence terms agreed first.

Differential Privacy vs Membership Inference (PDF)

Why it matters

AI models can memorise the data they learn from. Researchers have pulled names, phone numbers and email addresses out of GPT-2, and made ChatGPT reveal its training text by asking it to repeat one word forever. In IBM's 2026 Cost of a Data Breach study, about one in five breached organisations had an AI model or application breached.

The slides set out four ways to attack an AI model, then test a defence with a mathematical guarantee against the first of them:

Membership inference. Was this person's record in the training data? This is the focus of the experiment.
Model inversion. Rebuilding private data, such as a face, from what a model has learned.
Data poisoning. Planting a hidden backdoor in the training data.
Adversarial examples. Small changes that fool a model, from stickers on a stop sign to prompt injection, which is first on the OWASP Top 10 for LLM Applications 2026.

How differential privacy works

Differential privacy is a mathematical promise: a model trained with your record gives almost the same answers as one trained without it, so nothing it reveals can single you out. It is already in use. The US Census Bureau applied it to the 2020 census, Apple and Google use it to learn from typing on phones, and in September 2025 Google released VaultGemma, a 1-billion-parameter open language model trained from scratch with differential privacy. The costs are real: private models are about five years behind normal ones, they need more computing, and someone has to decide how much privacy is enough.

The standard training method, DP-SGD, follows three rules at every step:

Clip. Cap how far any one record can pull the model, so no one counts for more than the rest.
Add noise. Add random noise to the combined update, so one record is lost in the crowd, like one voice in a busy café.
Count the cost. Track the privacy budget, ε (epsilon). The smaller ε is, the stronger the privacy.

Close-up of computer memory modules — Memory modules

The experiment

We wrote a neural network and DP-SGD from scratch in NumPy and trained them on 3,000 synthetic patient records, each with 50 measurements and a yes or no diagnosis. A normal model and three private models each learned from the same 1,000 records. The attacker was then given 2,000 records, half of them from the training set, and had to say which ones the model had seen. Its rule was simple: if the model is very sure about a record, call it a member; if not, call it a non-member. This is the loss-threshold attack of Yeom and colleagues (2018).

Each setting ran five times. The privacy budget was measured with Google's dp-accounting library (its RDP accountant) at δ = 10⁻⁵.

The results

Model	Attack success (lower is safer)	Accuracy on new patients
Normal model	72%	79%
Private, ε ≈ 8 (light noise)	53%	83%
Private, ε ≈ 4 (more noise)	52%	80%
Private, ε ≈ 2 (most noise)	53%	75%

Figures are the mean of five runs. Pure guessing scores about 50 to 52%. The normal model was 100% accurate on its own training records but only 79% on new ones: it had memorised its patients, and that is what the attack exploited. Every private model brought the attack down to about the level of guessing. Privacy works like a dial: more noise means safer data and, past a point, lower accuracy.

How often the attack guessed rightMembership inference, lower is safer

Normal model72%
Private, ε ≈ 853%
Private, ε ≈ 452%
Private, ε ≈ 253%
Pure guessingabout 50%

Mean of five runs on 3,000 synthetic patient records. Guessing scores about 50 to 52%.

Accuracy on new patientsHigher is better

Normal model79%
Private, ε ≈ 883%
Private, ε ≈ 480%
Private, ε ≈ 275%

Mean of five runs. More noise means stronger privacy and, past a point, lower accuracy.

What we learned

Memorising is not learning. The leaky model was also worse on new patients, 79% against 83% for the lightly private one.
A guess is not a promise. Training for less time also beat this simple attack, but only differential privacy carries a proof that holds against any attack.
Privacy needs data. A first try with 400 training examples gave ε = 36, which is almost no privacy.
Two practical fixes. A one-step mathematical shortcut made training fast enough, and a lower learning rate steadied it.

Who needs it

Hospitals, banks, phone makers and AI assistant providers: anyone who trains models on personal information. Rules are tightening too. From 10 December 2026, Australian privacy policies must explain automated decisions that use personal information, and the EU AI Act's rules for high-risk AI start to apply in December 2027 and August 2028. IBM's 2026 study puts the average cost of one data breach at US$4.99 million.

The handle and combination dial of an old safe door — Safe door and dial

Is it fair, and is it honest?

Fairness. Noise hurts small groups most, such as patients with rare diseases.
Privacy washing. "We use differential privacy" means little with ε = 36, which protects almost nothing.
Who decides ε? Engineers choose it, not the people whose data is used.
Private is not the same as good. A private model can still be biased, or used for surveillance.

The slides call for three things: published ε values, independent testing and laws with teeth.

Who it is for

Technical leaders, privacy and risk teams, and researchers deciding how to train AI on sensitive data, and what to ask a supplier who says their model is private.

How we can help

We test AI systems for data leakage, prompt injection and excessive permissions through AI security testing, build private AI assistants that keep your data under your control, and help you set up AI governance that stands up to scrutiny. See our research work, or read our guide to prompt injection.

Can AI keep a secret?