Four separate regimes reach AI in a healthcare setting, and they ask different questions. HIPAA asks where the patient data went. FDA oversight asks whether the software is a device. State AI statutes ask about disclosure, oversight and consequential decisions. The EU AI Act asks whether the system is high-risk and whether you can prove it is managed.
None of them asks "is your AI good." All four ask what you can show. That distinction is where most healthcare AI programmes are caught short.
HIPAA: it follows the data, not the technology
HIPAA has nothing to say about AI as a category. It applies to protected health information, which means it reaches an AI system precisely as far as PHI travels into it — and that is usually further than the architecture diagram suggests.
Five places it lands that teams routinely miss:
- Prompts. A clinician pasting a note into an assistant has moved PHI, whatever the tool is called.
- Retrieved context. A retrieval layer pulling chart excerpts into a model's context window is disclosing PHI to whatever sits at the other end of that call.
- Logs. Prompt and completion logging is on by default in a great many platforms, and log retention is frequently longer than anyone building the feature assumed.
- Evaluation datasets. Real notes copied into a test set are still PHI, and test sets are the least governed data in most organisations.
- Vendor-side retention and training. What the contract permits is often not what the sales conversation implied — and the contract is what governs.
The practical work is a data-flow map for each AI system showing every point PHI enters, rests or leaves, and a business associate agreement covering each party that touches it. We go through the pipeline in detail in where health data rules actually touch your model pipeline.
FDA: the device question is about claims and reviewability
Whether a piece of clinical software is a regulated device turns on what it claims to do and how far the clinician can independently check its reasoning. Software that supports a clinician who can review the basis of a recommendation sits differently from software that directs a diagnosis or a treatment decision.
Two things are worth stating plainly. First, this boundary is genuinely contested at the edges, it has moved, and it is a question for regulatory counsel reading the FDA's current guidance — not for an article, including this one. Second, and more practically: the device question is not the whole risk question. A tool that sits outside device regulation can still cause harm, still processes PHI, still influences care, and is still squarely in scope of the other three regimes on this page. "Not a device" is a regulatory classification, not an assurance.
Where it does bite hardest in practice is change control. Models that are updated — retrained, re-prompted, or silently upgraded behind a vendor's API — raise a version-management question that most clinical governance processes were not built for.
State AI statutes: the strictest applicable rule tends to win
A growing number of US states have enacted AI legislation, and they do not agree with one another. The two most often raised with us are Texas's Responsible AI Governance Act and Colorado's AI legislation; several others impose disclosure duties on AI used in a healthcare context.
For a multi-state provider the practical consequence is structural rather than legal: it is usually cheaper to operate to the strictest applicable standard than to maintain per-state behaviour in a clinical system. It also means that where a system is deployed matters as much as where the organisation is headquartered — which is why our healthcare intake asks specifically about Texas deployment rather than about Texas incorporation.
Effective dates and scope in this area have moved more than once. Check the statute, not a summary.
The EU AI Act: it can reach a US provider
The Act applies to AI whose output is used in the EU, whoever operates it. A US health system with EU patients, an EU research collaboration or an EU-facing product can be in scope, and health-related AI falls within its high-risk provisions in several defined cases.
What high-risk classification brings is documentation, risk management, data governance, logging, human oversight and accuracy obligations — the machinery, not a threshold score. Its obligations phase in on a published timetable that has itself been subject to amendment proposals, so verify the current schedule against the primary text. Our comparison of the two regimes is in high-risk AI in healthcare.
The two places providers are most often caught short
Human oversight that is real
Every regime on this page leans on meaningful human review, and every clinical AI deployment claims it. The claim is tested by volume and by authority.
A clinician reviewing forty AI-drafted notes an hour at the end of a shift is not providing meaningful oversight of any of them, whatever the policy says — and that is a workflow problem, not a clinician problem. The questions worth asking of your own deployment: what is the review volume per clinician per hour, what is the median time spent per item, what information is in front of them when they decide, and can you name a case where a clinician overruled the system and what happened to it. If nobody has ever overruled it, either the model is perfect or the review is nominal.
Evaluation on the wrong population
A model validated on a population that does not resemble yours has been validated for somebody else. Age distribution, comorbidity profile, payer mix, language, documentation conventions and the local dialect of clinical shorthand all move performance, and vendor benchmarks rarely disclose enough to tell whether they resemble your patients.
The defensible position is a local evaluation on your own population, with the date, the sample and the subgroup breakdown recorded — and a re-evaluation schedule, because your population changes. A single accuracy number carries almost none of this; what to ask instead is a separate read, as is what a defensible fairness evaluation measures.
What to be able to show, per system
Six answers cover most of what all four regimes ask:
- What the system does, and what clinical decision it influences.
- Where PHI enters, rests and leaves — including logs, evaluation data and vendor retention.
- What evaluation was performed, on which population, on what date, with what subgroup results.
- How a clinician can see the basis of an output and disagree with it, and the volume they carry.
- What happens when it is wrong: detection, escalation, patient-safety route, and the incident record.
- What the vendor contract actually allocates to whom.
An organisation that can produce those six for each clinical AI system is in a strong position under every regime on this page. One that can produce none of them has a governance gap that no further reading closes.
How we approach the domain — what we look at, and against which frameworks — is set out on the healthcare page, and the standard our own work is measured against is published in full as the verification methodology.
Frequently asked questions
- Does HIPAA apply to AI?
- HIPAA applies to protected health information, not to AI as a category — so it reaches an AI system exactly to the extent that PHI moves through it. In practice that means prompts, retrieved context, logs, evaluation datasets and vendor-side retention are all in scope, and those are the places health systems most often lose track of it.
- Is our clinical AI a regulated medical device?
- It depends on what it claims to do and how much the clinician can independently check it. Software that supports a clinician who can review the basis of the recommendation is treated differently from software that directs a diagnosis or a treatment. This boundary is genuinely contested at the edges and is a question for regulatory counsel against the FDA's current guidance, not for a summary.
- Do state AI laws apply to healthcare providers?
- Several do, and they do not agree with each other. Texas's Responsible AI Governance Act and Colorado's AI legislation are the two most often raised with us. The practical consequence for a multi-state provider is that the strictest applicable rule tends to set the operating standard, and that where a system is deployed matters as much as where the organisation is based.
- Does the EU AI Act apply to a US health system?
- It can. The Act reaches AI whose output is used in the EU, regardless of where the operator sits — so a US provider with EU patients, EU research collaborations or an EU-facing product can be in scope. Health-related AI falls into its high-risk provisions in several defined cases.
- What do we need to be able to show?
- For each clinical AI system: what it does, what decision it influences, where PHI goes, what evaluation was done and on which population, how a clinician can disagree with it, and what happens when it is wrong. That set answers most of what all four regimes ask.
