Large language models in medicine: opportunities and risks

LLMs can shorten clinical documentation, but their output still needs review. Discover opportunities, deployment risks and how to validate them safely.

1 Sept 2026 9 min read

Large language models in medicine: opportunities and risks

A 2024 JAMA study revealed that 72% of family physicians currently use generative AI for clinical tasks. Large language models (LLMs) have moved from research labs to exam rooms in less than 18 months—transforming how doctors document, triage, and synthesize evidence.

The promise is clear: LLMs can draft SOAP notes, summarize complex histories, and answer clinical questions in seconds. But the deployment comes with risks: hallucinations, bias, and regulatory uncertainty. This article examines what LLMs can realistically achieve in healthcare, where they fail, and what safeguards clinicians should demand before adoption.

You'll find evidence-based performance metrics, use cases already in practice, and the six questions every health system must answer before deploying an LLM in clinical workflows.

What are large language models and why they matter in healthcare.

Large language models are neural networks trained on billions of text tokens, including medical literature, EHR notes, drug databases, and clinical guidelines. Unlike rule-based systems, LLMs learn probabilistic patterns in language, enabling them to generate coherent text, answer questions, and complete partial sentences.

The breakthrough came in 2017 with the transformer architecture. GPT-4, Claude, and Med-PaLM 2 now achieve passing scores on USMLE exams. More relevant to front-line clinicians: LLMs can draft structured clinical notes from unstructured conversation transcripts in under 30 seconds.

Why this matters: physicians spend a significant portion of their workday on documentation. AI clinical documentation systems powered by LLMs promise to reclaim that time—if they're deployed responsibly.

Current applications: where LLMs are already working

Ambient clinical documentation

This is the highest-volume use case. An LLM listens to the patient encounter, transcribes speech, and generates a structured note—typically SOAP format. DAX Copilot (Nuance/Microsoft) and Abridge are live in hundreds of US health systems.

A study in Nature Medicine (2024) showed a 52% reduction in after-hours documentation among physicians using ambient scribes. The LLM doesn't replace clinical judgment; it structures the raw conversation so the doctor edits rather than types from scratch.

Differential diagnosis support

Med-PaLM 2, trained on medical corpora, scored 86.5% on MedQA (USMLE-style questions). In pilot deployments, it presents differential diagnoses with supporting evidence. But it doesn't prescribe—the clinician validates every suggestion.

Patient message triage and response drafting

LLMs can classify patient portal messages by urgency and draft replies. A Mayo Clinic pilot found that 78% of drafted responses required only minor edits. This reduces the message backlog, but human review remains mandatory—liability stays with the physician.

Literature synthesis

Summarizing 30 PubMed abstracts into a clinical summary used to take an hour. LLMs do it in 90 seconds. Tools like Elicit and Consensus are used by academic physicians for systematic review preparation, though peer review is non-negotiable.

Performance benchmarks: how accurate are medical LLMs today

General-purpose LLMs (GPT-4, Claude 3) achieve 75–82% accuracy on medical Q&A benchmarks. Specialized models like Med-PaLM 2 push accuracy to 86–90%. But accuracy on exams doesn't equate to safety in practice.

LLMs like GPT-4 can produce clinically unsafe recommendations when tested on real EHR cases—confusing drug dosages, missing contraindications, or suggesting obsolete treatments. When prompted with ambiguous symptoms, they can hallucinate diagnoses.

In other words: LLMs are powerful assistants, not autonomous clinicians. They work best when the task is structured (e.g., transcription → SOAP) and the human validates every output.

Natural language processing techniques like named entity recognition and sentiment analysis underpin these systems—but they remain probabilistic, not deterministic.

The five implementation risks every health system must mitigate

1. Hallucination (fabricated data)

LLMs sometimes generate plausible-sounding but entirely false information—citing nonexistent studies, inventing lab values, or recommending discontinued drugs. This happens in 2–8% of outputs, depending on the prompt.

2. Bias and health disparities

LLMs trained on biased datasets reproduce those biases. A 2023 Stanford study found that dermatology LLMs underperformed on non-Caucasian skin conditions by 14 percentage points because training images were predominantly of white patients.

3. Privacy and GDPR compliance

LLMs hosted by third-party vendors may log inputs for model training. If you paste a patient transcript into ChatGPT, you will likely have violated HIPAA and GDPR.

4. Medicolegal liability

If an LLM drafts a note that omits a red-flag symptom and harm results, who is liable? Courts haven't settled this. Most jurisdictions hold the signing physician responsible, not the software vendor.

5. Over-reliance and deskilling

Trainees using LLMs may skip learning to structure notes independently. A JAMA Medical Education commentary (2024) warned of "diagnostic atrophy" if residents defer pattern recognition to AI.

What to ask vendors before deploying an LLM in your clinic

1. What is the model's validation cohort? Demand peer-reviewed accuracy data on real clinical notes, not synthetic test sets.

2. Where is my data stored and who owns it? Reject vendors who train their models on your patient data without explicit consent.

3. Can I customize the output template? A rigid SOAP structure may not fit pediatrics, psychiatry, or aesthetic medicine. Tools like MedicMic offer per-specialty templates.

4. What happens when the LLM is unsure? Does it flag low-confidence outputs, or does it guess?

5. Who has reviewed this for clinical safety? Look for independent clinical validation—ideally published—not just vendor white papers.

6. What is your incident response protocol? If the LLM hallucinates a dangerous dosage, how fast does the vendor patch the model and notify customers?

A glossary of AI terms can help you decode vendor marketing claims.

LLMs vs. traditional NLP: what's different

Traditional medical NLP relied on rule-based parsers and ontologies (SNOMED, RxNorm). An engineer coded explicit rules: "If sentence contains 'chest pain' + 'radiating' + 'left arm,' tag as angina."

LLMs learn patterns probabilistically. You don't code rules—you train the model on millions of examples. This makes LLMs vastly more flexible (they handle synonyms, slang, and ambiguous phrasing) but also less predictable.

For structured tasks—extracting medications from a discharge summary—rule-based NLP is still more reliable. For generative tasks—drafting a patient-facing explanation—LLMs excel.

Most clinical AI systems now use hybrid architectures: rule-based extraction for safety-critical data, LLMs for narrative generation.

The physician's role in the LLM era: editor, not typist

LLMs won't replace clinicians. But they will redefine what "documentation" means. The physician's job shifts from transcription to validation: reviewing the AI-generated note, correcting errors, and adding clinical nuance the model missed.

This is faster than typing from scratch and can offer significant time savings—but it's not zero-effort. You still need 2–3 minutes per note to verify accuracy.

The risk is that time pressure pushes some doctors to approve notes without reading them. That's negligence, not innovation. Health systems must build in protected time for AI review.

Regulatory landscape: FDA, CE Mark, and clinical AI

As of mid-2024, the FDA classifies most LLM scribes as "administrative tools," not medical devices—so they don't require premarket approval. But if an LLM interprets clinical data or suggests diagnoses, it may fall under SaMD (Software as a Medical Device) regulation.

The EU's AI Act (effective 2025) classifies clinical decision support as "high-risk AI," requiring conformity assessment. GDPR Article 22 restricts fully automated decisions with legal effect—so an LLM can't auto-approve a prescription without human oversight.

Translation for clinicians: even if a tool is legally compliant, you remain liable for its output. Regulatory approval is a floor, not a ceiling.

Preguntas frecuentes

No. Current LLMs achieve 75–90% accuracy on medical Q&A benchmarks, but they hallucinate 2–8% of the time and lack real-world clinical judgment. They assist—they don't diagnose. The clinician remains responsible for every decision.

Yes, and courts treat them like any other EHR entry. If the note is inaccurate and harm results, the signing physician is liable. Always review and edit LLM output before finalizing.

Only if deployed carelessly. Cloud-based LLMs that log inputs (e.g., ChatGPT) violate HIPAA and GDPR. Use BAA-covered, on-premise, or privacy-first vendors that delete recordings post-processing in line with data protection principles.

Request validation data stratified by age, sex, race, and language. If the vendor can't provide it, the system hasn't been audited for bias. Independent clinical validation is essential.

MedicMic supports per-specialty templates. If a vendor offers only rigid SOAP output, it is not suitable for psychiatry, pediatrics, or aesthetic medicine.

You catch it during review. If a hallucination reaches the EHR, it's a documentation error attributable to the clinician. This is why human-in-the-loop review is legally and clinically mandatory.

Possibly, if overused during training. Residency programs should require manual note-writing before allowing LLM assistance, to prevent "diagnostic atrophy" and ensure competency.