Medical dictation with local AI: doctor's letters from your templates, without the cloud

Medical dictation with local AI, step by step

Medical dictation with local AI lets a doctor dictate findings, assessment and plan after the consultation, while a speech recognition model on a workstation in the practice writes the text and a language model turns it into a doctor's letter in the practice's own template. The doctor reads the draft, corrects it and signs it. For a practice, one workstation with an NVIDIA RTX PRO 4500 Blackwell card runs both models, and audio, transcript and letter stay on hardware the practice owns.

This guide is for practice owners, practice managers and the medical directors of medical care centres and clinic departments who dictate letters today and want the recordings to stay in the building. It covers dictation after the visit, with one speaker and a letter template. Recording the whole consultation is described in our guide to an on-premise AI scribe for medical practices, nursing reports in our guide to voice documentation in care homes, and dictation in offices and law firms in local AI dictation as a Dragon alternative. The legal parts are general information on EU law as of October 2026, not legal advice.

Three terms are used throughout. A speech recognition model turns speech into text; Whisper from OpenAI and Parakeet from NVIDIA are the open ones used here. A language model is a second AI that writes text, here a structured letter from the raw dictation. A GPU, or graphics card, is the part of the workstation that runs both models quickly.

From the dictation to the signed doctor's letter

Steps as described in this article; models and tools from the Whisper repository, the Parakeet model card and the OpenWhispr pages, read 30 October 2026.

The text reaches the practice software in one of two ways. For short entries, a dictation app on the PC types straight into the open field. OpenWhispr, an open app under the MIT licence for Windows, macOS and Linux, does this and can send the audio to the practice workstation: its models page lists "Any OpenAI-compatible API" as a transcription provider and names "self-hosted Whisper servers". For letters, the doctor dictates the whole content in one go, for example after a block of appointments; the recording goes to the workstation, which returns transcript and letter draft together. How the draft then enters the patient record depends on the practice software. Some products import documents, others take text only by paste, so ask your vendor which way it offers.

How accurate speech recognition is for German medical dictation

On general German speech the open models are close to each other. The Open ASR Leaderboard paper (arXiv 2510.06961v3) measured a word error rate (WER), the share of words a model gets wrong, of 4.97 per cent for Whisper large-v3 and 4.90 per cent for Parakeet TDT 0.6B v3 on public German test sets, on an NVIDIA A100. That is about one word in twenty; our comparison of German speech recognition, accuracy and dialects has the details.

Medical German is harder. A preprint from Hochschule Pforzheim, written with a software company (arXiv 2601.19945, January 2026), tested 29 models on four simulated German doctor-patient conversations, among them a patient with a strong regional dialect and a doctor with a non-native accent. The best system, a commercial cloud service, reached 3.0 per cent WER, and Mistral's two Voxtral models 7.1 and 8.5 per cent. For Whisper large-v3 the paper gives 12.6 per cent, taken from a table row labelled as a distilled version without naming the checkpoint, so that figure needs care. The authors name acoustically difficult surroundings and medical terminology as the main causes of errors, and report that Whisper large-v3's error rate rises markedly with complex accents. Four conversations are a small sample.

Dictation is the friendlier case: one speaker, close to the microphone, in a quiet room. A 2024 master's thesis at the University of Vienna built such a set, 11.5 hours of German surgical dictation, single sentences from anonymised patient records dictated by two speakers of Austrian Standard German with a dictation microphone. It kept numbers and abbreviations in the reference text, because "subtle distinctions, such as precise dosage amounts or specific medical shorthand, may substantially modify the meaning", and reports that fine-tuning reduced "the Word Error Rate (WER) for the Whisper-medium model by 14.52%".

We found no published figure for open models on German doctor's letters dictated in a practice. Take 30 of your own dictations with the drug names, abbreviations and numbers of your specialty, and count the errors that would change the meaning. Whisper's model card adds that its predictions "may include texts that are not actually spoken in the audio input (i.e. hallucination)", one more reason why the person who signs reads every letter.

Spelling of terms can be steered. OpenAI's guide to its speech service says "Use a prompt to improve recognition of names, acronyms, formatting, or recording-specific vocabulary" and suggests "a dictionary of the correct spellings"; for its hosted Whisper model the prompt is limited to 224 tokens. vLLM's transcription interface accepts the same optional prompt, so a short list of the drug names and terms your practice uses most can go with every dictation. Fine-tuning on your own dictations is the larger step, after a pilot.

Letter templates in the practice's own format

The language model does what a typist does with a template. It receives the transcript and an instruction that describes the letter: addressee and patient fields, the sections the practice uses, such as diagnoses, history, findings, assessment, therapy and recommended procedure, the standard phrases and the closing. Each letter type gets its own template, for example a reply to a referring colleague, a findings report, a report for an insurer or a discharge letter in a clinic department.

The instruction also sets the rules that keep the letter faithful to the dictation:

• use only what was dictated, and mark a section as empty when nothing was said about it;
• keep drug names, doses, dates and laboratory values exactly as transcribed;
• mark words the transcript shows as unclear instead of guessing them;
• add no diagnosis, code or recommendation that the doctor did not dictate.

gpt-oss-20b, OpenAI's open model under the Apache 2.0 licence, has 13.76 GB of weight files, 12.8 GiB, and its configuration allows 131,072 tokens of context, the units in which models count text, far more than a letter with its dictation and template needs. Whether its German letters are good enough shows in a test with the same 30 dictations; the next step up is gpt-oss-120b on a 96 GB card. A language model can change meaning when it rephrases, so the doctor checks the letter against the dictation before signing.

You can order the workstation alone, the setup of Whisper, the dictation app and the letter model alone, or both together: the workstation delivered with everything installed and configured; write to us and we suggest the best solution for your tasks and budget.

Health data, the GDPR and the practice's own duties

A dictated letter is data concerning health, which the GDPR defines in Article 4(15) as "personal data related to the physical or mental health of a natural person". Article 9(1) prohibits processing such data unless an exception applies; point (h) of Article 9(2) covers medical diagnosis and the provision of care, under the conditions of Article 9(3), which ties it to professionals subject to professional secrecy. In a dictation only the doctor speaks, after the visit. Recording the consultation itself, with the patient's agreement and an impact assessment, is set out in our AI scribe guide. Your data-protection officer confirms which basis applies to your dictation workflow.

When a cloud service recognises the speech, it processes the recordings on the practice's behalf, and under Article 28(1) "the controller shall use only processors" able to show appropriate technical and organisational measures; contract, subprocessors and location of processing then belong in your review, as our guide to GDPR, cloud hosting and subprocessors explains. With local recognition no outside provider processes the audio for that step. The practice's own duties stay: Article 32(1) asks for security "appropriate to the risk", including "the pseudonymisation and encryption of personal data". For a dictation workstation that means an encrypted disk and connection, named administrator accounts, a lockable room and audio deleted once the letter is signed. Rules on medical confidentiality are set nationally and by the medical chambers; your data-protection officer and your chamber confirm what applies.

Dictation software under the Medical Device Regulation

The Medical Device Regulation, Regulation (EU) 2017/745, decides by the intended purpose that the manufacturer states. The guidance MDCG 2019-11 rev.1, endorsed in June 2025 by the MDCG, the body of Member State representatives set up under Article 103 of the MDR, has two passages that fit dictation. Its section 3.1 says that software "only intended for non-medical purposes", such as "invoicing, staff planning, e-mailing, web or voice messaging, data parsing, word processing, and back-up", does not qualify as medical device software. Its Annex I counts speech-to-text systems among the communication systems used in healthcare and describes such systems as normally based on general-purpose software, outside the medical device definition unless they carry a medical purpose. The guidance has no example on dictation or letter writing.

Transcribing what the doctor said and placing it in the letter template is documentation. A letter model that adds a diagnosis, proposes a therapy or flags abnormal results moves towards a medical purpose, and the stricter tests of the same guidance and of Rule 11 come into play, as our AI scribe guide and our page on a private LLM for hospitals describe. Keep the instruction to what was dictated, and ask your regulatory adviser to confirm that your set-up stays on the documentation side. Under Article 2(12) of the MDR the intended purpose also follows from what the supplier says in its labelling and promotional material, so we describe and set up the letter model for documentation only: it fills your template from the dictation and adds no diagnosis or therapy suggestion.

Hardware for a practice and for a clinic

A dictation needs little GPU time. The closest published figure is the A100 measurement in the Open ASR Leaderboard paper, where Whisper large-v3 transcribed 145.5 seconds of audio per second of computing, so one minute of dictation took about 0.4 seconds by our arithmetic. RTX PRO cards have no such published figure. The size of the workstation follows from the letter model, the letter length and how many doctors finish a dictation at the same moment.

GPU memory from NVIDIA's datasheets; Whisper memory from OpenAI's repository (6 GB = 5.6 GiB, 10 GB = 9.3 GiB); sessions by our users-per-card rule (90 per cent of driver-visible memory, less 3 GiB per card, less the weights): 25.7 GiB usable on the RTX PRO 4500 and 83.0 GiB on a 96 GB card; gpt-oss-20b 12.8 GiB with 0.1875 GiB per 8K session, gpt-oss-120b 60.8 GiB with 1.125 GiB per 32K session (16-bit cache). Our estimates, not measurements.

Dictation only. One NVIDIA RTX PRO 4000 Blackwell with 24 GB and 145 W is enough for Whisper large-v3 as the speech service of a practice, by our estimate. It cannot hold the letter model as well: by our rule a 24 GB card leaves 18.5 GiB, and Whisper turbo with gpt-oss-20b would take 18.4 of them, with no room for a letter draft. Our page to buy an RTX PRO 4000 workstation lists its data.

Dictation and letters. The NVIDIA RTX PRO 4500 Blackwell with 32 GB and 200 W is the workstation we recommend for a practice. Of the 25.7 GiB our rule leaves, gpt-oss-20b takes 12.8 and Whisper turbo about 5.6; the remaining 7.3 GiB holds 38 letter drafts of 8,192 tokens at the same moment. It runs from an office socket and belongs in a lockable room; our page to buy an RTX PRO 4500 workstation gives the configuration.

Larger model. A medical care centre with several specialties and longer reports can use one NVIDIA RTX PRO 6000 Blackwell with 96 GB, as the 600 W Workstation Edition or the 300 W Max-Q. It carries gpt-oss-120b next to Whisper large-v3, with 11 sessions of 32,768 tokens by our rule. With the Workstation Edition the machine draws about 1.1 kW at the wall by our estimate, within an ordinary 230 V circuit.

Clinic. For several departments the service moves to a server room. A 2U server with four NVIDIA RTX PRO 4500 Blackwell Server Edition cards draws about 1.4 kW by our estimate, as our guide to a four-card RTX PRO 4500 Server Edition server works out; the entry server on our AI servers page, with two RTX PRO 6000 Server Edition cards, about 1.8 kW. Our AI scribe guide sizes such servers by number of doctors.

Tell us how many doctors dictate, in which specialties and with which practice software, and we suggest a configuration for your practice or clinic.

Notes for the IT team or the practice's IT provider

1. Speech endpoint. vLLM serves /v1/audio/transcriptions, "compatible with OpenAI's Transcriptions API", after pip install vllm[audio]; uploads are limited to 25 MB by default through VLLM_MAX_AUDIO_CLIP_FILESIZE_MB, and the optional prompt carries the vocabulary list (vLLM documentation of 19 October 2026).
2. Clients. OpenWhispr is pointed at the endpoint as a custom OpenAI-compatible provider; its README states "No data collection, no telemetry". Check that no cloud provider, including OpenWhispr Cloud, is configured on any PC and that voice fingerprinting stays off, as our article on telemetry in self-hosted AI tools describes.
3. Access. vLLM's --api-key protects only the paths under /v1, /v2, /inference and /cohere; /metrics, /health and /invocations answer without the key, and /invocations gives the same inference as /v1 (vLLM security guide, read 31 October 2026). Keep the services behind an LLM gateway with keys and a query log, with TLS between PCs and workstation.
4. Blackwell builds. RTX PRO Blackwell cards have compute capability 12.0, and container images must be built for it; Parakeet's model card lists Blackwell among the supported architectures.
5. Retention. Delete the audio once the letter is signed, unless your impact assessment sets another period, and restrict access to gateway logs, which contain health data. Our guides to a DPIA for an internal LLM and to an AI server without an IT department cover the rest.

FAQ