At a recent doctor's appointment, I was asked to consent to an artificial intelligence assistant listening to my consultation. The assistant was Heidi, one of a growing number of AI medical scribes. It listens to the conversation, produces a transcript and turns it into structured clinical notes for the doctor to review, which, on the face of it, is exactly the sort of work we should give to an AI. Doctors spend time typing notes, referral letters and reports. A machine that handles the paperwork while the doctor concentrates on the patient offers an obvious human benefit: less administration, better attention and, potentially, better care.
I politely declined. He said he'd used Heidi for more than a year, understood the technology and had previously worked with blockchains. More importantly, he said he wouldn't conduct the consultation without it. That rather changed the nature of the consent being requested.
In the past week, Digital Rights Watch launched a campaign calling for patients to have a legally protected right to refuse AI scribes without losing access to healthcare. The group says people are already being refused care for precisely that reason, something to which I can personally attest.
My concern wasn't that an AI had entered the consulting room. I use AI constantly. It was that I was being asked to approve a system whose operation the person introducing it couldn't clearly explain. Where would the conversation be processed? What would be retained? Which other companies provided the infrastructure? Who owned the service, and who had funded it? I'd followed Heidi’s funding trail through global venture capital firms and institutional investors, including the Australian superannuation fund HESTA.
My doctor had little to say about that. Instead, he assured me that nothing was recorded, which was a technical impossibility.
An AI scribe can't transcribe a sound it hasn't captured. The conversation must enter a microphone, become digital data and be processed before a transcript or clinical note can exist. The real distinction is not between “recorded” and “not recorded,” it's between processed and retained.
Tom Sulston, Head of Policy at Digital Rights Watch, suggested that “retained” was probably the word my doctor was reaching for.
“The audio does have to be recorded in order to be transmitted to the AI scribe,” Sulston told me. “Most services then delete the audio after completing the transcription.”
That sounds like a sensible privacy safeguard. But Sulston raised an uncomfortable trade-off: if the original recording is destroyed, how can anyone later verify that the transcript is correct? AI systems can mishear words, omit details and even invent information. Deleting the audio may protect the patient from one risk while exposing them to another.
The more immediate problem was whether I’d really been offered a choice. When I asked Sulston whether consent could be meaningful if refusing the AI affected access to the consultation, his answer was direct. “No, in short. Consent gathered under duress, such as threatening withdrawal of treatment, does not constitute valid consent.” Meaningful consent requires more than asking a question and receiving a yes. People need enough information to understand what they are agreeing to, and they must be able to decline without unreasonable consequences. Privacy guidance generally treats consent as meaningful only when it’s informed and voluntary.
This can be challenging in a consulting room. A patient may be ill, worried or in pain. Few are going to pause the appointment to investigate hosting locations, retention policies, third-party processors and pages of privacy legalese … except patients like me of course.
They’ll get a much simpler explanation: “It just writes my notes.”
This isn't merely a healthcare problem. It reflects a wider bargain running through modern technology.
As we previously reported, patients are broadly open to medical AI when it assists clinicians rather than replaces them, but they place a high value on transparency and explainability.
Convenience removes friction, but it can also remove understanding. Streaming gives us instant access while weakening ownership. Smart devices eliminate manual effort while making ordinary objects dependent on distant servers. AI produces answers, summaries and decisions without necessarily revealing where the information went or how the result was reached. Technology becomes most interesting when it stops being a gadget and becomes part of daily life. At that point, the important question is no longer simply whether it works, it's on what have we become dependent?
My own response has been a gradual move toward self-hosting. It began because I wanted my music back. Streaming had made almost everything available, while turning a collection I once owned into access granted by subscription. Rebuilding it on local storage eventually led to hosting my documents, home automation and AI.
The services I run at home aren't curiosities I occasionally tinker with; they have become part of the plumbing of daily life. A password manager like Vaultwarden can hold the keys to everything. Document archives like Paperless-ngx can contain everything from AI-tagged medical letters, invoices and records. Home Assistant runs the lights, sensors, routines and small acts of automation that make the house feel responsive rather than merely connected. My personal media libraries in Plex, Jellyfin, Audiobookshelf and Immich remain available because I hold the files, not because a subscription remains active. And local AI lets me work with private material on a machine whose location, network access and storage I can inspect.
That's why self-hosting feels less like owning gadgets and more like reclaiming agency. These systems save time, reduce friction and make the house work better, but they also remove a collection of opaque dependencies from services I now consider essential.
That doesn't make local technology automatically safer.
Sulston pointed out that a clinic running its own AI would inherit responsibility for patching, backups and security. A specialist cloud provider may do those jobs far better than a small medical practice. He suggested a possible middle ground: shared “community AI” infrastructure operated for groups of clinics.
That wider perspective matters, because sovereignty isn't achieved simply by dragging every service into the spare room. Nor is cloud computing inherently careless or exploitative. The human benefit is choice backed by understanding.
Heidi may handle medical information responsibly. It may save doctors time and improve the experience of consultations. The problem in that room wasn’t that an AI was listening. It was that convenience had moved ahead of comprehension.
In an AI-first world, the most important question may no longer be whether a machine can help us. It may be whether we understand whose machine is listening, whose machine is thinking about us, and what happens when we say no.
