Episode 19 — Who's responsible when the AI gets it wrong
S01:E19

Episode 19 — Who's responsible when the AI gets it wrong

•

Episode description

Iza is standing over the medicine drawer one evening, photographing a box with her phone. And halfway through she catches herself on a thought worth having about every app with artificial intelligence inside it: how do I actually know that whatever comes up on that screen is true?

This episode isn’t about whether AI makes mistakes. It does. It’s about what happens AFTERWARDS — who gets the last word, where responsibility sits, and how to spot an app that tells you honestly what it doesn’t know.

The most important sentence in the episode is built into the app itself: mojApteczka is not a medical device within the meaning of Regulation MDR 2017/745. That isn’t legal cover pasted into the terms and conditions — it’s a design choice with consequences, and those consequences are what we talk about.

What we cover:

  • Three steps in a scan, and the first one matters least. A model looks at the photo and recognises the shapes of letters. It’s the medicines register that says what the medicine actually is. And the third step is a person: the result lands on a screen to check and correct, and until you tap save, there’s nothing in your cabinet.
  • The app can’t create a medicine that isn’t in the register. If the model invents something, there’s nowhere to put it. It sounds dull. It’s the foundation the whole thing sits on.
  • What the interaction database knows, and what it can’t. More than three hundred thousand documented interactions; over one point three million entries once each pair is recorded in both directions. It describes a possible interaction between two substances and how severe it might be. It doesn’t know your age, what you’re being treated for, or whether your GP decided quite deliberately that the combination is fine in your case. It’s a list of questions for the pharmacist, not a verdict.
  • 61% of products carry low confidence in the data. Product characteristics are written very cautiously — quite rightly. We talk about what to do honestly with information you aren’t sure of yourself, and why those markings are muted rather than turned up.
  • Why the warning got shorter. The same block of text on every one of eight medicine cards stops being read. One sentence stays visible; the full source list sits one tap away behind an icon.
  • Safety markings are OFF by default. We say plainly why — and admit we don’t know whether that’s the final answer.
  • Prescription scanning works differently from scanning a box. All the reading happens on the phone. The photo never goes anywhere. Only the medicine name goes to the lookup — exactly as if you’d typed it in yourself.

Where the data comes from. Next to the safety information, the sources are named: the Summary of Product Characteristics, LactMed (NIH), OpenFDA and the European Medicines Agency. For interactions, a specific scientific database with its year of publication: DDInter 2.0 (Nucleic Acids Research, 2025). “Our algorithm detected” is a sentence that means nothing.

What the app does NOT do: it doesn’t diagnose, it doesn’t treat, it doesn’t judge whether a medicine is right for you, it doesn’t set doses and it doesn’t tell you to take or skip anything. That verdict belongs to your GP and your pharmacist. The app’s job is to make sure you know what you’ve got at home and what to ask about.

One line to take away: artificial intelligence can suggest, but it can’t decide. Don’t ask whether an app has AI in it. Ask what happens when that AI gets it wrong.

Links:

⸻

About AI in this episode. The dialogue was written by Tomasz Szuster. The voices you hear are AI-generated synthetic voices — not recordings of the hosts. Full details are in the show description and in the episode outro. We say so plainly, because we think it should be the standard for any podcast made this way.

0:00Cold open: how do I know it's true?
1:12Step one: a model reads the photo
1:41Why the model doesn't know anything
2:23The third step is you
3:30But who is responsible, legally?
3:52"Not a medical device" — the European MDR
5:14Four sources, named
5:56Interactions: what the database can't know
7:4161% of products carry low confidence
8:33Why the warning got shorter
9:35Safety markings are off by default
10:17Prescription scanning stays on the phone
11:11AI can suggest, it can't decide
12:32Outro, CTA and AI disclosure