Episode 19 — Who's responsible when the AI gets it wrong

Last night I was standing over the medicine drawer, taking a photo of a box with my phone,

and halfway through I caught myself thinking,

how do I actually know that whatever comes up on that screen is true?

Hello, I'm Tomasz Szuster. I built the mojApteczka app.

And Iza's question there is probably the most honest question you can put to anyone

building something with artificial intelligence inside it.

Because I see it from the other side, you know.

Everyone's saying, our app has got AI in it now, as though that were an argument all by itself.

It is.

It very much isn't.

If anything, it's a fresh reason not to trust the thing.

When a person gets it wrong, I know who to be cross with.

When a model gets it wrong, then what?

That's exactly what today's about.

Not, does AI make mistakes, because it does.

It's what happens afterwards.

Who gets the last word?

Right.

Let's start at the beginning, then.

I photograph a box.

What happens?

The photo goes to a model that can look at an image and read it.

It pulls out the name, the dose, the expiry date and the barcode.

That's the bit that looks like magic.

And that's where I'd stop if I were in marketing.

Most people do stop there.

But it's only the first of three steps.

And honestly, it's the least important one.

Why the least important?

Because the model doesn't know anything.

It recognises the shapes of letters in a photograph.

So the second step is that I take that barcode and look it up in a separate medicines database.

And from there, I pull the manufacturer, the form it comes in, the ATC group and the links to the leaflets.

So it isn't the model telling me what the medicine is?

No.

The model says, this box probably has such and such written on it.

The register says what the medicine actually is.

Those are two completely different things.

And that distinction is the foundation the whole app sits on.

All right.

And the third step?

The third step is you.

Me?

Nothing saves itself.

The result lands on a screen where everything's already filled in.

But everything can be changed.

And until you tap save, there's nothing in your cabinet at all.

So the app never tells me I've added a medicine for you?

Never.

The result goes to a screen for you to check and correct.

The decision sits with you.

And there's one more thing that sounds dull and is, to my mind, the most important part of the whole scan.

The app can't create a medicine that isn't in the register.

Hang on.

So if the model invents something…

There's nowhere to put it.

Without a match in the register, the app won't add it as a medicine.

Right.

That does sound honest, to be fair.

By the way, everything we're discussing here are actual features in the mojApteczka app.

Not a pitch.

Just context.

Now the question that genuinely nags at me.

Because all of that is engineering.

I'm asking about responsibility.

Legally.

If the app shows me something that's wrong, then what?

That's the question I had to answer before I wrote a single line.

And the answer is in the app itself, in plain text.

Meaning?

When you open the drug interaction screen, there's a sentence at the bottom.

The mojApteczka app is not a medical device within the meaning of the European Medical Devices Regulation, MDR 2017 745.

And is that good news or bad news?

It's a choice.

A deliberate one.

In practice, it works like this.

If a product claims it diagnoses or treats, it falls into an entirely different regulatory regime.

A notification.

A notified body.

Oversight.

And it makes no odds whether it's a piece of hardware or an app.

So, in plain terms, you could have gone down the road of,

the app will tell you whether this medicine is right for you.

I could have tried.

And I didn't.

Because then I'd be promising something I can't honestly guarantee for every medicine,

every person, every condition.

The verdict belongs to the GP and the pharmacist.

I can be your memory.

I can't be your oracle.

You know what's odd about that?

It sounds like admitting a weakness.

It does.

And it's the opposite.

It's terribly easy to fall into the other trap,

showing confidence you haven't actually got,

because it looks better.

I'd rather show you the boundary and then stand behind it.

All right.

But if you're not the oracle,

how am I meant to know what to trust in what you're showing me?

Because I tell you where I got it.

Next to the safety information for a medicine,

the source is spelled out.

The summary of product characteristics from the medicines regulator.

LactMed from the American National Institutes of Health.

Open FDA.

And the European Medicines Agency.

Four sources.

Named.

Named.

And for interactions,

there's a specific scientific database credited,

with its year of publication.

Because our algorithm detected is a sentence that means nothing at all.

This came from here.

Go and check it yourself.

That means something.

Hold on, though.

There's something I don't follow.

You show possible interactions between medicines.

That already sounds like something a pharmacist does.

And that's a very fine line.

So let me say precisely what happens.

I take the medicines in your cabinet.

I make every possible pair out of them.

And I check each pair against a scientific database.

How big a database?

More than 300,000 documented interactions.

And since a pair has to be recorded in both directions,

that comes to over 1.3 million entries.

And all of that sits on the phone?

No, that sits on the server side and gets queried.

But the more important bit is what isn't in there.

You aren't in there.

What do you mean, I'm not in there?

That database describes a possible interaction between substance A and substance B,

and how severe it might be.

It doesn't know your age, what you're being treated for,

what's already been taken off your list,

or whether your GP decided quite deliberately

that in your case, the combination is fine.

So the app could frighten me over nothing?

It could, which is why it's sorted by severity,

labelled with where it came from,

and signed off with a line saying,

this isn't medical advice.

It's meant to be a list of questions for your pharmacist,

not a verdict.

Right.

So I go into the chemists and say,

the app flagged these two.

Would you have a look?

That's precisely the scenario I built it for.

Not, the app said I can't.

Just, the app drew my attention to something.

I'd like the opinion of someone who actually knows.

And when the data's thin?

Because we don't know the same amount about every medicine.

Good question, and I've got an uncomfortable answer.

For roughly 61% of products,

the confidence in the data is low.

61%?

That's more than half.

More than half.

It comes from the fact that the product characteristics are written very cautiously.

Quite rightly, too.

But for me, it's a design problem.

What do you do with information you aren't sure of yourself?

And what do you do?

I mute it.

Where confidence is low, the icon is grey, with no colour fill.

Only the better documented ones get full colour.

Mute it?

That sounds like hiding it.

It looks similar, and it's the reverse.

If I show you a strong warning where the data's thin,

I'm training you to ignore warnings.

And then you'll miss the better documented one,

the sort that's genuinely worth taking to the pharmacist.

Ah.

We had exactly that problem with the notice saying the information is for guidance only.

It was on every medicine card, the whole thing, with the full list of sources.

On a list of eight medicines, that's the same block of text eight times over.

So people stop reading it.

Precisely.

So I left one sentence visible, for guidance only, not medical advice,

and tucked the full source list behind an icon.

It's always there, one tap away, but it isn't drowning itself out.

That's interesting, because that isn't a legal decision.

That's a decision about making the thing actually work.

And that's the whole difference.

You can tick the compliance box.

You can paste in 300 pages of terms and be technically in the right.

I'm after something else.

That the person standing over the drawer at 10 at night knows what this app does not know.

Hmm.

And I'll tell you one more thing most app makers wouldn't say out loud.

Those safety markings are switched off by default.

You have to turn them on in settings yourself.

Hang on.

Why?

That sounds like something that ought to be on from the start.

Because it's information that means something when somebody reaches for it deliberately.

If I push it at everyone, on every medicine card, we're back where we started.

Noise nobody reads.

Noise.

I'd rather a person decided once that they want it.

So, fewer warnings, but one somebody actually reads.

That's the trade.

And honestly, I don't know whether that's the final answer.

But I know why it's that way for now.

Tell me one more thing.

Because scanning a prescription is a different matter entirely.

There's data on there I wouldn't hand to anybody.

Which is why that feature works differently from scanning a box.

The whole reading of the prescription happens on your phone.

The photo of the prescription doesn't go anywhere.

Anywhere meaning where, exactly?

Anywhere.

Not to my server.

Not to any model in the cloud.

It stays on the device.

Only the medicine name goes to the look-up.

Exactly as if you'd typed it into the search yourself.

So the numbers, the patient details, all of that...

stays with you.

And again, nothing adds itself.

You confirm each item from the prescription separately.

Do you know what?

I've a feeling we've been saying one sentence all episode,

just in different ways.

Which is?

That artificial intelligence can suggest, but it can't decide.

Yes.

And if I were to leave listeners one thing from this episode,

that'd be it.

Don't ask whether an app has got AI in it.

Ask what happens when that AI gets it wrong.

Whether there's a screen where you can correct it.

Whether you can see where the data came from.

Whether somebody has written down plainly what the tool does not do.

And if none of those are there?

Then you've got less to go on than you need to judge how far to trust it.

And that's your decision.

You ought to be making it with your eyes open.

Right.

I'll add mine.

From a completely non-technical angle.

Leon is nine, and he already understands that something appearing on a screen doesn't make it true.

Basher is five, and she doesn't understand that yet.

And honestly, half the grown-ups don't either.

Which is why that sentence has to be in the app, not in the terms and conditions.

And it's why, when I'm unsure about a medicine, I still take the box down to the chemists.

And so you should.

The app's job is to make sure you know what you've got at home and what to ask about.

The answer belongs to the pharmacist.

If you'd like to give it a go, you'll find the link in the episode description.

The app is free, and your medicine cabinet is there offline too.

And if you enjoyed this episode, share it with someone whose medicine cabinet at home is, well, chaos.

One more thing before we go.

This podcast is made with the help of artificial intelligence.

I wrote the dialogue myself, but the voices you're hearing aren't ours.

They're AI-generated synthetic voices.

We're telling you this straight out because we think it should be the standard for any podcast made this way.

Full details are in the show notes and in every episode description.

So, in an episode about saying where your data comes from, we say where our voices come from.

It'd be a bit odd if we didn't.

Until next time.

Until next time.