WEBVTT 00:00.000 --> 00:06.260 Last night I was standing over the medicine drawer, taking a photo of a box with my phone, 00:07.000 --> 00:10.060 and halfway through I caught myself thinking, 00:10.520 --> 00:16.360 how do I actually know that whatever comes up on that screen is true? 00:17.620 --> 00:21.920 Hello, I'm Tomasz Szuster. I built the mojApteczka app. 00:22.780 --> 00:27.020 And Iza's question there is probably the most honest question you can put to anyone 00:27.020 --> 00:29.860 building something with artificial intelligence inside it. 00:30.500 --> 00:33.000 Because I see it from the other side, you know. 00:33.900 --> 00:40.640 Everyone's saying, our app has got AI in it now, as though that were an argument all by itself. 00:41.260 --> 00:42.100 It is. 00:42.620 --> 00:44.460 It very much isn't. 00:45.260 --> 00:48.620 If anything, it's a fresh reason not to trust the thing. 00:49.500 --> 00:53.220 When a person gets it wrong, I know who to be cross with. 00:53.740 --> 00:57.260 When a model gets it wrong, then what? 00:57.260 --> 00:59.980 That's exactly what today's about. 01:00.720 --> 01:03.660 Not, does AI make mistakes, because it does. 01:04.300 --> 01:05.900 It's what happens afterwards. 01:06.780 --> 01:08.060 Who gets the last word? 01:09.100 --> 01:09.420 Right. 01:10.020 --> 01:11.640 Let's start at the beginning, then. 01:12.520 --> 01:13.880 I photograph a box. 01:14.420 --> 01:15.200 What happens? 01:15.200 --> 01:19.600 The photo goes to a model that can look at an image and read it. 01:20.240 --> 01:24.200 It pulls out the name, the dose, the expiry date and the barcode. 01:24.940 --> 01:26.740 That's the bit that looks like magic. 01:27.700 --> 01:30.440 And that's where I'd stop if I were in marketing. 01:31.080 --> 01:32.640 Most people do stop there. 01:33.240 --> 01:35.260 But it's only the first of three steps. 01:35.620 --> 01:38.360 And honestly, it's the least important one. 01:38.360 --> 01:40.420 Why the least important? 01:41.240 --> 01:43.840 Because the model doesn't know anything. 01:44.640 --> 01:47.100 It recognises the shapes of letters in a photograph. 01:48.000 --> 01:52.740 So the second step is that I take that barcode and look it up in a separate medicines database. 01:53.480 --> 01:59.820 And from there, I pull the manufacturer, the form it comes in, the ATC group and the links to the leaflets. 01:59.820 --> 02:03.700 So it isn't the model telling me what the medicine is? 02:03.900 --> 02:04.060 No. 02:04.520 --> 02:08.700 The model says, this box probably has such and such written on it. 02:09.260 --> 02:12.360 The register says what the medicine actually is. 02:13.020 --> 02:14.960 Those are two completely different things. 02:15.420 --> 02:18.860 And that distinction is the foundation the whole app sits on. 02:20.160 --> 02:20.760 All right. 02:21.120 --> 02:22.520 And the third step? 02:23.000 --> 02:24.860 The third step is you. 02:25.520 --> 02:25.860 Me? 02:26.740 --> 02:27.800 Nothing saves itself. 02:27.800 --> 02:31.540 The result lands on a screen where everything's already filled in. 02:31.960 --> 02:34.120 But everything can be changed. 02:34.840 --> 02:38.320 And until you tap save, there's nothing in your cabinet at all. 02:39.160 --> 02:42.680 So the app never tells me I've added a medicine for you? 02:43.320 --> 02:43.640 Never. 02:44.440 --> 02:47.540 The result goes to a screen for you to check and correct. 02:48.080 --> 02:49.620 The decision sits with you. 02:50.480 --> 02:56.500 And there's one more thing that sounds dull and is, to my mind, the most important part of the whole scan. 02:56.500 --> 03:00.720 The app can't create a medicine that isn't in the register. 03:01.660 --> 03:02.200 Hang on. 03:02.680 --> 03:04.820 So if the model invents something… 03:05.540 --> 03:06.580 There's nowhere to put it. 03:07.200 --> 03:10.460 Without a match in the register, the app won't add it as a medicine. 03:11.300 --> 03:11.820 Right. 03:12.480 --> 03:14.980 That does sound honest, to be fair. 03:14.980 --> 03:20.500 By the way, everything we're discussing here are actual features in the mojApteczka app. 03:20.980 --> 03:21.600 Not a pitch. 03:21.980 --> 03:23.020 Just context. 03:24.020 --> 03:26.720 Now the question that genuinely nags at me. 03:27.300 --> 03:29.680 Because all of that is engineering. 03:30.540 --> 03:32.260 I'm asking about responsibility. 03:33.140 --> 03:33.560 Legally. 03:33.560 --> 03:38.060 If the app shows me something that's wrong, then what? 03:39.140 --> 03:42.400 That's the question I had to answer before I wrote a single line. 03:43.180 --> 03:46.660 And the answer is in the app itself, in plain text. 03:47.360 --> 03:47.660 Meaning? 03:48.260 --> 03:52.140 When you open the drug interaction screen, there's a sentence at the bottom. 03:52.140 --> 04:02.100 The mojApteczka app is not a medical device within the meaning of the European Medical Devices Regulation, MDR 2017 745. 04:03.340 --> 04:06.620 And is that good news or bad news? 04:06.720 --> 04:07.440 It's a choice. 04:08.060 --> 04:08.920 A deliberate one. 04:09.540 --> 04:11.120 In practice, it works like this. 04:11.540 --> 04:17.220 If a product claims it diagnoses or treats, it falls into an entirely different regulatory regime. 04:17.220 --> 04:18.220 A notification. 04:18.960 --> 04:19.940 A notified body. 04:20.520 --> 04:21.080 Oversight. 04:21.680 --> 04:24.620 And it makes no odds whether it's a piece of hardware or an app. 04:25.180 --> 04:28.620 So, in plain terms, you could have gone down the road of, 04:28.820 --> 04:31.700 the app will tell you whether this medicine is right for you. 04:32.120 --> 04:32.960 I could have tried. 04:33.620 --> 04:34.320 And I didn't. 04:34.900 --> 04:39.600 Because then I'd be promising something I can't honestly guarantee for every medicine, 04:39.980 --> 04:41.720 every person, every condition. 04:42.260 --> 04:45.320 The verdict belongs to the GP and the pharmacist. 04:45.320 --> 04:46.940 I can be your memory. 04:47.680 --> 04:48.960 I can't be your oracle. 04:49.380 --> 04:51.300 You know what's odd about that? 04:52.020 --> 04:54.120 It sounds like admitting a weakness. 04:54.800 --> 04:55.240 It does. 04:55.720 --> 04:56.540 And it's the opposite. 04:57.220 --> 04:59.460 It's terribly easy to fall into the other trap, 04:59.880 --> 05:01.840 showing confidence you haven't actually got, 05:02.240 --> 05:03.300 because it looks better. 05:04.020 --> 05:07.360 I'd rather show you the boundary and then stand behind it. 05:07.840 --> 05:08.260 All right. 05:08.760 --> 05:10.640 But if you're not the oracle, 05:11.360 --> 05:14.740 how am I meant to know what to trust in what you're showing me? 05:14.740 --> 05:16.760 Because I tell you where I got it. 05:17.380 --> 05:19.280 Next to the safety information for a medicine, 05:19.600 --> 05:21.160 the source is spelled out. 05:21.700 --> 05:24.860 The summary of product characteristics from the medicines regulator. 05:25.700 --> 05:28.800 LactMed from the American National Institutes of Health. 05:29.600 --> 05:30.440 Open FDA. 05:31.200 --> 05:32.980 And the European Medicines Agency. 05:33.680 --> 05:34.640 Four sources. 05:35.320 --> 05:35.740 Named. 05:36.180 --> 05:36.560 Named. 05:37.060 --> 05:37.900 And for interactions, 05:38.280 --> 05:40.700 there's a specific scientific database credited, 05:41.100 --> 05:42.300 with its year of publication. 05:42.300 --> 05:47.260 Because our algorithm detected is a sentence that means nothing at all. 05:47.880 --> 05:48.760 This came from here. 05:49.200 --> 05:50.240 Go and check it yourself. 05:50.880 --> 05:52.160 That means something. 05:52.500 --> 05:53.260 Hold on, though. 05:54.020 --> 05:55.660 There's something I don't follow. 05:56.420 --> 05:59.260 You show possible interactions between medicines. 05:59.840 --> 06:02.980 That already sounds like something a pharmacist does. 06:02.980 --> 06:04.440 And that's a very fine line. 06:04.940 --> 06:06.800 So let me say precisely what happens. 06:06.800 --> 06:09.100 I take the medicines in your cabinet. 06:09.800 --> 06:11.880 I make every possible pair out of them. 06:12.420 --> 06:15.240 And I check each pair against a scientific database. 06:16.000 --> 06:16.960 How big a database? 06:17.840 --> 06:20.760 More than 300,000 documented interactions. 06:21.700 --> 06:24.180 And since a pair has to be recorded in both directions, 06:24.700 --> 06:27.440 that comes to over 1.3 million entries. 06:27.440 --> 06:29.860 And all of that sits on the phone? 06:30.560 --> 06:33.040 No, that sits on the server side and gets queried. 06:33.600 --> 06:36.180 But the more important bit is what isn't in there. 06:36.840 --> 06:37.720 You aren't in there. 06:38.320 --> 06:40.260 What do you mean, I'm not in there? 06:40.840 --> 06:45.540 That database describes a possible interaction between substance A and substance B, 06:45.980 --> 06:47.280 and how severe it might be. 06:47.280 --> 06:50.720 It doesn't know your age, what you're being treated for, 06:51.100 --> 06:52.820 what's already been taken off your list, 06:53.300 --> 06:55.460 or whether your GP decided quite deliberately 06:55.460 --> 06:58.300 that in your case, the combination is fine. 06:59.060 --> 07:01.340 So the app could frighten me over nothing? 07:01.980 --> 07:04.680 It could, which is why it's sorted by severity, 07:05.260 --> 07:06.560 labelled with where it came from, 07:06.940 --> 07:08.440 and signed off with a line saying, 07:08.800 --> 07:10.340 this isn't medical advice. 07:10.940 --> 07:13.640 It's meant to be a list of questions for your pharmacist, 07:13.900 --> 07:14.540 not a verdict. 07:15.360 --> 07:15.780 Right. 07:15.780 --> 07:18.840 So I go into the chemists and say, 07:19.420 --> 07:20.840 the app flagged these two. 07:21.080 --> 07:21.920 Would you have a look? 07:22.520 --> 07:24.720 That's precisely the scenario I built it for. 07:25.260 --> 07:27.060 Not, the app said I can't. 07:27.400 --> 07:29.860 Just, the app drew my attention to something. 07:30.360 --> 07:32.600 I'd like the opinion of someone who actually knows. 07:33.300 --> 07:34.600 And when the data's thin? 07:35.400 --> 07:38.120 Because we don't know the same amount about every medicine. 07:38.800 --> 07:41.080 Good question, and I've got an uncomfortable answer. 07:41.880 --> 07:43.860 For roughly 61% of products, 07:43.860 --> 07:46.660 the confidence in the data is low. 07:47.320 --> 07:48.400 61%? 07:48.400 --> 07:50.360 That's more than half. 07:50.460 --> 07:51.160 More than half. 07:51.940 --> 07:55.660 It comes from the fact that the product characteristics are written very cautiously. 07:56.240 --> 07:57.120 Quite rightly, too. 07:57.680 --> 07:59.740 But for me, it's a design problem. 08:00.600 --> 08:03.540 What do you do with information you aren't sure of yourself? 08:03.540 --> 08:06.120 And what do you do? 08:06.120 --> 08:07.080 I mute it. 08:07.080 --> 08:11.480 Where confidence is low, the icon is grey, with no colour fill. 08:12.160 --> 08:14.780 Only the better documented ones get full colour. 08:15.120 --> 08:15.720 Mute it? 08:16.100 --> 08:17.720 That sounds like hiding it. 08:18.060 --> 08:20.040 It looks similar, and it's the reverse. 08:20.040 --> 08:23.400 If I show you a strong warning where the data's thin, 08:23.760 --> 08:25.700 I'm training you to ignore warnings. 08:26.240 --> 08:28.880 And then you'll miss the better documented one, 08:29.260 --> 08:32.400 the sort that's genuinely worth taking to the pharmacist. 08:32.620 --> 08:33.040 Ah. 08:33.580 --> 08:38.080 We had exactly that problem with the notice saying the information is for guidance only. 08:38.080 --> 08:43.120 It was on every medicine card, the whole thing, with the full list of sources. 08:44.020 --> 08:49.080 On a list of eight medicines, that's the same block of text eight times over. 08:49.540 --> 08:51.320 So people stop reading it. 08:51.620 --> 08:52.160 Precisely. 08:52.740 --> 08:57.280 So I left one sentence visible, for guidance only, not medical advice, 08:57.700 --> 09:00.400 and tucked the full source list behind an icon. 09:01.020 --> 09:05.380 It's always there, one tap away, but it isn't drowning itself out. 09:05.380 --> 09:09.100 That's interesting, because that isn't a legal decision. 09:09.800 --> 09:13.080 That's a decision about making the thing actually work. 09:13.460 --> 09:14.900 And that's the whole difference. 09:15.560 --> 09:17.200 You can tick the compliance box. 09:17.700 --> 09:22.200 You can paste in 300 pages of terms and be technically in the right. 09:22.880 --> 09:24.060 I'm after something else. 09:24.600 --> 09:30.560 That the person standing over the drawer at 10 at night knows what this app does not know. 09:30.880 --> 09:31.240 Hmm. 09:31.240 --> 09:34.780 And I'll tell you one more thing most app makers wouldn't say out loud. 09:35.380 --> 09:38.400 Those safety markings are switched off by default. 09:39.200 --> 09:41.860 You have to turn them on in settings yourself. 09:42.220 --> 09:42.700 Hang on. 09:43.000 --> 09:43.460 Why? 09:44.020 --> 09:47.080 That sounds like something that ought to be on from the start. 09:47.440 --> 09:51.280 Because it's information that means something when somebody reaches for it deliberately. 09:52.060 --> 09:56.180 If I push it at everyone, on every medicine card, we're back where we started. 09:56.820 --> 09:58.360 Noise nobody reads. 09:58.360 --> 09:59.360 Noise. 09:59.360 --> 10:01.960 I'd rather a person decided once that they want it. 10:02.740 --> 10:07.040 So, fewer warnings, but one somebody actually reads. 10:07.640 --> 10:08.360 That's the trade. 10:09.080 --> 10:12.220 And honestly, I don't know whether that's the final answer. 10:12.780 --> 10:14.740 But I know why it's that way for now. 10:14.740 --> 10:16.780 Tell me one more thing. 10:17.720 --> 10:20.920 Because scanning a prescription is a different matter entirely. 10:21.820 --> 10:24.440 There's data on there I wouldn't hand to anybody. 10:25.260 --> 10:28.180 Which is why that feature works differently from scanning a box. 10:28.840 --> 10:31.440 The whole reading of the prescription happens on your phone. 10:32.180 --> 10:34.000 The photo of the prescription doesn't go anywhere. 10:34.000 --> 10:37.120 Anywhere meaning where, exactly? 10:37.960 --> 10:38.380 Anywhere. 10:39.080 --> 10:39.940 Not to my server. 10:40.260 --> 10:41.580 Not to any model in the cloud. 10:42.140 --> 10:43.360 It stays on the device. 10:44.120 --> 10:46.120 Only the medicine name goes to the look-up. 10:46.560 --> 10:48.780 Exactly as if you'd typed it into the search yourself. 10:49.560 --> 10:53.780 So the numbers, the patient details, all of that... 10:53.780 --> 10:54.600 stays with you. 10:55.340 --> 10:57.500 And again, nothing adds itself. 10:58.240 --> 11:00.920 You confirm each item from the prescription separately. 11:01.880 --> 11:02.940 Do you know what? 11:02.940 --> 11:06.960 I've a feeling we've been saying one sentence all episode, 11:07.360 --> 11:08.700 just in different ways. 11:09.640 --> 11:10.480 Which is? 11:11.120 --> 11:15.620 That artificial intelligence can suggest, but it can't decide. 11:16.280 --> 11:16.800 Yes. 11:17.380 --> 11:20.200 And if I were to leave listeners one thing from this episode, 11:20.680 --> 11:21.380 that'd be it. 11:22.040 --> 11:24.480 Don't ask whether an app has got AI in it. 11:25.220 --> 11:27.460 Ask what happens when that AI gets it wrong. 11:28.060 --> 11:30.100 Whether there's a screen where you can correct it. 11:30.760 --> 11:32.780 Whether you can see where the data came from. 11:32.940 --> 11:37.520 Whether somebody has written down plainly what the tool does not do. 11:37.520 --> 11:39.780 And if none of those are there? 11:40.260 --> 11:43.800 Then you've got less to go on than you need to judge how far to trust it. 11:43.800 --> 11:45.540 And that's your decision. 11:45.540 --> 11:48.700 You ought to be making it with your eyes open. 11:48.800 --> 11:49.120 Right. 11:49.520 --> 11:50.560 I'll add mine. 11:51.080 --> 11:53.580 From a completely non-technical angle. 11:55.040 --> 12:01.040 Leon is nine, and he already understands that something appearing on a screen doesn't make it true. 12:01.040 --> 12:05.900 Basher is five, and she doesn't understand that yet. 12:05.900 --> 12:09.660 And honestly, half the grown-ups don't either. 12:10.280 --> 12:15.200 Which is why that sentence has to be in the app, not in the terms and conditions. 12:15.200 --> 12:21.460 And it's why, when I'm unsure about a medicine, I still take the box down to the chemists. 12:21.860 --> 12:22.880 And so you should. 12:23.520 --> 12:28.440 The app's job is to make sure you know what you've got at home and what to ask about. 12:28.440 --> 12:31.320 The answer belongs to the pharmacist. 12:32.840 --> 12:36.540 If you'd like to give it a go, you'll find the link in the episode description. 12:37.480 --> 12:41.120 The app is free, and your medicine cabinet is there offline too. 12:42.260 --> 12:48.880 And if you enjoyed this episode, share it with someone whose medicine cabinet at home is, well, chaos. 12:50.300 --> 12:51.820 One more thing before we go. 12:52.600 --> 12:55.660 This podcast is made with the help of artificial intelligence. 12:55.660 --> 13:00.940 I wrote the dialogue myself, but the voices you're hearing aren't ours. 13:01.380 --> 13:03.820 They're AI-generated synthetic voices. 13:04.780 --> 13:10.700 We're telling you this straight out because we think it should be the standard for any podcast made this way. 13:11.800 --> 13:15.860 Full details are in the show notes and in every episode description. 13:17.040 --> 13:23.820 So, in an episode about saying where your data comes from, we say where our voices come from. 13:23.820 --> 13:25.900 It'd be a bit odd if we didn't. 13:26.400 --> 13:27.380 Until next time. 13:28.280 --> 13:29.380 Until next time.