Aleksandra Zuraw: [00:00:28] Hi everyone. Today’s episode is sponsored by Visiopharm, Denmark based leader in artificial intelligence-driven image analysis, tissue mining, and precision pathology, and my guest is JT he is Visiopharm’s team lead for computer vision and artificial intelligence. With his degree in biomedical engineering, his choice to apply his technical knowledge and harness the power of machine learning to benefit healthcare was not a surprise. A [00:01:28] personal event, further narrowed his choice of where to apply his skills to the field of pathology. Today, we’re going to talk about why and how AI is taking over the tissue image analysis field, and if there’s still space for classical image analysis methods, listen in, and I hope you’ll enjoy it.
[00:01:51] How are you Jeppe?
Jeppe Thagaard: [00:01:52] Good, I’m good. Thank you. Thank you for having me.
AZ: [00:01:55] So let’s start with a brief introduction. Tell the listeners about your background and what led you on this digital pathology path, because Visiopharm is on the digital pathology path.
JT: [00:02:08] Yeah. So my background is in medicine and technology.
[00:02:12] So I have a biomedical engineering background with 30% were in med school and then the rest of that at the technical university. So quite early, I knew that I was going to do machine learning and I wanted to do it in health. That I knew quite early [00:02:28] on in my career. And then I think, at some point I was in the US for one external stay, and I got a call from my mother saying that she had cancer and I had been there three days.
[00:02:40] And of course, as an engineer, you, start looking into what kind of information she had and all that. And that led me into kind of the pathology report. And I thought that was a lot of quantitative information there. And then that combination together with, my connection, with the founders of Visiopharm, Michael Grunkin, and Johan Doré let me on the path of digital pathology.
AZ: [00:03:03] You said that you knew it, that you’re going to do machine learning early in your studies, but machine learning in pathology is not so old. How did you know that you wanted to do this?
JT: [00:03:15] Yeah, machine learning is older than applied to pathology. And I quickly realized that there was immense power in machine learning for any problem. And I think [00:03:28] we see that now in every field. At some point it also struck me in digital pathology there was this perfect storm of untapped data, which was the requirement for machine learning. And the application needs there is very important.
AZ: [00:03:47] Yes. I remember that in 2017 and even still 2018 AI in tissue image analysis was something very cutting edge and fast forward two years, it seems to be mainstream. Now, how did it happen in such a short time? You say that machine learning has been there around, but not so much in pathology. At least in my experience, it was very fast.
JT: [00:04:14] Yeah. So I think especially the field of deep learning that really got kick-started in 2016, previous to the Camelyn challenges, 2016 and 2017, there were not that many papers out there [00:04:28] using deep learning for image analysis and pathology. I think you can almost count it on one hand, papers that use deep learning.
[00:04:37] Of course there were a lot of papers using traditional approaches and machine learning. But you’re absolutely right. There now, it’s of every problem that we have, we throw deep learning
AZ: [00:04:48] at it.
[00:04:49] So what changed in this period of time? Was it software capabilities or was it hardware or was it the mentality of people that they adopted deep learning?
[00:05:00] Now, basically, if you speak about image analysis and you still talk about the classical computer vision, people think you’re old school.
JT: [00:05:10] Yeah. I mean, Visiopharm is not a new company. It was founded in 2001, and since the beginning, Visiopharm was doing image analysis and using machine learning, more traditional approaches.
[00:05:25] So what changed was this perfect storm. I [00:05:28] think with both the whole slide scanning were more common, getting more common, so we get more data, but it’s also the accessibility of deep learning frameworks, the software, and the hardware that definitely got more accessible to the common man. So I’ll say, and I think that applies to every field, not only pathology.
AZ: [00:05:51] So we’re talking about classical computer vision, classical image analysis versus AI-based image analysis methods. Let’s explain what are the differences?
JT: [00:06:03] Okay. Yeah. So the way I usually explain it, is that we have this overarching field of image analysis and there, we can get the computer to mimic certain tasks that the human could do.
[00:06:16] And in the simplest form, we can take the data and we can understand the problem. And after that, we can write rules that solve the problem, and these we call rule-based systems. [00:06:28] So that could be, for example, traditional computer image processing, and then setting a threshold and in a certain intensity scale, something like that.
[00:06:37] So when we go into machine learning, that’s where the machine starts to learn something. And specifically the early days of machine learning in the pathology, where that we were still creating these features, we were still doing some manipulation of the image to enhance the signals of what we were looking for.
[00:06:54] But the rules from getting from those features to the output, those we’ll learn by training data, and that could be anything like a random forest classifier, a bayesian classifier, all these traditional machine learning approaches. So the kind of the latest thing is deep learning and deep learning is basically just that we learn everything from the input data to the output data.
[00:07:21] So instead of us trying to create features or engineer features that enhance the [00:07:28] signal, we’re looking for, we simply take the raw input data, we might do some normalization of that, but then we take that and we learn both the features and the rules to get to an output. So it’s just the amount in that pipeline that you actually learned from the data. That’s the difference.
AZ: [00:07:46] So we give examples now and let the system figure out what the features are instead of defining them.
JT: [00:07:53] Exactly. Yeah. So it’s really this teach by example where we don’t have to do any feature engineering. We can augment that with some feature engineering, but usually, we get along way with just inputting the raw image data.
AZ: [00:08:10] and I know that artificial intelligence is a broader concept than just deep learning, but now it’s in pathology, it’s associated with using deep learning. So I use it interchangeably here.
JT: [00:08:24] I think that’s perfectly fine, that you can do that. And [00:08:28] that’s how everybody’s doing it now.
AZ: [00:08:29] So when considering different methods for tissue image analysis, we often hear that AI is more powerful. That’s why so many companies, including Visiopharm, have an AI tool, AI module, or are only doing AI. What does that mean, that it’s powerful? What does this power consist of?
JT: [00:08:50] I think there are two aspects of that first is it can solve problems that we couldn’t solve before.
[00:08:56] I think that’s the key factor. So the assembly applications in pathology, where we didn’t know, knew how to write the rules and the computer. To get to an output that was good enough for use. So this is the power of deep learning that it learns these rules to a greater extent than you could program it.
AZ: [00:09:19] So probably means that you can work with something that you couldn’t visually see and define before.
JT: [00:09:26] Exactly. And pathologists that have [00:09:28] years and years of training. And I think that was the kind of the specialty in general in medical imaging is that medical experts they’ve been trained for many years and recognize the stuff, recognizing patterns.
[00:09:41] And some of the rules that they use, the human cognition use was just too difficult to write manually into a computer. So this is the one aspect of power. I think it’s the main reason why we use it. The other powerful aspect of deep learning is the way that we can approach problems.
[00:09:58] So instead of we have, we are having to. Spend a lot of time writing the code. We spend a lot of time collecting the data. So it’s both enabling another way of developing algorithms. And it also, who can do that. I think that’s too meaningful, an aspect of deep learning.
AZ: [00:10:18] Do you think that called writing was narrowing who could do that, and data labeling is opening it to more people or that [00:10:28] we just change the profession that does the job because now data labeling his own pathologists. They’re also a limited resource and they have limited time.
JT: [00:10:38] Yeah, I think that’s a good point. I think. I think it also always has been this collaboration between a computer scientist and the pathologist.
[00:10:46] And I think, it needs to be that, and it still is. It’s just that the way that we create for we used 80% of the time of creating an algorithm was writing the code experimentally with the code. And now 80% of developing is probably collecting the data queue, seeing it, cleaning up, making sure that there’s something that the computer can learn from.
[00:11:08] And this exercise of doing that is not only a pathologist that can do that. I think it’s still a combination of a data scientist and a pathologist.
AZ: [00:11:18] That is true. So I believe for any kind of image analysis development, you always have to work together because there needs to be a bridge [00:11:28] between computer science and pathology.
[00:11:30] And like you said, now, the way it’s, amount of work or different before it was the people coding. Now it’s the people labeling the data. I also think that it doesn’t have to be a pathologist per se, labeling the data. You can train other people to do that and supervise it. So this can be leveraged.
JT: [00:11:52] Exactly. And that’s what we also spend a lot of time on creating software that enables both improved ways of laboring, but also different approaches to labeling. So we have been working a lot with creating new ways of using immunohistochemistry or immunofluorescence to generate quantitative ground truth labeling.
AZ: [00:12:13] This is so important: ground truth. When it’s generated by people, it’s not always the same and it’s not always objective. And if you have different ways of labeling that are more objective, that [00:12:28] definitely improves the data you’re working with. So how does AI help the users of image analysis software? How does AI help the users of Visiopharm in your case?
JT: [00:12:40] So I think it’s a little bit back to the previous topic as well. We have made the platform, so it’s flexible to do whatever you want to do, but we have also enabled more users to have access to more powerful computational models. And that is that maybe we can get to a really fast point with traditional image analysis, but there you have to know something about image analysis, image processing, and that kept a cap on who can do that, who was trained to do that. So with deep learning, we can get better results, even faster by just showing it examples. And this is exactly what broadens the user base, right? Because then [00:13:28] people not necessarily trained in the basics of machine learning or image processing, they could get started with using deep learning for their problems.
[00:13:37] And I think that’s a major breakthrough in the accessibility of this technology. So how I see it is that you can quickly get started with something with minimal effort. And then what we have also done is try to make it a platform. So if you want to grow your skills, you can do that on the platform as well.
[00:13:57] So if you want to use kind of your start on a basic level, you start using deep learning, then you learn more of how it works, you learn more of the concept and then you get can get more access to the underlying parameters of a neural net. And how we did it is that of course there’s a lot of open-source material and how to learn to use deep learning, but we also have our training Academy that goes hand in hand growing people into being deep learning experts in pathology, actually.
AZ: [00:14:26] So can anyone [00:14:28] do AI. What are the entry requirements? You said that before it was somebody who already had knowledge of image analysis to use the software. Now it can be anyone really?
JT: [00:14:38] I think the entry-level is everyone. Yeah. I think if you know what to look for and how, what questions you need to ask, and you can label stuff, you can actually get 80 or 90% of the way.
AZ: [00:14:52] So labeling stuff, we talk about annotations, and for different classes that we want to detect, and this is now enabled into the software in this AI module.
[00:15:05] Yes. So what was Visiopharms journey to incorporating AI into the offering?
JT: [00:15:12] Yeah so it happens at one point where we have been working with deep learning, on a research field, outside the platform. But we saw the potential of what it could do if we needed to scale this up to more users and [00:15:28] also internally.
[00:15:29] So I can tell that my table was very full with requests for new algorithms. So we said, all right, if we need to make all these algorithms, we need to have more people doing it, and one way of doing that is making it easier to use. And that kind of kick-started the platform, the development of the deep learning module.
[00:15:51] And what we then decided was not to create something separate from the existing platform. As I said, Visiopharm has been a company for 20 years in doing digital pathology for 20 years. So we had a lot of experience when how to handle images, how to preprocess images, how to do post-processing as well.
[00:16:13] So we said, or where does deep learning fit into this framework? So we incorporated it into the stomach of the entire platform. So you still have access to all the things that you actually need to get to an algorithm. [00:16:28] And then you have this powerful tool in there that is deep learning.
AZ: [00:16:31] So the modules that are classical image analysis based are staying, and this is made into an ecosystem that can play together if I understand correctly.
JT: [00:16:44] Exactly. So, so you can a more maybe, for advanced users can combine it with traditional approaches, but any digital pathology problem that we need to create an algorithm for actually goes a fixed set of phases. So we need to of course extract the image from the whole slide.
[00:17:05] We need to read the horse. Like first that’s a major aspect of Jesus’ pathology. We need to reprocess it in some way, so we can put it into a classifier, but a neural network. But the new network also gives us an output and that output we can handle in multiple ways. So we call that post-processing where you’re in for a lot of kind of object logic [00:17:28] to your algorithm.
[00:17:30] And then you have in the end you need to take those results and then quantify something. So that’s getting density measures, getting cell counts, getting area based measurement, all these kinds of stuff that you want to do, spatial analysis. You still have to do it in a kind of combination with deep learning.
[00:17:48] So to sum that up deep learning fits in where we see that it’s more, most powerful and it solves what is best to do.
AZ: [00:17:58] So from what I hear is it definitely combines with all the other classical computer vision methods, you can use it to also derive features to later use just the classical computer vision methods.
It’s interesting because sometimes when you read about it when you listen about AI, you think it’s a standalone thing that you’re either do classical or you do AI and here it’s definitely a combination.
JT: [00:18:28] Yeah. So that’s also why we kept the traditional approaches in this, is that not every problem with highest deep learning.
[00:18:36] So if you have something that’s very simple, you should also pick the simplest model for that. And our platform kind of supports that way of working. So we think that it’s always the problem that should pick the method and not the method that you pick the problem.
AZ: [00:18:52] So you mentioned Camelyon challenge. Did Visiopharm’s team take part in the challenge?
JT: [00:18:58] Yeah, so we actually were doing work already in the communion 16 challenge and also in communion 17 challenge,
AZ: [00:19:07] Because you said you started working on deep learning before the challenge. And then did you do well in the challenge?
JT: [00:19:14] In the 17 edition, we got fifth place.
AZ: [00:19:18] Oh, wow. Good. Congratulations!
JT: [00:19:21] Those are a good job.
AZ: [00:19:24] So there was a lot of development from 16 to 17. [00:19:28] And from what I have heard, all the winning algorithms are all the first places. First ten places. This is all deep learning, right?
JT: [00:19:37] Yeah, exactly. For recognizing what is cancer metastasis? So then for those people who don’t know that the Camelyon challenge that were popular challenges put out to the community where we needed to detect metastasis in lymph node sections from, for breast cancer patients.
[00:19:54] And that’s also what kick-started our, the clinical applications the deep learning that we have today. So we actually continued working that since and have a clinically approved app of them for the European market.
AZ: [00:20:09] Great. Your journey in digital pathology started with a personal event with your mother being sick.
[00:20:18] And this is where it started. Now, how many years forward? Six years forward. Where do you think [00:20:28] it’s going to go? AI? What is it going to give us that we don’t have yet? Or are we at the end of the journey? Maybe that’s it. How do you see the development of the field?
JT: [00:20:40] So AI in general, or AI in digital pathology?
AZ: [00:20:44] In digital pathology.
JT: [00:20:45] Yeah. So I think we’re first getting started with using AI indigenous of pathology. I think there are so many low-hanging fruits that we can use, or can tackle with deep learning. So three major topics are of course predictive markers. So using deep learning to predict who needs, what treatment, and couple that with the companion diagnostics, I think that’s going to be a major breakthrough going forward.
[00:21:13] I think there’ll be already there, but with her too, we are getting there with PD-L1. But as more and more biomarkers come out as companion diagnostics for targeted drugs, I think deep learning plays a major role in digital pathology there
[00:21:27] . Second [00:21:28] one is pronounced sticks. So simply just analyzing a slide. Are they, it’s an agency also using new markers to quantify the patients and make a prognosis based on quantitative data.
[00:21:43] And then the final thing is workflow optimization and quality assessment of the digital pathology workflow. I think that is many tasks that we can solve with the current state of deep learning now. And so that’s kind in the near future of where we are going. In the longer term, I can see that it, and we already see this actually, even in the basic research, that it gives us more insight into the underlying biology and pathology of samples that either we use high plex, or we use, even higher plex kind of CyTOF technology, where we have 50, 50 Plex micros per cell, and goes more into detail on the understanding of the [00:22:28] biology
[00:22:29] in my interview with the Dr. Ralph course, he mentioned this as well, that those who are not digital yet, where we’ll be forced to go digital because of the advances in technology.
[00:22:42] Yeah, I think the field is getting there. Of course, there’s a need to be an incentive to go digital. And I think AI is the nail in the coffin of that incentive to enable us to do more than we can with the analog glass slide.
AZ: [00:22:58] So you’re satisfied with this method,
JT: [00:23:01] With the current methods and what we can do with them?
AZ: [00:23:04] Yes. Did it fulfill your expectations?
JT: [00:23:09] Yeah. I’m very much looking forward to the next. Five to 10 years, that’s going to be very exciting in this field.
AZ: [00:23:16] And obviously you’re looking at it with your computer scientists’ eyes. I’m looking at it with my pathologist’s eyes, and I am very satisfied with this as well. With the lower barrier to [00:23:28] entry. This is my main gain from AI. I don’t have to know. Which features I’m looking for. I can give examples. I know what I’m looking for. Like you said, everybody who knows that what they’re looking for can get started. So this is a great advantage for me, and I think for pathologists. Thank you so much for your time in explaining this and have a great day.
JT: [00:23:56] Thank you very much for having me.
AZ: [00:23:58] I want to thank Visiopharm for sponsoring this episode. To learn more about their offer, please visit their website at [00:24:28] visiopharm.com.
