Radiology Partners’ approach to clinical AI applications: w/ Nina Kottler, MD, MS, FSIIM | Radiology Partners

Nina: So we in healthcare have to figure out what are the best practices.

How do we make sure that AI is being created in the right way? How to make sure we’re testing it in the right way? How are we deploying in the right way? How are we educating people about it? And we have to be able to figure out how to manage that and normalize that. Even the regulation is trying to do that as we are trying to keep up in health care or we are keeping up with the rate of technology evolution.

How is it that we could actually improve the quality and decrease the cost of care? Instead of just saying, Oh, we’re the best by raising our hand, which is what a lot of people did. Or they said, we’re the best use us because we have the best operational metrics. Operational metrics are not going to help patients.

We need patient care metrics and quality metrics.

Intro: [00:01:00] Learn about the newest digital pathology trends in science and industry. Meet the most interesting people in the niche and gain insights relevant to your own projects. Here is where pathology meets computer science. You are listening to the digital pathology podcast with your host, Dr. Aleksandra Zuraw.

Welcome my digital pathology trailblazers.

Aleks: Welcome my digital pathology trailblazers. Today, I have a radiologist on the show again. My guest is Dr. Nina Kottler. She is a practicing radiologist and associate chief medical officer for clinical AI at Radiology Partners. And Radiology Partners is the largest radiology practice in the U.S. And not only is she in this largest practice in the US, my guess is also in the American College of Radiology Metrics [00:02:00] Committee, American College of Radiology Quality and Safety Conference Planning Committee, then in the Society of Imaging Informatics in Medicine Machine Learning Committee, and then this is the same Society of Imaging Informatics in Medicine (SIIM).

She’s in the program committee for SIIM, and she’s also in the Radiology Society of North America Educational Exhibits Committee. Nina, welcome to the show. I don’t know. You are like very prominent doctor and AI personality in the medical imaging society. I’m also super lucky to have you because when I was reading up on your bio, there’s one specific award.

That I noticed among all the other awards and this one is called a trailblazer award. And I always welcome my digital pathology interested people, my subscribers, [00:03:00] those who listen, they are my digital pathology trailblazers. So when I saw that you’re, you got a trailblazer award in 2018, I was like, this is my guest. Let’s see if she wants to join, and she wanted to join.

Welcome to the show. 

Nina: That is perfect. I welcome all digital trailblazers.

Aleks: Welcome. So we start with you other than, all the committees that you’re part of and all your professional position is, and just let the digital pathology trailblazer know. Who you are? How did you get to work with AI at the level you’re working with AI?

Because this is not just, oh, I’m interested and read up a few papers. There is a long trail in your education that I already know of, but if you could share it, that would be great.

Nina: Sure. And I don’t want it to scare anyone ’cause I do have a background in applied mathematics, but that’s not what makes me have expertise [00:04:00] in artificial intelligence.

Anyone can develop it. Alright, so where did I come from? I did not actually think I wanted to become a physician. It just never occurred to me. And my father… 

Aleks: It just so happened accidentally. 

Nina: Yeah. I noticed. So I got into mathematics. I was good at math. And so I was going to get my PhD in applied mathematics.

And even when I was in my college I was working with one of my attendings or one of the professors there, and he asked me to help him with a graduate program where he was trying to create a model, a mathematical model of the cornea. So I started modeling the cornea and learning a little bit about optics and how the eye works.

And then when I was in graduate school, I was working on a mathematical model of the kidney. So we were looking at how solvents and solutes flow through the kidney and we were creating differential equations to model it. But I found the kidney was far more fascinating [00:05:00] than the differential equation. So with all of those things, I ended up thinking, what is it that I’m going to be when I’m older?

And and I thought…

Aleks: When I grow up… 

Nina: When I grew up, I just didn’t know when I was younger, I was like if I’m good at math, I’ll go into math. But yeah. I thought a little bit more about it and realized that being in medicine was really fascinating. So from there, I had to go back, do all the medical stuff, went to medical school.

And radiology was a great combination of the two, between technology, it’s mathematical, and there’s also the physical components. So all of that combined together. And frankly, when I was in medical school, for any clinicians out there, where you go through rotation, and every rotation that we were on, because this was, gosh it was in the 1990s, it was a very long time ago.

There was no digital radiology back then. Everything was analog. And you didn’t have access to any of your imaging and you didn’t have access to [00:06:00] a report. So every rotation, you went down into the reading room and you talked to the radiologist. And I just… 

Aleks: It’s like today, right? The day where all computers broke.

Nina: Oh, like today? Absolutely. 

Aleks: Yeah. 

Nina: Can talk to… 

Aleks: But like some CrowdStrike bug or whatever that was. And all the institutions like banks. I heard, so by the way, my husband has not a similar, but like a parallel thing that he did. He is a clinical pathologist and he was a chemical engineer first.

And then he went, he decided he wanted to be a doctor, like out of the blue applied to two medical schools and then became a doctor. But he basically told me that mass general had to cancel their surgeries today because of this computer outage. So that’s like the, in the olden days, right? With everything was analog.

Nina: And in some ways there were components of analog that were better in that people had to talk with people much more closely. We provided a much greater history, digital [00:07:00] is the only way to scale. So in the future with AI and all of this, we have to go back to the past and look at what worked really well back then, which was the person to person communication, the understanding about the patient, the discussion back and forth.

And the robust consultation, but also moving to the future where we are digital, we can scale and we can do that more easily.

Aleks: This is funny because like you said, like this was taken away by digital and now we are relearning to recreate the benefits of that on digital communicators, right? The less study.

So I’m a toxicologic pathologist, evaluate studies for drug development and the last study I was very much recreating that experience, like sending screenshots to people across the organization and trying to just have this super dynamic fast interaction that you were able to do when you had somebody next to you in the same room.

You could just ask we have to [00:08:00] relearn so yeah, we’re bringing back that analog benefits into the digital world to scale.

Nina: That is the goal, because not all technology has been deployed in the most optimal way. And so if we think about how do we go back to take those optimal components and build it into our technology in the future, that’s going to be the way forward.

Aleks: And I am sure you guys already did part of this at Radiology Partners because this is the largest radiology practice in the U.S. And so how do you scale there and how do you approach AI implementation and innovation at Radiology Partners. Maybe a little bit of a few words about how you operate? How did this practice become the largest in the U.S. and in the world of, smaller groups that are servicing hospitals?

Nina: Yes. We didn’t always start that way. Like everything, we started when [00:09:00] it was just two co founders, Rich and Anthony. So Anthony Gabriel and our current CEO, Rich Whitney. It was the two of them that had this idea that radiology in the US was very disparate.

The average size of a radiology practice was about nine people at the time. And there was scale happening all around the world in healthcare, especially with insurance companies and with hospitals consolidating. And it’s really difficult if you’re so separate to be able to improve the quality and drive value.

So they thought why don’t we try to bring groups together? Because at scale, any group could outperform anyone else and provide that value back into healthcare that we needed. So in 2013, I joined Radiology Partners. I was the first radiologist. So it was Rich, Anthony, and myself. And because of that, they call me Rad1.

We now have 600 or more radiologists. We do about probably [00:10:00] more than 10 percent of the imaging done in the U.S. is done by our practice. And we’re still growing. The way that we got there was by concentrating on things like driving value. How is it that we can actually improve the quality and decrease the cost of care?

Instead of just saying, Oh, we’re the best by raising our hand, which is what a lot of people did. Or they said, we’re the best use us because we have the best operational metrics. Operational metrics are not going to help patients. We need patient care metrics and quality metrics. And so the idea was.

Why don’t we come up with something that you could actually measure that was a true quality metric that would help patients that could drive value. And that’s what we did. And that was really successful and that’s what’s driven us to grow among a few other things, culture and maintaining some local leadership and education, all of these things together.

But it has been a 10, 11 year journey now.

Aleks: So what were the [00:11:00] metrics that you came up with? What was the thing that set you apart? Also when you talk about what’s relevant in like your improvements to the patients? Because I think. Yeah. It’s like you say, very much when you, and in radiology, not so much as in pathology because pathology is a little bit later in this journey.

And it’s still a discussion, oh, do we want to invest the money or not? And the metrics deciding, do we want to invest the money is more like operational money type of return on investment and not really patient care driven. And so what do you guys have? on the Covey measure?

Nina: It was actually not easy to initially come up with something because there’s no one that had done it.

And so we had to identify what are the things that we could do that really drove quality and things like peer [00:12:00] review or turnaround times, which were very operational, were not those things. So we got together and formed what we call a clinical value team. And that the mission of that clinical value team was to come up with these metrics that we could not only say, not only know would improve quality, but also we could measure.

And that’s the harder part. You have to be able to measure it to say you’re doing well. And what we looked at was decreasing variability. Variability is the enemy of quality. It’s the enemy of efficiency. And if you could decrease variability, you will automatically increase value. And we looked at the variability in, humans are variable, so we looked at the variability at how we reported things.

And there were lots of guidelines about population health. How do you follow up certain lesions that had been, extended out through journal articles and people would try to follow. But people are doing that at variable rates. So for example,  [00:13:00] if you have a lung nodule or if you have an aneurysm of Rheorda, triple-A.

What are the guidelines that improve population health? We should all be following these guidelines. It ended up that most of the time people were not following these guidelines. And it’s because it wasn’t easy for them to do. We tried to make an easier way to provide those guidelines immediately to our radiologists so that they could follow them, and we created a way to measure if they were.

And we went from, in fact, for Triple-A, that, that was the one that people were following least. Not only in our practice, but all across the U.S., we actually went to an academic medical center. They said, oh, we want to see what you’re doing. Can you evaluate us? We went to a very high-end academic medical center.

You would think they’re following guidelines 100 percent of the time. Zero. Zero. 0 percent of the time are they following them. 

Aleks: And they like asked you to evaluate them? 

Nina: Yeah, because they wanted to improve. 

Aleks: That’s amazing…

Nina: Like we all should, that’s a learning opportunity and not a not a judgment [00:14:00] because if we’re, we have to be able to hold a mirror up to ourselves and look back and say, we’re doing something that isn’t the best, let’s figure out a way to improve it.

And I think they were great for doing that. So we started rolling that out and as we did, we were rolling it out using change management and reminding people to do things because that’s the processes we have. And I realized that just wouldn’t scale. We all realized that wouldn’t scale. If you have one or two of these, people will remember them.

But if you have 10 or 20, like there’s no way that we, humans aren’t good at that. So we realized at that point we had to create technology that would help us.

Nina: And that was my first foray into artificial intelligence. Back in 2017.

Aleks: Yes. So this was already, so I can totally relate because also like from the books that I read recently, I think I shared with you that the price we pay about the price of healthcare. [00:15:00] 

Of course, I forgot the author, but I will Google it in the meantime. But anyway, there he says that… Like everybody operates with the knowledge they gained whenever they were trained, right? And he was looking into the opioid, the amount of opioids being prescribed is like not a derivative of some like ill will or, negligence or anything.

This is what the people were taught in residency. And this was drilled into them. And this is the knowledge that they go into the world with. And then research advances. And you live and you work and there is very little time to catch up on these things that you might not know even exist.

So you say 2017 NLP,and this was like a lot before our ChatGPT and the large language models. So can you maybe walk us [00:16:00] through the evolution of the natural language processing and AI that you were using and also I assume later computer vision came into play for radiology. It’s more at play than in pathology and pathology we’re just starting.

So how did this, how did it, how did you start with what methods and how did it evolve when new methods were available? Specifically the large language models, the deep learnings for for computer vision. And now we are actually venturing into the large vision models as well.

Nina: And the multimodal, which combines large vision and large language with others. 

Aleks: Yes. Exactly. 

Nina: It’s super exciting. And that the technology, the development of technology is accelerating massively.

Nina: So we’ve had artificial intelligence around since the 1960s. Now it is not the AI that we know today.

And in fact, even in healthcare and in radiology, we’ve been using a [00:17:00] form of artificial intelligence Since the 90’s with the old-fashioned CAD, which is computer-aided detection. We did it for breast imaging or our MAMO studies, right? We had all 2D MAMO back then. And the CAD was a rule-based system.

It wasn’t a machine learning. That was old fashioned computer aided detection. Didn’t work that great because we had to, as humans, program the computer to say, if a lesion looks like X, Y, and Z, if the borders are A, B, and C, then tell us that you think it’s something we should look at. And there were a lot of positives and people didn’t love it because it wasn’t that great.

That was in the 80s, 90s. Then in 2000s, early 2000s, we started machine learning. That didn’t get to foreign healthcare until the 2010s where we started doing deep learning. Now deep learning is a type of machine learning and that’s different from the old fashioned CAD because instead of programming how the computer [00:18:00] learns, you teach it by example.

The way that you teach your kids, like when my kids are growing up, I wouldn’t tell them that a ball was a circular shape and a certain color and it bounced and we threw it. I would show them a football and then I would show them a tennis ball and then I’d show them a basketball. I showed them all a different kind of ball, and they’d say, I’d call it ball.

And then they would understand what a ball was. So that was machine learning. And deep learning had more capability because our compute was better and and cheaper. So that was convolutional neural networks are the main, architecture for deep learning that we used with computer vision and that most people use with computer vision in healthcare.

That was in the two thousands a little bit later and also natural language processing, that’s the kind that we used for language. And so when we were creating that first AI solution that you mentioned, it was NLP, natural language processing. That was back in 2017. After that [00:19:00] actually in the sameyear, that year, 2017, is when this famous paper came out by Google, which is called “Attention is All You Need.”

And that was the architecture behind Transformers. And it’s a new way of thinking about machine learning. And this is the technology that is behind OpenAI, ChatGPT, all of the new large language models that are out there, it’s using an attention mechanism to look at different components of what you say, the whole thing in in total, and then put that information together.

And you can do the same thing now with vision. If you take pieces of an image. And you look at them all and then you put that information together, that is a transformer, a vision transformer model. So it was large language models came out, I would say in the 20, early 2020s, right? CHAT GPT was November, 2022.

After that, the, there started to be a whole bunch of applications [00:20:00] in medical imaging. People were using CHAT GPT, but they were also using other versions and they were modifying it. And now, as of, I’d say 2023, we’re starting to create and combine large vision models with large language models, with a generative AI, that’s the part that can give you an answer, and you can communicate with it simply.

Those are now the AI of the present day within the world, but of the present and future day within healthcare. 

Aleks: Definitely.  And we lag a little bit in healthcare because when ChatGPT came out in 2022 I remember giving some webinars and I tried to find publications and there was like nothing there was maybe five publications.

And now you have a lot more, but most of them are still like from the knowledge. 

Nina: [00:21:00] Where it came out, where… 

Aleks: We didn’t have methods to restrict the hallucinations, like the retrieval augmented generation, where the fine-tuning of the model was not yet something that people now it’s like part of the development process.

And I call it consumer AI. It’s implemented as it’s being developed, but in the medical field. publications that, so I have every week on Friday, I have a little, I call it digital, digipath digest, and I have this PubMed alert that sends me alerts about papers about digital pathology and AI, and I just started seeing this trend this month that you have more ChatGPT papers.

But when you look into those papers one out of like four actually describes. the wave that the consumer AI is writing right now. So like that fine tuning, making it more specific [00:22:00] for domains and the rest is still, Oh, it can hallucinate. Oh, it is not trained specific to the domain. So I’m like, okay, there is a lag.

And obviously, this lag is because the peer review process takes time. And also because the use in medicine and is slower than just consumer use. So you are more restricted. And then, one thing that I did look up, the book The Price We Pay is by Dr. Marty Macari. If somebody wants to grab this one.

Nina: Awesome.

Aleks: Okay. So…

Nina: Just to comment on that.

Nina: Completely fascinating to me because you said healthcare, we’ve always been behind because we have to be more careful. We’ve got patient lives at stake. There are regulations that oversee things. And in general, technology has been maybe 10, 20 years behind in healthcare, the rest of the world.

The rest of all the specialties and organizations that are out there. [00:23:00] And when this is crazy… 

Aleks: 20 years in technology years is like… 

Nina: Yes. 

Aleks: Like a hundred years. 

Nina: It is. 

Aleks: I don’t know. It’s crazy. 

Nina: And it’s the same.

Aleks: We stil use fax.

Nina: Yeah. I mean, when, even during COVID, my kids were speaking to each other and speaking to their friends through their computer systems, through their video games, and I’m still faxing stuff to people.

So we tend to be very far behind when it comes to healthcare because we’re careful. Now the benefit of that is that we know where to go. Everything has already been optimized by the time we get there. And it’s already, there’s a thousand papers written about it. And here we are now, we’re at the tip of the spear with AI.

We are implementing this as everyone else is. So like you said, there’s no journal articles about it. There’s no book that is being written that by the time it’s written, it’s not out of date that we can follow that will tell us what to do. So we in healthcare have to figure out what are the best practices.

How do we make sure that AI [00:24:00] is being created in the right way? How to make sure we’re testing it in the right way? How are we deploying it in the right way? How are we educating people about it? All of these things are the wild west right now. And we have to be able to figure out how to manage that and normalize that.

Even the regulation is trying to do that as we are keeping, trying to keep up in healthcare, or we are keeping up. with the rate of technology evolution.

Aleks: You’re so right. So I wrote the book that they have behind, Digital Pathology 101. And the last year, end of last year, it’s already out, the AI chapter is outdated.

I have to add all the information about the large language model and generative AI. And yeah, totally. 

Nina: And even some of that is changing now. So the, these transformer models, this transformer architecture that came out in 2017 ends up being very compute hungry. So it’s expensive. And digital pathology, the images are very high resolution. [00:25:00]

And if you want to have many images, so you’ve got lots of slices and they’re all high resolution, it’s just computationally too expensive to be able to do right now. There’s a quadratic equation that’s associated with a cost for transformers. So we’re going to need to transform again. And there’s new models called state space models that people are talking about that have a more linear compute.

Not a quadratic formula, and the cost of compute over time will come down, but we’re going to be continuing to evolve. We are not near to be, we’re not nearly at the end of our journey.

Aleks:  Yeah. I learned that with, when deep learning was introduced to image analysis in the commercially available software.

So that was like, let’s say I don’t know, Alex net was the deep learning architecture that outperformed everything. And one of those image competitions that was, I think, 2012 and 2016, 17, the commercial [00:26:00] image analysis software programs for just, for researchers, for working with pathology images started implementing this.

And that was the time, okay, we didn’t have to do that classical machine learning with thresholding. And like you described, we did not have to describe the ball as a round object. We could just give an example of the ball and then Oh my goodness, this is like a game changer. And now the large language models and different things.

So now I’m okay, if we hit the wall, let’s just wait a month. There is going to be something else. 

Nina: Yeah. 

Aleks: And the question is, unlike the discussion that we have, like, how can we benefit from it in medicine in probably not real time, but like first enough, because then it’s. Being so behind is to the disadvantage of patients, but you don’t want to [00:27:00] go like with the newest flow because then you have not tested enough.

Nina: That’s the hard part. If you look at, I don’t know if you’ve heard of Gartner Hype Cycle, it’s, Gartner is an information technology and consulting firm. And they created this curve that explains the maturity of any new technology. And at first. It starts at the bottom at zero, then it goes up into the, this, what they call the top of the hype.

And this is where everyone thinks AI is going to take over the world. No one’s going to have jobs anymore. And all of a sudden we come into reality and we come back down because there’s negative press. We realize it doesn’t do everything we need. Maybe some people get it harmed them. Or the cars, the automatic cars driving crash into someone and we’re like, Oh no, the world’s falling out.

We go really far down and then we end up coming back up and we get to this plateau. And it takes a while generally for the entire population to get to that plateau. In healthcare, since we are [00:28:00] tip of the spear with technology, especially in radiology, primarily right now in radiology is leading the way.

We have to have early adopters who are willing to go along that curve and help pull us, pull the rest of the group further ahead. And that means we need to have people that are willing to invest in it to figure out how to do these things. And it’s the reason why I’ve developed the expertise or any expertise that I have.

It’s not because I am thinking about it or just doing research on it, it’s because we’re deploying it. We’re deploying it in a safe way. We’re figuring out what works, what doesn’t work. We’re going through that innovation sort of PDSA cycle to continuously iterate. And that’s not something that we always do in, at least in radiology and healthcare.

Nina: So many of our processes are very well formed. That they just they work or you just follow something that someone else is doing. You don’t have to continually iterate and put dollars into that iteration. And when you’re iterating on something, [00:29:00] it may fail. Like you probably an innovation cycle, you try 10 things and nine of them might fail.

So you have to get through that cycle quickly so that the 10th one that makes it through and you figure out how to do it. You can get benefit from and then you take those lessons and you bring them out to the world, which is what we’re trying to do because we’ve been investing.

Aleks: Which is what I love about your activities in all the societies, because educating people along the way is part of pulling everybody with you, like they don’t need to have the investment or the capacity or the expertise in house to be able to be part of your journey, which is what’s happening around us.

Like we’re like part of this journey and there are always this red light questions or hallucinations or whatever, like whatever the limitation of the technology is. Healthcare professional [00:30:00] is going to think, Oh, Okay, it has potential, but is there a risk that’s going to harm my patients?

And seeing, seeing radiology in general go through it gives everybody else okay, they did it. We can do it.

Aleks: And pathology is like logical follower because we are an image based specialty, right? Images are a little different. Our workflows are different. That’s okay. But the main thing, okay, we both look at images.

We can both look at those images remotely. How did they do it? Can we do it like that as well? You know what I’ve noticed that even though our specialties are so, are image based, like we don’t know about our workflows that, that well. And I think we know about our workflows more than other specialties know about their respective workflows. [00:31:00]

Aleks: A good book that I think we already talked about it as well, is that Deep Medicine by Eric Topol. Yes, because you told me you guys spoke at the same stages and you knew each other. And something I love about this book is basically he paints the picture off. A specialty of the workflow, a specialty and how AI and this one is already outdated 2019, right.

But he talks about radiology, he talks about pathology, he talks about cardiology. So it gives people perspective. Okay. How do these specialties work and where can they apply AI, which way of applying AI. They do. So that brings me to the question.

Aleks: How do you guys deploy your AI’s?

How do you make all the data available for the tools that you are using? What are the tools? How do you coordinate it all?

Nina: That’s yeah, there’s a lot there [00:32:00] and a lot we had to… 

Aleks: Feel free to pick whichever part you want to talk about first.

Nina: Not a problem. You know, when I was in graduate school in my specialty or my subspecialty within graduate school, cause I was in applied mathematics, but I was studying optimization theory.

And one of the things that I really learned then is that it’s too hard to take a very large problem and solve it on its own. The best way to solve a problem is to break it down into its individual components and look at the individual components and try to solve for that smaller piece and optimize to that which is the most important.

So in radiology, we’ve done that. We early on as AI started coming out, people were saying I’ve got a computer vision model that can look for lung nodules and I’ve got another computer vision model that can do, I don’t know, something else. All of it was random. There was no framework for it.

Nina: And we wanted to, we really needed to create a framework to think about what the potential [00:33:00] of AI would be and not just take what was available, but drive the market. And so we, what we did was we thought about what do we do in our workflow? You said, radiology and pathology. We know our workflow really well.

Nina: So we broke that down into what we call the life cycle of an exam. And it’s also the life cycle of a patient going through this process. So the first thing that happens. is a patient goes to a referring clinician and they say, I have this problem, I have this pain, I have something, and the referring clinician, 85 percent or more of the time, they’re ordering an imaging study to help.

So that’s step number one. They order a study. Then step number two is you’ve gotta schedule the patient, you’ve gotta schedule the scanner. Step number three, you scan and acquire those images. Step number four, after the images are acquired, it goes off to a system For us, it’s a pack system. We review those images and we open up the study through a work list.

So what is the order of that work list? Then it goes on to [00:34:00] how, when I open up the, how do I get information about the patient and the history? Then finally, I get to the point where I’m actually look at the images. There’ve been already, five steps before that, but yet everyone was concentrating in this one area of detection and diagnosis and quantification of things on the image.

And then there are steps after that because we have to then create a report. We’ve got to communicate that report. We have to do a follow-up. We do peer learning. So all of those steps, we separated out. And said, let’s think about how we can use AI in each one. And where is the value?

Nina: Because if you start with something that doesn’t have a massive amount of value or an ROI for people to pay for it, it’s not going to go very far.

And that’s why we’ve had some trouble in getting things started is no one has money in healthcare and they couldn’t figure out how do we afford to pay for these extra tools, whereas humans are actually pretty good at detecting things. You get an ROI for sure, but it’s maybe a 20 percent ROI.

Whereas in other components [00:35:00] of workflow, where we’re not good at it at all, like gathering information from multiple sources and putting that together, or remembering to do things, those are things computers are great at, the ROI on that is more like 90%. So it’s really important, I think, first, to look at your workflow, if you could split that up and think about the use cases to drive the market toward that end for what you really need.

So that, that’s number one. The number two, another big component of this, is that we don’t realize the full story is not the AI tool. The AI tool is evaluating your data, whatever data it is, and providing a result. But in order to do that, you’ve got to get your data to the AI, and then you’ve got to get the AI result back into your workflow in a way that you can use it.

And that is not easy.

Nina: There is a whole component of getting your data to the AI system That I call orchestration or data orchestration. And the AI, at least in radiology, [00:36:00] most of the AI systems out there, they don’t review every image on the study. Very unlike a radiologist, it’s unlike a human. So for example, a CAT scan or CT scan, maybe a CT scan of the abdomen and pelvis.

It has multiple different series. It has an initial scout view. It’s got an axial thin slice view. It’s got an axial thick slice view. It’s got coronal and sagittal views. It’s got bone images. It has maybe five or ten different series. And we want to send the right one to the AI. The AI is actually only looking at one of those.

So how do you make sure it’s getting the right series and reviewing the right images? Because if you send it the wrong ones, you are going to get a very bad output. The orchestration is actually really complex, and it’s something that as you guys are going into digital pathology, it’s really important to think about, how do you know what is in your data?

In order to send the right piece of data, you actually need the right image [00:37:00] of the right series for the right study, for the right patient, to the right AI, and you need to do it in the right amount of time before, before you’re interpreting that exam and in the right setting. All of these things have to happen, and the only way they can happen is if you know what’s in your images.

Now, we use DICOM to be able to help us determine that.

Nina: Our DICOM is a structure that says in general what are in the images, but it’s not perfect by far. And there are, you could probably drive a truck through components of it. So we have to create a whole nother system to manage. to make sure we’re getting the right images there.

And that’s what I call data orchestration. It’s actually quite difficult. I’ll pause there for a second because that’s a lot of information. 

Aleks: Yes. So the first thing about workflow. Oh my goodness. That’s a gold nugget. Because every especially, I don’t know in, in radiology, but in pathology, many of the players, defenders they do not, especially [00:38:00] like the algorithm and software vendors, they do not come from the healthcare background.

So you have people who like know the technology very well, and then think, Oh, it’s going to be amazing to apply it in healthcare. Where would it be amazing to apply it? Without knowing the workflow and I want to emphasize that, okay, we are both like, I’m a veterinary pathologist working in tox path, toxicologic pathology.

It’s so niche that even diagnostic veterinary pathologists don’t really know how I work. Even though we like took the same boards, have the same background, but we went like different specialty routes. And educationally are very close. Imagine people coming from a different place, they like, don’t know what the workflow is.

Yeah. And then, you pick something, like you said, many of the algorithms [00:39:00] were random. That’s how I feel about different things that, okay, it can help marginally here. It’s actually performing well. Let’s like make it into a tool. And. Let’s even try to get the regulatory approval and the question is, okay, where’s the ROI?

Yeah. Who’s paying for this? And then when nobody’s paying, then that was it was a nice exercise. What are we going to do with it? Fantastic. We have a proof of concept that cost a lot of money. So that’s my comment on your first point. And the second point and the data orchestration. Yes. This is because in the scientific world and all the publications coming out, you make your like little digital lab, you have the models, you train them, they are performing well, better, great results.

How are you going to take this model from the air of your cloud and put it somewhere? So one comment I read today in that in those abstracts is okay. [00:40:00] And you mentioned that the new architecture, it has to be lighter. It has to be easier to deploy because it’s going to go into a medical device, and.

Often is not like just this software component alone. It’s something that’s, I don’t know, does imaging, like a dermatoscope or the x-ray machine or whatever, like you want this AI to be deployed either in this machine or like in the workflow. How are you going to do this? So you say DICOM is not perfect.

We are just starting with DICOM.

Nina: Dicom, I mean it’s fantastic. It has helped us to get where we are today. But because the technology is so specific, if we wanted to send the whole study to an AI and have the AI interpret the entire study, if the AI had the capability of doing that, DICOM would be totally fine.

It tells us enough about what is in the image and it was something that brought us from a place of having no idea what was in the study until a human looked at it. [00:41:00] So it is an awesome step. It’s just now we need so much more detail. That we need to improve that. And part of the issue is that some of this information is coming from humans putting the information in.

So on a CT scanner, someone might have created a protocol. I’ve seen all kinds of crazy protocol names. There is one, a CTCS birthday special. How do you know what that is? And within that birthday special they’re naming the protocols. I’ve seen a series name that had a smiley face at the end of it, like that’s not a standardized thing.

And then that gets taken from the scanner and put into the DICOM. And then we don’t really know what that means. So we have to think about it at a much higher level now that the technology that we need is looking not just at the study level, but at the series level and the image level. It actually gets far more complicated.

So and that’s just the first part of getting the data to the AI. You also mentioned getting the data back. [00:42:00] Now once you send the data to the AI system, right now all we’re doing it is sending it back to our PAC system. But that is not necessarily the future.

Nina: In fact, that’s not how I work as a radiologist.

If I get a piece of data, if I’m looking at, let’s say, a trauma study, a patient gets in a motor vehicle collision, and I have a CT scan of the chest, the abdomen, and pelvis, and I’m looking for a different pathology, and one of the first things I see is that they’ve got a fracture of the first rib. When I see a fracture of the first rib, that tells me, because of the education and experience I’ve had, that is a very, look into all kinds of things, because it’s a very high speed injury, so I look for, yeah of course I look for a pneumothorax in the lungs and contusion, but you also look for vascular injuries, you look for vertebral fractures, you look for other things.

That you might not otherwise and right now the AI is not doing that. The AI is just saying there’s a rib fracture, but what if [00:43:00] the result of that AI sent it to another AI that was looking for these vertebral compression fractures or looking for these vascular injuries and said, guess what AI, I found a first rib fracture.

You should decrease your threshold for calling these other things because we know that they’re going to be more likely in this setting and none of that coordination is happening now. So there’s a whole sort of in-between sort of an orchestration phase two that could happen even before that result comes back to the physician.

Aleks: Do you think that LLMs can help? And I’m specifically referring to a concept I recently learned about the concept of agents. At that, where you have a virtual programmer and you can couple them all together and tell one, okay, when this, then go to the other one, then go to the other one and then come back with your, your information.

That still seems to me like super, they’re doing it in non-medical space, but that kind of, it brought this concept [00:44:00] to me when you were talking about data orchestration, that’s yeah, how do you, break down the workflow even further in the diagnostic realm where you just make those all those decision trees that we saw in medical or veterinary books.

Okay, if this, then that, how do you program it into the AI? And make the AI, you probably would need an AI for every single of those components. Or maybe, I don’t know, that’s what comes to my mind when we talk about it.

Nina: It’s a fantastic insight, really perfect analogy. So agents, what are agents?

Agents are essentially, you can think of them as a brain. They’re the smart component that figures out from a question that you ask, right? Right now, we ask. ChatGPT, a question we say, Oh, I want to understand a world population [00:45:00] and what is happening and in this area of the world. So when you ask it a question, it has to break down that question into individual components.

Just like we talked about when you break down your whole specialty into its individual components because it’s easier to solve, breaks it down into individual components. And then it figures out where do I need to go to get the information for these different parts? So step one, I need to go to the internet and grab some information about population.

I need to go to another site to figure out something else. And there’s a brain behind that says, I’ve got these tools in my tool belt. I have this question and I have to take these steps. How am I going to go do that? And that’s what an agent does. It’s a large language model that can break down things into its multiple different steps.

And then go out into its ecosystem, whatever it has access to, the databases, the internet, and it will grab those things. And as it grabs those things, it brings the information back to a central large language model [00:46:00] that then puts it all back together and gives you an answer. And in the setting of healthcare.

If you can imagine even going beyond what we’re talking about here, where you would take the information from one AI and give it to another, if you imagine that you have access to the electronic health record if you have access to a database of a pathology. You have access to information about the latest and greatest journal articles because no large language model is being constantly updated.

The information ChatGPT has is from what, 2021? So it doesn’t have anything, any information. So you need access to that. So think about all of these databases as tools. And then in the center of that, you’ve got an agent that when it gets a question, it can reach out and grab one of those tools and then combine all of that information together and give you an answer.

The robustness of that is based on what you have access to in your ecosystem. And the more tools you have in your tool belt [00:47:00] or these things that you have that the agent can grab, the better the result you’re going to get. And that will enable us to combine all of the information that we have in healthcare in a super smart way.

It’s the thing I’m absolutely most excited about. AI is a tool within this. It’s the central brain as well. But it’s the ecosystem and the combination of all of these together that make us great. The analogy that I give is if you think about any person. A single person is only as good as that single person.

But if you bring together a team of people and that team is coordinated by a coach that knows how to get them to work well together, that team is going to outperform any person at any moment in time. And that’s what we want to do in healthcare. We want to bring our teammates, which are all of our different databases, the AI models, all of our sources of information together under one roof.

And you could have some smart brain in the middle that accesses it and then provides us that information back to where we need to get it when we need to get it. [00:48:00] 

Aleks: This is amazing. And so I, of course, had a bunch of other questions that I will not ask because in the sake of time we will have to wrap up this episode, but I wanted to ask you where like, where can people learn more from you?

Aleks: Do you have a platform where you share those insights, informations, conferences you go to like let’s spread this information.

Nina: Yeah, and this is how I learned about AI. Besides, like, when I didn’t have the opportunity to test everything, I go out there and I go to conferences. So the Society of Imaging Informatics in Medicine is a wonderful conference, they have an annual meeting.

Aleks: And I am going to go next year. 

Nina: Yeah, and they also have another meeting coming up in October, I think it’s October 20th to 22nd. It’s going to be at Boston University and that’s called their CMIMI Conference, which is more for research [00:49:00] but about AI computer vision or computer imaging in medicine.

There are other ones, of course, the RSNA. 

Aleks: Are you going? 

Nina: I’m actually doing the keynote at that.

Aleks: See everybody who is interested in this and wants to listen to this keynote, that’s the conference to go. And let’s see if I can organize my schedule to go as well. That would be great.

Nina: And if you can’t, there’s RSNA that’s coming up after that.

I’m giving a keynote there as well. It’s not just about me though. I have some information that there are so many people out there that can provide robust information, so go do meetings, ACR SIM, RSNA, those are the ones in the U. S. anyway follow people on social media. A lot of people post great things.

Aleks: Are you on social media? Where are you? 

Nina: I am. 

Aleks: Can we follow you? 

Nina: Search for me @radkottler, R A D K O T T L E R, that’s my name. You can also search for me. I have a bunch of recordings or things that are recorded that are probably available on YouTube. 

Aleks: Yeah, I want to like the recordings [00:50:00] available to people because you have, I mean you have the insights, so when I listen to you, I know that you have very much thought through how to teach it.

It’s not just that you have the knowledge but you have thought through your analogies. You have. You have broken it down into components that I would like to share with my digital pathology trailblazers to make it understandable. I mean, I didn’t, so the paper, the attention is all you need. It’s okay, everybody knows the title, but how does a transformer really work?

How, so I could understand it easily for text. Okay. Because, you can look from above and you have grammar. How does it work for images? And you just have this analogy. Okay. It goes like piece by piece and figures out the context. That’s how the transformer looks. And they will give this to my digital pathology trailblazers.

If [00:51:00] they want to binge watch your content, where do they go?

Nina: Absolutely. I’ll be happy to help you. I haven’t gathered it all in one location, but we can work on that.

Aleks: Do you have a channel? Where do you have them? I can put the site together if you want them in one place so that I can share it with my people.

Nina: Yeah, sure. I have a bunch at one point, I have a, Instagram account that I created, an @radkottler – Instagram account, and in Instagram you can’t put the link there, but you can type the link, you can connect to it. So I typed a whole bunch in there, but I stopped after a while. We could start there and, and then I, I’m sure that the ones that are coming up for CMIMI, which is I think an hour long and another 45 minutes at RSNA, those are going to be at the end of this year, they’ll probably be recorded.

You could get a virtual access to those as well. We probably can’t take them and put them on a website, but if anyone signs up, [00:52:00] you can watch those.

Aleks: See, let’s do this then. Okay. So you say Instagram would be the best place to start for any recordings and information. 

Nina: And then the other is just search Kottler AI videos on Google.

Okay. There’ll probably be a bunch that will come up there. 

Aleks: Okay, my next step is going to be to do that, exactly that, and see if I can put them on one page so that I can, if they’re already available, let’s make it a repository. I think my digital pathology trailblazers can benefit a lot from the way you’re teaching.

Everybody can benefit from the knowledge, but I think there is a specific way of teaching those complex topics that you have mastered. So thank you so much. I’ve learned a lot during this episode. And I will figure out how to go to, to CMIMI and maybe we can meet in person there. And then I can ask you the questions [00:53:00] that are left on my long question list.

I totally underestimate what it means to record a podcast episode with Dr. Nina Kottler.

Nina: I happen to be very chatty. I apologize for that. Hopefully it was helpful. 

Aleks: No, don’t. It was amazing. Thank you so much. I wish you a fantastic day and I hope to see you in real life in October. 

Nina: Wonderful. Thank you all.

Aleks: Thank you so much for staying till the end. It means you are a true digital pathology trailblazer. I found this conversation fascinating. We ran a little bit out of time when we were recording, but I think we’re going to meet again. I think there is not enough conversation between radiology and pathology.

Whenever I get another chance to meet with Nina, I’m going to discuss more in depth all things AI and what we can learn from radiology. Although radiology is not pathology, so not everything is going to be applicable. But [00:54:00] if you would like to hear an opinion of another radiologist on digitization, going digital with the whole specialty, I’m going to link to the episode with Todd Randolph.

So go ahead, listen to this one and I talk to you in the next episode.