[00:01:36] Aleksandra Zuraw: Today, my guest is Heather Couture. Heather is a machine learning, artificial intelligence image analysis specialist at her own company, Pixel Scientia Labs. And I came across Heather on social media, on LinkedIn, where I saw her weekly. I don’t know, Heather, how often do you publish the literature reviews?
[00:01:57] Heather Couture: LinkedIn, I do a few days a week and newsletters, I do twice a month.
[00:02:03] Aleksandra: Mm-hmm [affirmative], so pretty often. There are literature reviews and the literature reviews were about machine learning and artificial intelligence used in pathology. So this, obviously, caught my attention and I’m a great fan of these reviews. They provide great value to, I think, the whole digital pathology community and Heather was kind enough to join me on the podcast today. Welcome, Heather. How are you today?
[00:02:29] Heather: I’m good. Thanks. Thanks for having me.
[00:02:31] Aleksandra: Let’s start with you. Let’s tell the listeners about you. What’s your background and just a little bit about yourself?
[00:02:38] Heather: Okay. My background is in computer science and within that, computer vision, machine learning, deep learning. I’ve been working in that field for about 16 years now. I did my PhD research on machine learning algorithms for pathology, and particular predicting molecular properties to breast cancer for me, H&E. So through that and through working with my own company, I have about a decade of experience in applying machine learning to pathology images.
[00:03:08] Aleksandra: So you are a computer vision specialist and your background is in computer vision. How did it happen that your research work was in digital pathology?
[00:03:20] Heather: So I worked for five years before going back to do a PhD. At the beginning of my degree, I was looking for an interesting application of computer vision. I’d already been working in that field for probably seven, eight years or so, but for me, it’s about the application of it. What important problems can I solve by using computer vision machine learning? And I was already in the Raleigh, North Carolina area, so I had applied to the PhD program at the University of North Carolina where they have a good medical imaging program. So that’s the area I was looking at for an interesting application and they do a lot of radiology image analysis, but the area that I actually fell into and found much more interesting is pathology, dealing with these massive high resolution images and the complexities within them and trying to automate the analysis for that.
[00:04:14] Aleksandra: What did you do before digital pathology?
[00:04:17] Heather: So I worked in a couple different application area areas of computer vision. One was for a company that looked at video, movies and TV shows and did analysis of that to recognize faces, recognize scenes, and to be able to search and categorize it. But even before that, my masters and an internship was actually more related to the space applications, so looking at craters and rocks on Mars and trying to detect and classify them, some from images that could be from a future Mars Rover, but also from satellite imagery of Mars. And again, high resolution imagery and looking at details within them, in this case, it was rocks and craters, but for me, these two satellite images of Mars and pathology images are certainly wildly different applications. But for me, they’re actually quite similar in that it’s the experts who understand the details and the algorithms that myself, as somewhat of an outsider, the algorithms are very much the same and a lot of the challenges in dealing with these images are the same.
[00:05:23] Aleksandra: Even though like you say, it sounds that they’re vastly different, it’s interesting to know that from computer vision perspective, the tasks are not really that different. So this is what you do at Pixel Scientia Labs, your company. How did you come up with the idea to create Pixel Scientia Labs?
[00:05:49] Heather: I originally started this company during my PhD because I was doing contract work on the side during my degree, so working for a couple different companies in the area. Some of it’s related to medical imaging and some not. But when I graduated, I started doing this full-time as a consultancy. Building my own company gives me the flexibility that I want, being able to work from home and raise a family, but also work on applications that matter and be to choose what those applications are. So I have a great deal of experience in pathology applications, so that’s one core area, but I bring the latest research in machine learning and computer vision and apply it to these problems that matter and help companies who are tackling these important problems to make better use of their algorithms, to accelerate their projects and to reach the important goals they’re trying to achieve. So I’m in the process of trying to scale this vision to build this into a company that stretches beyond just myself.
[00:06:48] Aleksandra: I’ve seen you have open positions, so if anybody’s interested in working for Pixel Scientia Labs, there are open positions on the website and I’m going to link to the website in the show notes. So you say the space images, so that’s the climate change part of your business and pathology. What is the proportion of the projects currently that you tackle for both of these areas? How much pathology? How much climate change?
[00:07:16] It’s mostly pathology. I’ve got one climate change remote sensing project going on right now. I would love to do more on climate change in the future, but my priority for the short term is to build the pathology side of
[00:07:29] this business.
[00:07:31] Do you think there is more assistance from machine learning and computer vision specialists needed in pathology than in climate change or why do you focus on pathology?
[00:07:42] Heather: Both domains really do need the machine learning expertise. It’s certainly not the only skill that’s needed in fighting cancer and climate change, but it’s one that can solve some important problems. The reason I focus more on pathology in the beginning is because that’s where the majority of my expertise is, having worked in this area for longer.
[00:08:01] Aleksandra: And what was your PhD about? What was your first machine learning pathology work about? Can you tell a bit more details and how it was?
[00:08:10] Heather: I worked on a couple different projects, mostly related to H&E images and trying to predict molecular properties. So the first one was related to melanoma and we were trying to predict mutation status of melanoma from H&E and this was just around the time deep learning was coming on the scene, so it wasn’t widely used yet. We were using more are the traditional machine learning tools, what we call handcrafted features, so that we would segment out individual cells and nuclei and characterize their shape and appearance and try to predict mutation status. And we couldn’t do it at the time. The tools we had weren’t accomplishing that.
[00:08:48] But the next project was related to breast cancer and we were able to get a larger data set and slowly deep learning was coming on the scene, the toolkits were becoming available, so it was much easier to apply those algorithms. And my main dissertation research was predicting genomic subtype and receptor status and grade of H&E breast cancer images. So we were able to show that the same algorithms can predict grade, something that a pathologist is an expert at, can also predict molecular properties that a pathologist cannot see from H&E because they require other analysis for. So at the time, this was one of the early projects in this work, in this area. We would’ve started this project 2014 or so and then I graduated in 2018.
[00:09:36] Aleksandra: It’s funny because these dates, they’re super recent dates and so many things have changed. And now every other paper you look at is trying to predict that for one cancer entity or another. And so recently, it was such a cutting edge thing and now you have companies trying to do this commercially.
[00:09:59] Heather: Yeah.
[00:09:59] Aleksandra: When you work in this pathology world with pathologists and life scientists, what are the most common misconceptions about artificial intelligence, machine learning and computer vision in general, in this environment, in the pathology world?
[00:10:18] Heather: So I’m probably more connected with the machine learning community, so I might answer this question a little bit in reverse, which is not quite what you’re asking. But coming from this side of it, one of the common misconceptions of machine learning practitioners and across medical imaging, not just in pathology, is that they can develop a useful solution on their own, which is definitely not true. If you take COVID as an example, when the pandemic started, there were a lot of machine learning practitioners developing solutions that could take x-ray or CT images and try and predict does this patient have COVID versus pneumonia. But a lot of them didn’t work with domain experts and a lot of this research has more recently been revealed that these solutions aren’t necessarily useful. One of the problems is just the data sets they were using. They were an ad hoc compilation of different data sets.
[00:11:12] Some of them, the diagnosis hadn’t been confirmed. In some cases, they’re throwing in images of children in a data set with adults, things like that, that if you included domain expert in planning a project, these types of things would be caught much earlier on because you want to be sure your project has a clinical use case, that you’re using the appropriate dataset, that you can identify any potential biases. So it’s very much a case where you need teamwork and I’m pretty sure that pathologists understand this. You guys are the experts. But on the machine learning side, I don’t always see the same awareness of this.
[00:11:53] Aleksandra: Yes and no, I think. What I have seen on our, the pathology side, what the misconceptions would be is applying the wrong computer vision task to your problem. What do I mean by that? Let’s say there is the instant segmentation problem and the semantic segmentation, and we’re not going to go into details what that is, but basically one would be good for regions and the other one would be good for objects. And mixing these things up and trying to apply different computer vision task to a pathology problem, then it should be, at least from my experience so far.
[00:12:39] Heather: Yeah. No, I definitely see that one as well, if I was talking to people with a pathology or medical background that you have to explain the terminology along the way, because machine learning people, it might seem obvious to them. It’s definitely not for everybody, nor should it be. We have to agree on the terminology and be able to speak the same language in order to work through a project like that.
[00:13:03] Aleksandra: That’s why I think the main experts from both sides, regardless from which side you tackle of this question, that’s something that has to be there for successful project.
[00:13:15] Heather: Yeah.
[00:13:16] Aleksandra: So what’s your mission at Pixel Scientia Labs and why is your mission important? Why are you doing it?
[00:13:24] Heather: Our mission is to fight cancer and climate change with AI. On the cancer side, cancer is one of the leading causes of death worldwide and it actually encompasses many different diseases. So the goal is to improve patient care with better diagnostics, better treatment selection, maybe better treatments, and also better understand the disease mechanisms along the way. On the climate change sides, I believe that climate change is the largest challenger our world faces today. There’s many different technologies that are needed to mitigate and adapt to it and it’s certainly going to take more than just machine learning, but machine learning can be a part of the solution. And so I think that both of these are really important things to tackle.
[00:14:07] Aleksandra: Do you think those problems can even be approached without machine learning and artificial intelligence at the moment, or is this a crucial component and without it, we’re just not going to be successful?
[00:14:22] Heather: I think some aspects of it can certainly be tackled without machine learning. It doesn’t have to be the only solution, but it can accelerate solutions. It can make analysis of images more efficient, less error prone, less subjective, different things like that. And in some cases, it can provide additional information that even an expert, let’s say a pathologist looking at H&E images, would talk about up being able to predict molecular properties of that. Well, it’s a way to screen for those, whereas the alternative is expensive and time consuming molecular analyses. There’s many different ways it can be part of the solution, but certainly only part of a much larger, more complex solution for these.
[00:15:11] Aleksandra: Do you have an example where you were not able to solve something without AI? Where only when you’ve got access to AI in the sense of deep learning, AI is a broader term than just deep learning, but let’s say were you not able to solve in this domain without deep learning?
[00:15:33] Heather: One example is the melanoma project that I mentioned earlier. The first project I was on during my PhD research and trying to use traditional handcrafted features to predict mutations in melanoma and the analysis we did with the features we had and the data we had, we couldn’t do it. But then you fast forward to last year and I read the deep learning paper where they are successfully doing the same analysis that we tried to do, but with deep learning and they’re successful. So I’m seeing a lot of that come up now, challenges that couldn’t be solved even for five years ago ,now suddenly are with large enough data sets and deep learning.
[00:16:19] Aleksandra: You mentioned that was your research, the prediction of mutations from H and E__ images. I assume that’s part of your offer, is that correct?
[00:16:30] Heather: Yeah, it’s part of it. I work with teams who are trying to improve their image analysis using machine learning for pathology or remote sensing images. So it’s not specific to any application. It’s more broadly with these types of images, with pathology images in particular, that can be H&E, IHC, immunofluorescence or even other modalities. The commonality is that it is an image of disease and through collecting a sufficiently large training set, you can train a machine learning model to predict some property of it. So I work with companies who are trying to improve the machine learning aspect of their projects.
[00:17:13] Aleksandra: So machine learning, image analysis, just to have that better for whatever the end goal or the end points are.
[00:17:21] Heather: Yeah. The end points definitely vary. A lot of the challenges along the way are quite similar across the companies I work with though, things like nuclei detection and classification, mitosis detection, segmenting different tissue types, stain normalization, and other techniques to enable a model to generalize images from a different scanner. These things come into a lot of different projects, even if the types images that are input or the goal of the analysis vary from project to project.
[00:17:54] Aleksandra: How do you work on those projects? How does it work? Somebody who has a problem in computer vision and image analysis, they approach you. Why would they approach you? Do they use some tools and just want your expertise as the machine learning expert into the project or do they want to have this image analysis done by you? What are the different frameworks you operate within?
[00:18:23] Heather: So I don’t provide an end to end solution. I work with the teams that my clients I already have, so they must have some kind of software team in house. It might be that they have data scientists and machine learning engineers, or might just be that they have data people and they really need help with the machine learning analysis. So within those setups, I work in either advisory or collaborative relationships. So I might just be guiding them with the high level direction of the project, providing pointers to the latest research, identifying a different path they should take or different analysis they should do or on other projects, I help with the implementation. So I might be helping them with a proof of concept or some kind of data results analysis. But in a either setting, I’m integrated with their team and their in-house team are doing the majority of the work.
[00:19:17] Aleksandra: I think it’s an interesting value proposition that is not really that common in the industry yet, because usually you have, like you say, you are not providing an end to end solution, which is the goal of most of the players in the market. They either have a tool that can be used for deep learning or some kind of product or solution that is end to end, and they want the client to use it from the beginning till the end, whereas your value proposition is kind of tool agnostic.
[00:19:55] Heather: Right.
[00:19:55] Aleksandra: That’s one of the things that differentiates you from the competition at the first glance, from what I have read and what I have seen you doing. Can you tell me a little more about it?
[00:20:08] Heather: Yeah. No, that’s definitely the case and one reason Pixel Scientia is still a very small company, so we don’t have the capacity to provide an end to end solution, but the other part of it is more related to impact. I want to work on projects that can make a difference, and those projects tend to be an interdisciplinary collaboration in a large team. So you need players from different fields, different expertise to all come together. It’s the pathology expertise and in some cases, genomics or epidemiology can come into play. You need the software experts, the data engineering experts, machine learning, and together that can make a successful project. So I try to work with those teams that are doing the important work and help make their machine learning algorithms more effective and more successful.
[00:21:04] Aleksandra: And can you tell me about something that was not so successful, some experience that it was a failure and you had a great lesson learned out of this failure?
[00:21:15] Heather: Yeah. For that one, the one I keep coming back to is that the melanoma project where we couldn’t do with deep learning. That’s certainly not the only time a project can fail, but that’s one example that I’ve seen it firsthand and then seen it succeed later on with deep learning technology. The other one is just related to data sets. If you can’t collect enough data and annotate it properly and have reliable annotations, that makes it harder to train a model. So it also means that if you’re in a small data regime, deep learning is not always the best solution. It might be that once you’re able to collect more data, it’ll become a more viable solution, but you also have to look beyond it in some use cases.
[00:22:04] Aleksandra: I think it’s very important to remember that you cannot really throw deep learning at everything, because there is this data component, size of data set, and quality of your data set that has to be taken into account. I think it is a little bit advertised as the solution for everything, at the moment, which, like you mentioned, it’s not.
[00:22:28] Heather: Right.
[00:22:29] Aleksandra: So one more question about this. So you work with experts that annotate data. How do you deal with the fact that they’re not annotated consistently or the way they’re annotated is not the correct way for the problem you’re trying to solve?
[00:22:48] Heather: It depends on the project because part of it depends on the annotation needs. Are they delineating individual structures in an image or are they clicking and saying this points associated with this annotation? So it depends on that and it depends on the downstream tasks. Definitely find that annotation and data cleaning is an iterative process, so it’s not that it gets done once and you stick with that. That would be great if it was. It would make development of these solutions a whole lot easier. But in a lot of cases, it’s very iterative. So you develop a first pass model with that dataset, you look at the output and some of the predictions that made aren’t going to agree with those labels that you have and so you need to investigate why. Is it because the model didn’t perform well in those cases? Is it because it doesn’t have enough training data of those types of examples?
[00:23:42] Is it that the label was perhaps incorrect? And in that case, why was it incorrect? It’s not necessarily that the expert who labeled it did a bad job. Maybe they didn’t have all the information or maybe the modality that you’re labeling from is different than the one that you’re training your model on. So it could be a different imaging modality, it could be that these are serial sections and so there’s another dimension that changes where those labels should be. But because of that, it’s definitely an iterative and data cleaning process, in most cases, to get these solutions working.
[00:24:18] Aleksandra: This is something I had to learn from. So I’ve been doing annotations for different purposes on pathology images since I started working in a commercial job after my PhD, which was 2016. And the way we were doing annotations for classical computer vision, for classical image analysis, very much differs from what you have to do for deep learning. And for the previous approach, we were able to generate trainings and say exactly how to annotate, what to annotate, because those annotations were mostly region of interest where the algorithms were applied. Whereas now for supervised deep learning, this is the examples the algorithm or the model learns from. So I definitely had to learn that this is an iterative process. It’s okay to not know everything at the beginning. And the goal is to have a better annotation pool and strategy at the end of the project, so that you have the best outcome.
[00:25:23] Heather: Right.
[00:25:24] Aleksandra: That was my personal experience with [inaudible] . So is there anything you wish you had known when you started out that would’ve protected you from some of the failures or accelerated your work?
[00:25:40] Heather: For me, it was probably less about the technical side, because you know the technical side and what kind of solutions develop, that I very much learn along the way and we don’t know all the answers now. Research keeps advancing very rapidly and I learn more every day. For me, the things I wish I knew were more on the business side, now that I’m running my own company, learning about sales and marketing and hiring and all those aspects of trying to run my own business. Those stand above the technical challenges at the moment.
[00:26:15] Aleksandra: These are, I would say, maybe something that’s common for scientists starting companies, because I think we’re not exposed to this problem in life at all, I think, before we actually start to offer our services.
[00:26:35] Heather: Definitely new for me. I knew there would be challenges, but I didn’t necessarily know what things I needed to learn or how to learn them or any of that. So it’s been a lot of different ways of gaining information over the last couple years, coaching programs, different things like that to teach me the skills that I was lacking.
[00:26:55] Aleksandra: Do you like it or do you feel like, oh, I wish I could just outsource it? How do you feel about it?
[00:27:01] Heather: I like some of it, but there’s other aspects, as I grow this business, I certainly help hope to outsource. The marketing side, I actually quite like it because I read papers, I write LinkedIn posts about them, I write newsletters and articles about them and that I quite enjoy and it serves the purpose of marketing my business. But the other side of it is things like sales, getting new business and having conversations in order to build my business. To me, that’s more of a challenge for myself.
[00:27:31] Aleksandra: I love this literature review series of yours on LinkedIn and this is how I got to know you, got to see you on LinkedIn. So definitely this marketing strategy worked and I am a fan and I will provide the link to that. But through this, like you say, you like doing it, you were doing it for your research anyway. So you’re basically up to date on what are the newest developments in the field of machine learning and pathology or at the intersection of those fields. What do you think is the future of pathology or what direction is it going? We cannot predict future, but what are the trends that you’re seeing are going to be next.
[00:28:12] Heather: In digital pathology more broadly, certainly AI coming on the scene and being used to assist pathologists in the field. AI can’t replace the expertise and high level knowledge of pathologists. They can certainly augment it, take time for these clinical applications to come through, but it’s already starting, Paige’s prostate software that helps locate cancer in whole slide images received de novo authorization from the FDA. That’s one example. There will be many more AI applications to come, but even within the AI area, the major one that I’ve been seeing and was a part of the early part stages of and seems to be accelerating is predicting molecular properties of tumors from H&E. We talk talked about mutation status earlier, but it can also be a receptor status, genomic subtypes, presence of a virus across many different cancer types and many different biomarkers. So that area seems to be exploding. Don’t know where it’s going to head past that, but there’s a lot more use cases for it and proven studies than there were a couple of years ago.
[00:29:23] Aleksandra: Definitely. The use cases that I keep hearing about is that this is so much less expensive that the current way of evaluating those parameters, that you can basically do it from the image, which saves lot of money, which gives access to a lot more people to do this. It’s not going to be so cost restrictive anymore.
[00:29:47] Heather: Yeah. It can also be faster to complete and as it doesn’t require using additional tissue. It’s just that original H&E image that is ubiquitous anyway. So just additional analysis based on that, it won’t necessarily have the same accuracy that the molecular analyses would, but it can be a screening mechanism for it. So from that, decide which patients require more additional analysis.
[00:30:17] Aleksandra: That’s important to say as well, that it may not have the same accuracy. Because I think to what we are comparing now, all those solutions, is to those other methods and due to method limitations, it may never reach that level. And does it then mean, okay, it’s not good enough or does it mean, okay, the purpose of this, like you say, would be a screening tool and not a diagnostic tool per se?
[00:30:42] Heather: The ground truth in training these methods is that molecular analyses. So because that’s the ground truth we’re trying to compare to, the deep learning version, operating on H&E will never match that exactly, if that’s the goal. But it may come closer and it may be that in some cases, it’s good enough for screening purposes. In other cases, it might not be, or maybe we just haven’t collected enough data yet to know.
[00:31:12] Aleksandra: That’s definitely something that will change the everyday pathology practice. So you say these tools are faster, but speaking of faster, from another perspective, from the technology and the data transmission perspective, those pathology images, like you mentioned at the beginning, are so much larger than radiology or any other static images. Is there something you see on the technology side that’s going to help with transmitting them or handling them better in an easier way? What I see now as a limiting factor is the speed of transferring those images. So often it’s easier to send a hard drive with images uploaded to it, then send the image via the network. Is there anything you’re seeing that may solve this challenge or alleviate?
[00:32:10] Heather: Yeah, that one’s a bit harder. I don’t have my eyes on that one as much. One solution would be to do the analysis locally, to be able to train an algorithm with data that you’ve accumulated from different institutions, but be able to deploy it locally so you don’t have the data transfer components.
[00:32:28] Aleksandra: So like in a federated learning manner.
[00:32:31] Heather: Yeah. You would then need the hardware locally in your lab, the GPU and compute capability there, which isn’t necessarily an easier problem to solve because the hardware is expensive to buy and to maintain and all that. It may vary from setting to setting as to which is more feasible to run it locally versus in the cloud. Compression is another one. I don’t know whether there’s any advancements there, but compression without losing the quality of the images would be the goal there that could help the transfer speed.
[00:33:07] Aleksandra: I think, in general, the research is more focused on the quality of the image analysis than on the technical side and people who are doing this research already have the capabilities. But I think to deploy it in underserved areas or to really diminish the cost, the hardware or processing technology would have to cut up with the quality of the image analysis that we are achieving.
[00:33:33] Heather: Yep. There’s definitely still challenges there to be tackled.
[00:33:39] Aleksandra: I’m staying positive. That was the same with AI was conceptually started in the fifties and it’s only now that we’re actually using it for these kind of tests that we were mentioning. So let’s just stay positive.
[00:33:52] Heather: Yes.
[00:33:53] Aleksandra: There are many brilliant people that are working on it and at some point, we’re going to get access to this technology.
[00:34:01] Heather: And once again, it comes back to experts in different areas. You’re an expert in pathology. I’m an expert in machine learning. We need people who can do data and hardware and all that to help solve this problem too.
[00:34:15] Aleksandra: Definitely a call to action to those from these areas that are listening to us right now, please get engaged in this field because we need you. We very much need you. Before we go, where can the listeners find you on the internet?
[00:34:30] Heather: The first place is my website, pixelscientia.com, that’s P-I-X-E-L S-C-I-E-N-T-I-A.com or on LinkedIn. You can find me as Heather D. Couture. Always happy to connect with new people.
[00:34:45] Aleksandra: Okay. And I’m going to be putting all those links in the show notes, so you’ll be able to click there. Thank you so much, Heather, for this time and for being a guest on my podcast.
[00:34:58] Heather: Thanks for having me. I had a great time chatting.
[00:35:00] Aleksandra: Have a great day.
[00:35:01] Heather: Thanks, you too.
