How Integration Helps to Adopt Digital Pathology

Hello and welcome to my talk, “How Integration Helps to Adopt Digital Pathology and Pathology AI.” I'm Professor Dr. Peter Schüffler from the Technical University of Munich. And I'm also co-founder of Paige. 

Our pathology department in Munich is located in the city center, as you can see on the right side, and it consists of around 100 people and 20 of them are pathologists. I'm a computer scientist and located in this building, making the bridge and filling the gap between computer science and pathology. Of course, our computational pathology endeavor is embedded in the whole institution, including the whole Technical University; the clinicum, and the pathology department, the computational center, translatum, computer science department, and the new Munich Data Science Institute.

In pathology, the task is to investigate a tissue of the human body to find specific diseases. Pathologists get the tissue from the surgery, for example, and embed this into paraffin blocks, into wax to say, to conserve it over 10 years. We conserve our blocks almost forever, so at least for 10 years. From these paraffin blocks, small slices are cut with a sharp knife and transferred to glass plates, glass slides. These glass slides with the tissue are then stained with chemicals to reveal the tissue and make it visible to the human eye because it's so thin without the staining it is of course transparent, but you want to see something. This here are very common and famous pathology slides, which are usually reddish or bluish with the hematoxylin and eosin staining.

And these glass lines are then investigated by pathologists under the light microscope at very high magnification. This is an example of a skin biopsy and the pathologist would zoom in into that image at a very high magnification to see the morphological structure and the cellular structure of the tissue.

The task now would be to find lesions and alterations of those cells within the whole slide, which might describe a disease. You can already imagine that this is, of course a very cumbersome task for pathologists. This is then the workflow for pathology; you look at the tissue under the microscope and you write up what you find in the pathology report for the patient. This is already very important that for every individual patient there exists a pathology report.

This is also important when it comes to learning, because we can use this information from these pathology reports to train algorithms for pathology AI. Unfortunately, today it's not that easy anymore. It's not a simple workflow. According to the new techniques in pathology, for example, molecular pathology and immunohistochemistry, the data load for a particular patient increases over time. It's getting more data. For example, nowadays we have special staining, we have sequencing data, we have metabolomics, maybe also FISH fluorescence imaging and so on.

For diseases, it can be very high data load pathologists have to look through. But, they don't have to do this alone, especially for difficult cases where you produce all these different data and you can also consult a tumor board or also a colleague for a second opinion. You can, as a whole team, discuss this patient in the case to get a conclusion for the treatment.

At the same time, we can observe over the last 15 years, as shown here on the left side, that the number of cases per pathologists is increasing steadily. By, for example, the number of physicians as shown here on the right side in pathology is decreasing and this is different to radiology, for example with the number of physicians is increasing.

This is not only a problem in the United States, we also notice from, for example, from China and also from Europe, for example in Germany. Here is a nice review paper which shows that the number of pathologists in in in Germany is also particularly low. Here on the left side you see the number of inhabitants per pathologists plotted in whole Europe. In particular in Germany, it has the second last place. Only in Poland, pathologists have to deal with more inhabitants. 

This leads to these workloads that you can sometimes see for pathologists. For example, here is an image where pathologist piled up all these cases she had to process within one week. Every white paper sheet is one work day for one week.

In this scenario, you can easily imagine that AI support for pathologists would be very helpful, right? Imagine you have AI which can also access all these different data modalities and make an interpretation of those data already and send this to the pathologist. This would be very helpful in the clinical setting. To train those algorithms, which are really helpful for pathologists in the clinical setting, we need access to all these large data sets that pathology departments usually inhibit, right? 

We call this dark data and at the Technical University in Munich we have our archives, our, our shelves full of these thousands of glass slides from these patients of collected over the last 10 years and even longer. What we are doing now is also digitizing this archive to produce a large data set which we can use for learning larger algorithms. For example, here are these two Leica scanners GT450 DX which we are currently using.

In the future, if you imagine how a pathology department is built up, you would see in the beginning, in the big basement, there is the wet laboratory where the tissue is processed. As I mentioned in the beginning and already here you could sign out patients for example, assign out cases but very analogously. Even the patient handling would be very analog.

In the second row there's the pathology informatics which uses for example barcodes for slides, uses a system for patient electronic health records, or a laboratory information system. In this scenario, and this is actually where most pathology departments nowadays are signing out a case is still analog.

Writing the pathology report is done on the computer. On top of this, there's the digital pathology where also the data itself are digitized. This means to scan the glass lights, making also quality control organize the glass lights in databases, store them in databases, and also for example, reconstruct the infrastructure of the department such that they can handle all these data streams, for example. Then pathologists could sign out on a digital computer screen. Alternatively, of course, also on a microscope.

This digital pathology itself is already very valuable. For example, it shows benefits in six categories, which are listed here. In the workflow, you could simultaneously access slides. They are not blocked if you order them and have them physically on your desk. You could faster search and retrieve cases, for example, prior cases of a patient. You could think of home office or remote working of pathologists, which is especially helpful in the current pandemic situation. You could think of easier collaboration and consultation if you can simply share digital slides via a link. Even teaching could also benefit from digital pathology if you think of courses and certificates. We can also think of digital tools and AI which can access the digital data and research where we can find new biomarkers, new quantifications, new patterns and features in these digital images. 

Digital pathology is not that easy as just simply putting one or two scanners in your lab. It also has to nicely integrate in your current existing workflow and the most important one being the laboratory information system, the LIS. And most LIS are very old, 10 or 20 years old, and then they are not designed to also handle ingests and deal with digital image data. All these things have to be developed together with the vendors. In, for example, an academic hospital, you could also develop this by yourself, and this is what we did here at Sloan Kettering.

This is a picture of what we did at Memorial Sloan Kettering, where I was before I came to ToUM. We installed a digital pathology workflow including and homegrown viewer for digital slides and also and homegrown honus broker for translational research. This is what we call HoBBIT, which has access to the clinical data and also has procedures to anonymize the images itself, but also the pathology report. This is what you see here in the middle from these patient identified data to the clean data, so to say our anonymous data or de-identified data. Those could be used for example to put them on to computing cluster to build these large scale models.

And the MSK viewer, so the viewer is also a vendor agnostic, so it can communicate or it can also visualize images from different scanners. This integrates in all these different hospital applications. This is an illustration of this integration. We had many, many different hospital applications, not only different scanners, but then the also the LIS, molecular pathology application, CBioPortal, Biobank research, and education applications and all these came with their different systems and different viewers, which made it more complicated for pathologists to interact in the end with those tools.

After installing our system then everything was unified and everything was easier to interact with the pathologists. Pathologists could use that MSK viewer to see the images from these different systems, but also to annotate the images from these different systems. And this is an example of how this looks. This is an old video. Here you see the MSK viewer in action, in a in a research platform. You would not see any patient identifier here, but you could still organize images for projects. Breast cancer or prostate cancer project to annotate the images with pathologists and zoom in and zoom out and all these tools that you would be used to from different viewers. 

We integrated that platform to these different hotel applications, for example, the LIS on the top left. Here you will see patient information as well when you open the images, you could see the barcode label and the images would flip. If you have visited them or not visited them to really do a sign out. On the bottom left there is for example the integration in CBioPortal again for research without any PHI. We could also then observe overtime the usage of digital pathology at Sloan Kettering, which you see on the top right. The yellow curves is the clinical usage. You open a case from the LIS and you see the steadily growing over time. The other applications are depicted with different colors.

Digital pathology itself provides a rich base for research. For example, you could think of how you efficiently stream data from the storage to the user, and this is not that obvious. You could think it's obvious, yes. But there's a lot of opportunity for research, for example, the storage of the images is usually done in the pyramidal shape. Would you use for example virtual deep zoom representation of those images of what you just use it as the images are natively done by the scanner? You can also think of caching strategies and non caching strategies, whatever you think of is best to stream those data. We could show for example that you could. Increase the time to change a slide from one second when you click on the slide to 1/5 or even a 10th of a second when you when you click on the slide and even the navigation so the streaming of the images itself, the navigation of the field of view was a little bit increased, of course not significantly, but we could see a little bit of a change here.

Digital pathology also provides opportunities for research in the diagnostic sense. For example, you would validate the digital platform if it's ready or can be used for primary diagnosis, and this is also what we did at Sloan Kettering.

In two major studies we showed first that yes, the digital platform was equivalent to the microscope in terms of diagnostic features, which pathologists usually look at. In the second study, we showed that it was also applicable in a remote setting, so this is actually a symbol for the home office. We used the viewer there, especially in the COVID-19 pandemic such as pathologist could do home office there.

Once we established a digital pathology, we can also think of the computational pathology and the pathology AI, which helps pathologists to sign out patients. In this setting, pathologists would still be able to go to the microscope, but the AI would give you enhanced information on a computer screen about the case.

What is pathology AI? In principle, this computer vision for pathology images. This means it is the ability of a computer to extract relevant information to solve a particular problem. The most basic problem in pathology is to determine if there is cancer on an image or not. Right, so this is only one problem in pathology, of course, we have many more, but this is a basic one.

In classical computer vision is of course, an old field in computer science, and we have many, many techniques, but it cannot easily be transferred to pathology. The reasons for this is mainly in in three domains. The first one is the data itself. The images or classical images are usually small because they they're coming from your mobile phone or from photographs which are not that large in terms of pixels. We have also millions or even billions out there we could access theoretically. In pathology images are still few because we need these special devices which are also very expensive, these scanners. The images itself are huge, 100,000 by 80,000 pixel is easily matched.

The annotations for the images to train algorithms are easy in classic computer vision, every kit can contribute for annotations, and we can also use crowdsourcing to get annotations. In pathology, we need trained experts, pathologists trained over 10 years, which have to tell us where cancer is, it's not obvious. The objects itself in classical computer vision usually are large, structured, which can also make computer vision easier, so to say. Then in pathology, where the objects are very, very tiny, it can be the search for the needle in haystack. And cancer doesn't have any orientation. There is no left and right and top and down and so on. It can be arbitrarily difficult.

To illustrate how large those images are compared to classical computer vision images. two slides for this. C410 is a very common data set in computer vision consists of 60,000 images from objects, but the whole data set fits into one tiny part of one digital pathology image. Imagenet, which is also very famous data set in computer vision with 40,000,000 different images of objects. This also fits into 500 whole slide image. This is what the pathology department produces in an afternoon easily. Then you can imagine how much data such algorithms have to handle.

Once we can digitize those images and we have access to them, then we can also see what we can do with this, right? Here is an illustration of data sets in computational pathology that we used over the last 15 years. We see that most studies used maybe 10 or 20, a few dozens or a few hundreds of pathology images, of digitized pathology images. For example, metastasis detection, lung segmentation, lung detection.

The reason for the number of these slides is back in the days, this was also when I learned computational pathology, digitizing images were still very expensive and special. Not many departments had these scanners, but also one reason is that all these data sets are exhaustively annotated. This is a typical scenario with these data sets that you ask pathologists to paint on those images, to outline, where the cancer is. The algorithms can learn from these annotations. But now, for example, when we have thousands of images, access to these thousands of images, you cannot ask pathologists to annotate those images anymore, because this is way too expensive. For example, we can use other techniques in machine learning, for example weekly supervised learning. Unsupervised learning.

In weekly supervised learning, which was very successful. We would learn from the pathology reports that I mentioned in the beginning that every single patient has and we would correlate pathology reports or the information from pathology reports to the images itself.

When this comes for free, so to say, we just have to digitize image entry reports. There is no additional annotation needed from pathologists. With this set up, our PhD student back in the days, Gapriella Campanella, he developed an cancer detection model, for example, for prostate cancer, but also for other cancers, which was trained on these thousands of images. Back in the days it was ten thousands of slides with weekly supervised learning.

What we developed, was multiple instance learning pipeline which basically takes all these different images right? For every single image, it patches the whole image into small patches exhaustively, so you would patch the whole tissue that you find on the whole image. Every individual patch is processed by a convolutional neural network and the network assigns a probability to every patch, the probability to show cancerous lesions or signs of cancer right in the beginning.

The network is initiated randomly and the probabilities are just distributed randomly. There is no neural network here but. You would still rank all the patches according to the probability and with the most cancerous patch on the top right. Here is where the information comes in. From the pathology report, we know if this is a cancerous slide or benign slide. If this is a cancerous slide, there must be at least one patch in the whole image which shows cancer, otherwise it would not be a cancerous slide.

This should be the topmost one. You can use this information or this difference to back propagates through the network and to adjust the weight to make this happen. On the other side, if this was a benign slide with the probability of zero of cancer, and you rank those patches, even the topmost one should have a probability of zero. You can use this information again to adjust the weights to make this happen.

If you repeat this thousands of times the network can learn the difference between cancerous patches and benign patches. Then you would apply this network to a new slide, patch it again, exhaustively, and for every patch you would assign a probability which results in a sort of a heat map over the whole slide, which looks like this maybe. You would then need an aggregator function, which takes all these probabilities and all of, for example all these patches and calculates a final diagnosis for the whole slide. This aggregator function again can be learned with a new network, for example, or can simply also be something like an average or a majority vote. 

With these networks or with this scenario, Gapriella trained that right and tested that and it was very luckily and also surprising that this works so very well on this large scale data set, right? Here is the area under the curve for cancer prediction, for slides. And this is already very high, right? It's 0.98 or 0.99 depending on the aggregator function that you use.

If you look at an image and you look at the region where the probability is high as to expose cancer, ovarian cancers, patches in fact are given by the model and you compare this to where a pathologist would locate the cancer, then you can also see that these are very similar regions which also stresses the fact that the model sees these similar morphological structures or signs that also pathologists sees. 

Here is also the very interesting insight in this research. Gapriella repeated this experiment with small data sets a few hundreds of images or a few thousands of images up to the final letter set size of 8000 images. On the right side you see the error plotted for the results of those of those network networks. In the beginning when data sets are very small. This multiple instance learning approach would not learn anything. It's almost random, almost random error. Over time when data sets are growing, the network is learning more and more up to this really high accuracy which we have seen before. 

This is remarkable that the network benefits from these large data. It really learns from those large datasets until the end and it learns to a very high accuracy which is already interesting to be used in clinics because it's not 60% and 80%, but it's really like this area under the curve of 99%. 

This is where we started to talk about clinical grade models which are interesting for clinics. But it's not done with one curve or with one experiment. This triggered a lot of different validation steps for example. For clinical grade model, you would also need to see if such a model can deal with the data which come from pathology departments as they as they are produced. It's not curated data. Pathologists and technical assistance produce data just as they come in. They might have pen annotations. They might be unsharp. They might also have air bubbles in the slide or cracks in the glass.

All these different disturbances where pathologists can easily deal with, they just ignore those artifacts and still do the diagnosis. Also, AI and models should be able to deal with those data and you would then go there and take this data and make your proper validation to see if the higher accuracy still holds true. Scanner agnosticy is also a very big issue. For example Leica scanners and Phillip scanners. We know they produce a little bit of a different image appearance, right. Also pathologists are agnostic for that, they just know this and also AI has to ignore this. You have to make the validation if this holds true. We did this and it was also very successful it was very scanner agnostic, but there are also two other very big studies and showing these, these high performing scenarios.

Prostate is only one. There's also breast cancer. Here, we see that this works not only on prostate cancer this is the example for other cancers as well which all work with these different large scale data sets and it's always the same multiple instance learning approach and always we see these higher accuracies.

It's not only cancer detection. This is a very nice review article which I found which shows many different models for different pathology tasks. In pathology. We are not only interested in two model detection and this is what we talked before prostate cancer detection, but this is only one task. For example, we also have subtyping of cancer or grading of cancer. And maybe survival prognosis, response prediction for therapies. 

Also very interesting is mutation prediction. Can we predict the presence of a mutation without sequence or without doing sequencing of that tissue, just based on H&E slides? For all these different domains and applications, these models listed and plotted many models which are already progressed in their development, so to say, right in the outer circle. For example, they are validated internally on internal data sets in the medium circle. They are validated externally on external data sets and in the middle they are also already approved by the FDA. And every cross is a different model solving a particular cancer in that problem and what it shows is that all these different groups research groups and models want to drift into the middle? They want to be clinically and applicable, and they're all going there. And this is really some movement in the community.

What does it mean to create this nice pathology AI? We need domain experts working hand in hand together, pathologists and computer scientists. We need to collect these real world clinically relevant data sets at scale, which we want to fill with scanning the archives and scanning prospectively just systematically. We need also new ways to learn from those data. We cannot annotate, for example, everything we need to use all these other techniques that exist in machine learning. And of course, we also need sufficient computational power to deal with these very, very large data sets. This was the answer I want to thank you for your attention, and if you're always, if you're ever interested in what we are doing in Munich more, feel free to catch me up. Thank you.