Beyond Observation: Tools, Software and Molecular Prediction

My name is Francesco Merola. I'm a researcher at the University of Molise and with great pleasure I present a brief speech here today, together with Francesco Martino, my friend and collaborator, on some IT tools that we can now rely on to approach the world of digital pathology.

The concept of digital pathology, the same lexical expression, is no longer a taboo today. As we can see from the PubMed chart, the number of scientific publications dealing with this topic has increased incredibly in recent years. And digital pathology is also often at the center of relations and discussions in pathology conferences as well. An increasing number of pathologists are using virtual slides and consequently software primarily for viewing these slides. But the advent of virtual slides has also allowed pathological anatomy to enter in the world of image analysis.

There are many platforms available nowadays for the analysis of digital images in pathology, but certainly the one preferred by most is QuPath, an open-source software, relatively user-friendly, which allows even those who do not have programming skills to do excellent things with virtual slides. QuPath is developed at the University of Edinburgh. The software was originally created at the Centre of Cancer Research and Cell Biology at Queens University in Belfast.

Colour Deconvolution

I don't have time here to show you an entire image analysis workflow with this platform, nor do I want to anticipate the content of the speech that will follow in a while on some practical application that we made by using this software. I won't say that among the things that I personally find more useful and interesting than this software can do, is the management of tissue microarrays, of which we make extensive use for research purposes, and in addition the management of colors with the possibility of easily performing at the convolution. This feature has allowed us in many cases to discern efficiently the immunohistochemical signals separating them from compounding signals such as, for example, the pigment in the case of melanocytic lesions.

Here you can see, in this example, how easy it is to isolate individual color channels. And it's just an example on how user-friendly the software can be.

Cell and Nuclei Segmentation

Segmentation is another fundamental step of an image analysis workflow. The control panel allows us to fine-tune the segmentation process and to adapt it also to not quite simple situations. Here, for example, you can see an example of squamous carcinoma of the oral cavity, well differentiated, where the abundance of keratin can be a problem. In the end, we managed to obtain excellent results, especially, most interestingly, all the parameters are easily exportable to subsequent analysis on the same type of samples.

Object Classification

And finally, the classification step. QuPath allows, after the recognition of the single elements, nuclei and cells, to extrapolate the features based on which we can build classifiers. By which, to automatically recognize the elements of the classes assigned to the object in question. For example, here we trained the software to recognize cancer cells and stromal cells, obtaining a classifier that automatically recognized the two classes in our slides. This is very useful when it comes to quantifying an immunochemical signal and you want to evaluate it, for example, for a particular class, let's say to automatically calculate the h-score of a tissue marker expressed in the tumor.

Feature Extraction

Well, the machine learning approach that we can do with QuPath has many limitations. Basically, it is based on a manual selection of features. Although the features that the software allows us to identify and to work with are very representative. This approach can have its advantages. For example, to the extent that it allows us to study individual cellular characteristics by associating them with other variables, such as, let's say, positivity for a tissue marker. This is in fact a road that we are pursuing now. Our intention is to explore the prediction value of immunohistochemical positivity of individual features or group of them, as Francesco will tell you in a while. Thank you.

What if we go beyond what people can see?

Hello everyone. I'm Francesco Martino. I am a doctoral student in surgical pathology at the University of Naples, Federico Sicondo. I graduated in biotechnology and then during master's degree, I joined the group of Professor Sibano. I'm working on image analysis. Now that I'm a PhD student, I mostly use QuPath, but I'm also working on a project with TensorFlow and Python. During our experience in digital pathology, while studying artificial intelligence applications and after reading some marvelous works in other fields, we started asking, what if we go beyond what people can see? Is it possible to use artificial intelligence to see what human eyes cannot? Well, someone else tried to answer this question, and I will show a couple of examples before presenting our results.

The first one is the study Classification and Mutation Prediction from Non-Small-Cell Lung Cancer Histopathology Images using Deep Learning. Here, the authors firstly developed a deep learning framework to automatically analyze whole-slide images, telling lung adenocarcinoma area from squamous cell carcinoma, and they obtained results comparable to pathologists. Then they validated this algorithm on external dataset, confirming its accuracy. Then they used this algorithm on lung adenocarcinoma to segment all the areas and to generate an algorithm that could predict gene mutation. They successfully predicted the mutation status of six genes, and furthermore, they confirmed the EGFF prediction on a dataset made with samples from the archive. Concerning methods, they used TensorFlow with Inception V3 network with Python.

However, this is not the only result in literature, and there is another example. Here, there is another example. In the study Prediction of BAP1 expression in neural melanoma using densely connected deep classification networks, the authors proposed a similar idea but developed the expression of BAP1-associated protein 1 in Uveal melanoma. In this case, they used whole slide images from 47 patients to obtain 7,000 patches that were used to train their network to effectively predict BAP1 positivity in neural melanoma.

In detail, they extracted the images, they manually annotated these images, and they trained a network that could predict the positivity not only evaluating the single nuclei but also evaluating the other part of the image. Although this study shows some limitations, this is still an important advancement in deep learning and in molecular prediction.

IHC Immunohistochemical Positivity Prediction Using an Open-Source Software

And here we come with our idea. Previously reported results need important programming skills, but could it be possible to predict the immunohistochemical positivity using an open-source and user-friendly software? Well, we decided to try our idea using QuPath.

As I said before, we used QuPath to perform our analysis. First, I want to clarify that I will not state the name of the biomarker for two reasons. The first one is that we currently have a paper around the revision. And the second one, this is the most important one, this is just a proof of concept, it's not meant to be used for IOD, and I do not want to carry a wrong message. For our project, we started from H&E stained slides, we performed a slide scanning with Aperio AT2, the Leica scanner, and then we de-stained our slides and colored them again with an immunohistochemistry. By this approach, we had the chance to manually annotate single cells in positive and negative according to the immunohistochemical result.

Here you can see a part of a slide, a magnification of a slide, with a series of bounding boxes. Red boxes are cells that resulted positive in immunohistochemistry, while green boxes are cells that resulted negative in immunohistochemistry. As you can see, nuclei are contoured. This is because we used QuPath tool to perform a nuclei segmentation and to calculate the series of features you saw before for each nuclei.

After segmentation, we saved the QuPath computed features and using the statistical analysis, we found the features that would be able to predict the immunohistochemical positivity. Later, we generated the false color map using QuPath tools, and a false color map is a map with a color associated to the intensity of a value. In QuPath, with the plasma ranging color, blue is a low intensity, a low value of the feature, Yellow is a high value of the feature. We found that the cells with higher value of the feature of interest were positive for the immunohistochemistry.

Here is a graphical representation of the result. As I said before, red boxes and green boxes are cells annotated as positive or negative according to the immunohistochemistry, while blue filling and yellow filling are cells predicted as negative or predicted as positive. Red boxes with yellow filling are true positive, while green boxes with blue filling are true negative.

Finally, we test our protocol on TMAs. On the left, there is an H&E slide. On the right, there is a immunohistochemistry. In the center, there is a full color map. I want to underline that H&E slide and the immunohistochemical stain and slide are not the same slide, because in this case it is not a destaining, but it is two nearly consecutive slice, so some differences are normal. However, as you can see in the first case, the case in the top part of the slide, is highly positive in immunohistochemistry, and the result is confirmed with the first color map. By the other hand, the second case is lowly positive in immunohistochemistry, and the same result is present in a false color map.

Future Perspectives

I showed just a part of the result, a minor part of the result, because I wanted just to show the idea. I wanted also to discuss the possible answer to the question: is it possible to predict immunohistochemical positivity using H&E slide with an open source and user-friendly software? Well, I think the answer is yes. I do believe the answer is yes, but some modifications are necessary. First, we foresee in the next feature to use multiple features at once, because in this part of the study we used just one feature. Another approach may be using neural networks to generate synthetic immunohistochemistry. For example, using generative adversarial networks. Finally, although this is just proof of concept, it is not the time, today is not the day for clinical application, we are confident that in the next feature this application will be part of the routine clinical practice. We’ll see what the feature will be and thank you for your attention.