Demystifying DICOM in Digital Pathology: Experiences from the National Pathology Imaging Co-operative

Thank you very much for the opportunity to speak to you today. I'm a histopathologist primarily with an interest in DICOM digital pathology and IT. Hopefully today's presentation will be covering the clinical side and not too much technical detail about DICOM. 

Outline

Here's a brief outline of my presentation. I'm going to start with a background to the NPIC project and then give a basic overview of DICOM and specifically how DICOM applies to digital pathology. We're going to look at the current state of DICOM via its advantages and disadvantages, look at a couple of future developments, and finish with the summary.

The need for a National Pathology Imaging Co-operative (NPIC)

Given that my experience of DICOM is through work with NPIC, the National Pathology Imaging Co-operative based here in Leeds in the UK, I'll give a brief overview of this project. Why is there a need for a National Pathology Imaging Co-operative? Currently there's generally a low uptake of digital pathology in the UK and NHS and also worldwide, but increasing need for high quality data, both digital images and their associated metadata across multiple sites to improve diagnostics and workflows and also to help address staff shortages in histopathology which is particularly relevant in the UK at the moment. 

The thinking behind this cooperative was to provide an environment for industry collaboration, encourage interoperability, and avoid isolated silos; by that we mean walled-off stores of data in proprietary formats that can't be accessed or transferred elsewhere. This is a recurring problem with systems throughout the NHS. Also important from the start of the project was the development of standards, regulation and quality assurance; and through collaboration with academic institutions and industry to provide new research and product opportunities. 

NPIC: Aims 

In summary, the NPIC project has three primary aims. Firstly, to drive the clinical use of digital pathology. It's no coincidence that clinical use is the first aim as clinical practice is and always has been at the heart of the project. Secondly, to provide a resource for artificial intelligence, AI development and evaluation, which in turn will assist the first aim for clinical diagnostics. And thirdly, to support further research and innovation. 

NPIC: Summary

The NPIC Project is one of five UK research and innovation centers of excellence in digital imaging and AI. The project began in 2019 and continues until 2023 and receives both public and industry funding as outlined here. The overall desire is to build a national digital pathology system as a platform for the NHS, again going back to the clinical basis of the project we discussed previously, on a world-leading scale. As outlined on the right, there's over 30 hospitals involved currently producing 3 petabytes of image data per year. 

Scaling Up Full Deployment in 3 NHSI Pathology Networks

The basis of the project is to provide full digitization of three NHS pathology networks in the north of England, so that's 100% the scanning of all slides produced in those laboratories with the aim of sharing knowledge, procedures and QA processes from the already established deployment in Leeds. And on the right there, are a few figures indicating the number of trusts, pathologists, people, and scanners that will be involved in this large project. 

National Networks in Multiple Other Areas with a VNA

Central to the storage of digital images in the project is the VNA or Vendor Neutral Archive at the heart of which will be the DICOM format. As the name suggests, this means we're not tied into a single manufacturer or system and covers the need for scalability and future proofing. 

This same archive can be used by any scanner, any software, and any AI tool. On the left there is just some indication of our partners in each of these areas in the project. From the outset, our vendors and partners were engaged on the understanding of this need for standardization and the bold statement was that DICOM compliance is an entry criterion for NPIC, indicating the emphasis that we put on DICOM as part of this project. 

Press Release

You may well be aware of this press release from Leica Biosystems in June of this year (2021), which announced the following work in collaboration with NPIC, the Leica Aperio GT450 scanners now support native imaging in DICOM format. We're delighted at all the work that's gone into this and the GT450 scanners are used in our deployment in the clinical regions of this project. We're delighted in the move towards DICOM with these. 

DICOM: The Basics

That was a brief summary of the NPIC project and where DICOM fits in. What is this DICOM all about? DICOM stands for digital imaging and communications in medicine, and the definition here from the DICOM website is it's the international standard for medical images and related information. It defines the formats for medical images that can be exchanged with the data and quality necessary for clinical use. 

DICOM originated in radiology in the 1980s. In the early days of digital radiology, imaging devices from different vendors produced proprietary file formats only compatible with their own PACS, that's the picture archiving and communication systems, and their own viewers, which made it practically impossible for radiology departments to replace their PACS systems with one from a different vendor without also exchanging all their equipment, which was obviously far from ideal. 

The DICOM standard was first published in 1993. Compliance with this standard meant that radiology equipment produced images on a standardized format and communicated with standard protocols. It's since been expanded to cover numerous medical specialties and is implemented in almost every radiology and cardiology imaging and radiotherapy device and increasingly in devices in other medical specialties including ophthalmology, dentistry, dermatology, and surgery. And now obviously pathology. 

These different specialties are covered by 34 DICOM working groups, which developed and published supplements to this standard pertinent to these different specialties and areas. 

DICOM Standard: Key Concepts

Here's a few key concepts about the DICOM standard itself. The standard incorporates firstly compressed pixel data, so that's the image itself as either a single image or multiple frames in JPEG format. Secondly, metadata, which is vitally important in clinical and AI applications. This can be identification of the patient and their demographics, but also details about the acquisition of the image itself. This is particularly important in digital pathology, which we'll come back to later. And thirdly, services. DICOM is not simply a file format. It also consists of services, most of which involve the transmission of data over a network. Again, the roots of this are in radiology, but most of these services can be utilized in digital pathology. 

Firstly, the store service is used to send images or other objects, such as reports, to a PACS system or workstation. This is the storage commitment service, which is a useful service used to confirm that an image has been permanently stored by a device to make sure that it's safe to delete the images locally. Secondly, the query and retrieve service enables a workstation to find lists of images and then retrieve them from a PACS system. Thirdly, the modality worklist service. This provides a list of imaging procedures that have been scheduled, including details about the subject of the procedures; the patient name and demographics, type of the procedure, such as the equipment type and the procedure description, and the procedure order: the referring physician accession number and reason for the exam. 

An image acquisition device such as a CT scanner or a slide scanner in digital pathology queries or service provider such as radiology information system or laboratory information system in digital pathology, to get this information, which is then used to populate details in the image metadata. 

Prior to the use of this service, the scanner operator was required to manually enter all of the relevant details, which creates obvious issues with accuracy and completeness of information. Finally, the print service which is used to send DICOM images to a printer, normally to print on x-ray film, which is obviously less relevant to digital pathology.

DICOM in Digital Pathology

How did DICOM develop to you encompass digital pathology? We've already mentioned the DICOM working groups, the working group for digital pathology is group #26. This group has been instrumental in the development of the standard for digital pathology. By two principle means, firstly by the production of two major DICOM supplements. These mapped out the amendments required to the DICOM standard for use in digital pathology. These were supplements 122 in 2000 and 145 in 2009. I'm not going to go into too much detail about these, but for those more technically minded, they are available on the DICOM website. 

These probably standards are pretty useless unless they were taken up by vendors and could be demonstrated in real world usage. That was where the “Connectathons” came along. Quite an amusing term, the “Connectathon,” but basically the idea is that there was active with participation by a number of hardware and software vendors, the names of which are shown on this slide. 

There were five of these “Connectathons” between 2017 and 2019. The participants range from the manufacturers of the scanners themselves on the right there. Then software companies involved in analysis of the images, viewing of the images, and archiving of the images. They were a great success. 

These “Connectathons” were held at digital pathology conferences such as Pathology Visions in the US and European Congress on Digital Pathology in Europe. And they really demonstrated that these vendors were willing to work together with their competitors on this unified standard. Really the vendors didn't have to participate. Most of these vendors already had their own proprietary format which worked well for them. It was really encouraging to see everyone really working with their competitors to achieve the goal of developing the DICOM standard. 

They had real-time side-by-side demonstrations at these conferences, again demonstrating to people in real time what was possible with the DICOM format. 

There's several fundamental differences between imaging and radiology and digital pathology that require a different approach, and it's probably useful to mention these at this point. The big one is that the whole slide images diagnostic resolution are very large. With compression, they're in the order of 3 to 500 megabytes. They can be much larger depending on the maximum scanning resolution used, and this may lead to issues with the storage and network capabilities. 

The DICOM standard allows elements more specific to histopathology. It allows categorization by specimen rather than study as is usually the case in radiology. It allows description of specimen processing, such as details on collection, fixation, embedding and staining. The storage of which within the DICOM file may be important for AI applications. 

It encompasses color management, again, much more relevant to digital pathology. We're all aware in light microscopy in pathology, how the different staining and colors vary between laboratories and so color management being built into a digital-first standard is important. 

It allows annotations. We'll come back to this again a bit later. Although a lot of manual annotations by the person reporting the image or the digital slide are very similar between pathology and radiology, the annotations involved in AI applications may be much more extensive. Also, it allows structured reports. 

The other major difference between radiology and pathology is the need for rapid pan and zoom to large degrees of magnification to emulate the function of a light microscope which isn't required in viewing standard radiology images. The image data in DICOM for digital pathology is stored as separate tiles which allows access to small regions of the image without having to load large amounts of data across a network, which I will now attempt to demonstrate.

Single Frame Organization

The simplest way to store two-dimensional image data is in a single frame organization where the data is stored in rows extending across the entire image. In this diagram, the pixels making up the image are represented by the purple squares at the top, so extend from the top left to the top right and then down to the next row. That's the standard sort of single frame organization of pixels stored in rows.

This has a major limitation in that to view a small portion of the image which is represented by this dark green square here, the much larger proportion of the image must be loaded. Just to get the pixel data for this square here, because it's stored in rows, the entire row of each of these five rows must be loaded. That's quite a lot of data just to look at that green square. 

In DICOM for whole slide imaging, the 2D image data is stored in tiles which may be squares or rectangles that represented as squares in this diagram here, rather than rows of pixels that we saw in the previous diagram. This is a more complicated way of storing the image data, but it has a major advantage that compared with a single frame organization. 

To view a small portion of the image again by demonstrated by this dark green square here, a much smaller portion of the image needs to be loaded. To view the pixels here, these four light green squares needed to be loaded as opposed to the entire five rows of pixels that we saw in the previous diagram. 

Pyramid of Image Data

This solves the problem of rapid panning, but what about rapid zooming, which is also very important in whole slide imaging? The solution proposed in the working group 26 supplements was that of tiled pyramids. The idea is that the whole slide image is composed of multiple images at different resolutions. In this diagram, there's four different images of different resolutions, so we have the highest resolution or baseline image made-up of these squared tiles here, and then three further intermediate zoom images here, here and here at lower resolutions. 

In this diagram, the altitude of the pyramid corresponds to the zoom level, so in terms of microscopy it's low power at the top and higher power at the bottom. Again, in this diagram, the green square is the retrieved portion of the image as an arbitrary resolution of this darker black line, somewhere between the highest baseline, highest resolution, and the first intermediate level, and the shaded areas on these tiles at this resolution and this resolution show the proportion of the slide that must be retrieved to allow rapid zooming. 

The advantage of the pyramid structure is that the entire highest resolution layer is very large, so actually storing even multiple lower magnification or lower resolution levels for faster zooming actually takes up negligible extra file space. Again, with the versatility of DICOM, there's no set number of intermediate levels and there's no set size to the tiles. As we've said before, they can be square or rectangle. 

This tile arrangement works around one restriction of DICOM in the single frame size limitation which is 64K by 64K. Using this tiled representation means that there's no change needed to the underlying DICOM coding and no change to existing DICOM toolkits or archives. 

DICOM allows two different types of storage. Firstly, tiled-full. That means the entire set of tiles for a particular field are stored in a pre-specified order. In this example, every single one of these square tiles would be stored in the file in the order that they're stated here. And then tiled-sparse. This is more versatile in that tiles can be anywhere and even overlap, and the position of each tile is explicitly specified. 

The tile-sparse means that not all tiles need to be present. In a particular scanner, for example, the white space surrounding the actual tissue doesn't necessarily need to be included in the tiled sparse storage, which would mean a reduction in the required file size. Rightly or wrongly, I equate this panning and zooming in digital pathology to that of Google Maps, in that you've got potentially a very high-resolution image at the sort of highest magnification of Google Maps. But in any particular area you don't require your computer to load this entire high resolution image in order for you to view a particular area. I might be wrong from a technical level, but I think it's helpful to equate that as a way of thinking about the pyramid storage of data in DICOM for digital pathology. 

Current State of DICOM: Advantages and Disadvantages

It might be a good time now to leave the more technical aspects of the format and consider the advantages and disadvantages of DICOM in its current state. As previously mentioned before, one of the main advantages of DICOM and digital pathology, particularly in the NPIC project, is that it prevents vendor locking and allows futureproofing of long term storage. Without having to either move existing images onto different storage or use different PACS systems or viewers or AI tools in the future to look at the same files. 

DICOM allows digital pathology to integrate into existing PACS and IT systems, and this might be a key feature in terms of procurement that it might not be as costly as setting up a system from the word go if it's going to be included in an existing radiology PACS system, for example. 

It allows collaboration across disciplines. Again, if they're on the same PACS system with the same viewer, this potentially allows both pathologists and radiologists to view radiology and pathology images alongside each other, which as we move forward, is going to be increasingly important for things like MDT. Discussion could be very useful.

DICOM has proven performance in viewing images, certainly comparable with many of the proprietary formats traditionally offered by vendors, and as mentioned, not all images of the slide need to be captured at the highest resolution, again allowing saving on file space for a particular slide and these are referred to in DICOM as sparse images. 

Importantly, is the metadata in the DICOM header. We've mentioned this already, but having such data about the patient themselves, about the technical aspects surrounding the creation of the slides, and the particular stains in as much detail as necessary, having all of that stored in the same file is very useful, particularly for things like development of AI tools. 

The DICOM standard is flexible. We've already alluded to this in terms of the tiles and number and shape of the tiles and resolution, but it allows things like color management, annotations, and even structured reports to be stored within the files themselves. 

There are obviously some disadvantages potentially has been reported that DICOM file format could lead to potentially slower scanning, or if a particular scanner does not export natively in DICOM format, it could require extra steps in the file creation once the slide itself has been scanned. 

There’s problem is populating the DICOM header. It requires integration with the LIMS, the laboratory information management systems for items such as identification and specimen preparation details and many existing LIMS do not support the modality worklist interface that has been so successful in radiology, so this this can be a problem and not having access to the useful metadata to store within the files themselves.

Although we had this as an advantage, the fact that the DICOM standard is flexible can also potentially be a disadvantage. If there are several possibilities of ways of storing the data that could potentially be a disadvantage and lead to slower read times if that's not thought about in advance with the particular viewers or storage systems that require it. 

Also what the fields are mandatory and optional, that can cause problems in populating the DICOM header with what to do with fields where the DICOM file itself is expecting particular fields to be filled in but the data is not able to be retrieved from the LIMS system. Existing PACS and IT systems may need to adapt to handle larger file sizes. 

Future Developments

Finally, just a very quick look at future developments. I've picked out just a couple of future developments. Firstly, another supplement from working group 26. This is supplement 222, whole slide imaging annotation which was published earlier this year. As I mentioned before, annotations on digital pathology images doesn't just mean a single pathologist either doing a measurement or drawing circles around a particular area of interest. It can also mean bulk annotations created by machine algorithms. 

For example, if an algorithm wants to pick out all of the nuclei in a particular field, then the coordinates of each of those shapes drawn around the nuclei would need to be stored. And this supplement allows that to happen and maps out how they will be stored in the DICOM format. 

There's a virtual hackathon ongoing this year, so hackathon is similar idea to the Connectathon, but this this is ongoing and we'll finish in October of this year and this allows the development and testing of software for generation and exchange and use of these whole side imaging annotations as mapped out in this supplement. Interesting to hear the feedback from different vendors and software and AI manufacturers as to how that's going. 

Again, we mentioned it in in the last slide, but integration with LIMS is vitally important. The development of this will allow metadata enhanced workflows really important for clinical diagnosis and academic research. 

Also, the thought of bidirectional transfer of data. For example regions of interest in measure and measurements. If these were measured on the viewer of the DICOM image, they could be directly transferred into the LIMS for example. Rather than measuring a particular area and then typing that value into the LIMS, there could be a direct transfer of a measurement into the LIMS, which in certain applications would be very useful, time saver and avoid typographical errors. As you've already mentioned, the integration with LIMS is essential for continuing AI development. 

Demystifying DICOM: Summary

And so that's all I have to say. I've just summarized what we've spoken about today. DICOM is well established international standard in medical imaging and has been since the 1980s. DICOM in digital pathology is becoming, and some might argue is already well established. And as we've discovered via things like the Connectathons, there's increasing adoption by the whole slide imaging vendors.

Really from our point of view, it's important going forward in any planning and procurement of digital pathology systems and services, it's important to discuss DICOM and question the manufacturers and the vendors as to where they're at with their DICOM compliance and if they're not already DICOM compliant, what is the road map for that compliance? 

Also in discussing this procurement, there's always the possibility of integrating into existing PACS systems. DICOM is a flexible, developing standard, and through this there's opportunities to get involved. The DICOM working group 26, it's got a lot of members from varying backgrounds, ranging from pathologists like myself through to sort of technical people and representatives from vendors and manufacturers and software vendors and is really an international group. They're always keen for new members who are involved and interested in this in this area.

Then also IHE PaLM, that's Integrating Healthcare Enterprise Pathology and Laboratory Medicine group, they will concentrate on the pathways and workflows to digital pathology and integrating it into our day-to-day work and they create several profiles for this to be followed and happen. 

Acknowledgements

Finally, just a couple of acknowledgments. Firstly to David Clunie and the other members of the working group, 26 for DICOM, who've really taught me in the ways of DICOM. And also to Darren Treanor and all the other staff at the NPIC project.