Interoperability is at the heart of our vision for the future of radiology. Rad AI Reporting leverages standards like HL7, FHIR, and FHIRcast to enhance radiology workflows.

HL7 (Health Level Seven International) is a set of international standards for the transfer of clinical and administrative data between software applications used by various healthcare providers.

Within HL7, FHIR (Fast Healthcare Interoperability Resources) is a standard describing data formats and elements (known as "resources") and an application programming interface (API) for exchanging electronic health records. FHIR aims to simplify implementation without sacrificing information integrity, providing a modern, web-based approach to healthcare interoperability.

FHIRcast builds on FHIR's foundational principles of interoperability and modern web technologies. FHIRcast is an open standard for synchronizing healthcare applications in real-time. It allows applications to communicate and share context, enabling them to stay in sync while the user works across multiple applications.

While many vendors have worked with HL7 and FHIR, not many have worked with FHIRcast just yet. We’ll go over what FHIRcast is, what problems it solves, and how Rad AI uses FHIRcast to push the boundaries of what’s possible in radiology reporting.

To start, let’s take a look at the strengths and weaknesses of FHIR.

What problems are solved by FHIR?

Common Data Model

FHIR provides a standardized and consistent data model known as "resources". Each resource represents a specific aspect of healthcare, such as patients, observations, imaging studies, and procedures. By using these standardized resources, FHIR ensures that different systems can understand and use the data in the same way. FHIR resources are format-agnostic, but commonly implemented using JSON.

Resources used by Rad AI Reporting include DiagnosticReport, Patient, ImagingStudy, ServiceRequest, and Observation.

Common Access Patterns

FHIR defines a RESTful API as part of its specification. Implementations of this RESTful API specification are often referred to as “FHIR servers”. These FHIR servers provide well-defined endpoints for retrieving FHIR resources, publishing updates to FHIR resources, etc.

Improvements Over HL7

The complexity of HL7 can be a barrier to adoption and effective implementation. The HL7 standards have steep learning curves and often require extensive customization, which can be time-consuming and expensive. FHIR is designed to be simpler and more intuitive. Its modular structure allows developers to use only the parts they need, reducing complexity and making it easier to implement.

Furthermore, conceptual differences from HL7 make FHIR easier to work with. HL7 messages are transactional in nature. To get the up-to-date content for a set of resources, a system needs to read the HL7 resources in order and apply those transactions one at a time to the underlying resources. This essentially forces systems to re-implement their own data store to get the full resource content.

FHIR breaks those constraints by providing a stateful interface instead of a transactional interface. When you request an ImagingStudy resource, you get the full up-to-date state of that resource. Underlying transactions (i.e. creating resources, updating resources, deleting resources) are handled by the FHIR server.

What problems are NOT solved by FHIR?

While FHIR addresses many challenges in healthcare data interoperability, there are certain problems that it does not fully solve, particularly related to real-time data sharing and synchronization across multiple systems.

Real-Time Sharing of Resources

FHIR does not inherently provide mechanisms for real-time notifications or updates when a resource changes in one system. This means that changes to a resource in one system are not automatically propagated to other systems.

Without a built-in method for real-time notifications, systems need to implement their own mechanisms to share updates, which can lead to inconsistencies and delays in data synchronization.

Real-Time Sharing of Context

FHIR does not provide native support for maintaining shared context across different systems. For example, if I open an ImagingStudy in my PACS, my Reporting app should open to the same ImagingStudy.

This lack of shared context can lead to workflow inefficiencies and discrepancies, as different systems may not be synchronized. In scenarios where multiple applications need to work together seamlessly, this can be a significant limitation.

Existing Solutions that FHIRcast Replaces

For real-time sharing of resources and context, healthcare systems often resort to bespoke data-sharing processes. One common approach is the "file-drop" method, which involves writing files to disk and monitoring those files for updates.

Let’s take a look at an example. We’ll assume a Radiologist is using a PACS and the Rad AI Reporting app, with their PACS acting as the “driver” application.

1. A user initiates an action in their PACS, such as opening an imaging study.

2. The PACS pulls the accession number from the imaging study and writes this information to an XML file.

   • This XML file is saved in a pre-defined directory that other systems have access to.

   • That XML file may look like:

<ImagingStudy>
    <AccessionNumber>123456</AccessionNumber>
</ImagingStudy>

1. The Rad AI Reporting app watches for changes to that XML file. When a file change occurs, Rad AI Reporting reads the XML file, extracts the accession number, and fetches the corresponding ImagingStudy resource from the FHIR server.

Problems with the File-Drop Method

• Latency: There can be delays between when the PACS writes the XML file and when other systems detect and process the change.

• Scalability: This method does not scale well, especially in environments with multiple systems needing to stay in sync.

• Standardization: This method lacks standardization, leading to inconsistencies in how different systems implement and interpret the XML files.

• File-system access: All applications must have file-system access to watch and read from files. This introduces complexities for healthcare applications that are built as web apps, as web apps don’t have access to the file-system.

What is FHIRcast?

FHIRcast is a standard developed to provide real-time synchronization and context sharing between healthcare applications. It builds on the foundations of FHIR resources, but focuses specifically on ensuring that multiple applications remain synchronized in real-time as users interact with them.

FHIRcast requires a centralized system to manage real-time synchronization and context sharing. This system is referred to as a “FHIRcast hub”.

Here’s how FHIRcast works:

1. Subscription setup: Applications subscribe to a FHIRcast hub for specific events. This is typically done during application startup.

2. Event triggering: When a user interacts with an application (e.g., opens a patient record or an imaging study), the application sends an event to the FHIRcast hub.

3. Real-time notification: The FHIRcast hub immediately broadcasts this event to all subscribed applications via WebSockets, ensuring they are notified in real-time.

4. Context synchronization: Subscribed applications receive the event notification and adjust their context accordingly. For example, if an imaging study is opened in one application, other applications might display related patient information or open the same study.

Example Use Case - Imaging Study Synchronization

1. A radiologist opens an imaging study in their PACS.

2. The PACS sends an event to the FHIRcast hub, indicating that the ImagingStudy FHIR resource has been opened.

3. The FHIRcast hub broadcasts this event to all subscribed applications, including the Rad AI Reporting application.

4. The Rad AI Reporting application receives the notification and automatically opens the same ImagingStudy, ready for the radiologist to dictate their findings into a new report.

Benefits of FHIRcast

• Standards based: By providing a standard interface, all applications can share data and context seamlessly.

• Centralized access control: FHIRcast provides built-in mechanisms for authentication and authorization, ensuring that only authorized applications can subscribe to and receive updates.

• Robust error handling: FHIRcast offers robust mechanisms for error handling and retry logic within the WebSocket protocol, ensuring reliable communication. If an application falls out-of-sync, FHIRcast provides mechanisms to get back in-sync with the current context.

Potential Drawbacks of FHIRcast

• Complexity of implementation: Implementing FHIRcast requires integrating with a FHIRcast hub and ensuring that all participating applications can communicate with it. While third-party libraries may perform some of the heavy-lifting here, integrating older or legacy systems that do not support modern web technologies like WebSockets or do not adhere to FHIR standards can be difficult.

• Single point of failure: The FHIRcast hub acts as a central point for managing events and notifications. If the hub experiences downtime or failures, the entire real-time synchronization mechanism can be disrupted.

Furthermore, the FHIRcast specification is relatively young. The initial 1.0 release only came out in 2019, and the specification is still evolving alongside the FHIR specification.

Despite these potential drawbacks, Rad AI Reporting has embraced FHIRcast as the go-to mechanism to integrate with other healthcare applications.

How Rad AI Reporting Uses FHIRcast for Integrations

Rad AI Reporting is one application among many that serve radiologists in their day-to-day workflows. We need the ability to integrate with PACS, EMRs, and any other healthcare applications radiologists use.

To solve this problem, Rad AI Reporting leans heavily on FHIRcast for vendor integrations. There are a wide-range of events we listen for and publish, and we have thorough documentation on these events and the workflows they relate to within the Rad AI Reporting application.

There’s a bit of a “chicken and egg” problem with FHIRcast. Because it’s relatively new, not many vendors have committed to integrating it into their products yet. Because there aren’t many vendors with FHIRcast enabled, it’s hard to justify building it into an existing product. Rad AI Reporting is pushing the healthcare ecosystem forward here. We’re actively building out our FHIRcast system and working with vendors to build out their FHIRcast integrations as we roll out to new customers.

We’ve also seen a lot of interest from new tools in the healthcare ecosystem. These are newer applications that want to integrate with vendors but don’t necessarily have the resources or experience to build custom integrations. For these applications, FHIRcast is a modern, low-barrier way to start integrating with other applications.

Another way Rad AI Reporting has built out FHIRcast integrations is through “hackathons”. These are events many individuals and companies attend to push the industry forward in regards to healthcare application interoperability. Recently, Rad AI Reporting has participated in two of these events.

IHE Connectathon - Trieste, 2024

IHE Connectathons provide a detailed implementation and testing process to enable the adoption of standards-based interoperability by vendors and users of healthcare information systems. During a Connectathon systems exchange information with corresponding systems in a structured and supervised peer-to-peer testing environment, performing transactions required for the roles (IHE actors) they have selected to perform in carefully defined interoperability use cases (IHE profiles).

Rad AI partnered with Agfa for this IHE Connectathon. We tested Rad AI Reporting’s FHIRcast capabilities with Agfa’s HealthCare Enterprise Imaging.

For our integration, Agfa would have the “driver” application, and Rad AI would have the “driven” application. This meant that Agfa would set the current context via FHIRcast events like DiagnosticReport-open, and Rad AI Reporting would update resources via FHIRcast events like DiagnosticReport-update.

Partnering with Agfa at the IHE Connectathon was a massive success. Both teams were able to pass all tests, becoming the first vendors to pass all Integrated Reporting Application tests.

Some of the workflows we were able to demonstrate include Interruptions Without Losing Progress and Sharing Measurements from PACS.

By using FHIRcast, these types of integrations are now open to all other vendors, making integration with Rad AI Reporting seamless and simple.

Interrupted Without Losing Progress

This demo shows how a radiologist could:

1. Start working on a report (the “original report”).

2. Get interrupted and start on another report (the “interrupting report”).

3. Sign and send off the interrupting report.

4. Switch back to the original report without losing progress.

Afga demonstrating the “Suspend/Resume” IRA Profile

Sharing Measurements from PACS

This demo shows how a radiologist could:

1. Make structured measurements in their PACS.

2. Use those structured measurements in the report.

Note that FHIR resources inserted into the report as “measurements” are stored as-is. You can see this in the image on the right, we’re storing ObservationComponent and ImagingSelection resources.

Agfa demonstrating the “Content Creator” IRA Profile by sending measurements from their PACS to Reporting

SIIM Hackathon - 2024

The Society for Imaging Informatics in Medicine (SIIM) is a global advisor and thought leader in imaging informatics, focusing on enterprise imaging, artificial intelligence, cybersecurity, infrastructure, and standards.

SIIM Hackathon Project Showcase. Martin’s project, “Integrated Reporting and Conferencing with FHIRcast” is the first project in the showcase

Rad AI Reporting did not officially participate in the event; however, we played a supportive role by assisting one of the participants, Martin Bellehumeur, with the integration of FHIRcast into his hackathon project.

This collaborative effort highlights our commitment to advancing interoperability and supporting the broader healthcare community, even outside of direct event participation.

Martin’s project was an integration between Open Health Imaging Foundation (OHIF) and Rad AI Reporting. The idea was to enable “FHIRcast Conferencing”, allowing conference leaders to share context and drive the applications for conference attendees through the use of FHIRcast.

The project showcases some of the deep integrations Rad AI Reporting can have with other vendors. Some of the workflows enabled by this integration include:

• Opening an imaging study in OHIF will create or open that report in Rad AI Reporting.

• Sending measurements taken from OHIF to Rad AI Reporting, and adding those measurements into the report.

• Selecting a measurement in Rad AI Reporting sharing that selection with OHIF, bringing that measurement into context.

Congratulations to Martin for his project’s 5th place win in the SIIM Hackathon!

Why Standards Matter

Modern healthcare technology standards like FHIR and FHIRcast are crucial for enhancing interoperability and communication between disparate healthcare systems. By providing a common framework for data exchange and real-time synchronization, these standards enable seamless sharing of patient information across various applications and platforms. This connectivity ensures that healthcare providers have timely access to comprehensive patient data, which facilitates more informed decision-making, reduces the risk of errors, and streamlines workflows. Ultimately, the adoption of standards like FHIR and FHIRcast leads to more coordinated and efficient patient care.

Key Features of Abstractive Summarization:

1. Condensation of Information: It aims to reduce the length of the original text while retaining the key points and essential information.

2. Generation of New Text: Instead of directly copying sentences from the source, it creates new sentences that convey the same meaning.

3. Paraphrasing: It often involves rephrasing the original content to make the summary more readable and succinct.

4. Synthesis: It integrates and combines information from different parts of the text to provide a coherent and unified summary.

Traditional methods in abstractive summarization, such as word frequency and discourse semantics, are based on relatively simple algorithms. Word frequency methods determine the importance of words or phrases by counting their occurrences in the text[3]. This approach assumes that words appearing more frequently are more important to the overall meaning of the text. Discourse semantics[4], on the other hand, focuses on the structure and meaning of the text at a higher level, aiming to produce summaries that reflect the underlying discourse.Recent advances in abstractive summarization have been driven by the development of sophisticated models and techniques. Pre-trained encoders, such as BERT (Bidirectional Encoder Representations from Transformers), have significantly improved the quality of summaries by leveraging large amounts of pre-trained data to better understand and represent the input text. These models can capture complex relationships between words and phrases, leading to more coherent and contextually accurate summaries.Sequence-to-sequence models[6,7,8], like BART (Bidirectional and Auto-Regressive Transformers) and Pegasus, have also revolutionized abstractive summarization. These models use transformer architectures to generate summaries, allowing them to consider the entire input text and produce more coherent and contextually relevant summaries. By learning to generate text one token at a time, these models can produce summaries that are more fluent and natural-sounding.Furthermore, the advent of Large Language Models (LLMs) such as GPT-3 (Generative Pre-trained Transformer 3) has further advanced the field of abstractive summarization[9,10,11]. LLMs are trained on massive amounts of text data and are capable of understanding and generating human-like text. This makes them particularly well-suited for summarization tasks, as they can generate summaries that are not only accurate but also stylistically similar to human-written summaries.The field of abstractive summarization has seen significant advancements in recent years, driven by the development of pre-trained encoders, sequence-to-sequence models, and Large Language Models. These advancements have led to substantial improvements in the quality and effectiveness of abstractive summarization systems, showcasing the rapid progress in the field of natural language processing.

Why Do We Need RAG with language models?

Even though LLMs are trained on a vast amount of data, they sometimes fall short due to:

• Outdated Data: The information they are trained on can become outdated.

  • Costly Retraining: Updating the model with new data requires expensive and time-consuming retraining.

• Limited Context: LLMs may lack in-depth industry- or organization-specific context.

• Inaccuracies: LLMs may generate responses that are incorrect, some as grave as hallucinations.

• Lack of Explainability: LLMs can’t verify, trace, or cite sources.

Introducing Retrieval-Augmented Generation (RAG)

RAG is a technique to ground your LLMs to generate responses to your queries based on a custom knowledge-base that you provide. RAG enhances LLMs by allowing them to access up-to-date information from specific sources or an organization’s own knowledge base. This keeps the model's responses accurate and relevant without needing to retrain the whole model.RAG can greatly benefit abstractive summarization with LLMs. The main benefits from RAG can be broadly classified into two categories - factual accuracy and personalization.

Factual accuracy

Retrieval-Augmented Generation (RAG) can boost the factual accuracy of large language models (LLMs) by integrating external, up-to-date information into their outputs. This process ensures that responses are derived from the most current and reliable data, enhancing their trustworthiness and precision.Sometimes, summarizing documents requires additional knowledge that isn't present in the original source document or query. Rather than depending solely on the LLM to utilize the extensive data it was trained on to infer this extra information, we can employ Retrieval-Augmented Generation (RAG) to supply the necessary context. By retrieving relevant information from external databases, RAG greatly diminishes the occurrence of hallucinations[12] or factually incorrect generations, thus enhancing the reliability of the content [13]. This combination of retrieval and generation capabilities allows for responses that are both contextually appropriate and informed by the latest and most accurate information. Consequently, RAG represents a significant advancement in the development of more intelligent and versatile language models. [13,14].

Personalization

Personalizing a Large Language Model (LLM) involves adapting its behavior to better suit the preferences and needs of individual users[15]. As Large Language Models (LLMs) have become more prevalent, there is a growing interest in how to utilize these models to improve personalization. In the context of summarization, tailoring a summary to user preferences is highly desired. One common approach is to incorporate all of historical user data into the model's input. However, this can lead to lengthy inputs that exceed system limits, resulting in delays and increased costs [16]. A no-brainer option is finetuning on user data but this usually ends up being time consuming and, usually causes over-specialization to the fine-tuning tasks, and harm the model’s pre-existing generalization ability on unseen tasks via in-context learning[32]. As a middle ground, current techniques focus on selectively retrieving relevant user data to generate prompts for specific tasks[17]. Recent work has shown promise in combining retrieval approaches with LLMs to improve personalized performance in recommender systems [18, 19, 20], as well as general NLP tasks [21, 22, 23].

RAG framework

The Retrieval-Augmented Generation (RAG) framework encompasses several essential steps. Initially, text inputs, such as documents or queries, undergo an embedding phase where they are converted into numerical representations called embeddings. These embeddings capture the semantic meaning of the text and facilitate efficient storage and retrieval. Subsequently, the embeddings are stored in a database, serving as the foundation for the RAG system's knowledge base. During the indexing phase, the embeddings are organized to enable rapid retrieval, creating an index that maps embeddings to their respective documents or passages. When a user query is presented, the RAG system searches the index to locate the most relevant embeddings. The system then retrieves the corresponding documents or passages from the database, which serve as context for generating a response. In some instances, the retrieved documents may be ranked based on their relevance to the query. Finally, using the retrieved documents and the input query, the RAG system generates a response, often employing pre-trained language models to produce text that aligns with the query's intent. Through these steps, the RAG framework enhances the accuracy and relevance of responses in natural language processing tasks. Let’s go through the three most researched areas here : Embedding creation, retrieval methods and generation.

Retrieval framework

1. Embedding creation

   Embeddings are numerical representations of objects or concepts. In NLP, embeddings are often used to represent words or phrases. These embeddings are generated by algorithms that analyze the context in which words appear in a large corpus of text. The goal is to capture semantic relationships between words, such that similar words have similar embeddings. This allows algorithms to understand the meaning of words based on their context and enables tasks like sentiment analysis, machine translation, and text summarization. For clarity, see the image below for a simple depiction of a high-dimensional vector space:

   

   Embedding models are algorithms specifically created to learn and produce embeddings for a given piece of information. There are different ways of creating embeddings. Sparse methods, like BM25 and TF-IDF, count how many times each word appears in a piece of text to create these representations. However, these representations are very sparse, meaning they are mostly zeros and can be as long as the number of unique words in a language. Sparse methods are easy to understand and use, but they might not be great at capturing the meaning of words. On the other hand, dense methods use more complex models, like neural networks, to create more compact and meaningful representations. These methods, such as REALM, DPR, and Sentence-Transformers, can understand the context and relationships between words better. However, they require more computational power and training time. Luckily, there are many pre-trained models available now, so you don't always have to train them from scratch. For a task like summarization, context has high importance and hence dense methods would produce significantly better embeddings for RAG. With numerous excellent models available, choosing the right embedding model for your RAG application can be challenging. A helpful starting point is the MTEB Leaderboard on Hugging Face [https://huggingface.co/spaces/mteb/leaderboard], which offers a current list of proprietary and open-source text embedding models. This leaderboard provides insights into each model's performance on various embedding tasks such as retrieval and summarization. However, be aware that these results are self-reported and may have been benchmarked on different datasets. Additionally, some models might include the MTEB datasets in their training, so it's advisable to evaluate the model's performance on your specific dataset before making a final decision.

2. Retrieval

   In the retrieval phase, the retriever searches through the vast knowledge base in the database to identify relevant information. The goal is to extract contextually relevant information that can enrich the generated content.The retrieval component aims to extract relevant information from a vast array of external knowledge sources. When the user enters a query or a prompt, it is this system (Retriever) that is responsible for accurately fetching the correct snippet of information that is used in responding to the user query. It involves two main phases, indexing and searching.

   • Indexing organizes documents to facilitate efficient retrieval, using either inverted indexes for sparse retrieval or dense vector encoding for dense retrieval [24,25,26].

   • The searching utilizes these indexes to fetch relevant documents on the user’s query, often incorporating the optional rerankers [26,27,28] to refine the ranking of the retrieved documents.

     According to LangChain’s 2023 State of AI survey, amongst the 6 most used retrieval searching strategies were Self Query, Contextual Compression, Multi-query and time weighted.

Source : LangChain State of AI 2023

• Similarity Search: Calculates distance between input and document embeddings for retrieval.

• Maximum Marginal Relevance (MMR): Reduces redundancy in retrieval, considering new information given previous results.

• Multi-query Retrieval: Uses language model to generate diverse queries for user input, retrieving relevant documents from each query and combining them for comprehensive results.

• Contextual Compression: Squeezes long documents to retain only important parts matching the search.

• Multi Vector Retrieval: Stores multiple vectors in a document for more efficient retrieval, matching with different types of information.

• Parent Document Retrieval: Stores small document chunks for better embeddings, retrieving larger documents using parent IDs during search.

• Self Query: System can ask itself questions, converting them into structured queries for more efficient and accurate searches.

• Time-weighted Retrieval: Supplements semantic similarity search with time delay, giving more weight to fresher or more used documents.

• Ensemble Techniques: Uses multiple retrieval methods in conjunction for improved results, with implementation depending on use cases.

The most optimal method to use will be task, data and domain dependent.

3. Generation

   The generation component utilizes the retrieved content to formulate coherent and contextually relevant responses with the prompting and inferencing phases. The input for language models is formulated through prompting, integrating the query from the retrieval phase. Methods like Chain of Thought (CoT) [29] or Rephrase and Respond (RaR) [30] guide better generation results. In the inferencing step, Large Language Models (LLMs) interpret the prompted input to generate accurate and in-depth responses that align with the query’s intent and integrate the extracted information [31].

Advancements in NLP have led to substantial improvements in the quality and effectiveness of text summarization systems, paving the way for more accurate and contextually relevant summaries. While abstractive summarization techniques continue to evolve, driven by sophisticated models and techniques, RAG further enhances summarization by incorporating external, up-to-date information. By fusing the precision of retrieval and the creativity of generation, we unlock a new paradigm in summarization. As this model navigates context with precision, crafts summaries with creative finesse, and proves its mettle across diverse datasets, it heralds a future where summarization is not merely a task but an art form — a nuanced and intelligent expression of content understanding in the digital age.

References

1. Yen-Chun Chen and Mohit Bansal. Fast abstractive summarization with reinforce-selected sentence rewriting. arXiv preprint arXiv:1805.11080, 2018.

2. Ani Nenkova, Lucy Vanderwende, and Kathleen McKeown. A compositional context sensitive multidocument summarizer: Exploring the factors that influence summarization. In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 573–580, 2006.

3. Ani Nenkova, Kathleen McKeown, et al. Automatic summarization. Foundations and Trends in Information Retrieval, 5(2–3):103–233, 2011.

4. Josef Steinberger, Massimo Poesio, Mijail A Kabadjov, and Karel Ježek. Two uses of anaphora resolution in summarization. Information Processing & Management, 43(6):1663–1680, 2007.

5. Yang Liu and Mirella Lapata. Text summarization with pretrained encoders. arXiv preprint arXiv:1908.08345, 2019.

6. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461,2019.

7. Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In International Conference on Machine Learning, pp. 11328–11339. PMLR, 2020a.

8. Yixin Liu, Pengfei Liu, Dragomir Radev, and Graham Neubig. BRIO: Bringing order to abstractive summarization. arXiv preprint arXiv:2203.16804, 2022b.

9. Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B Hashimoto. Benchmarking large language models for news summarization. Transactions of the Association for Computational Linguistics, 12:39–57, 2024.

10. Liyan Tang, Zhaoyi Sun, Betina Idnay, Jordan G Nestor, Ali Soroush, Pierre A Elias, Ziyang Xu, Ying Ding, Greg Durrett, Justin F Rousseau, et al. Evaluating large language models on medical evidence summarization. NPJ Digital Medicine, 6(1):158, 2023.

11. Dave Van Veen, Cara Van Uden, Louis Blankemeier, Jean-Benoit Delbrouck, Asad Aali, Christian Bluethgen, Anuj Pareek, Malgorzata Polacin, Eduardo Pontes Reis, Anna Seehofnerova, et al. Clinical text summarization: Adapting large language models can outperform human experts. Research Square, 2023.

12. *Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X.,Qin, B., Liu, T.: A survey on hallucination in large language models: Principles, taxonomy,challenges, and open questions (Nov 2023). https://doi.org/10.48550/ARXIV.2311.05232*

13. Zhang, Y., Khalifa, M., Logeswaran, L., Lee, M., Lee, H., Wang, L.: Merging Generated and Retrieved Knowledge for Open-Domain QA. In: Bouamor, H., Pino, J., Bali, K.(eds.) Proceedings of the 2023 Conference on Empirical Methods in Natural LanguageProcessing. pp. 4710–4728. Association for Computational Linguistics, Singapore (Dec 2023). https://doi.org/10.18653/v1/2023.emnlp-main.286, https://aclanthology.org/2023.emnlp-main.286

14. Yao, J.Y., Ning, K.P., Liu, Z.H., Ning, M.N., Yuan, L.: Llm lies: Hallucinations are not bugs,but features as adversarial examples. arXiv preprint arXiv:2310.01469 (2023)

15. *https://arxiv.org/pdf/2310.20081v1*

16. Jiuhai Chen, Lichang Chen, Chen Zhu, and Tianyi Zhou. 2023. How Many Demonstrations Do You Need for In-context Learning? arXiv:2303.08119 [cs.AI]

17. Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. 2023.LaMP: When Large Language Models Meet Personalization. arXiv preprint arXiv:2304.11406 (2023).

18. Zheng Chen. 2023. PALR: Personalization Aware LLMs for Recommendation. arXiv preprint arXiv:2305.07622 (2023).

19. Jinming Li, Wentao Zhang, Tian Wang, Guanglei Xiong, Alan Lu, and Gerard Medioni. 2023. GPT4Rec: A generative framework for personalized recommendation and user interests interpretation. arXiv preprint arXiv:2304.03879 (2023).

20. Jiajing Xu, Andrew Zhai, and Charles Rosenberg. 2022. Rethinking personalized ranking at Pinterest: An end-to-end approach. In Proceedings of the 16th ACM Conference on Recommender Systems. 502–505.Shiran Dudy. 2022. Personalization and Relevance in NLG. In Companion Proceedings of the Web Conference 2022. 1178–1178.

21. Lucie Flek. 2020. Returning the N to NLP: Towards contextually personalized classification models. In Proceedings of the 58th annual meeting of the association for computational linguistics. 7828–7838.

22. Hongjin Qian, Xiaohe Li, Hanxun Zhong, Yu Guo, Yueyuan Ma, Yutao Zhu, Zhanliang Liu, Zhicheng Dou, and Ji-Rong Wen. 2021. Pchatbot: a large-scale dataset for personalized chatbot. In Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval. 2470–2477.

23. Omid Rafieian and Hema Yoganarasimhan. 2022. AI and Personalization. Available at SSRN 4123356 (2022).

24. *Blagojevic, V.: Enhancing RAG Pipelines in Haystack: Introducing DiversityRanker and LostInTheMiddleRanker (Aug 2023), https://towardsdatascience.com/enhancing-rag-pipelines-in-haystack-45f14e2bc9f5*

25. Douze, M., Guzhva, A., Deng, C., Johnson, J., Szilvasy, G., Mazaré, P.E., Lomeli, M., Hosseini, L., Jégou, H.: The faiss library (2024)

26. Khattab, O., Zaharia, M.: Colbert: Efficient and effective passage search via contextualized late interaction over bert (Apr 2020). https://doi.org/10.48550/ARXIV.2004. 12832

27. Lyu, Y., Li, Z., Niu, S., Xiong, F., Tang, B., Wang, W., Wu, H., Liu, H., Xu, T., Chen, E., Luo, Y., Cheng, P., Deng, H., Wang, Z., Lu, Z.: Crud-rag: A comprehensive chinese benchmark for retrieval-augmented generation of large language models (Jan 2024).

https://doi

1. . org/10.48550/ARXIV.2401.17043

2. *Tang, Y., Yang, Y.: Multihop-rag: Benchmarking retrieval-augmented generation for multihop queries (Jan 2024). https://doi.org/10.48550/ARXIV.2401.15391*

3. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., Zhou, D.: Chain-of-thought prompting elicits reasoning in large language models (Jan 2022). https://doi.org/10.48550/ARXIV.2201.11903

4. Deng, Y., Zhang, W., Chen, Z., Gu, Q.: Rephrase and respond: Let large language models ask better questions for themselves (Nov 2023). https://doi.org/10.48550/ARXIV.2311.04205

5. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis,M., Yih, W.t., Rocktäschel, T., Riedel, S., Kiela, D.: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Tech. rep. (Apr 2021), http://arxiv.org/abs/ 2005.11401, arXiv:2005.11401 [cs] type: article

6. https://arxiv.org/html/2211.00635v3#:~:text=The most common practice to,tasks via in-context learning.

It seems like every team that I’ve been on holds some flavor of daily “stand-up” meeting. Especially once hybrid and remote work became necessarily en vogue, a daily standing meeting seemed a panacea to a whole host of modern organizational ailments. Thus a new breed of meeting was born, justified with all the original big-meeting killing rationale of Agile™️ processes, but often missing the forest for the trees, driving teams back into the daily slump of boring meetings. It’s worth raising the question of whether your own well-meaning daily exercise has decayed into something that no longer serves its intended purpose - it happens easily!

Stand by me!

As my engineering dinosaur feathers grow in, I recall a distant time when my team would gather around our lead’s desk in a tight circle every day in the morning. Each of my teammates had straggled in, oriented themselves to what was on their plate for the day, and came ready to talk. We exchanged information with tight, cross-talking fluidity - and the name wasn’t a coincidence! It was an actual “standing” meeting, in that we would be on our feet, giving us all a little more pressure to keep the meeting quick, functional, and effective.

Fast forward a few years, and though most of the team was still in the same office, technology allowed teams to become vastly distributed across time and space. Remote communication made the distributed standup possible for the first time, but it became much trickier to pull off effectively. For one thing, there simply was no “first thing in the morning” that worked for everyone, so while SF engineers were just starting their day, folks on the East Coast were already finished with lunch. For a hybrid team, if part of the team WAS in-person, it meant the remote folks needed to be represented by a laptop or situated far against the wall on a screen. And for the remote callers on the other side of that screen, the daily ritual became entirely detached, with A/V issues often making their compatriots seem even further away. I remember being the guy on the screen when I first transitioned into a remote role - couldn’t help but feel like the kid standing just outside the circle on the playground.

Eventually, fully remote teams became far more the norm for me, and customary stand-ups followed, though by this point they had become something pretty far removed from the rose-tinted desk huddles of my distant memory. The daily video call was seen as an exercise de rigeur, even as its effectiveness dwindled.

So why are we still imposing this small daily burden? Well, it does have its good points…

When it’s good

• Team members can effectively communicate important status updates, speaking to a tight circle of relevant, interested, people. Information transfer happens fast, eliminating the need for folks to chase each other down later.

• Managers can predict team demands, paying attention to who needs to be unblocked, and arranging exceptional support, just in time.

• By virtue of sharing every morning together, remote teams can build rapport, making the other informal communication that teams thrive on happen naturally.

When it’s bad

• The audience in the room has grown so large and generalized, that many people just aren’t concerned with a large portion of the others’ day-to-day work, and can’t even really remember what everyone has said even if they wanted to. Peripheral players get included just for the sake of inclusion and team building, even if their work is not relevant to others on a day-to-day basis. Meetings drag on endlessly so everyone gets their turn.

• Team members start viewing their “share” as less of a way to exchange vital information others need, and more as a laundry list of boxes that have been, or will be checked.

• Most of the communication devolves into people speaking directly to their manager, with an inherent (usually false) assumption that every other team member is listening intently and effectively, absorbing each cumulative update.

When it’s ugly

• The standup is considered the one and only time during the day that the team gets together, and so starts filling the place of other, more casual and focused small group decision meetings. As the duration stretches out longer, participants’ attention wanders to the myriad distractions lying in other windows. Everyone looks effectively busy, while important decisions get lost in the chaos.

• Managers view the standup as an opportunity to re-prioritize an IC’s work on a day-to-day basis, eroding their pillar of autonomy before the first cup of coffee.

• Status exchange effectiveness grinds to near zero, with team members having to chase each other down for followup conversations, despite holding a daily, expensive meeting

What to do?

Knowing that these meetings can quickly lose their effectiveness once a distributed team grows, what can be done to retain a stand-up meeting’s value?

Keep the guest list tight

Once there are about 7 people in the circle, the group starts to reach the limits of attention and memory. One of our Engineering Managers, Catherine Fleres, likes to abide by the “Two-Pizza Rule” - if it takes more than that to adequately feed everyone in a meeting, there are too many people at the table! Strive to keep standup groups to just relevant stakeholders in a focused topic. Organize into squads, and split meetings ruthlessly. If there are other, larger team-wide broadcasts that need to be made, consider alternate channels or meeting formats.

Consider actually standing

Teammates think I’m crazy when I suggest this, because it involves an extremely awkward coordinated dance of adjusting cameras, changing desk heights, and just generally looking silly in a room alone. Strange as it seems, the physiological pressure of being in a less-than-leisurely body pose can help keep shares nice and short - direct and to the point.

Take discussions offline… ruthlessly …

It starts innocently enough: a status update sparks a question, and the answer is detailed and complex, which sparks a follow-up from the original speaker. Before they even realize it, the two participants have taken the meeting off the rails, and everybody else’s eyes are slowly glazing over as they hash out the finer points. Develop a team culture for tabling those important conversations compassionately, and following up after the group has dispersed. It can feel rude to have to police this, so it’s especially important that everybody is bought in and held accountable, and understands a shared signal to move it along. Encourage followups to happen in a separate “space”, so that there is a clear signal that uninterested parties are not obligated to attend.

Optimize for broadband communication

Standing meetings can sometimes grow large to accommodate the schedule of a single individual who has become the hub from which all others are spokes. The entire meeting degrades from n to n communication to serial bursts of 1:1 exchanges with that single person. This is often indicative of larger organizational chokepoints, and a good hint it’s time to delegate responsibility more broadly and spread the load.

Make space for other group collaboration

Is the daily standup the only place where group communication happens on your team? Do decisions get rushed, because they have to be made while a roomful of disinterested folks wait for their turn? Do important announcements get forgotten, buried in a laundry list of mundane updates? Make room for the longer discussions, note their importance, and encourage them to happen in a properly framed meeting venue with strictly the interested parties.

Double down on autonomy

It’s common to justify a daily gathering as a way of tracking team accountability. Tickets closed, code merged, charts burnt down. Around the table, the ICs get the message that they need to justify their existence (perhaps to their manager?), and so their shares become a list of yesterday’s accomplishments. The meetings transform into a status dump for a theoretically rapt listener than effective team communication. Think about this - in your own daily meetings, to what audience is the speaker framing their share? Do participants seem like they’re collaborating and engaging with each other, or mostly to the manager? If your meeting culture expects every detail to be reported up the chain on a daily cadence, it can easily undermine an IC’s sense of work autonomy. It leaves the door open for unwelcome scrutiny, micro-management, and ultimately starts to wear on motivation and engagement.

While managerial demands for daily status checks seem reasonable, they are often unnecessary, and reinforce the notion that work could deviate wildly from sprint plans at any time. I like the analogy to a cruise ship; you can make course corrections on the scale of sprint cycles, but the ship, like your dev’s psyches, may be too big to turn on a dime. Escaping the notion of “stand-up as status report” can help managers plan more realistically, and workers feel safe in predictable work expectations. Consider incorporating automated status tracking into organization tooling, so that a developer’s normal workflow triggers updates in a board or charts visible to managers. This can help alleviate the concern that work isn’t happening, even over the distances that remote teams require.

If all else fails, Slack up!

A common solution to the woes of unfortunate stand-ups is to switch to an “async” model, where team members leave their updates in a Slack channel or other team collaboration tool, as they come online for the day. This format alleviates a lot of the pains of having to convey status to others without making awkward time demands, so is a favorite of the meeting-averse. However, this is hardly a suitable replacement. In theory, each team member is consuming everyone else’s updates, but in practice that rarely holds. The features that defined the format get lost - psychological pressure keeping updates brief and effective has vanished, the laundry list mentality gets exacerbated, and it’s way too easy to let the audience in the channel grow too large. This is often the last iteration of the meeting format before the whole exercise is scrapped as unhelpful.

Standing together as a team

Whether you’re analyzing user experience or plotting out team processes, success or failure depends greatly on how your stakeholders feel. When engineers report “feeling” burnout, often the first thing they call out is energy-sapping meetings - being held captive on camera while other people talk about irrelevant distractions. On the other hand, some teams will point to a daily meeting as the time they feel most like a single, sleek operating unit, and this carries through the rest of their day. A well-run standup can go a long way towards maintaining sanity in your schedule, realistic expectations of your peers, and predictable synergy on your team. Invest in doing it right.

Subscribe now

Identifying the problem and having a clear end result.

The first step in creating a meaningful application is identifying a concrete problem. This might seem straightforward, but it's where many teams need to improve. The importance of this foundational step cannot be overstated. Successful ML applications are not just about leveraging advanced algorithms or having access to vast datasets; it's about applying these tools to solve a specific, well-defined problem that has real-world implications for patients and providers.

A practical example of this concept is Rad AI Omni Impressions, a tool designed to assist radiologists by generating high-precision summaries of their reports known as ‘Impressions’. The creation of these ‘Impressions’ is normally repetitive and time consuming for radiologists. Impressions specifically aims to alleviate the burden during this phase of the workflow, focusing on tangible goals such as reducing fatigue and enhancing efficiency.

Similarly, while ML has tremendous transformative potential, it is worth cautioning against the allure of technology for technology's sake (Topol, 2020). The key is not just to innovate but to innovate with purpose and, as discussed later, with a strong proposed Return on Investment (ROI) over the status quo. The development of ML applications must be driven by a clear understanding of the challenges they aim to address and the tangible benefits they seek to deliver.

The process of problem identification involves not only understanding the clinical need but also engaging with stakeholders, including healthcare professionals, patients, and policymakers. This engagement ensures that the ML solution is not only technologically advanced but also practical, usable, and aligned with the needs of those it aims to serve. Furthermore, setting clear, achievable goals is essential. These goals should be specific, measurable, attainable, relevant, and time-bound (SMART), providing a roadmap for the project's development and implementation.

A real-world example of this approach can be seen in the development of ML applications for diagnosing diabetic retinopathy. The problem is clear and well-defined: early detection of this condition can prevent severe visual impairment. The goal, therefore, is to develop a tool that can accurately and efficiently screen patients for early signs of diabetic retinopathy, facilitating timely intervention. This example underscores the importance of beginning with a concrete problem and having a clear end result in mind.

Ensuring data represents real-world situations.

Data is undoubtedly the lifeblood of any AI application, but it's a whole different ball game when it comes to healthcare. Here, sheer volume of data isn't the only game in town. It's the diversity and quality of that data that truly makes a difference, ensuring it accurately reflects the myriad complexities and nuances of real-world medical scenarios.

For LLMs and other contemporary model architectures to truly excel, they need more than just a large amount of data. They need the correct data. Just like in any educational setting where the quality of materials determines the quality of learning, LLMs require high-quality, well-curated data to thrive. This isn't just about feeding them information; it's about educating them with data that's as close to the real deal as possible.

Drawing from the insights provided in the paper "Exploring the Impact of Data Quality on Large Language Models" (arXiv:2306.11644) it becomes evident that the 'nutrition' we provide to these AI models through data significantly impacts their performance across the board. The paper eloquently discusses how nuanced, high-quality datasets serve as the optimal nourishment that enables LLMs, particularly in the healthcare domain, to achieve better performance. It's akin to the difference between fast food and a balanced diet for humans; the quality of the input directly affects the output.

In practice, this involves using targeted datasets created in real-world workflows, and applying domain knowledge to highlight the most meaningful parts of this data. At Rad AI, we have access to hundreds of millions of radiology reports. However, effectively utilizing them is more complex than simply feeding them into a model training pipeline. To focus the data in a way that reflects what you want your model to learn, you must first understand the data you're starting with.

Imagine teaching a medical student with outdated textbooks and then expecting them to perform complex surgeries. The outcome likely wouldn't be great. The same goes for models. If we're aiming for AI applications that not only support but enhance healthcare delivery, we must prioritize the impeccable quality and variety of data we provide. This ensures that these models can handle the complexities of healthcare, offering data-efficient approaches that significantly surpass their predecessors.

Have an ROI pitch.

The theoretical promise of AI in healthcare is potentially enormous. However, the tangible results are often elusive. This is one of many reasons why ROI is a core component of anything worth building. To be clear, we are not advocating ROI in purely financial terms. In fact, the best clinical-facing offerings tend to start with non-financial ROI and then supply financial benefits as an additional consideration.

It is crucial that you not only have a solid idea of your ROI proposition but also work on how you will communicate this to the stakeholders in the healthcare institutions you will need to work with. It will not matter if your model is 99% accurate if you cannot or do not convey how this moves the needle in practical terms.

For instance, when we converse with medical providers and administrators, we don't discuss vague outcomes or intangible benefits, even though these areas hold considerable potential. Instead, we focus on concrete results. For Omni Impressions, we highlight quantifiable reductions in fatigue and increased accuracy in reporting. For Rad AI Continuity, we emphasize measurable patient follow-up rates and real-world patient outcome statistics where Continuity played a key role.

Developing ML models for healthcare is a multifaceted endeavor that goes beyond technical achievement and requires a highly multidisciplinary team. Your ROI pitch should be grounded in tangible benefits like improved patient outcomes, operational efficiency, and financial savings. Further, the more you are able to tie this to quantifiable outcomes and existing industry KPIs, the more straightforward your conversations will be. A significant caveat to this, though, is that you should not feel limited to these metrics; there are, of course, a multitude of benefits that will be just as, if not more, impactful but difficult to quantify.

Build with innovation partners.

Once you are at the point of having end users engage with some version of your product, you want to start looking for and building with individuals and teams that are not just end users of your product but are willing to innovate alongside your team. The part that makes all the difference in the world here is engagement—the vision of the potential benefits of what your project could be and not necessarily what it currently is.

The building of these applications tends to be a very iterative process. Therefore, you will want people who will not only have some patience for the inevitable pitfalls as biases and edge cases are worked through. The trade of here is you are asking for additional help from your end users who can not only point out things you have not thought of but also provide ample amounts of constructive feedback to help hammer your product from a dented piece of metal to a smooth and shiny offering that provides many more upside benefits than the original prototype.

This process is exemplified by several radiology groups that have been partners of Rad AI for a long time, some since the company's earliest days. The radiologists we collaborate with not only act as end users but also help us tailor our tools to be as effective and purposeful as possible. We actively engage with our users and value their feedback, striving to make the development process beneficial for both parties.

While this refinement process can occur without such engagement, and you should only offload some of the debugging process onto your end users, a more collaborative approach in the iterative process can accelerate progress on core issues and result in a superior end product.

Test and monitor thoroughly.

Thorough testing and validation are non-negotiable. This process begins with offline validation and should ideally include human validation to assess the application's real-world efficacy. The advent of generative AI (GenAI) introduces the need for innovative performance monitoring metrics, necessitating creative approaches to assessment.

In our deployments, we utilize a wide range of both offline and online metrics to maintain quality and consistency. Besides traditional text-based metrics like BLEU and ROUGE, we've also implemented our own custom metrics. These focus on the content and meaning of the text generated by our models, rather than just token-level accuracy. Creating custom metrics is a domain-specific and time-consuming process, but it can be a worthwhile investment. This is especially true for generative models, where more traditional metrics may fall short.

Deployment is just the beginning. The real test comes when the application faces production data. Monitoring performance necessitates a blend of qualitative feedback and quantitative metrics, focusing on the solution’s performance and the synergy between the AI and its human users. Adjusting to feedback and data drift is an ongoing process essential for the application's evolution.

Embrace Human-in-the-Loop systems.

While the allure of full automation is strong, the reality of healthcare demands a much more grounded approach. Human-in-the-loop systems represent a balanced integration of AI and human expertise, capitalizing on the strengths of each. These systems have proven to be the most effective in scenarios requiring complex reasoning or nuanced judgment. From both ethical and practical considerations, clinicians are still ultimately responsible for how they choose to use AI-powered tools. As powerful as they are becoming, these new features are still just that: tools. These systems should be looked at as additional tools used as input to the overall decision-making process, not something that is meant to supplant it.

We've deployed numerous ML driven applications and features. However, most still incorporate human input within the workflow. Despite their capabilities, modern language models aren't yet ready to operate independently. For instance, in Omni Imressions, a radiologist still reviews and signs off on the impression. In Continuity, a medical professional ensures the validity of patient follow-ups. While some argue that this approach limits the potential of ML augmented workflows, the success of Rad AI and many others in the industry proves otherwise. These ML-augmented workflows continue to offer significant benefits relative to baseline.

Expanding on this, we also establish systems that interpret human interaction with our model outputs, which can then be used as direct human feedback. Beyond just analytics, this approach enables you to feed production data back into your models. This allows for ongoing improvement, ensures data remains current, and minimizes issues with data drift and models becoming outdated.

There are many additional considerations such as appropriate transparency on limitations, training on how to use, and the potential limitations that should also be considerations in the deployment of AI-powered technologies both in and outside of the healthcare domain.

The responsibility for appropriate and responsible implementation of these powerful tools does not rest on any one party, as with so many other areas of society. This burden is a shared commodity. The people involved at all stages of the process, ranging from developers, clinicians, regulators, and, in some instances, the patients, should all be vigilant to how these autonomous systems are changing how we approach healthcare (Herrington, 2023).

Conclusion

Returning to the example of a screening solution for diabetic retinopathy, the solution needs to be introduced at the right steps in the patient journey (at times where retinal images are being acquired, or being made available in retrospect). Deployment and training needs to be done with specific clinical stakeholders (potentially optometrists, ophthalmic technicians, nurses, primary care physicians or Physician Assistants (PAs), depending on the complexity of product training and use, and the route of introduction into the clinical workflow). Product and data security and privacy need to be ensured, alongside any necessary clearances for that specific geographic market.

The production data used for the screening solution needs to be readily available in a consistent format effectively represented by the ML models’ training data, or acquired specifically for the purpose of screening. Clinical stakeholders and executive leadership at optometry offices, primary care offices and/or health systems need to buy into the necessity and benefit of adoption—and will likely need to see specific measurable ROI beyond the improvements to patient care.

Incorporating artificial intelligence into the healthcare sector comes with its set of difficulties, but it also offers significant opportunities for important changes. While developers must overcome these hurdles, which might involve acquiring new knowledge and adopting novel approaches, successful outcomes generally justify the additional effort involved.

Subscribe now

Share

Leave a comment

Subscribe now