Evaluation of Generative Medical Artificial Intelligence based on Fine-tuned Large Language Models (LLM)

Abstract

The integration of large language models (LLMs) into medical applications has opened new frontiers in clinical decision support and patient management. Fine-tuned generative models can significantly improve diagnostics and personalized care but require rigorous evaluation for reliability and accuracy. While LLMs excel in natural language generation, their medical applications face challenges, including accuracy, clinical relevance, and ethical use. Ensuring these models align with real-world practices and data privacy is crucial for adoption. This study fine-tunes a GPT-style LLM using a curated medical dataset, comprising publicly available resources (e.g., PubMed articles, clinical guidelines, medical textbooks, electronic health records (EHR). Etc.) and anonymized patient records. The dataset was curated by removing irrelevant data, anonymizing sensitive information, and annotating key medical concepts for improved model learning. The model's performance was evaluated across tasks like generating medical summaries and proposing diagnoses, using accuracy, precision, recall, F1 score, and human evaluations as key metrics. The fine-tuned model achieved an accuracy of 85% in generating patient summaries and 80% for differential diagnostic suggestions. The F1 score was 0.82, reflecting a good balance of precision and recall. However, in complex cases, accuracy dropped to 70%, and factual inaccuracies emerged. Ethical concerns, such as biases in training data, were also noted, emphasizing the need for ongoing improvements and oversight. Generative medical AI based on fine-tuned LLMs holds significant potential for enhancing healthcare delivery. Nonetheless, challenges related to accuracy, transparency, and ethical compliance must be addressed before integration into clinical practice. Future work will focus on improving model robustness and alignment with regulatory standards.