Undisclosed Project #3

My Role: Tech Lead

Technologies

Project Overview

This project aimed to develop a prototype for a digital, AI-assisted system that simplifies the process of creating medical assessments. The focus was on exploring key AI use cases, such as transcription and report generation, while designing a lightweight and scalable solution. Given the constraints of limited resources and a tight timeline, the team worked intensively to deliver a functional proof of concept.

My Role

As the Tech Lead and sole developer, I was responsible for all technical decisions and implementation. My role encompassed system architecture, AI model selection and integration, frontend and backend development, and optimizing workflows for performance and scalability.

Technical Project Description

The system was designed with a modular and scalable architecture to process long (>1 hour) medical interview recordings efficiently. The key components were:

System Design and Architecture

• Designed a lightweight, modular system optimized for rapid iteration and future scalability.
• Built a distributed processing pipeline to handle large audio transcriptions and AI-powered assessments efficiently.
• Ensured the system could be deployed both in cloud and on-premise environments for strict data privacy compliance.

AI-Powered Medical Assessment Workflow

1. Transcription:

   • Implemented WhisperX for speech-to-text conversion, including speaker diarization and word alignment.
   • Optimized transcription pipelines to handle long-form audio efficiently.

2. Preprocessing & Summarization:

   • Developed multiple preprocessing strategies, including summarization, bullet points, semantic segmentation, and Retrieval-Augmented Generation (RAG) to enhance transcript usability.
   • Each preprocessing strategy was implemented as a separate LangChain chain for flexibility.

3. Medical Assessment Decision Support:

   • Integrated LLaMA and other generative AI models to draft structured medical assessments.
   • Used various AI techniques to extract meaningful insights from transcripts, enabling semi-automated decision-making.

Frontend and User Interaction

• Developed a Progressive Web App (PWA) with Next.js and React, ensuring a responsive and user-friendly interface.
• Implemented key features such as:
  • Secure audio file uploads and real-time transcription visualization.
  • Editable assessment previews for user validation.
  • Search and filtering for efficient data retrieval.

Backend and Data Management

• Built the backend with Supabase to manage authentication, storage, and CRUD operations.
• Integrated Azure Container Instances for flexible AI workload deployment.
• Ensured data privacy compliance through strict access control mechanisms.

AI Experimentation and Evaluation

• Integrated LangFuse for tracking model performance, experiment logging, and dataset evaluations.
• Designed evaluation datasets with real transcripts and applied metrics like Word Error Rate (WER), F1-score, and accuracy to measure system effectiveness.
• Implemented dataset runs in a CI/CD pipeline, allowing for automated AI model performance benchmarking.

Workflow Automation and Scalability

• Used Prefect to orchestrate the end-to-end AI pipeline, including:
  • Retries, queuing, and rollbacks for robust data processing.
  • Parallel execution to ensure scalability and responsiveness under heavy workloads.

Challenges

• Processing Long Audio Files: Optimizing transcription and AI processing for multi-hour recordings while maintaining accuracy.
• High Performance UI for Large Data: Handling large transcriptions efficiently in the frontend without performance bottlenecks.
• AI Model Evaluation: Designing an effective framework to compare multiple AI models and improve accuracy iteratively.
• Data Privacy: Ensuring compliance with strict medical data regulations, including an on-premise deployment option.

Achievements

• Rapid Development: Delivered a working proof of concept in record time, demonstrating the feasibility of AI-assisted medical assessment workflows.
• Significant Efficiency Gains: Automated transcription and report generation reduced manual workload significantly.
• Scalability & Flexibility: The modular architecture allows for easy expansion and adaptation to new medical assessment criteria.

Current Status

The prototype successfully demonstrated the effectiveness of AI-powered medical assessments and has laid the foundation for a potential full-scale implementation. Future work includes refining the AI models, improving UI/UX, and integrating more advanced clinical decision-support features.