Undisclosed Project #4

My Role: Tech Lead & Sole Developer

Technologies

Project Overview

This project was developed as an extension to a previous project (Undisclosed Project #2). The primary goal was to enhance and streamline the workflow by building a web application that allows users to upload data, process it efficiently, and visualize results interactively. The entire application was designed with performance, scalability, and ease of use in mind, leveraging modern web technologies and Azure cloud services.

My Role

As the sole developer and tech lead, I was responsible for all technical decisions, architectural design, and implementation. This included selecting the technology stack, optimizing data processing pipelines, and ensuring high performance when handling large datasets. I worked closely with stakeholders to align the application with business requirements and user needs.

Technical Project Description

The application follows a standard cloud-based architecture, integrating multiple Azure services for data storage, processing, and authentication. The core functionalities include:

• Authentication & User Management: Implemented using Microsoft Entra External ID, ensuring secure access for different user roles.
• Frontend: Built with Next.js 15 (App Router), React, and Tailwind CSS. Shadcn/UI was used for modern, accessible UI components.
• Backend & Database: Utilized Prisma with an Azure PostgreSQL database for structured data management.
• File Storage & Processing:
  • Large data files are uploaded to an Azure Storage Account.
  • Azure Synapse Analytics extracts relevant information from uploaded files into the database.
• Data Visualization: Integrated Deck.gl for high-performance map rendering, ensuring smooth interaction even with hundreds of thousands of data points.
• Search Optimization:
  • Used FlexSearch for client-side search functionality.
  • Implemented background indexing tasks to keep the UI responsive during large data loads.
• Performance Enhancements:
  • Optimized data handling using HyParquet, allowing efficient querying of Parquet files.
  • Only parts of a file are read initially to enable quick rendering of maps while loading continues in the background.
• Observability & Monitoring: Implemented Sentry for logging and error tracking to ensure stability and quick issue resolution.
• Deployment & Hosting:
  • The frontend is hosted on Vercel, leveraging server-side rendering (SSR) for better performance.
  • Backend components run on Azure Static Web Apps and Azure Synapse Analytics for scalable data processing.

Challenges

• Handling Large Datasets: The application needed to handle hundreds of thousands of data entries efficiently. This was achieved by:
  • Using Parquet files to store structured data compactly.
  • Ensuring only relevant portions of the data are read into memory when displaying maps.
• Optimizing Map Performance:
  • Deck.gl was integrated to provide smooth rendering and interactivity, even with large datasets.
• Fast & Efficient Search Implementation:
  • A frontend search had to be implemented for mapping data.
  • FlexSearch was chosen for its high performance, but indexing had to be offloaded to background tasks to ensure the UI remained responsive.
• Ensuring High Availability & Observability:
  • The application was designed with Azure-native observability tools.
  • Sentry was used for error tracking, helping quickly diagnose and resolve issues in production.

Achievements

• Extended the LMO Scoring Project: Successfully built an application that enhances the existing scoring system with an interactive and scalable interface.
• Massive Performance Gains:
  • Achieved 20x speedup in data processing compared to handling data manually.
  • Reduced memory consumption by 50%, enabling smooth handling of large datasets.
• Improved Data Visualization:
  • Integrated Deck.gl, allowing visualization of significantly larger datasets.
• Optimized Search Capabilities:
  • Implemented background search indexing, significantly improving UI responsiveness.
• Robust Authentication & Security:
  • Integrated Microsoft Entra External ID for secure authentication and role-based access control.
• Scalable & Cloud-Native Deployment:
  • Fully deployed using Azure services and Vercel, ensuring high availability and low operational overhead.

Current Status

The application is live and in active use, continuously improving based on user feedback. It provides a significant efficiency boost for end-users, making the scoring and data analysis workflow seamless and scalable. Future iterations may include further optimizations and expanded functionality based on additional business needs.