Undisclosed Project #2

My Role: Software Developer & Data Scientist

Technologies

Project Overview

This project was carried out for a prominent strategy consultancy specializing in media and publishing. The goal was to optimize last-mile delivery processes through geo-based scoring methods, enabling the analysis and visualization of large datasets in an interactive mapping interface. The existing solution, originally built with Python and pandas, was slow, memory-intensive, and difficult to maintain. By re-engineering the codebase, I achieved a 20x performance improvement, reduced memory usage by 50%, and enhanced scalability, enabling the visualization of datasets exceeding 500,000 entries.

Beyond technical development, I also advised the client on productization strategies and the feasibility of incorporating AI-based features for future use cases.

My Role

As the sole developer and technical lead, I made all technology-related decisions based on the client’s requirements. My responsibilities included:

• Refactoring and optimizing the entire computational pipeline
• Implementing scalable and efficient data processing techniques
• Designing and developing an interactive web application
• Ensuring smooth handling of massive geospatial datasets
• Consulting on productization and future AI integration

Technical Project Description

Codebase Optimization

• Refactored the entire Python-based codebase, eliminating inefficiencies and improving maintainability.
• Introduced software tests to ensure the accuracy and consistency of scoring computations.
• Applied advanced vectorization techniques, optimizing large-scale calculations and significantly improving runtime.
• Reimplemented the core computational logic using efficient libraries and FFT-based convolution/sum-pooling, enabling the performance gains.

Interactive Mapping

• Migrated the existing Folium-based visualization to Kepler.gl, allowing for smooth rendering of hundreds of thousands of data points.
• Designed and implemented a Next.js-based web application with React, Tailwind, and deck.gl for an interactive user experience.
• Optimized frontend performance for handling large datasets efficiently, including implementing lazy loading and background indexing.

Geo-Based Scoring System

• Developed and optimized scoring algorithms to rank buildings based on profitability for last-mile delivery.
• Integrated geospatial data processing with libraries such as GeoPandas and PyProj to ensure accuracy and performance.
• Ensured results were visualized effectively through dynamic mapping solutions.

Consulting on Productization

• Provided guidance on a scalable cloud architecture for deployment.
• Evaluated AI-driven predictive modeling as a future enhancement.
• Helped the client structure their offering for easier integration into their existing business processes.

Challenges

• Performance Bottlenecks: The legacy code was extremely inefficient, requiring a complete overhaul.
• Scalability Issues: The original system struggled with datasets exceeding a few thousand entries, whereas the new solution handles over 500,000 seamlessly.
• Code Maintainability: The original implementation was a tangle of nested functions, which I replaced with a modular, testable, and well-structured codebase.
• Frontend Performance: Ensuring fast loading and smooth interaction for massive datasets required background indexing and lazy loading techniques.

Achievements

• 20x Performance Improvement: Optimized computations for a dramatic speed increase.
• 50% Memory Reduction: Reduced resource consumption, enabling processing of much larger datasets.
• Scalable Interactive Mapping: Successfully transitioned from Folium to Kepler.gl, allowing seamless visualization of hundreds of thousands of points.
• Robust Testing: Introduced unit and integration tests with pytest, ensuring reliability.
• Client Enablement: The consultancy can now deliver faster and more accurate insights to their customers, giving them a competitive edge.

Current Status

The core system is fully operational, significantly improving the client’s ability to analyze last-mile delivery feasibility. Ongoing discussions are in place regarding further enhancements, including AI-driven predictive analytics and automated recommendations for delivery route optimization. The interactive mapping solution has been well received, and future iterations may include additional cloud-based capabilities for further scalability.