https://advenn.github.io/blog/tags/data-processing/Recent content in Data-Processing on Asadbek KurbonovHugo -- 0.148.0en-usMon, 24 Nov 2025 22:00:00 +0100https://advenn.github.io/blog/posts/go-python-architecture/Mon, 24 Nov 2025 22:00:00 +0100https://advenn.github.io/blog/posts/go-python-architecture/<p><em>This is Part 2 of a series. Read <a href="../pandas-vs-polars-in-production/">Part 1: Pandas vs Polars in Production - Performance Comparison</a> for the background on the Polars migration.</em></p> <hr> <p>After migrating from Pandas to Polars, CPU performance improved dramatically—but a memory problem persisted. Despite extensive debugging, I couldn&rsquo;t identify the root cause. So I made a pragmatic decision: architect around it.</p> <p>This is the story of splitting a monolithic Python application into a Go orchestration service with Python workers, not because I fully understood the problem, but because I needed production to be stable.</p>https://advenn.github.io/blog/posts/pandas-vs-polars-in-production/Sun, 23 Nov 2025 23:02:39 +0100https://advenn.github.io/blog/posts/pandas-vs-polars-in-production/<p>When performance bottlenecks started affecting my production data pipeline, I decided to test whether Polars could deliver on its performance promises. This is what I learned from migrating a real production workload from Pandas to Polars.</p> <h2 id="the-workload">The Workload</h2> <p>The application was a data aggregation service running as a Kubernetes pod with the following constraints:</p> <ul> <li><strong>Resources</strong>: 2 CPUs, 3 GB RAM</li> <li><strong>Execution frequency</strong>: Every 2-2.5 minutes</li> <li><strong>Data volume</strong>: 5,000-7,000 rows × 100-150 columns per run</li> <li><strong>Operations</strong>: Multiple database calls, API requests, DataFrame merges, arithmetic operations (additions, multiplications), and group-by aggregations</li> <li><strong>Web server</strong>: FastAPI with Uvicorn handling production traffic</li> </ul> <p>All operations were properly vectorized-no row-by-row iteration. The pipeline combined data from various sources into a single DataFrame, transformed it, and output the results.</p>