Libraries, speed & errors
Polars & Arrow
Polars, calamine and Arrow for faster reads and columnar pipelines.
Start with the full guides
The long-form treatment of polars & arrow, with the reasoning the shorter pages assume.
Step-by-step how-tos
One task each, with code you can copy, run and adapt.
- How-toClean and Aggregate Excel Data with Polars ExpressionsClean a messy Excel export with Polars expressions — trim and normalise text, parse amounts typed with currency symbols and thousands separators, find the values that failed, add conditional columns, then aggregate with group_by, window shares and a pivot, and write the summary back to Excel.
- How-toClean Wide Excel Sheets with Polars Column SelectorsWide Excel sheets with a column per month, region or measure are tedious to clean column by column. Polars column selectors pick columns by type, name pattern or position and combine with set operations, so one expression can trim every text column, fill every numeric gap, total every monthly column, rename a whole block and unpivot it — and keep working when next month adds a column.
- How-toConvert pandas Excel Code to PolarsPort an existing pandas Excel report script to Polars step by step: read_excel and to_excel equivalents, filtering, new columns, groupby and merge, pivot and fillna, the index that Polars does not have, NaN versus null, apply calls that become expressions, and a test that proves the new script produces the same workbook figures as the old one.
- How-toJoin and Pivot Excel Data with PolarsCombine and reshape workbook data with Polars: read several sheets, join them with validate to catch duplicate keys, find unmatched rows with an anti join, group and aggregate, pivot quarters into columns, unpivot back, run the pipeline lazily, and write a formatted result with write_excel — with the pandas equivalents alongside.
- How-toRead an Excel File with polars.read_excelRead xlsx into Polars: pick sheets, set the header row, control column types with schema_overrides, handle dates and nulls, and convert to pandas when you need to.
- How-toRead Multiple Excel Sheets with PolarsRead the month sheets of a workbook into one Polars DataFrame — list sheet names without loading data, pick sheets by pattern, force consistent column types across sheets, stack them with the right concat mode, turn sheet names into dates and repeat across many workbooks.
- How-toScan and Filter Many Excel Files Lazily with PolarsPolars has scan_csv and scan_parquet but no scan_excel, so a folder of workbooks cannot be queried lazily as-is. This page reads only the columns you need from each file with the calamine engine, tags rows with their source file, builds one LazyFrame for filtering and aggregation, and adds a Parquet cache that turns every later run into a true lazy scan with projection and predicate pushdown.
- How-toUse Polars Window Functions on Excel DataAdd group-level figures to every row of an Excel table without collapsing it: Polars over() for share of group total, rank within group, running totals, month-on-month change and top N per group, the order_by and mapping_strategy options that control how windows behave, and the same calculations side by side with pandas transform.
- How-toWrite a Polars DataFrame to Excel with FormattingUse DataFrame.write_excel to produce a formatted workbook: number formats, autofit, Excel tables, conditional formats, multiple sheets, totals rows and charts.
- How-toConvert Excel Files to Parquet with PythonTurn slow, repeated Excel reads into fast columnar ones: convert workbooks to Parquet with pandas or Polars, keep types, partition by period, and refresh incrementally.
- How-toSpeed Up pandas Excel Reads with the calamine EngineSwap openpyxl for calamine in pandas.read_excel: install it, measure the difference, understand the trade-offs, and know which reads it will not help.
Compare the options
Which tool or approach to pick, and why.
- ComparisonExcel vs CSV vs Parquet for Python Data PipelinesExcel is a delivery format, not a storage format. Compare types, size and read speed, and use Parquet in the middle of the pipeline with .xlsx only at the end.
- Comparisonpandas vs Polars for Excel WorkflowsBoth read Excel through calamine and write through xlsxwriter, so the real difference is the transform. Compare the APIs, the speed, and how to move between them.
When it breaks
The errors this topic produces, with the cause explained rather than just the fix.
Follow it through a track
Prefer an ordered route? These tracks cover polars & arrow along the way, in the order that builds on itself.
Related topics
Subjects that keep turning up on the same pages as this one, most overlap first.
Polars & Arrow is part of Libraries, speed & errors.