[{"data":1,"prerenderedAt":1977},["ShallowReactive",2],{"doc:\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fexport-a-large-database-table-to-excel-in-chunks":3,"surround:\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fexport-a-large-database-table-to-excel-in-chunks":1948,"ancestors:\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fexport-a-large-database-table-to-excel-in-chunks":1957,"children:\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fexport-a-large-database-table-to-excel-in-chunks":1962,"related:\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fexport-a-large-database-table-to-excel-in-chunks":1963},{"id":4,"title":5,"body":6,"dateModified":1915,"datePublished":1915,"description":1916,"extension":1917,"faq":1918,"keyword":1927,"kind":1928,"meta":1929,"navigation":140,"path":1937,"seo":1938,"slug":1941,"stem":1942,"topics":1943,"type":1946,"__hash__":1947},"docs\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fexport-a-large-database-table-to-excel-in-chunks\u002Findex.md","Export a Large Database Table to Excel in Chunks",{"type":7,"value":8,"toc":1903},"minimark",[9,32,37,71,81,85,98,267,281,285,291,786,806,946,950,957,1023,1188,1196,1200,1207,1432,1435,1488,1492,1495,1749,1768,1772,1821,1825,1838,1842,1848,1854,1860,1866,1870,1899],[10,11,12,13,17,18,21,22,25,26,31],"p",{},"\"Can you send me the whole table in Excel?\" is a request that is easy to fulfil for ten thousand rows and surprisingly easy to get wrong for two million. The straightforward approach — ",[14,15,16],"code",{},"pd.read_sql"," into a DataFrame, then ",[14,19,20],{},"to_excel"," — holds the entire result in memory twice over (once as the DataFrame, once as the workbook being built), and fails outright when the row count passes Excel's limit of 1,048,576 rows per sheet. The robust approach streams: fetch rows from the database in chunks, write each chunk to the workbook and discard it, and start a new sheet before the limit. With XlsxWriter's ",[14,23,24],{},"constant_memory"," mode, memory stays flat no matter how many rows go through. This page builds that export, with the database side done properly too. It belongs to ",[27,28,30],"a",{"href":29},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002F","Moving Data Between Excel and Databases",".",[33,34,36],"h2",{"id":35},"prerequisites","Prerequisites",[38,39,44],"pre",{"className":40,"code":41,"language":42,"meta":43,"style":43},"language-bash shiki shiki-themes github-light github-dark-high-contrast","pip install pandas sqlalchemy xlsxwriter openpyxl\n","bash","",[14,45,46],{"__ignoreMap":43},[47,48,51,55,59,62,65,68],"span",{"class":49,"line":50},"line",1,[47,52,54],{"class":53},"sMTad","pip",[47,56,58],{"class":57},"srMev"," install",[47,60,61],{"class":57}," pandas",[47,63,64],{"class":57}," sqlalchemy",[47,66,67],{"class":57}," xlsxwriter",[47,69,70],{"class":57}," openpyxl\n",[10,72,73,76,77,80],{},[14,74,75],{},"warehouse.db"," is a SQLite database with a ",[14,78,79],{},"movements"," table of 250,000 stock movements: id, timestamp, SKU, warehouse, quantity and unit cost. SQLite keeps the example self-contained; the same code works against PostgreSQL, SQL Server or MySQL by changing the connection URL, and the server-side cursor note in Step 1 matters most for those.",[33,82,84],{"id":83},"step-1-fetch-the-table-in-chunks","Step 1 — Fetch the table in chunks",[10,86,87,89,90,93,94,97],{},[14,88,16],{}," with ",[14,91,92],{},"chunksize"," returns an iterator of DataFrames instead of one large frame. With SQLAlchemy, add ",[14,95,96],{},"stream_results=True"," so the database driver uses a server-side cursor and does not pull the whole result into the client before the first chunk is returned:",[38,99,103],{"className":100,"code":101,"language":102,"meta":43,"style":43},"language-python shiki shiki-themes github-light github-dark-high-contrast","import pandas as pd\nfrom sqlalchemy import create_engine, text\n\nengine = create_engine(\"sqlite:\u002F\u002F\u002Fwarehouse.db\")\nquery = text(\"SELECT id, moved_at, sku, warehouse, qty, unit_cost FROM movements ORDER BY id\")\n\nwith engine.connect().execution_options(stream_results=True) as conn:\n    sizes = [len(chunk) for chunk in pd.read_sql(query, conn, chunksize=50_000)]\nprint(len(sizes), \"chunks:\", sizes)\n","python",[14,104,105,121,135,142,160,176,181,208,247],{"__ignoreMap":43},[47,106,107,111,115,118],{"class":49,"line":50},[47,108,110],{"class":109},"s-kum","import",[47,112,114],{"class":113},"skGVy"," pandas ",[47,116,117],{"class":109},"as",[47,119,120],{"class":113}," pd\n",[47,122,124,127,130,132],{"class":49,"line":123},2,[47,125,126],{"class":109},"from",[47,128,129],{"class":113}," sqlalchemy ",[47,131,110],{"class":109},[47,133,134],{"class":113}," create_engine, text\n",[47,136,138],{"class":49,"line":137},3,[47,139,141],{"emptyLinePlaceholder":140},true,"\n",[47,143,145,148,151,154,157],{"class":49,"line":144},4,[47,146,147],{"class":113},"engine ",[47,149,150],{"class":109},"=",[47,152,153],{"class":113}," create_engine(",[47,155,156],{"class":57},"\"sqlite:\u002F\u002F\u002Fwarehouse.db\"",[47,158,159],{"class":113},")\n",[47,161,163,166,168,171,174],{"class":49,"line":162},5,[47,164,165],{"class":113},"query ",[47,167,150],{"class":109},[47,169,170],{"class":113}," text(",[47,172,173],{"class":57},"\"SELECT id, moved_at, sku, warehouse, qty, unit_cost FROM movements ORDER BY id\"",[47,175,159],{"class":113},[47,177,179],{"class":49,"line":178},6,[47,180,141],{"emptyLinePlaceholder":140},[47,182,184,187,190,194,196,200,203,205],{"class":49,"line":183},7,[47,185,186],{"class":109},"with",[47,188,189],{"class":113}," engine.connect().execution_options(",[47,191,193],{"class":192},"sa561","stream_results",[47,195,150],{"class":109},[47,197,199],{"class":198},"sP0c6","True",[47,201,202],{"class":113},") ",[47,204,117],{"class":109},[47,206,207],{"class":113}," conn:\n",[47,209,211,214,216,219,222,225,228,231,234,237,239,241,244],{"class":49,"line":210},8,[47,212,213],{"class":113},"    sizes ",[47,215,150],{"class":109},[47,217,218],{"class":113}," [",[47,220,221],{"class":198},"len",[47,223,224],{"class":113},"(chunk) ",[47,226,227],{"class":109},"for",[47,229,230],{"class":113}," chunk ",[47,232,233],{"class":109},"in",[47,235,236],{"class":113}," pd.read_sql(query, conn, ",[47,238,92],{"class":192},[47,240,150],{"class":109},[47,242,243],{"class":198},"50_000",[47,245,246],{"class":113},")]\n",[47,248,250,253,256,258,261,264],{"class":49,"line":249},9,[47,251,252],{"class":198},"print",[47,254,255],{"class":113},"(",[47,257,221],{"class":198},[47,259,260],{"class":113},"(sizes), ",[47,262,263],{"class":57},"\"chunks:\"",[47,265,266],{"class":113},", sizes)\n",[10,268,269,270,272,273,275,276,280],{},"Five chunks of 50,000 rows each arrive one at a time. The ",[14,271,92],{}," controls how many rows pandas builds into each DataFrame; ",[14,274,96],{}," controls whether the ",[277,278,279],"em",{},"driver"," buffers the full result first. Without it, drivers such as psycopg for PostgreSQL fetch every row into client memory at the moment the query executes, and chunking in pandas only splits up data that is already all in memory. SQLite does not buffer that way, so the option is harmless here and essential there. Ordering by the primary key makes the export deterministic, which matters when someone compares two exports.",[33,282,284],{"id":283},"step-2-write-each-chunk-straight-to-disk","Step 2 — Write each chunk straight to disk",[10,286,287,288,290],{},"XlsxWriter's ",[14,289,24],{}," mode writes each row to a temporary file as soon as the next row starts, instead of keeping the whole sheet in memory. The cost is that rows must be written in order, top to bottom — exactly what streaming chunks does:",[38,292,294],{"className":100,"code":293,"language":102,"meta":43,"style":43},"import xlsxwriter\n\ndef export(conn, query, path, chunksize=50_000, max_rows=1_048_575):\n    book = xlsxwriter.Workbook(path, {\"constant_memory\": True})\n    head = book.add_format({\"bold\": True, \"bg_color\": \"#EEF2FF\", \"bottom\": 1})\n    when = book.add_format({\"num_format\": \"yyyy-mm-dd hh:mm\"})\n    money = book.add_format({\"num_format\": \"#,##0.00\"})\n    sheet, row, sheets, total = None, 0, 0, 0\n    for chunk in pd.read_sql(query, conn, chunksize=chunksize, parse_dates=[\"moved_at\"]):\n        for rec in chunk.itertuples(index=False):\n            if sheet is None or row > max_rows:\n                sheets += 1\n                sheet = book.add_worksheet(f\"Movements {sheets}\")\n                sheet.write_row(0, 0, list(chunk.columns), head)\n                sheet.set_column(1, 1, 17)\n                sheet.freeze_panes(1, 0)\n                row = 1\n            sheet.write_number(row, 0, rec.id)\n            sheet.write_datetime(row, 1, rec.moved_at.to_pydatetime(), when)\n            sheet.write_string(row, 2, rec.sku)\n            sheet.write_string(row, 3, rec.warehouse)\n            sheet.write_number(row, 4, rec.qty)\n            sheet.write_number(row, 5, rec.unit_cost, money)\n            row += 1\n            total += 1\n    book.close()\n    return sheets, total\n\nwith engine.connect().execution_options(stream_results=True) as conn:\n    print(export(conn, query, \"movements.xlsx\"))\n",[14,295,296,303,307,334,355,395,414,432,456,488,512,538,550,582,602,621,635,645,656,667,679,690,701,712,722,732,738,747,752,771],{"__ignoreMap":43},[47,297,298,300],{"class":49,"line":50},[47,299,110],{"class":109},[47,301,302],{"class":113}," xlsxwriter\n",[47,304,305],{"class":49,"line":123},[47,306,141],{"emptyLinePlaceholder":140},[47,308,309,312,316,319,321,323,326,328,331],{"class":49,"line":137},[47,310,311],{"class":109},"def",[47,313,315],{"class":314},"s_Opv"," export",[47,317,318],{"class":113},"(conn, query, path, chunksize",[47,320,150],{"class":109},[47,322,243],{"class":198},[47,324,325],{"class":113},", max_rows",[47,327,150],{"class":109},[47,329,330],{"class":198},"1_048_575",[47,332,333],{"class":113},"):\n",[47,335,336,339,341,344,347,350,352],{"class":49,"line":144},[47,337,338],{"class":113},"    book ",[47,340,150],{"class":109},[47,342,343],{"class":113}," xlsxwriter.Workbook(path, {",[47,345,346],{"class":57},"\"constant_memory\"",[47,348,349],{"class":113},": ",[47,351,199],{"class":198},[47,353,354],{"class":113},"})\n",[47,356,357,360,362,365,368,370,372,375,378,380,383,385,388,390,393],{"class":49,"line":162},[47,358,359],{"class":113},"    head ",[47,361,150],{"class":109},[47,363,364],{"class":113}," book.add_format({",[47,366,367],{"class":57},"\"bold\"",[47,369,349],{"class":113},[47,371,199],{"class":198},[47,373,374],{"class":113},", ",[47,376,377],{"class":57},"\"bg_color\"",[47,379,349],{"class":113},[47,381,382],{"class":57},"\"#EEF2FF\"",[47,384,374],{"class":113},[47,386,387],{"class":57},"\"bottom\"",[47,389,349],{"class":113},[47,391,392],{"class":198},"1",[47,394,354],{"class":113},[47,396,397,400,402,404,407,409,412],{"class":49,"line":178},[47,398,399],{"class":113},"    when ",[47,401,150],{"class":109},[47,403,364],{"class":113},[47,405,406],{"class":57},"\"num_format\"",[47,408,349],{"class":113},[47,410,411],{"class":57},"\"yyyy-mm-dd hh:mm\"",[47,413,354],{"class":113},[47,415,416,419,421,423,425,427,430],{"class":49,"line":183},[47,417,418],{"class":113},"    money ",[47,420,150],{"class":109},[47,422,364],{"class":113},[47,424,406],{"class":57},[47,426,349],{"class":113},[47,428,429],{"class":57},"\"#,##0.00\"",[47,431,354],{"class":113},[47,433,434,437,439,442,444,447,449,451,453],{"class":49,"line":210},[47,435,436],{"class":113},"    sheet, row, sheets, total ",[47,438,150],{"class":109},[47,440,441],{"class":198}," None",[47,443,374],{"class":113},[47,445,446],{"class":198},"0",[47,448,374],{"class":113},[47,450,446],{"class":198},[47,452,374],{"class":113},[47,454,455],{"class":198},"0\n",[47,457,458,461,463,465,467,469,471,474,477,479,482,485],{"class":49,"line":249},[47,459,460],{"class":109},"    for",[47,462,230],{"class":113},[47,464,233],{"class":109},[47,466,236],{"class":113},[47,468,92],{"class":192},[47,470,150],{"class":109},[47,472,473],{"class":113},"chunksize, ",[47,475,476],{"class":192},"parse_dates",[47,478,150],{"class":109},[47,480,481],{"class":113},"[",[47,483,484],{"class":57},"\"moved_at\"",[47,486,487],{"class":113},"]):\n",[47,489,491,494,497,499,502,505,507,510],{"class":49,"line":490},10,[47,492,493],{"class":109},"        for",[47,495,496],{"class":113}," rec ",[47,498,233],{"class":109},[47,500,501],{"class":113}," chunk.itertuples(",[47,503,504],{"class":192},"index",[47,506,150],{"class":109},[47,508,509],{"class":198},"False",[47,511,333],{"class":113},[47,513,515,518,521,524,526,529,532,535],{"class":49,"line":514},11,[47,516,517],{"class":109},"            if",[47,519,520],{"class":113}," sheet ",[47,522,523],{"class":109},"is",[47,525,441],{"class":198},[47,527,528],{"class":109}," or",[47,530,531],{"class":113}," row ",[47,533,534],{"class":109},">",[47,536,537],{"class":113}," max_rows:\n",[47,539,541,544,547],{"class":49,"line":540},12,[47,542,543],{"class":113},"                sheets ",[47,545,546],{"class":109},"+=",[47,548,549],{"class":198}," 1\n",[47,551,553,556,558,561,564,567,571,574,577,580],{"class":49,"line":552},13,[47,554,555],{"class":113},"                sheet ",[47,557,150],{"class":109},[47,559,560],{"class":113}," book.add_worksheet(",[47,562,563],{"class":109},"f",[47,565,566],{"class":57},"\"Movements ",[47,568,570],{"class":569},"sSjpA","{",[47,572,573],{"class":113},"sheets",[47,575,576],{"class":569},"}",[47,578,579],{"class":57},"\"",[47,581,159],{"class":113},[47,583,585,588,590,592,594,596,599],{"class":49,"line":584},14,[47,586,587],{"class":113},"                sheet.write_row(",[47,589,446],{"class":198},[47,591,374],{"class":113},[47,593,446],{"class":198},[47,595,374],{"class":113},[47,597,598],{"class":198},"list",[47,600,601],{"class":113},"(chunk.columns), head)\n",[47,603,605,608,610,612,614,616,619],{"class":49,"line":604},15,[47,606,607],{"class":113},"                sheet.set_column(",[47,609,392],{"class":198},[47,611,374],{"class":113},[47,613,392],{"class":198},[47,615,374],{"class":113},[47,617,618],{"class":198},"17",[47,620,159],{"class":113},[47,622,624,627,629,631,633],{"class":49,"line":623},16,[47,625,626],{"class":113},"                sheet.freeze_panes(",[47,628,392],{"class":198},[47,630,374],{"class":113},[47,632,446],{"class":198},[47,634,159],{"class":113},[47,636,638,641,643],{"class":49,"line":637},17,[47,639,640],{"class":113},"                row ",[47,642,150],{"class":109},[47,644,549],{"class":198},[47,646,648,651,653],{"class":49,"line":647},18,[47,649,650],{"class":113},"            sheet.write_number(row, ",[47,652,446],{"class":198},[47,654,655],{"class":113},", rec.id)\n",[47,657,659,662,664],{"class":49,"line":658},19,[47,660,661],{"class":113},"            sheet.write_datetime(row, ",[47,663,392],{"class":198},[47,665,666],{"class":113},", rec.moved_at.to_pydatetime(), when)\n",[47,668,670,673,676],{"class":49,"line":669},20,[47,671,672],{"class":113},"            sheet.write_string(row, ",[47,674,675],{"class":198},"2",[47,677,678],{"class":113},", rec.sku)\n",[47,680,682,684,687],{"class":49,"line":681},21,[47,683,672],{"class":113},[47,685,686],{"class":198},"3",[47,688,689],{"class":113},", rec.warehouse)\n",[47,691,693,695,698],{"class":49,"line":692},22,[47,694,650],{"class":113},[47,696,697],{"class":198},"4",[47,699,700],{"class":113},", rec.qty)\n",[47,702,704,706,709],{"class":49,"line":703},23,[47,705,650],{"class":113},[47,707,708],{"class":198},"5",[47,710,711],{"class":113},", rec.unit_cost, money)\n",[47,713,715,718,720],{"class":49,"line":714},24,[47,716,717],{"class":113},"            row ",[47,719,546],{"class":109},[47,721,549],{"class":198},[47,723,725,728,730],{"class":49,"line":724},25,[47,726,727],{"class":113},"            total ",[47,729,546],{"class":109},[47,731,549],{"class":198},[47,733,735],{"class":49,"line":734},26,[47,736,737],{"class":113},"    book.close()\n",[47,739,741,744],{"class":49,"line":740},27,[47,742,743],{"class":109},"    return",[47,745,746],{"class":113}," sheets, total\n",[47,748,750],{"class":49,"line":749},28,[47,751,141],{"emptyLinePlaceholder":140},[47,753,755,757,759,761,763,765,767,769],{"class":49,"line":754},29,[47,756,186],{"class":109},[47,758,189],{"class":113},[47,760,193],{"class":192},[47,762,150],{"class":109},[47,764,199],{"class":198},[47,766,202],{"class":113},[47,768,117],{"class":109},[47,770,207],{"class":113},[47,772,774,777,780,783],{"class":49,"line":773},30,[47,775,776],{"class":198},"    print",[47,778,779],{"class":113},"(export(conn, query, ",[47,781,782],{"class":57},"\"movements.xlsx\"",[47,784,785],{"class":113},"))\n",[10,787,788,789,792,793,796,797,799,800,802,803,805],{},"Each row is written with a typed method — ",[14,790,791],{},"write_datetime"," with a date format, ",[14,794,795],{},"write_number"," with a money format — so the timestamps arrive in Excel as real dates and the costs as numbers, not text. ",[14,798,476],{}," makes pandas convert the SQLite text timestamps while reading. The header row, column width and frozen pane are set when each sheet is created, because in ",[14,801,24],{}," mode nothing can be changed in a row once a later row has been written. Writing cell by cell looks slow, but XlsxWriter is fast at it; the generic ",[14,804,20],{}," would not work here anyway, because it cannot append to a sheet across chunks.",[807,808,816,817,816,821,816,825,816,831,816,841,816,852,816,859,816,863,816,869,816,874,816,878,816,885,816,890,816,893,816,897,816,900,816,905,816,912,816,916,816,919,816,923,816,926,816,931,816,938,816,943],"svg",{"viewBox":809,"role":810,"ariaLabelledBy":811,"xmlns":814,"style":815},"4 0 752 125","img",[812,813],"ch-flow-t","ch-flow-d","http:\u002F\u002Fwww.w3.org\u002F2000\u002Fsvg","width:100%;max-width:752px;height:auto;display:block;margin:1.5rem auto;font-family:Inter,ui-sans-serif,system-ui,sans-serif","\n  ",[818,819,820],"title",{"id":812},"A streaming export from database to workbook",[822,823,824],"desc",{"id":813},"The query runs with a server-side cursor, pandas yields chunks of rows, each row is written with a typed XlsxWriter call in constant memory mode and flushed to disk, and a new sheet starts before Excel's row limit.",[826,827],"rect",{"x":697,"y":446,"width":828,"height":829,"fill":830},"752","125","#ffffff",[826,832],{"x":833,"y":834,"width":835,"height":836,"rx":837,"fill":838,"stroke":839,"style":840},"20.0","16.0","153.0","93.0","12","#e7ebef","var(--line,#cdd5e6)","stroke-width:2px",[842,843,845],"g",{"transform":844},"translate(84.5,29.0) scale(1.000)",[846,847],"path",{"d":848,"fill":849,"stroke":850,"style":851},"M5 6c0-1.7 3.1-3 7-3s7 1.3 7 3-3.1 3-7 3-7-1.3-7-3z M5 6v12c0 1.7 3.1 3 7 3s7-1.3 7-3V6 M5 12c0 1.7 3.1 3 7 3s7-1.3 7-3","none","var(--text,#172033)","stroke-width:1.80px;stroke-linecap:round;stroke-linejoin:round",[853,854,858],"text",{"x":855,"y":856,"style":857},"96.5","75.5","font-size:13.5px;font-weight:700;fill:var(--text,#172033);text-anchor:middle","Server-side cursor",[853,860,96],{"x":855,"y":861,"style":862},"95.0","font-size:11.5px;font-weight:400;fill:var(--text,#172033);text-anchor:middle",[49,864],{"x1":865,"y1":866,"x2":867,"y2":866,"stroke":868,"style":840},"178.0","62.5","200.0","var(--brand,#5b5cf0)",[870,871],"polygon",{"points":872,"fill":873},"204.0,62.5 195.0,67.5 195.0,57.5","#5b5cf0",[826,875],{"x":876,"y":834,"width":835,"height":836,"rx":837,"fill":877,"stroke":868,"style":840},"209.0","#f0f4ff",[842,879,881],{"transform":880},"translate(273.5,29.0) scale(1.000)",[846,882],{"d":883,"fill":849,"stroke":884,"style":851},"M12 3l9 5-9 5-9-5z M3 13l9 5 9-5 M3 17l9 5 9-5","var(--brand-strong,#4338ca)",[853,886,889],{"x":887,"y":856,"style":888},"285.5","font-size:13.5px;font-weight:700;fill:var(--brand-strong,#4338ca);text-anchor:middle","Chunks",[853,891,892],{"x":887,"y":861,"style":862},"read_sql(chunksize=…)",[49,894],{"x1":895,"y1":866,"x2":896,"y2":866,"stroke":868,"style":840},"367.0","389.0",[870,898],{"points":899,"fill":873},"393.0,62.5 384.0,67.5 384.0,57.5",[826,901],{"x":902,"y":834,"width":835,"height":836,"rx":837,"fill":903,"stroke":904,"style":840},"398.0","#d9f4f1","var(--teal,#0f9488)",[842,906,908],{"transform":907},"translate(462.5,29.0) scale(1.000)",[846,909],{"d":910,"fill":849,"stroke":911,"style":851},"M13 2L4 14h7l-1 8 9-12h-7z","var(--teal-ink,#0b6157)",[853,913,24],{"x":914,"y":856,"style":915},"474.5","font-size:13.5px;font-weight:700;fill:var(--teal-ink,#0b6157);text-anchor:middle",[853,917,918],{"x":914,"y":861,"style":862},"rows flushed in order",[49,920],{"x1":921,"y1":866,"x2":922,"y2":866,"stroke":868,"style":840},"556.0","578.0",[870,924],{"points":925,"fill":873},"582.0,62.5 573.0,67.5 573.0,57.5",[826,927],{"x":928,"y":834,"width":835,"height":836,"rx":837,"fill":929,"stroke":930,"style":840},"587.0","#fdefd8","var(--gold,#b4740a)",[842,932,934],{"transform":933},"translate(651.5,29.0) scale(1.000)",[846,935],{"d":936,"fill":849,"stroke":937,"style":851},"M3 5h18v14H3z M3 10h18 M3 15h18 M9 5v14","var(--gold-ink,#7a4e06)",[853,939,942],{"x":940,"y":856,"style":941},"663.5","font-size:13.5px;font-weight:700;fill:var(--gold-ink,#7a4e06);text-anchor:middle","Sheet rollover",[853,944,945],{"x":940,"y":861,"style":862},"before 1,048,576 rows",[33,947,949],{"id":948},"step-3-roll-over-before-the-row-limit","Step 3 — Roll over before the row limit",[10,951,952,953,956],{},"An Excel sheet holds at most 1,048,576 rows, header included. The ",[14,954,955],{},"export"," function counts rows per sheet and starts a new sheet with its own header when the next row would exceed the limit. To see the rollover without generating a million rows, lower the limit:",[38,958,960],{"className":100,"code":959,"language":102,"meta":43,"style":43},"with engine.connect().execution_options(stream_results=True) as conn:\n    sheets, total = export(conn, query, \"movements_split.xlsx\", max_rows=100_000)\nprint(sheets, \"sheets for\", total, \"rows\")\n",[14,961,962,980,1005],{"__ignoreMap":43},[47,963,964,966,968,970,972,974,976,978],{"class":49,"line":50},[47,965,186],{"class":109},[47,967,189],{"class":113},[47,969,193],{"class":192},[47,971,150],{"class":109},[47,973,199],{"class":198},[47,975,202],{"class":113},[47,977,117],{"class":109},[47,979,207],{"class":113},[47,981,982,985,987,990,993,995,998,1000,1003],{"class":49,"line":123},[47,983,984],{"class":113},"    sheets, total ",[47,986,150],{"class":109},[47,988,989],{"class":113}," export(conn, query, ",[47,991,992],{"class":57},"\"movements_split.xlsx\"",[47,994,374],{"class":113},[47,996,997],{"class":192},"max_rows",[47,999,150],{"class":109},[47,1001,1002],{"class":198},"100_000",[47,1004,159],{"class":113},[47,1006,1007,1009,1012,1015,1018,1021],{"class":49,"line":137},[47,1008,252],{"class":198},[47,1010,1011],{"class":113},"(sheets, ",[47,1013,1014],{"class":57},"\"sheets for\"",[47,1016,1017],{"class":113},", total, ",[47,1019,1020],{"class":57},"\"rows\"",[47,1022,159],{"class":113},[807,1024,816,1030,816,1033,816,1036,816,1041,816,1047,816,1051,816,1057,816,1061,816,1065,816,1069,816,1073,816,1077,816,1081,816,1085,816,1089,816,1092,816,1096,816,1098,816,1101,816,1103,816,1106,816,1108,816,1111,816,1114,816,1117,816,1119,816,1123,816,1125,816,1127,816,1129,816,1132,816,1134,816,1137,816,1140,816,1143,816,1145,816,1148,816,1150,816,1153,816,1155,816,1158,816,1160,816,1162,816,1165,816,1168,816,1170,816,1173,816,1175,816,1178,816,1180,816,1183,816,1185],{"viewBox":1025,"role":810,"ariaLabelledBy":1026,"xmlns":814,"style":1029},"189 0 381 168",[1027,1028],"ch-split-t","ch-split-d","width:100%;max-width:381px;height:auto;display:block;margin:1.5rem auto;font-family:Inter,ui-sans-serif,system-ui,sans-serif",[818,1031,1032],{"id":1027},"Three sheets from one table",[822,1034,1035],{"id":1028},"With a limit of 100,000 data rows per sheet, the 250,000 movements are written to Movements 1 with ids 1 to 100,000, Movements 2 with ids 100,001 to 200,000 and Movements 3 with ids 200,001 to 250,000. Each sheet has its own header row.",[826,1037],{"x":1038,"y":446,"width":1039,"height":1040,"fill":830},"189","381","168",[826,1042],{"x":1043,"y":834,"width":1044,"height":1045,"rx":446,"fill":838,"stroke":839,"style":1046},"205.6","32.0","24.0","stroke-width:1px",[826,1048],{"x":1049,"y":834,"width":1050,"height":1045,"rx":446,"fill":838,"stroke":839,"style":1046},"237.6","97.7",[853,1052,1056],{"x":1053,"y":1054,"style":1055},"286.5","33.0","font-size:11px;font-weight:600;fill:var(--muted,#5b6780);text-anchor:middle","A",[826,1058],{"x":1059,"y":834,"width":1060,"height":1045,"rx":446,"fill":838,"stroke":839,"style":1046},"335.3","69.1",[853,1062,1064],{"x":1063,"y":1054,"style":1055},"369.8","B",[826,1066],{"x":1067,"y":834,"width":1068,"height":1045,"rx":446,"fill":838,"stroke":839,"style":1046},"404.4","71.8",[853,1070,1072],{"x":1071,"y":1054,"style":1055},"440.3","C",[826,1074],{"x":1075,"y":834,"width":1076,"height":1045,"rx":446,"fill":838,"stroke":839,"style":1046},"476.2","78.2",[853,1078,1080],{"x":1079,"y":1054,"style":1055},"515.3","D",[826,1082],{"x":1043,"y":1083,"width":1044,"height":1084,"rx":446,"fill":838,"stroke":839,"style":1046},"40.0","28.0",[853,1086,392],{"x":1087,"y":1088,"style":1055},"221.6","59.0",[826,1090],{"x":1049,"y":1083,"width":1050,"height":1084,"rx":446,"fill":877,"stroke":868,"style":1091},"stroke-width:1.5px",[853,1093,1095],{"x":1053,"y":1088,"style":1094},"font-size:10.5px;font-weight:700;fill:var(--brand-strong,#4338ca);text-anchor:middle","Sheet",[826,1097],{"x":1059,"y":1083,"width":1060,"height":1084,"rx":446,"fill":877,"stroke":868,"style":1091},[853,1099,1100],{"x":1063,"y":1088,"style":1094},"First id",[826,1102],{"x":1067,"y":1083,"width":1068,"height":1084,"rx":446,"fill":877,"stroke":868,"style":1091},[853,1104,1105],{"x":1071,"y":1088,"style":1094},"Last id",[826,1107],{"x":1075,"y":1083,"width":1076,"height":1084,"rx":446,"fill":877,"stroke":868,"style":1091},[853,1109,1110],{"x":1079,"y":1088,"style":1094},"Data rows",[826,1112],{"x":1043,"y":1113,"width":1044,"height":1084,"rx":446,"fill":838,"stroke":839,"style":1046},"68.0",[853,1115,675],{"x":1087,"y":1116,"style":1055},"87.0",[826,1118],{"x":1049,"y":1113,"width":1050,"height":1084,"rx":446,"fill":830,"stroke":839,"style":1046},[853,1120,1122],{"x":1053,"y":1116,"style":1121},"font-size:10.5px;font-weight:400;fill:var(--text,#172033);text-anchor:middle","Movements 1",[826,1124],{"x":1059,"y":1113,"width":1060,"height":1084,"rx":446,"fill":830,"stroke":839,"style":1046},[853,1126,392],{"x":1063,"y":1116,"style":1121},[826,1128],{"x":1067,"y":1113,"width":1068,"height":1084,"rx":446,"fill":830,"stroke":839,"style":1046},[853,1130,1131],{"x":1071,"y":1116,"style":1121},"100,000",[826,1133],{"x":1075,"y":1113,"width":1076,"height":1084,"rx":446,"fill":903,"stroke":904,"style":1091},[853,1135,1131],{"x":1079,"y":1116,"style":1136},"font-size:10.5px;font-weight:700;fill:var(--teal-ink,#0b6157);text-anchor:middle",[826,1138],{"x":1043,"y":1139,"width":1044,"height":1084,"rx":446,"fill":838,"stroke":839,"style":1046},"96.0",[853,1141,686],{"x":1087,"y":1142,"style":1055},"115.0",[826,1144],{"x":1049,"y":1139,"width":1050,"height":1084,"rx":446,"fill":830,"stroke":839,"style":1046},[853,1146,1147],{"x":1053,"y":1142,"style":1121},"Movements 2",[826,1149],{"x":1059,"y":1139,"width":1060,"height":1084,"rx":446,"fill":830,"stroke":839,"style":1046},[853,1151,1152],{"x":1063,"y":1142,"style":1121},"100,001",[826,1154],{"x":1067,"y":1139,"width":1068,"height":1084,"rx":446,"fill":830,"stroke":839,"style":1046},[853,1156,1157],{"x":1071,"y":1142,"style":1121},"200,000",[826,1159],{"x":1075,"y":1139,"width":1076,"height":1084,"rx":446,"fill":903,"stroke":904,"style":1091},[853,1161,1131],{"x":1079,"y":1142,"style":1136},[826,1163],{"x":1043,"y":1164,"width":1044,"height":1084,"rx":446,"fill":838,"stroke":839,"style":1046},"124.0",[853,1166,697],{"x":1087,"y":1167,"style":1055},"143.0",[826,1169],{"x":1049,"y":1164,"width":1050,"height":1084,"rx":446,"fill":830,"stroke":839,"style":1046},[853,1171,1172],{"x":1053,"y":1167,"style":1121},"Movements 3",[826,1174],{"x":1059,"y":1164,"width":1060,"height":1084,"rx":446,"fill":830,"stroke":839,"style":1046},[853,1176,1177],{"x":1063,"y":1167,"style":1121},"200,001",[826,1179],{"x":1067,"y":1164,"width":1068,"height":1084,"rx":446,"fill":830,"stroke":839,"style":1046},[853,1181,1182],{"x":1071,"y":1167,"style":1121},"250,000",[826,1184],{"x":1075,"y":1164,"width":1076,"height":1084,"rx":446,"fill":903,"stroke":904,"style":1091},[853,1186,1187],{"x":1079,"y":1167,"style":1136},"50,000",[10,1189,1190,1191,1195],{},"With a limit of 100,000 data rows, the 250,000 movements land on three sheets — 100,000, 100,000 and 50,000 rows — each with a header and frozen top row. In real use leave the default; the function then only splits tables that genuinely exceed one sheet. For very large exports ask whether Excel is the right format at all: a 3-million-row workbook is slow to open and hard to use, and a CSV or Parquet file, or a filtered extract, usually serves the requester better. ",[27,1192,1194],{"href":1193},"\u002Fgetting-started-with-python-excel-automation\u002Fchoosing-a-python-excel-library\u002Fexcel-vs-csv-vs-parquet-for-python-data-pipelines\u002F","Excel vs CSV vs Parquet for Python Data Pipelines"," helps with that conversation.",[33,1197,1199],{"id":1198},"step-4-measure-memory-against-the-simple-approach","Step 4 — Measure memory against the simple approach",[10,1201,1202,1203,1206],{},"The point of streaming is flat memory. ",[14,1204,1205],{},"tracemalloc"," measures the peak memory Python allocates during each approach:",[38,1208,1210],{"className":100,"code":1209,"language":102,"meta":43,"style":43},"import tracemalloc\n\ndef peak_mb(fn):\n    tracemalloc.start()\n    fn()\n    peak = tracemalloc.get_traced_memory()[1]\n    tracemalloc.stop()\n    return round(peak \u002F 1024 \u002F 1024, 1)\n\ndef simple():\n    df = pd.read_sql(query, engine, parse_dates=[\"moved_at\"])\n    df.to_excel(\"movements_simple.xlsx\", index=False, engine=\"xlsxwriter\")\n\ndef streamed():\n    with engine.connect().execution_options(stream_results=True) as conn:\n        export(conn, query, \"movements_streamed.xlsx\")\n\nprint(\"simple:   \", peak_mb(simple), \"MB peak\")\nprint(\"streamed: \", peak_mb(streamed), \"MB peak\")\n",[14,1211,1212,1219,1223,1233,1238,1243,1258,1263,1290,1294,1304,1325,1353,1357,1366,1385,1395,1399,1416],{"__ignoreMap":43},[47,1213,1214,1216],{"class":49,"line":50},[47,1215,110],{"class":109},[47,1217,1218],{"class":113}," tracemalloc\n",[47,1220,1221],{"class":49,"line":123},[47,1222,141],{"emptyLinePlaceholder":140},[47,1224,1225,1227,1230],{"class":49,"line":137},[47,1226,311],{"class":109},[47,1228,1229],{"class":314}," peak_mb",[47,1231,1232],{"class":113},"(fn):\n",[47,1234,1235],{"class":49,"line":144},[47,1236,1237],{"class":113},"    tracemalloc.start()\n",[47,1239,1240],{"class":49,"line":162},[47,1241,1242],{"class":113},"    fn()\n",[47,1244,1245,1248,1250,1253,1255],{"class":49,"line":178},[47,1246,1247],{"class":113},"    peak ",[47,1249,150],{"class":109},[47,1251,1252],{"class":113}," tracemalloc.get_traced_memory()[",[47,1254,392],{"class":198},[47,1256,1257],{"class":113},"]\n",[47,1259,1260],{"class":49,"line":183},[47,1261,1262],{"class":113},"    tracemalloc.stop()\n",[47,1264,1265,1267,1270,1273,1276,1279,1282,1284,1286,1288],{"class":49,"line":210},[47,1266,743],{"class":109},[47,1268,1269],{"class":198}," round",[47,1271,1272],{"class":113},"(peak ",[47,1274,1275],{"class":109},"\u002F",[47,1277,1278],{"class":198}," 1024",[47,1280,1281],{"class":109}," \u002F",[47,1283,1278],{"class":198},[47,1285,374],{"class":113},[47,1287,392],{"class":198},[47,1289,159],{"class":113},[47,1291,1292],{"class":49,"line":249},[47,1293,141],{"emptyLinePlaceholder":140},[47,1295,1296,1298,1301],{"class":49,"line":490},[47,1297,311],{"class":109},[47,1299,1300],{"class":314}," simple",[47,1302,1303],{"class":113},"():\n",[47,1305,1306,1309,1311,1314,1316,1318,1320,1322],{"class":49,"line":514},[47,1307,1308],{"class":113},"    df ",[47,1310,150],{"class":109},[47,1312,1313],{"class":113}," pd.read_sql(query, engine, ",[47,1315,476],{"class":192},[47,1317,150],{"class":109},[47,1319,481],{"class":113},[47,1321,484],{"class":57},[47,1323,1324],{"class":113},"])\n",[47,1326,1327,1330,1333,1335,1337,1339,1341,1343,1346,1348,1351],{"class":49,"line":540},[47,1328,1329],{"class":113},"    df.to_excel(",[47,1331,1332],{"class":57},"\"movements_simple.xlsx\"",[47,1334,374],{"class":113},[47,1336,504],{"class":192},[47,1338,150],{"class":109},[47,1340,509],{"class":198},[47,1342,374],{"class":113},[47,1344,1345],{"class":192},"engine",[47,1347,150],{"class":109},[47,1349,1350],{"class":57},"\"xlsxwriter\"",[47,1352,159],{"class":113},[47,1354,1355],{"class":49,"line":552},[47,1356,141],{"emptyLinePlaceholder":140},[47,1358,1359,1361,1364],{"class":49,"line":584},[47,1360,311],{"class":109},[47,1362,1363],{"class":314}," streamed",[47,1365,1303],{"class":113},[47,1367,1368,1371,1373,1375,1377,1379,1381,1383],{"class":49,"line":604},[47,1369,1370],{"class":109},"    with",[47,1372,189],{"class":113},[47,1374,193],{"class":192},[47,1376,150],{"class":109},[47,1378,199],{"class":198},[47,1380,202],{"class":113},[47,1382,117],{"class":109},[47,1384,207],{"class":113},[47,1386,1387,1390,1393],{"class":49,"line":623},[47,1388,1389],{"class":113},"        export(conn, query, ",[47,1391,1392],{"class":57},"\"movements_streamed.xlsx\"",[47,1394,159],{"class":113},[47,1396,1397],{"class":49,"line":637},[47,1398,141],{"emptyLinePlaceholder":140},[47,1400,1401,1403,1405,1408,1411,1414],{"class":49,"line":647},[47,1402,252],{"class":198},[47,1404,255],{"class":113},[47,1406,1407],{"class":57},"\"simple:   \"",[47,1409,1410],{"class":113},", peak_mb(simple), ",[47,1412,1413],{"class":57},"\"MB peak\"",[47,1415,159],{"class":113},[47,1417,1418,1420,1422,1425,1428,1430],{"class":49,"line":658},[47,1419,252],{"class":198},[47,1421,255],{"class":113},[47,1423,1424],{"class":57},"\"streamed: \"",[47,1426,1427],{"class":113},", peak_mb(streamed), ",[47,1429,1413],{"class":57},[47,1431,159],{"class":113},[10,1433,1434],{},"The simple approach peaked at about 237 MB for this 250,000-row table — the full DataFrame plus XlsxWriter's in-memory sheet — and grows linearly with the table. The streamed export peaked at about 39 MB, set mainly by the 50,000-row chunk, and stays there however long the table is. On a server running several jobs, or in a container with a memory limit, that difference decides whether the export finishes at all. Smaller chunks lower the peak further at a small cost in speed; 10,000 to 100,000 rows per chunk is a sensible range.",[807,1436,816,1442,816,1445,816,1448,816,1452,816,1456,816,1462,816,1468,816,1473,816,1476,816,1480,816,1483],{"viewBox":1437,"role":810,"ariaLabelledBy":1438,"xmlns":814,"style":1441},"4 0 744 100",[1439,1440],"ch-mem-t","ch-mem-d","width:100%;max-width:744px;height:auto;display:block;margin:1.5rem auto;font-family:Inter,ui-sans-serif,system-ui,sans-serif",[818,1443,1444],{"id":1439},"Peak Python memory exporting 250,000 rows",[822,1446,1447],{"id":1440},"Reading the whole table into a DataFrame and writing it with to_excel peaked at about 237 megabytes in this run. Streaming in chunks of 50,000 rows with constant memory mode peaked at about 39 megabytes and does not grow with the table.",[826,1449],{"x":697,"y":446,"width":1450,"height":1451,"fill":830},"744","100",[826,1453],{"x":833,"y":834,"width":1454,"height":1084,"rx":1455,"fill":838,"stroke":839,"style":1046},"206.4","7",[853,1457,1461],{"x":1458,"y":1459,"style":1460},"123.2","34.5","font-size:12.5px;font-weight:700;fill:var(--text,#172033);text-anchor:middle","read_sql + to_excel",[826,1463],{"x":1464,"y":834,"width":1465,"height":1084,"rx":1455,"fill":1466,"stroke":1467,"style":1091},"236.4","321.7","#fce9e9","var(--danger,#dc2626)",[853,1469,1472],{"x":1470,"y":1459,"style":1471},"566.0","font-size:12px;font-weight:700;fill:var(--accent-ink,#be185d);text-anchor:start","≈ 237 MB, grows with rows",[826,1474],{"x":833,"y":1475,"width":1454,"height":1084,"rx":1455,"fill":838,"stroke":839,"style":1046},"56.0",[853,1477,1479],{"x":1458,"y":1478,"style":1460},"74.5","chunks + constant_memory",[826,1481],{"x":1464,"y":1475,"width":1482,"height":1084,"rx":1455,"fill":903,"stroke":904,"style":1091},"53.3",[853,1484,1487],{"x":1485,"y":1478,"style":1486},"297.7","font-size:12px;font-weight:700;fill:var(--teal-ink,#0b6157);text-anchor:start","≈ 39 MB, flat",[33,1489,1491],{"id":1490},"step-5-check-the-export-is-complete","Step 5 — Check the export is complete",[10,1493,1494],{},"A large export should prove it contains every row. Count rows per sheet in the written file without loading it fully, and compare with the database:",[38,1496,1498],{"className":100,"code":1497,"language":102,"meta":43,"style":43},"from openpyxl import load_workbook\n\nwith engine.connect() as conn:\n    expected = conn.execute(text(\"SELECT COUNT(*) FROM movements\")).scalar()\nwb = load_workbook(\"movements_split.xlsx\", read_only=True)\nper_sheet = {ws.title: sum(1 for _ in ws.iter_rows(min_row=2, values_only=True)) for ws in wb.worksheets}\nwb.close()\nprint(per_sheet, \"| total\", sum(per_sheet.values()), \"| database\", expected)\nwb = load_workbook(\"movements_split.xlsx\", read_only=True)\nfirst = next(wb.worksheets[0].iter_rows(min_row=2, max_row=2, values_only=True))\nwb.close()\nprint(\"first row types:\", [type(v).__name__ for v in first])\n",[14,1499,1500,1512,1516,1527,1543,1566,1623,1628,1651,1671,1714,1718],{"__ignoreMap":43},[47,1501,1502,1504,1507,1509],{"class":49,"line":50},[47,1503,126],{"class":109},[47,1505,1506],{"class":113}," openpyxl ",[47,1508,110],{"class":109},[47,1510,1511],{"class":113}," load_workbook\n",[47,1513,1514],{"class":49,"line":123},[47,1515,141],{"emptyLinePlaceholder":140},[47,1517,1518,1520,1523,1525],{"class":49,"line":137},[47,1519,186],{"class":109},[47,1521,1522],{"class":113}," engine.connect() ",[47,1524,117],{"class":109},[47,1526,207],{"class":113},[47,1528,1529,1532,1534,1537,1540],{"class":49,"line":144},[47,1530,1531],{"class":113},"    expected ",[47,1533,150],{"class":109},[47,1535,1536],{"class":113}," conn.execute(text(",[47,1538,1539],{"class":57},"\"SELECT COUNT(*) FROM movements\"",[47,1541,1542],{"class":113},")).scalar()\n",[47,1544,1545,1548,1550,1553,1555,1557,1560,1562,1564],{"class":49,"line":162},[47,1546,1547],{"class":113},"wb ",[47,1549,150],{"class":109},[47,1551,1552],{"class":113}," load_workbook(",[47,1554,992],{"class":57},[47,1556,374],{"class":113},[47,1558,1559],{"class":192},"read_only",[47,1561,150],{"class":109},[47,1563,199],{"class":198},[47,1565,159],{"class":113},[47,1567,1568,1571,1573,1576,1579,1581,1583,1586,1589,1591,1594,1597,1599,1601,1603,1606,1608,1610,1613,1615,1618,1620],{"class":49,"line":178},[47,1569,1570],{"class":113},"per_sheet ",[47,1572,150],{"class":109},[47,1574,1575],{"class":113}," {ws.title: ",[47,1577,1578],{"class":198},"sum",[47,1580,255],{"class":113},[47,1582,392],{"class":198},[47,1584,1585],{"class":109}," for",[47,1587,1588],{"class":113}," _ ",[47,1590,233],{"class":109},[47,1592,1593],{"class":113}," ws.iter_rows(",[47,1595,1596],{"class":192},"min_row",[47,1598,150],{"class":109},[47,1600,675],{"class":198},[47,1602,374],{"class":113},[47,1604,1605],{"class":192},"values_only",[47,1607,150],{"class":109},[47,1609,199],{"class":198},[47,1611,1612],{"class":113},")) ",[47,1614,227],{"class":109},[47,1616,1617],{"class":113}," ws ",[47,1619,233],{"class":109},[47,1621,1622],{"class":113}," wb.worksheets}\n",[47,1624,1625],{"class":49,"line":183},[47,1626,1627],{"class":113},"wb.close()\n",[47,1629,1630,1632,1635,1638,1640,1642,1645,1648],{"class":49,"line":210},[47,1631,252],{"class":198},[47,1633,1634],{"class":113},"(per_sheet, ",[47,1636,1637],{"class":57},"\"| total\"",[47,1639,374],{"class":113},[47,1641,1578],{"class":198},[47,1643,1644],{"class":113},"(per_sheet.values()), ",[47,1646,1647],{"class":57},"\"| database\"",[47,1649,1650],{"class":113},", expected)\n",[47,1652,1653,1655,1657,1659,1661,1663,1665,1667,1669],{"class":49,"line":249},[47,1654,1547],{"class":113},[47,1656,150],{"class":109},[47,1658,1552],{"class":113},[47,1660,992],{"class":57},[47,1662,374],{"class":113},[47,1664,1559],{"class":192},[47,1666,150],{"class":109},[47,1668,199],{"class":198},[47,1670,159],{"class":113},[47,1672,1673,1676,1678,1681,1684,1686,1689,1691,1693,1695,1697,1700,1702,1704,1706,1708,1710,1712],{"class":49,"line":490},[47,1674,1675],{"class":113},"first ",[47,1677,150],{"class":109},[47,1679,1680],{"class":198}," next",[47,1682,1683],{"class":113},"(wb.worksheets[",[47,1685,446],{"class":198},[47,1687,1688],{"class":113},"].iter_rows(",[47,1690,1596],{"class":192},[47,1692,150],{"class":109},[47,1694,675],{"class":198},[47,1696,374],{"class":113},[47,1698,1699],{"class":192},"max_row",[47,1701,150],{"class":109},[47,1703,675],{"class":198},[47,1705,374],{"class":113},[47,1707,1605],{"class":192},[47,1709,150],{"class":109},[47,1711,199],{"class":198},[47,1713,785],{"class":113},[47,1715,1716],{"class":49,"line":514},[47,1717,1627],{"class":113},[47,1719,1720,1722,1724,1727,1730,1733,1736,1739,1741,1744,1746],{"class":49,"line":540},[47,1721,252],{"class":198},[47,1723,255],{"class":113},[47,1725,1726],{"class":57},"\"first row types:\"",[47,1728,1729],{"class":113},", [",[47,1731,1732],{"class":198},"type",[47,1734,1735],{"class":113},"(v).",[47,1737,1738],{"class":198},"__name__",[47,1740,1585],{"class":109},[47,1742,1743],{"class":113}," v ",[47,1745,233],{"class":109},[47,1747,1748],{"class":113}," first])\n",[10,1750,1751,1752,1755,1756,1759,1760,1763,1764,31],{},"Reading with ",[14,1753,1754],{},"read_only=True"," streams the sheets, so even a very large export can be checked cheaply. The total across sheets matches the database count, and the first data row comes back with an integer, a ",[14,1757,1758],{},"datetime",", strings and numbers — proof the cells were written typed. If rows are being added to the table while the export runs, the count can legitimately differ; export from a snapshot or with a fixed ",[14,1761,1762],{},"WHERE id \u003C= :max_id"," to make the comparison exact. Counting rows quickly is covered in ",[27,1765,1767],{"href":1766},"\u002Fadvanced-data-transformation-and-cleaning\u002Fworking-with-large-excel-files-in-python\u002Fcount-rows-in-an-excel-file-without-loading-it\u002F","Count Rows in an Excel File Without Loading It",[33,1769,1771],{"id":1770},"common-pitfalls","Common pitfalls",[1773,1774,1775,1786,1794,1803,1815],"ul",{},[1776,1777,1778,1782,1783,1785],"li",{},[1779,1780,1781],"strong",{},"Chunking in pandas but not in the driver."," Without ",[14,1784,96],{},", the driver may load everything first.",[1776,1787,1788,1793],{},[1779,1789,1790,1792],{},[14,1791,20],{}," per chunk."," It cannot append to a sheet; write rows yourself with XlsxWriter.",[1776,1795,1796,1799,1800,1802],{},[1779,1797,1798],{},"Formatting rows after writing more rows."," In ",[14,1801,24],{}," mode, set headers and widths when the sheet is created.",[1776,1804,1805,1808,1809,1811,1812,1814],{},[1779,1806,1807],{},"Text instead of dates and numbers."," Use ",[14,1810,791],{}," and ",[14,1813,795],{}," with formats.",[1776,1816,1817,1820],{},[1779,1818,1819],{},"Ignoring the row limit."," Roll over to a new sheet before 1,048,576 rows — or question Excel as the format.",[33,1822,1824],{"id":1823},"conclusion","Conclusion",[10,1826,1827,1828,1831,1832,1834,1835,1837],{},"To export a table too big for memory, stream it: read with ",[14,1829,1830],{},"pd.read_sql(..., chunksize=...)"," over a connection with ",[14,1833,96],{}," so the database uses a server-side cursor, write each row with typed XlsxWriter calls in ",[14,1836,24],{}," mode, and start a new sheet, with its own header, before Excel's 1,048,576-row limit. Memory then stays flat whatever the table size, and a final row count per sheet against the database proves the export is complete.",[33,1839,1841],{"id":1840},"frequently-asked-questions","Frequently asked questions",[10,1843,1844,1847],{},[1779,1845,1846],{},"How do I export a large SQL table to Excel without running out of memory?","\nRead it in chunks with pandas read_sql and chunksize, using a server-side cursor, and write each row with XlsxWriter in constant_memory mode so rows are flushed to disk as they are written.",[10,1849,1850,1853],{},[1779,1851,1852],{},"What is the maximum number of rows in an Excel sheet?","\n1,048,576 rows including the header. Larger exports must be split across several sheets or files.",[10,1855,1856,1859],{},[1779,1857,1858],{},"Why does read_sql with chunksize still use a lot of memory?","\nMany database drivers fetch the whole result into memory when the query runs. Use a connection with stream_results=True so the driver streams rows from a server-side cursor.",[10,1861,1862,1865],{},[1779,1863,1864],{},"Can I use to_excel to write chunks to the same sheet?","\nNot reliably; to_excel writes a whole DataFrame. Write rows yourself with XlsxWriter, keeping track of the next row number.",[33,1867,1869],{"id":1868},"related","Related",[1773,1871,1872,1878,1885,1892],{},[1776,1873,1874,1875,1877],{},"Up one level: ",[27,1876,30],{"href":29}," — getting data in and out.",[1776,1879,1880,1884],{},[27,1881,1883],{"href":1882},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fexport-sql-query-results-to-excel-with-python\u002F","Export SQL Query Results to Excel with Python"," — the simple case.",[1776,1886,1887,1891],{},[27,1888,1890],{"href":1889},"\u002Fformatting-and-charting-excel-reports-with-python\u002Fbuilding-excel-reports-with-xlsxwriter\u002Fwrite-a-million-rows-to-excel-with-xlsxwriter-constant-memory\u002F","Write a Million Rows to Excel with XlsxWriter constant_memory"," — the writing side in depth.",[1776,1893,1894,1898],{},[27,1895,1897],{"href":1896},"\u002Fadvanced-data-transformation-and-cleaning\u002Fworking-with-large-excel-files-in-python\u002Fstream-excel-rows-into-a-database-with-openpyxl\u002F","Stream Excel Rows into a Database with openpyxl"," — the opposite direction.",[1900,1901,1902],"style",{},"html pre.shiki code .sMTad, html code.shiki .sMTad{--shiki-default:#6F42C1;--shiki-dark:#FFB757}html pre.shiki code .srMev, html code.shiki .srMev{--shiki-default:#032F62;--shiki-dark:#ADDCFF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .s-kum, html code.shiki .s-kum{--shiki-default:#D73A49;--shiki-dark:#FF9492}html pre.shiki code .skGVy, html code.shiki .skGVy{--shiki-default:#24292E;--shiki-dark:#F0F3F6}html pre.shiki code .sa561, html code.shiki .sa561{--shiki-default:#E36209;--shiki-dark:#FFB757}html pre.shiki code .sP0c6, html code.shiki .sP0c6{--shiki-default:#005CC5;--shiki-dark:#91CBFF}html pre.shiki code .s_Opv, html code.shiki .s_Opv{--shiki-default:#6F42C1;--shiki-dark:#DBB7FF}html pre.shiki code .sSjpA, html code.shiki .sSjpA{--shiki-default:#005CC5;--shiki-dark:#FF9492}",{"title":43,"searchDepth":123,"depth":123,"links":1904},[1905,1906,1907,1908,1909,1910,1911,1912,1913,1914],{"id":35,"depth":123,"text":36},{"id":83,"depth":123,"text":84},{"id":283,"depth":123,"text":284},{"id":948,"depth":123,"text":949},{"id":1198,"depth":123,"text":1199},{"id":1490,"depth":123,"text":1491},{"id":1770,"depth":123,"text":1771},{"id":1823,"depth":123,"text":1824},{"id":1840,"depth":123,"text":1841},{"id":1868,"depth":123,"text":1869},"2026-10-02","Export a database table too big to load at once into Excel with flat memory use: stream rows in chunks from SQLAlchemy with a server-side cursor, write them straight to disk with XlsxWriter constant_memory mode, roll over to a new sheet before the 1,048,576-row limit, keep dates and numbers typed, and check the row counts and memory afterwards.","md",[1919,1921,1923,1925],{"q":1846,"a":1920},"Read it in chunks with pandas read_sql and chunksize, using a server-side cursor, and write each row with XlsxWriter in constant_memory mode so rows are flushed to disk as they are written.",{"q":1852,"a":1922},"1,048,576 rows including the header. Larger exports must be split across several sheets or files.",{"q":1858,"a":1924},"Many database drivers fetch the whole result into memory when the query runs. Use a connection with stream_results=True so the driver streams rows from a server-side cursor.",{"q":1864,"a":1926},"Not reliably; to_excel writes a whole DataFrame. Write rows yourself with XlsxWriter, keeping track of the next row number.","export a large database table to excel in chunks",null,{"breadcrumb":1930},[1931,1933,1936],{"name":1932,"item":1275},"Home",{"name":1934,"item":1935},"Advanced Data Transformation and Cleaning for Excel Automation","\u002Fadvanced-data-transformation-and-cleaning\u002F",{"name":30,"item":29},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fexport-a-large-database-table-to-excel-in-chunks",{"title":1939,"description":1940},"Export a Large Database Table to Excel in Chunks with Python","Stream a big SQL table into Excel: pandas read_sql chunksize or SQLAlchemy stream_results, XlsxWriter constant_memory, sheet rollover at the 1,048,576-row limit, typed cells, and memory and row-count checks.","export-a-large-database-table-to-excel-in-chunks","advanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fexport-a-large-database-table-to-excel-in-chunks\u002Findex",[1944,1945],"data-sources","large-files","how-to","Fp0L1GwI46hw7cXID2kADLFjknXVL9irJHW9GJ4qzTI",[1949,1953],{"title":1950,"path":1951,"stem":1952,"children":-1},"Download New Excel Files from an SFTP Server","\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fdownload-new-excel-files-from-an-sftp-server","advanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fdownload-new-excel-files-from-an-sftp-server\u002Findex",{"title":1954,"path":1955,"stem":1956,"children":-1},"Export Database Lookup Tables as Excel Dropdown Lists","\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fexport-database-lookup-tables-as-excel-dropdown-lists","advanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fexport-database-lookup-tables-as-excel-dropdown-lists\u002Findex",[1958,1960],{"path":1959,"title":1934},"\u002Fadvanced-data-transformation-and-cleaning",{"path":1961,"title":30},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases",[],[1964,1967,1970,1973,1974,1975],{"path":1965,"title":1966,"kind":1946},"\u002Fautomating-reporting-workflows\u002Fpublishing-excel-reports-to-cloud-storage\u002Fread-an-excel-file-directly-from-amazon-s3-with-pandas","Read an Excel File Directly from Amazon S3 with pandas",{"path":1968,"title":1969,"kind":1946},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fcache-api-responses-when-building-excel-reports","Cache API Responses When Building Excel Reports",{"path":1971,"title":1972,"kind":1946},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fcompare-an-excel-file-with-a-database-table","Compare an Excel File with a Database Table",{"path":1951,"title":1950,"kind":1946},{"path":1955,"title":1954,"kind":1946},{"path":1976,"title":1883,"kind":1946},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fexport-sql-query-results-to-excel-with-python",1790972096627]