[{"data":1,"prerenderedAt":2055},["ShallowReactive",2],{"doc:\u002Fadvanced-data-transformation-and-cleaning\u002Fworking-with-large-excel-files-in-python\u002Fremove-duplicate-rows-across-many-large-excel-files":3,"surround:\u002Fadvanced-data-transformation-and-cleaning\u002Fworking-with-large-excel-files-in-python\u002Fremove-duplicate-rows-across-many-large-excel-files":2021,"ancestors:\u002Fadvanced-data-transformation-and-cleaning\u002Fworking-with-large-excel-files-in-python\u002Fremove-duplicate-rows-across-many-large-excel-files":2030,"children:\u002Fadvanced-data-transformation-and-cleaning\u002Fworking-with-large-excel-files-in-python\u002Fremove-duplicate-rows-across-many-large-excel-files":2035,"related:\u002Fadvanced-data-transformation-and-cleaning\u002Fworking-with-large-excel-files-in-python\u002Fremove-duplicate-rows-across-many-large-excel-files":2036},{"id":4,"title":5,"body":6,"dateModified":1987,"datePublished":1987,"description":1988,"extension":1989,"faq":1990,"keyword":1999,"kind":2000,"meta":2001,"navigation":109,"path":2010,"seo":2011,"slug":2014,"stem":2015,"topics":2016,"type":2019,"__hash__":2020},"docs\u002Fadvanced-data-transformation-and-cleaning\u002Fworking-with-large-excel-files-in-python\u002Fremove-duplicate-rows-across-many-large-excel-files\u002Findex.md","Remove Duplicate Rows Across Many Large Excel Files",{"type":7,"value":8,"toc":1975},"minimark",[9,33,38,69,76,80,87,253,265,269,272,862,869,1021,1025,1032,1468,1483,1487,1490,1712,1715,1793,1846,1850,1864,1868,1903,1907,1910,1914,1920,1926,1932,1938,1942,1971],[10,11,12,13,17,18,21,22,26,27,32],"p",{},"Weekly exports often overlap. The system emits \"the last 30 days\" every week, so each record appears in four or five files; someone re-sends a file with a different name; two branches export the same shared customers. Combining a year of such files with ",[14,15,16],"code",{},"pd.concat"," and ",[14,19,20],{},"drop_duplicates"," works until the combined data no longer fits comfortably in memory — or until you need to know ",[23,24,25],"em",{},"which"," file each duplicate came from. A streaming approach handles any number of files with a small, fixed memory cost: read one row at a time, compute a fingerprint from the fields that define \"the same record\", keep only the set of fingerprints seen so far, and write each first-seen row straight to the output. This page builds that, logs every duplicate, and handles the \"keep the latest version\" case. It belongs to ",[28,29,31],"a",{"href":30},"\u002Fadvanced-data-transformation-and-cleaning\u002Fworking-with-large-excel-files-in-python\u002F","Working with Large Excel Files in Python",".",[34,35,37],"h2",{"id":36},"prerequisites","Prerequisites",[39,40,45],"pre",{"className":41,"code":42,"language":43,"meta":44,"style":44},"language-bash shiki shiki-themes github-light github-dark-high-contrast","pip install openpyxl xlsxwriter pandas\n","bash","",[14,46,47],{"__ignoreMap":44},[48,49,52,56,60,63,66],"span",{"class":50,"line":51},"line",1,[48,53,55],{"class":54},"sMTad","pip",[48,57,59],{"class":58},"srMev"," install",[48,61,62],{"class":58}," openpyxl",[48,64,65],{"class":58}," xlsxwriter",[48,67,68],{"class":58}," pandas\n",[10,70,71,72,75],{},"The ",[14,73,74],{},"exports\u002F"," folder holds eight exports of up to 15,000 orders each — 119,500 rows in total — covering overlapping windows, so most orders appear in two or three files. Some exports write order IDs in lower case, and later exports carry updated statuses for some orders. The task: one workbook with each order exactly once.",[34,77,79],{"id":78},"step-1-decide-what-makes-two-rows-the-same","Step 1 — Decide what makes two rows \"the same\"",[10,81,82,83,86],{},"Duplicates are defined by a ",[23,84,85],{},"key",", not by entire rows. Here an order is identified by its order ID — but the same order exported twice may differ in other fields (an updated status, a different export date), and the ID may be written in different case. Normalise the key fields and fingerprint them:",[39,88,92],{"className":89,"code":90,"language":91,"meta":44,"style":44},"language-python shiki shiki-themes github-light github-dark-high-contrast","import hashlib\n\nKEY = [\"Order ID\"]\n\ndef fingerprint(record):\n    parts = [str(record[k]).strip().upper() for k in KEY]\n    return hashlib.blake2b(\"\\x1f\".join(parts).encode(), digest_size=8).digest()\n\nprint(fingerprint({\"Order ID\": \"SO-100042\"}) == fingerprint({\"Order ID\": \" so-100042 \"}))\n","python",[14,93,94,104,111,130,135,148,179,212,217],{"__ignoreMap":44},[48,95,96,100],{"class":50,"line":51},[48,97,99],{"class":98},"s-kum","import",[48,101,103],{"class":102},"skGVy"," hashlib\n",[48,105,107],{"class":50,"line":106},2,[48,108,110],{"emptyLinePlaceholder":109},true,"\n",[48,112,114,118,121,124,127],{"class":50,"line":113},3,[48,115,117],{"class":116},"sP0c6","KEY",[48,119,120],{"class":98}," =",[48,122,123],{"class":102}," [",[48,125,126],{"class":58},"\"Order ID\"",[48,128,129],{"class":102},"]\n",[48,131,133],{"class":50,"line":132},4,[48,134,110],{"emptyLinePlaceholder":109},[48,136,138,141,145],{"class":50,"line":137},5,[48,139,140],{"class":98},"def",[48,142,144],{"class":143},"s_Opv"," fingerprint",[48,146,147],{"class":102},"(record):\n",[48,149,151,154,157,159,162,165,168,171,174,177],{"class":50,"line":150},6,[48,152,153],{"class":102},"    parts ",[48,155,156],{"class":98},"=",[48,158,123],{"class":102},[48,160,161],{"class":116},"str",[48,163,164],{"class":102},"(record[k]).strip().upper() ",[48,166,167],{"class":98},"for",[48,169,170],{"class":102}," k ",[48,172,173],{"class":98},"in",[48,175,176],{"class":116}," KEY",[48,178,129],{"class":102},[48,180,182,185,188,191,195,197,200,204,206,209],{"class":50,"line":181},7,[48,183,184],{"class":98},"    return",[48,186,187],{"class":102}," hashlib.blake2b(",[48,189,190],{"class":58},"\"",[48,192,194],{"class":193},"sSjpA","\\x1f",[48,196,190],{"class":58},[48,198,199],{"class":102},".join(parts).encode(), ",[48,201,203],{"class":202},"sa561","digest_size",[48,205,156],{"class":98},[48,207,208],{"class":116},"8",[48,210,211],{"class":102},").digest()\n",[48,213,215],{"class":50,"line":214},8,[48,216,110],{"emptyLinePlaceholder":109},[48,218,220,223,226,228,231,234,237,240,243,245,247,250],{"class":50,"line":219},9,[48,221,222],{"class":116},"print",[48,224,225],{"class":102},"(fingerprint({",[48,227,126],{"class":58},[48,229,230],{"class":102},": ",[48,232,233],{"class":58},"\"SO-100042\"",[48,235,236],{"class":102},"}) ",[48,238,239],{"class":98},"==",[48,241,242],{"class":102}," fingerprint({",[48,244,126],{"class":58},[48,246,230],{"class":102},[48,248,249],{"class":58},"\" so-100042 \"",[48,251,252],{"class":102},"}))\n",[10,254,255,256,17,259,262,263,32],{},"The fingerprint is an 8-byte BLAKE2 hash of the normalised key fields, joined with a separator character that cannot appear in the data. Normalising first — trimming and upper-casing — means ",[14,257,258],{},"so-100042",[14,260,261],{},"SO-100042"," count as the same order. Storing 8-byte hashes instead of the key strings keeps memory small: a million fingerprints take a few tens of megabytes in a Python set, regardless of how many columns the rows have. With 64 bits, the chance of two different keys colliding is negligible even for billions of rows. If the key is several columns, list them all in ",[14,264,117],{},[34,266,268],{"id":267},"step-2-stream-every-file-write-unique-rows-as-you-go","Step 2 — Stream every file, write unique rows as you go",[10,270,271],{},"Read each workbook in openpyxl's read-only mode, one row at a time, and write rows whose fingerprint has not been seen to an XlsxWriter workbook in constant-memory mode. Duplicates go to a log instead:",[39,273,275],{"className":89,"code":274,"language":91,"meta":44,"style":44},"from pathlib import Path\nfrom openpyxl import load_workbook\nimport xlsxwriter\n\nfiles = sorted(Path(\"exports\").glob(\"orders_week*.xlsx\"))\nseen = set()\ndupes = []\n\nout = xlsxwriter.Workbook(\"orders_unique.xlsx\", {\"constant_memory\": True})\nws_out = out.add_worksheet(\"Orders\")\ndate_fmt = out.add_format({\"num_format\": \"yyyy-mm-dd\"})\nout_row = 0\n\nfor f in files:\n    wb = load_workbook(f, read_only=True)\n    rows = wb.active.iter_rows(values_only=True)\n    header = list(next(rows))\n    if out_row == 0:\n        ws_out.write_row(0, 0, header + [\"Source file\"])\n        out_row = 1\n    for excel_row, values in enumerate(rows, start=2):\n        record = dict(zip(header, values))\n        fp = fingerprint(record)\n        if fp in seen:\n            dupes.append((record[\"Order ID\"], f.name, excel_row))\n            continue\n        seen.add(fp)\n        for c, v in enumerate(values):\n            if hasattr(v, \"year\"):\n                ws_out.write_datetime(out_row, c, v, date_fmt)\n            else:\n                ws_out.write(out_row, c, v)\n        ws_out.write(out_row, len(values), f.name)\n        out_row += 1\n    wb.close()\nout.close()\nprint(f\"read {sum(1 for _ in dupes) + out_row - 1:,} rows from {len(files)} files: \"\n      f\"{out_row - 1:,} unique, {len(dupes):,} duplicates\")\n",[14,276,277,290,302,309,313,338,351,361,365,392,409,430,441,446,459,479,499,519,536,564,575,603,622,633,647,658,664,670,686,703,709,717,723,735,745,751,757,823],{"__ignoreMap":44},[48,278,279,282,285,287],{"class":50,"line":51},[48,280,281],{"class":98},"from",[48,283,284],{"class":102}," pathlib ",[48,286,99],{"class":98},[48,288,289],{"class":102}," Path\n",[48,291,292,294,297,299],{"class":50,"line":106},[48,293,281],{"class":98},[48,295,296],{"class":102}," openpyxl ",[48,298,99],{"class":98},[48,300,301],{"class":102}," load_workbook\n",[48,303,304,306],{"class":50,"line":113},[48,305,99],{"class":98},[48,307,308],{"class":102}," xlsxwriter\n",[48,310,311],{"class":50,"line":132},[48,312,110],{"emptyLinePlaceholder":109},[48,314,315,318,320,323,326,329,332,335],{"class":50,"line":137},[48,316,317],{"class":102},"files ",[48,319,156],{"class":98},[48,321,322],{"class":116}," sorted",[48,324,325],{"class":102},"(Path(",[48,327,328],{"class":58},"\"exports\"",[48,330,331],{"class":102},").glob(",[48,333,334],{"class":58},"\"orders_week*.xlsx\"",[48,336,337],{"class":102},"))\n",[48,339,340,343,345,348],{"class":50,"line":150},[48,341,342],{"class":102},"seen ",[48,344,156],{"class":98},[48,346,347],{"class":116}," set",[48,349,350],{"class":102},"()\n",[48,352,353,356,358],{"class":50,"line":181},[48,354,355],{"class":102},"dupes ",[48,357,156],{"class":98},[48,359,360],{"class":102}," []\n",[48,362,363],{"class":50,"line":214},[48,364,110],{"emptyLinePlaceholder":109},[48,366,367,370,372,375,378,381,384,386,389],{"class":50,"line":219},[48,368,369],{"class":102},"out ",[48,371,156],{"class":98},[48,373,374],{"class":102}," xlsxwriter.Workbook(",[48,376,377],{"class":58},"\"orders_unique.xlsx\"",[48,379,380],{"class":102},", {",[48,382,383],{"class":58},"\"constant_memory\"",[48,385,230],{"class":102},[48,387,388],{"class":116},"True",[48,390,391],{"class":102},"})\n",[48,393,395,398,400,403,406],{"class":50,"line":394},10,[48,396,397],{"class":102},"ws_out ",[48,399,156],{"class":98},[48,401,402],{"class":102}," out.add_worksheet(",[48,404,405],{"class":58},"\"Orders\"",[48,407,408],{"class":102},")\n",[48,410,412,415,417,420,423,425,428],{"class":50,"line":411},11,[48,413,414],{"class":102},"date_fmt ",[48,416,156],{"class":98},[48,418,419],{"class":102}," out.add_format({",[48,421,422],{"class":58},"\"num_format\"",[48,424,230],{"class":102},[48,426,427],{"class":58},"\"yyyy-mm-dd\"",[48,429,391],{"class":102},[48,431,433,436,438],{"class":50,"line":432},12,[48,434,435],{"class":102},"out_row ",[48,437,156],{"class":98},[48,439,440],{"class":116}," 0\n",[48,442,444],{"class":50,"line":443},13,[48,445,110],{"emptyLinePlaceholder":109},[48,447,449,451,454,456],{"class":50,"line":448},14,[48,450,167],{"class":98},[48,452,453],{"class":102}," f ",[48,455,173],{"class":98},[48,457,458],{"class":102}," files:\n",[48,460,462,465,467,470,473,475,477],{"class":50,"line":461},15,[48,463,464],{"class":102},"    wb ",[48,466,156],{"class":98},[48,468,469],{"class":102}," load_workbook(f, ",[48,471,472],{"class":202},"read_only",[48,474,156],{"class":98},[48,476,388],{"class":116},[48,478,408],{"class":102},[48,480,482,485,487,490,493,495,497],{"class":50,"line":481},16,[48,483,484],{"class":102},"    rows ",[48,486,156],{"class":98},[48,488,489],{"class":102}," wb.active.iter_rows(",[48,491,492],{"class":202},"values_only",[48,494,156],{"class":98},[48,496,388],{"class":116},[48,498,408],{"class":102},[48,500,502,505,507,510,513,516],{"class":50,"line":501},17,[48,503,504],{"class":102},"    header ",[48,506,156],{"class":98},[48,508,509],{"class":116}," list",[48,511,512],{"class":102},"(",[48,514,515],{"class":116},"next",[48,517,518],{"class":102},"(rows))\n",[48,520,522,525,528,530,533],{"class":50,"line":521},18,[48,523,524],{"class":98},"    if",[48,526,527],{"class":102}," out_row ",[48,529,239],{"class":98},[48,531,532],{"class":116}," 0",[48,534,535],{"class":102},":\n",[48,537,539,542,545,548,550,553,556,558,561],{"class":50,"line":538},19,[48,540,541],{"class":102},"        ws_out.write_row(",[48,543,544],{"class":116},"0",[48,546,547],{"class":102},", ",[48,549,544],{"class":116},[48,551,552],{"class":102},", header ",[48,554,555],{"class":98},"+",[48,557,123],{"class":102},[48,559,560],{"class":58},"\"Source file\"",[48,562,563],{"class":102},"])\n",[48,565,567,570,572],{"class":50,"line":566},20,[48,568,569],{"class":102},"        out_row ",[48,571,156],{"class":98},[48,573,574],{"class":116}," 1\n",[48,576,578,581,584,586,589,592,595,597,600],{"class":50,"line":577},21,[48,579,580],{"class":98},"    for",[48,582,583],{"class":102}," excel_row, values ",[48,585,173],{"class":98},[48,587,588],{"class":116}," enumerate",[48,590,591],{"class":102},"(rows, ",[48,593,594],{"class":202},"start",[48,596,156],{"class":98},[48,598,599],{"class":116},"2",[48,601,602],{"class":102},"):\n",[48,604,606,609,611,614,616,619],{"class":50,"line":605},22,[48,607,608],{"class":102},"        record ",[48,610,156],{"class":98},[48,612,613],{"class":116}," dict",[48,615,512],{"class":102},[48,617,618],{"class":116},"zip",[48,620,621],{"class":102},"(header, values))\n",[48,623,625,628,630],{"class":50,"line":624},23,[48,626,627],{"class":102},"        fp ",[48,629,156],{"class":98},[48,631,632],{"class":102}," fingerprint(record)\n",[48,634,636,639,642,644],{"class":50,"line":635},24,[48,637,638],{"class":98},"        if",[48,640,641],{"class":102}," fp ",[48,643,173],{"class":98},[48,645,646],{"class":102}," seen:\n",[48,648,650,653,655],{"class":50,"line":649},25,[48,651,652],{"class":102},"            dupes.append((record[",[48,654,126],{"class":58},[48,656,657],{"class":102},"], f.name, excel_row))\n",[48,659,661],{"class":50,"line":660},26,[48,662,663],{"class":98},"            continue\n",[48,665,667],{"class":50,"line":666},27,[48,668,669],{"class":102},"        seen.add(fp)\n",[48,671,673,676,679,681,683],{"class":50,"line":672},28,[48,674,675],{"class":98},"        for",[48,677,678],{"class":102}," c, v ",[48,680,173],{"class":98},[48,682,588],{"class":116},[48,684,685],{"class":102},"(values):\n",[48,687,689,692,695,698,701],{"class":50,"line":688},29,[48,690,691],{"class":98},"            if",[48,693,694],{"class":116}," hasattr",[48,696,697],{"class":102},"(v, ",[48,699,700],{"class":58},"\"year\"",[48,702,602],{"class":102},[48,704,706],{"class":50,"line":705},30,[48,707,708],{"class":102},"                ws_out.write_datetime(out_row, c, v, date_fmt)\n",[48,710,712,715],{"class":50,"line":711},31,[48,713,714],{"class":98},"            else",[48,716,535],{"class":102},[48,718,720],{"class":50,"line":719},32,[48,721,722],{"class":102},"                ws_out.write(out_row, c, v)\n",[48,724,726,729,732],{"class":50,"line":725},33,[48,727,728],{"class":102},"        ws_out.write(out_row, ",[48,730,731],{"class":116},"len",[48,733,734],{"class":102},"(values), f.name)\n",[48,736,738,740,743],{"class":50,"line":737},34,[48,739,569],{"class":102},[48,741,742],{"class":98},"+=",[48,744,574],{"class":116},[48,746,748],{"class":50,"line":747},35,[48,749,750],{"class":102},"    wb.close()\n",[48,752,754],{"class":50,"line":753},36,[48,755,756],{"class":102},"out.close()\n",[48,758,760,762,764,767,770,773,776,778,781,784,787,789,792,794,796,799,802,805,808,811,813,815,818,820],{"class":50,"line":759},37,[48,761,222],{"class":116},[48,763,512],{"class":102},[48,765,766],{"class":98},"f",[48,768,769],{"class":58},"\"read ",[48,771,772],{"class":193},"{",[48,774,775],{"class":116},"sum",[48,777,512],{"class":102},[48,779,780],{"class":116},"1",[48,782,783],{"class":98}," for",[48,785,786],{"class":102}," _ ",[48,788,173],{"class":98},[48,790,791],{"class":102}," dupes) ",[48,793,555],{"class":98},[48,795,527],{"class":102},[48,797,798],{"class":98},"-",[48,800,801],{"class":116}," 1",[48,803,804],{"class":98},":,",[48,806,807],{"class":193},"}",[48,809,810],{"class":58}," rows from ",[48,812,772],{"class":193},[48,814,731],{"class":116},[48,816,817],{"class":102},"(files)",[48,819,807],{"class":193},[48,821,822],{"class":58}," files: \"\n",[48,824,826,829,831,833,835,837,839,841,843,846,848,850,853,855,857,860],{"class":50,"line":825},38,[48,827,828],{"class":98},"      f",[48,830,190],{"class":58},[48,832,772],{"class":193},[48,834,435],{"class":102},[48,836,798],{"class":98},[48,838,801],{"class":116},[48,840,804],{"class":98},[48,842,807],{"class":193},[48,844,845],{"class":58}," unique, ",[48,847,772],{"class":193},[48,849,731],{"class":116},[48,851,852],{"class":102},"(dupes)",[48,854,804],{"class":98},[48,856,807],{"class":193},[48,858,859],{"class":58}," duplicates\"",[48,861,408],{"class":102},[10,863,864,865,868],{},"Only one row of one file is in memory at a time, plus the set of fingerprints. Each unique row is written immediately with a source-file column, so the output says where every record came from; dates are written as real dates. The duplicate log keeps the order ID, file and Excel row number of every skipped row — the evidence anyone needs to check a surprising count. The result: 119,500 rows read, 60,000 unique orders written and 59,500 duplicates logged. Files are processed in name order, which here is also date order, so each order is kept from the ",[23,866,867],{},"first"," export it appeared in.",[870,871,879,880,879,884,879,888,879,895,879,905,879,916,879,923,879,928,879,934,879,939,879,943,879,950,879,956,879,960,879,964,879,968,879,971,879,976,879,983,879,988,879,991,879,995,879,998,879,1003,879,1010,879,1015,879,1018],"svg",{"viewBox":872,"role":873,"ariaLabelledBy":874,"xmlns":877,"style":878},"4 0 752 142","img",[875,876],"dd-stream-t","dd-stream-d","http:\u002F\u002Fwww.w3.org\u002F2000\u002Fsvg","width:100%;max-width:752px;height:auto;display:block;margin:1.5rem auto;font-family:Inter,ui-sans-serif,system-ui,sans-serif","\n  ",[881,882,883],"title",{"id":875},"Streaming de-duplication",[885,886,887],"desc",{"id":876},"Each file is read row by row in read-only mode. A fingerprint is computed from the normalised key fields. If it is already in the seen set, the row is logged as a duplicate with its file and row. Otherwise the fingerprint is added and the row is written immediately to the output workbook.",[889,890],"rect",{"x":891,"y":544,"width":892,"height":893,"fill":894},"4","752","142","#ffffff",[889,896],{"x":897,"y":898,"width":899,"height":900,"rx":901,"fill":902,"stroke":903,"style":904},"20.0","16.0","153.0","110.5","12","#e7ebef","var(--line,#cdd5e6)","stroke-width:2px",[906,907,909],"g",{"transform":908},"translate(84.5,37.8) scale(1.000)",[910,911],"path",{"d":912,"fill":913,"stroke":914,"style":915},"M4 4h16v16H4z M4 9h16 M4 14h16 M10 4v16","none","var(--text,#172033)","stroke-width:1.80px;stroke-linecap:round;stroke-linejoin:round",[917,918,922],"text",{"x":919,"y":920,"style":921},"96.5","84.2","font-size:13.5px;font-weight:700;fill:var(--text,#172033);text-anchor:middle","Read row",[917,924,927],{"x":919,"y":925,"style":926},"103.8","font-size:11.5px;font-weight:400;fill:var(--text,#172033);text-anchor:middle","openpyxl read-only",[50,929],{"x1":930,"y1":931,"x2":932,"y2":931,"stroke":933,"style":904},"178.0","71.2","200.0","var(--brand,#5b5cf0)",[935,936],"polygon",{"points":937,"fill":938},"204.0,71.2 195.0,76.2 195.0,66.2","#5b5cf0",[889,940],{"x":941,"y":898,"width":899,"height":900,"rx":901,"fill":942,"stroke":933,"style":904},"209.0","#f0f4ff",[906,944,946],{"transform":945},"translate(273.5,29.0) scale(1.000)",[910,947],{"d":948,"fill":913,"stroke":949,"style":915},"M5 9h14 M4 15h14 M10 4l-2 16 M16 4l-2 16","var(--brand-strong,#4338ca)",[917,951,955],{"x":952,"y":953,"style":954},"285.5","75.5","font-size:13.5px;font-weight:700;fill:var(--brand-strong,#4338ca);text-anchor:middle","Fingerprint",[917,957,959],{"x":952,"y":958,"style":926},"95.0","normalised key →",[917,961,963],{"x":952,"y":962,"style":926},"112.5","8-byte hash",[50,965],{"x1":966,"y1":931,"x2":967,"y2":931,"stroke":933,"style":904},"367.0","389.0",[935,969],{"points":970,"fill":938},"393.0,71.2 384.0,76.2 384.0,66.2",[889,972],{"x":973,"y":898,"width":899,"height":900,"rx":901,"fill":974,"stroke":975,"style":904},"398.0","#fdefd8","var(--gold,#b4740a)",[906,977,979],{"transform":978},"translate(462.5,37.8) scale(1.000)",[910,980],{"d":981,"fill":913,"stroke":982,"style":915},"M11 4a7 7 0 1 0 .01 0z M16 16l5 5","var(--gold-ink,#7a4e06)",[917,984,987],{"x":985,"y":920,"style":986},"474.5","font-size:13.5px;font-weight:700;fill:var(--gold-ink,#7a4e06);text-anchor:middle","Seen before?",[917,989,990],{"x":985,"y":925,"style":926},"yes → duplicate log",[50,992],{"x1":993,"y1":931,"x2":994,"y2":931,"stroke":933,"style":904},"556.0","578.0",[935,996],{"points":997,"fill":938},"582.0,71.2 573.0,76.2 573.0,66.2",[889,999],{"x":1000,"y":898,"width":899,"height":900,"rx":901,"fill":1001,"stroke":1002,"style":904},"587.0","#d9f4f1","var(--teal,#0f9488)",[906,1004,1006],{"transform":1005},"translate(651.5,29.0) scale(1.000)",[910,1007],{"d":1008,"fill":913,"stroke":1009,"style":915},"M6 3h9l4 4v14H6z M15 3v4h4 M9.5 11.5l4.5 6 M14 11.5l-4.5 6","var(--teal-ink,#0b6157)",[917,1011,1014],{"x":1012,"y":953,"style":1013},"663.5","font-size:13.5px;font-weight:700;fill:var(--teal-ink,#0b6157);text-anchor:middle","Write row",[917,1016,1017],{"x":1012,"y":958,"style":926},"XlsxWriter",[917,1019,1020],{"x":1012,"y":962,"style":926},"constant_memory",[34,1022,1024],{"id":1023},"step-3-keep-the-latest-copy-instead-of-the-first","Step 3 — Keep the latest copy instead of the first",[10,1026,1027,1028,1031],{},"Keeping the first copy is wrong when later exports carry corrections — here, later exports have updated statuses. To keep the ",[23,1029,1030],{},"latest"," copy without loading everything, make two passes: first find which file holds the newest copy of each key, then write only those rows:",[39,1033,1035],{"className":89,"code":1034,"language":91,"meta":44,"style":44},"latest = {}                                      # fingerprint -> index of the newest file containing it\nfor i, f in enumerate(files):                    # files are in export order\n    wb = load_workbook(f, read_only=True)\n    rows = wb.active.iter_rows(values_only=True)\n    header = list(next(rows))\n    key_cols = [header.index(k) for k in KEY]\n    for values in rows:\n        latest[fingerprint({k: values[c] for k, c in zip(KEY, key_cols)})] = i\n    wb.close()\n\nout = xlsxwriter.Workbook(\"orders_latest.xlsx\", {\"constant_memory\": True})\nws_out = out.add_worksheet(\"Orders\")\nr, written = 1, set()\nfor i, f in enumerate(files):\n    wb = load_workbook(f, read_only=True)\n    rows = wb.active.iter_rows(values_only=True)\n    header = list(next(rows))\n    if i == 0:\n        ws_out.write_row(0, 0, header + [\"Source file\"])\n    for values in rows:\n        fp = fingerprint(dict(zip(header, values)))\n        if latest[fp] == i and fp not in written:\n            written.add(fp)\n            ws_out.write_row(r, 0, [v.date().isoformat() if hasattr(v, \"year\") else v for v in values] + [f.name])\n            r += 1\n    wb.close()\nout.close()\nprint(f\"{r - 1:,} unique orders, each from its latest export\")\n",[14,1036,1037,1051,1068,1084,1100,1114,1134,1146,1173,1177,1181,1202,1214,1230,1243,1259,1275,1289,1302,1322,1332,1351,1376,1381,1423,1432,1436,1440],{"__ignoreMap":44},[48,1038,1039,1042,1044,1047],{"class":50,"line":51},[48,1040,1041],{"class":102},"latest ",[48,1043,156],{"class":98},[48,1045,1046],{"class":102}," {}                                      ",[48,1048,1050],{"class":1049},"s-wDw","# fingerprint -> index of the newest file containing it\n",[48,1052,1053,1055,1058,1060,1062,1065],{"class":50,"line":106},[48,1054,167],{"class":98},[48,1056,1057],{"class":102}," i, f ",[48,1059,173],{"class":98},[48,1061,588],{"class":116},[48,1063,1064],{"class":102},"(files):                    ",[48,1066,1067],{"class":1049},"# files are in export order\n",[48,1069,1070,1072,1074,1076,1078,1080,1082],{"class":50,"line":113},[48,1071,464],{"class":102},[48,1073,156],{"class":98},[48,1075,469],{"class":102},[48,1077,472],{"class":202},[48,1079,156],{"class":98},[48,1081,388],{"class":116},[48,1083,408],{"class":102},[48,1085,1086,1088,1090,1092,1094,1096,1098],{"class":50,"line":132},[48,1087,484],{"class":102},[48,1089,156],{"class":98},[48,1091,489],{"class":102},[48,1093,492],{"class":202},[48,1095,156],{"class":98},[48,1097,388],{"class":116},[48,1099,408],{"class":102},[48,1101,1102,1104,1106,1108,1110,1112],{"class":50,"line":137},[48,1103,504],{"class":102},[48,1105,156],{"class":98},[48,1107,509],{"class":116},[48,1109,512],{"class":102},[48,1111,515],{"class":116},[48,1113,518],{"class":102},[48,1115,1116,1119,1121,1124,1126,1128,1130,1132],{"class":50,"line":150},[48,1117,1118],{"class":102},"    key_cols ",[48,1120,156],{"class":98},[48,1122,1123],{"class":102}," [header.index(k) ",[48,1125,167],{"class":98},[48,1127,170],{"class":102},[48,1129,173],{"class":98},[48,1131,176],{"class":116},[48,1133,129],{"class":102},[48,1135,1136,1138,1141,1143],{"class":50,"line":181},[48,1137,580],{"class":98},[48,1139,1140],{"class":102}," values ",[48,1142,173],{"class":98},[48,1144,1145],{"class":102}," rows:\n",[48,1147,1148,1151,1153,1156,1158,1161,1163,1165,1168,1170],{"class":50,"line":214},[48,1149,1150],{"class":102},"        latest[fingerprint({k: values[c] ",[48,1152,167],{"class":98},[48,1154,1155],{"class":102}," k, c ",[48,1157,173],{"class":98},[48,1159,1160],{"class":116}," zip",[48,1162,512],{"class":102},[48,1164,117],{"class":116},[48,1166,1167],{"class":102},", key_cols)})] ",[48,1169,156],{"class":98},[48,1171,1172],{"class":102}," i\n",[48,1174,1175],{"class":50,"line":219},[48,1176,750],{"class":102},[48,1178,1179],{"class":50,"line":394},[48,1180,110],{"emptyLinePlaceholder":109},[48,1182,1183,1185,1187,1189,1192,1194,1196,1198,1200],{"class":50,"line":411},[48,1184,369],{"class":102},[48,1186,156],{"class":98},[48,1188,374],{"class":102},[48,1190,1191],{"class":58},"\"orders_latest.xlsx\"",[48,1193,380],{"class":102},[48,1195,383],{"class":58},[48,1197,230],{"class":102},[48,1199,388],{"class":116},[48,1201,391],{"class":102},[48,1203,1204,1206,1208,1210,1212],{"class":50,"line":432},[48,1205,397],{"class":102},[48,1207,156],{"class":98},[48,1209,402],{"class":102},[48,1211,405],{"class":58},[48,1213,408],{"class":102},[48,1215,1216,1219,1221,1223,1225,1228],{"class":50,"line":443},[48,1217,1218],{"class":102},"r, written ",[48,1220,156],{"class":98},[48,1222,801],{"class":116},[48,1224,547],{"class":102},[48,1226,1227],{"class":116},"set",[48,1229,350],{"class":102},[48,1231,1232,1234,1236,1238,1240],{"class":50,"line":448},[48,1233,167],{"class":98},[48,1235,1057],{"class":102},[48,1237,173],{"class":98},[48,1239,588],{"class":116},[48,1241,1242],{"class":102},"(files):\n",[48,1244,1245,1247,1249,1251,1253,1255,1257],{"class":50,"line":461},[48,1246,464],{"class":102},[48,1248,156],{"class":98},[48,1250,469],{"class":102},[48,1252,472],{"class":202},[48,1254,156],{"class":98},[48,1256,388],{"class":116},[48,1258,408],{"class":102},[48,1260,1261,1263,1265,1267,1269,1271,1273],{"class":50,"line":481},[48,1262,484],{"class":102},[48,1264,156],{"class":98},[48,1266,489],{"class":102},[48,1268,492],{"class":202},[48,1270,156],{"class":98},[48,1272,388],{"class":116},[48,1274,408],{"class":102},[48,1276,1277,1279,1281,1283,1285,1287],{"class":50,"line":501},[48,1278,504],{"class":102},[48,1280,156],{"class":98},[48,1282,509],{"class":116},[48,1284,512],{"class":102},[48,1286,515],{"class":116},[48,1288,518],{"class":102},[48,1290,1291,1293,1296,1298,1300],{"class":50,"line":521},[48,1292,524],{"class":98},[48,1294,1295],{"class":102}," i ",[48,1297,239],{"class":98},[48,1299,532],{"class":116},[48,1301,535],{"class":102},[48,1303,1304,1306,1308,1310,1312,1314,1316,1318,1320],{"class":50,"line":538},[48,1305,541],{"class":102},[48,1307,544],{"class":116},[48,1309,547],{"class":102},[48,1311,544],{"class":116},[48,1313,552],{"class":102},[48,1315,555],{"class":98},[48,1317,123],{"class":102},[48,1319,560],{"class":58},[48,1321,563],{"class":102},[48,1323,1324,1326,1328,1330],{"class":50,"line":566},[48,1325,580],{"class":98},[48,1327,1140],{"class":102},[48,1329,173],{"class":98},[48,1331,1145],{"class":102},[48,1333,1334,1336,1338,1341,1344,1346,1348],{"class":50,"line":577},[48,1335,627],{"class":102},[48,1337,156],{"class":98},[48,1339,1340],{"class":102}," fingerprint(",[48,1342,1343],{"class":116},"dict",[48,1345,512],{"class":102},[48,1347,618],{"class":116},[48,1349,1350],{"class":102},"(header, values)))\n",[48,1352,1353,1355,1358,1360,1362,1365,1367,1370,1373],{"class":50,"line":605},[48,1354,638],{"class":98},[48,1356,1357],{"class":102}," latest[fp] ",[48,1359,239],{"class":98},[48,1361,1295],{"class":102},[48,1363,1364],{"class":98},"and",[48,1366,641],{"class":102},[48,1368,1369],{"class":98},"not",[48,1371,1372],{"class":98}," in",[48,1374,1375],{"class":102}," written:\n",[48,1377,1378],{"class":50,"line":624},[48,1379,1380],{"class":102},"            written.add(fp)\n",[48,1382,1383,1386,1388,1391,1394,1396,1398,1400,1403,1406,1409,1411,1413,1415,1418,1420],{"class":50,"line":635},[48,1384,1385],{"class":102},"            ws_out.write_row(r, ",[48,1387,544],{"class":116},[48,1389,1390],{"class":102},", [v.date().isoformat() ",[48,1392,1393],{"class":98},"if",[48,1395,694],{"class":116},[48,1397,697],{"class":102},[48,1399,700],{"class":58},[48,1401,1402],{"class":102},") ",[48,1404,1405],{"class":98},"else",[48,1407,1408],{"class":102}," v ",[48,1410,167],{"class":98},[48,1412,1408],{"class":102},[48,1414,173],{"class":98},[48,1416,1417],{"class":102}," values] ",[48,1419,555],{"class":98},[48,1421,1422],{"class":102}," [f.name])\n",[48,1424,1425,1428,1430],{"class":50,"line":649},[48,1426,1427],{"class":102},"            r ",[48,1429,742],{"class":98},[48,1431,574],{"class":116},[48,1433,1434],{"class":50,"line":660},[48,1435,750],{"class":102},[48,1437,1438],{"class":50,"line":666},[48,1439,756],{"class":102},[48,1441,1442,1444,1446,1448,1450,1452,1455,1457,1459,1461,1463,1466],{"class":50,"line":672},[48,1443,222],{"class":116},[48,1445,512],{"class":102},[48,1447,766],{"class":98},[48,1449,190],{"class":58},[48,1451,772],{"class":193},[48,1453,1454],{"class":102},"r ",[48,1456,798],{"class":98},[48,1458,801],{"class":116},[48,1460,804],{"class":98},[48,1462,807],{"class":193},[48,1464,1465],{"class":58}," unique orders, each from its latest export\"",[48,1467,408],{"class":102},[10,1469,1470,1471,1474,1475,1478,1479,1482],{},"The first pass stores one small integer per fingerprint — the index of the newest file that contains it. The second pass writes each row only from that file. Memory is still proportional to the number of unique keys, not to the data. The cost is reading every file twice; with fast readers that is usually acceptable, and still far cheaper than loading all the files together. (For brevity this pass writes dates as ISO text; use ",[14,1472,1473],{},"write_datetime"," as in Step 2 in real code.) The ",[14,1476,1477],{},"written"," set guards against a key that appears twice ",[23,1480,1481],{},"within"," one file.",[34,1484,1486],{"id":1485},"step-4-check-the-result","Step 4 — Check the result",[10,1488,1489],{},"Verify both outputs with pandas — which is fine at this stage, because each output holds only unique rows:",[39,1491,1493],{"className":89,"code":1492,"language":91,"meta":44,"style":44},"import pandas as pd\n\nfirst = pd.read_excel(\"orders_unique.xlsx\")\nlast = pd.read_excel(\"orders_latest.xlsx\")\nfor name, df in ((\"first\", first), (\"latest\", last)):\n    ids = df[\"Order ID\"].str.upper()\n    print(f\"{name:6} rows {len(df):,}  duplicate ids {ids.duplicated().sum()}  \"\n          f\"shipped {int((df['Status'] == 'shipped').sum()):,}\")\n\nlog = pd.DataFrame(dupes, columns=[\"Order ID\", \"File\", \"Row\"])\nprint(log[\"File\"].value_counts().sort_index().to_dict())\n",[14,1494,1495,1508,1512,1526,1539,1563,1578,1626,1664,1668,1700],{"__ignoreMap":44},[48,1496,1497,1499,1502,1505],{"class":50,"line":51},[48,1498,99],{"class":98},[48,1500,1501],{"class":102}," pandas ",[48,1503,1504],{"class":98},"as",[48,1506,1507],{"class":102}," pd\n",[48,1509,1510],{"class":50,"line":106},[48,1511,110],{"emptyLinePlaceholder":109},[48,1513,1514,1517,1519,1522,1524],{"class":50,"line":113},[48,1515,1516],{"class":102},"first ",[48,1518,156],{"class":98},[48,1520,1521],{"class":102}," pd.read_excel(",[48,1523,377],{"class":58},[48,1525,408],{"class":102},[48,1527,1528,1531,1533,1535,1537],{"class":50,"line":132},[48,1529,1530],{"class":102},"last ",[48,1532,156],{"class":98},[48,1534,1521],{"class":102},[48,1536,1191],{"class":58},[48,1538,408],{"class":102},[48,1540,1541,1543,1546,1548,1551,1554,1557,1560],{"class":50,"line":137},[48,1542,167],{"class":98},[48,1544,1545],{"class":102}," name, df ",[48,1547,173],{"class":98},[48,1549,1550],{"class":102}," ((",[48,1552,1553],{"class":58},"\"first\"",[48,1555,1556],{"class":102},", first), (",[48,1558,1559],{"class":58},"\"latest\"",[48,1561,1562],{"class":102},", last)):\n",[48,1564,1565,1568,1570,1573,1575],{"class":50,"line":150},[48,1566,1567],{"class":102},"    ids ",[48,1569,156],{"class":98},[48,1571,1572],{"class":102}," df[",[48,1574,126],{"class":58},[48,1576,1577],{"class":102},"].str.upper()\n",[48,1579,1580,1583,1585,1587,1589,1591,1594,1597,1599,1602,1604,1606,1609,1611,1613,1616,1618,1621,1623],{"class":50,"line":181},[48,1581,1582],{"class":116},"    print",[48,1584,512],{"class":102},[48,1586,766],{"class":98},[48,1588,190],{"class":58},[48,1590,772],{"class":193},[48,1592,1593],{"class":102},"name",[48,1595,1596],{"class":98},":6",[48,1598,807],{"class":193},[48,1600,1601],{"class":58}," rows ",[48,1603,772],{"class":193},[48,1605,731],{"class":116},[48,1607,1608],{"class":102},"(df)",[48,1610,804],{"class":98},[48,1612,807],{"class":193},[48,1614,1615],{"class":58},"  duplicate ids ",[48,1617,772],{"class":193},[48,1619,1620],{"class":102},"ids.duplicated().sum()",[48,1622,807],{"class":193},[48,1624,1625],{"class":58},"  \"\n",[48,1627,1628,1631,1634,1636,1639,1642,1645,1648,1650,1653,1656,1658,1660,1662],{"class":50,"line":214},[48,1629,1630],{"class":98},"          f",[48,1632,1633],{"class":58},"\"shipped ",[48,1635,772],{"class":193},[48,1637,1638],{"class":116},"int",[48,1640,1641],{"class":102},"((df[",[48,1643,1644],{"class":58},"'Status'",[48,1646,1647],{"class":102},"] ",[48,1649,239],{"class":98},[48,1651,1652],{"class":58}," 'shipped'",[48,1654,1655],{"class":102},").sum())",[48,1657,804],{"class":98},[48,1659,807],{"class":193},[48,1661,190],{"class":58},[48,1663,408],{"class":102},[48,1665,1666],{"class":50,"line":219},[48,1667,110],{"emptyLinePlaceholder":109},[48,1669,1670,1673,1675,1678,1681,1683,1686,1688,1690,1693,1695,1698],{"class":50,"line":394},[48,1671,1672],{"class":102},"log ",[48,1674,156],{"class":98},[48,1676,1677],{"class":102}," pd.DataFrame(dupes, ",[48,1679,1680],{"class":202},"columns",[48,1682,156],{"class":98},[48,1684,1685],{"class":102},"[",[48,1687,126],{"class":58},[48,1689,547],{"class":102},[48,1691,1692],{"class":58},"\"File\"",[48,1694,547],{"class":102},[48,1696,1697],{"class":58},"\"Row\"",[48,1699,563],{"class":102},[48,1701,1702,1704,1707,1709],{"class":50,"line":411},[48,1703,222],{"class":116},[48,1705,1706],{"class":102},"(log[",[48,1708,1692],{"class":58},[48,1710,1711],{"class":102},"].value_counts().sort_index().to_dict())\n",[10,1713,1714],{},"Both outputs contain 60,000 orders with no duplicate IDs. But they disagree sharply on status: keeping the first copy leaves 12,880 orders marked shipped, keeping the latest gives 54,050, because each order is taken from its most recent export, after its status had moved on. Which is right depends on the question — but for a current order book, keeping the first copy would report tens of thousands of shipped orders as still open. The duplicate log, summarised by file — 8,500 repeats in each export after the first — shows how much each export overlapped with the ones before it — a useful sanity check that the files really are the overlapping windows you think they are. Writing the log to its own sheet, or to a CSV for very large runs, keeps the evidence with the result.",[870,1716,879,1721,879,1724,879,1727,879,1730,879,1735,879,1742,879,1748,879,1754,879,1759,879,1763,879,1767,879,1769,879,1775,879,1780,879,1784,879,1787,879,1790],{"viewBox":1717,"role":873,"ariaLabelledBy":1718,"xmlns":877,"style":878},"4 0 752 207",[1719,1720],"dd-keep-t","dd-keep-d",[881,1722,1723],{"id":1719},"Keeping the first copy or the latest",[885,1725,1726],{"id":1720},"Keeping the first copy needs one pass and the set of fingerprints, but keeps outdated values when later exports carry corrections. Keeping the latest needs two passes and a fingerprint-to-file map, and keeps the most recent version of every record.",[889,1728],{"x":891,"y":544,"width":892,"height":1729,"fill":894},"207",[889,1731],{"x":897,"y":898,"width":1732,"height":1733,"rx":1734,"fill":942,"stroke":933,"style":904},"351.0","175.0","14",[906,1736,1738],{"transform":1737},"translate(182.5,29.0) scale(1.083)",[910,1739],{"d":1740,"fill":913,"stroke":949,"style":1741},"M13 2L4 14h7l-1 8 9-12h-7z","stroke-width:1.66px;stroke-linecap:round;stroke-linejoin:round",[917,1743,1747],{"x":1744,"y":1745,"style":1746},"195.5","78.0","font-size:14px;font-weight:700;fill:var(--brand-strong,#4338ca);text-anchor:middle","keep first",[50,1749],{"x1":1750,"y1":1751,"x2":1752,"y2":1751,"stroke":933,"style":1753},"36.0","91.0","355.0","stroke-width:1px",[917,1755,1758],{"x":1744,"y":1756,"style":1757},"111.0","font-size:12px;font-weight:400;fill:var(--text,#172033);text-anchor:middle","one pass over the files",[917,1760,1762],{"x":1744,"y":1761,"style":1757},"137.0","set of 8-byte fingerprints",[917,1764,1766],{"x":1744,"y":1765,"style":1757},"163.0","older status kept",[889,1768],{"x":967,"y":898,"width":1732,"height":1733,"rx":1734,"fill":1001,"stroke":1002,"style":904},[906,1770,1772],{"transform":1771},"translate(551.5,29.0) scale(1.083)",[910,1773],{"d":1774,"fill":913,"stroke":1009,"style":1741},"M20 11a8 8 0 1 0-2.3 5.7 M20 4v7h-7",[917,1776,1779],{"x":1777,"y":1745,"style":1778},"564.5","font-size:14px;font-weight:700;fill:var(--teal-ink,#0b6157);text-anchor:middle","keep latest",[50,1781],{"x1":1782,"y1":1751,"x2":1783,"y2":1751,"stroke":1002,"style":1753},"405.0","724.0",[917,1785,1786],{"x":1777,"y":1756,"style":1757},"two passes over the files",[917,1788,1789],{"x":1777,"y":1761,"style":1757},"fingerprint → newest file index",[917,1791,1792],{"x":1777,"y":1765,"style":1757},"updated status kept",[870,1794,879,1800,879,1803,879,1806,879,1810,879,1815,879,1821,879,1826,879,1831,879,1834,879,1838,879,1841],{"viewBox":1795,"role":873,"ariaLabelledBy":1796,"xmlns":877,"style":1799},"4 0 744 100",[1797,1798],"dd-status-t","dd-status-d","width:100%;max-width:744px;height:auto;display:block;margin:1.5rem auto;font-family:Inter,ui-sans-serif,system-ui,sans-serif",[881,1801,1802],{"id":1797},"Shipped orders after de-duplication",[885,1804,1805],{"id":1798},"Both outputs contain the same 60,000 orders. Keeping each order's first copy leaves 12,880 marked as shipped. Keeping each order's latest copy gives 54,050 shipped, because later exports carry the updated status.",[889,1807],{"x":891,"y":544,"width":1808,"height":1809,"fill":894},"744","100",[889,1811],{"x":897,"y":898,"width":1812,"height":1813,"rx":1814,"fill":902,"stroke":903,"style":1753},"133.6","28.0","7",[917,1816,1820],{"x":1817,"y":1818,"style":1819},"86.8","34.5","font-size:12.5px;font-weight:700;fill:var(--text,#172033);text-anchor:middle","keep first copy",[889,1822],{"x":1823,"y":898,"width":1824,"height":1813,"rx":1814,"fill":974,"stroke":975,"style":1825},"163.6","110.2","stroke-width:1.5px",[917,1827,1830],{"x":1828,"y":1818,"style":1829},"281.8","font-size:12px;font-weight:700;fill:var(--gold-ink,#7a4e06);text-anchor:start","12,880 shipped",[889,1832],{"x":897,"y":1833,"width":1812,"height":1813,"rx":1814,"fill":902,"stroke":903,"style":1753},"56.0",[917,1835,1837],{"x":1817,"y":1836,"style":1819},"74.5","keep latest copy",[889,1839],{"x":1823,"y":1833,"width":1840,"height":1813,"rx":1814,"fill":1001,"stroke":1002,"style":1825},"462.3",[917,1842,1845],{"x":1843,"y":1836,"style":1844},"634.0","font-size:12px;font-weight:700;fill:var(--teal-ink,#0b6157);text-anchor:start","54,050 shipped",[34,1847,1849],{"id":1848},"when-pandas-is-the-better-tool","When pandas is the better tool",[10,1851,1852,1853,1855,1856,1859,1860,32],{},"If the combined data fits comfortably in memory — say, a few hundred thousand rows on a normal laptop — ",[14,1854,16],{}," followed by ",[14,1857,1858],{},"drop_duplicates(subset=..., keep=\"last\")"," is simpler, and normalising keys first works the same way: add a column with the normalised key and de-duplicate on that. The streaming version earns its place when the files keep growing, when memory is limited (a container, a shared server), or when you need a per-row duplicate log. Both give the same answer; the test in Step 4 is the same either way. Duplicate detection within a single file is covered in ",[28,1861,1863],{"href":1862},"\u002Fadvanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002Ffind-duplicate-rows-in-excel-with-python\u002F","Find Duplicate Rows in Excel with Python",[34,1865,1867],{"id":1866},"common-pitfalls","Common pitfalls",[1869,1870,1871,1879,1885,1891,1897],"ul",{},[1872,1873,1874,1878],"li",{},[1875,1876,1877],"strong",{},"Duplicates defined on whole rows."," Updated fields make true duplicates look different; define a key.",[1872,1880,1881,1884],{},[1875,1882,1883],{},"Unnormalised keys."," Case and spaces make the same key look different.",[1872,1886,1887,1890],{},[1875,1888,1889],{},"Keeping the first copy when later files correct earlier ones."," Use the two-pass latest approach.",[1872,1892,1893,1896],{},[1875,1894,1895],{},"No record of what was dropped."," Log every duplicate with file and row.",[1872,1898,1899,1902],{},[1875,1900,1901],{},"Processing files in an arbitrary order."," Sort by export date so \"first\" and \"latest\" mean what you intend.",[34,1904,1906],{"id":1905},"conclusion","Conclusion",[10,1908,1909],{},"To remove duplicates across many large Excel files, stream them: read each workbook row by row in read-only mode, fingerprint every record from its normalised key fields, keep only the set of fingerprints in memory, and write first-seen rows straight to an XlsxWriter workbook in constant-memory mode while logging every duplicate with its file and row. When later exports correct earlier ones, a two-pass variant keeps each record's latest copy. Memory then depends on the number of unique keys, not on the size or number of files.",[34,1911,1913],{"id":1912},"frequently-asked-questions","Frequently asked questions",[10,1915,1916,1919],{},[1875,1917,1918],{},"How do I remove duplicates across several Excel files in Python?","\nRead the files in order and keep a set of keys already seen; write a row only if its key is new. For data that fits in memory, pd.concat followed by drop_duplicates on the key columns does the same.",[10,1921,1922,1925],{},[1875,1923,1924],{},"How do I de-duplicate Excel data that is too big for memory?","\nStream rows with openpyxl read-only mode, keep a set of compact hashes of the key fields, and write unique rows directly to the output with XlsxWriter constant_memory mode.",[10,1927,1928,1931],{},[1875,1929,1930],{},"How do I keep the latest version of a duplicated record?","\nMake a first pass recording, for each key, the newest file that contains it, then a second pass writing each row only from that file.",[10,1933,1934,1937],{},[1875,1935,1936],{},"Why are obvious duplicates not detected?","\nUsually because the key differs in case or spacing, or because duplicates are compared on whole rows where some fields changed. Normalise the key and compare on the key only.",[34,1939,1941],{"id":1940},"related","Related",[1869,1943,1944,1950,1957,1964],{},[1872,1945,1946,1947,1949],{},"Up one level: ",[28,1948,31],{"href":30}," — big files, fast code.",[1872,1951,1952,1956],{},[28,1953,1955],{"href":1954},"\u002Fadvanced-data-transformation-and-cleaning\u002Fcleaning-excel-data-with-pandas\u002Fkeep-the-latest-record-per-key-with-pandas\u002F","Keep the Latest Record per Key with pandas"," — the in-memory version of Step 3.",[1872,1958,1959,1963],{},[28,1960,1962],{"href":1961},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmerging-and-joining-excel-dataframes\u002Fappend-new-rows-to-a-master-excel-file-without-duplicates\u002F","Append New Rows to a Master Excel File Without Duplicates"," — de-duplicating incrementally.",[1872,1965,1966,1970],{},[28,1967,1969],{"href":1968},"\u002Fadvanced-data-transformation-and-cleaning\u002Fworking-with-large-excel-files-in-python\u002Fwrite-large-dataframes-to-excel-with-write-only-mode\u002F","Write Large DataFrames to Excel with Write-Only Mode"," — streaming output with openpyxl.",[1972,1973,1974],"style",{},"html pre.shiki code .sMTad, html code.shiki .sMTad{--shiki-default:#6F42C1;--shiki-dark:#FFB757}html pre.shiki code .srMev, html code.shiki .srMev{--shiki-default:#032F62;--shiki-dark:#ADDCFF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .s-kum, html code.shiki .s-kum{--shiki-default:#D73A49;--shiki-dark:#FF9492}html pre.shiki code .skGVy, html code.shiki .skGVy{--shiki-default:#24292E;--shiki-dark:#F0F3F6}html pre.shiki code .sP0c6, html code.shiki .sP0c6{--shiki-default:#005CC5;--shiki-dark:#91CBFF}html pre.shiki code .s_Opv, html code.shiki .s_Opv{--shiki-default:#6F42C1;--shiki-dark:#DBB7FF}html pre.shiki code .sSjpA, html code.shiki .sSjpA{--shiki-default:#005CC5;--shiki-dark:#FF9492}html pre.shiki code .sa561, html code.shiki .sa561{--shiki-default:#E36209;--shiki-dark:#FFB757}html pre.shiki code .s-wDw, html code.shiki .s-wDw{--shiki-default:#6A737D;--shiki-dark:#BDC4CC}",{"title":44,"searchDepth":106,"depth":106,"links":1976},[1977,1978,1979,1980,1981,1982,1983,1984,1985,1986],{"id":36,"depth":106,"text":37},{"id":78,"depth":106,"text":79},{"id":267,"depth":106,"text":268},{"id":1023,"depth":106,"text":1024},{"id":1485,"depth":106,"text":1486},{"id":1848,"depth":106,"text":1849},{"id":1866,"depth":106,"text":1867},{"id":1905,"depth":106,"text":1906},{"id":1912,"depth":106,"text":1913},{"id":1940,"depth":106,"text":1941},"2026-10-02","De-duplicate records spread over dozens of overlapping Excel exports without loading them all at once: stream rows file by file, fingerprint each record on normalised key fields, keep only a set of hashes in memory, write unique rows straight to an output workbook, log every duplicate with its source file and row, and choose between keeping the first or the latest copy.","md",[1991,1993,1995,1997],{"q":1918,"a":1992},"Read the files in order and keep a set of keys already seen; write a row only if its key is new. For data that fits in memory, pd.concat followed by drop_duplicates on the key columns does the same.",{"q":1924,"a":1994},"Stream rows with openpyxl read-only mode, keep a set of compact hashes of the key fields, and write unique rows directly to the output with XlsxWriter constant_memory mode.",{"q":1930,"a":1996},"Make a first pass recording, for each key, the newest file that contains it, then a second pass writing each row only from that file.",{"q":1936,"a":1998},"Usually because the key differs in case or spacing, or because duplicates are compared on whole rows where some fields changed. Normalise the key and compare on the key only.","remove duplicate rows across many large excel files",null,{"breadcrumb":2002},[2003,2006,2009],{"name":2004,"item":2005},"Home","\u002F",{"name":2007,"item":2008},"Advanced Data Transformation and Cleaning for Excel Automation","\u002Fadvanced-data-transformation-and-cleaning\u002F",{"name":31,"item":30},"\u002Fadvanced-data-transformation-and-cleaning\u002Fworking-with-large-excel-files-in-python\u002Fremove-duplicate-rows-across-many-large-excel-files",{"title":2012,"description":2013},"Remove Duplicates Across Many Large Excel Files in Python","Stream overlapping Excel exports, hash normalised key fields to spot duplicates with little memory, write unique rows with XlsxWriter, log duplicates by file and row, and keep the first or latest copy.","remove-duplicate-rows-across-many-large-excel-files","advanced-data-transformation-and-cleaning\u002Fworking-with-large-excel-files-in-python\u002Fremove-duplicate-rows-across-many-large-excel-files\u002Findex",[2017,2018],"large-files","data-cleaning","how-to","2p6OzxI-9lgExvPJh8G0TgB4qaCSYkM2DgWsYTIb_Eo",[2022,2026],{"title":2023,"path":2024,"stem":2025,"children":-1},"Reduce the File Size of an Excel Workbook with Python","\u002Fadvanced-data-transformation-and-cleaning\u002Fworking-with-large-excel-files-in-python\u002Freduce-the-file-size-of-an-excel-workbook-with-python","advanced-data-transformation-and-cleaning\u002Fworking-with-large-excel-files-in-python\u002Freduce-the-file-size-of-an-excel-workbook-with-python\u002Findex",{"title":2027,"path":2028,"stem":2029,"children":-1},"Speed Up openpyxl with read_only Mode","\u002Fadvanced-data-transformation-and-cleaning\u002Fworking-with-large-excel-files-in-python\u002Fspeed-up-openpyxl-with-read-only-mode","advanced-data-transformation-and-cleaning\u002Fworking-with-large-excel-files-in-python\u002Fspeed-up-openpyxl-with-read-only-mode\u002Findex",[2031,2033],{"path":2032,"title":2007},"\u002Fadvanced-data-transformation-and-cleaning",{"path":2034,"title":31},"\u002Fadvanced-data-transformation-and-cleaning\u002Fworking-with-large-excel-files-in-python",[],[2037,2040,2043,2046,2049,2052],{"path":2038,"title":2039,"kind":2019},"\u002Fadvanced-data-transformation-and-cleaning\u002Freading-excel-with-polars-and-arrow\u002Fspeed-up-pandas-excel-reads-with-the-calamine-engine","Speed Up pandas Excel Reads with the calamine Engine",{"path":2041,"title":2042,"kind":2019},"\u002Fautomating-reporting-workflows\u002Ftesting-and-packaging-excel-automation-scripts\u002Fprofile-a-slow-excel-report-script","Profile a Slow Excel Report Script in Python",{"path":2044,"title":2045,"kind":2019},"\u002Fformatting-and-charting-excel-reports-with-python\u002Fbuilding-excel-reports-with-xlsxwriter\u002Fwrite-a-million-rows-to-excel-with-xlsxwriter-constant-memory","Write a Million Rows to Excel with xlsxwriter",{"path":2047,"title":2048,"kind":2019},"\u002Fgetting-started-with-python-excel-automation\u002Fchoosing-a-python-excel-library\u002Fbenchmark-python-excel-read-and-write-speed","Benchmark Python Excel Read and Write Speed",{"path":2050,"title":2051,"kind":2019},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fexport-a-large-database-table-to-excel-in-chunks","Export a Large Database Table to Excel in Chunks",{"path":2053,"title":2054,"kind":2019},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fquery-excel-files-with-sql-using-duckdb","Query Excel Files with SQL Using DuckDB",1790972106777]