[{"data":1,"prerenderedAt":2336},["ShallowReactive",2],{"doc:\u002Fadvanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002Frun-a-benfords-law-check-on-excel-data":3,"surround:\u002Fadvanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002Frun-a-benfords-law-check-on-excel-data":2303,"ancestors:\u002Fadvanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002Frun-a-benfords-law-check-on-excel-data":2312,"children:\u002Fadvanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002Frun-a-benfords-law-check-on-excel-data":2317,"related:\u002Fadvanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002Frun-a-benfords-law-check-on-excel-data":2318},{"id":4,"title":5,"body":6,"dateModified":2270,"datePublished":2270,"description":2271,"extension":2272,"faq":2273,"keyword":2282,"kind":2283,"meta":2284,"navigation":331,"path":2292,"seo":2293,"slug":2296,"stem":2297,"topics":2298,"type":2301,"__hash__":2302},"docs\u002Fadvanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002Frun-a-benfords-law-check-on-excel-data\u002Findex.md","Run a Benford's Law Check on Excel Data",{"type":7,"value":8,"toc":2257},"minimark",[9,19,24,59,65,69,84,279,283,290,474,485,488,575,579,582,788,792,795,991,994,1160,1164,1171,1342,1345,1401,1500,1503,1514,1600,1604,1607,2130,2133,2137,2178,2182,2185,2189,2195,2201,2207,2213,2217,2253],[10,11,12,13,18],"p",{},"In many naturally occurring sets of numbers — invoice amounts, expense claims, populations, river lengths — the leading digit is 1 about 30% of the time and 9 less than 5% of the time. That pattern is Benford's law, and auditors use it as a screening test: data that departs from it markedly is worth a closer look, because people who invent or adjust numbers tend to spread digits more evenly or cluster them below approval limits. It does not prove anything on its own; it tells you where to look. This page runs the test on an expense ledger with pandas. It is part of ",[14,15,17],"a",{"href":16},"\u002Fadvanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002F","Validating Excel Data with Python",".",[20,21,23],"h2",{"id":22},"prerequisites","Prerequisites",[25,26,31],"pre",{"className":27,"code":28,"language":29,"meta":30,"style":30},"language-bash shiki shiki-themes github-light github-dark-high-contrast","pip install pandas scipy openpyxl xlsxwriter\n","bash","",[32,33,34],"code",{"__ignoreMap":30},[35,36,39,43,47,50,53,56],"span",{"class":37,"line":38},"line",1,[35,40,42],{"class":41},"sMTad","pip",[35,44,46],{"class":45},"srMev"," install",[35,48,49],{"class":45}," pandas",[35,51,52],{"class":45}," scipy",[35,54,55],{"class":45}," openpyxl",[35,57,58],{"class":45}," xlsxwriter\n",[10,60,61,64],{},[32,62,63],{},"expense_claims.xlsx"," has a year of expense claims: claim ID, employee, category and amount, from a couple of pounds for a coffee to several thousand for equipment. Claims of 5,000 or more need a director's approval. That rule is the kind of thing Benford analysis is good at exposing.",[20,66,68],{"id":67},"why-the-leading-digit-is-not-uniform","Why the leading digit is not uniform",[10,70,71,72,76,77,79,80,83],{},"Benford's law says the probability that the first significant digit is ",[73,74,75],"em",{},"d"," is log₁₀(1 + 1\u002F",[73,78,75],{},"). It holds for data that spans several orders of magnitude and arises from multiplying or growing processes: going from 1,000 to 2,000 needs a 100% increase, from 9,000 to 10,000 only 11%, so values spend longer with a leading 1. It does ",[73,81,82],{},"not"," apply to data with a narrow range (heights of adults), assigned numbers (phone numbers, invoice numbers) or values bounded by rules (a fixed fee, a price list).",[85,86,94,95,94,99,94,103,94,111,94,118,94,125,94,129,94,133,94,136,94,140,94,143,94,147,94,150,94,154,94,158,94,164,94,168,94,172,94,175,94,179,94,183,94,187,94,191,94,195,94,203,94,208,94,213,94,217,94,222,94,226,94,231,94,235,94,240,94,243,94,248,94,252,94,257,94,261,94,266,94,270,94,275],"svg",{"viewBox":87,"role":88,"ariaLabelledBy":89,"xmlns":92,"style":93},"4 0 748 252","img",[90,91],"bf-expected-t","bf-expected-d","http:\u002F\u002Fwww.w3.org\u002F2000\u002Fsvg","width:100%;max-width:748px;height:auto;display:block;margin:1.5rem auto;font-family:Inter,ui-sans-serif,system-ui,sans-serif","\n  ",[96,97,98],"title",{"id":90},"Expected share of each first digit under Benford's law",[100,101,102],"desc",{"id":91},"A column chart of the expected percentage of values whose first digit is 1 to 9. The share falls from 30.1 percent for 1 to 17.6 for 2, 12.5 for 3, 9.7 for 4, 7.9 for 5, 6.7 for 6, 5.8 for 7, 5.1 for 8 and 4.6 percent for 9.",[104,105],"rect",{"x":106,"y":107,"width":108,"height":109,"fill":110},"4","0","748","252","#ffffff",[37,112],{"x1":113,"y1":114,"x2":115,"y2":114,"stroke":116,"style":117},"58.1","214.0","736.0","var(--line,#cdd5e6)","stroke-width:1px",[119,120,124],"text",{"x":121,"y":122,"style":123},"33.0","218.0","font-size:11px;font-weight:400;fill:var(--muted,#5b6780);text-anchor:middle","0%",[37,126],{"x1":113,"y1":127,"x2":115,"y2":127,"stroke":116,"style":128},"166.5","stroke-width:1px;stroke-dasharray:5 4",[119,130,132],{"x":121,"y":131,"style":123},"170.5","10%",[37,134],{"x1":113,"y1":135,"x2":115,"y2":135,"stroke":116,"style":128},"119.0",[119,137,139],{"x":121,"y":138,"style":123},"123.0","20%",[37,141],{"x1":113,"y1":142,"x2":115,"y2":142,"stroke":116,"style":128},"71.5",[119,144,146],{"x":121,"y":145,"style":123},"75.5","30%",[37,148],{"x1":113,"y1":149,"x2":115,"y2":149,"stroke":116,"style":128},"24.0",[119,151,153],{"x":121,"y":152,"style":123},"28.0","40%",[37,155],{"x1":113,"y1":114,"x2":115,"y2":114,"stroke":156,"style":157},"var(--muted,#5b6780)","stroke-width:1.5px",[119,159,163],{"x":160,"y":161,"style":162},"95.8","232.0","font-size:11.5px;font-weight:400;fill:var(--text,#172033);text-anchor:middle","1",[119,165,167],{"x":166,"y":161,"style":162},"171.1","2",[119,169,171],{"x":170,"y":161,"style":162},"246.4","3",[119,173,106],{"x":174,"y":161,"style":162},"321.7",[119,176,178],{"x":177,"y":161,"style":162},"397.0","5",[119,180,182],{"x":181,"y":161,"style":162},"472.4","6",[119,184,186],{"x":185,"y":161,"style":162},"547.7","7",[119,188,190],{"x":189,"y":161,"style":162},"623.0","8",[119,192,194],{"x":193,"y":161,"style":162},"698.3","9",[104,196],{"x":197,"y":198,"width":199,"height":200,"rx":171,"fill":201,"stroke":202,"style":157},"77.8","71.0","36.0","143.0","#f0f4ff","var(--brand,#5b5cf0)",[119,204,207],{"x":160,"y":205,"style":206},"65.0","font-size:10.5px;font-weight:700;fill:var(--brand-strong,#4338ca);text-anchor:middle","30.1",[104,209],{"x":210,"y":211,"width":199,"height":212,"rx":171,"fill":201,"stroke":202,"style":157},"153.1","130.4","83.6",[119,214,216],{"x":166,"y":215,"style":206},"124.4","17.6",[104,218],{"x":219,"y":220,"width":199,"height":221,"rx":171,"fill":201,"stroke":202,"style":157},"228.4","154.6","59.4",[119,223,225],{"x":170,"y":224,"style":206},"148.6","12.5",[104,227],{"x":228,"y":229,"width":199,"height":230,"rx":171,"fill":201,"stroke":202,"style":157},"303.7","167.9","46.1",[119,232,234],{"x":174,"y":233,"style":206},"161.9","9.7",[104,236],{"x":237,"y":238,"width":199,"height":239,"rx":171,"fill":201,"stroke":202,"style":157},"379.0","176.5","37.5",[119,241,242],{"x":177,"y":131,"style":206},"7.9",[104,244],{"x":245,"y":246,"width":199,"height":247,"rx":171,"fill":201,"stroke":202,"style":157},"454.4","182.2","31.8",[119,249,251],{"x":181,"y":250,"style":206},"176.2","6.7",[104,253],{"x":254,"y":255,"width":199,"height":256,"rx":171,"fill":201,"stroke":202,"style":157},"529.7","186.4","27.5",[119,258,260],{"x":185,"y":259,"style":206},"180.4","5.8",[104,262],{"x":263,"y":264,"width":199,"height":265,"rx":171,"fill":201,"stroke":202,"style":157},"605.0","189.8","24.2",[119,267,269],{"x":189,"y":268,"style":206},"183.8","5.1",[104,271],{"x":272,"y":273,"width":199,"height":274,"rx":171,"fill":201,"stroke":202,"style":157},"680.3","192.2","21.8",[119,276,278],{"x":193,"y":277,"style":206},"186.2","4.6",[20,280,282],{"id":281},"step-1-extract-the-first-significant-digit","Step 1 — Extract the first significant digit",[10,284,285,286,289],{},"The first ",[73,287,288],{},"significant"," digit of 0.045 is 4, and of 1,250.00 it is 1. Converting to a string and taking the first character gets both wrong: \"0\" for the first, and for negative numbers \"-\". Use logarithms instead, after taking the absolute value:",[25,291,295],{"className":292,"code":293,"language":294,"meta":30,"style":30},"language-python shiki shiki-themes github-light github-dark-high-contrast","import numpy as np\nimport pandas as pd\n\nclaims = pd.read_excel(\"expense_claims.xlsx\")\namount = claims[\"Amount\"].abs()\n\ndef first_digit(x):\n    x = x[x > 0]\n    return (x \u002F 10 ** np.floor(np.log10(x))).astype(int)\n\nprint(first_digit(pd.Series([0.045, 1250.00, 9.99, 10.0, 999_999.0])).tolist())\n","python",[32,296,297,313,326,333,351,368,373,386,407,433,438],{"__ignoreMap":30},[35,298,299,303,307,310],{"class":37,"line":38},[35,300,302],{"class":301},"s-kum","import",[35,304,306],{"class":305},"skGVy"," numpy ",[35,308,309],{"class":301},"as",[35,311,312],{"class":305}," np\n",[35,314,316,318,321,323],{"class":37,"line":315},2,[35,317,302],{"class":301},[35,319,320],{"class":305}," pandas ",[35,322,309],{"class":301},[35,324,325],{"class":305}," pd\n",[35,327,329],{"class":37,"line":328},3,[35,330,332],{"emptyLinePlaceholder":331},true,"\n",[35,334,336,339,342,345,348],{"class":37,"line":335},4,[35,337,338],{"class":305},"claims ",[35,340,341],{"class":301},"=",[35,343,344],{"class":305}," pd.read_excel(",[35,346,347],{"class":45},"\"expense_claims.xlsx\"",[35,349,350],{"class":305},")\n",[35,352,354,357,359,362,365],{"class":37,"line":353},5,[35,355,356],{"class":305},"amount ",[35,358,341],{"class":301},[35,360,361],{"class":305}," claims[",[35,363,364],{"class":45},"\"Amount\"",[35,366,367],{"class":305},"].abs()\n",[35,369,371],{"class":37,"line":370},6,[35,372,332],{"emptyLinePlaceholder":331},[35,374,376,379,383],{"class":37,"line":375},7,[35,377,378],{"class":301},"def",[35,380,382],{"class":381},"s_Opv"," first_digit",[35,384,385],{"class":305},"(x):\n",[35,387,389,392,394,397,400,404],{"class":37,"line":388},8,[35,390,391],{"class":305},"    x ",[35,393,341],{"class":301},[35,395,396],{"class":305}," x[x ",[35,398,399],{"class":301},">",[35,401,403],{"class":402},"sP0c6"," 0",[35,405,406],{"class":305},"]\n",[35,408,410,413,416,419,422,425,428,431],{"class":37,"line":409},9,[35,411,412],{"class":301},"    return",[35,414,415],{"class":305}," (x ",[35,417,418],{"class":301},"\u002F",[35,420,421],{"class":402}," 10",[35,423,424],{"class":301}," **",[35,426,427],{"class":305}," np.floor(np.log10(x))).astype(",[35,429,430],{"class":402},"int",[35,432,350],{"class":305},[35,434,436],{"class":37,"line":435},10,[35,437,332],{"emptyLinePlaceholder":331},[35,439,441,444,447,450,453,456,458,461,463,466,468,471],{"class":37,"line":440},11,[35,442,443],{"class":402},"print",[35,445,446],{"class":305},"(first_digit(pd.Series([",[35,448,449],{"class":402},"0.045",[35,451,452],{"class":305},", ",[35,454,455],{"class":402},"1250.00",[35,457,452],{"class":305},[35,459,460],{"class":402},"9.99",[35,462,452],{"class":305},[35,464,465],{"class":402},"10.0",[35,467,452],{"class":305},[35,469,470],{"class":402},"999_999.0",[35,472,473],{"class":305},"])).tolist())\n",[10,475,476,477,480,481,484],{},"Floating-point quirks can make ",[32,478,479],{},"x \u002F 10**k"," land a hair below an integer; ",[32,482,483],{},"astype(int)"," truncates, so for values like 1000.0 check the result is 1, as above.",[10,486,487],{},"Standard practice is to leave out small amounts, below 10, because they are dominated by fixed prices and rounding rather than by the processes Benford describes:",[25,489,491],{"className":292,"code":490,"language":294,"meta":30,"style":30},"tested = claims[amount >= 10].copy()\ntested[\"D1\"] = first_digit(tested[\"Amount\"].abs())\nprint(f\"{len(claims)} claims, {len(tested)} tested (amount >= 10)\")\n",[32,492,493,511,532],{"__ignoreMap":30},[35,494,495,498,500,503,506,508],{"class":37,"line":38},[35,496,497],{"class":305},"tested ",[35,499,341],{"class":301},[35,501,502],{"class":305}," claims[amount ",[35,504,505],{"class":301},">=",[35,507,421],{"class":402},[35,509,510],{"class":305},"].copy()\n",[35,512,513,516,519,522,524,527,529],{"class":37,"line":315},[35,514,515],{"class":305},"tested[",[35,517,518],{"class":45},"\"D1\"",[35,520,521],{"class":305},"] ",[35,523,341],{"class":301},[35,525,526],{"class":305}," first_digit(tested[",[35,528,364],{"class":45},[35,530,531],{"class":305},"].abs())\n",[35,533,534,536,539,542,545,549,552,555,558,561,563,565,568,570,573],{"class":37,"line":328},[35,535,443],{"class":402},[35,537,538],{"class":305},"(",[35,540,541],{"class":301},"f",[35,543,544],{"class":45},"\"",[35,546,548],{"class":547},"sSjpA","{",[35,550,551],{"class":402},"len",[35,553,554],{"class":305},"(claims)",[35,556,557],{"class":547},"}",[35,559,560],{"class":45}," claims, ",[35,562,548],{"class":547},[35,564,551],{"class":402},[35,566,567],{"class":305},"(tested)",[35,569,557],{"class":547},[35,571,572],{"class":45}," tested (amount >= 10)\"",[35,574,350],{"class":305},[20,576,578],{"id":577},"step-2-observed-against-expected","Step 2 — Observed against expected",[10,580,581],{},"Count the first digits, turn them into proportions, and set them beside the Benford expectation:",[25,583,585],{"className":292,"code":584,"language":294,"meta":30,"style":30},"digits = np.arange(1, 10)\nexpected = np.log10(1 + 1 \u002F digits)\nobserved = tested[\"D1\"].value_counts(normalize=True).reindex(digits, fill_value=0).to_numpy()\n\ntable = pd.DataFrame({\"Digit\": digits, \"Count\": tested[\"D1\"].value_counts().reindex(digits, fill_value=0).to_numpy(),\n                      \"Observed %\": observed * 100, \"Expected %\": expected * 100})\ntable[\"Difference\"] = table[\"Observed %\"] - table[\"Expected %\"]\nprint(table.round(2).to_string(index=False))\n",[32,586,587,606,630,667,671,707,736,765],{"__ignoreMap":30},[35,588,589,592,594,597,599,601,604],{"class":37,"line":38},[35,590,591],{"class":305},"digits ",[35,593,341],{"class":301},[35,595,596],{"class":305}," np.arange(",[35,598,163],{"class":402},[35,600,452],{"class":305},[35,602,603],{"class":402},"10",[35,605,350],{"class":305},[35,607,608,611,613,616,618,621,624,627],{"class":37,"line":315},[35,609,610],{"class":305},"expected ",[35,612,341],{"class":301},[35,614,615],{"class":305}," np.log10(",[35,617,163],{"class":402},[35,619,620],{"class":301}," +",[35,622,623],{"class":402}," 1",[35,625,626],{"class":301}," \u002F",[35,628,629],{"class":305}," digits)\n",[35,631,632,635,637,640,642,645,649,651,654,657,660,662,664],{"class":37,"line":328},[35,633,634],{"class":305},"observed ",[35,636,341],{"class":301},[35,638,639],{"class":305}," tested[",[35,641,518],{"class":45},[35,643,644],{"class":305},"].value_counts(",[35,646,648],{"class":647},"sa561","normalize",[35,650,341],{"class":301},[35,652,653],{"class":402},"True",[35,655,656],{"class":305},").reindex(digits, ",[35,658,659],{"class":647},"fill_value",[35,661,341],{"class":301},[35,663,107],{"class":402},[35,665,666],{"class":305},").to_numpy()\n",[35,668,669],{"class":37,"line":335},[35,670,332],{"emptyLinePlaceholder":331},[35,672,673,676,678,681,684,687,690,693,695,698,700,702,704],{"class":37,"line":353},[35,674,675],{"class":305},"table ",[35,677,341],{"class":301},[35,679,680],{"class":305}," pd.DataFrame({",[35,682,683],{"class":45},"\"Digit\"",[35,685,686],{"class":305},": digits, ",[35,688,689],{"class":45},"\"Count\"",[35,691,692],{"class":305},": tested[",[35,694,518],{"class":45},[35,696,697],{"class":305},"].value_counts().reindex(digits, ",[35,699,659],{"class":647},[35,701,341],{"class":301},[35,703,107],{"class":402},[35,705,706],{"class":305},").to_numpy(),\n",[35,708,709,712,715,718,721,723,726,729,731,733],{"class":37,"line":370},[35,710,711],{"class":45},"                      \"Observed %\"",[35,713,714],{"class":305},": observed ",[35,716,717],{"class":301},"*",[35,719,720],{"class":402}," 100",[35,722,452],{"class":305},[35,724,725],{"class":45},"\"Expected %\"",[35,727,728],{"class":305},": expected ",[35,730,717],{"class":301},[35,732,720],{"class":402},[35,734,735],{"class":305},"})\n",[35,737,738,741,744,746,748,751,754,756,759,761,763],{"class":37,"line":375},[35,739,740],{"class":305},"table[",[35,742,743],{"class":45},"\"Difference\"",[35,745,521],{"class":305},[35,747,341],{"class":301},[35,749,750],{"class":305}," table[",[35,752,753],{"class":45},"\"Observed %\"",[35,755,521],{"class":305},[35,757,758],{"class":301},"-",[35,760,750],{"class":305},[35,762,725],{"class":45},[35,764,406],{"class":305},[35,766,767,769,772,774,777,780,782,785],{"class":37,"line":388},[35,768,443],{"class":402},[35,770,771],{"class":305},"(table.round(",[35,773,167],{"class":402},[35,775,776],{"class":305},").to_string(",[35,778,779],{"class":647},"index",[35,781,341],{"class":301},[35,783,784],{"class":402},"False",[35,786,787],{"class":305},"))\n",[20,789,791],{"id":790},"step-3-measure-conformity","Step 3 — Measure conformity",[10,793,794],{},"Two measures are used. The mean absolute deviation (MAD) is the average absolute gap between observed and expected proportions; Mark Nigrini's widely used thresholds for the first-digit test are below 0.006 for close conformity, up to 0.012 acceptable, up to 0.015 marginal, and above that nonconformity. The chi-square test gives a p-value, but with thousands of rows it flags even trivial differences, which is why MAD is preferred for large ledgers:",[25,796,798],{"className":292,"code":797,"language":294,"meta":30,"style":30},"from scipy.stats import chisquare\n\nmad = np.abs(observed - expected).mean()\nverdict = (\"close conformity\" if mad \u003C 0.006 else \"acceptable\" if mad \u003C 0.012\n           else \"marginal\" if mad \u003C 0.015 else \"nonconformity\")\ncounts = table[\"Count\"].to_numpy()\nchi2, p = chisquare(counts, f_exp=expected * counts.sum())\nprint(f\"MAD {mad:.4f} ({verdict}); chi-square {chi2:.1f}, p = {p:.2g}\")\n",[32,799,800,813,817,832,872,896,910,932],{"__ignoreMap":30},[35,801,802,805,808,810],{"class":37,"line":38},[35,803,804],{"class":301},"from",[35,806,807],{"class":305}," scipy.stats ",[35,809,302],{"class":301},[35,811,812],{"class":305}," chisquare\n",[35,814,815],{"class":37,"line":315},[35,816,332],{"emptyLinePlaceholder":331},[35,818,819,822,824,827,829],{"class":37,"line":328},[35,820,821],{"class":305},"mad ",[35,823,341],{"class":301},[35,825,826],{"class":305}," np.abs(observed ",[35,828,758],{"class":301},[35,830,831],{"class":305}," expected).mean()\n",[35,833,834,837,839,842,845,848,851,854,857,860,863,865,867,869],{"class":37,"line":335},[35,835,836],{"class":305},"verdict ",[35,838,341],{"class":301},[35,840,841],{"class":305}," (",[35,843,844],{"class":45},"\"close conformity\"",[35,846,847],{"class":301}," if",[35,849,850],{"class":305}," mad ",[35,852,853],{"class":301},"\u003C",[35,855,856],{"class":402}," 0.006",[35,858,859],{"class":301}," else",[35,861,862],{"class":45}," \"acceptable\"",[35,864,847],{"class":301},[35,866,850],{"class":305},[35,868,853],{"class":301},[35,870,871],{"class":402}," 0.012\n",[35,873,874,877,880,882,884,886,889,891,894],{"class":37,"line":353},[35,875,876],{"class":301},"           else",[35,878,879],{"class":45}," \"marginal\"",[35,881,847],{"class":301},[35,883,850],{"class":305},[35,885,853],{"class":301},[35,887,888],{"class":402}," 0.015",[35,890,859],{"class":301},[35,892,893],{"class":45}," \"nonconformity\"",[35,895,350],{"class":305},[35,897,898,901,903,905,907],{"class":37,"line":370},[35,899,900],{"class":305},"counts ",[35,902,341],{"class":301},[35,904,750],{"class":305},[35,906,689],{"class":45},[35,908,909],{"class":305},"].to_numpy()\n",[35,911,912,915,917,920,923,925,927,929],{"class":37,"line":375},[35,913,914],{"class":305},"chi2, p ",[35,916,341],{"class":301},[35,918,919],{"class":305}," chisquare(counts, ",[35,921,922],{"class":647},"f_exp",[35,924,341],{"class":301},[35,926,610],{"class":305},[35,928,717],{"class":301},[35,930,931],{"class":305}," counts.sum())\n",[35,933,934,936,938,940,943,945,948,951,953,955,957,960,962,965,967,970,973,975,978,980,982,985,987,989],{"class":37,"line":388},[35,935,443],{"class":402},[35,937,538],{"class":305},[35,939,541],{"class":301},[35,941,942],{"class":45},"\"MAD ",[35,944,548],{"class":547},[35,946,947],{"class":305},"mad",[35,949,950],{"class":301},":.4f",[35,952,557],{"class":547},[35,954,841],{"class":45},[35,956,548],{"class":547},[35,958,959],{"class":305},"verdict",[35,961,557],{"class":547},[35,963,964],{"class":45},"); chi-square ",[35,966,548],{"class":547},[35,968,969],{"class":305},"chi2",[35,971,972],{"class":301},":.1f",[35,974,557],{"class":547},[35,976,977],{"class":45},", p = ",[35,979,548],{"class":547},[35,981,10],{"class":305},[35,983,984],{"class":301},":.2g",[35,986,557],{"class":547},[35,988,544],{"class":45},[35,990,350],{"class":305},[10,992,993],{},"The two measures disagree here, and the disagreement is the lesson. By MAD the ledger is in close conformity: averaged over nine digits, the deviations are small. The chi-square test, with over 4,000 claims, says the pattern is very unlikely to be chance (p ≈ 0.005). Look at the table instead of either number: eight digits sit within about one percentage point of Benford, and one, digit 4, is two points high, about 85 more claims than expected. A targeted manipulation affects a narrow band of amounts, and an average over all digits dilutes it. Always read the per-digit differences, not just the summary statistic.",[85,995,94,1000,94,1003,94,1006,94,1009,94,1014,94,1020,94,1024,94,1028,94,1031,94,1034,94,1037,94,1040,94,1043,94,1046,94,1049,94,1052,94,1055,94,1058,94,1060,94,1063,94,1065,94,1067,94,1069,94,1071,94,1073,94,1075,94,1077,94,1079,94,1085,94,1090,94,1095,94,1099,94,1104,94,1109,94,1114,94,1119,94,1124,94,1128,94,1132,94,1136,94,1140,94,1144,94,1148,94,1152,94,1156],{"viewBox":996,"role":88,"ariaLabelledBy":997,"xmlns":92,"style":93},"4 0 748 280",[998,999],"bf-observed-t","bf-observed-d",[96,1001,1002],{"id":998},"Observed first digits against Benford",[100,1004,1005],{"id":999},"A column chart comparing the observed share of each first digit in the expense claims with the Benford expectation. The two agree closely for every digit except 4, where 11.7 percent of claims start with 4 against an expected 9.7 percent.",[104,1007],{"x":106,"y":107,"width":108,"height":1008,"fill":110},"280",[104,1010],{"x":1011,"y":1012,"width":1013,"height":1013,"rx":171,"fill":201,"stroke":202,"style":157},"64.0","18.0","14.0",[119,1015,1019],{"x":1016,"y":1017,"style":1018},"84.0","29.5","font-size:12px;font-weight:600;fill:var(--text,#172033);text-anchor:start","Observed",[104,1021],{"x":1022,"y":1012,"width":1013,"height":1013,"rx":171,"fill":1023,"stroke":116,"style":157},"172.4","#e7ebef",[119,1025,1027],{"x":1026,"y":1017,"style":1018},"192.4","Expected",[37,1029],{"x1":113,"y1":1030,"x2":115,"y2":1030,"stroke":116,"style":117},"242.0",[119,1032,124],{"x":121,"y":1033,"style":123},"246.0",[37,1035],{"x1":113,"y1":1036,"x2":115,"y2":1036,"stroke":116,"style":128},"194.5",[119,1038,132],{"x":121,"y":1039,"style":123},"198.5",[37,1041],{"x1":113,"y1":1042,"x2":115,"y2":1042,"stroke":116,"style":128},"147.0",[119,1044,139],{"x":121,"y":1045,"style":123},"151.0",[37,1047],{"x1":113,"y1":1048,"x2":115,"y2":1048,"stroke":116,"style":128},"99.5",[119,1050,146],{"x":121,"y":1051,"style":123},"103.5",[37,1053],{"x1":113,"y1":1054,"x2":115,"y2":1054,"stroke":116,"style":128},"52.0",[119,1056,153],{"x":121,"y":1057,"style":123},"56.0",[37,1059],{"x1":113,"y1":1030,"x2":115,"y2":1030,"stroke":156,"style":157},[119,1061,163],{"x":160,"y":1062,"style":162},"260.0",[119,1064,167],{"x":166,"y":1062,"style":162},[119,1066,171],{"x":170,"y":1062,"style":162},[119,1068,106],{"x":174,"y":1062,"style":162},[119,1070,178],{"x":177,"y":1062,"style":162},[119,1072,182],{"x":181,"y":1062,"style":162},[119,1074,186],{"x":185,"y":1062,"style":162},[119,1076,190],{"x":189,"y":1062,"style":162},[119,1078,194],{"x":193,"y":1062,"style":162},[104,1080],{"x":1081,"y":1082,"width":1083,"height":1084,"rx":171,"fill":201,"stroke":202,"style":157},"70.4","104.2","24.4","137.8",[104,1086],{"x":1087,"y":1088,"width":1083,"height":1089,"rx":171,"fill":201,"stroke":202,"style":157},"145.7","157.9","84.1",[104,1091],{"x":1092,"y":1093,"width":1083,"height":1094,"rx":171,"fill":201,"stroke":202,"style":157},"221.0","181.7","60.3",[104,1096],{"x":1097,"y":255,"width":1083,"height":1098,"rx":171,"fill":201,"stroke":202,"style":157},"296.4","55.6",[104,1100],{"x":1101,"y":1102,"width":1083,"height":1103,"rx":171,"fill":201,"stroke":202,"style":157},"371.7","204.0","38.0",[104,1105],{"x":1106,"y":1107,"width":1083,"height":1108,"rx":171,"fill":201,"stroke":202,"style":157},"447.0","212.6","29.4",[104,1110],{"x":1111,"y":1112,"width":1083,"height":1113,"rx":171,"fill":201,"stroke":202,"style":157},"522.3","215.9","26.1",[104,1115],{"x":1116,"y":1117,"width":1083,"height":1118,"rx":171,"fill":201,"stroke":202,"style":157},"597.7","218.7","23.3",[104,1120],{"x":1121,"y":1122,"width":1083,"height":1123,"rx":171,"fill":201,"stroke":202,"style":157},"673.0","221.6","20.4",[104,1125],{"x":1126,"y":1127,"width":1083,"height":200,"rx":171,"fill":1023,"stroke":116,"style":157},"96.8","99.0",[104,1129],{"x":1130,"y":1131,"width":1083,"height":212,"rx":171,"fill":1023,"stroke":116,"style":157},"172.1","158.4",[104,1133],{"x":1134,"y":1135,"width":1083,"height":221,"rx":171,"fill":1023,"stroke":116,"style":157},"247.4","182.6",[104,1137],{"x":1138,"y":1139,"width":1083,"height":230,"rx":171,"fill":1023,"stroke":116,"style":157},"322.7","195.9",[104,1141],{"x":1142,"y":1143,"width":1083,"height":239,"rx":171,"fill":1023,"stroke":116,"style":157},"398.0","204.5",[104,1145],{"x":1146,"y":1147,"width":1083,"height":247,"rx":171,"fill":1023,"stroke":116,"style":157},"473.4","210.2",[104,1149],{"x":1150,"y":1151,"width":1083,"height":256,"rx":171,"fill":1023,"stroke":116,"style":157},"548.7","214.4",[104,1153],{"x":1154,"y":1155,"width":1083,"height":265,"rx":171,"fill":1023,"stroke":116,"style":157},"624.0","217.8",[104,1157],{"x":1158,"y":1159,"width":1083,"height":274,"rx":171,"fill":1023,"stroke":116,"style":157},"699.3","220.2",[20,1161,1163],{"id":1162},"step-4-drill-into-the-digit-that-stands-out","Step 4 — Drill into the digit that stands out",[10,1165,1166,1167,1170],{},"A test result is only useful if it leads to transactions. Here digit 4 is over-represented; the next question is where, and a first-two-digits breakdown sharpens it. The two-digit test uses 90 bins, 10 to 99, with expected share log₁₀(1 + 1\u002F",[73,1168,1169],{},"dd","):",[25,1172,1174],{"className":292,"code":1173,"language":294,"meta":30,"style":30},"x = tested[\"Amount\"].abs()\ntested[\"D12\"] = (x \u002F 10 ** (np.floor(np.log10(x)) - 1)).astype(int)\ntwo = tested[\"D12\"].value_counts(normalize=True).rename(\"Observed\")\ntwo = two.to_frame().assign(Expected=np.log10(1 + 1 \u002F two.index))\ntwo[\"Excess\"] = (two[\"Observed\"] - two[\"Expected\"]) * 100\nprint(two.sort_values(\"Excess\", ascending=False).head(4).round(3).to_string())\n",[32,1175,1176,1189,1222,1249,1276,1311],{"__ignoreMap":30},[35,1177,1178,1181,1183,1185,1187],{"class":37,"line":38},[35,1179,1180],{"class":305},"x ",[35,1182,341],{"class":301},[35,1184,639],{"class":305},[35,1186,364],{"class":45},[35,1188,367],{"class":305},[35,1190,1191,1193,1196,1198,1200,1202,1204,1206,1208,1211,1213,1215,1218,1220],{"class":37,"line":315},[35,1192,515],{"class":305},[35,1194,1195],{"class":45},"\"D12\"",[35,1197,521],{"class":305},[35,1199,341],{"class":301},[35,1201,415],{"class":305},[35,1203,418],{"class":301},[35,1205,421],{"class":402},[35,1207,424],{"class":301},[35,1209,1210],{"class":305}," (np.floor(np.log10(x)) ",[35,1212,758],{"class":301},[35,1214,623],{"class":402},[35,1216,1217],{"class":305},")).astype(",[35,1219,430],{"class":402},[35,1221,350],{"class":305},[35,1223,1224,1227,1229,1231,1233,1235,1237,1239,1241,1244,1247],{"class":37,"line":328},[35,1225,1226],{"class":305},"two ",[35,1228,341],{"class":301},[35,1230,639],{"class":305},[35,1232,1195],{"class":45},[35,1234,644],{"class":305},[35,1236,648],{"class":647},[35,1238,341],{"class":301},[35,1240,653],{"class":402},[35,1242,1243],{"class":305},").rename(",[35,1245,1246],{"class":45},"\"Observed\"",[35,1248,350],{"class":305},[35,1250,1251,1253,1255,1258,1260,1262,1265,1267,1269,1271,1273],{"class":37,"line":335},[35,1252,1226],{"class":305},[35,1254,341],{"class":301},[35,1256,1257],{"class":305}," two.to_frame().assign(",[35,1259,1027],{"class":647},[35,1261,341],{"class":301},[35,1263,1264],{"class":305},"np.log10(",[35,1266,163],{"class":402},[35,1268,620],{"class":301},[35,1270,623],{"class":402},[35,1272,626],{"class":301},[35,1274,1275],{"class":305}," two.index))\n",[35,1277,1278,1281,1284,1286,1288,1291,1293,1295,1297,1300,1303,1306,1308],{"class":37,"line":353},[35,1279,1280],{"class":305},"two[",[35,1282,1283],{"class":45},"\"Excess\"",[35,1285,521],{"class":305},[35,1287,341],{"class":301},[35,1289,1290],{"class":305}," (two[",[35,1292,1246],{"class":45},[35,1294,521],{"class":305},[35,1296,758],{"class":301},[35,1298,1299],{"class":305}," two[",[35,1301,1302],{"class":45},"\"Expected\"",[35,1304,1305],{"class":305},"]) ",[35,1307,717],{"class":301},[35,1309,1310],{"class":402}," 100\n",[35,1312,1313,1315,1318,1320,1322,1325,1327,1329,1332,1334,1337,1339],{"class":37,"line":370},[35,1314,443],{"class":402},[35,1316,1317],{"class":305},"(two.sort_values(",[35,1319,1283],{"class":45},[35,1321,452],{"class":305},[35,1323,1324],{"class":647},"ascending",[35,1326,341],{"class":301},[35,1328,784],{"class":402},[35,1330,1331],{"class":305},").head(",[35,1333,106],{"class":402},[35,1335,1336],{"class":305},").round(",[35,1338,171],{"class":402},[35,1340,1341],{"class":305},").to_string())\n",[10,1343,1344],{},"The spike is at 47, 48 and 49: amounts from 4,700 to 4,999, just under the 5,000 approval limit. Now look at who submits them:",[25,1346,1348],{"className":292,"code":1347,"language":294,"meta":30,"style":30},"near_limit = tested[tested[\"Amount\"].between(4700, 4999.99)]\nprint(near_limit.groupby(\"Employee\").size().sort_values(ascending=False).head(5).to_string())\n",[32,1349,1350,1376],{"__ignoreMap":30},[35,1351,1352,1355,1357,1360,1362,1365,1368,1370,1373],{"class":37,"line":38},[35,1353,1354],{"class":305},"near_limit ",[35,1356,341],{"class":301},[35,1358,1359],{"class":305}," tested[tested[",[35,1361,364],{"class":45},[35,1363,1364],{"class":305},"].between(",[35,1366,1367],{"class":402},"4700",[35,1369,452],{"class":305},[35,1371,1372],{"class":402},"4999.99",[35,1374,1375],{"class":305},")]\n",[35,1377,1378,1380,1383,1386,1389,1391,1393,1395,1397,1399],{"class":37,"line":315},[35,1379,443],{"class":402},[35,1381,1382],{"class":305},"(near_limit.groupby(",[35,1384,1385],{"class":45},"\"Employee\"",[35,1387,1388],{"class":305},").size().sort_values(",[35,1390,1324],{"class":647},[35,1392,341],{"class":301},[35,1394,784],{"class":402},[35,1396,1331],{"class":305},[35,1398,178],{"class":402},[35,1400,1341],{"class":305},[85,1402,94,1408,94,1411,94,1414,94,1419,94,1424,94,1430,94,1434,94,1438,94,1441,94,1446,94,1448,94,1451,94,1454,94,1458,94,1460,94,1463,94,1466,94,1469,94,1471,94,1474,94,1477,94,1480,94,1482,94,1485,94,1488,94,1492,94,1496],{"viewBox":1403,"role":88,"ariaLabelledBy":1404,"xmlns":92,"style":1407},"242 0 275 200",[1405,1406],"bf-drill-t","bf-drill-d","width:100%;max-width:275px;height:auto;display:block;margin:1.5rem auto;font-family:Inter,ui-sans-serif,system-ui,sans-serif",[96,1409,1410],{"id":1405},"From a digit to the claims behind it",[100,1412,1413],{"id":1406},"A drill-down table. The first-digit test shows digit 4 over-represented. The two-digit test narrows it to 47, 48 and 49. Those are amounts from 4,700 to 4,999 just below the 5,000 approval limit, and three employees submitted most of them.",[104,1415],{"x":1416,"y":107,"width":1417,"height":1418,"fill":110},"242","275","200",[104,1420],{"x":1421,"y":1422,"width":1423,"height":152,"rx":107,"fill":201,"stroke":202,"style":157},"258.8","16.0","107.4",[119,1425,1429],{"x":1426,"y":1427,"style":1428},"312.5","35.0","font-size:11px;font-weight:700;fill:var(--brand-strong,#4338ca);text-anchor:middle","Test",[104,1431],{"x":1432,"y":1422,"width":1433,"height":152,"rx":107,"fill":201,"stroke":202,"style":157},"366.3","134.9",[119,1435,1437],{"x":1436,"y":1427,"style":1428},"433.7","Finding",[104,1439],{"x":1421,"y":1440,"width":1423,"height":152,"rx":107,"fill":110,"stroke":116,"style":117},"44.0",[119,1442,1445],{"x":1426,"y":1443,"style":1444},"63.0","font-size:11px;font-weight:400;fill:var(--text,#172033);text-anchor:middle","First digit",[104,1447],{"x":1432,"y":1440,"width":1433,"height":152,"rx":107,"fill":110,"stroke":116,"style":117},[119,1449,1450],{"x":1436,"y":1443,"style":1444},"4 over-represented",[104,1452],{"x":1421,"y":1453,"width":1423,"height":152,"rx":107,"fill":110,"stroke":116,"style":117},"72.0",[119,1455,1457],{"x":1426,"y":1456,"style":1444},"91.0","First two digits",[104,1459],{"x":1432,"y":1453,"width":1433,"height":152,"rx":107,"fill":110,"stroke":116,"style":117},[119,1461,1462],{"x":1436,"y":1456,"style":1444},"47, 48, 49 spike",[104,1464],{"x":1421,"y":1465,"width":1423,"height":152,"rx":107,"fill":110,"stroke":116,"style":117},"100.0",[119,1467,1468],{"x":1426,"y":135,"style":1444},"Amounts",[104,1470],{"x":1432,"y":1465,"width":1433,"height":152,"rx":107,"fill":110,"stroke":116,"style":117},[119,1472,1473],{"x":1436,"y":135,"style":1444},"4,700–4,999",[104,1475],{"x":1421,"y":1476,"width":1423,"height":152,"rx":107,"fill":110,"stroke":116,"style":117},"128.0",[119,1478,1479],{"x":1426,"y":1042,"style":1444},"Rule nearby",[104,1481],{"x":1432,"y":1476,"width":1433,"height":152,"rx":107,"fill":110,"stroke":116,"style":117},[119,1483,1484],{"x":1436,"y":1042,"style":1444},"approval at 5,000",[104,1486],{"x":1421,"y":1487,"width":1423,"height":152,"rx":107,"fill":110,"stroke":116,"style":117},"156.0",[119,1489,1491],{"x":1426,"y":1490,"style":1444},"175.0","Who",[104,1493],{"x":1432,"y":1487,"width":1433,"height":152,"rx":107,"fill":1494,"stroke":1495,"style":157},"#fce9e9","var(--danger,#dc2626)",[119,1497,1499],{"x":1436,"y":1490,"style":1498},"font-size:11px;font-weight:700;fill:var(--accent-ink,#be185d);text-anchor:middle","three employees",[10,1501,1502],{},"Three employees account for most of the claims just below the limit. That is not proof of anything — some roles legitimately buy equipment in that price range — but it is a specific, short list for someone to review, which is the whole purpose of the test.",[10,1504,1505,1506,1509,1510,1513],{},"Two companion tests are cheap to add. The ",[73,1507,1508],{},"duplicate amounts"," test lists the exact amounts that occur most often; a single figure repeated dozens of times is either a standard charge, which is fine, or a claim being resubmitted. The ",[73,1511,1512],{},"last two digits"," test checks rounding: invented numbers end in 00 or 50 far more often than real receipts do.",[25,1515,1517],{"className":292,"code":1516,"language":294,"meta":30,"style":30},"print(tested[\"Amount\"].value_counts().head(3).to_string())\ncents = (tested[\"Amount\"] * 100).round().astype(int) % 100\nprint(f\"amounts ending .00: {(cents == 0).mean():.1%} (about 1% expected for real receipts)\")\n",[32,1518,1519,1535,1566],{"__ignoreMap":30},[35,1520,1521,1523,1526,1528,1531,1533],{"class":37,"line":38},[35,1522,443],{"class":402},[35,1524,1525],{"class":305},"(tested[",[35,1527,364],{"class":45},[35,1529,1530],{"class":305},"].value_counts().head(",[35,1532,171],{"class":402},[35,1534,1341],{"class":305},[35,1536,1537,1540,1542,1545,1547,1549,1551,1553,1556,1558,1561,1564],{"class":37,"line":315},[35,1538,1539],{"class":305},"cents ",[35,1541,341],{"class":301},[35,1543,1544],{"class":305}," (tested[",[35,1546,364],{"class":45},[35,1548,521],{"class":305},[35,1550,717],{"class":301},[35,1552,720],{"class":402},[35,1554,1555],{"class":305},").round().astype(",[35,1557,430],{"class":402},[35,1559,1560],{"class":305},") ",[35,1562,1563],{"class":301},"%",[35,1565,1310],{"class":402},[35,1567,1568,1570,1572,1574,1577,1579,1582,1585,1587,1590,1593,1595,1598],{"class":37,"line":328},[35,1569,443],{"class":402},[35,1571,538],{"class":305},[35,1573,541],{"class":301},[35,1575,1576],{"class":45},"\"amounts ending .00: ",[35,1578,548],{"class":547},[35,1580,1581],{"class":305},"(cents ",[35,1583,1584],{"class":301},"==",[35,1586,403],{"class":402},[35,1588,1589],{"class":305},").mean()",[35,1591,1592],{"class":301},":.1%",[35,1594,557],{"class":547},[35,1596,1597],{"class":45}," (about 1% expected for real receipts)\"",[35,1599,350],{"class":305},[20,1601,1603],{"id":1602},"step-5-write-the-results-with-a-chart","Step 5 — Write the results with a chart",[10,1605,1606],{},"The workbook holds the digit table with a column chart of observed against expected, the two-digit table, and the claims behind the finding:",[25,1608,1610],{"className":292,"code":1609,"language":294,"meta":30,"style":30},"with pd.ExcelWriter(\"benford_check.xlsx\", engine=\"xlsxwriter\") as xw:\n    out = table.assign(**{\"Observed %\": table[\"Observed %\"] \u002F 100, \"Expected %\": table[\"Expected %\"] \u002F 100,\n                          \"Difference\": table[\"Difference\"] \u002F 100})\n    out.to_excel(xw, sheet_name=\"First digit\", index=False)\n    ws = xw.sheets[\"First digit\"]\n    pct = xw.book.add_format({\"num_format\": \"0.0%\"})\n    ws.set_column(2, 4, 12, pct)\n    ws.write(11, 0, f\"MAD {mad:.4f}: {verdict}\")\n    chart = xw.book.add_chart({\"type\": \"column\"})\n    chart.add_series({\"name\": \"Observed\", \"categories\": [\"First digit\", 1, 0, 9, 0], \"values\": [\"First digit\", 1, 2, 9, 2]})\n    line = xw.book.add_chart({\"type\": \"line\"})\n    line.add_series({\"name\": \"Benford expected\", \"categories\": [\"First digit\", 1, 0, 9, 0],\n                     \"values\": [\"First digit\", 1, 3, 9, 3], \"marker\": {\"type\": \"circle\"}})\n    chart.combine(line)\n    chart.set_y_axis({\"num_format\": \"0%\"})\n    chart.set_title({\"name\": \"First digits against Benford's law\"})\n    ws.insert_chart(\"G2\", chart)\n    two.sort_values(\"Excess\", ascending=False).round(4).to_excel(xw, sheet_name=\"First two digits\")\n    near_limit.sort_values([\"Employee\", \"Amount\"]).to_excel(xw, sheet_name=\"Claims 4,700-4,999\", index=False)\nprint(\"written\")\n",[32,1611,1612,1640,1685,1702,1725,1739,1760,1779,1817,1837,1904,1922,1962,2006,2012,2027,2042,2054,2086,2118],{"__ignoreMap":30},[35,1613,1614,1617,1620,1623,1625,1628,1630,1633,1635,1637],{"class":37,"line":38},[35,1615,1616],{"class":301},"with",[35,1618,1619],{"class":305}," pd.ExcelWriter(",[35,1621,1622],{"class":45},"\"benford_check.xlsx\"",[35,1624,452],{"class":305},[35,1626,1627],{"class":647},"engine",[35,1629,341],{"class":301},[35,1631,1632],{"class":45},"\"xlsxwriter\"",[35,1634,1560],{"class":305},[35,1636,309],{"class":301},[35,1638,1639],{"class":305}," xw:\n",[35,1641,1642,1645,1647,1650,1653,1655,1657,1660,1662,1664,1666,1668,1670,1672,1674,1676,1678,1680,1682],{"class":37,"line":315},[35,1643,1644],{"class":305},"    out ",[35,1646,341],{"class":301},[35,1648,1649],{"class":305}," table.assign(",[35,1651,1652],{"class":301},"**",[35,1654,548],{"class":305},[35,1656,753],{"class":45},[35,1658,1659],{"class":305},": table[",[35,1661,753],{"class":45},[35,1663,521],{"class":305},[35,1665,418],{"class":301},[35,1667,720],{"class":402},[35,1669,452],{"class":305},[35,1671,725],{"class":45},[35,1673,1659],{"class":305},[35,1675,725],{"class":45},[35,1677,521],{"class":305},[35,1679,418],{"class":301},[35,1681,720],{"class":402},[35,1683,1684],{"class":305},",\n",[35,1686,1687,1690,1692,1694,1696,1698,1700],{"class":37,"line":328},[35,1688,1689],{"class":45},"                          \"Difference\"",[35,1691,1659],{"class":305},[35,1693,743],{"class":45},[35,1695,521],{"class":305},[35,1697,418],{"class":301},[35,1699,720],{"class":402},[35,1701,735],{"class":305},[35,1703,1704,1707,1710,1712,1715,1717,1719,1721,1723],{"class":37,"line":335},[35,1705,1706],{"class":305},"    out.to_excel(xw, ",[35,1708,1709],{"class":647},"sheet_name",[35,1711,341],{"class":301},[35,1713,1714],{"class":45},"\"First digit\"",[35,1716,452],{"class":305},[35,1718,779],{"class":647},[35,1720,341],{"class":301},[35,1722,784],{"class":402},[35,1724,350],{"class":305},[35,1726,1727,1730,1732,1735,1737],{"class":37,"line":353},[35,1728,1729],{"class":305},"    ws ",[35,1731,341],{"class":301},[35,1733,1734],{"class":305}," xw.sheets[",[35,1736,1714],{"class":45},[35,1738,406],{"class":305},[35,1740,1741,1744,1746,1749,1752,1755,1758],{"class":37,"line":370},[35,1742,1743],{"class":305},"    pct ",[35,1745,341],{"class":301},[35,1747,1748],{"class":305}," xw.book.add_format({",[35,1750,1751],{"class":45},"\"num_format\"",[35,1753,1754],{"class":305},": ",[35,1756,1757],{"class":45},"\"0.0%\"",[35,1759,735],{"class":305},[35,1761,1762,1765,1767,1769,1771,1773,1776],{"class":37,"line":375},[35,1763,1764],{"class":305},"    ws.set_column(",[35,1766,167],{"class":402},[35,1768,452],{"class":305},[35,1770,106],{"class":402},[35,1772,452],{"class":305},[35,1774,1775],{"class":402},"12",[35,1777,1778],{"class":305},", pct)\n",[35,1780,1781,1784,1787,1789,1791,1793,1795,1797,1799,1801,1803,1805,1807,1809,1811,1813,1815],{"class":37,"line":388},[35,1782,1783],{"class":305},"    ws.write(",[35,1785,1786],{"class":402},"11",[35,1788,452],{"class":305},[35,1790,107],{"class":402},[35,1792,452],{"class":305},[35,1794,541],{"class":301},[35,1796,942],{"class":45},[35,1798,548],{"class":547},[35,1800,947],{"class":305},[35,1802,950],{"class":301},[35,1804,557],{"class":547},[35,1806,1754],{"class":45},[35,1808,548],{"class":547},[35,1810,959],{"class":305},[35,1812,557],{"class":547},[35,1814,544],{"class":45},[35,1816,350],{"class":305},[35,1818,1819,1822,1824,1827,1830,1832,1835],{"class":37,"line":409},[35,1820,1821],{"class":305},"    chart ",[35,1823,341],{"class":301},[35,1825,1826],{"class":305}," xw.book.add_chart({",[35,1828,1829],{"class":45},"\"type\"",[35,1831,1754],{"class":305},[35,1833,1834],{"class":45},"\"column\"",[35,1836,735],{"class":305},[35,1838,1839,1842,1845,1847,1849,1851,1854,1857,1859,1861,1863,1865,1867,1869,1871,1873,1875,1878,1881,1883,1885,1887,1889,1891,1893,1895,1897,1899,1901],{"class":37,"line":435},[35,1840,1841],{"class":305},"    chart.add_series({",[35,1843,1844],{"class":45},"\"name\"",[35,1846,1754],{"class":305},[35,1848,1246],{"class":45},[35,1850,452],{"class":305},[35,1852,1853],{"class":45},"\"categories\"",[35,1855,1856],{"class":305},": [",[35,1858,1714],{"class":45},[35,1860,452],{"class":305},[35,1862,163],{"class":402},[35,1864,452],{"class":305},[35,1866,107],{"class":402},[35,1868,452],{"class":305},[35,1870,194],{"class":402},[35,1872,452],{"class":305},[35,1874,107],{"class":402},[35,1876,1877],{"class":305},"], ",[35,1879,1880],{"class":45},"\"values\"",[35,1882,1856],{"class":305},[35,1884,1714],{"class":45},[35,1886,452],{"class":305},[35,1888,163],{"class":402},[35,1890,452],{"class":305},[35,1892,167],{"class":402},[35,1894,452],{"class":305},[35,1896,194],{"class":402},[35,1898,452],{"class":305},[35,1900,167],{"class":402},[35,1902,1903],{"class":305},"]})\n",[35,1905,1906,1909,1911,1913,1915,1917,1920],{"class":37,"line":440},[35,1907,1908],{"class":305},"    line ",[35,1910,341],{"class":301},[35,1912,1826],{"class":305},[35,1914,1829],{"class":45},[35,1916,1754],{"class":305},[35,1918,1919],{"class":45},"\"line\"",[35,1921,735],{"class":305},[35,1923,1925,1928,1930,1932,1935,1937,1939,1941,1943,1945,1947,1949,1951,1953,1955,1957,1959],{"class":37,"line":1924},12,[35,1926,1927],{"class":305},"    line.add_series({",[35,1929,1844],{"class":45},[35,1931,1754],{"class":305},[35,1933,1934],{"class":45},"\"Benford expected\"",[35,1936,452],{"class":305},[35,1938,1853],{"class":45},[35,1940,1856],{"class":305},[35,1942,1714],{"class":45},[35,1944,452],{"class":305},[35,1946,163],{"class":402},[35,1948,452],{"class":305},[35,1950,107],{"class":402},[35,1952,452],{"class":305},[35,1954,194],{"class":402},[35,1956,452],{"class":305},[35,1958,107],{"class":402},[35,1960,1961],{"class":305},"],\n",[35,1963,1965,1968,1970,1972,1974,1976,1978,1980,1982,1984,1986,1988,1990,1993,1996,1998,2000,2003],{"class":37,"line":1964},13,[35,1966,1967],{"class":45},"                     \"values\"",[35,1969,1856],{"class":305},[35,1971,1714],{"class":45},[35,1973,452],{"class":305},[35,1975,163],{"class":402},[35,1977,452],{"class":305},[35,1979,171],{"class":402},[35,1981,452],{"class":305},[35,1983,194],{"class":402},[35,1985,452],{"class":305},[35,1987,171],{"class":402},[35,1989,1877],{"class":305},[35,1991,1992],{"class":45},"\"marker\"",[35,1994,1995],{"class":305},": {",[35,1997,1829],{"class":45},[35,1999,1754],{"class":305},[35,2001,2002],{"class":45},"\"circle\"",[35,2004,2005],{"class":305},"}})\n",[35,2007,2009],{"class":37,"line":2008},14,[35,2010,2011],{"class":305},"    chart.combine(line)\n",[35,2013,2015,2018,2020,2022,2025],{"class":37,"line":2014},15,[35,2016,2017],{"class":305},"    chart.set_y_axis({",[35,2019,1751],{"class":45},[35,2021,1754],{"class":305},[35,2023,2024],{"class":45},"\"0%\"",[35,2026,735],{"class":305},[35,2028,2030,2033,2035,2037,2040],{"class":37,"line":2029},16,[35,2031,2032],{"class":305},"    chart.set_title({",[35,2034,1844],{"class":45},[35,2036,1754],{"class":305},[35,2038,2039],{"class":45},"\"First digits against Benford's law\"",[35,2041,735],{"class":305},[35,2043,2045,2048,2051],{"class":37,"line":2044},17,[35,2046,2047],{"class":305},"    ws.insert_chart(",[35,2049,2050],{"class":45},"\"G2\"",[35,2052,2053],{"class":305},", chart)\n",[35,2055,2057,2060,2062,2064,2066,2068,2070,2072,2074,2077,2079,2081,2084],{"class":37,"line":2056},18,[35,2058,2059],{"class":305},"    two.sort_values(",[35,2061,1283],{"class":45},[35,2063,452],{"class":305},[35,2065,1324],{"class":647},[35,2067,341],{"class":301},[35,2069,784],{"class":402},[35,2071,1336],{"class":305},[35,2073,106],{"class":402},[35,2075,2076],{"class":305},").to_excel(xw, ",[35,2078,1709],{"class":647},[35,2080,341],{"class":301},[35,2082,2083],{"class":45},"\"First two digits\"",[35,2085,350],{"class":305},[35,2087,2089,2092,2094,2096,2098,2101,2103,2105,2108,2110,2112,2114,2116],{"class":37,"line":2088},19,[35,2090,2091],{"class":305},"    near_limit.sort_values([",[35,2093,1385],{"class":45},[35,2095,452],{"class":305},[35,2097,364],{"class":45},[35,2099,2100],{"class":305},"]).to_excel(xw, ",[35,2102,1709],{"class":647},[35,2104,341],{"class":301},[35,2106,2107],{"class":45},"\"Claims 4,700-4,999\"",[35,2109,452],{"class":305},[35,2111,779],{"class":647},[35,2113,341],{"class":301},[35,2115,784],{"class":402},[35,2117,350],{"class":305},[35,2119,2121,2123,2125,2128],{"class":37,"line":2120},20,[35,2122,443],{"class":402},[35,2124,538],{"class":305},[35,2126,2127],{"class":45},"\"written\"",[35,2129,350],{"class":305},[10,2131,2132],{},"Showing the expected values as a line over observed columns makes a spike obvious without any statistics.",[20,2134,2136],{"id":2135},"common-pitfalls","Common pitfalls",[2138,2139,2140,2148,2154,2160,2166,2172],"ul",{},[2141,2142,2143,2147],"li",{},[2144,2145,2146],"strong",{},"Taking the first character of a string."," It fails for decimals below 1 and negative numbers; use the logarithm.",[2141,2149,2150,2153],{},[2144,2151,2152],{},"Testing data Benford does not describe."," Narrow ranges, assigned numbers and price lists will \"fail\" for innocent reasons.",[2141,2155,2156,2159],{},[2144,2157,2158],{},"Including small amounts."," Below 10, fixed prices dominate; exclude them and say so.",[2141,2161,2162,2165],{},[2144,2163,2164],{},"Relying on chi-square alone with big data."," Tiny differences become \"significant\"; use MAD for large ledgers.",[2141,2167,2168,2171],{},[2144,2169,2170],{},"Treating a result as proof."," Nonconformity says where to look. The explanation is in the transactions.",[2141,2173,2174,2177],{},[2144,2175,2176],{},"Too few rows."," With a few hundred values, random variation swamps the pattern; the test needs thousands.",[20,2179,2181],{"id":2180},"conclusion","Conclusion",[10,2183,2184],{},"Extract the first significant digit with logarithms after taking absolute values, exclude amounts below 10, and compare the observed digit shares with log₁₀(1 + 1\u002Fd). Measure conformity with the mean absolute deviation against Nigrini's thresholds, use chi-square with care on large data, then drill in with the first-two-digits test to specific amount ranges and the people or suppliers behind them. Write the tables, a chart of observed against expected and the transactions for review to Excel.",[20,2186,2188],{"id":2187},"frequently-asked-questions","Frequently asked questions",[10,2190,2191,2194],{},[2144,2192,2193],{},"What is Benford's law?","\nAn observation that in many real-world datasets the leading digit d occurs with probability log10(1 + 1\u002Fd): about 30% of values start with 1 and under 5% with 9. It applies to data spanning several orders of magnitude.",[10,2196,2197,2200],{},[2144,2198,2199],{},"How do I get the first digit of a number in pandas?","\nTake the absolute value, then divide by 10 raised to the floor of its base-10 logarithm and convert to int. That handles decimals below 1 and large numbers, which string slicing does not.",[10,2202,2203,2206],{},[2144,2204,2205],{},"What MAD value indicates nonconformity?","\nFor the first-digit test, Nigrini's thresholds are below 0.006 close conformity, 0.006 to 0.012 acceptable, 0.012 to 0.015 marginal, and above 0.015 nonconformity.",[10,2208,2209,2212],{},[2144,2210,2211],{},"Does failing a Benford test mean there is fraud?","\nNo. It means the data does not follow the expected pattern, which can have innocent causes such as a price list or a policy threshold. It identifies where to look more closely.",[20,2214,2216],{"id":2215},"related","Related",[2138,2218,2219,2225,2232,2239,2246],{},[2141,2220,2221,2222,2224],{},"Up one level: ",[14,2223,17],{"href":16}," — checks to run before trusting a sheet.",[2141,2226,2227,2231],{},[14,2228,2230],{"href":2229},"\u002Fadvanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002Ffind-gaps-in-a-numbered-sequence-with-pandas\u002F","Find Gaps in a Numbered Sequence with pandas"," — another audit test.",[2141,2233,2234,2238],{},[14,2235,2237],{"href":2236},"\u002Fadvanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002Fprofile-an-excel-dataset-with-pandas\u002F","Profile an Excel Dataset with pandas"," — a first look at any file.",[2141,2240,2241,2245],{},[14,2242,2244],{"href":2243},"\u002Fadvanced-data-transformation-and-cleaning\u002Fcleaning-excel-data-with-pandas\u002Fdetect-outliers-in-excel-data-with-pandas\u002F","Detect Outliers in Excel Data with pandas"," — unusual values, rather than unusual digits.",[2141,2247,2248,2252],{},[14,2249,2251],{"href":2250},"\u002Fformatting-and-charting-excel-reports-with-python\u002Fcreating-charts-in-excel-with-openpyxl\u002Fadd-a-combo-chart-with-a-secondary-axis-in-openpyxl\u002F","Add a Combo Chart with a Secondary Axis in openpyxl"," — column-and-line charts.",[2254,2255,2256],"style",{},"html pre.shiki code .sMTad, html code.shiki .sMTad{--shiki-default:#6F42C1;--shiki-dark:#FFB757}html pre.shiki code .srMev, html code.shiki .srMev{--shiki-default:#032F62;--shiki-dark:#ADDCFF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .s-kum, html code.shiki .s-kum{--shiki-default:#D73A49;--shiki-dark:#FF9492}html pre.shiki code .skGVy, html code.shiki .skGVy{--shiki-default:#24292E;--shiki-dark:#F0F3F6}html pre.shiki code .s_Opv, html code.shiki .s_Opv{--shiki-default:#6F42C1;--shiki-dark:#DBB7FF}html pre.shiki code .sP0c6, html code.shiki .sP0c6{--shiki-default:#005CC5;--shiki-dark:#91CBFF}html pre.shiki code .sSjpA, html code.shiki .sSjpA{--shiki-default:#005CC5;--shiki-dark:#FF9492}html pre.shiki code .sa561, html code.shiki .sa561{--shiki-default:#E36209;--shiki-dark:#FFB757}",{"title":30,"searchDepth":315,"depth":315,"links":2258},[2259,2260,2261,2262,2263,2264,2265,2266,2267,2268,2269],{"id":22,"depth":315,"text":23},{"id":67,"depth":315,"text":68},{"id":281,"depth":315,"text":282},{"id":577,"depth":315,"text":578},{"id":790,"depth":315,"text":791},{"id":1162,"depth":315,"text":1163},{"id":1602,"depth":315,"text":1603},{"id":2135,"depth":315,"text":2136},{"id":2180,"depth":315,"text":2181},{"id":2187,"depth":315,"text":2188},{"id":2215,"depth":315,"text":2216},"2026-09-29","Test whether the leading digits of amounts in an Excel ledger follow Benford's law: extract first digits correctly, compare observed and expected frequencies, measure conformity with the mean absolute deviation and a chi-square test, drill into over-represented digits such as amounts just below an approval limit, and chart the result in Excel.","md",[2274,2276,2278,2280],{"q":2193,"a":2275},"An observation that in many real-world datasets the leading digit d occurs with probability log10(1 + 1\u002Fd): about 30% of values start with 1 and under 5% with 9. It applies to data spanning several orders of magnitude.",{"q":2199,"a":2277},"Take the absolute value, then divide by 10 raised to the floor of its base-10 logarithm and convert to int. That handles decimals below 1 and large numbers, which string slicing does not.",{"q":2205,"a":2279},"For the first-digit test, Nigrini's thresholds are below 0.006 close conformity, 0.006 to 0.012 acceptable, 0.012 to 0.015 marginal, and above 0.015 nonconformity.",{"q":2211,"a":2281},"No. It means the data does not follow the expected pattern, which can have innocent causes such as a price list or a policy threshold. It identifies where to look more closely.","run a benfords law check on excel data",null,{"breadcrumb":2285},[2286,2288,2291],{"name":2287,"item":418},"Home",{"name":2289,"item":2290},"Advanced Data Transformation and Cleaning for Excel Automation","\u002Fadvanced-data-transformation-and-cleaning\u002F",{"name":17,"item":16},"\u002Fadvanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002Frun-a-benfords-law-check-on-excel-data",{"title":2294,"description":2295},"Benford's Law Test on Excel Data with Python (pandas)","Run a Benford's law first-digit test on Excel amounts with pandas: extract leading digits, expected log10(1+1\u002Fd) frequencies, MAD conformity thresholds, chi-square, first-two-digit test, drill-down, and an Excel chart.","run-a-benfords-law-check-on-excel-data","advanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002Frun-a-benfords-law-check-on-excel-data\u002Findex",[2299,2300],"data-validation","pandas","how-to","hcJdgHSMIT1_1nlLbqUg1xT0YrKb2SbkiYVBUZfOXms",[2304,2308],{"title":2305,"path":2306,"stem":2307,"children":-1},"Reconcile Excel Totals Against Control Totals with Python","\u002Fadvanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002Freconcile-excel-totals-with-control-totals","advanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002Freconcile-excel-totals-with-control-totals\u002Findex",{"title":2309,"path":2310,"stem":2311,"children":-1},"Validate Check Digits in Excel Data with Python","\u002Fadvanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002Fvalidate-check-digits-in-excel-data-with-python","advanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python\u002Fvalidate-check-digits-in-excel-data-with-python\u002Findex",[2313,2315],{"path":2314,"title":2289},"\u002Fadvanced-data-transformation-and-cleaning",{"path":2316,"title":17},"\u002Fadvanced-data-transformation-and-cleaning\u002Fvalidating-excel-data-with-python",[],[2319,2321,2324,2327,2330,2333],{"path":2320,"title":2244,"kind":2301},"\u002Fadvanced-data-transformation-and-cleaning\u002Fcleaning-excel-data-with-pandas\u002Fdetect-outliers-in-excel-data-with-pandas",{"path":2322,"title":2323,"kind":2301},"\u002Fadvanced-data-transformation-and-cleaning\u002Fapplying-conditional-formatting-with-openpyxl\u002Fhighlight-cells-containing-text-with-openpyxl","Highlight Cells Containing Specific Text with openpyxl",{"path":2325,"title":2326,"kind":2301},"\u002Fadvanced-data-transformation-and-cleaning\u002Fcleaning-excel-data-with-pandas\u002Ffind-near-duplicate-names-within-one-excel-column","Find Near-Duplicate Names Within One Excel Column",{"path":2328,"title":2329,"kind":2301},"\u002Fadvanced-data-transformation-and-cleaning\u002Fextracting-data-from-messy-excel-sheets\u002Fread-dropdown-lists-from-an-existing-workbook","Read Dropdown Lists and Validation Rules from an Existing Workbook",{"path":2331,"title":2332,"kind":2301},"\u002Fadvanced-data-transformation-and-cleaning\u002Fhandling-missing-data-in-excel-reports\u002Ffind-and-report-missing-values-in-an-excel-file","Find and Report Missing Values in an Excel File",{"path":2334,"title":2335,"kind":2301},"\u002Fadvanced-data-transformation-and-cleaning\u002Fhandling-missing-data-in-excel-reports\u002Fflag-rows-with-too-many-missing-values","Flag Rows with Too Many Missing Values",1790972100800]