[{"data":1,"prerenderedAt":2151},["ShallowReactive",2],{"doc:\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fcache-api-responses-when-building-excel-reports":3,"surround:\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fcache-api-responses-when-building-excel-reports":2121,"ancestors:\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fcache-api-responses-when-building-excel-reports":2129,"children:\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fcache-api-responses-when-building-excel-reports":2133,"related:\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fcache-api-responses-when-building-excel-reports":2134},{"id":4,"title":5,"body":6,"dateModified":2087,"datePublished":2087,"description":2088,"extension":2089,"faq":2090,"keyword":2099,"kind":2100,"meta":2101,"navigation":95,"path":2110,"seo":2111,"slug":2114,"stem":2115,"topics":2116,"type":2119,"__hash__":2120},"docs\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fcache-api-responses-when-building-excel-reports\u002Findex.md","Cache API Responses When Building Excel Reports",{"type":7,"value":8,"toc":2075},"minimark",[9,19,24,59,66,70,73,334,337,341,351,683,710,854,925,928,932,935,1044,1062,1066,1069,1487,1509,1513,1516,1866,1881,1956,1960,2000,2004,2010,2014,2020,2026,2032,2038,2042,2071],[10,11,12,13,18],"p",{},"A report that pulls its data from an API is slow to develop. Every time you change a column width or fix a label, the script fetches thousands of records again, waits on rate limits, and sometimes fails halfway because the service hiccuped. In production the same thing happens on a smaller scale: several reports call the same endpoint for the same data within minutes of each other. A cache solves both. During development it makes reruns instant and offline; in production it removes duplicate calls and makes reruns after a failure cheap. The danger is the obvious one — a report built from stale data without anyone knowing. This page adds a cache with explicit expiry rules, shows a hand-rolled version for requests that are not simple GETs, and makes the workbook state how old its data is. It belongs to ",[14,15,17],"a",{"href":16},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002F","Moving Data Between Excel and Databases",".",[20,21,23],"h2",{"id":22},"prerequisites","Prerequisites",[25,26,31],"pre",{"className":27,"code":28,"language":29,"meta":30,"style":30},"language-bash shiki shiki-themes github-light github-dark-high-contrast","pip install requests requests-cache pandas openpyxl\n","bash","",[32,33,34],"code",{"__ignoreMap":30},[35,36,39,43,47,50,53,56],"span",{"class":37,"line":38},"line",1,[35,40,42],{"class":41},"sMTad","pip",[35,44,46],{"class":45},"srMev"," install",[35,48,49],{"class":45}," requests",[35,51,52],{"class":45}," requests-cache",[35,54,55],{"class":45}," pandas",[35,57,58],{"class":45}," openpyxl\n",[10,60,61,62,65],{},"The examples call three endpoints of a sales API: exchange rates, monthly sales per region, and a stock query sent as a POST with a JSON body. They run against a small local stand-in for the API (address in ",[32,63,64],{},"BASE",") that counts how many requests actually reach it — the number that caching is meant to reduce. The examples were run with requests-cache 1.3.",[20,67,69],{"id":68},"step-1-measure-the-problem","Step 1 — Measure the problem",[10,71,72],{},"A typical report script calls the API in several places, often for the same data. Count the requests it makes:",[25,74,78],{"className":75,"code":76,"language":77,"meta":30,"style":30},"language-python shiki shiki-themes github-light github-dark-high-contrast","import requests\n\ndef build_report(session):\n    rates = session.get(f\"{BASE}\u002Frates\", timeout=30).json()[\"rates\"]\n    frames = {}\n    for region in [\"North\", \"South\", \"East\", \"North\"]:          # North requested twice\n        frames[region] = session.get(f\"{BASE}\u002Fsales\", params={\"region\": region}, timeout=30).json()[\"rows\"]\n    return rates, frames\n\nstart = hits[\"n\"]\nbuild_report(requests.Session())\nbuild_report(requests.Session())                                 # a rerun after changing a format\nprint(\"requests reaching the API:\", hits[\"n\"] - start)\n","python",[32,79,80,90,97,110,162,173,212,263,272,277,293,299,308],{"__ignoreMap":30},[35,81,82,86],{"class":37,"line":38},[35,83,85],{"class":84},"s-kum","import",[35,87,89],{"class":88},"skGVy"," requests\n",[35,91,93],{"class":37,"line":92},2,[35,94,96],{"emptyLinePlaceholder":95},true,"\n",[35,98,100,103,107],{"class":37,"line":99},3,[35,101,102],{"class":84},"def",[35,104,106],{"class":105},"s_Opv"," build_report",[35,108,109],{"class":88},"(session):\n",[35,111,113,116,119,122,125,128,132,135,138,141,144,148,150,153,156,159],{"class":37,"line":112},4,[35,114,115],{"class":88},"    rates ",[35,117,118],{"class":84},"=",[35,120,121],{"class":88}," session.get(",[35,123,124],{"class":84},"f",[35,126,127],{"class":45},"\"",[35,129,131],{"class":130},"sSjpA","{",[35,133,64],{"class":134},"sP0c6",[35,136,137],{"class":130},"}",[35,139,140],{"class":45},"\u002Frates\"",[35,142,143],{"class":88},", ",[35,145,147],{"class":146},"sa561","timeout",[35,149,118],{"class":84},[35,151,152],{"class":134},"30",[35,154,155],{"class":88},").json()[",[35,157,158],{"class":45},"\"rates\"",[35,160,161],{"class":88},"]\n",[35,163,165,168,170],{"class":37,"line":164},5,[35,166,167],{"class":88},"    frames ",[35,169,118],{"class":84},[35,171,172],{"class":88}," {}\n",[35,174,176,179,182,185,188,191,193,196,198,201,203,205,208],{"class":37,"line":175},6,[35,177,178],{"class":84},"    for",[35,180,181],{"class":88}," region ",[35,183,184],{"class":84},"in",[35,186,187],{"class":88}," [",[35,189,190],{"class":45},"\"North\"",[35,192,143],{"class":88},[35,194,195],{"class":45},"\"South\"",[35,197,143],{"class":88},[35,199,200],{"class":45},"\"East\"",[35,202,143],{"class":88},[35,204,190],{"class":45},[35,206,207],{"class":88},"]:          ",[35,209,211],{"class":210},"s-wDw","# North requested twice\n",[35,213,215,218,220,222,224,226,228,230,232,235,237,240,242,244,247,250,252,254,256,258,261],{"class":37,"line":214},7,[35,216,217],{"class":88},"        frames[region] ",[35,219,118],{"class":84},[35,221,121],{"class":88},[35,223,124],{"class":84},[35,225,127],{"class":45},[35,227,131],{"class":130},[35,229,64],{"class":134},[35,231,137],{"class":130},[35,233,234],{"class":45},"\u002Fsales\"",[35,236,143],{"class":88},[35,238,239],{"class":146},"params",[35,241,118],{"class":84},[35,243,131],{"class":88},[35,245,246],{"class":45},"\"region\"",[35,248,249],{"class":88},": region}, ",[35,251,147],{"class":146},[35,253,118],{"class":84},[35,255,152],{"class":134},[35,257,155],{"class":88},[35,259,260],{"class":45},"\"rows\"",[35,262,161],{"class":88},[35,264,266,269],{"class":37,"line":265},8,[35,267,268],{"class":84},"    return",[35,270,271],{"class":88}," rates, frames\n",[35,273,275],{"class":37,"line":274},9,[35,276,96],{"emptyLinePlaceholder":95},[35,278,280,283,285,288,291],{"class":37,"line":279},10,[35,281,282],{"class":88},"start ",[35,284,118],{"class":84},[35,286,287],{"class":88}," hits[",[35,289,290],{"class":45},"\"n\"",[35,292,161],{"class":88},[35,294,296],{"class":37,"line":295},11,[35,297,298],{"class":88},"build_report(requests.Session())\n",[35,300,302,305],{"class":37,"line":301},12,[35,303,304],{"class":88},"build_report(requests.Session())                                 ",[35,306,307],{"class":210},"# a rerun after changing a format\n",[35,309,311,314,317,320,323,325,328,331],{"class":37,"line":310},13,[35,312,313],{"class":134},"print",[35,315,316],{"class":88},"(",[35,318,319],{"class":45},"\"requests reaching the API:\"",[35,321,322],{"class":88},", hits[",[35,324,290],{"class":45},[35,326,327],{"class":88},"] ",[35,329,330],{"class":84},"-",[35,332,333],{"class":88}," start)\n",[10,335,336],{},"Two runs of a small script made ten requests, two of them duplicates within a single run. Scale that to hundreds of regions or products and a few dozen development reruns and the API sees thousands of calls — which means slow development, rate-limit errors, and, for paid APIs, real cost.",[20,338,340],{"id":339},"step-2-a-transparent-cache-with-requests-cache","Step 2 — A transparent cache with requests-cache",[10,342,343,346,347,350],{},[32,344,345],{},"requests_cache.CachedSession"," is a drop-in replacement for ",[32,348,349],{},"requests.Session"," that stores responses in a local SQLite file and answers repeat requests from it. Expiry can be set per URL pattern:",[25,352,354],{"className":75,"code":353,"language":77,"meta":30,"style":30},"import requests_cache\nfrom datetime import timedelta\n\ncached = requests_cache.CachedSession(\n    \"report_cache\",                              # report_cache.sqlite next to the script\n    expire_after=timedelta(hours=1),             # default for everything\n    urls_expire_after={\n        f\"{BASE}\u002Frates\": timedelta(days=1),      # rates change daily\n        f\"{BASE}\u002Fsales*\": timedelta(minutes=15),\n    },\n    allowable_codes=(200,),                      # never cache errors\n    allowable_methods=(\"GET\",),\n)\ncached.cache.clear()\n\nstart = hits[\"n\"]\nbuild_report(cached)\nfirst = hits[\"n\"] - start\nbuild_report(cached)\nprint(\"first run:\", first, \"requests | second run:\", hits[\"n\"] - start - first, \"requests\")\nr = cached.get(f\"{BASE}\u002Frates\")\nprint(\"from cache:\", r.from_cache, \"| expires:\", r.expires is not None)\n",[32,355,356,363,376,380,390,401,425,435,466,494,499,517,532,537,543,548,561,567,586,591,628,653],{"__ignoreMap":30},[35,357,358,360],{"class":37,"line":38},[35,359,85],{"class":84},[35,361,362],{"class":88}," requests_cache\n",[35,364,365,368,371,373],{"class":37,"line":92},[35,366,367],{"class":84},"from",[35,369,370],{"class":88}," datetime ",[35,372,85],{"class":84},[35,374,375],{"class":88}," timedelta\n",[35,377,378],{"class":37,"line":99},[35,379,96],{"emptyLinePlaceholder":95},[35,381,382,385,387],{"class":37,"line":112},[35,383,384],{"class":88},"cached ",[35,386,118],{"class":84},[35,388,389],{"class":88}," requests_cache.CachedSession(\n",[35,391,392,395,398],{"class":37,"line":164},[35,393,394],{"class":45},"    \"report_cache\"",[35,396,397],{"class":88},",                              ",[35,399,400],{"class":210},"# report_cache.sqlite next to the script\n",[35,402,403,406,408,411,414,416,419,422],{"class":37,"line":175},[35,404,405],{"class":146},"    expire_after",[35,407,118],{"class":84},[35,409,410],{"class":88},"timedelta(",[35,412,413],{"class":146},"hours",[35,415,118],{"class":84},[35,417,418],{"class":134},"1",[35,420,421],{"class":88},"),             ",[35,423,424],{"class":210},"# default for everything\n",[35,426,427,430,432],{"class":37,"line":214},[35,428,429],{"class":146},"    urls_expire_after",[35,431,118],{"class":84},[35,433,434],{"class":88},"{\n",[35,436,437,440,442,444,446,448,450,453,456,458,460,463],{"class":37,"line":265},[35,438,439],{"class":84},"        f",[35,441,127],{"class":45},[35,443,131],{"class":130},[35,445,64],{"class":134},[35,447,137],{"class":130},[35,449,140],{"class":45},[35,451,452],{"class":88},": timedelta(",[35,454,455],{"class":146},"days",[35,457,118],{"class":84},[35,459,418],{"class":134},[35,461,462],{"class":88},"),      ",[35,464,465],{"class":210},"# rates change daily\n",[35,467,468,470,472,474,476,478,481,483,486,488,491],{"class":37,"line":274},[35,469,439],{"class":84},[35,471,127],{"class":45},[35,473,131],{"class":130},[35,475,64],{"class":134},[35,477,137],{"class":130},[35,479,480],{"class":45},"\u002Fsales*\"",[35,482,452],{"class":88},[35,484,485],{"class":146},"minutes",[35,487,118],{"class":84},[35,489,490],{"class":134},"15",[35,492,493],{"class":88},"),\n",[35,495,496],{"class":37,"line":279},[35,497,498],{"class":88},"    },\n",[35,500,501,504,506,508,511,514],{"class":37,"line":295},[35,502,503],{"class":146},"    allowable_codes",[35,505,118],{"class":84},[35,507,316],{"class":88},[35,509,510],{"class":134},"200",[35,512,513],{"class":88},",),                      ",[35,515,516],{"class":210},"# never cache errors\n",[35,518,519,522,524,526,529],{"class":37,"line":301},[35,520,521],{"class":146},"    allowable_methods",[35,523,118],{"class":84},[35,525,316],{"class":88},[35,527,528],{"class":45},"\"GET\"",[35,530,531],{"class":88},",),\n",[35,533,534],{"class":37,"line":310},[35,535,536],{"class":88},")\n",[35,538,540],{"class":37,"line":539},14,[35,541,542],{"class":88},"cached.cache.clear()\n",[35,544,546],{"class":37,"line":545},15,[35,547,96],{"emptyLinePlaceholder":95},[35,549,551,553,555,557,559],{"class":37,"line":550},16,[35,552,282],{"class":88},[35,554,118],{"class":84},[35,556,287],{"class":88},[35,558,290],{"class":45},[35,560,161],{"class":88},[35,562,564],{"class":37,"line":563},17,[35,565,566],{"class":88},"build_report(cached)\n",[35,568,570,573,575,577,579,581,583],{"class":37,"line":569},18,[35,571,572],{"class":88},"first ",[35,574,118],{"class":84},[35,576,287],{"class":88},[35,578,290],{"class":45},[35,580,327],{"class":88},[35,582,330],{"class":84},[35,584,585],{"class":88}," start\n",[35,587,589],{"class":37,"line":588},19,[35,590,566],{"class":88},[35,592,594,596,598,601,604,607,609,611,613,615,618,620,623,626],{"class":37,"line":593},20,[35,595,313],{"class":134},[35,597,316],{"class":88},[35,599,600],{"class":45},"\"first run:\"",[35,602,603],{"class":88},", first, ",[35,605,606],{"class":45},"\"requests | second run:\"",[35,608,322],{"class":88},[35,610,290],{"class":45},[35,612,327],{"class":88},[35,614,330],{"class":84},[35,616,617],{"class":88}," start ",[35,619,330],{"class":84},[35,621,622],{"class":88}," first, ",[35,624,625],{"class":45},"\"requests\"",[35,627,536],{"class":88},[35,629,631,634,636,639,641,643,645,647,649,651],{"class":37,"line":630},21,[35,632,633],{"class":88},"r ",[35,635,118],{"class":84},[35,637,638],{"class":88}," cached.get(",[35,640,124],{"class":84},[35,642,127],{"class":45},[35,644,131],{"class":130},[35,646,64],{"class":134},[35,648,137],{"class":130},[35,650,140],{"class":45},[35,652,536],{"class":88},[35,654,656,658,660,663,666,669,672,675,678,681],{"class":37,"line":655},22,[35,657,313],{"class":134},[35,659,316],{"class":88},[35,661,662],{"class":45},"\"from cache:\"",[35,664,665],{"class":88},", r.from_cache, ",[35,667,668],{"class":45},"\"| expires:\"",[35,670,671],{"class":88},", r.expires ",[35,673,674],{"class":84},"is",[35,676,677],{"class":84}," not",[35,679,680],{"class":134}," None",[35,682,536],{"class":88},[10,684,685,686,689,690,693,694,697,698,701,702,705,706,709],{},"The first run makes four requests — the duplicate North call within the run is already served from the cache — and the second run makes none: everything comes from ",[32,687,688],{},"report_cache.sqlite",". ",[32,691,692],{},"urls_expire_after"," gives each endpoint an expiry that matches how often its data changes: exchange rates for a day, sales for fifteen minutes, everything else for an hour. ",[32,695,696],{},"allowable_codes=(200,)"," is important: without it, a cached ",[32,699,700],{},"503"," error could be served back on the next run and make a temporary outage look permanent. Every response carries ",[32,703,704],{},"from_cache"," and ",[32,707,708],{},"expires",", which the report uses in Step 5.",[711,712,720,721,720,725,720,729,720,737,720,747,720,758,720,765,720,770,720,776,720,781,720,785,720,792,720,797,720,800,720,804,720,807,720,812,720,819,720,824,720,827,720,831,720,834,720,839,720,846,720,851],"svg",{"viewBox":713,"role":714,"ariaLabelledBy":715,"xmlns":718,"style":719},"4 0 752 125","img",[716,717],"cache-flow-t","cache-flow-d","http:\u002F\u002Fwww.w3.org\u002F2000\u002Fsvg","width:100%;max-width:752px;height:auto;display:block;margin:1.5rem auto;font-family:Inter,ui-sans-serif,system-ui,sans-serif","\n  ",[722,723,724],"title",{"id":716},"How a cached request is answered",[726,727,728],"desc",{"id":717},"The script asks the CachedSession for a URL. If a fresh, successful response is stored, it is returned from the local SQLite file. Otherwise the request goes to the API, and a 200 response is stored with an expiry chosen by URL pattern.",[730,731],"rect",{"x":732,"y":733,"width":734,"height":735,"fill":736},"4","0","752","125","#ffffff",[730,738],{"x":739,"y":740,"width":741,"height":742,"rx":743,"fill":744,"stroke":745,"style":746},"20.0","16.0","153.0","93.0","12","#e7ebef","var(--line,#cdd5e6)","stroke-width:2px",[748,749,751],"g",{"transform":750},"translate(84.5,29.0) scale(1.000)",[752,753],"path",{"d":754,"fill":755,"stroke":756,"style":757},"M9 8l-4 4 4 4 M15 8l4 4-4 4","none","var(--text,#172033)","stroke-width:1.80px;stroke-linecap:round;stroke-linejoin:round",[759,760,764],"text",{"x":761,"y":762,"style":763},"96.5","75.5","font-size:13.5px;font-weight:700;fill:var(--text,#172033);text-anchor:middle","Script request",[759,766,769],{"x":761,"y":767,"style":768},"95.0","font-size:11.5px;font-weight:400;fill:var(--text,#172033);text-anchor:middle","session.get(url)",[37,771],{"x1":772,"y1":773,"x2":774,"y2":773,"stroke":775,"style":746},"178.0","62.5","200.0","var(--brand,#5b5cf0)",[777,778],"polygon",{"points":779,"fill":780},"204.0,62.5 195.0,67.5 195.0,57.5","#5b5cf0",[730,782],{"x":783,"y":740,"width":741,"height":742,"rx":743,"fill":784,"stroke":775,"style":746},"209.0","#f0f4ff",[748,786,788],{"transform":787},"translate(273.5,29.0) scale(1.000)",[752,789],{"d":790,"fill":755,"stroke":791,"style":757},"M11 4a7 7 0 1 0 .01 0z M16 16l5 5","var(--brand-strong,#4338ca)",[759,793,796],{"x":794,"y":762,"style":795},"285.5","font-size:13.5px;font-weight:700;fill:var(--brand-strong,#4338ca);text-anchor:middle","Fresh in cache?",[759,798,799],{"x":794,"y":767,"style":768},"expiry per URL pattern",[37,801],{"x1":802,"y1":773,"x2":803,"y2":773,"stroke":775,"style":746},"367.0","389.0",[777,805],{"points":806,"fill":780},"393.0,62.5 384.0,67.5 384.0,57.5",[730,808],{"x":809,"y":740,"width":741,"height":742,"rx":743,"fill":810,"stroke":811,"style":746},"398.0","#d9f4f1","var(--teal,#0f9488)",[748,813,815],{"transform":814},"translate(462.5,29.0) scale(1.000)",[752,816],{"d":817,"fill":755,"stroke":818,"style":757},"M5 6c0-1.7 3.1-3 7-3s7 1.3 7 3-3.1 3-7 3-7-1.3-7-3z M5 6v12c0 1.7 3.1 3 7 3s7-1.3 7-3V6 M5 12c0 1.7 3.1 3 7 3s7-1.3 7-3","var(--teal-ink,#0b6157)",[759,820,823],{"x":821,"y":762,"style":822},"474.5","font-size:13.5px;font-weight:700;fill:var(--teal-ink,#0b6157);text-anchor:middle","Yes → local file",[759,825,826],{"x":821,"y":767,"style":768},"no network call",[37,828],{"x1":829,"y1":773,"x2":830,"y2":773,"stroke":775,"style":746},"556.0","578.0",[777,832],{"points":833,"fill":780},"582.0,62.5 573.0,67.5 573.0,57.5",[730,835],{"x":836,"y":740,"width":741,"height":742,"rx":743,"fill":837,"stroke":838,"style":746},"587.0","#fdefd8","var(--gold,#b4740a)",[748,840,842],{"transform":841},"translate(651.5,29.0) scale(1.000)",[752,843],{"d":844,"fill":755,"stroke":845,"style":757},"M7 18h10a4 4 0 0 0 .6-8A6 6 0 0 0 6.1 11.4 3.4 3.4 0 0 0 7 18z","var(--gold-ink,#7a4e06)",[759,847,850],{"x":848,"y":762,"style":849},"663.5","font-size:13.5px;font-weight:700;fill:var(--gold-ink,#7a4e06);text-anchor:middle","No → API",[759,852,853],{"x":848,"y":767,"style":768},"store only 200s",[711,855,720,861,720,864,720,867,720,871,720,877,720,883,720,890,720,895,720,898,720,902,720,905,720,910,720,913,720,917,720,920],{"viewBox":856,"role":714,"ariaLabelledBy":857,"xmlns":718,"style":860},"4 0 744 140",[858,859],"cache-count-t","cache-count-d","width:100%;max-width:744px;height:auto;display:block;margin:1.5rem auto;font-family:Inter,ui-sans-serif,system-ui,sans-serif",[722,862,863],{"id":858},"Requests that reached the API for two runs of the report",[726,865,866],{"id":859},"Without a cache, two runs of the report sent ten requests to the API. With requests-cache, the first run sent four, because the duplicate North request was served from the cache, and the second run sent none.",[730,868],{"x":732,"y":733,"width":869,"height":870,"fill":736},"744","140",[730,872],{"x":739,"y":740,"width":873,"height":874,"rx":875,"fill":744,"stroke":745,"style":876},"166.9","28.0","7","stroke-width:1px",[759,878,882],{"x":879,"y":880,"style":881},"103.4","34.5","font-size:12.5px;font-weight:700;fill:var(--text,#172033);text-anchor:middle","no cache (two runs)",[730,884],{"x":885,"y":740,"width":886,"height":874,"rx":875,"fill":887,"stroke":888,"style":889},"196.9","455.6","#fce9e9","var(--danger,#dc2626)","stroke-width:1.5px",[759,891,894],{"x":892,"y":880,"style":893},"660.4","font-size:12px;font-weight:700;fill:var(--accent-ink,#be185d);text-anchor:start","10 requests",[730,896],{"x":739,"y":897,"width":873,"height":874,"rx":875,"fill":744,"stroke":745,"style":876},"56.0",[759,899,901],{"x":879,"y":900,"style":881},"74.5","cached — first run",[730,903],{"x":885,"y":897,"width":904,"height":874,"rx":875,"fill":837,"stroke":838,"style":889},"182.2",[759,906,909],{"x":907,"y":900,"style":908},"387.1","font-size:12px;font-weight:700;fill:var(--gold-ink,#7a4e06);text-anchor:start","4 requests",[730,911],{"x":739,"y":912,"width":873,"height":874,"rx":875,"fill":744,"stroke":745,"style":876},"96.0",[759,914,916],{"x":879,"y":915,"style":881},"114.5","cached — second run",[730,918],{"x":885,"y":912,"width":919,"height":874,"rx":875,"fill":810,"stroke":811,"style":889},"6.8",[759,921,924],{"x":922,"y":915,"style":923},"211.7","font-size:12px;font-weight:700;fill:var(--teal-ink,#0b6157);text-anchor:start","0 requests",[10,926,927],{},"The saving grows with the number of reruns: a morning spent adjusting a report's layout might run the script thirty times, and with the cache only the first run touches the API. It also makes the script usable offline — on a train, or while the API is down for maintenance — as long as the cached responses have not expired.",[20,929,931],{"id":930},"step-3-errors-are-not-cached","Step 3 — Errors are not cached",[10,933,934],{},"Make sure a temporary failure does not get stuck in the cache. Request a failing endpoint twice:",[25,936,938],{"className":75,"code":937,"language":77,"meta":30,"style":30},"start = hits[\"n\"]\nfor _ in range(2):\n    resp = cached.get(f\"{BASE}\u002Fsales\", params={\"region\": \"broken\"})\n    print(resp.status_code, \"from cache:\", resp.from_cache)\nprint(\"requests reaching the API:\", hits[\"n\"] - start)\n",[32,939,940,952,973,1013,1026],{"__ignoreMap":30},[35,941,942,944,946,948,950],{"class":37,"line":38},[35,943,282],{"class":88},[35,945,118],{"class":84},[35,947,287],{"class":88},[35,949,290],{"class":45},[35,951,161],{"class":88},[35,953,954,957,960,962,965,967,970],{"class":37,"line":92},[35,955,956],{"class":84},"for",[35,958,959],{"class":88}," _ ",[35,961,184],{"class":84},[35,963,964],{"class":134}," range",[35,966,316],{"class":88},[35,968,969],{"class":134},"2",[35,971,972],{"class":88},"):\n",[35,974,975,978,980,982,984,986,988,990,992,994,996,998,1000,1002,1004,1007,1010],{"class":37,"line":99},[35,976,977],{"class":88},"    resp ",[35,979,118],{"class":84},[35,981,638],{"class":88},[35,983,124],{"class":84},[35,985,127],{"class":45},[35,987,131],{"class":130},[35,989,64],{"class":134},[35,991,137],{"class":130},[35,993,234],{"class":45},[35,995,143],{"class":88},[35,997,239],{"class":146},[35,999,118],{"class":84},[35,1001,131],{"class":88},[35,1003,246],{"class":45},[35,1005,1006],{"class":88},": ",[35,1008,1009],{"class":45},"\"broken\"",[35,1011,1012],{"class":88},"})\n",[35,1014,1015,1018,1021,1023],{"class":37,"line":112},[35,1016,1017],{"class":134},"    print",[35,1019,1020],{"class":88},"(resp.status_code, ",[35,1022,662],{"class":45},[35,1024,1025],{"class":88},", resp.from_cache)\n",[35,1027,1028,1030,1032,1034,1036,1038,1040,1042],{"class":37,"line":164},[35,1029,313],{"class":134},[35,1031,316],{"class":88},[35,1033,319],{"class":45},[35,1035,322],{"class":88},[35,1037,290],{"class":45},[35,1039,327],{"class":88},[35,1041,330],{"class":84},[35,1043,333],{"class":88},[10,1045,1046,1047,1049,1050,1054,1055,705,1058,1061],{},"Both requests reach the API and neither is served from the cache, because only ",[32,1048,510],{}," responses are stored. The next run, after the service recovers, will fetch real data. Combine this with retries for transient errors — the session from ",[14,1051,1053],{"href":1052},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Ffetch-paginated-api-data-into-excel\u002F","Fetch Paginated API Data into Excel"," handles ",[32,1056,1057],{},"429",[32,1059,1060],{},"5xx"," — and the cache never has to deal with failures at all.",[20,1063,1065],{"id":1064},"step-4-a-hand-rolled-cache-for-post-queries","Step 4 — A hand-rolled cache for POST queries",[10,1067,1068],{},"Some APIs take their query as a POST body, which HTTP caches deliberately do not cache. For those, a small file cache keyed by a hash of the request is easy to write and easy to reason about:",[25,1070,1072],{"className":75,"code":1071,"language":77,"meta":30,"style":30},"import hashlib\nimport json\nimport time\nfrom pathlib import Path\n\nCACHE_DIR = Path(\"api_cache\")\nCACHE_DIR.mkdir(exist_ok=True)\n\ndef cached_post(url, payload, max_age_s=900, refresh=False):\n    key = hashlib.sha256(json.dumps([url, payload], sort_keys=True).encode()).hexdigest()[:16]\n    path = CACHE_DIR \u002F f\"{key}.json\"\n    if path.exists() and not refresh and time.time() - path.stat().st_mtime \u003C max_age_s:\n        return json.loads(path.read_text()), True\n    resp = requests.post(url, json=payload, timeout=30)\n    resp.raise_for_status()\n    path.write_text(resp.text)\n    return resp.json(), False\n\nstart = hits[\"n\"]\nq = {\"skus\": [\"DRL-01\", \"SAW-02\"], \"warehouse\": \"LDS\"}\nprint([cached_post(f\"{BASE}\u002Fstock\", q)[1] for _ in range(3)], \"| requests:\", hits[\"n\"] - start)\nsame_q_reordered = {\"warehouse\": \"LDS\", \"skus\": [\"DRL-01\", \"SAW-02\"]}\nprint(\"reordered keys hit the cache:\", cached_post(f\"{BASE}\u002Fstock\", same_q_reordered)[1])\n",[32,1073,1074,1081,1088,1095,1107,1111,1127,1144,1148,1173,1198,1226,1258,1269,1294,1299,1304,1314,1318,1330,1368,1424,1454],{"__ignoreMap":30},[35,1075,1076,1078],{"class":37,"line":38},[35,1077,85],{"class":84},[35,1079,1080],{"class":88}," hashlib\n",[35,1082,1083,1085],{"class":37,"line":92},[35,1084,85],{"class":84},[35,1086,1087],{"class":88}," json\n",[35,1089,1090,1092],{"class":37,"line":99},[35,1091,85],{"class":84},[35,1093,1094],{"class":88}," time\n",[35,1096,1097,1099,1102,1104],{"class":37,"line":112},[35,1098,367],{"class":84},[35,1100,1101],{"class":88}," pathlib ",[35,1103,85],{"class":84},[35,1105,1106],{"class":88}," Path\n",[35,1108,1109],{"class":37,"line":164},[35,1110,96],{"emptyLinePlaceholder":95},[35,1112,1113,1116,1119,1122,1125],{"class":37,"line":175},[35,1114,1115],{"class":134},"CACHE_DIR",[35,1117,1118],{"class":84}," =",[35,1120,1121],{"class":88}," Path(",[35,1123,1124],{"class":45},"\"api_cache\"",[35,1126,536],{"class":88},[35,1128,1129,1131,1134,1137,1139,1142],{"class":37,"line":214},[35,1130,1115],{"class":134},[35,1132,1133],{"class":88},".mkdir(",[35,1135,1136],{"class":146},"exist_ok",[35,1138,118],{"class":84},[35,1140,1141],{"class":134},"True",[35,1143,536],{"class":88},[35,1145,1146],{"class":37,"line":265},[35,1147,96],{"emptyLinePlaceholder":95},[35,1149,1150,1152,1155,1158,1160,1163,1166,1168,1171],{"class":37,"line":274},[35,1151,102],{"class":84},[35,1153,1154],{"class":105}," cached_post",[35,1156,1157],{"class":88},"(url, payload, max_age_s",[35,1159,118],{"class":84},[35,1161,1162],{"class":134},"900",[35,1164,1165],{"class":88},", refresh",[35,1167,118],{"class":84},[35,1169,1170],{"class":134},"False",[35,1172,972],{"class":88},[35,1174,1175,1178,1180,1183,1186,1188,1190,1193,1196],{"class":37,"line":279},[35,1176,1177],{"class":88},"    key ",[35,1179,118],{"class":84},[35,1181,1182],{"class":88}," hashlib.sha256(json.dumps([url, payload], ",[35,1184,1185],{"class":146},"sort_keys",[35,1187,118],{"class":84},[35,1189,1141],{"class":134},[35,1191,1192],{"class":88},").encode()).hexdigest()[:",[35,1194,1195],{"class":134},"16",[35,1197,161],{"class":88},[35,1199,1200,1203,1205,1208,1211,1214,1216,1218,1221,1223],{"class":37,"line":295},[35,1201,1202],{"class":88},"    path ",[35,1204,118],{"class":84},[35,1206,1207],{"class":134}," CACHE_DIR",[35,1209,1210],{"class":84}," \u002F",[35,1212,1213],{"class":84}," f",[35,1215,127],{"class":45},[35,1217,131],{"class":130},[35,1219,1220],{"class":88},"key",[35,1222,137],{"class":130},[35,1224,1225],{"class":45},".json\"\n",[35,1227,1228,1231,1234,1237,1239,1242,1244,1247,1249,1252,1255],{"class":37,"line":301},[35,1229,1230],{"class":84},"    if",[35,1232,1233],{"class":88}," path.exists() ",[35,1235,1236],{"class":84},"and",[35,1238,677],{"class":84},[35,1240,1241],{"class":88}," refresh ",[35,1243,1236],{"class":84},[35,1245,1246],{"class":88}," time.time() ",[35,1248,330],{"class":84},[35,1250,1251],{"class":88}," path.stat().st_mtime ",[35,1253,1254],{"class":84},"\u003C",[35,1256,1257],{"class":88}," max_age_s:\n",[35,1259,1260,1263,1266],{"class":37,"line":310},[35,1261,1262],{"class":84},"        return",[35,1264,1265],{"class":88}," json.loads(path.read_text()), ",[35,1267,1268],{"class":134},"True\n",[35,1270,1271,1273,1275,1278,1281,1283,1286,1288,1290,1292],{"class":37,"line":539},[35,1272,977],{"class":88},[35,1274,118],{"class":84},[35,1276,1277],{"class":88}," requests.post(url, ",[35,1279,1280],{"class":146},"json",[35,1282,118],{"class":84},[35,1284,1285],{"class":88},"payload, ",[35,1287,147],{"class":146},[35,1289,118],{"class":84},[35,1291,152],{"class":134},[35,1293,536],{"class":88},[35,1295,1296],{"class":37,"line":545},[35,1297,1298],{"class":88},"    resp.raise_for_status()\n",[35,1300,1301],{"class":37,"line":550},[35,1302,1303],{"class":88},"    path.write_text(resp.text)\n",[35,1305,1306,1308,1311],{"class":37,"line":563},[35,1307,268],{"class":84},[35,1309,1310],{"class":88}," resp.json(), ",[35,1312,1313],{"class":134},"False\n",[35,1315,1316],{"class":37,"line":569},[35,1317,96],{"emptyLinePlaceholder":95},[35,1319,1320,1322,1324,1326,1328],{"class":37,"line":588},[35,1321,282],{"class":88},[35,1323,118],{"class":84},[35,1325,287],{"class":88},[35,1327,290],{"class":45},[35,1329,161],{"class":88},[35,1331,1332,1335,1337,1340,1343,1346,1349,1351,1354,1357,1360,1362,1365],{"class":37,"line":593},[35,1333,1334],{"class":88},"q ",[35,1336,118],{"class":84},[35,1338,1339],{"class":88}," {",[35,1341,1342],{"class":45},"\"skus\"",[35,1344,1345],{"class":88},": [",[35,1347,1348],{"class":45},"\"DRL-01\"",[35,1350,143],{"class":88},[35,1352,1353],{"class":45},"\"SAW-02\"",[35,1355,1356],{"class":88},"], ",[35,1358,1359],{"class":45},"\"warehouse\"",[35,1361,1006],{"class":88},[35,1363,1364],{"class":45},"\"LDS\"",[35,1366,1367],{"class":88},"}\n",[35,1369,1370,1372,1375,1377,1379,1381,1383,1385,1388,1391,1393,1395,1397,1399,1401,1403,1405,1408,1411,1414,1416,1418,1420,1422],{"class":37,"line":630},[35,1371,313],{"class":134},[35,1373,1374],{"class":88},"([cached_post(",[35,1376,124],{"class":84},[35,1378,127],{"class":45},[35,1380,131],{"class":130},[35,1382,64],{"class":134},[35,1384,137],{"class":130},[35,1386,1387],{"class":45},"\u002Fstock\"",[35,1389,1390],{"class":88},", q)[",[35,1392,418],{"class":134},[35,1394,327],{"class":88},[35,1396,956],{"class":84},[35,1398,959],{"class":88},[35,1400,184],{"class":84},[35,1402,964],{"class":134},[35,1404,316],{"class":88},[35,1406,1407],{"class":134},"3",[35,1409,1410],{"class":88},")], ",[35,1412,1413],{"class":45},"\"| requests:\"",[35,1415,322],{"class":88},[35,1417,290],{"class":45},[35,1419,327],{"class":88},[35,1421,330],{"class":84},[35,1423,333],{"class":88},[35,1425,1426,1429,1431,1433,1435,1437,1439,1441,1443,1445,1447,1449,1451],{"class":37,"line":655},[35,1427,1428],{"class":88},"same_q_reordered ",[35,1430,118],{"class":84},[35,1432,1339],{"class":88},[35,1434,1359],{"class":45},[35,1436,1006],{"class":88},[35,1438,1364],{"class":45},[35,1440,143],{"class":88},[35,1442,1342],{"class":45},[35,1444,1345],{"class":88},[35,1446,1348],{"class":45},[35,1448,143],{"class":88},[35,1450,1353],{"class":45},[35,1452,1453],{"class":88},"]}\n",[35,1455,1457,1459,1461,1464,1467,1469,1471,1473,1475,1477,1479,1482,1484],{"class":37,"line":1456},23,[35,1458,313],{"class":134},[35,1460,316],{"class":88},[35,1462,1463],{"class":45},"\"reordered keys hit the cache:\"",[35,1465,1466],{"class":88},", cached_post(",[35,1468,124],{"class":84},[35,1470,127],{"class":45},[35,1472,131],{"class":130},[35,1474,64],{"class":134},[35,1476,137],{"class":130},[35,1478,1387],{"class":45},[35,1480,1481],{"class":88},", same_q_reordered)[",[35,1483,418],{"class":134},[35,1485,1486],{"class":88},"])\n",[10,1488,1489,1490,1493,1494,1497,1498,1501,1502,1505,1506,18],{},"The key is a hash of the URL and the payload serialised with ",[32,1491,1492],{},"sort_keys=True",", so the same query with its keys in a different order hits the same cache entry; the first of the three calls goes to the API, the next two are answered from disk, and the reordered query is a cache hit too — one request in total. Freshness is the file's age compared with ",[32,1495,1496],{},"max_age_s",", and ",[32,1499,1500],{},"refresh=True"," bypasses the cache. Only successful responses are written, because ",[32,1503,1504],{},"raise_for_status()"," raises before the file is saved. Keep the cache directory out of version control — it can contain business data — by adding it to ",[32,1507,1508],{},".gitignore",[20,1510,1512],{"id":1511},"step-5-force-fresh-data-for-production-and-say-how-old-it-is","Step 5 — Force fresh data for production, and say how old it is",[10,1514,1515],{},"Caching is for development and for repeat calls; the production run should usually start fresh and the workbook should say when its data was fetched. requests-cache can bypass the cache on demand, and the response tells you whether it came from the cache:",[25,1517,1519],{"className":75,"code":1518,"language":77,"meta":30,"style":30},"import os\nfrom datetime import datetime, timezone\nimport pandas as pd\n\nFRESH = os.environ.get(\"REPORT_FRESH\") == \"1\"\nsession = requests_cache.CachedSession(\"report_cache\", expire_after=timedelta(hours=1), allowable_codes=(200,))\n\nresp = session.get(f\"{BASE}\u002Fsales\", params={\"region\": \"North\"}, force_refresh=FRESH)\nsales = pd.DataFrame(resp.json()[\"rows\"])\nfetched = resp.created_at if resp.from_cache else datetime.now(timezone.utc)\nnote = f\"Data fetched {fetched:%Y-%m-%d %H:%M} UTC\" + (\" (from cache)\" if resp.from_cache else \"\")\nwith pd.ExcelWriter(\"north_sales.xlsx\", engine=\"openpyxl\") as xw:\n    sales.to_excel(xw, sheet_name=\"Sales\", index=False, startrow=2)\n    xw.sheets[\"Sales\"][\"A1\"] = note\nprint(note)\n",[32,1520,1521,1528,1539,1552,1556,1578,1621,1625,1672,1686,1708,1778,1806,1839,1859],{"__ignoreMap":30},[35,1522,1523,1525],{"class":37,"line":38},[35,1524,85],{"class":84},[35,1526,1527],{"class":88}," os\n",[35,1529,1530,1532,1534,1536],{"class":37,"line":92},[35,1531,367],{"class":84},[35,1533,370],{"class":88},[35,1535,85],{"class":84},[35,1537,1538],{"class":88}," datetime, timezone\n",[35,1540,1541,1543,1546,1549],{"class":37,"line":99},[35,1542,85],{"class":84},[35,1544,1545],{"class":88}," pandas ",[35,1547,1548],{"class":84},"as",[35,1550,1551],{"class":88}," pd\n",[35,1553,1554],{"class":37,"line":112},[35,1555,96],{"emptyLinePlaceholder":95},[35,1557,1558,1561,1563,1566,1569,1572,1575],{"class":37,"line":164},[35,1559,1560],{"class":134},"FRESH",[35,1562,1118],{"class":84},[35,1564,1565],{"class":88}," os.environ.get(",[35,1567,1568],{"class":45},"\"REPORT_FRESH\"",[35,1570,1571],{"class":88},") ",[35,1573,1574],{"class":84},"==",[35,1576,1577],{"class":45}," \"1\"\n",[35,1579,1580,1583,1585,1588,1591,1593,1596,1598,1600,1602,1604,1606,1609,1612,1614,1616,1618],{"class":37,"line":175},[35,1581,1582],{"class":88},"session ",[35,1584,118],{"class":84},[35,1586,1587],{"class":88}," requests_cache.CachedSession(",[35,1589,1590],{"class":45},"\"report_cache\"",[35,1592,143],{"class":88},[35,1594,1595],{"class":146},"expire_after",[35,1597,118],{"class":84},[35,1599,410],{"class":88},[35,1601,413],{"class":146},[35,1603,118],{"class":84},[35,1605,418],{"class":134},[35,1607,1608],{"class":88},"), ",[35,1610,1611],{"class":146},"allowable_codes",[35,1613,118],{"class":84},[35,1615,316],{"class":88},[35,1617,510],{"class":134},[35,1619,1620],{"class":88},",))\n",[35,1622,1623],{"class":37,"line":214},[35,1624,96],{"emptyLinePlaceholder":95},[35,1626,1627,1630,1632,1634,1636,1638,1640,1642,1644,1646,1648,1650,1652,1654,1656,1658,1660,1663,1666,1668,1670],{"class":37,"line":265},[35,1628,1629],{"class":88},"resp ",[35,1631,118],{"class":84},[35,1633,121],{"class":88},[35,1635,124],{"class":84},[35,1637,127],{"class":45},[35,1639,131],{"class":130},[35,1641,64],{"class":134},[35,1643,137],{"class":130},[35,1645,234],{"class":45},[35,1647,143],{"class":88},[35,1649,239],{"class":146},[35,1651,118],{"class":84},[35,1653,131],{"class":88},[35,1655,246],{"class":45},[35,1657,1006],{"class":88},[35,1659,190],{"class":45},[35,1661,1662],{"class":88},"}, ",[35,1664,1665],{"class":146},"force_refresh",[35,1667,118],{"class":84},[35,1669,1560],{"class":134},[35,1671,536],{"class":88},[35,1673,1674,1677,1679,1682,1684],{"class":37,"line":274},[35,1675,1676],{"class":88},"sales ",[35,1678,118],{"class":84},[35,1680,1681],{"class":88}," pd.DataFrame(resp.json()[",[35,1683,260],{"class":45},[35,1685,1486],{"class":88},[35,1687,1688,1691,1693,1696,1699,1702,1705],{"class":37,"line":279},[35,1689,1690],{"class":88},"fetched ",[35,1692,118],{"class":84},[35,1694,1695],{"class":88}," resp.created_at ",[35,1697,1698],{"class":84},"if",[35,1700,1701],{"class":88}," resp.from_cache ",[35,1703,1704],{"class":84},"else",[35,1706,1707],{"class":88}," datetime.now(timezone.utc)\n",[35,1709,1710,1713,1715,1717,1720,1722,1725,1728,1731,1734,1737,1739,1742,1744,1747,1749,1752,1754,1757,1760,1763,1766,1769,1771,1773,1776],{"class":37,"line":295},[35,1711,1712],{"class":88},"note ",[35,1714,118],{"class":84},[35,1716,1213],{"class":84},[35,1718,1719],{"class":45},"\"Data fetched ",[35,1721,131],{"class":130},[35,1723,1724],{"class":88},"fetched:",[35,1726,1727],{"class":84},"%",[35,1729,1730],{"class":88},"Y",[35,1732,1733],{"class":84},"-%",[35,1735,1736],{"class":88},"m",[35,1738,1733],{"class":84},[35,1740,1741],{"class":88},"d ",[35,1743,1727],{"class":84},[35,1745,1746],{"class":88},"H:",[35,1748,1727],{"class":84},[35,1750,1751],{"class":88},"M",[35,1753,137],{"class":130},[35,1755,1756],{"class":45}," UTC\"",[35,1758,1759],{"class":84}," +",[35,1761,1762],{"class":88}," (",[35,1764,1765],{"class":45},"\" (from cache)\"",[35,1767,1768],{"class":84}," if",[35,1770,1701],{"class":88},[35,1772,1704],{"class":84},[35,1774,1775],{"class":45}," \"\"",[35,1777,536],{"class":88},[35,1779,1780,1783,1786,1789,1791,1794,1796,1799,1801,1803],{"class":37,"line":301},[35,1781,1782],{"class":84},"with",[35,1784,1785],{"class":88}," pd.ExcelWriter(",[35,1787,1788],{"class":45},"\"north_sales.xlsx\"",[35,1790,143],{"class":88},[35,1792,1793],{"class":146},"engine",[35,1795,118],{"class":84},[35,1797,1798],{"class":45},"\"openpyxl\"",[35,1800,1571],{"class":88},[35,1802,1548],{"class":84},[35,1804,1805],{"class":88}," xw:\n",[35,1807,1808,1811,1814,1816,1819,1821,1824,1826,1828,1830,1833,1835,1837],{"class":37,"line":310},[35,1809,1810],{"class":88},"    sales.to_excel(xw, ",[35,1812,1813],{"class":146},"sheet_name",[35,1815,118],{"class":84},[35,1817,1818],{"class":45},"\"Sales\"",[35,1820,143],{"class":88},[35,1822,1823],{"class":146},"index",[35,1825,118],{"class":84},[35,1827,1170],{"class":134},[35,1829,143],{"class":88},[35,1831,1832],{"class":146},"startrow",[35,1834,118],{"class":84},[35,1836,969],{"class":134},[35,1838,536],{"class":88},[35,1840,1841,1844,1846,1849,1852,1854,1856],{"class":37,"line":539},[35,1842,1843],{"class":88},"    xw.sheets[",[35,1845,1818],{"class":45},[35,1847,1848],{"class":88},"][",[35,1850,1851],{"class":45},"\"A1\"",[35,1853,327],{"class":88},[35,1855,118],{"class":84},[35,1857,1858],{"class":88}," note\n",[35,1860,1861,1863],{"class":37,"line":545},[35,1862,313],{"class":134},[35,1864,1865],{"class":88},"(note)\n",[10,1867,1868,1869,1872,1873,1876,1877,18],{},"Setting ",[32,1870,1871],{},"REPORT_FRESH=1"," in the production scheduler makes every request skip the cache with ",[32,1874,1875],{},"force_refresh=True"," — and still stores the new responses for any later call in the same run. In development the variable is unset and the cache does its job. Either way the first line of the sheet records when the data was fetched and whether it came from the cache, so nobody mistakes a development workbook for a current report. Keeping such switches in configuration rather than code is covered in ",[14,1878,1880],{"href":1879},"\u002Fautomating-reporting-workflows\u002Ftesting-and-packaging-excel-automation-scripts\u002Fkeep-excel-report-settings-in-a-config-file\u002F","Keep Excel Report Settings in a Config File",[711,1882,720,1887,720,1890,720,1893,720,1896,720,1901,720,1907,720,1913,720,1918,720,1923,720,1927,720,1931,720,1933,720,1939,720,1944,720,1948,720,1950,720,1953],{"viewBox":1883,"role":714,"ariaLabelledBy":1884,"xmlns":718,"style":719},"4 0 752 207",[1885,1886],"cache-modes-t","cache-modes-d",[722,1888,1889],{"id":1885},"Development runs versus production runs",[726,1891,1892],{"id":1886},"In development, the cache answers repeat requests instantly and offline, with expiry per endpoint. In production, a flag forces fresh requests while still storing the responses, and the workbook records when the data was fetched.",[730,1894],{"x":732,"y":733,"width":734,"height":1895,"fill":736},"207",[730,1897],{"x":739,"y":740,"width":1898,"height":1899,"rx":1900,"fill":784,"stroke":775,"style":746},"351.0","175.0","14",[748,1902,1904],{"transform":1903},"translate(182.5,29.0) scale(1.083)",[752,1905],{"d":754,"fill":755,"stroke":791,"style":1906},"stroke-width:1.66px;stroke-linecap:round;stroke-linejoin:round",[759,1908,1912],{"x":1909,"y":1910,"style":1911},"195.5","78.0","font-size:14px;font-weight:700;fill:var(--brand-strong,#4338ca);text-anchor:middle","development",[37,1914],{"x1":1915,"y1":1916,"x2":1917,"y2":1916,"stroke":775,"style":876},"36.0","91.0","355.0",[759,1919,1922],{"x":1909,"y":1920,"style":1921},"111.0","font-size:12px;font-weight:400;fill:var(--text,#172033);text-anchor:middle","cache answers reruns",[759,1924,1926],{"x":1909,"y":1925,"style":1921},"137.0","works offline",[759,1928,1930],{"x":1909,"y":1929,"style":1921},"163.0","expiry per endpoint",[730,1932],{"x":803,"y":740,"width":1898,"height":1899,"rx":1900,"fill":810,"stroke":811,"style":746},[748,1934,1936],{"transform":1935},"translate(551.5,29.0) scale(1.083)",[752,1937],{"d":1938,"fill":755,"stroke":818,"style":1906},"M13 2L4 14h7l-1 8 9-12h-7z",[759,1940,1943],{"x":1941,"y":1910,"style":1942},"564.5","font-size:14px;font-weight:700;fill:var(--teal-ink,#0b6157);text-anchor:middle","production (REPORT_FRESH=1)",[37,1945],{"x1":1946,"y1":1916,"x2":1947,"y2":1916,"stroke":811,"style":876},"405.0","724.0",[759,1949,1875],{"x":1941,"y":1920,"style":1921},[759,1951,1952],{"x":1941,"y":1925,"style":1921},"responses still stored",[759,1954,1955],{"x":1941,"y":1929,"style":1921},"'Data fetched …' note in the sheet",[20,1957,1959],{"id":1958},"common-pitfalls","Common pitfalls",[1961,1962,1963,1974,1980,1986,1994],"ul",{},[1964,1965,1966,1970,1971,1973],"li",{},[1967,1968,1969],"strong",{},"Caching errors."," Restrict caching to ",[32,1972,510],{}," responses.",[1964,1975,1976,1979],{},[1967,1977,1978],{},"One expiry for everything."," Match expiry to how often each endpoint's data changes.",[1964,1981,1982,1985],{},[1967,1983,1984],{},"Stale data in production without a trace."," Force refresh for scheduled runs and record the fetch time in the workbook.",[1964,1987,1988,1991,1992,18],{},[1967,1989,1990],{},"Cache keys that depend on dictionary order."," Serialise payloads with ",[32,1993,1492],{},[1964,1995,1996,1999],{},[1967,1997,1998],{},"Committing the cache."," It may hold business data; ignore it in version control.",[20,2001,2003],{"id":2002},"conclusion","Conclusion",[10,2005,2006,2007,2009],{},"Caching API responses makes Excel report development fast and production reruns cheap. ",[32,2008,345],{}," adds a transparent SQLite-backed cache with expiry per URL pattern; restrict it to successful GET responses. For POST queries, a small file cache keyed by a hash of the URL and a sorted-key payload does the same job. Force fresh requests for production runs with a configuration flag, and write the fetch time — and whether it came from the cache — into the workbook so readers know how current the figures are.",[20,2011,2013],{"id":2012},"frequently-asked-questions","Frequently asked questions",[10,2015,2016,2019],{},[1967,2017,2018],{},"How do I cache API calls in a Python report script?","\nReplace requests.Session with requests_cache.CachedSession, which stores responses in a local SQLite file and serves repeat requests from it until they expire.",[10,2021,2022,2025],{},[1967,2023,2024],{},"How do I set different cache times for different endpoints?","\nPass urls_expire_after to CachedSession with URL patterns mapped to expiry times, for example a day for exchange rates and fifteen minutes for sales data.",[10,2027,2028,2031],{},[1967,2029,2030],{},"Can requests-cache cache POST requests?","\nNot by default, because POST requests are not normally cacheable. Write a small file cache keyed by a hash of the URL and the JSON payload instead.",[10,2033,2034,2037],{},[1967,2035,2036],{},"How do I make sure a scheduled report uses fresh data?","\nPass force_refresh=True for production runs, controlled by an environment variable or config setting, and write the fetch time into the workbook.",[20,2039,2041],{"id":2040},"related","Related",[1961,2043,2044,2050,2057,2064],{},[1964,2045,2046,2047,2049],{},"Up one level: ",[14,2048,17],{"href":16}," — getting data in and out.",[1964,2051,2052,2056],{},[14,2053,2055],{"href":2054},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Ffetch-api-data-into-excel-with-python-requests\u002F","Fetch API Data into Excel with Python requests"," — the requests that get cached.",[1964,2058,2059,2063],{},[14,2060,2062],{"href":2061},"\u002Fautomating-reporting-workflows\u002Ferror-handling-and-logging-in-excel-automation\u002Fskip-a-report-when-the-input-file-has-not-changed\u002F","Skip a Report When the Input File Has Not Changed"," — avoiding work at the report level.",[1964,2065,2066,2070],{},[14,2067,2069],{"href":2068},"\u002Fautomating-reporting-workflows\u002Ftesting-and-packaging-excel-automation-scripts\u002Fstore-credentials-for-excel-report-scripts-safely\u002F","Store Credentials for Excel Report Scripts Safely"," — the API keys behind the requests.",[2072,2073,2074],"style",{},"html pre.shiki code .sMTad, html code.shiki .sMTad{--shiki-default:#6F42C1;--shiki-dark:#FFB757}html pre.shiki code .srMev, html code.shiki .srMev{--shiki-default:#032F62;--shiki-dark:#ADDCFF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html pre.shiki code .s-kum, html code.shiki .s-kum{--shiki-default:#D73A49;--shiki-dark:#FF9492}html pre.shiki code .skGVy, html code.shiki .skGVy{--shiki-default:#24292E;--shiki-dark:#F0F3F6}html pre.shiki code .s_Opv, html code.shiki .s_Opv{--shiki-default:#6F42C1;--shiki-dark:#DBB7FF}html pre.shiki code .sSjpA, html code.shiki .sSjpA{--shiki-default:#005CC5;--shiki-dark:#FF9492}html pre.shiki code .sP0c6, html code.shiki .sP0c6{--shiki-default:#005CC5;--shiki-dark:#91CBFF}html pre.shiki code .sa561, html code.shiki .sa561{--shiki-default:#E36209;--shiki-dark:#FFB757}html pre.shiki code .s-wDw, html code.shiki .s-wDw{--shiki-default:#6A737D;--shiki-dark:#BDC4CC}",{"title":30,"searchDepth":92,"depth":92,"links":2076},[2077,2078,2079,2080,2081,2082,2083,2084,2085,2086],{"id":22,"depth":92,"text":23},{"id":68,"depth":92,"text":69},{"id":339,"depth":92,"text":340},{"id":930,"depth":92,"text":931},{"id":1064,"depth":92,"text":1065},{"id":1511,"depth":92,"text":1512},{"id":1958,"depth":92,"text":1959},{"id":2002,"depth":92,"text":2003},{"id":2012,"depth":92,"text":2013},{"id":2040,"depth":92,"text":2041},"2026-10-02","Stop a report script from calling the same API hundreds of times while you develop it or rerun it: a transparent HTTP cache with requests-cache and expiry rules per endpoint, a hand-rolled file cache for responses that are not plain GETs, caching only successful responses, forcing a refresh for the production run, and recording in the workbook how old the cached data was.","md",[2091,2093,2095,2097],{"q":2018,"a":2092},"Replace requests.Session with requests_cache.CachedSession, which stores responses in a local SQLite file and serves repeat requests from it until they expire.",{"q":2024,"a":2094},"Pass urls_expire_after to CachedSession with URL patterns mapped to expiry times, for example a day for exchange rates and fifteen minutes for sales data.",{"q":2030,"a":2096},"Not by default, because POST requests are not normally cacheable. Write a small file cache keyed by a hash of the URL and the JSON payload instead.",{"q":2036,"a":2098},"Pass force_refresh=True for production runs, controlled by an environment variable or config setting, and write the fetch time into the workbook.","cache api responses when building excel reports",null,{"breadcrumb":2102},[2103,2106,2109],{"name":2104,"item":2105},"Home","\u002F",{"name":2107,"item":2108},"Advanced Data Transformation and Cleaning for Excel Automation","\u002Fadvanced-data-transformation-and-cleaning\u002F",{"name":17,"item":16},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fcache-api-responses-when-building-excel-reports",{"title":2112,"description":2113},"Cache API Responses When Building Excel Reports in Python","Cache API calls behind an Excel report: requests-cache CachedSession with per-URL expiry, a simple file cache keyed by request, only caching 200s, forced refresh, and an as-of note in the workbook.","cache-api-responses-when-building-excel-reports","advanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fcache-api-responses-when-building-excel-reports\u002Findex",[2117,2118],"data-sources","reliable-jobs","how-to","exRTdVKt9wm81_qa8iL3o3x1ofAo_9olcQWuR6_4Tlw",[2122,2125],{"title":17,"path":2123,"stem":2124,"children":-1},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases","advanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Findex",{"title":2126,"path":2127,"stem":2128,"children":-1},"Compare an Excel File with a Database Table","\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fcompare-an-excel-file-with-a-database-table","advanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fcompare-an-excel-file-with-a-database-table\u002Findex",[2130,2132],{"path":2131,"title":2107},"\u002Fadvanced-data-transformation-and-cleaning",{"path":2123,"title":17},[],[2135,2138,2139,2142,2145,2148],{"path":2136,"title":2137,"kind":2119},"\u002Fautomating-reporting-workflows\u002Fpublishing-excel-reports-to-cloud-storage\u002Fread-an-excel-file-directly-from-amazon-s3-with-pandas","Read an Excel File Directly from Amazon S3 with pandas",{"path":2127,"title":2126,"kind":2119},{"path":2140,"title":2141,"kind":2119},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fdownload-new-excel-files-from-an-sftp-server","Download New Excel Files from an SFTP Server",{"path":2143,"title":2144,"kind":2119},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fexport-a-large-database-table-to-excel-in-chunks","Export a Large Database Table to Excel in Chunks",{"path":2146,"title":2147,"kind":2119},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fexport-database-lookup-tables-as-excel-dropdown-lists","Export Database Lookup Tables as Excel Dropdown Lists",{"path":2149,"title":2150,"kind":2119},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fexport-sql-query-results-to-excel-with-python","Export SQL Query Results to Excel with Python",1790972096564]