[{"data":1,"prerenderedAt":156},["ShallowReactive",2],{"topic:data-sources":3},[4,13,20,26,32,38,43,49,54,59,64,70,75,80,85,90,95,100,105,110,115,121,127,133,139,145,150],{"path":5,"title":6,"description":7,"type":8,"topics":9,"kind":12},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases","Moving Data Between Excel and Databases","Excel as the interface, the database as the source of truth: query to workbook with SQLAlchemy, load a spreadsheet into a table safely, keep types intact in both directions, and refresh on a schedule.","guide",[10,11],"data-sources","pandas",null,{"path":14,"title":15,"description":16,"type":17,"topics":18,"kind":12},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fcache-api-responses-when-building-excel-reports","Cache API Responses When Building Excel Reports","Stop a report script from calling the same API hundreds of times while you develop it or rerun it: a transparent HTTP cache with requests-cache and expiry rules per endpoint, a hand-rolled file cache for responses that are not plain GETs, caching only successful responses, forcing a refresh for the production run, and recording in the workbook how old the cached data was.","how-to",[10,19],"reliable-jobs",{"path":21,"title":22,"description":23,"type":17,"topics":24,"kind":12},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fcompare-an-excel-file-with-a-database-table","Compare an Excel File with a Database Table","Find every difference between a spreadsheet and the database table it is supposed to match: normalise both sides so formatting noise does not count, outer-join on the key, list rows only in Excel or only in the database, compare fields with tolerances for numbers and dates, write a highlighted difference report, and preview the SQL that would bring the table in line.",[10,25],"data-validation",{"path":27,"title":28,"description":29,"type":17,"topics":30,"kind":12},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fdownload-new-excel-files-from-an-sftp-server","Download New Excel Files from an SFTP Server","Collect the spreadsheets a partner drops on an SFTP server with paramiko: connect with a pinned host key, list the folder with sizes and modification times, ignore partial uploads and files still being written, download only files not fetched before — or fetched before but changed since — into a temporary name and rename, check each workbook opens, keep a small state file, and test it all against a local SFTP server.",[10,31],"scheduling",{"path":33,"title":34,"description":35,"type":17,"topics":36,"kind":12},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fexport-a-large-database-table-to-excel-in-chunks","Export a Large Database Table to Excel in Chunks","Export a database table too big to load at once into Excel with flat memory use: stream rows in chunks from SQLAlchemy with a server-side cursor, write them straight to disk with XlsxWriter constant_memory mode, roll over to a new sheet before the 1,048,576-row limit, keep dates and numbers typed, and check the row counts and memory afterwards.",[10,37],"large-files",{"path":39,"title":40,"description":41,"type":17,"topics":42,"kind":12},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fexport-database-lookup-tables-as-excel-dropdown-lists","Export Database Lookup Tables as Excel Dropdown Lists","Build an Excel input template whose dropdowns come straight from the database: read the active cost centres, projects and expense types, write them to a hidden lists sheet, give each list a named range, attach data validation to the input columns, show codes with their descriptions while storing a code that maps back cleanly, refresh the lists in an existing template, and check the returned file against the database.",[10,25],{"path":44,"title":45,"description":46,"type":17,"topics":47,"kind":12},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fexport-sql-query-results-to-excel-with-python","Export SQL Query Results to Excel with Python","Run a parameterised query with SQLAlchemy and write the result to a formatted workbook: one sheet per result set, correct date and money formats, sized columns, and chunked reads for big results.",[10,48,11],"writing-excel",{"path":50,"title":51,"description":52,"type":17,"topics":53,"kind":12},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Ffetch-api-data-into-excel-with-python-requests","Fetch API Data into Excel with Python requests","Pull JSON from an HTTP API into a workbook: a session with timeouts and retries, pagination that terminates, json_normalize for nested records, and a cache so a rerun does not hammer the endpoint.",[10,48],{"path":55,"title":56,"description":57,"type":17,"topics":58,"kind":12},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Ffetch-paginated-api-data-into-excel","Fetch Paginated API Data into Excel","Pull every record from a REST API that returns results a page at a time and write them to Excel: page-number, offset and cursor pagination, following next links, handling 429 rate limits with Retry-After, retrying transient errors, flattening nested JSON with json_normalize, de-duplicating across pages, and checking you received as many records as the API says exist.",[10],{"path":60,"title":61,"description":62,"type":17,"topics":63,"kind":12},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fgenerate-sql-insert-statements-from-an-excel-sheet","Generate SQL INSERT Statements from an Excel Sheet","Turn an Excel sheet into a reviewable .sql script for a database administrator to run: map pandas types to SQL literals, quote text safely by doubling single quotes, write NULL for blanks, dates as ISO literals, batch rows into multi-row INSERTs inside one transaction, add an upsert variant for PostgreSQL, and prove the script by running it against a scratch database and comparing every value with the sheet.",[10,48],{"path":65,"title":66,"description":67,"type":17,"topics":68,"kind":12},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fload-an-excel-file-into-a-sql-database-with-pandas","Load an Excel File into a SQL Database with pandas","Take a spreadsheet people filled in and get it into a table safely: normalise the columns, pin the types, stage the rows, validate them, then merge in one transaction that either succeeds or leaves nothing behind.",[10,69,11],"reading-excel",{"path":71,"title":72,"description":73,"type":17,"topics":74,"kind":12},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fload-excel-data-into-mongodb-with-python","Load Excel Data into MongoDB with Python","Turn spreadsheet rows into well-shaped MongoDB documents with pandas and PyMongo: nested fields from dotted column names, real dates and numbers instead of strings, blanks left out rather than stored as NaN, upserts on a business key with bulk_write so reloading a file is safe, a unique index, and a check of what was inserted, updated and skipped.",[10],{"path":76,"title":77,"description":78,"type":17,"topics":79,"kind":12},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fload-excel-into-sqlite-with-a-defined-schema","Load Excel into SQLite with a Defined Schema","Load Excel sheets into SQLite tables that you designed — proper types, primary keys, NOT NULL and CHECK constraints, foreign keys and STRICT typing — instead of whatever pandas.to_sql guesses: create the schema first, convert dates and decimals deliberately, insert in one transaction so a bad row rolls the whole load back, and report exactly which rows broke which rule.",[10,11],{"path":81,"title":82,"description":83,"type":17,"topics":84,"kind":12},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fquery-a-database-for-ids-listed-in-an-excel-sheet","Query a Database for IDs Listed in an Excel Sheet","Someone sends a spreadsheet of customer or order IDs and wants the matching records. Clean the IDs, pass them to SQL safely as parameters, split long lists into chunks the database driver accepts, switch to a temporary table for very long lists, and report the IDs that were not found — in the order the sheet listed them.",[10,11],{"path":86,"title":87,"description":88,"type":17,"topics":89,"kind":12},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fquery-excel-files-with-sql-using-duckdb","Query Excel Files with SQL Using DuckDB","Run SQL directly against .xlsx files with DuckDB's excel extension — read a range below a report header, stop at the first empty row, handle text in numeric columns, combine several workbooks, join Excel with pandas DataFrames and write query results back to a workbook.",[10,37],{"path":91,"title":92,"description":93,"type":17,"topics":94,"kind":12},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fread-excel-attachments-into-a-database-nightly","Read Excel Attachments into a Database Nightly","Load the spreadsheets that arrive by email into a database every night without duplicates or half-loaded files: take the saved attachments from an inbox folder, skip files already loaded by their content hash, validate each workbook's columns and values, write each file in its own transaction with an upsert on the business key, record every load in a log table, move rejected files to a quarantine folder, and report the night's results.",[10,31],{"path":96,"title":97,"description":98,"type":17,"topics":99,"kind":12},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Frefresh-an-excel-report-from-a-database-on-a-schedule","Refresh an Excel Report from a Database on a Schedule","Keep a workbook current without anyone asking: an incremental query with a watermark, an atomic write nobody can catch half-finished, a freshness stamp readers can see, and a check that the refresh actually ran.",[10,31],{"path":101,"title":102,"description":103,"type":17,"topics":104,"kind":12},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fscrape-a-web-page-table-into-excel-with-pandas","Scrape a Web Page Table into Excel with pandas","Turn an HTML table on a web page into a clean Excel sheet. Fetch the page with requests, pick the right table out of several with read_html, flatten two-row headers, strip footnote markers and dashes so numbers become numbers, keep the links, and save the result with its source and time.",[10,11],{"path":106,"title":107,"description":108,"type":17,"topics":109,"kind":12},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fsync-google-sheets-with-excel-using-python","Sync Google Sheets with Excel Using Python","Move data both ways between Google Sheets and .xlsx: service-account auth with gspread, reading a tab into pandas, writing back, exporting a whole spreadsheet and scheduling it.",[10],{"path":111,"title":112,"description":113,"type":17,"topics":114,"kind":12},"\u002Fadvanced-data-transformation-and-cleaning\u002Fmoving-data-between-excel-and-databases\u002Fupsert-excel-rows-into-a-database-table","Upsert Excel Rows into a Database Table with Python","Load a maintained Excel sheet into a database table without duplicating rows — insert new keys, update changed ones and leave unchanged rows alone. Use a staging table and INSERT … ON CONFLICT, a to_sql method for smaller loads, and checks for duplicate keys and rows deleted from the sheet.",[10,11],{"path":116,"title":117,"description":118,"type":17,"topics":119,"kind":12},"\u002Fadvanced-data-transformation-and-cleaning\u002Fworking-with-large-excel-files-in-python\u002Fstream-excel-rows-into-a-database-with-openpyxl","Stream Excel Rows into a Database with openpyxl","Load a large Excel sheet into a database without ever holding it in memory — read rows with openpyxl's read-only mode, convert and validate each row, insert in batches inside a transaction, divert bad rows to a rejects table and make the load safe to rerun.",[37,10,120],"openpyxl",{"path":122,"title":123,"description":124,"type":17,"topics":125,"kind":12},"\u002Fautomating-reporting-workflows\u002Fbuilding-finance-and-operations-reports\u002Fbuild-a-cash-position-report-from-bank-statements","Build a Cash Position Report from Bank Statements","Produce a daily cash position across several banks and currencies from their statement exports with pandas: map each bank's own CSV layout to one standard, prove every statement adds up from opening to closing balance, convert to the reporting currency, compare each account with its minimum balance and overdraft limit, and write a one-page position with the largest movements of the day.",[126,10],"report-layouts",{"path":128,"title":129,"description":130,"type":17,"topics":131,"kind":12},"\u002Fautomating-reporting-workflows\u002Femailing-excel-reports-with-smtplib\u002Fdownload-excel-attachments-from-an-email-inbox","Download Excel Attachments from an Email Inbox with Python","Pick up the spreadsheets suppliers and colleagues email in, without anyone saving them by hand. Search a mailbox over IMAP, save only genuine Excel attachments under safe file names, decode international file names, skip messages already processed, and move handled mail to a folder so the job can run every few minutes.",[132,10],"email-delivery",{"path":134,"title":135,"description":136,"type":17,"topics":137,"kind":12},"\u002Fautomating-reporting-workflows\u002Fpublishing-excel-reports-to-cloud-storage\u002Fread-an-excel-file-directly-from-amazon-s3-with-pandas","Read an Excel File Directly from Amazon S3 with pandas","Load a workbook from S3 into a DataFrame without a temporary file: BytesIO reads, paginated prefix scans, readable error translation, and a Parquet cache for repeat runs.",[10,138,69,11],"publishing",{"path":140,"title":141,"description":142,"type":17,"topics":143,"kind":12},"\u002Fgetting-started-with-python-excel-automation\u002Fhandling-excel-file-formats-and-conversions\u002Fconvert-xml-to-excel-with-pandas","Convert XML to Excel with pandas read_xml","Turn an XML export — orders, invoices, a feed from a legacy system — into a workbook with one sheet per record type. Use pd.read_xml with XPath and namespaces, keep nested line items linked to their parent record, preserve codes with leading zeros, and stream very large files with iterparse.",[144,10,11],"file-formats",{"path":146,"title":147,"description":148,"type":17,"topics":149,"kind":12},"\u002Fgetting-started-with-python-excel-automation\u002Fhandling-excel-file-formats-and-conversions\u002Fread-an-excel-file-from-a-url-or-bytes-in-python","Read an Excel File from a URL or Bytes in Python","Load a workbook straight from HTTP, S3, a database blob or an upload: BytesIO, content-type checks, authentication, retries, streaming downloads and caching.",[69,10],{"path":151,"title":152,"description":153,"type":17,"topics":154,"kind":12},"\u002Fgetting-started-with-python-excel-automation\u002Fusing-python-in-excel\u002Fget-external-data-into-python-in-excel","Get External Data into Python in Excel with Power Query","Python in Excel cannot read files or call the internet, so bring CSVs, databases and web data in with Power Query, then analyse them with xl() — tables versus connection-only queries, refresh order, parameters and size limits.",[155,10],"python-in-excel",1790972052271]