Guide

Automating Reporting WorkflowsDeep dive

Upload an Excel Report to Amazon S3 with boto3

Q: Do I have to write the file to disk before uploading?

No. Write the workbook into an io.BytesIO buffer and pass it to upload_fileobj. That avoids temporary files entirely, which matters in a container with a read-only or ephemeral filesystem.

Publish a generated .xlsx to S3 from Python: upload straight from memory, set the right content type, make the write atomic, verify it, and share it with a presigned URL.

S3 is the natural home for a generated report that other systems also consume: cheap, versioned, and reachable from anything. Getting a workbook there is one boto3 call — and then four details decide whether the result is reliable. This guide covers uploading straight from memory, setting the content type so browsers treat the object as a spreadsheet, making the write atomic so readers never catch a half-uploaded file, verifying the transfer, and sharing it without opening the bucket to the world. It is the object-storage path from Publishing Excel Reports to Cloud Storage.

Prerequisites

Bash

pip install boto3 pandas xlsxwriter

Credentials should come from the environment rather than the code. On EC2, ECS or Lambda that means an attached IAM role and nothing in the script at all; elsewhere, environment variables the scheduler injects:

Bash

export AWS_REGION=eu-west-1
export REPORT_BUCKET=acme-reports

The minimum policy the job needs is narrow. Grant PutObject, GetObject and DeleteObject on the report prefix only — the delete is required because the atomic pattern cleans up its staging object:

Json

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["s3:PutObject", "s3:GetObject", "s3:DeleteObject"],
    "Resource": "arn:aws:s3:::acme-reports/reports/*"
  }]
}

Step 1 — Upload straight from memory

Building the workbook in a BytesIO avoids a temporary file entirely, which matters in a container whose filesystem is read-only or ephemeral:

Python

import io
import os
import boto3
import pandas as pd

XLSX = "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"

s3 = boto3.client("s3")
bucket = os.environ["REPORT_BUCKET"]

df = pd.DataFrame({
    "region": ["North", "South", "West"],
    "revenue": [159.92, 247.50, 137.44],
})

buffer = io.BytesIO()
with pd.ExcelWriter(buffer, engine="xlsxwriter") as writer:
    df.to_excel(writer, sheet_name="Summary", index=False)
buffer.seek(0)

s3.upload_fileobj(
    buffer, bucket, "reports/regional-latest.xlsx",
    ExtraArgs={"ContentType": XLSX},
)
print("uploaded")

Two details carry weight. The buffer.seek(0) rewinds the cursor after writing — without it, upload_fileobj reads from the end and uploads zero bytes, producing an empty object with a perfectly successful response.

And the content type is not cosmetic. Omit it and S3 stores the object as binary/octet-stream; a browser following a link then downloads it as a nameless blob instead of handing it to Excel. Set it once, in a constant, so it cannot drift between call sites.

Step 2 — Make the write atomic

A direct upload to the final key leaves a growing, partially written object visible for the whole transfer. Anyone who opens the link in that window gets a corrupt file. Stage, then promote:

Python

import io
import boto3

XLSX = "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"
s3 = boto3.client("s3")

def publish(buffer, bucket, key):
    """Upload atomically: stage, promote with a server-side copy, clean up."""
    staging = f"{key}.uploading"
    buffer.seek(0)

    s3.upload_fileobj(buffer, bucket, staging, ExtraArgs={"ContentType": XLSX})
    try:
        s3.copy_object(
            Bucket=bucket,
            Key=key,
            CopySource={"Bucket": bucket, "Key": staging},
            ContentType=XLSX,
            MetadataDirective="REPLACE",
        )
    finally:
        s3.delete_object(Bucket=bucket, Key=staging)

    return f"s3://{bucket}/{key}"

MetadataDirective="REPLACE" is required whenever you set ContentType on the copy — with the default COPY, the argument is ignored and the promoted object inherits the source metadata. The finally guarantees the staging object is removed even when the copy fails, so a failed run does not leave litter that the next one trips over.

Write the dated key and the stable key in one function, so history and links stay in step:

Python

from datetime import date

def publish_report(buffer, bucket, name, on=None):
    on = on or date.today()
    dated = f"reports/{on:%Y/%m}/{name}-{on:%Y-%m}.xlsx"
    latest = f"reports/{name}-latest.xlsx"

    publish(buffer, bucket, dated)

    # Server-side copy: the bytes never leave S3 for this second write.
    s3.copy_object(
        Bucket=bucket, Key=latest,
        CopySource={"Bucket": bucket, "Key": dated},
        ContentType=XLSX, MetadataDirective="REPLACE",
    )
    return dated, latest

The second write is a server-side copy rather than a second upload — no bytes cross the network again, which on a large report is the difference between one transfer and two.

Step 3 — Verify the object

A successful call means the request was accepted. Confirm the object is what you sent:

Python

import hashlib
import boto3

s3 = boto3.client("s3")

def verify(buffer, bucket, key):
    """Check the stored object matches the buffer that was uploaded."""
    buffer.seek(0)
    payload = buffer.read()

    head = s3.head_object(Bucket=bucket, Key=key)

    if head["ContentLength"] != len(payload):
        raise RuntimeError(
            f"size mismatch: sent {len(payload)}, stored {head['ContentLength']}"
        )

    etag = head["ETag"].strip('"')
    if "-" in etag:
        # Multipart upload: the ETag is a digest of digests, not the file MD5.
        return "size-verified"

    if etag != hashlib.md5(payload).hexdigest():
        raise RuntimeError("checksum mismatch — the stored object is corrupt")
    return "checksum-verified"

The multipart branch is the part people get wrong. boto3 switches to multipart automatically above a threshold, and for those objects the ETag has a -N suffix and is not the file's MD5 — comparing it directly reports corruption on every large file. If you want a real checksum on large objects, ask S3 for one explicitly:

Python

s3.upload_fileobj(
    buffer, bucket, key,
    ExtraArgs={"ContentType": XLSX, "ChecksumAlgorithm": "SHA256"},
)
head = s3.head_object(Bucket=bucket, Key=key, ChecksumMode="ENABLED")
print(head.get("ChecksumSHA256"))

Presigned URLs give time-limited access to a single object using your credentials, so the bucket stays private:

Python

import boto3

s3 = boto3.client("s3")

def share_link(bucket, key, hours=48, filename=None):
    """Time-limited download link for one object."""
    params = {"Bucket": bucket, "Key": key}
    if filename:
        # Force a friendly filename in the browser's save dialog.
        params["ResponseContentDisposition"] = f'attachment; filename="{filename}"'

    return s3.generate_presigned_url(
        "get_object", Params=params, ExpiresIn=hours * 3600
    )

url = share_link("acme-reports", "reports/regional-latest.xlsx",
                 hours=48, filename="Regional report August 2026.xlsx")

Two operational cautions. The link is a bearer token — anyone who has it can download until it expires, so keep the window short and treat it as sensitive. And a presigned URL cannot outlive the credentials that signed it; a URL signed by a role session with a one-hour lifetime stops working after that hour regardless of the ExpiresIn you asked for. For long-lived links, sign with a longer-lived identity or re-sign on each run.

Common pitfalls and fixes

Symptom	Cause	Fix
Object is 0 bytes	Buffer not rewound	`buffer.seek(0)` before uploading.
Browser downloads a nameless blob	Content type not set	Pass the spreadsheetml `ContentType`.
Content type lost on the copy	`MetadataDirective` left as `COPY`	Set it to `REPLACE`.
Checksum "mismatch" on large files	Multipart ETag is not an MD5	Compare size, or use `ChecksumAlgorithm`.
`AccessDenied` on cleanup	Policy grants no `DeleteObject`	Add it for the report prefix.
Presigned link expires early	Role session shorter than `ExpiresIn`	Sign with a longer-lived identity.
Readers get a truncated file	Uploaded straight to the final key	Stage and promote.
`NoCredentialsError` under cron	Environment differs from the shell	Use an instance role, or export credentials in the job.

Performance and scale notes

boto3 switches to multipart uploads automatically above a threshold and runs the parts concurrently, so large reports already parallelise without any work from you. The knobs live on TransferConfig:

Python

import boto3
from boto3.s3.transfer import TransferConfig

config = TransferConfig(
    multipart_threshold=16 * 1024 * 1024,   # start multipart at 16 MB
    multipart_chunksize=16 * 1024 * 1024,
    max_concurrency=8,
    use_threads=True,
)

s3.upload_fileobj(buffer, bucket, key,
                  ExtraArgs={"ContentType": XLSX}, Config=config)

Raising max_concurrency helps on a fast link and hurts on a constrained one, where parts start competing for the same bandwidth. Measure before tuning.

Three habits matter more than the knobs. Reuse the client — boto3.client("s3") builds a session and resolves credentials, so creating one per upload inside a loop is pure overhead:

Python

s3 = boto3.client("s3")            # once, at module level

for region in regions:
    publish(build_report(region), bucket, f"reports/{region}-latest.xlsx")

Copy server-side rather than re-uploading. Promoting a dated object to the latest key with copy_object moves no bytes over your connection at all.

Parallelise across reports, not within one. A fan-out job producing one workbook per region — the shape described in generating one Excel report per region — gains far more from a thread pool over regions than from tuning a single transfer:

Python

from concurrent.futures import ThreadPoolExecutor

def build_and_publish(region):
    buffer = build_report(region)                # returns a BytesIO
    return publish(buffer, bucket, f"reports/{region}-latest.xlsx")

with ThreadPoolExecutor(max_workers=6) as pool:
    for uri in pool.map(build_and_publish, ["north", "south", "west", "east"]):
        print("published", uri)

Threads are the right choice here rather than processes, because the work is network-bound and boto3 releases the GIL during I/O. Watch memory, though: six workers each holding a workbook buffer means six copies resident at once, which for large reports is the real constraint — the streaming techniques in writing large DataFrames with write-only mode keep each one smaller.

Conclusion

Publishing an Excel report to S3 is upload_fileobj plus four disciplines. Build the workbook in a BytesIO and rewind it. Set the spreadsheetml content type so browsers treat the object as a spreadsheet. Stage the upload under a temporary key and promote it with a server-side copy, so readers never see a partial file. Verify size — and a checksum when the object is not multipart. Then share with a short-lived presigned URL rather than making the bucket public, and let a dated key plus a stable latest key give you history and durable links at the same time.

Frequently asked questions

Do I have to write the file to disk before uploading? No. Write the workbook into an io.BytesIO buffer and pass it to upload_fileobj. That avoids temporary files entirely, which matters in a container with a read-only or ephemeral filesystem.

Which content type should an .xlsx have?application/vnd.openxmlformats-officedocument.spreadsheetml.sheet. Without it S3 serves the object as binary/octet-stream and browsers download it with a generic name instead of opening it as a spreadsheet.

Why does the ETag not match my file's MD5? Because the upload was multipart. For multipart objects the ETag is a digest of the part digests with a dash and the part count appended, so compare sizes instead, or enable S3's own checksum algorithms.

How do I share the report without making the bucket public? Generate a presigned URL. It grants time-limited access to one object using your credentials, so the bucket stays private and the link expires on its own.

Should each run overwrite the same key? Write a dated key for the history and copy it over a stable latest key for links. Enabling bucket versioning gives you a safety net on top, so an accidental overwrite is recoverable.

Up to the parent: Publishing Excel Reports to Cloud Storage — the naming, atomicity and retry patterns applied here.
Upload an Excel Report to SharePoint with Python — the same job against Microsoft Graph.
Write Excel Files to a Network Share from Python — the on-premise equivalent.
Retry a Failed Excel Report Job in Python — backoff for the transient upload failures.
Generate One Excel Report per Region in a Loop — the fan-out this upload step serves.