Schedule Excel Reports with Celery Beat

Web applications built with Django or Flask often already run Celery for background work — sending emails, processing uploads. When the same application needs a weekly Excel report, adding it to Celery keeps everything in one place and avoids a second scheduling system: the report uses the application's models and settings, runs on the workers that are already monitored, and is scheduled by Celery beat, the component that puts tasks on the queue at set times. This page, part of Scheduling Python Excel Scripts with Cron, sets up the task and the schedule, tests them with a real worker, and covers the operational rule that catches most teams: only one beat may run.

Prerequisites

pip install celery pandas openpyxl
# plus a broker in production, usually Redis or RabbitMQ

In production Celery needs a message broker, and the examples would use redis://…. Here the app uses Celery's in-memory broker and result backend, which work within a single process — enough to run a real worker thread in a test and exercise the task exactly as a deployed worker would. sales_source.csv stands in for the application's database.

Step 1 — The Celery app and its time zone

Configure the time zone once on the app. Beat evaluates every crontab entry in that zone:

from celery import Celery

app = Celery("reports", broker="memory://", backend="cache+memory://")
app.conf.update(
    timezone="Europe/London",
    enable_utc=True,
    task_acks_late=True,
    worker_prefetch_multiplier=1,
)
print(app.conf.timezone, app.conf.enable_utc)

enable_utc=True keeps internal timestamps in UTC while timezone decides how schedules are read, so "07:00" means 07:00 in London all year. task_acks_late makes the worker acknowledge a task only after it finishes, so a report interrupted by a worker restart is delivered again rather than lost; worker_prefetch_multiplier=1 stops one worker reserving several long reports at once. In Django these settings live in settings.py with a CELERY_ prefix.

Step 2 — A report task that retries and stops

The task builds the workbook and returns where it saved it. Transient failures — the database restarting, a network share briefly unavailable — are retried automatically with exponential backoff; time limits stop a stuck run:

from datetime import date
from pathlib import Path
import pandas as pd

@app.task(bind=True, autoretry_for=(ConnectionError, TimeoutError), retry_backoff=1,
          retry_backoff_max=300, retry_jitter=False, max_retries=4,
          soft_time_limit=600, time_limit=660)
def build_weekly_report(self, report_date: str | None = None) -> str:
    report_date = date.fromisoformat(report_date) if report_date else date.today()
    rows = load_sales(report_date)
    out = Path("reports") / f"weekly_sales_{report_date:%Y-%m-%d}.xlsx"
    out.parent.mkdir(exist_ok=True)
    rows.to_excel(out, index=False)
    return str(out)

attempts = {"n": 0}

def load_sales(report_date):
    attempts["n"] += 1
    if attempts["n"] < 3:
        raise ConnectionError("database is restarting")
    return pd.read_csv("sales_source.csv")

print(build_weekly_report.name, build_weekly_report.max_retries)

autoretry_for lists the exceptions worth retrying; anything else — a KeyError from a renamed column — fails at once, because retrying cannot fix it. retry_backoff=1 waits 1, 2, 4, 8 seconds between attempts, capped at retry_backoff_max; in production leave retry_jitter on so many failing tasks do not retry in lockstep. soft_time_limit raises an exception inside the task after ten minutes so it can clean up, and time_limit kills it a minute later if it does not. The task takes the report date as an optional ISO string — task arguments travel as JSON, and a string survives that where a date object would not.

Step 3 — Test it with a real worker

celery.contrib.testing.worker.start_worker runs a worker in a background thread against the app's broker, so the test goes through serialisation, the queue and the retry machinery — not just a function call:

import time
from celery.contrib.testing.worker import start_worker

started = time.perf_counter()
with start_worker(app, perform_ping_check=False, pool="solo", loglevel="WARNING"):
    result = build_weekly_report.delay("2026-09-28")
    path = result.get(timeout=30)
print(path, "after", attempts["n"], "attempts in", f"{time.perf_counter() - started:.1f}s")
print(pd.read_excel(path)["Region"].tolist())

The first two attempts failed with the simulated database error, Celery retried after one and then two seconds, and the third attempt built the workbook — a few seconds in all, three of them spent in the backoff waits. This is the behaviour a deployed worker shows, which a plain function call cannot demonstrate. In a pytest suite the same pattern works with the celery_app and celery_worker fixtures that Celery provides.

Step 4 — The beat schedule

Beat reads beat_schedule: a dictionary of entries, each naming a task, a schedule and optional arguments. crontab takes the familiar fields by name:

from celery.schedules import crontab

app.conf.beat_schedule = {
    "weekly-sales-monday-0700": {
        "task": build_weekly_report.name,
        "schedule": crontab(minute=0, hour=7, day_of_week="mon"),
        "options": {"expires": 3 * 3600},
    },
    "daily-stock-weekdays-0630": {
        "task": build_weekly_report.name,
        "schedule": crontab(minute=30, hour=6, day_of_week="mon-fri"),
        "args": [None],
    },
}

from datetime import datetime
from zoneinfo import ZoneInfo

schedule = app.conf.beat_schedule["weekly-sales-monday-0700"]["schedule"]
schedule.app = app
last = datetime(2026, 10, 19, 7, 0, tzinfo=ZoneInfo("Europe/London"))
for _ in range(3):
    start, delta, _now = schedule.remaining_delta(last)
    last = start + delta
    print(last.astimezone(ZoneInfo("Europe/London")).strftime("%a %d %b %H:%M %Z"))

remaining_delta is what beat uses to decide when an entry is next due; computing a few runs from it confirms the entry means Monday at 07:00 London time, before and after the clocks change on 25 October. expires discards the message if no worker has picked it up within three hours, so a backlog after an outage does not produce a stale Monday report on Wednesday. Start beat as its own process in production with celery -A reports beat, next to the workers started with celery -A reports worker. Beat only sends messages; it never runs the report itself, so it needs almost no resources.

Step 5 — Run exactly one beat

Beat has no built-in coordination: every beat process that is running sends every scheduled task. Two beats — one per web container, or an old one left running after a deployment — means every report is built and emailed twice. Run beat as a single dedicated process or container, separate from the scalable workers, and make sure deployments stop the old one before starting the new one:

celery -A reports worker --loglevel=INFO --concurrency=2     # scale these as needed
celery -A reports beat --loglevel=INFO                        # exactly one of these

For schedules that non-developers should change without a deployment, django-celery-beat stores entries in the database with an admin interface; beat is started with its DatabaseScheduler, and the one-beat rule still applies. Making the report task idempotent — writing to a dated file name and checking whether today's report already went out before emailing — turns an accidental duplicate into a harmless no-op, as described in Prevent Overlapping Runs of an Excel Report Job.

Performance notes

Workbook building is CPU-bound and can take minutes for large reports, which is long for a worker that also serves quick web tasks. Route report tasks to their own queue — task_routes = {"reports.build_weekly_report": {"queue": "reports"}} — and run a separate worker for that queue with a concurrency of one or two, so a Monday-morning batch of reports cannot delay password-reset emails. Keep the result small: return the path or the object-storage key of the workbook, never the workbook bytes, which would be stored in the result backend for every run.

Common pitfalls

• More than one beat. Every beat sends every task; run exactly one.
• Retrying every exception. List transient exceptions in autoretry_for and let data errors fail immediately.
• Passing dates and DataFrames as arguments. Task arguments are serialised; pass ISO strings and IDs.
• No time limits. A hung report holds a worker forever; set soft and hard limits.
• Forgetting the time zone. Without timezone, crontab entries are read in UTC.

Conclusion

Set the Celery app's timezone, write the report as a task with autoretry_for on transient errors, exponential backoff, soft and hard time limits and a date string argument, and test it through a real in-process worker with start_worker. Declare the schedule in beat_schedule with crontab(minute=0, hour=7, day_of_week="mon"), check its next runs, give messages an expiry, and run exactly one beat process alongside the workers.

Frequently asked questions

How do I schedule a task every Monday at 7am with Celery? Add an entry to app.conf.beat_schedule with "schedule": crontab(minute=0, hour=7, day_of_week="mon") and the task's name, set app.conf.timezone to your zone, and run celery beat alongside the workers.

Why is my Celery beat task running twice? More than one beat process is running — for example one started in every web container. Each beat sends every scheduled task, so run exactly one.

How do I make a Celery task retry automatically? Pass autoretry_for with the exception types to retry, plus retry_backoff and max_retries, to the task decorator. Celery then re-queues the task with increasing delays when one of those exceptions is raised.

How can I test a Celery task without a broker? Use Celery's in-memory broker and the start_worker helper from celery.contrib.testing.worker, which runs a real worker thread in the test process. In pytest, the celery_app and celery_worker fixtures do the same.

Related

• Up one level: Scheduling Python Excel Scripts with Cron — every way to run reports on a schedule.
• Generate Large Excel Exports in a Background Job — the same Celery set-up for on-demand exports.
• Schedule Recurring Excel Reports with APScheduler — scheduling inside a single process.
• Export a Django Queryset to Excel — building the workbook from application data.