Fix UnicodeDecodeError When Reading CSV Files Saved by Excel

Someone exports a sheet from Excel with Save As › CSV and sends it over. pd.read_csv("export.csv") fails at once:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfc in position 22: invalid start byte

or, for a different export option, can't decode byte 0xff in position 0. Python reads text files as UTF-8 by default, and Excel offers several "CSV" and text formats, most of which are not UTF-8. The byte in the message is a strong clue to which one you have. This page shows how to identify the format from the first bytes, read each correctly, and avoid the two "fixes" that make the error go away while quietly corrupting the data. It belongs to Troubleshooting Common Python Excel Errors.

Prerequisites

pip install pandas charset-normalizer

The three sample files contain the same customers with accented names and euro prices, saved the way Excel saves its three text formats on a Western-European Windows machine: CSV (Comma delimited), CSV UTF-8 (Comma delimited) and Unicode Text.

Step 1 — Read the first bytes

The quickest diagnosis needs no library: look at the start of the file.

from pathlib import Path

for name in ("excel_csv.csv", "excel_csv_utf8.csv", "excel_unicode.txt"):
    head = Path(name).read_bytes()[:16]
    print(f"{name:20} {head.hex(' ')}")

Three patterns, three formats:

• ef bb bf at the start is the UTF-8 byte order mark — CSV UTF-8.
• ff fe at the start is the UTF-16 little-endian byte order mark — Unicode Text, which is also tab-separated despite the name people give it.
• No marker, and bytes such as fc (ü), e9 (é) or 80 (€) later on, means a single-byte Windows code page — plain CSV, written in the code page of the machine that saved it, usually Windows-1252 (cp1252) in Western Europe and the Americas.

Step 2 — Read each format correctly

import pandas as pd

ansi = pd.read_csv("excel_csv.csv", encoding="cp1252")
utf8 = pd.read_csv("excel_csv_utf8.csv", encoding="utf-8-sig")
utf16 = pd.read_csv("excel_unicode.txt", encoding="utf-16", sep="\t")
for label, df in [("cp1252", ansi), ("utf-8-sig", utf8), ("utf-16", utf16)]:
    print(f"{label:10} {list(df.columns)}  {df.iloc[0].tolist()}")

All three now produce the same frame: Müller GmbH, Köln, €1250. The UTF-16 file needs both the encoding and sep="\t"; with only the encoding it reads as one column. The utf-8-sig codec decodes UTF-8 and strips the byte order mark if present. pandas actually strips a leading BOM even with the default utf-8, but Python's csv module and plain open() do not, which leaves the first header as '\ufeffCustomer' and makes row["Customer"] raise a KeyError:

import csv

with open("excel_csv_utf8.csv", encoding="utf-8", newline="") as fh:
    print(next(csv.reader(fh))[0].encode("unicode_escape"))
with open("excel_csv_utf8.csv", encoding="utf-8-sig", newline="") as fh:
    print(next(csv.reader(fh))[0])

Step 3 — Avoid the two silent fixes

Two popular answers make the error disappear without making the data right:

latin = pd.read_csv("excel_csv.csv", encoding="latin-1")
replaced = pd.read_csv("excel_csv.csv", encoding_errors="replace")
print("latin-1 :", latin.iloc[0].tolist())
print("replace :", replaced.iloc[0].tolist())

latin-1 never raises — it maps every byte to some character — so accented letters come out right, but the euro sign (byte 0x80 in Windows-1252) becomes the invisible control character \x80, and so do curly quotes and dashes. encoding_errors="replace" keeps reading as UTF-8 and substitutes � for every byte it cannot decode, so Köln becomes K�ln and the original letter is gone for good. Both produce plausible-looking frames that fail later, in a lookup that never matches or a report that prints garbage.

Step 4 — When you do not know the encoding

For files from many sources, try the likely encodings in order of strictness and keep the first that decodes cleanly. UTF-8 is strict — Windows-1252 text almost never decodes as valid UTF-8 by accident — so trying it first is safe:

def read_excel_csv(path, candidates=("utf-8-sig", "cp1252")):
    head = Path(path).read_bytes()[:4]
    if head.startswith((b"\xff\xfe", b"\xfe\xff")):
        return pd.read_csv(path, encoding="utf-16", sep="\t"), "utf-16"
    for enc in candidates:
        try:
            return pd.read_csv(path, encoding=enc), enc
        except UnicodeDecodeError:
            continue
    raise ValueError(f"{path}: none of {candidates} decoded the file")

for name in ("excel_csv.csv", "excel_csv_utf8.csv", "excel_unicode.txt"):
    df, used = read_excel_csv(name)
    print(f"{name:20} read as {used:10} {df.iloc[0, 0]}")

Note that cp1252 itself rarely fails — only five byte values are undefined in it — so it must come last; put the code page your senders actually use there (cp1250 for Central Europe, cp1251 for Cyrillic). Statistical detectors such as charset-normalizer can help with unknown files, but treat their answer as a guess: on this short sample it labelled the Windows-1252 file as mac_latin2, an old Macintosh encoding that turns ü into a different letter altogether. Short files simply do not contain enough text for statistics to be reliable. Detection is no substitute for knowing where files come from.

Step 5 — Fix it at the source

The durable fix is to ask for CSV UTF-8 (Comma delimited) when you can, which every current Excel offers in the Save As list, or better, for the .xlsx itself, which has no encoding question at all — pd.read_excel reads text as Unicode. If you produce CSVs that go back to Excel users, write them with encoding="utf-8-sig" so Excel decodes them correctly, as explained in Write CSV Files That Open Correctly in Excel. European exports often bring a second problem with them — semicolons and decimal commas — covered in Read CSV Exports with Semicolons and Decimal Commas.

Step 6 — Repair text that was already decoded wrongly

Sometimes the damage happened upstream: a system read a UTF-8 file as Windows-1252, saved the result, and the sheet now contains Müller and Café. That pattern — Ã followed by another character — is the signature of UTF-8 bytes decoded with a single-byte code page, and it can usually be reversed by doing the opposite round trip:

garbled = pd.Series(["Müller GmbH", "Café Señor", "Plain ASCII"])

def undo_mojibake(text):
    try:
        return text.encode("cp1252").decode("utf-8")
    except (UnicodeEncodeError, UnicodeDecodeError):
        return text

print(garbled.map(undo_mojibake).tolist())

Re-encoding with cp1252 recovers the original bytes, and decoding them as UTF-8 recovers the text; values that were never garbled raise an error in one direction or the other and are returned unchanged. Check a sample by eye before applying it to a whole column, because a value that legitimately contains Ã would be "repaired" too. The ftfy package automates this and handles many more patterns, as described in Fix Garbled Characters in Excel Data.

Performance notes

The encoding barely affects reading speed; decoding cp1252 or UTF-16 costs about the same as UTF-8. The try-in-order helper reads the file a second time when the first encoding fails, which only matters for very large files — for those, decode just the first megabyte to choose the encoding (Path(p).read_bytes()[:1_000_000].decode(enc)), then read the whole file once with the winner. A UTF-8 character can be cut in half at the one-megabyte boundary, so trim a few bytes from the end of the sample, or decode it with the incremental decoder from the codecs module, before treating a failure as proof.

Common pitfalls

• Using latin-1 because it never fails. It mis-decodes €, curly quotes and dashes from Windows files.
• encoding_errors="replace". Destroys characters; use it only to locate the problem, never in production.
• Ignoring the BOM in the csv module. The first column name gains \ufeff; use utf-8-sig.
• Reading Unicode Text as CSV. It is UTF-16 and tab-separated.
• Assuming one code page. A Polish or Greek colleague's "CSV" uses cp1250 or cp1253; ask, or record the source per sender.

Conclusion

A UnicodeDecodeError on a CSV from Excel means the file is not plain UTF-8. Look at the first bytes: EF BB BF is CSV UTF-8 (read with utf-8-sig), FF FE is Unicode Text (UTF-16 with tabs), and no marker means the Windows code page, usually cp1252. Read with that encoding, try strict encodings before permissive ones when the source is unknown, never paper over the error with latin-1 or errors="replace", and ask for CSV UTF-8 or the .xlsx itself wherever you can.

Frequently asked questions

Why do I get "utf-8 codec can't decode byte 0xfc" reading an Excel CSV? The file was saved with Excel's plain CSV option, which uses the Windows code page rather than UTF-8. Byte 0xfc is ü in Windows-1252. Read it with pd.read_csv(path, encoding="cp1252").

What does "can't decode byte 0xff in position 0" mean? The file starts with FF FE, the UTF-16 byte order mark, which Excel's Unicode Text format writes. Read it with encoding="utf-16" and sep="\t", since that format is tab-separated.

Why does my first column name start with \ufeff? The file is CSV UTF-8 with a byte order mark, and it was read with plain utf-8 by the csv module or open(). Use encoding="utf-8-sig", which removes the mark.

Is encoding="latin-1" a safe fix? No. It never raises an error, but it decodes Windows-specific characters such as the euro sign, curly quotes and dashes as invisible control characters. Use cp1252 for files saved by Western-European Excel.

Related

• Up one level: Troubleshooting Common Python Excel Errors — fixes for common failures.
• Fix Garbled Characters in Excel Data — repairing text that was decoded wrongly earlier.
• Convert CSV Files to Excel with Python — once the CSV reads cleanly.
• Remove Non-Printable Characters from Excel Data — cleaning up stray control characters.