Download the scripts and tests (ZIP)
Python scripts, tests, and ready-to-run merge, join, and date practice files with expected results. No signup required.
A CSV export opens correctly in the application that created it, but your Python workflow stops with UnicodeDecodeError. The file may use a different text encoding. This guide converts a file from a known source encoding to UTF-8, checks its comma-separated records, and writes a separate file. It keeps IDs such as 001, quoted fields, and the original line endings.
Use this as a preparation step before merging CSV exports or joining two CSV files by key. It does not repair text that was already decoded incorrectly and saved with garbled characters.
Tested with Python 3.10 on Windows on September 14, 2026. The example data is synthetic; no customer files or external services are used.
Decide the source encoding first
Check the export application’s settings or ask whoever generated the file. An encoding is a rule for interpreting bytes. A successful decode is not proof that you chose the right rule: some encodings accept bytes that represent different characters elsewhere. Our script requires --from-encoding and never tries a list until one happens to succeed.
| Confirmed source | Argument | What to check |
|---|---|---|
| Windows-1252 export | --from-encoding cp1252 |
Accented names and the euro sign |
| UTF-8, optionally with its signature | --from-encoding utf-8-sig |
A leading signature is removed |
| UTF-16 with a byte-order mark | --from-encoding utf-16 |
The export really includes its byte-order mark |
| Korean Windows code page 949 | --from-encoding cp949 |
Confirm the producer’s setting; language alone is insufficient |
Python documents its codec registry and error handling. This script uses strict decoding so invalid byte sequences stop conversion; it offers no option to silently discard or replace undecodable bytes.
Create a reproducible Windows-1252 export
Save the following as make_sample.py in a new working folder and run python make_sample.py. It creates legacy.csv only if that filename is unused.
from pathlib import Path
text = 'id,name,note\r\n001,André,"paid €12, cash"\r\n002,Zoë,"line one\r\nline two"\r\n'
with Path('legacy.csv').open('xb') as stream:
stream.write(text.encode('cp1252'))
There are two data records. The newline inside the second note is part of a field, so counting physical lines would give the wrong answer. The sample deliberately includes an ID with leading zeros, a comma inside quotes, and characters outside ASCII.
Save the complete converter
Save this as convert_csv_encoding.py next to the sample. The downloadable ZIP places it in examples/; include that prefix if running it directly from the extracted archive.
"""Validate a comma-separated CSV and write a new UTF-8 copy, without guessing."""
import argparse
import csv
import io
from pathlib import Path
def convert_csv_encoding(source, output, source_encoding, with_bom=False):
source, output = Path(source), Path(output)
# Whole-file validation happens before exclusive output creation.
text = source.read_bytes().decode(source_encoding, errors='strict')
if text.startswith('\ufeff'):
text = text[1:]
reader = csv.reader(io.StringIO(text, newline=''), strict=True)
header = next(reader, None)
if not header or any(not name.strip() for name in header):
raise ValueError('Expected a nonblank header')
if len(set(header)) != len(header):
raise ValueError('Duplicate header names')
count = 0
for number, row in enumerate(reader, 2):
if len(row) != len(header):
raise ValueError(f'Record {number}: expected {len(header)} fields, got {len(row)}')
count += 1
encoded = text.encode('utf-8-sig' if with_bom else 'utf-8', errors='strict')
with output.open('xb') as stream:
stream.write(encoded)
return count
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('source', type=Path)
parser.add_argument('output', type=Path)
parser.add_argument('--from-encoding', required=True)
parser.add_argument('--with-bom', action='store_true')
args = parser.parse_args()
try:
count = convert_csv_encoding(args.source, args.output, args.from_encoding, args.with_bom)
except (OSError, ValueError, LookupError, csv.Error) as exc:
parser.exit(1, f'Conversion failed: {exc}\n')
print(f'Wrote {count} data records to {args.output}')
if __name__ == '__main__':
main()
The validation pass uses Python’s CSV reader with strict parsing and newline translation disabled. Our extra checks require a nonblank, unique header and the same number of fields in every record. It expects comma separation and double quotes. It does not infer a delimiter or prove that field values are correct for your business rules.
Run and inspect the result
python convert_csv_encoding.py legacy.csv converted.csv --from-encoding cp1252
Expected message:
Wrote 2 data records to converted.csv
The decoded contents of the output are:
id,name,note
001,André,"paid €12, cash"
002,Zoë,"line one
line two"
The displayed block cannot show the distinction between CRLF and LF. The converter preserves those decoded characters because it writes the encoded text directly, rather than rewriting records through a CSV writer. Only a leading Unicode signature is removed before output encoding.
Run this independent check to confirm exact bytes, record values, and source preservation:
import csv
from pathlib import Path
expected = 'id,name,note\r\n001,André,"paid €12, cash"\r\n002,Zoë,"line one\r\nline two"\r\n'
assert Path('legacy.csv').read_bytes() == expected.encode('cp1252')
assert Path('converted.csv').read_bytes() == expected.encode('utf-8')
with open('converted.csv', encoding='utf-8', newline='') as stream:
rows = list(csv.reader(stream))
assert rows == [
['id', 'name', 'note'],
['001', 'André', 'paid €12, cash'],
['002', 'Zoë', 'line one\r\nline two'],
]
print('Verified original bytes and both output records')
For a destination application that specifically expects a UTF-8 signature, choose a different output filename and add --with-bom. The default creates UTF-8 without that signature. Neither option controls whether a spreadsheet later interprets 001 as a number; import that column as text in the receiving application.
Failure cases worth testing
An invalid byte sequence, unknown codec name, malformed quote, duplicate header, or wrong field count stops before output creation. Empty files and blank records are rejected. A header-only file is allowed and returns zero data records. The tests in the ZIP cover these cases, Korean text, UTF-16 and UTF-8 signatures, multiline fields, and attempts to overwrite the source or an existing destination.
If converted.csv already exists, the command refuses it. Inspect the earlier result and select a new filename for another attempt. Do not automatically remove a file just because a conversion command failed.
Limits and next steps
This version holds the input bytes, decoded text, and encoded output in memory at different stages; large files can require several times their disk size. It is intended for exports that comfortably fit available memory, with no performance benchmark claimed. Disk errors during the final write can still leave a partial output file. Treat success as the printed completion message followed by your checks; output existence alone is insufficient. Parent folders must already exist.
The operation preserves text characters, not the source’s byte sequence, since changing encoding is the purpose. A leading U+FEFF is treated as a signature, so a file using it as intentional first-header content is outside this script’s contract. Strict CSV parsing also does not reject every unusual convention; inspect the header and representative values before downstream use.
After verifying the output, you can select and reorder its columns or convert its records to JSON. Keep the original export so that an incorrect encoding choice can be corrected from the original bytes.