Download the scripts and tests (ZIP)
Python scripts, tests, and ready-to-run merge, join, and date practice files with expected results. No signup required.
Combine a folder of daily CSV exports into one report while keeping a single header row. This script checks the whole batch first, refuses mismatched columns, and preserves text IDs such as 001. It uses Python’s standard library and does not upload your files.
Tested: Python 3.10 on Windows, September 7, 2026. The tests cover UTF-8 with and without a byte-order mark, a comma inside a quoted field, a line break inside a quoted field, inconsistent headers, missing fields, and an existing output file. Other Python versions have not been run for this article.
If a UTF-8 export uses tabs or semicolons, convert its delimiter with an expected-header check before adding it to the merge folder. A matching filename extension does not guarantee a matching table format.
Start with two small exports
Run the downloadable practice first
The Download the scripts and tests (ZIP) link on this page now includes the two input files and an independently specified expected result. Extract the entire archive and open a terminal in the extracted folder containing examples and fixtures. Check python --version; this practice was tested with Python 3.10 on Windows on September 18, 2026. No extra packages are needed.
python examples/merge_csv.py fixtures/merge/inputs practice-merged.csv
Expect Merged 2 rows. Compare the CSV records in practice-merged.csv with fixtures/merge/expected/merged.csv: one header, ID 001 with Seoul, Korea, then ID 002 with a two-line note. The output includes a UTF-8 byte-order mark; the expected file does not, so compare parsed records rather than requiring identical bytes. An embedded newline belongs to its quoted field and is not another record.
Keep practice-merged.csv outside the input folder. If it already exists, inspect it and choose a different output name; the script will refuse to replace it. fixtures/README.md includes the complete practice instructions. On a Windows installation using the Python launcher, substitute py -3.10 for python; on systems using python3, substitute that command. The paths after the command stay the same.
To verify the downloaded examples, run python -m unittest discover -s . -p test_examples.py -v from the extracted root. The practice test checks the actual command, expected records, unchanged inputs, and refusal to overwrite. For your own exports, work on copies and confirm their encoding and header order first. The manually created files below describe the same sample.
This workflow expects UTF-8 input. If a producer confirms another encoding, convert that CSV to UTF-8 first and verify accented names, quoted notes, and text IDs before adding it to the batch. Do not suppress decoding errors: a file that loads after discarding bytes can still contain damaged data.
Create a folder named exports. Save this as exports/a.csv using UTF-8:
id,note
001,"Seoul, Korea"
Save this as exports/b.csv. The second record intentionally occupies two physical lines:
id,note
002,"café
second line"
The target is two data records and one header, even though one record contains an embedded newline. Joining files as plain text would repeat the header; splitting every input line on commas would break the first record. The example below parses records before writing them.
The complete script
Save the following as merge_csv.py outside exports:
"""Validate a CSV batch before writing a new combined file. Python 3.10+."""
import csv
from pathlib import Path
def merge_csv(source, output):
source, output = Path(source).resolve(), Path(output).resolve()
files = sorted(p for p in source.glob('*.csv') if p.resolve() != output)
if not files:
raise ValueError('No input CSV files')
if output.exists():
raise FileExistsError(output)
header = None
# First pass avoids creating a report from a known-invalid batch.
for path in files:
with path.open(encoding='utf-8-sig', newline='') as handle:
reader = csv.reader(handle, strict=True)
current = next(reader, None)
if not current or len(set(current)) != len(current) or any(not c.strip() for c in current):
raise ValueError(f'{path.name}: missing, empty, or duplicate column names')
if header is None:
header = current
if current != header:
raise ValueError(f'{path.name}: headers differ, including order')
for row in reader:
if len(row) != len(header):
raise ValueError(f'{path.name}: wrong field count near line {reader.line_num}')
count = 0
# Exclusive creation refuses to replace an existing report.
with output.open('x', encoding='utf-8-sig', newline='') as handle:
writer = csv.writer(handle)
writer.writerow(header)
for path in files:
with path.open(encoding='utf-8-sig', newline='') as incoming:
reader = csv.reader(incoming, strict=True)
next(reader)
for row in reader:
writer.writerow(row)
count += 1
return count
if __name__ == '__main__':
import argparse
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('source')
parser.add_argument('output')
args = parser.parse_args()
print(f'Merged {merge_csv(args.source, args.output)} rows')
Run this from the directory that contains the script and exports:
python merge_csv.py exports combined.csv
Expected terminal output:
Merged 2 rows
The new file has one id,note header, 001 and 002 remain text in the file, and the quoted note remains one field. To check the result without a spreadsheet interpreting the IDs, save this short check as verify.py and run python verify.py:
import csv
with open("combined.csv", encoding="utf-8-sig", newline="") as handle:
rows = list(csv.reader(handle))
assert rows[0] == ["id", "note"]
assert [row[0] for row in rows[1:]] == ["001", "002"]
assert len(rows) == 3
print("Two records; IDs preserved")
What the checks protect
| Situation | Result |
|---|---|
| Matching header names in the same order | Combine the data records |
Header order changes to note,id |
Stop before creating output |
| Empty or repeated column name | Stop and name the source file |
| A record has fewer or more fields | Stop and report the nearby physical line |
| Output already exists | Refuse to overwrite it |
| Output path is inside the source folder | Exclude that path from input discovery |
The script compares headers exactly. It does not silently trim or rename them: id and ID may represent different contracts with the exporting system. Fix the export or deliberately map its columns in a separate step. The row count excludes the header. Blank physical rows outside quoted fields fail the field-count check rather than disappearing silently.
Diagnose a rejected batch before retrying
Read the final line of the traceback first. The merge script reports processing failures as Python exceptions; unlike the join guide, it does not prefix them with Join failed. These checks were reproduced with Python 3.10 on Windows on September 19, 2026.
| Message fragment | Inspect first | Recovery |
|---|---|---|
No input CSV files |
The source path and direct .csv files inside it |
Point to the folder that actually contains the exports; the script does not search subfolders |
headers differ, including order |
Every column name and its position in each file | Correct the export contract or explicitly reorder columns on copies before retrying |
wrong field count near line |
Quoting and the delimiter around that physical line | Repair or re-export the affected record; a quoted multiline cell can span several lines |
FileExistsError |
The chosen output path | Inspect the existing result and choose a fresh output name |
UnicodeDecodeError |
The encoding specified by the producer | Convert a copy with that known encoding; do not discard undecodable bytes |
For a controlled failure, create a separate folder bad-exports with these invented files. Save a.csv as:
id,note
001,first
Save b.csv as:
note,id
second,002
From the extracted download folder run:
python examples/merge_csv.py bad-exports rejected-merge.csv
With a fresh output name, this exits with code 1, reports b.csv: headers differ, including order, and creates no output. On a copy, change both the second file’s header and field order to id,note and 002,second. Changing only the header would mislabel the data. Rerun to a new name such as repaired-merge.csv; expect Merged 2 rows, with IDs 001 and 002 in the first column. Preserve the original failed export for comparison.
The line number in a CSV parsing error can differ from a record number when fields contain newlines; Python documents this distinction for reader.line_num. A validation failure leaves no newly created report, but a later write failure can leave a partial file. This merge script has no completion marker: compare counts and inspect the report before using it downstream.
Encoding and spreadsheet behavior
The input decoder accepts UTF-8 files with or without a BOM. The writer emits UTF-8 with a BOM. This is an explicit format choice; it is not automatic detection of every possible encoding. If an older export uses a different encoding, obtain its documented encoding before changing the reader.
Keeping 001 in the CSV does not force a spreadsheet to treat that column as text. If your spreadsheet turns it into 1, configure the import column type there. Do not convert identifiers to integers in an intermediate step if their leading zeros matter. For details on record parsing and newline handling, see the official csv documentation.
Limits before using it in a daily job
This example reads each input twice: validation first, then writing. Keep the source folder unchanged during both passes. A writer changing files between passes can invalidate the checks. A disk failure during writing can also leave a partial output, which must be reviewed before a retry. For concurrent production workflows, snapshot inputs and write a temporary report before committing it to the final destination.
Only direct *.csv matches are included; it does not recursively search subfolders. Every repeated business record is retained. Combining exports and removing duplicate orders are different operations: choose a business key and conflict policy before adding deduplication. The script also preserves cell text that a spreadsheet might interpret as a formula. Use trusted sample data; handling untrusted spreadsheet-bound content requires an explicit formula-injection policy.
Need smaller uploads? Split a CSV while keeping each header.
Next, inspect your exports with a read-only duplicate-file report or validate JSONL records before an import.
Code in this article is original Dadidan Lab example code. You may use and adapt it, including commercially. It is provided without warranty. Corrections: contact Dadidan Lab.