Python Build a Multi-Format Data Import Tool

Import CSV and JSON with different parsers but one validated internal record format.

How It Works

A multi-format importer benefits from a small conversion layer: parse each supported format, normalize field names, then validate each record. This keeps application logic independent from file format and makes error reporting easier. Do not use file extensions as proof that the content is safe.

For large imports, parse records iteratively and consider logging invalid rows separately.

Step-by-Step Example

import csv
import json
from io import StringIO

csv_rows = list(csv.DictReader(StringIO("name,quantity\nPens,3\n")))
json_rows = json.loads('[{"name": "Paper", "quantity": 5}]')

def normalize_record(item):
    name = str(item["name"]).strip()
    quantity = int(item["quantity"])
    if not name or quantity < 0:
        raise ValueError("Invalid item record")
    return {"name": name, "quantity": quantity}

print([normalize_record(item) for item in csv_rows + json_rows])

Understanding the Example

The expression 'list(csv.DictReader(StringIO("name,quantity\\nPens,3\\n")))' supplies the value stored as 'csv_rows'. This value is then used or checked by the later statements.
The expression 'json.loads(\'[{"name": "Paper", "quantity": 5}]\')' supplies the value stored as 'json_rows'. This value is then used or checked by the later statements.

A successful run of this example prints the following output:

[{'name': 'Pens', 'quantity': 3}, {'name': 'Paper', 'quantity': 5}]

Where This Fits in a Real Program

In a real import pipeline, parsing is only the first stage. Convert each field to the intended type, reject missing or unexpected values and report which record failed. Keep the original source file unchanged until the normalized data has passed validation and an export has been checked.

Practice: Try It Yourself

Add a validation error that reports the filename and row number without exposing sensitive content.

For this exercise, predict the result before running the changed code. Then explain the difference between the original and modified cases using the actual values, not just whether the program completed.

Common Beginner Mistakes

Assuming every CSV row value already has the same type as a parsed JSON number.
Catching all conversion errors and silently removing malformed records from reports.

Key Takeaways

Import CSV and JSON with different parsers but one validated internal record format.
A multi-format importer benefits from a small conversion layer: parse each supported format, normalize field names, then validate each record.
For large imports, parse records iteratively and consider logging invalid rows separately.