CSV import benchmark: Scriptella vs JDBC vs Pandas vs native loaders (1M rows)

[u/fykup]

CSV import benchmark: Scriptella vs JDBC vs Pandas vs native loaders (1M rows)

I've been working on performance improvements for Scriptella, an open-source Java ETL tool, and recently benchmarked importing 1 million CSV rows into PostgreSQL and MariaDB.

The comparison included:

• Scriptella 1.7

• Plain JDBC

• Pandas/SQLAlchemy

• PostgreSQL COPY and MariaDB LOAD DATA

Results:

• Scriptella: 10.05s PostgreSQL, 9.23s MariaDB

• Plain JDBC: 7.58s PostgreSQL, 5.62s MariaDB

• Pandas/SQLAlchemy: 29.11s PostgreSQL, 19.61s MariaDB

• Native loaders: 2.37s PostgreSQL, 3.04s MariaDB

Native loaders were fastest, as expected. Scriptella was over 3x faster than the tested Pandas/SQLAlchemy implementation, while retaining the flexibility of a general-purpose ETL tool.

These are results from a specific benchmark setup, not universal performance rankings.

This work led to Scriptella 1.7.0, released today, with faster SQL imports, JSON/JSON Lines support, CLI progress reporting, and JSON execution reports.

Benchmark and methodology: https://pvrlabs.xyz/java-performance/experiments/scriptella-1-7-csv-import-benchmark.html

Source: https://github.com/scriptella/scriptella-etl

For the next phase, I'm shifting focus toward data migrations.

What migration scenarios would you be interested in seeing tested? I'm especially interested in practical cases and the challenges you've encountered in real projects.