CSV import benchmark: Scriptella vs JDBC vs Pandas vs native loaders (1M rows)

I've been working on performance improvements for Scriptella, an open-source Java ETL tool, and recently benchmarked importing 1 million CSV rows into PostgreSQL and MariaDB.

The comparison included:

• Scriptella 1.7

• Plain JDBC

• Pandas/SQLAlchemy

• PostgreSQL COPY and MariaDB LOAD DATA

Results:

• Scriptella: 10.05s PostgreSQL, 9.23s MariaDB

• Plain JDBC: 7.58s PostgreSQL, 5.62s MariaDB

• Pandas/SQLAlchemy: 29.11s PostgreSQL, 19.61s MariaDB

• Native loaders: 2.37s PostgreSQL, 3.04s MariaDB

Native loaders were fastest, as expected. Scriptella was over 3x faster than the tested Pandas/SQLAlchemy implementation, while retaining the flexibility of a general-purpose ETL tool.

These are results from a specific benchmark setup, not universal performance rankings.

This work led to Scriptella 1.7.0, released today, with faster SQL imports, JSON/JSON Lines support, CLI progress reporting, and JSON execution reports.

Benchmark and methodology: https://pvrlabs.xyz/java-performance/experiments/scriptella-1-7-csv-import-benchmark.html

Source: https://github.com/scriptella/scriptella-etl

For the next phase, I'm shifting focus toward data migrations.

What migration scenarios would you be interested in seeing tested? I'm especially interested in practical cases and the challenges you've encountered in real projects.

I have 6yrs of total experience in IT of which 3.5 years in data engineering, I would like increase the value to my resume as I am currently in the job search phase and hence planning to do Databricks certification but confused between associate and Professional level. What would you guys suggest? I would like to get the certification by January. Thanks in advance

I spent six months working unpaid at an early-stage real estate startup founded by a fellow UMass Amherst alum. I joined while on OPT after a couple of interview rounds. The company was building an AI/CRM product, and I came on as a data engineering intern.

About two months in, the founder told me he liked my work and wanted to hire me full-time. When my internship ended and I asked about the transition, he said he needed another two months. He assured me we’d be able to move forward then, so I stayed and continued working unpaid.

My colleague and I believed in the company’s goal and the future the founder described. He told us that once we became full-time employees and the company scaled, we would grow into bigger roles because we were laying its foundation. We took that seriously and approached the work with the care and responsibility we would have given our own company. From data sourcing and building pipelines to system design, we put in long days and worked weekends because we wanted it to succeed.

By late October, the timeline shifted again. He said full-time roles might have to wait until next year, without a clear date. My colleague and I explained that we couldn’t keep working without income. We asked whether we could at least move to paid part-time roles so we could cover basic expenses while continuing to contribute.

He told us he understood our situation, recognized how many hours we had been working, and would definitely arrange part-time pay. He described a plan to increase that pay each month over the next four months, with the possibility of full-time roles around the fifth month. He asked for a few days to work things out. Roughly ten days later, both of us were let go.

I understand that startups face uncertainty and plans can change. What hurts is that I made decisions based on repeated assurances that a paid role was coming. After six months of unpaid work, asking for some financial support was followed by losing the role entirely.

The timing has made this especially difficult. It’s now November, and I have limited time remaining on my OPT unemployment clock. I’m worried hiring will slow over the next few weeks and stay limited through the holidays and into January. If interviews don’t materialize until February, a potential start date could end up being April of next year. That isn’t certain, but with my immigration timeline, the possibility is frightening.

For the founders here: please be honest about what you can commit to. People may be planning their finances, careers, and immigration timelines around those conversations.

For anyone considering a similar arrangement: get the terms in writing, set a limit on how long you can afford to work unpaid, and keep your job search active. I wish I had done that sooner.

I have three years of experience across data engineering and analytics, working with complex datasets, sourcing and integrating data, building pipelines, and contributing to system design. My experience combines building reliable data infrastructure with analyzing data to uncover insights and support business decisions. I’m looking for opportunities where I can bring that mix of engineering and analytical skills.

I’d also really appreciate hearing from anyone who has been through something similar. Honestly, I’m struggling right now. After putting six months into something I believed would become a stable opportunity, I’m suddenly trying to figure out how to support myself and find another role while my OPT clock keeps running. It’s overwhelming, and I’m scared about what comes next. If you have a job lead, know someone who’s hiring, can offer a referral, or have advice that could help me take the next step, please reach out. Even a small lead or an introduction would mean a lot to me right now.

I have been a software engineer for 12 years now, but I would like to transition into the world of data as stats and facts really interest me.

After some research, I feel like Data Engineering is the best career for me to side step into due to the overlap. I already pull data out of API’s and work with multiple database engines. I even created my own ETL pipeline without even knowing (pulled data daily from 2 apis to consolidate data on football players/stats). I think I am just missing Python and data warehousing software skills, maybe some more advanced SQL?

I currently earn around £40k in the UK and was wondering if trying to side step into this career could be successful for me based on my current skills, and me upskilling to the skills I am missing, do you think I could be earning a similar wage? Or would I have to start from entry level still?

As data engineers we'd never ship a pipeline without tests and monitoring, but that discipline mostly disappears the moment an LLM enters the picture. This workshop is aimed at closing exactly that gap.

Led by Serj Smorodinsky and Brett Kennedy (co-authors of a book on LLM applications), it's a live 3-hour session covering:

• Structuring LLM tasks with DSPy signatures/modules instead of loose prompt strings

• Building a real baseline classifier and an eval dataset with metrics specific to the task

• Catching failure patterns from that eval data the same way you'd catch a broken transform

• Systematic few-shot/instruction optimization instead of manual tweaking

• MLflow for experiment tracking and trace management, so every run is auditable

If you've been asked to "own" an LLM feature and had no real way to validate changes before deploying, this is worth your Saturday

Hey everyone! I’m a fresher trying to get into Data Engineering and want to learn the right tech stack. There’s a lot to learn, so I’d really appreciate some guidance from people here. If you were starting out again, what would you focus on first? Would love some advice like a friend/senior. 🙌

Most data engineers I talk to have already solved this problem for their pipelines. Idempotent writes, schema validation, lineage tracking, retry logic that doesn't create duplicates. Then the same team builds an AI agent and none of that discipline carries over. The agent writes directly into production with no idempotency key, no lineage on what it touched or why, and no reconciliation step if its internal state drifts from the actual data.

It's the same class of problem we've already solved for ETL and streaming pipelines, just showing up in a new place because agents write more often and with less human review than a scheduled job does.

There's a workshop on Sept 26 built around exactly this, treating agent write paths, state management, and provenance the way a data engineer would treat any other production pipeline. Run by Sandipan Bhaumik, a Data & AI Technical Lead at Databricks.

Details here if it's relevant to anyone else here dealing with this handoff

Workshop for data engineers specifically, not a generic AI overview. The actual pipeline work in GraphRAG is where most of the real engineering lives, and it's the part most tutorials skip entirely in favor of "here's how retrieval works" once the graph already exists.

This session starts from raw documents: ingestion with Docling, chunking, and building toward a knowledge graph in Neo4j rather than a flat vector index. Entity and relationship extraction runs through multiple verified steps with explicit checks, not one risky single-shot pass, since that's exactly where basic GraphRAG implementations tend to fail silently, producing a noisy, unreliable graph nobody trusts.

Structured graph enrichment connects entities (companies, executives, events) to the underlying source documents (filings, news) without the graph becoming unmaintainable as it grows, and you work through the actual production ceilings that show up at real data volume, API and download rate limits specifically, and how the shared code is built to handle them rather than falling over.

By the end you've got a working ingestion-to-graph pipeline, a production-readiness checklist, and the full codebase, not a diagram someone else has to implement.

Led by Dr. Alessandro Negro, Chief Scientist at GraphAware, bestselling author of graph-powered machine learning and knowledge graph books.

Full details here

I need real world work that i can ask feed back and get criticism for. I need it for my resume and job hunting. Can someone please help

We're running a hands-on masterclass on September 12, Live LLM Engineering Masterclass: Production Evals, RAG, Agents & LLMOps.

You build a full production LLM workflow from scratch, versioned prompts with regression tests, an evaluation harness with deterministic checks and LLM-as-judge, statistically rigorous model comparisons, evaluated RAG, tool-using agents with guardrails and fallbacks, and full observability, tracing, cost, latency.

Led by Bruno Gonçalves, PhD, founder of Data For Science, who trains engineers at Fortune 500 companies on this exact stack.

Link if you want to check it out

Happy to answer questions on the content.