Can an LLM do the slow, risky part of a data migration? A TU Berlin research benchmark built to answer that with numbers instead of opinions.
- My part: system design and generating the test data the benchmark runs on
The question
Sooner or later every company moves its data: off a legacy database, from one CRM to another. Someone has to write the logic that maps every old field onto a new one. It’s slow, fiddly work, and a small mistake quietly loses data.
So can a language model write that logic? Everyone has an opinion. AidMi answers with numbers.
How a run works
The source data goes into a Postgres staging database. A strategy looks at it together with the target schema and asks a model to write dbt models: plain SQL that maps source onto target. Some strategies plan first and execute after, some write and then critique their own work, some let several attempts vote.
Then the pipeline runs that SQL and checks the result. Did it run at all? How many target columns are covered? Are the types right? How many rows survived? What did it cost? A campaign repeats this for every combination of strategy, model and prompt, and a hunch becomes a table.
My part
Five of us built it as a research course at TU Berlin. I mostly worked on the system design: how data, strategies, the dbt run and the evaluators plug together, so a new strategy or metric doesn’t mean touching everything else. I also built the test data, which turned out to matter more than I expected.
What I took from it
Calling the model was the easy part. A benchmark is only as honest as its test data and its scoring: change either one a little, and the numbers tell a different story.