education

AidMi

TU Berlin research project

Can an LLM do the slow, risky part of a data migration? A TU Berlin research benchmark built to answer that with numbers instead of opinions.

  • My part: system design and generating the test data the benchmark runs on

The question

Sooner or later every company moves its data: off a legacy database, from one CRM to another. Someone has to write the logic that maps every old field onto a new one. It’s slow, fiddly work, and a small mistake quietly loses data.

So can a language model write that logic? Everyone has an opinion. AidMi answers with numbers.

How a run works

The source data goes into a Postgres staging database. A strategy looks at it together with the target schema and asks a model to write dbt models: plain SQL that maps source onto target. Some strategies plan first and execute after, some write and then critique their own work, some let several attempts vote.

Then the pipeline runs that SQL and checks the result. Did it run at all? How many target columns are covered? Are the types right? How many rows survived? What did it cost? A campaign repeats this for every combination of strategy, model and prompt, and a hunch becomes a table.

My part

Five of us built it as a research course at TU Berlin. I mostly worked on the system design: how data, strategies, the dbt run and the evaluators plug together, so a new strategy or metric doesn’t mean touching everything else. I also built the test data, which turned out to matter more than I expected.

What I took from it

Calling the model was the easy part. A benchmark is only as honest as its test data and its scoring: change either one a little, and the numbers tell a different story.

How it’s built

INPUTSSTAGEGENERATERUNSCORESource datae.g. a SalesforceexportTarget schemae.g. Pipedrive,optionalCampaign gridstrategies ×models × promptsPostgresstaging · sourceas tablesStrategyper table,plan→execute,critique, voteLLMhosted or localdbt modelsSQL that mapssource → targetdbt runagainst stagingEvaluatorsruns? coverage,types, rows keptReportper strategy andmodelCosttokens, dollars1target2prompts345
  1. 1Realistic test data is loaded into a Postgres staging database.
  2. 2A strategy looks at the staged data and the target schema and prompts the model in its own way, for example planning first, or writing and then critiquing.
  3. 3What comes out is a set of dbt models, plain SQL that maps the source onto the target.
  4. 4dbt runs them, and evaluators check whether they ran, how many target columns were covered, type mismatches and how much data survived.
  5. 5Results from every run in the campaign are combined into one report.
A campaign repeats this loop over every combination in its grid, which turns "can an LLM do this?" into numbers.