What is LakeQuest?
LakeQuest evaluates systems that answer natural-language questions grounded in heterogeneous data lakes. The benchmark spans three domains: AIML, bank, and drug.
Systems are tested on answer correctness and, where applicable, evidence grounding over the source lake artifacts.
Getting started
The public starter kit contains the participant-facing submission format, sample predictor, and local validation utilities.
Benchmark tracks
LakeQuest is hosted as three Codabench tracks because Codabench phases inside one competition must be sequential.
Stay connected
@inproceedings{solodko2026lakequest,
title = {{LakeQuest}: A Three-Domain Benchmark for Grounded Question Answering across Data Lakes},
author = {Solodko, Michael and Gong, Steven and Yu, Guangwei and Gorti, Satya Krishna and Cresswell, Jesse C. and Zhong, Victor},
booktitle = {Proceedings of the Conference on Language Modeling (COLM)},
year = {2026}
}
Benchmark Splits
LakeQuest uses separate open development, open test, and closed held-out code-evaluation tracks.
| Lake | Open Dev | Open Test | Closed Test |
|---|---|---|---|
| AIML | 1,727 | 1,725 | 1,725 |
| Bank | 1,021 | 1,020 | 1,020 |
| Drug | 1,151 | 1,150 | 1,150 |
| Total | 3,899 | 3,895 | 3,895 |
Leaderboard
Official scores are maintained on Codabench. The table below mirrors the public metric layout.
Open Test
| # | Model | Team | Overall | Answer F1 | Coverage |
|---|---|---|---|---|---|
| - | Official submissions | Codabench | View live leaderboard | ||
Evaluation
Prediction submissions contain one predictions.jsonl file at the archive root. Each line must include a question ID and answer text.
{"qa_id": "example-id", "answer": "answer text"}
The main public score emphasizes answer quality. Optional evidence fields may be used for diagnostic grounding metrics when enabled by the organizers.
Tracks
| Track | Questions | Answers | Submission |
|---|---|---|---|
| Open Dev | Released | Released | Prediction zip |
| Open Test | Released | Hidden | Prediction zip |
| Closed Test | Hidden | Hidden | Runnable project zip |