How to Evaluate and Balance the Complexity of Semantic Models in Malloy Development

·Sep 01, 2026 07:18 AM(edited)·

I attended the semantic layer live session recently conducted by motherduck where you talked about how you built a semantic layer , do you need one ? etc . You mentioned that the LLM writes malloy and then sql is compiled deterministically and executed One trap I keep falling into is overfitting the malloy semantic model to the eval set ; meaning I add more docs and views to the model in order to increase accuracy on my evals , but those changes only work for those specific set of question answer pairs ; if I add another question it fails . So my Question is what's the framework you use to evaluate changes to the semantic layer and how do you strike the balance between having a poorly defined semantic model vs overly defined semantic model ( extreme case where you have a registered view for every question on the domain)

4 comments

· Sorted by Oldest

Alex M.
·
·
Howdy! Great questions! Generalization is definitely still a hard problem. In our experiment, we used a pretty representative sample of the question set. Our goal was actually to have a training question for each "domain" of the set. When going into production, I think the highest ROI is really to increase that training set so that it covers a large fraction of the types of questions that people ask. Then if someone asks a question that didn't get a good answer, you'd want a feedback mechanism where you get notified when that happens so you can add more training questions. What model were you using for your evals? It does require a fairly new model for this to work well, but it can be a small new model.
Alex M.
·
·
You can also add to your prompt to favor composing queries from existing views rather than creating new ones (especially in your prompt that builds the semantic layer). Maybe even a subagent reviewer to look for similar views and combine them (like a code refactoring pass)
neel.y
·
·
I was using 3.5 Flash Lite (Gemini) , so you are saying that start with a set of questions that represent different classes of the eval set (could be business domain , or query profile like multi hop joins , simple aggreagates) and then let the model generalise and continuously improve your semantic model using real user interactions Regarding the LLM model , in your experiments have you tried this with non-frontier models because I feel like the more powerful the model the less handholding it needs whereas with a weaker model you need more scaffolding around it to make it reliably good at this stuff
Alex M.
·
·
Yes, spot on! I think the continuous improvement loop is the most important. In our step that produced the semantic layer, we used a large frontier model (Fable 5). To query against it, we used a smaller but still modern one. I think it is the build/improve step that really need the max intelligence.