neel.y

Commented on How to Evaluate and Balance the Complexity of Sema...·Posted inHelp

I was using 3.5 Flash Lite (Gemini) , so you are saying that start with a set of questions that represent different classes of the eval set (could be business domain , or query profile like multi hop joins , simple aggreagates) and then let the model generalise and continuously improve your semantic model using real user interactions Regarding the LLM model , in your experiments have you tried this with non-frontier models because I feel like the more powerful the model the less handholding it needs whereas with a weaker model you need more scaffolding around it to make it reliably good at this stuff

Posted in Help·

neel.y

How to Evaluate and Balance the Complexity of Semantic Models in Malloy Development

I attended the semantic layer live session recently conducted by motherduck where you talked about how you built a semantic layer , do you need one ? etc . You mentioned that the LLM writes malloy and then sql is compiled deterministically and executed One trap I keep falling into is overfitting the malloy semantic model to the eval set ; meaning I add more docs and views to the model in order to increase accuracy on my evals , but those changes only work for those specific set of question answer pairs ; if I add another question it fails . So my Question is what's the framework you use to evaluate changes to the semantic layer and how do you strike the balance between having a poorly defined semantic model vs overly defined semantic model ( extreme case where you have a registered view for every question on the domain)

4Comments

About

Start Date
Title