Show And Tell

Hi all! 👋 I just published the first article in a series on evals for data agents: what hillclimbing is and how to use it to measure and improve your agent’s accuracy. The next articles apply it hands-on with The Honest Agent, an open-source eval tool I built, running on MotherDuck’s MCP server. Feedback welcome, especially if you’re running evals on your own agents!

Interesting, I will take a look on it!

Built a thing I kept needing and could not find. Most fake data tools give you a fake column or a fake table. None of them keep the relationships intact, so the moment you join anything the numbers stop making sense. model2data takes a DBML schema and generates synthetic data that respects the keys and resolves foreign keys to real parent rows. It runs on DuckDB by default, so there is nothing to configure and no database access needed. It also scaffolds a dbt project on top: seeds, sources, staging models and tests. I use it for data modelling demos and for getting a stack running before anyone has granted me a single credential. MIT, pip install model2data. github.com/JB-Analytica/model2data Happy to hear where it breaks.

sspaeti shared a new post: https://daily.dev/posts/A9YKsFjox

OWASP top 10 was only the start - https://tomkornblit.substack.com/p/the-owasp-top-10-for-llms-was-only

Hi All - I've put together a flowchart on how I use duckdb and motherduck in my Corral AI project. I'd love to get some feedback on it - maybe in the next Office Hours. claude.ai/code/artifact/…

Built something that uses DuckDB in a way I have not seen elsewhere, and this felt like the right room for it. corral is a code auditing tool. It plants deliberate faults in your code, runs your own test suite against each one in a sandbox, and reports which faults your tests failed to catch. The interesting part for this channel is not the auditing — it is where the results live. CI runners are amnesiac. A runner boots, measures something about your code, and is destroyed. Whatever it learned dies with it, and the usual fix is to stand up a database: a server, a schema, credentials in CI, a bill, all to hold what is really an append-only list of small facts. So corral does not stand anything up. Every verdict is written as one gzipped JSON file and committed to an orphan git branch in the repository itself. An orphan branch shares no history with main, never appears in your diffs, and lives somewhere every CI runner already has credentials for. Then DuckDB reads it in place, straight off GitHub over HTTPS, with no clone and nothing running: import duckdb, json, urllib.request repo = "pdbethke/corralai" names = [f["name"] for f in json.load(urllib.request.urlopen( f"api.github.com/repos/{repo}/contents/scans?…")) if f["name"].endswith(".json.gz")] urls = [f"raw.githubusercontent.com/{repo}/…/{n}" for n in names] con = duckdb.connect(); con.execute("INSTALL httpfs; LOAD httpfs") print(con.sql(f"SELECT kind, count(*) FROM read_json_auto({urls!r}, union_by_name=true) GROUP BY 1")) That runs against our own record right now. 104 entries as I write this: 2 audits, 43 cold reviews by other models, and 59 rulings where a person confirmed or refuted what a reviewer claimed. Some findings are still open and the record says so. A git branch is a strange database and I want to be honest about the tradeoffs. It is append-only by construction, every entry carries the hash of the one before it, and a clean clone plus one command tells you whether the chain holds. It is also unindexed, and every query reads every file, so this works because the row counts are small and the entries are one per audit rather than one per event. If that stops being true the same rows push to a real warehouse, which is where MotherDuck comes in: one repository's branch is a record, and md: is the view across every repository that pushes to it. The push is the same verb either way, and the destination is a DSN: corral certify --repo . --push md:my_database -- pytest -q corral certify --repo . --push ./audits.duckdb -- pytest -q First push auto-creates the schema: corral_scans, corral_audits (one row per audited file), corral_mutants (one row per planted fault and its fate), corral_model_calls (tokens and latency per call), corral_events, plus corral_reviews, corral_findings and corral_adjudications for the review side. There is also a corral_seal view that cross-references a signed statement's hash against the rows it produced, so a row and the receipt it came from cannot quietly disagree. Rows are append-only. Overwriting is how a trend is lost, and a trend is the entire reason to keep them. The questions that need the warehouse are the ones a single run cannot answer. Which files keep shipping demonstrated gaps. Whether a kill rate moved or just resampled, since the faults are generated fresh each run and one number is a sample. How much of what changed could be audited at all, which is the honest denominator nobody prints. The one I did not expect to care about: the warehouse is how the models get graded. corral models rank reads those rows back and ranks each seat by its own metric — a test writer by proven catches over survivors, a reviewer by claims that held of those checked, a verifier by verdicts that agreed with the outcome. Every number comes from execution or from a human ruling, never from a model's account of itself. It is a leaderboard whose evidence is a table you can query, and it prints an insufficient-evidence marker rather than a rank when a seat has fewer than five observations. One real number, so this is not an abstract pitch. Pointed at pallets/flask yesterday, at app.py: 40 faults planted, flask's own suite run against each in a sandbox, 29 killed and 11 survived, and a second model then wrote a test that actually caught 10 of those 11. Kill rate 0.72. That entry is on the branch and the query above will return it. The signed statement also went to Sigstore's transparency log at index 2759598612, so the receipt is checkable against a log I do not run. v1.0.0-rc.13, Elastic-2.0, Go. Honest state of things: it is a release candidate, I am the only maintainer, and seven rounds of cold review found real bugs in it including two ways a CI gate could pass an audit that measured nothing. Those are fixed and the reviews are entries on that same public branch, which felt like the only defensible thing to do with them. corralai.dev github.com/pdbethke/corralai Happy to talk about the branch-as-storage idea specifically. I am not certain it is a good idea at scale and would genuinely like to hear where it breaks.

The agent weeped as it crawled into the sandbox - https://tomkornblit.substack.com/p/the-agent-didnt-get-hacked-it-got

I’ve integrated MotherDuck into SQL Cell, a Google Sheets add-on I’ve been building. You can query MotherDuck tables from the sidebar, profile the result, and write it back into the sheet. One use case I’ve found especially useful is joining a table in MotherDuck with a small mapping table maintained in Sheets. The warehouse has the data, while the spreadsheet has the business-specific context. You can combine both in the same query. A note for anyone attempting something similar in a constrained frame. An Apps Script sidebar is a sandboxed iframe inside docs.google.com, so it is not cross-origin isolated and never can be — COOP/COEP belong to the top-level document, which is Google's. That rules out motherduck/wasm-client, which needs SharedArrayBuffer. What does work is the motherduck extension installed into the DuckDB-WASM instance that was already there: INSTALL motherduck FROM the version-keyed repo, LOAD, SET motherduck_token, ATTACH 'md:'. Both api.motherduck.com and ext.motherduck.com send permissive CORS headers, so an arbitrary origin is fine. It fails on the v1.4.3 extension build and works on v1.5.2+. The engine version was the whole story. Thanks for shipping a build that loads without cross-origin isolation. It's the reason this exists at all. The result profiler borrows your Inspector's layout, since it turns out to be the right one: a row per column with type, a distribution silhouette, and either a range or a cardinality. It runs SUMMARIZE against the materialised result rather than the source, so profiling a query incurs no additional network overhead. Anyway, very happy to finally have MotherDuck working inside Sheets. Here is the marketplace link workspace.google.com/marketplace/app/sql_cell/178915154116

What kind of tax are we getting hit with by using AI agents without any security?tomkornblit.substack.com/p/an-ai-agent-doesnt-need-to-be-compromised

Hi All! I'm Peter. I've been a dev and software architect over 30 years. Right now my focus is on AI accountability in software tests and CI/CD. I built a system with DuckDB at its heart that integrates MotherDuck as well. Its a system that does decorralated, signed mutant testing of test suites, recording all the results and telemetry. My thesis is - no one (or AI) should be the judge of oneself. In this day and age, the same agents are writing the code, the tests, and certifying the releases. I created Corral to address this issue, and DuckDB and Motherduck are a big part of it. corralai.dev

Can you trust AI? tomkornblit.substack.com/p/when-giving-ai-agents-more-access

Hi friends! Adam from the MotherDuck support team. Wanted to share a cool project that's been incubating over here for the last month. Pardon appearances - this was an unexpected RESULT . For those of you in this channel stuck on your projects, Nathaniel T. and I have been hosting Builder Office Hours. We'd love to help you on your MotherDuck journey. Please come by, show off your latest data adventures, and work with us to help you progress. Upcoming office hours (register at the link): August 14th, 1PM ET / 10AM PT August 18th, 10AM ET / 7AM PT

Hi All, Excited to be here! Quack! I am a Data Engineer, and have been a big fan of the Duck "Universe" since it first released. Ive recently built DataCharter - a local, Apache-2.0 DuckDB explorer that gives AI agents read-only, PII-masked SQL over your data. evals that measure how much your context improves agent accuracy. Wrote it up as a local-first implementation, inspired by the MotherDuck Guides launch - curious what the community thinks: ▎ datacharter.dev/blog/measure-your-context-lift

Hi everyone! I’ve been working with DuckDB for a long time, and just finished up an article about how I used it to build a query transpiler from DuckDB to other dialects. MotherDuck obviously doesn’t fall directly into this category because it already uses DuckDB’s dialect (Woo hoo), but for other dialects like Postgres, ClickHouse, it’s very relevant. Wrote about how I built it, here: www.cocoalemana.com/blog/building-a-duckdb-transpiler/