Tia M.

Hey Patrick, tokens for the same service account don’t interfere with each other, so one won’t kick the other off or take the instance down. Both tokens connect to the same Duckling, so it’s really the same as running both scripts with a single token, they’ll share that Duckling’s CPU and memory. However, if both scripts are heavy at the same time they may slow each other down a bit, and if they’re both writing to the same tables you could see occasional write conflicts. If you want the two workloads fully isolated, the way to do that is to put them on separate service accounts, since each service account gets its own Duckling. If either script only reads data, a read scaling token would route it to separate read-only Ducklings. Let me know if this makes sense!

Thanks Patrick, makes sense. We already have a feature request open for table-level security on DuckLake, and I’ve added your use case to it. Let me know if anything comes up!

Hey Patrick! A view would work great for this use case, without external code A view doesn’t store data, only the query. So if you create a view in a regular MotherDuck database pointing at a table in lake_gold, For example

CREATE VIEW my_db.main.my_view AS SELECT * FROM lake_gold.main.my_table;

every time you query it, it reads the current state of the lake and a full CREATE OR REPLACE TABLE on the lake table shows up right away. However, MotherDuck doesn’t support triggers or materialized views, the view covers that need, since it’s always current. If the underlying lake table is dropped or renamed, or a column the view references by name is removed, you’d need to recreate the view. Additionally, If you share the database holding the view with other users, they’ll also need access to lake_gold for the view to work. If you ever need a stored copy instead, you can refresh it on a schedule with CREATE OR REPLACE TABLE ... AS SELECT * FROM lake_gold.... Let me know if this makes sense!

Hey William A., we are investigating this internally and I shall get back to you once we understand what occurred!

Hey William, I'm taking a look!

Hey Rishabh we have completed our investigation on the incident you experienced and want to share this RCA: On September 16 at 16:50 UTC, an internal maintenance process consumed more resources than expected due to an inefficient execution strategy. This starved the capacity that new compute instances need to start up, which surfaced as connection timeouts. We stopped the job at 17:16 UTC, after which service returned to normal. We have since identified and validated a fix that makes this maintenance work run efficiently and prevents it from competing with live query traffic. We know your workloads depend on MotherDuck being consistently available, and we take that seriously. We're sorry for the impact and are continuing to invest in platform stability.

Hey Rishabh, we are looking into this!

Hey John J. DuckDB has a mysql extension, so you can attach the database on your VPS and write into MotherDuck in one go

INSTALL mysql; LOAD mysql;
ATTACH 'host=<your-vps> port=3306 user=<user> password=<pw> database=<db>'
AS mysql_db (TYPE mysql, READ_ONLY);
ATTACH 'md:';
CREATE OR REPLACE TABLE bronze.main.orders AS
SELECT * FROM mysql_db.my_schema.orders;

You can run that from a DuckDB client on whatever box already runs Dagster. Or you can put it in a Flight and let MotherDuck run it on a schedule, as Sam suggested. We have a cookbook that does this for Postgres and the MySQL version should work the same way! motherduck.com/docs/cookbook/flight-postgres-ingest. Let me know if you have any other questions!

Hey John J. Great advice already from Sam and Jesse! What database is on the VPS? If it's Postgres or MySQL, you can attach it from a DuckDB client and write straight into MotherDuck in a single statement.

Hey Tanvir, I can confirm the retention change went through, all your databases are showing 0 days now. But I want to flag that it won't fully fix this on its own. The storage won't drop right away, since those bytes move into failsafe, which is a fixed 7-day window we can't configure. So it'll look flat for about a week before it falls off. And it won't settle as low as you'd want, because the storage is coming from lake_cache and the driver is your refresh loop. You're running CREATE OR REPLACE TABLE ... FROM iceberg_scan(...) about every 5 minutes across 10 tables, and each rewrite keeps a copy of the previous version, so as long as that cadence continues you'll keep a large multiple of your actual data. What would actually close the gap is making it a transient database. Transient databases have no historical snapshots and a one-day failsafe, and since lake_cache is a cache you rebuild from Iceberg anyway,. However, transient can only be set when the database is created, so it would mean CREATE DATABASE lake_cache_v2 (TRANSIENT), repointing the pipeline, and dropping the old one. Would you be open to that? More on how the lifecycle works here: motherduck.com/docs/concepts/storage-lifecycle