Tia M.

Hey John J. DuckDB has a mysql extension, so you can attach the database on your VPS and write into MotherDuck in one go

INSTALL mysql; LOAD mysql;
ATTACH 'host=<your-vps> port=3306 user=<user> password=<pw> database=<db>'
AS mysql_db (TYPE mysql, READ_ONLY);
ATTACH 'md:';
CREATE OR REPLACE TABLE bronze.main.orders AS
SELECT * FROM mysql_db.my_schema.orders;

You can run that from a DuckDB client on whatever box already runs Dagster. Or you can put it in a Flight and let MotherDuck run it on a schedule, as Sam suggested. We have a cookbook that does this for Postgres and the MySQL version should work the same way! motherduck.com/docs/cookbook/flight-postgres-ingest. Let me know if you have any other questions!

Hey John J. Great advice already from Sam and Jesse! What database is on the VPS? If it's Postgres or MySQL, you can attach it from a DuckDB client and write straight into MotherDuck in a single statement.

Hey Tanvir, I can confirm the retention change went through, all your databases are showing 0 days now. But I want to flag that it won't fully fix this on its own. The storage won't drop right away, since those bytes move into failsafe, which is a fixed 7-day window we can't configure. So it'll look flat for about a week before it falls off. And it won't settle as low as you'd want, because the storage is coming from lake_cache and the driver is your refresh loop. You're running CREATE OR REPLACE TABLE ... FROM iceberg_scan(...) about every 5 minutes across 10 tables, and each rewrite keeps a copy of the previous version, so as long as that cadence continues you'll keep a large multiple of your actual data. What would actually close the gap is making it a transient database. Transient databases have no historical snapshots and a one-day failsafe, and since lake_cache is a cache you rebuild from Iceberg anyway,. However, transient can only be set when the database is created, so it would mean CREATE DATABASE lake_cache_v2 (TRANSIENT), repointing the pipeline, and dropping the old one. Would you be open to that? More on how the lifecycle works here: motherduck.com/docs/concepts/storage-lifecycle

Thanks for the update, we are looking into what occurred and I'll let you know what we find!

Which account and user are you getting this error ?

Hey Greg, Looking into it!

Thanks for reporting this. I shall pass this on to the team!

Hey Staël, looking into it!

Hey Guru, Our team was able to get this fixed. Let me know if you have any other questions!

Hey Guru, We are taking a look at this!