Weird Column Corruption in SQLMesh State Table Possibly Due to DuckLake Version Compatibility

Aaron A. · 2026-04-14T01:53:13.679Z

Ever see this before? On a SQLMesh state table. I have been trying pretty hard to figure out the root cause, best I've got is a forward-compatibility edge case in DuckLake 0.3 and/or SQLMesh (local) with DuckLake 0.4 on remote (MD). It was just this one column on one table that got corrupted, kind of a weird thing with the concatenation. I'd need to look deeper into DuckLake and probably SQLMesh source to fully understand what went on here, which I am probably not going to do tbh... maybe I will. Just wondering if this is familiar to anyone. Thank you!

11 comments

· Sorted by Oldest

Aedrian E.
·
·
Hey Aaron A., we've seen this before and from past investigation, this is due to concurrent writes when creating tables in DuckLake. To prevent the likelihood of this occurring, we suggest running your DDL statements on a single thread. We believe this should be fixed once we provide support for DuckDB 1.5.2, so please stay tuned!
Aedrian E.
·
·
Just to confirm - have you recovered your DuckLake to a previous state? Please let us know if you need some help there
Aaron A.
·
·
Ah, makes perfect sense and that aligns well with the conditions that caused this. Yes, I was able to make the DuckLake available again by fixing the column definition
🙌1
Aaron A.
·
·
Thank you for the very quick reply!
Aedrian E.
·
·
Awesome! You're welcome!
Aaron A.
·
(edited)·
We are running into this issue still. (no surprise since I didn't fix the issue on our end yet). I will turn off concurrent dlt writes to DuckLake for now, think that's probably the ultimate thing. Loosely related: Please let me know if there is any way for our company to be aware of releases before they happen. Is there a defined release schedule? We of course would prefer to be able to do our loads concurrently rather than sequentially if possible. But this may be bad pipeline design on my end also. Thanks so much!
Aedrian E.
·
·
Hey Aaron, we generally have releases every Wednesday, unless there's a reason to hold off on it. After the release, we update the release notes.
We are running into this issue still. (no surprise since I didn't fix the issue on our end yet). I will turn off concurrent dlt writes to DuckLake for now, think that's probably the ultimate thing.
Our expectation with the fix is that the corruption should not happen at all even with concurrent writes. DuckLake has a mechanism in place to perform retries when there is a collision. We'll let you know once the fix is in place so you can go back to doing concurrent writes. Would that work for you?
Aaron A.
·
(edited)·
Yeah absolutely, this is no big deal btw as our current ingest volume is slim. Only happens sometimes and honestly I think it might be related to these being kicked off in isolated Azure Function tasks (so potentially worse than just multiple threads, esp. if one fails partway through and the other tasks don't know about it). Likewise I am moving away from storing SQLMesh state in the lake, doesn't make much sense I think... Really appreciate the insight and timely response 🙂 Truly
Aedrian E.
·
·
You're welcome! 🫡
Aedrian E.
·
·
Hey Aaron A., just letting you know we have completed the release yesterday, so feel free to switch to concurrent writes 🙂
🙌1
Aaron A.
·
·
Thank you for letting me know!
🫡1

Aedrian E.
·
·
Hey Aaron A., we've seen this before and from past investigation, this is due to concurrent writes when creating tables in DuckLake. To prevent the likelihood of this occurring, we suggest running your DDL statements on a single thread. We believe this should be fixed once we provide support for DuckDB 1.5.2, so please stay tuned!
Aedrian E.
·
·
Just to confirm - have you recovered your DuckLake to a previous state? Please let us know if you need some help there
Aaron A.
·
·
Ah, makes perfect sense and that aligns well with the conditions that caused this. Yes, I was able to make the DuckLake available again by fixing the column definition
🙌1
Aaron A.
·
·
Thank you for the very quick reply!
Aedrian E.
·
·
Awesome! You're welcome!
Aaron A.
·
(edited)·
We are running into this issue still. (no surprise since I didn't fix the issue on our end yet). I will turn off concurrent dlt writes to DuckLake for now, think that's probably the ultimate thing. Loosely related: Please let me know if there is any way for our company to be aware of releases before they happen. Is there a defined release schedule? We of course would prefer to be able to do our loads concurrently rather than sequentially if possible. But this may be bad pipeline design on my end also. Thanks so much!
Aedrian E.
·
·
Hey Aaron, we generally have releases every Wednesday, unless there's a reason to hold off on it. After the release, we update the release notes.
We are running into this issue still. (no surprise since I didn't fix the issue on our end yet). I will turn off concurrent dlt writes to DuckLake for now, think that's probably the ultimate thing.
Our expectation with the fix is that the corruption should not happen at all even with concurrent writes. DuckLake has a mechanism in place to perform retries when there is a collision. We'll let you know once the fix is in place so you can go back to doing concurrent writes. Would that work for you?
Aaron A.
·
(edited)·
Yeah absolutely, this is no big deal btw as our current ingest volume is slim. Only happens sometimes and honestly I think it might be related to these being kicked off in isolated Azure Function tasks (so potentially worse than just multiple threads, esp. if one fails partway through and the other tasks don't know about it). Likewise I am moving away from storing SQLMesh state in the lake, doesn't make much sense I think... Really appreciate the insight and timely response 🙂 Truly
Aedrian E.
·
·
You're welcome! 🫡
Aedrian E.
·
·
Hey Aaron A., just letting you know we have completed the release yesterday, so feel free to switch to concurrent writes 🙂
🙌1
Aaron A.
·
·
Thank you for letting me know!
🫡1