Connor C.

ok, awesome, appreciate it! We're also checking our logs.

Thanks for letting us know, Ajjit N.. We'll look into this ASAP for you

Ajjit N. just confirming that things have been running without problems now?

Ajjit N. we just got confirmation that the fix has been rolled out in the Estuary client! Your pipelines will run without encountering this problem now.

Hi Ajjit N.! we released the version of the MotherDuck client with the fix a few hours ago, and are chatting with Estuary right now to enable it in their system. Stay tuned!

For this use case, I would recommend a standard duckling for your service accounts doing the ingestion

Ajjit N. we have identified the root cause for this issue, and will have a fix shipped in this week's release. Here's what was happening: • We use short-lived tokens for authorization between the client and the server, and they get refreshed automatically when they expire. We identified an extremely rare race condition during this refresh path where, under a specific timing of background activity, the refresh could end up waiting on a lock it already held, producing a deadlock that prevented the client from obtaining a new token and reaching the server. The fix removes that potential loop and will ship with this week's release. Let us know if you have any questions!

I'll send you a DM for the inbound access so our engineering team can dig deeper. Thanks for setting that up!

Hi Ajit, thanks for reaching out. We've been working on this issue in collaboration with Estuary...they actually just sent over a backend core dump from one of your materializations to help us identify the root cause and work on a fix. Can you try disabling and then enabling the Estuary pipeline? We have seen that this often clears the hanging client-side commit from Estuary and allows the pipeline to run.