@Marostegui and I will be absent on tuesday, hosts have been depooled and are ready.

data-persistence hosts handled, ready whenever you are @cmooney

Since https://github.com/wikimedia/operations-software-spicerack/blob/v8.8.0/CHANGELOG.rst has been tested, this can be considered quite done, only remains the improvement of exception carrying when a SQL command fails on a host, tracked in T370419

sure thing!

Could be considered as redundant of T367283 which would offer a more specific angle.

done!

I've updated

• dbstore1009
• db1164 (m1)
• db1176 (m5)

In T370029#9984960, @Volans wrote:

The issue is due to the fact that cumin2002 has python3-wmfmariadbpy at version 0.10 while cumin1002 and most of the host with that package has version 0.11.2.
[...] how do you manage the versioning and upgrade of this package?
Full list of hosts with the old version available in Debmonitor: https://debmonitor.wikimedia.org/packages/python3-wmfmariadbpy

This server has been down for a few days, @wiki_willy please let me know if I can help

thanks! will roll the change there as well

dbstore1009 has replication up to date on all 3 instances

We could probably try and rely on read_default_file like we do in other tools

debug is already in progress, this task is to track what's been done

In T362529#9980552, @BTullis wrote:

@ABran-WMF, @fnegri - I believe that we are ready to run the sre.wikireplicas.add-wiki cookbook for this wiki, which should make it available on both clouddb* and an-redacteddb1001 hosts.
Are you happy for us (myself and @Stevemunene ) to run that now, or is there anything else that either of you feels need be done first?

execution collided with T367856 on s7, stopped and repooling will resume monday.

I am unable to reach it via management interface either, it might need a bit of hands on

dbhost repooling
dbproxy reloaded
backuphost checked and looks green

@Zabe it seems we were missing the "storage layer" task we usually get. Anyway, this is done on our side.

db1190 repooling
dbproxy reloaded

This is done, we'll iterate and monitor elsewhere if needed.

Exporter new configuration has been merged, it's rolling out on production. Rules have been merged as well

we've not seen any regression since you released the update, I think you're good to go!

In T367278#9956653, @Marostegui wrote:

We could also go for Misc hosts in codfw as testing.

will do!

@fgiunchedi fyi I've started rolling out the backport version on clouddb hosts, I've left aside clouddb1019 as @fnegri told me it was kind of unstable recently. I'll let it sit for the week-end before merging anything and go for the remaining hosts that need that version installed. @jcrespo don't worry about this change, it's very safe and monitored on my side, it'll cover backup hosts that have mysqld-exporter installed as well.

I'm affraid thats an answer I don't have @BTullis maybe @Marostegui or @Ladsgroup knows.

Amazing 🎉 lets maye try the first deployment from a canary cumin host so we're 100% sure that there is no breaking change.

Ah @BTullis I see the issue you face, I had the same one, sorry for not spotting it sooner!

db hosts as well, repooling

@Marostegui yep no worries! @BTullis you can remove and create dgit/$distro-wikimedia branches no problem. Our current packages have their version known and we can always revert to that if needed to. The specific instructions are here. @MatthewVernon has produced an extensive doc which I tried to follow and proof check last time I went through it. Please let me know if there is a missing link or component in that chapter that would help. Otherwise I'll just add a header saying that its OK to have to delete/overwrite dgit/$branch!

noted, thanks @jcrespo !

Today and yesterday we had another event of critical alarming firing for that reason

I limited myself to warning states on the first PS → I can trigger a page for that context indeed, will iterate

This will be unlocked by T362893