[07:31:52] <wikibugs>	 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10fgiunchedi)
[07:33:42] <wikibugs>	 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10fgiunchedi)
[07:35:25] <wikibugs>	 10DBA, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row B - https://phabricator.wikimedia.org/T286061 (10fgiunchedi)
[07:37:42] <wikibugs>	 10DBA, 10Infrastructure-Foundations, 10SRE, 10netops, 10cloud-services-team (Kanban): Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10fgiunchedi)
[07:39:32] <wikibugs>	 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10fgiunchedi)
[09:15:52] <wikibugs>	 10DBA: iw_url in interwiki is varbinary(127) in production but blob in code - https://phabricator.wikimedia.org/T277118 (10Kormat)
[09:22:45] <wikibugs>	 10DBA: iw_url in interwiki is varbinary(127) in production but blob in code - https://phabricator.wikimedia.org/T277118 (10Kormat)
[09:25:36] <wikibugs>	 10DBA: iw_url in interwiki is varbinary(127) in production but blob in code - https://phabricator.wikimedia.org/T277118 (10Kormat) a:03Kormat
[09:25:49] <wikibugs>	 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10cmooney)
[09:34:25] <wikibugs>	 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10cmooney) @Dwisehaupt I have to apologize for a newbie error I made here.  The FR-Tech server's interfaces don't show what they're connected to in Netb...
[09:46:06] <wikibugs>	 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10Volans)
[09:46:18] <wikibugs>	 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10cmooney)
[09:50:45] <wikibugs>	 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10Volans)
[10:01:23] <wikibugs>	 10DBA, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row B - https://phabricator.wikimedia.org/T286061 (10Volans)
[10:06:19] <wikibugs>	 10DBA, 10SRE, 10ops-eqiad: Upgrade db1104 firmware - https://phabricator.wikimedia.org/T286226 (10LSobanski) @wiki_willy we're good to move ahead with this one. The host needs to be downtimed and shut down in advance, what (EU friendly) time would work for you?
[10:16:30] <wikibugs>	 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10cmooney)
[10:17:00] <wikibugs>	 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10cmooney)
[10:17:28] <wikibugs>	 10DBA, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row B - https://phabricator.wikimedia.org/T286061 (10cmooney)
[10:19:05] <wikibugs>	 10DBA, 10Infrastructure-Foundations, 10SRE, 10netops, 10cloud-services-team (Kanban): Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10cmooney)
[10:19:17] <wikibugs>	 10DBA: fa_deleted_timestamp and fa_timestamp are binary(14) in code but varbinary(14) in production - https://phabricator.wikimedia.org/T277116 (10Kormat) a:03Kormat
[10:44:47] <wikibugs>	 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10cmooney)
[11:17:02] <wikibugs>	 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10MoritzMuehlenhoff)
[11:19:06] <wikibugs>	 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10MoritzMuehlenhoff)
[11:20:44] <wikibugs>	 10DBA, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row B - https://phabricator.wikimedia.org/T286061 (10MoritzMuehlenhoff)
[11:21:44] <wikibugs>	 10DBA, 10Infrastructure-Foundations, 10SRE, 10netops, 10cloud-services-team (Kanban): Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10MoritzMuehlenhoff)
[11:27:06] <wikibugs>	 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10MoritzMuehlenhoff)
[11:35:59] <wikibugs>	 10DBA: fa_deleted_timestamp and fa_timestamp are binary(14) in code but varbinary(14) in production - https://phabricator.wikimedia.org/T277116 (10Kormat)
[11:36:27] <wikibugs>	 10DBA: DB maintenance work to do while eqiad is passive (June 2021) - https://phabricator.wikimedia.org/T285139 (10Kormat)
[11:36:29] <wikibugs>	 10DBA: job_cmd is varbinary(255) in production while being varbinary(60) in code since 2007 - https://phabricator.wikimedia.org/T278621 (10Kormat)
[11:36:31] <wikibugs>	 10DBA: iw_url in interwiki is varbinary(127) in production but blob in code - https://phabricator.wikimedia.org/T277118 (10Kormat)
[11:36:34] <wikibugs>	 10DBA: fa_deleted_timestamp and fa_timestamp are binary(14) in code but varbinary(14) in production - https://phabricator.wikimedia.org/T277116 (10Kormat) 05Open→03Stalled Stalling this until we switch back to eqiad.
[11:36:36] <wikibugs>	 10DBA, 10Datasets-General-or-Unknown, 10Patch-For-Review, 10Sustainability (Incident Followup), and 2 others: Detect object, schema and data drifts between mediawiki HEAD, production masters and replicas - https://phabricator.wikimedia.org/T104459 (10Kormat)
[11:45:22] <wikibugs>	 10DBA, 10Infrastructure-Foundations, 10SRE, 10netops, 10cloud-services-team (Kanban): Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10Legoktm)
[11:48:40] <wikibugs>	 10DBA, 10Infrastructure-Foundations, 10SRE, 10netops, 10cloud-services-team (Kanban): Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10cmooney)
[11:52:08] <wikibugs>	 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10Kormat)
[11:54:38] <wikibugs>	 10DBA, 10SRE, 10Wikimedia-Mailing-lists: Mailman3 schema change:  Switch autoresponse_text fields to Text - https://phabricator.wikimedia.org/T286552 (10Legoktm)
[11:54:48] <wikibugs>	 10DBA, 10SRE, 10Wikimedia-Mailing-lists, 10Schema-change: Mailman3 schema change:  Switch autoresponse_text fields to Text - https://phabricator.wikimedia.org/T286552 (10Legoktm)
[11:59:12] <wikibugs>	 10DBA: iw_url in interwiki is varbinary(127) in production but blob in code - https://phabricator.wikimedia.org/T277118 (10Kormat) Deployed to an s6 eqiad replica (db1096:3316). Will leave to soak overnight.
[12:18:06] <Amir1>	 okay the image table migration is going way slower than I thought, it'll take 50 days with this speed, I can decrease it to forty days or so
[12:19:58] <Amir1>	 2.27% is done so far
[12:20:18] <wikibugs>	 10DBA, 10SRE, 10Wikimedia-Mailing-lists, 10Schema-change: Mailman3 schema change:  Switch autoresponse_text fields to Text - https://phabricator.wikimedia.org/T286552 (10LSobanski) p:05Triage→03Medium Let us know when full tested and ready to go. Preferably after the DC switch back as our schedule is p...
[12:20:23] <RhinosF1>	 Amir1: is this gonna be needed for 1.37 update too?
[12:21:01] <Amir1>	 not sure, the default is still php serialization 
[12:21:14] <kormat>	 Amir1: have you checked to make sure the 'turbo' button is set to 'on'?
[12:21:18] <Amir1>	 once the default changes then running the script would be needed
[12:21:41] <Amir1>	 kormat: reminds me of this masterpiece https://bash.toolforge.org/quip/AVi2Z1eBQMK9DA-FJpXK
[12:21:57] <kormat>	 :D
[12:22:01] <RhinosF1>	 Amir1: I'm gonna cry when I see 1.37 changelog
[12:22:35] <Amir1>	 I can reduce the sleep time (it's 5s between each batch) but that in total is only 1/3rd of the time, If I remove it totally, then the second dc (and the cloud) start to lag behind
[12:23:00] <Amir1>	 RhinosF1: haha, maybe we should change it in 1.38 :D
[12:23:25] <Reedy>	 the cloud will lag!?!?!
[12:23:50] <RhinosF1>	 Amir1: there seems to loads planned already. I'm pretty sure half our extensions are gonna break.
[12:24:35] <Amir1>	 Reedy: more than usual :D my most priority there is the second dc
[12:24:44] <Reedy>	 It's only 4 months to 1.37 is due to be released...
[12:24:50] <Reedy>	 So only 2-2.5 months more dev time
[12:25:01] <Reedy>	 We're already on .12
[12:25:07] <Amir1>	 because they don't show up in waitForReplication() that's why we have an extra sleep there
[12:25:10] <RhinosF1>	 It's gone fast
[12:25:36] <RhinosF1>	 Reedy: it better be nice because it's gonna be my job again
[12:25:50] <Amir1>	 Reedy: for this case, honestly I think it's just changing the default config, there's not much needed to be done but we can always postpone it 
[12:25:51] <RhinosF1>	 I got bullseye to think about too soon
[12:25:53] <legoktm>	 is it a problem if the second DC (eqiad) is lagging?
[12:26:15] <RhinosF1>	 legoktm: depends if it finishes before switchback or if an emergency happens
[12:26:19] <Amir1>	 legoktm: it used to make Manuel sad in previous runs
[12:26:27] <Amir1>	 specially since it'll accumulate 
[12:26:33] <Reedy>	 Amir1: I think it's reasonable. Don't remove the config or the back compat in this version, so those that can't run the script yet don't need to be forced to etc
[12:26:37] <legoktm>	 gotcha
[12:28:03] <Amir1>	 Reedy: sounds good to me
[12:29:11] <Reedy>	 TBH, it's maybe the sort of thing we might want to keep to the next LTS
[12:29:14] <Reedy>	 Maybe. Maybe not
[12:29:48] <Amir1>	 what is the next LTS? 1.39?
[12:29:50] <RhinosF1>	 If it takes 50 days for wikimedia, I don't want to think how long it will take us.
[12:29:51] <Reedy>	 Yeah
[12:30:08] <Amir1>	 RhinosF1: oh that's 50 days for pdf of commons only
[12:30:11] <Reedy>	 lol
[12:30:12] <RhinosF1>	 We can barely get scripts to stay running for more than 12 hours at the moment
[12:30:23] <RhinosF1>	 Amir1: oh god
[12:30:38] * Reedy grins
[12:31:05] <kormat>	 legoktm: also bear in mind that we're doing a ton of maintenance (like schema changes) while we're in codfw, which already adds lag.
[12:31:40] <legoktm>	 ah, yeah
[12:31:59] <Amir1>	 tbf, the pdf is 80% of storage alone (while being only 2% of files), that's why it's slow, if we get it to the rest I can use a higher batch size and smaller sleep time
[12:32:03] <Reedy>	 #progress
[12:32:18] <Amir1>	 we should just use mongodb
[12:32:24] <Amir1>	 that'll solve all our issues
[12:32:37] <Amir1>	 #HypeDrivenDevelopment
[12:32:38] <kormat>	 i've been lobbying marostegui for us to move to sqlite3
[12:32:54] <kormat>	 it would instantly solve all replication issues for us
[12:32:56] <Amir1>	 oh alter tables on sqlite are awesome lol
[12:33:32] <RhinosF1>	 SQLite has 0 concurrency
[12:33:50] <RhinosF1>	 When i use it for developing stuff, I end up wondering why it's always Locke
[12:33:52] <RhinosF1>	 Locked
[12:33:53] <kormat>	 RhinosF1: more resistant to data corruption, you say? sign me up!
[12:34:22] <RhinosF1>	 kormat: the dB errors dashboards would be a million times the size
[12:35:22] <RhinosF1>	 I mean the database wouldn't grow as fast
[12:35:31] <majavah>	 wiki replicas: have a cronjob rsync the database file to nfs
[12:38:45] <wikibugs>	 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10ema)
[12:40:53] <wikibugs>	 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10ema)
[12:40:57] <Amir1>	 majavah: you're a genius
[12:41:18] <wikibugs>	 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10ema)
[12:45:06] <wikibugs>	 10DBA, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row B - https://phabricator.wikimedia.org/T286061 (10ema)
[12:46:14] <wikibugs>	 10DBA, 10Infrastructure-Foundations, 10SRE, 10netops, 10cloud-services-team (Kanban): Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10ema)
[12:51:35] <wikibugs>	 10DBA, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row B - https://phabricator.wikimedia.org/T286061 (10ema)
[12:52:13] <wikibugs>	 10DBA, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row B - https://phabricator.wikimedia.org/T286061 (10ema)
[13:03:31] <wikibugs>	 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10ssingh)
[13:06:12] <wikibugs>	 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10ssingh)
[13:12:08] <wikibugs>	 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10Vgutierrez)
[13:23:43] <wikibugs>	 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10jbond)
[13:24:41] <wikibugs>	 10DBA, 10Infrastructure-Foundations, 10SRE, 10netops, 10cloud-services-team (Kanban): Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10Vgutierrez)
[13:26:18] <wikibugs>	 10DBA, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row B - https://phabricator.wikimedia.org/T286061 (10jbond)
[13:28:24] <wikibugs>	 10DBA, 10Infrastructure-Foundations, 10SRE, 10Traffic, and 2 others: Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10jbond)
[13:29:47] <wikibugs>	 10DBA, 10Infrastructure-Foundations, 10SRE, 10Traffic, and 2 others: Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10jbond)
[13:43:51] <wikibugs>	 10DBA, 10Infrastructure-Foundations, 10SRE, 10Traffic, and 2 others: Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10jbond)
[14:19:39] <wikibugs>	 10DBA, 10Patch-For-Review: Rename dbstore1004 to db1183 and place it on m5 - https://phabricator.wikimedia.org/T284622 (10Kormat) 05Open→03Stalled Stalling until {T286032} is done, then it will be moved to s7.
[14:19:45] <wikibugs>	 10DBA, 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: dbstore1004 85% disk space used. - https://phabricator.wikimedia.org/T283125 (10Kormat)
[14:19:46] <wikibugs>	 10DBA, 10Infrastructure-Foundations, 10SRE, 10Traffic, and 2 others: Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10Kormat)
[16:04:50] <wikibugs>	 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10Dwisehaupt)
[16:06:41] <wikibugs>	 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10Dwisehaupt) @cmooney Not a problem. I have updated the task to remove the pre/post tasks we were looking at. Always good for us to think about failure...
[16:49:34] <robh>	 Any DBAs around to depool db1104 via https://phabricator.wikimedia.org/T286226 for firmware upgrade?
[16:53:52] <wikibugs>	 10DBA, 10SRE, 10ops-eqiad: Upgrade db1104 firmware - https://phabricator.wikimedia.org/T286226 (10RobH) I can handle this, and a firmware upgrade takes anywhere from 5 to 30 minutes (depending on if its only bios, etc..)  This is just bios, so I can handle this anytime this week.    @LSobanski: Please have s...
[16:54:14] <robh>	 marostegui: shoudl that be assigned to kormat?
[16:54:23] <robh>	 its a firmware update, so i suspect that was accidental?
[16:55:16] <kormat>	 robh: feel free to grab it :)
[16:55:44] <robh>	 well, now it technicallys hould go back to dba team for someone there to depool and then reasisgn to me
[16:55:57] <robh>	 cuz im blocked on that, it was requested a good window, and anytime this week works for me...
[16:56:02] <robh>	 hrmm, elmme edit my reply to clarify that
[16:56:23] <kormat>	 if i depool it now, would that work for you?
[16:57:22] <robh>	 yep!
[16:57:31] <robh>	 I have the firmware stuff in front of me, now is perfection
[16:57:34] <kormat>	 alright, gimme 3 mins
[16:57:50] <robh>	 i'll then push new idrac and then new bios, will take like... 30 min tops if its crazy slow, but i suspect more like 15
[16:57:53] <robh>	 cool, thank you!
[17:00:42] <kormat>	 robh: depooled, task assigned to you
[17:00:52] <wikibugs>	 10DBA, 10SRE, 10ops-eqiad: Upgrade db1104 firmware - https://phabricator.wikimedia.org/T286226 (10Kormat) a:05Kormat→03RobH Machine is depooled and ready for firmware upgrade.
[17:00:58] <robh>	 thanks, ill update idrac and then bios now
[17:02:56] <robh>	 oh man, so old
[17:03:03] <robh>	 r630 old iidrac interface is ugly
[17:03:20] <robh>	 idrac uploading.
[17:07:39] <robh>	 and it disappeared...wtf
[17:07:46] <robh>	 hit 'install' and nothing in job queue
[17:07:58] <robh>	 oh well, i echo my issues in dcops for remainder of this ; D
[17:14:47] <robh>	 this is gonna be longer than i expected
[17:14:54] <robh>	 its still downloading hte idrac update in the jobqueue
[17:30:29] <wikibugs>	 10DBA, 10SRE, 10ops-eqiad: Upgrade db1104 firmware - https://phabricator.wikimedia.org/T286226 (10RobH) So for best practices, I tend to upload a new idrac firmware before anything else.  Since idrac handles the firmware updates of other things, it just seems safest.  That being stated, this host is taking f...
[17:40:05] <robh>	 kormat: so..... if you can connect to that, you may as well return to service but now the idrac isn't accessible to me via https to update bios =[
[17:40:22] <robh>	 i'll update the task now with details, but basically the updated idrac firmware breaks ssl in new browsers for that old host
[17:40:39] <robh>	 so an onsite needs to connect with crash cart and roll it back manually
[17:41:00] <robh>	 but both chome and ff complain now since i updated it, which sucks.
[17:42:48] <robh>	 kormat: actually, leave it off for a bit longer sorry for second ping im going to poke around web for potential solutions
[17:44:01] <wikibugs>	 10DBA, 10SRE, 10ops-eqiad: Upgrade db1104 firmware - https://phabricator.wikimedia.org/T286226 (10RobH) This has gone exceedingly poorly.  I updated the idrac firmware, and idrac is accessible via SSH but not via HTTPS, as its self signed cert changed to one unsupported (even with bypass option via advanced...
[17:57:40] <wikibugs>	 10DBA, 10SRE, 10ops-eqiad: Upgrade db1104 firmware - https://phabricator.wikimedia.org/T286226 (10RobH) I downloaded the 2.80.80.80 idrac firmware file, but the idrac was 2.4.something.  It seems it had to increment it from the 2.4.x up to 2.61.61.61 and that version also introduced the https error.  I was a...
[18:12:17] <kormat>	 robh: np
[18:12:33] <wikibugs>	 10DBA, 10SRE, 10ops-eqiad: Upgrade db1104 firmware - https://phabricator.wikimedia.org/T286226 (10RobH) a:05RobH→03LSobanski I am at a loss on how to proceed for this, without connecting a crash cart and rolling back the idrac version via crash cart.  Basically the latest idrac firmware made the https id...
[18:27:35] <wikibugs>	 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10Bstorm)
[18:35:22] <wikibugs>	 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10Bstorm) cloudstore servers are both in the same rack.  They are a cluster, and it will simply be offline. We will make a task to verify that it come...
[18:42:04] <wikibugs>	 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10Bstorm)
[18:46:51] <wikibugs>	 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10Bstorm)
[18:47:50] <wikibugs>	 10DBA, 10SRE, 10ops-eqiad: Upgrade db1104 firmware - https://phabricator.wikimedia.org/T286226 (10Kormat) @RobH : as the host is still accessible, i'm going to repool it in the meantime.
[18:48:03] <wikibugs>	 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10Bstorm) Switching cloudmetrics to just eat the brief outage. I don't think it will be a big deal. We can just check it after.
[18:49:32] <wikibugs>	 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10Bstorm) The WMCS-owned dbproxy1018 and 1019 make up the entire cluster, so that's just an outage no matter what for wikireplicas. It will just need do...
[18:57:43] <wikibugs>	 10DBA, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row B - https://phabricator.wikimedia.org/T286061 (10Bstorm) @Gehel I'm not sure who to tag in for cloudelastic here. If there are 2 of them in this row, that could be something that requires attention for redundancy....
[19:08:02] <wikibugs>	 10DBA, 10Infrastructure-Foundations, 10SRE, 10Traffic, and 2 others: Switch buffer re-partition - Eqiad Row A - https://phabricator.wikimedia.org/T286032 (10Bstorm)
[19:08:34] <wikibugs>	 10DBA, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row B - https://phabricator.wikimedia.org/T286061 (10Gehel) Ryan should be around tomorrow to double check, but cloudelastic should be resilient to a row failure. Worst case the service will be down for the duration of...
[19:20:30] <wikibugs>	 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, and 2 others: Switch buffer re-partition - Eqiad Row D - https://phabricator.wikimedia.org/T286069 (10Bstorm)
[19:30:11] <wikibugs>	 10DBA, 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10Bstorm)
[22:34:11] <wikibugs>	 10DBA, 10Commons, 10MediaWiki-File-management, 10MW-1.37-notes (1.37.0-wmf.14; 2021-07-12), and 4 others: Address "image" table capacity problems by storing pdf/djvu text outside file metadata - https://phabricator.wikimedia.org/T275268 (10Ladsgroup) Update: This will take ~fifty days to finish ^
[23:32:54] <wikibugs>	 10Data-Persistence-Backup, 10DC-Ops, 10ops-eqiad: hw troubleshooting: system right cp board missing in new host backup1006 - https://phabricator.wikimedia.org/T286625 (10RobH)
[23:35:53] <wikibugs>	 10Data-Persistence-Backup, 10DC-Ops, 10ops-eqiad: hw troubleshooting: system right cp board missing in new host backup1006 - https://phabricator.wikimedia.org/T286625 (10RobH)
[23:37:17] <wikibugs>	 10Data-Persistence-Backup, 10DC-Ops, 10ops-eqiad: hw troubleshooting: system right cp board missing in new host backup1006 - https://phabricator.wikimedia.org/T286625 (10RobH) a:03Cmjohnson Assigning this to Chris for him to pop this chassis open and investigate if everything is seated.  He returns from hi...