[01:13:33] Is there a reason the ChanServ ACL in this channel was set to private? [06:44:11] legoktm: no, not particularly I guess [06:44:39] happy to change it so that is consistent with the other chans, do you have the command handy by any chance? [06:46:09] volans: /msg ChanServ SET #wikimedia-sre-foundations PUBACL ON [06:50:02] thanks majavah, done [09:01:01] 10SRE-tools, 10DBA, 10Infrastructure-Foundations, 10Spicerack, 10Datacenter-Switchover: switchdc should verify active/active DBs are read-write in both datacenters - https://phabricator.wikimedia.org/T287129 (10LSobanski) Certainly makes sense. To be sure I understand the expectations, who owns making th... [09:02:38] 10netops, 10Analytics, 10DBA, 10Infrastructure-Foundations, and 2 others: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10cmooney) [09:26:11] 10netops, 10Data-Persistence-Backup, 10Infrastructure-Foundations, 10SRE, 10bacula: Understand (and mitigate) the backup speed differences between backup1002->backup2002 and backup2002->backup1002 - https://phabricator.wikimedia.org/T274234 (10cmooney) Thanks @jcrespo. Yes this makes perfect sense. Due... [11:41:24] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE: Switch buffer re-partition - Eqiad Row B - https://phabricator.wikimedia.org/T286061 (10Marostegui) [14:40:55] 10netops, 10Analytics, 10DBA, 10Infrastructure-Foundations, and 2 others: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10ops-monitoring-bot) Icinga downtime set by mmandere@cumin2002 for 1:00:00 4 host(s) and their services with reason: Eqiad row C maintenance ` cp[... [14:42:58] 10netops, 10Analytics, 10DBA, 10Infrastructure-Foundations, and 2 others: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10Vgutierrez) [14:44:31] hi! i just logged into mwmaint1002.eqiad.wmnet to run the media moderation script (https://phabricator.wikimedia.org/T258603) but got a message saying NOT to run maintenance scripts there. Is there a better server to connect to? [14:45:44] It should say in the message [14:46:30] all I see is this in the message "Please connect to the server in the active data center instead." [14:46:34] thanks! [14:46:54] it's mwmaint2002.codfw.wmnet [14:47:02] thank you volans! [14:47:06] Looks like 2002 though [14:47:15] thank you RhinosF1! [14:47:54] There's probably a mailing list you should be on though to hear stuff like that. I'm just a nosey normal person. [14:48:03] i connected and don't see that message anymore :) [14:48:20] yeah, probably, i have a bunch of gmail filters set up so i might have missed it [14:49:31] 10netops, 10Analytics, 10DBA, 10Infrastructure-Foundations, and 2 others: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10herron) [14:50:24] 10netops, 10Analytics, 10DBA, 10Infrastructure-Foundations, and 2 others: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10ops-monitoring-bot) Icinga downtime set by mmandere@cumin2002 for 1:00:00 1 host(s) and their services with reason: Eqiad row C maintenance ` lvs... [14:52:35] I appreciate you being nosey RhinosF1 [14:52:42] I like it [14:52:54] You learn a lot from watching stuff happen [14:55:44] 10netops, 10Analytics, 10DBA, 10Infrastructure-Foundations, and 2 others: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10Vgutierrez) [14:56:57] mepps: FYI the mwmaint server is automatically switched as part of the datacenter switchover. Hence given that we switched to codfw now the codfw's mwmaint server is the one to be used and the other presents the banner to not use it. [14:57:11] AFAIK it was not announced separatly [14:57:23] cool, that's helpful volans [14:59:45] volans are the maintenance boxes supposed to have kafkacat installed? I ask because the documentation for the script I'm running suggests using it but it isn't available to my user [15:01:08] mepps: the kafkacat debian package is currently installed on 46 hosts, but not the mwmaint ones. [15:01:16] thanks volans [15:01:27] althought that seems to be a question more for the Analytics team [15:03:00] that's helpful to know volans [15:04:08] 10netops, 10Analytics, 10DBA, 10Infrastructure-Foundations, and 2 others: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10MoritzMuehlenhoff) [15:07:54] 10netops, 10Analytics, 10DBA, 10Infrastructure-Foundations, and 2 others: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10aborrero) [15:10:26] 10netops, 10Analytics, 10DBA, 10Infrastructure-Foundations, 10SRE: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10herron) [15:22:13] 10netops, 10Analytics, 10DBA, 10Infrastructure-Foundations, 10SRE: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10aborrero) [15:23:10] 10netops, 10Analytics, 10DBA, 10Infrastructure-Foundations, 10SRE: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10cmooney) [15:23:47] 10netops, 10Infrastructure-Foundations, 10SRE: Adjust egress buffer allocations on ToR switches - https://phabricator.wikimedia.org/T284592 (10cmooney) [15:27:33] 10netops, 10DC-Ops, 10SRE, 10ops-codfw, 10Wikimedia-Incident: asw-a2-codfw unresponsive - https://phabricator.wikimedia.org/T286787 (10Papaul) switch shipped out today tracking information below Tracking Number: 1ZA19A021295420730 [15:45:34] second part of the SSO tech blog postings: https://techblog.wikimedia.org/2021/07/22/the-rollout-of-single-sign-on-sso-at-the-wikimedia-foundation/ [15:45:46] nice! [16:49:33] 10CAS-SSO, 10Infrastructure-Foundations, 10SRE, 10User-jbond: Add logout.d script for lists.wikimedia.org - https://phabricator.wikimedia.org/T286906 (10Legoktm) Currently lists.wm.o uses its own independent user database, so I think we'd need to take the LDAP uid, get the associated email address, and the... [17:02:19] 10netops, 10Analytics, 10DBA, 10Infrastructure-Foundations, 10SRE: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10cmooney) All went very well with the change, this time I ran rapid ping from the CR to see if any packet loss was observed, and did detect some loss,... [17:02:35] 10netops, 10Analytics, 10DBA, 10Infrastructure-Foundations, 10SRE: Switch buffer re-partition - Eqiad Row C - https://phabricator.wikimedia.org/T286065 (10cmooney) 05Open→03Resolved [17:02:43] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Adjust egress buffer allocations on ToR switches - https://phabricator.wikimedia.org/T284592 (10cmooney) [17:31:37] moritzm: \o/ [17:31:41] awesome :) [17:34:16] XioNoX, topranks I know lots of stuff are going on right now so apologies to pile on, but from the emails I fwd'ed to you last week the CCD fo the new path for eqiad-esams is today [17:36:55] paravoid: sry somehow missed, looking now thanks for heads up. [17:37:31] I haven't received anything more than the emails I fwd'ed to you [17:37:39] but heads-up maybe it happens later today or something [17:39:59] yeah. it's sufficiently vague I don't think it's worth taking the link out of service pro-actively. [17:41:12] we can probably let is happen and allow BFD/OSPF/BGP to do their jobs. If there was more clarity about the timing it may make sense to take it out of action in advance, but I'm not sure we want to have the link down for an extended period when we don't know the particulars. [17:41:27] I'll see if I can contact them to get an update on status anyway. [17:42:25] nod [17:42:33] I think it's fine to just wait without pinging them for now [17:42:49] I can ping them tomorrow or next week myself too, since I've been doing that for a while now anyway :P [17:43:51] 10CAS-SSO, 10Infrastructure-Foundations, 10SRE, 10User-jbond: Add logout.d script for lists.wikimedia.org - https://phabricator.wikimedia.org/T286906 (10MoritzMuehlenhoff) >>! In T286906#7230602, @Legoktm wrote: > Also to clarify, when we mean "logout", we actually mean "disable account" right? No, the c... [17:46:37] paravoid: sry seen that too late had just fired off a mail. [17:46:50] I'll fwd to you now, had left you off as I'm sure you've enough hitting your inbox. [21:30:16] 10Packaging, 10Infrastructure-Foundations, 10MediaWiki-extensions-Score, 10serviceops: Update Lilypond in Shellbox container to >= 2.22.0, - https://phabricator.wikimedia.org/T287212 (10Legoktm) a:03Legoktm