[03:51:57] <jinxer-wm>	 (HAProxyEdgeTrafficDrop) firing: 66% request drop in text@eqsin during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=eqsin&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop
[03:56:57] <jinxer-wm>	 (HAProxyEdgeTrafficDrop) resolved: 69% request drop in text@eqsin during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=eqsin&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop
[03:58:57] <jinxer-wm>	 (HAProxyEdgeTrafficDrop) firing: 64% request drop in text@eqsin during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=eqsin&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop
[04:03:56] <jinxer-wm>	 (HAProxyEdgeTrafficDrop) resolved: 69% request drop in text@eqsin during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=eqsin&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop
[06:42:10] <wikibugs>	 10netops, 10Infrastructure-Foundations: Set consistent MTUs - https://phabricator.wikimedia.org/T315838 (10ayounsi) p:05Triage→03Low
[06:42:34] <wikibugs>	 10netops, 10Infrastructure-Foundations: Set consistent MTUs - https://phabricator.wikimedia.org/T315838 (10ayounsi)
[09:55:23] <wikibugs>	 10Traffic, 10MediaWiki-General, 10SRE, 10MW-1.39-notes (1.39.0-wmf.23; 2022-08-01), 10Patch-For-Review: Roll out query parameter normalization - https://phabricator.wikimedia.org/T314868 (10ori)
[10:04:49] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad, 10Sustainability (Incident Followup): eqiad row C switch fabric recabling - https://phabricator.wikimedia.org/T313384 (10ayounsi) I see that the subtask got resolved, nice!  Please run the new/additional cables without connecting them. Once done...
[13:39:43] <wikibugs>	 10Varnish, 10Analytics-Radar, 10SRE, 10Traffic-Icebox, 10Privacy: Add request_id to webrequest logs as well as other event records ingested into Hadoop - https://phabricator.wikimedia.org/T113817 (10Ottomata) a:05Ottomata→03None
[13:50:52] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10netbox, 10Patch-For-Review: Represent sub-interface and bridge device assocations in Netbox - https://phabricator.wikimedia.org/T296832 (10cmooney) I had a good discussion with @jbond on irc about how we model the host interfaces in Netbox, and I think bas...
[15:11:52] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE Observability (FY2022/2023-Q1): Rancid on netmon1003 unable to login to network devices - https://phabricator.wikimedia.org/T314936 (10lmata)
[16:02:13] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: Link from lsw1-e1-eqiad to lsw1-f2-eqiad down - https://phabricator.wikimedia.org/T315052 (10wiki_willy) a:05Cmjohnson→03Jclark-ctr
[16:21:59] <brett>	 kwakuofori: Would it make sense to set up a traffic team RIPE atlas account rather than have it all under mine?
[16:22:36] <brett>	 Not sure what WMF policy is for credential sharing - I don't like it but feel it might be better than putting everything under a single users' account
[16:22:46] <brett>	 (I don't see any sort of group membership feature)
[16:23:05] <kwakuofori>	 yes, makes sense. let's discuss how we satisfy the email requirement for the account
[16:24:23] <mutante>	 you can send an email to techsupport@  to get a mailing list / alias
[16:34:04] <brett>	 Sounds good. ripeatlas@wikimedia.org?
[16:34:54] <paravoid>	 for what?
[16:35:29] <paravoid>	 (sorry for being terse, trying to follow Amir too) our Atlas account is under our LIR IIRC
[16:36:45] <brett>	 paravoid: For latency measurements, which are paid for in credits
[16:37:31] <paravoid>	 are you using the wikimedia ripe atlas account?
[16:37:39] <paravoid>	 or a personal one?
[16:38:20] <brett>	 paravoid: I wasn't aware that there was a wikimedia account. Presently using my wikimedia email
[16:38:49] <brett>	 kwakuofori: Would using that account work?
[16:40:26] <paravoid>	 brett: I used the "share access" feature and added your work email
[16:40:51] <paravoid>	 I think there is another setting somewhere to give you access to that account
[16:41:14] <brett>	 oh nice
[16:41:17] <paravoid>	 access has always been a little weird
[16:41:49] <mutante>	 should that be in pwstore repo?
[16:42:29] <brett>	 Is it a shared account or a membership-based group? Haven't been able to see anything resembling the latter
[16:42:37] <paravoid>	 I'm not entirely sure
[16:42:46] <brett>	 really nicely designed :D
[16:43:15] <paravoid>	 I set up our anchors back in the day under my account, and at the time they had in the roadmap something about org accounts
[16:43:42] <paravoid>	 we are ripe members and have multiple LIR accounts with them, so it doesn't make sense for that to be all disjoint
[16:43:50] <paravoid>	 I'm not sure if they ever fixed that
[16:45:13] <paravoid>	  Management Sharing
[16:45:14] <paravoid>	 The LIR us.wmf has shared administration of this probe.
[16:45:17] <paravoid>	 that's for our anchors
[16:45:25] <paravoid>	 so that exists
[16:45:31] <paravoid>	 not sure how the credits are issued though
[17:56:35] <cdanis>	 (I can shed some small insight on this in a bit)
[17:57:05] <wikibugs>	 10Traffic: ATS Read While Writer feature is wrongly configured - https://phabricator.wikimedia.org/T315911 (10Vgutierrez)
[18:14:27] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE Observability (FY2022/2023-Q1): Rancid on netmon1003 unable to login to network devices - https://phabricator.wikimedia.org/T314936 (10andrea.denisse) Fixed in the following patches:  1. [[ https://gerrit.wikimedia.org/r/822196 | #822196 - netmon: Create...
[18:15:00] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE Observability (FY2022/2023-Q1): Rancid on netmon1003 unable to login to network devices - https://phabricator.wikimedia.org/T314936 (10andrea.denisse) 05Open→03Resolved
[20:39:32] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: Link from lsw1-e1-eqiad to lsw1-f2-eqiad down - https://phabricator.wikimedia.org/T315052 (10Jclark-ctr) replaced qsfp in e1 port 54
[20:44:27] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: Link from lsw1-e1-eqiad to lsw1-f2-eqiad down - https://phabricator.wikimedia.org/T315052 (10cmooney) Thanks @Jclark-ctr that seems to have done it:  ` cmooney@lsw1-e1-eqiad> show interfaces et-0/0/54     Aug 22 20:37:45 Physical interface: et-0/0...
[20:45:10] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: Link from lsw1-e1-eqiad to lsw1-f3-eqiad down - https://phabricator.wikimedia.org/T315052 (10cmooney) 05Open→03Resolved
[23:08:33] <wikibugs>	 10Traffic, 10Performance-Team, 10SRE, 10SRE-swift-storage, and 2 others: Progressive Multi-DC roll out - https://phabricator.wikimedia.org/T279664 (10tstarling) The rollout was reverted back to stage 0 on August 15 due to T315271. I just reverted the revert, so it will be running on mediawiki.org once the...