[04:06:23] <wikibugs>	 10Traffic, 10Anti-Harassment, 10SRE: Enable automatic redirection to the mobile version of votewiki - https://phabricator.wikimedia.org/T288938 (10Niharika) 05Open→03Resolved a:03Niharika Thank you indeed!
[04:47:08] <wikibugs>	 10Traffic, 10DC-Ops, 10SRE, 10Sustainability (Incident Followup): Audit eqiad & codfw LVS network links - https://phabricator.wikimedia.org/T286881 (10Krinkle)
[04:47:36] <wikibugs>	 10netops, 10DC-Ops, 10SRE, 10ops-codfw, 10Wikimedia-Incident: asw-a2-codfw unresponsive - https://phabricator.wikimedia.org/T286787 (10Krinkle)
[04:47:46] <wikibugs>	 10Traffic, 10DC-Ops, 10SRE, 10ops-codfw, 10Sustainability (Incident Followup): lvs2007, lvs2009, lvs2010 should not be on the same row A switch - https://phabricator.wikimedia.org/T286879 (10Krinkle)
[04:47:52] <wikibugs>	 10Traffic, 10SRE, 10Sustainability (Incident Followup): LVS should handle losing a NIC on eqiad and codfw - https://phabricator.wikimedia.org/T286924 (10Krinkle)
[04:49:21] <wikibugs>	 10netops, 10DC-Ops, 10SRE, 10ops-codfw, 10Wikimedia-Incident: asw-a2-codfw unresponsive - https://phabricator.wikimedia.org/T286787 (10Krinkle)
[04:49:27] <wikibugs>	 10Traffic, 10SRE, 10Sustainability (Incident Followup): LVS should handle losing a NIC on eqiad and codfw - https://phabricator.wikimedia.org/T286924 (10Krinkle)
[04:49:33] <wikibugs>	 10Traffic, 10DC-Ops, 10SRE, 10Sustainability (Incident Followup): Audit eqiad & codfw LVS network links - https://phabricator.wikimedia.org/T286881 (10Krinkle)
[08:22:59] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10serviceops, 10Kubernetes: kubernetes1005 BGP down for 3 weeks - https://phabricator.wikimedia.org/T289111 (10ayounsi) p:05Triage→03High
[08:47:40] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10serviceops, 10Kubernetes: kubernetes1005 BGP down for 3 weeks - https://phabricator.wikimedia.org/T289111 (10JMeybohm) a:03JMeybohm
[09:19:12] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10serviceops, and 2 others: kubernetes1005 BGP down for 3 weeks - https://phabricator.wikimedia.org/T289111 (10JMeybohm) This happened while I was running docker pull tests 2021-07-21 ~15:04Z and kubernetes1005 is one of the dedicated sessionstore nodes runnin...
[11:09:11] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10serviceops, and 2 others: kubernetes1005 BGP down for 3 weeks - https://phabricator.wikimedia.org/T289111 (10JMeybohm) K8s event logs (https://logstash.wikimedia.org/goto/b16700661b703799af5ac188db2d3f5c) are pretty clear on that I created a lot of disk pres...
[13:33:48] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10serviceops, and 2 others: kubernetes1005 BGP down for 3 weeks - https://phabricator.wikimedia.org/T289111 (10JMeybohm) 05Open→03Resolved Ok, really dumb situation! A bunch of (failing) sessionstore Pods are clogging all resources on kubernetes1005, leavi...
[14:23:27] <wikibugs>	 10netops, 10Alerting, 10Infrastructure-Foundations, 10SRE, 10SRE Observability (FY2021/2022-Q1): Ingest Cron and Root Alerts Into Logstash - https://phabricator.wikimedia.org/T274377 (10lmata)
[14:26:52] <wikibugs>	 10netops, 10Alerting, 10Infrastructure-Foundations, 10SRE: Ingest Cron and Root Alerts Into Logstash - https://phabricator.wikimedia.org/T274377 (10lmata)
[16:55:22] <wikibugs>	 10Traffic, 10 Data-Engineering, 10Analytics, 10Growth-Team, and 3 others: Migrated Server-side EventLogging events recording http.client_ip as 127.0.0.1 - https://phabricator.wikimedia.org/T288853 (10Mholloway) >>! In T288853#7289672, @Tgr wrote: > IMO there is value in client-side and server-side events i...
[20:28:57] <wikibugs>	 10Traffic, 10 Data-Engineering, 10Analytics, 10Growth-Team, and 4 others: Migrated Server-side EventLogging events recording http.client_ip as 127.0.0.1 - https://phabricator.wikimedia.org/T288853 (10Milimetric) I'm just making sure I didn't miss anything: it looks to me like the instrumentation's just not...
[21:47:48] <wikibugs>	 10Traffic, 10 Data-Engineering, 10Analytics, 10Growth-Team, and 4 others: Migrated Server-side EventLogging events recording http.client_ip as 127.0.0.1 - https://phabricator.wikimedia.org/T288853 (10Tgr) >>! In T288853#7292309, @Mholloway wrote: > Just to be clear, you're advocating here for getting `http...