[10:35:48] <wikibugs>	 06Traffic: Map ISPs in Maxmind db, used in turnilo/superset, to use in requestctl rule - https://phabricator.wikimedia.org/T392219#10779756 (10Fabfur) Other possibilities to achieve the same result are:  - Looking up MaxMind DB entries directly from LUA (in HAProxy) without using map files: https://github.com/an...
[11:07:35] <wikibugs>	 06Traffic, 06Experimentation Lab, 13Patch-For-Review: SDS 2.4.4 Edge Uniques Production Cookie Deployment - https://phabricator.wikimedia.org/T391411#10779850 (10Vgutierrez)
[11:19:57] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw: Upgrade codfw E/F Juniper equipment to Junos 23.x - https://phabricator.wikimedia.org/T393001 (10ayounsi) 03NEW
[12:21:07] <wikibugs>	 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: Enable gNMI on SRX devices and fasw - https://phabricator.wikimedia.org/T390052#10780029 (10ayounsi) While the equivalent of this is needed on the pfw:  ` [edit security zones security-zone production interfaces lo0.0 host-inbound-traffic system-ser...
[13:18:23] <wikibugs>	 06Traffic, 06[Archived]Wikidata Dev Team, 10Prod-Kubernetes, 06SRE, and 5 others: Frequent 500 Errors and Timeouts When Adding Statements to New Item or Lexeme-typed Properties - https://phabricator.wikimedia.org/T374230#10780239 (10ArthurPSmith) Hi - has this been done yet? I'm ready to test it on live Wi...
[14:32:03] <wikibugs>	 06Traffic: haproxykafka service isn't restarted when upgraded - https://phabricator.wikimedia.org/T393016 (10Fabfur) 03NEW
[14:32:18] <wikibugs>	 06Traffic: haproxykafka service isn't restarted when upgraded - https://phabricator.wikimedia.org/T393016#10780631 (10Fabfur)
[14:48:17] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: Link down between cr3-ulsfo and cr4-ulsfo - https://phabricator.wikimedia.org/T390731#10780708 (10RobH) New remote hands entered to get this fixed: Case Order #01053614    > Directions for remote hands to repair our link between cr3 an...
[15:56:30] <wikibugs>	 06Traffic, 06Infrastructure-Foundations, 10Spicerack, 10SRE-tools, 13Patch-For-Review: Spicerack's Icinga module should provide a way to skip specific services in sub-optimal but desired state - https://phabricator.wikimedia.org/T392848#10780975 (10elukey) I filed https://gerrit.wikimedia.org/r/c/operati...
[16:04:11] <wikibugs>	 06Traffic, 06[Archived]Wikidata Dev Team, 10Prod-Kubernetes, 06SRE, and 5 others: Frequent 500 Errors and Timeouts When Adding Statements to New Item or Lexeme-typed Properties - https://phabricator.wikimedia.org/T374230#10781006 (10ArthurPSmith) Since it's well after 10:00 UTC I gave it a try - problem is...
[17:01:17] <wikibugs>	 06Traffic, 06[Archived]Wikidata Dev Team, 10Prod-Kubernetes, 06SRE, and 5 others: Frequent 500 Errors and Timeouts When Adding Statements to New Item or Lexeme-typed Properties - https://phabricator.wikimedia.org/T374230#10781355 (10ArthurPSmith) Hmm, it seems to have resolved now. Maybe I'll try another o...
[17:11:51] <wikibugs>	 06Traffic, 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE HAProxy Migration: Add HAproxy termination field to webrequest - https://phabricator.wikimedia.org/T387454#10781416 (10Ahoelzl)
[17:37:34] <brett>	 ryankemper: As gerrit seems to be toast for the forseeable future, I don't think we'll make our schedule. Do you want to reschedule?
[17:43:06] <wikibugs>	 06Traffic, 06[Archived]Wikidata Dev Team, 10Prod-Kubernetes, 06SRE, and 5 others: Frequent 500 Errors and Timeouts When Adding Statements to New Item or Lexeme-typed Properties - https://phabricator.wikimedia.org/T374230#10781527 (10ArthurPSmith) Nope - new property frozen also as soon as I added an exampl...
[17:58:29] <inflatador>	 brett ryankemper do we need to reschedule the teardown due to the gerrit outage?
[17:58:57] <inflatador>	 oops! I see you've addressed this
[17:58:58] <ryankemper>	 brett: inflatador: yes just hopping online now but if gerrit is down let’s definitely push to tomorrow
[17:58:59] <brett>	 Sounds like we do
[18:01:54] <inflatador>	 Cool, I'll head to lunch then. Will check the updated invite when I get back
[18:12:36] <brett>	 ryankemper: I have a free schedule tomorrow
[20:22:51] <jinxer-wm>	 FIRING: FermMSS: Unexpected MSS value on 10.2.2.27:80 @ ms-fe1016 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=eqiad&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS
[20:27:51] <jinxer-wm>	 RESOLVED: FermMSS: Unexpected MSS value on 10.2.2.27:80 @ ms-fe1016 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=eqiad&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS
[20:28:51] <jinxer-wm>	 FIRING: FermMSS: Unexpected MSS value on 10.2.2.27:80 @ ms-fe1016 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=eqiad&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS
[20:33:06] <jinxer-wm>	 RESOLVED: FermMSS: Unexpected MSS value on 10.2.2.27:80 @ ms-fe1016 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=eqiad&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS
[20:52:02] <sukhe>	 hmmm
[20:54:16] <mutante>	 sukhe: that machine was reimaged today
[20:54:28] <mutante>	 in https://phabricator.wikimedia.org/T388886
[20:54:46] <mutante>	 if that was what you were wondering
[20:57:06] <sukhe>	 ah thanks mutante
[20:58:32] <mutante>	 doesnt explain why the MSS value alerts.. but .. yea.. 
[21:06:56] <sukhe>	 in cases of reimages/reboots, this may be a false positive (because of when the MSS clamping happens), so the timing does match up
[21:07:21] <sukhe>	 which it was in this case and hence the automatic resolve
[21:07:35] <sukhe>	 but I wasn't aware of the reimage until you pointed it out
[21:09:36] <mutante>	 ok, good
[21:23:27] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: Link down between cr3-ulsfo and cr4-ulsfo - https://phabricator.wikimedia.org/T390731#10782143 (10RobH) a:05RobH→03cmooney @cmooney :    >        "Created by: mmariscalmata The following has been completed: >  > Retrieve package #1...
[21:27:07] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: Link down between cr3-ulsfo and cr4-ulsfo - https://phabricator.wikimedia.org/T390731#10782147 (10cmooney) Thanks @RobH.  It looks good so far, this is the graph we need to keep an eye on:  https://grafana.wikimedia.org/goto/SVEEkIbHR...