[01:16:51] FIRING: FermMSS: Unexpected MSS value on 10.2.1.27:80 @ ms-fe2010 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [01:21:52] RESOLVED: FermMSS: Unexpected MSS value on 10.2.1.27:80 @ ms-fe2010 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [03:10:25] 06Traffic, 06Commons, 06DBA, 06SRE: Unable to save edits or delete pages on Commons – database lag - https://phabricator.wikimedia.org/T402749#11122138 (10Zache) > However, I remain concerned that a determined attacker or a widely used non-compliant script could create the same load again. This risk hi... [04:19:51] FIRING: FermMSS: Unexpected MSS value on 10.2.1.27:80 @ ms-fe2010 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [04:24:51] RESOLVED: FermMSS: Unexpected MSS value on 10.2.1.27:80 @ ms-fe2010 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [04:46:40] 06Traffic, 06SRE, 10Wikidata, 10Wikidata-Query-Service: Find a solution for SPARQL federation that is blocked by stricter user agent policy enforcement - https://phabricator.wikimedia.org/T402959#11122213 (10Abbe98) Affected SPARQL backends appear to at least include Fuseki and Virtuoso. [05:06:51] FIRING: FermMSS: Unexpected MSS value on 10.2.1.27:80 @ ms-fe2010 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [05:11:51] RESOLVED: FermMSS: Unexpected MSS value on 10.2.1.27:80 @ ms-fe2010 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [05:39:53] 10netops, 06DC-Ops, 06Infrastructure-Foundations: Take advantage of 10Gb NICs in the new network stack - https://phabricator.wikimedia.org/T360297#11122271 (10ayounsi) 05Open→03Resolved a:03ayounsi Closing that task as more specific subtasks exists. [06:56:16] 06Traffic, 06Commons, 06DBA, 06SRE: Unable to save edits or delete pages on Commons – database lag - https://phabricator.wikimedia.org/T402749#11122371 (10Josve05a) (I meant to edit my comment but deleted it… ugh) [07:55:51] FIRING: FermMSS: Unexpected MSS value on 10.2.1.27:80 @ ms-fe2010 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [07:59:34] 06Traffic, 06Data-Engineering: Reduce noise from duplicate sequence-gap alerts on HaProxy-webrequests - https://phabricator.wikimedia.org/T401383#11122421 (10Antoine_Quhen) 05Resolved→03Open The sequence for bad request id is no more 0. That's good news. But I'm still observing a gap after each bad reques... [08:00:51] RESOLVED: FermMSS: Unexpected MSS value on 10.2.1.27:80 @ ms-fe2010 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [08:26:36] 06Traffic, 06Commons, 06DBA, 06SRE: Unable to save edits or delete pages on Commons – database lag - https://phabricator.wikimedia.org/T402749#11122458 (10Aklapper) @Josve05a: I could re-post your comment from my bugmail copy, if you want me to? [08:28:48] Hello, planning to Add dse-k8s-codfw nodes to BGP in the next 10 mins would this be an appropriate time or should I do it a bit later on? [08:41:37] WDYM with adding dse-k8s-codfw nodes to BGP? [08:43:56] the config referenced on https://wikitech.wikimedia.org/wiki/Kubernetes/Clusters/Add_or_remove_nodes#Step_1:_Add_node_to_BGP ie. set the BGP value to true on netbox and run homer from the cumin host [08:44:36] for the ctrl hosts and worker hosts [08:46:05] oh right.. that's for netops then :) [08:47:14] ack, thanks vgutierrez [08:50:51] FIRING: FermMSS: Unexpected MSS value on 10.2.1.27:80 @ ms-fe2010 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [08:55:51] RESOLVED: FermMSS: Unexpected MSS value on 10.2.1.27:80 @ ms-fe2010 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [09:38:23] 06Traffic, 06Commons, 06DBA, 06SRE: Unable to save edits or delete pages on Commons – database lag - https://phabricator.wikimedia.org/T402749#11122675 (10Bugreporter) >>! In T402749#11122138, @Zache wrote: >> However, I remain concerned that a determined attacker or a widely used non-compliant script... [09:39:39] 06Traffic, 06SRE, 10Wikidata, 10Wikidata-Query-Service: Find a solution for SPARQL federation that is blocked by stricter user agent policy enforcement - https://phabricator.wikimedia.org/T402959#11122678 (10Lucas_Werkmeister_WMDE) [09:52:51] FIRING: FermMSS: Unexpected MSS value on 10.2.1.27:80 @ ms-fe2010 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [09:57:51] RESOLVED: FermMSS: Unexpected MSS value on 10.2.1.27:80 @ ms-fe2010 - https://wikitech.wikimedia.org/wiki/LVS#LVSRealserverMSS_alert - https://grafana.wikimedia.org/d/Y9-MQxNSk/ipip-encapsulated-services?orgId=1&viewPanel=4&var-site=codfw&var-cluster=swift - https://alerts.wikimedia.org/?q=alertname%3DFermMSS [10:01:27] 06Traffic, 06Data-Engineering: Request for a new request dataset for caching research - https://phabricator.wikimedia.org/T401331#11122841 (10yazhuoz) @ssingh Thanks for your reply! I'm not sure if it's possible, but it would be super helpful if the logs could include some user-agent and referer info (even in... [10:04:43] FIRING: [5x] HaproxyKafkaSocketDroppedMessages: Unexpected rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [10:09:43] RESOLVED: [8x] HaproxyKafkaSocketDroppedMessages: Unexpected rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages