[09:57:42] 06Traffic, 10DNS, 06SRE, 07Language codes: Redirect legacy language codes for Toki Pona to tok.wikipedia.org - https://phabricator.wikimedia.org/T404507#11180272 (10Tamzin) There are any number of historical links, on-wiki, on the mailing lists, and indeed on Phabricator, referring to tokipona.wikipedia.or... [10:13:39] so after running the cookbook to restart pybal, I am not moving forward here [10:13:39] [9/15, retrying in 27.00s] Attempt to run 'spicerack.icinga.IcingaHosts.wait_for_optimal..check' raised: Not all services are recovered: lvs1020:PyBal backends health check [10:14:13] however here: https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?host=all&type=detail&servicestatustypes=16&hoststatustypes=3&serviceprops=2097162 I see teh alert is gone [10:14:52] mmm could be a false positive [10:15:07] is the cookbook stuck? [10:15:46] it will get to its 15/15 attempts [10:15:53] ==> Failed to downtime hosts: Not all services are recovered: lvs1020:PyBal backends health check [10:16:15] but the restart has happened [10:16:27] that's for `proxoid_4260: Servers urldownloader2004.wikimedia.org are marked down but pooled: k8s-ingress-dse_30443: Servers dse-k8s-worker2002.codfw.wmnet are marked down but pooled` [10:16:27] I wonder if I can just abort then ? [10:16:54] fabfur: yes, but lvs1020 is not complainaing any more [10:17:02] so I assume it restarted ok [10:17:13] I'd let it timeout, in case there are some cleanup tasks after [10:17:27] it is here [10:17:27] ==> Failed to downtime hosts: Not all services are recovered: lvs1020:PyBal backends health check [10:17:27] Type "go" to proceed or "abort" to interrupt the execution [10:17:38] I'd "go" [10:17:39] if I go, I believe it will timeout again [10:17:51] ah cool it is happy [10:17:57] :) [10:18:00] alright let me proceed with codfw secondary [10:18:04] if he's happy we're happy :) [10:18:51] haha [10:18:59] effie: pybal still complaining about urldownloader though :( [10:19:55] elukey: on 1020 ? [10:20:02] yeah [10:20:16] https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?host=all&type=detail&servicestatustypes=16&hoststatustypes=3&serviceprops=2097162 [10:20:18] it is not here [10:20:20] sigh [10:20:28] where/what do you see? [10:20:57] [proxoid_4260 ProxyFetch] WARN: urldownloader1003.wikimedia.org (enabled/down/pooled): Fetch failed etc.. [10:21:04] in /var/log/pybal.log [10:22:51] sigh [10:26:42] fabfur: I may need some help here [10:29:48] one thing that I noticed is that proxoid doesn't have the ipip_encapsulation [10:30:41] I do not know what/how it is supposed to be [10:30:54] lets move this to the other channel [10:34:30] ack [11:38:57] 06Traffic, 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q4:rack/setup/install cp20[43-58] codfw - https://phabricator.wikimedia.org/T392851#11180482 (10Jhancock.wm) @elukey I forgot to comment when i finished up last week. I got 2049 fixed and go 50-52 ready to go. However, we've had an issue in the... [15:20:12] 06Traffic, 06DC-Ops, 10ops-codfw, 06SRE, 13Patch-For-Review: Q4:rack/setup/install cp20[43-58] codfw - https://phabricator.wikimedia.org/T392851#11181269 (10elukey) @Jhancock.wm thanks! I tried 2049 today and I ended up with: ` [15:22:18] 06Traffic, 06SRE, 13Patch-For-Review, 07User-notice: Block traffic from user-agents not honoring our policy - https://phabricator.wikimedia.org/T400119#11181281 (10bd808) [15:40:36] 06Traffic, 10DNS, 06SRE, 07Language codes: Redirect legacy language codes for Toki Pona to tok.wikipedia.org - https://phabricator.wikimedia.org/T404507#11181396 (10Pppery) > Both of these codes are also referenced in any old revisions containing [[tp:]] or [[tokipona:]] langlinks, so there's a backwards c... [15:40:39] 06Traffic, 10DNS, 06SRE, 07Language codes: Redirect legacy language codes for Toki Pona to tok.wikipedia.org - https://phabricator.wikimedia.org/T404507#11181397 (10Pppery) [16:13:51] 06Traffic, 10DNS, 06SRE, 07Language codes: Redirect legacy language codes for Toki Pona to tok.wikipedia.org - https://phabricator.wikimedia.org/T404507#11181616 (10taavi) [16:14:16] 06Traffic, 10DNS, 06SRE, 07Language codes: Redirect legacy language codes for Toki Pona to tok.wikipedia.org - https://phabricator.wikimedia.org/T404507#11181623 (10taavi) Per the task description. This is not actually blocked on the wiki creation, nor is this required for the wiki itself to actually funct... [16:17:07] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad: eqiad: rows C/D Upgrade Tracking - https://phabricator.wikimedia.org/T404609 (10RobH) 03NEW p:05Triage→03Medium [16:18:46] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad: eqiad: rows C/D Upgrade Tracking - https://phabricator.wikimedia.org/T404609#11181650 (10RobH) @cmooney: What do you think is the best way to go about migrating these connections on upcoming C/D updates? The new switch will be online in the ra... [16:40:44] ryankemper: Would you like to schedule time to handle the wdqs lvs setup? [17:36:09] FIRING: LVSHighRX: Excessive RX traffic on lvs5005:9100 (ens1f0np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs5005 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [17:46:09] RESOLVED: LVSHighRX: Excessive RX traffic on lvs5005:9100 (ens1f0np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs5005 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [19:37:28] 06Traffic, 10DNS, 06SRE, 07Language codes: Redirect legacy language codes for Toki Pona to tok.wikipedia.org - https://phabricator.wikimedia.org/T404507#11182387 (10Dzahn) This type of redirect/rewrite would likely have to be handled in the appserver apache config rather than the CDN. That would mean serv... [19:56:40] FIRING: [9x] VarnishHighThreadCount: Varnish's thread count on cp1100:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [20:01:40] FIRING: [19x] VarnishHighThreadCount: Varnish's thread count on cp1100:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [20:01:43] FIRING: [6x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [20:06:21] 06Traffic, 10Community-Tech (Sea Lion Squad), 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review, 07SEO: Suppress mobile redirect for Googlebot Smartphone on Commons - https://phabricator.wikimedia.org/T397267#11182523 (10Krinkle) Googlebot is now (back) to indexing the mobile version of Wikimedia... [20:06:40] FIRING: [19x] VarnishHighThreadCount: Varnish's thread count on cp1100:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [20:06:43] FIRING: [8x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [20:11:40] FIRING: [20x] VarnishHighThreadCount: Varnish's thread count on cp1100:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [20:16:40] FIRING: [20x] VarnishHighThreadCount: Varnish's thread count on cp1100:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [20:16:43] RESOLVED: [8x] HaproxyKafkaSocketDroppedMessages: Sustained high rate of dropped messages from HaproxyKafka - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaSocketDroppedMessages - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaSocketDroppedMessages [20:31:40] FIRING: [18x] VarnishHighThreadCount: Varnish's thread count on cp1100:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [20:39:13] 06Traffic, 10DNS, 06serviceops, 06SRE, 07Language codes: Redirect legacy language codes for Toki Pona to tok.wikipedia.org - https://phabricator.wikimedia.org/T404507#11182687 (10A_smart_kitten) [20:41:40] FIRING: [33x] VarnishHighThreadCount: Varnish's thread count on cp1100:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [20:58:36] hello traffic, anyone comfortable reviewing https://gerrit.wikimedia.org/r/1187126 in v.gutierrez's absence? [21:01:40] RESOLVED: [16x] VarnishHighThreadCount: Varnish's thread count on cp1100:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [21:11:06] rzl: I'll take a look in a moment [21:12:15] thanks! [21:31:21] brett: much appreciated :) [21:31:32] my pleasure [21:32:35] 06Traffic, 07Documentation: Document x-cache-status header on Wikitech - https://phabricator.wikimedia.org/T404654 (10aaron) 03NEW