[03:29:28] 06serviceops, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review, 07Wikimedia-Performance-recommendation: Add support for JIT in PHP 8.4 images - https://phabricator.wikimedia.org/T384294#11011717 (10Krinkle) [03:29:37] 06serviceops, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review, 07Wikimedia-Performance-recommendation: Add support for JIT in PHP 8.4 images - https://phabricator.wikimedia.org/T384294#11011719 (10Krinkle) [03:29:43] 06serviceops, 06Data-Engineering, 06Data-Engineering-Radar, 10Dumps-Generation, 06MediaWiki-Platform-Team: Migrate WMF production from PHP 7.4 to PHP 8.1 - https://phabricator.wikimedia.org/T319432#11011720 (10Krinkle) [03:35:17] 06serviceops, 06Data-Engineering, 06Data-Engineering-Radar, 10Dumps-Generation, 06MediaWiki-Platform-Team: Migrate WMF production from PHP 7.4 to PHP 8.1 - https://phabricator.wikimedia.org/T319432#11011722 (10Novem_Linguae) Looks like we've ticked most the boxes and resolved most the subtasks. Might be... [04:56:13] 06serviceops, 06Data-Engineering, 06Data-Engineering-Radar, 10Dumps-Generation, 06MediaWiki-Platform-Team: Migrate WMF production from PHP 7.4 to PHP 8.1 - https://phabricator.wikimedia.org/T319432#11011816 (10Krinkle) [07:55:56] 06serviceops: Create Grafana and Logstash dashboards for MediaWiki migrations/upgrades - https://phabricator.wikimedia.org/T383875#11012057 (10jijiki) [07:55:59] 06serviceops, 10MediaWiki-Platform-Team (Radar): Performance assessment of PHP 8.1 - https://phabricator.wikimedia.org/T383805#11012059 (10jijiki) [07:56:04] 06serviceops, 13Patch-For-Review: MediaWiki on PHP 8.1 production traffic ramp-up - https://phabricator.wikimedia.org/T383845#11012058 (10jijiki) [07:59:54] 06serviceops, 06Data-Engineering, 06Data-Engineering-Radar, 10Dumps-Generation, 06MediaWiki-Platform-Team: Migrate WMF production from PHP 7.4 to PHP 8.1 - https://phabricator.wikimedia.org/T319432#11012061 (10jijiki) [08:00:50] 06serviceops, 06Data-Engineering, 06Data-Engineering-Radar, 10Dumps-Generation, 06MediaWiki-Platform-Team: Migrate WMF production from PHP 7.4 to PHP 8.1 - https://phabricator.wikimedia.org/T319432#11012072 (10jijiki) [08:01:39] 06serviceops, 10MediaWiki-Platform-Team (Radar): Performance assessment of PHP 8 - https://phabricator.wikimedia.org/T383805#11012086 (10jijiki) [08:01:48] 06serviceops, 10MediaWiki-Platform-Team (Radar): Performance assessment of PHP 8 - https://phabricator.wikimedia.org/T383805#11012088 (10jijiki) [08:01:55] 06serviceops, 13Patch-For-Review: MediaWiki on PHP 8.1 production traffic ramp-up - https://phabricator.wikimedia.org/T383845#11012089 (10jijiki) [12:55:22] 06serviceops, 10MediaWiki-Special-pages, 07Wikimedia-production-error: MediaWiki periodic job update-special-pages-s5 failed - https://phabricator.wikimedia.org/T399681#11013096 (10A_smart_kitten) Hey #serviceops, please could an SRE grab the stack trace for this failure if/when you have the time? [13:15:17] 06serviceops, 10MediaWiki-Special-pages, 07Wikimedia-production-error: MediaWiki periodic job update-special-pages-s5 failed - https://phabricator.wikimedia.org/T399681#11013153 (10akosiaris) There you go. ` mgwiktionary Uncategorizedcategories [QueryPage] Wikimedia\Rdbms\DBConnectionError from line... [13:25:03] 06serviceops, 10MediaWiki-Special-pages, 07Wikimedia-production-error: MediaWiki periodic job update-special-pages-s5 failed - https://phabricator.wikimedia.org/T399681#11013199 (10A_smart_kitten) Hmm… what's interesting about this task & the other two most-recent #mediawiki-special-pages cron-job-failure ta... [13:30:56] 06serviceops, 07affects-Kiwix-and-openZIM, 07Wikimedia-Performance-recommendation: large amount of traffic to the action=parse API from MWOffliner - https://phabricator.wikimedia.org/T324866#11013254 (10JArguello-WMF) [13:35:08] Anyone know what this is about? [13:35:09] https://www.irccloud.com/pastebin/KjkbB1HY/ [13:35:23] it's responsible for a month old ProbeDown [13:43:00] 06serviceops, 10MediaWiki-Special-pages, 07Wikimedia-production-error: MediaWiki periodic job update-special-pages-s5 failed - https://phabricator.wikimedia.org/T399681#11013296 (10akosiaris) >>! In T399681#11013197, @A_smart_kitten wrote: > Hmm… what's interesting about this task & the other two most-recent... [14:01:38] urandom: yes, your curl call isn't sending an SNI [14:02:21] speaking for that last openssl connect error [14:03:01] what happens is that the envoy implementing the gateway doesn't know where to send your request (since it's to an IP address and not a hostname) and thus just resets the TLS handshake. [14:03:29] don't you love how verbose and self explaining are errors at the TLS layer? [14:04:19] Ok, that makes sense, and I have it working with curl [14:04:32] but is this also the problem for that ProbeDown? [14:04:47] https://alerts.wikimedia.org/?q=%40state%3Dactive&q=%40cluster%3Dwikimedia.org&q=alertname%3DProbeDown [14:05:02] seems weird this would be the only one... [14:05:55] akosiaris: and yeah, in #-olly I basically said that I suspected the error itself was a red herring [14:06:24] * akosiaris looking [14:08:57] 06serviceops, 10Citoid, 06Data-Engineering, 06Data-Engineering-Radar, and 8 others: Migrate node-based services in production to node18 - https://phabricator.wikimedia.org/T349118#11013379 (10MSantos) [14:09:17] 06serviceops, 10Citoid, 06Data-Engineering, 06Data-Engineering-Radar, and 8 others: Migrate node-based services in production to node18 - https://phabricator.wikimedia.org/T349118#11013386 (10Jgiannelos) [15:00:12] 06serviceops, 13Patch-For-Review, 07PHP 8.1 support: Update PCRE in PHP 8.1 images to PCRE 10.39 or newer - https://phabricator.wikimedia.org/T386006#11013782 (10Scott_French) [15:14:58] 06serviceops: Create Grafana and Logstash dashboards for MediaWiki migrations/upgrades - https://phabricator.wikimedia.org/T383875#11013831 (10jijiki) I have installed nginx-exporter, but sadly the stats we are getting are, well, poor ` nginx_connections_accepted 32013 nginx_connections_active 17 nginx_connectio... [16:10:41] 06serviceops, 06MediaWiki-Engineering, 13Patch-For-Review: Prepare WMF PHP 8.3 packages for bullseye - https://phabricator.wikimedia.org/T398245#11014180 (10MSantos) [16:49:01] 06serviceops, 06MediaWiki-Engineering, 13Patch-For-Review: Prepare WMF PHP 8.3 packages for bullseye - https://phabricator.wikimedia.org/T398245#11014359 (10Scott_French) [16:49:47] 06serviceops, 06MediaWiki-Engineering, 13Patch-For-Review: Prepare WMF PHP 8.3 packages for bullseye - https://phabricator.wikimedia.org/T398245#11014365 (10Scott_French) [17:45:13] 06serviceops: Configure production MediaWiki image builds for PHP 8.3 - https://phabricator.wikimedia.org/T399884 (10Scott_French) 03NEW [17:45:52] 06serviceops: Configure production MediaWiki image builds for PHP 8.3 - https://phabricator.wikimedia.org/T399884#11014632 (10Scott_French) [17:45:54] 06serviceops, 13Patch-For-Review: Prepare PHP 8.3 production images - https://phabricator.wikimedia.org/T398246#11014633 (10Scott_French) [17:46:01] 06serviceops, 06MediaWiki-Engineering, 13Patch-For-Review: Prepare WMF PHP 8.3 packages for bullseye - https://phabricator.wikimedia.org/T398245#11014634 (10Scott_French) [17:46:24] 06serviceops: Configure production MediaWiki image builds for PHP 8.3 - https://phabricator.wikimedia.org/T399884#11014635 (10Scott_French) [17:46:25] 06serviceops, 06MediaWiki-Platform-Team, 07Epic: Migrate Wikimedia production from PHP 8.1 to PHP 8.3 - https://phabricator.wikimedia.org/T360995#11014636 (10Scott_French) [17:46:45] 06serviceops: Configure production MediaWiki image builds for PHP 8.3 - https://phabricator.wikimedia.org/T399884#11014638 (10Scott_French) 05Open→03Stalled [17:47:39] 06serviceops, 13Patch-For-Review: Prepare PHP 8.3 production images - https://phabricator.wikimedia.org/T398246#11014651 (10Scott_French) 05Open→03Resolved These images now exist in the registry (though it may take up to an hour for them to appear in the https://docker-registry.wikimedia.org UI). [18:29:36] 06serviceops, 07Datacenter-Switchover: 🚀 Southward Datacenter Switchover (Sept. 2025) - https://phabricator.wikimedia.org/T399891 (10jasmine_) 03NEW p:05Triage→03Medium [18:29:53] 06serviceops, 07Datacenter-Switchover: 🚀 Southward Datacenter Switchover (Sept. 2025) - https://phabricator.wikimedia.org/T399891#11014830 (10jasmine_) p:05Medium→03Low [18:41:39] 06serviceops, 06collaboration-services, 06Infrastructure-Foundations, 13Patch-For-Review: Upgrade httpd images to bullseye or bookworm - https://phabricator.wikimedia.org/T378128#11014876 (10Scott_French) Alright, both Shellbox and MediaWiki are back on the "normal" httpd images. Once miscweb does the same... [18:45:03] 06serviceops, 07Datacenter-Switchover: 🚀 Southward Datacenter Switchover (Sept. 2025) - https://phabricator.wikimedia.org/T399891#11014881 (10jasmine_) p:05Low→03Medium [19:10:22] 06serviceops, 10MoveComms-Support, 07Datacenter-Switchover: MoveComms support for Southward DC Switchover (September 2025) - https://phabricator.wikimedia.org/T399894 (10jasmine_) 03NEW [19:10:40] 06serviceops, 10MediaWiki-Special-pages, 07Wikimedia-production-error: MediaWiki periodic job update-special-pages-s5 failed - https://phabricator.wikimedia.org/T399681#11014954 (10A_smart_kitten) >>! In T399681#11013296, @akosiaris wrote: >>>! In T399681#11013197, @A_smart_kitten wrote: >> Hmm… what's inter... [19:37:49] 06serviceops, 07Datacenter-Switchover: Assess switchover behavior for mw-wikifunctions - https://phabricator.wikimedia.org/T397874#11015040 (10jasmine_) [19:37:51] 06serviceops, 07Datacenter-Switchover: 🚀 Southward Datacenter Switchover (Sept. 2025) - https://phabricator.wikimedia.org/T399891#11015041 (10jasmine_) [19:40:00] 06serviceops, 07Datacenter-Switchover: Assess switchover behavior for mw-wikifunctions - https://phabricator.wikimedia.org/T397874#11015042 (10jasmine_) a:03jasmine_ [21:41:05] 06serviceops, 06MediaWiki-Engineering: Prepare WMF PHP 8.3 packages for bullseye - https://phabricator.wikimedia.org/T398245#11015362 (10Reedy)