[00:38:57] 06serviceops, 06SRE: mwscript-cleanup.service failure - https://phabricator.wikimedia.org/T390790#10701976 (10RLazarus) a:03RLazarus [04:57:02] 06serviceops, 06Abstract Wikipedia team: Provide guidance on how to use apache bench to benchmark requests not through SSL for production services - https://phabricator.wikimedia.org/T390099#10702206 (10ecarg) Yes ty, @akosiaris~ We want to perform requests such as: ` curl https://wikifunctions.discovery.wmne... [05:49:44] 06serviceops, 10RESTBase Sunsetting, 07User-notice-archive: Switchover plan from RESTbase to REST Gateway for rest_v1/page/html and rest_v1/page/title endpoints - https://phabricator.wikimedia.org/T374683#10702239 (10Legoktm) >>! In T374683#10623045, @daniel wrote: > Report for what incident? The missing... [06:56:52] 06serviceops: docker-registry.wikimedia.org keeps serving bad blobs - https://phabricator.wikimedia.org/T390251#10702282 (10elukey) I've rebooted registry2004 to keep the cluster in the same state, since yesterday I had to do the same for 2005 to figure out why nginx wasn't logging (root cause was the root parti... [08:25:23] 06serviceops, 06Abstract Wikipedia team: Provide guidance on how to use apache bench to benchmark requests not through SSL for production services - https://phabricator.wikimedia.org/T390099#10702514 (10akosiaris) Cool. Look at T389375#10692618 for an almost identical example of how to use `siege` (again ab ju... [09:06:38] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10702693 (10Clement_Goubert) Yeah sure, fine by me, at least it's the last in the range so easy to keep it separated :D [10:49:43] 06serviceops, 07Wikimedia-production-error: Error for wmf.14 (while wmf.21 should be live) Uncaught MediaWiki\Config\ConfigException: Failed to load configuration from etcd: (curl error: 1) Unsupported protocol in /srv/mediawiki/php-1.44.0-wmf.14/includes/con... - https://phabricator.wikimedia.org/T389877#10703140 [11:43:36] 06serviceops, 06cloud-services-team, 10Cloud-VPS: OOM livelock stalls - https://phabricator.wikimedia.org/T358634#10703281 (10jijiki) [11:46:22] 06serviceops, 06cloud-services-team, 10Cloud-VPS: OOM livelock stalls - https://phabricator.wikimedia.org/T358634#10703290 (10jijiki) 05Open→03Stalled [11:47:00] 06serviceops, 10observability, 10Prod-Kubernetes, 07Kubernetes: Increase visibility of kubernetes network status - https://phabricator.wikimedia.org/T356877#10703291 (10jijiki) 05Open→03Stalled [12:18:18] 06serviceops, 06DC-Ops, 10ops-codfw, 06SRE: Q3:rack/setup/install wikikube-worker2248-2331, wikikube-ctrl2004-2005 - https://phabricator.wikimedia.org/T384970#10703399 (10Jhancock.wm) 05Open→03Resolved i'm gonna mark this task as resolved but i'll keep worker2331 on my list to check back on once in... [12:30:15] 06serviceops, 07Datacenter-Switchover, 07User-notice: MoveComms support for March 2025 Datacentre switchover - https://phabricator.wikimedia.org/T387444#10703433 (10jijiki) 05In progress→03Resolved [12:33:13] 06serviceops: Create Grafana and Logstash dashboards for MediaWiki migrations/upgrades - https://phabricator.wikimedia.org/T383875#10703443 (10jijiki) [12:33:30] 06serviceops: Create Grafana and Logstash dashboards for MediaWiki migrations/upgrades - https://phabricator.wikimedia.org/T383875#10703444 (10jijiki) 05Open→03Stalled p:05Triage→03Medium [13:08:02] 06serviceops, 06Abstract Wikipedia team, 07Wikimedia-production-error: Partial mw-wikifunctions outage; 404s on load.php and others? - https://phabricator.wikimedia.org/T390854#10703583 (10Jdforrester-WMF) This one fails consistently, for instance: https://www.wikifunctions.org/w/load.php?lang=en&modules=ext... [13:20:54] 06serviceops, 06Infrastructure-Foundations, 10Prod-Kubernetes: Kubernetes dependency upgrade in spicerack - https://phabricator.wikimedia.org/T390857 (10Volans) 03NEW [13:29:57] 06serviceops: wikikube-worker2[248-331] implementation tracking - https://phabricator.wikimedia.org/T390859 (10Clement_Goubert) 03NEW [13:32:43] 06serviceops: wikikube-worker2[248-331] implementation tracking - https://phabricator.wikimedia.org/T390859#10703696 (10Clement_Goubert) [13:36:41] 06serviceops, 06Abstract Wikipedia team, 07Wikimedia-production-error: Partial mw-wikifunctions outage; 404s on load.php and others? - https://phabricator.wikimedia.org/T390854#10703718 (10akosiaris) https://gerrit.wikimedia.org/r/c/operations/puppet/+/1133363 is the reason for this, it was reverted in https... [13:37:04] !log depool cp3066 for debugging T390854 [13:40:24] 06serviceops: wikikube-ctrl200[4-5] implementation tracking - https://phabricator.wikimedia.org/T390861 (10Clement_Goubert) 03NEW [13:40:34] 06serviceops: wikikube-ctrl200[4-5] implementation tracking - https://phabricator.wikimedia.org/T390861#10703747 (10Clement_Goubert) [13:41:03] 06serviceops, 06Abstract Wikipedia team, 06Traffic, 07Wikimedia-production-error: Partial mw-wikifunctions outage; 404s on load.php and others? - https://phabricator.wikimedia.org/T390854#10703770 (10akosiaris) Adding #traffic, since this involves ATS [13:42:40] 06serviceops: wikikube-worker2[248-331] implementation tracking - https://phabricator.wikimedia.org/T390859#10703776 (10Clement_Goubert) [13:47:00] 06serviceops, 06Abstract Wikipedia team: Provide guidance on how to use apache bench to benchmark requests not through SSL for production services - https://phabricator.wikimedia.org/T390099#10703800 (10Jdforrester-WMF) >>! In T390099#10702514, @akosiaris wrote: > Cool. Look at T389375#10692618 for an almost i... [13:48:45] 06serviceops, 06Infrastructure-Foundations: Redis dependency upgrade in spicerack - https://phabricator.wikimedia.org/T390864 (10Volans) 03NEW [13:51:52] 06serviceops, 06Abstract Wikipedia team, 06Traffic, 07Wikimedia-production-error: Partial mw-wikifunctions outage; 404s on load.php and others? - https://phabricator.wikimedia.org/T390854#10703816 (10akosiaris) `lang=bash deploy1003:~$ siege -c 2 -r 100 --no-parser --no-follow -H "Host: www.wikifunctions.... [13:59:25] 06serviceops, 06Abstract Wikipedia team, 06Traffic, 07Wikimedia-production-error: Partial mw-wikifunctions outage; 404s on load.php and others? - https://phabricator.wikimedia.org/T390854#10703865 (10akosiaris) After depooling the node and running 3 consecutive invocations of the above, no 404s observed at... [14:04:57] 06serviceops, 06cloud-services-team, 10Cloud-VPS: OOM livelock stalls - https://phabricator.wikimedia.org/T358634#10703926 (10Andrew) p:05Triage→03Medium [15:03:35] 06serviceops, 13Patch-For-Review: Migrate mw-script to PHP 8.1 - https://phabricator.wikimedia.org/T387917#10704445 (10Scott_French) [15:21:50] 06serviceops, 06SRE, 13Patch-For-Review: mwscript-cleanup.service failure - https://phabricator.wikimedia.org/T390790#10704587 (10RLazarus) 05Open→03Resolved ` Apr 02 15:20:03 deploy1003 systemd[1]: Starting Remove lingering Helm releases from completed maintenance scripts.... Apr 02 15:20:04 deploy1... [15:59:22] 06serviceops, 06Data-Engineering, 06Data-Engineering-Radar, 10Dumps-Generation, 06MediaWiki-Platform-Team: Migrate WMF production from PHP 7.4 to PHP 8.1 - https://phabricator.wikimedia.org/T319432#10704726 (10taavi) [16:31:47] 06serviceops, 06Abstract Wikipedia team: Provide guidance on how to use apache bench to benchmark requests not through SSL for production services - https://phabricator.wikimedia.org/T390099#10704910 (10akosiaris) >>! In T390099#10703800, @Jdforrester-WMF wrote: >>>! In T390099#10702514, @akosiaris wrote: >> C... [18:53:32] 06serviceops: docker-registry.wikimedia.org keeps serving bad blobs - https://phabricator.wikimedia.org/T390251#10705647 (10Scott_French) This happened again today during a backport deployment (stated at https://sal.toolforge.org/log/Glpq95UB8tZ8Ohr00jdL). Now, somewhat surprisingly for a backport, this was a f...