[00:24:24] 06cloud-services-team, 10Toolforge: re-enable wmtran tool at toolforge - https://phabricator.wikimedia.org/T392408 (10Gryllida) 03NEW [01:02:05] 06cloud-services-team, 10Toolforge: re-enable wmtran tool at toolforge - https://phabricator.wikimedia.org/T392408#10758211 (10Wargo) > My tool is only a PHP/JS page, it does not require any changes. Web apps also uses job engines, even if you don't know you use it. Use these instructions: https://wikitech.w... [04:35:46] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.drain_node (exit_code=99) [08:56:39] 06cloud-services-team, 10Toolforge, 03Wikimedia-Hackathon-2025: [Session] Introducing and exploring Toolforge UI with prospective users - https://phabricator.wikimedia.org/T383149#10758502 (10Sarai-WMF) >>! In T383149#10754006, @debt wrote: > Got it - it's been moved to 10am on Saturday. :) Sounds good too!... [11:15:16] 06cloud-services-team, 10Toolforge: re-enable wmtran tool at toolforge - https://phabricator.wikimedia.org/T392408#10758623 (10Gryllida) $ toolforge webservice --backend=gridengine stop $ toolforge webservice --backend=kubernetes start What is the "type"? [12:16:17] FIRING: [2x] KernelErrors: Server cloudcephmon1004 logged kernel errors - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/KernelErrors - https://grafana.wikimedia.org/d/b013af4c-d405-4d9f-85d4-985abb3dec0c/wmcs-kernel-errors?orgId=1&var-instance=cloudcephmon1004 - https://alerts.wikimedia.org/?q=alertname%3DKernelErrors [12:16:25] 06cloud-services-team: KernelErrors Server cloudcephmon1004 logged kernel errors - https://phabricator.wikimedia.org/T392423 (10phaultfinder) 03NEW [12:21:41] PROBLEM - MD RAID on cloudcephmon1004 is CRITICAL: CRITICAL: State: degraded, Active: 3, Working: 3, Failed: 1, Spare: 0 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook%23Hardware_Raid_Information_Gathering [12:21:42] ACKNOWLEDGEMENT - MD RAID on cloudcephmon1004 is CRITICAL: CRITICAL: State: degraded, Active: 3, Working: 3, Failed: 1, Spare: 0 nagiosadmin RAID handler auto-ack: https://phabricator.wikimedia.org/T392424 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook%23Hardware_Raid_Information_Gathering [12:39:11] FIRING: SmartNotHealthy: Disk not healthy - https://wikitech.wikimedia.org/wiki/SMART#Alerts - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcephmon1004 - https://alerts.wikimedia.org/?q=alertname%3DSmartNotHealthy [14:13:10] 10Tool-Pageviews: Data problems with dumps and siteviews tool - https://phabricator.wikimedia.org/T384636#10758768 (10agray) April 2025 update: It looks like the siteviews data has been updated retroactively and is now consistent with the dumps. Siteviews is consistent with dump data for every month 2024-01 thro... [15:38:20] 10Tool-paulina: Editing interface - https://phabricator.wikimedia.org/T392429 (10Pepe_piton) 03NEW [15:39:15] 10Tool-paulina: Editing interface - https://phabricator.wikimedia.org/T392429#10758842 (10Pepe_piton) p:05Triage→03Medium a:03Pepe_piton [16:16:18] FIRING: [2x] KernelErrors: Server cloudcephmon1004 logged kernel errors - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/KernelErrors - https://grafana.wikimedia.org/d/b013af4c-d405-4d9f-85d4-985abb3dec0c/wmcs-kernel-errors?orgId=1&var-instance=cloudcephmon1004 - https://alerts.wikimedia.org/?q=alertname%3DKernelErrors [16:34:56] 10Tool-gitlab-content, 07Epic: Implement a reverse proxy for gitlab.wikimedia.org raw content that supports mime-type specification - https://phabricator.wikimedia.org/T392431 (10bd808) 03NEW [16:39:11] FIRING: SmartNotHealthy: Disk not healthy - https://wikitech.wikimedia.org/wiki/SMART#Alerts - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcephmon1004 - https://alerts.wikimedia.org/?q=alertname%3DSmartNotHealthy [16:42:08] 10Tool-gitlab-content, 07Epic, 03Wikimedia-Hackathon-2025: Implement a reverse proxy for gitlab.wikimedia.org raw content that supports mime-type specification - https://phabricator.wikimedia.org/T392431#10758892 (10bd808) p:05Triage→03Medium a:03bd808 I plan to work on a implementation for this idea d... [20:16:18] FIRING: [2x] KernelErrors: Server cloudcephmon1004 logged kernel errors - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/KernelErrors - https://grafana.wikimedia.org/d/b013af4c-d405-4d9f-85d4-985abb3dec0c/wmcs-kernel-errors?orgId=1&var-instance=cloudcephmon1004 - https://alerts.wikimedia.org/?q=alertname%3DKernelErrors [20:39:11] FIRING: SmartNotHealthy: Disk not healthy - https://wikitech.wikimedia.org/wiki/SMART#Alerts - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcephmon1004 - https://alerts.wikimedia.org/?q=alertname%3DSmartNotHealthy [20:48:08] 06cloud-services-team, 10Toolforge: re-enable wmtran tool at toolforge - https://phabricator.wikimedia.org/T392408#10759036 (10Wargo) ` webservice --help ` perhaps `php7.4` or `php8.2` [21:20:29] 10Quarry, 07Regression: Bad resultset number case is not handled - https://phabricator.wikimedia.org/T218470#10759098 (10Pppery) [21:29:35] 06cloud-services-team, 10Toolforge: re-enable wmtran tool at toolforge - https://phabricator.wikimedia.org/T392408#10759109 (10Gryllida) This worked. Thanks. Will it not require me to do this again, now, like after a reboot and such?