[02:14:02] (03PS1) 10Wandji collins: Workflow to deploy frontend and backend to toolforge [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1181250 (https://phabricator.wikimedia.org/T402610) [02:14:26] (03CR) 10CI reject: [V:04-1] Workflow to deploy frontend and backend to toolforge [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1181250 (https://phabricator.wikimedia.org/T402610) (owner: 10Wandji collins) [04:21:45] 10Cloud-VPS (Project-requests): Request creation of eseap VPS project - https://phabricator.wikimedia.org/T401957#11112852 (10Raymond_Ndibe) a:03Raymond_Ndibe [04:32:48] !log raymond-ndibe@cloudcumin1001 eseap START - Cookbook wmcs.vps.create_project for project eseap in eqiad1 (T401957) [04:32:50] raymond-ndibe@cloudcumin1001: Unknown project "eseap" [04:32:50] T401957: Request creation of eseap VPS project - https://phabricator.wikimedia.org/T401957 [04:33:20] !log raymond-ndibe@cloudcumin1001 eseap END (FAIL) - Cookbook wmcs.vps.create_project (exit_code=99) for project eseap in eqiad1 (T401957) [04:33:20] raymond-ndibe@cloudcumin1001: Unknown project "eseap" [04:37:32] !log raymond-ndibe@cloudcumin1001 eseap START - Cookbook wmcs.vps.create_project for project eseap in eqiad1 (T401957) [04:37:32] raymond-ndibe@cloudcumin1001: Unknown project "eseap" [04:38:09] (03update) 10group_199_bot_333a6c67971a471aeb1cf0b14ccf9f49: projects: added project eseap [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/265 (https://phabricator.wikimedia.org/T401957) [04:38:13] (03open) 10group_199_bot_333a6c67971a471aeb1cf0b14ccf9f49: projects: added project eseap [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/265 (https://phabricator.wikimedia.org/T401957) [04:41:44] raymond-ndibe@cloudcumin1001 create_project (PID 3809184) is awaiting input [04:42:24] (03approved) 10raymond-ndibe: projects: added project eseap [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/265 (https://phabricator.wikimedia.org/T401957) (owner: 10group_199_bot_333a6c67971a471aeb1cf0b14ccf9f49) [04:43:18] (03merge) 10raymond-ndibe: projects: added project eseap [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/265 (https://phabricator.wikimedia.org/T401957) (owner: 10group_199_bot_333a6c67971a471aeb1cf0b14ccf9f49) [04:46:04] !log raymond-ndibe@cloudcumin1001 eseap END (PASS) - Cookbook wmcs.vps.create_project (exit_code=0) for project eseap in eqiad1 (T401957) [04:46:08] T401957: Request creation of eseap VPS project - https://phabricator.wikimedia.org/T401957 [04:52:21] 10Cloud-VPS (Project-requests), 13Patch-For-Review: Request creation of eseap VPS project - https://phabricator.wikimedia.org/T401957#11112862 (10Raymond_Ndibe) `sudo cookbook wmcs.vps.create_project --user robertsky --user chlod --cluster-name eqiad1 --project eseap --task-id T401957 --description "To hos... [04:52:30] 10Cloud-VPS (Project-requests), 13Patch-For-Review: Request creation of eseap VPS project - https://phabricator.wikimedia.org/T401957#11112863 (10Raymond_Ndibe) 05Open→03Resolved [09:22:29] FIRING: NodeTextfileStale: Stale textfile for cloudvirt2004-dev:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [09:27:29] FIRING: [8x] NodeTextfileStale: Stale textfile for cloudcephosd1021:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [09:29:47] FIRING: NodeTextfileStale: Stale textfile for clouddb1025:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [09:32:29] FIRING: [23x] NodeTextfileStale: Stale textfile for cloudcephmon2004-dev:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [09:35:24] FIRING: [2x] NodeTextfileStale: Stale textfile for an-redacteddb1001:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [09:37:43] FIRING: [44x] NodeTextfileStale: Stale textfile for cloudbackup1001-dev:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [09:40:20] FIRING: [4x] NodeTextfileStale: Stale textfile for an-redacteddb1001:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [09:42:38] FIRING: [56x] NodeTextfileStale: Stale textfile for cloudbackup1001-dev:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [09:47:01] FIRING: NodeTextfileStale: Stale textfile for cloudcumin2001:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [09:47:33] FIRING: [78x] NodeTextfileStale: Stale textfile for cloudbackup1001-dev:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [09:50:37] FIRING: [5x] NodeTextfileStale: Stale textfile for an-redacteddb1001:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [09:52:58] FIRING: [2x] NodeTextfileStale: Stale textfile for cloudcumin1001:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [09:53:24] FIRING: [98x] NodeTextfileStale: Stale textfile for cloudbackup1001-dev:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [09:57:33] FIRING: [112x] NodeTextfileStale: Stale textfile for cloudbackup1001-dev:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [10:02:29] FIRING: [119x] NodeTextfileStale: Stale textfile for cloudbackup1001-dev:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [10:07:29] FIRING: [127x] NodeTextfileStale: Stale textfile for cloudbackup1001-dev:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [10:35:52] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=0) (T401693) [10:35:59] T401693: Put cloudcephosd10[42-47] in service - https://phabricator.wikimedia.org/T401693 [13:35:15] 06cloud-services-team, 10Toolforge: toolforge logs appears to suffer from intermittent latency - https://phabricator.wikimedia.org/T402736#11113071 (10DamianZaremba) [13:49:44] FIRING: [5x] NodeTextfileStale: Stale textfile for an-redacteddb1001:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [13:51:29] FIRING: [2x] NodeTextfileStale: Stale textfile for cloudcumin1001:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [14:07:44] FIRING: [127x] NodeTextfileStale: Stale textfile for cloudbackup1001-dev:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [14:09:32] 06cloud-services-team, 10Toolforge: toolforge logs appears to suffer from intermittent latency - https://phabricator.wikimedia.org/T402736#11113104 (10DamianZaremba) As a workaround I have the job emitting a known log line and then poll the logs API until that line in seen, this isn't ideal as it adds extra re... [14:27:01] 06cloud-services-team, 10Cloud-VPS: Monitoring/metrics for trove instances - https://phabricator.wikimedia.org/T402738 (10DamianZaremba) 03NEW [15:42:14] 06cloud-services-team, 10Toolforge (Toolforge iteration 23): [components-api,beta] Config not updated from remote source - https://phabricator.wikimedia.org/T401868#11113227 (10DamianZaremba) For now I've basically replaced `source_url` with re-creating the config directly and then triggering a deploy - https:... [15:51:46] 06cloud-services-team, 10Toolforge (Toolforge iteration 23): [components-api,beta] Config not updated from remote source - https://phabricator.wikimedia.org/T401868#11113231 (10DamianZaremba) Example run using the above: https://github.com/cluebotng/component-configs/actions/runs/17190716103/job/48765956567 D... [16:16:55] (03PS2) 10Rehan_khan_78: Split main.css into multiple CSS files by page and update templates [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1181245 (https://phabricator.wikimedia.org/T316197) [16:23:33] 10Tool-bridgebot, 10Education-Program-Dashboard: Should ignore wikibugs messages on #wikimedia-ed - https://phabricator.wikimedia.org/T330341#11113256 (10Jony) 05Open→03Resolved a:03Jony [17:19:31] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add (T401693) [17:19:39] T401693: Put cloudcephosd10[42-47] in service - https://phabricator.wikimedia.org/T401693 [17:23:00] PROBLEM - Host cloudcephosd1046 is DOWN: PING CRITICAL - Packet loss = 100% [17:23:52] 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE: Put cloudcephosd10[42-47] in service - https://phabricator.wikimedia.org/T401693#11113335 (10Andrew) [17:25:30] RECOVERY - Host cloudcephosd1046 is UP: PING OK - Packet loss = 0%, RTA = 0.33 ms [17:26:10] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [17:27:29] FIRING: [127x] NodeTextfileStale: Stale textfile for cloudbackup1001-dev:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [17:31:09] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [17:34:52] (03PS2) 10Wandji collins: Workflow to deploy frontend and backend to toolforge [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1181250 (https://phabricator.wikimedia.org/T402610) [17:35:22] (03CR) 10CI reject: [V:04-1] Workflow to deploy frontend and backend to toolforge [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1181250 (https://phabricator.wikimedia.org/T402610) (owner: 10Wandji collins) [17:37:29] FIRING: [126x] NodeTextfileStale: Stale textfile for cloudbackup1001-dev:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [17:42:29] FIRING: [126x] NodeTextfileStale: Stale textfile for cloudbackup1001-dev:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [17:47:29] FIRING: [126x] NodeTextfileStale: Stale textfile for cloudbackup1001-dev:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [17:49:44] FIRING: [5x] NodeTextfileStale: Stale textfile for an-redacteddb1001:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [17:51:44] FIRING: [2x] NodeTextfileStale: Stale textfile for cloudcumin1001:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [18:04:29] FIRING: [5x] NodeTextfileStale: Stale textfile for an-redacteddb1001:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [18:07:29] FIRING: [28x] NodeTextfileStale: Stale textfile for clouddumps1001:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [18:26:29] FIRING: [2x] NodeTextfileStale: Stale textfile for cloudcumin1001:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [18:36:29] RESOLVED: [2x] NodeTextfileStale: Stale textfile for cloudcumin1001:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [18:37:29] RESOLVED: [2x] NodeTextfileStale: Stale textfile for clouddumps1001:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [18:39:29] RESOLVED: NodeTextfileStale: Stale textfile for an-redacteddb1001:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [19:06:52] (03PS3) 10Wandji collins: Workflow to deploy frontend and backend to toolforge [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1181250 (https://phabricator.wikimedia.org/T402610) [19:13:55] (03CR) 10Eugene233: [C:03+2] Workflow to deploy frontend and backend to toolforge [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1181250 (https://phabricator.wikimedia.org/T402610) (owner: 10Wandji collins) [19:14:43] (03Merged) 10jenkins-bot: Workflow to deploy frontend and backend to toolforge [labs/tools/WdTmCollab] - 10https://gerrit.wikimedia.org/r/1181250 (https://phabricator.wikimedia.org/T402610) (owner: 10Wandji collins) [19:28:21] (03PS3) 10Aklapper: Split main.css into multiple CSS files by page and update templates [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1181245 (https://phabricator.wikimedia.org/T316197) (owner: 10Rehan_khan_78) [21:15:22] (03open) 10josefanthony: Fix search field alignment and improve dropdown selection UX [toolforge-repos/paulina] - 10https://gitlab.wikimedia.org/toolforge-repos/paulina/-/merge_requests/7 [21:23:53] 10Tool-paulina: Add autocomplete suggestions to search bar using Wikidata wbsearchentities API - https://phabricator.wikimedia.org/T402458#11113435 (10JosefAnthony) 05Open→03In progress Started work on this task, implementation is in progress. Merge Request: !7 (https://gitlab.wikimedia.org/toolforge-repos/p... [22:56:22] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/42 (owner: 10l10n-bot) [22:56:24] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/42 (owner: 10l10n-bot) [22:56:46] (03update) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/42 (owner: 10l10n-bot)