[00:00:15] !log andrew@cloudcumin1001 projectnameandid END (FAIL) - Cookbook wmcs.vps.create_project (exit_code=99) for project projectnameandid in codfw1dev [00:00:16] andrew@cloudcumin1001: Unknown project "projectnameandid" [00:05:00] (03open) 10andrew: Revert "projects: added project uuidtest" [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/155 [00:05:09] (03merge) 10andrew: Revert "projects: added project uuidtest" [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/155 [00:05:21] (03update) 10andrew: projects: added project projectnameandid [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/154 (owner: 10group_199_bot_333a6c67971a471aeb1cf0b14ccf9f49) [00:05:37] (03merge) 10andrew: projects: added project projectnameandid [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/154 (owner: 10group_199_bot_333a6c67971a471aeb1cf0b14ccf9f49) [00:06:08] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for main branch [00:06:11] !log andrew@cloudcumin1001 admin END (ERROR) - Cookbook wmcs.openstack.tofu (exit_code=97) running tofu plan for main branch [00:06:30] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan for main branch [00:06:42] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch [00:07:21] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [00:08:39] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [00:22:14] (03open) 10andrew: Revert "projects: added project projectnameandid" [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/156 [00:22:17] (03merge) 10andrew: Revert "projects: added project projectnameandid" [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/156 [00:22:22] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.tofu running tofu plan+apply for main branch [00:24:49] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan+apply for main branch [00:50:43] 06cloud-services-team, 10Cloud-VPS, 10Toolforge: Block web crawlers from accessing Cloud Services - https://phabricator.wikimedia.org/T226688#10573944 (10AntiCompositeNumber) This seems to be a recurring problem for a number of tools. Unfortunately user-agent-based filtering doesn't work for a lot (of the m... [01:33:07] (03update) 10raymond-ndibe: [toolforge-deploy] maintain-harbor use robot account [repos/cloud/toolforge/toolforge-deploy] (maintain_harbor_use_robot_account) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/674 (https://phabricator.wikimedia.org/T352417 https://phabricator.wikimedia.org/T361698) [01:40:22] (03update) 10raymond-ndibe: [toolforge-deploy] add maintain-harbor quota config and tests [repos/cloud/toolforge/toolforge-deploy] (maintain_harbor_use_robot_account) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/674 (https://phabricator.wikimedia.org/T352417) [01:40:44] (03update) 10raymond-ndibe: [toolforge-deploy] add maintain-harbor quota config and tests [repos/cloud/toolforge/toolforge-deploy] (maintain_harbor_use_robot_account) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/674 (https://phabricator.wikimedia.org/T352417) [01:40:52] (03update) 10raymond-ndibe: [toolforge-deploy] add maintain-harbor quota config and tests [repos/cloud/toolforge/toolforge-deploy] (maintain_harbor_use_robot_account) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/674 (https://phabricator.wikimedia.org/T352417) [01:42:45] (03update) 10raymond-ndibe: [toolforge-deploy] add maintain-harbor quota config and tests [repos/cloud/toolforge/toolforge-deploy] (maintain_harbor_use_robot_account) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/674 (https://phabricator.wikimedia.org/T352417) [01:45:02] (03update) 10raymond-ndibe: [toolforge-deploy] add maintain-harbor quota config and tests [repos/cloud/toolforge/toolforge-deploy] (maintain_harbor_use_robot_account) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/674 (https://phabricator.wikimedia.org/T352417) [01:47:12] (03update) 10raymond-ndibe: [toolforge-deploy] add maintain-harbor quota config and tests [repos/cloud/toolforge/toolforge-deploy] (maintain_harbor_use_robot_account) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/674 (https://phabricator.wikimedia.org/T352417) [01:50:07] (03update) 10raymond-ndibe: [toolforge-deploy] maintain-harbor use robot account [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/672 (https://phabricator.wikimedia.org/T361698) [01:52:13] (03update) 10raymond-ndibe: [toolforge-deploy] add maintain-harbor quota config and tests [repos/cloud/toolforge/toolforge-deploy] (maintain_harbor_use_robot_account) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/674 (https://phabricator.wikimedia.org/T352417) [02:43:34] (03update) 10raymond-ndibe: [toolforge-deploy] add maintain-harbor quota config and tests [repos/cloud/toolforge/toolforge-deploy] (maintain_harbor_use_robot_account) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/674 (https://phabricator.wikimedia.org/T352417) [04:51:47] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [05:01:45] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 29349 bytes in 6.709 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [05:06:47] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [05:17:47] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 29322 bytes in 9.834 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [05:20:47] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [05:52:47] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 29364 bytes in 8.942 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [05:55:47] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [06:56:47] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 29359 bytes in 9.461 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [06:59:47] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [07:01:45] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 29360 bytes in 7.879 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [07:04:47] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [07:24:41] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 29358 bytes in 2.997 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [07:27:47] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [07:40:41] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 28953 bytes in 3.886 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [07:43:47] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [07:50:41] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 28953 bytes in 2.553 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [07:54:47] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [08:12:47] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 29222 bytes in 9.435 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [08:15:47] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [08:31:43] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 29358 bytes in 5.803 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [08:35:47] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [08:49:47] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 29306 bytes in 8.878 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [08:52:47] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [09:03:39] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 28942 bytes in 0.518 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [09:20:53] (03open) 10dcaro: admission: remove all volumes in any case [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/25 [10:21:33] (03update) 10dcaro: admission: remove all volumes in any case [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/25 [10:28:56] (03update) 10raymond-ndibe: [toolforge-deploy] add maintain-harbor quota config and tests [repos/cloud/toolforge/toolforge-deploy] (maintain_harbor_use_robot_account) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/674 (https://phabricator.wikimedia.org/T352417) [10:30:34] (03update) 10raymond-ndibe: [toolforge-deploy] add maintain-harbor quota config and tests [repos/cloud/toolforge/toolforge-deploy] (maintain_harbor_use_robot_account) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/674 (https://phabricator.wikimedia.org/T352417) [11:08:20] (03update) 10raymond-ndibe: [toolforge-deploy] add maintain-harbor quota config and tests [repos/cloud/toolforge/toolforge-deploy] (maintain_harbor_use_robot_account) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/674 (https://phabricator.wikimedia.org/T352417) [11:15:02] (03update) 10raymond-ndibe: [toolforge-deploy] add maintain-harbor quota config and tests [repos/cloud/toolforge/toolforge-deploy] (maintain_harbor_use_robot_account) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/674 (https://phabricator.wikimedia.org/T352417) [11:39:07] (03update) 10raymond-ndibe: [toolforge-deploy] add maintain-harbor quota config and tests [repos/cloud/toolforge/toolforge-deploy] (maintain_harbor_use_robot_account) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/674 (https://phabricator.wikimedia.org/T352417) [12:00:17] (03update) 10raymond-ndibe: [toolforge-deploy] add maintain-harbor quota config and tests [repos/cloud/toolforge/toolforge-deploy] (maintain_harbor_use_robot_account) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/674 (https://phabricator.wikimedia.org/T352417) [12:06:13] (03update) 10raymond-ndibe: [toolforge-deploy] add maintain-harbor quota config and tests [repos/cloud/toolforge/toolforge-deploy] (maintain_harbor_use_robot_account) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/674 (https://phabricator.wikimedia.org/T352417) [12:09:55] (03update) 10raymond-ndibe: [toolforge-deploy] add maintain-harbor quota config and tests [repos/cloud/toolforge/toolforge-deploy] (maintain_harbor_use_robot_account) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/674 (https://phabricator.wikimedia.org/T352417) [12:10:39] (03update) 10raymond-ndibe: [toolforge-deploy] add maintain-harbor quota config and tests [repos/cloud/toolforge/toolforge-deploy] (maintain_harbor_use_robot_account) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/674 (https://phabricator.wikimedia.org/T352417) [12:22:17] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/8 [12:23:49] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [labs/tools/weapon-of-mass-description] - 10https://gerrit.wikimedia.org/r/1122124 (owner: 10L10n-bot) [12:25:20] (03update) 10raymond-ndibe: [toolforge-deploy] add maintain-harbor quota config and tests [repos/cloud/toolforge/toolforge-deploy] (maintain_harbor_use_robot_account) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/674 (https://phabricator.wikimedia.org/T352417) [12:29:35] 10VPS-Projects, 06Content-Transform-Team-WIP, 10Parsoid, 07Essential-Work, 10Parsoid-Read-Views (Wiktionary Q3 FY24-25): upgrade nodejs on parsing-qa-02 - https://phabricator.wikimedia.org/T349941#10574952 (10MSantos) [12:29:52] 10VPS-Projects, 06Content-Transform-Team-WIP, 07Essential-Work, 10Parsoid-Read-Views (Wiktionary Q3 FY24-25): Create a bookwork-imaged VM ctt-qa-03 to replace parsing-qa-03 - https://phabricator.wikimedia.org/T383249#10574953 (10MSantos) [12:30:14] 10VPS-Projects, 06Content-Transform-Team-WIP, 07Essential-Work, 10Parsoid-Read-Views (Wiktionary Q3 FY24-25): Request increased quota for wikitextexp Cloud VPS project - https://phabricator.wikimedia.org/T383251#10574954 (10MSantos) [12:30:25] 10Cloud-VPS (Quota-requests), 06Content-Transform-Team-WIP, 07Essential-Work, 10Parsoid-Read-Views (Wiktionary Q3 FY24-25): Bump up quota for wikitextexp to let us spin up a more powerful test server - https://phabricator.wikimedia.org/T383252#10574955 (10MSantos) [12:33:11] FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [12:34:12] (03update) 10raymond-ndibe: [toolforge-deploy] add maintain-harbor quota config and tests [repos/cloud/toolforge/toolforge-deploy] (maintain_harbor_use_robot_account) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/674 (https://phabricator.wikimedia.org/T352417) [12:38:11] RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [12:40:04] (03update) 10raymond-ndibe: [toolforge-deploy] add maintain-harbor quota config and tests [repos/cloud/toolforge/toolforge-deploy] (maintain_harbor_use_robot_account) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/674 (https://phabricator.wikimedia.org/T352417) [12:43:40] (03update) 10dcaro: admission: remove all volumes in any case [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/25 [12:56:50] (03update) 10raymond-ndibe: [toolforge-deploy] add maintain-harbor quota config and tests [repos/cloud/toolforge/toolforge-deploy] (maintain_harbor_use_robot_account) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/674 (https://phabricator.wikimedia.org/T352417) [13:04:52] (03update) 10raymond-ndibe: [toolforge-deploy] add maintain-harbor quota config and tests [repos/cloud/toolforge/toolforge-deploy] (maintain_harbor_use_robot_account) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/674 (https://phabricator.wikimedia.org/T352417) [13:08:54] (03update) 10dcaro: admission: remove all volumes in any case [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/25 [13:11:01] (03update) 10raymond-ndibe: [toolforge-deploy] add maintain-harbor quota config and tests [repos/cloud/toolforge/toolforge-deploy] (maintain_harbor_use_robot_account) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/674 (https://phabricator.wikimedia.org/T352417) [13:16:33] (03update) 10raymond-ndibe: [toolforge-deploy] add maintain-harbor quota config and tests [repos/cloud/toolforge/toolforge-deploy] (maintain_harbor_use_robot_account) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/674 (https://phabricator.wikimedia.org/T352417) [13:30:27] (03update) 10raymond-ndibe: [toolforge-deploy] add maintain-harbor quota config and tests [repos/cloud/toolforge/toolforge-deploy] (maintain_harbor_use_robot_account) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/674 (https://phabricator.wikimedia.org/T352417) [13:37:33] (03open) 10dcaro: volume-admission: add first test for disallowed path [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/675 [13:42:47] (03update) 10raymond-ndibe: [ maintain-harbor ] add job for managing harbor quotas [repos/cloud/toolforge/maintain-harbor] (refactor_config) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/22 (https://phabricator.wikimedia.org/T352417) (owner: 10sstefanova) [13:43:34] (03update) 10raymond-ndibe: [toolforge-deploy] add maintain-harbor quota config and tests [repos/cloud/toolforge/toolforge-deploy] (maintain_harbor_use_robot_account) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/674 (https://phabricator.wikimedia.org/T352417) [13:46:18] (03update) 10dcaro: volume-admission: add first test for disallowed path [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/675 [13:46:30] (03update) 10dcaro: admission: remove all volumes in any case [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/25 [13:46:42] (03update) 10dcaro: admission: remove all volumes in any case [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/25 [13:47:04] (03update) 10dcaro: volume-admission: add first test for disallowed path [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/675 [13:49:27] (03update) 10dcaro: admission: remove all volumes in any case [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/25 [13:51:07] (03update) 10dcaro: volume-admission: add first test for disallowed path [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/675 [13:52:26] (03update) 10raymond-ndibe: [ maintain-harbor ] add job for managing harbor quotas [repos/cloud/toolforge/maintain-harbor] (refactor_config) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/22 (https://phabricator.wikimedia.org/T352417) (owner: 10sstefanova) [13:52:53] (03update) 10dcaro: scheduled jobs: add timeout option [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/129 [13:53:19] (03update) 10dcaro: scheduled job: add timeout parameter [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/76 (https://phabricator.wikimedia.org/T306391) [13:55:58] (03update) 10raymond-ndibe: [ maintain-harbor ] add job for managing harbor quotas [repos/cloud/toolforge/maintain-harbor] (refactor_config) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/22 (https://phabricator.wikimedia.org/T352417) (owner: 10sstefanova) [14:02:34] (03update) 10raymond-ndibe: [ maintain-harbor ] add job for managing harbor quotas [repos/cloud/toolforge/maintain-harbor] (refactor_config) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/22 (https://phabricator.wikimedia.org/T352417) (owner: 10sstefanova) [14:04:43] (03update) 10raymond-ndibe: [toolforge-deploy] add maintain-harbor quota config and tests [repos/cloud/toolforge/toolforge-deploy] (maintain_harbor_use_robot_account) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/674 (https://phabricator.wikimedia.org/T352417) [14:09:31] FIRING: ToolsToolsDBReplicationError: ToolsDB replication is broken on tools-db-5 (errno 2003) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationError [14:09:31] FIRING: ToolsToolsDBWritableState: There should be exactly one writable MariaDB instance instead of 0 - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsToolsDBWritableState - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBWritableState [14:09:56] FIRING: SystemdUnitDown: The service unit disable-tool.service is in failed status on host cloudcontrol1005. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1005 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [14:11:41] 06cloud-services-team, 10Cloud-VPS, 10VPS-Projects, 10Catalyst: metricsinfra: send alerts for the catalyst project to catalyst@w.o email - https://phabricator.wikimedia.org/T386416#10575167 (10dcaro) @EBomani Maybe the easiest way would be to stop puppet, in 24h you should see an alert popping up (plus ema... [14:14:32] (03update) 10dcaro: deploy-token: prevent accidental token overwrites [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/49 (owner: 10sstefanova) [14:14:56] RESOLVED: SystemdUnitDown: The service unit disable-tool.service is in failed status on host cloudcontrol1005. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcontrol1005 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [14:22:07] (03update) 10dcaro: [jobs-api] remove wait_for_job from runtime methods [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/138 (https://phabricator.wikimedia.org/T359804) (owner: 10raymond-ndibe) [14:23:04] (03update) 10dcaro: [jobs-api] remove wait_for_job from runtime methods [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/138 (https://phabricator.wikimedia.org/T359804) (owner: 10raymond-ndibe) [14:26:01] RESOLVED: ToolsToolsDBWritableState: There should be exactly one writable MariaDB instance instead of 0 - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsToolsDBWritableState - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBWritableState [14:27:01] RESOLVED: ToolsToolsDBReplicationError: ToolsDB replication is broken on tools-db-5 (errno 2003) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationError [14:42:53] (03update) 10dcaro: [jobs-api] replace load with diff_job runtime method [repos/cloud/toolforge/jobs-api] (save_business_models_to_db) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/143 (https://phabricator.wikimedia.org/T359804) (owner: 10raymond-ndibe) [14:51:55] (03update) 10raymond-ndibe: [jobs-api] create seperate api.py and move flask things there [repos/cloud/toolforge/jobs-api] (diff_job_runtime_method) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/91 (https://phabricator.wikimedia.org/T359804) [15:04:49] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge: [toolsdb] Remove apt pinning and upgrade to latest version - https://phabricator.wikimedia.org/T385885#10575335 (10fnegri) I upgraded the primary today. Shutting down mariadb took longer than I expected (about 10 minutes). Logs from the upgrade: {P73513}... [15:31:25] (03update) 10dcaro: admission: deny any unknown hostPath volumes [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/25 [15:31:58] (03update) 10dcaro: admission: deny when unknown hostPath volumes passed [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/25 [15:32:09] (03update) 10dcaro: admission: deny when unknown hostPath volumes passed [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/25 [16:08:57] 10Tool-toolwatch: Implementing alert system to notify maintainers of downtime - https://phabricator.wikimedia.org/T368816#10575681 (10Gopavasanth) 05Open→03Resolved a:03Hrideshmg As per: https://github.com/gopavasanth/ToolWatch/pull/24 Thanks! @Hrideshmg [16:16:51] 10Tool-toolwatch: Show Success Notification on Value Change and Save - https://phabricator.wikimedia.org/T387136 (10Gopavasanth) 03NEW [16:17:07] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge: [toolsdb] Remove apt pinning and upgrade to latest version - https://phabricator.wikimedia.org/T385885#10575711 (10fnegri) > I merged the patch and the pinning is now working correctly: unattended-upgrades is not upgrading the package automatically, but a... [16:21:43] 10Tool-toolwatch: Enhance Notification Frequency and Graph Updates for Downtime Alerts - https://phabricator.wikimedia.org/T387138 (10Gopavasanth) 03NEW [16:30:59] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10575781 (10Jgreen) I'm unable to log into Wikitech (user jgreen), my password seems to work but google auth 2FA does not. How can I reset 2FA for my account? [16:32:08] 10Tool-toolwatch, 06Toolforge-standards-committee, 07Privacy: toolwatch loads third party resources - https://phabricator.wikimedia.org/T378901#10575788 (10Gopavasanth) 05Open→03Resolved Missed following up on this somehow, Now it's fixed at: https://github.com/gopavasanth/ToolWatch/commit/3c3a1902b7... [16:33:27] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10575793 (10Ladsgroup) I will have to rename your account to `Jgreen (wmf)` https://meta.wikimedia.org/wiki/Special:CentralAuth?target=Jgreen%20(wmf) Then you can use the SUL credentials (bot... [16:34:39] 06cloud-services-team, 10Toolforge: dotnet: migrate to Heroku buildpack - https://phabricator.wikimedia.org/T387140 (10fnegri) 03NEW [16:35:12] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10575809 (10Jgreen) >>! In T376267#10575793, @Ladsgroup wrote: > I will have to rename your account to `Jgreen (wmf)` https://meta.wikimedia.org/wiki/Special:CentralAuth?target=Jgreen%20(wmf)... [16:36:06] 06cloud-services-team, 10Toolforge: [builds-builder,dotnet] migrate to Heroku buildpack - https://phabricator.wikimedia.org/T387140#10575812 (10dcaro) [16:37:33] 10VPS-project-Phabricator, 06collaboration-services: phabricator.wmcloud.org should be more clearly marked as a testing instance - https://phabricator.wikimedia.org/T283088#10575817 (10LSobanski) [16:37:48] 10VPS-project-Phabricator, 06collaboration-services: phabricator.wmcloud.org should be more clearly marked as a testing instance - https://phabricator.wikimedia.org/T283088#10575820 (10LSobanski) p:05Triage→03Low [16:38:12] 06cloud-services-team, 10Toolforge: migrate from apt buildpack to Heroku's .deb packages buildpack - https://phabricator.wikimedia.org/T387141 (10fnegri) 03NEW [16:40:06] 06cloud-services-team, 10Toolforge: [builds-builder,apt] migrate from apt buildpack to Heroku's .deb packages buildpack - https://phabricator.wikimedia.org/T387141#10575849 (10dcaro) [16:46:03] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10575854 (10Ladsgroup) >>! In T376267#10575809, @Jgreen wrote: >>>! In T376267#10575793, @Ladsgroup wrote: >> I will have to rename your account to `Jgreen (wmf)` https://meta.wikimedia.org/w... [16:49:20] (03approved) 10dcaro: [jobs-api] custom resource definition deployment templates [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/101 (https://phabricator.wikimedia.org/T359650) (owner: 10raymond-ndibe) [17:08:04] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10575913 (10Jgreen) >>! In T376267#10575854, @Ladsgroup wrote: >>>! In T376267#10575809, @Jgreen wrote: >>>>! In T376267#10575793, @Ladsgroup wrote: >>> I will have to rename your account to... [17:11:51] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10575920 (10Ladsgroup) >>! In T376267#10575913, @Jgreen wrote: >>>! In T376267#10575854, @Ladsgroup wrote: >>>>! In T376267#10575809, @Jgreen wrote: >>>>>! In T376267#10575793, @Ladsgroup wro... [17:18:31] (03update) 10dcaro: deploy-token: prevent accidental token overwrites [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/49 (owner: 10sstefanova) [17:19:44] 10Tool-toolwatch, 06Toolforge-standards-committee, 07Privacy: toolwatch loads third party resources - https://phabricator.wikimedia.org/T378901#10575944 (10LucasWerkmeister) https://cdnjs.toolforge.org/npm/chart.js is a 404 URL – Toolforge cdnjs is not a drop-in replacement for jsdelivr. (The usual appro... [18:00:01] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10576131 (10Jgreen) >>! In T376267#10575920, @Ladsgroup wrote: >> My password no longer works with either "jgreen" or "Jgreen (wmf)" - not sure what to do? > The wikitech password and 2FA hav... [18:03:39] FIRING: [3x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:08:39] FIRING: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:13:28] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic, 13Patch-For-Review: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10576188 (10bd808) The [[https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/thread/DQ575HDEJMANQE6IHTP5Q... [18:13:39] RESOLVED: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:33:07] 06cloud-services-team, 10Toolforge: [builds-builder,apt] migrate from apt buildpack to Heroku's .deb packages buildpack - https://phabricator.wikimedia.org/T387141#10576355 (10bd808) Related tickets: * {T356500} * {T357251} * {T361409} * {T363027} * {T369563} [18:43:39] 10wikitech.wikimedia.org: Decide what to do with SUL attached Wikitech accounts that Bitu associates with a different SUL account - https://phabricator.wikimedia.org/T386026#10576403 (10bd808) 05Open→03Resolved The [[https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/thread/DQ575H... [19:08:41] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/8 (owner: 10l10n-bot) [19:08:43] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/8 (owner: 10l10n-bot) [19:11:18] FIRING: KernelErrors: Server cloudvirt1043 logged kernel errors - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/KernelErrors - https://grafana.wikimedia.org/d/b013af4c-d405-4d9f-85d4-985abb3dec0c/wmcs-kernel-errors?orgId=1&var-instance=cloudvirt1043 - https://alerts.wikimedia.org/?q=alertname%3DKernelErrors [19:11:42] 06cloud-services-team: KernelErrors Server cloudvirt1043 logged kernel errors - https://phabricator.wikimedia.org/T387156 (10phaultfinder) 03NEW [20:10:56] (03approved) 10raymond-ndibe: scheduled job: add timeout parameter [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/76 (https://phabricator.wikimedia.org/T306391) (owner: 10dcaro) [20:10:57] (03update) 10raymond-ndibe: scheduled job: add timeout parameter [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/76 (https://phabricator.wikimedia.org/T306391) (owner: 10dcaro) [20:11:01] (03merge) 10raymond-ndibe: scheduled job: add timeout parameter [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/76 (https://phabricator.wikimedia.org/T306391) (owner: 10dcaro) [20:11:50] (03approved) 10raymond-ndibe: scheduled jobs: add timeout option [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/129 (owner: 10dcaro) [20:11:52] (03update) 10raymond-ndibe: scheduled jobs: add timeout option [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/129 (owner: 10dcaro) [20:12:08] (03update) 10raymond-ndibe: scheduled jobs: add timeout option [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/129 (owner: 10dcaro) [20:12:26] (03merge) 10raymond-ndibe: scheduled jobs: add timeout option [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/129 (owner: 10dcaro) [20:12:53] (03update) 10raymond-ndibe: volume-admission: add first test for disallowed path [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/675 (owner: 10dcaro) [20:13:05] (03update) 10raymond-ndibe: volume-admission: add first test for disallowed path [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/675 (owner: 10dcaro) [20:14:16] (03update) 10raymond-ndibe: volume-admission: add first test for disallowed path [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/675 (owner: 10dcaro) [20:14:22] (03update) 10raymond-ndibe: admission: deny when unknown hostPath volumes passed [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/25 (owner: 10dcaro) [20:14:57] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: jobs-api: bump to 0.0.349-20250205000226-031f1bc7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/668 [20:15:00] (03update) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: jobs-api: bump to 0.0.349-20250205000226-031f1bc7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/668 (https://phabricator.wikimedia.org/T306391) [20:19:30] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [20:23:24] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [20:25:28] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10576825 (10Dillon) |**Wikitech account/LDAP:**| DHardy-WMF| |**SUL account**|Dillon / dhardy| |**Account linked on IDM** |Y| |**I have visited [[ https://wikitech.wikimedia.org/wiki/MediaWik... [20:32:34] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [20:39:02] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [20:40:39] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-api [20:44:41] (03open) 10raymond-ndibe: d/changelog: bump to 16.1.7 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/84 (https://phabricator.wikimedia.org/T306391) [20:49:42] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-api [20:52:17] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli [20:52:37] (03update) 10raymond-ndibe: jobs-api: bump to 0.0.349-20250205000226-031f1bc7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/668 (https://phabricator.wikimedia.org/T306391) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [20:52:43] (03approved) 10raymond-ndibe: jobs-api: bump to 0.0.349-20250205000226-031f1bc7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/668 (https://phabricator.wikimedia.org/T306391) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [20:52:48] (03merge) 10raymond-ndibe: jobs-api: bump to 0.0.349-20250205000226-031f1bc7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/668 (https://phabricator.wikimedia.org/T306391) (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [20:58:59] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli [20:59:31] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli [21:02:09] (03PS1) 10Gerrit Patch Uploader: fix spacing in large removal message [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1122246 [21:02:09] (03CR) 10Gerrit Patch Uploader: "This commit was uploaded using the Gerrit Patch Uploader [1]." [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1122246 (owner: 10Gerrit Patch Uploader) [21:07:44] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli [21:12:15] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component jobs-cli [21:13:28] (03PS2) 10Krinkle: Console.msgs: Fix spacing in "Large removal" messages [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1122246 (owner: 10Gerrit Patch Uploader) [21:15:15] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10577012 (10Peachey88) @Dillon the file hasn't attached properly to the task so isn't visible to users other than yourself, The easiest way to fix it is probably: Go to https://phabricator.w... [21:17:41] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10577019 (10Dillon) >>! In T376267#10577012, @Peachey88 wrote: > @Dillon the file hasn't attached properly to the task so isn't visible to users other than yourself, The easiest way to fix it... [21:18:15] (03PS3) 10Gerrit Patch Uploader: Fix more spacing in large removal message [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1122246 [21:18:15] (03CR) 10Gerrit Patch Uploader: "This commit was uploaded using the Gerrit Patch Uploader [1]." [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1122246 (owner: 10Gerrit Patch Uploader) [21:20:37] !log raymond-ndibe@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component jobs-cli [21:45:23] (03PS4) 10Krinkle: Console.msgs: Fix spacing in "Large removal" messages [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1122246 (owner: 10Gerrit Patch Uploader) [21:45:25] (03CR) 10Krinkle: [C:03+2] Console.msgs: Fix spacing in "Large removal" messages [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1122246 (owner: 10Gerrit Patch Uploader) [21:45:54] (03Merged) 10jenkins-bot: Console.msgs: Fix spacing in "Large removal" messages [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1122246 (owner: 10Gerrit Patch Uploader) [22:01:11] FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [22:06:11] RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [22:10:07] 06cloud-services-team, 06DC-Ops, 10decommission-hardware, 10ops-eqiad, 06SRE: decommission cloudgw100[12] - https://phabricator.wikimedia.org/T386810#10577244 (10VRiley-WMF) [22:19:42] 06cloud-services-team, 06DC-Ops, 10decommission-hardware, 10ops-eqiad, 06SRE: decommission cloudgw100[12] - https://phabricator.wikimedia.org/T386810#10577268 (10VRiley-WMF) Unracked and removed the following servers. However, the script has failed and returning an for both servers. @Andrew would you be... [23:06:47] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10577419 (10Ladsgroup) I created `DHardy-WMF` for you: ` ladsgroup@mwmaint2002:~$ mwscript extensions/CentralAuth/maintenance/createLocalAccount.php --wiki=labswiki "DHardy-WMF" User 'DHardy... [23:11:18] FIRING: KernelErrors: Server cloudvirt1043 logged kernel errors - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/KernelErrors - https://grafana.wikimedia.org/d/b013af4c-d405-4d9f-85d4-985abb3dec0c/wmcs-kernel-errors?orgId=1&var-instance=cloudvirt1043 - https://alerts.wikimedia.org/?q=alertname%3DKernelErrors [23:17:56] 06cloud-services-team: KernelErrors Server cloudvirt1043 logged kernel errors - https://phabricator.wikimedia.org/T387156#10577441 (10Andrew) 05Open→03Declined ` [1549405.992360] perf: interrupt took too long (6430 > 6150), lowering kernel.perf_event_max_sample_rate to 31000 [1549406.049708] {3}[Hardware... [23:24:09] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic, 13Patch-For-Review: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10577449 (10Ladsgroup) There are 31650 users that are not attached yet and out of those, 11237 have conflicting usernames. I'm a... [23:24:11] FIRING: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [23:27:00] (03update) 10raymond-ndibe: d/changelog: bump to 16.1.7 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/84 (https://phabricator.wikimedia.org/T306391) [23:27:00] (03update) 10raymond-ndibe: d/changelog: bump to 16.1.7 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/84 (https://phabricator.wikimedia.org/T306391) [23:27:11] (03approved) 10raymond-ndibe: d/changelog: bump to 16.1.7 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/84 (https://phabricator.wikimedia.org/T306391) [23:27:16] (03merge) 10raymond-ndibe: d/changelog: bump to 16.1.7 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/84 (https://phabricator.wikimedia.org/T306391) [23:29:11] RESOLVED: Temperature: Inlet Temp issue on clouddumps1001:9290 - https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook - https://grafana.wikimedia.org/d/ZA1I-IB4z/ipmi-sensor-state?orgId=1&viewPanel=92&var-server=clouddumps1001 - https://alerts.wikimedia.org/?q=alertname%3DTemperature [23:40:41] (03update) 10raymond-ndibe: [toolforge-deploy] maintain-harbor use robot account [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/672 (https://phabricator.wikimedia.org/T361698) [23:41:45] (03update) 10raymond-ndibe: [toolforge-weld] update jobs custom resources version in k8sclient [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/75 (https://phabricator.wikimedia.org/T359650) [23:41:49] (03approved) 10raymond-ndibe: [toolforge-weld] update jobs custom resources version in k8sclient [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/75 (https://phabricator.wikimedia.org/T359650)