[00:03:32] RESOLVED: AlertLintProblem: Linting problems found for OpensearchHealth - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://prometheus-alerts.wmcloud.org/?q=alertname%3DAlertLintProblem [00:12:43] (03update) 10raymond-ndibe: models: support webservice job-type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T428898) [00:24:36] 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10Castor, 10Continuous-Integration-Infrastructure, 06Test Platform (Arusha 30): Apply the 4x-iops Cinder volume type to the Castor cache volume (integration-castor06) - https://phabricator.wikimedia.org/T437127#12325721 (10Mhurd) I think retyping witho... [00:33:17] FIRING: AlertLintProblem: Linting problems found for OpensearchHealth - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://prometheus-alerts.wmcloud.org/?q=alertname%3DAlertLintProblem [00:40:01] 10Toolforge, 06tools-platform-team: [jobs-api] Generate the openapi definition from the code - https://phabricator.wikimedia.org/T390138#12325730 (10Raymond_Ndibe) a:03Raymond_Ndibe [00:40:31] 06tools-platform-team: Enforce minimum 1 replica for jobs - https://phabricator.wikimedia.org/T438103 (10Raymond_Ndibe) 03NEW [00:40:38] 06tools-platform-team: Enforce minimum 1 replica for jobs - https://phabricator.wikimedia.org/T438103#12325743 (10Raymond_Ndibe) a:03Raymond_Ndibe [00:46:13] 10Toolforge, 06tools-platform-team: Enforce minimum 1 replica for jobs - https://phabricator.wikimedia.org/T438103#12325744 (10JJMC89) [01:08:17] RESOLVED: AlertLintProblem: Linting problems found for OpensearchHealth - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://prometheus-alerts.wmcloud.org/?q=alertname%3DAlertLintProblem [01:34:17] FIRING: AlertLintProblem: Linting problems found for OpensearchHealth - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://prometheus-alerts.wmcloud.org/?q=alertname%3DAlertLintProblem [01:36:45] (03update) 10raymond-ndibe: support --webservice option [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/143 (https://phabricator.wikimedia.org/T428898) [01:52:14] (03update) 10raymond-ndibe: test continuous job webservice [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1131 (https://phabricator.wikimedia.org/T428898) [02:04:32] RESOLVED: AlertLintProblem: Linting problems found for OpensearchHealth - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://prometheus-alerts.wmcloud.org/?q=alertname%3DAlertLintProblem [02:33:17] FIRING: AlertLintProblem: Linting problems found for OpensearchHealth - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://prometheus-alerts.wmcloud.org/?q=alertname%3DAlertLintProblem [03:08:17] RESOLVED: AlertLintProblem: Linting problems found for OpensearchHealth - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://prometheus-alerts.wmcloud.org/?q=alertname%3DAlertLintProblem [03:33:17] FIRING: AlertLintProblem: Linting problems found for OpensearchHealth - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://prometheus-alerts.wmcloud.org/?q=alertname%3DAlertLintProblem [04:08:17] RESOLVED: AlertLintProblem: Linting problems found for OpensearchHealth - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://prometheus-alerts.wmcloud.org/?q=alertname%3DAlertLintProblem [04:33:17] FIRING: AlertLintProblem: Linting problems found for OpensearchHealth - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://prometheus-alerts.wmcloud.org/?q=alertname%3DAlertLintProblem [04:58:12] 10Tool-ghanasupremecases1, 10Toolforge, 06tools-platform-team: Toolforge web service returns “No healthy upstream” when deploying FastAPI + React application - https://phabricator.wikimedia.org/T437785#12325976 (10komla) @Sunkanmi12 does the issue still persist? I don't see `No healthy upstream` when I visi... [05:03:17] RESOLVED: AlertLintProblem: Linting problems found for OpensearchHealth - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://prometheus-alerts.wmcloud.org/?q=alertname%3DAlertLintProblem [05:04:47] FIRING: AlertLintProblem: Linting problems found for OpensearchHealth - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://prometheus-alerts.wmcloud.org/?q=alertname%3DAlertLintProblem [05:09:47] RESOLVED: AlertLintProblem: Linting problems found for OpensearchHealth - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://prometheus-alerts.wmcloud.org/?q=alertname%3DAlertLintProblem [05:33:17] FIRING: AlertLintProblem: Linting problems found for OpensearchHealth - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://prometheus-alerts.wmcloud.org/?q=alertname%3DAlertLintProblem [06:03:17] RESOLVED: AlertLintProblem: Linting problems found for OpensearchHealth - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://prometheus-alerts.wmcloud.org/?q=alertname%3DAlertLintProblem [06:04:47] FIRING: AlertLintProblem: Linting problems found for OpensearchHealth - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://prometheus-alerts.wmcloud.org/?q=alertname%3DAlertLintProblem [06:05:02] RESOLVED: AlertLintProblem: Linting problems found for OpensearchHealth - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://prometheus-alerts.wmcloud.org/?q=alertname%3DAlertLintProblem [06:14:34] (03open) 10phedenskog: Add coldstart data for the Postgres core job [toolforge-repos/phpunit-results-cache] - 10https://gitlab.wikimedia.org/toolforge-repos/phpunit-results-cache/-/merge_requests/15 (https://phabricator.wikimedia.org/T437796) [06:33:17] FIRING: AlertLintProblem: Linting problems found for OpensearchHealth - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://prometheus-alerts.wmcloud.org/?q=alertname%3DAlertLintProblem [07:04:54] 10PAWS, 10Toolforge, 06tools-infrastructure-team: Dumps NFS mount down - https://phabricator.wikimedia.org/T438062#12326136 (10fgiunchedi) It looks like yesterday (15/09/2026) at ~14 something caused nfs clients to disconnect from clouddumps1002 and come back, note that at the time both hosts were pooled for... [07:08:17] RESOLVED: AlertLintProblem: Linting problems found for OpensearchHealth - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://prometheus-alerts.wmcloud.org/?q=alertname%3DAlertLintProblem [07:13:56] (03update) 10dcaro: auth: generalize the writer authorization [repos/cloud/toolforge/logs-api] (add-get-events-endpoint) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/43 (https://phabricator.wikimedia.org/T432571) [07:33:17] FIRING: AlertLintProblem: Linting problems found for OpensearchHealth - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://prometheus-alerts.wmcloud.org/?q=alertname%3DAlertLintProblem [07:44:05] (03approved) 10arthurtaylor: Add coldstart data for the Postgres core job [toolforge-repos/phpunit-results-cache] - 10https://gitlab.wikimedia.org/toolforge-repos/phpunit-results-cache/-/merge_requests/15 (https://phabricator.wikimedia.org/T437796) (owner: 10phedenskog) [07:44:22] (03merge) 10arthurtaylor: Add coldstart data for the Postgres core job [toolforge-repos/phpunit-results-cache] - 10https://gitlab.wikimedia.org/toolforge-repos/phpunit-results-cache/-/merge_requests/15 (https://phabricator.wikimedia.org/T437796) (owner: 10phedenskog) [08:00:17] 10Cloud-Services: Unlink SUL account "TanvirSdq" from stale developer account on toolsadmin (Striker) - https://phabricator.wikimedia.org/T438120 (10TanvirSdq) 03NEW The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832... [08:03:17] RESOLVED: AlertLintProblem: Linting problems found for OpensearchHealth - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://prometheus-alerts.wmcloud.org/?q=alertname%3DAlertLintProblem [08:03:32] FIRING: AlertLintProblem: Linting problems found for OpensearchHealth - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://prometheus-alerts.wmcloud.org/?q=alertname%3DAlertLintProblem [08:04:47] RESOLVED: AlertLintProblem: Linting problems found for OpensearchHealth - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://prometheus-alerts.wmcloud.org/?q=alertname%3DAlertLintProblem [08:07:17] 10PAWS, 10Toolforge, 06tools-infrastructure-team: Dumps NFS mount down - https://phabricator.wikimedia.org/T438062#12326343 (10fgiunchedi) Immediate follow ups: * nothing alerted even though e.g. `mnt-nfs-dumps.mount` unit was failed on tools-bastion-15, albeit shortly before this task. verify that (auto)mo... [08:26:11] (03update) 10dcaro: global: add deploy events [repos/cloud/toolforge/logs-api] (add-get-events-endpoint) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/42 (https://phabricator.wikimedia.org/T432571) [08:33:17] FIRING: AlertLintProblem: Linting problems found for OpensearchHealth - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://prometheus-alerts.wmcloud.org/?q=alertname%3DAlertLintProblem [08:44:25] (03approved) 10dcaro: models: support webservice job-type [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/264 (https://phabricator.wikimedia.org/T428898) (owner: 10raymond-ndibe) [08:49:20] (03update) 10dcaro: alerts-api: configure prometheus hostnames [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1419 (https://phabricator.wikimedia.org/T432865) (owner: 10fnegri) [08:49:29] (03update) 10dcaro: alerts-api: bump to 0.0.4-20260915131028-71347d7a [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1422 (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [08:50:09] (03update) 10dcaro: alerts-api: configure prometheus hostnames [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1419 (https://phabricator.wikimedia.org/T432865) (owner: 10fnegri) [08:51:39] (03approved) 10dcaro: alerts-api: configure prometheus hostnames [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1419 (https://phabricator.wikimedia.org/T432865) (owner: 10fnegri) [08:51:54] (03open) 10arthurtaylor: Update timing cache data before server deployment [toolforge-repos/phpunit-results-cache] - 10https://gitlab.wikimedia.org/toolforge-repos/phpunit-results-cache/-/merge_requests/16 (https://phabricator.wikimedia.org/T437796) [08:55:20] (03update) 10arthurtaylor: Update timing cache data before server deployment [toolforge-repos/phpunit-results-cache] - 10https://gitlab.wikimedia.org/toolforge-repos/phpunit-results-cache/-/merge_requests/16 (https://phabricator.wikimedia.org/T437796) [09:03:32] RESOLVED: AlertLintProblem: Linting problems found for OpensearchHealth - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://prometheus-alerts.wmcloud.org/?q=alertname%3DAlertLintProblem [09:05:02] FIRING: AlertLintProblem: Linting problems found for OpensearchHealth - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://prometheus-alerts.wmcloud.org/?q=alertname%3DAlertLintProblem [09:08:17] RESOLVED: AlertLintProblem: Linting problems found for OpensearchHealth - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://prometheus-alerts.wmcloud.org/?q=alertname%3DAlertLintProblem [09:15:01] (03update) 10dcaro: global: add deploy events [repos/cloud/toolforge/logs-api] (add-get-events-endpoint) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/42 (https://phabricator.wikimedia.org/T432571) [09:16:31] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/80 [09:16:34] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/80 [09:17:13] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [09:17:14] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/40 [09:18:16] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/28 [09:18:18] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/28 [09:27:42] (03approved) 10dcaro: Fetch alerts from Prometheus [repos/cloud/toolforge/alerts-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/3 (https://phabricator.wikimedia.org/T432865) (owner: 10fnegri) [09:29:52] (03approved) 10dcaro: Keep polling when a prometheus host is unreachable [repos/cloud/toolforge/alerts-api] (fran-T432865-505040f5) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts-api/-/merge_requests/4 (https://phabricator.wikimedia.org/T432865) (owner: 10fnegri) [09:33:17] FIRING: AlertLintProblem: Linting problems found for OpensearchHealth - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://prometheus-alerts.wmcloud.org/?q=alertname%3DAlertLintProblem [09:33:40] (03update) 10dcaro: auth: generalize the writer authorization [repos/cloud/toolforge/logs-api] (add-get-events-endpoint) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/43 (https://phabricator.wikimedia.org/T432571) [10:03:17] RESOLVED: AlertLintProblem: Linting problems found for OpensearchHealth - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://prometheus-alerts.wmcloud.org/?q=alertname%3DAlertLintProblem [10:04:47] FIRING: AlertLintProblem: Linting problems found for OpensearchHealth - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://prometheus-alerts.wmcloud.org/?q=alertname%3DAlertLintProblem [10:09:47] RESOLVED: AlertLintProblem: Linting problems found for OpensearchHealth - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://prometheus-alerts.wmcloud.org/?q=alertname%3DAlertLintProblem [10:14:31] (03update) 10renovatebot: Update pnpm to v12.4.1 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/20 [10:14:41] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [10:14:42] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/16 [10:14:53] (03update) 10renovatebot: Update pnpm to v12.4.1 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/32 [10:14:59] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28 [10:15:01] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/28 [10:15:12] (03update) 10renovatebot: Update pnpm to v12.4.1 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/27 [10:15:18] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [10:15:20] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/23 [10:16:55] (03update) 10renovatebot: Update pnpm to v12.4.1 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/59 [10:17:01] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 [10:17:02] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/55 [10:17:40] (03CR) 10Volans: [C:03+1] "General approach looks ok to me, couple of questions inline." [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1341953 (owner: 10Majavah) [10:18:08] (03update) 10renovatebot: Update pnpm to v12.4.1 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/44 [10:19:06] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/42 [10:19:08] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/42 [10:19:41] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/30 [10:19:43] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/30 [10:19:50] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/33 [10:19:51] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/33 [10:33:17] FIRING: AlertLintProblem: Linting problems found for OpensearchHealth - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://prometheus-alerts.wmcloud.org/?q=alertname%3DAlertLintProblem [10:38:03] FIRING: InstanceDown: Project tools instance tools-k8s-worker-nfs-16 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [10:43:23] FIRING: ToolforgeKubernetesNodeNotReady: Kubernetes node tools-k8s-worker-nfs-16 is not ready - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady [10:55:43] 10Tool-wikitermbase, 07good first task: Replace deprecated document.execCommand('copy') with navigator.clipboard.writeText() - https://phabricator.wikimedia.org/T437964#12327242 (10Rohitpal08) Hi @Gerges, I have implemented the requested fix to replace the deprecated `document.execCommand('copy')` with `navig... [11:03:17] RESOLVED: AlertLintProblem: Linting problems found for OpensearchHealth - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://prometheus-alerts.wmcloud.org/?q=alertname%3DAlertLintProblem [11:04:47] FIRING: AlertLintProblem: Linting problems found for OpensearchHealth - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://prometheus-alerts.wmcloud.org/?q=alertname%3DAlertLintProblem [11:09:47] RESOLVED: AlertLintProblem: Linting problems found for OpensearchHealth - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://prometheus-alerts.wmcloud.org/?q=alertname%3DAlertLintProblem [11:14:03] (03update) 10renovatebot: Update pnpm to v12.4.1 [toolforge-repos/namehistory] - 10https://gitlab.wikimedia.org/toolforge-repos/namehistory/-/merge_requests/20 [11:14:59] (03update) 10renovatebot: Update pnpm to v12.4.1 [toolforge-repos/allblocksever] - 10https://gitlab.wikimedia.org/toolforge-repos/allblocksever/-/merge_requests/32 [11:15:50] (03update) 10renovatebot: Update pnpm to v12.4.1 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/27 [11:16:59] (03update) 10renovatebot: Update pnpm to v12.4.1 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/59 [11:33:17] FIRING: AlertLintProblem: Linting problems found for OpensearchHealth - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://prometheus-alerts.wmcloud.org/?q=alertname%3DAlertLintProblem [11:39:37] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [o11y] Automate daily count of successful SSH logins as a measure of user activity - https://phabricator.wikimedia.org/T423549#12327494 (10aputhin) 05In progress→03Stalled p:05Medium→03Low a:05komla→03None This is currently paused until we a... [11:49:10] 10Tool-wikitermbase, 07patch-welcome: Add a button to group citations using {{ترقيم استشهادات}} - https://phabricator.wikimedia.org/T437813#12327563 (10Osps7) I’ve started looking into this task and reviewed the WikiTermBase gadget code. It looks like the main changes can be implemented in `gadget/SearchTerm.... [11:57:38] 10Tool-wikitermbase, 07patch-welcome: Add a button to group citations using {{ترقيم استشهادات}} - https://phabricator.wikimedia.org/T437813#12327579 (10Gerges) Hi @Osps7, This feature needs to be added to both the web interface and the gadget interface; the generated wikitext should look like this: ` {{ترقيم ا... [12:03:17] RESOLVED: AlertLintProblem: Linting problems found for OpensearchHealth - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://prometheus-alerts.wmcloud.org/?q=alertname%3DAlertLintProblem [12:04:15] (03update) 10dcaro: global: add deploy events [repos/cloud/toolforge/logs-api] (add-get-events-endpoint) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/42 (https://phabricator.wikimedia.org/T432571) [12:04:47] FIRING: AlertLintProblem: Linting problems found for OpensearchHealth - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://prometheus-alerts.wmcloud.org/?q=alertname%3DAlertLintProblem [12:06:05] (03update) 10dcaro: global: add deploy events [repos/cloud/toolforge/logs-api] (generalize_logs_write_auth) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/42 (https://phabricator.wikimedia.org/T432571) [12:09:47] RESOLVED: AlertLintProblem: Linting problems found for OpensearchHealth - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://prometheus-alerts.wmcloud.org/?q=alertname%3DAlertLintProblem [12:12:08] (03update) 10dcaro: Draft: global: add deploy events [repos/cloud/toolforge/logs-api] (generalize_logs_write_auth) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/logs-api/-/merge_requests/42 (https://phabricator.wikimedia.org/T432571) [12:17:19] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/28 [12:17:24] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/flaggedrevspromotioncheck] - 10https://gitlab.wikimedia.org/toolforge-repos/flaggedrevspromotioncheck/-/merge_requests/30 [12:17:28] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/33 [12:28:39] !log dcaro@acme tools START - Cookbook wmcs.openstack.cloudvirt.vm_console [12:28:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:33:17] FIRING: AlertLintProblem: Linting problems found for OpensearchHealth - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://prometheus-alerts.wmcloud.org/?q=alertname%3DAlertLintProblem [12:33:46] !log dcaro@acme tools END (ERROR) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=255) [12:33:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:43:52] 06cloud-services-team, 10Quarry, 06tools-platform-team, 06RoadToWiki, and 3 others: "1 rows" should be "1 row" - https://phabricator.wikimedia.org/T419564#12327744 (10aputhin) 05Stalled→03Open [12:46:18] 06cloud-services-team, 10Striker, 06tools-platform-team, 13Patch-For-Review: Remove unused WIKITECH_USER setting from Striker - https://phabricator.wikimedia.org/T438014#12327754 (10aputhin) p:05Triage→03Low [12:46:40] 10Tool-wikitermbase, 07good first task: Replace deprecated document.execCommand('copy') with navigator.clipboard.writeText() - https://phabricator.wikimedia.org/T437964#12327757 (10ForzaGreen) Thank you very @Rohitpal08 for the resolution ! and thank you @Gerges for reporting ! I merged the pull request. We wi... [12:48:35] 10Tool-wikitermbase, 07good first task: Replace deprecated document.execCommand('copy') with navigator.clipboard.writeText() - https://phabricator.wikimedia.org/T437964#12327774 (10ForzaGreen) 05Open→03In progress [12:58:28] 10PAWS, 10Toolforge, 06tools-infrastructure-team, 06Data-Platform-SRE: Dumps NFS mount down - https://phabricator.wikimedia.org/T438062#12327804 (10fgiunchedi) + #data-platform-sre @BTullis @bking @brouberol JFYI I'll be poking clouddumps to try and get to the bottom of this [12:59:28] !log dcaro@acme tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16 [12:59:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:02:07] !log dcaro@acme tools END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-16 [13:02:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:03:03] RESOLVED: InstanceDown: Project tools instance tools-k8s-worker-nfs-16 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [13:03:17] RESOLVED: AlertLintProblem: Linting problems found for OpensearchHealth - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://prometheus-alerts.wmcloud.org/?q=alertname%3DAlertLintProblem [13:03:23] RESOLVED: ToolforgeKubernetesNodeNotReady: Kubernetes node tools-k8s-worker-nfs-16 is not ready - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesNodeNotReady - https://grafana.wmcloud.org/d/8GiwHDL4k/kubernetes-cluster-overview?orgId=1 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesNodeNotReady [13:04:47] FIRING: AlertLintProblem: Linting problems found for OpensearchHealth - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://prometheus-alerts.wmcloud.org/?q=alertname%3DAlertLintProblem [13:09:47] RESOLVED: AlertLintProblem: Linting problems found for OpensearchHealth - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://prometheus-alerts.wmcloud.org/?q=alertname%3DAlertLintProblem [13:17:47] 10Toolforge, 06tools-platform-team: toolforge opensearch prometheus exporter is broken - https://phabricator.wikimedia.org/T438160 (10taavi) 03NEW [13:29:31] 10Toolforge, 06tools-platform-team: toolforge opensearch prometheus exporter is broken - https://phabricator.wikimedia.org/T438160#12327988 (10taavi) 05Open→03Resolved a:03taavi started failing on Sept 14, after a wmf-auto-restart: ` Sep 14 15:16:02 tools-opensearch-3 wmf-auto-restart[1981859]: INFO:... [13:43:50] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: jobs-api: support job stop/pause - https://phabricator.wikimedia.org/T436638#12328091 (10aputhin) p:05Triage→03Medium [13:50:45] (03update) 10raymond-ndibe: add jobs-api backend to webservice-cli [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/95 (https://phabricator.wikimedia.org/T431421) [13:51:33] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [webservice-cli] use jobs-api to start the webservice - https://phabricator.wikimedia.org/T431421#12328143 (10aputhin) [13:51:38] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 13Patch-For-Review: jobs-api: support job stop/pause - https://phabricator.wikimedia.org/T436638#12328144 (10aputhin) [13:51:49] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [webservice-cli] use jobs-api to start the webservice - https://phabricator.wikimedia.org/T431421#12328147 (10aputhin) Depends on T436638 [13:52:39] (03update) 10raymond-ndibe: add jobs-api backend to webservice-cli [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/95 (https://phabricator.wikimedia.org/T431421) [13:52:47] (03update) 10raymond-ndibe: add jobs-api backend to webservice-cli [repos/cloud/toolforge/webservice-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/95 (https://phabricator.wikimedia.org/T431421) [14:28:19] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [webservice-cli] use jobs-api to start the webservice - https://phabricator.wikimedia.org/T431421#12328434 (10aputhin) [14:28:26] 10Toolforge (Push-to-Deploy), 06tools-platform-team, 07Epic, 13Patch-For-Review: [jobs-api,webservice] Run webservices via the jobs framework - https://phabricator.wikimedia.org/T348755#12328435 (10aputhin) [14:32:30] 10Toolforge, 06tools-platform-team, 07Epic, 07OKR-Work: Add a dedicated Toolforge alertmanager instance - https://phabricator.wikimedia.org/T435636#12328507 (10aputhin) p:05Triage→03Low Not needed for the MVP, might be a follow up task for the Alert Service. We should re-assess later. [14:35:22] FIRING: HAProxyBackendUnavailable: HAProxy service wikireplica-db-web-s3 backend clouddb1022.eqiad.wmnet is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [14:40:22] RESOLVED: HAProxyBackendUnavailable: HAProxy service wikireplica-db-web-s3 backend clouddb1022.eqiad.wmnet is DOWN - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [14:57:24] 10Cloud-VPS, 06tools-infrastructure-team: Change mount propagation for /mnt/nfs in wikiqlever - https://phabricator.wikimedia.org/T433671#12328813 (10fgiunchedi) @Physikerwelt would you be able to assist with changing the container mount options? thank you ! [15:07:25] 10Cloud-VPS, 06tools-infrastructure-team: Change mount propagation for /mnt/nfs in wikiqlever - https://phabricator.wikimedia.org/T433671#12328926 (10Physikerwelt) I was hoping to wind down the WikiQlever project as soon as we have a non-split Qlever instance in production. Is there some query-unsplit.wikidata... [15:13:14] !log andrew@cloudcumin1001 cloudvirt-canary START - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary on eqiad1, with recreate False, for hosts list: ['cloudvirt1067'] [15:13:40] !log andrew@cloudcumin1001 cloudvirt-canary END (PASS) - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary (exit_code=0) on eqiad1, with recreate False, for hosts list: ['cloudvirt1067'] [15:21:46] 10Cloud-VPS, 06tools-infrastructure-team: Change mount propagation for /mnt/nfs in wikiqlever - https://phabricator.wikimedia.org/T433671#12329012 (10Fnielsen) What does "Is there some query-unsplit.wikidata.org we could use for scholia" mean? [15:25:46] 06tools-infrastructure-team, 10Data-Persistence-Misc: clouddb1023 memory alert - https://phabricator.wikimedia.org/T438200 (10Andrew) 03NEW [15:27:47] 10Data-Services, 06tools-infrastructure-team: clouddb1023 memory alert - https://phabricator.wikimedia.org/T438200#12329057 (10Andrew) [15:29:52] 10Cloud-VPS, 06tools-infrastructure-team: Change mount propagation for /mnt/nfs in wikiqlever - https://phabricator.wikimedia.org/T433671#12329072 (10Physikerwelt) WikiQlever consumes all Wikidata updates and updates the SPARQL endpoint accordingly. T429150 reports of two Qlever endpoints query-next.wikidata... [15:35:25] FIRING: SystemdUnitFailed: prometheus-node-textfile-openstack_magnum_capi_worker_stats_exporter.service on cloudcontrol1007:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:35:55] 06tools-platform-team: jobs-cli: "jobs load" should warn and skip duplicate job names in one file - https://phabricator.wikimedia.org/T438202 (10Raymond_Ndibe) 03NEW [15:36:23] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387#12329114 (10BTullis) I've had a look at this (with LLM assistance) and I think that I found something useful. The 12 host... [15:36:44] 10Toolforge, 06tools-platform-team: jobs-cli: "jobs load" should warn and skip duplicate job names in one file - https://phabricator.wikimedia.org/T438202#12329115 (10JJMC89) [15:37:58] (03open) 10raymond-ndibe: jobs-cli: warn on duplicates and report loaded count on load [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/181 (https://phabricator.wikimedia.org/T438202) [15:45:25] FIRING: [2x] SystemdUnitFailed: mariadb.service on clouddb1023:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:50:05] (03update) 10raymond-ndibe: support --webservice option [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/143 (https://phabricator.wikimedia.org/T428898) [15:50:24] (03update) 10raymond-ndibe: support --webservice option [repos/cloud/toolforge/jobs-cli] (load_dedup_and_count) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/143 (https://phabricator.wikimedia.org/T428898) [15:53:10] 10Cloud-VPS (Debian Bullseye Deprecation), 10a Wikimedia CLI: Migrate mwcli away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435778#12329276 (10komla) @Addshore thank you! [15:54:03] (03update) 10raymond-ndibe: test continuous job webservice [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1131 (https://phabricator.wikimedia.org/T428898) [16:03:21] 10Tool-ghanasupremecases1, 10Toolforge, 06tools-platform-team: Toolforge web service returns “No healthy upstream” when deploying FastAPI + React application - https://phabricator.wikimedia.org/T437785#12329344 (10dcaro) >>! In T437785#12316995, @SincerelySaeed5000 wrote: > let FastAPI serve the built React... [16:05:25] RESOLVED: [4x] SystemdUnitFailed: mariadb.service on clouddb1023:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:06:05] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: jobs-cli: "jobs load" should warn and skip duplicate job names in one file - https://phabricator.wikimedia.org/T438202#12329373 (10aputhin) p:05Triage→03Medium a:03Raymond_Ndibe [16:30:17] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387#12329592 (10Andrew) >>! In T429387#12329114, @BTullis wrote: > I've had a look at this (with LLM assistance) and I think... [16:38:59] 10Data-Services, 06tools-infrastructure-team: clouddb1023 memory alert - https://phabricator.wikimedia.org/T438200#12329643 (10fnegri) This triggered the following alert in Icinga, which I don't remember seeing before. So I wonder if something changed in this clouddb host (or in multiple ones). ` Service War... [16:41:46] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387#12329678 (10Andrew) This is worth a read, I haven't tried all the suggestions (and certainly haven't tried all of them tog... [16:46:29] 10Data-Services, 06tools-infrastructure-team: clouddb1023 memory alert - https://phabricator.wikimedia.org/T438200#12329700 (10fnegri) Last 7 days for all clouddb hosts: {F103287118} This was generated with the following query in Grafana: `lang=promql node_memory_MemFree_bytes{instance=~"clouddb.*"} ` [17:00:49] 10Data-Services, 06tools-infrastructure-team: clouddb1023 memory alert - https://phabricator.wikimedia.org/T438200#12329816 (10fnegri) I double checked the icinga alert, and it's based on `available` memory, not `free` memory. The right PromQL to use is then: `promql node_memory_MemAvailable_bytes{instance=~"... [17:18:36] 10Cloud-VPS, 06tools-infrastructure-team, 13Patch-For-Review: cloudceph HEALTH_WARN, multiple OSD(s) experiencing slow operations in BlueStore - https://phabricator.wikimedia.org/T429387#12329926 (10BTullis) These cumin queries are useful, too. This query breaks down the hosts, by which SSDs they have. ` su... [17:47:38] (03PS1) 10MusikAnimal: GlobalContribs: ensure DB names from CentralAuth get the _p suffix [labs/xtools] - 10https://gerrit.wikimedia.org/r/1342317 (https://phabricator.wikimedia.org/T344090) [17:48:12] (03CR) 10CI reject: [V:04-1] GlobalContribs: ensure DB names from CentralAuth get the _p suffix [labs/xtools] - 10https://gerrit.wikimedia.org/r/1342317 (https://phabricator.wikimedia.org/T344090) (owner: 10MusikAnimal) [17:51:26] (03PS2) 10MusikAnimal: GlobalContribs: ensure DB names from CentralAuth get the _p suffix [labs/xtools] - 10https://gerrit.wikimedia.org/r/1342317 (https://phabricator.wikimedia.org/T344090) [17:52:17] (03CR) 10MusikAnimal: [C:03+2] GlobalContribs: ensure DB names from CentralAuth get the _p suffix [labs/xtools] - 10https://gerrit.wikimedia.org/r/1342317 (https://phabricator.wikimedia.org/T344090) (owner: 10MusikAnimal) [17:53:01] (03Merged) 10jenkins-bot: GlobalContribs: ensure DB names from CentralAuth get the _p suffix [labs/xtools] - 10https://gerrit.wikimedia.org/r/1342317 (https://phabricator.wikimedia.org/T344090) (owner: 10MusikAnimal) [17:53:59] (03PS1) 10MusikAnimal: 3.26.1 version bump [labs/xtools] - 10https://gerrit.wikimedia.org/r/1342320 [17:54:49] (03CR) 10MusikAnimal: [C:03+2] 3.26.1 version bump [labs/xtools] - 10https://gerrit.wikimedia.org/r/1342320 (owner: 10MusikAnimal) [17:56:02] (03Merged) 10jenkins-bot: 3.26.1 version bump [labs/xtools] - 10https://gerrit.wikimedia.org/r/1342320 (owner: 10MusikAnimal) [18:07:24] FIRING: SystemdUnitFailed: magnum-api.service on cloudcontrol2010-dev:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:12:24] FIRING: SystemdUnitFailed: magnum-api.service on cloudcontrol2006-dev:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:15:55] 10Tool-curator: Curator: when using {{camera.make}} and {{camera.model}} in template, make it unknown instead of erroring it - https://phabricator.wikimedia.org/T438224 (10PantheraLeo1359531) 03NEW [18:19:49] 10Tool-curator: Curator: add "taken with {{camera.make}} {{camera.model}}" at the end of the title template by standard - https://phabricator.wikimedia.org/T438226 (10PantheraLeo1359531) 03NEW [18:20:05] 10Tool-curator: Curator: when using {{camera.make}} and {{camera.model}} in template, make it unknown instead of erroring it - https://phabricator.wikimedia.org/T438224#12330316 (10PantheraLeo1359531) https://phabricator.wikimedia.org/T438226 is related to this request :) [18:35:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [18:40:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [19:17:54] RESOLVED: SystemdUnitFailed: magnum-api.service on cloudcontrol2006-dev:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:40:42] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-rdb01 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436480#12331099 (10Southparkfan) Hi @Ladsgroup! I am currently working on getting the Beta Cluster off Bullseye. As the author https://gerrit.wikimedia.... [21:56:07] 10Tool-spacemedia, 10Toolforge, 06tools-platform-team: Can't deploy anymore on Toolforge (pod stays pending) - https://phabricator.wikimedia.org/T438255 (10Don-vip) 03NEW [21:59:08] 06tools-platform-team, 10Toolhub: Upgrade Toolhub base image to Trixie - https://phabricator.wikimedia.org/T433065#12331192 (10bd808) [22:00:28] 10Striker, 06tools-platform-team, 10Toolhub: "GO TO TOOL" targeting toolsadmin.wikimedia.org pages fails because of remote cross-origin-opener-policy header value - https://phabricator.wikimedia.org/T432262#12331194 (10bd808) [22:09:04] 10Tool-spacemedia, 10Toolforge, 06tools-platform-team: Can't deploy anymore on Toolforge (pod stays pending) - https://phabricator.wikimedia.org/T438255#12331209 (10Don-vip) [22:19:32] 10Tool-spacemedia, 10Toolforge, 06tools-platform-team: Can't deploy anymore on Toolforge (pod stays pending) - https://phabricator.wikimedia.org/T438255#12331264 (10Don-vip) As a workaround I managed to start my tools with lower requests/limits: ` toolforge webservice --cpu 2500m --mem 4500Mi --mount none b... [23:42:25] (03update) 10raymond-ndibe: test continuous job webservice [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1131 (https://phabricator.wikimedia.org/T428898) [23:43:15] (03update) 10raymond-ndibe: test continuous job webservice [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1131 (https://phabricator.wikimedia.org/T428898) [23:44:05] (03update) 10raymond-ndibe: test continuous job webservice [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1131 (https://phabricator.wikimedia.org/T428898)