[00:15:00] FIRING: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [00:16:46] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41745119449 (https://github.com/cluebotng/component-configs/commits/1ec1f58d43747a66c7dced1860e116768b3953e6) [00:16:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [00:17:33] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41745119511 (https://github.com/cluebotng/component-configs/commits/1ec1f58d43747a66c7dced1860e116768b3953e6) [00:17:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [00:18:02] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41745119500 (https://github.com/cluebotng/component-configs/commits/1ec1f58d43747a66c7dced1860e116768b3953e6) [00:18:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [00:25:42] RESOLVED: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [00:29:00] 10Tool-commons-delinquent: Tool-commons-delinquent: HTTP Code 500 error - https://phabricator.wikimedia.org/T437416#12513904 (10SD_hehua) Not recovered. [00:35:00] FIRING: [2x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [00:40:25] !log tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41748064637 (https://github.com/cluebotng/component-configs/commits/cff94477a84651d691f840e8ded909cba65ec7a2) [00:40:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [00:41:35] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41748064641 (https://github.com/cluebotng/component-configs/commits/cff94477a84651d691f840e8ded909cba65ec7a2) [00:41:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [00:43:52] 10Cloud-VPS, 06tools-infrastructure-team, 10WM-Bot: wm-bot.wmflabs.org now incorrectly redirects to wm-bot.wmcloud.org - https://phabricator.wikimedia.org/T443531#12513954 (10bd808) By design any `$HOSTNAME.wmcloud.org` web proxy also 301 redirects inbound `$HOSTNAME.wmflabs.org` requests to the same backend... [00:45:42] FIRING: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [00:58:15] 10Cloud-VPS, 06tools-infrastructure-team, 10WM-Bot: wm-bot.wmflabs.org now incorrectly redirects to wm-bot.wmcloud.org - https://phabricator.wikimedia.org/T443531#12514000 (10Krinkle) I would suggest maybe: * copy the data over the new directory in a freenode subfolder. This will be the canonical place. * ~~... [01:15:12] (03update) 10renovatebot: Update Rust crate replica-pools to v0.1.3 [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/94 [01:15:13] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/93 [01:15:32] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/65 [01:15:42] RESOLVED: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [01:16:20] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-rangeblock] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-rangeblock/-/merge_requests/40 [01:23:47] 10Cloud-VPS, 06tools-infrastructure-team, 10WM-Bot: wm-bot.wmflabs.org now incorrectly redirects to wm-bot.wmcloud.org - https://phabricator.wikimedia.org/T443531#12514043 (10MacFan4000) I'll note that I can access freenode stuff via https://bots.wmflabs.org which was also setup as a proxy. [01:28:01] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [01:28:06] FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [01:33:01] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [01:33:06] RESOLVED: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [01:40:42] !log tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41755274531 (https://github.com/cluebotng/component-configs/commits/c68d0cf2fead4e93381cc05f1756241a916a24d6) [01:40:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [01:41:29] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41755274447 (https://github.com/cluebotng/component-configs/commits/c68d0cf2fead4e93381cc05f1756241a916a24d6) [01:41:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [01:42:06] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41755274459 (https://github.com/cluebotng/component-configs/commits/c68d0cf2fead4e93381cc05f1756241a916a24d6) [01:42:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [01:45:01] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [01:45:05] FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [01:45:42] FIRING: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [01:50:01] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [01:50:05] RESOLVED: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [02:20:42] RESOLVED: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [02:25:26] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41760170800 (https://github.com/cluebotng/component-configs/commits/ee3a8262f4918cfcf2e22ccfe0d728ebf8143df2) [02:25:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [02:39:04] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41761692440 (https://github.com/cluebotng/component-configs/commits/3cdc869413ae6a77faaac98309e3b59dab578a53) [02:39:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [02:44:08] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41761998436 (https://github.com/cluebotng/component-configs/commits/c16528be19dec3e78a0c5b289774e8fd56d8ab32) [02:44:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [02:45:42] FIRING: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [02:54:46] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41763262663 (https://github.com/cluebotng/component-configs/commits/d15c2921f9532c4723d817aa328532a85783a46c) [02:54:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [02:59:25] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41763724637 (https://github.com/cluebotng/component-configs/commits/1681819283fff40cec7eedcf2e2ddd6b6e555828) [02:59:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [03:00:12] !log tools.cluebot-syncer Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41764208830 (https://github.com/cluebotng/component-configs/commits/85b5ce73ff9dbeca337eb18aa04b06b92e36e12f) [03:00:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot-syncer/SAL [03:00:22] !log tools.cluebot3 Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41764208889 (https://github.com/cluebotng/component-configs/commits/85b5ce73ff9dbeca337eb18aa04b06b92e36e12f) [03:00:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebot3/SAL [03:02:35] !log tools.toolforge-functional-runner Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41764208828 (https://github.com/cluebotng/component-configs/commits/85b5ce73ff9dbeca337eb18aa04b06b92e36e12f) [03:02:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.toolforge-functional-runner/SAL [03:02:40] !log tools.cluebotng-editsets Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41764208884 (https://github.com/cluebotng/component-configs/commits/85b5ce73ff9dbeca337eb18aa04b06b92e36e12f) [03:02:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-editsets/SAL [03:03:10] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41764161174 (https://github.com/cluebotng/component-configs/commits/efdda3a013777f4e684be77dc6e05704036d4f0a) [03:03:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [03:03:17] !log tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41764208797 (https://github.com/cluebotng/component-configs/commits/85b5ce73ff9dbeca337eb18aa04b06b92e36e12f) [03:03:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [03:04:29] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41764208829 (https://github.com/cluebotng/component-configs/commits/85b5ce73ff9dbeca337eb18aa04b06b92e36e12f) [03:04:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [03:04:40] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41764208847 (https://github.com/cluebotng/component-configs/commits/85b5ce73ff9dbeca337eb18aa04b06b92e36e12f) [03:04:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [03:04:56] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41764208843 (https://github.com/cluebotng/component-configs/commits/85b5ce73ff9dbeca337eb18aa04b06b92e36e12f) [03:04:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [03:07:01] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [03:07:05] FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [03:08:14] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41764208826 (https://github.com/cluebotng/component-configs/commits/85b5ce73ff9dbeca337eb18aa04b06b92e36e12f) [03:08:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [03:12:01] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [03:12:05] RESOLVED: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [03:19:00] FIRING: NovafullstackSustainedFailures: Novafullstack tests have been failing for more than 5hours in eqiad - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NovafullstackSustainedFailures - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-nova-fullstack?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DNovafullstackSustainedFailures [03:25:42] RESOLVED: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [03:45:42] FIRING: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [03:55:00] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [03:59:27] (03approved) 10raymond-ndibe: node22: add new pre-built image [repos/cloud/toolforge/image-config] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config/-/merge_requests/25 (https://phabricator.wikimedia.org/T442893 https://phabricator.wikimedia.org/T443499) (owner: 10dcaro) [04:08:05] FIRING: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [04:12:01] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [04:13:05] FIRING: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [04:15:42] RESOLVED: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [04:17:01] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [04:18:05] RESOLVED: [2x] TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [04:20:01] FIRING: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [04:20:05] FIRING: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [04:25:01] RESOLVED: QuarryDown: Quarry application is unreachable - https://prometheus-alerts.wmcloud.org/?q=alertname%3DQuarryDown [04:25:05] RESOLVED: TargetDown: Job app is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [04:27:06] FIRING: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [04:29:27] !log tools.cluebotng-review Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41773053027 (https://github.com/cluebotng/component-configs/commits/da4b9b8d6a0e7e83007541e61941093c6c9db0b3) [04:29:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [04:31:08] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41773053080 (https://github.com/cluebotng/component-configs/commits/da4b9b8d6a0e7e83007541e61941093c6c9db0b3) [04:31:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [04:32:06] RESOLVED: TargetDown: Job flask is unreachable in project quarry instance quarry.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [04:45:42] FIRING: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [05:15:22] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41777591788 (https://github.com/cluebotng/component-configs/commits/e816c7dcfa9f55dfe3171cd83dcb396d239b99eb) [05:15:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [05:20:57] RESOLVED: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [05:45:42] FIRING: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [05:53:23] 10Toolforge, 06tools-platform-team: [components-api] deployment hangs forever on an unexpected builds-api error - https://phabricator.wikimedia.org/T443557 (10Raymond_Ndibe) 03NEW [05:53:32] 10Toolforge, 06tools-platform-team: [components-api] deployment hangs forever on an unexpected builds-api error - https://phabricator.wikimedia.org/T443557#12514197 (10Raymond_Ndibe) a:03Raymond_Ndibe [05:59:58] 10Toolforge, 06tools-platform-team: [all] design a robust network policy for all our services - https://phabricator.wikimedia.org/T443558 (10Raymond_Ndibe) 03NEW [06:01:59] 10Toolforge, 06tools-platform-team: [all] design a robust network policy for all our services - https://phabricator.wikimedia.org/T443558#12514212 (10Raymond_Ndibe) [06:25:42] RESOLVED: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [06:36:10] !log tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41786107416 (https://github.com/cluebotng/component-configs/commits/c8dca63c87878642bde2b1434d3dba8287dae06a) [06:36:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [06:36:54] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41786107372 (https://github.com/cluebotng/component-configs/commits/c8dca63c87878642bde2b1434d3dba8287dae06a) [06:36:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [06:45:42] FIRING: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [06:53:31] (03CR) 10Lokal Profil: [C:03+2] Use flex sticky-footer so footer never overlaps content [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1275119 (owner: 10Lokal Profil) [06:55:45] (03Merged) 10jenkins-bot: Use flex sticky-footer so footer never overlaps content [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1275119 (owner: 10Lokal Profil) [07:15:42] RESOLVED: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [07:17:12] FIRING: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [07:22:12] RESOLVED: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [07:23:36] (03open) 10raymond-ndibe: deploy: only retry a build start on a 409 [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/212 (https://phabricator.wikimedia.org/T443557) [07:45:42] FIRING: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [07:48:35] (03PS2) 10Lokal Profil: api: Let FormatHtmllist use the shared output loop [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1362226 (https://phabricator.wikimedia.org/T443563) [07:49:05] (03PS2) 10Lokal Profil: api: Rename Language to RegionNames [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1362179 (https://phabricator.wikimedia.org/T443562) [07:49:42] (03PS2) 10Lokal Profil: maintenance: Share the script bootstrap [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1362210 (https://phabricator.wikimedia.org/T443561) [07:49:44] (03PS2) 10Lokal Profil: Refresh the countries cache after each harvest [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/1362211 (https://phabricator.wikimedia.org/T443561) [07:55:00] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [07:55:42] RESOLVED: AlertLintProblem: Linting problems found for CloudVirtDisabled - https://wikitech.wikimedia.org/wiki/Alertmanager#Alert_linting_found_problems - TODO - https://alerts.wikimedia.org/?q=alertname%3DAlertLintProblem [08:02:59] !log filippo@cloudcumin1001 admin START - Cookbook wmcs.openstack.roll_reboot_cloudnets [08:05:25] PROBLEM - Host cloudnet1006 is DOWN: PING CRITICAL - Packet loss = 100% [08:07:53] RECOVERY - Host cloudnet1006 is UP: PING OK - Packet loss = 0%, RTA = 0.25 ms [08:35:38] 10Toolforge (Push-to-Deploy), 06tools-platform-team: Create scheduled job deployment tutorial - https://phabricator.wikimedia.org/T437901#12514455 (10komla) Noted! Addressing them. Thanks [08:56:34] 10Cloud-VPS, 06tools-infrastructure-team, 10Stashbot: stashbot reconnects after 15 minutes on ping timeout - https://phabricator.wikimedia.org/T443568 (10fgiunchedi) 03NEW [09:07:44] 10Cloud-VPS, 06tools-infrastructure-team, 10WM-Bot: wm-bot reconnects after 15 minutes on ping timeout - https://phabricator.wikimedia.org/T443569 (10fgiunchedi) 03NEW [09:24:35] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [components-api] deployment hangs forever on an unexpected builds-api error - https://phabricator.wikimedia.org/T443557#12514585 (10dcaro) p:05Triage→03High [09:40:34] FIRING: ToolsNfsAlmostFull: Toolforge NFS is almost full - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsNfsAlmostFull - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsNfsAlmostFull [09:42:27] (03update) 10dcaro: add pagination [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/75 (https://phabricator.wikimedia.org/T440294) [09:43:06] FIRING: [4x] PuppetAgentNoResources: No Puppet resources found on instance cvn-apache12 on project cvn - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [09:46:35] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance scratch-2 on project cloudinfra-nfs - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [09:53:06] FIRING: [4x] PuppetAgentNoResources: No Puppet resources found on instance cvn-apache12 on project cvn - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [09:53:33] 06cloud-services-team, 10PAWS, 06tools-platform-team: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T435704#12514637 (10TLepage-WMF) GitHub PR approved! ✅ [09:55:30] RESOLVED: NovafullstackSustainedFailures: Novafullstack tests have been failing for more than 5hours in eqiad - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NovafullstackSustainedFailures - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-nova-fullstack?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DNovafullstackSustainedFailures [09:58:06] RESOLVED: [4x] PuppetAgentNoResources: No Puppet resources found on instance cvn-apache12 on project cvn - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [10:04:35] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance quarry-bastion on project quarry - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [10:05:06] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance extdist-07 on project extdist - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [10:05:35] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance etcd-discovery-1 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [10:06:35] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance metricsinfra-puppetserver-1 on project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [10:06:41] (03update) 10dcaro: add noop [repos/cloud/toolforge/maintain-harbor] (add_pagination) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/73 (https://phabricator.wikimedia.org/T440295) [10:07:21] (03update) 10dcaro: add noop [repos/cloud/toolforge/maintain-harbor] (add_pagination) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/73 (https://phabricator.wikimedia.org/T440295) [10:07:24] (03update) 10dcaro: add noop [repos/cloud/toolforge/maintain-harbor] (add_pagination) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/73 (https://phabricator.wikimedia.org/T440295) [10:07:35] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance project-proxy-puppetserver-1 on project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [10:10:10] 06cloud-services-team, 10PAWS, 06tools-platform-team: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T435704#12514683 (10fnegri) 05In progress→03Resolved [10:18:04] 10Cloud-VPS, 06tools-infrastructure-team, 10WM-Bot: wm-bot.wmflabs.org now incorrectly redirects to wm-bot.wmcloud.org - https://phabricator.wikimedia.org/T443531#12514710 (10taavi) a:03taavi This seems like a regression from {T429930}, where the redirect @bd808 mentions is now applied regardless of whethe... [10:18:35] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance paws-puppetserver-1 on project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [10:20:34] RESOLVED: ToolsNfsAlmostFull: Toolforge NFS is almost full - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsNfsAlmostFull - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsNfsAlmostFull [10:20:39] 06tools-platform-team: [all] first of the month dependency upgrades for 03-11-2026 - https://phabricator.wikimedia.org/T443392#12514719 (10aputhin) p:05Triage→03High [10:25:14] (03update) 10dcaro: jobs: delete also untagged artifacts [repos/cloud/toolforge/maintain-harbor] (add_noop) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/74 (https://phabricator.wikimedia.org/T440343) [10:27:54] 10Cloud-VPS, 06tools-infrastructure-team: Ensure cron job to update ipblocks data on Cloud VPS requestctl instance is ran regularly - https://phabricator.wikimedia.org/T443575 (10taavi) 03NEW [10:27:56] 10Toolforge, 06tools-platform-team, 07Epic, 07OKR-Work: [hypothesis] ST5.4.1 Extend logging capabilities - https://phabricator.wikimedia.org/T432564#12514745 (10dcaro) 05In progress→03Resolved [10:33:22] (03update) 10dcaro: jobs: skip artifacts that don't exist on deletion [repos/cloud/toolforge/maintain-harbor] (delete_untagged) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/76 (https://phabricator.wikimedia.org/T440343) [10:34:07] 10Cloud-VPS, 06tools-infrastructure-team: Occasional high ping from irb-1120.cloudsw1-c8-eqiad.eqiad1.wikimediacloud.org - https://phabricator.wikimedia.org/T443577 (10fgiunchedi) 03NEW [10:37:11] 10Cloud-VPS, 06tools-infrastructure-team, 10WM-Bot, 13Patch-For-Review: wm-bot.wmflabs.org now incorrectly redirects to wm-bot.wmcloud.org - https://phabricator.wikimedia.org/T443531#12514793 (10taavi) 05Open→03Resolved [10:38:07] 10Cloud-VPS, 06tools-infrastructure-team: Occasional high ping from irb-1120.cloudsw1-c8-eqiad.eqiad1.wikimediacloud.org - https://phabricator.wikimedia.org/T443577#12514800 (10fgiunchedi) [10:38:53] (03update) 10dcaro: jobs: skip artifacts that don't exist on deletion [repos/cloud/toolforge/maintain-harbor] (delete_untagged) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/76 (https://phabricator.wikimedia.org/T440343) [10:38:59] 10Cloud-VPS, 06tools-infrastructure-team: Occasional high ping from irb-1120.cloudsw1-c8-eqiad.eqiad1.wikimediacloud.org - https://phabricator.wikimedia.org/T443577#12514806 (10fgiunchedi) [10:39:26] 10Toolforge, 06tools-platform-team: maintain-harbor: delete_stale_artifacts deletes images in use - https://phabricator.wikimedia.org/T442154#12514809 (10dcaro) a:03TLepage-WMF [10:39:47] (03update) 10dcaro: bugfix: delete_stale_artifacts no longer deletes images in use [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/77 (https://phabricator.wikimedia.org/T442154) (owner: 10tlepage) [10:42:25] (03approved) 10dcaro: bugfix: delete_stale_artifacts no longer deletes images in use [repos/cloud/toolforge/maintain-harbor] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-harbor/-/merge_requests/77 (https://phabricator.wikimedia.org/T442154) (owner: 10tlepage) [10:44:26] (03CR) 10Thibaut Le Page: [C:03+1] node22: add new pre-built image [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1362082 (https://phabricator.wikimedia.org/T443499) (owner: 10David Caro) [10:44:55] (03CR) 10David Caro: [C:03+2] node22: add new pre-built image [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1362082 (https://phabricator.wikimedia.org/T443499) (owner: 10David Caro) [10:45:26] (03merge) 10dcaro: node22: add new pre-built image [repos/cloud/toolforge/image-config] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config/-/merge_requests/25 (https://phabricator.wikimedia.org/T442893 https://phabricator.wikimedia.org/T443499) [10:45:37] (03Merged) 10jenkins-bot: node22: add new pre-built image [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1362082 (https://phabricator.wikimedia.org/T443499) (owner: 10David Caro) [10:46:58] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: image-config: bump to 0.0.30-20261104104540-e82d3242 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1498 (https://phabricator.wikimedia.org/T442893) [10:50:07] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [maintain-harbor] delete untagged artifacts too - https://phabricator.wikimedia.org/T440343#12514873 (10dcaro) @TLepage-WMF I just did an update of the patches, waiting for reviews from @Raymond_Ndibe, feel free to take over adapting/merging if I don't... [11:01:25] 10Cloud-VPS, 06tools-infrastructure-team: Error message from Puppet Service for wlm-it-visual - https://phabricator.wikimedia.org/T443580 (10Ysogo) 03NEW [11:02:23] 10Cloud-VPS, 06tools-infrastructure-team: Error message from Puppet Service for wlm-it-visual - https://phabricator.wikimedia.org/T443580#12514935 (10Ysogo) [11:14:41] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component image-config [11:19:28] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component image-config [11:19:37] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component image-config [11:21:38] 10Toolforge, 06tools-platform-team: [toolsbeta] update all the components - https://phabricator.wikimedia.org/T443582 (10dcaro) 03NEW [11:29:45] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config [11:30:02] 10Cloud-VPS, 06tools-infrastructure-team: Error message from Puppet Service for wlm-it-visual - https://phabricator.wikimedia.org/T443580#12515010 (10Volans) 05Open→03Resolved a:03Volans @Ysogo this was due to a maintenance on our side, see the related announcement here: https://lists.wikimedia.org/h... [11:49:06] 10Toolforge, 06tools-platform-team: [all] design a robust network policy for all our services - https://phabricator.wikimedia.org/T443558#12515086 (10DamianZaremba) T362539 for api gateway T400940 request for tool support (handled directly with NetworkPolicy objects currently) [11:50:47] 10superset.wmcloud.org: Superset WMCloud fails to access s[0-9] clusters - https://phabricator.wikimedia.org/T443585 (10Urbanecm) 03NEW [11:51:08] 10Cloud-Services, 10superset.wmcloud.org: Superset WMCloud fails to access s[0-9] clusters - https://phabricator.wikimedia.org/T443585#12515111 (10Urbanecm) The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ and rep... [11:51:30] 10Data-Services, 10superset.wmcloud.org, 06tools-platform-team: Superset WMCloud fails to access s[0-9] clusters - https://phabricator.wikimedia.org/T443585#12515113 (10Urbanecm) [11:55:00] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [12:31:22] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component image-config [12:37:22] !log dcaro@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component image-config [12:47:42] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component image-config [12:57:39] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component image-config [12:58:20] (03approved) 10dcaro: image-config: bump to 0.0.30-20261104104540-e82d3242 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1498 (https://phabricator.wikimedia.org/T442893) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [12:58:27] (03merge) 10dcaro: image-config: bump to 0.0.30-20261104104540-e82d3242 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1498 (https://phabricator.wikimedia.org/T442893) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [13:04:52] 10Data-Services, 10superset.wmcloud.org: Superset WMCloud fails to access s[0-9] clusters - https://phabricator.wikimedia.org/T443585#12515444 (10aputhin) Removing our tag, as this is now maintained by community members who coordinate via the #superset.wmcloud.org tag, as per https://lists.wikimedia.org/hyperk... [13:43:50] 10Toolforge, 06tools-platform-team: [all] first of the month dependency upgrades for November 2026 - https://phabricator.wikimedia.org/T443532#12515688 (10aputhin) p:05Triage→03High [13:47:51] 10Tool-delintbot: Add margin: auto to tables when replacing center tags - https://phabricator.wikimedia.org/T443603 (10Redmin) 03NEW [13:48:04] 10Tool-delintbot: Add margin: auto to tables when replacing center tags - https://phabricator.wikimedia.org/T443603#12515719 (10Redmin) p:05Triage→03Medium [13:51:48] 10superset.wmcloud.org: Superset WMCloud Error when clicking o the translation stats - https://phabricator.wikimedia.org/T442327#12515755 (10ForzaGreen) Hi @Aklapper , Superset is partially back now. See this comment for details: https://phabricator.wikimedia.org/T416373#12509047 I can log in and view some dashb... [14:04:03] 10Cloud-VPS, 06tools-infrastructure-team: Make profile::conftool::requestctl_client compatible with Cloud VPS - https://phabricator.wikimedia.org/T443608 (10taavi) 03NEW [14:25:17] (03PS1) 10Majavah: Add fake WMCS WAF hiddenparma root token [labs/private] - 10https://gerrit.wikimedia.org/r/1363244 [14:25:41] (03CR) 10Majavah: [V:03+2 C:03+2] Add fake WMCS WAF hiddenparma root token [labs/private] - 10https://gerrit.wikimedia.org/r/1363244 (owner: 10Majavah) [14:25:49] 10Tool-commons-delinquent: Tool-commons-delinquent: HTTP Code 500 error - https://phabricator.wikimedia.org/T437416#12515930 (10grin) It is known, waits to be fixed. Happens due to the changes of the database replicas. [14:26:28] 10Cloud-VPS, 06tools-infrastructure-team: Make profile::conftool::requestctl_client compatible with Cloud VPS - https://phabricator.wikimedia.org/T443608#12515932 (10taavi) a:03taavi [14:29:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:36:03] (03open) 10r4356th: Add margin: auto to tables when replacing center tags that surround them [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/47 (https://phabricator.wikimedia.org/T443603) [14:36:49] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41848610075 (https://github.com/cluebotng/component-configs/commits/8c9d33b12cfd03687a781e4cc65df3014ba5ac3f) [14:36:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [14:39:40] (03merge) 10r4356th: Add margin: auto to tables when replacing center tags that surround them [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/47 (https://phabricator.wikimedia.org/T443603) [14:40:16] 10Tool-delintbot: Add margin: auto to tables when replacing center tags - https://phabricator.wikimedia.org/T443603#12516057 (10Redmin) 05Open→03Resolved a:03Redmin [14:48:37] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41850597334 (https://github.com/cluebotng/component-configs/commits/6d4c24a0f2ad8cc6b8220fb6195cac8eca56120e) [14:48:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [14:49:58] 10Cloud-VPS, 06tools-infrastructure-team: Occasional high ping from irb-1120.cloudsw1-c8-eqiad.eqiad1.wikimediacloud.org - https://phabricator.wikimedia.org/T443577#12516128 (10fgiunchedi) [14:50:11] 10Striker, 06tools-platform-team, 07OKR-Work: Striker: Show active tool alerts - https://phabricator.wikimedia.org/T441098#12516129 (10TLepage-WMF) 05Open→03In progress [14:50:40] 10Striker, 06tools-platform-team, 07OKR-Work: Striker: See past alerts - https://phabricator.wikimedia.org/T441099#12516140 (10TLepage-WMF) 05Open→03In progress [15:01:34] 10Striker, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: Draft: Measure adoption of reliability views - https://phabricator.wikimedia.org/T441328#12516280 (10TLepage-WMF) 05Open→03In progress [15:24:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:42:59] 10superset.wmcloud.org: Superset WMCloud fails to access s[0-9] clusters - https://phabricator.wikimedia.org/T443585#12516735 (10JJMC89) [15:54:44] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41861784896 (https://github.com/cluebotng/component-configs/commits/e820717bfa1e2145b54da8fff522fc828da9457f) [15:54:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [15:55:01] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [16:08:32] (03open) 10r4356th: Upgrade to Python 3.15 [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/48 (https://phabricator.wikimedia.org/T442326) [16:19:41] !log tools.cluebotng-trainer Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41865423884 (https://github.com/cluebotng/component-configs/commits/86b7159004732c324a71b91e87663d2e1cea32df) [16:19:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-trainer/SAL [16:20:10] !log tools.cluebotng-monitoring Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41865536484 (https://github.com/cluebotng/component-configs/commits/358d261c41034263816250714b7bda000b460afa) [16:20:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [16:20:13] (03update) 10r4356th: Upgrade to Python 3.15 [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/48 (https://phabricator.wikimedia.org/T442326) [16:32:04] (03approved) 10dcaro: toolforge-cd,utils: do not add the 'Needs review' label to bot merge requests [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/101 (https://phabricator.wikimedia.org/T443238) (owner: 10raymond-ndibe) [16:42:00] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [image-config,pre-built] add node22 pre-built image - https://phabricator.wikimedia.org/T443499#12517359 (10fnegri) Anything left to do here given the patch is now merged, and I see the image in `toolforge jobs images`? [16:42:56] 06cloud-services-team (Hardware), 06tools-infrastructure-team, 06DC-Ops, 06SRE, 06Data-Platform-SRE (2026-10-30 - 2026-11-20): How to configure storage on new dumps servers - https://phabricator.wikimedia.org/T441202#12517370 (10Gehel) [16:47:42] 06cloud-services-team (Hardware), 06tools-infrastructure-team, 06DC-Ops, 06SRE, and 2 others: How to configure storage on new dumps servers - https://phabricator.wikimedia.org/T441202#12517437 (10Gehel) p:05Triage→03High [16:50:40] (03update) 10raymond-ndibe: deploy: only retry a build start on a 409 [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/212 (https://phabricator.wikimedia.org/T443557) [16:50:51] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [image-config,pre-built] add node22 pre-built image - https://phabricator.wikimedia.org/T443499#12517453 (10dcaro) >>! In T443499#12517359, @fnegri wrote: > Anything left to do here given the patch is now merged, and I see the image in `toolforge jobs i... [16:51:24] (03update) 10raymond-ndibe: deploy: only retry a build start on a 409 [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/212 (https://phabricator.wikimedia.org/T443557) [16:51:31] 10Toolforge, 06tools-platform-team: [components-api] leaks internal authentication header requirement on bad client request - https://phabricator.wikimedia.org/T443490#12517481 (10aputhin) >>! In T443490#12516850, @DamianZaremba wrote: > "Leaking" was in reference to an internal implementation detail that isn'... [16:51:44] (03approved) 10raymond-ndibe: deploy: only retry a build start on a 409 [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/212 (https://phabricator.wikimedia.org/T443557) [16:52:19] (03update) 10raymond-ndibe: deploy: only retry a build start on a 409 [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/212 (https://phabricator.wikimedia.org/T443557) [16:52:33] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41870962440 (https://github.com/cluebotng/component-configs/commits/3c1b5d1defc502bca33e6c53fe4f68d97f33cfba) [16:52:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [16:53:15] (03merge) 10raymond-ndibe: deploy: only retry a build start on a 409 [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/212 (https://phabricator.wikimedia.org/T443557) [16:55:25] !log tools.cluebotng-review Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41871475015 (https://github.com/cluebotng/component-configs/commits/2013e4c5bd958fdebe0a0c4511369f5a2457b368) [16:55:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [16:55:44] 10Toolforge, 06tools-platform-team, 07OKR-Work: alerts-api: Email templates should deep-link to Striker - https://phabricator.wikimedia.org/T443658 (10TLepage-WMF) 03NEW [16:56:10] 10Toolforge, 06tools-platform-team, 07OKR-Work: alerts-api: Email templates should deep-link to Striker - https://phabricator.wikimedia.org/T443658#12517564 (10TLepage-WMF) [16:56:11] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: components-api: bump to 0.0.232-20261104165326-b4ab3772 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1497 (https://phabricator.wikimedia.org/T362077 https://phabricator.wikimedia.org/T443557) [16:56:12] 10Toolforge, 06tools-platform-team, 07Epic, 07OKR-Work: ST5.4.3 Implement Toolforge UI MVP - https://phabricator.wikimedia.org/T440589#12517565 (10TLepage-WMF) [16:56:13] (03approved) 10raymond-ndibe: toolforge-cd,utils: do not add the 'Needs review' label to bot merge requests [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/101 (https://phabricator.wikimedia.org/T443238) [16:56:33] (03merge) 10raymond-ndibe: toolforge-cd,utils: do not add the 'Needs review' label to bot merge requests [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/101 (https://phabricator.wikimedia.org/T443238) [16:56:36] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: components-api: bump to 0.0.232-20261104165326-b4ab3772 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1497 (https://phabricator.wikimedia.org/T362077 https://phabricator.wikimedia.org/T443557) [16:56:40] (03update) 10raymond-ndibe: toolforge-cd,utils: auto-create phab tasks for the monthly upgrade MRs [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/102 (https://phabricator.wikimedia.org/T443234) [16:56:41] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [cicd] don't add the 'Needs review' label to auto-created MRs - https://phabricator.wikimedia.org/T443238#12517571 (10Raymond_Ndibe) 05In progress→03Resolved [16:57:40] 10Striker, 06tools-platform-team, 07OKR-Work: Striker: See past alerts - https://phabricator.wikimedia.org/T441099#12517590 (10TLepage-WMF) [16:57:41] 10Toolforge, 06tools-platform-team, 07OKR-Work: alerts-api: Email templates should deep-link to Striker - https://phabricator.wikimedia.org/T443658#12517589 (10TLepage-WMF) [16:57:44] (03update) 10raymond-ndibe: components-api: bump to 0.0.232-20261104165326-b4ab3772 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1497 (https://phabricator.wikimedia.org/T362077 https://phabricator.wikimedia.org/T443557) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [16:58:44] 10Toolforge, 06tools-platform-team, 07OKR-Work: alerts-api: Email templates should deep-link to Striker - https://phabricator.wikimedia.org/T443658#12517614 (10TLepage-WMF) [16:58:47] 10Toolforge, 06tools-platform-team, 07Epic, 07OKR-Work: ST5.4.3 Implement Toolforge UI MVP - https://phabricator.wikimedia.org/T440589#12517616 (10TLepage-WMF) [16:58:49] 10Striker, 06tools-platform-team, 07OKR-Work, 13Patch-For-Review: Draft: Measure adoption of reliability views - https://phabricator.wikimedia.org/T441328#12517615 (10TLepage-WMF) [17:06:19] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [image-config,pre-built] add node22 pre-built image - https://phabricator.wikimedia.org/T443499#12517685 (10fnegri) > I wanted to send an email or something, wdyt? Up to you, not strictly needed IMHO, but maybe we could follow up in the existing email... [17:12:01] 10Quarry, 06tools-platform-team: [quarry] started failing when results directory does not exist - https://phabricator.wikimedia.org/T443411#12517767 (10dcaro) a:03dcaro [17:16:54] 10Quarry, 06tools-platform-team: [quarry] started failing when results directory does not exist - https://phabricator.wikimedia.org/T443411#12517820 (10github-toolforge-bot) david-caro opened https://github.com/toolforge/quarry/pull/113 [17:29:09] 10Toolforge, 06tools-platform-team: [builds-api] returns incorrect parameter values for use_latest_versions & use_deprecated_versions causing [components-api] to not re-use existing builds - https://phabricator.wikimedia.org/T442364#12517949 (10DamianZaremba) Hi @dcaro, I'm in agreement for the context outline... [17:35:07] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [image-config,pre-built] add node22 pre-built image - https://phabricator.wikimedia.org/T443499#12518049 (10dcaro) Just added a note in the changelog, the docs update automatically: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Changelog#2026 [17:37:02] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41878343537 (https://github.com/cluebotng/component-configs/commits/d03a8439ccb347e558a42e94a674edbc6d1b087d) [17:37:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [17:38:21] 06cloud-services-team (Hardware), 06tools-infrastructure-team, 06DC-Ops, 06SRE, and 2 others: How to configure storage on new dumps servers - https://phabricator.wikimedia.org/T441202#12518079 (10BTullis) Thanks for the ping and apologies for being slow to respond. I'm looking into this now, too. I'm noti... [17:42:25] !log tools.cluebotng-staging Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41879214249 (https://github.com/cluebotng/component-configs/commits/230211c76403808093c776115bc37480e740de36) [17:42:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-staging/SAL [17:42:47] 10Toolforge, 06tools-platform-team: [builds-api] returns incorrect parameter values for use_latest_versions & use_deprecated_versions causing [components-api] to not re-use existing builds - https://phabricator.wikimedia.org/T442364#12518191 (10dcaro) p:05Medium→03High Agree [17:44:20] !log tools.cluebotng Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41879214126 (https://github.com/cluebotng/component-configs/commits/230211c76403808093c776115bc37480e740de36) [17:44:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL [17:44:41] 10Striker, 06tools-platform-team, 10ChangeProp, 10Citoid, and 22 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12518207 (10dduvall) [17:55:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:56:44] (03open) 10r4356th: Add a script to retrieve the latest data from the database [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/49 (https://phabricator.wikimedia.org/T440299) [17:57:03] (03update) 10r4356th: Add a script to retrieve the latest data about blocked edit attempts from the database [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/49 (https://phabricator.wikimedia.org/T440299) [17:57:18] (03merge) 10r4356th: Add a script to retrieve the latest data about blocked edit attempts from the database [toolforge-repos/delintbot] - 10https://gitlab.wikimedia.org/toolforge-repos/delintbot/-/merge_requests/49 (https://phabricator.wikimedia.org/T440299) [17:58:58] (03approved) 10dcaro: Store build options as annotations [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/169 (owner: 10damian) [17:59:11] (03merge) 10dcaro: Store build options as annotations [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/169 (owner: 10damian) [17:59:34] 10Tool-delintbot, 13Patch-For-Review: Store data about blocked edit attempts - https://phabricator.wikimedia.org/T440299#12518356 (10Redmin) [17:59:45] 10Tool-delintbot, 13Patch-For-Review: Store data about blocked edit attempts - https://phabricator.wikimedia.org/T440299#12518358 (10Redmin) 05Open→03Resolved a:03Redmin [18:00:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [18:00:47] 10Toolforge, 06tools-platform-team: [builds-api] returns incorrect parameter values for use_latest_versions & use_deprecated_versions causing [components-api] to not re-use existing builds - https://phabricator.wikimedia.org/T442364#12518378 (10dcaro) It's missing the components-api side though, I'm off for th... [18:05:44] (03update) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: builds-api: bump to 0.0.225-20261104175921-6e4276df [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1499 (https://phabricator.wikimedia.org/T442364) [18:05:45] (03open) 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce: builds-api: bump to 0.0.225-20261104175921-6e4276df [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1499 (https://phabricator.wikimedia.org/T442364) [18:05:58] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-api [18:08:24] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [image-config,pre-built] add node22 pre-built image - https://phabricator.wikimedia.org/T443499#12518425 (10dcaro) 05Open→03Resolved [18:08:36] (03open) 10damian: _check_for_matching_build - check the build's builder is current [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/213 (https://phabricator.wikimedia.org/T442364) [18:09:37] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [18:10:12] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component builds-api [18:10:29] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.copy_istio_images_to_registry for Istio 1.31.3 [18:10:56] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.copy_istio_images_to_registry (exit_code=0) for Istio 1.31.3 [18:14:28] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [18:14:50] (03approved) 10dcaro: builds-api: bump to 0.0.225-20261104175921-6e4276df [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1499 (https://phabricator.wikimedia.org/T442364) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [18:14:55] (03merge) 10dcaro: builds-api: bump to 0.0.225-20261104175921-6e4276df [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1499 (https://phabricator.wikimedia.org/T442364) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [18:17:41] (03update) 10damian: _check_for_matching_build - check the build's builder is current [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/213 (https://phabricator.wikimedia.org/T442364) [18:19:27] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [builds-api] returns incorrect parameter values for use_latest_versions & use_deprecated_versions causing [components-api] to not re-use existing builds - https://phabricator.wikimedia.org/T442364#12518475 (10DamianZaremba) https://gitlab.wikimedia.org/... [18:20:17] (03update) 10damian: _check_for_matching_build - check the build's builder is current [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/213 (https://phabricator.wikimedia.org/T442364) [18:23:54] (03update) 10damian: _check_for_matching_build - check the build's builder is current [repos/cloud/toolforge/components-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/213 (https://phabricator.wikimedia.org/T442364) [18:25:04] (03update) 10taavi: istio-system: Update Helm repository URL [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1500 (https://phabricator.wikimedia.org/T442885) [18:25:06] (03update) 10taavi: istio-system: Update to 1.31.3 on non-prod environments [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1501 (https://phabricator.wikimedia.org/T442885) [18:25:09] (03update) 10taavi: istio-system: Update to 1.31.3 on tools [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1502 (https://phabricator.wikimedia.org/T442885) [18:25:17] (03open) 10taavi: istio-system: Update Helm repository URL [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1500 (https://phabricator.wikimedia.org/T442885) [18:25:18] (03open) 10taavi: istio-system: Update to 1.31.3 on non-prod environments [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1501 (https://phabricator.wikimedia.org/T442885) [18:25:19] (03open) 10taavi: istio-system: Update to 1.31.3 on tools [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1502 (https://phabricator.wikimedia.org/T442885) [18:25:24] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component istio-system [18:26:23] 10Toolforge, 06tools-platform-team: [components-api] requires re-formatting generated code - https://phabricator.wikimedia.org/T443675 (10DamianZaremba) 03NEW [18:40:57] (03update) 10taavi: istio-system: Update Helm repository URL [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1500 (https://phabricator.wikimedia.org/T442885) [18:40:57] (03update) 10taavi: istio-system: Update to 1.31.3 on tools [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1502 (https://phabricator.wikimedia.org/T442885) [18:40:58] (03update) 10taavi: istio-system: Update to 1.31.3 on non-prod environments [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1501 (https://phabricator.wikimedia.org/T442885) [18:41:49] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component istio-system [18:42:14] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component istio-system [18:45:16] (03update) 10taavi: istio-system: Update Helm repository URL [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1500 (https://phabricator.wikimedia.org/T442885) [18:45:17] (03merge) 10taavi: istio-system: Update Helm repository URL [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1500 (https://phabricator.wikimedia.org/T442885) [18:45:19] (03update) 10taavi: istio-system: Update to 1.31.3 on non-prod environments [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1501 (https://phabricator.wikimedia.org/T442885) [18:45:25] (03merge) 10taavi: istio-system: Update to 1.31.3 on non-prod environments [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1501 (https://phabricator.wikimedia.org/T442885) [18:45:27] (03update) 10taavi: istio-system: Update to 1.31.3 on tools [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1502 (https://phabricator.wikimedia.org/T442885) [18:45:36] (03merge) 10taavi: istio-system: Update to 1.31.3 on tools [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1502 (https://phabricator.wikimedia.org/T442885) [18:47:59] !log taavi@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component istio-system [18:52:40] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [builds-api] returns incorrect parameter values for use_latest_versions & use_deprecated_versions causing [components-api] to not re-use existing builds - https://phabricator.wikimedia.org/T442364#12518629 (10DamianZaremba) Had to re-format the generate... [18:53:10] 10Toolforge, 06tools-platform-team: `config generate with some existing jobs works` failure - https://phabricator.wikimedia.org/T443680 (10taavi) 03NEW [18:59:39] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [builds-api] returns incorrect parameter values for use_latest_versions & use_deprecated_versions causing [components-api] to not re-use existing builds - https://phabricator.wikimedia.org/T442364#12518671 (10DamianZaremba) We can also un-wind https://g... [19:02:10] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-sessionstore06 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436471#12518683 (10Southparkfan) 05Open→03In progress a:03Southparkfan [19:02:35] (03open) 10damian: Revert "builds-api: use different tags for latest/deprecated" [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1503 (https://phabricator.wikimedia.org/T442364) [19:14:30] (03merge) 10sadrettin: Düzenleme özetinde çıkarılan kategoriyi çizgisiz yaz [toolforge-repos/kategori] - 10https://gitlab.wikimedia.org/toolforge-repos/kategori/-/merge_requests/10 (owner: 10project_5175_bot_1dc5cd2b4a7cadb6d1af8e29974ce1a7) [19:45:01] FIRING: [3x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [20:05:01] FIRING: [2x] OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [20:16:02] 10Cloud-VPS (Debian Bullseye Deprecation), 10Tool-bub2: Migrate bub2 away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435761#12519025 (10Andrew) 05Open→03Resolved a:03Andrew [20:18:41] 10Cloud-VPS (Debian Bullseye Deprecation), 10Capacity Exchange: Migrate capacity-exchange away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435763#12519045 (10Andrew) 05Open→03Resolved a:03Andrew [20:21:33] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate dumps away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435767#12519066 (10Andrew) 05Open→03Resolved a:03Andrew [20:23:14] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate fastcci away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435771#12519092 (10Andrew) 05Open→03Resolved a:03Andrew [20:25:14] 10Cloud-VPS (Debian Bullseye Deprecation), 10WMF-Memory-Bank: Migrate foundationmemory away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435772#12519108 (10Andrew) 05Open→03Resolved a:03Andrew [20:26:14] (03open) 10project_5175_bot_1dc5cd2b4a7cadb6d1af8e29974ce1a7: Kuyruktaki işler gizlenir; türler, aynı adlı kategori, üst kategori düzenleme, toplu QID, geri alma [toolforge-repos/kategori] - 10https://gitlab.wikimedia.org/toolforge-repos/kategori/-/merge_requests/11 [20:26:15] (03update) 10project_5175_bot_1dc5cd2b4a7cadb6d1af8e29974ce1a7: Kuyruktaki işler gizlenir; türler, aynı adlı kategori, üst kategori düzenleme, toplu QID, geri alma [toolforge-repos/kategori] - 10https://gitlab.wikimedia.org/toolforge-repos/kategori/-/merge_requests/11 [20:26:45] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate imagebulk away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435773#12519115 (10Andrew) 05Open→03Resolved a:03Andrew [20:27:09] (03merge) 10sadrettin: Kuyruktaki işler gizlenir; türler, aynı adlı kategori, üst kategori düzenleme, toplu QID, geri alma [toolforge-repos/kategori] - 10https://gitlab.wikimedia.org/toolforge-repos/kategori/-/merge_requests/11 (owner: 10project_5175_bot_1dc5cd2b4a7cadb6d1af8e29974ce1a7) [20:28:19] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate incubator away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435774#12519120 (10Andrew) 05Open→03Resolved a:03Andrew [20:35:37] (03update) 10raymond-ndibe: components-api: bump to 0.0.232-20261104165326-b4ab3772 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1497 (https://phabricator.wikimedia.org/T362077 https://phabricator.wikimedia.org/T443557) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [20:36:09] !log raymond-ndibe@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component components-api [20:39:16] !log raymond-ndibe@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component components-api [20:45:22] 10Cloud-VPS (Debian Bullseye Deprecation), 10ISA: Migrate isa away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435776#12519238 (10Andrew) 05Open→03Resolved a:03Andrew [20:46:31] 10Cloud-VPS (Debian Bullseye Deprecation), 10Matrix: Migrate matrix away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435777#12519246 (10Andrew) 05Open→03Resolved a:03Andrew [20:47:33] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component components-api [20:48:15] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate signwriting away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435781#12519256 (10Andrew) 05Open→03Resolved a:03Andrew [20:49:22] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api [20:50:48] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate teyora away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435786#12519281 (10Andrew) 05Open→03Resolved a:03Andrew [20:51:50] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate wildcat away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435790#12519290 (10Andrew) 05Open→03Resolved a:03Andrew [20:52:53] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate wmcs-uptime away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435759#12519314 (10Andrew) 05Open→03Resolved a:03Andrew [20:54:23] 10Cloud-VPS (Debian Bullseye Deprecation): Migrate text-to-speech away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T435785#12519337 (10Andrew) 05Open→03Resolved a:03Andrew [20:56:58] 10Tool-wikinewsie: Multimedia focus original reporting on Wikinewsie - https://phabricator.wikimedia.org/T435264#12519345 (10Pharos) >>! In T435264#12439746, @E_mln_e wrote: > Following on our discussion on Signal, my suggestion was that commons category are oftentimes related to the news - e.g., festivals, cult... [20:57:00] !log raymond-ndibe@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component components-api [20:58:16] !log raymond-ndibe@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component components-api [21:02:33] 10Striker, 06tools-platform-team, 10Charts, 06Language and Product Localization, and 11 others: Migrate node-based services in production to node22 - https://phabricator.wikimedia.org/T393434#12519362 (10Jdforrester-WMF) [21:03:40] 10Striker, 06tools-platform-team, 10Citoid, 10CXServer, and 21 others: Migrate node-based services in production to node26 - https://phabricator.wikimedia.org/T437512#12519364 (10Jdforrester-WMF) [21:08:41] FIRING: CloudVPSDesignateLeaks: Detected 12 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:17:01] (03open) 10project_5175_bot_1dc5cd2b4a7cadb6d1af8e29974ce1a7: Son değişiklikler, ana sayfada istatistik çubuğu ve görsel (P18) önerileri [toolforge-repos/kategori] - 10https://gitlab.wikimedia.org/toolforge-repos/kategori/-/merge_requests/12 [21:18:01] (03merge) 10sadrettin: Son değişiklikler, ana sayfada istatistik çubuğu ve görsel (P18) önerileri [toolforge-repos/kategori] - 10https://gitlab.wikimedia.org/toolforge-repos/kategori/-/merge_requests/12 (owner: 10project_5175_bot_1dc5cd2b4a7cadb6d1af8e29974ce1a7) [21:22:44] (03open) 10project_5175_bot_1dc5cd2b4a7cadb6d1af8e29974ce1a7: Sayfanın üstündeki kategori.toolforge.org başlığını kaldır [toolforge-repos/kategori] - 10https://gitlab.wikimedia.org/toolforge-repos/kategori/-/merge_requests/13 [21:22:46] (03update) 10project_5175_bot_1dc5cd2b4a7cadb6d1af8e29974ce1a7: Sayfanın üstündeki kategori.toolforge.org başlığını kaldır [toolforge-repos/kategori] - 10https://gitlab.wikimedia.org/toolforge-repos/kategori/-/merge_requests/13 [21:23:42] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-sessionstore06 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436471#12519535 (10Southparkfan) The plan: - [x] Spin up `deployment-sessionstore07` - [x] In `deployment-sessionstore` prefix hiera: set `prof... [21:23:56] (03update) 10project_5175_bot_1dc5cd2b4a7cadb6d1af8e29974ce1a7: Sayfanın üstündeki kategori.toolforge.org başlığını kaldır [toolforge-repos/kategori] - 10https://gitlab.wikimedia.org/toolforge-repos/kategori/-/merge_requests/13 [21:24:24] (03merge) 10sadrettin: Sayfanın üstündeki kategori.toolforge.org başlığını kaldır [toolforge-repos/kategori] - 10https://gitlab.wikimedia.org/toolforge-repos/kategori/-/merge_requests/13 (owner: 10project_5175_bot_1dc5cd2b4a7cadb6d1af8e29974ce1a7) [21:24:38] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:35:19] (03approved) 10raymond-ndibe: components-api: bump to 0.0.232-20261104165326-b4ab3772 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1497 (https://phabricator.wikimedia.org/T362077 https://phabricator.wikimedia.org/T443557) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [21:35:31] (03merge) 10raymond-ndibe: components-api: bump to 0.0.232-20261104165326-b4ab3772 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1497 (https://phabricator.wikimedia.org/T362077 https://phabricator.wikimedia.org/T443557) (owner: 10group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce) [21:36:12] 10Toolforge, 06tools-platform-team, 13Patch-For-Review: [components-api] deployment hangs forever on an unexpected builds-api error - https://phabricator.wikimedia.org/T443557#12519592 (10Raymond_Ndibe) 05Open→03Resolved [21:38:36] (03open) 10project_5175_bot_1dc5cd2b4a7cadb6d1af8e29974ce1a7: Son tarama zamanı en üstteki çubuğun solunda [toolforge-repos/kategori] - 10https://gitlab.wikimedia.org/toolforge-repos/kategori/-/merge_requests/14 [21:39:38] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-7:443 has failed probes (http_admin_beta_toolforge_org_ip6) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/k8s-haproxy - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:40:00] (03merge) 10sadrettin: Son tarama zamanı en üstteki çubuğun solunda [toolforge-repos/kategori] - 10https://gitlab.wikimedia.org/toolforge-repos/kategori/-/merge_requests/14 (owner: 10project_5175_bot_1dc5cd2b4a7cadb6d1af8e29974ce1a7) [21:43:56] (03open) 10project_5175_bot_1dc5cd2b4a7cadb6d1af8e29974ce1a7: Soldaki Gezinti bölmesi kapatılıp açılabilir [toolforge-repos/kategori] - 10https://gitlab.wikimedia.org/toolforge-repos/kategori/-/merge_requests/15 [21:45:20] (03merge) 10sadrettin: Soldaki Gezinti bölmesi kapatılıp açılabilir [toolforge-repos/kategori] - 10https://gitlab.wikimedia.org/toolforge-repos/kategori/-/merge_requests/15 (owner: 10project_5175_bot_1dc5cd2b4a7cadb6d1af8e29974ce1a7) [21:48:41] RESOLVED: CloudVPSDesignateLeaks: Detected 36 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:49:42] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Replace deployment-sessionstore06 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436471#12519637 (10Southparkfan) 05In progress→03Resolved [21:58:57] 06tools-infrastructure-team, 10Beta-Cluster-Infrastructure: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482#12519708 (10Southparkfan) [22:57:07] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-etcd05 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436473#12519969 (10Southparkfan) 05Open→03In progress a:03Southparkfan [23:03:35] 10Cloud-VPS (Debian Bullseye Deprecation), 10Beta-Cluster-Infrastructure: Replace deployment-echostore02 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436479#12519986 (10Southparkfan) 05Open→03In progress a:03Southparkfan [23:08:55] !log tools.cluebotng-monitoring Deployment completed: https://github.com/cluebotng/component-configs/actions/runs/41930463906 (https://github.com/cluebotng/component-configs/commits/b495272b56a26a89b4498d24e762398799fe3f61) [23:08:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-monitoring/SAL [23:47:33] !log tools.cluebotng-review Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41935942779 (https://github.com/cluebotng/component-configs/commits/0acc26b8c3ae99ca82d89767fec31ad7b93e5262) [23:47:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng-review/SAL [23:47:38] !log tools.cluebotng Deployment failed: https://github.com/cluebotng/component-configs/actions/runs/41935942732 (https://github.com/cluebotng/component-configs/commits/0acc26b8c3ae99ca82d89767fec31ad7b93e5262) [23:47:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cluebotng/SAL