[06:28:37] !log tools restart haproxy on haproxy-8 [06:28:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:34:41] !status mostly stable T431946 ongoing [12:34:42] T431946: 2026-07-11 Incident - Toolforge haproxies flapping - https://phabricator.wikimedia.org/T431946 [13:59:14] !log gergesshamon@tools-bastion-15 tools.wikimonitor-beta [DEPLOY] Starting deployment | ref=fix/isolate-cookie-store-per-wiki-server [13:59:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikimonitor-beta/SAL [14:00:31] !log gergesshamon@tools-bastion-15 tools.wikimonitor-beta [DEPLOY] Build triggered successfully | ref=fix/isolate-cookie-store-per-wiki-server [14:00:31] !log gergesshamon@tools-bastion-15 tools.wikimonitor-beta [DEPLOY] Restarting buildservice | ref=fix/isolate-cookie-store-per-wiki-server [14:00:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikimonitor-beta/SAL [14:00:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikimonitor-beta/SAL [14:00:43] !log gergesshamon@tools-bastion-15 tools.wikimonitor-beta [DEPLOY] Deployment completed successfully | ref=fix/isolate-cookie-store-per-wiki-server [14:00:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikimonitor-beta/SAL [14:14:20] !log gergesshamon@tools-bastion-15 tools.wikimonitor-beta [DEPLOY] Starting deployment | ref=perf/broadcast-remove-completablefuture-overhead [14:14:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikimonitor-beta/SAL [14:15:36] !log gergesshamon@tools-bastion-15 tools.wikimonitor-beta [DEPLOY] Build triggered successfully | ref=perf/broadcast-remove-completablefuture-overhead [14:15:36] !log gergesshamon@tools-bastion-15 tools.wikimonitor-beta [DEPLOY] Restarting buildservice | ref=perf/broadcast-remove-completablefuture-overhead [14:15:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikimonitor-beta/SAL [14:15:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikimonitor-beta/SAL [14:15:43] !log gergesshamon@tools-bastion-15 tools.wikimonitor-beta [DEPLOY] Deployment completed successfully | ref=perf/broadcast-remove-completablefuture-overhead [14:15:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikimonitor-beta/SAL [14:58:26] !log gergesshamon@tools-bastion-15 tools.wikimonitor-beta [DEPLOY] Starting deployment | ref=perf/cache-compiled-regex-patterns [14:58:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikimonitor-beta/SAL [14:59:39] !log gergesshamon@tools-bastion-15 tools.wikimonitor-beta [DEPLOY] Build triggered successfully | ref=perf/cache-compiled-regex-patterns [14:59:40] !log gergesshamon@tools-bastion-15 tools.wikimonitor-beta [DEPLOY] Restarting buildservice | ref=perf/cache-compiled-regex-patterns [14:59:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikimonitor-beta/SAL [14:59:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikimonitor-beta/SAL [14:59:51] !log gergesshamon@tools-bastion-15 tools.wikimonitor-beta [DEPLOY] Deployment completed successfully | ref=perf/cache-compiled-regex-patterns [14:59:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikimonitor-beta/SAL [15:13:25] !log gergesshamon@tools-bastion-15 tools.wikimonitor-beta [DEPLOY] Starting deployment | ref=v1.6.0-beta [15:13:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikimonitor-beta/SAL [15:14:26] !log gergesshamon@tools-bastion-15 tools.wikimonitor-beta [DEPLOY] Build triggered successfully | ref=v1.6.0-beta [15:14:26] !log gergesshamon@tools-bastion-15 tools.wikimonitor-beta [DEPLOY] Restarting buildservice | ref=v1.6.0-beta [15:14:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikimonitor-beta/SAL [15:14:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikimonitor-beta/SAL [15:14:36] !log gergesshamon@tools-bastion-15 tools.wikimonitor-beta [DEPLOY] Deployment completed successfully | ref=v1.6.0-beta [15:14:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikimonitor-beta/SAL [21:01:39] !log tools restart haproxy on haproxy-8 [21:01:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [21:53:10] !log tools.wikibugs Updated channels.yaml to: cba60b5284e61a691df2e9cf38bb64b7ec54061bfec27c61d405101f80d1084c [21:53:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL [22:01:13] Hm.. This !log command suggest deployment worked but the gitlab ci job failed [22:01:37] curl … https://wikibugs.toolforge.org/api/config/refresh [22:01:37] curl: (22) The requested URL returned error: 502 Bad Gateway [22:01:37] ERROR: Job failed: command terminated with exit code 1 [22:01:43] https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/jobs/895209 [22:01:53] Did I retry already and failed again [22:01:56] Maybe that's normal