[01:26:33] 10Beta-Cluster-Infrastructure: Maps don't display on the beta cluster - https://phabricator.wikimedia.org/T420299 (10matmarex) 03NEW [01:28:49] 10Beta-Cluster-Infrastructure: Maps don't display on the beta cluster - https://phabricator.wikimedia.org/T420299#11716762 (10matmarex) This seems to be because it's trying to display map tiles from , which is down. {F72917575} @awight I'm afraid that you were the last person to... [01:33:05] 10Beta-Cluster-Infrastructure: Maps don't display on the beta cluster - https://phabricator.wikimedia.org/T420299#11716776 (10matmarex) [01:40:32] FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [01:45:32] RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [02:13:32] FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [02:18:32] RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [02:42:32] FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [02:47:32] RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [03:18:32] FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [03:23:32] RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [03:42:32] FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [03:52:32] RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [04:42:32] FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [04:52:32] RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [05:43:32] FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [05:48:32] RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [06:39:50] (03open) 10jhuneidi: Add catalyst-ci-client to trusted runners [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/153 (https://phabricator.wikimedia.org/T419092) [06:40:23] (03merge) 10jhuneidi: Add catalyst-ci-client to trusted runners [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/153 (https://phabricator.wikimedia.org/T419092) [06:42:32] FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [06:47:32] RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [07:01:09] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 06Traffic, 13Patch-For-Review: ATS: align ATS and Gerrit Apache timeouts to reenable connection re-use - https://phabricator.wikimedia.org/T417998#11717078 (10ABran-WMF) it's merged! I'll resolve {T246763}. Thanks for pointing out that c... [07:02:17] 10Gerrit, 06Release-Engineering-Team, 14Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 07ci-test-error (WMF-deployed Build Failure), 13Patch-For-Review: Jenkins job failing intermittently due to Gerrit HTTP 502 errors when interacting with repo... - https://phabricator.wikimedia.org/T246763#11717081 [07:09:23] (03PS1) 10Robert Vogel: Add new inter-extension dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1254000 [07:10:55] (03CR) 10CI reject: [V:04-1] Add new inter-extension dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1254000 (owner: 10Robert Vogel) [07:11:32] FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [07:12:53] 10Gerrit, 06Release-Engineering-Team, 14Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 07ci-test-error (WMF-deployed Build Failure), 13Patch-For-Review: Jenkins job failing intermittently due to Gerrit HTTP 502 errors when interacting with repo... - https://phabricator.wikimedia.org/T246763#11717085 [07:21:32] RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [07:21:40] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 06Traffic: Gerrit: Debug connection re-use on Gerrit's httpd causing Gerrit interface to be very slow - https://phabricator.wikimedia.org/T420189#11717097 (10ABran-WMF) I've merged patch with the Jetty timeout alignment made by @hashar in {T2... [07:23:35] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 06Traffic: Gerrit: Debug connection re-use on Gerrit's httpd causing Gerrit interface to be very slow - https://phabricator.wikimedia.org/T420189#11717100 (10ABran-WMF) [07:23:56] 10Gerrit, 06Release-Engineering-Team, 14Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 07ci-test-error (WMF-deployed Build Failure), 13Patch-For-Review: Jenkins job failing intermittently due to Gerrit HTTP 502 errors when interacting with repo... - https://phabricator.wikimedia.org/T246763#11717102 [07:33:04] Project beta-code-update-eqiad build #592000: 04FAILURE in 4.2 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/592000/ [07:42:32] FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [07:43:50] 10Gerrit, 06Release-Engineering-Team, 14Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 07ci-test-error (WMF-deployed Build Failure), 13Patch-For-Review: Jenkins job failing intermittently due to Gerrit HTTP 502 errors when interacting with rep... - https://phabricator.wikimedia.org/T246763#11717159 [07:45:22] Yippee, build fixed! [07:45:22] Project beta-code-update-eqiad build #592001: 09FIXED in 2 min 22 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/592001/ [07:47:32] RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [08:08:30] 10GitLab, 06Community-Tech, 10Wikimedia OCR: Migrate wikimedia/wikimedia-ocr from GitHub to GitLab - https://phabricator.wikimedia.org/T420317#11717240 (10Samwilson) [08:08:57] 10GitLab (Project Migration), 06Community-Tech, 10Wikimedia OCR: Migrate wikimedia/wikimedia-ocr from GitHub to GitLab - https://phabricator.wikimedia.org/T420317#11717242 (10Samwilson) [08:12:48] 10GitLab (Project Migration), 06Community-Tech, 10Wikimedia OCR: Migrate wikimedia/wikimedia-ocr from GitHub to GitLab - https://phabricator.wikimedia.org/T420317#11717272 (10Samwilson) Actually I'd forgotten about T387062, so I think creating https://gitlab.wikimedia.org/toolforge-repos/ocr is probably the... [08:14:25] 10GitLab (Project Migration), 06Community-Tech, 10Wikimedia OCR: Migrate wikimedia/wikimedia-ocr from GitHub to GitLab - https://phabricator.wikimedia.org/T420317#11717275 (10Samwilson) [08:37:39] Yippee, build fixed! [08:37:39] Project beta-scap-sync-world build #249567: 09FIXED in 2 min 20 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/249567/ [08:43:12] Yippee, build fixed! [08:43:13] Project mediawiki-core-doxygen build #18654: 09FIXED in 12 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/18654/ [08:56:32] FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [09:01:32] RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [09:18:19] 10Gerrit, 06collaboration-services, 13Patch-For-Review: gerrit: create a reboot gerrit cookbook - https://phabricator.wikimedia.org/T420194#11717446 (10ABran-WMF) 05Open→03In progress a:03ABran-WMF [09:18:32] FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [09:19:03] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T413811#11717452 (10A_smart_kitten) [09:21:32] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T413811#11717474 (10A_smart_kitten) (See T383948#11717445 & the comment before it for the reason for me adding that task as a train-blocker) [09:23:32] RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [09:31:10] (03CR) 10Hashar: [C:04-1] "That `locale.Error: unsupported locale setting` is suspicious, there should always be some locale available and I guess the reason is the " [integration/quibble] - 10https://gerrit.wikimedia.org/r/1250501 (https://phabricator.wikimedia.org/T418234) (owner: 10Zfilipin) [09:43:11] 10Beta-Cluster-Infrastructure, 10Cassandra, 06Data-Persistence: Cassandra killed by oom-killer and prometheus scrapes failing intermittently on deployment-sessionstore06 - https://phabricator.wikimedia.org/T415021#11717520 (10hashar) ` $ facter -d -l trace --custom-dir /var/lib/puppet/lib/facter ` Tha... [09:43:32] FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [09:44:40] 10GitLab (Account Approval), 06Release-Engineering-Team (Doing 😎): Requesting GitLab account activation for Kengkong1 - https://phabricator.wikimedia.org/T419773#11717521 (10Aklapper) 05Open→03Resolved a:03Aklapper Approved. Happy hacking! [09:45:06] (03CR) 10Hashar: [C:03+2] "Lets fly, thank you for the reviews!" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1250584 (https://phabricator.wikimedia.org/T419683) (owner: 10Hashar) [09:46:16] (03CR) 10Hashar: [C:03+2] Add Python 3.12 and 3.13 testing [integration/quibble] - 10https://gerrit.wikimedia.org/r/1250540 (https://phabricator.wikimedia.org/T419675) (owner: 10Hashar) [09:52:42] 10Phabricator, 06Release-Engineering-Team (Priority Backlog 📥): Update to Phorge/Arcanist upstream (2026.xx) - https://phabricator.wikimedia.org/T410849#11717552 (10Aklapper) > Temporary comment: Should pull Phorge's 9ae0020a857e, the commit after seems to be buggy Fixed now so my previous comment is obsolete... [09:53:32] RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [09:55:55] (03CR) 10Hashar: Add new inter-extension dependency (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/1254000 (owner: 10Robert Vogel) [09:57:09] Yippee, build fixed! [09:57:09] Project EntitySchema-phpmetrics build #162: 09FIXED in 8.8 sec: https://integration.wikimedia.org/ci/job/EntitySchema-phpmetrics/162/ [10:00:28] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 06Traffic, 13Patch-For-Review: Gerrit: Debug connection re-use on Gerrit's httpd causing Gerrit interface to be very slow - https://phabricator.wikimedia.org/T420189#11717566 (10ABran-WMF) After merging [[ https://gerrit.wikimedia.org/r/c/op... [10:02:20] (03Merged) 10jenkins-bot: Split BrowserTests duration reports [integration/quibble] - 10https://gerrit.wikimedia.org/r/1250584 (https://phabricator.wikimedia.org/T419683) (owner: 10Hashar) [10:02:22] (03CR) 10CI reject: [V:04-1] Add Python 3.12 and 3.13 testing [integration/quibble] - 10https://gerrit.wikimedia.org/r/1250540 (https://phabricator.wikimedia.org/T419675) (owner: 10Hashar) [10:06:39] 10Phabricator, 06Infrastructure-Foundations, 10SRE-tools: offboard-user: Migrate Phabricator API access from user.query() to user.search() - https://phabricator.wikimedia.org/T420324#11717595 (10Aklapper) FYI pretty similar tasks: https://phabricator.wikimedia.org/maniphest/query/lV7c54v0tL3z/#R [10:09:40] (03CR) 10Robert Vogel: Add new inter-extension dependency (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/1254000 (owner: 10Robert Vogel) [10:10:50] (03PS2) 10Robert Vogel: Add new inter-extension dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1254000 [10:12:09] (03CR) 10Robert Vogel: Add new inter-extension dependency (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1254000 (owner: 10Robert Vogel) [10:13:13] 06Project-Admins, 07EngProd-Virtual-Hackathon: Consider archiving #engprod-virtual-hackathon - https://phabricator.wikimedia.org/T420326 (10A_smart_kitten) 03NEW [10:27:59] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T413811#11717691 (10Ladsgroup) >>! In T413811#11717474, @A_smart_kitten wrote: > (See T383948#11717445 & the comment before it for the rea... [10:40:32] FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [10:50:32] RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [11:04:59] 06Project-Admins, 06Release-Engineering-Team (Doing 😎): Consider archiving #engprod-virtual-hackathon - https://phabricator.wikimedia.org/T420326#11717848 (10Aklapper) 05Open→03Resolved a:03Aklapper I agree. Thanks for finding that one! [11:05:23] 06Release-Engineering-Team (Seen), 10Release Pipeline (Blubber): Allow new blubber builders to be implemented in yaml - https://phabricator.wikimedia.org/T201875#11717855 (10Aklapper) [11:05:24] 10Continuous-Integration-Infrastructure, 10Quibble: Feature request: Evaluate "require" field from "extension.json" in automated test environment - https://phabricator.wikimedia.org/T185736#11717857 (10Aklapper) [11:10:32] FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [11:15:32] RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [11:26:05] (03CR) 10Hashar: Add new inter-extension dependency (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1254000 (owner: 10Robert Vogel) [11:35:03] maintenance-disconnect-full-disks build 789930 integration-agent-docker-1061 (/: 25%, /srv: 98%, /var/lib/docker: 31%): OFFLINE due to disk space [11:40:03] maintenance-disconnect-full-disks build 789931 integration-agent-docker-1061 (/: 25%, /srv: 48%, /var/lib/docker: 30%): RECOVERY disk space OK [12:10:32] FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [12:15:28] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T413811#11718123 (10Aklapper) [12:18:38] 10GitLab (Project Migration), 06Community-Tech, 10Wikimedia OCR: Migrate wikimedia/wikimedia-ocr from GitHub to GitLab - https://phabricator.wikimedia.org/T420317#11718151 (10A_smart_kitten) [12:20:32] RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [12:24:15] 10Continuous-Integration-Config, 10MediaWiki-extensions-WikimediaEvents, 06Test Kitchen: Make WikimediaEvents depend on TestKitchen - https://phabricator.wikimedia.org/T419679#11718191 (10phuedx) I met with @hashar about this and this is what we'd have to do to make this work. As I understand it, we'd have t... [12:41:32] FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [12:46:32] RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [13:02:04] (03PS1) 10Sbisson: ArticleGuidance: register SpamBlacklist as phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1254194 [13:02:33] (03PS2) 10Sbisson: ArticleGuidance: register SpamBlacklist as phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1254194 [13:06:16] 10Continuous-Integration-Infrastructure, 07Jenkins: Wikibase job in Jenkins do not include the full log - https://phabricator.wikimedia.org/T420347 (10Peter) 03NEW [13:32:33] 10Continuous-Integration-Infrastructure, 07Jenkins: Wikibase job in Jenkins do not include the full log - https://phabricator.wikimedia.org/T420347#11718537 (10Peter) Also as @hashar pointed out, the full log is available at https://integration.wikimedia.org/ci/job/quibble-with-Wikibase-extensions-browser-test... [14:10:30] (03CR) 10Jforrester: "00:03:21.033 py313-unit: FAIL code 1 (13.41=setup[11.24]+cmd[2.17] seconds)" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1250540 (https://phabricator.wikimedia.org/T419675) (owner: 10Hashar) [14:10:32] FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [14:11:37] (03PS3) 10Hashar: Add Python 3.12 and 3.13 testing [integration/quibble] - 10https://gerrit.wikimedia.org/r/1250540 (https://phabricator.wikimedia.org/T419675) [14:15:32] RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [14:22:59] 10Gerrit, 06collaboration-services, 06Infrastructure-Foundations, 07Puppet: Edit puppet-merge to use gerrit.discovery.wmnet instead of gerrit.wikimedia.org? - https://phabricator.wikimedia.org/T420184#11718810 (10ABran-WMF) [14:23:29] 10Gerrit, 06collaboration-services, 06Infrastructure-Foundations, 07Puppet: Edit puppet-merge to use gerrit.discovery.wmnet instead of gerrit.wikimedia.org? - https://phabricator.wikimedia.org/T420184#11718811 (10ABran-WMF) p:05Triage→03Low [14:23:59] 10Phabricator: Have Phabricator link Gerrit change-id to Gerrit search query - https://phabricator.wikimedia.org/T420363 (10hashar) 03NEW [14:24:45] 10Phabricator: Have Phabricator link Gerrit change-id to Gerrit search query - https://phabricator.wikimedia.org/T420363#11718823 (10hashar) It might just be wrong, I had that idea a second ago as I copy pasted a Change-Id into a Phabricator comment and the preview below the comment did not link it. Maybe it is... [14:24:52] 10Gerrit, 06collaboration-services, 06Infrastructure-Foundations, 07Puppet: Change puppet-merge git origin to use gerrit.discovery.wmnet instead of gerrit.wikimedia.org - https://phabricator.wikimedia.org/T420184#11718825 (10ABran-WMF) [14:25:33] (03open) 10viktoriahillerudwmse: Add Piper to trusted runners [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/154 [14:26:50] brennen: hey, https://www.mediawiki.org/wiki/Talk:Phabricator/Help#Account_Disabled_-_THORzero9 seems to be your doing? [14:28:51] (03PS3) 10Jforrester: Zuul: [mediawiki/extensions/ArticleGuidance]: Add SpamBlacklist as phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1254194 (https://phabricator.wikimedia.org/T420015) (owner: 10Sbisson) [14:29:41] 10Continuous-Integration-Infrastructure, 07Jenkins: Wikibase job in Jenkins do not include the full log - https://phabricator.wikimedia.org/T420347#11718856 (10hashar) I am pretty sure it is due to https://gerrit.wikimedia.org/r/c/integration/quibble/+/1235103 which nests `>>> Start:` / `>>> Finish:` when Quib... [14:30:49] (03CR) 10CI reject: [V:04-1] Zuul: [mediawiki/extensions/ArticleGuidance]: Add SpamBlacklist as phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1254194 (https://phabricator.wikimedia.org/T420015) (owner: 10Sbisson) [14:31:34] (03PS4) 10Jforrester: Zuul: [mediawiki/extensions/ArticleGuidance]: Add SpamBlacklist as phan dep [integration/config] - 10https://gerrit.wikimedia.org/r/1254194 (https://phabricator.wikimedia.org/T420015) (owner: 10Sbisson) [14:31:37] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/ArticleGuidance]: Add SpamBlacklist as phan dep [integration/config] - 10https://gerrit.wikimedia.org/r/1254194 (https://phabricator.wikimedia.org/T420015) (owner: 10Sbisson) [14:33:21] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/ArticleGuidance]: Add SpamBlacklist as phan dep [integration/config] - 10https://gerrit.wikimedia.org/r/1254194 (https://phabricator.wikimedia.org/T420015) (owner: 10Sbisson) [14:36:50] !log Zuul: [mediawiki/extensions/ArticleGuidance]: Add SpamBlacklist as phan dep, for T420015 [14:36:52] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:36:53] T420015: Source validation - https://phabricator.wikimedia.org/T420015 [14:37:39] 10Continuous-Integration-Infrastructure, 07Jenkins: Wikibase job in Jenkins do not include the full log - https://phabricator.wikimedia.org/T420347#11718908 (10hashar) The Jenkins plugin configuration is done via https://integration.wikimedia.org/ci/configure under {nav Collapsing Console Sections}: {F7296349... [14:38:35] (03CR) 10Hashar: [C:03+2] "This most probably has caused *T420347 - Wikibase job in Jenkins do not include the full log*." [integration/quibble] - 10https://gerrit.wikimedia.org/r/1235103 (owner: 10Hashar) [14:40:15] Phabricator needs a short restart at 15:00 UTC (in 20 minutes) [14:43:32] FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [14:49:12] 10Phabricator (phabricator-next), 06Release-Engineering-Team, 06collaboration-services: Deploy Phab/Phorge 2026-03-17 - https://phabricator.wikimedia.org/T420366 (10brennen) 03NEW [14:49:46] (03open) 10brennen: update submodules for 2026-03-17 deploy [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/99 (https://phabricator.wikimedia.org/T420366) [14:53:32] RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [14:54:46] 10Phabricator, 06collaboration-services, 06DC-Ops, 10ops-codfw: phab2002: SEL System Event:, System Board Front LED Panel, Critical, management controller unavailable - https://phabricator.wikimedia.org/T420228#11719008 (10Jhancock.wm) soft rebooted the idrac [15:01:42] (03merge) 10brennen: update submodules for 2026-03-17 deploy [repos/phabricator/deployment] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/99 (https://phabricator.wikimedia.org/T420366) [15:04:00] 10GitLab (Integrations), 10Phabricator (phabricator-next), 06Release-Engineering-Team, 06collaboration-services: Update expired gitlab_api_key for Phorge GitLab related changes widget - https://phabricator.wikimedia.org/T418935#11719076 (10Aklapper) [15:04:29] Phabricator maintenance finished [15:05:33] 10Phabricator (phabricator-next), 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Deploy Phab/Phorge 2026-03-17 - https://phabricator.wikimedia.org/T420366#11719087 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=94285b4c-4e9a-4c2a-aa8a-070b49254cd1) set by jel... [15:06:00] 10Phabricator (phabricator-next), 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Deploy Phab/Phorge 2026-03-17 - https://phabricator.wikimedia.org/T420366#11719092 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=acded186-68d8-426d-8eac-7a88925196f5) set by jel... [15:12:48] (03merge) 10dancy: Add Piper to trusted runners [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/154 (owner: 10viktoriahillerudwmse) [15:18:43] 10Phabricator, 06collaboration-services, 06DC-Ops, 10ops-codfw: phab2002: SEL System Event:, System Board Front LED Panel, Critical, management controller unavailable - https://phabricator.wikimedia.org/T420228#11719169 (10Aklapper) I see an entry `ipmi_sdr_cache_open: internal IPMI error` for `phab2002` a... [15:23:46] 10Phabricator (phabricator-next), 06Release-Engineering-Team (Doing 😎), 07CSS, 07dark-mode: Bot comment styles are not dark-mode compatible - https://phabricator.wikimedia.org/T414117#11719203 (10Aklapper) 05Stalled→03Resolved This just got deployed. [15:27:10] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work, 13Patch-For-Review: Put a limit on demos created by ci - https://phabricator.wikimedia.org/T417304#11719217 (10jnuche) 05Open→03Resolved CI pipelines from Abstract Wikipedia now have a limit of 15 non-deleted... [15:27:26] 10Phabricator (2026-03-17), 06Release-Engineering-Team (Doing 😎), 06collaboration-services: Deploy Phab/Phorge 2026-03-17 - https://phabricator.wikimedia.org/T420366#11719222 (10Aklapper) 05Open→03Resolved a:03brennen This deployment included / fixed / touched / put fairy dust on: * Dark Mode suppo... [15:29:14] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: Disaster recovery for k8s upgrade - https://phabricator.wikimedia.org/T419580#11719239 (10jnuche) a:03jnuche [15:29:45] 10Gerrit, 06collaboration-services, 13Patch-For-Review: gerrit: create a reboot gerrit cookbook - https://phabricator.wikimedia.org/T420194#11719246 (10ABran-WMF) 05In progress→03Resolved [15:45:32] FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [15:55:32] RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [16:03:18] (03open) 10sebastian-berlin-wmse: Add Matcha to trusted runners [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/155 [16:09:05] 10Gerrit, 06Release-Engineering-Team, 14Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 07ci-test-error (WMF-deployed Build Failure), 13Patch-For-Review: Jenkins job failing intermittently due to Gerrit HTTP 502 errors when interacting with rep... - https://phabricator.wikimedia.org/T246763#11719533 [16:28:58] (03CR) 10Jforrester: [C:03+2] Add Python 3.12 and 3.13 testing [integration/quibble] - 10https://gerrit.wikimedia.org/r/1250540 (https://phabricator.wikimedia.org/T419675) (owner: 10Hashar) [16:31:52] (03CR) 10CI reject: [V:04-1] Add Python 3.12 and 3.13 testing [integration/quibble] - 10https://gerrit.wikimedia.org/r/1250540 (https://phabricator.wikimedia.org/T419675) (owner: 10Hashar) [16:36:14] 10Phabricator, 06collaboration-services, 10VPS-project-Phabricator: Phabricator test project requires email verification but can't send email - https://phabricator.wikimedia.org/T388022#11719821 (10A_smart_kitten) >>! In T388022#11715419, @Dzahn wrote: > @Aklapper The last question above was about a configur... [16:37:56] (03PS3) 10Jforrester: Zuul: [BlueSpicePermissionManager] Add …ConfigManager & …UserManager deps [integration/config] - 10https://gerrit.wikimedia.org/r/1254000 (owner: 10Robert Vogel) [16:39:02] (03CR) 10Jforrester: [C:03+2] Zuul: [BlueSpicePermissionManager] Add …ConfigManager & …UserManager deps [integration/config] - 10https://gerrit.wikimedia.org/r/1254000 (owner: 10Robert Vogel) [16:39:09] 10Continuous-Integration-Config, 10MediaWiki-extensions-WikimediaEvents, 06Test Kitchen: Make WikimediaEvents depend on TestKitchen - https://phabricator.wikimedia.org/T419679#11719850 (10JVanderhoop-WMF) p:05Triage→03High [16:41:56] (03Merged) 10jenkins-bot: Zuul: [BlueSpicePermissionManager] Add …ConfigManager & …UserManager deps [integration/config] - 10https://gerrit.wikimedia.org/r/1254000 (owner: 10Robert Vogel) [16:43:10] !log Zuul: [BlueSpicePermissionManager] Add …ConfigManager & …UserManager deps [16:43:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:47:32] FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [16:49:20] (03CR) 10Jforrester: [C:03+2] build: Updating flatted to 3.4.1 [performance/fresnel] - 10https://gerrit.wikimedia.org/r/1252409 (owner: 10Libraryupgrader) [16:56:16] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: Disaster recovery for k8s upgrade - https://phabricator.wikimedia.org/T419580#11720024 (10jnuche) Disk quotas for both `catalyst` and `catalyst-dev` will need to be raised by 760GB from the current 1200GB to a **total o... [16:57:07] Project mediawiki-core-doxygen build #18662: 04FAILURE in 12 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/18662/ [16:58:11] 10Phabricator, 06collaboration-services, 06DC-Ops, 10ops-codfw: phab2002: SEL System Event:, System Board Front LED Panel, Critical, management controller unavailable - https://phabricator.wikimedia.org/T420228#11720040 (10Jhancock.wm) yes. that matches the time. this error can be from a firmware issue. [17:00:31] (03merge) 10dancy: Add Matcha to trusted runners [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/155 (owner: 10sebastian-berlin-wmse) [17:02:32] RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [17:16:32] FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [17:21:32] RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [17:30:19] Yippee, build fixed! [17:30:19] Project mediawiki-core-doxygen build #18663: 09FIXED in 12 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/18663/ [17:35:36] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team (Priority Backlog 📥), 07ci-test-error, and 2 others: Various CI jobs failing after "mkdir: cannot create directory ‘log’: Permission denied" - https://phabricator.wikimedia.org/T282893#11720282 (10matmarex) I feel like it's been... [18:15:32] FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [18:20:32] RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [18:24:58] (03PS1) 10Ebernhardson: Zuul: search: Update opensearch plugins for java 11/17 [integration/config] - 10https://gerrit.wikimedia.org/r/1254282 [18:26:30] (03CR) 10CI reject: [V:04-1] Zuul: search: Update opensearch plugins for java 11/17 [integration/config] - 10https://gerrit.wikimedia.org/r/1254282 (owner: 10Ebernhardson) [18:27:18] (03PS2) 10Ebernhardson: Zuul: search: Update opensearch plugins for java 11/17 [integration/config] - 10https://gerrit.wikimedia.org/r/1254282 (https://phabricator.wikimedia.org/T420407) [18:28:45] (03CR) 10CI reject: [V:04-1] Zuul: search: Update opensearch plugins for java 11/17 [integration/config] - 10https://gerrit.wikimedia.org/r/1254282 (https://phabricator.wikimedia.org/T420407) (owner: 10Ebernhardson) [18:33:46] (03PS3) 10Ebernhardson: Zuul: search: Update opensearch plugins for java 11/17 [integration/config] - 10https://gerrit.wikimedia.org/r/1254282 (https://phabricator.wikimedia.org/T420407) [18:45:52] (03PS1) 10Sbisson: ArticleGuidance: register dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/1254289 [19:09:51] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: setup 2 contint machines for jenkins - https://phabricator.wikimedia.org/T418521#11720597 (10Dzahn) name to talk to the new jenkins: ` [dns1004:~] $ host jenkins.discovery.wmnet jenkins.disco... [19:13:32] FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [19:18:32] RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [19:31:00] 10Continuous-Integration-Config, 06Growth-Team, 10MediaWiki-extensions-CentralAuth, 06MediaWiki-Platform-Team: Add CentralAuth to gated extensions - https://phabricator.wikimedia.org/T333541#11720625 (10Michael) Growth Team would like to see something like this happen, because GrowthExperiments has substan... [19:33:24] 10GitLab (Account Approval), 06Release-Engineering-Team (Doing 😎): Requesting GitLab account activation for Kengkong1 - https://phabricator.wikimedia.org/T419773#11720637 (10Kengkong1) sweet! Thanks! [19:55:32] FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [19:59:30] 06Release-Engineering-Team (Radar), 10Scap, 10MW-on-K8s, 06ServiceOps new, 10ServiceOps-SharedInfra: Pushing mediawiki-multiversion Docker image from deploy server takes 4 minutes - https://phabricator.wikimedia.org/T341441#11720726 (10RLazarus) p:05Triage→03Medium [20:00:32] RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [20:02:32] FIRING: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [20:03:40] huh: live-migration error for ^ [20:09:21] hrm, but I can ssh in it's just...unhappy [20:11:41] thcipriani: deployment-sessionstore06 "dies" every time puppet runs. There is some (java?) thing firing during factor runs that eats all the ram on the box. [20:12:32] RESOLVED: InstanceDown: Project deployment-prep instance deployment-sessionstore06 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [20:12:32] or.. maybe not per h.ashar's last post on T415021. But it it know to be very sick right now. [20:12:32] T415021: Cassandra killed by oom-killer and prometheus scrapes failing intermittently on deployment-sessionstore06 - https://phabricator.wikimedia.org/T415021 [20:20:03] !log Resize deployment-sessionstore06 from g4.cores1.ram2.disk20 to g4.cores2.ram4.disk20 (T415021) [20:20:05] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:20:06] T415021: Cassandra killed by oom-killer and prometheus scrapes failing intermittently on deployment-sessionstore06 - https://phabricator.wikimedia.org/T415021 [20:20:23] (03open) 10jforrester: Draft: Don't move phpcbf after minus-x in fix, it breaks for macOS devs [repos/ci-tools/libup] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/111 (https://phabricator.wikimedia.org/T214652) [20:21:11] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-sessionstore06 is down - https://phabricator.wikimedia.org/T420227#11720781 (10A_smart_kitten) >>! In T420227#11716276, @bd808 wrote: > Boo. Leaving it open did not stop {T420284} from being filed. My first guess would be that maybe... [20:24:23] 10Beta-Cluster-Infrastructure, 10Cassandra, 06Data-Persistence: Cassandra killed by oom-killer and prometheus scrapes failing intermittently on deployment-sessionstore06 - https://phabricator.wikimedia.org/T415021#11720786 (10bd808) Before: `lang=shell-session bd808@deployment-sessionstore06:~$ free -h... [20:30:19] 10Continuous-Integration-Config, 06Growth-Team, 10MediaWiki-extensions-CentralAuth, 06MediaWiki-Platform-Team: Add integration test suite for CentralAuth as a merge requirement - https://phabricator.wikimedia.org/T333541#11720823 (10bd808) [20:58:11] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-sessionstore06 is down - https://phabricator.wikimedia.org/T420287#11720900 (10bd808) [20:58:12] 10Beta-Cluster-Infrastructure, 10Cassandra, 06Data-Persistence: Cassandra killed by oom-killer and prometheus scrapes failing intermittently on deployment-sessionstore06 - https://phabricator.wikimedia.org/T415021#11720901 (10bd808) [20:59:16] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-sessionstore06 is down - https://phabricator.wikimedia.org/T420287#11720915 (10bd808) p:05Triage→03Low Side effect of OOM issues that should go away if we fix that problem [21:02:03] (03PS1) 10Fomafix: Zuul: [mediawiki/extensions/JsonForms] Add quibble jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1254329 [21:15:11] (03CR) 10Jforrester: "I'll split this into the two different commits." [integration/config] - 10https://gerrit.wikimedia.org/r/1254282 (https://phabricator.wikimedia.org/T420407) (owner: 10Ebernhardson) [21:20:42] (03PS4) 10Jforrester: Zuul: search: Update opensearch plugins for java 11/17 [integration/config] - 10https://gerrit.wikimedia.org/r/1254282 (https://phabricator.wikimedia.org/T420407) (owner: 10Ebernhardson) [21:20:42] (03PS1) 10Jforrester: jjb: Provide search maven jobs on Java 17 too [integration/config] - 10https://gerrit.wikimedia.org/r/1254334 (https://phabricator.wikimedia.org/T420407) [21:20:44] (03PS1) 10Jforrester: jjb: Drop search maven jobs on Java 8, now unused [integration/config] - 10https://gerrit.wikimedia.org/r/1254335 (https://phabricator.wikimedia.org/T420407) [21:21:28] (03CR) 10Jforrester: [C:03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/1254334 (https://phabricator.wikimedia.org/T420407) (owner: 10Jforrester) [21:22:21] (03CR) 10CI reject: [V:04-1] jjb: Drop search maven jobs on Java 8, now unused [integration/config] - 10https://gerrit.wikimedia.org/r/1254335 (https://phabricator.wikimedia.org/T420407) (owner: 10Jforrester) [21:22:59] (03Merged) 10jenkins-bot: jjb: Provide search maven jobs on Java 17 too [integration/config] - 10https://gerrit.wikimedia.org/r/1254334 (https://phabricator.wikimedia.org/T420407) (owner: 10Jforrester) [21:24:49] (03CR) 10Jforrester: [C:03+2] Zuul: search: Update opensearch plugins for java 11/17 [integration/config] - 10https://gerrit.wikimedia.org/r/1254282 (https://phabricator.wikimedia.org/T420407) (owner: 10Ebernhardson) [21:26:17] (03Merged) 10jenkins-bot: Zuul: search: Update opensearch plugins for java 11/17 [integration/config] - 10https://gerrit.wikimedia.org/r/1254282 (https://phabricator.wikimedia.org/T420407) (owner: 10Ebernhardson) [21:27:10] !log Zuul: search: Update opensearch plugins for Java 11/17, for T420407 [21:27:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:27:13] T420407: Migrate opensearch plugins to 2.19.5 - https://phabricator.wikimedia.org/T420407 [21:28:39] (03CR) 10Jforrester: [C:03+2] "Deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/1254282 (https://phabricator.wikimedia.org/T420407) (owner: 10Ebernhardson) [21:41:34] (03PS1) 10Jforrester: Zuul: [search/glent] Drop Java 8 testing, add Java 17 testing [integration/config] - 10https://gerrit.wikimedia.org/r/1254341 (https://phabricator.wikimedia.org/T420407) [21:41:36] (03PS1) 10Jforrester: jjb: Add support for running Java 25 for search projects [integration/config] - 10https://gerrit.wikimedia.org/r/1254342 [21:41:36] (03PS1) 10Jforrester: Zuul: [search/*] Add experimental Java 25 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1254343 [21:43:04] (03CR) 10Jforrester: "Waiting for Erik's confirmation before deploying." [integration/config] - 10https://gerrit.wikimedia.org/r/1254341 (https://phabricator.wikimedia.org/T420407) (owner: 10Jforrester) [22:00:54] (03CR) 10Jforrester: [C:04-1] "I've given advice on I5ee64af3f3573265c4424e772beb6b1c9aeb1105 as to why your patch is failing phan." [integration/config] - 10https://gerrit.wikimedia.org/r/1254289 (owner: 10Sbisson) [22:05:50] Declaring victory would be a horrible jinx to put on myself, but... doubling the ram on deployment-sessionstore06 seems to have made it much happier. The newer Cassandra version and java runtime look to have just pushed things about 512Mb past what the older flavor could support. [22:44:58] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/JsonForms] Add quibble jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1254329 (owner: 10Fomafix) [22:46:36] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/JsonForms] Add quibble jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1254329 (owner: 10Fomafix) [22:50:36] !log Zuul: [mediawiki/extensions/JsonForms] Add quibble jobs [22:50:37] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:18:49] belated thanks bd808 <3 [23:52:22] ^