[00:06:13] Project beta-scap-sync-world build #206144: 04STILL FAILING in 1 min 10 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/206144/ [00:16:18] Project beta-scap-sync-world build #206145: 04STILL FAILING in 1 min 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/206145/ [00:20:02] Project beta-update-databases-eqiad build #84652: 04STILL FAILING in 2.3 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/84652/ [00:26:19] Project beta-scap-sync-world build #206146: 04STILL FAILING in 1 min 10 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/206146/ [00:36:10] Project beta-scap-sync-world build #206147: 04STILL FAILING in 1 min 5 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/206147/ [00:46:16] Project beta-scap-sync-world build #206148: 04STILL FAILING in 1 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/206148/ [00:56:21] Project beta-scap-sync-world build #206149: 04STILL FAILING in 1 min 10 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/206149/ [01:06:31] Project beta-scap-sync-world build #206150: 04STILL FAILING in 1 min 26 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/206150/ [01:16:22] Project beta-scap-sync-world build #206151: 04STILL FAILING in 1 min 9 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/206151/ [01:20:02] Project beta-update-databases-eqiad build #84653: 04STILL FAILING in 2.5 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/84653/ [01:26:14] Project beta-scap-sync-world build #206152: 04STILL FAILING in 1 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/206152/ [01:36:08] Project beta-scap-sync-world build #206153: 04STILL FAILING in 1 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/206153/ [01:46:17] Project beta-scap-sync-world build #206154: 04STILL FAILING in 1 min 10 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/206154/ [01:56:16] Project beta-scap-sync-world build #206155: 04STILL FAILING in 1 min 9 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/206155/ [02:06:13] Project beta-scap-sync-world build #206156: 04STILL FAILING in 1 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/206156/ [02:16:21] Project beta-scap-sync-world build #206157: 04STILL FAILING in 1 min 13 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/206157/ [02:20:02] Project beta-update-databases-eqiad build #84654: 04STILL FAILING in 2.5 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/84654/ [02:26:20] Project beta-scap-sync-world build #206158: 04STILL FAILING in 1 min 13 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/206158/ [02:36:12] Project beta-scap-sync-world build #206159: 04STILL FAILING in 1 min 5 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/206159/ [02:46:13] Project beta-scap-sync-world build #206160: 04STILL FAILING in 1 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/206160/ [02:56:09] Project beta-scap-sync-world build #206161: 04STILL FAILING in 1 min 3 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/206161/ [03:06:05] Project beta-scap-sync-world build #206162: 04STILL FAILING in 58 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/206162/ [03:16:14] Project beta-scap-sync-world build #206163: 04STILL FAILING in 1 min 7 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/206163/ [03:20:02] Project beta-update-databases-eqiad build #84655: 04STILL FAILING in 2.5 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/84655/ [03:26:16] Project beta-scap-sync-world build #206164: 04STILL FAILING in 1 min 8 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/206164/ [03:36:10] Project beta-scap-sync-world build #206165: 04STILL FAILING in 1 min 3 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/206165/ [03:46:16] Project beta-scap-sync-world build #206166: 04STILL FAILING in 1 min 9 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/206166/ [03:56:16] Project beta-scap-sync-world build #206167: 04STILL FAILING in 1 min 9 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/206167/ [04:06:13] Project beta-scap-sync-world build #206168: 04STILL FAILING in 1 min 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/206168/ [04:16:20] Project beta-scap-sync-world build #206169: 04STILL FAILING in 1 min 13 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/206169/ [04:20:02] Project beta-update-databases-eqiad build #84656: 04STILL FAILING in 2.4 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/84656/ [04:26:19] Project beta-scap-sync-world build #206170: 04STILL FAILING in 1 min 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/206170/ [04:36:08] Project beta-scap-sync-world build #206171: 04STILL FAILING in 1 min 1 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/206171/ [04:46:11] Project beta-scap-sync-world build #206172: 04STILL FAILING in 1 min 4 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/206172/ [04:56:13] Project beta-scap-sync-world build #206173: 04STILL FAILING in 1 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/206173/ [05:06:07] Project beta-scap-sync-world build #206174: 04STILL FAILING in 1 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/206174/ [05:16:16] Project beta-scap-sync-world build #206175: 04STILL FAILING in 1 min 9 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/206175/ [05:20:02] Project beta-update-databases-eqiad build #84657: 04STILL FAILING in 2.3 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/84657/ [05:26:16] Project beta-scap-sync-world build #206176: 04STILL FAILING in 1 min 8 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/206176/ [05:36:12] Project beta-scap-sync-world build #206177: 04STILL FAILING in 1 min 9 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/206177/ [05:46:25] Project beta-scap-sync-world build #206178: 04STILL FAILING in 1 min 16 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/206178/ [05:56:22] Project beta-scap-sync-world build #206179: 04STILL FAILING in 1 min 13 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/206179/ [06:06:19] Project beta-scap-sync-world build #206180: 04STILL FAILING in 1 min 13 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/206180/ [06:16:16] Project beta-scap-sync-world build #206181: 04STILL FAILING in 1 min 9 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/206181/ [06:20:02] Project beta-update-databases-eqiad build #84658: 04STILL FAILING in 2.3 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/84658/ [06:26:13] Project beta-scap-sync-world build #206182: 04STILL FAILING in 1 min 5 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/206182/ [06:36:11] Project beta-scap-sync-world build #206183: 04STILL FAILING in 1 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/206183/ [06:46:16] Project beta-scap-sync-world build #206184: 04STILL FAILING in 1 min 8 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/206184/ [06:56:16] Project beta-scap-sync-world build #206185: 04STILL FAILING in 1 min 7 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/206185/ [07:06:15] Project beta-scap-sync-world build #206186: 04STILL FAILING in 1 min 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/206186/ [07:11:29] 10Gerrit, 06Language and Product Localization: Delete two branches from mediawiki/extensions/Translate Gerrit extension - https://phabricator.wikimedia.org/T392787#10810320 (10Nikerabbit) [07:15:53] 10Beta-Cluster-Infrastructure: EtcdConfig failed to fetch data: (curl error: 60) SSL peer certificate or SSH remote key was not OK - https://phabricator.wikimedia.org/T393855#10810325 (10hashar) `wmf-config/LabsServices.php` has `_etcd._tcp.svc.deployment-prep.eqiad1.wikimedia.cloud`. From https://horizon.wikime... [07:17:05] Project beta-scap-sync-world build #206187: 04STILL FAILING in 1 min 55 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/206187/ [07:20:02] Project beta-update-databases-eqiad build #84659: 04STILL FAILING in 2.7 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/84659/ [07:26:54] Project beta-scap-sync-world build #206188: 04STILL FAILING in 1 min 12 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/206188/ [07:36:12] Project beta-scap-sync-world build #206189: 04STILL FAILING in 1 min 4 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/206189/ [07:49:06] 10Beta-Cluster-Infrastructure, 06Infrastructure-Foundations, 10observability, 07Grafana: deployment-etcd05: Function lookup() did not find a value for the name 'prometheus::instances_defaults' - https://phabricator.wikimedia.org/T393866 (10hashar) 03NEW [07:49:35] Project beta-scap-sync-world build #206190: 04STILL FAILING in 2 min 33 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/206190/ [07:54:14] 10Beta-Cluster-Infrastructure: EtcdConfig failed to fetch data: (curl error: 60) SSL peer certificate or SSH remote key was not OK - https://phabricator.wikimedia.org/T393855#10810382 (10hashar) On `deployment-etcd05.deployment-prep.eqiad1.wikimedia.cloud` in journalctl: ` May 11 18:58:01 deployment-etcd05 etcd[... [07:54:58] 10Beta-Cluster-Infrastructure, 06Infrastructure-Foundations, 10observability, 07Grafana: deployment-etcd05: Function lookup() did not find a value for the name 'prometheus::instances_defaults' - https://phabricator.wikimedia.org/T393866#10810385 (10taavi) > hieradata/common/prometheus.yaml Loading that fil... [07:58:19] Project beta-scap-sync-world build #206191: 04STILL FAILING in 1 min 53 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/206191/ [07:58:41] !log Disabled https://integration.wikimedia.org/ci/job/beta-scap-sync-world/ due to a failure with Etcd/expired certificate # T393855 [07:58:43] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [07:58:43] T393855: EtcdConfig failed to fetch data: (curl error: 60) SSL peer certificate or SSH remote key was not OK - https://phabricator.wikimedia.org/T393855 [08:13:06] (03CR) 10Hashar: [C:03+2] "That looks fine, I just kept using `$(pwd)` out of a bad habit 😊" [integration/config] - 10https://gerrit.wikimedia.org/r/1143967 (https://phabricator.wikimedia.org/T393847) (owner: 10Bartosz Dziewoński) [08:15:08] !log Updated jobs for "Replace all uses of `$(pwd)` with `$PWD`" | https://gerrit.wikimedia.org/r/c/integration/config/+/1143967/ [08:15:09] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:15:14] (03Merged) 10jenkins-bot: Replace all uses of `$(pwd)` with `$PWD` [integration/config] - 10https://gerrit.wikimedia.org/r/1143967 (https://phabricator.wikimedia.org/T393847) (owner: 10Bartosz Dziewoński) [08:20:03] Project beta-update-databases-eqiad build #84660: 04STILL FAILING in 3.7 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/84660/ [08:28:05] !log Disabled https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/ due to a failure with Etcd/expired certificate # T393855 [08:28:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:28:07] T393855: EtcdConfig failed to fetch data: (curl error: 60) SSL peer certificate or SSH remote key was not OK - https://phabricator.wikimedia.org/T393855 [08:32:56] (03CR) 10Hashar: "I will split this change in smaller ones since that would require jobs to be updated, different images to be updated and in turn more jobs" [integration/config] - 10https://gerrit.wikimedia.org/r/1143969 (https://phabricator.wikimedia.org/T393847) (owner: 10Bartosz Dziewoński) [08:36:29] 10Release-Engineering-Team (Priority Backlog 📥), 10MW-on-K8s, 06serviceops, 06SRE: Automated validation of mediawiki-multiversion images - https://phabricator.wikimedia.org/T288629#10810495 (10JMeybohm) 05Open→03Resolved a:03JMeybohm All related changes have been reverted [08:37:50] (03PS2) 10Hashar: Silence `git (init|fetch|checkout|submodule)` in examples [integration/config] - 10https://gerrit.wikimedia.org/r/1143969 (https://phabricator.wikimedia.org/T393847) (owner: 10Bartosz Dziewoński) [08:37:50] (03PS1) 10Hashar: jjb: silence git [integration/config] - 10https://gerrit.wikimedia.org/r/1144453 (https://phabricator.wikimedia.org/T393847) [08:38:20] (03CR) 10Hashar: [C:03+2] Silence `git (init|fetch|checkout|submodule)` in examples [integration/config] - 10https://gerrit.wikimedia.org/r/1143969 (https://phabricator.wikimedia.org/T393847) (owner: 10Bartosz Dziewoński) [08:39:12] (03CR) 10Hashar: [C:03+2] "Jobs updated:" [integration/config] - 10https://gerrit.wikimedia.org/r/1144453 (https://phabricator.wikimedia.org/T393847) (owner: 10Hashar) [08:39:30] 10Gerrit, 06collaboration-services: Gerrit is unresponsive (2025-05-06) - https://phabricator.wikimedia.org/T393498#10810507 (10Jelto) 05Open→03Resolved a:03Jelto Gerrits availability has improved over the last days (see T392467#10806961). I'll resolve the task. Future work is being tracked in the op... [08:39:42] (03Merged) 10jenkins-bot: Silence `git (init|fetch|checkout|submodule)` in examples [integration/config] - 10https://gerrit.wikimedia.org/r/1143969 (https://phabricator.wikimedia.org/T393847) (owner: 10Bartosz Dziewoński) [08:41:17] (03Merged) 10jenkins-bot: jjb: silence git [integration/config] - 10https://gerrit.wikimedia.org/r/1144453 (https://phabricator.wikimedia.org/T393847) (owner: 10Hashar) [08:52:22] (03PS1) 10Hashar: dockerfiles: silence git [integration/config] - 10https://gerrit.wikimedia.org/r/1144475 (https://phabricator.wikimedia.org/T393847) [08:52:41] (03CR) 10Hashar: [C:03+2] dockerfiles: silence git [integration/config] - 10https://gerrit.wikimedia.org/r/1144475 (https://phabricator.wikimedia.org/T393847) (owner: 10Hashar) [08:59:34] (03Merged) 10jenkins-bot: dockerfiles: silence git [integration/config] - 10https://gerrit.wikimedia.org/r/1144475 (https://phabricator.wikimedia.org/T393847) (owner: 10Hashar) [09:14:20] 06Release-Engineering-Team, 06collaboration-services, 06Infrastructure-Foundations, 06SRE, 10vm-requests: codfw: 1VM request for zuul3+ - https://phabricator.wikimedia.org/T393873#10810632 (10Jelto) [09:15:05] 06Release-Engineering-Team, 06collaboration-services, 06Infrastructure-Foundations, 06SRE, 10vm-requests: codfw: 1VM request for zuul3+ - https://phabricator.wikimedia.org/T393873#10810634 (10Jelto) [09:22:02] 10Beta-Cluster-Infrastructure, 06Infrastructure-Foundations, 10observability, 07Grafana: deployment-etcd05: Function lookup() did not find a value for the name 'prometheus::instances_defaults' - https://phabricator.wikimedia.org/T393866#10810643 (10fgiunchedi) I'm ok to move `prometheus::instances_default`... [09:30:51] (03CR) 10Hashar: [C:03+2] "I have split your original patch to smaller changes :)" [integration/config] - 10https://gerrit.wikimedia.org/r/1143969 (https://phabricator.wikimedia.org/T393847) (owner: 10Bartosz Dziewoński) [09:33:09] (03PS3) 10Hashar: dockerfiles: silence git in ci-src-setup [integration/config] - 10https://gerrit.wikimedia.org/r/1144483 (https://phabricator.wikimedia.org/T393847) [09:45:18] (03PS4) 10Hashar: dockerfiles: silence git in ci-src-setup [integration/config] - 10https://gerrit.wikimedia.org/r/1144483 (https://phabricator.wikimedia.org/T393847) [09:47:31] (03PS5) 10Hashar: dockerfiles: silence git in ci-src-setup [integration/config] - 10https://gerrit.wikimedia.org/r/1144483 (https://phabricator.wikimedia.org/T393847) [09:50:23] (03CR) 10Hashar: dockerfiles: silence git in ci-src-setup (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/1144483 (https://phabricator.wikimedia.org/T393847) (owner: 10Hashar) [09:51:01] (03PS2) 10Hashar: Omit noisy `ls` debugging commands when not needed [integration/config] - 10https://gerrit.wikimedia.org/r/1143972 (https://phabricator.wikimedia.org/T282893) (owner: 10Bartosz Dziewoński) [09:52:04] (03CR) 10Hashar: [C:03+2] Omit noisy `ls` debugging commands when not needed [integration/config] - 10https://gerrit.wikimedia.org/r/1143972 (https://phabricator.wikimedia.org/T282893) (owner: 10Bartosz Dziewoński) [09:52:33] !log Updating all jobs for https://gerrit.wikimedia.org/r/c/integration/config/+/1143972 "Omit noisy `ls` debugging commands when not needed" # T282893 & T393847 [09:52:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:52:36] T282893: Various CI jobs failing after "mkdir: cannot create directory ‘log’: Permission denied" - https://phabricator.wikimedia.org/T282893 [09:52:37] T393847: Improve brevity of Jenkins console output - https://phabricator.wikimedia.org/T393847 [09:54:42] 10Continuous-Integration-Config, 07Developer Productivity, 13Patch-For-Review: Improve brevity of Jenkins console output - https://phabricator.wikimedia.org/T393847#10810759 (10hashar) I have split one of the changes to smaller ones and deployed most of them. The last one remaining is https://gerrit.wikimedi... [09:55:05] (03CR) 10Hashar: [C:03+2] doc: Remove quotes from env var value [integration/quibble] - 10https://gerrit.wikimedia.org/r/1141863 (owner: 10Jakob) [10:11:51] (03Merged) 10jenkins-bot: doc: Remove quotes from env var value [integration/quibble] - 10https://gerrit.wikimedia.org/r/1141863 (owner: 10Jakob) [12:17:25] 10Scap (SpiderPig 🕸️): SpiderPig fails to show warning for Depends-On confusion in summary view - https://phabricator.wikimedia.org/T393885 (10Tgr) 03NEW [12:18:24] 10Scap (SpiderPig 🕸️): SpiderPig fails to show warning for Depends-On confusion in summary view - https://phabricator.wikimedia.org/T393885#10811156 (10Tgr) [12:30:43] !log Disable publishing noise from rWSWF, T143162, T267223 [12:30:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:30:47] T143162: Reduce task notification noise/frequency of changes to associated open patchsets - https://phabricator.wikimedia.org/T143162 [12:30:47] T267223: Move /random endpoint from RESTBase to Wikifeeds - https://phabricator.wikimedia.org/T267223 [12:32:33] 10Scap (SpiderPig 🕸️): SpiderPig UI: "show sensitive information" checkbox should show further clarification - https://phabricator.wikimedia.org/T393889 (10Tgr) 03NEW [12:45:09] mwext-codehealth-master-non-voting (non voting) seems rather backlogged [12:45:42] Only 51 of them. /s [12:46:36] ISTR we run it only one at a time to avoid issues with upstream API being overloaded, which was fine when it was an experimental tool only on a handful of repos, but now that it's been rolled out more widely… [12:47:12] Yeah... [12:47:31] * Reedy files a task [12:47:35] On it. [12:47:38] T224990 [12:47:39] T224990: Enable codehealth pipeline for more extensions - https://phabricator.wikimedia.org/T224990 [12:47:59] that task hasn't been touched in 4.5 years :P [12:49:09] Now it has. [12:49:13] 10Continuous-Integration-Config, 06Code-Health-Metrics: Enable codehealth pipeline for more extensions - https://phabricator.wikimedia.org/T224990#10811348 (10Jdforrester-WMF) Note that as of right now there's a long back-log of the post-merge job (`mwext-codehealth-master-non-voting`), ~6 hours / 51. https:/... [12:51:28] 10Phabricator-Bot-Requests: Request a bot for a Slack integration - https://phabricator.wikimedia.org/T393773#10811367 (10DSantamaria) Hi @Aklapper , Are we good, or do I need to provide something else? [13:07:52] 10Scap (SpiderPig 🕸️): Spiderpig UI should link to Logtsash dashboards - https://phabricator.wikimedia.org/T393895 (10Tgr) 03NEW [13:26:59] 10Scap (SpiderPig 🕸️): Detect stuck gerrit merges in SpiderPig - https://phabricator.wikimedia.org/T393896 (10Tgr) 03NEW [13:30:16] 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10Scap (SpiderPig 🕸️), 07OKR-Work: Dismiss interaction notification if someone responded to the interaction - https://phabricator.wikimedia.org/T393026#10811615 (10Tgr) I was just about to suggest the same. Is this expected to work already? It didn't seem to w... [13:30:23] (03PS3) 10Hashar: Omit noisy `ls` debugging commands when not needed [integration/config] - 10https://gerrit.wikimedia.org/r/1143972 (https://phabricator.wikimedia.org/T282893) (owner: 10Bartosz Dziewoński) [13:30:38] (03CR) 10Hashar: [C:03+2] Omit noisy `ls` debugging commands when not needed [integration/config] - 10https://gerrit.wikimedia.org/r/1143972 (https://phabricator.wikimedia.org/T282893) (owner: 10Bartosz Dziewoński) [13:32:46] (03Merged) 10jenkins-bot: Omit noisy `ls` debugging commands when not needed [integration/config] - 10https://gerrit.wikimedia.org/r/1143972 (https://phabricator.wikimedia.org/T282893) (owner: 10Bartosz Dziewoński) [13:35:03] maintenance-disconnect-full-disks build 700990 integration-agent-docker-1048 (/: 25%, /srv: 97%, /var/lib/docker: 33%): OFFLINE due to disk space [13:38:48] (03PS2) 10Hashar: build: Updating mediawiki/mediawiki-codesniffer to 47.0.0 [integration/docroot] - 10https://gerrit.wikimedia.org/r/1143220 (owner: 10Libraryupgrader) [13:40:03] maintenance-disconnect-full-disks build 700991 integration-agent-docker-1048 (/: 25%, /srv: 33%, /var/lib/docker: 31%): RECOVERY disk space OK [13:44:29] (03CR) 10Hashar: [C:03+2] build: Updating mediawiki/mediawiki-codesniffer to 47.0.0 [integration/docroot] - 10https://gerrit.wikimedia.org/r/1143220 (owner: 10Libraryupgrader) [13:48:15] 10Scap (SpiderPig 🕸️): Spiderpig UI should link to Logstash dashboards - https://phabricator.wikimedia.org/T393895#10811740 (10Lucas_Werkmeister_WMDE) [13:49:20] 10Gerrit: Sticky search query in Gerrit search form - https://phabricator.wikimedia.org/T393899 (10DAlangi_WMF) 03NEW [14:27:41] James_F: I'll summarize on the buster > bullseye task T335766 [14:27:43] T335766: Migrate Quibble images from buster to bullseye - https://phabricator.wikimedia.org/T335766 [14:33:45] 10Beta-Cluster-Infrastructure, 10observability, 07Grafana: deployment-etcd05: Function lookup() did not find a value for the name 'prometheus::instances_defaults' - https://phabricator.wikimedia.org/T393866#10811902 (10joanna_borun) [14:34:16] (03CR) 10Bartosz Dziewoński: [C:03+1] dockerfiles: silence git in ci-src-setup (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/1144483 (https://phabricator.wikimedia.org/T393847) (owner: 10Hashar) [14:34:25] 06Release-Engineering-Team, 06collaboration-services, 06Infrastructure-Foundations, 06SRE, 10vm-requests: codfw: 1VM request for zuul3+ - https://phabricator.wikimedia.org/T393873#10811905 (10joanna_borun) p:05Triage→03Medium [14:34:54] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Migrate Quibble images from buster to bullseye - https://phabricator.wikimedia.org/T335766#10811906 (10hashar) Production has pretty much migrated everything and maintenance scripts are the last one to move. Our Bu... [14:34:56] 10Continuous-Integration-Config, 07Developer Productivity, 13Patch-For-Review: Improve brevity of Jenkins console output - https://phabricator.wikimedia.org/T393847#10811907 (10matmarex) Wow, thank you! [14:35:21] MatmaRex: Timo and James say thank you for the changes you have made to the Jenkins console output 🏆 [14:36:26] really it looks like i mde you do all the work by proposing half-correct patches ;) thanks for making it all work [14:36:59] btw what runs the example-run.sh files, and why are they all named "example"? that was one thing i couldn't figure out [14:38:03] oh I have merely split your patch in chunks that were easier for me to deploy :) [14:38:16] the example-run.sh files are only used to locally test an image [14:38:27] we used those a lot in the early days when we were creating all those base images [14:38:45] so locally we could update the Dockerfile, build the image and invoke the shell to have a live test [14:38:51] rinse and repeat until happy [14:39:16] they are thus not used by CI at all [14:39:26] (03update) 10dancy: spiderpig: Accept unconsumed TOTP tokens from the T-1, T+0, and T+1 windows [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/794 (https://phabricator.wikimedia.org/T392815 https://phabricator.wikimedia.org/T393651) [14:39:33] (03merge) 10dancy: spiderpig: Accept unconsumed TOTP tokens from the T-1, T+0, and T+1 windows [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/794 (https://phabricator.wikimedia.org/T392815 https://phabricator.wikimedia.org/T393651) [14:41:39] (03open) 10dancy: Release 4.163.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/796 [14:42:49] (03CR) 10Hashar: [C:03+2] "Great! :]" [integration/config] - 10https://gerrit.wikimedia.org/r/1144483 (https://phabricator.wikimedia.org/T393847) (owner: 10Hashar) [14:44:14] (03merge) 10dancy: Release 4.163.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/796 [14:44:27] (03Merged) 10jenkins-bot: dockerfiles: silence git in ci-src-setup [integration/config] - 10https://gerrit.wikimedia.org/r/1144483 (https://phabricator.wikimedia.org/T393847) (owner: 10Hashar) [14:48:31] aha [14:52:21] 10Scap (SpiderPig 🕸️), 13Patch-For-Review: Accept unconsumed TOTP tokens from the T-1, T+0, and T+1 windows - https://phabricator.wikimedia.org/T393651#10811987 (10dancy) 05In progress→03Resolved Deployed via scap 4.163.0 [14:58:10] 10Scap (SpiderPig 🕸️): Spiderpig UI should link to Logstash dashboards - https://phabricator.wikimedia.org/T393895#10812021 (10dancy) There is already a link to the mediawiki-errors dashboard on the top right of every SpiderPig page. Do you know the url of the mediawiki-debug dashboard? [15:01:07] images are uploading [15:01:41] 10Scap (SpiderPig 🕸️): Spiderpig UI should link to Logstash dashboards - https://phabricator.wikimedia.org/T393895#10812058 (10Lucas_Werkmeister_WMDE) That would be https://logstash.wikimedia.org/app/dashboards#/view/mwdebug1002 (linked from [Backport windows/Deployers](https://wikitech.wikimedia.org/wiki/Backpo... [15:17:09] 10Beta-Cluster-Infrastructure, 10observability, 07Grafana, 13Patch-For-Review: deployment-etcd05: Function lookup() did not find a value for the name 'prometheus::instances_defaults' - https://phabricator.wikimedia.org/T393866#10812181 (10bd808) The patch would set: `lang=yaml prometheus::instances_default... [15:17:46] 10Phabricator (Upstream), 07Upstream: Phabricator workboard import: Array for %Ls conversion is empty. Query: projectPHID IN (%Ls) - https://phabricator.wikimedia.org/T392168#10812188 (10Aklapper) p:05Triage→03Medium a:03Aklapper ` [2025-05-12 14:31:14] EXCEPTION: (AphrontParameterQueryException) Array f... [15:18:39] 10Beta-Cluster-Infrastructure, 10observability, 07Grafana, 13Patch-For-Review: deployment-etcd05: Function lookup() did not find a value for the name 'prometheus::instances_defaults' - https://phabricator.wikimedia.org/T393866#10812207 (10bd808) The `prometheus::instances:` settings seem to also be in comm... [15:19:19] (03PS1) 10Hashar: jjb: silence git in ci-src-setup [integration/config] - 10https://gerrit.wikimedia.org/r/1144596 (https://phabricator.wikimedia.org/T393847) [15:21:59] !log Added `prometheus::instances` and `prometheus::instances_defaults` hiera settings to "deployment-etcd" Prefix Puppet via Horizon (T393866) [15:22:01] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:22:02] T393866: deployment-etcd05: Function lookup() did not find a value for the name 'prometheus::instances_defaults' - https://phabricator.wikimedia.org/T393866 [15:23:51] 10Beta-Cluster-Infrastructure: EtcdConfig failed to fetch data: (curl error: 60) SSL peer certificate or SSH remote key was not OK - https://phabricator.wikimedia.org/T393855#10812294 (10bd808) Once I got Puppet working in {T393866} it looks to me like it fixed the cert issue: ` Notice: /Stage[main]/Main/Cfssl::... [15:24:45] 10Beta-Cluster-Infrastructure, 10observability, 07Grafana, 13Patch-For-Review: deployment-etcd05: Function lookup() did not find a value for the name 'prometheus::instances_defaults' - https://phabricator.wikimedia.org/T393866#10812296 (10bd808) >>! In T393866#10812265, @Stashbot wrote: > {nav icon=file, n... [15:28:03] !log Forced puppet run on deployment-etcd02.deployment-prep.eqiad1.wikimedia.cloud to fix Puppet run (T393866) [15:28:05] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:28:05] T393866: deployment-etcd05: Function lookup() did not find a value for the name 'prometheus::instances_defaults' - https://phabricator.wikimedia.org/T393866 [15:28:08] !log Forced puppet run on deployment-etcd05.deployment-prep.eqiad1.wikimedia.cloud to fix Puppet run (T393866) [15:28:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:28:55] bd808: I usually get an email once a day of puppet is not running properly on a deployment-prep node. Do you know why that wasn't happening for this etcd host? [15:30:27] dancy: I don't, but I do know both hosts have been on the https://prometheus-alerts.wmcloud.org/?q=project%3Ddeployment-prep&q=%40state%3Dactive board for a while. [15:30:35] https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/abae441df308e1d7c3da73f19f5a416194c76dc4%5E%21/deployment-prep/_.yaml suggests that those emails were turned off in 2021 [15:30:54] hmmm.. maybe it's devtools hosts that I get the emails about. [15:31:11] and perhaps we should turn the deployment-prep ones back on so we find out about these problems sooner. [15:32:12] we can spam people, but until someone feels it is their job to keep Beta Cluster healthy it's just spam. [15:32:54] * bd808 sees the cirrussearch* nodes are failing puppet too [15:35:50] 10Beta-Cluster-Infrastructure: EtcdConfig failed to fetch data: (curl error: 60) SSL peer certificate or SSH remote key was not OK - https://phabricator.wikimedia.org/T393855#10812423 (10bd808) 05Open→03Resolved a:03bd808 I think the lack of EtcdConfig complaints here means the certs are fixed. `lang=s... [15:50:03] !log `sudo /usr/local/sbin/clean-stale-puppet-certs --clean` on deployment-puppetserver-1.deployment-prep.eqiad1.wikimedia.cloud. Attempting to fix a "Found non-revoked Puppet certificates for 1 deleted instances" Prometheus alert. [15:50:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:54:47] 10Beta-Cluster-Infrastructure: Puppet failing on deployment-cirrussearch{12,13,14}.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T393924 (10bd808) 03NEW [15:55:24] !log Updated Jenkins jobs to silence git in ci-src-setup | https://gerrit.wikimedia.org/r/1144596 | T393847 [15:55:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:55:27] T393847: Improve brevity of Jenkins console output - https://phabricator.wikimedia.org/T393847 [15:57:43] bd808: We know that releng is supposed to be the owner of beta so I think those notifications being delivered to us is reasonable. It's only spam if we don't do anything about it. [15:57:54] 10Beta-Cluster-Infrastructure, 10CirrusSearch, 06Discovery-Search, 07Puppet: Puppet failing on deployment-cirrussearch{12,13,14}.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T393924#10812549 (10bd808) @bking Could you make some time to look into these failures? https://opensta... [15:57:54] (03CR) 10Hashar: [C:03+2] "I have updated all the jobs!" [integration/config] - 10https://gerrit.wikimedia.org/r/1144596 (https://phabricator.wikimedia.org/T393847) (owner: 10Hashar) [15:59:10] (03CR) 10CI reject: [V:04-1] jjb: silence git in ci-src-setup [integration/config] - 10https://gerrit.wikimedia.org/r/1144596 (https://phabricator.wikimedia.org/T393847) (owner: 10Hashar) [15:59:53] 00:00:17.393 docker: Error response from daemon: manifest for docker-registry.wikimedia.org/releng/tox-buster:4.8.0-3-s4 not found: manifest unknown: manifest unknown. [15:59:54] pfff [16:00:01] dancy: fair enough. One of the challenges today would be only emailing RelEng folks and not the dozens of other deployment-prep admins. Maybe we could setup more targeted notifications using the Prometheus alerting system. [16:00:36] Agreed [16:00:48] 10Beta-Cluster-Infrastructure, 10CirrusSearch, 06Data-Platform-SRE, 06Discovery-Search, 07Puppet: Puppet failing on deployment-cirrussearch{12,13,14}.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T393924#10812563 (10bking) [16:00:51] Maybe even a "Beta Cluster errors" mailing list that folks could opt into? :shrug: [16:01:18] 10Beta-Cluster-Infrastructure, 10CirrusSearch, 06Data-Platform-SRE, 06Discovery-Search, 07Puppet: Puppet failing on deployment-cirrussearch{12,13,14}.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T393924#10812564 (10bking) Thanks for the ping, @bd808 . I'm not sure when we'll... [16:01:48] bd808: I like that idea. [16:07:49] (03approved) 10dduvall: blubber.yaml: Use buildkit:wmf-v0.21.1 [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/122 (https://phabricator.wikimedia.org/T393731) (owner: 10dancy) [16:08:22] 10Beta-Cluster-Infrastructure: Setup mailing list for automated monitoring reports from Beta Cluster project - https://phabricator.wikimedia.org/T393926 (10bd808) 03NEW [16:10:56] (03merge) 10dancy: blubber.yaml: Use buildkit:wmf-v0.21.1 [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/122 (https://phabricator.wikimedia.org/T393731) [16:16:12] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 13Patch-For-Review: buildkit v0.21.1 released - https://phabricator.wikimedia.org/T393731#10812656 (10dancy) 05Open→03Resolved [16:26:36] (03PS1) 10Hashar: dockerfiles: stop build tox-buster images [integration/config] - 10https://gerrit.wikimedia.org/r/1144611 (https://phabricator.wikimedia.org/T335765) [16:27:33] (03CR) 10Hashar: [C:03+2] "The tox-buster images failed to build and I am thus removing them with f572aebe1cb72cb0cb1843f60a4f61a5e28c113d" [integration/config] - 10https://gerrit.wikimedia.org/r/1144483 (https://phabricator.wikimedia.org/T393847) (owner: 10Hashar) [16:27:56] (03CR) 10CI reject: [V:04-1] dockerfiles: stop build tox-buster images [integration/config] - 10https://gerrit.wikimedia.org/r/1144611 (https://phabricator.wikimedia.org/T335765) (owner: 10Hashar) [16:28:43] (03CR) 10Jforrester: "Ouch." [integration/config] - 10https://gerrit.wikimedia.org/r/1144611 (https://phabricator.wikimedia.org/T335765) (owner: 10Hashar) [16:29:09] (03PS2) 10Hashar: jjb: silence git in ci-src-setup [integration/config] - 10https://gerrit.wikimedia.org/r/1144596 (https://phabricator.wikimedia.org/T393847) [16:30:14] (03CR) 10Hashar: "Hopefully a lot of those repositories are to be archived or have moved to Gitlab. Or maybe I can move them to releng/tox-v3 but that woul" [integration/config] - 10https://gerrit.wikimedia.org/r/1144611 (https://phabricator.wikimedia.org/T335765) (owner: 10Hashar) [16:30:58] (03CR) 10CI reject: [V:04-1] jjb: silence git in ci-src-setup [integration/config] - 10https://gerrit.wikimedia.org/r/1144596 (https://phabricator.wikimedia.org/T393847) (owner: 10Hashar) [16:31:31] (03CR) 10Hashar: "The tox-buster images are no more buildable I have thuse frozen then in If572aebe1cb72cb0cb1843f60a4f61a5e28c113d . 2nd patchset is removi" [integration/config] - 10https://gerrit.wikimedia.org/r/1144596 (https://phabricator.wikimedia.org/T393847) (owner: 10Hashar) [16:32:12] 10Beta-Cluster-Infrastructure, 10CirrusSearch, 06Data-Platform-SRE, 06Discovery-Search, 07Puppet: Puppet failing on deployment-cirrussearch{12,13,14}.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T393924#10812752 (10taavi) The likely fix for this will be migrating that file d... [16:32:53] 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10Scap (SpiderPig 🕸️), 07OKR-Work: Dismiss interaction notification if someone responded to the interaction - https://phabricator.wikimedia.org/T393026#10812759 (10dancy) I just tested https://spiderpig.wikimedia.org which was running scap 4.163.0. I used spi... [16:33:37] (03PS2) 10Hashar: dockerfiles: stop build tox-buster images [integration/config] - 10https://gerrit.wikimedia.org/r/1144611 (https://phabricator.wikimedia.org/T335765) [16:33:37] (03PS3) 10Hashar: jjb: silence git in ci-src-setup [integration/config] - 10https://gerrit.wikimedia.org/r/1144596 (https://phabricator.wikimedia.org/T393847) [16:34:24] (03update) 10dancy: backport: Include full complaint in _prompt_for_approval_or_exit calls [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/797 (https://phabricator.wikimedia.org/T393885) [16:34:25] (03open) 10dancy: backport: Include full complaint in _prompt_for_approval_or_exit calls [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/797 (https://phabricator.wikimedia.org/T393885) [16:34:53] (03CR) 10CI reject: [V:04-1] jjb: silence git in ci-src-setup [integration/config] - 10https://gerrit.wikimedia.org/r/1144596 (https://phabricator.wikimedia.org/T393847) (owner: 10Hashar) [16:34:54] (03CR) 10CI reject: [V:04-1] dockerfiles: stop build tox-buster images [integration/config] - 10https://gerrit.wikimedia.org/r/1144611 (https://phabricator.wikimedia.org/T335765) (owner: 10Hashar) [16:35:41] (03update) 10dancy: backport: Include full complaint in _prompt_for_approval_or_exit calls [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/797 (https://phabricator.wikimedia.org/T393885) [16:45:13] (03PS3) 10Hashar: dockerfiles: stop building tox-buster images [integration/config] - 10https://gerrit.wikimedia.org/r/1144611 (https://phabricator.wikimedia.org/T335765) [16:45:14] (03PS4) 10Hashar: jjb: silence git in ci-src-setup [integration/config] - 10https://gerrit.wikimedia.org/r/1144596 (https://phabricator.wikimedia.org/T393847) [16:46:34] !log Reenabled beta-scap-sync-world and beta-update-databases-eqiad Jenkins jobs [16:46:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:47:30] (03CR) 10Hashar: [C:03+2] dockerfiles: stop building tox-buster images [integration/config] - 10https://gerrit.wikimedia.org/r/1144611 (https://phabricator.wikimedia.org/T335765) (owner: 10Hashar) [16:48:10] !log Updated Jenkins jobs to silence git in ci-src-setup (take 2) | https://gerrit.wikimedia.org/r/1144596 | T393847 [16:48:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:48:13] T393847: Improve brevity of Jenkins console output - https://phabricator.wikimedia.org/T393847 [16:48:23] (03approved) 10jnuche: backport: Include full complaint in _prompt_for_approval_or_exit calls [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/797 (https://phabricator.wikimedia.org/T393885) (owner: 10dancy) [16:48:49] (03Merged) 10jenkins-bot: dockerfiles: stop building tox-buster images [integration/config] - 10https://gerrit.wikimedia.org/r/1144611 (https://phabricator.wikimedia.org/T335765) (owner: 10Hashar) [16:49:53] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥): Phabricator "Impossible Cluster Write" - https://phabricator.wikimedia.org/T393931 (10thcipriani) 03NEW [16:50:18] (03update) 10dancy: backport: Include full complaint in _prompt_for_approval_or_exit calls [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/797 (https://phabricator.wikimedia.org/T393885) [16:52:17] 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10Scap (SpiderPig 🕸️), 07OKR-Work: Dismiss interaction notification if someone responded to the interaction - https://phabricator.wikimedia.org/T393026#10812888 (10Tgr) I will see if I can reproduce it. [16:52:22] (03merge) 10dancy: backport: Include full complaint in _prompt_for_approval_or_exit calls [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/797 (https://phabricator.wikimedia.org/T393885) [16:52:29] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Buster Deprecation): Replace deployment-maps-master01 with a Bullseye or Bookworm instance - https://phabricator.wikimedia.org/T361381#10812890 (10bd808) This is still happening, and I'm currently struggling to figure out how to get verbose errors to explain t... [16:53:13] (03open) 10dancy: Release 4.164.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/798 [16:55:08] (03CR) 10Hashar: [C:03+2] "I have deployed those as well :)" [integration/config] - 10https://gerrit.wikimedia.org/r/1144596 (https://phabricator.wikimedia.org/T393847) (owner: 10Hashar) [16:55:13] (03merge) 10dancy: Release 4.164.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/798 [16:55:45] 10Continuous-Integration-Config, 07Developer Productivity, 13Patch-For-Review: Improve brevity of Jenkins console output - https://phabricator.wikimedia.org/T393847#10812919 (10hashar) 05Open→03Resolved a:03hashar I have deployed everything! :-] Thank you for the improvements! [16:56:42] (03Merged) 10jenkins-bot: jjb: silence git in ci-src-setup [integration/config] - 10https://gerrit.wikimedia.org/r/1144596 (https://phabricator.wikimedia.org/T393847) (owner: 10Hashar) [16:59:58] Yippee, build fixed! [16:59:58] Project beta-scap-sync-world build #206192: 09FIXED in 14 min: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/206192/ [17:09:54] Yippee, build fixed! [17:09:54] Project beta-update-databases-eqiad build #84661: 09FIXED in 2 min 48 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/84661/ [17:31:49] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Seen), 10WMF-General-or-Unknown: Invent a method for applying pre-merge operations/mediawiki-config changes to Beta Cluster wikis - https://phabricator.wikimedia.org/T136828#10813191 (10bd808) [17:45:03] 10Continuous-Integration-Config, 10Wikidata, 10ci-test-error (WMF-deployed Build Failure), 05MW-1.44-release: Wikibase REL1_44 gate-and-submit always times out - https://phabricator.wikimedia.org/T393869#10813252 (10Jdforrester-WMF) [17:57:41] 10Phabricator (Upstream), 07Upstream: Phabricator workboard import: Array for %Ls conversion is empty. Query: projectPHID IN (%Ls) - https://phabricator.wikimedia.org/T392168#10813331 (10Aklapper) I was initially looking at the SQL query being something like ` SELECT `project`.* FROM `project` `project` WHERE... [18:21:14] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Buster Deprecation): Replace deployment-maps-master01 with a Bullseye or Bookworm instance - https://phabricator.wikimedia.org/T361381#10813400 (10Andrew) The issue is this bit in the post install script: ` if ! getent passwd ${VAR_UG_PKG_NAME} > /de... [18:23:36] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Buster Deprecation): Replace deployment-maps-master01 with a Bullseye or Bookworm instance - https://phabricator.wikimedia.org/T361381#10813405 (10Andrew) > I'm currently struggling to figure out how to get verbose errors to explain the error exit status 6 res... [18:33:33] 10Continuous-Integration-Config, 07Developer Productivity: Improve brevity of Jenkins console output - https://phabricator.wikimedia.org/T393847#10813427 (10matmarex) We're down from 142 to 105 lines. Current results: {P75926} [18:36:08] 10Phabricator (Upstream), 07Upstream: Phabricator workboard import: Array for %Ls conversion is empty. Query: projectPHID IN (%Ls) - https://phabricator.wikimedia.org/T392168#10813433 (10Aklapper) [18:42:39] 10Scap (SpiderPig 🕸️): Spiderpig UI should link to Logstash dashboards - https://phabricator.wikimedia.org/T393895#10813449 (10Tgr) Duh, sorry, banner blindness. I guess because of the icon next to it I just assumed it's some sort of login/profile link, without really looking at it. [18:48:42] (03PS1) 10Bartosz Dziewoński: Use `set -x` / `set +x` a bit more for verbose checks [integration/config] - 10https://gerrit.wikimedia.org/r/1144636 (https://phabricator.wikimedia.org/T393847) [18:48:59] 10Continuous-Integration-Config, 07Developer Productivity, 13Patch-For-Review: Improve brevity of Jenkins console output - https://phabricator.wikimedia.org/T393847#10813461 (10matmarex) Please humor me with one more thing, I promise this is the last one :) [18:50:00] (03PS2) 10Bartosz Dziewoński: Use `set -x` / `set +x` a bit more for verbose checks [integration/config] - 10https://gerrit.wikimedia.org/r/1144636 (https://phabricator.wikimedia.org/T393847) [19:05:39] 10Phabricator-Bot-Requests, 10Release-Engineering-Team (Doing 😎): Request a bot for a Slack integration - https://phabricator.wikimedia.org/T393773#10813566 (10Aklapper) p:05Triage→03Medium a:03Aklapper [19:07:56] 10Phabricator-Bot-Requests, 10Release-Engineering-Team (Doing 😎): Request a bot for a Slack integration - https://phabricator.wikimedia.org/T393773#10813584 (10Aklapper) 05Open→03Resolved @DSantamaria: See non-public P75927 for the access token. For more info, see https://www.mediawiki.org/wiki/Phabric... [19:12:05] 10Phabricator: Visualize numerous queries/data in Facts (Charts) - https://phabricator.wikimedia.org/T393297#10813611 (10Aklapper) p:05Triage→03Low [19:12:38] 10Phabricator: Visualize numerous queries/data in Facts (Charts) - https://phabricator.wikimedia.org/T393297#10813616 (10Aklapper) [20:05:42] (03approved) 10dancy: Draft: Allow deployments to specify custom k8s clusters [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/795 (https://phabricator.wikimedia.org/T388761) (owner: 10swfrench) [20:29:14] (03open) 10dancy: backport: Augment "Would you like to see the diff?" prompt [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/799 [20:29:16] (03update) 10dancy: backport: Augment "Would you like to see the diff?" prompt [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/799 [20:42:55] (03update) 10dancy: backport: Augment "Would you like to see the diff?" prompt [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/799 [20:43:38] 10Scap (SpiderPig 🕸️): SpiderPig show submodule diffs - https://phabricator.wikimedia.org/T393964 (10thcipriani) 03NEW [20:50:25] (03open) 10dancy: backport: Use status.submoduleSummary=true when showing diffs [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/800 (https://phabricator.wikimedia.org/T393964) [20:50:27] (03update) 10dancy: backport: Use status.submoduleSummary=true when showing diffs [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/800 (https://phabricator.wikimedia.org/T393964) [20:55:28] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Buster Deprecation): Replace deployment-maps-master01 with a Bullseye or Bookworm instance - https://phabricator.wikimedia.org/T361381#10813948 (10bd808) >>! In T361381#10813400, @Andrew wrote: > At first blush, I think that usermod is just wrong -- since when... [21:03:26] (03update) 10dancy: backport: Use status.submoduleSummary=true when showing diffs [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/800 (https://phabricator.wikimedia.org/T393964) [21:04:22] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Buster Deprecation): Replace deployment-maps-master01 with a Bullseye or Bookworm instance - https://phabricator.wikimedia.org/T361381#10813975 (10bd808) 05Open→03Resolved I have a feeling that everything maps is still busted in deployment-prep, but Pu... [21:08:40] (03update) 10dancy: backport: Use status.submoduleSummary=true when showing diffs [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/800 (https://phabricator.wikimedia.org/T393964) [21:29:08] 10Beta-Cluster-Infrastructure: Setup mailing list for automated monitoring reports from Beta Cluster project - https://phabricator.wikimedia.org/T393926#10814051 (10bd808) We actually already have https://lists.wikimedia.org/postorius/lists/betacluster-alerts.lists.wikimedia.org/ which was setup for {T1125}. Add... [21:30:44] 10Beta-Cluster-Infrastructure: Setup mailing list for automated monitoring reports from Beta Cluster project - https://phabricator.wikimedia.org/T393926#10814058 (10bd808) I think we may be rediscovering the work from {T789} from first principles. [21:45:15] why am I still the only owner of betacluster-alerts? :P [21:45:41] fixed ^ bd808 is now co-owner [22:01:09] 10Beta-Cluster-Infrastructure: Setup mailing list for automated monitoring reports from Beta Cluster project - https://phabricator.wikimedia.org/T393926#10814212 (10bd808) @greg made me an admin of betacluster-alerts@lists.wikimedia.org [22:15:15] 10Beta-Cluster-Infrastructure, 06cloud-services-team, 10Cloud-VPS: Route deployment-prep Prometheus alerts to the betacluster-alerts@lists.wikimedia.org mailing list - https://phabricator.wikimedia.org/T393975 (10bd808) 03NEW [22:25:22] 10Beta-Cluster-Infrastructure, 06cloud-services-team, 10Cloud-VPS: Route deployment-prep Prometheus alerts to the betacluster-alerts@lists.wikimedia.org mailing list - https://phabricator.wikimedia.org/T393975#10814317 (10bd808) I think what is needed here is a row poked into the prometheusconfig database as... [22:39:29] 10Gerrit, 07Upstream: Sticky search query in Gerrit search form - https://phabricator.wikimedia.org/T393899#10814386 (10Paladox) I've done: https://gerrit-review.googlesource.com/c/gerrit/+/474382 [23:14:25] I got my first cold marketing email from gitlab. I guess their institutional memory isn't very long. [23:15:02] "I work with teams trying to modernize without giving up that open-core ethos" sigh [23:19:51] heh. I wonder what the open-core ethos is?