[01:13:36] 06Release-Engineering-Team, 10WikimediaDebug, 13Patch-For-Review: Support WebExtensions Manifest v3 - https://phabricator.wikimedia.org/T312694#10435866 (10bd808) It turns out that unsurprisingly the manifest v3 implementations between these two runtimes are different. There is active work at the W3C to crea... [05:29:03] 10Phabricator, 06collaboration-services: phabricator long running queries - https://phabricator.wikimedia.org/T383068#10436133 (10Marostegui) >>! In T383068#10435304, @Dzahn wrote: > This comment seems interesting: ". The MySQL setting optimizer_search_depth=0 may fix the weird explosive complexity here." Pha... [05:37:16] 10Phabricator, 06collaboration-services: phabricator long running queries - https://phabricator.wikimedia.org/T383068#10436141 (10Marostegui) [05:54:34] 10Phabricator, 06collaboration-services: phabricator long running queries - https://phabricator.wikimedia.org/T383068#10436153 (10Marostegui) I'll have to reload the proxy for the switch and the connections will be gone. I've saved the queries in case we want to investigate/disable/optimize them: {P71819} We... [05:55:37] 10Phabricator, 06collaboration-services, 06Data-Persistence: phabricator long running queries - https://phabricator.wikimedia.org/T383068#10436154 (10Marostegui) [06:51:54] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Rohansingh1010 - https://phabricator.wikimedia.org/T382397#10436207 (10Roahnsingh1010) 05Stalled→03Open [09:19:47] 10Beta-Cluster-Infrastructure: Multiple kafka Cergen certs expired in beta cluster - https://phabricator.wikimedia.org/T383096#10436586 (10elukey) Hi! So PKI works for deployment-prep, we can apply the prod's settings where applicable. I briefly checked and this is what I found: === kafka mirror === Afaics it... [09:33:33] 10Phabricator, 06SRE, 06Traffic: Phabricator should cache tasks for a few minutes for logged-out users - https://phabricator.wikimedia.org/T274228#10436598 (10kostajh) Hi, wondering if there's interest to move this forward. @DLynch and I have a [[ https://github.com/kemayo/loosephabric | tool that integrates... [10:10:37] 06Release-Engineering-Team, 10dev-images, 06Growth-Team, 10MediaWiki-Docker, and 2 others: Since the update to bookworm, unable to start mediawiki-web on my M2 Mac - https://phabricator.wikimedia.org/T382709#10436671 (10Urbanecm_WMF) >>! In T382709#10428228, @Mhurd wrote: > What version of Docker/Docker De... [11:12:31] (03CR) 10Hashar: [C:03+2] Remove Flow from being listed on doc.wikimedia.org frontpage [integration/docroot] - 10https://gerrit.wikimedia.org/r/1108446 (https://phabricator.wikimedia.org/T379671) (owner: 10Aklapper) [11:12:59] (03Merged) 10jenkins-bot: Remove Flow from being listed on doc.wikimedia.org frontpage [integration/docroot] - 10https://gerrit.wikimedia.org/r/1108446 (https://phabricator.wikimedia.org/T379671) (owner: 10Aklapper) [11:14:33] 10GitLab (Pipeline Services Migration🐤), 06collaboration-services, 13Patch-For-Review: move os-reports.wikimedia.org to kubernetes - https://phabricator.wikimedia.org/T350794#10436808 (10Arnoldokoth) @JMeybohm I think that could work too but if the rsync fails in the second container then the entire pod woul... [12:12:48] (03open) 10marostegui: deployments-calendar.yaml: Remove Arnaud [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/141 [12:13:26] 10GitLab: Gitlab MFA reset for Esuvat Bomani - https://phabricator.wikimedia.org/T383123 (10MBywater-WMF) 03NEW [12:30:29] (03merge) 10hashar: deployments-calendar.yaml: Remove Arnaud [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/141 (owner: 10marostegui) [12:34:17] 10GitLab (Pipeline Services Migration🐤), 06collaboration-services, 13Patch-For-Review: move os-reports.wikimedia.org to kubernetes - https://phabricator.wikimedia.org/T350794#10436993 (10JMeybohm) >>! In T350794#10436808, @Arnoldokoth wrote: > @JMeybohm I think that could work too but if the rsync fails in t... [12:37:37] 10GitLab: Gitlab MFA reset for Esuvat Bomani - https://phabricator.wikimedia.org/T383123#10436997 (10Aklapper) 05Open→03Stalled Hi, please note that the Phabricator account https://phabricator.wikimedia.org/p/EBomani-WMF/ has zero connected authentication methods, so it's effectively unusable, and this reque... [13:11:51] (03PS1) 10Abijeet Patro: Review access change [extensions/Translate] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1108743 [13:13:04] (03PS2) 10Abijeet Patro: Review access change for Tacsipacsi and jhsoby [extensions/Translate] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1108743 [13:17:04] 06Release-Engineering-Team, 06collaboration-services: Low disk space: doc1003 / doc2002 - https://phabricator.wikimedia.org/T382610#10437066 (10hashar) The doc1003 instance was erroneously created with **a single partition**, the generated documentations are stored under `/srv/doc` and `/srv` should be a stand... [13:19:38] (03CR) 10Tacsipacsi: "What does this mean? If it means that we can submit changes, then thanks, but I’m fine not having the responsibility that comes with submi" [extensions/Translate] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1108743 (owner: 10Abijeet Patro) [13:38:12] 06Release-Engineering-Team, 06collaboration-services: Low disk space: doc1003 / doc2002 - https://phabricator.wikimedia.org/T382610#10437142 (10hashar) Side track: 85G of 98G usage on /srv/doc is due to mediawiki/core I have filed T383128 to clean it up :) [13:57:26] (03PS1) 10Hashar: zuul: drop JS doc for fundraising/ branches [integration/config] - 10https://gerrit.wikimedia.org/r/1108747 (https://phabricator.wikimedia.org/T383128) [13:59:22] (03CR) 10Hashar: [C:03+2] zuul: drop JS doc for fundraising/ branches [integration/config] - 10https://gerrit.wikimedia.org/r/1108747 (https://phabricator.wikimedia.org/T383128) (owner: 10Hashar) [14:01:10] (03Merged) 10jenkins-bot: zuul: drop JS doc for fundraising/ branches [integration/config] - 10https://gerrit.wikimedia.org/r/1108747 (https://phabricator.wikimedia.org/T383128) (owner: 10Hashar) [14:11:04] 06Release-Engineering-Team, 06collaboration-services: Low disk space: doc1003 / doc2002 - https://phabricator.wikimedia.org/T382610#10437236 (10hashar) I have cleaned up a bunch of old MediaWiki documentation on `doc1003`, eventually the rsync replicated the deletions to `doc2002` and they now have plenty more... [14:15:39] (03PS1) 10Jforrester: Zuul: [mediawiki/extensions/WikiLambda] Only run standalone jobs in master [integration/config] - 10https://gerrit.wikimedia.org/r/1108751 [14:29:44] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/WikiLambda] Only run standalone jobs in master [integration/config] - 10https://gerrit.wikimedia.org/r/1108751 (owner: 10Jforrester) [14:31:24] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/WikiLambda] Only run standalone jobs in master [integration/config] - 10https://gerrit.wikimedia.org/r/1108751 (owner: 10Jforrester) [14:33:12] !log Zuul: [mediawiki/extensions/WikiLambda] Only run standalone jobs in master [14:33:13] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:45:50] James_F: we have a node18 "npm audit" job for Wikibase at https://integration.wikimedia.org/ci/view/All/job/wikibase-daily-npm-audit-daily-node18-npmaudit/ [14:45:54] which I guess runs on a daily or so basis [14:45:58] but in JJB it is no more defined [14:46:04] though there is a node20 version available :) [14:46:07] should I swap them? [14:46:45] I mean I can rename the node18 one node20 and refresh it [14:46:46] hmm [14:47:06] * hashar does that [14:48:31] !log Manually renamed wikibase-daily-npm-audit-daily-node18-npmaudit to node20 variant and refresh the config with JJB [14:48:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:48:38] https://integration.wikimedia.org/ci/view/All/job/wikibase-daily-npm-audit-daily-node20-npmaudit/ [14:48:48] * Lucas_WMDE hasn’t heard of that job before [14:50:09] * Lucas_WMDE doesn’t see any emails from it at wikimedia-ci-status@wikimedia[.]de either [14:51:10] ah [14:52:00] well it fails because well hmm there are "vulnerabilitiesi" [14:52:55] that job is only running for Wikibase [14:53:41] the npm audit thing was done by Kunal back in 2018 or so [14:53:49] Addshore did an experiment shortly before covid [14:53:58] and eventually muted the job via https://gerrit.wikimedia.org/r/c/integration/config/+/582798 [14:54:37] I am going to bye it [14:55:27] ah https://gerrit.wikimedia.org/r/c/integration/config/+/573235 [14:55:40] that is the source and T244001 was the task [14:55:41] T244001: Investigate reminding people to update `npm audit` - https://phabricator.wikimedia.org/T244001 [14:56:08] which apparently is now done via DependABot on Github [14:56:48] and a GitHub action that then uses Addbot to geenerate a change in Gerrit [14:57:02] as seen on eg https://gerrit.wikimedia.org/r/q/owner:addshorewiki%252Baddbot-wikitech@gmail.com [14:57:19] hashar: I thought I swapped them? Sorry! [14:57:19] when we also have https://libraryupgrader2.wmcloud.org/ [14:57:22] anyway [14:57:30] I am deleting the job :) [14:57:34] Yeah, maybe just drop them. [14:57:36] James_F: nop! But it is not a big deal :) [14:57:39] I am deleting it [15:02:46] James_F: oh also happy new year and https://phabricator.wikimedia.org/project/view/7592/ !!! [15:04:32] (03PS1) 10Hashar: jjb: rm Wikibase npm audit job [integration/config] - 10https://gerrit.wikimedia.org/r/1108760 (https://phabricator.wikimedia.org/T244001) [15:07:10] (03PS2) 10Hashar: jjb: rm Wikibase npm audit job [integration/config] - 10https://gerrit.wikimedia.org/r/1108760 (https://phabricator.wikimedia.org/T244001) [15:07:16] (03CR) 10Hashar: [C:03+2] jjb: rm Wikibase npm audit job [integration/config] - 10https://gerrit.wikimedia.org/r/1108760 (https://phabricator.wikimedia.org/T244001) (owner: 10Hashar) [15:08:34] (03Merged) 10jenkins-bot: jjb: rm Wikibase npm audit job [integration/config] - 10https://gerrit.wikimedia.org/r/1108760 (https://phabricator.wikimedia.org/T244001) (owner: 10Hashar) [15:12:14] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (Zuul upgrade): Deleted unused Jenkins jobs - https://phabricator.wikimedia.org/T383134 (10hashar) 03NEW [15:21:16] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (Zuul upgrade): Delete unused Jenkins jobs - https://phabricator.wikimedia.org/T383134#10437457 (10hashar) [15:37:44] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (Zuul upgrade): Delete unused Jenkins jobs - https://phabricator.wikimedia.org/T383134#10437510 (10Jdforrester-WMF) Some of them are matrix-creations that we never run in practice. [15:38:50] 10GitLab: Gitlab MFA reset for Esuvat Bomani - https://phabricator.wikimedia.org/T383123#10437515 (10EBomani-WMF) Hello, Yes, that is correct, the phabricator account I use is Ebomani . I unfortunately did not get to save the recovery codes but I just tried the SSH... [15:41:06] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (Zuul upgrade): Delete unused Jenkins jobs - https://phabricator.wikimedia.org/T383134#10437518 (10hashar) [15:42:02] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 06cloud-services-team, 10Cloud-VPS, and 2 others: Various CI jobs running in the integration Cloud VPS project failing due to transient DNS lookup failures, often for our own hosts... - https://phabricator.wikimedia.org/T374830#10437519 [15:44:38] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 06cloud-services-team, 10Cloud-VPS, and 2 others: Various CI jobs running in the integration Cloud VPS project failing due to transient DNS lookup failures, often for our own hosts... - https://phabricator.wikimedia.org/T374830#10437522 [15:50:54] (03update) 10lwatson: Draft: web: improve JobViewerPage UI [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/608 (https://phabricator.wikimedia.org/T379413) [15:51:17] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (Zuul upgrade): Delete unused Jenkins jobs - https://phabricator.wikimedia.org/T383134#10437534 (10hashar) I went to install the [[ https://plugins.jenkins.io/extra-columns/ | Extra Columns Jenkins plugin ]] to create an [[ https://integr... [15:53:25] 10GitLab: Gitlab MFA reset for Esuvat Bomani - https://phabricator.wikimedia.org/T383123#10437545 (10MBywater-WMF) HI @Aklapper, Esuvat confirmed they were able to login using SSH codes, so this task can be closed. Thanks for your help! Meg [15:53:26] (03PS1) 10Hashar: jjb: remove quibble-integration Matrix job [integration/config] - 10https://gerrit.wikimedia.org/r/1108778 (https://phabricator.wikimedia.org/T383134) [15:54:34] (03CR) 10CI reject: [V:04-1] jjb: remove quibble-integration Matrix job [integration/config] - 10https://gerrit.wikimedia.org/r/1108778 (https://phabricator.wikimedia.org/T383134) (owner: 10Hashar) [16:01:42] (03PS2) 10Hashar: jjb: remove quibble-integration Matrix job [integration/config] - 10https://gerrit.wikimedia.org/r/1108778 (https://phabricator.wikimedia.org/T383134) [16:05:37] (03CR) 10Hashar: [C:03+2] "Bye job!" [integration/config] - 10https://gerrit.wikimedia.org/r/1108778 (https://phabricator.wikimedia.org/T383134) (owner: 10Hashar) [16:05:51] (03Merged) 10jenkins-bot: jjb: remove quibble-integration Matrix job [integration/config] - 10https://gerrit.wikimedia.org/r/1108778 (https://phabricator.wikimedia.org/T383134) (owner: 10Hashar) [16:06:00] (03CR) 10Jforrester: "<3" [integration/config] - 10https://gerrit.wikimedia.org/r/1108778 (https://phabricator.wikimedia.org/T383134) (owner: 10Hashar) [16:06:11] (03PS2) 10Hashar: Remove debian-glue-nocheck job [integration/config] - 10https://gerrit.wikimedia.org/r/1108780 (https://phabricator.wikimedia.org/T215458) [16:06:25] that is going to be a lot of mess this week [16:08:18] (03open) 10jelto: remove eoghan from office hours and gitlab maintenance window [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/142 [16:09:55] A lot *less* [16:10:43] OH MAN [16:10:52] we only have two kind of Matrix jobs left [16:10:55] \o/ [16:12:02] 06Release-Engineering-Team, 06Data-Engineering, 06Data-Platform-SRE, 10Dumps-Generation, and 4 others: WE 5.4 KR - Hypothesis 5.4.4 - Q3 FY24/25 - Migrate current-generation dumps to run on kubernetes - https://phabricator.wikimedia.org/T352650#10437672 (10BTullis) [16:18:17] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (Zuul upgrade), 13Patch-For-Review: Delete unused Jenkins jobs - https://phabricator.wikimedia.org/T383134#10437688 (10Jdforrester-WMF) [16:19:12] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Security-Team, 07Security: Disable php-composer-security-docker emails to security-admin-feed@lists - https://phabricator.wikimedia.org/T309655#10437689 (10hashar) 05Declined→03Open I am reopening this task cause it as some valuable history. **Su... [16:19:19] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Security-Team, 07Security: Disable php-composer-security-docker emails to security-admin-feed@lists - https://phabricator.wikimedia.org/T309655#10437693 (10hashar) [16:19:22] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (Zuul upgrade), 13Patch-For-Review: Delete unused Jenkins jobs - https://phabricator.wikimedia.org/T383134#10437692 (10hashar) [16:23:40] 06Gerrit-Privilege-Requests, 10LDAP-Access-Requests, 06SRE, 10SRE-Access-Requests: Offboard Muhammad Jazirahly from WMF systems - https://phabricator.wikimedia.org/T383056#10437705 (10Dzahn) 05Open→03In progress p:05Triage→03High a:03Dzahn [16:24:41] 06Gerrit-Privilege-Requests, 10LDAP-Access-Requests, 06SRE, 10SRE-Access-Requests: Offboard Muhammad Jazirahly from WMF systems - https://phabricator.wikimedia.org/T383056#10437709 (10Dzahn) [16:25:25] 06Gerrit-Privilege-Requests, 10LDAP-Access-Requests, 06SRE, 10SRE-Access-Requests: Offboard Muhammad Jazirahly from WMF systems - https://phabricator.wikimedia.org/T383056#10437714 (10Dzahn) [16:27:10] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06Security-Team, 07Security: Disable php-composer-security-docker emails to security-admin-feed@lists - https://phabricator.wikimedia.org/T309655#10437744 (10sbassett) >>! In T309655#10437689, @hashar wrote: > I am thus proposing to delete the jobs give... [16:30:46] 06Gerrit-Privilege-Requests, 10LDAP-Access-Requests, 06SRE, 10SRE-Access-Requests: Offboard Muhammad Jazirahly from WMF systems - https://phabricator.wikimedia.org/T383056#10437768 (10Dzahn) [16:48:49] 06Gerrit-Privilege-Requests, 10Infrastructure Security, 10LDAP-Access-Requests, 06SRE, and 2 others: Offboard Muhammad Jazirahly from WMF systems - https://phabricator.wikimedia.org/T383056#10437813 (10Dzahn) [16:49:58] 06Gerrit-Privilege-Requests, 10Infrastructure Security, 10LDAP-Access-Requests, 06SRE, and 2 others: Offboard Muhammad Jazirahly from WMF systems - https://phabricator.wikimedia.org/T383056#10437816 (10Dzahn) ` [ldap-maint1001:~] $ offboard-user -l muja ... Is not member of any LDAP group Is not a member i... [17:04:11] 06Release-Engineering-Team, 10dev-images, 06Growth-Team, 10MediaWiki-Docker, and 3 others: Since the update to bookworm, unable to start mediawiki-web on my M2 Mac - https://phabricator.wikimedia.org/T382709#10437869 (10bd808) 05Open→03Resolved a:03bd808 >>! In T382709#10436671, @Urbanecm_WMF wro... [17:06:06] 06Release-Engineering-Team, 10dev-images, 06Growth-Team, 10MediaWiki-Docker, and 3 others: Since the update to bookworm, unable to start mediawiki-web on my M2 Mac - https://phabricator.wikimedia.org/T382709#10437888 (10Mhurd) @Urbanecm_WMF It would be interesting to see if the issue is present on Doc... [17:06:39] 06Gerrit-Privilege-Requests, 10Infrastructure Security, 10LDAP-Access-Requests, 06SRE, and 2 others: Offboard Muhammad Jazirahly from WMF systems - https://phabricator.wikimedia.org/T383056#10437893 (10Dzahn) @Muehlenhoff @SLyngshede-WMF @WMDE-leszek I ... - manually removed from wmde, nda and airflow-w... [17:20:10] 06Gerrit-Privilege-Requests, 10Infrastructure Security, 10LDAP-Access-Requests, 06SRE, and 2 others: Offboard Muhammad Jazirahly from WMF systems - https://phabricator.wikimedia.org/T383056#10437944 (10Dzahn) Half an hour after the puppet patch was merged I ran the `sre.idm.logout` cookbook. [17:20:56] 06Release-Engineering-Team, 10dev-images, 06Growth-Team, 10MediaWiki-Docker, and 3 others: Since the update to bookworm, unable to start mediawiki-web on my M2 Mac - https://phabricator.wikimedia.org/T382709#10437946 (10bd808) >>! In T382709#10437888, @Mhurd wrote: > @Urbanecm_WMF > > It would be inte... [17:21:59] 06Release-Engineering-Team, 10dev-images, 06Growth-Team, 10MediaWiki-Docker, and 3 others: Since the update to bookworm, unable to start mediawiki-web on my M2 Mac - https://phabricator.wikimedia.org/T382709#10437947 (10Mhurd) Ah makes sense 👍 [17:27:48] 06Gerrit-Privilege-Requests, 10Infrastructure Security, 10LDAP-Access-Requests, 06SRE, and 2 others: Offboard Muhammad Jazirahly from WMF systems - https://phabricator.wikimedia.org/T383056#10437970 (10Dzahn) a:05Dzahn→03None [17:28:04] 06Gerrit-Privilege-Requests, 10Infrastructure Security, 10LDAP-Access-Requests, 06SRE, and 2 others: Offboard Muhammad Jazirahly from WMF systems - https://phabricator.wikimedia.org/T383056#10437972 (10Dzahn) [17:28:12] 06Gerrit-Privilege-Requests, 10Infrastructure Security, 10LDAP-Access-Requests, 06SRE, and 2 others: Offboard Muhammad Jazirahly from WMF systems - https://phabricator.wikimedia.org/T383056#10437973 (10Dzahn) p:05High→03Medium [18:00:50] 10GitLab: Gitlab MFA reset for Esuvat Bomani - https://phabricator.wikimedia.org/T383123#10438108 (10Aklapper) @EBomani-WMF: Glad you found a way! On an unrelated note, I am not sure what happened to your Phabricator account https://phabricator.wikimedia.org/p/EBomani-WMF/ as it has zero authentication providers... [18:14:34] (03approved) 10brennen: remove eoghan from office hours and gitlab maintenance window [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/142 (owner: 10jelto) [18:30:03] (03merge) 10brennen: remove eoghan from office hours and gitlab maintenance window [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/142 (owner: 10jelto) [18:30:56] (03open) 10taavi: repos: Add Striker [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/51 [18:41:22] (03merge) 10taavi: repos: Add Striker [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/51 [18:48:33] 10Beta-Cluster-Infrastructure: prometheus-openstack-stale-puppet-certs crashing on deployment-puppetserver-1.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T383153 (10bd808) 03NEW [18:49:24] 10Beta-Cluster-Infrastructure, 06cloud-services-team, 10Cloud-VPS: prometheus-openstack-stale-puppet-certs crashing on deployment-puppetserver-1.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T383153#10438323 (10bd808) [18:59:33] FIRING: DatasourceError: Queue (Jenkins jobs + Zuul functions) - https://grafana.wikimedia.org/alerting/grafana/b9a8470a-ebab-46f7-9be2-22b5e74a528b/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [19:00:19] 10Beta-Cluster-Infrastructure, 06cloud-services-team, 10Cloud-VPS: prometheus-openstack-stale-puppet-certs crashing on deployment-puppetserver-1.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T383153#10438358 (10bd808) a:03taavi After @taavi cleaned the certs causing the script... [19:09:32] RESOLVED: DatasourceError: Queue (Jenkins jobs + Zuul functions) - https://grafana.wikimedia.org/alerting/grafana/b9a8470a-ebab-46f7-9be2-22b5e74a528b/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [19:24:58] 10Beta-Cluster-Infrastructure, 06cloud-services-team, 10Cloud-VPS: Future growth of deployment-prep? - https://phabricator.wikimedia.org/T381420#10438434 (10Andrew) 05Open→03Resolved a:03Andrew We've now placed the order for new cloudvirts so I'm going to close this task for now. [19:25:26] 10Continuous-Integration-Infrastructure, 10GitLab (CI & Job Runners), 06cloud-services-team, 10Cloud-VPS, and 2 others: Future testing-infra growth on cloud-vps - https://phabricator.wikimedia.org/T381419#10438458 (10Andrew) 05Open→03Resolved a:03Andrew We've now placed the order for new cloudvir... [19:30:35] 10MediaWiki-Releasing, 07Documentation, 05MW-1.43-notes, 05MW-1.43-release, 13Patch-For-Review: Finish up RELEASE-NOTES for 1.43.0 - https://phabricator.wikimedia.org/T372325#10438479 (10Atieno) a:03Atieno [20:04:52] (03CR) 10Hashar: [C:03+2] "Job Was last build in December 2019" [integration/config] - 10https://gerrit.wikimedia.org/r/1108780 (https://phabricator.wikimedia.org/T215458) (owner: 10Hashar) [20:06:28] (03Merged) 10jenkins-bot: Remove debian-glue-nocheck job [integration/config] - 10https://gerrit.wikimedia.org/r/1108780 (https://phabricator.wikimedia.org/T215458) (owner: 10Hashar) [20:20:01] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.44.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T382363#10438633 (10thcipriani) p:05Triage→03Medium a:03brennen [20:20:02] Project beta-update-databases-eqiad build #81669: 04FAILURE in 2.1 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/81669/ [20:21:10] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.44.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T382364#10438638 (10thcipriani) p:05Triage→03Medium a:03hashar [20:21:48] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.44.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T382365#10438643 (10thcipriani) p:05Triage→03Medium a:03jeena [20:22:18] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.44.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T382366#10438648 (10thcipriani) p:05Triage→03Medium a:03jnuche [20:22:58] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.44.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T382367#10438655 (10thcipriani) p:05Triage→03Medium a:03Aklapper [20:24:45] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.44.0-wmf.17 deployment blockers - https://phabricator.wikimedia.org/T382368#10438662 (10thcipriani) p:05Triage→03Medium a:03dancy [20:38:45] PROBLEM - Host doc1003 is DOWN: PING CRITICAL - Packet loss = 100% [20:47:13] RECOVERY - Host doc1003 is UP: PING OK - Packet loss = 0%, RTA = 0.56 ms [20:47:28] 10Release-Engineering-Team (Priority Backlog 📥): Delete wmf branches from Gerrit repositories - https://phabricator.wikimedia.org/T303828#10438761 (10hashar) > Subject: [RelEng] PASS: train-clean > > ... > 05:00:01 No eligible versions to remove. Cause there was no train last week :] See you next week! [20:57:32] (03PS1) 10Fomafix: Zuul: [mediawiki/extensions/CollapsibleVector] Add Echo phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1108830 [21:00:39] (03PS2) 10Fomafix: Zuul: [mediawiki/extensions/CollapsibleVector] Add Vector phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1108830 [21:01:55] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.44.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T382363#10438784 (10brennen) [21:03:20] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 07Jenkins, 10Quality-and-Test-Engineering-Team (Test Infrastructure): Move beta cluster automatic deployment to a dedicated infrastructure - https://phabricator.wikimedia.org/T256168#10438786 (10hashar) [21:09:44] 06Project-Admins, 07Tracking-Neverending: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706#10438802 (10Seddon) Hey I have two batch requests on behalf of the apps engineering group. iOS Team: Can you add @HNordeenWMF, @tsevener, @mazevedo and @GOlso... [21:20:02] Project beta-update-databases-eqiad build #81670: 04STILL FAILING in 2.1 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/81670/ [21:22:06] (03open) 10catrope: Bump Codex to 1.19.1 and Vue to 3.4.28 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/52 (https://phabricator.wikimedia.org/T383073) [21:22:34] cdanis: ^ that failure might be related to your work [21:22:36] 21:20:02 Wikimedia\Rdbms\DBConnectionError from line 1129 of /srv/mediawiki-staging/php-master/includes/libs/rdbms/loadbalancer/LoadBalancer.php: Cannot access the database: Precondition failed: Attempted to create a span with the currently active span as the implicit parent, but no span was active. Use createRootSpan() to create a span with no parent (i.e. a root span). (deployment-db11) [21:22:44] looking [21:23:17] oh [21:23:23] this is a maintenance script, yes? [21:23:26] yeah [21:23:28] it's update.php [21:24:37] we will probably need a backport [21:24:55] that should be running master [21:24:58] ack [21:25:06] 06Release-Engineering-Team, 06collaboration-services: Low disk space: doc1003 / doc2002 - https://phabricator.wikimedia.org/T382610#10438860 (10Arnoldokoth) doc1003 is also good now. ` aokoth@doc1003:~$ sudo df -ht ext4 Filesystem Size Used Avail Use% Mounted on /dev/vda1 117G 7.7G 104G 7% / /d... [21:25:15] Reedy: well, I think we probably broke group0 maint scripts for similar reasons [21:25:42] ah, I thought you meant a backport to fix it on beta [21:26:03] You mean it needs .11-ing when we have a fix [21:26:09] yes, indeed [21:26:18] and I think we have a fix for prod, but possibly not for the tests [21:27:08] 06Release-Engineering-Team, 06collaboration-services: Low disk space: doc1003 / doc2002 - https://phabricator.wikimedia.org/T382610#10438864 (10Dzahn) [21:30:29] 06Release-Engineering-Team, 06collaboration-services: Low disk space: doc1003 / doc2002 - https://phabricator.wikimedia.org/T382610#10438874 (10Arnoldokoth) 05In progress→03Resolved [21:34:25] Reedy: advice appreciated on how to fix https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1105949 [21:38:16] I'd probably either mark the test as skipped/ignored for a while (ie until you revsit it)... [21:38:31] Or... around the check you've added/changed, evaludate defined( 'MW_PHPUNIT_TEST' ) [21:39:34] ah I got it [21:39:37] haha [21:39:43] yeah I had just finished testing that 😅 [21:39:58] Reedy: ptal https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1105949 for [21:40:48] yeah, that would look reasonable... see what jenkins has to say :P [21:41:05] indeed [21:47:06] 10GitLab (Pipeline Services Migration🐤), 06collaboration-services, 13Patch-For-Review: move os-reports.wikimedia.org to kubernetes - https://phabricator.wikimedia.org/T350794#10438912 (10Arnoldokoth) @JMeybohm Well, I guess that could work. The only other reason I would have to justify a separate cronjob i... [21:49:11] is it really necessary to run all the selenium tests on every patch 😰 [21:52:27] 10Continuous-Integration-Infrastructure (Zuul upgrade): Identify the different classes of jobs - https://phabricator.wikimedia.org/T383067#10438929 (10hashar) [21:58:09] 10Continuous-Integration-Infrastructure (Zuul upgrade): Identify the different classes of jobs - https://phabricator.wikimedia.org/T383067#10438963 (10hashar) [21:58:19] so hmm [21:58:23] there is a bit of work to do [21:59:11] which would be my fifth CI migration of doom (, JJB, Nodepool, Docker) [21:59:23] over 13 years [22:00:03] I can't tell I am good at building things that last, but maybe that is a sin of our field [22:00:19] and instead I should build concreted bridges like the Chinese and Romans! [22:03:24] This is weird, locally this is one the most commit. Clicking on rebase is also failing because it's on the most recent commit. Do you know what's happening? https://gerrit.wikimedia.org/r/c/operations/dumps/+/1108844/1#message-73c756bad33a0df299abd6d01decefa4fb15655c [22:04:20] there are two different merge checks [22:04:42] there is one done by Gerrit itself done entirely in memory (afaik) and reflected at the top left of the UI [22:04:53] that change is [Active] so it is not in a merge conflict [22:05:12] the other check is Zuul attempting to merge yur patch against the up-to-date tip of the target branch (master for this change) [22:05:22] and that is done using good old `git merge` [22:05:30] but there is a namespace clash [22:06:44] that is because a change to `operations/dumps/dcat` was proposed earlier [22:07:22] !log Deleted /srv/zuul/git/operations/dumps/dcat on both contint1002 and contint2002 # T157818 [22:07:24] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:07:24] T157818: zuul-merger fails when repository names overlaps - https://phabricator.wikimedia.org/T157818 [22:09:20] yaya [22:09:22] Amir1: fixed! thanks [22:09:24] thanks! [22:09:50] Amir1: that issue has been around forever, I first noticed it iin 2017 [22:09:56] but never went to get it fixed [22:10:04] thanksfully someone else fixed it for me upstream! [22:10:11] and when we upgrade.That will be a thing from the past [22:10:12] e [22:10:13] past [22:10:15] bah [22:10:28] ooh, nice [22:20:02] Project beta-update-databases-eqiad build #81671: 04STILL FAILING in 2.1 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/81671/ [22:28:39] 10Continuous-Integration-Infrastructure (Zuul upgrade): Identify the different classes of jobs - https://phabricator.wikimedia.org/T383067#10439047 (10hashar) 05Open→03In progress [22:28:42] next one should pass... [23:00:06] Reedy: yup the beta update job got fixed :) [23:00:10] https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/81672/console [23:00:16] thank you [23:00:30] * hashar sleeps [23:10:31] 06Project-Admins, 07Tracking-Neverending: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706#10439176 (10thcipriani) >>! In T706#10438801, @Seddon wrote: > Hey I have two batch requests on behalf of the apps engineering group. > > iOS Team: Can you ad... [23:11:02] Yippee, build fixed! [23:11:02] Project beta-update-databases-eqiad build #81672: 09FIXED in 11 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/81672/ [23:39:17] hi folks, fundraising is hoping to start running CiviCRM tests on bookworm / PHP 8.2 [23:39:32] I see it's using docker-registry.wikimedia.org/releng/civicrm:0.4.2-s2 [23:40:28] So I'd like to see if I can make a new version of that civicrm container, but I'm not sure where it is hosted [23:41:07] it's not under https://gitlab.wikimedia.org/repos/releng/dev-images , right ? [23:48:15] https://github.com/wikimedia/integration-config/blob/master/dockerfiles/civicrm/control [23:48:19] https://github.com/wikimedia/integration-config/tree/master/dockerfiles/civicrm [23:48:44] (03approved) 10egardner: Bump Codex to 1.19.1 and Vue to 3.4.28 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/52 (https://phabricator.wikimedia.org/T383073) (owner: 10catrope) [23:48:51] (03merge) 10egardner: Bump Codex to 1.19.1 and Vue to 3.4.28 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/52 (https://phabricator.wikimedia.org/T383073) (owner: 10catrope) [23:49:26] thanks Reedy! [23:50:43] it's using ci-bullseye anyway, so it should have PHP 8.2 already? [23:53:35] but yeah... updating it to use ci-bookworm... update changelog, rebuild image and update usage