[00:00:59] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Bitu, 06Infrastructure-Foundations: User corvus cannot activate ssh key in Bitu - https://phabricator.wikimedia.org/T395857#10877920 (10thcipriani) hrm, pasting what I see in log output for the public key it seems valid: ` ssh-keygen -lv -f <(echo '... [01:10:49] (03PS1) 10Hslater: Add OOJSPlus dependency to UnifiedTaskOverview [integration/config] - 10https://gerrit.wikimedia.org/r/1152866 [06:18:49] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Bitu, 06Infrastructure-Foundations: User corvus cannot activate ssh key in Bitu - https://phabricator.wikimedia.org/T395857#10878318 (10SLyngshede-WMF) a:03SLyngshede-WMF [06:57:50] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Bitu, 06Infrastructure-Foundations: User corvus cannot activate ssh key in Bitu - https://phabricator.wikimedia.org/T395857#10878362 (10SLyngshede-WMF) Trying to step through the code, every things looks correct until we try to save the key in LDAP. T... [07:08:29] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Bitu, 06Infrastructure-Foundations: User corvus cannot activate ssh key in Bitu - https://phabricator.wikimedia.org/T395857#10878382 (10SLyngshede-WMF) Actual error: ` >>> ldap_user.entry_commit_changes() False >>> ldap_user.entry_cursor.errors [Oper... [07:26:15] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Bitu, 06Infrastructure-Foundations: User corvus cannot activate ssh key in Bitu - https://phabricator.wikimedia.org/T395857#10878409 (10SLyngshede-WMF) @Corvus Hi, sorry for the inconvenience, your SSH key should work now. We had to remove the LDAP ob... [07:26:23] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Bitu, 06Infrastructure-Foundations: User corvus cannot activate ssh key in Bitu - https://phabricator.wikimedia.org/T395857#10878410 (10SLyngshede-WMF) 05Open→03Resolved p:05Triage→03High [08:30:43] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Bitu, 06Infrastructure-Foundations: User corvus cannot activate ssh key in Bitu - https://phabricator.wikimedia.org/T395857#10878520 (10taavi) @SLyngshede-WMF do you happen to have an idea how that account got in that state in the first place? Or... [08:41:04] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Bitu, 06Infrastructure-Foundations: User corvus cannot activate ssh key in Bitu - https://phabricator.wikimedia.org/T395857#10878579 (10SLyngshede-WMF) Yes, the account is fairly old and the system that created it wasn't to careful about ObjectCla... [08:49:50] 06Project-Admins, 10Release-Engineering-Team (Doing 😎): Create a new SLO Phabricator tag - https://phabricator.wikimedia.org/T395537#10878605 (10Aklapper) 05Open→03Resolved a:03Aklapper For now I called it `#SRE-SLO` but we can rename in the future (plus if this becomes cross-teams, change to the Tag... [08:57:38] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Bitu, 06Infrastructure-Foundations: User corvus cannot activate ssh key in Bitu - https://phabricator.wikimedia.org/T395857#10878636 (10SLyngshede-WMF) Note to self: https://github.com/cannatag/ldap3/blob/2b4d94e71dd26d3aeb937a350777c8be88e5c7ca/l... [09:14:23] 10Phabricator, 06Security-Team, 07Security: Audit members of acl*security for more than 12 months of no activity (May 2025) - https://phabricator.wikimedia.org/T368224#10878690 (10Aklapper) [09:14:44] 10Phabricator, 06Security-Team, 07Security: Audit members of acl*security for more than 12 months of no activity (May 2025) - https://phabricator.wikimedia.org/T368224#10878692 (10Aklapper) @Jly: Posted the SQL query results in non-public P76908 [09:49:44] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Bitu, 06Infrastructure-Foundations: User corvus cannot activate ssh key in Bitu - https://phabricator.wikimedia.org/T395857#10878822 (10SLyngshede-WMF) We've been do a bit more digging, the issue seems to stem from the OpenDJ to OpenLDAP migration... [09:52:31] !log `samtar@deployment-cache-text08:~$ sudo -i puppet agent -tv` for T395808 [09:52:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:53:17] !log `samtar@deployment-cache-text08:~$ sudo service varnish-frontend restart` for T395808 [09:53:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:44:20] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Bitu, 06Infrastructure-Foundations: User corvus cannot activate ssh key in Bitu - https://phabricator.wikimedia.org/T395857#10878968 (10MoritzMuehlenhoff) >>! In T395857#10878822, @SLyngshede-WMF wrote: > We've been do a bit more digging, the issu... [11:03:26] 10Gerrit, 06collaboration-services, 10Wikimedia-GitHub: github mirror out of sync - https://phabricator.wikimedia.org/T395887 (10Marostegui) 03NEW [11:16:06] 10Gerrit, 06collaboration-services, 10Wikimedia-GitHub: github mirror out of sync - https://phabricator.wikimedia.org/T395887#10879088 (10Jelto) CC from IRC: > Right after the last commit I found the error in Gerrit log "Error while deleting task f25ad588bb6cc6c345e1efca0b571c8bb262aad3 [CONTEXT PLUGIN="re... [11:33:38] 10Gerrit, 06collaboration-services, 10Wikimedia-GitHub: github mirror out of sync - https://phabricator.wikimedia.org/T395887#10879204 (10SomeRandomDeveloper) The repos of MW Core and various extensions I checked are also affected. [12:16:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance deployment-mwmaint03 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [12:16:39] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-mwmaint03 on project deployment-prep - https://phabricator.wikimedia.org/T395897 (10wmcs-alerts) 03NEW [12:22:11] 10Phabricator-Bot-Requests, 10Release-Engineering-Team (Doing 😎): Create Phabricator bot account for Cloud VPS monitoring - https://phabricator.wikimedia.org/T395104#10879324 (10taavi) 05Open→03Resolved [12:22:49] 10Beta-Cluster-Infrastructure, 06cloud-services-team, 10Cloud-VPS: Consider setting up an https://github.com/knyar/phalerts instance in metricsinfra - https://phabricator.wikimedia.org/T394446#10879326 (10taavi) 05Open→03Resolved Done for all deployment-prep alerts. Customizing which alerts are sent... [12:36:28] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance deployment-mwmaint03 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [12:36:33] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-mwmaint03 on project deployment-prep - https://phabricator.wikimedia.org/T395897#10879353 (10wmcs-alerts) [12:39:55] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-mwmaint03 on project deployment-prep - https://phabricator.wikimedia.org/T395897#10879358 (10taavi) ` Jun 03 11:30:16 deployment-mwmaint03 puppet-agent[2629852]: Using environment 'production' Jun 03 11:30:16 deployment-mwmaint03 pup... [14:06:52] 10Release-Engineering-Team (Doing 😎), 06Security-Team, 10Stashbot, 07SecTeam-Processed, and 2 others: stashbot comments leak security issue titles (when mentioning tasks the bot is subscribed on) - https://phabricator.wikimedia.org/T301082#10879587 (10sbassett) p:05Triage→03Medium [14:07:00] 10Release-Engineering-Team (Doing 😎), 06Security-Team, 10Stashbot, 07SecTeam-Processed, and 2 others: stashbot comments leak security issue titles (when mentioning tasks the bot is subscribed on) - https://phabricator.wikimedia.org/T301082#10879592 (10sbassett) [14:25:15] 06Project-Admins, 07Tracking-Neverending: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706#10879671 (10JVanderhoop-WMF) Hello! Can I be added to acl*Project-Admins? As the Product Manager of the Experiment Platform Team, I'm looking to groom our boa... [14:35:28] FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance deployment-webperf21 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [14:35:33] 10Beta-Cluster-Infrastructure: Last Puppet run was over 24 hours ago on instance deployment-webperf21 in project deployment-prep - https://phabricator.wikimedia.org/T395908 (10wmcs-alerts) 03NEW [14:36:41] 10Gerrit, 06collaboration-services, 10Wikimedia-GitHub: github mirror out of sync - https://phabricator.wikimedia.org/T395887#10879755 (10hashar) a:03hashar The replicas are around: ` $ ssh -p 29418 hashar@gerrit.wikimedia.org replication list Remote: github Url: git@github.com:wikimedia/${name} Remote: r... [15:02:29] 10Gerrit, 06collaboration-services, 10Wikimedia-GitHub, 13Patch-For-Review: github mirror out of sync - https://phabricator.wikimedia.org/T395887#10879856 (10hashar) That broke replication (T395887) because: * gerrit2003 uses a different username (T338470 should be completed) * its ssh host key was not kn... [15:11:32] 10Gerrit, 06collaboration-services, 10Wikimedia-GitHub, 13Patch-For-Review: github mirror out of sync - https://phabricator.wikimedia.org/T395887#10879909 (10hashar) 05Open→03Resolved I have seen replication pass through so I guess it is fixed. The replication is going on right now, it will eventua... [15:16:15] 10Phabricator, 06Security-Team, 07Security: Audit members of acl*security for more than 12 months of no activity (May 2025) - https://phabricator.wikimedia.org/T368224#10879938 (10Jly) a:03Jly Thanks! I will take a look at it and get back to you soon. [15:32:44] TheresNoTime: thanks for unblocking yourself :) [15:52:23] 10Continuous-Integration-Infrastructure: Create WMF CI images and jobs for Node.js 24 - https://phabricator.wikimedia.org/T395923 (10Jdforrester-WMF) 03NEW [15:52:31] 10Continuous-Integration-Infrastructure: Create WMF CI images and jobs for Node.js 22 - https://phabricator.wikimedia.org/T363653#10880350 (10Jdforrester-WMF) [15:52:39] (03PS1) 10Jforrester: Docker: Provide initial Node 24 images [integration/config] - 10https://gerrit.wikimedia.org/r/1153267 (https://phabricator.wikimedia.org/T395923) [15:53:33] (03CR) 10CI reject: [V:04-1] Docker: Provide initial Node 24 images [integration/config] - 10https://gerrit.wikimedia.org/r/1153267 (https://phabricator.wikimedia.org/T395923) (owner: 10Jforrester) [15:53:53] 10Continuous-Integration-Config: Upgrade all CI jobs for WMF-deployed projects from Node 20 to Node 22 - https://phabricator.wikimedia.org/T395924 (10Jdforrester-WMF) 03NEW [15:54:22] 10Continuous-Integration-Config: Upgrade all CI jobs for WMF-deployed projects from Node 22 to Node 24 - https://phabricator.wikimedia.org/T395926 (10Jdforrester-WMF) 03NEW [15:54:30] 10Continuous-Integration-Config: Upgrade all CI jobs for WMF-deployed projects from Node 20 to Node 22 - https://phabricator.wikimedia.org/T395924#10880408 (10Jdforrester-WMF) [15:54:42] 10Continuous-Integration-Config: Upgrade all CI jobs for WMF-deployed projects from Node 22 to Node 24 - https://phabricator.wikimedia.org/T395926#10880413 (10Jdforrester-WMF) [15:54:46] 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Create WMF CI images and jobs for Node.js 24 - https://phabricator.wikimedia.org/T395923#10880414 (10Jdforrester-WMF) [15:54:50] 10Continuous-Integration-Infrastructure: Create WMF CI images and jobs for Node.js 22 - https://phabricator.wikimedia.org/T363653#10880416 (10Jdforrester-WMF) [15:54:54] 10Continuous-Integration-Config: Upgrade all CI jobs for WMF-deployed projects from Node 20 to Node 22 - https://phabricator.wikimedia.org/T395924#10880415 (10Jdforrester-WMF) [15:54:58] 10Continuous-Integration-Config, 13Patch-For-Review: Upgrade all CI jobs for WMF-deployed projects from Node 18 to Node 20 - https://phabricator.wikimedia.org/T343827#10880417 (10Jdforrester-WMF) [15:55:02] 10Beta-Cluster-Infrastructure: Last Puppet run was over 24 hours ago on instance deployment-webperf21 in project deployment-prep - https://phabricator.wikimedia.org/T395927 (10bd808) 03NEW [15:55:16] (03PS2) 10Jforrester: Docker: Provide initial Node 24 images [integration/config] - 10https://gerrit.wikimedia.org/r/1153267 (https://phabricator.wikimedia.org/T395923) [15:56:46] 10Beta-Cluster-Infrastructure: Last Puppet run was over 24 hours ago on instance deployment-webperf21 in project deployment-prep - https://phabricator.wikimedia.org/T395927#10880460 (10bd808) The space seems to be taken up by logs in /var/log and quite possibly the same log messages being recorded in a number of... [15:57:00] 10Beta-Cluster-Infrastructure: Last Puppet run was over 24 hours ago on instance deployment-webperf21 in project deployment-prep - https://phabricator.wikimedia.org/T395927#10880463 (10bd808) This looks like more of {T391273} [15:57:50] (03PS1) 10Jforrester: jjb: Provide Node 24 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1153276 (https://phabricator.wikimedia.org/T395923) [15:58:03] (03CR) 10Jforrester: [C:03+2] Docker: Provide initial Node 24 images [integration/config] - 10https://gerrit.wikimedia.org/r/1153267 (https://phabricator.wikimedia.org/T395923) (owner: 10Jforrester) [15:59:10] 06Project-Admins, 06Release-Engineering-Team, 10MediaWiki-extensions-General, 10MediaWiki-General, and 2 others: Create a security pre-release Phabricator policy manageable by the Security Team - https://phabricator.wikimedia.org/T393403#10880482 (10sbassett) >>! In T393403#10873299, @Aklapper wrote: >... [15:59:51] (03CR) 10CI reject: [V:04-1] jjb: Provide Node 24 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1153276 (https://phabricator.wikimedia.org/T395923) (owner: 10Jforrester) [16:00:06] (03Merged) 10jenkins-bot: Docker: Provide initial Node 24 images [integration/config] - 10https://gerrit.wikimedia.org/r/1153267 (https://phabricator.wikimedia.org/T395923) (owner: 10Jforrester) [16:00:10] !log Docker: Provide initial Node 24 images, for T395923 [16:00:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:00:23] T395923: Create WMF CI images and jobs for Node.js 24 - https://phabricator.wikimedia.org/T395923 [16:01:56] 10Beta-Cluster-Infrastructure: Last Puppet run was over 24 hours ago on instance deployment-webperf21 in project deployment-prep - https://phabricator.wikimedia.org/T395927#10880523 (10bd808) `lang=shell-session root@deployment-webperf21:/var/log# rm messages.1 user.log.1 syslog.1 root@deployment-webperf21:/var/... [16:02:20] 10Beta-Cluster-Infrastructure: Last Puppet run was over 24 hours ago on instance deployment-webperf21 in project deployment-prep - https://phabricator.wikimedia.org/T395927#10880529 (10bd808) →14Duplicate dup:03T391273 [16:02:22] 10Beta-Cluster-Infrastructure, 10NavigationTiming, 06SRE Observability: navtiming: Loss of Kafka connection fills multiple log files with identical stack traces - https://phabricator.wikimedia.org/T391273#10880527 (10bd808) [16:12:29] 10Beta-Cluster-Infrastructure, 10NavigationTiming, 06SRE Observability: navtiming: Loss of Kafka connection fills multiple log files with identical stack traces - https://phabricator.wikimedia.org/T391273#10880566 (10bd808) >>! In T391273#10847883, @Krinkle wrote: > For what it's worth, there never has been... [16:19:52] 10Beta-Cluster-Infrastructure, 10NavigationTiming, 06SRE Observability: navtiming: Loss of Kafka connection fills multiple log files with identical stack traces - https://phabricator.wikimedia.org/T391273#10880607 (10bd808) In `::profile::webperf::processors`: `lang=puppet # statsv is on main kafka, not... [16:21:03] 10Beta-Cluster-Infrastructure, 10NavigationTiming, 06SRE Observability: navtiming: Loss of Kafka connection fills multiple log files with identical stack traces - https://phabricator.wikimedia.org/T391273#10880614 (10taavi) I think the ultimate issue here is that `profile::webperf::processors` expects a Kafk... [16:25:33] 10Beta-Cluster-Infrastructure, 10NavigationTiming, 06SRE Observability: navtiming: Loss of Kafka connection fills multiple log files with identical stack traces - https://phabricator.wikimedia.org/T391273#10880647 (10bd808) `::webperf::navtiming` leads to modules/webperf/templates/navtiming.systemd.erb where... [16:30:28] RESOLVED: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance deployment-webperf21 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [16:30:34] 10Beta-Cluster-Infrastructure: Last Puppet run was over 24 hours ago on instance deployment-webperf21 in project deployment-prep - https://phabricator.wikimedia.org/T395908#10880672 (10wmcs-alerts) [16:52:44] (03update) 10dduvall: artifacts: Support `exclude` patterns for copies/requirements [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/123 [16:52:47] (03update) 10dduvall: artifacts: Support `exclude` patterns for copies/requirements [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/123 [16:52:47] (03update) 10dduvall: artifacts: Support `exclude` patterns for copies/requirements [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/123 [16:56:18] (03update) 10dduvall: artifacts: Support `exclude` patterns for copies/requirements [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/123 [16:56:26] (03update) 10dduvall: artifacts: Support `exclude` patterns for copies/requirements [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/123 [16:56:38] (03update) 10dduvall: artifacts: Support `exclude` patterns for copies/requirements [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/123 [16:57:10] (03update) 10dduvall: artifacts: Support `exclude` patterns for copies/requirements [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/123 [17:02:47] 10Beta-Cluster-Infrastructure, 10NavigationTiming, 06SRE Observability: navtiming: Loss of Kafka connection fills multiple log files with identical stack traces - https://phabricator.wikimedia.org/T391273#10880784 (10bd808) Let's try this and see what happens: `lang=yaml kafka_clusters: jumbo-deployment-pr... [17:04:24] !log Added jumbo-eqiad and main-eqiad aliases to kafka_clusters hiera config (T391273) [17:04:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:04:26] T391273: navtiming: Loss of Kafka connection fills multiple log files with identical stack traces - https://phabricator.wikimedia.org/T391273 [17:07:23] 10Beta-Cluster-Infrastructure, 10NavigationTiming, 06SRE Observability: navtiming: Loss of Kafka connection fills multiple log files with identical stack traces - https://phabricator.wikimedia.org/T391273#10880797 (10bd808) Bah. Things don't survive the round trips from one system to another. ` Error: Could... [17:08:40] !log Manually expanded (duplicated) jumbo-eqiad and main-eqiad aliases in kafka_clusters hiera config (T391273) [17:08:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:09:43] !log Forced puppet run on deployment-webperf21 to pick up Kafka config changes (T391273) [17:09:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:09:47] T391273: navtiming: Loss of Kafka connection fills multiple log files with identical stack traces - https://phabricator.wikimedia.org/T391273 [17:12:27] 10Beta-Cluster-Infrastructure, 10NavigationTiming, 06SRE Observability: navtiming: Loss of Kafka connection fills multiple log files with identical stack traces - https://phabricator.wikimedia.org/T391273#10880828 (10bd808) ` Notice: /Stage[main]/Webperf::Navtiming/Systemd::Service[navtiming]/Systemd::Unit[n... [17:16:18] 10Beta-Cluster-Infrastructure, 10NavigationTiming, 06SRE Observability: navtiming: Loss of Kafka connection fills multiple log files with identical stack traces - https://phabricator.wikimedia.org/T391273#10880849 (10bd808) [17:16:19] 10Beta-Cluster-Infrastructure: Last Puppet run was over 24 hours ago on instance deployment-webperf21 in project deployment-prep - https://phabricator.wikimedia.org/T395908#10880852 (10bd808) →14Duplicate dup:03T391273 [17:20:56] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services: puppetize setup of new zuul VMs - https://phabricator.wikimedia.org/T395938 (10Dzahn) 03NEW [17:21:21] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Radar), 06collaboration-services, 06Infrastructure-Foundations, and 2 others: eqiad/codfw: 6 VM request for Zuul upgrade project - https://phabricator.wikimedia.org/T393873#10880896 (10Dzahn) Further setup from here on... [17:22:21] 10Beta-Cluster-Infrastructure, 10NavigationTiming, 06SRE Observability: navtiming: Loss of Kafka connection fills multiple log files with identical stack traces - https://phabricator.wikimedia.org/T391273#10880899 (10taavi) >>! In T391273#10880797, @bd808 wrote: > Bah. Things don't survive the round trips fr... [17:25:10] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services: puppetize setup of new zuul VMs - https://phabricator.wikimedia.org/T395938#10880930 (10Dzahn) partman: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1147878 in-setup: https://gerrit.wikimedia.org/r/c/operations/puppet/+/... [17:25:33] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-mwmaint03 on project deployment-prep - https://phabricator.wikimedia.org/T395897#10880934 (10bd808) 05Open→03Resolved a:03Clement_Goubert I think this was fixed by https://gerrit.wikimedia.org/r/c/operations/puppet/+/1153108 [17:29:39] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-mwmaint03 on project deployment-prep - https://phabricator.wikimedia.org/T395897#10880943 (10Clement_Goubert) Yeah sorry it took longer than expected to fix my mess. [17:35:09] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 13Patch-For-Review: puppetize setup of new zuul VMs - https://phabricator.wikimedia.org/T395938#10880972 (10Dzahn) ` Notice: /Stage[main]/Profile::Zuul::User/Group[zuul]/ensure: created Notice: /Stage[main]/Profile::Zuul::User/... [17:39:25] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06SRE, 10SRE-Access-Requests: Requesting access to contint-roots for Corvus - https://phabricator.wikimedia.org/T395167#10881004 (10Dzahn) a:05Corvus→03KFrancis [17:51:26] 10Beta-Cluster-Infrastructure, 10NavigationTiming, 06SRE Observability: navtiming: Loss of Kafka connection fills multiple log files with identical stack traces - https://phabricator.wikimedia.org/T391273#10881067 (10bd808) 05Open→03Resolved a:03bd808 I think this is fixed. There is a different log... [17:53:56] 10Beta-Cluster-Infrastructure: deployment-webperf21 logspam from /usr/bin/confd: "WARNING Found no templates" - https://phabricator.wikimedia.org/T395945 (10bd808) 03NEW [17:57:45] 10Beta-Cluster-Infrastructure: CampaignEvents: Error 1091: Can't DROP COLUMN `event_type`; check that it exists - https://phabricator.wikimedia.org/T394418#10881116 (10bd808) 05Open→03Resolved Let's call it sunspots and move on with our lives. If there is a bug here that isn't just timing related in depl... [18:16:58] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 06Data-Persistence: Request mariadb database for Zuul - https://phabricator.wikimedia.org/T394844#10881230 (10Dzahn) [18:17:01] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 06Data-Persistence: Request mariadb database for Zuul - https://phabricator.wikimedia.org/T394844#10881231 (10Dzahn) ` QPS: 10 - 50 Size: < 500 MB DB Name: zuul User: zuul Accessed from server (s): zuul1001.eqiad.wmnet Backup... [18:17:08] 06Release-Engineering-Team, 10Scap, 10Dumps-Generation: scap needs to be k8s-cluster aware - https://phabricator.wikimedia.org/T388761#10881232 (10Scott_French) Although changes to mediawiki-dumps-legacy will be needed before this feature can actually be put to use there (details in T389786#10881115), we wer... [18:17:17] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services, 06Data-Persistence: Request mariadb database for Zuul - https://phabricator.wikimedia.org/T394844#10881233 (10Dzahn) a:05Dzahn→03None [18:32:53] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Bitu, 06Infrastructure-Foundations: User corvus cannot activate ssh key in Bitu - https://phabricator.wikimedia.org/T395857#10881282 (10hashar) Thank you @jhathaway for the debugging and @SLyngshede-WMF for the fix :) [19:35:35] 10Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.45.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T392174#10881484 (10dduvall) [20:13:06] https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php81/19822/console seems stuck. I'm going to abort it. [20:27:15] (03PS1) 10Hashar: Switch to prepare-workspace-openshift [integration/config] (zuul3) - 10https://gerrit.wikimedia.org/r/1153348 [20:29:25] would you like me to upgrade jenkins for bullseye (contint)? this has not been execute, but it could: 'jenkins': '2.492.3' will be upgraded to '2.504.2' [20:30:04] (just noticing while trying to get jenkins for bookworm) [20:43:12] (03PS1) 10Hashar: Add directories and Ansible inventory in pre playbook [integration/config] (zuul3) - 10https://gerrit.wikimedia.org/r/1153353 [20:43:50] we now have jenkins on bookworm (for releases-jenkins) [20:44:14] and it's that 2.504.2 version [20:54:12] mutante: let's not upgrade existing hosts at this time. Looks low risk, but we usually handle that as part of security upgrades. [20:55:20] and thanks for new bookworm host with jenkins for releases (cc jnuche for your A.M.) [20:56:56] thcipriani: ACK, not touching bullseye / contint / active releases host [20:57:10] releases2003 is the upgraded host and the backup [20:57:18] <3 [20:59:14] in our APT repo: new component called 'thirdparty/jenkins' for bookworm-wikimedia [21:05:13] nice, thirdparty/ci is...no more? [21:05:22] or just undesirable? [21:06:23] " [21:06:23] This component includes the external containerd and the external docker-ce on Buster/Bullseye. These are no longer needed on Bookworm and later since Bookworm directly includes current versions of containerd and docker.io. [21:06:27] As such I think it would be better to name this less confusingly and rather call it thirdparty/jenkins going forward. [21:06:33] per ^ [21:07:03] because it's only jenkins left in it [21:07:22] gotcha [21:07:37] we also have that current version of docker.io on the new zuul VMs [21:11:34] 06Project-Admins, 06Release-Engineering-Team, 10MediaWiki-extensions-General, 10MediaWiki-General, and 2 others: Create a security pre-release Phabricator policy manageable by the Security Team - https://phabricator.wikimedia.org/T393403#10881871 (10Mstyles) You can update the View Policy by clicking o... [21:12:42] 06Project-Admins, 06Release-Engineering-Team, 10MediaWiki-extensions-General, 10MediaWiki-General, and 2 others: Create a security pre-release Phabricator policy manageable by the Security Team - https://phabricator.wikimedia.org/T393403#10881873 (10sbassett) Ah, ok, that's what I wanted to confirm. T... [21:19:50] 10Deployments, 10Release-Engineering-Team (Radar), 06serviceops, 07Wikimedia-production-error: httpb sometimes fails upon deployment with a HTTP 503 - https://phabricator.wikimedia.org/T380958#10881891 (10Mstyles) ` 21:11:43 Started check-testservers 21:11:43 Executing check 'check_testservers_baremetal-1_... [22:31:22] (03PS1) 10Hashar: Remove add-build-sshkey which was for static hosts [integration/config] (zuul3) - 10https://gerrit.wikimedia.org/r/1153378 [22:34:46] (03PS1) 10Hashar: Remove docker-run-hello-world [integration/config] (zuul3) - 10https://gerrit.wikimedia.org/r/1153379 [22:37:44] (03PS1) 10Hashar: Remove cleanup as well [integration/config] (zuul3) - 10https://gerrit.wikimedia.org/r/1153382 [23:03:47] (03PS2) 10Jforrester: jjb: Provide Node 24 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1153276 (https://phabricator.wikimedia.org/T395923) [23:07:01] (03PS3) 10Jforrester: jjb: Provide Node 24 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1153276 (https://phabricator.wikimedia.org/T395923) [23:07:13] (03CR) 10Jforrester: [C:03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/1153276 (https://phabricator.wikimedia.org/T395923) (owner: 10Jforrester) [23:08:29] (03Merged) 10jenkins-bot: jjb: Provide Node 24 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1153276 (https://phabricator.wikimedia.org/T395923) (owner: 10Jforrester) [23:17:08] (03PS1) 10Jforrester: Zuul: Provide experimental Node 24 jobs where Node 22 ones exist [integration/config] - 10https://gerrit.wikimedia.org/r/1153387 (https://phabricator.wikimedia.org/T395926) [23:17:19] 10Continuous-Integration-Infrastructure, 13Patch-For-Review: Create WMF CI images and jobs for Node.js 24 - https://phabricator.wikimedia.org/T395923#10882234 (10Jdforrester-WMF) 05Open→03Resolved a:03Jdforrester-WMF [23:18:52] (03CR) 10Jforrester: [C:03+2] Zuul: Provide experimental Node 24 jobs where Node 22 ones exist [integration/config] - 10https://gerrit.wikimedia.org/r/1153387 (https://phabricator.wikimedia.org/T395926) (owner: 10Jforrester) [23:20:06] (03Merged) 10jenkins-bot: Zuul: Provide experimental Node 24 jobs where Node 22 ones exist [integration/config] - 10https://gerrit.wikimedia.org/r/1153387 (https://phabricator.wikimedia.org/T395926) (owner: 10Jforrester) [23:20:35] !log Zuul: Provide experimental Node 24 jobs where Node 22 ones exist, for T395926 [23:20:38] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:20:38] T395926: Upgrade all CI jobs for WMF-deployed projects from Node 22 to Node 24 - https://phabricator.wikimedia.org/T395926 [23:41:15] (03CR) 10Jforrester: "check experimental" [software/gerrit] (deploy/wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1148868 (https://phabricator.wikimedia.org/T390666) (owner: 10Hashar) [23:43:38] (03PS1) 10Corvus: Add start-zuul-console to pre-run playbook [integration/config] (zuul3) - 10https://gerrit.wikimedia.org/r/1153391 [23:49:13] (03PS1) 10Jforrester: Zuul: [wikimedia/portals/deploy] Drop tests, this repo isn't testable [integration/config] - 10https://gerrit.wikimedia.org/r/1153393 [23:49:13] (03PS1) 10Jforrester: Zuul: [oojs/ui] Upgrade test suite to Node 24 and Node 22 [integration/config] - 10https://gerrit.wikimedia.org/r/1153394 (https://phabricator.wikimedia.org/T395926) [23:49:14] (03PS1) 10Jforrester: Zuul: [oojs/core] Upgrade test suite to Node 24 and Node 22 [integration/config] - 10https://gerrit.wikimedia.org/r/1153395 (https://phabricator.wikimedia.org/T395926) [23:49:15] (03PS1) 10Jforrester: Zuul: [wikimedia/portals] Upgrade test suite to Node 24 and Node 22 [integration/config] - 10https://gerrit.wikimedia.org/r/1153396 (https://phabricator.wikimedia.org/T395926) [23:49:17] (03PS1) 10Jforrester: Zuul: [wikipeg] Upgrade test suite to Node 24 and Node 22 [integration/config] - 10https://gerrit.wikimedia.org/r/1153397 (https://phabricator.wikimedia.org/T395926) [23:49:20] (03PS1) 10Jforrester: Zuul: [mediawiki/services/parsoid/testreduce] Upgrade test suite to Node 24 and Node 22 [integration/config] - 10https://gerrit.wikimedia.org/r/1153398 (https://phabricator.wikimedia.org/T395926) [23:49:21] (03PS1) 10Jforrester: Zuul: [mediawiki/services/texvcjs] Upgrade test suite to Node 24 and Node 22 [integration/config] - 10https://gerrit.wikimedia.org/r/1153399 (https://phabricator.wikimedia.org/T395926) [23:49:25] (03PS1) 10Jforrester: Zuul: [operations/software/gerrit] Upgrade test suite to Node 24 and Node 22 [integration/config] - 10https://gerrit.wikimedia.org/r/1153400 (https://phabricator.wikimedia.org/T395926) [23:50:38] (03CR) 10CI reject: [V:04-1] Zuul: [wikimedia/portals] Upgrade test suite to Node 24 and Node 22 [integration/config] - 10https://gerrit.wikimedia.org/r/1153396 (https://phabricator.wikimedia.org/T395926) (owner: 10Jforrester) [23:50:48] (03CR) 10CI reject: [V:04-1] Zuul: [mediawiki/services/texvcjs] Upgrade test suite to Node 24 and Node 22 [integration/config] - 10https://gerrit.wikimedia.org/r/1153399 (https://phabricator.wikimedia.org/T395926) (owner: 10Jforrester) [23:50:48] (03CR) 10CI reject: [V:04-1] Zuul: [mediawiki/services/parsoid/testreduce] Upgrade test suite to Node 24 and Node 22 [integration/config] - 10https://gerrit.wikimedia.org/r/1153398 (https://phabricator.wikimedia.org/T395926) (owner: 10Jforrester) [23:51:06] (03CR) 10CI reject: [V:04-1] Zuul: [operations/software/gerrit] Upgrade test suite to Node 24 and Node 22 [integration/config] - 10https://gerrit.wikimedia.org/r/1153400 (https://phabricator.wikimedia.org/T395926) (owner: 10Jforrester) [23:51:17] (03CR) 10CI reject: [V:04-1] Zuul: [wikipeg] Upgrade test suite to Node 24 and Node 22 [integration/config] - 10https://gerrit.wikimedia.org/r/1153397 (https://phabricator.wikimedia.org/T395926) (owner: 10Jforrester) [23:51:21] (03CR) 10Jforrester: [C:03+2] Zuul: [wikimedia/portals/deploy] Drop tests, this repo isn't testable [integration/config] - 10https://gerrit.wikimedia.org/r/1153393 (owner: 10Jforrester) [23:52:34] (03Merged) 10jenkins-bot: Zuul: [wikimedia/portals/deploy] Drop tests, this repo isn't testable [integration/config] - 10https://gerrit.wikimedia.org/r/1153393 (owner: 10Jforrester) [23:52:58] (03PS2) 10Jforrester: Zuul: [wikimedia/portals] Upgrade test suite to Node 24 and Node 22 [integration/config] - 10https://gerrit.wikimedia.org/r/1153396 (https://phabricator.wikimedia.org/T395926) [23:52:58] (03PS2) 10Jforrester: Zuul: [wikipeg] Upgrade test suite to Node 24 and Node 22 [integration/config] - 10https://gerrit.wikimedia.org/r/1153397 (https://phabricator.wikimedia.org/T395926) [23:52:58] (03PS2) 10Jforrester: Zuul: [mediawiki/services/parsoid/testreduce] Upgrade test suite to Node 24 and Node 22 [integration/config] - 10https://gerrit.wikimedia.org/r/1153398 (https://phabricator.wikimedia.org/T395926) [23:52:58] (03PS2) 10Jforrester: Zuul: [mediawiki/services/texvcjs] Upgrade test suite to Node 24 and Node 22 [integration/config] - 10https://gerrit.wikimedia.org/r/1153399 (https://phabricator.wikimedia.org/T395926) [23:52:59] (03PS2) 10Jforrester: Zuul: [operations/software/gerrit] Upgrade test suite to Node 24 and Node 22 [integration/config] - 10https://gerrit.wikimedia.org/r/1153400 (https://phabricator.wikimedia.org/T395926) [23:53:08] !log Zuul: [wikimedia/portals/deploy] Drop tests, this repo isn't testable [23:53:09] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:53:19] (03CR) 10Jforrester: [C:03+2] Zuul: [oojs/ui] Upgrade test suite to Node 24 and Node 22 [integration/config] - 10https://gerrit.wikimedia.org/r/1153394 (https://phabricator.wikimedia.org/T395926) (owner: 10Jforrester) [23:53:22] (03CR) 10Jforrester: [C:03+2] Zuul: [oojs/core] Upgrade test suite to Node 24 and Node 22 [integration/config] - 10https://gerrit.wikimedia.org/r/1153395 (https://phabricator.wikimedia.org/T395926) (owner: 10Jforrester) [23:54:22] (03CR) 10CI reject: [V:04-1] Zuul: [mediawiki/services/parsoid/testreduce] Upgrade test suite to Node 24 and Node 22 [integration/config] - 10https://gerrit.wikimedia.org/r/1153398 (https://phabricator.wikimedia.org/T395926) (owner: 10Jforrester) [23:54:38] (03CR) 10Jforrester: [C:03+2] Zuul: [wikimedia/portals] Upgrade test suite to Node 24 and Node 22 [integration/config] - 10https://gerrit.wikimedia.org/r/1153396 (https://phabricator.wikimedia.org/T395926) (owner: 10Jforrester) [23:54:42] (03CR) 10Jforrester: [C:03+2] Zuul: [wikipeg] Upgrade test suite to Node 24 and Node 22 [integration/config] - 10https://gerrit.wikimedia.org/r/1153397 (https://phabricator.wikimedia.org/T395926) (owner: 10Jforrester) [23:55:24] (03Merged) 10jenkins-bot: Zuul: [oojs/ui] Upgrade test suite to Node 24 and Node 22 [integration/config] - 10https://gerrit.wikimedia.org/r/1153394 (https://phabricator.wikimedia.org/T395926) (owner: 10Jforrester) [23:55:26] (03Merged) 10jenkins-bot: Zuul: [oojs/core] Upgrade test suite to Node 24 and Node 22 [integration/config] - 10https://gerrit.wikimedia.org/r/1153395 (https://phabricator.wikimedia.org/T395926) (owner: 10Jforrester) [23:55:50] !log Zuul: [oojs/*i] Upgrade test suite to Node 24 and Node 22, for T395926 [23:55:52] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:55:53] T395926: Upgrade all CI jobs for WMF-deployed projects from Node 22 to Node 24 - https://phabricator.wikimedia.org/T395926 [23:55:54] (03Merged) 10jenkins-bot: Zuul: [wikimedia/portals] Upgrade test suite to Node 24 and Node 22 [integration/config] - 10https://gerrit.wikimedia.org/r/1153396 (https://phabricator.wikimedia.org/T395926) (owner: 10Jforrester) [23:55:55] (03Merged) 10jenkins-bot: Zuul: [wikipeg] Upgrade test suite to Node 24 and Node 22 [integration/config] - 10https://gerrit.wikimedia.org/r/1153397 (https://phabricator.wikimedia.org/T395926) (owner: 10Jforrester) [23:56:10] !log Zuul: [wikipeg] Upgrade test suite to Node 24 and Node 22, for T395926 [23:56:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:56:30] !log Zuul: [wikimedia/portals] Upgrade test suite to Node 24 and Node 22, for T395926 [23:56:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:57:11] (03PS3) 10Jforrester: Zuul: [mediawiki/services/parsoid/testreduce] Upgrade test suite to Node 24 & 22 [integration/config] - 10https://gerrit.wikimedia.org/r/1153398 (https://phabricator.wikimedia.org/T395926) [23:57:11] (03PS3) 10Jforrester: Zuul: [mediawiki/services/texvcjs] Upgrade test suite to Node 24 and Node 22 [integration/config] - 10https://gerrit.wikimedia.org/r/1153399 (https://phabricator.wikimedia.org/T395926) [23:57:11] (03PS3) 10Jforrester: Zuul: [operations/software/gerrit] Upgrade test suite to Node 24 and Node 22 [integration/config] - 10https://gerrit.wikimedia.org/r/1153400 (https://phabricator.wikimedia.org/T395926) [23:57:24] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/services/parsoid/testreduce] Upgrade test suite to Node 24 & 22 [integration/config] - 10https://gerrit.wikimedia.org/r/1153398 (https://phabricator.wikimedia.org/T395926) (owner: 10Jforrester) [23:57:27] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/services/texvcjs] Upgrade test suite to Node 24 and Node 22 [integration/config] - 10https://gerrit.wikimedia.org/r/1153399 (https://phabricator.wikimedia.org/T395926) (owner: 10Jforrester) [23:57:31] (03CR) 10Jforrester: [C:03+2] Zuul: [operations/software/gerrit] Upgrade test suite to Node 24 and Node 22 [integration/config] - 10https://gerrit.wikimedia.org/r/1153400 (https://phabricator.wikimedia.org/T395926) (owner: 10Jforrester) [23:59:34] (03Merged) 10jenkins-bot: Zuul: [mediawiki/services/parsoid/testreduce] Upgrade test suite to Node 24 & 22 [integration/config] - 10https://gerrit.wikimedia.org/r/1153398 (https://phabricator.wikimedia.org/T395926) (owner: 10Jforrester) [23:59:35] (03Merged) 10jenkins-bot: Zuul: [mediawiki/services/texvcjs] Upgrade test suite to Node 24 and Node 22 [integration/config] - 10https://gerrit.wikimedia.org/r/1153399 (https://phabricator.wikimedia.org/T395926) (owner: 10Jforrester) [23:59:36] (03Merged) 10jenkins-bot: Zuul: [operations/software/gerrit] Upgrade test suite to Node 24 and Node 22 [integration/config] - 10https://gerrit.wikimedia.org/r/1153400 (https://phabricator.wikimedia.org/T395926) (owner: 10Jforrester) [23:59:39] !log Zuul: [mediawiki/services/] Upgrade test suite to Node 24 & 22, for T395926 [23:59:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL