[03:54:51] (03PS1) 10Zoranzoki21: Zuul: [mediawiki/extensions/SemanticDependency] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/956093 (https://phabricator.wikimedia.org/T342002) [06:50:51] hashar: cool, thanks for looking into it! [07:33:58] (03PS2) 10Hashar: dockerfiles: align shellcheck image version with package [integration/config] - 10https://gerrit.wikimedia.org/r/956083 [07:34:07] (03CR) 10Hashar: [C: 03+2] dockerfiles: align shellcheck image version with package [integration/config] - 10https://gerrit.wikimedia.org/r/956083 (owner: 10Hashar) [07:35:38] (03Merged) 10jenkins-bot: dockerfiles: align shellcheck image version with package [integration/config] - 10https://gerrit.wikimedia.org/r/956083 (owner: 10Hashar) [07:37:11] (03PS2) 10Hashar: dockerfiles: shellcheck: upgrade to 0.8.0 [integration/config] - 10https://gerrit.wikimedia.org/r/956084 [07:37:51] (03CR) 10Hashar: [C: 03+2] "Successfully published image docker-registry.discovery.wmnet/releng/shellcheck:0.5.0-1" [integration/config] - 10https://gerrit.wikimedia.org/r/956083 (owner: 10Hashar) [07:37:59] (03CR) 10Hashar: [C: 03+2] dockerfiles: shellcheck: upgrade to 0.8.0 [integration/config] - 10https://gerrit.wikimedia.org/r/956084 (owner: 10Hashar) [07:39:30] (03Merged) 10jenkins-bot: dockerfiles: shellcheck: upgrade to 0.8.0 [integration/config] - 10https://gerrit.wikimedia.org/r/956084 (owner: 10Hashar) [07:42:41] (03CR) 10Hashar: [C: 03+2] "Successfully published image docker-registry.discovery.wmnet/releng/shellcheck:0.8.0-1" [integration/config] - 10https://gerrit.wikimedia.org/r/956084 (owner: 10Hashar) [07:44:05] (03PS1) 10Hashar: jjb: update shellcheck job to use v0.8.0 [integration/config] - 10https://gerrit.wikimedia.org/r/956362 [07:44:29] (03CR) 10Hashar: [C: 03+2] jjb: update shellcheck job to use v0.8.0 [integration/config] - 10https://gerrit.wikimedia.org/r/956362 (owner: 10Hashar) [07:45:33] (03Merged) 10jenkins-bot: jjb: update shellcheck job to use v0.8.0 [integration/config] - 10https://gerrit.wikimedia.org/r/956362 (owner: 10Hashar) [08:31:24] 10Release-Engineering-Team (Priority Backlog 📥), 10Scap, 10Patch-For-Review: scap backport: prompt before merging if another deploy is ongoing - https://phabricator.wikimedia.org/T345247 (10CodeReviewBot) jnuche closed https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/27 backport tests: a... [08:38:39] 10GitLab (Pipeline Services Migration🐤), 10collaboration-services, 10Patch-For-Review: Move micro sites from Ganeti to Kubernetes and from Gerrit to GitLab - https://phabricator.wikimedia.org/T300171 (10Jelto) [08:54:32] (03PS2) 10Hashar: jjb: add editor-analytics [integration/config] - 10https://gerrit.wikimedia.org/r/955953 (https://phabricator.wikimedia.org/T336391) (owner: 10Hnowlan) [08:54:34] (03PS2) 10Hashar: zuul: add editor-analytics [integration/config] - 10https://gerrit.wikimedia.org/r/955948 (https://phabricator.wikimedia.org/T336391) (owner: 10Hnowlan) [08:55:02] (03CR) 10Hashar: [C: 03+2] "INFO:jenkins_jobs.builder:Number of jobs generated: 6" [integration/config] - 10https://gerrit.wikimedia.org/r/955953 (https://phabricator.wikimedia.org/T336391) (owner: 10Hnowlan) [08:56:10] (03Merged) 10jenkins-bot: jjb: add editor-analytics [integration/config] - 10https://gerrit.wikimedia.org/r/955953 (https://phabricator.wikimedia.org/T336391) (owner: 10Hnowlan) [08:59:55] (03CR) 10Hashar: [C: 03+2] zuul: add editor-analytics [integration/config] - 10https://gerrit.wikimedia.org/r/955948 (https://phabricator.wikimedia.org/T336391) (owner: 10Hnowlan) [09:01:54] (03Merged) 10jenkins-bot: zuul: add editor-analytics [integration/config] - 10https://gerrit.wikimedia.org/r/955948 (https://phabricator.wikimedia.org/T336391) (owner: 10Hnowlan) [09:03:19] !log Reloaded Zuul for "add editor-analytics" https://gerrit.wikimedia.org/r/c/integration/config/+/955948/ | T336391 [09:03:21] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:03:21] T336391: AQS 2.0: Page Analytics Service: Deploy to staging and production - https://phabricator.wikimedia.org/T336391 [09:52:15] (03CR) 10Hashar: [C: 03+2] Zuul: [mediawiki/extensions/SemanticDependency] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/956093 (https://phabricator.wikimedia.org/T342002) (owner: 10Zoranzoki21) [09:53:22] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/SemanticDependency] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/956093 (https://phabricator.wikimedia.org/T342002) (owner: 10Zoranzoki21) [09:55:16] 10GitLab (Infrastructure), 10collaboration-services: Switchover gitlab-replica (gitlab2002 -> gitlab1003) - September 2023 - https://phabricator.wikimedia.org/T345590 (10Jelto) [09:56:01] (03Abandoned) 10Michael Große: Run EntitySchema CI on master once a day [integration/config] - 10https://gerrit.wikimedia.org/r/948558 (https://phabricator.wikimedia.org/T333660) (owner: 10Michael Große) [09:56:09] (03CR) 10Michael Große: Run EntitySchema CI on master once a day (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/948558 (https://phabricator.wikimedia.org/T333660) (owner: 10Michael Große) [10:32:18] 10Continuous-Integration-Config, 10Browser-Tests, 10Ruby, 10User-zeljkofilipin: Drop Ruby Selenium CI jobs; we don't support them any more - https://phabricator.wikimedia.org/T220035 (10Jdforrester-WMF) 05Stalled→03Resolved a:03Jdforrester-WMF [11:29:23] (03CR) 10Hashar: [C: 03+2] utils/jjb-diff.sh: emit a changelog for JJB changes [integration/config] - 10https://gerrit.wikimedia.org/r/955971 (owner: 10Hashar) [11:31:33] (03Merged) 10jenkins-bot: utils/jjb-diff.sh: emit a changelog for JJB changes [integration/config] - 10https://gerrit.wikimedia.org/r/955971 (owner: 10Hashar) [11:31:40] 10GitLab (Pipeline Services Migration🐤), 10Math, 10Mathoid: Migrate Mathoid to GitLab - https://phabricator.wikimedia.org/T344747 (10Jdforrester-WMF) OK, this is ready to do, but I don't have the rights to import into mediawiki/services for some reason. @thcipriani, do you want to do so? [11:42:29] 10GitLab (Infrastructure), 10collaboration-services: Switchover gitlab-replica (gitlab2002 -> gitlab1003) - September 2023 - https://phabricator.wikimedia.org/T345590 (10Jelto) [11:57:44] 10Phabricator: Implement "your turn to comment" feature for Phabricator tasks - https://phabricator.wikimedia.org/T346034 (10kostajh) [12:15:07] 10Project-Admins, 10Campaign-Tools: Create project tag for Event-Discovery - https://phabricator.wikimedia.org/T345938 (10Aklapper) 05Open→03Resolved a:03Aklapper @ifried: Hi! For future reference, please follow the link on https://www.mediawiki.org/wiki/Phabricator/Creating_and_renaming_projects (linked... [12:16:13] 10Phabricator (Upstream): Implement "your turn to comment" feature for Phabricator tasks after a username @mention - https://phabricator.wikimedia.org/T346034 (10Aklapper) [12:16:23] 10Phabricator (Upstream): Implement "your turn to comment" feature for Phabricator tasks after a username @mention - https://phabricator.wikimedia.org/T346034 (10Aklapper) [12:16:43] (03CR) 10Hashar: [C: 03+2] Drop --run=all in favor of an empty list [integration/quibble] - 10https://gerrit.wikimedia.org/r/903255 (owner: 10Hashar) [12:24:02] !log Upgrading pip to 23.2.1 in tox jobs | https://gerrit.wikimedia.org/r/c/integration/config/+/955919 | T345607 [12:24:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:24:05] T345607: tox 4 no more support [tox:jenkins] configuration section - https://phabricator.wikimedia.org/T345607 [12:24:10] (03CR) 10Hashar: [C: 03+2] jjb: switch tox jobs to pip v23.2.1 [integration/config] - 10https://gerrit.wikimedia.org/r/955919 (https://phabricator.wikimedia.org/T345607) (owner: 10Hashar) [12:25:16] (03CR) 10CI reject: [V: 04-1] jjb: switch tox jobs to pip v23.2.1 [integration/config] - 10https://gerrit.wikimedia.org/r/955919 (https://phabricator.wikimedia.org/T345607) (owner: 10Hashar) [12:28:50] !log Rolled back upgrade 23.2.1 for tox jobs | https://gerrit.wikimedia.org/r/c/integration/config/+/955919 | T345607 [12:28:52] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:34:37] (03CR) 10CI reject: [V: 04-1] Drop --run=all in favor of an empty list [integration/quibble] - 10https://gerrit.wikimedia.org/r/903255 (owner: 10Hashar) [12:47:09] Project beta-scap-sync-world build #120183: 04FAILURE in 1 min 59 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/120183/ [12:52:21] Project beta-scap-sync-world build #120184: 04STILL FAILING in 3 min 22 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/120184/ [12:52:45] (03PS1) 10Hashar: dockerfiles: tox: install virtualenv==20.21.1 [integration/config] - 10https://gerrit.wikimedia.org/r/956430 (https://phabricator.wikimedia.org/T345695) [12:53:04] (03CR) 10Hashar: [C: 03+2] dockerfiles: tox: install virtualenv==20.21.1 [integration/config] - 10https://gerrit.wikimedia.org/r/956430 (https://phabricator.wikimedia.org/T345695) (owner: 10Hashar) [12:54:49] (03Merged) 10jenkins-bot: dockerfiles: tox: install virtualenv==20.21.1 [integration/config] - 10https://gerrit.wikimedia.org/r/956430 (https://phabricator.wikimedia.org/T345695) (owner: 10Hashar) [12:56:00] 10Release-Engineering-Team (Radar), 10Browser-Tests: wdio-cucumber-framework fails on NodeJS due to fibers@2.x under nodejs10 - https://phabricator.wikimedia.org/T229033 (10zeljkofilipin) [12:58:32] Project beta-scap-sync-world build #120185: 04STILL FAILING in 3 min 22 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/120185/ [12:58:43] (03PS2) 10Hashar: jjb: tox jobs to pip v23.2.1 virtualenv 20.21.1 [integration/config] - 10https://gerrit.wikimedia.org/r/955919 (https://phabricator.wikimedia.org/T345695) [13:00:50] (03PS2) 10Hashar: jjb: zuul-cloner job to pip v23.2.1 virtualenv 20.21.1 [integration/config] - 10https://gerrit.wikimedia.org/r/955920 (https://phabricator.wikimedia.org/T345695) [13:01:43] 12:47:07 12:47:07 sudo -u mwdeploy -n -- /usr/bin/rsync -l deployment-deploy03.deployment-prep.eqiad1.wikimedia.cloud::common/wikiversions*.{json,php} /srv/mediawiki (ran as mwdeploy@deployment-mediawiki12.deployment-prep.eqiad1.wikimedia.cloud) returned [10]: rsync: getaddrinfo: deployment-deploy03.deployment-prep.eqiad1.wikimedia.cloud 873: Temporary failure in name resolution [13:01:49] I guess something is broken on WMCS infra :) [13:02:19] the other builds fail similarly Temporary failure in name resolution [13:02:21] internal DNS switchover I guess [13:02:34] it is always DNS :D [13:07:14] Project beta-scap-sync-world build #120186: 04STILL FAILING in 2 min 8 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/120186/ [13:16:21] Project beta-scap-sync-world build #120187: 04STILL FAILING in 1 min 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/120187/ [13:26:13] Yippee, build fixed! [13:26:14] Project beta-scap-sync-world build #120188: 09FIXED in 1 min 7 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/120188/ [13:36:26] 10Release-Engineering-Team (Priority Backlog 📥), 10Scap, 10Patch-For-Review: scap backport: prompt before merging if another deploy is ongoing - https://phabricator.wikimedia.org/T345247 (10CodeReviewBot) jnuche merged https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/179 backport: serialize c... [13:40:38] 10Release-Engineering-Team (Priority Backlog 📥), 10Scap, 10Patch-For-Review: scap backport: prompt before merging if another deploy is ongoing - https://phabricator.wikimedia.org/T345247 (10CodeReviewBot) jnuche opened https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/180 backport tests: add t... [13:43:11] (03CR) 10Hashar: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/955920 (https://phabricator.wikimedia.org/T345695) (owner: 10Hashar) [13:43:46] !log Upgrading tox jobs to pip 23.2.1 and virtualenv 20.21.1 | T345695 [13:43:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:43:49] T345695: Deal with tox 4 upgrading - https://phabricator.wikimedia.org/T345695 [13:45:50] (03CR) 10Hashar: [C: 03+2] jjb: zuul-cloner job to pip v23.2.1 virtualenv 20.21.1 [integration/config] - 10https://gerrit.wikimedia.org/r/955920 (https://phabricator.wikimedia.org/T345695) (owner: 10Hashar) [13:46:18] (03CR) 10Hashar: [C: 03+2] jjb: tox jobs to pip v23.2.1 virtualenv 20.21.1 [integration/config] - 10https://gerrit.wikimedia.org/r/955919 (https://phabricator.wikimedia.org/T345695) (owner: 10Hashar) [13:48:26] (03Merged) 10jenkins-bot: jjb: tox jobs to pip v23.2.1 virtualenv 20.21.1 [integration/config] - 10https://gerrit.wikimedia.org/r/955919 (https://phabricator.wikimedia.org/T345695) (owner: 10Hashar) [13:48:29] (03Merged) 10jenkins-bot: jjb: zuul-cloner job to pip v23.2.1 virtualenv 20.21.1 [integration/config] - 10https://gerrit.wikimedia.org/r/955920 (https://phabricator.wikimedia.org/T345695) (owner: 10Hashar) [13:57:10] 10Phabricator, 10Dumps-Generation: Compress phabricator dump - https://phabricator.wikimedia.org/T262744 (10jnuche) I think a regular scap deployment should do the trick here. @brennen can you confirm? I can take care of it if that's the case. [14:00:49] (03PS8) 10Hashar: dockerfiles: upgrade tox to 4.8.0 [integration/config] - 10https://gerrit.wikimedia.org/r/955397 (https://phabricator.wikimedia.org/T345695) [14:05:13] 10Phabricator (Upstream), 10Upstream: Visually indicate when a Phabricator user is new (Welcome culture) - https://phabricator.wikimedia.org/T64324 (10Aklapper) p:05Lowest→03Low a:03Aklapper I wrote a quick Proof of Concept patch in P52415 which needs further clean up but is probably the most minimal pos... [14:07:18] (03CR) 10Hashar: [C: 03+2] "Tested against integration/config.git (which requires a python2.7 venv) and creating virtualenv with the locally built image:" [integration/config] - 10https://gerrit.wikimedia.org/r/955397 (https://phabricator.wikimedia.org/T345695) (owner: 10Hashar) [14:09:00] (03Merged) 10jenkins-bot: dockerfiles: upgrade tox to 4.8.0 [integration/config] - 10https://gerrit.wikimedia.org/r/955397 (https://phabricator.wikimedia.org/T345695) (owner: 10Hashar) [14:12:46] 10Phabricator, 10Dumps-Generation: Compress phabricator dump - https://phabricator.wikimedia.org/T262744 (10brennen) > @brennen can you confirm? I can take care of it if that's the case. That'd do the trick, assuming that the tools submodule is updated on wmf/stable [[https://gitlab.wikimedia.org/repos/phabri... [14:13:49] !log Building Docker images for upgrading tox to 4.8.0 | https://gerrit.wikimedia.org/r/c/integration/config/+/955397/ | T345695 | T345607 [14:13:52] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:13:53] T345695: Deal with tox 4 upgrading - https://phabricator.wikimedia.org/T345695 [14:13:53] T345607: tox 4 no more support [tox:jenkins] configuration section - https://phabricator.wikimedia.org/T345607 [14:21:46] 10Beta-Cluster-Infrastructure, 10Dumps-Generation, 10MediaWiki-ContentHandler, 10Patch-For-Review: XMLDumps broken on deployment-mwmaint02 due to Jade Extension related content - https://phabricator.wikimedia.org/T345874 (10Milimetric) @Ladsgroup: would you prefer that to the settings change? I'm happy to... [14:45:35] (03CR) 10Hashar: [C: 03+2] "Successfully published image docker-registry.discovery.wmnet/releng/tox-mysqld:4.8.0-1" [integration/config] - 10https://gerrit.wikimedia.org/r/955397 (https://phabricator.wikimedia.org/T345695) (owner: 10Hashar) [14:47:13] 10Continuous-Integration-Config, 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review: Deal with tox 4 upgrading - https://phabricator.wikimedia.org/T345695 (10hashar) The releng CI images having tox 4.8.0 are: | docker-registry.wikimedia.org/releng/tox-mysqld:4.8.0-1 | docker-registry.wikime... [14:47:28] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Patch-For-Review, 10Upstream: tox 4 no more support [tox:jenkins] configuration section - https://phabricator.wikimedia.org/T345607 (10hashar) The releng CI images having tox 4.8.0 and support for `tox:jenkins` tox configuration section... [14:48:20] 10Continuous-Integration-Config, 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review: Deal with tox 4 upgrading - https://phabricator.wikimedia.org/T345695 (10hashar) [14:48:48] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Patch-For-Review, 10Upstream: tox 4 no more support [tox:jenkins] configuration section - https://phabricator.wikimedia.org/T345607 (10hashar) 05Open→03Resolved a:03hashar Solved by the creation of a tox plugin: https://gerrit.wik... [14:50:52] (03PS1) 10Hashar: jjb: update integration/config to tox 4.8.0 [integration/config] - 10https://gerrit.wikimedia.org/r/956453 (https://phabricator.wikimedia.org/T345695) [14:53:27] (03CR) 10Hashar: [C: 03+2] "I have updated the jobs before sending this change to verify that was working :)" [integration/config] - 10https://gerrit.wikimedia.org/r/956453 (https://phabricator.wikimedia.org/T345695) (owner: 10Hashar) [14:55:08] (03Merged) 10jenkins-bot: jjb: update integration/config to tox 4.8.0 [integration/config] - 10https://gerrit.wikimedia.org/r/956453 (https://phabricator.wikimedia.org/T345695) (owner: 10Hashar) [15:06:38] 10Release-Engineering-Team (Priority Backlog 📥), 10Scap, 10Patch-For-Review: scap backport: prompt before merging if another deploy is ongoing - https://phabricator.wikimedia.org/T345247 (10CodeReviewBot) jnuche merged https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/180 backport tests: add t... [15:25:13] 10Release-Engineering-Team (Priority Backlog 📥), 10Scap: scap backport: prompt before merging if another deploy is ongoing - https://phabricator.wikimedia.org/T345247 (10jnuche) 05Open→03Resolved Deployed to prod. Scap should now let the deployer know about concurrent backports before the changes are merged. [15:37:19] 10GitLab (Pipeline Services Migration🐤), 10Math, 10Mathoid: Migrate Mathoid to GitLab - https://phabricator.wikimedia.org/T344747 (10dancy) @Jdforrester-WMF I bumped your role to `Owner` in the repos/mediawiki tree. Lemme know if that gets you moving. [15:59:06] (03CR) 10Krinkle: "recheck - not sure if the Flow/AbuseFilter failure is related." [integration/quibble] - 10https://gerrit.wikimedia.org/r/952016 (owner: 10Krinkle) [16:10:51] 10GitLab (Pipeline Services Migration🐤), 10Math, 10Mathoid, 10Patch-For-Review: Migrate Mathoid to GitLab - https://phabricator.wikimedia.org/T344747 (10CodeReviewBot) jforrester opened https://gitlab.wikimedia.org/repos/mediawiki/services/mathoid/-/merge_requests/1 build: Install standard node 16 CI for... [16:19:25] PROBLEM - Check systemd state on doc2002 is CRITICAL: CRITICAL - degraded: The following units failed: rsync-doc-host-data-sync.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [16:19:26] 10GitLab (Pipeline Services Migration🐤), 10Math, 10Mathoid, 10Patch-For-Review: Migrate Mathoid to GitLab - https://phabricator.wikimedia.org/T344747 (10Jdforrester-WMF) >>! In T344747#9157033, @dancy wrote: > @Jdforrester-WMF I bumped your role to `Owner` in the repos/mediawiki tree. Lemme know if that g... [16:20:01] The GitLab runners in WMCS are experiencing some networking issues and are unreachable at the moment. We are looking into this but may not resolve the problem today. The default for untrusted loads is to use public cloud runners so this should only impact jobs explicitly set to use WMCS runners. Follow https://phabricator.wikimedia.org/T346060 for updates. [16:21:09] sobanski: maybe worth wikitech-l? [16:35:18] am I right in thinking scap doesn't work on bookworm yet? context is T344136 where me and denisse are upgrading netmon hosts to bookworm [16:35:19] T344136: Upgrade LibreNMS to 23.7.0 or higher - https://phabricator.wikimedia.org/T344136 [16:37:40] dancy jnuche ^ rings a bell? [16:39:34] to be clear, the failure ATM is in bootstrapping the scap target [16:39:36] e.g [16:39:37] 2023-09-11T16:26:42.676000+00:00 netmon2002 puppet-agent[1667]: (/Stage[main]/Scap/Exec[bootstrap-scap-target]/returns) rsync: link_stat "/scap-wheels/bookworm" (in scap-install-staging) failed: No such file or directory (2) [16:40:33] (03CR) 10Hashar: Remove MW_QUIBBLE_CI env variable (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/952016 (owner: 10Krinkle) [16:41:12] (03CR) 10Hashar: [C: 03+2] "The AbuseFilter/Flow test failure is certainly unrelated (and happens on another change https://gerrit.wikimedia.org/r/c/integration/quibb" [integration/quibble] - 10https://gerrit.wikimedia.org/r/903255 (owner: 10Hashar) [16:42:39] godog: there is no scap build for bookworm, it is thus not supported [16:42:44] (I checked the source) [16:43:22] I guess you can file it as a task against #scap in Phabricator with the above message error [16:43:43] thank you hashar, I'll do that [16:43:59] it is probably not that complicated to add bookworm support (with some gotcha such as validating scap works with python3.11) [16:44:12] definitely surprising to me we're the first with scap+bookworm, and I guess someone has to be [16:44:27] I don't quite know hwo scap is deployed in production, but that involves a build step in CI with the target OS [16:44:36] godog: That should be something easy to remedy, especially if you're willing to help test! [16:44:45] and bookworm does not show up in the scap source code [16:45:00] ah here is the scap wizard :-] [16:45:27] I am off for dinner etc & [16:45:33] dancy: thank you <3 definitely willing to help [16:45:36] enjoy hashar [16:45:40] dancy: filing task ATM [16:45:49] 🏆 [16:45:59] OK. No need to include the error message. It's not unexpected. [16:46:10] cheers, even simpler [16:46:40] 10Scap: Add bookworm support/build to scap - https://phabricator.wikimedia.org/T346071 (10fgiunchedi) [16:46:45] ^ [16:46:52] Thanks for fielding the question hashar! [16:48:20] 10Release-Engineering-Team, 10Scap: Add bookworm support/build to scap - https://phabricator.wikimedia.org/T346071 (10dancy) p:05Triage→03Low a:03dancy [16:48:41] 10Release-Engineering-Team, 10Scap: Add bookworm support/build to scap - https://phabricator.wikimedia.org/T346071 (10dancy) @jnuche I'll take the first stab at this. [16:51:58] RhinosF1: I guess there's no harm in doing that, thanks for the suggestion [16:52:32] (Queue (Jenkins jobs + Zuul functions) alert) firing: - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [17:00:48] PROBLEM - Work requests waiting in Zuul Gearman server on contint2002 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [400.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [17:02:23] 10Release-Engineering-Team, 10Scap: Add bookworm support/build to scap - https://phabricator.wikimedia.org/T346071 (10jnuche) @dancy thanks! [17:02:32] (Queue (Jenkins jobs + Zuul functions) alert) resolved: - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [17:03:51] (03CR) 10CI reject: [V: 04-1] Drop --run=all in favor of an empty list [integration/quibble] - 10https://gerrit.wikimedia.org/r/903255 (owner: 10Hashar) [17:12:26] RECOVERY - Work requests waiting in Zuul Gearman server on contint2002 is OK: OK: Less than 100.00% above the threshold [200.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [17:13:02] RECOVERY - Check systemd state on doc2002 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [17:37:07] 10Beta-Cluster-Infrastructure, 10Dumps-Generation, 10MediaWiki-ContentHandler, 10Patch-For-Review: XMLDumps broken on deployment-mwmaint02 due to Jade Extension related content - https://phabricator.wikimedia.org/T345874 (10Ladsgroup) I was wondering if we could simply change the content model to json for... [17:46:12] hi releng! I'd like to update the tests for fundraising-civicrm to run against php7.4 rather than 7.3 [17:46:22] so we can get e.g. this patch passing: https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/956480/ [17:47:13] I see there's a wikimedia-fundraising-civicrm-bullseye-docker job defined in wm-fundraising.yaml [17:49:20] so I'll just try to edit the layout.yaml to make that one run in test & gate-and-submit [17:51:03] ok, also noted that there were some issues with mariadb on the bullseye tests as per https://phabricator.wikimedia.org/T307178 [17:51:17] running check experimental now to see if those are fixed [18:03:26] 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments: 1.41.0-wmf.26 deployment blockers - https://phabricator.wikimedia.org/T343728 (10matmarex) [18:05:33] ok, darn, looks like we still need to fix something there: /src/wikimedia/fundraising/crm/bin/ci-create-dbs.sh [18:05:36] 14:00:00 ERROR 1698 (28000): Access denied for user 'root'@'localhost' [18:28:23] 10Phabricator, 10Release-Engineering-Team (Radar), 10DBA, 10Data-Persistence-Backup, and 2 others: phabricator->phorge migration - database handling - https://phabricator.wikimedia.org/T335080 (10brennen) Good to close this one? [18:29:02] 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments: 1.41.0-wmf.26 deployment blockers - https://phabricator.wikimedia.org/T343728 (10matmarex) [18:30:05] 10Beta-Cluster-Infrastructure, 10MediaWiki-General, 10VisualEditor, 10VisualEditor-MediaWiki, 10Patch-For-Review: Can't save edits using VisualEditor on beta cluster: "Exception caught: Provided specification is not an array." - https://phabricator.wikimedia.org/T346080 (10matmarex) (Turns out to be just... [18:30:52] 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments, 10User-brennen: 1.41.0-wmf.27 deployment blockers - https://phabricator.wikimedia.org/T345888 (10brennen) [18:31:13] 10GitLab (Integrations), 10GitLab-Test, 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), and 2 others: Experiment with GitLab-Phabricator integration - https://phabricator.wikimedia.org/T265617 (10brennen) a:05brennen→03None [18:38:06] 10GitLab (Pipeline Services Migration🐤), 10Math, 10Mathoid, 10Patch-For-Review: Migrate Mathoid to GitLab - https://phabricator.wikimedia.org/T344747 (10CodeReviewBot) physikerwelt merged https://gitlab.wikimedia.org/repos/mediawiki/services/mathoid/-/merge_requests/1 build: Install standard node 16 CI fo... [18:38:12] 10Continuous-Integration-Infrastructure, 10Wikimedia-Fundraising-CiviCRM, 10Patch-For-Review: CiviCRM CI jobs fails when migrating from Stretch to Bullseye - https://phabricator.wikimedia.org/T307178 (10Ejegg) Looks like all of those innodb_ variables being set in ci-create-dbs are being set to the default v... [18:53:43] 10GitLab (Pipeline Services Migration🐤), 10Math, 10Mathoid, 10Patch-For-Review: Migrate Mathoid to GitLab - https://phabricator.wikimedia.org/T344747 (10CodeReviewBot) dancy opened https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/43 Add repos/mediawiki/services/mathoid [18:55:54] 10GitLab (Pipeline Services Migration🐤), 10Math, 10Mathoid, 10Patch-For-Review: Migrate Mathoid to GitLab - https://phabricator.wikimedia.org/T344747 (10CodeReviewBot) dancy merged https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/43 Add repos/mediawiki/services/mathoid [18:56:46] 10Beta-Cluster-Infrastructure, 10MediaWiki-General, 10VisualEditor, 10VisualEditor-MediaWiki: Can't save edits using VisualEditor on beta cluster: "Exception caught: Provided specification is not an array." - https://phabricator.wikimedia.org/T346080 (10matmarex) 05Open→03Resolved [19:11:58] 10Project-Admins, 10Campaign-Tools: Create project tag for Event-Discovery - https://phabricator.wikimedia.org/T345938 (10ifried) Thank you, @Aklapper! [19:21:04] 10Release-Engineering-Team, 10Release Pipeline (Blubber): Blubber: Add workarounds for Bookworm's Python pip restrictions - https://phabricator.wikimedia.org/T346090 (10dancy) [19:21:24] 10Release-Engineering-Team, 10Scap: Add bookworm support/build to scap - https://phabricator.wikimedia.org/T346071 (10dancy) [19:21:26] 10Release-Engineering-Team, 10Release Pipeline (Blubber): Blubber: Add workarounds for Bookworm's Python pip restrictions - https://phabricator.wikimedia.org/T346090 (10dancy) [19:25:51] 10Release-Engineering-Team (Seen), 10local-charts, 10Developer Productivity, 10MW-1.35-notes (1.35.0-wmf.5; 2019-11-05): mediawiki/core/.pipeline/dev_prereq.sh: fails if VisualEditor lib/ve submodule is at a commit not present in shallow clone - https://phabricator.wikimedia.org/T237310 (10brennen) [19:32:16] 10Release-Engineering-Team, 10Release Pipeline (Blubber): Blubber: Add workaround for Bookworm's Python pip restrictions - https://phabricator.wikimedia.org/T346090 (10dancy) [19:35:57] (03PS1) 10Jforrester: Zuul: [mediawiki/services/mathoid] Mark as archived, moved to GitLab [integration/config] - 10https://gerrit.wikimedia.org/r/956493 (https://phabricator.wikimedia.org/T344747) [19:35:59] (03PS1) 10Jforrester: jjb: Drop mathoid custom jobs, repo is now on GitLab [integration/config] - 10https://gerrit.wikimedia.org/r/956494 (https://phabricator.wikimedia.org/T344747) [19:36:09] 10Phabricator, 10Release-Engineering-Team (Radar), 10DBA, 10Data-Persistence-Backup, and 2 others: phabricator->phorge migration - database handling - https://phabricator.wikimedia.org/T335080 (10Marostegui) Yep! [19:37:19] 10Release-Engineering-Team, 10Release Pipeline (Blubber): Blubber: Add workaround for Bookworm's Python pip restrictions - https://phabricator.wikimedia.org/T346090 (10dancy) [19:37:35] 10Phabricator, 10Release-Engineering-Team (Escape Goats🐐), 10collaboration-services, 10User-brennen: Migrate phabricator.wikimedia.org to Phorge as upstream - https://phabricator.wikimedia.org/T333885 (10brennen) [19:37:37] 10Phabricator, 10Release-Engineering-Team (Radar), 10DBA, 10Data-Persistence-Backup, and 2 others: phabricator->phorge migration - database handling - https://phabricator.wikimedia.org/T335080 (10brennen) 05In progress→03Resolved [19:40:53] 10Beta-Cluster-Infrastructure, 10MediaWiki-General, 10VisualEditor, 10VisualEditor-MediaWiki: Can't save edits using VisualEditor on beta cluster: "Exception caught: Provided specification is not an array." - https://phabricator.wikimedia.org/T346080 (10DLynch) Is this also related to why we're getting `[Z... [19:51:11] 10Beta-Cluster-Infrastructure, 10MediaWiki-General, 10VisualEditor, 10VisualEditor-MediaWiki: Can't save edits using VisualEditor on beta cluster: "Exception caught: Provided specification is not an array." - https://phabricator.wikimedia.org/T346080 (10matmarex) Seems unrelated… [20:01:12] (03PS1) 10Jdlrobson: Revert "Add selenium daily job to Vector" [integration/config] - 10https://gerrit.wikimedia.org/r/956061 (https://phabricator.wikimedia.org/T345114) [20:01:21] (03CR) 10CI reject: [V: 04-1] Revert "Add selenium daily job to Vector" [integration/config] - 10https://gerrit.wikimedia.org/r/956061 (https://phabricator.wikimedia.org/T345114) (owner: 10Jdlrobson) [20:02:51] (03PS2) 10Jdlrobson: Revert "Add selenium daily job to Vector" [integration/config] - 10https://gerrit.wikimedia.org/r/956061 (https://phabricator.wikimedia.org/T345114) [20:03:19] (03PS3) 10Jdlrobson: Revert "Add selenium daily job to Vector" [integration/config] - 10https://gerrit.wikimedia.org/r/956061 (https://phabricator.wikimedia.org/T345114) [20:07:40] in https://www.mediawiki.org/w/index.php?title=Git%2FReviewers&diff=6070080&oldid=6054878 I thought I had the right regex to ensure that abstractSchemaChanges related stuff would hit my inbox (basically, like a manual query at https://gerrit.wikimedia.org/r/q/project:%2522%255Emediawiki/.*%2524%2522+directory:abstractSchemaChanges ). But, what I saw was that https://gerrit.wikimedia.org/r/c/mediawiki/core/+/954900 auto-added me as a [20:07:40] reviewer whereas [20:10:03] https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikimediaEvents/+/931704 and https://gerrit.wikimedia.org/r/c/mediawiki/extensions/AntiSpoof/+/945669 didn't. I had even dug into https://github.com/valhallasw/gerrit-reviewer-bot and fnmatch to be relatively confident that it would work. But I must have overlooked something. Anyone have a quick idea on how to make the regex do the right thing? [20:13:52] 10GitLab (Upstream pit of despair 🕳️), 10Release-Engineering-Team: Cosmetic error when navigating to merge requests - https://phabricator.wikimedia.org/T340202 (10brennen) [20:13:55] 10GitLab (Upstream pit of despair 🕳️), 10Release-Engineering-Team, 10Upstream: GitLab merge request pages show an error when logged out - https://phabricator.wikimedia.org/T340062 (10brennen) [20:14:37] 10GitLab (Upstream pit of despair 🕳️), 10Release-Engineering-Team (Seen), 10Upstream: GitLab merge request pages show an error when logged out - https://phabricator.wikimedia.org/T340062 (10brennen) [20:15:10] 10GitLab (Upstream pit of despair 🕳️), 10Release-Engineering-Team (Seen), 10Upstream: GitLab loses track which link you clicked when session expires - https://phabricator.wikimedia.org/T340011 (10brennen) [20:17:13] 10Phabricator, 10Release-Engineering-Team (Escape Goats🐐), 10VPS-project-Phabricator, 10Patch-For-Review, 10User-brennen: After a deployment, Phabricator errors out with `Unable to load the "Arcanist" library. Put "arcanist/" next to "phabricator/" on disk.` - https://phabricator.wikimedia.org/T314460 (10... [20:32:23] 10Beta-Cluster-Infrastructure, 10MediaWiki-General, 10VisualEditor, 10VisualEditor-MediaWiki: Can't save edits using VisualEditor on beta cluster: "Exception caught: Provided specification is not an array." - https://phabricator.wikimedia.org/T346080 (10matmarex) That's {T346094} [20:33:10] 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments: 1.41.0-wmf.26 deployment blockers - https://phabricator.wikimedia.org/T343728 (10matmarex) [20:42:34] (03CR) 10Jeena Huneidi: scap backport: check for already merged dependency (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/837983 (owner: 10Jeena Huneidi) [21:06:45] (03PS4) 10Jforrester: jjb: Drop selenium-daily-beta-Vector [integration/config] - 10https://gerrit.wikimedia.org/r/956061 (https://phabricator.wikimedia.org/T345114) (owner: 10Jdlrobson) [21:07:08] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/956061 (https://phabricator.wikimedia.org/T345114) (owner: 10Jdlrobson) [21:07:25] (03CR) 10Jforrester: [C: 03+2] Zuul: [mediawiki/services/mathoid] Mark as archived, moved to GitLab [integration/config] - 10https://gerrit.wikimedia.org/r/956493 (https://phabricator.wikimedia.org/T344747) (owner: 10Jforrester) [21:08:50] (03Merged) 10jenkins-bot: jjb: Drop selenium-daily-beta-Vector [integration/config] - 10https://gerrit.wikimedia.org/r/956061 (https://phabricator.wikimedia.org/T345114) (owner: 10Jdlrobson) [21:08:52] (03CR) 10Jforrester: [C: 03+2] jjb: Drop mathoid custom jobs, repo is now on GitLab [integration/config] - 10https://gerrit.wikimedia.org/r/956494 (https://phabricator.wikimedia.org/T344747) (owner: 10Jforrester) [21:08:54] (03Merged) 10jenkins-bot: Zuul: [mediawiki/services/mathoid] Mark as archived, moved to GitLab [integration/config] - 10https://gerrit.wikimedia.org/r/956493 (https://phabricator.wikimedia.org/T344747) (owner: 10Jforrester) [21:09:49] !log Zuul: [mediawiki/services/mathoid] Mark as archived, moved to GitLab for T344747 [21:09:52] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:09:52] T344747: Migrate Mathoid to GitLab - https://phabricator.wikimedia.org/T344747 [21:11:22] (03Merged) 10jenkins-bot: jjb: Drop mathoid custom jobs, repo is now on GitLab [integration/config] - 10https://gerrit.wikimedia.org/r/956494 (https://phabricator.wikimedia.org/T344747) (owner: 10Jforrester) [21:42:01] 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments: 1.41.0-wmf.26 deployment blockers - https://phabricator.wikimedia.org/T343728 (10matmarex) [21:47:55] 10Release-Engineering-Team, 10Release Pipeline (Blubber): Blubber: Add workaround for Bookworm's Python pip restrictions - https://phabricator.wikimedia.org/T346090 (10dduvall) From my review comment: > This seems like an OK solution for now, as I don't think the risk of breaking the distro's Python installati... [22:24:50] 10Release-Engineering-Team, 10Release Pipeline (Blubber): Blubber: Add workaround for Bookworm's Python pip restrictions - https://phabricator.wikimedia.org/T346090 (10CodeReviewBot) dancy merged https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/61 Python: Set PIP_BREAK_SYSTEM_PACKAGES [22:36:03] 10Release-Engineering-Team, 10Release Pipeline (Blubber): Blubber: Add workaround for Bookworm's Python pip restrictions - https://phabricator.wikimedia.org/T346090 (10hashar) **error: externally-managed-environment** The reason pip warns is because people would often find a misguided recommendation to run it...