[00:00:39] FIRING: DatasourceError: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:05:39] RESOLVED: DatasourceError: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [02:14:25] 10GitLab (Pipeline Services MigrationšŸ¤), 10ircservserv: Migrate ircservserv-config to GitLab - https://phabricator.wikimedia.org/T344744#9997486 (10bd808) The target repo could pretty easily be https://gitlab.wikimedia.org/toolforge-repos/ircservserv-config and created via #Striker. [03:43:41] 10Continuous-Integration-Config: Drop rust testing from gerrit CI, under-supported and only used by a few tools (unlike GitLab CI) - https://phabricator.wikimedia.org/T355363#9997516 (10Legoktm) Feel free to remove the CI jobs, I'll take care of moving the two repos to GitLab soon enough. [03:45:41] 10GitLab (Pipeline Services MigrationšŸ¤), 10ircservserv: Migrate ircservserv-config to GitLab - https://phabricator.wikimedia.org/T344744#9997518 (10Legoktm) Yeah, let's do it, and the source code too. [08:19:23] (03CR) 10Arthur taylor: "recheck" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1055243 (owner: 10Hashar) [08:35:53] (03CR) 10Arthur taylor: "resumbit" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1055243 (owner: 10Hashar) [08:35:59] (03CR) 10Arthur taylor: "resubmit" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1055243 (owner: 10Hashar) [09:13:47] (03CR) 10Hashar: dockerfiles: change NPM_CONFIG_cache to all lower case (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1055257 (https://phabricator.wikimedia.org/T370427) (owner: 10Hashar) [09:16:48] 10Beta-Cluster-Infrastructure, 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 13Patch-For-Review: Replace deployment-ircd02 with a Bullseye or Bookworm host - https://phabricator.wikimedia.org/T369919#9997739 (10Southparkfan) 05Openā†’03In progress After merging https://gerrit.wikimedia.... [09:22:25] 10Beta-Cluster-Infrastructure, 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation): Rebuild or delete deployment-docker-changeprop01 - https://phabricator.wikimedia.org/T369913#9997765 (10Southparkfan) 05Openā†’03Invalid Instance does not exist anymore? [09:22:29] 10Beta-Cluster-Infrastructure, 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation): Rebuild or delete deployment-docker-cpjobqueue01 - https://phabricator.wikimedia.org/T369914#9997772 (10Southparkfan) Instance is offline, seems to be superseded by `deployment-changeprop-1.deployment-prep.eqiad1.... [09:47:19] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/CheckUser] Add GlobalBlocking dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1055284 (https://phabricator.wikimedia.org/T370463) (owner: 10Dreamy Jazz) [09:47:31] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/CheckUser] Add MobileFrontend dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1055291 (https://phabricator.wikimedia.org/T369861) (owner: 10Dreamy Jazz) [09:48:29] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/CheckUser] Add GlobalBlocking dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1055284 (https://phabricator.wikimedia.org/T370463) (owner: 10Dreamy Jazz) [09:48:37] (03CR) 10Hashar: [C:03+2] Fix --change which never worked [integration/quibble] - 10https://gerrit.wikimedia.org/r/1055243 (owner: 10Hashar) [09:48:38] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/CheckUser] Add MobileFrontend dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1055291 (https://phabricator.wikimedia.org/T369861) (owner: 10Dreamy Jazz) [09:58:46] (03CR) 10Krinkle: [C:03+2] dockerfiles: change NPM_CONFIG_cache to all lower case [integration/config] - 10https://gerrit.wikimedia.org/r/1055257 (https://phabricator.wikimedia.org/T370427) (owner: 10Hashar) [09:59:03] (03CR) 10Krinkle: [C:03+1] dockerfiles: change NPM_CONFIG_cache to all lower case [integration/config] - 10https://gerrit.wikimedia.org/r/1055257 (https://phabricator.wikimedia.org/T370427) (owner: 10Hashar) [09:59:16] 10GitLab (Pipeline Services MigrationšŸ¤), 10Math, 10Texvcjs: Migrate texvcjs to GitLab - https://phabricator.wikimedia.org/T348384#9997921 (10Physikerwelt) 05Openā†’03Declined As announced on the mailinglist. gerrit will continue to exist. Thus, it's probably best to only use gitlab for mathoid, which h... [09:59:53] (03CR) 10Krinkle: [C:03+1] dockerfiles: change NPM_CONFIG_cache to all lower case (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1055257 (https://phabricator.wikimedia.org/T370427) (owner: 10Hashar) [10:02:08] (03CR) 10Hashar: dockerfiles: change NPM_CONFIG_cache to all lower case (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1055257 (https://phabricator.wikimedia.org/T370427) (owner: 10Hashar) [10:04:42] (03Merged) 10jenkins-bot: Fix --change which never worked [integration/quibble] - 10https://gerrit.wikimedia.org/r/1055243 (owner: 10Hashar) [10:06:01] (03Merged) 10jenkins-bot: Fix MW_SKIP_EXTERNAL_DEPENDENCIES to be set for is_vendor not use_vendor [integration/quibble] - 10https://gerrit.wikimedia.org/r/1054966 (https://phabricator.wikimedia.org/T370380) (owner: 10Krinkle) [10:18:48] (03CR) 10Jforrester: [C:03+1] "LGTM; want me to merge/build/deploy/test?" [integration/config] - 10https://gerrit.wikimedia.org/r/1055257 (https://phabricator.wikimedia.org/T370427) (owner: 10Hashar) [10:31:06] 10Beta-Cluster-Infrastructure, 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation): Replace or delete deployment-mediawiki[11-12].deployement-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T361387#9998010 (10Southparkfan) a:03Southparkfan [10:39:15] (03PS1) 10Jforrester: Zuul: Drop rust testing from last two repos [integration/config] - 10https://gerrit.wikimedia.org/r/1055402 (https://phabricator.wikimedia.org/T355363) [10:39:17] (03PS1) 10Jforrester: jjb: Drop rust-test, rust-doc-publish [integration/config] - 10https://gerrit.wikimedia.org/r/1055403 (https://phabricator.wikimedia.org/T355363) [10:39:18] (03PS1) 10Jforrester: Docker: [rust] Drop, now unused [integration/config] - 10https://gerrit.wikimedia.org/r/1055404 (https://phabricator.wikimedia.org/T355363) [10:44:04] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Quota-requests): Increase RAM and volume quotas for deployment-prep - https://phabricator.wikimedia.org/T370510 (10Southparkfan) 03NEW [10:56:32] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Quota-requests): Increase RAM and volume quotas for deployment-prep - https://phabricator.wikimedia.org/T370510#9998092 (10Slst2020) +1 [11:09:50] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Buster Deprecation), 13Patch-For-Review: Remove or replace deployment-jobrunner04.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370487#9998126 (10Southparkfan) `deployment-jobrunner04` has been shut down. As... [11:10:13] Project beta-scap-sync-world build #164218: 04FAILURE in 15 min: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/164218/ [11:11:44] ^ ok, that's on me [11:12:14] T370487 [11:12:14] T370487: Remove or replace deployment-jobrunner04.deployment-prep.eqiad1.wikimedia.cloud (Buster deprecation) - https://phabricator.wikimedia.org/T370487 [11:15:21] fixed [11:16:33] jobrunner04 is not active, but because it's still in the scap target list (and will stay there until fully removed from PuppetDB?), the build was failing - rebooted the machine till fully able to delete the instance [11:19:53] Yippee, build fixed! [11:19:53] Project beta-scap-sync-world build #164219: 09FIXED in 5 min 12 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/164219/ [12:28:31] Krinkle: hi, for the npm_config_cache change https://gerrit.wikimedia.org/r/c/integration/config/+/1055257 you have changed your CR+2 to a CR+1 was that intentional? [12:28:56] I am not familiar with npm config and I suspect I might ave screwed up something [12:28:59] s/ave/have/ [12:29:23] Southparkfan: kudos on fixing the beta-scap job ;) [12:33:09] hashar: intentional in that I'm not the one deploying. [12:33:20] ahhh [12:33:33] I thought you caught something last minute and voted +1 to prevent the merge [12:33:34] :D [12:33:37] I am doing it [12:33:43] (03CR) 10Hashar: [C:03+2] dockerfiles: change NPM_CONFIG_cache to all lower case [integration/config] - 10https://gerrit.wikimedia.org/r/1055257 (https://phabricator.wikimedia.org/T370427) (owner: 10Hashar) [12:33:45] Thx for checking [12:34:33] (03CR) 10Hashar: [C:03+2] "Timo changed his vote from CR+2 to CR+1 as an indication he was not going to rebuild the images himself." [integration/config] - 10https://gerrit.wikimedia.org/r/1055257 (https://phabricator.wikimedia.org/T370427) (owner: 10Hashar) [12:34:58] (03Merged) 10jenkins-bot: dockerfiles: change NPM_CONFIG_cache to all lower case [integration/config] - 10https://gerrit.wikimedia.org/r/1055257 (https://phabricator.wikimedia.org/T370427) (owner: 10Hashar) [12:35:19] and after that I guess I will cut and deploy a new Quibble release for MW_SKIP_EXTERNAL_DEPENDENCIES [12:42:15] (03PS1) 10Hashar: jjb: switch jobs to set NPM_CONFIG_cache to lower case [integration/config] - 10https://gerrit.wikimedia.org/r/1055429 (https://phabricator.wikimedia.org/T370427) [13:08:24] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10wmde-wikidata-tech, 07ci-test-error, 13Patch-For-Review: [QB] Build fails on EACCESS issues - https://phabricator.wikimedia.org/T370427#9998402 (10hashar) Interestingly that would have been fixed byhaving `HOME=/tmp` (T365871) but htt... [13:11:35] (03PS1) 10Hashar: jjb: switch Quibble jobs to use HOME=/tmp (2) [integration/config] - 10https://gerrit.wikimedia.org/r/1055435 (https://phabricator.wikimedia.org/T365871) [13:12:19] (03CR) 10Hashar: [C:03+2] "The rest of the jobs are switched with I332af7e3e10baa3ed0b5979e2b690ba60d2a7c79" [integration/config] - 10https://gerrit.wikimedia.org/r/1037453 (https://phabricator.wikimedia.org/T365871) (owner: 10Hashar) [13:15:11] (03CR) 10Hashar: [C:03+2] jjb: switch Quibble jobs to use HOME=/tmp (2) [integration/config] - 10https://gerrit.wikimedia.org/r/1055435 (https://phabricator.wikimedia.org/T365871) (owner: 10Hashar) [13:16:25] (03Merged) 10jenkins-bot: jjb: switch Quibble jobs to use HOME=/tmp (2) [integration/config] - 10https://gerrit.wikimedia.org/r/1055435 (https://phabricator.wikimedia.org/T365871) (owner: 10Hashar) [13:22:08] (03PS2) 10Hashar: jjb: switch jobs to set NPM_CONFIG_cache to lower case [integration/config] - 10https://gerrit.wikimedia.org/r/1055429 (https://phabricator.wikimedia.org/T370427) [13:25:06] (03open) 10matmarex: Update dependencies [repos/ci-tools/patchdemo] - 10https://gitlab.wikimedia.org/repos/ci-tools/patchdemo/-/merge_requests/626 [13:25:26] hashar: I broke it, but then I fixed it as well ;) [13:25:48] (03merge) 10matmarex: Update dependencies [repos/ci-tools/patchdemo] - 10https://gitlab.wikimedia.org/repos/ci-tools/patchdemo/-/merge_requests/626 [13:37:38] (03CR) 10Hashar: [C:03+2] "I have updated the two jobs triggered by wikidata/query-builder and did a `recheck` on https://gerrit.wikimedia.org/r/c/wikidata/query-bui" [integration/config] - 10https://gerrit.wikimedia.org/r/1055429 (https://phabricator.wikimedia.org/T370427) (owner: 10Hashar) [13:38:18] !log Updating Jenkins jobs for "NPM_CONFIG_cache to lower case" | https://gerrit.wikimedia.org/r/c/integration/config/+/1055429 | T370427 [13:38:21] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:38:21] T370427: [QB] Build fails on EACCESS issues - https://phabricator.wikimedia.org/T370427 [13:39:33] (03Merged) 10jenkins-bot: jjb: switch jobs to set NPM_CONFIG_cache to lower case [integration/config] - 10https://gerrit.wikimedia.org/r/1055429 (https://phabricator.wikimedia.org/T370427) (owner: 10Hashar) [13:40:41] 10Release-Engineering-Team (Radar), 06collaboration-services, 06SRE, 06Traffic: implement anti-abuse features for GitLab (Move GitLab behind the CDN) - https://phabricator.wikimedia.org/T366882#9998507 (10Jelto) [13:41:12] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10wmde-wikidata-tech, 07ci-test-error, 13Patch-For-Review: [QB] Build fails on EACCESS issues - https://phabricator.wikimedia.org/T370427#9998502 (10hashar) 05Openā†’03Resolved I have updated the two jobs triggered by `wikidata/qu... [13:47:27] (03PS1) 10Hashar: release: Quibble 1.9.4 [integration/quibble] - 10https://gerrit.wikimedia.org/r/1055442 (https://phabricator.wikimedia.org/T370380) [13:48:05] (03PS2) 10Hashar: release: Quibble 1.9.4 [integration/quibble] - 10https://gerrit.wikimedia.org/r/1055442 (https://phabricator.wikimedia.org/T370380) [13:48:38] (03PS3) 10Hashar: release: Quibble 1.9.4 [integration/quibble] - 10https://gerrit.wikimedia.org/r/1055442 (https://phabricator.wikimedia.org/T370380) [13:49:07] (03PS1) 10Hashar: release: Start 1.9.5 cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/1055444 [14:23:01] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Quota-requests): Increase RAM and volume quotas for deployment-prep - https://phabricator.wikimedia.org/T370510#9998617 (10Andrew) 05Openā†’03Resolved a:03Andrew Done. Thank you so much for working on this project, @Southparkfan [14:26:25] (03CR) 10Hashar: [C:03+2] release: Quibble 1.9.4 [integration/quibble] - 10https://gerrit.wikimedia.org/r/1055442 (https://phabricator.wikimedia.org/T370380) (owner: 10Hashar) [14:29:59] (03PS1) 10Hashar: dockerfiles: update Quibble to 1.9.4 [integration/config] - 10https://gerrit.wikimedia.org/r/1055452 (https://phabricator.wikimedia.org/T370380) [14:38:02] (03PS1) 10Hashar: jjb: switch jobs from Quibble 1.9.3 to 1.9.4 [integration/config] - 10https://gerrit.wikimedia.org/r/1055453 (https://phabricator.wikimedia.org/T370380) [14:58:55] Yippee, build fixed! [14:58:55] Project beta-scap-sync-world build #164239: 09FIXED in 3 min 51 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/164239/ [15:12:59] !log Switch Jenkins jobs to Quibble 1.9.4 [15:13:01] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:13:05] I swear [15:13:16] next time I work on automatizing that system [15:19:31] (03CR) 10Hashar: [C:03+2] jjb: switch jobs from Quibble 1.9.3 to 1.9.4 [integration/config] - 10https://gerrit.wikimedia.org/r/1055453 (https://phabricator.wikimedia.org/T370380) (owner: 10Hashar) [15:20:37] 10Continuous-Integration-Config, 10MediaWiki-Vendor, 10Quibble, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: mediawiki/core and mediawiki/vendor both skip composer.lock checks - https://phabricator.wikimedia.org/T370380#9998805 (10hashar) The fix for `MW_SKIP_EXTERNAL_DEPENDENCIES` has been app... [15:20:42] (03Merged) 10jenkins-bot: jjb: switch jobs from Quibble 1.9.3 to 1.9.4 [integration/config] - 10https://gerrit.wikimedia.org/r/1055453 (https://phabricator.wikimedia.org/T370380) (owner: 10Hashar) [16:12:20] 10GitLab-Application-Security-Pipeline, 10Security Team AppSec, 06Security-Team, 13Patch-For-Review, 07Security: Attempt to condense trivy scanning output and avoid false positive exit code - https://phabricator.wikimedia.org/T367440#9998962 (10sbassett) >>! In T367440#9990781, @mmartorana wrote: > As fo... [16:56:56] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Buster Deprecation): Migrate deployment-prep away from Debian Buster to Bullseye/Bookworm - https://phabricator.wikimedia.org/T327742#9999065 (10Andrew) [16:56:59] if I need to take a puppet compiler node out of jenkins rotation, should I just kill the local agent, or is there a more preferred approach? [17:12:43] (03PS2) 10Snwachukwu: Edit Repo Config [node-rdkafka-statsd] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1055476 [17:16:46] jhathaway: you should be able to mark an agent as offline via a page like https://integration.wikimedia.org/ci/computer/pcc%2Dworker1001.puppet%2Ddiffs.eqiad1.wikimedia.cloud/ [17:24:13] (03CR) 10Ottomata: [C:03+2] Edit Repo Config [node-rdkafka-statsd] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1055476 (owner: 10Snwachukwu) [17:24:14] (03CR) 10Ottomata: [V:03+2 C:03+2] Edit Repo Config [node-rdkafka-statsd] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1055476 (owner: 10Snwachukwu) [17:53:10] thanks bd808 I don't see a button anywhere, perhaps I don't have the necessary privileges [18:09:46] I think only the ciadmin ldap group can disconnect agents [18:11:37] I can offline a node, killing the agent or shutting down the node would also be effective. If you'll need to do this regularly, we could also get you added to that group. [18:21:19] thcipriani: thanks, I'll just kill the agent for now [18:21:40] sounds good :) [18:47:24] (03open) 10don-vip: Replace deprecated "sonar.login" by "sonar.token" [repos/ci-tools/sonarqube] - 10https://gitlab.wikimedia.org/repos/ci-tools/sonarqube/-/merge_requests/9 [19:58:13] jhathaway: Jenkins might auto reconnect the agent, I am not sure how it behaves ;) [19:58:34] true, I moved the jar out of the way, and it just recreated it [19:58:36] jhathaway: else add yourself to the `ciadmin` ldap group and you get blessed with full pers! [19:58:41] yeah it is magic [19:59:09] cursed magic ;) [19:59:12] the idea was to move the list of jenkins admins to a ldap group [19:59:21] and SRE are not included by default to avoid potential mistakes [19:59:29] I think [19:59:29] understandable [19:59:41] much like I guess SRE are probably not admin of Gerrit by default [21:09:08] 10GitLab (Pipeline Services MigrationšŸ¤), 10ircservserv: Migrate ircservserv-config to GitLab - https://phabricator.wikimedia.org/T344744#9999625 (10bd808) 05Openā†’03In progress a:03bd808 [22:23:05] Project beta-update-databases-eqiad build #77569: 04FAILURE in 3 min 4 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/77569/ [23:32:17] Yippee, build fixed! [23:32:17] Project beta-update-databases-eqiad build #77570: 09FIXED in 12 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/77570/ [23:39:14] 10GitLab (Pipeline Services MigrationšŸ¤), 10ircservserv, 13Patch-For-Review: Migrate ircservserv-config to GitLab - https://phabricator.wikimedia.org/T344744#9999804 (10bd808) Once @Legoktm and/or @JJMC89 approve and/or merge my pending merge requests we can start tearing down the gerrit repos.