[00:00:33] (DatasourceError) firing: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:05:33] (DatasourceError) resolved: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:20:58] 10Phabricator: Changing a non-access request ticket into an access request ticket does not create restricted blocker - https://phabricator.wikimedia.org/T103194 (10Pppery) It refers to T87467 (which bitrotted away at some point and no longer works, but some remnants of it persist in the extensions repo) [00:31:31] 10GitLab, 10Diffusion-Repository-Administrators, 10Projects-Cleanup: Delete GitLab repo toolforge-repos/povoconta and its Diffusion mirror (code hosted in GitHub instead) - https://phabricator.wikimedia.org/T348325 (10bd808) > For unknown reasons https://gitlab.wikimedia.org/toolforge-repos/povoconta/-/commi... [00:36:04] 10GitLab, 10Diffusion-Repository-Administrators, 10Projects-Cleanup: Delete GitLab repo toolforge-repos/povoconta and its Diffusion mirror (code hosted in GitHub instead) - https://phabricator.wikimedia.org/T348325 (10bd808) >>! In T348325#9231111, @Aklapper wrote: > Permission-wise I can do both of it, I'm... [09:29:22] (03CR) 10Hashar: dockerfiles: Add firefox-esr to quibble-buster images (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/987790 (owner: 10Krinkle) [09:29:45] (03PS2) 10Hashar: dockerfiles: Add firefox-esr to quibble-buster images [integration/config] - 10https://gerrit.wikimedia.org/r/987790 (owner: 10Krinkle) [09:30:00] (03CR) 10Hashar: [C: 03+2] dockerfiles: Add firefox-esr to quibble-buster images [integration/config] - 10https://gerrit.wikimedia.org/r/987790 (owner: 10Krinkle) [09:31:16] (03Merged) 10jenkins-bot: dockerfiles: Add firefox-esr to quibble-buster images [integration/config] - 10https://gerrit.wikimedia.org/r/987790 (owner: 10Krinkle) [10:09:04] 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments: 1.42.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T350089 (10daniel) ##### Risky Patch! 🚂🔥 * **Change**: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/965563 * **Summary**: ** This patch refactors the... [10:23:29] (03PS1) 10Hashar: jjb: update Quibble jobs to latest images [integration/config] - 10https://gerrit.wikimedia.org/r/987951 [10:23:57] (03CR) 10Hashar: [C: 03+2] "Jobs updated by https://gerrit.wikimedia.org/r/c/integration/config/+/987951" [integration/config] - 10https://gerrit.wikimedia.org/r/987790 (owner: 10Krinkle) [10:38:02] !log Updated Quibble Jenkins jobs to latest images ( https://gerrit.wikimedia.org/r/c/integration/config/+/987790 and https://gerrit.wikimedia.org/r/c/integration/config/+/983892 ) [10:38:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:39:03] OF COURSE IT FAILES [10:39:05] as expected [10:40:21] !log Rollback Quibble Jenkins jobs update due to broken `git` introduced by https://gerrit.wikimedia.org/r/c/integration/config/+/983892 [10:40:23] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:41:57] (03CR) 10Hashar: "The primary I hate Debian/Apt/Docker is that this change ends up causing an upgrade of every single Debian packages in every single images" [integration/config] - 10https://gerrit.wikimedia.org/r/983892 (owner: 10Ahmon Dancy) [10:42:12] (03PS1) 10Hashar: Revert "jjb: update Quibble jobs to latest images" [integration/config] - 10https://gerrit.wikimedia.org/r/987952 [10:42:30] (03CR) 10Hashar: [C: 03+2] Revert "jjb: update Quibble jobs to latest images" [integration/config] - 10https://gerrit.wikimedia.org/r/987952 (owner: 10Hashar) [10:43:54] (03CR) 10Hashar: [C: 03+2] "deployed and immediately reverted via https://gerrit.wikimedia.org/r/c/integration/config/+/987952/1" [integration/config] - 10https://gerrit.wikimedia.org/r/987951 (owner: 10Hashar) [10:46:38] 10Phabricator: When an account is disabled, remove them as watchers - https://phabricator.wikimedia.org/T256094 (10Aklapper) > When an account is disabled, remove them as watchers Well, what if an account got enabled again later on and they lost all the projects they watched? I don't think this is necessarily a... [10:47:28] (03Merged) 10jenkins-bot: jjb: update Quibble jobs to latest images [integration/config] - 10https://gerrit.wikimedia.org/r/987951 (owner: 10Hashar) [10:47:30] (03Merged) 10jenkins-bot: Revert "jjb: update Quibble jobs to latest images" [integration/config] - 10https://gerrit.wikimedia.org/r/987952 (owner: 10Hashar) [10:49:39] 10Continuous-Integration-Config, 10Release-Engineering-Team: Deal with git safe.directory in CI Docker images - https://phabricator.wikimedia.org/T354409 (10hashar) [10:49:49] 10Continuous-Integration-Config, 10Release-Engineering-Team: Deal with git safe.directory in CI Docker images - https://phabricator.wikimedia.org/T354409 (10hashar) [11:28:08] 10Continuous-Integration-Config, 10Release-Engineering-Team: Deal with git safe.directory in CI Docker images - https://phabricator.wikimedia.org/T354409 (10hashar) My guess is the `src` directory is created on the host and owned by `jenkins-deploy` / uid 2947 while the container runs `git` as `nobody` / uid... [14:49:18] 10Release-Engineering-Team, 10Traffic, 10collaboration-services, 10Patch-For-Review: CI on gitlab for eBPF / networking heavy projects - https://phabricator.wikimedia.org/T353279 (10CodeReviewBot) vgutierrez merged https://gitlab.wikimedia.org/repos/sre/tcp-mss-clamper/-/merge_requests/11 clamper: Provide... [15:28:30] 10Release-Engineering-Team, 10Traffic, 10collaboration-services, 10Patch-For-Review: CI on gitlab for eBPF / networking heavy projects - https://phabricator.wikimedia.org/T353279 (10Vgutierrez) https://gitlab.wikimedia.org/repos/sre/tcp-mss-clamper/-/merge_requests/11 performs integration tests using qemu... [16:03:11] (03CR) 10Ahmon Dancy: Add rsync to ci-{buster,bullseye,bookworm} images. (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/983892 (owner: 10Ahmon Dancy) [16:54:03] 10Release-Engineering-Team, 10Traffic, 10collaboration-services: Jobs killed on GitLab DO runners - https://phabricator.wikimedia.org/T354247 (10BCornwall) [17:19:03] 10Release-Engineering-Team, 10Traffic, 10collaboration-services: Jobs killed on GitLab DO runners - https://phabricator.wikimedia.org/T354247 (10dancy) Try adding ` tags: [memory-optimized] ` to the failing job. [17:36:45] 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments: 1.42.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T350088 (10thcipriani) 05Open→03Resolved [17:48:33] (03PS1) 10JHathaway: operations-puppet: Rebuild to grab, puppet-lint-wmf_styleguide-check (1.1.4) [integration/config] - 10https://gerrit.wikimedia.org/r/988025 [17:52:31] (03CR) 10JHathaway: "kindly review" [integration/config] - 10https://gerrit.wikimedia.org/r/988025 (owner: 10JHathaway) [18:06:40] (03CR) 10Jbond: [C: 03+1] operations-puppet: Rebuild to grab, puppet-lint-wmf_styleguide-check (1.1.4) [integration/config] - 10https://gerrit.wikimedia.org/r/988025 (owner: 10JHathaway) [18:15:35] 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments: 1.42.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T354432 (10thcipriani) p:05Triage→03Medium a:03jnuche [19:27:09] 10Phabricator: Username change request - https://phabricator.wikimedia.org/T354447 (10KimHolgerKelting) [19:41:51] looks like CI is a bit backed up (but still working) [19:53:52] 10Continuous-Integration-Infrastructure: gate-and-submit backlogged due to waiting for castor-save-workspace-cache - https://phabricator.wikimedia.org/T353925 (10Reedy) Happening again in similar circumstances... Do we really only have one host doing one of these jobs at a time? {F41651812 size=full} {F416518... [20:41:01] (03PS1) 10Jdlrobson: Flow should no longer generate JSDuck documentation [integration/config] - 10https://gerrit.wikimedia.org/r/988080 (https://phabricator.wikimedia.org/T138401) [21:09:55] (03CR) 10Krinkle: [C: 03+2] "thx, deploying shortly. This also avoids a CI failure for doc/ not existing after "npm run doc"." [integration/config] - 10https://gerrit.wikimedia.org/r/988080 (https://phabricator.wikimedia.org/T138401) (owner: 10Jdlrobson) [21:11:24] (03Merged) 10jenkins-bot: Flow should no longer generate JSDuck documentation [integration/config] - 10https://gerrit.wikimedia.org/r/988080 (https://phabricator.wikimedia.org/T138401) (owner: 10Jdlrobson) [21:15:18] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/988080 [21:15:20] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:15:28] Jdlrobson: done :) [22:05:21] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥): Create a separate memory optimized gitlab runner pool for memory hungry jobs - https://phabricator.wikimedia.org/T345000 (10BCornwall) I went ahead and added some documentation at https://wikitech.wikimedia.org/wiki/GitLab/Gitlab_Run... [22:05:23] 10Release-Engineering-Team, 10Traffic, 10collaboration-services: Jobs killed on GitLab DO runners - https://phabricator.wikimedia.org/T354247 (10BCornwall) @dancy You're a wizard. Thanks so much for this, it's now happily building. :) I added some docs to https://wikitech.wikimedia.org/wiki/GitLab/Gitlab_Ru... [22:05:35] 10Release-Engineering-Team, 10Traffic, 10collaboration-services: Jobs killed on GitLab DO runners - https://phabricator.wikimedia.org/T354247 (10BCornwall) 05Open→03Resolved a:03BCornwall [22:28:30] 10Project-Admins: Create project tag for Lingua Libre - https://phabricator.wikimedia.org/T354458 (10Poslovitch) [22:55:49] 10Release-Engineering-Team (Priority Backlog 📥), 10Projects-Cleanup, 10collaboration-services: Migrate SRE repositories to GitLab - Archiving unused Gerrit repositories - https://phabricator.wikimedia.org/T343707 (10thcipriani) >>! In T343707#9431758, @LSobanski wrote: > The recent commits are related to mig... [23:00:53] (03PS1) 10Thcipriani: GitLab Migration: wmfmariadbpy wmfdb wmfbackups [integration/config] - 10https://gerrit.wikimedia.org/r/988109 (https://phabricator.wikimedia.org/T343707) [23:03:35] (03CR) 10Thcipriani: [C: 03+2] GitLab Migration: wmfmariadbpy wmfdb wmfbackups [integration/config] - 10https://gerrit.wikimedia.org/r/988109 (https://phabricator.wikimedia.org/T343707) (owner: 10Thcipriani) [23:05:06] (03Merged) 10jenkins-bot: GitLab Migration: wmfmariadbpy wmfdb wmfbackups [integration/config] - 10https://gerrit.wikimedia.org/r/988109 (https://phabricator.wikimedia.org/T343707) (owner: 10Thcipriani) [23:05:38] (03PS1) 10Thcipriani: Revert "Add a banner for the 2023 developer survey" [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/987995 [23:07:37] !log reloading zuul to deploy https://gerrit.wikimedia.org/r/988109 [23:07:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:09:23] (03CR) 10Thcipriani: [C: 03+2] Revert "Add a banner for the 2023 developer survey" [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/987995 (owner: 10Thcipriani) [23:10:23] (03Merged) 10jenkins-bot: Revert "Add a banner for the 2023 developer survey" [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/987995 (owner: 10Thcipriani) [23:15:56] 10Release-Engineering-Team, 10Traffic, 10collaboration-services: Jobs killed on GitLab DO runners - https://phabricator.wikimedia.org/T354247 (10BCornwall) Hm, I'm still running into this every now and then. @dancy, Is there a possibility of raising the amount of memory even more? [23:46:17] 10Gerrit, 10Release-Engineering-Team: JenkinsCiPluginsRepository : No plugins available for Gerrit version 3.5.6 - https://phabricator.wikimedia.org/T354464 (10thcipriani) [23:46:41] 10Gerrit, 10Release-Engineering-Team: JenkinsCiPluginsRepository : No plugins available for Gerrit version 3.5.6 - https://phabricator.wikimedia.org/T354464 (10thcipriani)