[00:00:34] (DatasourceError) firing: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:05:34] (DatasourceError) resolved: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:05:48] (DatasourceError) firing: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:11:00] (DatasourceError) resolved: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [05:21:06] (03PS1) 10Hashar: (DO NOT INSTALL) upgrade pip/setuptools in fullrun [integration/quibble] - 10https://gerrit.wikimedia.org/r/963863 [05:23:50] (03CR) 10CI reject: [V: 04-1] (DO NOT INSTALL) upgrade pip/setuptools in fullrun [integration/quibble] - 10https://gerrit.wikimedia.org/r/963863 (owner: 10Hashar) [07:26:06] (03PS2) 10Hashar: ci-fullrun: use a virtualenv to setup Quibble [integration/quibble] - 10https://gerrit.wikimedia.org/r/963863 [07:29:01] (03CR) 10CI reject: [V: 04-1] ci-fullrun: use a virtualenv to setup Quibble [integration/quibble] - 10https://gerrit.wikimedia.org/r/963863 (owner: 10Hashar) [07:31:25] (03PS3) 10Hashar: ci-fullrun: use a virtualenv to setup Quibble [integration/quibble] - 10https://gerrit.wikimedia.org/r/963863 [07:33:58] (03CR) 10CI reject: [V: 04-1] ci-fullrun: use a virtualenv to setup Quibble [integration/quibble] - 10https://gerrit.wikimedia.org/r/963863 (owner: 10Hashar) [07:45:35] (03PS1) 10Hashar: Revert "dockerfiles: keep pip and wheels in Quibble images" [integration/config] - 10https://gerrit.wikimedia.org/r/963815 [07:57:38] (03PS2) 10Hashar: dockerfiles: quibble-buster: use venv instead of pip/wheels [integration/config] - 10https://gerrit.wikimedia.org/r/963815 [08:00:28] (03CR) 10Hashar: [C: 03+2] dockerfiles: quibble-buster: use venv instead of pip/wheels [integration/config] - 10https://gerrit.wikimedia.org/r/963815 (owner: 10Hashar) [08:01:42] (03Merged) 10jenkins-bot: dockerfiles: quibble-buster: use venv instead of pip/wheels [integration/config] - 10https://gerrit.wikimedia.org/r/963815 (owner: 10Hashar) [08:05:29] (03PS1) 10Hashar: jjb: switch integration/quibble job to new Quibble image [integration/config] - 10https://gerrit.wikimedia.org/r/963947 [08:18:10] (03PS4) 10Hashar: ci-fullrun: use a virtualenv to setup Quibble [integration/quibble] - 10https://gerrit.wikimedia.org/r/963863 [08:20:56] (03CR) 10CI reject: [V: 04-1] ci-fullrun: use a virtualenv to setup Quibble [integration/quibble] - 10https://gerrit.wikimedia.org/r/963863 (owner: 10Hashar) [08:35:47] (03PS5) 10Hashar: ci-fullrun: use a virtualenv to setup Quibble [integration/quibble] - 10https://gerrit.wikimedia.org/r/963863 [08:36:17] (03CR) 10Hashar: [C: 03+2] jjb: switch integration/quibble job to new Quibble image [integration/config] - 10https://gerrit.wikimedia.org/r/963947 (owner: 10Hashar) [08:37:45] (03Merged) 10jenkins-bot: jjb: switch integration/quibble job to new Quibble image [integration/config] - 10https://gerrit.wikimedia.org/r/963947 (owner: 10Hashar) [08:38:27] (03CR) 10CI reject: [V: 04-1] ci-fullrun: use a virtualenv to setup Quibble [integration/quibble] - 10https://gerrit.wikimedia.org/r/963863 (owner: 10Hashar) [08:44:26] (03PS6) 10Hashar: ci-fullrun: use a virtualenv to setup Quibble [integration/quibble] - 10https://gerrit.wikimedia.org/r/963863 [08:47:11] (03CR) 10CI reject: [V: 04-1] ci-fullrun: use a virtualenv to setup Quibble [integration/quibble] - 10https://gerrit.wikimedia.org/r/963863 (owner: 10Hashar) [08:51:04] (03PS7) 10Hashar: ci-fullrun: use a virtualenv to setup Quibble [integration/quibble] - 10https://gerrit.wikimedia.org/r/963863 [09:19:35] (03PS8) 10Hashar: ci-fullrun: use a virtualenv to setup Quibble [integration/quibble] - 10https://gerrit.wikimedia.org/r/963863 [10:10:29] (03PS9) 10Hashar: ci-fullrun: use a virtualenv to setup Quibble [integration/quibble] - 10https://gerrit.wikimedia.org/r/963863 [10:50:09] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Jenkins: Tox verbose outputs has poor contrast in Jenkins console output - https://phabricator.wikimedia.org/T347241 (10hashar) An alternative, we run tox in verbose mode `tox -v`, I guess the tox 4 image can remove the verbose flag... [10:57:32] (03PS1) 10Hashar: dockerfiles: lower tox 4 verbosity [integration/config] - 10https://gerrit.wikimedia.org/r/963992 (https://phabricator.wikimedia.org/T347241) [14:21:59] 10GitLab, 10Diffusion-Repository-Administrators, 10Projects-Cleanup: Delete GitLab repo toolforge-repos/povoconta and its Diffusion mirror (code hosted in GitHub instead) - https://phabricator.wikimedia.org/T348325 (10Aklapper) p:05Triage→03Low [14:22:17] 10GitLab, 10Diffusion-Repository-Administrators, 10Projects-Cleanup: Delete GitLab repo toolforge-repos/povoconta and its Diffusion mirror (code hosted in GitHub instead) - https://phabricator.wikimedia.org/T348325 (10Aklapper) [14:32:51] 10Project-Admins, 10Wikipedia-Android-App-Backlog, 10Wikipedia-iOS-App-Backlog: More intuitive names for Phabricator tags "Wikipedia-Android-App-Backlog" and "Wikipedia-iOS-App-Backlog" - https://phabricator.wikimedia.org/T347364 (10Aklapper) [14:38:01] 10GitLab, 10Diffusion-Repository-Administrators, 10Projects-Cleanup: Delete GitLab repo toolforge-repos/povoconta and its Diffusion mirror (code hosted in GitHub instead) - https://phabricator.wikimedia.org/T348325 (10Aklapper) Permission-wise I can do both of it, I'm just vaguely wondering if there's anythi... [16:15:16] (03CR) 10Xqt: [C: 03+1] dockerfiles: lower tox 4 verbosity [integration/config] - 10https://gerrit.wikimedia.org/r/963992 (https://phabricator.wikimedia.org/T347241) (owner: 10Hashar) [16:17:41] (03PS1) 10Hashar: Move install args to a standalone function [integration/quibble] - 10https://gerrit.wikimedia.org/r/964035 [18:20:27] 10Release-Engineering-Team (Priority Backlog 📥), 10wikimedia.biterg.io: Bitergia: ssh and Hatstall access for thcipriani - https://phabricator.wikimedia.org/T335700 (10Aklapper) [18:20:54] 10Release-Engineering-Team (Priority Backlog 📥), 10wikimedia.biterg.io: Adjust data update workflow in Bitergia DB to grimoirelab-sortinghat 0.8 changes - https://phabricator.wikimedia.org/T335577 (10Aklapper) 05Open→03Resolved This took a while to sort out but took an interesting turn: `sortinghat` (and i... [18:29:05] 10Release-Engineering-Team (Priority Backlog 📥), 10wikimedia.biterg.io: Bitergia: ssh and Hatstall access for thcipriani - https://phabricator.wikimedia.org/T335700 (10Aklapper) 05Stalled→03Open * Updated ssh access request in https://support.bitergia.com/support/tickets/891 now that T335577 got resolved a... [19:17:37] (03PS1) 10Hashar: Skip PostgreSQL test when it is not installed [integration/quibble] - 10https://gerrit.wikimedia.org/r/964068 [19:33:08] (03CR) 10Hashar: [C: 03+2] "That is not idea, but I think tox v4 debug log can be skipped :) Thanks for the +1!" [integration/config] - 10https://gerrit.wikimedia.org/r/963992 (https://phabricator.wikimedia.org/T347241) (owner: 10Hashar) [19:39:10] (03Merged) 10jenkins-bot: dockerfiles: lower tox 4 verbosity [integration/config] - 10https://gerrit.wikimedia.org/r/963992 (https://phabricator.wikimedia.org/T347241) (owner: 10Hashar) [19:39:33] Not sure why it took 6 minutes to merge .. [19:40:01] !log Updating docker-pkg files for https://gerrit.wikimedia.org/r/c/integration/config/+/963992/ "lower tox 4 verbosity" # T347241 [19:40:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:40:04] T347241: Tox verbose outputs has poor contrast in Jenkins console output - https://phabricator.wikimedia.org/T347241 [19:41:24] ah that is castor [19:44:21] :( [19:46:53] eg 4 min 30 sec waiting; [19:46:57] https://integration.wikimedia.org/ci/job/castor-save-workspace-cache/4021021/ [19:47:28] 10Release-Engineering-Team (Priority Backlog 📥), 10wikimedia.biterg.io: Bitergia: ssh and Hatstall access for thcipriani - https://phabricator.wikimedia.org/T335700 (10thcipriani) >>! In T335700#9231781, @Aklapper wrote: > * Updated ssh access request in https://support.bitergia.com/support/tickets/891 now tha... [19:49:37] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Jenkins, 10Patch-For-Review: Tox verbose outputs has poor contrast in Jenkins console output - https://phabricator.wikimedia.org/T347241 (10hashar) Successfully published image docker-registry.wikimedia.org/releng/tox-pywikibot:4.8... [19:55:32] (03PS1) 10Hashar: jjb: update tox v4 jobs to decrease verbosity [integration/config] - 10https://gerrit.wikimedia.org/r/964075 (https://phabricator.wikimedia.org/T347241) [19:56:32] (03CR) 10Hashar: [C: 03+2] jjb: update tox v4 jobs to decrease verbosity [integration/config] - 10https://gerrit.wikimedia.org/r/964075 (https://phabricator.wikimedia.org/T347241) (owner: 10Hashar) [19:57:49] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Jenkins, 10Patch-For-Review: Tox verbose outputs has poor contrast in Jenkins console output - https://phabricator.wikimedia.org/T347241 (10hashar) Among other jobs, two pywikibot related jobs got updated to drop `-v` when invoking... [19:58:15] (03Merged) 10jenkins-bot: jjb: update tox v4 jobs to decrease verbosity [integration/config] - 10https://gerrit.wikimedia.org/r/964075 (https://phabricator.wikimedia.org/T347241) (owner: 10Hashar) [20:06:01] Is this evidence of a gitlab problem, or do I just need to be patient? https://gitlab.wikimedia.org/repos/cloud/cloud-vps/horizon/deploy/-/jobs/150157 [20:06:18] I don't think it's config on my end since I've built that same project many times [20:08:02] 10Release-Engineering-Team (Priority Backlog 📥), 10wikimedia.biterg.io: Bitergia: ssh and Hatstall access for thcipriani - https://phabricator.wikimedia.org/T335700 (10Aklapper) Also new here, kind of. :) Added you to both tickets. [20:13:13] andrewbogott: hrm, interesting. So it looks like everything should be set up right to use the trusted runners (in the projects.json, tagged right afaict). [20:13:57] and of course I'm logged out [20:14:03] I'm guessing that there aren't any trusted runners? [20:14:09] but definitely don't know where to look for that [20:17:05] well I know to look in the admin area. Can you see them in: https://gitlab.wikimedia.org/repos/cloud/cloud-vps/horizon/deploy/-/settings/ci_cd#js-runners-settings ? [20:17:17] settings > CI/CD [20:17:38] I see a bunch online, now why don't they pick up this job [20:18:10] I can just tag and try again :) [20:19:46] oh, I think you need to add protected tags [20:21:56] those runners have the "protected" setting checked, so if you're building from a tag it'll need to be protected (I..think :)) https://docs.gitlab.com/ee/ci/runners/configure_runners.html#prevent-runners-from-revealing-sensitive-information [20:22:45] so in here you can add `*` as a protected tag: https://gitlab.wikimedia.org/repos/cloud/cloud-vps/horizon/deploy/-/settings/repository [20:22:49] would that have changed since last week? Or changed because I'm pushing from a different system? [20:22:52] then any time you push a tag it should get picked up [20:23:53] that shouldn't have changed in the past week. And I could be missing something else obvious. dancy or dduvall know what's actually happening :) [20:24:04] OK, I added * and I'll try a new tag [20:24:15] unless the change is retroactive... [20:24:53] * dancy looks [20:25:13] retroactive would be surprising. And computers never surprise me. [20:25:44] I canceled the old one, the new job seems to be running. [20:25:56] So as long as i'm not curious about why this worked before I think I'm all set :) [20:26:23] I...am curious. You were able to trigger this by pushing a tag before? [20:26:42] https://gitlab.wikimedia.org/repos/cloud/cloud-vps/horizon/deploy/-/pipelines [20:26:57] lots of previous successes there [20:27:18] But I think I did have a protected tag wildcard in place before. I don't know why it would've vanished. [20:27:47] ooh, that's curious [20:28:30] I have no suggestions. [20:29:01] based on a few jobs that I spot checked all of the previous jobs were using the https://phabricator.wikimedia.org/T346307 issue [20:29:37] ah, that makes perfect sense [20:29:40] Thanks Taavi! [20:29:53] so the bug is that it was working before? [20:29:54] ^ [20:29:54] Things working when they shouldn't have been is a good recipe for confusion. [20:30:04] ok :) [20:30:26] (^ was supposed to be for "Thanks taavi!" but also +1 it seems to be a bug :)) [20:31:38] good explanation, sorry to be curious :D [20:43:55] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.41.0-wmf.29 deployment blockers - https://phabricator.wikimedia.org/T347080 (10thcipriani) 05Open→03Resolved [20:47:57] 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments: 1.42.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T348354 (10thcipriani) p:05Triage→03Medium a:03brennen [21:32:39] 10GitLab (Project Migration), 10SRE, 10Traffic, 10Patch-For-Review: Migrate Traffic repositories from Gerrit to Gitlab - https://phabricator.wikimedia.org/T347623 (10CodeReviewBot) brett opened https://gitlab.wikimedia.org/repos/sre/acme-chief/-/merge_requests/4 Draft: Implement Gitlab CI and Blubber config [23:26:26] (03PS1) 10Krinkle: zuul: Add php8.2 job to tools-guc [integration/config] - 10https://gerrit.wikimedia.org/r/964096 [23:34:12] (03CR) 10Krinkle: [C: 03+2] zuul: Add php8.2 job to tools-guc [integration/config] - 10https://gerrit.wikimedia.org/r/964096 (owner: 10Krinkle) [23:35:48] (03Merged) 10jenkins-bot: zuul: Add php8.2 job to tools-guc [integration/config] - 10https://gerrit.wikimedia.org/r/964096 (owner: 10Krinkle) [23:36:51] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/964096 [23:36:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL