[00:12:47] <wikibugs>	 (03open) 10dreamyjazz: make-release: Stop branching SimilarEditors [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/268 (https://phabricator.wikimedia.org/T436880)
[05:05:06] <wikibugs>	 (03PS1) 10Hashar: Add dedicated coverage job for CVNBot [integration/config] - 10https://gerrit.wikimedia.org/r/1334328
[05:05:27] <wikibugs>	 (03CR) 10Hashar: [C:03+2] Add dedicated coverage job for CVNBot [integration/config] - 10https://gerrit.wikimedia.org/r/1334328 (owner: 10Hashar)
[05:09:18] <wikibugs>	 (03Merged) 10jenkins-bot: Add dedicated coverage job for CVNBot [integration/config] - 10https://gerrit.wikimedia.org/r/1334328 (owner: 10Hashar)
[05:09:52] <wikibugs>	 (03CR) 10Hashar: "The coverage got overwritten by CVNBot cause I did a copy paste and was not thinking twice when setting it up. I have regenerated Quibble " [integration/quibble] - 10https://gerrit.wikimedia.org/r/1333836 (owner: 10Hashar)
[05:13:03] <wikibugs>	 (03CR) 10Hashar: [C:03+2] "I have regenerated the Quibble coverage job using:" [integration/config] - 10https://gerrit.wikimedia.org/r/1334328 (owner: 10Hashar)
[05:14:56] <wikibugs>	 (03CR) 10Hashar: [C:03+2] jjb: update tox jobs to default to python 3.11 [integration/config] - 10https://gerrit.wikimedia.org/r/1334023 (https://phabricator.wikimedia.org/T436857) (owner: 10Hashar)
[05:15:50] <hashar>	 !log Updating tox jobs to change default python from 3.9 to 3.11 | https://gerrit.wikimedia.org/r/c/integration/config/+/1334023 | T436857
[05:15:52] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[05:15:53] <stashbot>	 T436857: Change tox-v3 tox-v4 default python from 3.9 to 3.11 - https://phabricator.wikimedia.org/T436857
[05:17:44] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: update tox jobs to default to python 3.11 [integration/config] - 10https://gerrit.wikimedia.org/r/1334023 (https://phabricator.wikimedia.org/T436857) (owner: 10Hashar)
[05:20:16] <wikibugs>	 (03open) 10phedenskog: Show wait as a share of run time on the castor total wait chart [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/58
[05:20:19] <wikibugs>	 (03merge) 10phedenskog: Show wait as a share of run time on the castor total wait chart [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/58
[05:28:15] <wikibugs>	 10Continuous-Integration-Config, 13Patch-For-Review: Change tox-v3 tox-v4 default python from 3.9 to 3.11 - https://phabricator.wikimedia.org/T436857#12284088 (10hashar) 05Open→03Resolved a:03hashar
[06:27:05] <hashar>	 !log Upgrading CI Jenkins on contint1003 # T436812
[06:27:07] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[06:38:11] <wmf-insecte>	 Yippee, build fixed!
[06:38:11] <wmf-insecte>	 Project mediawiki-core-doxygen build #22642: 09FIXED in 11 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/22642/
[07:24:30] <wikibugs>	 10GitLab, 06Release-Engineering-Team (Radar), 06Collaboration-Services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12284248 (10ABran-WMF)
[07:25:53] <hashar>	 !log integration: granted sudo access to  Phedenskog 
[07:25:54] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[08:02:43] <wikibugs>	 (03approved) 10jforrester: make-release: Stop branching SimilarEditors [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/268 (https://phabricator.wikimedia.org/T436880) (owner: 10dreamyjazz)
[08:05:33] <wikibugs>	 10GitLab: kokkuri:setup-variables failing for repos/releng/release!268 (due to being a fork?) - https://phabricator.wikimedia.org/T436899 (10Jdforrester-WMF) 03NEW
[08:09:10] <wikibugs>	 (03PS2) 10Jforrester: jjb, Zuul: [mediawiki/services/similar-users] Archive service [integration/config] - 10https://gerrit.wikimedia.org/r/1334110 (https://phabricator.wikimedia.org/T368269) (owner: 10Dreamy Jazz)
[08:09:20] <wikibugs>	 (03CR) 10Jforrester: [C:03+2] jjb, Zuul: [mediawiki/services/similar-users] Archive service [integration/config] - 10https://gerrit.wikimedia.org/r/1334110 (https://phabricator.wikimedia.org/T368269) (owner: 10Dreamy Jazz)
[08:11:44] <wikibugs>	 (03Merged) 10jenkins-bot: jjb, Zuul: [mediawiki/services/similar-users] Archive service [integration/config] - 10https://gerrit.wikimedia.org/r/1334110 (https://phabricator.wikimedia.org/T368269) (owner: 10Dreamy Jazz)
[08:13:43] <James_F>	 !log Zuul: [mediawiki/services/similar-users] Archive service, for T368269
[08:13:45] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[08:13:46] <stashbot>	 T368269: Archive the similar-users service - https://phabricator.wikimedia.org/T368269
[08:25:47] <wikibugs>	 (03open) 10jnuche: jenkins-rel: update plugins to address vulnerabilities [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/124 (https://phabricator.wikimedia.org/T436812)
[08:26:12] <wikibugs>	 (03merge) 10jnuche: jenkins-rel: update plugins to address vulnerabilities [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/124 (https://phabricator.wikimedia.org/T436812)
[08:26:20] <awight>	 Channel topic needs a fix to "libera_logs", that is 404 now
[08:27:53] <awight>	 arnaudb: Is it expected that gitlab.wmo ssh is down?
[08:29:03] <arnaudb>	 awight: is it down for you? I just cloned from gitlab-ssh.wikimedia.org
[08:29:35] <arnaudb>	 you might have missed the https://wikitech.wikimedia.org/wiki/GitLab/Migration if "gitlab-ssh" sounds new to you :D
[08:29:37] <awight>	 I have a URL like this: git@gitlab.wikimedia.org:repos/wmde/docker-dev.git -- should I change it
[08:29:43] <awight>	 aha ha, sorry for the false alarm then
[08:29:49] <arnaudb>	 no problem, happy to help :)
[08:30:24] <awight>	 I probably remember dismissing the banner, even :-|
[08:30:41] <arnaudb>	 :D
[08:33:57] <wikibugs>	 (03CR) 10Hashar: [C:03+2] build: Updating npm dependencies [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1334514 (owner: 10Libraryupgrader)
[08:56:17] <wikibugs>	 (03CR) 10Hashar: [C:03+2] zuul: Add CI job configuration for WandaVoice [integration/config] - 10https://gerrit.wikimedia.org/r/1333883 (owner: 10Sanjai Siddharthan)
[08:57:59] <wikibugs>	 (03Merged) 10jenkins-bot: zuul: Add CI job configuration for WandaVoice [integration/config] - 10https://gerrit.wikimedia.org/r/1333883 (owner: 10Sanjai Siddharthan)
[08:58:02] <wikibugs>	 06Release-Engineering-Team, 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: GitLab agents failing to connect - https://phabricator.wikimedia.org/T436902 (10jnuche) 03NEW
[08:58:40] <wikibugs>	 06Release-Engineering-Team, 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: GitLab agents failing to connect - https://phabricator.wikimedia.org/T436902#12284451 (10jnuche)
[08:59:17] <wikibugs>	 (03CR) 10Hashar: [C:03+2] "I did a `recheck` on WandaVoice change I6766773ca41ac2dcc534132ff8a935cd82b9afc3" [integration/config] - 10https://gerrit.wikimedia.org/r/1333883 (owner: 10Sanjai Siddharthan)
[09:09:29] <wikibugs>	 06Release-Engineering-Team, 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work, 13Patch-For-Review: GitLab agents failing to connect - https://phabricator.wikimedia.org/T436902#12284491 (10jnuche) Agents are now upgraded but the same problem persist: `     gitlab-agent:...
[09:13:15] <wikibugs>	 10GitLab, 06Release-Engineering-Team (Radar), 06Collaboration-Services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12284509 (10jnuche) hi there, our GitLab agents are failing to connect for the last hour or so, https://gitlab.wikimedia.org/repos/test-platform/catalyst...
[09:20:07] <wikibugs>	 10GitLab, 06Release-Engineering-Team: kokkuri:setup-variables failing for repos/releng/release!268 (due to being a fork?) - https://phabricator.wikimedia.org/T436899#12284530 (10Jdforrester-WMF)
[09:31:55] <wikibugs>	 (03PS5) 10Hashar: search: switch opensearch plugins to java21 [integration/config] - 10https://gerrit.wikimedia.org/r/1333700 (https://phabricator.wikimedia.org/T435823) (owner: 10DCausse)
[09:35:14] <wikibugs>	 (03CR) 10Hashar: [C:03+2] "Rebased and I have amended the commit message to list the jobs deleted and created (from `./utils/jjb-diff.sh`)." [integration/config] - 10https://gerrit.wikimedia.org/r/1333700 (https://phabricator.wikimedia.org/T435823) (owner: 10DCausse)
[09:37:24] <wikibugs>	 (03Merged) 10jenkins-bot: search: switch opensearch plugins to java21 [integration/config] - 10https://gerrit.wikimedia.org/r/1333700 (https://phabricator.wikimedia.org/T435823) (owner: 10DCausse)
[09:39:05] <wikibugs>	 10GitLab, 06Release-Engineering-Team, 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: GitLab agents failing to connect - https://phabricator.wikimedia.org/T436902#12284691 (10jnuche)
[09:53:14] <wikibugs>	 10Continuous-Integration-Config, 06Java-Scala-Standardization: Maven does not have color output on CI - https://phabricator.wikimedia.org/T436915 (10hashar) 03NEW
[09:58:59] <wikibugs>	 (03PS1) 10Hashar: dockerfiles: force maven color output when on CI [integration/config] - 10https://gerrit.wikimedia.org/r/1334776 (https://phabricator.wikimedia.org/T436915)
[09:59:57] <wikibugs>	 (03CR) 10Hashar: [C:03+2] dockerfiles: force maven color output when on CI [integration/config] - 10https://gerrit.wikimedia.org/r/1334776 (https://phabricator.wikimedia.org/T436915) (owner: 10Hashar)
[10:01:39] <wikibugs>	 (03Merged) 10jenkins-bot: dockerfiles: force maven color output when on CI [integration/config] - 10https://gerrit.wikimedia.org/r/1334776 (https://phabricator.wikimedia.org/T436915) (owner: 10Hashar)
[10:11:49] <wikibugs>	 (03CR) 10Hashar: [C:03+2] "java versions:" [integration/config] - 10https://gerrit.wikimedia.org/r/1334776 (https://phabricator.wikimedia.org/T436915) (owner: 10Hashar)
[10:13:29] <wikibugs>	 (03PS1) 10Hashar: jjb: update maven jobs to force color output on CI [integration/config] - 10https://gerrit.wikimedia.org/r/1334782 (https://phabricator.wikimedia.org/T436915)
[10:15:21] <wikibugs>	 (03CR) 10Hashar: [C:03+2] jjb: update maven jobs to force color output on CI [integration/config] - 10https://gerrit.wikimedia.org/r/1334782 (https://phabricator.wikimedia.org/T436915) (owner: 10Hashar)
[10:17:57] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: update maven jobs to force color output on CI [integration/config] - 10https://gerrit.wikimedia.org/r/1334782 (https://phabricator.wikimedia.org/T436915) (owner: 10Hashar)
[10:18:39] <wikibugs>	 10Continuous-Integration-Config, 06Java-Scala-Standardization, 13Patch-For-Review: Maven does not have color output on CI - https://phabricator.wikimedia.org/T436915#12284874 (10hashar) 05Open→03Resolved
[10:21:56] <wikibugs>	 10Continuous-Integration-Config, 06Java-Scala-Standardization, 13Patch-For-Review: Maven does not have color output on CI - https://phabricator.wikimedia.org/T436915#12284878 (10hashar) 05Resolved→03Open Despite testing locally, of course when I run it on Jenkins the color is not enabled bah
[10:27:37] <wikibugs>	 (03PS1) 10Dreamy Jazz: Zuul: [mediawiki/extensions/SimilarEditors] Archive extension [integration/config] - 10https://gerrit.wikimedia.org/r/1334790 (https://phabricator.wikimedia.org/T436880)
[10:29:45] <wikibugs>	 (03PS1) 10Hashar: jjb: update maven jobs to force color output [2] [integration/config] - 10https://gerrit.wikimedia.org/r/1334792 (https://phabricator.wikimedia.org/T436915)
[10:30:10] <wikibugs>	 (03CR) 10Hashar: [C:03+2] jjb: update maven jobs to force color output [2] [integration/config] - 10https://gerrit.wikimedia.org/r/1334792 (https://phabricator.wikimedia.org/T436915) (owner: 10Hashar)
[10:32:18] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: update maven jobs to force color output [2] [integration/config] - 10https://gerrit.wikimedia.org/r/1334792 (https://phabricator.wikimedia.org/T436915) (owner: 10Hashar)
[10:32:47] <wikibugs>	 10Continuous-Integration-Config, 06Java-Scala-Standardization, 13Patch-For-Review: Maven does not have color output on CI - https://phabricator.wikimedia.org/T436915#12284929 (10hashar) 05Open→03Resolved
[11:12:45] <wikibugs>	 (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/SimilarEditors] Archive extension [integration/config] - 10https://gerrit.wikimedia.org/r/1334790 (https://phabricator.wikimedia.org/T436880) (owner: 10Dreamy Jazz)
[11:14:33] <wikibugs>	 (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/SimilarEditors] Archive extension [integration/config] - 10https://gerrit.wikimedia.org/r/1334790 (https://phabricator.wikimedia.org/T436880) (owner: 10Dreamy Jazz)
[11:25:07] <wikibugs>	 (03PS1) 10Matthias Mullie: zuul: Add skins/MinervaNeue dependency for ReaderExperiments [integration/config] - 10https://gerrit.wikimedia.org/r/1334824
[11:26:11] <hashar>	 !log Deleted coverage report for SimilarEditors ( /srv/doc/cover-extensions/SimilarEditors ), extension is being archived # T436880
[11:26:13] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[11:26:14] <stashbot>	 T436880: Archive the SimilarEditors extension - https://phabricator.wikimedia.org/T436880
[11:27:34] <hashar>	 Dreamy_Jazz: done! :)
[11:27:58] <Dreamy_Jazz>	 Thanks :D
[11:36:38] <wikibugs>	 (03CR) 10Hashar: [C:03+2] zuul: Add skins/MinervaNeue dependency for ReaderExperiments [integration/config] - 10https://gerrit.wikimedia.org/r/1334824 (owner: 10Matthias Mullie)
[11:38:09] <wikibugs>	 (03Merged) 10jenkins-bot: zuul: Add skins/MinervaNeue dependency for ReaderExperiments [integration/config] - 10https://gerrit.wikimedia.org/r/1334824 (owner: 10Matthias Mullie)
[12:17:25] <wikibugs>	 10GitLab, 06Release-Engineering-Team, 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work, 13Patch-For-Review: GitLab agents failing to connect - https://phabricator.wikimedia.org/T436902#12285449 (10ABran-WMF)
[12:17:27] <wikibugs>	 10GitLab, 06Release-Engineering-Team (Radar), 06Collaboration-Services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12285450 (10ABran-WMF)
[12:19:04] <wikibugs>	 10GitLab, 06Release-Engineering-Team (Radar), 06Collaboration-Services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12285456 (10ABran-WMF) thanks for mentioning it @jnuche  I sent 2 patches, one that will enable websockets on [[ https://gerrit.wikimedia.org/r/1334854 |...
[12:26:55] <wikibugs>	 (03open) 10phedenskog: Show test-only time per extension in the gated selenium job [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/59
[12:27:50] <wikibugs>	 (03merge) 10phedenskog: Show test-only time per extension in the gated selenium job [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/59
[12:31:24] <wikibugs>	 (03open) 10arnaudb: Send a User-Agent when fetching the JWKS from GitLab [repos/releng/reggie] - 10https://gitlab.wikimedia.org/repos/releng/reggie/-/merge_requests/113 (https://phabricator.wikimedia.org/T425441)
[12:33:15] <wikibugs>	 (03PS2) 10Hashar: Rename cmd to cli [integration/quibble] - 10https://gerrit.wikimedia.org/r/1293713 (https://phabricator.wikimedia.org/T300727)
[12:34:25] <wikibugs>	 (03CR) 10Hashar: "This was requested by Adam back in 2022 via T300727  and got forgotten.  Feel free to drop your self from reviewers ;)" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1293713 (https://phabricator.wikimedia.org/T300727) (owner: 10Hashar)
[13:05:17] <wikibugs>	 06Release-Engineering-Team, 10dev-images, 10MediaWiki-Core-Profiler, 10MediaWiki-Docker, 06MediaWiki-Core-Platform-Team (Kanban): Replace php-tideways_xhprof with php-xhprof in MediaWiki-Docker - https://phabricator.wikimedia.org/T401115#12285745 (10pmiazga) Moving back to in-progress, looks like this is...
[13:08:09] <wikibugs>	 10GitLab, 06Release-Engineering-Team (Radar), 06Collaboration-Services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12285756 (10ABran-WMF) Another issue was [[ https://wikimedia.slack.com/archives/C05H0JYT85V/p1788437724411059 | mentioned ]] by @trueg. As discussed in...
[13:35:50] <atsukoito>	 Hi! Is there something happening to a docker registry? I'm getting `https://registry.cloud.releng.team/v2/repos/data-engineering/airflow-dags/cache/build-test-image/blobs/uploads/: 500 Internal Server Error` from a gitlab worker
[13:36:04] <atsukoito>	 Link to a job is https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/jobs/949104
[13:38:00] <atsukoito>	 pinging hashar jnuche 
[13:41:24] <jnuche>	 atsukoito: hi, that is most likely T425441 and https://gitlab.wikimedia.org/repos/releng/reggie/-/merge_requests/113
[13:41:25] <stashbot>	 T425441: gitlab behind CDN - https://phabricator.wikimedia.org/T425441
[13:48:02] <atsukoito>	 jnuche: I was thinking about it, but it is `registry.cloud.releng.team`, not gitlab itself
[13:50:47] <jnuche>	 atsukoito: yeah, `registry.cloud.releng.team` is reggie (see MR above). reggie is reaching out to gilab itself, getting a 403 and returning a 500 to your pipeline (CI pipeline -> reggie -> gitlab -> boom)
[13:51:14] <atsukoito>	 whoa!
[13:51:56] <atsukoito>	 jnuche: thanks, now I know
[13:57:12] <wikibugs>	 (03PS1) 10Hashar: Require Python 3.11 [integration/quibble] - 10https://gerrit.wikimedia.org/r/1334901 (https://phabricator.wikimedia.org/T429715)
[13:57:14] <wikibugs>	 (03PS1) 10Hashar: Use Parenthesized Context Managers [integration/quibble] - 10https://gerrit.wikimedia.org/r/1334902 (https://phabricator.wikimedia.org/T429715)
[14:05:16] <atsukoito>	 junche: is there any blockers to merge and deploy MR113 (the one above), except the fact that CI is blocked? (because of this issue)
[14:05:42] <atsukoito>	 jnuche: (sorry typo)
[14:06:25] <wikibugs>	 06Release-Engineering-Team, 10Quibble, 13Patch-For-Review: Upgrade Quibble code base to Python3.11 - https://phabricator.wikimedia.org/T429715#12286090 (10hashar)
[14:08:06] <wikibugs>	 06Release-Engineering-Team, 10Quibble, 13Patch-For-Review: Upgrade Quibble code base to Python3.11 - https://phabricator.wikimedia.org/T429715#12286097 (10hashar)
[14:08:17] <jnuche>	 atsukoito: yeah, we can force-merge than change but I think we won't be able to push the changes (because reggie itself is currently unstable) I've asked if it's possible to disable the UA check temporarily but no response yet. Also I think the overall conversation is happening in a channel I'm not in, so I can't tell what the current status is for sure
[14:10:15] <arnaudb>	 jnuche: I don't think traffic will want to disable that rule temporarily, I'll revert the move behind CDN to give us time to fix these remaining bits
[14:10:39] <jnuche>	 arnaudb: ack, ty
[14:10:51] <atsukoito>	 I see, thanks for the update!
[14:15:11] <wikibugs>	 06Release-Engineering-Team, 10Quibble, 13Patch-For-Review: Upgrade Quibble code base to Python3.11 - https://phabricator.wikimedia.org/T429715#12286121 (10hashar)
[14:21:01] <wikibugs>	 10GitLab, 06Release-Engineering-Team (Radar), 06Collaboration-Services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12286140 (10ABran-WMF) Gitlab primary instance is back to where it was this morning to simplify debugging the missing user-agent issue.
[14:24:28] <wikibugs>	 (03PS1) 10Hashar: Use str.removesuffix() for clarity [integration/quibble] - 10https://gerrit.wikimedia.org/r/1334924 (https://phabricator.wikimedia.org/T429715)
[14:25:18] <wikibugs>	 (03update) 10arnaudb: Send a User-Agent when fetching the JWKS from GitLab [repos/releng/reggie] - 10https://gitlab.wikimedia.org/repos/releng/reggie/-/merge_requests/113 (https://phabricator.wikimedia.org/T425441)
[14:25:24] <wikibugs>	 (03update) 10arnaudb: Send a User-Agent when fetching the JWKS from GitLab [repos/releng/reggie] - 10https://gitlab.wikimedia.org/repos/releng/reggie/-/merge_requests/113 (https://phabricator.wikimedia.org/T425441)
[14:30:05] <wikibugs>	 (03merge) 10jnuche: Send a User-Agent when fetching the JWKS from GitLab [repos/releng/reggie] - 10https://gitlab.wikimedia.org/repos/releng/reggie/-/merge_requests/113 (https://phabricator.wikimedia.org/T425441) (owner: 10arnaudb)
[15:12:06] <wmcs-alerts>	 FIRING: PuppetAgentFailure: Puppet agent failure detected on instance deployment-cp-text09 in project deployment-prep   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure
[16:19:51] <btullis>	 Hi. I got a failure from `btullis@deploy1003:~$ scap sync-world --k8s-only --k8s-confirm-diff -D full_image_build:true` and I'm not quite sure what to do about it. It looks like it might have been related to pushing the new mediawiki images to the docker-registry.
[16:21:44] <btullis>	 Should I just try again, or would anyone want to check the logs, or has anyone seen this kind of thing elsewhere recently?
[16:53:45] <dancy>	 btullis: I'll take a look at your build log
[16:54:45] <dancy>	 No space left on device
[16:56:08] <dancy>	 I'm trying to run some cleanup
[16:58:24] <btullis>	 dancy: Ah, many thanks.
[16:58:52] <btullis>	 Oh, it on `/srv` on the deployment host?
[16:59:10] <dancy>	 I'm assuming so, though it's currently reporting 12GB free.
[17:00:12] <dancy>	 Please try your deployment again. I'll monitor filesystem usage while it runs
[17:01:26] <btullis>	 Thanks. Running now.
[17:02:44] <dancy>	 ok.. free space is rapidly dropping right now
[17:02:55] <dancy>	 and hit 0.
[17:03:15] <dancy>	 So, 12GB of free space is inadequate.
[17:05:38] <btullis>	 OK, the script is still running right now. It hasn't quit out.
[17:06:28] <dancy>	 That's good news.  I'm working on a fix to scap in the meantime that will address one issue w/ space recovery
[17:11:06] <btullis>	 Failed in the same way. I've got to step out for the day now, but I'll have another crack at this tomorrow. Thanks again.
[17:11:34] <dancy>	 OK. Is there anything outstanding that will affect  the next deployer?
[17:12:42] <wikibugs>	 (03update) 10dancy: clean-images: Fix the endless re-execution as docker_user [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1306
[17:12:43] <wikibugs>	 (03open) 10dancy: clean-images: Fix the endless re-execution as docker_user [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1306
[17:15:51] <wikibugs>	 (03merge) 10dancy: clean-images: Fix the endless re-execution as docker_user [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1306
[17:18:12] <wikibugs>	 (03update) 10dancy: Release 4.289.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1307
[17:18:13] <wikibugs>	 (03open) 10dancy: Release 4.289.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1307
[17:20:15] <wikibugs>	 (03merge) 10dancy: Release 4.289.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1307
[17:32:03] <dancy>	 btullis: 116GB free in /srv now.  You should be good to go tomorrow.
[17:42:15] <btullis>	 Great. Many thanks. No, nothing outstanding as far as I know. This was just about a new mediawiki-cli image to pick up new dump scripts.
[17:43:18] <dancy>	 Do you want me to run that sync-world for you?
[18:32:02] <wikibugs>	 (03open) 10dancy: spiderpig: Give each job a queue [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1308 (https://phabricator.wikimedia.org/T412941)
[18:33:39] <wikibugs>	 (03update) 10dancy: spiderpig: Give each job a queue [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1308 (https://phabricator.wikimedia.org/T412941)
[18:36:39] <wikibugs>	 (03merge) 10dancy: spiderpig: Give each job a queue [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1308 (https://phabricator.wikimedia.org/T412941)
[18:37:42] <wikibugs>	 (03open) 10dancy: jobrunner.py: Fix Jobrunner._clear_orphaned_jobs [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1309
[18:40:50] <wikibugs>	 (03merge) 10dancy: jobrunner.py: Fix Jobrunner._clear_orphaned_jobs [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1309
[18:42:19] <wikibugs>	 (03open) 10dancy: deploy-service: Add load_catalog() [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1310 (https://phabricator.wikimedia.org/T412941)
[18:45:04] <wikibugs>	 (03merge) 10dancy: deploy-service: Add load_catalog() [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1310 (https://phabricator.wikimedia.org/T412941)
[18:46:19] <wikibugs>	 (03update) 10dancy: spiderpig: Show the output that an interaction prompt asks about [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1311 (https://phabricator.wikimedia.org/T412941)
[18:46:20] <wikibugs>	 (03open) 10dancy: spiderpig: Show the output that an interaction prompt asks about [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1311 (https://phabricator.wikimedia.org/T412941)
[18:49:37] <wikibugs>	 (03merge) 10dancy: spiderpig: Show the output that an interaction prompt asks about [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1311 (https://phabricator.wikimedia.org/T412941)
[19:41:55] <wikibugs>	 06Release-Engineering-Team: Configure logstash error checks for automated supervision of Pretrain deployments - https://phabricator.wikimedia.org/T435419#12287500 (10Scott_French) No issues encountered during a test deployment that exercised the new checks. Thanks for those!  One oddity I'm having a hard time ex...
[19:47:07] <wikibugs>	 (03CR) 10Michael Große: [C:03+1] "We probably should make it habit to double-check these dependencies every once in a while." [integration/config] - 10https://gerrit.wikimedia.org/r/1334082 (https://phabricator.wikimedia.org/T436682) (owner: 10Aude)
[19:49:00] <wikibugs>	 06Release-Engineering-Team: Configure logstash error checks for automated supervision of Pretrain deployments - https://phabricator.wikimedia.org/T435419#12287531 (10Scott_French) Huh ... Odd. Yeah, if I'm understanding [[ https://logstash.wikimedia.org/goto/96759b82493fbac323882ce4a53cd1e0 | these scap logs ]]...
[21:03:35] <wikibugs>	 (03open) 10dancy: spiderpig: Deploy a service [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1312 (https://phabricator.wikimedia.org/T412941)
[22:26:28] <wikibugs>	 (03open) 10dancy: logstash_checker.py: Match the fields of a query exactly [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1313 (https://phabricator.wikimedia.org/T435419)
[22:27:30] <wikibugs>	 (03update) 10dancy: logstash_checker.py: Match the fields of a query exactly [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1313 (https://phabricator.wikimedia.org/T435419)
[22:27:33] <wikibugs>	 (03update) 10dancy: logstash_checker.py: Match the fields of a query exactly [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1313 (https://phabricator.wikimedia.org/T435419)
[22:30:24] <wikibugs>	 (03update) 10dancy: logstash_checker.py: Match the fields of a query exactly [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1313 (https://phabricator.wikimedia.org/T435419)
[22:36:30] <wikibugs>	 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T430837#12288013 (10dancy) 05Open→03Resolved
[22:43:23] <wikibugs>	 (03update) 10dancy: spiderpig: Deploy a service [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1312 (https://phabricator.wikimedia.org/T412941)
[22:56:53] <wikibugs>	 (03update) 10dancy: logstash: Match the fields of a query exactly [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1313 (https://phabricator.wikimedia.org/T435419)
[22:57:02] <wikibugs>	 (03update) 10dancy: logstash: Match the fields of a query exactly [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1313 (https://phabricator.wikimedia.org/T435419)
[22:58:32] <wikibugs>	 (03update) 10dancy: spiderpig: Deploy a service [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1312 (https://phabricator.wikimedia.org/T412941)
[23:17:14] <wikibugs>	 (03PS1) 10Krinkle: Fix accidental inclusion of Logstash requests and other non-MediaWiki traffic [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1335275
[23:18:00] <wikibugs>	 (03PS2) 10Krinkle: Fix accidental inclusion of Logstash requests and other non-MediaWiki traffic [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1335275
[23:19:52] <wikibugs>	 (03PS3) 10Krinkle: Fix accidental inclusion of Logstash requests and other non-MediaWiki traffic [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1335275
[23:22:38] <wikibugs>	 (03PS4) 10Krinkle: Fix accidental inclusion of Logstash requests and other non-MediaWiki traffic [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1335275
[23:23:02] <Krinkle>	 I don't suppose anybody else has run into this when profiling with XHGui, Logstash, or Excimer? 
[23:24:51] <wikibugs>	 (03PS5) 10Krinkle: Fix inclusion non-MediaWiki request noise in the list [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1335275
[23:42:46] <wikibugs>	 (03PS1) 10Apehitkey: zuul: add Thilio Liojo to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1335286
[23:44:38] <wikibugs>	 (03CR) 10Sohom Datta: [C:03+1] "+1, long-standing English Wikipedia contributor who has shown interest in contributing to the technical side of the project!" [integration/config] - 10https://gerrit.wikimedia.org/r/1335286 (owner: 10Apehitkey)
[23:47:16] <wikibugs>	 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: CI build slow and timing out sometimes - https://phabricator.wikimedia.org/T434024#12288059 (10Lars) Just to follow up on the power management sett...