[00:01:35] <jinxer-wm>	 (DatasourceError) firing: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError   - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError
[00:06:50] <jinxer-wm>	 (DatasourceError) resolved: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError   - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError
[08:44:48] <wikibugs>	 (03CR) 10Hashar: Add rsync to ci-{buster,bullseye,bookworm} images. (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/983892 (owner: 10Ahmon Dancy)
[09:13:32] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team: Deal with git safe.directory in CI Docker images - https://phabricator.wikimedia.org/T354409 (10hashar)
[09:13:35] <wikibugs>	 10Release-Engineering-Team (Priority Backlog 📥), 10Security-Team, 10SecTeam-Processed, 10Security, 10Vuln-Misconfiguration: Debian security update for git silently broke mediawiki-i18n-check-docker - https://phabricator.wikimedia.org/T329266 (10hashar)
[09:17:22] <wikibugs>	 (03PS1) 10Hashar: dockerfiles: disable git safe.directory [integration/config] - 10https://gerrit.wikimedia.org/r/988408 (https://phabricator.wikimedia.org/T354409)
[09:18:28] <jakob_WMDE>	 bonjour hashar, we're seeing the npm cache corruption errors again in our CI e.g. https://integration.wikimedia.org/ci/job/mwgate-node16-docker/99445/console - should I create a ticket?
[09:19:52] <wikibugs>	 (03CR) 10Hashar: "The generic task is T335354 and that git `safe.directory` "feature" has been hitting us all other the place. In all/most case we end up di" [integration/config] - 10https://gerrit.wikimedia.org/r/988408 (https://phabricator.wikimedia.org/T354409) (owner: 10Hashar)
[09:20:45] <hashar>	 jakob_WMDE: I will nuke it :)
[09:20:55] <jakob_WMDE>	 thank you! :)
[09:22:50] <hashar>	 !log integration: integration-castor05: rm -fR /srv/castor/castor-mw-ext-and-skins/master/mwgate-node16-docker/
[09:22:52] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[09:23:40] <hashar>	 jakob_WMDE: it is gone!
[09:24:16] <jakob_WMDE>	 yay!
[09:46:52] <wikibugs>	 10Gerrit, 10Release-Engineering-Team (Quid Pro Crow 🦃): Add MoTD to gerrit - https://phabricator.wikimedia.org/T351109 (10hashar) The Gerrit plugin displaying the banner has been removed by https://gerrit.wikimedia.org/r/c/operations/software/gerrit/+/987995  End users who dismissed the banner have a `wm-motd-...
[10:07:17] <wikibugs>	 10Release-Engineering-Team (Radar), 10DBA, 10MediaWiki-libs-Rdbms, 10Observability-Logging, and 2 others: Transaction profiler logs full query which is truncated by logstash - https://phabricator.wikimedia.org/T349140 (10Ladsgroup) 05Open→03Resolved a:03Ladsgroup This is fixed.
[10:09:30] <wikibugs>	 10Release-Engineering-Team (Radar), 10DBA, 10MediaWiki-libs-Rdbms, 10Observability-Logging, and 2 others: Transaction profiler logs full query which is truncated by logstash - https://phabricator.wikimedia.org/T349140 (10hashar) @Ladsgroup thank you for the patches and to have verified the truncated logs v...
[10:12:03] <Amir1>	 hashar: There are some jsonTruncated left but they are php parser stuff
[10:12:12] <Amir1>	 and very small, like 100 in span of a week
[10:12:40] <Amir1>	 https://logstash.wikimedia.org/app/discover#/doc/logstash-*/logstash-mediawiki-1-7.0.0-1-2024.01.08?id=IaR16IwBRtLP5wy61ABE
[10:13:39] <wikibugs>	 10Project-Admins: Create project tag for Lingua Libre - https://phabricator.wikimedia.org/T354458 (10Aklapper) 05Open→03Resolved a:03Aklapper Done in https://phabricator.wikimedia.org/project/view/3393/ and https://phabricator.wikimedia.org/project/view/6913/
[10:37:30] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team (Quid Pro Crow 🦃), 10Language-Team (Language-2024-January-March), 10affects-translatewiki.net: jenkins.job=mediawiki-i18n-check-docker fails without a real error - https://phabricator.wikimedia.org/T331098 (10Pginer-WMF)
[11:06:37] <wikibugs>	 10Phabricator (phabricator-next), 10Patch-For-Review: Disallow creating projects with an empty description - https://phabricator.wikimedia.org/T344610 (10Aklapper) Note to myself: https://github.com/Wikimedia-Sverige/project-start/blob/main/phab.py is fine.
[11:16:05] <wikibugs>	 10Project-Admins: Create project tag for Lingua Libre - https://phabricator.wikimedia.org/T354458 (10Poslovitch) Thanks!
[13:02:39] <andre>	 brennen: Hej hej. Could I get a review of https://gerrit.wikimedia.org/r/c/phabricator/antivandalism/+/984213 and maybe ask for another full Phab deployment this week?  *deer eyes*
[13:48:04] <wikibugs>	 10GitLab (Pipeline Services Migration🐤), 10Wikimedia-Design, 10collaboration-services, 10Design: move design.wikimedia.org to kubernetes - https://phabricator.wikimedia.org/T350791 (10Jelto)
[13:51:52] <wikibugs>	 10GitLab (Pipeline Services Migration🐤), 10Wikimedia-Design, 10collaboration-services, 10Design: move design.wikimedia.org to kubernetes - https://phabricator.wikimedia.org/T350791 (10Jelto) All three design services (design-landing-page, design-blog and design-strategy) were deployed to the wikikube clust...
[14:14:53] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] "Let's do this." [integration/config] - 10https://gerrit.wikimedia.org/r/988408 (https://phabricator.wikimedia.org/T354409) (owner: 10Hashar)
[14:17:25] <wikibugs>	 (03Merged) 10jenkins-bot: dockerfiles: disable git safe.directory [integration/config] - 10https://gerrit.wikimedia.org/r/988408 (https://phabricator.wikimedia.org/T354409) (owner: 10Hashar)
[14:19:47] <James_F>	 !log Docker: Publishing new CI images with git safe.directory disabled for T354409
[14:19:50] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[14:19:51] <stashbot>	 T354409: Deal with git safe.directory in CI Docker images - https://phabricator.wikimedia.org/T354409
[14:48:24] <hashar>	 James_F: I think there are a couple images that fail to build
[14:48:39] <hashar>	 `lintr` and the one for cargo
[14:48:40] <James_F>	 It's still going…
[14:48:44] <hashar>	 s/cargo/rust/
[14:48:55] <hashar>	 but those build failures are I think unrelated to the recent patches
[14:48:58] <James_F>	 Oh, right. Joy. Existing build issue this uncovers?
[14:48:59] <James_F>	 Right.
[14:50:35] <hashar>	 I am dropping the lintr image
[14:50:51] <James_F>	 Do we use it?
[14:51:51] <hashar>	 for Gerrit repo analytics/wmde/WDCM  , last touched in 2020
[14:51:58] <James_F>	 Fun.
[14:52:05] <James_F>	 Wikidata community metrics?
[14:52:22] <James_F>	 Ah, no, https://www.wikidata.org/wiki/Wikidata:Wikidata_Concepts_Monitor
[14:52:57] <James_F>	 And per the talk page it stopped working a year ago, so yeah, kill it.
[14:53:29] <hashar>	 OHH
[14:56:55] <Krinkle>	 hashar: once jobs are updated to use the latest quibble images that include the git safe.directory change, does that mean firefox-esr will also be present? Or was that separately reverted?
[14:58:53] <James_F>	 Krinkle: It'll be part of the new images, yes.
[14:59:13] <James_F>	 And then I need to land https://gerrit.wikimedia.org/r/c/integration/config/+/987814 to push out php-ast 1.1.1, so lots of image changes right now. :-(
[14:59:46] <hashar>	 so that analytics/wmde/WDCM repo is in Puppet so I guess it is production
[14:59:55] <hashar>	 I will just ignore it and the related releng/lintr docker image
[15:00:29] <James_F>	 We can keep the repo in prod but drop its CI.
[15:01:22] <hashar>	 Krinkle: some image n got created which included firefox-esr  but the image is broken because its parent image got rebuild to use a newer git version
[15:02:02] <hashar>	 Krinkle: the one James is building would still apt install firefox-esr  AND   has a system-wide git config to disable the git feature introduced in the new parent image
[15:02:45] <hashar>	 the whole issue is that those images are not reproducible and get magically "rebased" with various unwanted changes
[15:03:22] <hashar>	 we'd need a way to compose an image by hand picking the layers we want in the final image
[15:03:24] <hashar>	 or 
[15:03:51] <hashar>	 pin every single Debian package but apt / Debian is not really thought for that use case
[15:19:48] <James_F>	 (Docker build still going, over an hour later.)
[15:21:58] <wikibugs>	 10Continuous-Integration-Config, 10Projects-Cleanup, 10Wikimedia-GitHub: Archive Wikidata Concepts Monitor repositories - https://phabricator.wikimedia.org/T354534 (10hashar)
[15:28:33] <wikibugs>	 (03PS1) 10Hashar: zuul: archive WDCM and lintr jobs/image [integration/config] - 10https://gerrit.wikimedia.org/r/988653 (https://phabricator.wikimedia.org/T354534)
[15:28:52] <hashar>	 James_F: for lintr / WDCM  ^^^ :)
[15:28:58] <James_F>	 Ack.
[15:29:14] <James_F>	 docker-registry.discovery.wmnet/releng/rust-coverage also fails BTW.
[15:29:23] <hashar>	 if it passes I will +2 it
[15:29:27] <hashar>	 for rust I have no clue
[15:29:51] <hashar>	 at first I am wondering why we went to start adopting rust when I am guessing in all cases we would have been fine with just python
[15:29:54] <hashar>	 but I digress
[15:30:21] <hashar>	 the failure might be easy to fix up, my idea was to file it in Phabricator and CC Kunal on it
[15:30:58] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] zuul: archive WDCM and lintr jobs/image [integration/config] - 10https://gerrit.wikimedia.org/r/988653 (https://phabricator.wikimedia.org/T354534) (owner: 10Hashar)
[15:31:26] <wikibugs>	 (03CR) 10Lucas Werkmeister (WMDE): Zuul: [mediawiki/extensions/Wikibase] Disable Phan on release branches (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/984222 (https://phabricator.wikimedia.org/T226945) (owner: 10Lucas Werkmeister (WMDE))
[15:33:16] <wikibugs>	 (03Merged) 10jenkins-bot: zuul: archive WDCM and lintr jobs/image [integration/config] - 10https://gerrit.wikimedia.org/r/988653 (https://phabricator.wikimedia.org/T354534) (owner: 10Hashar)
[15:33:58] <wikibugs>	 10Continuous-Integration-Config, 10Projects-Cleanup, 10Wikidata, 10Wikidata Analytics, and 3 others: Archive Wikidata Concepts Monitor repositories - https://phabricator.wikimedia.org/T354534 (10Lucas_Werkmeister_WMDE)
[15:34:22] <hashar>	 !log gerrit: arrchived analytics/wmde/WDCM* repositories # T354534
[15:34:25] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[15:34:25] <stashbot>	 T354534: Archive Wikidata Concepts Monitor repositories - https://phabricator.wikimedia.org/T354534
[15:35:19] <hashar>	 Lucas_WMDE: hopefully WDCM is actually no more used, at least the Gerrit repositories did not get any activity since November 2020 :)
[15:37:46] <Lucas_WMDE>	 we’re cleaning up some related things but I have no idea about the details
[15:38:08] <Lucas_WMDE>	 I sure hope whatever you just did is reversible
[15:39:25] <Lucas_WMDE>	 (since you linked to T351072: as I understand it, that task is specifically about the clone of one git repo on one machine; it doesn’t necessarily imply that all of WDCM should be thrown away)
[15:39:26] <stashbot>	 T351072: Remove the WDCM clone (stats1007) - https://phabricator.wikimedia.org/T351072
[15:40:07] <hashar>	 Lucas_WMDE: yeah that can be reverted if needed
[15:40:12] <Lucas_WMDE>	 ok
[15:40:19] <wikibugs>	 10Continuous-Integration-Config, 10Projects-Cleanup, 10Wikidata, 10Wikidata Analytics, and 3 others: Archive Wikidata Concepts Monitor repositories - https://phabricator.wikimedia.org/T354534 (10AndrewTavis_WMDE) Hey @hashar 👋 For clarification, could I just check that what we're discussing here would shut...
[15:40:29] <hashar>	 it is merely because the releng/lintr docker image fails to build for whatever reason
[15:40:58] <hashar>	 and instead of investigating the build failure (which might be easy) I went to get rid of the image entirely since the sole repo using it is apparently abandoned :)
[15:42:05] <hashar>	 for T351072, my guess it is about removing the git clone from Puppet, removing the clone from that stats* machine and verify it is not breaking anything (which it should not since the task states all cronjobs have been removed)
[15:47:39] <hashar>	 !log phabricator: archiving diffusions repos rAWWO rAWWS rAWWB rAWWD rAWWU rAWCM rAWWV rAWWH rAWWJ rWDGD rAWWT # T354534
[15:47:42] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[15:47:42] <stashbot>	 T354534: Archive Wikidata Concepts Monitor repositories - https://phabricator.wikimedia.org/T354534
[15:48:28] <hashar>	 !log Reloaded Zuul for https://gerrit.wikimedia.org/r/988653 # T354534
[15:48:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[15:55:12] <wikibugs>	 10Continuous-Integration-Config, 10Projects-Cleanup, 10Wikidata, 10Wikidata Analytics, and 3 others: Archive Wikidata Concepts Monitor repositories - https://phabricator.wikimedia.org/T354534 (10hashar) 05Open→03Resolved a:03hashar >>! In T354534#9441977, @AndrewTavis_WMDE wrote: > Hey @hashar 👋 For...
[15:56:32] <wikibugs>	 (03CR) 10Hashar: "I haven't had a chance to process this change yet, notably I don't know what kind of side effect / breakage that will lead to :)" [integration/config] - 10https://gerrit.wikimedia.org/r/983476 (https://phabricator.wikimedia.org/T347407) (owner: 10Kosta Harlan)
[16:02:53] <legoktm>	 James_F, hashar: I'll double check, but I think we can drop the Rust jobs/images. I moved all my Rust projects to GitLab already
[16:06:35] <wikibugs>	 10Gerrit, 10Upstream: Syntax highlightning of Embedded Puppet (EPP) files in Gerrit - https://phabricator.wikimedia.org/T339096 (10joanna_borun)
[16:06:43] <James_F>	 Awesome.
[16:17:33] <wikibugs>	 10GitLab (Account Approval), 10Release-Engineering-Team: Requesting GitLab account activation for Deltaspace42 - https://phabricator.wikimedia.org/T354011 (10brennen) 05Open→03Resolved a:03brennen
[16:19:05] <wikibugs>	 (03CR) 10Kosta Harlan: jjb: Install MediaWiki in codehealth coverage jobs (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/983476 (https://phabricator.wikimedia.org/T347407) (owner: 10Kosta Harlan)
[16:24:59] <wikibugs>	 10Release-Engineering-Team, 10Gerrit (Gerrit 3.6): JenkinsCiPluginsRepository : No plugins available for Gerrit version 3.5.6 - https://phabricator.wikimedia.org/T354464 (10hashar) 05Open→03Declined That is the plugin manager plugin which is not able to retrieve the list of plugins for Gerrit 3.5.x series...
[16:31:58] <wikibugs>	 10Continuous-Integration-Config, 10Projects-Cleanup, 10Wikidata, 10Wikidata Analytics, and 2 others: Archive Wikidata Concepts Monitor repositories - https://phabricator.wikimedia.org/T354534 (10AndrewTavis_WMDE) Thanks for the information, @hashar! I'll communicate the above to the rest of the team over h...
[16:33:47] <wikibugs>	 10Phabricator, 10collaboration-services, 10Patch-For-Review: automate data syncing between phabricator servers? - https://phabricator.wikimedia.org/T354221 (10Jelto) p:05Triage→03Medium
[16:34:25] <wikibugs>	 10Phabricator, 10collaboration-services, 10Patch-For-Review: automate data syncing between phabricator servers? - https://phabricator.wikimedia.org/T354221 (10Jelto) a:03Dzahn
[16:35:35] <wikibugs>	 (03PS2) 10Jforrester: dockerfiles: [php-ast] Install php-ast v1.1.1 and cascade [integration/config] - 10https://gerrit.wikimedia.org/r/987814
[16:56:55] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] dockerfiles: [php-ast] Install php-ast v1.1.1 and cascade [integration/config] - 10https://gerrit.wikimedia.org/r/987814 (owner: 10Jforrester)
[16:59:09] <wikibugs>	 (03Merged) 10jenkins-bot: dockerfiles: [php-ast] Install php-ast v1.1.1 and cascade [integration/config] - 10https://gerrit.wikimedia.org/r/987814 (owner: 10Jforrester)
[16:59:29] <James_F>	 !log Docker: [php-ast] Install php-ast v1.1.1 and cascade
[16:59:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[17:13:16] <wikibugs>	 (03CR) 10Lucas Werkmeister (WMDE): "If it helps, I could also create a separate Phabricator task for the issue in Wikibase release branches; I noticed it while backporting th" [integration/config] - 10https://gerrit.wikimedia.org/r/984222 (https://phabricator.wikimedia.org/T226945) (owner: 10Lucas Werkmeister (WMDE))
[17:14:33] <strainu>	 Hey folks, your team is marked as code steward for the operations/media wiki-config repo.
[17:14:50] <strainu>	 Is this the right place to request a review for https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/972473?
[17:15:04] <strainu>	 It already has a +1
[17:16:32] <Reedy>	 strainu: Usually you mostly just need to schedule it for deploy/backport
[17:17:14] <strainu>	 Can you point me to some instructions Reedy?
[17:18:52] <James_F>	 strainu: https://wikitech.wikimedia.org/wiki/Wikimedia_site_requests
[17:19:08] <James_F>	 Or https://meta.wikimedia.org/wiki/Requesting_wiki_configuration_changes
[17:19:48] <strainu>	 Thanks!
[17:21:41] <James_F>	 Always. :-)
[18:15:16] <wikibugs>	 10Phabricator (phabricator-next): Deploy Phabricator/Phorge 2024-01-09 - https://phabricator.wikimedia.org/T354545 (10brennen)
[18:20:02] <wikibugs>	 (03PS1) 10Jforrester: jjb: Update all PHP-based images for new ast (and git and firefox-esr) [integration/config] - 10https://gerrit.wikimedia.org/r/988683
[18:23:51] <wikibugs>	 (03PS2) 10Jforrester: jjb: Update all PHP-based images for new ast (and git and firefox-esr) [integration/config] - 10https://gerrit.wikimedia.org/r/988683
[18:26:48] <wikibugs>	 10Phabricator (phabricator-next), 10Release-Engineering-Team, 10User-brennen: Deploy Phabricator/Phorge 2024-01-09 - https://phabricator.wikimedia.org/T354545 (10brennen)
[18:42:33] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/988683 (owner: 10Jforrester)
[18:44:24] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: Update all PHP-based images for new ast (and git and firefox-esr) [integration/config] - 10https://gerrit.wikimedia.org/r/988683 (owner: 10Jforrester)
[19:02:19] <wikibugs>	 (03PS1) 10Jforrester: jjb: Update misc. jobs for git bump [integration/config] - 10https://gerrit.wikimedia.org/r/988706
[19:35:59] <wikibugs>	 10Phabricator, 10collaboration-services, 10PM: Merge the Phabricator Priority values "Low" and "Lowest" - https://phabricator.wikimedia.org/T228759 (10BCornwall) @Aklapper Thanks for bringing that up. I did briefly attempt to bring this up some time ago but I didn't get much traction. @mark Is this something...
[19:36:22] <wikibugs>	 10Release-Engineering-Team, 10Fundraising-Backlog, 10MediaWiki-extensions-FundraiserLandingPage, 10donate.wikimedia.org, and 3 others: donate.wikimedia.org landing pages don't have descriptions in search engine results - https://phabricator.wikimedia.org/T254808 (10Ejegg) @Pcoombe I notice the first redire...
[20:17:01] <wikibugs>	 10Release-Engineering-Team, 10Fundraising-Backlog, 10MediaWiki-extensions-FundraiserLandingPage, 10donate.wikimedia.org, and 3 others: donate.wikimedia.org landing pages don't have descriptions in search engine results - https://phabricator.wikimedia.org/T254808 (10hashar) >>! In T254808#9361374, @Ejegg wr...
[20:52:38] <wikibugs>	 10GitLab (Project Migration), 10Release-Engineering-Team, 10User-brennen: Add a /repos/gadgets namespace in Wikimedia Gitlab - https://phabricator.wikimedia.org/T353491 (10brennen)
[21:20:01] <wmf-insecte>	 Project beta-update-databases-eqiad build #73025: 04FAILURE in 0.72 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/73025/
[22:30:29] <wmf-insecte>	 Yippee, build fixed!
[22:30:29] <wmf-insecte>	 Project beta-update-databases-eqiad build #73026: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/73026/
[22:33:37] <wikibugs>	 10Beta-Cluster-Infrastructure, 10MediaWiki-Platform-Team, 10MediaWiki-ResourceLoader: Source map error: Error: request failed with status 404 - https://phabricator.wikimedia.org/T352167 (10tstarling) A source map 404 is not a bug in itself.   >>! In T352167#9377439, @AlexisJazz wrote: > Is this also a startu...