[02:38:36] 10MediaWiki-Releasing, 10Parsoid, 05MW-1.42-release: Missing composer.json patch causes upgrade failures - https://phabricator.wikimedia.org/T382783#10431248 (10Debo) Note that my experience was a patch upgrade from 1.42.3 to 1.42.4, not using a docker image. [07:30:29] 06Release-Engineering-Team, 06collaboration-services: Low disk space: doc1003 / doc2002 - https://phabricator.wikimedia.org/T382610#10431429 (10LSobanski) p:05Triageβ†’03High [08:01:56] 10GitLab (Pipeline Services Migration🐀), 06collaboration-services, 13Patch-For-Review: move os-reports.wikimedia.org to kubernetes - https://phabricator.wikimedia.org/T350794#10431488 (10JMeybohm) Out of curiosity: Why that kubectl CronJob, additional roles etc. ? Wouldn't it be sufficient to add a second co... [09:03:56] !log gerrit: flushed diff_intraline, diff_summary, gerrit_file_diff and git_file_diff caches after having turned on diff3 style # T359821 [09:03:58] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:03:58] T359821: Configure Gerrit to use conflictStyle diff3 - https://phabricator.wikimedia.org/T359821 [09:12:34] 10Gerrit (Gerrit 3.10): Configure Gerrit to use conflictStyle diff3 - https://phabricator.wikimedia.org/T359821#10431677 (10hashar) I ended up flushing various diff caches. The link in my previous comment ( https://gerrit.wikimedia.org/r/c/mediawiki/extensions/EntitySchema/+/1046598/1/src/MediaWiki/Hooks/LoadEx... [09:55:17] 10Gerrit (Gerrit 3.10): Configure Gerrit to use conflictStyle diff3 - https://phabricator.wikimedia.org/T359821#10431786 (10Lucas_Werkmeister_WMDE) Yes, it would be pretty terrible if it changed the contents of old patch sets… hopefully we’ll see the effect next time someone backports / cherry-picks with a merge... [09:58:10] (03PS1) 10Hashar: Rollback wikimedia-portals-build to Node 18 [integration/config] - 10https://gerrit.wikimedia.org/r/1108399 (https://phabricator.wikimedia.org/T382179) [10:00:06] (03CR) 10Hashar: [C:03+2] "https://integration.wikimedia.org/ci/job/wikimedia-portals-build/ sends email notification upon failure which is a bit noisy :)" [integration/config] - 10https://gerrit.wikimedia.org/r/1108399 (https://phabricator.wikimedia.org/T382179) (owner: 10Hashar) [10:01:54] (03Merged) 10jenkins-bot: Rollback wikimedia-portals-build to Node 18 [integration/config] - 10https://gerrit.wikimedia.org/r/1108399 (https://phabricator.wikimedia.org/T382179) (owner: 10Hashar) [10:04:52] (03PS1) 10Hashar: Use node18-test image for Node 18 [integration/config] - 10https://gerrit.wikimedia.org/r/1108401 (https://phabricator.wikimedia.org/T382179) [10:05:20] (03CR) 10Hashar: [C:03+2] Use node18-test image for Node 18 [integration/config] - 10https://gerrit.wikimedia.org/r/1108401 (https://phabricator.wikimedia.org/T382179) (owner: 10Hashar) [10:06:11] (03CR) 10Hashar: [C:03+2] Rollback wikimedia-portals-build to Node 18 (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1108399 (https://phabricator.wikimedia.org/T382179) (owner: 10Hashar) [10:07:29] (03Merged) 10jenkins-bot: Use node18-test image for Node 18 [integration/config] - 10https://gerrit.wikimedia.org/r/1108401 (https://phabricator.wikimedia.org/T382179) (owner: 10Hashar) [10:16:18] (03update) 10hashar: fab: drop cd /tmp and run docker-pkg as normal user [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/70 (https://phabricator.wikimedia.org/T382285) [11:26:57] 10Continuous-Integration-Config, 10Quibble: Look at why wmf-quibble-selenium-php74 takes 20 minutes - https://phabricator.wikimedia.org/T382409#10432086 (10hashar) Yes my task is a dupe. I only filed a task here to publicly capture my investigation. [11:27:05] 10Continuous-Integration-Config, 10Quibble: Look at why wmf-quibble-selenium-php74 takes 20 minutes - https://phabricator.wikimedia.org/T382409#10432087 (10hashar) 05Openβ†’03Declined [11:47:29] RECOVERY - Host doc2002 is UP: PING OK - Packet loss = 0%, RTA = 30.64 ms [13:00:18] I think I might have done a dumb thing by +2'ing https://gerrit.wikimedia.org/r/c/mediawiki/vendor/+/1108396 [13:01:24] Just remove your +2 [13:01:28] And/or replace with -2 [13:01:31] won't get merged [13:02:10] codders: probably haven't got long left though ;) [13:02:31] okay. removed that [13:02:49] I got nervous - was suddenly seeing a lot of failures here: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/1101486 [13:03:01] but maybe that's not connected - the gate-and-submit wasn't done [13:03:56] It does depend on it [13:04:04] But that doesn't mean the failures are because of the vendor patch [13:04:18] ah. yeah. so maybe smart that I find out what's going on there before I plus-2 :) [13:07:03] You can do a no-op wikibase patch depends-on the vendor patch to confirm the vendor patch isn't seemingly breaking anything [13:07:13] There's very few functional code changes init [13:07:23] yeah - I'll try that in a sec. Just need to change trains now [13:07:28] thanks for helping avert disaster :) [13:13:22] 10Gerrit, 10Trust and Safety Product Sprint (Sprint Chimes (Dec. 9 - Jan. 17)): Create trust-and-safety-product Gerrit group - https://phabricator.wikimedia.org/T382243#10432310 (10kostajh) [13:19:51] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for [harsv567] - https://phabricator.wikimedia.org/T382570#10432335 (10Aklapper) 05Openβ†’03Resolved GitLab Account got approved somehow in the meantime thus resolving [13:24:03] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Rohank1010 - https://phabricator.wikimedia.org/T382397#10432337 (10Aklapper) 05Openβ†’03Stalled There is no account named `rohank1010` in GitLab so this task is not actionable. However there is an unapproved Git... [13:54:08] (03PS1) 10Jforrester: Zuul: Drop CI for REL1_41 branch, now EOL [integration/config] - 10https://gerrit.wikimedia.org/r/1108434 (https://phabricator.wikimedia.org/T376550) [14:34:39] 06Gerrit-Privilege-Requests, 10Phabricator, 10LDAP-Access-Requests, 10SRE-Access-Requests: Offboard Muhammad Jazirahly from WMF systems - https://phabricator.wikimedia.org/T383056 (10WMDE-leszek) 03NEW [14:50:12] (03CR) 10Jforrester: [C:03+2] Zuul: Drop CI for REL1_41 branch, now EOL [integration/config] - 10https://gerrit.wikimedia.org/r/1108434 (https://phabricator.wikimedia.org/T376550) (owner: 10Jforrester) [14:52:38] (03Merged) 10jenkins-bot: Zuul: Drop CI for REL1_41 branch, now EOL [integration/config] - 10https://gerrit.wikimedia.org/r/1108434 (https://phabricator.wikimedia.org/T376550) (owner: 10Jforrester) [14:58:26] !log Zuul: Drop CI for REL1_41 branch, now EOL per T376550 [14:58:28] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:58:28] T376550: Formally EOL MW 1.41 - https://phabricator.wikimedia.org/T376550 [15:12:46] 06Gerrit-Privilege-Requests, 10LDAP-Access-Requests, 10SRE-Access-Requests: Offboard Muhammad Jazirahly from WMF systems - https://phabricator.wikimedia.org/T383056#10432815 (10Aklapper) [15:12:51] 06Gerrit-Privilege-Requests, 10LDAP-Access-Requests, 10SRE-Access-Requests: Offboard Muhammad Jazirahly from WMF systems - https://phabricator.wikimedia.org/T383056#10432816 (10Aklapper) [15:14:37] (03open) 10jforrester: branches: Drop REL1_41, now EOL [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/50 [15:15:29] (03merge) 10jforrester: branches: Drop REL1_41, now EOL [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/50 [15:58:26] (03approved) 10dancy: fab: drop cd /tmp and run docker-pkg as normal user [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/70 (https://phabricator.wikimedia.org/T382285) (owner: 10hashar) [16:01:12] 10Gerrit: Gerrit notification emails are missing the content of inline comments on unchanged files - https://phabricator.wikimedia.org/T355259#10432941 (10Paladox) A upstream issue was filed that I think is related to this: https://issues.gerritcodereview.com/issues/387440281 [16:02:02] 10Gerrit: Gerrit notification emails are missing the content of inline comments on unchanged files - https://phabricator.wikimedia.org/T355259#10432943 (10Paladox) Bug caused by https://gerrit-review.googlesource.com/c/gerrit/+/300007/ at least according to https://gerrit-review.googlesource.com/c/gerrit/+/30000... [16:09:09] (03PS1) 10Aklapper: Remove Flow from being listed on doc.wikimedia.org frontpage [integration/docroot] - 10https://gerrit.wikimedia.org/r/1108446 (https://phabricator.wikimedia.org/T379671) [16:10:39] andre: Is this a Phab new feature? https://phabricator.wikimedia.org/mail/ [16:10:55] I was looking for the phab component called Mail and noticed there was another search result :) [16:12:38] Krinkle: AFAIK it has always existed, just not sure what it's really good for :P [16:24:07] 10MediaWiki-Releasing, 10ExtensionDistributor, 05MW-1.43-release: Mark REL1_43 in ExtensionDistributor as a stable release - https://phabricator.wikimedia.org/T372331#10433060 (10MacFan4000) 05Openβ†’03Resolved [16:34:05] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 06cloud-services-team, 10Cloud-VPS, and 2 others: Various CI jobs running in the integration Cloud VPS project failing due to transient DNS lookup failures, often for our own hosts... - https://phabricator.wikimedia.org/T374830#10433117 [16:39:00] 10Continuous-Integration-Infrastructure (Zuul upgrade): Identify the different classes of jobs - https://phabricator.wikimedia.org/T383067 (10hashar) 03NEW [16:42:03] 10Phabricator, 06collaboration-services: phabricator long running queries - https://phabricator.wikimedia.org/T383068 (10Marostegui) 03NEW [17:02:09] (03open) 10esanders: Don't add wikimedia/mediawiki to server configs [repos/ci-tools/libup] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/66 [17:21:40] 06Release-Engineering-Team, 06collaboration-services, 06Security-Team, 07SecTeam-Processed, and 2 others: CI token credentials exposed by public git config - https://phabricator.wikimedia.org/T382230#10433341 (10sbassett) [17:22:45] (03CR) 10Pppery: [C:03+1] Remove Flow from being listed on doc.wikimedia.org frontpage [integration/docroot] - 10https://gerrit.wikimedia.org/r/1108446 (https://phabricator.wikimedia.org/T379671) (owner: 10Aklapper) [17:55:55] 10Phabricator, 06collaboration-services: phabricator long running queries - https://phabricator.wikimedia.org/T383068#10433518 (10Dzahn) We have a couple puppetized scripts that fetch various metrics from Phabricator but these queries don't look like any of them. I am going to investigate some more but it doe... [18:19:09] !log Issued a new Puppet client cert for traindev01.deployment-prep.eqiad1.wikimedia.cloud [18:19:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:20:45] andrewbogott: ^ giving traindev01.deployment-prep.eqiad1.wikimedia.cloud a new cert fixed it. I wonder if the automated code for the project-local puppetserver switch is just broken these days? [18:21:25] oh, maybe these are new VMs that never worked? [18:21:53] That seemed to be the case for that instance anyway [18:22:26] like it worked well enough at some point that ssh was setup, but it was not properl attached to the project local puppetserver [18:22:52] ok, I'll go through the list an see if that's all of them [18:25:01] * bd808 puts his keyboard down on deployment-restbase-bullseye.deployment-prep.eqiad1.wikimedia.cloud [18:26:40] nope, it's all different broken puppet catalogs [18:26:44] at least, so far it is [18:27:53] please fix the restbase hosts if you're willing! I'll start at the end of the list with etcd02 [18:38:37] !log Fixed incorrect deployment-restbase prefix puppet setting that was causing puppet run failures [18:38:38] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:52:12] bd808: any idea what these alerts like "Puppet CA certificate kafka_jumbo-deployment-prep_broker is about to expire in -574d 22h 6m 12s" are about? Is that just a deleted VM with an orphaned cert, or is kafka abusing the puppetserver for some kind of kakfa marshalling magic? [18:53:07] !log Fixed missing profile::swift::global_account_keys::{codfw, eqiad} placeholders breaking deployment-ms-* puppet runs [18:53:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:54:58] andrewbogott: no idea. maybe ottomata would at least know what to start looking for? [18:55:09] ok [18:55:21] I will at least not delete them :) [18:55:55] The things I am seeing at the moment are just good old fashioned puppet rot/never setup to work in the first place [18:55:59] I need to step out for a bit. Feel free to stop fixing puppet as soon as you're tired of it and I'll check the dash when I'm back and do what's left. [18:56:00] Thanks! [18:57:00] * bd808 hasn't done this clean up buster puppet run slog for a long time [18:58:44] andrewbogott: the certificate used for tls used to be signed by puppetmaster via cergen https://wikitech.wikimedia.org/wiki/Cergen [18:59:04] I don't know if that happens anymore in prod (probably not). and likely the deployment prep kafka brokers haven't been touchedin ages [19:00:03] I would say about 3 years from the "Puppet CA certificate kafka_mirror_maker is about to expire in -574d 23h 12m 12s" warnings [19:00:14] 10Gerrit, 06Release-Engineering-Team, 10Projects-Cleanup: Prevent Gerrit archived repositories from being replicated to GitHub - https://phabricator.wikimedia.org/T351543#10433764 (10thcipriani) Other idea: we could have some bot go around and move self-service archived repos into All-Archived-Projects Β―\_(ツ... [19:05:32] (03approved) 10brennen: bookworm-apache2: Workaround Apache2 crash under QEMU emulation [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/71 (https://phabricator.wikimedia.org/T382709) (owner: 10bd808) [19:06:39] (03merge) 10brennen: fab: drop cd /tmp and run docker-pkg as normal user [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/70 (https://phabricator.wikimedia.org/T382285) (owner: 10hashar) [19:08:13] (03update) 10brennen: bookworm-apache2: Workaround Apache2 crash under QEMU emulation [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/71 (https://phabricator.wikimedia.org/T382709) (owner: 10bd808) [19:10:02] (03merge) 10brennen: bookworm-apache2: Workaround Apache2 crash under QEMU emulation [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/71 (https://phabricator.wikimedia.org/T382709) (owner: 10bd808) [19:11:27] !log Added placeholders for `graphite_host` and `statsd` to deployment-webperf Prefix Puppet [19:11:27] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:13:45] 10Phabricator, 06collaboration-services: phabricator long running queries - https://phabricator.wikimedia.org/T383068#10433797 (10Aklapper) > Is this known? It wasn't known to me. Only thing I could find in old upstream which sounds slightly related is https://secure.phabricator.com/T12993 [19:15:48] !log Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/71 (T382709) [19:15:50] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:15:50] T382709: Since the update to bookworm, unable to start mediawiki-web on my M2 Mac - https://phabricator.wikimedia.org/T382709 [19:25:03] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Buster Deprecation): Replace deployment-maps-master01 with a Bullseye or Bookworm instance - https://phabricator.wikimedia.org/T361381#10433836 (10bd808) 05Resolvedβ†’03Open Not sure why @andrew closed this as resolved. `lang=shell-session bd808@deployment-m... [19:27:07] !log Added `postgresql::postgis::postgresql_postgis_package: ignored` to deployment-maps Prefix Puppet to work around default parameter problem (T361381) [19:27:09] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:27:10] T361381: Replace deployment-maps-master01 with a Bullseye or Bookworm instance - https://phabricator.wikimedia.org/T361381 [19:31:47] !log Issued new Puppet cert for deployment-maps-master02.deployment-prep.eqiad1.wikimedia.cloud (T361381) [19:31:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:32:14] !log Added `postgresql::postgis::postgresql_postgis_package: postgresql-15-postgis-3` to deployment-maps Prefix Puppet to work around default parameter problem (T361381) [19:32:16] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:32:16] T361381: Replace deployment-maps-master01 with a Bullseye or Bookworm instance - https://phabricator.wikimedia.org/T361381 [19:35:21] !log Manually generated missing en_US.UTF-8 locale on deployment-maps-master02.deployment-prep.eqiad1.wikimedia.cloud (T361381) [19:35:23] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:38:07] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Buster Deprecation): Replace deployment-maps-master01 with a Bullseye or Bookworm instance - https://phabricator.wikimedia.org/T361381#10433901 (10bd808) `counterexample $ sudo -i puppet agent -tv ... Error: Execution of '/usr/bin/apt-get -q -y -o DPkg::Option... [19:38:52] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Buster Deprecation): Replace deployment-maps-master01 with a Bullseye or Bookworm instance - https://phabricator.wikimedia.org/T361381#10433905 (10bd808) p:05Triageβ†’03High [19:42:03] (03open) 10cdanis: Fix OTEL configuration [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/596 [19:50:21] 10Phabricator: Document project picture size constraints - https://phabricator.wikimedia.org/T383083 (10thcipriani) 03NEW [20:09:21] ottomata: so what do you suggest be done (if anything) about those certs and/or kafka in deployment-prep? [20:21:52] 10Phabricator: Document project picture size constraints - https://phabricator.wikimedia.org/T383083#10434056 (10Aklapper) I'd love to know the pixel dimensions of the original custom image. If I understand correctly, a custom image was uploaded as project profile image, but not rendered correctly and the myste... [20:43:00] 06Release-Engineering-Team, 06collaboration-services: Low disk space: doc1003 / doc2002 - https://phabricator.wikimedia.org/T382610#10434155 (10Arnoldokoth) doc2002 looks alright now. ` aokoth@doc2002:~$ sudo df -ht ext4 Filesystem Size Used Avail Use% Mounted on /dev/vda1 117G 7.3G 104G 7% / /... [20:45:16] 10Phabricator: Document project picture size constraints - https://phabricator.wikimedia.org/T383083#10434158 (10HCoplin-WMF) Original image dimensions were 4672β€ŠΓ—β€Š5941, 3.8 MB. I then trimmed it down to be more square, resulting in 4667x4763, 4.3 MB. The final image that worked was 400x400, 91 KB. It seems li... [20:52:28] andrewbogott: the alert board looks better. That maps instance needs someone who knows about maps to actually take a look I think. It appeared to have never had a full puppet run since it was created in July. [20:52:56] I reopened the T361381 ticket about it [20:52:56] T361381: Replace deployment-maps-master01 with a Bullseye or Bookworm instance - https://phabricator.wikimedia.org/T361381 [20:53:09] OK, thanks for reopening. I guess we'll see if anyone cares/notices that it doesn't work :( [20:54:46] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Buster Deprecation): Replace deployment-maps-master01 with a Bullseye or Bookworm instance - https://phabricator.wikimedia.org/T361381#10434225 (10bd808) [20:56:55] andrewbogott: there is a very recent T381565 for the prod hosts, so maybe it will get fixed by accident... [20:56:55] T381565: Move maps servers to Bookworm - https://phabricator.wikimedia.org/T381565 [21:02:33] 10Phabricator: Document project picture size constraints - https://phabricator.wikimedia.org/T383083#10434360 (10thcipriani) >>! In T383083#10434158, @HCoplin-WMF wrote: > Original image dimensions were 4672β€ŠΓ—β€Š5941, 3.8 MB (https://phabricator.wikimedia.org/F58028490). I then trimmed it down to be more square, r... [21:06:27] 10Phabricator: Document project picture size constraints - https://phabricator.wikimedia.org/T383083#10434371 (10Aklapper) I can reproduce locally, it's definitely the dimensions. It is not conflicting with Phab's `storage.mysql-engine.max-size` setting (which, uhm, differs between production and GitLab) and it'... [21:06:48] 10Phabricator: Document project picture dimension constraints - https://phabricator.wikimedia.org/T383083#10434372 (10Aklapper) p:05Triageβ†’03Low [21:14:33] 10Beta-Cluster-Infrastructure, 06Data-Engineering, 06Data-Engineering-Radar: Many kafka errors in beta/deployment-prep - https://phabricator.wikimedia.org/T346402#10434472 (10VirginiaPoundstone) [21:33:45] 06Release-Engineering-Team, 06Data-Engineering, 06Data-Platform-SRE, 10Dumps-Generation, and 4 others: WE 5.4 KR - Hypothesis 5.4.4 - Q3 FY24/55 - Migrate current-generation dumps to run on kubernetes - https://phabricator.wikimedia.org/T352650#10434814 (10VirginiaPoundstone) [21:41:37] 10Beta-Cluster-Infrastructure: deployment-webperf21 puppet failure: Could not find class role::webperf::processors_and_site - https://phabricator.wikimedia.org/T358332#10435215 (10bd808) 05Openβ†’03Invalid Fixed by magic in the intervening months: `lang=shell-session bd808@deployment-webperf21:~$ sudo -i p... [21:44:28] 10Beta-Cluster-Infrastructure: Puppet failing on deployment-sessionstore04 due to lack of free memory - https://phabricator.wikimedia.org/T332524#10435290 (10bd808) 05Openβ†’03Resolved a:03Southparkfan Replaced by https://openstack-browser.toolforge.org/server/deployment-sessionstore06.deployment-prep.eq... [21:44:38] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Buster Deprecation): Replace deployment-maps-master01 with a Bullseye or Bookworm instance - https://phabricator.wikimedia.org/T361381#10435300 (10bd808) [21:44:44] 10Phabricator, 06collaboration-services: phabricator long running queries - https://phabricator.wikimedia.org/T383068#10435304 (10Dzahn) This comment seems interesting: ". The MySQL setting optimizer_search_depth=0 may fix the weird explosive complexity here." [21:44:52] 10Beta-Cluster-Infrastructure, 10Maps: deployment-maps-master01 puppet failures - https://phabricator.wikimedia.org/T311609#10435303 (10bd808) β†’14Duplicate dup:03T361381 [21:45:12] 10Beta-Cluster-Infrastructure: Puppet failing on deployment-kafka-jumbo-3 due to memory pressure - https://phabricator.wikimedia.org/T286358#10435332 (10bd808) 05Openβ†’03Resolved a:03Ottomata Replaced by https://openstack-browser.toolforge.org/server/deployment-kafka-jumbo-5.deployment-prep.eqiad1.wiki... [21:55:33] 10Beta-Cluster-Infrastructure, 07Puppet: puppetmaster config in deployment-prep may be inadvertently breaking store,logstash reports? - https://phabricator.wikimedia.org/T218175#10435529 (10bd808) 05Openβ†’03Resolved a:03bd808 At some point in the long life of this bug we moved to Puppet 7 and a new se... [22:05:02] 10Beta-Cluster-Infrastructure: Multiple kafka Cergen certs expired in beta cluster - https://phabricator.wikimedia.org/T383096#10435542 (10bd808) @elukey would you know if we just need to delete the old certs manually or if there is likely to be more config needed to switch the beta cluster nodes to the PKI syst... [22:34:44] 06Release-Engineering-Team, 06collaboration-services, 10dev-images, 10docker-pkg, 13Patch-For-Review: Permission denied while running dev-images/fab deploy_devimages - https://phabricator.wikimedia.org/T382285#10435677 (10brennen) 05Openβ†’03Resolved a:03hashar