[00:25:55] <wikibugs>	 10Release-Engineering-Team, 10User-brennen: Some traffic seems to be reaching 1.39.0-wmf.19 code - https://phabricator.wikimedia.org/T313770 (10tstarling) I confirmed that mw1450 was affected and was reporting wmf.19 in Special:Version.  I confirmed that restart-php-fpm-all is still functional by running it on...
[00:31:44] <wikibugs>	 10Release-Engineering-Team, 10User-brennen: Some traffic seems to be reaching 1.39.0-wmf.19 code - https://phabricator.wikimedia.org/T313770 (10brennen) [[https://logstash.wikimedia.org/goto/1c4896b5d0bb010d76c5d48d0a35c362|From logstash]], looks like last canary restart was on the 20th.  {b616a7ff} touches ca...
[00:32:50] <wikibugs>	 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments: 1.39.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T308075 (10brennen)
[00:32:52] <wikibugs>	 10Release-Engineering-Team, 10Scap, 10User-brennen: scap no longer restarts php-fpm on canary servers - https://phabricator.wikimedia.org/T313770 (10brennen) 05Open→03In progress
[00:34:55] <brennen>	 do we have a procedure for blocking all deploys until further notice?
[00:56:50] <wikibugs>	 10Release-Engineering-Team, 10CampaignEvents, 10Campaign-Registration, 10Campaign-Tools (Campaign-Tools-Sprint-17), 10MW-1.39-notes (1.39.0-wmf.19; 2022-07-04): Release V0 of the CampaignEvents extension to the Beta Cluster - https://phabricator.wikimedia.org/T311752 (10Daimona)
[01:12:38] <brennen>	 (i s'pose i should have said "mediawiki" deploys.  anyway, sent some mail.)
[01:31:42] <wikibugs>	 10Release-Engineering-Team, 10Scap, 10User-brennen: scap no longer restarts php-fpm on canary servers - https://phabricator.wikimedia.org/T313770 (10brennen) a:03brennen
[01:35:09] <wikibugs>	 10Release-Engineering-Team, 10Scap, 10User-brennen: scap no longer restarts php-fpm on canary servers - https://phabricator.wikimedia.org/T313770 (10matmarex) Is this related to the issues we were seeing in backport windows where sometimes the patches would not take effect on some servers? (most recently tha...
[04:54:20] <wikibugs>	 10Release-Engineering-Team, 10Scap, 10User-brennen: scap no longer restarts php-fpm on canary servers - https://phabricator.wikimedia.org/T313770 (10brennen) a:05brennen→03None Un-cookie-licking.
[06:30:29] <RhinosF1>	 brennen: add a manual scap lock
[06:30:53] <RhinosF1>	 That was done last time when deploy got wiped
[06:53:29] <wikibugs>	 10Release-Engineering-Team, 10Scap: Catalog of errors from scap (or subprocesses) related to php-fpm restart always being enabled - https://phabricator.wikimedia.org/T311181 (10Joe)
[06:53:36] <wikibugs>	 10Release-Engineering-Team, 10Scap: Scap pool counter error while backporting - https://phabricator.wikimedia.org/T310835 (10Joe) 05Open→03In progress p:05Triage→03Medium a:03Joe
[07:50:14] <wikibugs>	 10Release-Engineering-Team, 10Scap: Catalog of errors from scap (or subprocesses) related to php-fpm restart always being enabled - https://phabricator.wikimedia.org/T311181 (10Joe)
[07:50:21] <wikibugs>	 10Release-Engineering-Team, 10Scap: Scap pool counter error while backporting - https://phabricator.wikimedia.org/T310835 (10Joe) 05In progress→03Resolved I deployed the new version of python3-poolcounter everywhere, tentatively resolving for now.
[08:34:44] <wikibugs>	 (03PS1) 10Giuseppe Lavagetto: Restart the canaries and testservers as well [tools/scap] - 10https://gerrit.wikimedia.org/r/817200 (https://phabricator.wikimedia.org/T313770)
[08:37:46] <wikibugs>	 10Release-Engineering-Team, 10Scap, 10Patch-For-Review, 10User-brennen: scap no longer restarts php-fpm on canary servers - https://phabricator.wikimedia.org/T313770 (10Joe) I think I have a patch that should be fixing the regression at https://gerrit.wikimedia.org/r/c/mediawiki/tools/scap/+/817200
[09:01:02] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Discovery-Search: Verify (and/or fix) Elasticsearch beta cluster problems - https://phabricator.wikimedia.org/T313521 (10dom_walden) >>! In T313521#8102091, @Gehel wrote: > Not clear what is broken. Nothing obvious was found.  @Gehel We are still seeing lots of errors of the fo...
[09:27:29] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Discovery-Search: Verify (and/or fix) Elasticsearch beta cluster problems - https://phabricator.wikimedia.org/T313521 (10dom_walden) @bking Do the 3 old elastic servers (elastic05,06,07) need to be removed from https://github.com/wikimedia/puppet/blob/production/hieradata/cloud...
[09:30:55] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Discovery-Search: Verify (and/or fix) Elasticsearch beta cluster problems - https://phabricator.wikimedia.org/T313521 (10dom_walden) Doing some investigation:  I ran `curl -s localhost:9200/_cluster/health?pretty` on all the elastic servers (deployment-elastic09,10,11) and they...
[09:49:01] <wikibugs>	 (03PS1) 10Giuseppe Lavagetto: Do not set the PHP variable, now unused [tools/scap] - 10https://gerrit.wikimedia.org/r/817212 (https://phabricator.wikimedia.org/T313770)
[09:53:23] <wikibugs>	 (03CR) 10Jaime Nuche: [C: 04-1] "Looks good, but let's also remove the default config value: https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/tools/scap/+/refs/hea" [tools/scap] - 10https://gerrit.wikimedia.org/r/817212 (https://phabricator.wikimedia.org/T313770) (owner: 10Giuseppe Lavagetto)
[09:57:08] <wikibugs>	 10GitLab (Infrastructure), 10Release-Engineering-Team, 10serviceops, 10serviceops-collab, and 2 others: GitLab major release: 15.x - https://phabricator.wikimedia.org/T309062 (10Jelto) I added a patch to bump `gitlab-ce` and `gitlab-runner` packages to `15.0` in https://gerrit.wikimedia.org/r/817211. Accor...
[10:24:58] <wikibugs>	 (03PS2) 10Giuseppe Lavagetto: Stop using utils.sudo_check_call in php restarts [tools/scap] - 10https://gerrit.wikimedia.org/r/817212 (https://phabricator.wikimedia.org/T313770)
[10:28:52] <wikibugs>	 (03CR) 10CI reject: [V: 04-1] Stop using utils.sudo_check_call in php restarts [tools/scap] - 10https://gerrit.wikimedia.org/r/817212 (https://phabricator.wikimedia.org/T313770) (owner: 10Giuseppe Lavagetto)
[10:35:55] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Discovery-Search: Verify (and/or fix) Elasticsearch beta cluster problems - https://phabricator.wikimedia.org/T313521 (10Daimona) Thanks, @dom_walden. I should note that I'm now able to search pages in the event namespace. Maybe the index was rebuilt since the last time I checked?
[10:41:47] <wikibugs>	 (03PS3) 10Giuseppe Lavagetto: Stop using utils.sudo_check_call in php restarts [tools/scap] - 10https://gerrit.wikimedia.org/r/817212 (https://phabricator.wikimedia.org/T313770)
[10:44:16] <wikibugs>	 (03CR) 10Robert Vogel: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/816700 (owner: 10Robert Vogel)
[10:47:39] <wikibugs>	 (03CR) 10CI reject: [V: 04-1] Stop using utils.sudo_check_call in php restarts [tools/scap] - 10https://gerrit.wikimedia.org/r/817212 (https://phabricator.wikimedia.org/T313770) (owner: 10Giuseppe Lavagetto)
[10:51:40] <wikibugs>	 (03PS4) 10Giuseppe Lavagetto: Stop using utils.sudo_check_call in php restarts [tools/scap] - 10https://gerrit.wikimedia.org/r/817212 (https://phabricator.wikimedia.org/T313770)
[10:55:58] <wikibugs>	 (03CR) 10CI reject: [V: 04-1] Stop using utils.sudo_check_call in php restarts [tools/scap] - 10https://gerrit.wikimedia.org/r/817212 (https://phabricator.wikimedia.org/T313770) (owner: 10Giuseppe Lavagetto)
[10:59:21] <wikibugs>	 (03PS5) 10Jaime Nuche: Stop using utils.sudo_check_call in php restarts [tools/scap] - 10https://gerrit.wikimedia.org/r/817212 (https://phabricator.wikimedia.org/T313770) (owner: 10Giuseppe Lavagetto)
[10:59:48] <wikibugs>	 (03PS6) 10Giuseppe Lavagetto: Stop using utils.sudo_check_call in php restarts [tools/scap] - 10https://gerrit.wikimedia.org/r/817212 (https://phabricator.wikimedia.org/T313770)
[11:22:39] <wikibugs>	 (03CR) 10Jaime Nuche: [C: 03+2] Stop using utils.sudo_check_call in php restarts [tools/scap] - 10https://gerrit.wikimedia.org/r/817212 (https://phabricator.wikimedia.org/T313770) (owner: 10Giuseppe Lavagetto)
[11:26:51] <wikibugs>	 (03Merged) 10jenkins-bot: Stop using utils.sudo_check_call in php restarts [tools/scap] - 10https://gerrit.wikimedia.org/r/817212 (https://phabricator.wikimedia.org/T313770) (owner: 10Giuseppe Lavagetto)
[11:44:51] <wikibugs>	 (03PS1) 10Giuseppe Lavagetto: Correctly split command in subprocess call [tools/scap] - 10https://gerrit.wikimedia.org/r/817249
[11:50:26] <wikibugs>	 (03CR) 10Jaime Nuche: [C: 03+2] Correctly split command in subprocess call [tools/scap] - 10https://gerrit.wikimedia.org/r/817249 (owner: 10Giuseppe Lavagetto)
[11:55:02] <wikibugs>	 (03Merged) 10jenkins-bot: Correctly split command in subprocess call [tools/scap] - 10https://gerrit.wikimedia.org/r/817249 (owner: 10Giuseppe Lavagetto)
[12:04:30] <wikibugs>	 (03CR) 10MarkAHershberger: Make composer-php80 run on gate-and-submit for MW core (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/816062 (https://phabricator.wikimedia.org/T300463) (owner: 10Brian Wolff)
[12:08:04] <wikibugs>	 (03PS1) 10Jaime Nuche: Release 4.11.4-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/817253
[12:16:49] <wikibugs>	 (03CR) 10Jaime Nuche: [C: 03+2] Release 4.11.4-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/817253 (owner: 10Jaime Nuche)
[12:21:19] <wikibugs>	 (03Merged) 10jenkins-bot: Release 4.11.4-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/817253 (owner: 10Jaime Nuche)
[12:33:28] <wikibugs>	 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments: 1.39.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T308075 (10Joe)
[12:33:39] <wikibugs>	 10Release-Engineering-Team, 10Scap, 10Patch-For-Review, 10User-brennen: scap no longer restarts php-fpm on canary servers - https://phabricator.wikimedia.org/T313770 (10Joe) 05In progress→03Resolved a:03Joe What happended is a combination of factors: * because of T224857#5467370, `scap pull` was not...
[12:39:07] <wikibugs>	 10Project-Admins: Create User-eakinloose project - https://phabricator.wikimedia.org/T313800 (10EAkinloose)
[14:25:48] <wikibugs>	 (03Abandoned) 10Giuseppe Lavagetto: Restart the canaries and testservers as well [tools/scap] - 10https://gerrit.wikimedia.org/r/817200 (https://phabricator.wikimedia.org/T313770) (owner: 10Giuseppe Lavagetto)
[14:40:57] <wikibugs>	 (03PS1) 10Lucas Werkmeister (WMDE): parameter_functions.py: Make WikibaseLexeme depend on WikibaseLexemeCirrusSearch [integration/config] - 10https://gerrit.wikimedia.org/r/817281
[14:42:03] <wikibugs>	 (03CR) 10Lucas Werkmeister (WMDE): "Related to future patch sets of I1bc87f7f5a, I7b3c60098b and upcoming changes." [integration/config] - 10https://gerrit.wikimedia.org/r/817281 (owner: 10Lucas Werkmeister (WMDE))
[14:43:26] <wikibugs>	 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for mfossati - https://phabricator.wikimedia.org/T313812 (10mfossati)
[14:43:46] <wikibugs>	 10GitLab (Infrastructure), 10Release-Engineering-Team, 10serviceops, 10serviceops-collab, and 2 others: GitLab major release: 15.x - https://phabricator.wikimedia.org/T309062 (10brennen) > 14.10.5 (current version) -> 15.0.4 -> 15.2.0 (or most recent version). >  > Do you think this is reasonable?  Yeah, f...
[14:44:14] <wikibugs>	 (03CR) 10Michael Große: [C: 03+1] parameter_functions.py: Make WikibaseLexeme depend on WikibaseLexemeCirrusSearch [integration/config] - 10https://gerrit.wikimedia.org/r/817281 (owner: 10Lucas Werkmeister (WMDE))
[14:50:29] <wikibugs>	 10GitLab (Infrastructure), 10Release-Engineering-Team, 10serviceops, 10serviceops-collab, and 2 others: GitLab major release: 15.x - https://phabricator.wikimedia.org/T309062 (10Jelto) a:03Jelto Ok I'll prepare the `15.0` packages tomorrow and start with the test instance and test runners.  We may have t...
[15:51:03] <wikibugs>	 10Release-Engineering-Team (The Decommission Mission 💀), 10SRE, 10SRE-Access-Requests, 10Patch-For-Review: Add dancy to phabricator-roots - https://phabricator.wikimedia.org/T313551 (10thcipriani)
[15:51:53] <wikibugs>	 10Release-Engineering-Team (The Decommission Mission 💀), 10SRE, 10SRE-Access-Requests, 10Patch-For-Review: Add dancy to phabricator-roots - https://phabricator.wikimedia.org/T313551 (10thcipriani) >>! In T313551#8095985, @dancy wrote: > Noting that my manager @thcipriani is on vacation right now.    Approv...
[16:17:33] <wikibugs>	 10Release-Engineering-Team (The Decommission Mission 💀), 10SRE, 10SRE-Access-Requests, 10Patch-For-Review: Add dancy to phabricator-roots - https://phabricator.wikimedia.org/T313551 (10Volans) 05Open→03Resolved a:03Volans The patch has been merged, it will be reflected in the fleet within the last 30...
[17:18:25] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (The Decommission Mission 💀), 10User-brennen: Deploy Phabricator with scap - https://phabricator.wikimedia.org/T313259 (10brennen)
[17:58:37] <icinga-wm>	 PROBLEM - Check systemd state on doc1002 is CRITICAL: CRITICAL - degraded: The following units failed: rsync-doc-doc2001.codfw.wmnet.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
[18:03:05] <mutante>	 ^ meh.. this is more like an annoyance
[18:03:21] <mutante>	 sometimes it's just that rsync is syncing and some files change on the source during sync
[18:04:25] <mutante>	 [doc1002:~] $ sudo systemctl start rsync-doc-doc2001.codfw.wmnet.service
[18:05:55] <icinga-wm>	 RECOVERY - Check systemd state on doc1002 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
[18:18:51] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (The Decommission Mission 💀), 10serviceops, 10serviceops-collab: Setup rsync for phab data on disk - https://phabricator.wikimedia.org/T313360 (10Dzahn)
[18:24:56] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (The Decommission Mission 💀), 10serviceops, 10serviceops-collab: Setup rsync for phab data on disk - https://phabricator.wikimedia.org/T313360 (10Dzahn) So I knew either we already had this code from last time or we had applied it in a special "migration" puppet ro...
[18:49:07] <ejegg>	 hi releng folks - I don't see a phab on a quick search, but I guess you're probably aware that php-security-checker has been giving false-positive warning emails for all the repos for about a week?
[18:49:37] <ejegg>	 Here's the output of one of the daily jobs: https://integration.wikimedia.org/ci/job/php-composer-security-docker/20930/console
[18:50:20] <ejegg>	 seems like it's posting to https://php-security-checker.wmcloud.org/ and getting 502 (which I also get just looking in a browser)
[18:51:20] <ejegg>	 I'm not sure where the emails are wired up, but fr-tech-failmail is getting one daily for each repo we manage saying those jobs are still failing
[18:52:30] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (The Decommission Mission 💀), 10serviceops, 10serviceops-collab: Setup rsync for phab data on disk - https://phabricator.wikimedia.org/T313360 (10Dzahn) checking the firewall rules. an iptables -L on phab1001 shows:   ` ACCEPT     tcp  --  phab1001.eqiad.wmnet  any...
[18:55:11] <mutante>	 ejegg: seems like more than one issue. for starters I can confirm 502 - Bad Gateway at https://php-security-checker.wmcloud.org/  that is for sure _one_ ticket worthy
[18:55:44] <mutante>	 then of course the other part where those emails go and all that
[18:56:21] <mutante>	 ejegg: let's find out who runs that cloud project.. I am looking
[18:56:50] <ejegg>	 thanks mutante !
[18:57:09] <ejegg>	 I see this ticket suggesting it move elsewhere: https://phabricator.wikimedia.org/T296967
[18:57:28] <mutante>	 finds https://phabricator.wikimedia.org/T296967
[18:57:30] <mutante>	 heh
[18:58:38] <mutante>	 I know that people have started testing MW on new PHP versions recently
[18:59:18] <mutante>	 makes me think it's possible that could influence a tool checking PHP security but could also be entirely unrelated issue with toolforge
[18:59:29] <mutante>	 or just a "restart webservice" kind of thing
[18:59:44] <mutante>	 feel like making a new one and just say the "here, look..502" ?
[19:00:10] <mutante>	 all the right tags and stuff..people will do
[19:02:52] <mutante>	 ejegg: so about the email part of this.. I tried putting the string "fr-tech-failmail" into the codesearch tool and I get https://codesearch.wmcloud.org/search/?q=fr-tech-failmail&i=nope&files=&excludeFiles=&repos=
[19:03:19] <mutante>	 so that's in the integration/config repo.. and this is the right channel for that
[19:03:36] <ejegg>	 ah cool
[19:03:37] <mutante>	 while the tool being down is more for -cloud 
[19:03:49] <ejegg>	 I do like getting those alerts when they're legit!
[19:04:00] <ejegg>	 So I'll follow up with -cloud about why it's down
[19:04:04] <ejegg>	 thanks for the tip!
[19:04:05] <mutante>	 I can offer making a patch there to remove that email address if you want that
[19:04:12] <ejegg>	 no, please leave it on
[19:04:16] <mutante>	 ok
[19:04:25] <ejegg>	 it's a good reminder to update composer packages now and then
[19:04:36] <mutante>	 alright,good
[19:04:43] <ejegg>	 :)
[19:10:11] <ejegg>	 Hi Krinkle ! I was asking about the php-security-checker over in -cloud and got pointed to https://openstack-browser.toolforge.org/project/packagist-mirror for the admins list, and found you on it.
[19:10:49] <ejegg>	 I guess it had been running on a stretch box and was deactivated last week
[19:11:09] <Krinkle>	 ejegg: define "php-security-checker", do you mean https://www.mediawiki.org/wiki/Continuous_integration/Phan/Phan-taint-check-plugin ?
[19:11:25] <Krinkle>	 aka phan-SecurityCheckPlugin
[19:11:38] <ejegg>	 Krinkle: I think it's something different - it just checks composer.json for package versions with vulnerabilities
[19:11:52] <ejegg>	 It was hosted at https://php-security-checker.wmcloud.org/
[19:12:23] <ejegg>	 and there are daily jenkins jobs that used that webservice to check for vulns in each repo
[19:12:46] <ejegg>	 and emailed the repo owners when vulns were found
[19:13:19] <ejegg>	 only now with that webservice down it is email repo managers every day saying their is a security check failure
[19:13:28] <Krinkle>	 packagist-mirror is an incomplete experiment from a few years ago, afaik it is not in active use nor is there a plan to do so. https://phabricator.wikimedia.org/T203529 It might e.g. speed up CI a bit or offer some limited downtime protection if upstream has another CDN outage.
[19:13:31] <ejegg>	 so I see this one ticket suggesting the service move to toolforge 
[19:13:31] <ejegg>	 https://phabricator.wikimedia.org/T296967
[19:13:55] <Krinkle>	 I'm not aware with what php-security-checker is, or why we need it, or why its VM was in that WMCS project.
[19:14:06] <ejegg>	 Krinkle: interesting....
[19:14:22] <Krinkle>	 I guess it's just what lego had handy
[19:14:34] <ejegg>	 Looks like it was written by legoktm, but he is probably busy with other fun projects these days
[19:14:37] <Krinkle>	 to avoid a temporary one-off WMCS project since we can't seem to delete WMCS projects
[19:14:43] <ejegg>	 oh fun
[19:14:54] <ejegg>	 so perhaps now orphaned
[19:14:57] <Krinkle>	 what do you need right now that would involve this service?
[19:15:14] <Krinkle>	 note that we have lib-up which sounds a lot like what you describe this did/would do.
[19:15:28] <Krinkle>	 https://www.mediawiki.org/wiki/Libraryupgrader
[19:15:59] <ejegg>	 Krinkle: OK, I guess someone should delete the jenkins jobs that depend on the webservice if it's not going to be fixed
[19:16:00] <mutante>	 "url is served by the security-checker1.packagist-mirror.eqiad1.wikimedia.cloud instance, which was shut down last week due to stretch" :p
[19:16:15] <ejegg>	 so there's this one: https://integration.wikimedia.org/ci/job/php-composer-security-docker/
[19:16:30] <ejegg>	 and about 5 parent projects
[19:16:47] <mutante>	 the reason is down now is that the distribution it ran on was so out of date that
[19:17:00] <mutante>	 it was deleted
[19:17:38] <mutante>	 did not expect that but it explains the 502. could argue it would be nice to service redirects to some "this tool is gone" page thugh
[19:18:15] <Krinkle>	 https://phabricator.wikimedia.org/T309655
[19:18:22] <mutante>	 it needs some kind of patch here https://codesearch.wmcloud.org/search/?q=fr-tech-failmail&i=nope&files=&excludeFiles=&repos=
[19:18:26] <mutante>	 to stop the fail mails
[19:18:34] <mutante>	 integration/config
[19:19:21] <Krinkle>	 https://phabricator.wikimedia.org/T309655
[19:19:26] <Krinkle>	 T309655
[19:19:26] <stashbot>	 T309655: Disable php-composer-security-docker emails to security-admin-feed@lists - https://phabricator.wikimedia.org/T309655
[19:19:27] <mutante>	 so maybe that can fix it for fr-tech and also T309655 at the same time
[19:19:39] <ejegg>	 mutante: yeah, or maybe just disable the scheduled job till that tool is fixed?
[19:19:54] <ejegg>	 Seems silly to just run a sure-to-fail thing in CI in the first place
[19:19:57] <Krinkle>	 afaik it hasn't been in use, and nobody is asking for it or relying on it, Ive also never seen it before today. afaik we can just remove it and not worry about it further.
[19:20:09] <Krinkle>	 It's not part of any CI job in my deifnition of a CI job.
[19:20:16] <Krinkle>	 it's a detached cron that nobody sees or knows about
[19:20:28] <Krinkle>	 but apparently emails someone when it fails.
[19:20:42] <Krinkle>	 if it's not part of dev workflow in Gerrit, it's not really a CI job :shrug:
[19:21:07] <mutante>	 who is the sender of those emails?
[19:21:16] <Krinkle>	 Jenkins job
[19:21:29] <mutante>	 so it should be a timer on contint1001?
[19:21:35] <mutante>	 or on the slaves
[19:21:36] <mutante>	 in cloud
[19:22:07] <Krinkle>	 what do you mean by "should"? I'm proposing to delete the jenkins job that currently runs on a 24h timer within Jenkins.
[19:22:24] <mutante>	 https://gerrit.wikimedia.org/r/c/integration/config/+/801808/
[19:22:28] <Krinkle>	 indeed
[19:22:53] <mutante>	 timer within Jenkins. ok, got you
[19:23:38] <ejegg>	 heh, fundraising tech used to use Jenkins as a glorified cron to run all sorts of critical donation processing tasks
[19:23:44] <ejegg>	 that was a bad idea
[19:25:24] <Krinkle>	 the subset of passive information that was useful has mostly been migrated to either an active signal on-commit in Gerrit/CI, or a production alert via Icinga/AlertManager for things that were essentially alerts already, or moved to a separate service that presents itself as a Phab task or Gerrit patch proposal (e.g. as LibUp does). The problem with cron Jenkins is mostly that it is outside the regular workflow so just tends to rot and 
[19:25:24] <Krinkle>	 be hard to assess seevrity/priority on.
[19:27:26] <Krinkle>	 ejegg: on behalf of fr-tech, I can at least remove the one that's spamming in your team's name, and leave the rest for sec team to decide on. apparently the task is undecided, though I"m not sure where the uncertainty is coming from. The reference to the appsec pipeline sounds nice. If it isn't vendor-locked in GitLab, perhaps sec team could run that as a Jenkins job as well to offer value to teams and services.
[19:57:11] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (The Decommission Mission 💀), 10serviceops, 10serviceops-collab: Setup rsync for phab data on disk - https://phabricator.wikimedia.org/T313360 (10Dzahn)
[20:01:41] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (The Decommission Mission 💀), 10serviceops, 10serviceops-collab: Setup rsync for phab data on disk - https://phabricator.wikimedia.org/T313360 (10Dzahn) regarding the UIDs and possible privilege issues after rsync:  - the owner of directories and files under /srv/r...
[20:33:58] <legoktm>	 ejegg: oh shit, I didn't realize it got shut down 
[20:34:35] <legoktm>	 ejegg: I meant to move it to Toolforge so I didn't have to deal with this, I will do that today or tomorrow
[21:00:35] <Krinkle>	 brennen: do you know what triggers scap doc updates? It https://doc.wikimedia.org/mw-tools-scap/index.html is a bit out of date with the repo, e.g. "Running tests" mentions `envlist` still which the repo hasn't since https://gerrit.wikimedia.org/r/c/mediawiki/tools/scap/+/665316 last year February.
[21:02:29] <wikibugs>	 (03PS1) 10Krinkle: docs: Fix grammar in "Background" section [tools/scap] - 10https://gerrit.wikimedia.org/r/817383
[21:05:27] <brennen>	 Krinkle: offhand i don't
[22:18:28] <ejegg>	 oops, was in a meeting so I didn't see the pings. Thanks legoktm !
[22:18:56] <ejegg>	 Krinkle: I guess we can leave that email address in there if it ill start working again soon
[22:19:18] <ejegg>	 it's nice to have those daily checks
[22:27:53] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (The Decommission Mission 💀), 10serviceops, 10serviceops-collab: Email tool maintainers about git-ssh deprecation on phabricator - https://phabricator.wikimedia.org/T313359 (10Aklapper)
[22:50:01] <icinga-wm>	 PROBLEM - Work requests waiting in Zuul Gearman server on contint2001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [400.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10
[23:06:41] <jinxer-wm>	 (Queue (Jenkins jobs + Zuul functions) alert) firing: Queue (Jenkins jobs + Zuul functions) alert   - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert
[23:11:26] <wikibugs>	 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T308075 (10brennen) End-of-workday status notes: Stable and quiet at group0; will plan to backport the fix for T313836 before group1.
[23:14:50] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (The Decommission Mission 💀), 10User-brennen: Deploy Phabricator with scap - https://phabricator.wikimedia.org/T313259 (10brennen) Paired with @jeena today experimenting with [[https://doc.wikimedia.org/mw-tools-scap/scap3/quickstart/setup.html#command-checks|scap3 c...