[00:02:43] (Queue (Jenkins jobs + Zuul functions) alert) firing: (2) Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [00:04:31] TheresNoTime: Indeed, NYC is a love place, much like London, not a boring quiet backwater like SF or Manchester. ;-) [00:09:41] 10Continuous-Integration-Config: Upgrade primary branch of all Wikimedia-deployed repos to a version of mediawiki-tools-phan including T270553 - https://phabricator.wikimedia.org/T295285 (10Jdforrester-WMF) Status as of 2022-08-01: * 284 repos using 0.11.1 * 74 repos still using 0.11.0 and LibUp can't upgrade t... [00:22:43] (Queue (Jenkins jobs + Zuul functions) alert) firing: (2) Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [00:58:48] 10Continuous-Integration-Config, 10Patch-For-Review: Upgrade primary branch of all Wikimedia-deployed repos to a version of mediawiki-tools-phan including T270553 - https://phabricator.wikimedia.org/T295285 (10Reedy) >>! In T295285#8122108, @Jdforrester-WMF wrote: > Status as of 2022-08-01: > > * 284 repos us... [01:17:43] (Queue (Jenkins jobs + Zuul functions) alert) resolved: Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [01:51:35] RECOVERY - Work requests waiting in Zuul Gearman server on contint2001 is OK: OK: Less than 100.00% above the threshold [200.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [02:07:45] legoktm: what's left regarding T291014, T290759 and friends? Looks like all patches landed [02:07:45] T291014: Terminate all implicit use of VipsScaler code from Wikimedia production so we can remove it without breaking things this time - https://phabricator.wikimedia.org/T291014 [02:07:46] T290759: Undeploy VipsScaler from Wikimedia wikis - https://phabricator.wikimedia.org/T290759 [02:07:56] James_F: ^ [02:08:40] Krinkle: Last time we tried it broke stuff. I think the uses are now gone, but… [02:08:56] Should probably be OK but I’m on a phone in an airport. [02:09:13] ack [02:12:05] Krinkle: something in PageImages broke last time: https://phabricator.wikimedia.org/T291014#7578533 [02:13:05] legoktm: thx, I missed the `Revert "Undeploy VipsScaler: I, II & III"` patch in the gerrit summary [02:13:25] so all merged != all done, because all merged is `all done + revert 1/3 patches` [02:13:31] which is too much done and so not done [02:16:32] yeah... [02:17:20] maybe someone who understands the file/thumb hierarchy will have an easier time on it [02:17:27] I spent a while and got nowhere besides my isError() patch [02:18:42] Krinkle: or disable it on a few (test) wikis first and see how it goes [02:32:03] legoktm: yeah.. or drive us crazy with Enabled = mt_rand() > 0.1; [02:32:28] not sure we'd notice the full range of issues otherwise if it's something commons specific [02:32:45] or just give it another go. [02:40:01] Krinkle: I think it should be reachable on client wikis since it was page images. Maybe even just disable with mwdebug and hit the URL on https://phabricator.wikimedia.org/T290973 again to get a stacktrace? [02:41:10] legoktm: oh you mean that exact issue, yes, that's a better way for that. [02:41:24] I thought maybe we were worried about additional uncovered Vips issues [02:41:50] nah I don't think so [02:42:12] I think it was undeployed for long enough that we would've caught anything important [02:42:34] (famous last words I guess) [02:48:15] ack [02:54:41] PROBLEM - Work requests waiting in Zuul Gearman server on contint2001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [400.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [03:02:40] (Queue (Jenkins jobs + Zuul functions) alert) firing: Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [03:13:48] _joe_: dancy: So... restart saga continues. The last changes to scap-pull mean that scap-pull now doesn't work on mwmaint1002/mwmaint2002 as it will end up prompting me for a sudo password for `mwdeploy` after `03:12:43 Checking if php-fpm restart needed` [03:15:50] Krinkle: I'll work on a fix first thing tomorrow [03:16:11] temporary workaround: `scap pull --no-php-restart` [03:16:14] thx! [03:17:40] (Queue (Jenkins jobs + Zuul functions) alert) firing: (2) Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [03:37:40] (Queue (Jenkins jobs + Zuul functions) alert) resolved: Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [03:54:10] RECOVERY - Work requests waiting in Zuul Gearman server on contint2001 is OK: OK: Less than 100.00% above the threshold [200.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [04:41:57] <_joe_> Krinkle: it's just a configuration problem, I'll fix it [04:42:13] <_joe_> I was even conscious of it, but I was in a hurry to fix the bigger problem [04:42:54] ack [07:55:36] !log cleared stuck beta deployment jobs T72597 [07:55:38] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [07:55:38] T72597: Jenkins Gearman plugin has deadlock on executor threads (was: Beta Cluster stopped receiving code updates (beta-update-databases-eqiad hung) - https://phabricator.wikimedia.org/T72597 [08:18:58] (03PS3) 10Jaime Nuche: beta: use scap-o-scap to install/update Scap [tools/scap] - 10https://gerrit.wikimedia.org/r/819002 [08:19:00] (03PS4) 10Jaime Nuche: debian package: remove from codebase [tools/scap] - 10https://gerrit.wikimedia.org/r/819039 [08:19:34] (03CR) 10Jaime Nuche: beta: use scap-o-scap to install/update Scap (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/819002 (owner: 10Jaime Nuche) [08:54:12] (03PS5) 10Jaime Nuche: debian package: remove from codebase [tools/scap] - 10https://gerrit.wikimedia.org/r/819039 [08:56:19] (03CR) 10Jaime Nuche: "Please note this section was also touched here: https://gerrit.wikimedia.org/r/c/mediawiki/tools/scap/+/819039" [tools/scap] - 10https://gerrit.wikimedia.org/r/808320 (owner: 10Hashar) [08:57:57] (03CR) 10Jaime Nuche: debian package: remove from codebase (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/819039 (owner: 10Jaime Nuche) [09:23:02] Project beta-code-update-eqiad build #402707: 04FAILURE in 1.5 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/402707/ [09:35:02] Yippee, build fixed! [09:35:03] Project beta-code-update-eqiad build #402708: 09FIXED in 2 min 1 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/402708/ [09:35:13] 10Deployments, 10Release-Engineering-Team (Doing), 10SRE, 10bacula, 10Parsoid (Tracking): Accidental removal of some files under /srv/deployment on deploy1002 - https://phabricator.wikimedia.org/T307349 (10elukey) 05Open→03Resolved We can close this task and see if any clean up is needed in the follo... [09:43:11] Project beta-scap-sync-world build #62307: 04FAILURE in 8 min 7 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/62307/ [09:52:57] anyone touching scap on beta? [09:54:11] `/usr/bin/scap: command not found` on `deployment-mediawiki12`, `deployment-snapshot03` and `deployment-mwmaint02` [09:54:58] when I ssh into mediawiki12 it seems to have a /usr/bin/scap [09:55:04] (during ^ run, `scap-cdb-rebuild` step, its running again now so we'll see..) [09:55:28] Yippee, build fixed! [09:55:28] Project beta-scap-sync-world build #62308: 09FIXED in 8 min 8 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/62308/ [09:55:28] I think I saw some “use scap to deploy scap” tickets flying around, possibly related to those? [09:55:32] yay, nevermind [09:56:02] Lucas_WMDE: yeaah I saw them too 🤷 may have just been transient ^^ [09:56:31] (transient, the nicer way of saying "who knows why that broke but oh well" /s) [09:56:36] ^^ [09:57:25] !bash transient, the nicer way of saying "who knows why that broke but oh well" /s [09:57:25] Lucas_WMDE: Stored quip at https://bash.toolforge.org/quip/CGD9XYIBa_6PSCT9HbBu [09:57:47] :p [09:59:10] https://gerrit.wikimedia.org/r/c/operations/puppet/+/817762 ? [10:00:02] maybe, or maybe "transient" :D [10:02:26] yeah, why not [10:17:00] Hey RelEng! Is there a procedure to archive a gerrit repo? Specifically https://gerrit.wikimedia.org/r/admin/repos/wikidata/query/flink-swift-plugin [10:21:18] 10Continuous-Integration-Config, 10MW-1.39-notes (1.39.0-wmf.25; 2022-08-15), 10Patch-For-Review: Upgrade primary branch of all Wikimedia-deployed repos to a version of mediawiki-tools-phan including T270553 - https://phabricator.wikimedia.org/T295285 (10Reedy) [11:25:11] TheresNoTime: sry about late response, I just saw your messages [11:25:27] yeah, the beta Puppet changes probably caused that temporary breakage [11:25:40] no worries! ^^ [11:25:40] sorry about the confusion it caused [11:32:47] * TheresNoTime lives in a state of confusion so it's all good :D [11:39:35] lol :) [11:40:54] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team (Radar): Stop triggering `beta-scap-sync-world` on `beta-mediawiki-config-update-eqiad` completion - https://phabricator.wikimedia.org/T314378 (10TheresNoTime) [11:42:23] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team (Radar): Stop triggering `beta-scap-sync-world` on `beta-mediawiki-config-update-eqiad` completion - https://phabricator.wikimedia.org/T314378 (10TheresNoTime) [13:58:30] 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments: 1.39.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T308076 (10matmarex) ##### Risky Patch! 🚂🔥 * **Change**: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/DiscussionTools/+/785171 Make reply links in... [14:25:51] 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments: Parsoid rt-testing is still broken, parsoid needs a revert - https://phabricator.wikimedia.org/T314395 (10cscott) p:05Triage→03Medium [14:27:13] 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments: Parsoid rt-testing is still broken, parsoid needs a revert - https://phabricator.wikimedia.org/T314395 (10RhinosF1) [14:36:56] 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments: Parsoid rt-testing is still broken, parsoid needs a revert - https://phabricator.wikimedia.org/T314395 (10brennen) Discussed with @ssastry on IRC. Cherry-pick and merge to `wmf/1.39.0-wmf.23` should be sufficient, as wmf.23 isn... [14:51:39] (03CR) 10Ahmon Dancy: [C: 03+2] beta: use scap-o-scap to install/update Scap [tools/scap] - 10https://gerrit.wikimedia.org/r/819002 (owner: 10Jaime Nuche) [14:58:02] (03Merged) 10jenkins-bot: beta: use scap-o-scap to install/update Scap [tools/scap] - 10https://gerrit.wikimedia.org/r/819002 (owner: 10Jaime Nuche) [15:08:59] (03CR) 10Ahmon Dancy: debian package: remove from codebase (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/819039 (owner: 10Jaime Nuche) [15:17:22] (03CR) 10Ahmon Dancy: [C: 03+2] scap: remove deb package jobs [integration/config] - 10https://gerrit.wikimedia.org/r/819028 (owner: 10Jaime Nuche) [15:19:14] (03Merged) 10jenkins-bot: scap: remove deb package jobs [integration/config] - 10https://gerrit.wikimedia.org/r/819028 (owner: 10Jaime Nuche) [15:22:11] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/819028 [15:22:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:23:33] !log Deleted beta-build-scap-deb and beta-publish-deb Jenkins jobs. (https://gerrit.wikimedia.org/r/c/integration/config/+/819028) [15:23:34] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:00:41] 10Release-Engineering-Team (Deployment Autopilot 🛩️), 10Scap, 10Patch-For-Review: Automated Tuesday Train via a timer - https://phabricator.wikimedia.org/T310395 (10dancy) [16:06:36] 10Project-Admins: Create project tag for - https://phabricator.wikimedia.org/T314406 (10JArguello-WMF) [16:17:19] 10Release-Engineering-Team (Deployment Autopilot 🛩️), 10Scap, 10Patch-For-Review: Automated Tuesday Train via a timer - https://phabricator.wikimedia.org/T310395 (10dancy) [16:37:57] 10Continuous-Integration-Config, 10Codex, 10Design-Systems-Team, 10Test-Coverage, 10User-DannyS712: Public codex code coverage report - https://phabricator.wikimedia.org/T303899 (10Catrope) >>! In T303899#7816452, @DannyS712 wrote: > Hmm, doesn't appear to be visible at https://doc.wikimedia.org/cover/ y... [17:01:12] 10Release-Engineering-Team, 10Platform Engineering, 10Similar Editors, 10Anti-Harassment (AHaT Sprint 13: The Magic Carp Hat): Configure SimilarEditors in production with Similarusers credentials - https://phabricator.wikimedia.org/T308670 (10ARamirez_WMF) [17:22:36] 10Release-Engineering-Team, 10Anti-Harassment, 10Platform Engineering, 10Similar Editors: Configure SimilarEditors in production with Similarusers credentials - https://phabricator.wikimedia.org/T308670 (10ARamirez_WMF) [17:35:31] 10Beta-Cluster-Infrastructure, 10service-runner: Service cannot make HTTPS requests due to missing ca-certificates in Docker image - https://phabricator.wikimedia.org/T309261 (10ori) 05Open→03Resolved a:03ori [17:40:12] 10Beta-Cluster-Infrastructure, 10service-runner: Service cannot make HTTPS requests due to missing ca-certificates in Docker image - https://phabricator.wikimedia.org/T309261 (10ori) a:05ori→03Jdforrester-WMF [17:49:28] * Krinkle sings a catchy Everly Brothers song like Let It Be Me or Be Bop A Lula with the alternate lyrics of "Scap oh Scap I miss you so...". [17:52:06] hehe [18:44:37] PROBLEM - Check systemd state on gerrit2002 is CRITICAL: CRITICAL - degraded: The following units failed: apache2.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [18:49:07] RECOVERY - Check systemd state on gerrit2002 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [18:59:36] PROBLEM - gerrit process on gerrit2002 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/lib/jvm/java-11-openjdk-amd64/bin/java .*-jar /var/lib/gerrit2/review_site/bin/gerrit.war daemon -d /var/lib/gerrit2/review_site https://wikitech.wikimedia.org/wiki/Gerrit [19:06:54] ACKNOWLEDGEMENT - gerrit process on gerrit2002 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/lib/jvm/java-11-openjdk-amd64/bin/java .*-jar /var/lib/gerrit2/review_site/bin/gerrit.war daemon -d /var/lib/gerrit2/review_site daniel_zahn setting up new host https://wikitech.wikimedia.org/wiki/Gerrit [19:08:19] gerrit2002 java[568306]: Error: Invalid or corrupt jarfile /var/lib/gerrit2/review_site/bin/gerrit.war :o [19:08:40] gerrit.war -> /srv/deployment/gerrit/gerrit/gerrit.war [19:08:49] ok..so we need to deploy gerrit to gerrit2002 next [19:09:00] before the service can start [19:23:09] (03PS1) 10Jforrester: Zuul: [design/codex] Switch coverage job back to -direct [integration/config] - 10https://gerrit.wikimedia.org/r/819698 [19:24:07] (03CR) 10Jforrester: [C: 03+2] Zuul: [design/codex] Switch coverage job back to -direct [integration/config] - 10https://gerrit.wikimedia.org/r/819698 (owner: 10Jforrester) [19:26:28] (03Merged) 10jenkins-bot: Zuul: [design/codex] Switch coverage job back to -direct [integration/config] - 10https://gerrit.wikimedia.org/r/819698 (owner: 10Jforrester) [19:26:38] (03PS1) 10Jforrester: Zuul: Run publish jobs on branches called 'main' too [integration/config] - 10https://gerrit.wikimedia.org/r/819702 [19:26:57] !log Zuul: [design/codex] Switch coverage job back to -direct [19:26:58] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:30:33] (03PS1) 10Jforrester: Zuul: [integration/config] Switch to test-prio [integration/config] - 10https://gerrit.wikimedia.org/r/819705 [19:31:16] (03PS2) 10Catrope: Zuul: Run publish jobs on branches called 'main' too [integration/config] - 10https://gerrit.wikimedia.org/r/819702 (https://phabricator.wikimedia.org/T303899) (owner: 10Jforrester) [19:36:02] (03CR) 10Catrope: Zuul: Run publish jobs on branches called 'main' too (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/819702 (https://phabricator.wikimedia.org/T303899) (owner: 10Jforrester) [19:36:47] 10Phabricator, 10Triagers: add MPhamWMF to Triagers group - https://phabricator.wikimedia.org/T314146 (10Aklapper) @MPhamWMF: For some more context: As our code is public it can be changed and improved by anyone outside of some team in some organization. Unless stewards/maintainers of a codebase are not //agai... [19:37:12] (03PS3) 10Jforrester: Zuul: Run publish jobs on branches called 'main' too [integration/config] - 10https://gerrit.wikimedia.org/r/819702 [19:38:36] 10Phabricator, 10Discovery-Search: Evaluate Discovery-Search Herald filter rule H143 - https://phabricator.wikimedia.org/T314429 (10Aklapper) [19:46:22] 10Project-Admins, 10Discovery, 10Discovery-Search: Clarify / clean up Discovery/Search related team project tags in Phabricator - https://phabricator.wikimedia.org/T314431 (10Aklapper) [19:46:58] 10Project-Admins, 10Discovery, 10Discovery-Search, 10User-AKlapper: Clarify / clean up Discovery/Search related team project tags in Phabricator - https://phabricator.wikimedia.org/T314431 (10Aklapper) [19:47:11] (03PS1) 10Dduvall: deploy-local: Render relative config files directly to their final paths [tools/scap] - 10https://gerrit.wikimedia.org/r/819720 (https://phabricator.wikimedia.org/T313950) [19:55:05] 10Project-Admins, 10Discovery, 10Discovery-Search, 10User-AKlapper: Clarify / clean up Discovery/Search related team project tags in Phabricator - https://phabricator.wikimedia.org/T314431 (10MPhamWMF) To my understanding, only #discovery-search is currently active. All other Discovery tags are historical... [20:11:05] 10Project-Admins: Create project tag for - https://phabricator.wikimedia.org/T314406 (10Aklapper) 05Open→03Stalled Hi, per https://wikitech.wikimedia.org/wiki/Event_Platform/EventStreams a Phab project tag already exists: https://phabricator.wikimedia.org/tag/event-platform/ If this new tag i... [20:15:03] (03CR) 10Jeena Huneidi: [C: 03+1] deploy-local: Render relative config files directly to their final paths [tools/scap] - 10https://gerrit.wikimedia.org/r/819720 (https://phabricator.wikimedia.org/T313950) (owner: 10Dduvall) [20:28:38] 10Project-Admins, 10Discovery-Search, 10User-AKlapper: Clarify / clean up Discovery/Search related team project tags in Phabricator - https://phabricator.wikimedia.org/T314431 (10Aklapper) Thanks for the quick answer and clarification! * Added Wikimedia-Portals to https://phabricator.wikimedia.org/project/p... [20:30:35] (03CR) 10Ahmon Dancy: [C: 03+2] deploy-local: Render relative config files directly to their final paths [tools/scap] - 10https://gerrit.wikimedia.org/r/819720 (https://phabricator.wikimedia.org/T313950) (owner: 10Dduvall) [20:32:49] 10Project-Admins: Create project tag for - https://phabricator.wikimedia.org/T314406 (10Ottomata) Yes, I believe #event-platform is the tag @JArguello-WMF wants and needs. [20:36:36] (03Merged) 10jenkins-bot: deploy-local: Render relative config files directly to their final paths [tools/scap] - 10https://gerrit.wikimedia.org/r/819720 (https://phabricator.wikimedia.org/T313950) (owner: 10Dduvall) [20:37:01] 10Gerrit, 10Release-Engineering-Team (The Decommission Mission 💀), 10SRE, 10serviceops, and 2 others: replacement for gerrit2001 - https://phabricator.wikimedia.org/T243027 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by dzahn@cumin2002 for host gerrit2002.wikimedia.org with OS b... [20:44:18] thcipriani: gerrit2002 was on bullseye. gerrit puppet role requires git-fat. git-fat has not been ported to python 3.. therefore.. we can't have gerrit on bullseye and reimaging it with buster ..sigh T279509 [20:44:19] T279509: git-fat needs to be ported to Python 3 - https://phabricator.wikimedia.org/T279509 [20:44:57] but also we didn't say we wanted to upgrade distro.. or maybe we did but not in this sprint [20:49:49] git-fat looks like dead tech -- https://github.com/jedbrown/git-fat/issues/92 [20:50:27] that's needed for ORES? Or something else? [20:58:34] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T308076 (10dancy) [21:01:03] (03CR) 10Catrope: Zuul: Run publish jobs on branches called 'main' too (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/819702 (owner: 10Jforrester) [21:11:09] RECOVERY - gerrit process on gerrit2002 is OK: PROCS OK: 1 process with regex args ^/usr/lib/jvm/java-11-openjdk-amd64/bin/java .*-jar /var/lib/gerrit2/review_site/bin/gerrit.war daemon -d /var/lib/gerrit2/review_site https://wikitech.wikimedia.org/wiki/Gerrit [21:11:35] 10Gerrit, 10Release-Engineering-Team (The Decommission Mission 💀), 10SRE, 10serviceops, and 2 others: replacement for gerrit2001 - https://phabricator.wikimedia.org/T243027 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by dzahn@cumin2002 for host gerrit2002.wikimedia.org with OS buste... [21:12:38] 10Project-Admins, 10Discovery-Search, 10User-AKlapper: Clarify / clean up Discovery/Search related team project tags in Phabricator - https://phabricator.wikimedia.org/T314431 (10Aklapper) @MPhamWMF Now that #Discovery is archived, it would be good if the open tasks at https://phabricator.wikimedia.org/manip... [21:24:22] ugh. Git-fat. ORES + everything java depends on it, currently. Scap technically has support for git-lfs, but it's mostly untested + the workflow with java would need to be reinvented a touch. [21:30:55] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T308076 (10Ladsgroup) [21:52:45] 10Project-Admins, 10Discovery-Search, 10User-AKlapper: Clarify / clean up Discovery/Search related team project tags in Phabricator - https://phabricator.wikimedia.org/T314431 (10MPhamWMF) @Aklapper, I am trying to prune my team's backlog to what we expect to do (in some reasonable timeframe). Based on the p... [21:57:49] (03PS1) 10Ahmon Dancy: Release 4.12.0-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/819757 [21:57:51] (03CR) 10Ahmon Dancy: [C: 03+2] Release 4.12.0-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/819757 (owner: 10Ahmon Dancy) [22:02:01] (03Merged) 10jenkins-bot: Release 4.12.0-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/819757 (owner: 10Ahmon Dancy) [22:03:17] 10Project-Admins, 10Discovery-Search, 10User-AKlapper: Clarify / clean up Discovery/Search related team project tags in Phabricator - https://phabricator.wikimedia.org/T314431 (10Reedy) >>! In T314431#8126270, @MPhamWMF wrote: > In other words, the tickets we want to address are a subset of the valid tickets... [22:03:27] 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments: Parsoid rt-testing is still broken, parsoid needs a revert - https://phabricator.wikimedia.org/T314395 (10ssastry) 05Open→03Resolved [22:03:30] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T308076 (10ssastry) [22:14:25] (03PS1) 10Ahmon Dancy: Add gerrit2002.wikimedia.org to scap targets list [software/gerrit] (deploy/wmf/stable-3.4) - 10https://gerrit.wikimedia.org/r/819760 (https://phabricator.wikimedia.org/T243027) [22:16:50] 10Gerrit, 10Release-Engineering-Team (The Decommission Mission 💀), 10Patch-For-Review: Bring up gerrit2002 - https://phabricator.wikimedia.org/T313250 (10Dzahn) https://gerrit.wikimedia.org/r/c/operations/puppet/+/819672 [22:18:04] (03CR) 10Dzahn: [C: 03+1] Add gerrit2002.wikimedia.org to scap targets list [software/gerrit] (deploy/wmf/stable-3.4) - 10https://gerrit.wikimedia.org/r/819760 (https://phabricator.wikimedia.org/T243027) (owner: 10Ahmon Dancy) [22:22:12] (03CR) 10Ahmon Dancy: [C: 03+2] Add gerrit2002.wikimedia.org to scap targets list [software/gerrit] (deploy/wmf/stable-3.4) - 10https://gerrit.wikimedia.org/r/819760 (https://phabricator.wikimedia.org/T243027) (owner: 10Ahmon Dancy) [22:23:44] (03Merged) 10jenkins-bot: Add gerrit2002.wikimedia.org to scap targets list [software/gerrit] (deploy/wmf/stable-3.4) - 10https://gerrit.wikimedia.org/r/819760 (https://phabricator.wikimedia.org/T243027) (owner: 10Ahmon Dancy) [22:42:26] 10Continuous-Integration-Config, 10MW-1.39-notes (1.39.0-wmf.25; 2022-08-15), 10Patch-For-Review: Upgrade primary branch of all Wikimedia-deployed repos to a version of mediawiki-tools-phan including T270553 - https://phabricator.wikimedia.org/T295285 (10Reedy) Only Wikibase left [23:22:30] PROBLEM - gerrit process on gerrit2002 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/lib/jvm/java-11-openjdk-amd64/bin/java .*-jar /var/lib/gerrit2/review_site/bin/gerrit.war daemon -d /var/lib/gerrit2/review_site https://wikitech.wikimedia.org/wiki/Gerrit [23:24:27] RECOVERY - gerrit process on gerrit2002 is OK: PROCS OK: 1 process with regex args ^/usr/lib/jvm/java-11-openjdk-amd64/bin/java .*-jar /var/lib/gerrit2/review_site/bin/gerrit.war daemon -d /var/lib/gerrit2/review_site https://wikitech.wikimedia.org/wiki/Gerrit