[00:36:28] PROBLEM - PHD should be supervising processes on phab1001 is CRITICAL: PROCS CRITICAL: 2 processes with UID = 497 (phd) https://wikitech.wikimedia.org/wiki/Phabricator [00:38:50] RECOVERY - PHD should be supervising processes on phab1001 is OK: PROCS OK: 4 processes with UID = 497 (phd) https://wikitech.wikimedia.org/wiki/Phabricator [00:45:44] PROBLEM - PHD should be supervising processes on phab1001 is CRITICAL: PROCS CRITICAL: 2 processes with UID = 497 (phd) https://wikitech.wikimedia.org/wiki/Phabricator [00:48:04] RECOVERY - PHD should be supervising processes on phab1001 is OK: PROCS OK: 22 processes with UID = 497 (phd) https://wikitech.wikimedia.org/wiki/Phabricator [01:16:00] PROBLEM - PHD should be supervising processes on phab1001 is CRITICAL: PROCS CRITICAL: 2 processes with UID = 497 (phd) https://wikitech.wikimedia.org/wiki/Phabricator [01:20:36] RECOVERY - PHD should be supervising processes on phab1001 is OK: PROCS OK: 3 processes with UID = 497 (phd) https://wikitech.wikimedia.org/wiki/Phabricator [02:11:40] PROBLEM - PHD should be supervising processes on phab1001 is CRITICAL: PROCS CRITICAL: 2 processes with UID = 497 (phd) https://wikitech.wikimedia.org/wiki/Phabricator [02:16:20] RECOVERY - PHD should be supervising processes on phab1001 is OK: PROCS OK: 7 processes with UID = 497 (phd) https://wikitech.wikimedia.org/wiki/Phabricator [06:39:06] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar), 10Code-Stewardship-Reviews: deployment-prep: Code stewardship request - https://phabricator.wikimedia.org/T215217 (10Ladsgroup) Pinging because one month has passed since the last comment on this. [06:48:39] o/ I added a note on https://wikitech.wikimedia.org/wiki/Deployments#Week_of_April_25 to mention that we merged a set of patches related to the elasticsearch upgrade [07:34:27] (03Abandoned) 10David Caro: operations-puppet: Add possibility to use custom facts [integration/config] - 10https://gerrit.wikimedia.org/r/778474 (owner: 10David Caro) [07:34:32] dcausse: I am pretty sure the notice might be missed. May you ping the train blocker task indicating it is a risky patch? ;) [07:34:51] dcausse: blocker task is https://phabricator.wikimedia.org/T305215 and you can copy paste the template from https://wikitech.wikimedia.org/wiki/Deployments/Risky_change_template ;] [07:35:10] hashar: thanks! will add a note there [07:46:01] 10Release-Engineering-Team (🌱 Spring Cleaning — April 2022), 10Release, 10Train Deployments: 1.39.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T305215 (10dcausse) * **Change**: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CirrusSearch/+/785199/ * **Summary**: ** This merges a bra... [08:16:04] dcausse: looks good :] [08:16:12] merci! [12:17:39] 10Phabricator, 10Release-Engineering-Team (🌱 Spring Cleaning — April 2022): Puppet failure on phabricator-stage-1001.devtools - https://phabricator.wikimedia.org/T299997 (10hashar) I have made backups: | `/var/lib/mysql` | `/var/lib/mysql-20220425` | `/srv/sqldata` | `/srv/sqldata-20220425` Then copied the f... [12:45:30] 10Phabricator, 10Release-Engineering-Team (🌱 Spring Cleaning — April 2022): Puppet failure on phabricator-stage-1001.devtools - https://phabricator.wikimedia.org/T299997 (10hashar) The config is hold in a json file last changed on Mar 1 11:12 ` lang=json,name=/srv/phab/phabricator/conf/local/www.json { "mys... [14:18:53] 10Release-Engineering-Team, 10MediaWiki-Docker, 10Performance-Team (Radar): Composer version in mediawiki-docker unsupported by mediawiki-vendor - https://phabricator.wikimedia.org/T306802 (10Krinkle) [14:19:04] 10Release-Engineering-Team, 10MediaWiki-Docker, 10Performance-Team (Radar): Composer version in mediawiki-docker unsupported by mediawiki-vendor - https://phabricator.wikimedia.org/T306802 (10Krinkle) p:05Triage→03High [14:19:50] 10Release-Engineering-Team, 10MediaWiki-Docker, 10Performance-Team (Radar): Composer version in mediawiki-docker unsupported by mediawiki-vendor - https://phabricator.wikimedia.org/T306802 (10Krinkle) My current workaround is: ` $ docker-compose exec -u root mediawiki bash root@3da127831b3d:/var/www/htm... [15:28:40] (03CR) 10Jforrester: [C: 03+2] Zuul: [mediawiki/extensions/RegularTooltips] Add basic quibble CI [integration/config] - 10https://gerrit.wikimedia.org/r/784779 (owner: 10Zoranzoki21) [15:30:40] 10Beta-Cluster-Infrastructure, 10Wikidata, 10Wikidata-Query-Service, 10wdwb-tech, 10Discovery-Search (Current work): Upgrade deployment-wdqs01 host to Buster - https://phabricator.wikimedia.org/T306054 (10dcausse) [15:31:05] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/RegularTooltips] Add basic quibble CI [integration/config] - 10https://gerrit.wikimedia.org/r/784779 (owner: 10Zoranzoki21) [15:31:27] !log Zuul: [mediawiki/extensions/RegularTooltips] Add basic quibble CI [15:31:28] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:31:46] 10Beta-Cluster-Infrastructure, 10Wikidata, 10Wikidata-Query-Service, 10wdwb-tech, 10Discovery-Search (Current work): Upgrade deployment-wdqs01 host to Buster - https://phabricator.wikimedia.org/T306054 (10bking) a:03bking [15:39:13] (03CR) 10Ahmon Dancy: "The change looks ok to me. Just a minor adjustment requested." [tools/scap] - 10https://gerrit.wikimedia.org/r/785154 (https://phabricator.wikimedia.org/T303801) (owner: 10Elukey) [15:56:45] 10Beta-Cluster-Infrastructure, 10Wikidata, 10Wikidata-Query-Service, 10wdwb-tech, 10Discovery-Search (Current work): Upgrade deployment-wdqs01 host to Buster - https://phabricator.wikimedia.org/T306054 (10EBernhardson) [17:15:59] (03CR) 10Ahmon Dancy: Update to only support helm 3 (032 comments) [releng/local-charts] - 10https://gerrit.wikimedia.org/r/784251 (owner: 10Majavah) [17:20:01] (03CR) 10Ahmon Dancy: "This was handled by https://gerrit.wikimedia.org/r/c/integration/config/+/777872" [integration/config] - 10https://gerrit.wikimedia.org/r/779451 (https://phabricator.wikimedia.org/T303408) (owner: 10Hnowlan) [17:23:14] (03CR) 10Ahmon Dancy: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/779450 (https://phabricator.wikimedia.org/T303408) (owner: 10Hnowlan) [17:26:02] (03CR) 10Ahmon Dancy: [C: 03+2] Add service pipeline entry for image-suggestions service [integration/config] - 10https://gerrit.wikimedia.org/r/779450 (https://phabricator.wikimedia.org/T303408) (owner: 10Hnowlan) [17:27:59] (03Merged) 10jenkins-bot: Add service pipeline entry for image-suggestions service [integration/config] - 10https://gerrit.wikimedia.org/r/779450 (https://phabricator.wikimedia.org/T303408) (owner: 10Hnowlan) [17:29:07] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/779450 [17:29:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:29:30] (03CR) 10Ahmon Dancy: "Deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/779450 (https://phabricator.wikimedia.org/T303408) (owner: 10Hnowlan) [17:32:25] (03PS1) 10Ahmon Dancy: Release 4.7.0-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/785899 [17:32:40] (03CR) 10Ahmon Dancy: [C: 03+2] Release 4.7.0-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/785899 (owner: 10Ahmon Dancy) [17:36:38] (03Merged) 10jenkins-bot: Release 4.7.0-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/785899 (owner: 10Ahmon Dancy) [17:42:35] 10Release-Engineering-Team, 10Scap, 10serviceops: Deploy Scap version 4.7.0 - https://phabricator.wikimedia.org/T306827 (10dancy) [17:47:21] (03CR) 10Ahmon Dancy: followup fix: scap prep can try to chmod dir not owned by user (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/781059 (owner: 10Ahmon Dancy) [17:55:23] 10Release-Engineering-Team (Priority Backlog 📥), 10Scap, 10SRE, 10Python3-Porting: git-fat needs to be ported to Python 3 - https://phabricator.wikimedia.org/T279509 (10dancy) @thcipriani Based on reading about git-lfs and git-fat (including outstanding issues on GitHub), I'm in favor of migrating to git-l... [17:56:37] 10Beta-Cluster-Infrastructure, 10Data-Engineering, 10Data-Engineering-Kanban, 10Event-Platform, 10Patch-For-Review: Upgrade event platform related VMs in deployment-prep to Debian bullsye (or buster) - https://phabricator.wikimedia.org/T304433 (10Ottomata) [18:03:01] 10Release-Engineering-Team (Priority Backlog 📥), 10Scap, 10SRE, 10Python3-Porting: git-fat needs to be ported to Python 3 - https://phabricator.wikimedia.org/T279509 (10Dzahn) T214229 - scap3 + git-fat results in git status with permissions errors T202100 - Intermittent git-fat failure during deploy T147... [18:10:23] (03CR) 10Ahmon Dancy: [C: 03+2] preserve user ssh env during Scap self-calls [tools/scap] - 10https://gerrit.wikimedia.org/r/785145 (https://phabricator.wikimedia.org/T302488) (owner: 10Jaime Nuche) [18:11:50] 10Phabricator, 10Project-Admins, 10Release-Engineering-Team (Radar), 10Security-Team, and 2 others: Move the #acl_security_volunteer policy outside of #acl_security - https://phabricator.wikimedia.org/T305890 (10sbassett) >>! In T305890#7865473, @DannyS712 wrote: > What about using a herald rule with a cus... [18:12:18] (03Merged) 10jenkins-bot: preserve user ssh env during Scap self-calls [tools/scap] - 10https://gerrit.wikimedia.org/r/785145 (https://phabricator.wikimedia.org/T302488) (owner: 10Jaime Nuche) [18:28:50] 10Beta-Cluster-Infrastructure, 10Wikidata, 10Wikidata-Query-Service, 10wdwb-tech, 10Discovery-Search (Current work): Upgrade deployment-wdqs01 host to Buster - https://phabricator.wikimedia.org/T306054 (10bking) I deleted instance `deployment-wdqs01 ` , please let us know if any further cleanup is required. [18:39:14] 10Release-Engineering-Team (Priority Backlog 📥), 10Scap, 10SRE, 10Python3-Porting: git-fat needs to be ported to Python 3 - https://phabricator.wikimedia.org/T279509 (10thcipriani) >>! In T279509#7877992, @dancy wrote: > I can help on the scap side. I haven't touched archiva yet. There is support in scap... [18:53:18] 10Beta-Cluster-Infrastructure, 10Wikimedia-Site-requests, 10Growth-Team (Current Sprint): Reopen beta eswiki - https://phabricator.wikimedia.org/T306833 (10Tgr) [18:56:26] 10Release-Engineering-Team (Priority Backlog 📥), 10Scap, 10SRE, 10Python3-Porting: git-fat needs to be ported to Python 3 - https://phabricator.wikimedia.org/T279509 (10Ottomata) > deploy directly from Gerrit ...say more :) The jar binaries are built by maven-release-plugin in a jenkins job and then uplo... [18:59:10] 10Beta-Cluster-Infrastructure, 10ContentTranslation, 10Wikimedia-Site-requests, 10Patch-For-Review, 10User-Luke081515: Put beta eswiki to read-only mode - https://phabricator.wikimedia.org/T109157 (10Tgr) I'm planning to undo this per {T306833}. If the spam becomes a problem, we can figure something out... [19:07:37] 10Beta-Cluster-Infrastructure, 10MediaWiki-Search, 10PageImages, 10Readers-Web-Backlog, and 3 others: PageImages ignores MediaWiki:Bad image list, (uses MediaWiki:Pageimages-denylist instead) displaying search results that are inappropriate for some readers - https://phabricator.wikimedia.org/T306246 (10Jdr... [19:11:51] 10Beta-Cluster-Infrastructure, 10Data-Engineering, 10Data-Engineering-Kanban, 10Event-Platform: Upgrade event platform related VMs in deployment-prep to Debian bullsye (or buster) - https://phabricator.wikimedia.org/T304433 (10Ottomata) [19:13:00] 10Beta-Cluster-Infrastructure, 10Data-Engineering, 10Data-Engineering-Kanban, 10Event-Platform: Upgrade event platform related VMs in deployment-prep to Debian bullsye (or buster) - https://phabricator.wikimedia.org/T304433 (10Ottomata) Update: all nodes have been replaced with either bullsye or buster! O... [19:16:24] 10Beta-Cluster-Infrastructure, 10Data-Engineering, 10Data-Engineering-Kanban, 10Event-Platform: Upgrade event platform related VMs in deployment-prep to Debian bullsye (or buster) - https://phabricator.wikimedia.org/T304433 (10Ottomata) Ah! I just needed to add the correct firewall security group. It works! [19:18:29] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Stretch Deprecation): Cloud VPS "deployment-prep" project Stretch deprecation - https://phabricator.wikimedia.org/T306068 (10Ottomata) [19:20:20] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Stretch Deprecation): Cloud VPS "deployment-prep" project Stretch deprecation - https://phabricator.wikimedia.org/T306068 (10Ottomata) [19:25:51] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Stretch Deprecation): Cloud VPS "deployment-prep" project Stretch deprecation - https://phabricator.wikimedia.org/T306068 (10Krinkle) [19:26:35] 10Beta-Cluster-Infrastructure, 10MediaWiki-Search, 10PageImages, 10Readers-Web-Backlog, and 3 others: PageImages ignores MediaWiki:Bad image list, (uses MediaWiki:Pageimages-denylist instead) displaying search results that are inappropriate for some readers - https://phabricator.wikimedia.org/T306246 (10Ale... [19:48:53] 10Release-Engineering-Team (Priority Backlog 📥), 10Scap, 10SRE, 10Python3-Porting: git-fat needs to be ported to Python 3 - https://phabricator.wikimedia.org/T279509 (10thcipriani) >>! In T279509#7878185, @Ottomata wrote: >> deploy directly from Gerrit > > ...say more :) > > The jar binaries are built by... [19:52:59] 10Release-Engineering-Team (Priority Backlog 📥), 10Scap, 10SRE, 10Python3-Porting: git-fat needs to be ported to Python 3 - https://phabricator.wikimedia.org/T279509 (10Ottomata) Can .jar .gitattributes be manged by git-lfs to download from Archiva API directly? E.g. this URL: http://archiva.wikimedia.org... [20:49:56] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar): Migrate deployment-prep away from Debian Stretch to Buster/Bullseye - https://phabricator.wikimedia.org/T278641 (10Jdforrester-WMF) [20:50:02] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Stretch Deprecation): Cloud VPS "deployment-prep" project Stretch deprecation - https://phabricator.wikimedia.org/T306068 (10Jdforrester-WMF) [20:50:15] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar): Migrate deployment-prep away from Debian Stretch to Buster/Bullseye - https://phabricator.wikimedia.org/T278641 (10Jdforrester-WMF) [20:50:53] 10Beta-Cluster-Infrastructure, 10Wikidata, 10Wikidata-Query-Service, 10wdwb-tech, 10Discovery-Search (Current work): Upgrade deployment-wdqs01 host to Buster - https://phabricator.wikimedia.org/T306054 (10Jdforrester-WMF) 05Open→03Resolved Thank you! [20:53:16] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar): Migrate deployment-prep away from Debian Stretch to Buster/Bullseye - https://phabricator.wikimedia.org/T278641 (10Jdforrester-WMF) [20:54:58] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar): Migrate deployment-prep away from Debian Stretch to Buster/Bullseye - https://phabricator.wikimedia.org/T278641 (10Jdforrester-WMF) [21:38:23] 10Beta-Cluster-Infrastructure, 10MediaWiki-Search, 10PageImages, 10Readers-Web-Backlog, and 3 others: PageImages ignores MediaWiki:Bad image list, (uses MediaWiki:Pageimages-denylist instead) displaying search results that are inappropriate for some readers - https://phabricator.wikimedia.org/T306246 (10Jdl... [22:07:48] 10Release-Engineering-Team (Next), 10Scap: scap proxies are CPU and/or network bound - https://phabricator.wikimedia.org/T305466 (10dancy) I'm still seeing some imbalance and cross-dc source selection with the latest code so I'm investigating again. [22:17:55] ^ ignore that. [22:23:40] 10Continuous-Integration-Config, 10Release-Engineering-Team (Seen), 10Vector (Vector (Tracking)): [CI] Replace npm-test job with npm-test + MediaWiki Core - https://phabricator.wikimedia.org/T252772 (10Jdlrobson) [22:27:54] 10Release-Engineering-Team, 10Scap, 10serviceops, 10User-brennen: Deploy Scap version 4.7.0 - https://phabricator.wikimedia.org/T306827 (10brennen) [22:55:46] dancy: I just saw your fixes for find_nearest_host in https://gerrit.wikimedia.org/r/c/mediawiki/tools/scap/+/779559. Nice work. :) I was super proud of that code when I wrote it, but I that that also predated IPv6 being used in the core network. [22:56:04] *think that [22:58:44] It was a massive improvement at the time ;) [22:58:49] :) not ;) [23:01:04] yeah, it worked better than the old `ping` method and also did not require a setuid binary. I think we improved on spread too at the time.