[05:14:33] PROBLEM - PHD should be supervising processes on phab1001 is CRITICAL: PROCS CRITICAL: 2 processes with UID = 497 (phd) https://wikitech.wikimedia.org/wiki/Phabricator [05:17:03] RECOVERY - PHD should be supervising processes on phab1001 is OK: PROCS OK: 7 processes with UID = 497 (phd) https://wikitech.wikimedia.org/wiki/Phabricator [09:35:10] (03PS1) 10Hashar: Json schema from Gerrit Java event classes [software/gerrit/plugins/events-wikimedia] - 10https://gerrit.wikimedia.org/r/814725 (https://phabricator.wikimedia.org/T304947) [09:39:26] (03PS1) 10Hashar: Rename jsonschema-generator -> events-wikimedia [integration/config] - 10https://gerrit.wikimedia.org/r/814728 (https://phabricator.wikimedia.org/T304947) [09:39:56] (03CR) 10Hashar: [C: 03+2] Rename jsonschema-generator -> events-wikimedia [integration/config] - 10https://gerrit.wikimedia.org/r/814728 (https://phabricator.wikimedia.org/T304947) (owner: 10Hashar) [09:41:57] (03Merged) 10jenkins-bot: Rename jsonschema-generator -> events-wikimedia [integration/config] - 10https://gerrit.wikimedia.org/r/814728 (https://phabricator.wikimedia.org/T304947) (owner: 10Hashar) [09:45:47] (03CR) 10Hashar: "recheck" [software/gerrit/plugins/events-wikimedia] - 10https://gerrit.wikimedia.org/r/814725 (https://phabricator.wikimedia.org/T304947) (owner: 10Hashar) [09:46:11] !log Reloaded Zuul for https://gerrit.wikimedia.org/r/814728 [09:46:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:57:55] (03PS1) 10Hashar: jjb: update php jobs to have php-pcov included [integration/config] - 10https://gerrit.wikimedia.org/r/814730 (https://phabricator.wikimedia.org/T280170) [09:58:19] !log Refreshing Jenkins jobs for https://gerrit.wikimedia.org/r/c/integration/config/+/814730 jjb: update php jobs to have php-pcov included [09:58:20] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:58:47] (03CR) 10Hashar: [C: 03+2] jjb: update php jobs to have php-pcov included [integration/config] - 10https://gerrit.wikimedia.org/r/814730 (https://phabricator.wikimedia.org/T280170) (owner: 10Hashar) [10:01:04] (03Merged) 10jenkins-bot: jjb: update php jobs to have php-pcov included [integration/config] - 10https://gerrit.wikimedia.org/r/814730 (https://phabricator.wikimedia.org/T280170) (owner: 10Hashar) [10:39:34] (03PS1) 10Hashar: jjb: update jobs having quibble-buster-php72 [integration/config] - 10https://gerrit.wikimedia.org/r/814745 (https://phabricator.wikimedia.org/T279833) [10:40:10] !log Refreshing Jenkins jobs for https://gerrit.wikimedia.org/r/814745 [10:40:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:40:23] (03CR) 10Hashar: [C: 03+2] jjb: update jobs having quibble-buster-php72 [integration/config] - 10https://gerrit.wikimedia.org/r/814745 (https://phabricator.wikimedia.org/T279833) (owner: 10Hashar) [10:43:38] (03Merged) 10jenkins-bot: jjb: update jobs having quibble-buster-php72 [integration/config] - 10https://gerrit.wikimedia.org/r/814745 (https://phabricator.wikimedia.org/T279833) (owner: 10Hashar) [12:57:16] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Release, 10Train Deployments: 1.39.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T308074 (10dom_walden) [12:58:35] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Release, 10Train Deployments: 1.39.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T308074 (10dom_walden) I have raised T313220 as a blocking task. It could lead to more exceptions on production and it will prevent users from importing article. [13:16:28] (03PS1) 10Hashar: Send events to Wikimedia EventGate [software/gerrit/plugins/events-wikimedia] - 10https://gerrit.wikimedia.org/r/814807 [13:18:31] (03CR) 10Hashar: "This is merely non sense coming from https://gerrit.wikimedia.org/r/plugins/gitiles/wikimedia-event-utilities/+/refs/heads/master "Genera" [software/gerrit/plugins/events-wikimedia] - 10https://gerrit.wikimedia.org/r/814807 (owner: 10Hashar) [13:40:49] !log lucaswerkmeister-wmde@deployment-deploy03:~$ sql wikishared --write < /srv/mediawiki-staging/php-master/extensions/CampaignEvents/db_patches/mysql/tables-generated.sql # T311752 [13:40:51] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:40:51] T311752: Release V0 of the CampaignEvents extension to the Beta Cluster - https://phabricator.wikimedia.org/T311752 [13:56:10] 10Release-Engineering-Team, 10CampaignEvents, 10Campaign-Registration, 10Campaign-Tools (Campaign-Tools-Sprint-16), and 2 others: Release V0 of the CampaignEvents extension to the Beta Cluster - https://phabricator.wikimedia.org/T311752 (10Daimona) [14:10:49] 10GitLab (CI & Job Runners), 10serviceops, 10serviceops-collab, 10Patch-For-Review: DNS/networking not working on Trusted Runners - https://phabricator.wikimedia.org/T311241 (10Jelto) >>! In T311241#8038385, @Dzahn wrote: > deployed the latest change, restarted docker on gitlab-runner2004, restarted the jo... [16:05:31] 10Phabricator, 10Project-Admins, 10Codex: Create custom Phabricator Forms for Codex tasks - https://phabricator.wikimedia.org/T312232 (10AnneT) 05Openβ†’03Declined We didn't want to use Phabulous because we expect this form to change occasionally and were concerned that people might use outdated links, but... [16:26:01] 10GitLab (CI & Job Runners), 10Release-Engineering-Team, 10Security-Team, 10SecTeam-Processed, 10user-sbassett: Allow registry.gitlab.com/security-products/**/* for gitlab shared runner docker images - https://phabricator.wikimedia.org/T312961 (10sbassett) [16:36:21] 10Release-Engineering-Team, 10Scap, 10Patch-For-Review: Scap pool counter error while backporting - https://phabricator.wikimedia.org/T310835 (10dancy) >>! In T310835#8076748, @Krinkle wrote: > I did three deployment sessions in the past week, each time encountered this during at least one of the syncs. > >... [16:47:17] (03CR) 10Ahmon Dancy: [C: 04-2] make-container-image: push -q to all docker-pusher calls [tools/release] - 10https://gerrit.wikimedia.org/r/802648 (owner: 10Ahmon Dancy) [16:52:30] someone willing to take a look at https://gerrit.wikimedia.org/r/c/mediawiki/tools/release/+/813343 this weeks branch happens? [16:57:55] James_F: ^ [17:00:49] * before this weeks branch happens [17:21:39] (03CR) 10Jforrester: [C: 03+2] Stop branching CongressLookup for Wikimedia production [tools/release] - 10https://gerrit.wikimedia.org/r/813343 (https://phabricator.wikimedia.org/T312894) (owner: 10Zabe) [17:22:06] Done. [17:22:16] thanks [17:23:15] (03Merged) 10jenkins-bot: Stop branching CongressLookup for Wikimedia production [tools/release] - 10https://gerrit.wikimedia.org/r/813343 (https://phabricator.wikimedia.org/T312894) (owner: 10Zabe) [17:45:59] (03CR) 10Dduvall: [C: 03+2] Set umask in file creating cmds [tools/scap] - 10https://gerrit.wikimedia.org/r/811785 (owner: 10Jeena Huneidi) [17:52:02] (03Merged) 10jenkins-bot: Set umask in file creating cmds [tools/scap] - 10https://gerrit.wikimedia.org/r/811785 (owner: 10Jeena Huneidi) [18:04:48] 10Gerrit, 10Release-Engineering-Team (The Decommission Mission πŸ’€), 10SRE, 10serviceops, 10serviceops-collab: replacement for gerrit2001 - https://phabricator.wikimedia.org/T243027 (10thcipriani) [18:08:50] (03PS1) 10Ebernhardson: Stop weekly release branching ApiFeatureUsage [tools/release] - 10https://gerrit.wikimedia.org/r/814872 (https://phabricator.wikimedia.org/T313248) [18:09:20] 10Release-Engineering-Team (The Decommission Mission πŸ’€): Decide how to bring up Gerrit2002 - https://phabricator.wikimedia.org/T313250 (10thcipriani) [18:09:31] 10Gerrit, 10Release-Engineering-Team (The Decommission Mission πŸ’€): Decide how to bring up Gerrit2002 - https://phabricator.wikimedia.org/T313250 (10thcipriani) [18:10:11] ^ mutante hashar that's my rough understanding of our options for brining up Gerrit2002β€”let me know of the task what I got wrong :D [18:11:36] thcipriani: option 1a: tell Gerrit that from now on the replica has a new address option 1b: tell Gerrit that from now on it shoudl replicate to 2 instead of 1 replica [18:11:53] 1b might require code changes to allow more than one replica [18:12:03] but is a bit nicer than just switching.. maybe [18:12:39] adding that on the ticket [18:12:52] 10Gerrit, 10Release-Engineering-Team (The Decommission Mission πŸ’€): Decide how to bring up Gerrit2002 - https://phabricator.wikimedia.org/T313250 (10thcipriani) [18:13:06] please do, added a pros and cons section [18:15:02] pros: [18:15:03] gerrit [18:15:04] cons: [18:15:06] gerrit [18:16:27] 1a) - don't tell Gerrit anything. simply change DNS so that gerrit-replica points to a new IP [18:20:40] 10Gerrit, 10Release-Engineering-Team (The Decommission Mission πŸ’€): Decide how to bring up Gerrit2002 - https://phabricator.wikimedia.org/T313250 (10Dzahn) Option 1 has 2 or 3 sub-options. 1a) - don't tell Gerrit anything specifically, only change DNS for gerrit-replica.wikimedia.org to point to the new IP / g... [18:21:18] https://phabricator.wikimedia.org/T313250#8085611 [18:28:53] can't you just plug gerrit2002 to the power grid? [18:30:10] power up machine, scap to gerrit*, switch IP gerrit-replica.wm.org points to.. wait an hour. done :o [18:30:13] if only it were that [18:34:09] Reedy: :D --- it's like you just summarized some substantial portion of my working life [18:42:23] 10GitLab (Project Migration), 10Phabricator, 10Release-Engineering-Team (The Decommission Mission πŸ’€), 10Epic, and 2 others: Migrate active repositories in Phabricator Differential to GitLab - https://phabricator.wikimedia.org/T191182 (10brennen) [18:44:24] mutante: hmm, does T164810 fit into or get overridden by plans for the current sprint? [18:44:25] T164810: Switch phabricator production to codfw - https://phabricator.wikimedia.org/T164810 [18:46:57] 10GitLab (Project Migration), 10Phabricator, 10Release-Engineering-Team (The Decommission Mission πŸ’€), 10Epic, and 2 others: Migrate active repositories in Phabricator Differential to GitLab - https://phabricator.wikimedia.org/T191182 (10brennen) (That may have been a mis-tag on my part, in that we probably... [18:47:30] (03CR) 10Arlolra: [C: 03+2] Get rid of a bunch of CLI options; config file is simpler [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/813945 (owner: 10Subramanya Sastry) [18:47:41] brennen: I would say neither, it's out of scope but also does not get overridden. it's just a separate thing. that is what I mentioned at the very end of the meeting. we are unblocked there now [18:48:00] but for now we just care about the hardware decom [18:48:10] (03CR) 10Arlolra: [C: 03+2] Add UI styling resource module [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/813272 (owner: 10Subramanya Sastry) [18:48:32] what's new on the ticket you are linking is that it should not be "stalled" anymore I guess [18:48:42] it was stalled because back then it was not technically possible yet [18:48:46] but now it should be [18:49:12] that is because "have a misc database in codfw" is possible now [18:52:35] cool, gotcha. [18:53:46] 10Phabricator, 10Release-Engineering-Team (The Decommission Mission πŸ’€), 10User-brennen: Deploy Phabricator with scap - https://phabricator.wikimedia.org/T313259 (10brennen) [18:54:28] (03Merged) 10jenkins-bot: Add UI styling resource module [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/813272 (owner: 10Subramanya Sastry) [18:54:30] (03Merged) 10jenkins-bot: Get rid of a bunch of CLI options; config file is simpler [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/813945 (owner: 10Subramanya Sastry) [19:01:51] 10Release-Engineering-Team (Next), 10tech-decision-forum, 10Code-Stewardship-Reviews, 10Documentation, 10User-AKlapper: Document checklist steps to undeploy / sunset a codebase on WMF servers (not: archiving) - https://phabricator.wikimedia.org/T294329 (10Amire80) Is there at least a draft checklist some... [19:14:17] mutante: gerrit1001= production gerrit, gerrit2001 = replica, gerrit2002 = new-replica ? [19:16:22] dancy: that is correct. no gerrit1002 because gerrit1001 is still fine to use [19:16:32] gerrit1001 was bought in 2019 but gerrit2001 is from 2016 [19:16:48] before prod was gerrit1001 it was "cobalt" [19:16:52] thx. I'm going to update the description of T243027 to clarify these bits. [19:16:53] T243027: replacement for gerrit2001 - https://phabricator.wikimedia.org/T243027 [19:17:13] cool [19:20:27] 10Gerrit, 10Release-Engineering-Team (The Decommission Mission πŸ’€), 10SRE, 10serviceops, 10serviceops-collab: replacement for gerrit2001 - https://phabricator.wikimedia.org/T243027 (10dancy) [19:21:15] 10Gerrit, 10Release-Engineering-Team (The Decommission Mission πŸ’€): Decide how to bring up Gerrit2002 - https://phabricator.wikimedia.org/T313250 (10dancy) [19:29:51] 10Release-Engineering-Team (Next), 10tech-decision-forum, 10Code-Stewardship-Reviews, 10Documentation, 10User-AKlapper: Document checklist steps to undeploy / sunset a codebase on WMF servers (not: archiving) - https://phabricator.wikimedia.org/T294329 (10thcipriani) Hrm, I can start a draft here anywayβ€”... [19:35:58] (03CR) 10Thcipriani: [C: 03+2] deployments-calendar.yaml: Add self to utc_morning_training [tools/release] - 10https://gerrit.wikimedia.org/r/813739 (owner: 10Samtar) [19:36:48] (03Merged) 10jenkins-bot: deployments-calendar.yaml: Add self to utc_morning_training [tools/release] - 10https://gerrit.wikimedia.org/r/813739 (owner: 10Samtar) [19:37:03] (03PS1) 10Ahmon Dancy: Release 4.11.0-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/814888 [19:37:05] (03CR) 10Ahmon Dancy: [C: 03+2] Release 4.11.0-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/814888 (owner: 10Ahmon Dancy) [19:41:42] (03Merged) 10jenkins-bot: Release 4.11.0-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/814888 (owner: 10Ahmon Dancy) [19:43:06] !log Upgrading scap to 4.10.0-1+0~20220718175214.344~1.gbpe518a1 in beta cluster [19:43:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:43:53] 10Release-Engineering-Team (Next), 10tech-decision-forum, 10Code-Stewardship-Reviews, 10Documentation, 10User-AKlapper: Document checklist steps to undeploy / sunset a codebase on WMF servers (not: archiving) - https://phabricator.wikimedia.org/T294329 (10Amire80) That's a start. My few tips: # Once it's... [23:28:27] (03PS2) 10Hashar: Send events to Wikimedia EventGate [software/gerrit/plugins/events-wikimedia] - 10https://gerrit.wikimedia.org/r/814807 [23:29:20] (03CR) 10Hashar: "And on commenting on a change I get in the error log:" [software/gerrit/plugins/events-wikimedia] - 10https://gerrit.wikimedia.org/r/814807 (owner: 10Hashar)