[00:05:38] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.38.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T293953 (10ssastry) >>! In T293953#7551832, @thcipriani wrote: >>>! In T293953#7551761, @ssastry wrote: >> All that said, if train operators prefer that we mitigate that fo... [00:31:14] dancy: I didn't get a chance to do the rest of the scap upgrade today, do you still want me to do it before tomorrow's train? it sounded like you wanted to test a sync-world beforehand? [01:04:16] 10GitLab, 10Striker, 10Tools: Replace Diffusion integration with Gitlab integration in Striker (toolsadmin) - https://phabricator.wikimedia.org/T296893 (10mmodell) I'd like to automate management of git repos as well, for the sake of easy management and tracking what exists in gitlab, diffusion, gerrit and g... [06:13:40] (Queue (Jenkins jobs + Zuul functions) alert) firing: Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org [06:22:07] PROBLEM - Work requests waiting in Zuul Gearman server on contint2001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [400.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [06:32:57] RECOVERY - Work requests waiting in Zuul Gearman server on contint2001 is OK: OK: Less than 100.00% above the threshold [200.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [06:33:40] (Queue (Jenkins jobs + Zuul functions) alert) firing: (2) Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org [06:53:40] (Queue (Jenkins jobs + Zuul functions) alert) resolved: Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org [08:50:41] 10Project-Admins: Create project tag for User-Slst2020 - https://phabricator.wikimedia.org/T297181 (10Slst2020) [09:11:13] 10Project-Admins: Create project tag for User-Slst2020 - https://phabricator.wikimedia.org/T297181 (10Zabe) [09:11:16] 10Phabricator (Upstream), 10Upstream: Per-user projects for personal work in progress tracking - https://phabricator.wikimedia.org/T555 (10Zabe) [09:38:26] 10Project-Admins, 10User-Kizule: Create project tag for User-Slst2020 - https://phabricator.wikimedia.org/T297181 (10Kizule) a:03Kizule Doing this right now... [09:41:54] 10Project-Admins, 10User-Kizule: Create project tag for User-Slst2020 - https://phabricator.wikimedia.org/T297181 (10Kizule) 05Open→03Resolved Requested public project #user-slst2020 has been created: https://phabricator.wikimedia.org/project/view/5658/ [Recommended practices for project and workboard man... [09:41:56] 10Phabricator (Upstream), 10Upstream: Per-user projects for personal work in progress tracking - https://phabricator.wikimedia.org/T555 (10Kizule) [10:01:41] (03CR) 10Jbond: [C: 03+1] "LGTM (will need someone from releng to +2)" [integration/config] - 10https://gerrit.wikimedia.org/r/743365 (https://phabricator.wikimedia.org/T296984) (owner: 10David Caro) [10:19:36] (03PS17) 10Kosta Harlan: [WIP] Run PHPUnit tests in parallel [integration/quibble] - 10https://gerrit.wikimedia.org/r/742200 (https://phabricator.wikimedia.org/T50217) [10:20:13] (03PS38) 10Kosta Harlan: [DNM] CI full run with extensions [integration/quibble] - 10https://gerrit.wikimedia.org/r/742201 [10:28:04] (03CR) 10jerkins-bot: [V: 04-1] [DNM] CI full run with extensions [integration/quibble] - 10https://gerrit.wikimedia.org/r/742201 (owner: 10Kosta Harlan) [15:11:45] Hi, question about gitlab at wmf: is it possible to create gitlab pages? [15:12:22] I suggest you ask in #wikimedia-gitlab, but I believe no, at least yet [15:14:47] majavah thanks! will do. [15:30:19] (03PS39) 10Kosta Harlan: [DNM] CI full run with extensions [integration/quibble] - 10https://gerrit.wikimedia.org/r/742201 [15:35:42] 10Continuous-Integration-Infrastructure, 10OOUI: Demos page for OOUI in php is broken - https://phabricator.wikimedia.org/T297035 (10matmarex) OOUI itself has required 7.2.9 since 2019: {68ef71f8f81c9203dfdca7af967266b446d54d44} This might be a problem with the setup on doc.wikimedia.org? Is it suddenly runni... [15:36:10] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.38.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T293953 (10Ladsgroup) Note that the blocker is in a weird state, it seems that there are several issues there. I think the db part is fixed but the memory part is not. Need... [15:37:58] (03CR) 10jerkins-bot: [V: 04-1] [DNM] CI full run with extensions [integration/quibble] - 10https://gerrit.wikimedia.org/r/742201 (owner: 10Kosta Harlan) [15:38:28] 10Continuous-Integration-Infrastructure, 10OOUI: Demos page for OOUI in php is broken - https://phabricator.wikimedia.org/T297035 (10Majavah) >>! In T297035#7553196, @matmarex wrote: > OOUI itself has required 7.2.9 since 2019: {68ef71f8f81c9203dfdca7af967266b446d54d44} > > This might be a problem with the se... [15:53:42] (03PS18) 10Kosta Harlan: [WIP] Run PHPUnit tests in parallel [integration/quibble] - 10https://gerrit.wikimedia.org/r/742200 (https://phabricator.wikimedia.org/T50217) [15:59:14] (03PS40) 10Kosta Harlan: [DNM] CI full run with extensions [integration/quibble] - 10https://gerrit.wikimedia.org/r/742201 [16:02:56] mutante: hey! I believe everything is ready, so let's switch doc from doc1001 to doc1002 at some point? [16:05:39] majavah: Appreciate you create those patches but they are the same I had uploaded and abandoned [16:06:07] sorry, I have another thing going on, will follow-up [16:06:19] cool, no worries [16:11:14] legoktm: Yes, I would like the scap 4.1.0 rollout at least to the deploy servers today. [16:15:45] (03CR) 10jerkins-bot: [V: 04-1] [DNM] CI full run with extensions [integration/quibble] - 10https://gerrit.wikimedia.org/r/742201 (owner: 10Kosta Harlan) [16:25:18] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.38.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T293953 (10ssastry) >>! In T293953#7551832, @thcipriani wrote: > If it's possible to mitigate this, that'd be ideal. We are working to take Parsoid out of the risky patch... [16:32:25] 10Continuous-Integration-Config, 10MW-on-K8s, 10MediaWiki-SettingsLoader, 10serviceops, 10Patch-For-Review: Install php-yaml for use by SettingsLoader - https://phabricator.wikimedia.org/T296331 (10dduvall) a:05dduvall→03Legoktm Reassigning to better reflect who's actually doing the work. [16:45:22] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.38.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T293953 (10cscott) The Parsoid "backport" is https://gerrit.wikimedia.org/r/c/mediawiki/vendor/+/744800. I assume that since the train hasn't rolled yet it's safe to merge... [17:20:22] 10Continuous-Integration-Config, 10MW-on-K8s, 10MediaWiki-SettingsLoader, 10serviceops, 10Patch-For-Review: Install php-yaml for use by SettingsLoader - https://phabricator.wikimedia.org/T296331 (10Legoktm) Yesterday I did all the appservers and api_appservers, today I should finish rolling it out everyw... [18:09:17] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.38.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T293953 (10dancy) [18:45:08] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 10SRE, 10serviceops, 10Patch-For-Review: replace doc1001.eqiad.wmnet with a buster VM and create the codfw equivalent - https://phabricator.wikimedia.org/T247653 (10Majavah) @hashar @Krinkle Content sync between instances, the je... [18:47:36] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.38.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T293953 (10EBernhardson) [18:56:20] dancy: so do you just want it on the deploy hosts for now, or should I upgrade it everywhere? [19:11:53] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 10SRE, 10serviceops, 10Patch-For-Review: replace doc1001.eqiad.wmnet with a buster VM and create the codfw equivalent - https://phabricator.wikimedia.org/T247653 (10Krinkle) 05Stalled→03Open [19:12:15] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 10SRE, 10serviceops, 10Patch-For-Review: replace doc1001.eqiad.wmnet with a buster VM and create the codfw equivalent - https://phabricator.wikimedia.org/T247653 (10Krinkle) a:05Dzahn→03hashar [19:24:55] legoktm: All hosts works for me. [19:25:04] :thumbsup: [19:27:57] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.38.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T293953 (10dancy) [19:35:28] {{done}} [19:35:44] 10Release-Engineering-Team, 10Scap, 10serviceops: Deploy Scap version 4.1.0 - https://phabricator.wikimedia.org/T296867 (10Legoktm) 05Open→03Resolved debmonitor confirms that it's everywhere now. [19:35:55] Thanks! [19:49:16] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar), 10MediaWiki-Parser: Templates rendering as links on beta cluster - https://phabricator.wikimedia.org/T173576 (10Jdlrobson) [20:03:13] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.38.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T293953 (10daniel) >>! In T293953#7553199, @Ladsgroup wrote: > Note that the blocker is in a weird state, it seems that there are several issues there. I think the db part... [20:28:19] (03PS1) 10BryanDavis: jjb: Define test and publish pipelines for developer-portal [integration/config] - 10https://gerrit.wikimedia.org/r/744871 [20:30:49] (03CR) 10jerkins-bot: [V: 04-1] jjb: Define test and publish pipelines for developer-portal [integration/config] - 10https://gerrit.wikimedia.org/r/744871 (owner: 10BryanDavis) [20:31:13] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.38.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T293953 (10daniel) [20:34:05] (03PS2) 10BryanDavis: jjb: Define test and publish pipelines for developer-portal [integration/config] - 10https://gerrit.wikimedia.org/r/744871 [20:40:14] (03CR) 10BryanDavis: "See Ia434672a91f0f35c025dc3584f0a5487e50516bc for the project side configuration for these pipelines." [integration/config] - 10https://gerrit.wikimedia.org/r/744871 (owner: 10BryanDavis) [21:10:05] PROBLEM - SSH on contint1001.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [21:22:08] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.38.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T293953 (10thcipriani) >>! In T293953#7553328, @ssastry wrote: >>>! In T293953#7551832, @thcipriani wrote: >> If it's possible to mitigate this, that'd be ideal. > > We a... [21:35:26] 10Continuous-Integration-Config, 10LibUp, 10Security-Team, 10Tools, 10SecTeam-Processed: Move php-security-checker.wmcloud.org to Toolforge - https://phabricator.wikimedia.org/T296967 (10Legoktm) My understanding is that composer-security-checker is the same functionality as https://github.com/fabpot/loc... [21:41:27] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T293953 (10brennen) [21:44:27] 10Continuous-Integration-Infrastructure, 10OOUI: Demos page for OOUI in php is broken - https://phabricator.wikimedia.org/T297035 (10Dylsss) [21:48:41] 10Continuous-Integration-Config, 10Security-Team: Add CI for mediawiki/extensions/SecurityApi - https://phabricator.wikimedia.org/T297243 (10Reedy) [21:53:06] (03PS1) 10STran: Add CI for mediawiki/extensions/SecurityAPI [integration/config] - 10https://gerrit.wikimedia.org/r/744883 (https://phabricator.wikimedia.org/T297243) [21:57:18] 10GitLab (Infrastructure), 10Release-Engineering-Team (Radar), 10serviceops: Self-reported GitLab SSH host key fingerprints don’t appear to match actual host key fingerprints - https://phabricator.wikimedia.org/T296944 (10brennen) p:05Triage→03Low We're using two separate sshd instances on gitlab1001 - o... [22:07:48] 10Continuous-Integration-Infrastructure, 10OOUI: Demos page for OOUI in php is broken - https://phabricator.wikimedia.org/T297035 (10Jdforrester-WMF) >>! In T297035#7553216, @Majavah wrote: >>>! In T297035#7553196, @matmarex wrote: >> OOUI itself has required 7.2.9 since 2019: {68ef71f8f81c9203dfdca7af967266b4... [22:11:08] RECOVERY - SSH on contint1001.mgmt is OK: SSH OK - OpenSSH_6.6 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [22:16:37] (03CR) 10SBassett: [C: 03+1] "LGTM" [integration/config] - 10https://gerrit.wikimedia.org/r/744883 (https://phabricator.wikimedia.org/T297243) (owner: 10STran) [22:31:02] (03PS1) 10Ahmon Dancy: Rename ScapCommon to ScapPull [tools/scap] - 10https://gerrit.wikimedia.org/r/744887 [22:31:08] (03PS1) 10Ahmon Dancy: sudo_check_call: Allow log level to be controlled [tools/scap] - 10https://gerrit.wikimedia.org/r/744888 [22:34:02] (03CR) 10Ahmon Dancy: [C: 03+2] Rename ScapCommon to ScapPull [tools/scap] - 10https://gerrit.wikimedia.org/r/744887 (owner: 10Ahmon Dancy) [22:34:27] (03CR) 10Ahmon Dancy: [C: 03+2] sudo_check_call: Allow log level to be controlled [tools/scap] - 10https://gerrit.wikimedia.org/r/744888 (owner: 10Ahmon Dancy) [22:34:34] 10Continuous-Integration-Infrastructure, 10OOUI: Demos page for OOUI in php is broken - https://phabricator.wikimedia.org/T297035 (10Tks4Fish) BTW I see here that I told Urbanecm on Nov. 23 about this page being down, but I'm pretty sure it was down even earlier when I tried to access it, just never reported a... [22:35:37] (03Merged) 10jenkins-bot: Rename ScapCommon to ScapPull [tools/scap] - 10https://gerrit.wikimedia.org/r/744887 (owner: 10Ahmon Dancy) [22:36:52] (03Merged) 10jenkins-bot: sudo_check_call: Allow log level to be controlled [tools/scap] - 10https://gerrit.wikimedia.org/r/744888 (owner: 10Ahmon Dancy) [22:38:11] 10Phabricator, 10Accessibility: Symbol for "Open" on Phabricator tasks suggests "alert" to me, and is inconsistent in its placement when task types are also used - https://phabricator.wikimedia.org/T297249 (10MBinder_WMF) [22:42:17] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T293953 (10Krinkle) [23:21:03] 10Diffusion, 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10Developer Productivity: Reduce task notification noise/frequency of changes to associated open patchsets - https://phabricator.wikimedia.org/T143162 (10thcipriani)