[00:09:23] 10Release-Engineering-Team (Priority Backlog 📥), 10GitLab (Auth & Access): Create a top level wmde group on Gitlab - https://phabricator.wikimedia.org/T291388 (10thcipriani) After discussing this with @brennen, I created a https://gitlab.wikimedia.org/repos/wmde that is owned by https://gitlab.wikimedia.org/pe... [00:09:54] 10Release-Engineering-Team (Priority Backlog 📥), 10GitLab (Auth & Access): Create a top level wmde group on Gitlab - https://phabricator.wikimedia.org/T291388 (10thcipriani) 05Stalled→03In progress [00:23:17] 10Release-Engineering-Team (Doing), 10Security Team AppSec, 10Security-Team, 10GitLab (CI & Job Runners), and 3 others: Migrate existing proof-of-concept node ci templates to slim node wm node docker images - https://phabricator.wikimedia.org/T294306 (10thcipriani) 05Open→03Invalid Sorry for the re-ope... [00:38:21] PROBLEM - SSH on contint1001.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [01:39:20] RECOVERY - SSH on contint1001.mgmt is OK: SSH OK - OpenSSH_6.6 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [01:46:26] (03PS1) 10Reedy: Enable extension-quibble for QuickInstantCommons [integration/config] - 10https://gerrit.wikimedia.org/r/739037 [01:48:14] (03CR) 10Reedy: [C: 03+2] Enable extension-quibble for QuickInstantCommons [integration/config] - 10https://gerrit.wikimedia.org/r/739037 (owner: 10Reedy) [01:50:14] (03Merged) 10jenkins-bot: Enable extension-quibble for QuickInstantCommons [integration/config] - 10https://gerrit.wikimedia.org/r/739037 (owner: 10Reedy) [01:50:51] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/739037 [01:50:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [01:59:49] (03PS1) 10Ahmon Dancy: sync_master: Use same CDB rebuild technique as main deploy server [tools/scap] - 10https://gerrit.wikimedia.org/r/739040 (https://phabricator.wikimedia.org/T295304) [02:00:39] (03CR) 10jerkins-bot: [V: 04-1] sync_master: Use same CDB rebuild technique as main deploy server [tools/scap] - 10https://gerrit.wikimedia.org/r/739040 (https://phabricator.wikimedia.org/T295304) (owner: 10Ahmon Dancy) [02:02:14] (03PS2) 10Ahmon Dancy: sync_master: Use same CDB rebuild technique as main deploy server [tools/scap] - 10https://gerrit.wikimedia.org/r/739040 (https://phabricator.wikimedia.org/T295304) [02:22:29] hi releng... in coming phab spam.... [06:40:36] train commit failed to merge https://gerrit.wikimedia.org/r/c/mediawiki/core/+/739043/ :/ [08:05:17] 10Project-Admins: Create a generic #SVG tag - https://phabricator.wikimedia.org/T287930 (10Aklapper) @Volker_E: Any idea when [not] to use #Wikimedia-SVG-rendering instead? [08:07:05] 10Project-Admins: Archive Tool-Database-Queries project - https://phabricator.wikimedia.org/T107699 (10Aklapper) 05Open→03Resolved a:03Aklapper No replies; seems to be unmaintained, plus https://wikitech.wikimedia.org/wiki/Obsolete:Tool:Query_service is historical. Archived. [08:13:21] 10Project-Admins, 10MediaViewer, 10Structured-Data-Backlog, 10PM: Convert MediaViewer tracking tasks to Phab workboard columns; use board columns for categories only instead of progress tracking - https://phabricator.wikimedia.org/T282946 (10Aklapper) @MarkTraceur any opinion? [08:30:33] 10Project-Admins, 10User-dcaro: Create tag projects worktype-project, origin-user, origin-alert, origin-team - https://phabricator.wikimedia.org/T295692 (10dcaro) > If nothing is flagged: Yes. See https://www.mediawiki.org/wiki/Phabricator/Help#Personal_lists_of_items That might do the trick, it would be grea... [08:35:58] 10Project-Admins: Create tag for server-side upload requests - https://phabricator.wikimedia.org/T295231 (10Aklapper) Makes a lot of sense to me; just wondering how to find all the docs to update... :-/ [09:07:20] 10Project-Admins: Create tag for server-side upload requests - https://phabricator.wikimedia.org/T295231 (10Aklapper) Hmm, or maybe that's only https://commons.wikimedia.org/wiki/Help:Server-side_upload#Process ? [09:11:56] (03CR) 10Hashar: [C: 03+2] jjb: remove intermediate docker-ci-src-setup macro [integration/config] - 10https://gerrit.wikimedia.org/r/738917 (owner: 10Hashar) [09:12:14] (03CR) 10Hashar: [C: 03+2] Upgrade jjb to 3.11.0 [integration/config] - 10https://gerrit.wikimedia.org/r/738921 (owner: 10Hashar) [09:13:00] !log Updating CI jobs for Jenkins Jobs Builder upgrade to 3.11.0 [09:13:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:14:07] (03Merged) 10jenkins-bot: jjb: remove intermediate docker-ci-src-setup macro [integration/config] - 10https://gerrit.wikimedia.org/r/738917 (owner: 10Hashar) [09:14:09] (03Merged) 10jenkins-bot: Upgrade jjb to 3.11.0 [integration/config] - 10https://gerrit.wikimedia.org/r/738921 (owner: 10Hashar) [09:19:07] 10Project-Admins, 10User-dcaro: Create tag projects worktype-project, origin-user, origin-alert, origin-team - https://phabricator.wikimedia.org/T295692 (10Aklapper) >>! In T295692#7505884, @dcaro wrote: >> If nothing is flagged: Yes. See https://www.mediawiki.org/wiki/Phabricator/Help#Personal_lists_of_items... [09:22:39] 10Project-Admins: Create #Croatian-Sites tag - https://phabricator.wikimedia.org/T291751 (10Aklapper) 05Open→03Resolved a:03Aklapper Requested public project #Croatian-Sites has been created: https://phabricator.wikimedia.org/project/view/5624/ Interested people are welcome to join the project as {icon us... [09:28:10] 10Phabricator, 10Project-Admins: Unarchive WMUA-tech project and create a custom security policy for its members - https://phabricator.wikimedia.org/T286866 (10Aklapper) I obviously don't know how to proceed here. :( [09:38:42] 10Project-Admins, 10User-dcaro: Create tag projects worktype-project, origin-user, origin-alert, origin-team - https://phabricator.wikimedia.org/T295692 (10dcaro) > though I'd still prefer some individual name prefix like user-dcaro-something in that case For the record, I'm ok with that too [09:44:44] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: beta-build-scap-deb failing - https://phabricator.wikimedia.org/T295719 (10hashar) [09:45:54] 10Project-Admins, 10Wikimedia-Small-Projects-User-Group: Create project tags for WikiSP-Projects and WikiSP-Milestone - https://phabricator.wikimedia.org/T293170 (10Aklapper) 05Open→03Resolved a:03Aklapper Requested public projects have been created: #Wikimedia-Small-Projects-User-Group-Projects (with `#... [09:52:10] (03PS1) 10Hashar: jjb: beta-build-scap-deb should capture any Lintian files [integration/config] - 10https://gerrit.wikimedia.org/r/739117 (https://phabricator.wikimedia.org/T295719) [10:11:17] (03CR) 10Hashar: "That is to get the XML reports attached to the builds :)" [integration/config] - 10https://gerrit.wikimedia.org/r/739117 (https://phabricator.wikimedia.org/T295719) (owner: 10Hashar) [10:16:41] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Patch-For-Review: beta-build-scap-deb failing - https://phabricator.wikimedia.org/T295719 (10hashar) Thanks for the rollback @dancy! That is indeed due to the Xunit plugin ( https://plugins.jenkins.io/xunit/ ). We have upgraded from vers... [10:19:30] hey, any clues on why https://integration.wikimedia.org/ci/job/debian-glue/2096/console is failing? [10:35:22] hashar maybe? I see T295719, I am having a similar issue on tools-webservice https://integration.wikimedia.org/ci/job/debian-glue/2096/console :/ [10:35:22] T295719: beta-build-scap-deb failing - https://phabricator.wikimedia.org/T295719 [11:09:03] 10Release-Engineering-Team (Seen), 10Scap, 10SRE: find a way to systematically update the deployment server name across all repos - https://phabricator.wikimedia.org/T197470 (10hnowlan) This happens because of how DEPLOY_HEAD retains the last-used deploy server name and unless explicitly told to ignore, it w... [11:16:27] 10Project-Admins, 10Croatian-Sites: Create #Croatian-Sites tag - https://phabricator.wikimedia.org/T291751 (10FriedrickMILBarbarossa) [11:29:02] majavah: ah yeah I broke it https://phabricator.wikimedia.org/T295719 :( [11:29:40] (03CR) 10Hashar: [C: 03+2] jjb: beta-build-scap-deb should capture any Lintian files [integration/config] - 10https://gerrit.wikimedia.org/r/739117 (https://phabricator.wikimedia.org/T295719) (owner: 10Hashar) [11:31:18] (03PS1) 10Hashar: jjb: temporarily disable Junit for debian-glue jobs [integration/config] - 10https://gerrit.wikimedia.org/r/739138 (https://phabricator.wikimedia.org/T295719) [11:31:39] (03Merged) 10jenkins-bot: jjb: beta-build-scap-deb should capture any Lintian files [integration/config] - 10https://gerrit.wikimedia.org/r/739117 (https://phabricator.wikimedia.org/T295719) (owner: 10Hashar) [11:32:44] (03CR) 10Hashar: [C: 03+2] "I have updated all the debian glue jos" [integration/config] - 10https://gerrit.wikimedia.org/r/739138 (https://phabricator.wikimedia.org/T295719) (owner: 10Hashar) [11:33:13] majavah: done [11:33:19] thanks! [11:33:24] though I think the job will no more fail when a lintian report has an error :( [11:33:27] I have to figure out a hack [11:34:34] (03Merged) 10jenkins-bot: jjb: temporarily disable Junit for debian-glue jobs [integration/config] - 10https://gerrit.wikimedia.org/r/739138 (https://phabricator.wikimedia.org/T295719) (owner: 10Hashar) [11:47:20] 10Project-Admins, 10Wikimedia-Small-Projects-User-Group: Create project tags for WikiSP-Projects and WikiSP-Milestone - https://phabricator.wikimedia.org/T293170 (10Galahad) @Aklapper thanks! Can you give permissions for create sub-projects and milestones on the projects? Regards. [11:48:42] 10Project-Admins, 10Wikimedia-Small-Projects-User-Group: Create project tags for WikiSP-Projects and WikiSP-Milestone - https://phabricator.wikimedia.org/T293170 (10Aklapper) Please see https://www.mediawiki.org/wiki/Phabricator/Creating_and_renaming_projects#Creating_new_projects [11:53:14] 10Project-Admins, 10Wikimedia-Small-Projects-User-Group: Create project tags for WikiSP-Projects and WikiSP-Milestone - https://phabricator.wikimedia.org/T293170 (10Galahad) Just for no bother you in creating new sub-projects. But it's okay if I'll create a new task for each sub-project/milestone :) Can you c... [11:55:56] 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10Galahad) Hi, I need to create sub-projects under #wikimedia-small-projects-user-group-milestone and #wikisp-projects from time to time. Please add me to the project admins group, tha... [12:19:26] (03PS1) 10Hashar: jjb: allow PATH override for lintian-junit-report [integration/config] - 10https://gerrit.wikimedia.org/r/739219 (https://phabricator.wikimedia.org/T295719) [12:19:29] (03PS1) 10Hashar: jjb: restore lintian Junit processing [integration/config] - 10https://gerrit.wikimedia.org/r/739220 (https://phabricator.wikimedia.org/T295719) [12:20:00] (03CR) 10Hashar: "I have updated the beta-build-scap-deb job and it passes! https://integration.wikimedia.org/ci/view/Beta/job/beta-build-scap-deb/91/consol" [integration/config] - 10https://gerrit.wikimedia.org/r/739220 (https://phabricator.wikimedia.org/T295719) (owner: 10Hashar) [12:20:31] (03CR) 10Hashar: "I have cherry picked the Puppet patch on integration-puppetmaster02 which patches the lintian-junit-report tool." [integration/config] - 10https://gerrit.wikimedia.org/r/739219 (https://phabricator.wikimedia.org/T295719) (owner: 10Hashar) [12:21:13] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Done by Wed 24 Nov 🔥), 10Patch-For-Review: beta-build-scap-deb failing - https://phabricator.wikimedia.org/T295719 (10hashar) a:03hashar [14:03:23] 10Project-Admins, 10Esperanto-Sites, 10User-Luke081515: Create "Esperanto-Sites" project to track Esperanto usage within Wikimedia projects - https://phabricator.wikimedia.org/T106974 (10FriedrickMILBarbarossa) [15:37:58] 10Phabricator, 10Developer-Advocacy, 10Epic: (Semi)automatically close Phabricator tickets with status "stalled" after a while - https://phabricator.wikimedia.org/T252522 (10Aklapper) a:05Aklapper→03None * Related upstream: https://secure.phabricator.com/T6491 : Herald rules based on age (hence not possi... [15:49:35] (03CR) 10Ahmon Dancy: [C: 03+2] jjb: allow PATH override for lintian-junit-report [integration/config] - 10https://gerrit.wikimedia.org/r/739219 (https://phabricator.wikimedia.org/T295719) (owner: 10Hashar) [15:49:55] (03CR) 10Ahmon Dancy: [C: 03+2] jjb: restore lintian Junit processing [integration/config] - 10https://gerrit.wikimedia.org/r/739220 (https://phabricator.wikimedia.org/T295719) (owner: 10Hashar) [15:52:17] (03Merged) 10jenkins-bot: jjb: allow PATH override for lintian-junit-report [integration/config] - 10https://gerrit.wikimedia.org/r/739219 (https://phabricator.wikimedia.org/T295719) (owner: 10Hashar) [15:52:27] (03Merged) 10jenkins-bot: jjb: restore lintian Junit processing [integration/config] - 10https://gerrit.wikimedia.org/r/739220 (https://phabricator.wikimedia.org/T295719) (owner: 10Hashar) [15:54:56] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Done by Wed 24 Nov 🔥), 10Patch-For-Review: beta-build-scap-deb failing - https://phabricator.wikimedia.org/T295719 (10dancy) Thanks for the hacking @hashar ! [16:39:02] 10Release-Engineering-Team (Radar), 10Observability-Logging, 10GitLab (Infrastructure), 10Patch-For-Review: Gitlab Sidekiq mapper parsing exceptions since 2021-11-15@1825 - https://phabricator.wikimedia.org/T295731 (10colewhite) 05Open→03Resolved a:03colewhite Reformatting the logs has eliminated the... [16:41:40] !log Cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/737970 again in deployment-prep, should only affect deployment-webperf11. [16:41:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:57:08] 10Project-Admins: Create tag for server-side upload requests - https://phabricator.wikimedia.org/T295231 (10Legoktm) I believe that link and the tasks video2commons creates are the main ways people file these tasks. Certainly at first we can rely on people who triage #wikimedia-site-requests to tag these tasks a... [17:02:07] 10Release-Engineering-Team (Seen), 10Scap, 10SRE: find a way to systematically update the deployment server name across all repos - https://phabricator.wikimedia.org/T197470 (10dancy) >>! In T197470#7506161, @hnowlan wrote: > For the immediate term if there are no objections I will replace all instances of `... [17:02:40] 10Release-Engineering-Team (Priority Backlog 📥), 10Scap, 10SRE: find a way to systematically update the deployment server name across all repos - https://phabricator.wikimedia.org/T197470 (10dancy) [17:06:03] (03PS1) 10Hashar: jjb: play with Jinja2 [integration/config] - 10https://gerrit.wikimedia.org/r/739282 [17:09:32] 10Continuous-Integration-Infrastructure, 10Performance-Team: Provide one or more Qemu agents in CI that use a newer version than 2.x - https://phabricator.wikimedia.org/T284774 (10dpifke) I've dropped a bullseye virtual machine image in my home directory on integration-agent-qemu-1003 and updated the instructi... [17:21:40] 10Release-Engineering-Team (Done by Wed 24 Nov 🔥), 10Release, 10Train Deployments: 1.38.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T293950 (10nray) [17:23:20] jeena: you need to +2 https://gerrit.wikimedia.org/r/c/mediawiki/core/+/739043/ instead of just commenting "recheck" to actually merge it [17:23:33] just did, thanks [17:31:59] !log Added deployment-docker-wikifunctions01 to deployment-prep for T289315 [17:32:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:32:02] T289315: Work out how we're going to have "production-like" versions of the wikifunctions evaluator and orchestrator services in Beta Cluster - https://phabricator.wikimedia.org/T289315 [17:34:44] that.. doesn't seem like a kubernetes node :/ [17:39:36] 10Project-Admins, 10User-Urbanecm: Create tag for server-side upload requests - https://phabricator.wikimedia.org/T295231 (10Urbanecm) 05Open→03Resolved a:03Urbanecm I think we're all in agreement here. Created as #server-side-upload-request. Going to update the documentation as well. [18:43:10] 10Project-Admins, 10User-dcaro: Create tag projects worktype-project, origin-user, origin-alert, origin-team - https://phabricator.wikimedia.org/T295692 (10MBinder_WMF) Fair warning: I've only really thought about this ticket at a high-level, and I am sure there are both more specific and more holistic approac... [18:44:31] 10Project-Admins, 10User-Urbanecm: Create tag for server-side upload requests - https://phabricator.wikimedia.org/T295231 (10Majavah) Is the new tag intended to be a fully separate process from normal site requests (in which case it should be a component) or an addition to tickets already tagged with #wikimedi... [18:46:35] 10Project-Admins, 10User-Urbanecm: Create tag for server-side upload requests - https://phabricator.wikimedia.org/T295231 (10Urbanecm) >>! In T295231#7507684, @Majavah wrote: > Is the new tag intended to be a fully separate process from normal site requests (in which case it should be a component) or an additi... [18:55:08] 10Project-Admins, 10User-Urbanecm: Create tag for server-side upload requests - https://phabricator.wikimedia.org/T295231 (10Legoktm) My request for a tag was so we could make a minimal change that helps surface these tickets better for hopefully better processing. I originally had some ideas about moving this... [19:14:29] 10Release-Engineering-Team (Done by Wed 24 Nov 🔥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T293950 (10jeena) 05Open→03In progress [19:24:15] 10Release-Engineering-Team (Done by Wed 24 Nov 🔥), 10GitLab (CI & Job Runners), 10User-brennen: runner-1002 is out of space - https://phabricator.wikimedia.org/T291221 (10dancy) [19:24:28] 10Release-Engineering-Team (Done by Wed 24 Nov 🔥), 10GitLab (CI & Job Runners), 10User-brennen: runner-1002 is out of space - https://phabricator.wikimedia.org/T291221 (10dancy) [19:24:39] 10Release-Engineering-Team (Done by Wed 24 Nov 🔥), 10GitLab (CI & Job Runners), 10Patch-For-Review: Run docker-gc resource monitor on gitlab runners - https://phabricator.wikimedia.org/T295707 (10dancy) 05Open→03In progress p:05Triage→03Medium [19:24:48] 10Release-Engineering-Team (Done by Wed 24 Nov 🔥), 10GitLab (CI & Job Runners), 10Patch-For-Review: Periodically run docker-gc on gitlab runners - https://phabricator.wikimedia.org/T295709 (10dancy) 05Open→03In progress p:05Triage→03Medium [19:31:03] 10Release-Engineering-Team (Done by Wed 24 Nov 🔥), 10GitLab (Auth & Access), 10Upstream, 10User-brennen: Gitlab 2fa password validation seems bugged - https://phabricator.wikimedia.org/T292431 (10brennen) 05In progress→03Resolved We're on GitLab 14.4.2 as of yesterday. This should be resolved, and I c... [19:35:34] 10Release-Engineering-Team (Done by Wed 24 Nov 🔥), 10GitLab (CI & Job Runners), 10Patch-For-Review: Periodically run docker-gc on gitlab runners - https://phabricator.wikimedia.org/T295709 (10dancy) a:03dancy [20:17:44] PROBLEM - SSH on contint1001.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [20:37:15] (03CR) 10Thcipriani: [C: 03+2] "TIL: jq Alternative Operator" [tools/release] - 10https://gerrit.wikimedia.org/r/737966 (owner: 10Ahmon Dancy) [20:39:11] (03Merged) 10jenkins-bot: deploy-promote get_train_task() Handle null task_id [tools/release] - 10https://gerrit.wikimedia.org/r/737966 (owner: 10Ahmon Dancy) [20:45:38] (03CR) 10Thcipriani: [C: 03+2] "Hrm, thought there was a bug for this, but couldn't find it in phab: either way, output looks much nicer." [tools/scap] - 10https://gerrit.wikimedia.org/r/738435 (owner: 10Ahmon Dancy) [20:46:23] (03Merged) 10jenkins-bot: Improve _check_fatals error reporting [tools/scap] - 10https://gerrit.wikimedia.org/r/738435 (owner: 10Ahmon Dancy) [21:06:09] (03PS1) 10Hashar: jjb: avoid chaining for docker run macros [integration/config] - 10https://gerrit.wikimedia.org/r/739350 [21:18:44] RECOVERY - SSH on contint1001.mgmt is OK: SSH OK - OpenSSH_6.6 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [22:36:48] 10MediaWiki-Releasing, 10MW-1.37-notes, 10MW-1.37-release, 10Patch-For-Review: Release MW 1.37.0 - https://phabricator.wikimedia.org/T289585 (10Reedy) [22:39:19] 10MediaWiki-Releasing, 10MW-1.37-notes, 10MW-1.37-release: Write and send release announcement for 1.37.0 - https://phabricator.wikimedia.org/T289594 (10Reedy) 05Open→03In progress a:03Reedy [22:39:22] 10MediaWiki-Releasing, 10MW-1.37-notes, 10MW-1.37-release, 10Patch-For-Review: Release MW 1.37.0 - https://phabricator.wikimedia.org/T289585 (10Reedy) [22:39:30] 10MediaWiki-Releasing, 10MW-1.37-notes, 10MW-1.37-release, 10Patch-For-Review: Release MW 1.37.0 - https://phabricator.wikimedia.org/T289585 (10Reedy) 05Open→03In progress [22:39:43] 10MediaWiki-Releasing, 10MW-1.37-notes, 10MW-1.37-release, 10Patch-For-Review: Release MW 1.37.0 - https://phabricator.wikimedia.org/T289585 (10Reedy) [22:50:40] 10MediaWiki-Releasing, 10MW-1.37-notes, 10MW-1.37-release: Write and send release announcement for 1.37.0 - https://phabricator.wikimedia.org/T289594 (10Reedy)