[00:15:33] (03CR) 10Jeena Huneidi: [C: 03+2] "LGTM" [integration/config] - 10https://gerrit.wikimedia.org/r/739908 (https://phabricator.wikimedia.org/T295994) (owner: 10Nikki Nikkhoui) [00:17:31] (03Merged) 10jenkins-bot: jjb, Zuul: [mediawiki/services/servicelib-node/spec] add test pipeline [integration/config] - 10https://gerrit.wikimedia.org/r/739908 (https://phabricator.wikimedia.org/T295994) (owner: 10Nikki Nikkhoui) [00:51:56] RECOVERY - SSH on contint1001.mgmt is OK: SSH OK - OpenSSH_6.6 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [01:19:43] 10Release-Engineering-Team (Done by Wed 24 Nov πŸ”₯), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T293950 (10Zabe) [03:10:23] 10Release-Engineering-Team (Done by Wed 24 Nov πŸ”₯), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T293950 (10Ladsgroup) [03:10:57] 10Release-Engineering-Team (Done by Wed 24 Nov πŸ”₯), 10Release, 10Train Deployments, 10User-Ladsgroup: 1.38.0-wmf.9 seems to have introduced a memory leak - https://phabricator.wikimedia.org/T296098 (10Ladsgroup) 05Openβ†’03Resolved a:03Ladsgroup >>! In T296098#7520999, @akosiaris wrote: > For what is wo... [03:17:59] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Done by Thu 04 Nov 🧟), 10Security-Team, 10User-brennen: Enable runners for projects under gitlab.wikimedia.org security group - https://phabricator.wikimedia.org/T294050 (10sbassett) >>! In T294050#7522134, @brennen wrote: > What project is this for?... [03:18:40] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Done by Thu 04 Nov 🧟), 10Security-Team, 10User-brennen: Enable runners for projects under gitlab.wikimedia.org security group - https://phabricator.wikimedia.org/T294050 (10sbassett) 05Openβ†’03Resolved [07:33:37] 10Release-Engineering-Team (Done by Wed 24 Nov πŸ”₯), 10Release, 10Train Deployments, 10User-Ladsgroup: 1.38.0-wmf.9 seems to have introduced a memory leak - https://phabricator.wikimedia.org/T296098 (10akosiaris) >>! In T296098#7522524, @Ladsgroup wrote: >>>! In T296098#7520999, @akosiaris wrote: >> For what... [08:08:37] 10Release-Engineering-Team (Done by Wed 24 Nov πŸ”₯), 10Patch-For-Review, 10Release, 10Train Deployments: 1.38.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T293950 (10hashar) 05In progressβ†’03Resolved The memory leak (T296098) does not appear anymore, it was the last step to claim this t... [09:26:02] 10MediaWiki-Releasing: Verify MediaWiki patch files apply in successive upgrades - https://phabricator.wikimedia.org/T263883 (10Legoktm) a:05Legoktmβ†’03None Un-cookie licking [10:47:10] 10Project-Admins, 10User-dcaro: Create tag projects worktype-project, origin-user, origin-alert, origin-team - https://phabricator.wikimedia.org/T295692 (10dcaro) 05Openβ†’03Resolved a:03dcaro [11:01:15] PROBLEM - SSH on contint1001.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [12:35:14] !log canceled one beta-scap-sync-world build, seemingly stuck since yesterday evening [12:35:16] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:36:12] hm, https://integration.wikimedia.org/ci/view/Beta/job/beta-mediawiki-config-update-eqiad/20030/console has a weird error [12:36:15] error: pack-objects died of signal 15 [12:36:19] fatal: failed to run repack [12:37:11] but beta-code-update-eqiad seems to be running at the moment, so let’s wait a bit and see if that works itself out [12:37:22] (I don’t really understand how the various beta jenkins jobs interact) [12:37:51] beta-code-update-eqiad updates core, extensions and skins but not config [12:37:58] beta-mediawiki-config-update-eqiad updates config [12:38:17] and then beta-scap-sync-world pushes both from deployment-deploy01 to all other servers [12:39:10] alright [12:39:16] looks like a new beta-scap-sync-world is running now [12:40:10] yeah, but if the mediawiki-config-update job failed, it does not have the changes you want it to have [12:42:43] looks like those git messages have been in the output for a while (e.g. https://integration.wikimedia.org/ci/view/Beta/job/beta-mediawiki-config-update-eqiad/20001/console) [12:42:55] let me ssh in and see what the git status looks like [12:43:58] git status on /srv/mediawiki-staging looks fine and has the latest change, at least [12:44:35] and /srv/mediawiki/wmf-config/InitialiseSettings-labs.php also has it [13:03:19] RECOVERY - SSH on contint1001.mgmt is OK: SSH OK - OpenSSH_6.6 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [13:55:07] 10Continuous-Integration-Config, 10Release-Engineering-Team (Doing), 10Patch-For-Review: Add a CI Job for running bundlesize tests in Vector - https://phabricator.wikimedia.org/T255149 (10kostajh) a:03kostajh [13:56:17] 10Continuous-Integration-Config, 10Release-Engineering-Team (Doing), 10Patch-For-Review: Move bundlesize test from npm script to MediaWikiIntegrationTest - https://phabricator.wikimedia.org/T255149 (10kostajh) [15:50:48] 10Beta-Cluster-Infrastructure, 10Abstract Wikipedia team, 10Patch-For-Review: Create a Beta Cluster version of Wikifunctions.org - https://phabricator.wikimedia.org/T284162 (10Jdforrester-WMF) 05Openβ†’03In progress a:03Jdforrester-WMF [15:57:24] 10Release-Engineering-Team (Doing), 10MediaWiki-Core-Tests, 10Patch-For-Review: Move bundlesize test from npm script to MediaWikiIntegrationTest - https://phabricator.wikimedia.org/T255149 (10kostajh) [15:58:13] 10Release-Engineering-Team (Doing), 10MediaWiki-Core-Tests, 10MediaWiki-ResourceLoader, 10Performance-Team, 10Patch-For-Review: Move bundlesize test from npm script to MediaWikiIntegrationTest - https://phabricator.wikimedia.org/T255149 (10kostajh) [16:24:24] 10Beta-Cluster-Infrastructure, 10MediaWiki-libs-HTTP, 10Beta-Cluster-reproducible: Exception: Invalid reverseProxy configured - https://phabricator.wikimedia.org/T296312 (10dom_walden) [16:58:19] 10Beta-Cluster-Infrastructure, 10MediaWiki-libs-HTTP, 10Beta-Cluster-reproducible: Exception: Invalid reverseProxy configured - https://phabricator.wikimedia.org/T296312 (10Joe) @Legoktm this seems related to your patches? [17:23:20] 10Beta-Cluster-Infrastructure, 10MediaWiki-libs-HTTP, 10Beta-Cluster-reproducible: Exception: Invalid reverseProxy configured - https://phabricator.wikimedia.org/T296312 (10Legoktm) a:03Legoktm Uh, yes. Looking at it now. [17:57:50] (03PS1) 10Nikki Nikkhoui: Zuul:[mediawiki/services/servicelib-node/spec] trigger on /spec changes [integration/config] - 10https://gerrit.wikimedia.org/r/740908 (https://phabricator.wikimedia.org/T295994) [18:30:01] (03PS1) 10Umherirrender: Zuul: [extensions/WikiEditor] Add ConfirmEdit as phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/740916 (https://phabricator.wikimedia.org/T296287) [18:35:26] (03PS2) 10Nikki Nikkhoui: Zuul:[mediawiki/services/servicelib-node/spec] trigger on /spec changes [integration/config] - 10https://gerrit.wikimedia.org/r/740908 (https://phabricator.wikimedia.org/T295994) [18:36:10] (03PS3) 10Nikki Nikkhoui: Zuul:[mediawiki/services/servicelib-node/spec] trigger on /spec changes [integration/config] - 10https://gerrit.wikimedia.org/r/740908 (https://phabricator.wikimedia.org/T295994) [18:46:25] (03CR) 10Jeena Huneidi: [C: 03+2] "LGTM" [integration/config] - 10https://gerrit.wikimedia.org/r/740908 (https://phabricator.wikimedia.org/T295994) (owner: 10Nikki Nikkhoui) [18:47:20] (03CR) 10Zabe: [C: 03+1] "Thanks" [integration/config] - 10https://gerrit.wikimedia.org/r/740916 (https://phabricator.wikimedia.org/T296287) (owner: 10Umherirrender) [18:48:30] (03Merged) 10jenkins-bot: Zuul:[mediawiki/services/servicelib-node/spec] trigger on /spec changes [integration/config] - 10https://gerrit.wikimedia.org/r/740908 (https://phabricator.wikimedia.org/T295994) (owner: 10Nikki Nikkhoui) [18:50:02] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/740908 [18:50:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:58:33] 10Continuous-Integration-Config, 10MW-on-K8s, 10MediaWiki-SettingsLoader: Install php-yaml for use by SettingsLoader - https://phabricator.wikimedia.org/T296331 (10dduvall) [19:59:03] 10Continuous-Integration-Config, 10MW-on-K8s, 10MediaWiki-SettingsLoader: Install php-yaml for use by SettingsLoader - https://phabricator.wikimedia.org/T296331 (10dduvall) 05Openβ†’03In progress p:05Triageβ†’03Medium [20:18:47] (03PS1) 10Dduvall: dockerfiles: Install php-yaml everywhere [integration/config] - 10https://gerrit.wikimedia.org/r/740928 (https://phabricator.wikimedia.org/T296331) [20:28:15] (03CR) 10Ppchelko: [C: 03+1] "+1 to the purpose of this. I don't know any details about how all this works, so can't comment on it." [integration/config] - 10https://gerrit.wikimedia.org/r/740928 (https://phabricator.wikimedia.org/T296331) (owner: 10Dduvall) [20:30:22] 10Continuous-Integration-Config, 10MW-on-K8s, 10MediaWiki-SettingsLoader, 10Patch-For-Review: Install php-yaml for use by SettingsLoader - https://phabricator.wikimedia.org/T296331 (10Majavah) Will 3rd party MW installations require php-yaml in the future? [20:30:55] 10Continuous-Integration-Config, 10MW-on-K8s, 10MediaWiki-SettingsLoader, 10Patch-For-Review: Install php-yaml for use by SettingsLoader - https://phabricator.wikimedia.org/T296331 (10Pchelolo) >>! In T296331#7524691, @Majavah wrote: > Will 3rd party MW installations require php-yaml in the future? It's r... [20:33:22] 10Continuous-Integration-Config, 10MW-on-K8s, 10MediaWiki-SettingsLoader, 10serviceops-radar, 10Patch-For-Review: Install php-yaml for use by SettingsLoader - https://phabricator.wikimedia.org/T296331 (10Dzahn) [20:41:19] 10Beta-Cluster-Infrastructure, 10Analytics, 10Beta-Cluster-reproducible, 10User-Urbanecm: 502, connect failed for intake-analytics.wikimedia.beta.wmflabs.org - https://phabricator.wikimedia.org/T289029 (10AlexisJazz) 05Resolvedβ†’03Open It's broken again. [21:36:13] (03PS1) 10Krinkle: doc: Add Codex, wikimedia-ui-base, less.php [integration/docroot] - 10https://gerrit.wikimedia.org/r/740937 [21:36:47] (03CR) 10Krinkle: [C: 03+2] doc: Add Codex, wikimedia-ui-base, less.php [integration/docroot] - 10https://gerrit.wikimedia.org/r/740937 (owner: 10Krinkle) [21:37:26] (03Merged) 10jenkins-bot: doc: Add Codex, wikimedia-ui-base, less.php [integration/docroot] - 10https://gerrit.wikimedia.org/r/740937 (owner: 10Krinkle) [22:04:05] (03CR) 10Ksdev: [C: 03+1] "This change is ready for review." [libs/ObjectFactory] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/740874 (owner: 10Ksdev) [22:08:34] (03CR) 10Legoktm: "Hi Ksdev, can you explain what you're trying to do?" [libs/ObjectFactory] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/740874 (owner: 10Ksdev) [22:42:19] (03CR) 10VolkerE: "Thanks for adding those. Late comments inside." [integration/docroot] - 10https://gerrit.wikimedia.org/r/740937 (owner: 10Krinkle) [22:44:20] (03CR) 10Hashar: "In order for docker-pkg to build the images, they must have the version bumped in each changelog files (dch -i -c dockerfiles/php72/change" [integration/config] - 10https://gerrit.wikimedia.org/r/740928 (https://phabricator.wikimedia.org/T296331) (owner: 10Dduvall) [22:48:17] 10Continuous-Integration-Config, 10MW-on-K8s, 10MediaWiki-SettingsLoader, 10serviceops-radar, 10Patch-For-Review: Install php-yaml for use by SettingsLoader - https://phabricator.wikimedia.org/T296331 (10Legoktm) We will need to upload packages of php-yaml to our php72 and php74 sections on apt.wm.o. Whe... [22:48:50] !log deployment-prep: reboot deployment-eventgate-3 (T289029) [22:48:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:48:53] T289029: 502, connect failed for intake-analytics.wikimedia.beta.wmflabs.org - https://phabricator.wikimedia.org/T289029 [22:50:06] 10Beta-Cluster-Infrastructure, 10Analytics, 10Beta-Cluster-reproducible, 10User-Urbanecm: 502, connect failed for intake-analytics.wikimedia.beta.wmflabs.org - https://phabricator.wikimedia.org/T289029 (10Urbanecm) 05Openβ†’03Resolved Did the same thing as before, works again now. [23:35:59] 10GitLab (Integrations), 10GitLab-Test, 10Phabricator, 10Release-Engineering-Team (Next), and 2 others: Experiment with GitLab-Phabricator integration - https://phabricator.wikimedia.org/T265617 (10Dzahn) [23:42:13] 10Gerrit, 10LibUp, 10MediaWiki-Gerrit-Group-Requests: Grant libraryupgrader +2 rights for some library bumps - https://phabricator.wikimedia.org/T174760 (10Zabe)