[07:22:52] (03CR) 10Nikerabbit: "Can we add templatedata and pagelist too?" [integration/config] - 10https://gerrit.wikimedia.org/r/757946 (https://phabricator.wikimedia.org/T222216) (owner: 10SBassett) [10:07:28] (03CR) 10Thiemo Kreuz (WMDE): [C: 03+1] Switch QUnit tests to use the apache backend [integration/config] - 10https://gerrit.wikimedia.org/r/757393 (https://phabricator.wikimedia.org/T299491) (owner: 10Awight) [10:11:15] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10User-brennen: 'No such file or directory' CI failures in multiple repos - https://phabricator.wikimedia.org/T300214 (10hashar) 05Open→03Resolved The summary of the issue is that we had several Jenkins agents connected to the same inst... [10:11:17] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen): Move all Wikimedia CI (WMCS integration project) instances from stretch to buster/bullseye - https://phabricator.wikimedia.org/T252071 (10hashar) [10:49:02] !log Added integration-agent-qemu-1003 with label `Qemu` # T284774 [10:49:04] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:49:04] T284774: Provide one or more Qemu agents in CI that use a newer version than 2.x - https://phabricator.wikimedia.org/T284774 [10:50:31] 10Release-Engineering-Team, 10Scap, 10serviceops: Deploy Scap version 4.2.2 - https://phabricator.wikimedia.org/T300392 (10Jelto) p:05Triage→03Medium a:03Jelto [11:14:29] 10Continuous-Integration-Infrastructure, 10Performance-Team: Provide one or more Qemu agents in CI that use a newer version than 2.x - https://phabricator.wikimedia.org/T284774 (10hashar) Looks like I have nuked `/srv` on `integration-agent-qemu-1003` and all the steps done by @dpifke in November have been los... [12:40:03] (03CR) 10Awight: [C: 03+1] dockerfiles: Install memcached and php-memcached for Quibble (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/757867 (https://phabricator.wikimedia.org/T300340) (owner: 10Kosta Harlan) [12:41:39] (03PS4) 10Kosta Harlan: dockerfiles: Install memcached and php-memcached for Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/757867 (https://phabricator.wikimedia.org/T300340) [12:41:42] (03CR) 10Kosta Harlan: dockerfiles: Install memcached and php-memcached for Quibble (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/757867 (https://phabricator.wikimedia.org/T300340) (owner: 10Kosta Harlan) [12:42:24] (03PS5) 10Kosta Harlan: dockerfiles: Install memcached and php-memcached for Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/757867 (https://phabricator.wikimedia.org/T300340) [12:42:42] (03CR) 10Awight: [C: 03+1] dockerfiles: Install memcached and php-memcached for Quibble [integration/config] - 10https://gerrit.wikimedia.org/r/757867 (https://phabricator.wikimedia.org/T300340) (owner: 10Kosta Harlan) [13:10:18] 10Beta-Cluster-Infrastructure, 10SRE, 10Wikidata, 10serviceops, and 2 others: Run mediawiki::maintenance scripts in Beta Cluster - https://phabricator.wikimedia.org/T125976 (10Ladsgroup) [13:36:50] 10Scap, 10Icinga, 10Observability-Alerting, 10SRE, 10observability: expose hosts in maintenance state so we can prevent scap from running on them - https://phabricator.wikimedia.org/T100777 (10lmata) [15:06:30] 10Beta-Cluster-Infrastructure, 10Traffic, 10Beta-Cluster-reproducible: Beta cluster down: Error: 502, Next Hop Connection Failed - https://phabricator.wikimedia.org/T300525 (10AlexisJazz) [15:07:00] 10Beta-Cluster-Infrastructure, 10Traffic, 10Beta-Cluster-reproducible: Beta cluster down: Error: 502, Next Hop Connection Failed - https://phabricator.wikimedia.org/T300525 (10AlexisJazz) [15:13:02] 10Release-Engineering-Team, 10Scap: scap overrides for deploy-local using -D parameter fail - https://phabricator.wikimedia.org/T300177 (10hnowlan) I think the changes have resolved this issue, I provisioned a new host without issue. Thanks very much for the quick fix! [15:14:15] 10Release-Engineering-Team, 10Scap: scap overrides for deploy-local using -D parameter fail - https://phabricator.wikimedia.org/T300177 (10hnowlan) 05In progress→03Resolved [15:14:17] 10Release-Engineering-Team, 10Scap, 10serviceops: Deploy Scap version 4.2.2 - https://phabricator.wikimedia.org/T300392 (10hnowlan) [15:14:20] 10Release-Engineering-Team, 10Scap, 10serviceops: Deploy Scap version 4.2.1 - https://phabricator.wikimedia.org/T300058 (10hnowlan) [15:17:44] Can someone please look at https://phabricator.wikimedia.org/T300525 [15:17:59] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Traffic, 10Beta-Cluster-reproducible: Beta cluster down: Error: 502, Next Hop Connection Failed - https://phabricator.wikimedia.org/T300525 (10RhinosF1) [15:18:17] It's very slow to error [15:18:22] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Beta-Cluster-reproducible: Beta cluster down: Error: 502, Next Hop Connection Failed - https://phabricator.wikimedia.org/T300525 (10Majavah) [15:20:23] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Beta-Cluster-reproducible: Beta cluster down: Error: 502, Next Hop Connection Failed - https://phabricator.wikimedia.org/T300525 (10Vgutierrez) hmm both URLs are currently working for me [15:22:24] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Beta-Cluster-reproducible: Beta cluster down: Error: 502, Next Hop Connection Failed - https://phabricator.wikimedia.org/T300525 (10ema) p:05Triage→03Medium Apparently deployment-mediawiki11.deployment-prep.eqiad1.wikimedia.cloud is gone and ha... [15:27:04] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Beta-Cluster-reproducible: Beta cluster down: Error: 502, Next Hop Connection Failed - https://phabricator.wikimedia.org/T300525 (10RhinosF1) Change was https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/35f8c97bd92ad240f2d4a52f... [15:37:50] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Beta-Cluster-reproducible: Beta cluster down: Error: 502, Next Hop Connection Failed - https://phabricator.wikimedia.org/T300525 (10AlexisJazz) Seems to be working again, thanks. :-) [16:20:02] Project beta-update-databases-eqiad build #56316: 04FAILURE in 2.1 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/56316/ [16:24:43] 10Release-Engineering-Team: Should UI regressions (e.g. no fatals) with the non-default skins ever block the train? - https://phabricator.wikimedia.org/T300169 (10Ammarpad) >>! In T300169#7655746, @hashar wrote: > In the ideal world an audit should be made to find out which skins are still actively used (some us... [16:35:51] maintenance-disconnect-full-disks build 356322 integration-agent-qemu-1001 (/: 96%, /srv: 40%, /var/lib/docker: 1%): OFFLINE due to disk space [16:40:56] maintenance-disconnect-full-disks build 356323 integration-agent-qemu-1001 (/: 56%, /srv: 40%, /var/lib/docker: 1%): RECOVERY disk space OK [16:43:40] 10Beta-Cluster-Infrastructure, 10Elasticsearch, 10Discovery-Search (Current work): Upgrade deployment-prep Elastic cluster to Debian Buster or newer - https://phabricator.wikimedia.org/T298252 (10MPhamWMF) [16:45:48] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar), 10Discovery, 10Discovery-Search (Current work): Deploy new elastic cluster nodes on deployment-prep - https://phabricator.wikimedia.org/T299797 (10bking) [16:47:30] looks like I broke beta deploys [16:47:42] gj Reedy [16:47:43] :) [16:48:31] is that the "Error: your composer.lock file is not up to date. Run "composer update --no-dev" to install newer dependencies" issue? [16:49:02] bumps needed in mw core's composer.json to match vendor [16:52:58] https://gerrit.wikimedia.org/r/c/mediawiki/core/+/758519 [17:20:03] Project beta-update-databases-eqiad build #56317: 04STILL FAILING in 3.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/56317/ [17:27:01] 10Continuous-Integration-Infrastructure, 10Performance-Team, 10Patch-For-Review: Provide one or more Qemu agents in CI that use a newer version than 2.x - https://phabricator.wikimedia.org/T284774 (10hashar) I have spent most of my day on that, my conclusions: 1) **reinventing the wheel** * our doc as way... [17:53:52] Reedy: id be more concerned if you didn't break beta [18:30:16] Yippee, build fixed! [18:30:16] Project beta-update-databases-eqiad build #56318: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/56318/ [18:32:28] Reedy: ^ I'll give you a gj for that one too and without the snark ;) [18:35:01] 10Project-Admins: Mark the #Contributors-Team group as inactive - https://phabricator.wikimedia.org/T300558 (10Aklapper) I agree. According to https://phabricator.wikimedia.org/maniphest/query/H6rXIVSOUXP4/#R , that would be https://phabricator.wikimedia.org/maniphest/?ids=167899,115598,115597,112984,104863,9003... [18:53:47] (03CR) 10SBassett: Add more tags and attributes to i18n security checker allow list (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/757946 (https://phabricator.wikimedia.org/T222216) (owner: 10SBassett) [18:55:50] (03PS3) 10SBassett: Add more tags and attributes to i18n security checker allow list [integration/config] - 10https://gerrit.wikimedia.org/r/757946 (https://phabricator.wikimedia.org/T222216) [18:56:37] (03CR) 10Jforrester: [C: 03+1] "LGTM. Want this deployed?" [integration/config] - 10https://gerrit.wikimedia.org/r/757946 (https://phabricator.wikimedia.org/T222216) (owner: 10SBassett) [18:57:41] 10Deployments, 10Performance-Team, 10bacula, 10Sustainability (Incident Followup): Local private files on deployment host should be backed up somewhere - https://phabricator.wikimedia.org/T69818 (10Krinkle) 05Open→03Resolved Thanks @jcrespo. LGTM! [18:57:59] (03CR) 10SBassett: Add more tags and attributes to i18n security checker allow list (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/757946 (https://phabricator.wikimedia.org/T222216) (owner: 10SBassett) [19:00:46] (03PS4) 10Jforrester: jjb: [mediawiki-i18n-check-docker] Allow some more tags and attributes [integration/config] - 10https://gerrit.wikimedia.org/r/757946 (https://phabricator.wikimedia.org/T222216) (owner: 10SBassett) [19:01:26] !log Re-configured Jenkins job mediawiki-i18n-check-docker to 9e3ea96c548d7a84be763d38c2d118bc861cf189 for T222216 [19:01:27] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:01:33] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/757946 (https://phabricator.wikimedia.org/T222216) (owner: 10SBassett) [19:03:31] (03Merged) 10jenkins-bot: jjb: [mediawiki-i18n-check-docker] Allow some more tags and attributes [integration/config] - 10https://gerrit.wikimedia.org/r/757946 (https://phabricator.wikimedia.org/T222216) (owner: 10SBassett) [19:10:19] (03CR) 10SBassett: jjb: [mediawiki-i18n-check-docker] Allow some more tags and attributes (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/757946 (https://phabricator.wikimedia.org/T222216) (owner: 10SBassett) [19:15:10] Project beta-mediawiki-config-update-eqiad build #87: 04FAILURE in 19 sec: https://integration.wikimedia.org/ci/job/beta-mediawiki-config-update-eqiad/87/ [19:23:18] Yippee, build fixed! [19:23:19] Project beta-mediawiki-config-update-eqiad build #88: 09FIXED in 1 min 47 sec: https://integration.wikimedia.org/ci/job/beta-mediawiki-config-update-eqiad/88/ [19:39:29] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.38.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T293961 (10Jdlrobson) Not a risky patch, but please note this week that we'll be rolling out the new Vector 2022 skin via some configuration changes (T299927). If there are... [19:48:13] 10GitLab (Auth & Access): Create top level 'sre' group on Gitlab - https://phabricator.wikimedia.org/T298620 (10brennen) Created: https://gitlab.wikimedia.org/repos/sre Members of `people/wmf-team-sre` are owners. [19:48:22] 10GitLab (Auth & Access): Create top level 'sre' group on Gitlab - https://phabricator.wikimedia.org/T298620 (10brennen) 05Open→03Resolved a:03brennen [20:09:37] 10Beta-Cluster-Infrastructure: Give steward rights on Beta Cluster to AlPaD - https://phabricator.wikimedia.org/T300151 (10Urbanecm) I'm sorry, but I have to **oppose** at this time. Creating a Phabricator ticket asking for steward rights at beta is the first Phabricator activity from you and I was unable to fin... [20:14:11] 10Beta-Cluster-Infrastructure: Give steward rights on Beta Cluster to AlPaD - https://phabricator.wikimedia.org/T300151 (10AlPaD) Hello. Thank you for your reply. No problem, I withdrawn this request. Best regards! [20:14:36] 10Beta-Cluster-Infrastructure: Give steward rights on Beta Cluster to AlPaD - https://phabricator.wikimedia.org/T300151 (10AlPaD) 05Open→03Declined [21:19:24] 10GitLab (Project Migration), 10Release-Engineering-Team (Doing), 10User-brennen: Create new GitLab project group: Generated Data Platform - https://phabricator.wikimedia.org/T296381 (10brennen) 05Open→03In progress **Actions taken:** - Created: https://gitlab.wikimedia.org/repos/generated-data-platfo... [22:03:29] (03CR) 10Dduvall: [C: 03+2] feature: build-time arguments for lives & runs user config [blubber] - 10https://gerrit.wikimedia.org/r/749569 (https://phabricator.wikimedia.org/T296046) (owner: 10BryanDavis) [22:06:05] (03CR) 10jerkins-bot: [V: 04-1] feature: build-time arguments for lives & runs user config [blubber] - 10https://gerrit.wikimedia.org/r/749569 (https://phabricator.wikimedia.org/T296046) (owner: 10BryanDavis) [22:21:51] 10GitLab (Auth & Access): Create subgroup for 'wikisp' - https://phabricator.wikimedia.org/T296110 (10brennen) 05Open→03Resolved a:03brennen > The Wikimedia-Small-Projects-User-Group-TechCom was acl (only members of the group can push into the repo) could be the same? That should be covered by the default... [22:25:34] 10GitLab (Project Migration), 10Release-Engineering-Team (Done by Wed 24 Nov 🧟), 10User-brennen, 10cloud-services-team (Kanban): Create top level 'cloud' group on Gitlab - https://phabricator.wikimedia.org/T293741 (10brennen) [22:34:10] dduvall: the jerkins failure on that blubber patch looks weird. The only reference to "service-checker" in that whole chart is the templates/tests/test-service-checker.yaml that is blowing up. Is this the first merge since I9a28bcd0ff3549688028e446b1655707d2d6787f landed 3 days ago? [22:34:29] bd808: yeah, see https://phabricator.wikimedia.org/T276949#7665908 [22:36:10] * bd808 subscribes [22:41:42] (03PS1) 10Dduvall: Workaround `helm3 test --logs` bug by omitting logs [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/758571 (https://phabricator.wikimedia.org/T276949) [22:43:41] (03PS2) 10Dduvall: Workaround `helm3 test --logs` bug by omitting logs [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/758571 (https://phabricator.wikimedia.org/T276949) [22:44:11] 10GitLab (Project Migration), 10Release-Engineering-Team: Create new GitLab project group: research - https://phabricator.wikimedia.org/T300074 (10brennen) 05Open→03Resolved a:03brennen Verified @fab as WMF employee in LDAP, and created: - https://gitlab.wikimedia.org/people/wmf-team-research - http... [22:44:52] (03CR) 10Ahmon Dancy: [C: 03+2] Workaround `helm3 test --logs` bug by omitting logs [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/758571 (https://phabricator.wikimedia.org/T276949) (owner: 10Dduvall) [22:46:01] (03Merged) 10jenkins-bot: Workaround `helm3 test --logs` bug by omitting logs [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/758571 (https://phabricator.wikimedia.org/T276949) (owner: 10Dduvall) [22:48:47] (03CR) 10Dduvall: [C: 03+2] feature: build-time arguments for lives & runs user config [blubber] - 10https://gerrit.wikimedia.org/r/749569 (https://phabricator.wikimedia.org/T296046) (owner: 10BryanDavis) [22:52:02] (03Merged) 10jenkins-bot: feature: build-time arguments for lives & runs user config [blubber] - 10https://gerrit.wikimedia.org/r/749569 (https://phabricator.wikimedia.org/T296046) (owner: 10BryanDavis) [23:43:29] PROBLEM - Check systemd state on doc1001 is CRITICAL: CRITICAL - degraded: The following units failed: rsync-doc-doc1002.eqiad.wmnet.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [23:52:48] bd808: ^ has been deployed. thanks for that feature! [23:53:54] dduvall: yay! I will have some fun using it in the 3 projects where I'm using that blubber container as dev environment pattern now. :) [23:54:59] 10Beta-Cluster-Infrastructure: Beta cluster MediaWiki code not updating - https://phabricator.wikimedia.org/T300591 (10Tgr) [23:58:49] bd808: yeah that's pretty cool [23:59:53] once it has a bit more testing I will write up the pattern somewhere and share it around