[00:29:14] 10Release-Engineering-Team (GitLab III: GitLab in LA đŸĒƒ): Provision Ingress with TLS for reggie to allow image reuse in pipelines - https://phabricator.wikimedia.org/T324361 (10dduvall) 05Open→03Resolved p:05Triage→03Medium Paired with @dancy and @jeena on this today. Reggie is now accessible via our clu... [01:31:23] 10Beta-Cluster-Infrastructure, 10Growth-Team, 10Growth-Team-Filtering, 10PageTriage, and 4 others: Set wgPageTriageEnableEnglishWikipediaFeatures to False on the Beta Cluster - https://phabricator.wikimedia.org/T321922 (10Stang) 05Open→03Resolved It should has been deployed. [04:58:18] 10Continuous-Integration-Config, 10Codex, 10Design-Systems-Team (Design-Systems-Sprint), 10Patch-For-Review: docs: Consider making the latest release branch the default for the live docs site - https://phabricator.wikimedia.org/T311097 (10Volker_E) @bmartinez I like the idea of being clearer about the diff... [05:16:53] 10Release-Engineering-Team, 10Fundraising-Backlog, 10Projects-Cleanup, 10Wikimedia-Fundraising-CiviCRM, 10Fundraising Sprint Wibbly Wobbly Timey Wimey: Decommission Fundraising's crm/civicrm git repo - https://phabricator.wikimedia.org/T314995 (10Eileenmcnaughton) @Aklapper you are correct - that repo st... [07:33:10] (03CR) 10Dom Walden: [C: 03+2] "I have added the password to jenkins credentials store." [integration/config] - 10https://gerrit.wikimedia.org/r/856618 (https://phabricator.wikimedia.org/T303739) (owner: 10Zfilipin) [07:34:59] (03Merged) 10jenkins-bot: WIP Run Selenium tests targeting production [integration/config] - 10https://gerrit.wikimedia.org/r/856618 (https://phabricator.wikimedia.org/T303739) (owner: 10Zfilipin) [08:20:08] !log Attached contint1002 as an agent of the CI Jenkins # T313832 [08:20:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:20:10] T313832: contint1002 service implementation tracking - https://phabricator.wikimedia.org/T313832 [08:23:37] (03PS1) 10Hashar: fab: update hostnames in comment [integration/config] - 10https://gerrit.wikimedia.org/r/866254 [08:40:51] (03PS1) 10Hashar: scap: drop default settings [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/866259 [08:44:57] (03PS2) 10Hashar: Remove scap settings having default values [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/866259 [08:44:59] (03PS1) 10Hashar: Use Puppet provided dsh group for the targets [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/866261 [08:54:35] (03PS1) 10Hashar: Review access change [integration/zuul/deploy] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/865751 [08:54:59] (03PS2) 10Hashar: Remove Gerrit Managers as owner [integration/zuul/deploy] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/865751 [08:55:05] (03CR) 10Hashar: [V: 03+2 C: 03+2] Remove Gerrit Managers as owner [integration/zuul/deploy] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/865751 (owner: 10Hashar) [08:56:03] (03PS1) 10Hashar: Review access change [integration/zuul/deploy] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/865752 [08:56:23] (03PS2) 10Hashar: Allow integration group to submit changes to branches [integration/zuul/deploy] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/865752 [08:56:31] (03CR) 10Hashar: [V: 03+2 C: 03+2] Allow integration group to submit changes to branches [integration/zuul/deploy] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/865752 (owner: 10Hashar) [08:57:04] (03CR) 10Hashar: [V: 03+2 C: 03+2] "From scap source code:" [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/866259 (owner: 10Hashar) [09:09:02] (03CR) 10Hashar: [V: 03+2 C: 03+2] Use Puppet provided dsh group for the targets [integration/zuul/deploy] - 10https://gerrit.wikimedia.org/r/866261 (owner: 10Hashar) [09:13:47] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10SRE, 10serviceops-collab, 10Patch-For-Review: contint1002 service implementation tracking - https://phabricator.wikimedia.org/T313832 (10hashar) [09:20:09] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10SRE, 10serviceops-collab, 10Patch-For-Review: contint1002 service implementation tracking - https://phabricator.wikimedia.org/T313832 (10hashar) [09:26:48] 10Phabricator: Request to add MusikAnimal to acl*phabricator - https://phabricator.wikimedia.org/T324716 (10Aklapper) 05Open→03Resolved a:03Aklapper Done! [10:14:37] (03PS2) 10Ladsgroup: jjb: Make wikimedia-portals-build job rebase [integration/config] - 10https://gerrit.wikimedia.org/r/856033 [10:20:28] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10SRE, 10serviceops-collab, 10Patch-For-Review: contint1002 service implementation tracking - https://phabricator.wikimedia.org/T313832 (10hashar) [10:20:35] (03CR) 10Ladsgroup: jjb: Make wikimedia-portals-build job rebase (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/856033 (owner: 10Ladsgroup) [10:23:13] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 10serviceops-collab: contint hardware refresh - https://phabricator.wikimedia.org/T294276 (10hashar) [10:23:23] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10SRE, 10serviceops-collab, 10Patch-For-Review: contint1002 service implementation tracking - https://phabricator.wikimedia.org/T313832 (10hashar) 05Open→03Resolved a:03hashar contint1002 is now attached as a Jenkins agent and run... [10:24:36] 10Release-Engineering-Team (Seen), 10serviceops, 10serviceops-collab: contint1001 hardware failures (remove contint1001 from production) - https://phabricator.wikimedia.org/T324698 (10hashar) contint1001 can be decommissioned [10:48:09] 10Release-Engineering-Team (Seen), 10serviceops-collab, 10Patch-For-Review: contint1001 hardware failures (remove contint1001 from production) - https://phabricator.wikimedia.org/T324698 (10LSobanski) [12:05:21] (03PS4) 10Hashar: Send Gerrit reviews with tag autogenerated:pipelinelib [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/859989 [12:05:52] (03CR) 10Hashar: "I have fixed yet another typo in the commit message. Otherwise it is ready for merging." [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/859989 (owner: 10Hashar) [12:26:41] (03CR) 10Nikerabbit: [C: 03+1] Grant l10n bot rights [phabricator/translations] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/863356 (https://phabricator.wikimedia.org/T321350) (owner: 10Hashar) [12:30:17] (03CR) 10Hashar: [V: 03+2 C: 03+2] Grant l10n bot rights [phabricator/translations] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/863356 (https://phabricator.wikimedia.org/T321350) (owner: 10Hashar) [12:31:42] (03PS3) 10Hashar: Add l10n-bot to phabricator/translations [integration/config] - 10https://gerrit.wikimedia.org/r/863804 (https://phabricator.wikimedia.org/T321350) [12:31:54] (03CR) 10Hashar: [C: 03+2] Add l10n-bot to phabricator/translations [integration/config] - 10https://gerrit.wikimedia.org/r/863804 (https://phabricator.wikimedia.org/T321350) (owner: 10Hashar) [12:33:32] (03Merged) 10jenkins-bot: Add l10n-bot to phabricator/translations [integration/config] - 10https://gerrit.wikimedia.org/r/863804 (https://phabricator.wikimedia.org/T321350) (owner: 10Hashar) [12:34:25] (03CR) 10Hashar: [C: 03+2] "deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/863804 (https://phabricator.wikimedia.org/T321350) (owner: 10Hashar) [12:44:28] 10Phabricator, 10Release-Engineering-Team: Work out how to bring in new Phabricator translations again - https://phabricator.wikimedia.org/T318763 (10Nikerabbit) [12:44:35] 10Phabricator, 10Release-Engineering-Team (Bonus Level 🕹ī¸), 10serviceops, 10serviceops-collab, 10Patch-For-Review: Deprecate git-ssh service on phabricator.wikimedia.org - https://phabricator.wikimedia.org/T296022 (10Nikerabbit) [12:46:03] 10Phabricator, 10Release-Engineering-Team, 10translatewiki.net, 10Language-Team (Language-2022-October-December), and 2 others: Phabricator translations can no more be received due to decommissioning of git-ssh - https://phabricator.wikimedia.org/T321350 (10Nikerabbit) 05In progress→03Resolved Proof: h... [14:23:44] 10Continuous-Integration-Infrastructure, 10PHP 8.2 support, 10Patch-For-Review: Create PHP 8.2 CI images and jobs for early testing - https://phabricator.wikimedia.org/T314093 (10Reedy) And it's now stable - https://www.php.net/archive/2022.php#2022-12-08-1 [14:24:32] 10Continuous-Integration-Infrastructure, 10PHP 8.2 support: Create PHP 8.2 CI images and jobs for early testing - https://phabricator.wikimedia.org/T314093 (10Reedy) [14:58:55] 10Gerrit, 10Sustainability (Incident Followup), 10Upstream: Investigate Gerrit h2 cache being way too large - https://phabricator.wikimedia.org/T323754 (10hashar) `gerrit2001.wikimedia.org` is the replica. It wrongfully got all materials from the primary rsynced in which means it has a snapshot of H2 databas... [15:05:30] 10Gerrit, 10Sustainability (Incident Followup), 10Upstream: Investigate Gerrit h2 cache being way too large - https://phabricator.wikimedia.org/T323754 (10hashar) After restarting Gerrit twice: ` 12M Dec 8 15:01 /var/lib/gerrit2/review_site/cache/accounts.h2.db 1.1M Dec 8 15:01 /var/lib/gerrit2/review_sit... [15:07:35] (03PS1) 10Jforrester: dockerfiles: Drop xdebug everywhere, no longer used [integration/config] - 10https://gerrit.wikimedia.org/r/866388 (https://phabricator.wikimedia.org/T319495) [15:07:37] (03PS1) 10Jforrester: [WIP] dockerfiles: [php82] Upgrade PHP to 8.2.0 now GM is out [integration/config] - 10https://gerrit.wikimedia.org/r/866389 (https://phabricator.wikimedia.org/T314093) [15:19:00] 10Gerrit, 10Sustainability (Incident Followup), 10Upstream: Investigate Gerrit h2 cache being way too large - https://phabricator.wikimedia.org/T323754 (10hashar) On gerrit1001 ` $ ls -lah /var/lib/gerrit2/review_site/{db/account_patch_reviews,cache/*}.h2.db|cut -b30- 19M Dec 8 15:07 /var/lib/gerrit2/revie... [15:19:39] 10Gerrit, 10Sustainability (Incident Followup), 10Upstream: Investigate Gerrit h2 cache being way too large - https://phabricator.wikimedia.org/T323754 (10hashar) The important parts are the big files have dramatically shrink: * `gerrit_file_diff.h2.db` went from 12G to 527M * `git_file_diff.h2.db` went from... [15:21:16] 10Gerrit, 10Release-Engineering-Team: Investigate dropping diff.h2.db and quota.repo_size.h2.db disk caches - https://phabricator.wikimedia.org/T324771 (10hashar) [15:21:26] 10Gerrit, 10Sustainability (Incident Followup), 10Upstream: Investigate Gerrit h2 cache being way too large - https://phabricator.wikimedia.org/T323754 (10hashar) 05Open→03Resolved [15:21:32] 10Gerrit, 10Release-Engineering-Team (GitLab III: GitLab in LA đŸĒƒ), 10SRE-OnFire, 10serviceops-collab, and 2 others: gerrit1001 running out of space on / - https://phabricator.wikimedia.org/T323262 (10hashar) [15:42:57] 10Continuous-Integration-Infrastructure, 10OOUI, 10Regression: OOUI PHP demos page is broken (again) - https://phabricator.wikimedia.org/T322357 (10Esanders) [15:45:24] PROBLEM - Check systemd state on contint1001 is CRITICAL: CRITICAL - degraded: The following units failed: wmf_auto_restart_zuul-merger.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [15:48:20] 10Release-Engineering-Team (Seen), 10serviceops-collab: contint1001 hardware failures (remove contint1001 from production) - https://phabricator.wikimedia.org/T324698 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=54d181e2-ed3c-4489-aee0-e61e18b0e850) set by cgoubert@cumin1001 for 365 days... [15:54:06] (03CR) 10Ahmon Dancy: [C: 03+2] Send Gerrit reviews with tag autogenerated:pipelinelib [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/859989 (owner: 10Hashar) [15:55:20] (03Merged) 10jenkins-bot: Send Gerrit reviews with tag autogenerated:pipelinelib [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/859989 (owner: 10Hashar) [16:07:59] (03CR) 10Hashar: "I have updated all the jobs running `./jjb-update 'selenium-daily*':" [integration/config] - 10https://gerrit.wikimedia.org/r/856618 (https://phabricator.wikimedia.org/T303739) (owner: 10Zfilipin) [16:19:41] 10Project-Admins, 10Content-Transform-Team: "Parsoid Tracking tasks" in Phabricator vs Content Transform Team - https://phabricator.wikimedia.org/T324679 (10JMcLeod_WMF) @Aklapper thanks for raising this. I have updated the description to "Tasks that the Content Transform Team tracks but for which they are not... [16:25:42] 10Continuous-Integration-Config, 10Codex, 10Design-Systems-Team (Design-Systems-Sprint), 10Patch-For-Review: docs: Consider making the latest release branch the default for the live docs site - https://phabricator.wikimedia.org/T311097 (10AnneT) I also really like the idea of a different, eye-catching logo... [16:26:25] 10Continuous-Integration-Config, 10Codex, 10Design-Systems-Team (Design-Systems-Sprint), 10Patch-For-Review: docs: Consider making the latest release branch the default for the live docs site - https://phabricator.wikimedia.org/T311097 (10AnneT) As discussed elsewhere, we all agree that /main and /latest a... [16:27:14] !log After six years of services `contint1001.wikimedia.org` has been shutdown. *waves* [16:27:16] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:27:30] Farewell [16:27:55] which got setup after gallium had hardware failure which might have been the original server [16:28:02] đŸĢĄ [16:28:04] so we are on our 3rd server since 2010 :-] [16:28:14] {{citation_needed} [16:29:12] I am stopping working a bit earlier today. I am fried [16:29:27] will play with openldap/jenkins containers tomorrow :-] [18:00:59] 10Release-Engineering-Team (GitLab III: GitLab in LA đŸĒƒ): Add Gitlab JWT support to Reggie - https://phabricator.wikimedia.org/T323394 (10dduvall) Which way are we going with this? FWIW I think fronting with nginx would allow offloading of both auth (via jwt-authorizer) and GET/HEAD request caching. Honestly, if... [18:10:51] (03CR) 10Jforrester: [C: 03+2] fab: update hostnames in comment [integration/config] - 10https://gerrit.wikimedia.org/r/866254 (owner: 10Hashar) [18:13:16] (03Merged) 10jenkins-bot: fab: update hostnames in comment [integration/config] - 10https://gerrit.wikimedia.org/r/866254 (owner: 10Hashar) [18:39:09] 10Beta-Cluster-Infrastructure: Replace deployment-prometheus02 - https://phabricator.wikimedia.org/T324782 (10TheresNoTime) [19:05:55] 10Beta-Cluster-Infrastructure: Replace deployment-prometheus02 - https://phabricator.wikimedia.org/T324782 (10TheresNoTime) With `role::prometheus::beta`, running puppet fails to install prometheus with `E: Unable to correct problems, you have held broken packages.` — will take a look later unless someone else g... [20:42:13] 10Continuous-Integration-Infrastructure, 10Patch-For-Review: Drop xdebug from all RelEng images - https://phabricator.wikimedia.org/T319495 (10Jdforrester-WMF) [22:25:38] 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ), 10Patch-For-Review, 10Release, 10Train Deployments: 1.40.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T320518 (10colewhite) We're seeing a [[ https://grafana-rw.wikimedia.org/d/000000561/logstash?orgId=1&from=1670527964115&to=167053491563... [22:31:20] 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ), 10Patch-For-Review, 10Release, 10Train Deployments: 1.40.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T320518 (10Ladsgroup) @Jdlrobson I think it's due to the module target changes. [22:36:37] 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ), 10Patch-For-Review, 10Release, 10Train Deployments: 1.40.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T320518 (10Ladsgroup) Is it because a module is marked to target mobile but something in the tree is not? If that's the case, I think I... [22:46:59] 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ), 10Patch-For-Review, 10Release, 10Train Deployments: 1.40.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T320518 (10Ladsgroup) Ah, it seems the patches didn't make it to branch cut. Backporting. [23:35:02] 10Release-Engineering-Team (Priority Backlog đŸ“Ĩ), 10Patch-For-Review, 10Release, 10Train Deployments: 1.40.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T320518 (10Ladsgroup) It's in a much better state now: https://grafana.wikimedia.org/d/000000561/logstash?from=1670529261667&orgId=1&to=...