[00:00:47] 10serviceops, 10SRE, 10SRE-OnFire, 10Shellbox, 10Sustainability (Incident Followup): Shellbox resource management - https://phabricator.wikimedia.org/T310557 (10RLazarus) With https://gerrit.wikimedia.org/r/813924 we ought to see smaller bursts in utilization, so I'm going to tentatively crank the shellb... [07:41:07] !log rolling restart of ats-be on cp[1080,1083,1085,1087,5006,6001,6006,6009,6011,6015] [07:41:15] wrong channel... [07:41:24] E_COFFEE [09:57:10] 10serviceops, 10Release-Engineering-Team, 10serviceops-collab, 10GitLab (Infrastructure), and 2 others: GitLab major release: 15.x - https://phabricator.wikimedia.org/T309062 (10Jelto) I added a patch to bump `gitlab-ce` and `gitlab-runner` packages to `15.0` in https://gerrit.wikimedia.org/r/817211. Accor... [12:02:18] 10serviceops, 10Data-Engineering, 10Event-Platform, 10SRE: eventstreams chart should use latest common_templates - https://phabricator.wikimedia.org/T310721 (10akosiaris) @ottomata, has there been any progress on this one? Anything (e.g. reviews) we can help with? [12:02:36] 10serviceops, 10Data Engineering Planning, 10Data-Engineering-Kanban, 10Event-Platform, and 2 others: eventgate chart should use common_templates - https://phabricator.wikimedia.org/T303543 (10akosiaris) @ottomata, has there been any progress on this one? Anything (e.g. reviews) we can help with? [12:53:00] 10serviceops, 10Patch-For-Review: Put conf100[789] in production - https://phabricator.wikimedia.org/T311407 (10akosiaris) conf100[789] are now proper members of both etcd and zookeeper clusters. I 've got a slow cumin based rolling restart of all kafka-related components (kafka, kafka-mirror-maker, burrow)... [12:53:39] 10serviceops: Decomission conf100[456] - https://phabricator.wikimedia.org/T311408 (10akosiaris) [12:53:42] 10serviceops, 10SRE: Update conf1* servers - https://phabricator.wikimedia.org/T310062 (10akosiaris) [12:53:52] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q3:(Need By: TBD) rack/setup/install conf100[789] - https://phabricator.wikimedia.org/T301272 (10akosiaris) [12:54:25] 10serviceops, 10Patch-For-Review: Put conf100[789] in production - https://phabricator.wikimedia.org/T311407 (10akosiaris) 05Open→03Resolved a:03akosiaris Gonna resolve this. Work to remove the old hosts is tracked in T311408 [14:43:48] 10serviceops, 10Release-Engineering-Team, 10serviceops-collab, 10GitLab (Infrastructure), and 2 others: GitLab major release: 15.x - https://phabricator.wikimedia.org/T309062 (10brennen) > 14.10.5 (current version) -> 15.0.4 -> 15.2.0 (or most recent version). > > Do you think this is reasonable? Yeah, f... [14:48:17] 10serviceops, 10Data Engineering Planning, 10Data-Engineering-Kanban, 10Event-Platform, and 2 others: eventgate chart should use common_templates - https://phabricator.wikimedia.org/T303543 (10JArguello-WMF) Hi @akosiaris, Andrew is Out of office and will be back on Fri, Jul 29. :) [14:50:30] 10serviceops, 10Release-Engineering-Team, 10serviceops-collab, 10GitLab (Infrastructure), and 2 others: GitLab major release: 15.x - https://phabricator.wikimedia.org/T309062 (10Jelto) a:03Jelto Ok I'll prepare the `15.0` packages tomorrow and start with the test instance and test runners. We may have t... [14:50:59] 10serviceops, 10Data Engineering Planning, 10Data-Engineering-Kanban, 10Event-Platform, and 2 others: eventgate chart should use common_templates - https://phabricator.wikimedia.org/T303543 (10akosiaris) >>! In T303543#8104853, @JArguello-WMF wrote: > Hi @akosiaris, Andrew is Out of office and will be back... [15:34:22] 10serviceops, 10Patch-For-Review, 10Performance-Team (Radar): Remove prometheus-phpfpm-exporter from physical hosts - https://phabricator.wikimedia.org/T313505 (10Joe) 05In progress→03Resolved [15:34:26] 10serviceops, 10Patch-For-Review, 10Performance-Team (Radar): Properly support php7.4 across the observability stack - https://phabricator.wikimedia.org/T312634 (10Joe) [15:35:06] 10serviceops, 10Parsoid, 10Performance-Team (Radar): Parsoid migration to php 7.4 - https://phabricator.wikimedia.org/T312638 (10Joe) p:05Triage→03High [15:46:24] 10serviceops, 10Parsoid, 10Performance-Team (Radar): Parsoid migration to php 7.4 - https://phabricator.wikimedia.org/T312638 (10ssastry) You can switch scandium to php 7.4 this week, and we can redo baseline test runs. [15:47:41] 10serviceops, 10Parsoid, 10Performance-Team (Radar): Parsoid migration to php 7.4 - https://phabricator.wikimedia.org/T312638 (10ssastry) But, please give me a heads up before you do that switch so we can make sure not to kick off test runs in that period OR ask you to wait for an additional few hours for an... [16:59:56] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install contint1002 - https://phabricator.wikimedia.org/T313830 (10RobH) [17:00:14] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install contint1002 - https://phabricator.wikimedia.org/T313830 (10RobH) [17:07:40] 10serviceops, 10SRE: contint1002 service implementation tracking - https://phabricator.wikimedia.org/T313832 (10RobH) [18:18:53] 10serviceops, 10Phabricator, 10serviceops-collab, 10Release-Engineering-Team (The Decommission Mission 💀): Setup rsync for phab data on disk - https://phabricator.wikimedia.org/T313360 (10Dzahn) [18:24:58] 10serviceops, 10Phabricator, 10serviceops-collab, 10Release-Engineering-Team (The Decommission Mission 💀): Setup rsync for phab data on disk - https://phabricator.wikimedia.org/T313360 (10Dzahn) So I knew either we already had this code from last time or we had applied it in a special "migration" puppet ro... [18:52:32] 10serviceops, 10Phabricator, 10serviceops-collab, 10Release-Engineering-Team (The Decommission Mission 💀): Setup rsync for phab data on disk - https://phabricator.wikimedia.org/T313360 (10Dzahn) checking the firewall rules. an iptables -L on phab1001 shows: ` ACCEPT tcp -- phab1001.eqiad.wmnet any... [19:57:13] 10serviceops, 10Phabricator, 10serviceops-collab, 10Release-Engineering-Team (The Decommission Mission 💀): Setup rsync for phab data on disk - https://phabricator.wikimedia.org/T313360 (10Dzahn) [20:01:43] 10serviceops, 10Phabricator, 10serviceops-collab, 10Release-Engineering-Team (The Decommission Mission 💀): Setup rsync for phab data on disk - https://phabricator.wikimedia.org/T313360 (10Dzahn) regarding the UIDs and possible privilege issues after rsync: - the owner of directories and files under /srv/r... [21:56:04] 10serviceops, 10Performance-Team, 10Patch-For-Review, 10Platform Team Workboards (Clinic Duty Team), 10Sustainability (MediaWiki-MultiDC): Harmonize CentralAuth and core session TTL and migrate CentralAuth sessions to Kask - https://phabricator.wikimedia.org/T313496 (10Eevans) >>! In T313496#8099717, @ts... [22:27:55] 10serviceops, 10Phabricator, 10serviceops-collab, 10Release-Engineering-Team (The Decommission Mission 💀): Email tool maintainers about git-ssh deprecation on phabricator - https://phabricator.wikimedia.org/T313359 (10Aklapper) [22:53:07] 10serviceops, 10DBA, 10Performance-Team, 10Patch-For-Review, 10Sustainability (MediaWiki-MultiDC): Harmonize CentralAuth and core session TTL and migrate CentralAuth sessions to Kask - https://phabricator.wikimedia.org/T313496 (10Krinkle) [22:55:16] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes202[01] - https://phabricator.wikimedia.org/T313870 (10RobH) [22:55:21] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install kubernetes202[01] - https://phabricator.wikimedia.org/T313870 (10RobH) [22:57:26] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: kubernetes202[01] implementation tracking - https://phabricator.wikimedia.org/T313871 (10RobH) [23:05:00] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install kubernetes102[01] - https://phabricator.wikimedia.org/T313873 (10RobH) [23:06:46] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install kubernetes102[01] - https://phabricator.wikimedia.org/T313873 (10RobH) [23:07:42] 10serviceops, 10SRE: kubernetes102[01] implemetation tracking - https://phabricator.wikimedia.org/T313874 (10RobH)