[00:46:06] 10serviceops, 10SRE, 10SRE-OnFire, 10Shellbox, 10Sustainability (Incident Followup): Shellbox resource management - https://phabricator.wikimedia.org/T310557 (10RLazarus) 05Open→03Resolved a:03RLazarus [01:40:56] 10serviceops, 10DBA, 10Performance-Team, 10Patch-For-Review, 10Sustainability (MediaWiki-MultiDC): Harmonize CentralAuth and core session TTL and migrate CentralAuth sessions to Kask - https://phabricator.wikimedia.org/T313496 (10tstarling) I don't really care whether it uses a separate Kask instance or... [02:33:29] 10serviceops, 10Patch-For-Review, 10Performance-Team (Radar): Properly support php7.4 across the observability stack - https://phabricator.wikimedia.org/T312634 (10Krinkle) It'd be nice to have a comparison. But with T244776 and T200108 still unresolved, it's not immediately obvious how safe it would be to a... [03:12:35] 10serviceops, 10MediaWiki-General, 10MediaWiki-libs-Metrics, 10SRE, and 4 others: MediaWiki Prometheus support - https://phabricator.wikimedia.org/T240685 (10Krinkle) >>! In T240685#8088120, @colewhite wrote: > From @tstarling's comment, I see a few action items: > # Create a service that can be a drop-in... [05:43:27] 10serviceops, 10DBA, 10Performance-Team, 10Patch-For-Review, 10Sustainability (MediaWiki-MultiDC): Harmonize CentralAuth and core session TTL and migrate CentralAuth sessions to Kask - https://phabricator.wikimedia.org/T313496 (10Joe) A few considerations: * I do prefer to have separate kask installation... [06:59:48] 10serviceops, 10Parsoid, 10Patch-For-Review, 10Performance-Team (Radar): Parsoid migration to php 7.4 - https://phabricator.wikimedia.org/T312638 (10Joe) Once we've moved scandium, my plan for parsoid would be to install the new hardware we got with php 7.4 only and progressively swap out old hardware for... [09:22:10] 10serviceops, 10Parsoid, 10Patch-For-Review, 10Performance-Team (Radar): Parsoid migration to php 7.4 - https://phabricator.wikimedia.org/T312638 (10Joe) @subbu would it be ok if we switched to php 7.4 tomorrow, July 28th, at 9:00 UTC? I have all the patches lined up to do that. [10:08:51] 10serviceops, 10SRE, 10serviceops-collab: contint1002 service implementation tracking - https://phabricator.wikimedia.org/T313832 (10LSobanski) [12:35:15] 10serviceops, 10Release-Engineering-Team, 10serviceops-collab, 10GitLab (CI & Job Runners), and 2 others: Changes to modules/gitlab_runner/templates/config-template.toml.erb have no effect on existing runners - https://phabricator.wikimedia.org/T311746 (10Jelto) All Runners (Trusted, Test and Shared) using... [12:47:19] 10serviceops, 10Infrastructure-Foundations, 10Scap, 10Patch-For-Review, 10Release-Engineering-Team (Deployment Autopilot 🛩ī¸): Use scap to deploy itself to scap targets - https://phabricator.wikimedia.org/T303559 (10jnuche) [13:27:53] 10serviceops, 10Parsoid, 10Patch-For-Review, 10Performance-Team (Radar): Parsoid migration to php 7.4 - https://phabricator.wikimedia.org/T312638 (10ssastry) Sure, that works for us. [14:20:37] 10serviceops, 10Parsoid, 10Patch-For-Review, 10Performance-Team (Radar): Parsoid migration to php 7.4 - https://phabricator.wikimedia.org/T312638 (10cscott) Let us know when it's done and we can kick off an rt-test run to kick the tires. [15:55:13] 10serviceops, 10Phabricator, 10serviceops-collab, 10Patch-For-Review, 10Release-Engineering-Team (The Decommission Mission 💀): move phabricator to new hardware generation - https://phabricator.wikimedia.org/T280597 (10Dzahn) 05Stalled→03In progress [15:59:30] 10serviceops, 10Phabricator, 10serviceops-collab, 10Patch-For-Review, 10Release-Engineering-Team (The Decommission Mission 💀): move phabricator to new hardware generation - https://phabricator.wikimedia.org/T280597 (10Dzahn) [15:59:38] 10serviceops, 10Phabricator, 10serviceops-collab, 10Patch-For-Review, 10Release-Engineering-Team (The Decommission Mission 💀): Setup rsync for phab data on disk - https://phabricator.wikimedia.org/T313360 (10Dzahn) 05Open→03In progress [16:16:41] 10serviceops, 10PHP 7.2 support, 10Patch-For-Review: Drop PHP 7.2 support from MediaWiki master branch, once Wikimedia production is on 7.4 - https://phabricator.wikimedia.org/T261872 (10Jdforrester-WMF) [17:50:49] 10serviceops: decomission mw2251-mw2258 - https://phabricator.wikimedia.org/T313730 (10RLazarus) N.B. this is only seven hosts, mw225[1-5,7-8] -- mw2256 was already decommed in T263065. [17:51:10] 10serviceops: decomission mw2251-mw2255, mw2257-mw2258 - https://phabricator.wikimedia.org/T313730 (10RLazarus) [18:25:39] 10serviceops, 10DC-Ops, 10SRE, 10masz, 10ops-codfw: Q1:rack/setup/install new memcached hosts - https://phabricator.wikimedia.org/T313963 (10RobH) [18:25:50] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install new memcached hosts - https://phabricator.wikimedia.org/T313963 (10RobH) [18:28:03] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install new memcached hosts - https://phabricator.wikimedia.org/T313963 (10RobH) @joe, Reassigning this to you per our IRC discussion. Pending needs from you/#serviceops: * Please populate racking details section with hostname, OS and all other f... [18:28:41] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install new memcached hosts - https://phabricator.wikimedia.org/T313963 (10RobH) [18:28:55] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install new codfw memcached hosts - https://phabricator.wikimedia.org/T313963 (10RobH) [18:29:56] 10serviceops, 10SRE: codfw (2) memcached host service implementation tracking - https://phabricator.wikimedia.org/T313965 (10RobH) [18:31:23] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install new codfw memcached hosts - https://phabricator.wikimedia.org/T313963 (10RobH) [18:31:50] 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install new eqiad memcached hosts - https://phabricator.wikimedia.org/T313963 (10RobH) [18:32:10] 10serviceops, 10SRE: eqiad (2) memcached host service implementation tracking - https://phabricator.wikimedia.org/T313965 (10RobH) [18:33:12] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install new codfw memcached hosts - https://phabricator.wikimedia.org/T313966 (10RobH) [18:33:39] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install new codfw memcached hosts - https://phabricator.wikimedia.org/T313966 (10RobH) @joe, Reassigning this to you per our IRC discussion. Pending needs from you/#serviceops: * Please populate racking details section with hostname, OS and all o... [18:35:07] 10serviceops, 10SRE: codfw (2) memcached host service implementation tracking - https://phabricator.wikimedia.org/T313968 (10RobH) [18:35:27] 10serviceops, 10SRE: codfw (2) memcached host service implementation tracking - https://phabricator.wikimedia.org/T313968 (10RobH) [18:36:33] 10serviceops, 10DC-Ops, 10SRE, 10ops-codfw: Q1:rack/setup/install new codfw memcached hosts - https://phabricator.wikimedia.org/T313966 (10RobH) [18:36:47] 10serviceops, 10SRE: codfw (2) memcached host service implementation tracking - https://phabricator.wikimedia.org/T313968 (10RobH) [19:46:31] 10serviceops, 10Patch-For-Review: decomission mw2251-mw2255, mw2257-mw2258 - https://phabricator.wikimedia.org/T313730 (10Dzahn) Reviewing the change above I was looking at the mw225* range in Netbox: https://netbox.wikimedia.org/search/?q=mw225&obj_type= And i noticed this ticket and the Gerrit change handl... [20:01:29] 10serviceops, 10DBA, 10Performance-Team, 10Patch-For-Review, 10Sustainability (MediaWiki-MultiDC): Harmonize CentralAuth and core session TTL and migrate CentralAuth sessions to Kask - https://phabricator.wikimedia.org/T313496 (10Eevans) >>! In T313496#8106860, @tstarling wrote: >> Since "better" in thi... [20:02:43] 10serviceops, 10DBA, 10Performance-Team, 10Patch-For-Review, 10Sustainability (MediaWiki-MultiDC): Harmonize CentralAuth and core session TTL and migrate CentralAuth sessions to Kask - https://phabricator.wikimedia.org/T313496 (10Eevans) >>! In T313496#8107095, @Joe wrote: > A few considerations: > * I d... [21:53:20] 10serviceops, 10Patch-For-Review: decommission mw2251-mw2255, mw2257-mw2258 - https://phabricator.wikimedia.org/T313730 (10Aklapper) [23:17:51] 10serviceops, 10Patch-For-Review: decommission mw2251-mw2255, mw2257-mw2258 - https://phabricator.wikimedia.org/T313730 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=ed218867-a364-4f33-bbd9-60f66ba67f36) set by rzl@cumin2002 for 2:00:00 on 7 host(s) and their services with reason: Decom `... [23:22:26] 10serviceops, 10DBA, 10Performance-Team, 10Patch-For-Review, 10Sustainability (MediaWiki-MultiDC): Harmonize CentralAuth and core session TTL and migrate CentralAuth sessions to Kask - https://phabricator.wikimedia.org/T313496 (10tstarling) I deployed the increased TTL and tested it by regenerating my en... [23:45:17] 10serviceops, 10Patch-For-Review: decommission mw2251-mw2255, mw2257-mw2258 - https://phabricator.wikimedia.org/T313730 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by rzl@cumin2002 for hosts: `mw[2251-2255,2257-2258].codfw.wmnet` - mw2251.codfw.wmnet (**PASS**) - Downtimed host on Icinga...