[07:50:22] 10Traffic, 10SRE: cp2027 powercycled - https://phabricator.wikimedia.org/T289908 (10fgiunchedi) p:05Triage→03Medium [08:04:15] !log pool cp2027 - T289908 [08:04:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:04:20] T289908: cp2027 powercycled - https://phabricator.wikimedia.org/T289908 [08:04:27] wrong window... expected result :) [08:05:38] 10Traffic, 10SRE: cp2027 powercycled - https://phabricator.wikimedia.org/T289908 (10Vgutierrez) 05Open→03Resolved a:03Vgutierrez all seems to be OK with cp2027, I just repooled it. Thanks @elukey! [08:07:51] 10Traffic, 10MediaWiki-Uploading, 10SRE, 10serviceops, 10Wikimedia-production-error: Unexpected upload speed to commons - https://phabricator.wikimedia.org/T288481 (10fgiunchedi) p:05Triage→03Medium [08:08:25] 10Traffic, 10SRE, 10SRE-swift-storage, 10Thumbor: Thumbnail of deleted image shown in "File history" after new image with same filename got uploaded - https://phabricator.wikimedia.org/T281780 (10fgiunchedi) p:05Triage→03Medium [08:27:47] 10Traffic, 10SRE, 10SRE-swift-storage, 10Thumbor: Thumbnail of deleted image shown in "File history" after new image with same filename got uploaded - https://phabricator.wikimedia.org/T281780 (10jcrespo) >>! In T281780#7315315, @AntiCompositeNumber wrote: > The thumbnail not existing in Swift is certainly... [08:37:44] 10Traffic, 10SRE: Prometheus Varnish exporter unit should depend on Varnish - https://phabricator.wikimedia.org/T283660 (10ema) @MMandere: is there anything left to do here? If not, let's close the task! [08:41:42] 10Traffic, 10SRE, 10SRE Observability: Prometheus Varnish exporter alert: add runbook and link to dashboard - https://phabricator.wikimedia.org/T289974 (10ema) [08:41:49] 10Traffic, 10SRE, 10SRE Observability: Prometheus Varnish exporter alert: add runbook and link to dashboard - https://phabricator.wikimedia.org/T289974 (10ema) p:05Triage→03Low [08:42:55] 10Traffic, 10SRE, 10SRE Observability: Prometheus Varnish exporter alert: add runbook and link to dashboard - https://phabricator.wikimedia.org/T289974 (10ema) [08:57:11] task failed successfully [09:11:51] 10Traffic, 10SRE: Prometheus Varnish exporter unit should depend on Varnish - https://phabricator.wikimedia.org/T283660 (10MMandere) @ema: All is set here. I will go ahead and close the task. [09:12:43] 10Traffic, 10SRE: Prometheus Varnish exporter unit should depend on Varnish - https://phabricator.wikimedia.org/T283660 (10MMandere) 05Open→03Resolved [12:39:43] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q1:(Need By: TBD) rack/setup/install cloudswift100[12] - https://phabricator.wikimedia.org/T289882 (10ayounsi) > (So row B in 10G, C8, and/or D5.) New cloud hosts only go in cloud racks, so no row B. [12:44:39] 10Traffic, 10SRE, 10Patch-For-Review: Deploy durum: check service for Wikidough - https://phabricator.wikimedia.org/T289536 (10ayounsi) Note that DNS PTRs are missing at least for: ` 185.71.138.139 185.71.138.141 185.71.138.140 ` [12:45:23] 10Traffic, 10SRE, 10Patch-For-Review: Deploy durum: check service for Wikidough - https://phabricator.wikimedia.org/T289536 (10ssingh) >>! In T289536#7318613, @ayounsi wrote: > Note that DNS PTRs are missing at least for: > ` > 185.71.138.139 > 185.71.138.141 > 185.71.138.140 > ` Oh right, good catch, thank... [12:48:47] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: Three ports on asw2-d-eqiad are not working as expected - https://phabricator.wikimedia.org/T247881 (10ayounsi) 05Stalled→03Resolved a:03ayounsi Noted, thanks! Yeah fine to close for now, and re-open if any issues. [13:40:28] 10Traffic, 10 Data-Engineering, 10Analytics, 10Growth-Team, and 4 others: Migrated Server-side EventLogging events recording http.client_ip as 127.0.0.1 - https://phabricator.wikimedia.org/T288853 (10Ottomata) Hello! > I believe that the top-level http and meta properties are in a sense "owned" by the int... [13:41:53] 10Traffic: Test haproxy as a WMF's CDN TLS terminator with real traffic - https://phabricator.wikimedia.org/T290005 (10Vgutierrez) [13:41:59] 10Traffic: Test haproxy as a WMF's CDN TLS terminator with real traffic - https://phabricator.wikimedia.org/T290005 (10Vgutierrez) p:05Triage→03Medium [13:45:52] 10Traffic, 10SRE: Test haproxy as a WMF's CDN TLS terminator with real traffic - https://phabricator.wikimedia.org/T290005 (10Vgutierrez) [14:39:18] 10Traffic, 10SRE: Test haproxy as a WMF's CDN TLS terminator with real traffic - https://phabricator.wikimedia.org/T290005 (10Vgutierrez) [14:45:11] 10Traffic, 10SRE: Test haproxy as a WMF's CDN TLS terminator with real traffic - https://phabricator.wikimedia.org/T290005 (10Vgutierrez) [15:03:07] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q1:(Need By: TBD) rack/setup/install cloudswift100[12] - https://phabricator.wikimedia.org/T289882 (10RobH) [15:03:41] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q1:(Need By: TBD) rack/setup/install cloudswift100[12] - https://phabricator.wikimedia.org/T289882 (10RobH) >>! In T289882#7318587, @ayounsi wrote: >> (So row B in 10G, C8, and/or D5.) > New cloud hosts only go in cloud racks, so no row... [15:05:17] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q1:(Need By: TBD) rack/setup/install cloudswift100[12] - https://phabricator.wikimedia.org/T289882 (10RobH) a:03nskaggs @nskaggs: Who in WMCS is going to be point on these servers? I ask so we can assign them this racking task, so t... [17:34:57] (VarnishTrafficDrop) firing: 58% GET drop in text@ during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [17:44:57] (VarnishTrafficDrop) resolved: 68% GET drop in text@ during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [17:53:27] 10Traffic, 10Privacy Engineering, 10SRE, 10Performance-Team (Radar), 10Privacy: Disable WMF-Last-Access cookies for wmfusercontent.org - https://phabricator.wikimedia.org/T210167 (10Krinkle) [18:33:23] 10Traffic, 10DNS, 10SRE: DNS entries for WikiLearn dev servers - https://phabricator.wikimedia.org/T289618 (10Asaf) 05Open→03Resolved Never mind, opened a new task with updated request. [23:18:18] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: rack spare switches in c1-eqiad - https://phabricator.wikimedia.org/T185337 (10wiki_willy) a:03Cmjohnson