[10:46:18] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10KOfori) [11:13:41] 10netops, 10Infrastructure-Foundations, 10SRE: Tighter control on exported BGP routes from MRs - https://phabricator.wikimedia.org/T348739 (10cmooney) p:05Triage→03Low [11:41:03] 10Traffic, 10Abstract Wikipedia team, 10SRE, 10Wikifunctions, and 2 others: Separate deployment for wikifunctions.org - https://phabricator.wikimedia.org/T347544 (10JMeybohm) [11:44:07] fabfur: you around by chance? I'd like to roll out https://gerrit.wikimedia.org/r/c/operations/puppet/+/965056/ and wonder what's the best way to check it is working before rolling out everywhere [11:57:31] * fabfur checking [11:58:23] I saw the comment from v.g, those domains are now ok? [12:01:18] seems that they are [12:02:29] oh, yeah sorry. [12:02:52] That change was crafted in parallel to actually adding all that dns and lvs stuff [12:03:59] you could disable puppet on all cp hosts, run the puppet agent with the change just on a test host (eg. in ulsfo) and test the change against it [12:05:52] so just curl'ing cp host directly? [12:07:12] I think passing the `Host: wikifunctions.org` header would do the trick [12:07:32] and `X-Forwarded-Proto: https` probably too [12:08:40] is there aparticular test host in ulsfo? [12:09:47] curl -kI -XGET https://localhost/wiki/Special:BlankPage -H Host:www.wikifunctions.org seems to work for me [12:10:42] do you want to do a HEAD or GET request? [12:11:18] GET [12:11:44] ok, so you can remove `-I` [12:12:16] oh, that I used to only get the reasponse headers printed [12:12:19] after the deployment do you expect the same answer, I suppose... [12:12:33] ok, I use `-v` for that usually [12:13:43] yeah, I expect the same answer apart from the server it comes from [12:14:12] which should then be a k8s pod every time [12:14:37] so with "test host" you mean just any cp in ulsfo? [12:15:06] yep [12:15:17] okay, will do. thanks! [12:15:33] the change is identical for text and upload hosts? [12:17:43] AIUI yes [12:18:26] oh, I copy pasted the wrong alias to SAL [12:27:09] LGTM, re-enabled puppet on the cp hosts. Thanks again [12:34:36] thanks! [12:55:45] 10Traffic, 10Abstract Wikipedia team, 10SRE, 10Wikifunctions, 10serviceops: Separate deployment for wikifunctions.org - https://phabricator.wikimedia.org/T347544 (10JMeybohm) [13:04:00] 10Traffic, 10Abstract Wikipedia team, 10SRE, 10Wikifunctions, 10serviceops: Separate deployment for wikifunctions.org - https://phabricator.wikimedia.org/T347544 (10JMeybohm) 05In progress→03Resolved All wikifunctions.org traffic from the edge as well as from function-orchestrator is now served by th... [14:58:46] 10Traffic, 10Abstract Wikipedia team, 10SRE, 10Wikifunctions, 10serviceops: Separate deployment for wikifunctions.org - https://phabricator.wikimedia.org/T347544 (10Jdforrester-WMF) Thank you! [15:00:42] (SystemdUnitFailed) firing: prometheus_gdnsd_stats.service Failed on dns2006:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:00:47] ^ expeted [15:05:42] (SystemdUnitFailed) resolved: prometheus_gdnsd_stats.service Failed on dns2006:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:26:19] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install cp11[00-15] - https://phabricator.wikimedia.org/T342159 (10VRiley-WMF) [16:34:18] 10Traffic, 10Abstract Wikipedia team, 10SRE, 10Wikifunctions, 10serviceops: Separate deployment for wikifunctions.org - https://phabricator.wikimedia.org/T347544 (10Jdforrester-WMF) [16:34:28] 10Traffic, 10Abstract Wikipedia team, 10MW-on-K8s, 10SRE, and 4 others: Migrate functions-orchestrator service to mw-api-int - https://phabricator.wikimedia.org/T347397 (10Jdforrester-WMF) [16:34:54] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install cp11[00-15] - https://phabricator.wikimedia.org/T342159 (10VRiley-WMF) [16:35:56] 10Traffic, 10SRE: Simplify maintenance of DNS/NTP hosts to reduce toil around reboots, reimages, and other work - https://phabricator.wikimedia.org/T347054 (10ssingh) [16:36:47] 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE: Remove static routes for ns[01] and replace their announcements with bird - https://phabricator.wikimedia.org/T348041 (10ssingh) The static routes have been removed and `ns[01]` are now announced via `bird`. Thanks to @ayounsi for his help with this! [16:36:51] 10Traffic, 10SRE: Simplify maintenance of DNS/NTP hosts to reduce toil around reboots, reimages, and other work - https://phabricator.wikimedia.org/T347054 (10ssingh) [16:37:26] 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE: Remove static routes for ns[01] and replace their announcements with bird - https://phabricator.wikimedia.org/T348041 (10ssingh) 05Open→03Resolved a:03ssingh [17:13:07] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install cp11[00-15] - https://phabricator.wikimedia.org/T342159 (10VRiley-WMF) [17:59:38] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install cp11[00-15] - https://phabricator.wikimedia.org/T342159 (10VRiley-WMF) [22:36:14] 10Traffic: Investigate why Traffic SLO Grafana dashboard has negative values on combined SLI - https://phabricator.wikimedia.org/T341606 (10BCornwall) @herron Thanks for all of your help. We've implemented varnish_sli_bad. I followed the formulae presented at the top of grafana-grizzly's slo_definitions.libsonn... [22:36:30] 10Traffic: Investigate why Traffic SLO Grafana dashboard has negative values on combined SLI - https://phabricator.wikimedia.org/T341606 (10BCornwall) 05Stalled→03In progress