[03:36:28] 06Traffic, 06SRE, 07User-notice: Block traffic from user-agents not honoring our policy - https://phabricator.wikimedia.org/T400119#11084530 (10Samwilson) Will GitLab CI be excluded from this policy? While working on T395398 I'm getting "429 Please set a proper user-agent…" in CI for URLs like https://wikis... [07:21:41] 10netops, 06Infrastructure-Foundations: lsw1-d2-codfw is unreachable through gNMI - https://phabricator.wikimedia.org/T401881 (10ayounsi) 03NEW p:05Triage→03High [07:46:39] 10netops, 06Infrastructure-Foundations: lsw1-d2-codfw is unreachable through gNMI - https://phabricator.wikimedia.org/T401881#11084791 (10cmooney) I was able to break it more!! I toggled the port number in the config, commited, then changed it back. Hoping perhaps this would force it to restart. Now: ` cmoo... [07:48:31] 10netops, 06Infrastructure-Foundations: lsw1-d2-codfw is unreachable through gNMI - https://phabricator.wikimedia.org/T401881#11084794 (10cmooney) Perhaps we could try one of these? ` cmooney@lsw1-d2-codfw> restart jsd ? Possible completions: <[Enter]> Execute this command all-members R... [08:16:43] 06Traffic: Patch httpbb to support dummy backend for non-blackbox tests - https://phabricator.wikimedia.org/T396839#11084836 (10Fabfur) 05Open→03Declined [08:17:13] 06Traffic: Create VTC tests for HAProxy - https://phabricator.wikimedia.org/T393770#11084837 (10Fabfur) 05In progress→03Declined Abandoned for T400244 [08:37:17] 10netops, 06Infrastructure-Foundations: lsw1-d2-codfw is unreachable through gNMI - https://phabricator.wikimedia.org/T401881#11084923 (10ayounsi) 05Open→03Resolved a:03ayounsi Nice, it worked! ` lsw1-d2-codfw> restart jsd gracefully JET Services Daemon signalled but still running, waiting 28 second... [10:15:08] 06Traffic: Possible SSL certificate expiration - https://phabricator.wikimedia.org/T401902 (10Josve05a) 03NEW [10:20:04] 06Traffic, 06SRE: Possible SSL certificate expiration - https://phabricator.wikimedia.org/T401902#11085478 (10Josve05a) [10:29:27] 06Traffic, 06SRE: Possible SSL certificate expiration - https://phabricator.wikimedia.org/T401902#11085524 (10Aklapper) Which website is this about? [10:32:13] 06Traffic: Possible SSL certificate expiration - https://phabricator.wikimedia.org/T401902#11085553 (10Vgutierrez) p:05Triage→03Low Do you know which specific hostname the volunteer is asking about? For context: we currently use two Certificate Authorities: **Let’s Encrypt** and **Google Trust Services**... [10:37:54] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: Move pfw1b-codfw to rack F5 - https://phabricator.wikimedia.org/T401297#11085585 (10ayounsi) From a quick look it does seem best to have two control links for proper redundancy. But I suggest that we do a test. In a maintenance window, unplug t... [12:59:20] FIRING: DnsboxServiceMismatch: Service ntp-a state mismatch on dns1004:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://grafana.wikimedia.org/d/96fb573c-0f3c-456a-886c-e50c29f3ed48/dns-box-service-state?var-site=eqiad&var-instance=dns1004:9100 - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [12:59:29] oh interesting [13:00:04] transient, it was due to the restart of ntpsec [13:00:08] good to see it works I guess :P [13:04:20] RESOLVED: DnsboxServiceMismatch: Service ntp-a state mismatch on dns1004:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://grafana.wikimedia.org/d/96fb573c-0f3c-456a-886c-e50c29f3ed48/dns-box-service-state?var-site=eqiad&var-instance=dns1004:9100 - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [13:12:50] FIRING: [2x] DnsboxServiceMismatch: Service ntp-a state mismatch on dns1004:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [13:17:50] RESOLVED: [2x] DnsboxServiceMismatch: Service ntp-a state mismatch on dns1004:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [13:28:50] FIRING: [2x] DnsboxServiceMismatch: Service ntp-b state mismatch on dns1005:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [13:29:37] stale alerts ^ :( [13:30:33] created silence for these for the next few hours as the restart completes [13:34:12] 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: Move pfw1b-codfw to rack F5 - https://phabricator.wikimedia.org/T401297#11086240 (10Papaul) @ayounsi I will have to work with fundraising to see when it will be best for us to do so. Thanks [13:48:08] 06Traffic, 10Prod-Kubernetes, 06serviceops, 07Kubernetes, 13Patch-For-Review: Handling inbound IPIP traffic on low traffic LVS k8s based realservers - https://phabricator.wikimedia.org/T352956#11086307 (10akosiaris) >>! In T352956#11083016, @Vgutierrez wrote: >>>! In T352956#11016142, @akosiaris wrote: >... [14:02:28] 06Traffic, 10Prod-Kubernetes, 06serviceops, 07Kubernetes, 13Patch-For-Review: Handling inbound IPIP traffic on low traffic LVS k8s based realservers - https://phabricator.wikimedia.org/T352956#11086373 (10Vgutierrez) you got that available as part of the sre.loadbalancer.migrate-service-ipip cookbook on... [16:33:33] 06Traffic, 06SRE, 07User-notice: Block traffic from user-agents not honoring our policy - https://phabricator.wikimedia.org/T400119#11086977 (10bd808) >>! In T400119#11084530, @Samwilson wrote: > ~~Will GitLab CI be excluded from this policy?~~ I know you added the ignore edit to this, but as this thread is... [17:03:52] 06Traffic: Possible SSL certificate expiration - https://phabricator.wikimedia.org/T401902#11087081 (10Josve05a) They replied that it was the Wikipedia.org with Let’s Encrypt. So I guess this is a no-issue then. [18:48:10] 06Traffic: Upgrade Traffic hosts to trixie - https://phabricator.wikimedia.org/T401832#11087467 (10ssingh) [18:48:13] 06Traffic, 06SRE, 13Patch-For-Review: Upgrade pdns-recursor to 5.x on all prod DNS hosts (all C:dnsrecursor and so possibly WMCS) - https://phabricator.wikimedia.org/T381608#11087466 (10ssingh) [22:04:22] 06Traffic, 10MediaWiki-extensions-QuickInstantCommons, 10MediaWiki-File-management, 06MediaWiki-Platform-Team, and 2 others: Make InstantCommons and other uses of ForeignApiRepo use WMF policy-compliant user agents - https://phabricator.wikimedia.org/T400881#11088017 (10Tgr) @Joe which format do you think...