[07:01:42] 06Traffic: Reimage one of each Traffic hosts before magru - https://phabricator.wikimedia.org/T359053#9788869 (10Fabfur) 05Open→03Resolved I'd say this could be closed... [09:03:33] 06Traffic, 10MoveComms-Support, 10MW-on-K8s, 06serviceops, and 2 others: Move 100% of external traffic to Kubernetes (excluding Votewiki and Commons) - https://phabricator.wikimedia.org/T362323#9789300 (10Clement_Goubert) [09:46:17] 10Acme-chief: acme-chief: add support for serving individual files over the puppet file system api - https://phabricator.wikimedia.org/T364589#9789524 (10Vgutierrez) file_metadata is already there and supports individual files. The only limitation is that it's currently expecting the parameters `links=manage&sou... [10:07:48] 10netops, 06Infrastructure-Foundations, 06SRE: Cloud IPv6 subnets - https://phabricator.wikimedia.org/T187929#9789580 (10taavi) >>! In T187929#9748100, @cmooney wrote: > The aggregate that is used for the cloud-private allocations should come from IPv6 space not announced to the internet/DFZ, or space that i... [10:18:37] 10Acme-chief, 13Patch-For-Review: acmechief: add support for providing files with they private key before the public key - https://phabricator.wikimedia.org/T364424#9789606 (10CodeReviewBot) vgutierrez opened https://gitlab.wikimedia.org/repos/sre/acme-chief/-/merge_requests/7 acme_chief,x509: Provide pri... [10:25:11] 10netops, 06cloud-services-team, 06Infrastructure-Foundations, 06SRE: CloudVPS: enable BGP in the neutron transport network - https://phabricator.wikimedia.org/T245606#9789660 (10taavi) 05Stalled→03Declined Closing this in favour of the slightly different approach in {T358868} that's likely going t... [10:28:35] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 07Epic: CloudVPS: network architecture - https://phabricator.wikimedia.org/T209460#9789669 (10taavi) 05Open→03Resolved Closing this task since I don't see a clear end goal here. Current ongoing and planned work is already... [10:43:46] 06Traffic, 10Observability-Metrics, 13Patch-For-Review: Add prometheus-https load balancer - https://phabricator.wikimedia.org/T326657#9789702 (10fgiunchedi) An example task of such migration is https://phabricator.wikimedia.org/T246998, which basically translates to: * provision a new oidc client for promet... [10:52:25] 10Acme-chief, 13Patch-For-Review: acme-chief: add support for serving individual files over the puppet file system api - https://phabricator.wikimedia.org/T364589#9789757 (10CodeReviewBot) vgutierrez opened https://gitlab.wikimedia.org/repos/sre/acme-chief/-/merge_requests/8 api: Stop requiring links/source_p... [11:19:44] there's a new dnsdist security issue affecting DoH workloads, but our internal build isn't affected, only for 1.9.x: https://www.openwall.com/lists/oss-security/2024/05/13/1 [12:01:16] yep thanks! [12:44:01] 06Traffic, 13Patch-For-Review: Use IPIP encapsulation on lvs<-->upload cluster - https://phabricator.wikimedia.org/T357257#9790016 (10Vgutierrez) [13:21:00] 06Traffic: Elevated 503 backend fetch failed reported by users - https://phabricator.wikimedia.org/T364691#9790207 (10Vgutierrez) we had a big spike of 503s on eqiad/drmrs/esams yesterday during EU morning: https://grafana.wikimedia.org/goto/J4YqQuYIR?orgId=1: {F52905505} [13:35:21] 06Traffic: Elevated 503 backend fetch failed reported by users - https://phabricator.wikimedia.org/T364691#9790288 (10Ladsgroup) >>! In T364691#9790207, @Vgutierrez wrote: > we had a big spike of 503s on eqiad/drmrs/esams yesterday during EU morning: https://grafana.wikimedia.org/goto/J4YqQuYIR?orgId=1: > {F5290... [13:37:07] 06Traffic: Elevated 503 backend fetch failed reported by users - https://phabricator.wikimedia.org/T364691#9790295 (10Vgutierrez) >>! In T364691#9790288, @Ladsgroup wrote: >>>! In T364691#9790207, @Vgutierrez wrote: >> we had a big spike of 503s on eqiad/drmrs/esams yesterday during EU morning: https://grafana.w... [13:40:14] 06Traffic: Elevated 503 backend fetch failed reported by users - https://phabricator.wikimedia.org/T364691#9790300 (10Ladsgroup) Haven't checked logged out, what I get is logged in users. [13:54:21] 06Traffic, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: GeoIP mapping experiments - https://phabricator.wikimedia.org/T332024#9790333 (10CDanis) [14:48:31] 10netops, 06cloud-services-team, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Create (or teach Andrew how to create) private connections+dns entries for new cloudcontrols - https://phabricator.wikimedia.org/T364559#9790593 (10cmooney) 05Open→03Resolved p:05Triage→03Medium >>! In T364559#... [18:27:09] FIRING: LVSHighRX: Excessive RX traffic on lvs6001:9100 (enp175s0f0np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs6001 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX [18:32:09] RESOLVED: LVSHighRX: Excessive RX traffic on lvs6001:9100 (enp175s0f0np0) - https://bit.ly/wmf-lvsrx - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=lvs6001 - https://alerts.wikimedia.org/?q=alertname%3DLVSHighRX