[08:42:34] 10netops, 10Infrastructure-Foundations, 10SRE: Maintain ROAs for currently unannounced BGP assignments - https://phabricator.wikimedia.org/T345601 (10cmooney) p:05Triage→03Low [08:43:24] 10netops, 10Infrastructure-Foundations, 10SRE: Maintain ROAs for currently unannounced BGP assignments - https://phabricator.wikimedia.org/T345601 (10ayounsi) Sounds good to me! [09:39:40] 10netops, 10Infrastructure-Foundations, 10SRE: Maintain ROAs for currently unannounced BGP assignments - https://phabricator.wikimedia.org/T345601 (10cmooney) 05Open→03Resolved a:03cmooney I've added ROAs for our newer RIPE /24 range and the old esams one now to help protect against hi-jack / misuse.... [13:08:11] 10Traffic, 10SRE, 10SRE-swift-storage, 10Thumbor: Cache thumbs in our caching infrastructure (e.g. ATS) - https://phabricator.wikimedia.org/T345334 (10MatthewVernon) Thanks. Is there any way that TTL cap could be raised for thumbnails? [13:12:43] 10Traffic, 10SRE, 10SRE-swift-storage, 10Thumbor: Cache thumbs in our caching infrastructure (e.g. ATS) - https://phabricator.wikimedia.org/T345334 (10Vgutierrez) Cache revalidation can further extend this period. After the initial 24-hour limit has passed, ATS will issue a conditional request to the backe... [13:45:05] 10Traffic, 10SRE, 10SRE-swift-storage, 10Thumbor: Cache thumbs in our caching infrastructure (e.g. ATS) - https://phabricator.wikimedia.org/T345334 (10MatthewVernon) Right, and swift does say 304 quite a lot; but that isn't very helpful for thumbs - swift can only say 304 because it's storing all these thu... [15:20:59] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Plan codfw row A/B top-of-rack switch refresh - https://phabricator.wikimedia.org/T327938 (10Papaul) netbox cable id update for ssw1-a8 to lsw1-a1 and lsw-a8 [16:47:45] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host doh1002.wikimedia.org with OS bookworm [17:08:58] 10Traffic, 10SRE, 10SRE-swift-storage, 10Thumbor: Cache thumbs in our caching infrastructure (e.g. ATS) - https://phabricator.wikimedia.org/T345334 (10Ladsgroup) Okay, I might not know what's going on so my apologies if I'm misunderstanding something. The front of CDN is not the target here, we were told... [17:44:53] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host doh1002.wikimedia.org with OS bookworm completed: - doh1002 (**PASS**) - Downtimed on Icinga/Al... [17:50:02] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10ssingh) Not directly related to bookworm but observed on the dnsdist 1.8.0 upgrade (part of bookworm) that results in a broken `latency_bucket` metric for the Wikimedia DNS hosts. Reported u... [17:57:57] 10Traffic, 10SRE, 10SRE-swift-storage, 10Thumbor: Cache thumbs in our caching infrastructure (e.g. ATS) - https://phabricator.wikimedia.org/T345334 (10BBlack) This topic probably deserves a ~hour meeting w/ Traffic to hash out some of the potential solutions and tradeoffs, but I'm gonna try to bullet-point... [18:31:28] 10Traffic, 10SRE, 10Epic: Deploy Wikimedia DNS: DNS-over-HTTPS (DoH) and DNS-over-TLS (DoT) public resolver - https://phabricator.wikimedia.org/T252132 (10ssingh) https://meta.wikimedia.org/wiki/Wikimedia_DNS is a detailed introduction of the project, including an FAQ. [18:52:49] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host doh1001.wikimedia.org with OS bookworm [19:32:23] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host doh1001.wikimedia.org with OS bookworm completed: - doh1001 (**PASS**) - Downtimed on Icinga/Al... [19:42:43] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10BCornwall) [20:54:13] 10Traffic: Package libvmod-re2 for Debian 12/Bookworm - https://phabricator.wikimedia.org/T345663 (10BCornwall) [20:56:42] 10Traffic: Package libvmod-re2 for Debian 12/Bookworm - https://phabricator.wikimedia.org/T345663 (10BCornwall) 05Open→03In progress p:05Triage→03Medium [21:01:55] 10Traffic: Package libvmod-re2 for Debian 12/Bookworm - https://phabricator.wikimedia.org/T345663 (10BCornwall)