[08:35:04] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Add per-output queue monitoring for Juniper network devices - https://phabricator.wikimedia.org/T326322 (10ayounsi) [12:58:54] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 2 others: Migrate internal traffic to k8s - https://phabricator.wikimedia.org/T333120 (10Jdforrester-WMF) [12:59:28] 10Traffic, 10Abstract Wikipedia team, 10MW-on-K8s, 10SRE, and 4 others: Migrate functions-orchestrator service to mw-api-int - https://phabricator.wikimedia.org/T347397 (10Jdforrester-WMF) 05Open→03In progress [13:29:34] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Add per-output queue monitoring for Juniper network devices - https://phabricator.wikimedia.org/T326322 (10ayounsi) To keep it somewhere for later, on Dell SONiC it should be on the `/openconfig-qos:qos/interfaces` path. Grouping it by sour... [13:30:18] 10Traffic, 10SRE, 10Patch-For-Review: Varnish mobile redirection misses some sites - https://phabricator.wikimedia.org/T344175 (10Fabfur) 05Open→03Stalled [13:31:18] 10Traffic, 10SRE, 10Patch-For-Review: Varnish mobile redirection misses some sites - https://phabricator.wikimedia.org/T344175 (10Fabfur) Hi, @nshahquinn if it's ok on your side I'll consider this as completed [13:32:46] 10Traffic, 10Abstract Wikipedia team, 10MW-on-K8s, 10SRE, and 4 others: Migrate functions-orchestrator service to mw-api-int - https://phabricator.wikimedia.org/T347397 (10JMeybohm) a:03JMeybohm It took me a while to figure this out, sorry. Due to wikifunctions having more strict firewall rules in genera... [13:38:38] 10Traffic, 10SRE: Simplify maintenance of DNS/NTP hosts to reduce toil around reboots, reimages, and other work - https://phabricator.wikimedia.org/T347054 (10ssingh) Thanks for the feedback everyone! I was waiting so that we can get most of the comments in before replying; responses inline: >>! In T347054#91... [13:38:53] 10netops, 10Infrastructure-Foundations, 10SRE: Upgrade asw1-eqsin - https://phabricator.wikimedia.org/T332395 (10ayounsi) FYI, the `mgmt_junos` bug (also present on the fasw) might not be fixed by an upgrade, but maybe with the solution exposed in https://www.reddit.com/r/Juniper/comments/mvq8hf/comment/j7gd... [13:41:51] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install cp1[098-113] - https://phabricator.wikimedia.org/T342159 (10VRiley-WMF) Updating naming as per requested. cp1100 - cp1115 [13:44:41] 10Traffic, 10SRE: Simplify maintenance of DNS/NTP hosts to reduce toil around reboots, reimages, and other work - https://phabricator.wikimedia.org/T347054 (10ssingh) Looking at `10.3.0.0/24` [[ https://netbox.wikimedia.org/ipam/prefixes/97/ip-addresses/ | in Netbox ]]: I plan to reserve `10.3.0.8/32` for `nt... [13:46:08] 10Traffic, 10MW-on-K8s, 10SRE, 10Wikidata, and 2 others: Serve Wikidata traffic via Kubernetes - https://phabricator.wikimedia.org/T347493 (10Lucas_Werkmeister_WMDE) [13:51:30] 10Traffic, 10SRE: Simplify maintenance of DNS/NTP hosts to reduce toil around reboots, reimages, and other work - https://phabricator.wikimedia.org/T347054 (10ayounsi) Thanks, as this VIP won't be critical we can skip the static routes and only allocate `10.3.0.8/32`. The existing "Reserved for XXX (backup st... [13:53:49] 10Traffic, 10SRE: Simplify maintenance of DNS/NTP hosts to reduce toil around reboots, reimages, and other work - https://phabricator.wikimedia.org/T347054 (10ssingh) >>! In T347054#9203404, @ayounsi wrote: > Thanks, as this VIP won't be critical we can skip the static routes and only allocate `10.3.0.8/32`. >... [13:53:59] 10Traffic, 10SRE: Allow purged to specify buffer length - https://phabricator.wikimedia.org/T346874 (10Fabfur) a:03Fabfur [13:55:20] 10Traffic, 10SRE: Simplify maintenance of DNS/NTP hosts to reduce toil around reboots, reimages, and other work - https://phabricator.wikimedia.org/T347054 (10ssingh) ` sukhe@re0.cr2-codfw# show routing-options static /* Anycast recdns - backup route */ route 10.3.0.0/30 { next-hop 208.80.153.77; read... [13:57:57] 10Traffic, 10SRE: Add README and build-specific Dockerfile to purged - https://phabricator.wikimedia.org/T347021 (10Fabfur) 05Open→03Resolved a:03Fabfur Done with * https://gerrit.wikimedia.org/r/c/operations/software/purged/+/958477 * https://gerrit.wikimedia.org/r/c/operations/software/purged/+/959049 [13:58:52] 10Traffic, 10SRE: Allow purged to specify buffer length - https://phabricator.wikimedia.org/T346874 (10Fabfur) 05Open→03Stalled Waiting for actual deployment to definitely closing this task [14:00:51] 10Traffic, 10SRE: Simplify maintenance of DNS/NTP hosts to reduce toil around reboots, reimages, and other work - https://phabricator.wikimedia.org/T347054 (10ssingh) For `route 10.3.0.0/30` above, `next-hop 208.80.153.77` is actually the old authdns host, so we are clearly not keeping the static routes update... [14:04:14] 10netops, 10Infrastructure-Foundations: Remove static routes for anycast prefixes - https://phabricator.wikimedia.org/T347494 (10ayounsi) [14:05:32] 10netops, 10Infrastructure-Foundations: Remove static routes for anycast prefixes - https://phabricator.wikimedia.org/T347494 (10ayounsi) a:03ayounsi [14:07:29] 10netops, 10Infrastructure-Foundations: Remove static routes for anycast prefixes - https://phabricator.wikimedia.org/T347494 (10ssingh) Thanks for the task! Do we plan to consolidate https://netbox.wikimedia.org/ipam/prefixes/97/ip-addresses/? That is, if we remove the redundant backup static routes, `10.3.0.... [14:12:21] 10netops, 10Infrastructure-Foundations: Remove static routes for anycast prefixes - https://phabricator.wikimedia.org/T347494 (10ayounsi) Yeah, actually you can use 10.3.0.2/32 for NTP, I won't go through renumbering the syslog VIP. [14:12:52] 10Traffic, 10SRE: Simplify maintenance of DNS/NTP hosts to reduce toil around reboots, reimages, and other work - https://phabricator.wikimedia.org/T347054 (10ayounsi) Thanks, I opened {T347494} to get rid of them. You can use 10.3.0.2/32 for the NTP VIP. [14:49:40] 10Traffic, 10SRE: Simplify maintenance of DNS/NTP hosts to reduce toil around reboots, reimages, and other work - https://phabricator.wikimedia.org/T347054 (10cmooney) Is there any reason we can't announce the "unicast" IPs in BGP too? I can't really see a good reason that any static routes are needed here. [14:55:24] 10Traffic, 10SRE: Simplify maintenance of DNS/NTP hosts to reduce toil around reboots, reimages, and other work - https://phabricator.wikimedia.org/T347054 (10ssingh) >>! In T347054#9203737, @cmooney wrote: > Is there any reason we can't announce the "unicast" IPs in BGP too? I can't really see a good reason... [15:05:04] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 2 others: Migrate internal traffic to k8s - https://phabricator.wikimedia.org/T333120 (10JMeybohm) [15:05:35] 10Traffic, 10Abstract Wikipedia team, 10MW-on-K8s, 10SRE, and 4 others: Migrate functions-orchestrator service to mw-api-int - https://phabricator.wikimedia.org/T347397 (10JMeybohm) 05In progress→03Resolved Direct access to mw-api is forbidden now. wikifunctions still working [16:14:51] 10Traffic, 10MW-on-K8s, 10SRE, 10Wikidata, and 2 others: Serve Wikidata traffic via Kubernetes - https://phabricator.wikimedia.org/T347493 (10Jdforrester-WMF) Possibly now solved by https://gerrit.wikimedia.org/r/c/operations/puppet/+/961351 ? [17:02:36] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host ncredir6002.drmrs.wmnet with OS bookworm [17:49:55] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host ncredir6002.drmrs.wmnet with OS bookworm completed: - ncredir6002 (**PASS**) - Downtimed on Ici... [18:20:59] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host ncredir6001.drmrs.wmnet with OS bookworm [18:22:59] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10BCornwall) [18:23:16] 10Traffic, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install cp11[00-15] - https://phabricator.wikimedia.org/T342159 (10BBlack) [19:10:08] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host ncredir6001.drmrs.wmnet with OS bookworm completed: - ncredir6001 (**PASS**) - Downtimed on Ici... [19:23:34] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10BCornwall) [19:26:34] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host ncredir4002.ulsfo.wmnet with OS bookworm [20:10:17] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host ncredir4002.ulsfo.wmnet with OS bookworm completed: - ncredir4002 (**PASS**) - Downtimed on Ici... [20:16:24] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host ncredir4001.ulsfo.wmnet with OS bookworm [20:54:33] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host ncredir4001.ulsfo.wmnet with OS bookworm completed: - ncredir4001 (**PASS**) - Downtimed on Ici... [20:58:16] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10BCornwall) [21:01:36] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host ncredir5002.eqsin.wmnet with OS bookworm [22:23:06] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host ncredir5002.eqsin.wmnet with OS bookworm completed: - ncredir5002 (**PASS**) - Downtimed on Ici... [22:25:30] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10BCornwall)