[07:48:51] 10netops, 10Infrastructure-Foundations, 10SRE: Juniper ZTP fails on certain devices due to DHCP binding on management router - https://phabricator.wikimedia.org/T345273 (10ayounsi) FYI there is now a pending diff for: ` [edit forwarding-options dhcp-relay] + /* T337345 */ + forward-snooped-clients non-... [08:24:46] 10netops, 10Infrastructure-Foundations, 10SRE, 10netbox: Netbox Juniper report - https://phabricator.wikimedia.org/T306238 (10ayounsi) @jbond from Juniper, does it make sens? > “If the customer would like to use OIDC they enter in their token for us to use and authenticate. The vast majority of users sign... [09:23:34] 10Traffic: Package and deploy ATS 9.2.1 - https://phabricator.wikimedia.org/T339134 (10Vgutierrez) [09:23:42] 10netops, 10Infrastructure-Foundations, 10SRE, 10netbox: Netbox Juniper report - https://phabricator.wikimedia.org/T306238 (10jbond) >>! In T306238#9132987, @ayounsi wrote: > @jbond from Juniper, does it make sens? >> “If the customer would like to use OIDC they enter in their token for us to use and authe... [09:24:28] 10Traffic, 10Thumbor, 10Patch-For-Review: Cannot download large (2GB) files with 10Mbps or slower network due to ATS timeout - https://phabricator.wikimedia.org/T341755 (10Vgutierrez) 05Open→03Resolved a:03Vgutierrez `vgutierrez@carrot:~$ curl -o /dev/null https://upload.wikimedia.org/wikipedia/commons... [09:39:01] 10netops, 10Infrastructure-Foundations, 10SRE: Juniper ZTP fails on certain devices due to DHCP binding on management router - https://phabricator.wikimedia.org/T345273 (10cmooney) >>! In T345273#9132938, @ayounsi wrote: > FYI there is now a pending diff for: > ` > [edit forwarding-options dhcp-relay] > +... [10:22:59] 10Traffic, 10SRE-swift-storage, 10Thumbor: Cache thumbs in our caching infrastructure (e.g. ATS) - https://phabricator.wikimedia.org/T345334 (10MatthewVernon) [10:23:55] 10Traffic, 10SRE-swift-storage, 10Thumbor: Cache thumbs in our caching infrastructure (e.g. ATS) - https://phabricator.wikimedia.org/T345334 (10MatthewVernon) [I spoke to @KOfori about this, and they suggested opening a phab task tagged traffic was the best next step] [11:01:41] 10Traffic, 10SRE, 10SRE-swift-storage, 10Thumbor: Cache thumbs in our caching infrastructure (e.g. ATS) - https://phabricator.wikimedia.org/T345334 (10Vgutierrez) Happy to provide assistance and guidance if needed but caching is technically controlled by the backend services and not by the CDN. the CDN imp... [11:27:03] 10netops, 10Infrastructure-Foundations, 10SRE: Juniper ZTP fails on certain devices due to DHCP binding on management router - https://phabricator.wikimedia.org/T345273 (10cmooney) 05Open→03Resolved [11:27:08] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-tools, 10Patch-For-Review: Setup zero touch provisioning (ZTP) for network devices - https://phabricator.wikimedia.org/T336485 (10cmooney) [11:44:49] hi traffic, I would like to put a service into lvs_setup - is now a good time? [11:49:24] vgutierrez / fabfur ^ [11:51:21] sure [11:51:39] Do you have a CR? [11:53:50] https://gerrit.wikimedia.org/r/c/operations/puppet/+/954003 [12:03:54] nice, lvs1019 & lvs1020 will be the impacted LVS [12:04:42] yep, thanks! [12:05:27] target lvs1020 first please [12:06:50] sure thing [12:09:45] vgutierrez: hmm, the docs (https://wikitech.wikimedia.org/wiki/LVS#Add_a_new_load_balanced_service - Configure the load balancers) say I should disable puppet on lvs, then merge the patch, then enable and run puppet on all LVS. That does not seem to make much sense [12:12:58] why not? [12:15:03] if you merge without disabling and you don't immediately proceed is going to trigger some alerts jayme [12:15:24] hence the recommendation [12:16:12] ah, okay. But if I'm doing it right away it shouldn't make any difference, right? So running puppet right afert puppet-merge I mean [12:17:11] running puppet and restarting LVS [12:17:19] pybal sorry [12:18:32] yeah [12:19:19] restarting pybal must be done more or less immediately after running puppet on all LVS IIUC [12:22:21] yep [12:22:29] Otherwise you'll get some alerts as well [12:30:42] all good I think. Thanks! [12:40:40] yep, all good. cheers [13:08:51] 10netops, 10Infrastructure-Foundations: Adjust routing policy to increase SSH session speed from East Asia to toolforge - https://phabricator.wikimedia.org/T334530 (10ayounsi) 05Open→03Resolved Rolled everywhere, another example, cr1-codfw: `name=before Prefix Nexthop MED Lclpref AS path... [13:32:21] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host doh2002.wikimedia.org with OS bookworm [13:56:26] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Add per-output queue monitoring for Juniper network devices - https://phabricator.wikimedia.org/T326322 (10ayounsi) We have data https://grafana.wikimedia.org/d/iUATvNzSz/network-queues ! And a doc: https://wikitech.wikimedia.org/wiki/Netwo... [14:17:53] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host doh2002.wikimedia.org with OS bookworm completed: - doh2002 (**PASS**) - Downtimed on Icinga/Al... [14:18:42] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10ssingh) [14:29:22] I'm cleaning out cruft related to now-removed stretch: In modules/varnish/files/tests/37-docker-registry-cl-head.vtc the test pulls an image which no longer exists, that should probably be updated to a newer image (or the test removed) [14:30:22] I'd like to move two services using ingress to state production in the service catalogue - would it be okay for to merge this and do the lvs dance today? https://gerrit.wikimedia.org/r/c/operations/puppet/+/954067 [14:30:41] hnowlan: yeah go for it, hth if I can [14:30:52] moritzm: looking [14:31:33] sukhe: thanks! [15:50:44] all good afaics, thanks [15:54:20] thank you :) [16:45:32] 10netops, 10Infrastructure-Foundations, 10SRE, 10netbox: Netbox Juniper report - https://phabricator.wikimedia.org/T306238 (10ayounsi) Thanks, I submitted the on-boarding form, let's see what happens now. [19:45:51] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host doh6002.wikimedia.org with OS bookworm [19:55:15] sukhe: you doing any work on doh6002? [20:00:14] nevermind, I see b.rett is re-imaging [20:07:02] topranks: yep, brett is doing it [20:32:29] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host doh6002.wikimedia.org with OS bookworm completed: - doh6002 (**WARN**) - Downtimed on Icinga/Al... [20:46:31] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10BCornwall) [20:47:06] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host doh5002.wikimedia.org with OS bookworm [22:17:16] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host doh5002.wikimedia.org with OS bookworm completed: - doh5002 (**PASS**) - Downtimed on Icinga/Al... [22:17:45] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bookworm - https://phabricator.wikimedia.org/T342154 (10BCornwall)