[00:26:41] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-tools: Setup zero touch provisioning (ZTP) for network devices - https://phabricator.wikimedia.org/T336485 (10Papaul) @Volans I checked on the dhcp server in the dhcp config file there is an option that is missing ' for configuration we have now is : `... [06:16:24] 10netops, 10Infrastructure-Foundations: Peering: prefer primary IXP for direcly connected networks - https://phabricator.wikimedia.org/T338201 (10ayounsi) p:05Triage→03High [09:20:51] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-tools: Setup zero touch provisioning (ZTP) for network devices - https://phabricator.wikimedia.org/T336485 (10Volans) @Papaul that's set already using option 66 that is a standard option, see `tftp-server-name` in `automation.conf`. And the previous comm... [12:43:34] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Users management on SONiC - https://phabricator.wikimedia.org/T338028 (10MoritzMuehlenhoff) It would appear to me that given that we have the full freedom of a Debian-based OS here (after all next to the FLOSS aspect the major perk of SONiC... [12:59:36] I've started an incident doc about yesterday's wikireplicas outage with respect to pybal/LVS - https://docs.google.com/document/d/1yo0pCpOSQ4waAPtWU06xMQUs7UMhM5wnAj2U4NOObj8/edit if anyone can add any more detail to the timeline or cause, that would be great. [13:00:48] 10Traffic, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar): GeoIP mapping experiments - https://phabricator.wikimedia.org/T332024 (10Ottomata) Temporarily disabling hadoop ingestion and canary events. The `ctx` field needs a schema. [13:08:18] 10Traffic, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: GeoIP mapping experiments - https://phabricator.wikimedia.org/T332024 (10Krinkle) [13:15:54] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-tools: Setup zero touch provisioning (ZTP) for network devices - https://phabricator.wikimedia.org/T336485 (10Papaul) @Volans ok i didn't look in the automation.conf file for the option 66 thanks [13:25:17] vgutierrez: o/ lemme know if https://gerrit.wikimedia.org/r/c/operations/puppet/+/924506 is ok to deploy (not urgent, event during the next days) [13:35:16] 10netops, 10Infrastructure-Foundations, 10SRE: Migrate row E/F network aggregation to dedicated Spine switches - https://phabricator.wikimedia.org/T322937 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=cc1782a9-2e71-42dd-977e-e3f886320173) set by cmooney@cumin1001 for 0:30:00 on 3 host(s... [14:06:35] 10netops, 10Infrastructure-Foundations, 10SRE: Migrate row E/F network aggregation to dedicated Spine switches - https://phabricator.wikimedia.org/T322937 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=ff119eb0-a98a-450e-a695-debe4cc1b997) set by cmooney@cumin1001 for 0:30:00 on 3 host(s... [14:16:12] 10Traffic, 10DC-Ops, 10SRE, 10ops-codfw, 10Patch-For-Review: Q4:rack/setup/install lvs2011, lvs2012, lvs2013, lvs2014 - https://phabricator.wikimedia.org/T326767 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host lvs2013.codfw.wmnet with OS bullseye [14:37:48] elukey: yeah I think so [14:41:08] vgutierrez: I was thinking to disable puppet on all cp nodes with vk, enable on one, run puppet and test etc.. [14:41:16] then re-enable on all [14:51:30] 10Traffic, 10DC-Ops, 10SRE, 10ops-codfw, 10Patch-For-Review: Q4:rack/setup/install lvs2011, lvs2012, lvs2013, lvs2014 - https://phabricator.wikimedia.org/T326767 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host lvs2013.codfw.wmnet with OS bullseye completed:... [15:09:59] ack so I think I am ready, ok to proceed with https://gerrit.wikimedia.org/r/c/operations/puppet/+/924506 ? [15:10:07] 1) disable puppet on c:varnishkafka [15:10:31] 2) run on a ulsfo node, and check the new unit (I think that all vk instances will restart due to the change in their unit) [15:10:38] 3) enable pupept [15:10:40] *puppet [17:06:27] 10Traffic, 10Maps, 10Product-Infrastructure-Team-Backlog-Deprecated, 10SRE, and 2 others: Support maps serving for affiliate sites via an allow list - https://phabricator.wikimedia.org/T261694 (10Galessandroni) Hi. In Vikidia (an European Wikipedia for kids) we have several problem to enroll it in Wikimedi... [20:09:40] 10Traffic, 10SRE, 10Wikibase Product Platform: Beta wikidata rejects PATCH requests - https://phabricator.wikimedia.org/T336659 (10WMDE-leszek) 05Open→03Resolved a:03WMDE-leszek This seems to be resolved. [21:38:49] sukhe: [nit] FYI 'enable-puppet "CR 926611" && run-puppet-agent -q' can be shortened and run slightly more atomically with: 'run-puppet-agent -q -e "CR 926611"' :)