[08:37:50] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-OnFire, and 2 others: Upgrade POPs asw to Junos 21 - https://phabricator.wikimedia.org/T316532 (10ayounsi) [11:21:05] 10CFSSL-PKI, 10Infrastructure-Foundations, 10Observability-Metrics: cfssl: export metricts from sql database - https://phabricator.wikimedia.org/T327768 (10jbond) [11:21:21] 10CFSSL-PKI, 10Infrastructure-Foundations, 10Observability-Metrics: cfssl: export metricts from sql database - https://phabricator.wikimedia.org/T327768 (10jbond) p:05Triage→03Medium [12:44:35] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-OnFire, 10Sustainability (Incident Followup): Upgrade POPs asw to Junos 21 - https://phabricator.wikimedia.org/T316532 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=795679e1-6c07-4196-8280-0cef7454587d) set by ayounsi@cumin1001 fo... [12:53:04] 10SRE-tools, 10Infrastructure-Foundations: sre.swift.roll-restart-reboot-proxies fails on thanos hosts, which lack nginx - https://phabricator.wikimedia.org/T327783 (10MatthewVernon) [13:00:13] 10SRE-tools, 10Infrastructure-Foundations: sre.swift.roll-restart-reboot-proxies fails on thanos hosts, which lack nginx - https://phabricator.wikimedia.org/T327783 (10MoritzMuehlenhoff) We ran into a similar issue in https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/875469, I'm inclined to simply respin... [14:08:18] 10SRE-tools, 10Infrastructure-Foundations: sre.swift.roll-restart-reboot-proxies fails on thanos hosts, which lack nginx - https://phabricator.wikimedia.org/T327783 (10MatthewVernon) I don't feel strongly, but I would like in the medium term for the thanos frontends and the swift frontends to be much more simi... [14:13:42] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-OnFire, 10Sustainability (Incident Followup): Upgrade POPs asw to Junos 21 - https://phabricator.wikimedia.org/T316532 (10ayounsi) [14:41:54] 10netops, 10Infrastructure-Foundations, 10SRE: Upgrade network devices to Junos 20+ - https://phabricator.wikimedia.org/T316539 (10ayounsi) [14:42:09] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-OnFire, 10Sustainability (Incident Followup): Upgrade POPs asw to Junos 21 - https://phabricator.wikimedia.org/T316532 (10ayounsi) 05Open→03Resolved a:03ayounsi All done! fpc2 didn't like the first "blank" reboot and required a power cycle using... [16:31:22] XioNoX, topranks: I've tried to add some notes on renumbering of things that came to mind in the pad. It's totally non-exhaustive :D [16:33:42] volans: thanks will have a look :) [19:00:18] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: codfw: Relocate servers to make space for new switches in rowA and rowB - https://phabricator.wikimedia.org/T326564 (10Papaul) We discuss this during today's meeting, we are going to put 1 spine in A1 and the other spine in A8. When we upgrade ro... [19:10:30] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: codfw: Relocate servers to make space for new switches in rowA and rowB - https://phabricator.wikimedia.org/T326564 (10Papaul) @BBlack Do you think you will have time for us to move lvs2007 this Thursday the 26th at 9:45am CT 2:45 pm UTC? Tha... [19:15:22] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: codfw: Relocate servers to make space for new switches in rowA and rowB - https://phabricator.wikimedia.org/T326564 (10BBlack) @Papaul - I can't make that slot for LVS, I have meetings a bit later that might get run over. @ssingh might be able t... [19:17:42] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: codfw: Relocate servers to make space for new switches in rowA and rowB - https://phabricator.wikimedia.org/T326564 (10ssingh) >>! In T326564#8554616, @BBlack wrote: > @Papaul - I can't make that slot for LVS, I have meetings a bit later that mig... [19:20:15] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: codfw: Relocate servers to make space for new switches in rowA and rowB - https://phabricator.wikimedia.org/T326564 (10Papaul) @BBlack @ssingh thank you. So the process is depool the server, power it down I move it and power it back no changes i... [20:04:53] 10Puppet, 10Analytics-Radar, 10Infrastructure-Foundations: modules/udp2log/manifests/instance/monitoring.pp has unreachable code - https://phabricator.wikimedia.org/T152104 (10Dzahn) It's been years since my last comment that it's been years.