[11:51:32] 06Traffic, 06Data Products, 06Data-Engineering, 10Observability-Logging, 13Patch-For-Review: Install new Benthos instance on cp hosts - https://phabricator.wikimedia.org/T358109#9583090 (10Fabfur) [11:53:35] 06Traffic: Generate netmapper file for abuse requestctl ipblocks - https://phabricator.wikimedia.org/T355158#9583094 (10Fabfur) 05Open→03Declined [14:04:39] 06Traffic, 06Data-Persistence, 06SRE, 10SRE-swift-storage, and 5 others: Change default image thumbnail size - https://phabricator.wikimedia.org/T355914#9583645 (10Ganesha811) @Jdlrobson thanks for laying out this plan, and thank you to all the WMF staff who are thinking about how best to do this! It looks... [14:53:44] 06Traffic, 06Data-Persistence, 06SRE, 10SRE-swift-storage, and 5 others: Change default image thumbnail size - https://phabricator.wikimedia.org/T355914#9583869 (10Jdlrobson) [14:56:08] 10netops, 06DBA, 06Infrastructure-Foundations, 06SRE, 10ops-codfw: Migrate servers in codfw rack B6 from asw-b6-codfw to lsw1-b6-codfw - https://phabricator.wikimedia.org/T355871#9583876 (10JMeybohm) [15:46:02] 10netops, 06DBA, 06Infrastructure-Foundations, 06SRE: Migrate servers in codfw rack B3 from asw-b3-codfw to lsw1-b3-codfw - https://phabricator.wikimedia.org/T355870#9584222 (10cmooney) 05Open→03Resolved a:03cmooney [15:46:12] 10netops, 06Infrastructure-Foundations, 06SRE, 10ops-codfw: Migrate hosts from codfw row A/B ASW to new LSW devices - https://phabricator.wikimedia.org/T355544#9584224 (10cmooney) [15:49:39] 10netops, 06DBA, 06Infrastructure-Foundations, 06SRE, 10ops-codfw: Migrate servers in codfw rack B6 from asw-b6-codfw to lsw1-b6-codfw - https://phabricator.wikimedia.org/T355871#9584239 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=1f99f40e-0648-48d6-a40a-a3ebae9e7b2b) set by cmoon... [16:04:41] 10netops, 06DBA, 06Infrastructure-Foundations, 06SRE, 10ops-codfw: Migrate servers in codfw rack B6 from asw-b6-codfw to lsw1-b6-codfw - https://phabricator.wikimedia.org/T355871#9584297 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=691919af-8b8a-4f2d-b390-eea3c6a54f5c) set by cmoon... [16:12:29] 10netops, 06DBA, 06Infrastructure-Foundations, 06SRE, 10ops-codfw: Migrate servers in codfw rack B6 from asw-b6-codfw to lsw1-b6-codfw - https://phabricator.wikimedia.org/T355871#9584311 (10cmooney) Works completed, all servers moved to the new switch and back responding to ping now. No issues. [16:44:36] 06Traffic, 13Patch-For-Review: Simplify maintenance of DNS/NTP hosts to reduce toil around reboots, reimages, and other work - https://phabricator.wikimedia.org/T347054#9584436 (10ssingh) Status as of today: we are now managing `authdns-update` state, that is, the list of hosts in `/etc/wikimedia-authdns.conf`... [18:21:24] 06Traffic, 13Patch-For-Review: Simplify maintenance of DNS/NTP hosts to reduce toil around reboots, reimages, and other work - https://phabricator.wikimedia.org/T347054#9584958 (10Volans) Even from itself? As in what happens if an operator runs `authdns-update` on a depooled host? [18:27:11] 06Traffic, 13Patch-For-Review: Simplify maintenance of DNS/NTP hosts to reduce toil around reboots, reimages, and other work - https://phabricator.wikimedia.org/T347054#9584973 (10Volans) Do we need to change anything on the `sre.dns.netbox` cookbook? It currently runs: ` cd {git} && utils/deploy-check.py -g {... [18:29:57] 06Traffic, 13Patch-For-Review: Simplify maintenance of DNS/NTP hosts to reduce toil around reboots, reimages, and other work - https://phabricator.wikimedia.org/T347054#9584989 (10ssingh) >>! In T347054#9584958, @Volans wrote: > Even from itself? As in what happens if an operator runs `authdns-update` on a dep... [18:31:44] 06Traffic, 13Patch-For-Review: Simplify maintenance of DNS/NTP hosts to reduce toil around reboots, reimages, and other work - https://phabricator.wikimedia.org/T347054#9584997 (10ssingh) >>! In T347054#9584973, @Volans wrote: > Do we need to change anything on the `sre.dns.netbox` cookbook? > It currently run... [18:33:19] 06Traffic, 13Patch-For-Review: Simplify maintenance of DNS/NTP hosts to reduce toil around reboots, reimages, and other work - https://phabricator.wikimedia.org/T347054#9585007 (10Volans) >>! In T347054#9584989, @ssingh wrote: >>>! In T347054#9584958, @Volans wrote: >> Even from itself? As in what happens if a... [18:35:28] 06Traffic, 13Patch-For-Review: Simplify maintenance of DNS/NTP hosts to reduce toil around reboots, reimages, and other work - https://phabricator.wikimedia.org/T347054#9585010 (10ssingh) >>! In T347054#9585007, @Volans wrote: >>>! In T347054#9584989, @ssingh wrote: >>>>! In T347054#9584958, @Volans wrote: >>>... [19:23:29] 06Traffic, 06Data-Engineering-Radar: Lock-in Varnish and VarnishKafka versions - https://phabricator.wikimedia.org/T304617#9585226 (10BCornwall) 05Open→03In progress a:03CDobbins [20:14:08] 06Traffic, 06DC-Ops, 06SRE, 10ops-eqiad: Decommission task for old cp hosts (cp1075-1090) - https://phabricator.wikimedia.org/T352253#9585418 (10dr0ptp4kt) @bking , @RKemper , and I met today. @bking has an action on this here ticket (@bking LMK in case I need to chime in on anything!). Thanks! [20:16:12] 06Traffic, 06DC-Ops, 06SRE, 10ops-eqiad: Decommission task for old cp hosts (cp1075-1090) - https://phabricator.wikimedia.org/T352253#9585422 (10bking) @wiki_willy I'm going to take over this work from @dr0ptp4kt . l'll make a phab task with the data you requested shortly. [20:33:09] 06Traffic, 06Data-Persistence, 06SRE, 10SRE-swift-storage, and 5 others: Change default image thumbnail size - https://phabricator.wikimedia.org/T355914#9585432 (10JScherer-WMF) I spoke with @DTorsani-WMF, @RHo and others on the WMF design team on this. Assuming the change only affects thumbnails that are... [22:00:28] 06Traffic, 06DC-Ops, 10SRE-swift-storage, 10ops-codfw, 10ops-eqiad: Reimage cookbook on new eqiad hosts stuck at PXE booting - https://phabricator.wikimedia.org/T350179#9585648 (10ssingh)