[09:33:50] 06Traffic, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 3 others: Reimage parse* hosts as kubernetes nodes - https://phabricator.wikimedia.org/T358752#9599659 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host parse2019.codfw.wmnet with OS bullseye comp... [09:42:25] 06Traffic, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 3 others: Reimage parse* hosts as kubernetes nodes - https://phabricator.wikimedia.org/T358752#9599689 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host parse2020.codfw.wmnet with OS bullseye [10:21:20] 06Traffic, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 3 others: Reimage parse* hosts as kubernetes nodes - https://phabricator.wikimedia.org/T358752#9599881 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host parse2020.codfw.wmnet with OS bullseye comp... [11:13:48] 06Traffic, 06Infrastructure-Foundations, 06SRE: Slowly ramping up traffic to the Brazil data center (magru) and related geo-maps - https://phabricator.wikimedia.org/T359054#9599986 (10ayounsi) I'd recommend to start by turning up a small country/region on that continent (Uruguay/Paraguay for example), ideall... [11:23:03] 06Traffic, 10MW-on-K8s, 06Release-Engineering-Team, 06SRE, 06serviceops: Move 60% of mediawiki external requests to mw on k8s - https://phabricator.wikimedia.org/T357508#9600053 (10Clement_Goubert) [14:16:55] 06Traffic, 06Data Products, 06Data-Engineering, 10Observability-Logging: Check statsv and eventlogging VarnishKafka instances - https://phabricator.wikimedia.org/T359178 (10Fabfur) [14:18:41] 06Traffic, 06Data Products, 06Data-Engineering, 10Observability-Logging: Create a new temporary topic on Kafka jumbo cluster to test Benthos - https://phabricator.wikimedia.org/T359172#9601109 (10brouberol) ` brouberol@kafka-jumbo1010:~$ kafka topics --create --topic webrequest_text_test --replication-fact... [14:18:50] 06Traffic, 06Data Products, 06Data-Engineering, 10Observability-Logging, 13Patch-For-Review: Install new Benthos instance on cp hosts - https://phabricator.wikimedia.org/T358109#9601115 (10brouberol) [14:19:12] 06Traffic, 06Data Products, 06Data-Engineering, 10Observability-Logging: Create a new temporary topic on Kafka jumbo cluster to test Benthos - https://phabricator.wikimedia.org/T359172#9601113 (10brouberol) 05Open→03Resolved All done! ` brouberol@kafka-jumbo1010:~$ kafka topics --describe --topic webr... [14:38:11] 06Traffic, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 3 others: Create parsoid mediawiki deployment and migrate parsoid-php.discovery.wmnet traffic to it - https://phabricator.wikimedia.org/T357392#9601203 (10akosiaris) [14:41:17] 06Traffic, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 3 others: Create parsoid mediawiki deployment and migrate parsoid-php.discovery.wmnet traffic to it - https://phabricator.wikimedia.org/T357392#9601211 (10akosiaris) We at [~50% mw-parsoid](https://grafana.wikimedia.org/d/VTCkm29Wz/envoy-telemetr... [15:03:00] 06Traffic, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 3 others: Reimage parse* hosts as kubernetes nodes - https://phabricator.wikimedia.org/T358752#9601359 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host mw1357.eqiad.wmnet with OS bullseye [15:03:31] 06Traffic, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 3 others: Reimage parse* hosts as kubernetes nodes - https://phabricator.wikimedia.org/T358752#9601362 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host mw1356.eqiad.wmnet with OS bullseye [15:06:49] 06Traffic, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 3 others: Reimage parse* hosts as kubernetes nodes - https://phabricator.wikimedia.org/T358752#9601375 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host parse1003.eqiad.wmnet with OS bullseye [15:07:22] 06Traffic, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 3 others: Reimage parse* hosts as kubernetes nodes - https://phabricator.wikimedia.org/T358752#9601378 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host parse1004.eqiad.wmnet with OS bullseye [15:38:59] 10netops, 06DBA, 06Infrastructure-Foundations, 06SRE, 10ops-codfw: Migrate servers in codfw rack B8 from asw-b8-codfw to lsw1-b8-codfw - https://phabricator.wikimedia.org/T355873#9601519 (10Joe) [15:39:50] 06Traffic, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 3 others: Reimage parse* hosts as kubernetes nodes - https://phabricator.wikimedia.org/T358752#9601524 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host mw1357.eqiad.wmnet with OS bullseye complet... [15:41:33] 06Traffic, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 3 others: Reimage parse* hosts as kubernetes nodes - https://phabricator.wikimedia.org/T358752#9601529 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host mw1356.eqiad.wmnet with OS bullseye complet... [15:44:25] 06Traffic, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 3 others: Reimage parse* hosts as kubernetes nodes - https://phabricator.wikimedia.org/T358752#9601543 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host parse1003.eqiad.wmnet with OS bullseye comp... [15:46:55] 06Traffic, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 3 others: Reimage parse* hosts as kubernetes nodes - https://phabricator.wikimedia.org/T358752#9601575 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host parse1004.eqiad.wmnet with OS bullseye comp... [15:55:14] 10netops, 06DBA, 06Infrastructure-Foundations, 06SRE, 10ops-codfw: Migrate servers in codfw rack B8 from asw-b8-codfw to lsw1-b8-codfw - https://phabricator.wikimedia.org/T355873#9601663 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=19e5ce18-f2ba-4d9e-a80a-2c957c2eecad) set by cmoon... [15:57:12] 06Traffic, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 3 others: Reimage parse* hosts as kubernetes nodes - https://phabricator.wikimedia.org/T358752#9601669 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host parse1002.eqiad.wmnet with OS bullseye [15:58:24] 10netops, 06DBA, 06Infrastructure-Foundations, 06SRE, 10ops-codfw: Migrate servers in codfw rack B8 from asw-b8-codfw to lsw1-b8-codfw - https://phabricator.wikimedia.org/T355873#9601682 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=f241631d-4830-4ac7-b5c1-29790ccbb916) set by cmoon... [15:58:48] 06Traffic, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 3 others: Reimage parse* hosts as kubernetes nodes - https://phabricator.wikimedia.org/T358752#9601678 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host parse1005.eqiad.wmnet with OS bullseye [15:59:40] 06Traffic, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 3 others: Reimage parse* hosts as kubernetes nodes - https://phabricator.wikimedia.org/T358752#9601689 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host parse1006.eqiad.wmnet with OS bullseye [16:00:33] 06Traffic, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 3 others: Reimage parse* hosts as kubernetes nodes - https://phabricator.wikimedia.org/T358752#9601695 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host parse1007.eqiad.wmnet with OS bullseye [16:01:25] 06Traffic, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 3 others: Reimage parse* hosts as kubernetes nodes - https://phabricator.wikimedia.org/T358752#9601700 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host parse1008.eqiad.wmnet with OS bullseye [16:05:05] 06Traffic, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 3 others: Reimage parse* hosts as kubernetes nodes - https://phabricator.wikimedia.org/T358752#9601738 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host parse1009.eqiad.wmnet with OS bullseye [16:13:54] fabfur: we're done in rack b8 if you want to repool dns2004 [16:13:55] thanks! [16:14:07] topranks: thanks! [16:14:11] nice job on another clean one :) [16:14:22] (fabfu.r is out now, so will repool) [16:15:01] tanks topranks [16:15:08] *thanks [16:15:23] I'll repool it later if you don't have any objection [16:15:29] 10netops, 06DBA, 06Infrastructure-Foundations, 06SRE, 10ops-codfw: Migrate servers in codfw rack B8 from asw-b8-codfw to lsw1-b8-codfw - https://phabricator.wikimedia.org/T355873#9601818 (10cmooney) All links moved without problem, servers back online and responding to ping now. [16:15:56] yeah there is no rush [16:16:13] fabfur: get off the internet [16:16:50] we should also think about moving the BGP peering from host -> CR to host -> switch at some stage, will discuss again [16:17:21] happy to [16:31:17] 06Traffic, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 3 others: Reimage parse* hosts as kubernetes nodes - https://phabricator.wikimedia.org/T358752#9602016 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host parse1002.eqiad.wmnet with OS bullseye comp... [16:32:59] 06Traffic, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 3 others: Reimage parse* hosts as kubernetes nodes - https://phabricator.wikimedia.org/T358752#9602044 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host parse1007.eqiad.wmnet with OS bullseye comp... [16:34:32] 06Traffic, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 3 others: Reimage parse* hosts as kubernetes nodes - https://phabricator.wikimedia.org/T358752#9602059 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host parse1008.eqiad.wmnet with OS bullseye comp... [16:36:24] 06Traffic, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 3 others: Reimage parse* hosts as kubernetes nodes - https://phabricator.wikimedia.org/T358752#9602072 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host parse1009.eqiad.wmnet with OS bullseye comp... [16:39:11] 06Traffic, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 3 others: Reimage parse* hosts as kubernetes nodes - https://phabricator.wikimedia.org/T358752#9602077 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host parse1006.eqiad.wmnet with OS bullseye exec... [16:39:52] 06Traffic, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 3 others: Reimage parse* hosts as kubernetes nodes - https://phabricator.wikimedia.org/T358752#9602087 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host parse1006.eqiad.wmnet with OS bullseye [16:41:23] 06Traffic, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 3 others: Reimage parse* hosts as kubernetes nodes - https://phabricator.wikimedia.org/T358752#9602092 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host parse1005.eqiad.wmnet with OS bullseye comp... [16:43:45] 06Traffic, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 3 others: Reimage parse* hosts as kubernetes nodes - https://phabricator.wikimedia.org/T358752#9602107 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host parse1010.eqiad.wmnet with OS bullseye [16:44:32] 06Traffic, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 3 others: Reimage parse* hosts as kubernetes nodes - https://phabricator.wikimedia.org/T358752#9602112 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host parse1011.eqiad.wmnet with OS bullseye [16:45:13] 06Traffic, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 3 others: Reimage parse* hosts as kubernetes nodes - https://phabricator.wikimedia.org/T358752#9602116 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host parse1012.eqiad.wmnet with OS bullseye [16:45:49] 06Traffic, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 3 others: Reimage parse* hosts as kubernetes nodes - https://phabricator.wikimedia.org/T358752#9602117 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host parse1013.eqiad.wmnet with OS bullseye [16:46:25] 06Traffic, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 3 others: Reimage parse* hosts as kubernetes nodes - https://phabricator.wikimedia.org/T358752#9602123 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host parse1014.eqiad.wmnet with OS bullseye [17:01:45] sukhe: I'm gonna start the move for lvs2011 unless there is a reason to hold off? [17:01:57] topranks: go for it, I think we are good [17:06:14] 06Traffic, 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Move lvs2011 from private1-a-codfw (row) to private1-a2-codfw (rack) vlan - https://phabricator.wikimedia.org/T352920#9602285 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=6010131f-b756-49c6-8082-62badba41... [17:06:44] topranks: let me know when you start :) [17:07:44] ok you did, gl! [17:08:18] 06Traffic, 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Move lvs2011 from private1-a-codfw (row) to private1-a2-codfw (rack) vlan - https://phabricator.wikimedia.org/T352920#9602297 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=c0fe6035-a553-49f8-8b94-3d7840e51... [17:09:21] 10netops, 06Infrastructure-Foundations, 06SRE: Icinga BFD check failing - https://phabricator.wikimedia.org/T359198 (10cmooney) p:05Triage→03Medium [17:09:44] sukhe: cool I've downtimed the host so I'm about to log on and disable puppet/pybal now [17:09:49] ok [17:11:05] done [17:11:56] 06Traffic, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 3 others: Reimage parse* hosts as kubernetes nodes - https://phabricator.wikimedia.org/T358752#9602348 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host parse1006.eqiad.wmnet with OS bullseye comp... [17:20:02] 10netops, 06Infrastructure-Foundations, 06SRE: Icinga BFD check failing - https://phabricator.wikimedia.org/T359198#9602418 (10andrea.denisse) a:03andrea.denisse [17:21:53] 06Traffic, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 3 others: Reimage parse* hosts as kubernetes nodes - https://phabricator.wikimedia.org/T358752#9602443 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host parse1010.eqiad.wmnet with OS bullseye comp... [17:23:41] 06Traffic, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 3 others: Reimage parse* hosts as kubernetes nodes - https://phabricator.wikimedia.org/T358752#9602458 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host parse1012.eqiad.wmnet with OS bullseye comp... [17:24:41] 06Traffic, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 3 others: Reimage parse* hosts as kubernetes nodes - https://phabricator.wikimedia.org/T358752#9602463 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host parse1011.eqiad.wmnet with OS bullseye comp... [17:26:28] 10netops, 06Infrastructure-Foundations, 06SRE: Icinga BFD check failing - https://phabricator.wikimedia.org/T359198#9602476 (10dcaro) Affecting also the cloudswitches {F42399814} [17:27:40] 10netops, 06Infrastructure-Foundations, 06SRE: Icinga BFD check failing - https://phabricator.wikimedia.org/T359198#9602493 (10dcaro) It's gone now :) [17:29:30] 10netops, 06Infrastructure-Foundations, 06SRE: Icinga BFD check failing - https://phabricator.wikimedia.org/T359198#9602506 (10andrea.denisse) [17:30:03] 10netops, 06Infrastructure-Foundations, 06SRE: Icinga BFD check failing - https://phabricator.wikimedia.org/T359198#9602503 (10fgiunchedi) I've bandaided the issue on alert2001, we'll need a more proper fix: ` # download-mibs # cd /var/lib/snmp && ln -s ../mibs ` [17:30:51] 10netops, 06Infrastructure-Foundations, 06SRE: Icinga BFD check failing - https://phabricator.wikimedia.org/T359198#9602522 (10Dzahn) There is this package on the alert hosts: ` ii snmp-mibs-downloader 1.2 all install and manage Management Information B... [17:32:46] 10netops, 06Infrastructure-Foundations, 06SRE: Icinga BFD check failing - https://phabricator.wikimedia.org/T359198#9602540 (10cmooney) >>! In T359198#9602522, @Dzahn wrote: > I guess the snmp-mibs-downloader just has to be automated to download stuff? Yeah on it's own that package installs but doesn't do a... [17:44:50] 06Traffic, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 3 others: Reimage parse* hosts as kubernetes nodes - https://phabricator.wikimedia.org/T358752#9602582 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host parse1015.eqiad.wmnet with OS bullseye [17:45:23] 06Traffic, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 3 others: Reimage parse* hosts as kubernetes nodes - https://phabricator.wikimedia.org/T358752#9602583 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host parse1016.eqiad.wmnet with OS bullseye [17:46:42] 06Traffic, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 3 others: Reimage parse* hosts as kubernetes nodes - https://phabricator.wikimedia.org/T358752#9602586 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host parse1017.eqiad.wmnet with OS bullseye [17:48:05] 06Traffic, 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Move lvs2011 from private1-a-codfw (row) to private1-a2-codfw (rack) vlan - https://phabricator.wikimedia.org/T352920#9602589 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmooney@cumin1002 for host... [17:48:33] 06Traffic, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 3 others: Reimage parse* hosts as kubernetes nodes - https://phabricator.wikimedia.org/T358752#9602590 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host parse1018.eqiad.wmnet with OS bullseye [17:49:10] 06Traffic, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 3 others: Reimage parse* hosts as kubernetes nodes - https://phabricator.wikimedia.org/T358752#9602595 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host parse1019.eqiad.wmnet with OS bullseye [17:53:37] 10netops, 06Infrastructure-Foundations, 06SRE: Icinga BFD check failing - https://phabricator.wikimedia.org/T359198#9602613 (10Dzahn) Looks like it's: `man 1 download-mibs` `download-mibs --help` and the config is at `/etc/snmp-mibs-downloader/snmp-mibs-downloader.conf` which has some kind of "AUTOLOAD" c... [17:55:57] 10netops, 06Infrastructure-Foundations, 06SRE: Icinga BFD check failing - https://phabricator.wikimedia.org/T359198#9602620 (10cmooney) >>! In T359198#9602613, @Dzahn wrote: > Looks like it's: > > `man 1 download-mibs` > `download-mibs --help` > > and the config is at `/etc/snmp-mibs-downloader/snmp-mibs-... [18:07:10] 06Traffic, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 3 others: Reimage parse* hosts as kubernetes nodes - https://phabricator.wikimedia.org/T358752#9602700 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host parse1014.eqiad.wmnet with OS bullseye exec... [18:18:17] 06Traffic, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 3 others: Reimage parse* hosts as kubernetes nodes - https://phabricator.wikimedia.org/T358752#9602716 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host parse1015.eqiad.wmnet with OS bullseye comp... [18:20:24] 06Traffic, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 3 others: Reimage parse* hosts as kubernetes nodes - https://phabricator.wikimedia.org/T358752#9602717 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host parse1016.eqiad.wmnet with OS bullseye comp... [18:22:39] 06Traffic, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 3 others: Reimage parse* hosts as kubernetes nodes - https://phabricator.wikimedia.org/T358752#9602723 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host parse1019.eqiad.wmnet with OS bullseye comp... [18:23:36] topranks: reimage treating you well? [18:23:57] so far so good yeah [18:24:05] ok! [18:24:21] first puppet run was ok, it's rebooting now [18:24:26] connectivity looks ok I already tested [18:24:30] awesome, good signs already [18:25:12] 06Traffic, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 3 others: Reimage parse* hosts as kubernetes nodes - https://phabricator.wikimedia.org/T358752#9602725 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host parse1018.eqiad.wmnet with OS bullseye comp... [18:27:53] 06Traffic, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 3 others: Reimage parse* hosts as kubernetes nodes - https://phabricator.wikimedia.org/T358752#9602731 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host parse1017.eqiad.wmnet with OS bullseye comp... [18:30:58] 06Traffic, 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Move lvs2011 from private1-a-codfw (row) to private1-a2-codfw (rack) vlan - https://phabricator.wikimedia.org/T352920#9602746 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmooney@cumin1002 for host lvs... [18:31:39] 06Traffic, 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Move lvs2011 from private1-a-codfw (row) to private1-a2-codfw (rack) vlan - https://phabricator.wikimedia.org/T352920#9602756 (10cmooney) Reimage looks good, BGP up and lvs2011 handling traffic again: ` cmooney@cumin1002:~$ sud... [18:31:59] nice work topranks! [18:32:41] 06Traffic: Reimage one of each Traffic hosts before magru - https://phabricator.wikimedia.org/T359053#9602760 (10ssingh) @cmooney reimaged `lvs2011` today and it seems to have gone fine. Sharing this here in case we want to check `lvs' in the list above. [18:33:31] sukhe: yep all looking good! [19:06:28] 10netops, 06Infrastructure-Foundations, 06SRE: Re-IP hosts on codfw row A and B to new per-rack vlans/subnets - https://phabricator.wikimedia.org/T354869#9602910 (10cmooney) [19:06:37] 06Traffic, 10netops, 06Infrastructure-Foundations, 06SRE: Move lvs2011 from private1-a-codfw (row) to private1-a2-codfw (rack) vlan - https://phabricator.wikimedia.org/T352920#9602909 (10cmooney) 05Open→03Resolved [19:16:28] 10netops, 06SRE, 13Patch-For-Review, 10SRE Observability (FY2023/2024-Q3): Icinga BFD check failing - https://phabricator.wikimedia.org/T359198#9602958 (10andrea.denisse) [21:41:54] 06Traffic, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 3 others: Reimage parse* hosts as kubernetes nodes - https://phabricator.wikimedia.org/T358752#9603411 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host parse1014.eqiad.wmnet with OS bullseye [23:00:20] 06Traffic: Restarting fifo-log-demux should not restart nginx - https://phabricator.wikimedia.org/T355905#9603614 (10BCornwall) 05In progress→03Resolved Deployed! [23:02:11] 06Traffic, 06Content-Transform-Team, 10MW-on-K8s, 06SRE, and 3 others: Reimage parse* hosts as kubernetes nodes - https://phabricator.wikimedia.org/T358752#9603617 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host parse1014.eqiad.wmnet with OS bullseye exec... [23:02:56] 06Traffic: Fix ncredir nginx access log tailing - https://phabricator.wikimedia.org/T358281#9603618 (10BCornwall) 05In progress→03Invalid Works for me, seems it might have been some transient error. I'll re-open if it pops up again. [23:34:37] 10netops, 06DBA, 06Infrastructure-Foundations, 06SRE, 10ops-codfw: Migrate servers in codfw rack B8 from asw-b8-codfw to lsw1-b8-codfw - https://phabricator.wikimedia.org/T355873#9603722 (10cmooney) 05Open→03Resolved a:03cmooney [23:34:49] 10netops, 06Infrastructure-Foundations, 06SRE, 10ops-codfw: Migrate hosts from codfw row A/B ASW to new LSW devices - https://phabricator.wikimedia.org/T355544#9603724 (10cmooney) [23:35:26] 10netops, 06Infrastructure-Foundations, 06SRE, 10ops-codfw: Decom asw-a-codfw switch stack - https://phabricator.wikimedia.org/T358244#9603728 (10cmooney) [23:35:35] 10netops, 06Infrastructure-Foundations, 06SRE, 10ops-codfw: Migrate hosts from codfw row A/B ASW to new LSW devices - https://phabricator.wikimedia.org/T355544#9603729 (10cmooney) [23:35:51] 10netops, 06Infrastructure-Foundations, 06SRE, 10ops-codfw: Migrate hosts from codfw row A/B ASW to new LSW devices - https://phabricator.wikimedia.org/T355544#9603725 (10cmooney) 05Open→03Resolved a:03cmooney Closing task. Big thanks to all the SRE teams for the help and co-operation getting this o...