[00:48:30] (SystemdUnitFailed) firing: generate_os_reports.service on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:48:30] (SystemdUnitFailed) firing: generate_os_reports.service on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:19:22] 10CAS-SSO, 06Infrastructure-Foundations: Migrate CAS to Bookworm - https://phabricator.wikimedia.org/T357748#9586157 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by slyngshede@cumin1002 for host idp-test1003.wikimedia.org with OS bookworm [07:23:02] (SystemdUnitFailed) firing: (2) netbox_report_accounting_run.service on netbox1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:48:02] (SystemdUnitFailed) firing: (2) netbox_report_accounting_run.service on netbox1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:51:55] 10CAS-SSO, 06Infrastructure-Foundations, 13Patch-For-Review: Migrate CAS to Bookworm - https://phabricator.wikimedia.org/T357748#9586182 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by slyngshede@cumin1002 for host idp-test1003.wikimedia.org with OS bookworm completed: - idp-test1003 (... [08:23:02] (SystemdUnitFailed) firing: (2) netbox_report_accounting_run.service on netbox1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:48:02] (SystemdUnitFailed) firing: (2) netbox_report_accounting_run.service on netbox1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:27:34] 10SRE-tools, 06Data-Persistence, 06Infrastructure-Foundations, 13Patch-For-Review: Automation to change a server's vlan - https://phabricator.wikimedia.org/T350152#9586525 (10Clement_Goubert) @ayounsi I've drained `kubernetes2023.codfw.wmnet` for you to test the cookbook [10:27:52] 10SRE-tools, 06Data-Persistence, 06Infrastructure-Foundations, 06serviceops-radar, 13Patch-For-Review: Automation to change a server's vlan - https://phabricator.wikimedia.org/T350152#9586526 (10Clement_Goubert) [12:10:18] 10netops, 06DBA, 06Infrastructure-Foundations, 06SRE, 10ops-codfw: Migrate servers in codfw rack B6 from asw-b6-codfw to lsw1-b6-codfw - https://phabricator.wikimedia.org/T355871#9586779 (10cmooney) 05Open→03Resolved a:03cmooney Closing task - thanks all for the co-operation! [12:10:28] 10netops, 06Infrastructure-Foundations, 06SRE, 10ops-codfw: Migrate hosts from codfw row A/B ASW to new LSW devices - https://phabricator.wikimedia.org/T355544#9586782 (10cmooney) [12:14:39] 10netops, 06Infrastructure-Foundations, 06SRE, 10SRE-swift-storage, 10ops-codfw: Migrate servers in codfw rack B7 from asw-b7-codfw to lsw1-b7-codfw - https://phabricator.wikimedia.org/T355872#9586789 (10ops-monitoring-bot) Draining ganeti2032.codfw.wmnet of running VMs [12:48:30] (SystemdUnitFailed) firing: generate_os_reports.service on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:57:33] 10netops, 06Infrastructure-Foundations, 06SRE, 10ops-codfw: Decom asw-a-codfw switch stack - https://phabricator.wikimedia.org/T358244#9586848 (10cmooney) After a quick discussion on irc I think we can't wipe the config for every unit in the VC over ssh to the master. So probably easiest to do that via se... [14:30:38] 10CAS-SSO, 06Infrastructure-Foundations, 13Patch-For-Review: Move CAS to Java 17 - https://phabricator.wikimedia.org/T357749#9587160 (10SLyngshede-WMF) 05Open→03Resolved [14:30:41] 10CAS-SSO, 06Infrastructure-Foundations, 13Patch-For-Review: Migrate CAS to Bookworm - https://phabricator.wikimedia.org/T357748#9587161 (10SLyngshede-WMF) [14:50:05] 10netops, 06Infrastructure-Foundations, 06SRE, 10ops-codfw: Decom asw-a-codfw switch stack - https://phabricator.wikimedia.org/T358244#9587275 (10Jhancock.wm) @cmooney they're on the old asw switches. Let me know when you want to move them back to the new lsw. [16:01:48] 10netops, 06Infrastructure-Foundations, 06SRE, 10SRE-swift-storage, 10ops-codfw: Migrate servers in codfw rack B7 from asw-b7-codfw to lsw1-b7-codfw - https://phabricator.wikimedia.org/T355872#9587603 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=ab1a9b14-3187-4d52-a4f6-be3c445a8081... [16:02:42] 10netops, 06Infrastructure-Foundations, 06SRE, 10SRE-swift-storage, 10ops-codfw: Migrate servers in codfw rack B7 from asw-b7-codfw to lsw1-b7-codfw - https://phabricator.wikimedia.org/T355872#9587615 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=12cd3c2a-9d8e-4ba6-a42e-1faa167de80d... [16:10:45] 10netops, 06Infrastructure-Foundations, 06SRE, 10SRE-swift-storage, 10ops-codfw: Migrate servers in codfw rack B7 from asw-b7-codfw to lsw1-b7-codfw - https://phabricator.wikimedia.org/T355872#9587654 (10cmooney) All hosts moved sucessfully. Showing up on switch, macs learnt and all responding to ping a... [16:20:48] 10netops, 06Infrastructure-Foundations, 06SRE, 10SRE-swift-storage, 10ops-codfw: Migrate servers in codfw rack B7 from asw-b7-codfw to lsw1-b7-codfw - https://phabricator.wikimedia.org/T355872#9587682 (10MatthewVernon) thanos and ms swift clusters OK post-move, thank you! [16:48:30] (SystemdUnitFailed) firing: generate_os_reports.service on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:55:05] 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic, 13Patch-For-Review: Move lvs2012 from private1-b-codfw (row) to private1-b2-codfw (rack) vlan - https://phabricator.wikimedia.org/T352918#9588268 (10cmooney) Ok I abandoned my previous change as my git skills weren't up to it. Prepping a new patc... [18:13:21] 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic, 13Patch-For-Review: Move lvs2012 from private1-b-codfw (row) to private1-b2-codfw (rack) vlan - https://phabricator.wikimedia.org/T352918#9588323 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=2521164e-4d59-47cb-8d79-8dd925725... [18:40:42] 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic, 13Patch-For-Review: Move lvs2012 from private1-b-codfw (row) to private1-b2-codfw (rack) vlan - https://phabricator.wikimedia.org/T352918#9588429 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=1baa9b0f-d917-4da6-83db-5cb28b50c... [18:56:32] 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic, 13Patch-For-Review: Move lvs2012 from private1-b-codfw (row) to private1-b2-codfw (rack) vlan - https://phabricator.wikimedia.org/T352918#9588508 (10cmooney) [18:57:31] 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic, 13Patch-For-Review: Move lvs2012 from private1-b-codfw (row) to private1-b2-codfw (rack) vlan - https://phabricator.wikimedia.org/T352918#9588511 (10cmooney) [19:10:03] 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic, 13Patch-For-Review: Move lvs2012 from private1-b-codfw (row) to private1-b2-codfw (rack) vlan - https://phabricator.wikimedia.org/T352918#9588555 (10cmooney) [19:14:43] 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic, 13Patch-For-Review: Move lvs2012 from private1-b-codfw (row) to private1-b2-codfw (rack) vlan - https://phabricator.wikimedia.org/T352918#9588573 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmooney@cumin1002 for host... [19:22:44] 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic, 13Patch-For-Review: Move lvs2012 from private1-b-codfw (row) to private1-b2-codfw (rack) vlan - https://phabricator.wikimedia.org/T352918#9588615 (10cmooney) [19:36:16] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack, 10FY2023/2024-Q3-Q4, 13Patch-For-Review: spicerack: tox fails to install PyYAML using python 3.11 on bookworm - https://phabricator.wikimedia.org/T345337#9588669 (10RKemper) a:05fnegri→03RKemper @fnegri @brouberol Yeah, Brian and I will work on... [20:09:00] 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic, 13Patch-For-Review: Move lvs2011 from private1-a-codfw (row) to private1-a2-codfw (rack) vlan - https://phabricator.wikimedia.org/T352920#9588735 (10cmooney) [20:47:32] 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic: Move lvs2012 from private1-b-codfw (row) to private1-b2-codfw (rack) vlan - https://phabricator.wikimedia.org/T352918#9588904 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmooney@cumin1002 for host lvs2012.codfw.wmnet with... [20:48:30] (SystemdUnitFailed) firing: generate_os_reports.service on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:58:26] 10netbox, 06DC-Ops, 06Infrastructure-Foundations: Netbox:Report:PhysicalHosts: mistmach model issue - https://phabricator.wikimedia.org/T358809 (10RobH) [20:58:47] 10netbox, 06DC-Ops, 06Infrastructure-Foundations: Netbox:Report:PhysicalHosts: mistmach model issue - https://phabricator.wikimedia.org/T358809#9588958 (10RobH) Is there a file where this mapping is maintained and if so, I can update in the future when I add new device models to netbox? [21:00:23] 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic: Move lvs2012 from private1-b-codfw (row) to private1-b2-codfw (rack) vlan - https://phabricator.wikimedia.org/T352918#9588964 (10cmooney) 05Open→03Resolved Moved to new vlan and BGP established between server and switch now. ` cmooney@lvs2012:/e... [21:00:31] 10netops, 06Infrastructure-Foundations, 06SRE: Re-IP hosts on codfw row A and B to new per-rack vlans/subnets - https://phabricator.wikimedia.org/T354869#9588966 (10cmooney) [21:53:07] 10netbox, 06DC-Ops, 06Infrastructure-Foundations: Netbox:Report:PhysicalHosts: mistmach model issue - https://phabricator.wikimedia.org/T358809#9589129 (10Volans) a:05Volans→03None There is no mapping, the reported device types are just not following the correct naming scheme, as you can see here compari...