[02:37:05] (PuppetFailure) firing: Puppet has failed on debmonitor2003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [02:39:04] (PuppetConstantChange) firing: Puppet performing a change on every puppet run on debmonitor2003:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [02:49:22] (SystemdUnitFailed) firing: (4) debmonitor-maintenance-gc.service Failed on debmonitor2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:37:34] (DiskSpace) firing: Disk space build2001:9100:/ 4.687% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=build2001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [03:52:34] (DiskSpace) resolved: Disk space build2001:9100:/ 4.687% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=build2001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [06:37:05] (PuppetFailure) firing: Puppet has failed on debmonitor2003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [06:39:04] (PuppetConstantChange) firing: Puppet performing a change on every puppet run on debmonitor2003:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [06:50:36] (SystemdUnitFailed) firing: (4) debmonitor-maintenance-gc.service Failed on debmonitor2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:49:18] 10netops, 10Data-Persistence, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack B5 from asw-b5-codfw to lsw1-b5-codfw - https://phabricator.wikimedia.org/T355549 (10Marostegui) [09:52:57] 10Puppet, 10Wikidata, 10wmde-wikidata-tech, 10Technical-Debt, 10Wikidata Analytics (Kanban): Remove the WDCM clone (stats1007) - https://phabricator.wikimedia.org/T351072 (10Manuel) [10:04:22] (SystemdUnitFailed) firing: (5) debmonitor-maintenance-gc.service Failed on debmonitor2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:29:22] (SystemdUnitFailed) firing: (6) debmonitor-maintenance-gc.service Failed on debmonitor2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:37:05] (PuppetFailure) firing: Puppet has failed on debmonitor2003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [10:39:04] (PuppetConstantChange) firing: Puppet performing a change on every puppet run on debmonitor2003:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [10:40:28] slyngs, moritzm: aren't all those alerts a bit spammy? missing downtime/silence? real issue? re-firing too often? [10:40:40] Already done [10:42:08] Oh, not the Puppet one, silenced as well [10:44:58] ack, thx [10:45:04] I'll fix it in the next days [10:45:40] Part of it is also my half-baked Puppet code for the debmonitor package deployment [10:46:02] ack [11:19:22] (SystemdUnitFailed) firing: (4) prometheus-ganeti-exporter.service Failed on ganeti2033:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:34:22] (SystemdUnitFailed) firing: (4) prometheus-ganeti-exporter.service Failed on ganeti2033:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:44:56] 10SRE-tools, 10Infrastructure-Foundations, 10Puppet-Core, 10SRE, and 5 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619 (10MoritzMuehlenhoff) [11:49:22] (SystemdUnitFailed) firing: (4) prometheus-ganeti-exporter.service Failed on ganeti2033:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:04:22] (SystemdUnitFailed) firing: (3) prometheus-ganeti-exporter.service Failed on ganeti2033:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:16:58] XioNoX: ^^^ FUI [12:18:27] looking, but we can probably downtime that host for quite some time [12:20:09] volans: looks like it recovered, doesn't show up in https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed&q=team%3Dinfrastructure-foundations [12:20:15] and looks fine on the host [12:20:34] no recovery here? [12:20:53] 10SRE-tools, 10Infrastructure-Foundations, 10Puppet-Core, 10SRE, and 5 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619 (10MoritzMuehlenhoff) [12:22:23] volans: I think it's because of the (3) ? [12:22:50] if they all recovered... dunno [12:22:54] seems weird [12:39:19] volans: In debmonitor we have get_client(), which assumes that the cli.py file is in BASE_DIR / utils / cli.py that's not really the case after splitting the code into two packages. Do we either want to make this configurable, and point it to the package content. Seems a little wrong somehow [12:41:39] slyngs: yes it's clearly wrong on multiple levels, on the repo iself for tests and self-consistency, on deployment given that one couild deploy in different ways, not only the debian package, etc... [12:42:21] for the debian packaging case I guess we could make the server package depend on the client one and use the path where the client installs it [12:42:44] an alternative way is to add the client repo as a git submodule in the server one [12:43:29] the tests weren't testing that method or did in a way that didn't require the file? IIRC the file was removed a while ago [12:43:44] having the server depend on the client sounds good to me [12:44:06] but that doens't solve installing from pypi/manually from the repo [12:44:50] unless installing from debian package will be the only supported installation method [12:45:37] also the path will change with debian versions, so I wonder how that would be managed, hardcoding a link of the file in the postinst? [12:46:39] The path shouldn't change with Debian version. [12:47:20] ash right doesn't have the minor version [12:47:20] /usr/lib/python3/dist-packages/debmonitor_client/cli.py [12:47:31] the alternative would be a debmonitor-common package both -client and -server depend on, but that seems a little overblown, since every server by itself would also want to submit to debmonitor [12:47:33] s/ash/ah/ [12:48:00] Yeah, but then we again have the issue: What if you don't install from a deb package [12:48:01] yeah that seems a bit overkill [12:48:13] (the 3 packages way) [12:48:27] slyngs: symlink I guess [12:48:34] of a known path in the server side [12:48:50] or configurable path to the client cli [12:49:15] but I'd rather have a method that by default has already the right path [12:49:25] when installing via deb package at least [12:49:30] We can do what Debian does for Jquery and just include a broken symlink with only works if the right package is installed [12:50:10] btw we could add the dependency on the client also in the server's setup.py deps [12:50:23] or we have a default path for cli.py and if that's not found, we error out with an explanation [12:50:25] and now that I double checked, we never released to pypi [12:50:30] (for the pypi use case) [12:50:33] so is down to source code and deb packages [12:50:36] at least for now [12:50:44] (for the source code use case :-) [12:51:00] Okay, so dependency on client package and ... symlink in the server package? [12:51:50] (PuppetFailure) resolved: Puppet has failed on debmonitor2003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [12:54:17] I don't know what's the best practice in deb packages for this situation, but I guess tht having a better check in the code to present a clear exception/error and then having the deps on the debian client package and creating the symlink might be a good idea [12:56:23] that sounds good to me [12:56:38] The code is already pretty good about it, the catches the exception and returns a error to the client [12:57:22] what does it say? [12:57:38] Unable to retrieve client code: ... [12:57:51] Unable to retrieve client code: [12:57:54] Yes [13:28:30] Like this: https://gerrit.wikimedia.org/r/c/operations/software/debmonitor/+/993083 [14:16:53] moritzm: among the things that needs to be migrated to cumin1002 it's httpbb, I see the timers are still active on 1001 and not on 1002 [14:17:50] oh indeed, can you please add a sub task under https://phabricator.wikimedia.org/T353419 and add ServiceOps to it? [14:17:59] sure [14:19:40] {done} [14:19:48] cheers [14:37:12] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate hosts from codfw row A/B ASW to new LSW devices - https://phabricator.wikimedia.org/T355544 (10cmooney) [14:39:10] 10netops, 10Data-Persistence, 10Data-Persistence-Backup, 10Infrastructure-Foundations, and 2 others: Migrate servers in codfw rack B4 from asw-b4-codfw to lsw1-b4-codfw - https://phabricator.wikimedia.org/T355860 (10Marostegui) [14:39:23] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A3 from asw-a3-codfw to lsw1-a3-codfw - https://phabricator.wikimedia.org/T355862 (10Marostegui) [14:39:44] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A4 from asw-a4-codfw to lsw1-a4-codfw - https://phabricator.wikimedia.org/T355863 (10Marostegui) [14:41:30] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A5 from asw-a5-codfw to lsw1-a5-codfw - https://phabricator.wikimedia.org/T355864 (10Marostegui) [14:41:44] 10SRE-tools, 10Infrastructure-Foundations, 10Puppet-Core, 10SRE, and 5 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619 (10MoritzMuehlenhoff) [14:41:50] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A6 from asw-a6-codfw to lsw1-a6-codfw - https://phabricator.wikimedia.org/T355866 (10Marostegui) [14:44:24] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A8 from asw-a8-codfw to lsw1-a8-codfw - https://phabricator.wikimedia.org/T355874 (10Marostegui) [14:44:36] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE: Migrate servers in codfw rack B3 from asw-b3-codfw to lsw1-b3-codfw - https://phabricator.wikimedia.org/T355870 (10Marostegui) [14:44:46] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack B6 from asw-b6-codfw to lsw1-b6-codfw - https://phabricator.wikimedia.org/T355871 (10Marostegui) [14:44:59] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack B8 from asw-b8-codfw to lsw1-b8-codfw - https://phabricator.wikimedia.org/T355873 (10Marostegui) [14:47:42] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A3 from asw-a3-codfw to lsw1-a3-codfw - https://phabricator.wikimedia.org/T355862 (10Marostegui) db2142 - x2 master db2103 - s1 master es2020 - es4 master [14:48:19] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A8 from asw-a8-codfw to lsw1-a8-codfw - https://phabricator.wikimedia.org/T355874 (10Marostegui) db2146 - slave db2106 - slave [14:48:41] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE: Migrate servers in codfw rack B3 from asw-b3-codfw to lsw1-b3-codfw - https://phabricator.wikimedia.org/T355870 (10Marostegui) db2108 - slave db2123 - slave es2021 - es4 master [14:50:17] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A4 from asw-a4-codfw to lsw1-a4-codfw - https://phabricator.wikimedia.org/T355863 (10Marostegui) db2183 - codfw backup master @jcrespo [14:51:51] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A5 from asw-a5-codfw to lsw1-a5-codfw - https://phabricator.wikimedia.org/T355864 (10Marostegui) db2121 - slave db2132 m1 master (not used) db2145 - slave db2104 - m2 master db2153 - slave db2154 - slave db2... [14:58:49] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A6 from asw-a6-codfw to lsw1-a6-codfw - https://phabricator.wikimedia.org/T355866 (10Marostegui) db2155 - slave db2156 - slave db2097 - backups slave @jcrespo db2105 - s3 master db2122 - slave db2133 - m2 ma... [15:02:28] meeting meeting [15:03:12] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack B6 from asw-b6-codfw to lsw1-b6-codfw - https://phabricator.wikimedia.org/T355871 (10Marostegui) db2098 - backup slave @jcrespo db2110 - slave db2111 - slave db2124 - slave db2134 - m3 master (not used) db20... [15:04:55] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack B8 from asw-b8-codfw to lsw1-b8-codfw - https://phabricator.wikimedia.org/T355873 (10Marostegui) db2148 - slave db2163 - slave db2185 zarcillo dc master (nothing required) db2164 - slave db2189 - slave es2029... [15:05:05] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A4 from asw-a4-codfw to lsw1-a4-codfw - https://phabricator.wikimedia.org/T355863 (10jcrespo) Thank you, I will shutdown media backups anyway every time one host is affected, not just this one, to minimize fa... [15:08:04] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A3 from asw-a3-codfw to lsw1-a3-codfw - https://phabricator.wikimedia.org/T355862 (10Marostegui) [15:13:13] 10netops, 10Ganeti, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Investigate Ganeti in routed mode - https://phabricator.wikimedia.org/T300152 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by ayounsi@cumin2002 for hosts: `sretest1005.eqiad.wmnet` - sretest1005.eqiad.wmnet (... [15:35:32] 10Mail, 10Infrastructure-Foundations, 10SRE: Puppetry - https://phabricator.wikimedia.org/T325395 (10jhathaway) p:05Triage→03Medium [15:36:26] 10Mail, 10Infrastructure-Foundations, 10SRE: Provision mta-inbound-lists - https://phabricator.wikimedia.org/T325404 (10jhathaway) p:05Triage→03Low [15:36:36] 10Mail, 10Infrastructure-Foundations, 10SRE: Provision mta-outbound-lists - https://phabricator.wikimedia.org/T325405 (10jhathaway) p:05Triage→03Medium [15:36:55] 10Mail, 10Infrastructure-Foundations, 10SRE: MTA Provisioning - https://phabricator.wikimedia.org/T325403 (10jhathaway) p:05Triage→03Medium [15:37:13] 10Mail, 10Infrastructure-Foundations, 10SRE: Replace Exim with Postfix on mail servers - https://phabricator.wikimedia.org/T325394 (10jhathaway) p:05Triage→03Medium [15:37:30] 10Mail, 10Infrastructure-Foundations: Email sent from wikipedia UI seems to use nondeliverable sender: 550 Administrative prohibition - https://phabricator.wikimedia.org/T207650 (10jhathaway) p:05Triage→03Medium [15:37:46] 10Mail, 10Infrastructure-Foundations: Troubleshooting Mail Delivery Issues from Coupa - https://phabricator.wikimedia.org/T306472 (10jhathaway) p:05Triage→03Medium [15:38:07] 10Mail, 10Infrastructure-Foundations: Some emails coming from Gerrit are being tagged as suspicious by Gmail - https://phabricator.wikimedia.org/T226884 (10jhathaway) p:05Triage→03Medium [15:38:16] 10Mail, 10Infrastructure-Foundations: Remove wikivoyage-ev.org mail aliases from wikivoyage.org & wikivoyage.de - https://phabricator.wikimedia.org/T319041 (10jhathaway) p:05Triage→03Low [15:41:49] 10Mail, 10Infrastructure-Foundations: Exim: add lists and auto-generated headers - https://phabricator.wikimedia.org/T347831 (10jhathaway) p:05Triage→03Low a:03jhathaway [15:43:39] 10netops, 10Infrastructure-Foundations, 10SRE: Put Dell SONiC switches in production - https://phabricator.wikimedia.org/T335028 (10ayounsi) p:05Triage→03Medium [15:43:44] 10CAS-SSO, 10Infrastructure-Foundations: Upgrade Apereo CAS to include PKCE functionality when it becomes available - https://phabricator.wikimedia.org/T350727 (10joanna_borun) p:05Triage→03Low [15:45:44] 10Mail, 10Infrastructure-Foundations: Received 4 mail notifications but only 2 actual mails - https://phabricator.wikimedia.org/T351027 (10jhathaway) @AlexisJazz were you able to reproduce? [15:45:54] 10Mail, 10Infrastructure-Foundations: Received 4 mail notifications but only 2 actual mails - https://phabricator.wikimedia.org/T351027 (10jhathaway) p:05Triage→03Low a:03jhathaway [15:48:07] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A4 from asw-a4-codfw to lsw1-a4-codfw - https://phabricator.wikimedia.org/T355863 (10cmooney) [15:48:49] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A6 from asw-a6-codfw to lsw1-a6-codfw - https://phabricator.wikimedia.org/T355866 (10Marostegui) [16:00:04] 10SRE-tools, 10Infrastructure-Foundations, 10Spicerack: Spicerack: migrate distributed locking to etcd v3 - https://phabricator.wikimedia.org/T352155 (10Volans) p:05Triage→03Medium [16:00:10] 10SRE-tools, 10Infrastructure-Foundations, 10Spicerack: Spicerack: adapt conftool module for etcd v3 - https://phabricator.wikimedia.org/T352153 (10Volans) p:05Triage→03Medium [16:01:39] 10SRE-tools, 10Infrastructure-Foundations, 10SRE, 10Spicerack: More structured cookbooks to reboot hosts - https://phabricator.wikimedia.org/T252807 (10MoritzMuehlenhoff) [16:02:04] 10SRE-tools, 10Infrastructure-Foundations, 10SRE, 10Spicerack: Migrate existing cookbooks related to rolling restarts/reboots to SREBatchBase - https://phabricator.wikimedia.org/T317855 (10MoritzMuehlenhoff) 05Open→03In progress p:05Triage→03Low [16:05:36] (SystemdUnitFailed) firing: generate_os_reports.service Failed on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:05:40] 10netbox, 10Infrastructure-Foundations, 10Patch-For-Review: Reduce the count of Netbox devices with incorrect status - https://phabricator.wikimedia.org/T320696 (10joanna_borun) p:05Triage→03Low [16:11:45] 10SRE-tools, 10Infrastructure-Foundations: Abstract a bit more the server provisioning process - https://phabricator.wikimedia.org/T351891 (10joanna_borun) p:05Triage→03Medium [16:16:54] 10netbox, 10Data-Persistence-Backup, 10Infrastructure-Foundations, 10bacula: Convert Netbox data (PostgresQL) longterm storage backups (bacula) into full backups rather than incrementals - https://phabricator.wikimedia.org/T316655 (10Volans) p:05Triage→03Medium a:03Volans [16:17:27] 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic, 10Patch-For-Review: Move lvs2012 from private1-b-codfw (row) to private1-b2-codfw (rack) vlan - https://phabricator.wikimedia.org/T352918 (10cmooney) [16:18:53] 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic, 10Patch-For-Review: Move lvs2011 from private1-a-codfw (row) to private1-a2-codfw (rack) vlan - https://phabricator.wikimedia.org/T352920 (10cmooney) [16:24:08] 10SRE-tools, 10Infrastructure-Foundations: Package pyGNMI and dictdiffer to be used by cookbooks - https://phabricator.wikimedia.org/T340045 (10MoritzMuehlenhoff) p:05Triage→03Medium [16:24:57] 10homer, 10Infrastructure-Foundations: Update Homer Puppet classes to allow to absent Homer resources - https://phabricator.wikimedia.org/T353932 (10joanna_borun) p:05Triage→03Low [16:27:03] 10netbox, 10Infrastructure-Foundations: Markdown bug in Netbox-next - https://phabricator.wikimedia.org/T340444 (10joanna_borun) p:05Triage→03Medium [16:27:12] 10netbox, 10Infrastructure-Foundations: Evaluate usage of Kubernetes/Wikikube Tags in netbox and replace them with something if possible - https://phabricator.wikimedia.org/T354169 (10ayounsi) Thanks for the task, we haven't forgot, but we're probably going to look at it after upgrading Netbox to make sure we... [16:27:30] 10netbox, 10Infrastructure-Foundations: Evaluate usage of Kubernetes/Wikikube Tags in netbox and replace them with something if possible - https://phabricator.wikimedia.org/T354169 (10ayounsi) [16:27:33] 10netbox, 10Infrastructure-Foundations, 10Patch-For-Review: Upgrade Netbox to 3.7.x - https://phabricator.wikimedia.org/T336275 (10ayounsi) [18:23:50] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate hosts from codfw row A/B ASW to new LSW devices - https://phabricator.wikimedia.org/T355544 (10cmooney) [20:09:22] (SystemdUnitFailed) firing: generate_os_reports.service Failed on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed