[10:29:18] 10puppet-compiler, 10Infrastructure-Foundations, 10SRE: compiler1003.puppet-diffs.eqiad1.wikimedia.cloud out of disk space - https://phabricator.wikimedia.org/T295253 (10ema) This is a recurring problem, see for example T273599 T222072 T295253. T222075 has ideas on how to tackle the issue. I've tried to acc... [11:20:42] 10puppet-compiler, 10Infrastructure-Foundations, 10SRE: compiler1003.puppet-diffs.eqiad1.wikimedia.cloud out of disk space - https://phabricator.wikimedia.org/T295253 (10Majavah) I cleaned up old jobs from `/srv/jenkins-workspace/puppet-compiler`. The cleanup jobs `delete-old-output-files.service` and `delet... [13:06:52] 10Puppet, 10Infrastructure-Foundations, 10User-jbond: Systemd time spec incompatibility: On(UnitIn)activeSec not compatable with 'Daily' - https://phabricator.wikimedia.org/T295284 (10jbond) p:05Triage→03Medium [13:20:15] 10puppet-compiler, 10Infrastructure-Foundations, 10SRE, 10observability, and 2 others: Prevent puppet catalog compiler workers from running out of disk space - https://phabricator.wikimedia.org/T222075 (10jbond) [13:22:49] 10puppet-compiler, 10Infrastructure-Foundations, 10SRE: compiler1003.puppet-diffs.eqiad1.wikimedia.cloud out of disk space - https://phabricator.wikimedia.org/T295253 (10jbond) after[[ https://gerrit.wikimedia.org/r/c/operations/puppet/+/737350 | joe's patch ]] the timers are now working again. I have also... [13:35:36] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10cloud-services-team (Kanban): cr-codfw: set up static route for 185.15.57.8/30 - https://phabricator.wikimedia.org/T295288 (10aborrero) [13:36:09] 10netops, 10Cloud-VPS, 10Infrastructure-Foundations, 10cloud-services-team (Kanban): cr-codfw: set up static route for 185.15.57.8/30 - https://phabricator.wikimedia.org/T295288 (10aborrero) p:05Triage→03Medium [15:30:00] 10netops, 10Infrastructure-Foundations, 10SRE: Can't commit on asw-b-codfw - https://phabricator.wikimedia.org/T295118 (10ayounsi) a:05ayounsi→03Papaul According to @akosiaris this is due to a failed hard drive, and it might not come back up from a reboot. @Papaul when you're back, let's replace FPC7 wi... [16:28:54] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Can't commit on asw-b-codfw - https://phabricator.wikimedia.org/T295118 (10wiki_willy) [17:57:59] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) Rows E/F network racking task - https://phabricator.wikimedia.org/T292095 (10wiki_willy) a:03Jclark-ctr [22:34:36] 10netops, 10Infrastructure-Foundations, 10SRE, 10Traffic, 10procurement: drmrs: primary software task - https://phabricator.wikimedia.org/T282788 (10colewhite) p:05Triage→03Medium [22:38:07] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE Observability (FY2021/2022-Q2), 10Sustainability (Incident Followup): Alert that should have paged did not reach VictorOps because of partial networking outage - https://phabricator.wikimedia.org/T294166 (10colewhite) p:05Triage→03Medium