[01:02:10] <jinxer-wm>	 (SystemdUnitFailed) firing: generate_os_reports.service on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[03:21:55] <jinxer-wm>	 (SystemdUnitFailed) firing: (2) netbox_report_accounting_run.service on netbox1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[03:46:55] <jinxer-wm>	 (SystemdUnitFailed) firing: (2) netbox_report_accounting_run.service on netbox1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[07:35:18] <wikibugs>	 10SRE-tools, 06Infrastructure-Foundations, 10Puppet-Core, 06SRE, and 5 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619#9610033 (10MoritzMuehlenhoff)
[07:37:49] <wikibugs>	 10SRE-tools, 06Infrastructure-Foundations, 10Puppet-Core, 06SRE, and 5 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619#9610034 (10MoritzMuehlenhoff)
[07:47:10] <jinxer-wm>	 (SystemdUnitFailed) firing: generate_os_reports.service on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[11:26:32] <wikibugs>	 10Packaging, 06Infrastructure-Foundations, 06SRE: Debian repository supporting multiple package versions - https://phabricator.wikimedia.org/T115758#9610647 (10MoritzMuehlenhoff) 05Open→03Resolved a:03MoritzMuehlenhoff We have established that with the system of separate component we introduced a few y...
[11:31:41] <fabfur>	 I was noticing that lately the CI timings are a little higher than usual, so _joe_ helped me finding this dashboard https://grafana.wikimedia.org/goto/FSgVZT0Sk?orgId=1
[11:31:59] <fabfur>	 looks like the average time got worse lately 
[11:32:18] <fabfur>	 did you noticed that too? Is something we already know? 
[11:47:10] <jinxer-wm>	 (SystemdUnitFailed) firing: generate_os_reports.service on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[12:32:38] <slyngs>	 fabfur: I did notice, I think I talked to someone about it, but can't remember what we concluded. jhathaway maybe we can talk about it later today, if nothing else comes up
[14:29:13] <wikibugs>	 10CAS-SSO, 10Bitu, 06Infrastructure-Foundations: Enable self-service IDP two-factor authentication management - https://phabricator.wikimedia.org/T359552 (10taavi) 03NEW
[15:47:10] <jinxer-wm>	 (SystemdUnitFailed) firing: generate_os_reports.service on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[17:17:45] <eoghan>	 Hi friends, the disk in pcc-worker1006.puppet-diffs.eqiad1.wikimedia.cloud is full, is it safe to clean up some directories in /srv/jenkins/puppet-compiler (what is using most of the space)? 
[17:21:10] <inflatador>	 not to give Emperor a heart attack, but has anyone considered putting the PCC output into swift as expiring objects? Might help with the disk space issues
[19:47:10] <jinxer-wm>	 (SystemdUnitFailed) firing: generate_os_reports.service on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[20:34:10] <wikibugs>	 10CAS-SSO, 10Bitu, 06Infrastructure-Foundations: Enable self-service IDP two-factor authentication management - https://phabricator.wikimedia.org/T359552#9613354 (10taavi)
[22:47:57] <ryankemper>	 inflatador and I ran into the same issue as eoghan on `pcc-worker1006`. We've went ahead and cleared away a few of the biggest directories, enough to reclaim ~7G. Hopefully we haven't stepped on any toes :)
[23:47:10] <jinxer-wm>	 (SystemdUnitFailed) firing: generate_os_reports.service on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed