[01:19:24] FIRING: SystemdUnitFailed: generate_os_reports.service on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:19:24] FIRING: SystemdUnitFailed: generate_os_reports.service on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:40:49] FIRING: PuppetConstantChange: Puppet performing a change on every puppet run on netbox2003:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [08:05:48] FIRING: [2x] PuppetConstantChange: Puppet performing a change on every puppet run on netbox1003:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [09:19:25] FIRING: SystemdUnitFailed: generate_os_reports.service on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:06:03] FIRING: [2x] PuppetConstantChange: Puppet performing a change on every puppet run on netbox1003:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [12:26:58] for some reason I am getting timeouts from CI for https://gerrit.wikimedia.org/r/c/operations/software/spicerack/+/1060854 [12:27:07] has anybody encountered a similar issue? [12:34:16] no idea, looks like py3.11 related ? maybe try PS disabling 3.11 to narrow down the issue ? [12:37:53] good suggestion [12:42:25] the weird thing is that now CI repeats all the tox workflow for every python version [12:43:38] so it is easy to get to timeout sigh [12:45:03] yes check https://integration.wikimedia.org/ci/job/tox/1676/console, that is a older CI for spicerack [12:45:30] only py39 ran, not 310/311 [12:57:31] elukey: hashar upgraded the CI image recently, so maybe it's related? There used to not be any 310 nor 311 supported [12:58:42] elukey: possible longer run fix is to just remove py39 and py310 from tox.ini as it's not really needed anymore [12:59:04] or just whatever we run on bullseye + most recent [13:00:23] XioNoX: we have 3.9 for bullseye https://packages.debian.org/bullseye/python3 [13:00:35] so maybe jsut remove 310 for now? [13:00:42] bookworm will have 311, so it would be nice to keep it [13:01:45] at least to unblock your workflow [13:03:35] I'll try to follow up with Releng, not sure if anybody works in these days [13:03:41] summer holidays etc.. [13:06:47] ah ok https://gerrit.wikimedia.org/r/c/integration/config/+/1060142/2/jjb/python-jobs.yaml [13:07:04] we jumped from tox 0.7.1 to 3.x? [13:08:13] mmm nono [13:19:25] FIRING: SystemdUnitFailed: generate_os_reports.service on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:46:19] Bryan pointed me to https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/thread/APVZDZTVIKSP3ZQL4U7UNTHGKC66LHUV/ [15:46:36] so now all python versions are supported in the same image, and spicerack's tox runs for all of them [16:06:03] FIRING: [2x] PuppetConstantChange: Puppet performing a change on every puppet run on netbox1003:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [16:08:15] that alert is going to be fixed as soon as https://gerrit.wikimedia.org/r/c/operations/puppet/+/1060075 is deployed ^ [16:21:02] opened https://phabricator.wikimedia.org/T372485 for the spicerack issue [16:27:43] so TL;DR: we cannot merge changes for spicerack atm, we can only if we manually v:+2 [17:19:25] FIRING: SystemdUnitFailed: generate_os_reports.service on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:06:03] FIRING: [2x] PuppetConstantChange: Puppet performing a change on every puppet run on netbox1003:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [21:19:25] FIRING: SystemdUnitFailed: generate_os_reports.service on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed