[01:50:07] FIRING: [8x] PuppetConstantChange: Puppet performing a change on every puppet run on puppetmaster1001:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [03:40:40] FIRING: [3x] SystemdUnitFailed: cowbuilder_update_buster-amd64.service on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:50:07] FIRING: [8x] PuppetConstantChange: Puppet performing a change on every puppet run on puppetmaster1001:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [06:30:25] FIRING: [5x] SystemdUnitFailed: cowbuilder_update_buster-amd64.service on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:34:47] ^ I've merged a patch to fix the cowbuilder alert; Buster was removed from the Debian mirrors over the weekend [06:36:00] FIRING: [5x] SystemdUnitFailed: cowbuilder_update_buster-amd64.service on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:30:25] FIRING: [4x] SystemdUnitFailed: debian-weekly-rebuild.service on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:50:08] FIRING: [8x] PuppetConstantChange: Puppet performing a change on every puppet run on puppetmaster1001:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [10:00:25] FIRING: [2x] SystemdUnitFailed: debian-weekly-rebuild.service on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:10:25] RESOLVED: [2x] SystemdUnitFailed: debian-weekly-rebuild.service on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:50:08] FIRING: [8x] PuppetConstantChange: Puppet performing a change on every puppet run on puppetmaster1001:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [13:56:47] 10SRE-tools, 06Infrastructure-Foundations, 06SRE, 07IPv6, 13Patch-For-Review: Enable ipv6 on ganeti2019-ganeti2024 - https://phabricator.wikimedia.org/T379890#11000393 (10MoritzMuehlenhoff) 05Open→03Resolved ganeti2019-ganeti2024 have been decommissioned as part of the last server reresh in codfw. [14:33:55] FIRING: MaxConntrack: Max conntrack at 80.41% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [14:38:55] RESOLVED: MaxConntrack: Max conntrack at 80.41% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [14:43:46] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack: Increase the default batch size of puppet.run() - https://phabricator.wikimedia.org/T397687#11000726 (10elukey) Possible related task: T280622 The concern may be that cookbooks running in parallel at the same time could put some strain the puppetserver... [14:45:58] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack: Increase the default batch size of puppet.run() - https://phabricator.wikimedia.org/T397687#11000746 (10joanna_borun) p:05Triage→03Medium [14:47:01] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack: Flaky spicerack icinga unit tests - https://phabricator.wikimedia.org/T397833#11000747 (10joanna_borun) p:05Triage→03Low [14:52:03] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack: Increase the default batch size of puppet.run() - https://phabricator.wikimedia.org/T397687#11000793 (10Volans) @JMeybohm do you have a specific use case that cannot/is hard to solve simply changing the `batch_size` of the call to `puppet.run()`? https:... [16:24:15] topranks: no actionable on the Meta emails for the BGP sessions in SGIX, right? [16:25:08] sukhe: correct, we aren't at the exchange anymore so that's expected [16:26:01] thanks! [16:27:36] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, and 2 others: eqsin purged consumers lag - https://phabricator.wikimedia.org/T399221#11001359 (10RobH) [17:50:08] FIRING: [8x] PuppetConstantChange: Puppet performing a change on every puppet run on puppetmaster1001:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [18:20:05] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, and 2 others: eqsin purged consumers lag - https://phabricator.wikimedia.org/T399221#11001862 (10cmooney) Just an update on this, we unfortunately did not have a spare optic of the right kind so dc-ops are ordering one with expedited delivery.... [21:50:08] FIRING: [8x] PuppetConstantChange: Puppet performing a change on every puppet run on puppetmaster1001:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange