[01:30:04] (PuppetConstantChange) firing: Puppet performing a change on every puppet run on testvm2005:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [01:46:31] 10CAS-SSO, 06Infrastructure-Foundations, 10Release-Engineering-Team (Priority Backlog 📥): Correct IDP login page Privacy Policy - https://phabricator.wikimedia.org/T350129#9648092 (10Pppery) a:03Pppery [02:42:14] (SystemdUnitFailed) firing: generate_os_reports.service on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:30:04] (PuppetConstantChange) firing: Puppet performing a change on every puppet run on testvm2005:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [06:42:14] (SystemdUnitFailed) firing: generate_os_reports.service on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:13:20] 10CAS-SSO, 06Infrastructure-Foundations, 13Patch-For-Review: Migrate CAS to Bookworm - https://phabricator.wikimedia.org/T357748#9648245 (10SLyngshede-WMF) VM creation: ` sudo cookbook sre.ganeti.makevm --vcpus 2 --memory 4 --disk 40 --network public --os bookworm --cluster codfw --group A -t T357748 -p 7... [09:00:00] 10netbox, 10ChangeProp, 10GitLab, 06Infrastructure-Foundations, and 3 others: Figure out a plan to move forward with regarding Redis License changes - https://phabricator.wikimedia.org/T360596#9648360 (10akosiaris) [09:00:59] 10netbox, 10ChangeProp, 10GitLab, 06Infrastructure-Foundations, and 4 others: Figure out a plan to move forward with regarding Redis License changes - https://phabricator.wikimedia.org/T360596#9648367 (10Peachey88) [09:02:20] 10netbox, 10ChangeProp, 10GitLab, 06Infrastructure-Foundations, and 4 others: Figure out a plan to move forward with regarding Redis License changes - https://phabricator.wikimedia.org/T360596#9648371 (10MoritzMuehlenhoff) [09:22:27] 10CAS-SSO, 06Infrastructure-Foundations, 13Patch-For-Review: Migrate CAS to Bookworm - https://phabricator.wikimedia.org/T357748#9648493 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by slyngshede@cumin1002 for host idp2003.wikimedia.org with OS bookworm [09:30:04] (PuppetConstantChange) firing: Puppet performing a change on every puppet run on testvm2005:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [09:59:16] 10CAS-SSO, 06Infrastructure-Foundations: Migrate CAS to Bookworm - https://phabricator.wikimedia.org/T357748#9648638 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by slyngshede@cumin1002 for host idp2003.wikimedia.org with OS bookworm completed: - idp2003 (**PASS**) - Removed from Puppe... [10:05:39] 10CAS-SSO, 06Infrastructure-Foundations: Migrate CAS to Bookworm - https://phabricator.wikimedia.org/T357748#9648658 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by slyngshede@cumin1002 for host idp1003.wikimedia.org with OS bookworm [10:34:08] 10CAS-SSO, 06Infrastructure-Foundations: Migrate CAS to Bookworm - https://phabricator.wikimedia.org/T357748#9648742 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by slyngshede@cumin1002 for host idp1003.wikimedia.org with OS bookworm completed: - idp1003 (**PASS**) - Removed from Puppe... [10:42:14] (SystemdUnitFailed) firing: generate_os_reports.service on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:53:15] 10netbox, 10ChangeProp, 06Commons, 10GitLab, and 7 others: Figure out a plan to move forward with regarding Redis License changes - https://phabricator.wikimedia.org/T360596#9648786 (10taavi) [10:53:27] 10netops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Decom asw-a-codfw switch stack - https://phabricator.wikimedia.org/T358244#9648799 (10cmooney) >>! In T358244#9636601, @ayounsi wrote: > FYI it's alerting for one of its PSU being down, but we don't really care anymore : >> asw-a-codfw> show syste... [11:13:12] 10netbox, 10netops, 06Infrastructure-Foundations: 14Netbox: bug preventing removing a parent bridge in custom script automation - 14https://phabricator.wikimedia.org/T359629#9648862 (10cmooney) 05Open→03Resolved 14This should be ok from here on. Thanks for the task and troubleshooting! [13:16:48] 10netbox, 10ChangeProp, 06Commons, 10GitLab, and 7 others: Figure out a plan to move forward with regarding Redis License changes - https://phabricator.wikimedia.org/T360596#9648979 (10Reedy) https://github.com/Snapchat/KeyDB already existed as a fork. https://github.com/Snapchat/KeyDB/issues/798 was filed... [13:30:04] (PuppetConstantChange) firing: Puppet performing a change on every puppet run on testvm2005:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [14:42:14] (SystemdUnitFailed) firing: generate_os_reports.service on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:21:07] 10netbox, 10ChangeProp, 06Commons, 10GitLab, and 8 others: Figure out a plan to move forward with regarding Redis License changes - https://phabricator.wikimedia.org/T360596#9649904 (10brennen) For GitLab: I //think// we currently run the bundled Redis in their Omnibus package. In that case, the easiest th... [16:06:07] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Add per-output queue monitoring for Juniper network devices - https://phabricator.wikimedia.org/T326322#9650260 (10cmooney) >>! In T326322#9130092, @ayounsi wrote: > @cmooney I came across https://www.juniper.net/documentation/us/en/softwar... [17:30:04] (PuppetConstantChange) firing: Puppet performing a change on every puppet run on testvm2005:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [18:42:14] (SystemdUnitFailed) firing: generate_os_reports.service on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:51:59] (SystemdUnitFailed) firing: (2) netbox_report_accounting_run.service on netbox1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:16:59] (SystemdUnitFailed) firing: (2) netbox_report_accounting_run.service on netbox1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:28:26] 10netbox, 10ChangeProp, 06Commons, 10GitLab, and 9 others: Figure out a plan to move forward with regarding Redis License changes - https://phabricator.wikimedia.org/T360596#9651946 (10Krinkle) [20:36:04] 10netbox, 10ChangeProp, 10GitLab, 06Infrastructure-Foundations, and 8 others: Figure out a plan to move forward with regarding Redis License changes - https://phabricator.wikimedia.org/T360596#9651962 (10Peachey88) [21:06:24] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack, 10cloud-services-team (FY2023/2024-Q3-Q4), 13Patch-For-Review: spicerack: tox fails to install PyYAML using python 3.11 on bookworm - https://phabricator.wikimedia.org/T345337#9652068 (10bking) We're going to upgrade curator (as well as its library)... [21:14:26] 10netbox, 10ChangeProp, 10GitLab, 06Infrastructure-Foundations, and 8 others: Figure out a plan to move forward with regarding Redis License changes - https://phabricator.wikimedia.org/T360596#9652082 (10Krinkle) In MediaWiki (as deployed at WMF), there exists 1 use of Redis, which is during file uploads v... [21:30:04] (PuppetConstantChange) firing: Puppet performing a change on every puppet run on testvm2005:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [22:00:41] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack, 10cloud-services-team (FY2023/2024-Q3-Q4), 13Patch-For-Review: spicerack: tox fails to install PyYAML using python 3.11 on bookworm - https://phabricator.wikimedia.org/T345337#9652267 (10Volans) >>! In T345337#9652068, @bking wrote: > We're going to... [22:03:02] 10netbox, 10ChangeProp, 10GitLab, 06Infrastructure-Foundations, and 8 others: Figure out a plan to move forward with regarding Redis License changes - https://phabricator.wikimedia.org/T360596#9652274 (10Ladsgroup) >>! In T360596#9652082, @Krinkle wrote: > In MediaWiki (as deployed at WMF), there exists 1... [22:10:53] 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack, 10cloud-services-team (FY2023/2024-Q3-Q4), 13Patch-For-Review: spicerack: tox fails to install PyYAML using python 3.11 on bookworm - https://phabricator.wikimedia.org/T345337#9652299 (10bking) > The linked task is this same one. Did you meant to li... [23:17:14] (SystemdUnitFailed) firing: generate_os_reports.service on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [23:32:53] 10netbox, 10ChangeProp, 10GitLab, 06Infrastructure-Foundations, and 8 others: Figure out a plan to move forward with regarding Redis License changes - https://phabricator.wikimedia.org/T360596#9652472 (10bd808)