[00:12:00] (NodeTextfileStale) firing: (5) Stale textfile for puppetserver1002:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [00:17:00] (NodeTextfileStale) firing: (6) Stale textfile for puppetserver1002:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [01:32:00] (NodeTextfileStale) firing: (7) Stale textfile for puppetserver1002:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [01:57:01] (NodeTextfileStale) firing: (9) Stale textfile for puppetserver1002:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [02:03:54] (SystemdUnitFailed) firing: production-images-weekly-rebuild.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:07:01] (NodeTextfileStale) firing: (10) Stale textfile for puppetserver1002:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [03:22:01] (NodeTextfileStale) firing: (11) Stale textfile for puppetserver1002:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [05:12:01] (NodeTextfileStale) firing: (13) Stale textfile for puppetserver1002:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [05:17:01] (NodeTextfileStale) firing: (14) Stale textfile for puppetserver1002:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [05:27:01] (NodeTextfileStale) firing: (14) Stale textfile for puppetserver1002:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [06:03:54] (SystemdUnitFailed) firing: production-images-weekly-rebuild.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:07:01] (NodeTextfileStale) firing: (14) Stale textfile for puppetserver1003:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [06:12:01] (NodeTextfileStale) firing: (15) Stale textfile for puppetserver1003:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [06:47:01] (NodeTextfileStale) firing: (16) Stale textfile for puppetserver1003:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [08:07:01] (NodeTextfileStale) firing: (17) Stale textfile for puppetserver1003:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [08:07:34] 10SRE-tools, 10Infrastructure-Foundations, 10Puppet-Core, 10SRE, and 2 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619 (10MoritzMuehlenhoff) [08:19:46] 10SRE-tools, 10Infrastructure-Foundations, 10Puppet-Core, 10SRE, and 2 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619 (10MoritzMuehlenhoff) [08:57:01] (NodeTextfileStale) firing: (19) Stale textfile for puppetserver1003:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [09:02:01] (NodeTextfileStale) firing: (20) Stale textfile for puppetserver1003:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [09:16:02] 10SRE-tools, 10Infrastructure-Foundations, 10Puppet-Core, 10SRE, 10Puppet (Puppet 7.0): Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619 (10MoritzMuehlenhoff) [09:52:01] (NodeTextfileStale) resolved: (20) Stale textfile for puppetserver1003:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [09:54:25] (SystemdUnitFailed) firing: (2) production-images-weekly-rebuild.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:58:21] 10SRE-tools, 10Infrastructure-Foundations, 10Puppet-Core, 10SRE, and 2 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619 (10MoritzMuehlenhoff) [10:19:25] (SystemdUnitFailed) firing: (4) production-images-weekly-rebuild.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:24:25] (SystemdUnitFailed) firing: (4) production-images-weekly-rebuild.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:28:58] 10CAS-SSO, 10Infrastructure-Foundations, 10Patch-For-Review: Create OpenID Connect client - https://phabricator.wikimedia.org/T350725 (10SLyngshede-WMF) ` catalyst: id: 5 service_class: 'OidcRegisteredService' service_id: 'https://catalyst-auth\.wmcloud\.org(/.*)?' profile_format: 'FLAT' ` [10:31:26] 10CAS-SSO, 10Cloud-VPS, 10Infrastructure-Foundations, 10Patch-For-Review: Create OpenID Connect client - https://phabricator.wikimedia.org/T350725 (10SLyngshede-WMF) The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile... [10:31:46] 10CAS-SSO, 10Cloud-VPS, 10Infrastructure-Foundations, 10Patch-For-Review: Create OpenID Connect client - https://phabricator.wikimedia.org/T350725 (10SLyngshede-WMF) [10:33:20] 10CAS-SSO, 10Cloud-VPS, 10Infrastructure-Foundations, 10cloud-services-team, 10Patch-For-Review: Create OpenID Connect client - https://phabricator.wikimedia.org/T350725 (10taavi) a:05SLyngshede-WMF→03taavi [10:35:41] 10CAS-SSO, 10Cloud-VPS, 10Infrastructure-Foundations, 10cloud-services-team, 10Patch-For-Review: Create OpenID Connect client - https://phabricator.wikimedia.org/T350725 (10taavi) I've configured the client to idp.wmcloud.org. The client ID is `catalyst` and the client secret is in P53310. [10:54:25] (SystemdUnitFailed) firing: (2) production-images-weekly-rebuild.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:02:58] topranks: volans: if you have a few (but probably more) minutes: https://gerrit.wikimedia.org/r/c/operations/puppet/+/973350 [11:03:22] Balto is working on automating it all, so hopefully that will be the last one [11:04:44] Sure thing let me have a look [11:05:17] 10Puppet, 10iPoid-Service: Rename FEED_API_KEY - https://phabricator.wikimedia.org/T350903 (10jijiki) 05Open→03Resolved a:03jijiki This was merged on Friday on the puppetmaster [11:05:42] if it makes it easier on Netbox you can use https://netbox.wikimedia.org/ipam/vlans/?site_id=6&vid= and filter on private1-, then on analytics1- and only display the name/prefixes columns [11:07:35] 10Puppet, 10iPoid-Service, 10serviceops: Rename FEED_API_KEY - https://phabricator.wikimedia.org/T350903 (10jijiki) [11:08:43] XioNoX: cool thanks [11:10:57] XioNoX: done [11:15:34] 10SRE-tools, 10Infrastructure-Foundations, 10Puppet-Core, 10SRE, 10Puppet (Puppet 7.0): Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619 (10MoritzMuehlenhoff) [11:18:36] 10SRE-tools, 10Infrastructure-Foundations, 10Puppet-Core, 10SRE, 10Puppet (Puppet 7.0): Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619 (10MoritzMuehlenhoff) [11:18:43] thx! [11:25:02] 10CAS-SSO, 10Cloud-VPS, 10Infrastructure-Foundations, 10cloud-services-team, 10Patch-For-Review: Create OpenID Connect client - https://phabricator.wikimedia.org/T350725 (10taavi) 05In progress→03Resolved [11:28:54] (SystemdUnitFailed) firing: (2) production-images-weekly-rebuild.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:30:30] "Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Function Call, (): found character '\t(TAB)' that cannot start any token. (Do not use \t(TAB) for indentation) while scanning for the next token at line 197 column 16 (file: /srv/puppet_code/environments/production/modules/network/manifests/constants.pp, line: 4, column: 21) on node [11:30:31] install1004.wikimedia.org" [11:30:44] looking into it, trying to find where that comes from [11:30:56] PCC didn't choke I'm surprised [11:38:35] jbond: if you're around, any idea what's up with the above? [11:39:02] XioNoX: one sec ill take a look [11:41:56] XioNoX: the new analytics1-e*-eqiad networks seem to use a tab instead of a space after the colon on the ipv4: lines [11:42:05] fix incomming [11:42:56] XioNoX: https://gerrit.wikimedia.org/r/c/operations/puppet/+/973750 [11:43:29] thx, I couldn't find them [11:44:03] np [11:44:17] jbond: how come the error doesn't show up on PCC? https://puppet-compiler.wmflabs.org/output/973350/405/install1004.wikimedia.org/index.html [11:52:41] XioNoX: ill have to do more testing but my guess is that the yaml parser in puppet7 jruby is a bit stricter [11:53:19] ok, yeah that would explain [11:53:59] (PuppetZeroResources) firing: Puppet has failed generate resources on ganeti1030:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [11:54:23] We may get a few of theses ^^^ im running puppet n failed now [11:54:32] the error mentioned above affected all puppet7 agents [12:02:58] volans, XioNoX: I've realised I never created the netboot cfg files for the new codfw private vlans either [12:03:18] depending on timing we might want to wait on Balto's patch? [12:03:23] volans: it seems that's probably the cause of that issue we were looking at whereby the d-i paused for input [12:03:28] https://gerrit.wikimedia.org/r/c/operations/puppet/+/973752 [12:03:41] XioNoX: yeah no preference really, not sure what the timeline is there [12:03:59] (PuppetZeroResources) resolved: Puppet has failed generate resources on ganeti1030:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [12:04:38] well, if you have it ready :) [12:05:51] I'll leave it up to you but yeah if it's ready let's get that included :D [12:06:43] +1 [12:19:26] (SystemdUnitFailed) firing: (3) production-images-weekly-rebuild.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:28:54] (SystemdUnitFailed) firing: (3) production-images-weekly-rebuild.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:32:56] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Connect two hosts in codfw row A/B for switch migration testing - https://phabricator.wikimedia.org/T345803 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmooney@cumin1001 for host sretest2004.codfw.wmnet with OS bullseye [12:42:42] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Connect two hosts in codfw row A/B for switch migration testing - https://phabricator.wikimedia.org/T345803 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmooney@cumin1001 for host sretest2004.codfw.wmnet with OS bullseye... [12:43:25] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Connect two hosts in codfw row A/B for switch migration testing - https://phabricator.wikimedia.org/T345803 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmooney@cumin1001 for host sretest2003.codfw.wmnet with OS bullseye [12:48:54] (SystemdUnitFailed) firing: (2) production-images-weekly-rebuild.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:59:25] (SystemdUnitFailed) firing: (3) production-images-weekly-rebuild.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:03:54] (SystemdUnitFailed) firing: (3) production-images-weekly-rebuild.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:12:46] 10CAS-SSO, 10Cloud-VPS, 10Infrastructure-Foundations, 10cloud-services-team: Create OpenID Connect client - https://phabricator.wikimedia.org/T350725 (10CCicalese_WMF) Works perfectly! Thank you! [13:26:15] 10SRE-tools, 10Infrastructure-Foundations, 10Puppet-Core, 10SRE, and 2 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619 (10MoritzMuehlenhoff) [13:28:46] 10SRE-tools, 10Infrastructure-Foundations, 10Puppet-Core, 10SRE, and 2 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619 (10MoritzMuehlenhoff) [13:31:38] 10netops, 10Infrastructure-Foundations, 10sre-alert-triage: Alert in need of triage: BGP status (instance cr2-eqdfw) - https://phabricator.wikimedia.org/T351083 (10LSobanski) [13:53:54] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Connect two hosts in codfw row A/B for switch migration testing - https://phabricator.wikimedia.org/T345803 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmooney@cumin1001 for host sretest2003.codfw.wmnet with OS bullseye... [13:57:57] 10SRE-tools, 10Infrastructure-Foundations, 10Puppet-Core, 10SRE, and 2 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619 (10MoritzMuehlenhoff) [14:28:12] 10SRE-tools, 10Infrastructure-Foundations, 10Puppet-Core, 10SRE, 10Puppet (Puppet 7.0): Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619 (10MoritzMuehlenhoff) [14:38:55] (SystemdUnitFailed) firing: (2) envoyproxy.service Failed on apt-staging2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:56:36] folks I'm not feeling so hot, been getting worse all day. gonna lie down for an hour or two get a break from the screen [14:57:20] get well! [14:59:25] (SystemdUnitFailed) firing: (3) envoyproxy.service Failed on apt-staging2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:01:30] i found a "fun" bug in the nftables puppetization: https://phabricator.wikimedia.org/T351094 [15:02:50] 10Puppet, 10MediaModeration (MediaModeration 2.0): Add mediamoderation_scan to the private tables list on puppet - https://phabricator.wikimedia.org/T351095 (10Dreamy_Jazz) [15:03:12] 10Puppet, 10MediaModeration (MediaModeration 2.0): Add mediamoderation_scan to the private tables list on puppet - https://phabricator.wikimedia.org/T351095 (10Dreamy_Jazz) a:03Dreamy_Jazz [15:03:23] 10Puppet, 10MediaModeration (MediaModeration 2.0), 10Trust and Safety Product Sprint: Add mediamoderation_scan to the private tables list on puppet - https://phabricator.wikimedia.org/T351095 (10Dreamy_Jazz) [15:03:31] 10Puppet, 10MediaModeration (MediaModeration 2.0), 10Trust and Safety Product Sprint: [S] Add mediamoderation_scan to the private tables list on puppet - https://phabricator.wikimedia.org/T351095 (10Dreamy_Jazz) [15:03:39] taavi: thanks in a metting now but wil try to send a fix after [15:03:41] 10Puppet, 10MediaModeration (MediaModeration 2.0), 10Trust and Safety Product Sprint (Sprint Bodhrán): [S] Add mediamoderation_scan to the private tables list on puppet - https://phabricator.wikimedia.org/T351095 (10Dreamy_Jazz) [15:03:46] have a good idea where the issues is [15:03:55] (SystemdUnitFailed) firing: (3) envoyproxy.service Failed on apt-staging2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:04:16] <3 [15:06:22] take care topranks [15:06:48] ineed [15:08:33] 10Puppet, 10MediaModeration (MediaModeration 2.0), 10Patch-For-Review, 10Trust and Safety Product Sprint (Sprint Bodhrán): [S] Add mediamoderation_scan to the private tables list on puppet - https://phabricator.wikimedia.org/T351095 (10Dreamy_Jazz) [15:09:04] 10Puppet, 10MediaModeration (MediaModeration 2.0), 10Patch-For-Review, 10Trust and Safety Product Sprint (Sprint Bodhrán): [S] Add mediamoderation_scan to the private tables list on puppet - https://phabricator.wikimedia.org/T351095 (10Dreamy_Jazz) [15:23:59] (PuppetFailure) firing: Puppet has failed on apt-staging2001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [15:48:13] 10Puppet, 10MobileFrontend (Tracking), 10User-Jdlrobson: Mobile site does not automatically redirect to desktop version (and not possible to use browser "use desktop view") - https://phabricator.wikimedia.org/T60425 (10jbond) [15:49:32] 10Puppet, 10Infrastructure-Foundations, 10SRE, 10conftool: confd fails to start after a reimage - https://phabricator.wikimedia.org/T244477 (10jbond) I have a feeling this is fixed we should see if its still present [15:55:30] 10Puppet, 10Infrastructure-Foundations, 10Puppet-Core, 10User-jbond: puppetlabs: create puppet 7 environment in WMCS to test code - https://phabricator.wikimedia.org/T294841 (10jbond) 05In progress→03Resolved this is available in the puppet-dev project [16:01:57] 10netbox, 10Infrastructure-Foundations, 10Puppet-Core, 10SRE, and 3 others: Netbox: use the netbox to also sync networks - https://phabricator.wikimedia.org/T329669 (10joanna_borun) [16:02:19] 10Puppet, 10Infrastructure-Foundations, 10SRE, 10User-Joe: Update puppet code to conform to puppet 4.x and later standards - https://phabricator.wikimedia.org/T181967 (10jbond) [16:03:04] 10Puppet, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review, 10User-Joe: Disable hiera autolookups - https://phabricator.wikimedia.org/T181971 (10jbond) 05Open→03Declined im going to close this as its [[ https://phabricator.wikimedia.org/T181971#5967526 | no longer possible ]] [16:03:39] 10netbox, 10Infrastructure-Foundations, 10Puppet-Core, 10SRE, and 3 others: Netbox: use the netbox to also sync networks - https://phabricator.wikimedia.org/T329669 (10joanna_borun) a:05jbond→03cmooney [16:04:35] 10Packaging, 10Infrastructure-Foundations, 10Patch-For-Review: apt: improve apt failover orchestration - https://phabricator.wikimedia.org/T330849 (10joanna_borun) [16:15:09] 10SRE-tools, 10Infrastructure-Foundations, 10Puppet-Core, 10SRE, and 2 others: Create cookbook to migrate servers from the puppetmasters to puppetservers - https://phabricator.wikimedia.org/T340739 (10jbond) 05Open→03Resolved a:03jbond this is complete [16:23:55] (SystemdUnitFailed) firing: (3) envoyproxy.service Failed on apt-staging2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:24:25] (SystemdUnitFailed) firing: (3) envoyproxy.service Failed on apt-staging2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:30:05] jhathaway: for the decom one lmk if you are gonna take it or should I, I didn't get it as we both were offering :D [16:30:41] will do [16:43:55] (SystemdUnitFailed) firing: (3) envoyproxy.service Failed on apt-staging2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:43:59] (PuppetFailure) resolved: Puppet has failed on apt-staging2001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [16:44:03] k, thx [17:00:47] 10SRE-tools, 10Infrastructure-Foundations, 10Puppet-Infrastructure, 10SRE, and 2 others: Update reimage cookbooks to work with puppet7 - https://phabricator.wikimedia.org/T348319 (10Volans) a:03Volans [17:43:55] (SystemdUnitFailed) firing: (3) envoyproxy.service Failed on apt-staging2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:08:55] (SystemdUnitFailed) firing: (2) envoyproxy.service Failed on apt-staging2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed