[01:51:49] FIRING: PuppetDisabled: Puppet disabled on apt-staging2001:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=misc&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled [02:13:11] zip: I finally understood you meant cloud == public clouds like AWS and not cloud == WMCS [02:14:11] re: Maxmind, it's a paid service yes, but the model is a downloadable database (laid out as a trie, so it's very efficient to query by IP address) [02:14:57] where are the logs you're looking at? our webrequest logs in the analytics cluster already have the maxmind ISP data joined into them [02:17:01] I think we're maybe overdue for some work in general on bot detection / traffic pattern characterization [05:51:49] FIRING: PuppetDisabled: Puppet disabled on apt-staging2001:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=misc&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled [07:38:35] 10netbox, 06Infrastructure-Foundations, 10Observability-Alerting, 10SRE Observability (FY2024/2025-Q2): Port netbox reports checks to Prometheus/Alertmanager - https://phabricator.wikimedia.org/T374823#10283525 (10SLyngshede-WMF) 05Open→03In progress [09:51:49] FIRING: PuppetDisabled: Puppet disabled on apt-staging2001:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=misc&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled [10:11:44] oh, it's been a while since I touched this but it was somewhere in monitoring. i'm out today so i'll circle back on monday [10:11:56] if i can throw that in to my queries that'd be amazing [10:13:24] (i apparently cannot tolerate leaving a notification unread for three days) [10:42:18] 10netops, 06DC-Ops, 06Infrastructure-Foundations: Take advantage of 10Gb NICs in the new network stack - https://phabricator.wikimedia.org/T360297#10283802 (10cmooney) >>! In T360297#10276313, @ayounsi wrote: > @cmooney @Papaul What do you think of: > # Keeping the new script presented previously for the "ea... [11:49:01] I put some information about ganeti1025 on #wikimedia-sre. I think ganeti1025 has seen better days.. [13:05:47] 10netops, 06Infrastructure-Foundations, 06SRE: ganeti1025 VMs unresponsive Nov 1 2024 - https://phabricator.wikimedia.org/T378809 (10cmooney) 03NEW p:05Triage→03Medium [13:34:03] 10netops, 06Infrastructure-Foundations, 06SRE: ganeti1025 VMs unresponsive Nov 1 2024 - https://phabricator.wikimedia.org/T378809#10284072 (10cmooney) [13:34:41] 10SRE-tools, 06DC-Ops, 06Infrastructure-Foundations, 10Spicerack: Upload redfish licenses to supermicro hosts - https://phabricator.wikimedia.org/T376121#10284086 (10elukey) [13:35:20] 10SRE-tools, 06DC-Ops, 06Infrastructure-Foundations, 10Spicerack: Upload redfish licenses to supermicro hosts - https://phabricator.wikimedia.org/T376121#10284087 (10elukey) 05Open→03Resolved Supermicro sent a new license for 1044 that worked, and I've ran successfully the provision cookbook. [13:51:49] FIRING: PuppetDisabled: Puppet disabled on apt-staging2001:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=misc&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled [14:01:51] 10netbox, 06Infrastructure-Foundations: Netbox: ImportPuppetDB uses wrong netmask for some hosts - https://phabricator.wikimedia.org/T378751#10284162 (10cmooney) Yeah this is an odd one. And tbh I'm not sure I 100% understand the original reasoning for this and if we still need it. I guess the goal is to tak... [14:27:19] 10netops, 06Infrastructure-Foundations, 06SRE: ganeti1025 VMs unresponsive Nov 1 2024 - https://phabricator.wikimedia.org/T378809#10284231 (10CDanis) I'm pretty confident this is the same as T348730, and I think it would be okay to return ganeti1025 to service and close this task as a dup [14:35:13] 10netops, 06Infrastructure-Foundations, 06SRE: ganeti1025 VMs unresponsive Nov 1 2024 - https://phabricator.wikimedia.org/T378809#10284244 (10cmooney) 05Open→03Resolved a:03cmooney >>! In T378809#10284231, @CDanis wrote: > I'm pretty confident this is the same as T348730, and I think it would be ok... [14:41:46] topranks: fine to leave the instance as-is :) [14:42:27] cool, yeah I agree [17:51:49] FIRING: PuppetDisabled: Puppet disabled on apt-staging2001:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=misc&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled [19:03:44] 10netbox, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 10Puppet-Core: Make netbox the source of truth for cloudceph networks - https://phabricator.wikimedia.org/T338329#10285243 (10taavi) [21:51:49] FIRING: PuppetDisabled: Puppet disabled on apt-staging2001:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=misc&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled