[09:58:16] 10netops, 06Infrastructure-Foundations: Downgrade pfw1-codfw to Junos 23.4R2-S3 - https://phabricator.wikimedia.org/T393996#11305617 (10cmooney) Just on this one we hit another issue with gnmi stats for this one. In a change made a few days ago we added the "openconfig:" prefix to the BGP stats collection, wh... [10:46:13] 10netops, 06Infrastructure-Foundations: Downgrade pfw1-codfw to Junos 23.4R2-S3 - https://phabricator.wikimedia.org/T393996#11305689 (10cmooney) a:05cmooney→03None [10:49:13] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Eqiad C/D refresh: 2 x test hosts for config validation - https://phabricator.wikimedia.org/T405560#11305694 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmooney@cumin1003 for host sretest1006.eqiad.wmnet... [11:18:41] FIRING: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting [12:09:29] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Eqiad C/D refresh: 2 x test hosts for config validation - https://phabricator.wikimedia.org/T405560#11305892 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmooney@cumin1003 for host sretest1006.eqiad.wmnet with... [12:19:03] RESOLVED: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting [13:56:35] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Eqiad C/D refresh: 2 x test hosts for config validation - https://phabricator.wikimedia.org/T405560#11306225 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmooney@cumin1003 for host sretest1006.eqiad.wmnet... [15:19:07] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Eqiad C/D refresh: 2 x test hosts for config validation - https://phabricator.wikimedia.org/T405560#11306704 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmooney@cumin1003 for host sretest1006.eqiad.wmnet with... [18:28:04] 10Packaging, 06Infrastructure-Foundations, 13Patch-For-Review: apt: improve apt failover orchestration - https://phabricator.wikimedia.org/T330849#11307814 (10CDanis) For the record, I needed something similar but different for DNS Discovery records: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1198583 [19:51:03] 07Puppet, 10MobileFrontend (Tracking): Mobile site does not automatically redirect to desktop version (and not possible to use browser "use desktop view") - https://phabricator.wikimedia.org/T60425#11308121 (10Krinkle) a:03Krinkle Done as of yesterday, via {T405931}. [19:51:13] 07Puppet, 10MobileFrontend (Tracking): Mobile site does not automatically redirect to desktop version (and not possible to use browser "use desktop view") - https://phabricator.wikimedia.org/T60425#11308125 (10Krinkle) 05Open→03Resolved [21:06:37] makevm consistently fails to create a VM in codfw..but has no problem in eqiad or esams for example. the error is just that cumin execution failed for the actual gnt-instance command. so more on the ganeti side. [21:13:00] mutante this is a silly suggestion, but do you have the right partman pattern for the codfw hosts? [21:13:48] yes, I am using a wildcard with * [21:14:24] if it's partman then I would end up with a VM but no OS and be able to reimage it [21:14:30] this one fails to create the VM [21:16:39] then it exists under VMs in netbox but does not exist in ganeti [21:17:37] if you don't manually delete it from netbox and try the cookbook again it would want to create the name in DNS a second time but with the next IP [21:20:53] will create a ticket for it later