[17:06:00] (03update) 10dcaro: quota: adapt the quota to the new default cpu [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/75 [17:08:16] 06cloud-services-team, 10Toolforge (Toolforge iteration 24): [infra,haproxy,ingress] 2025-09-23 Ingress hitting the backend session limit and started replying with 5xxs - https://phabricator.wikimedia.org/T405280#11206981 (10taavi) 05In progress→03Resolved [17:08:49] 10Tool-paulina: Outreachy 31: Features to edit author and work data on Wikidata directly from Paulina - https://phabricator.wikimedia.org/T392429#11206982 (10Pepe_piton) [17:14:59] (03update) 10dcaro: [jobs-api] split job models to oneoff, scheduled and continuous [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/154 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136) (owner: 10raymond-ndibe) [17:16:42] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1043.eqiad.wmnet}' [17:16:43] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1044.eqiad.wmnet}' [17:17:49] FIRING: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1043 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [17:19:39] (03update) 10dcaro: [jobs-api] split job models to oneoff, scheduled and continuous [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/154 (https://phabricator.wikimedia.org/T389118 https://phabricator.wikimedia.org/T390136) (owner: 10raymond-ndibe) [17:19:49] (03update) 10dcaro: quota: adapt the quota to the new default cpu [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/75 [17:22:48] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1044.eqiad.wmnet}' [17:22:49] RESOLVED: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1043 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [17:22:50] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1045.eqiad.wmnet}' [17:24:38] (03update) 10dcaro: jobs-api: bump to 0.0.416-20250923131926-45b7b4f9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/977 (https://phabricator.wikimedia.org/T404726) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:25:58] (03update) 10dcaro: jobs-api: bump to 0.0.416-20250923131926-45b7b4f9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/977 (https://phabricator.wikimedia.org/T404726) (owner: 10group_203_bot_f4d95069bb2675e4ce1fff090c1c1620) [17:29:43] 06cloud-services-team, 10PAWS: Grant membership in cloud-vps project 'PAWS' to vivian rook for volunteer work - https://phabricator.wikimedia.org/T400733#11207084 (10taavi) 05Stalled→03Declined Marking as declined as my understanding is that there's no clear way forward for this so keeping it open seem... [17:30:21] 06cloud-services-team: Systems and service continuity and availability constraints - https://phabricator.wikimedia.org/T234656#11207090 (10taavi) 05Stalled→03Invalid I don't see anything actionable here. [17:31:01] 06cloud-services-team, 10Cloud-VPS: designate: database having deadlock problems - https://phabricator.wikimedia.org/T270762#11207093 (10taavi) 05Open→03Resolved I don't think we've seen this recently. [17:37:42] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1045.eqiad.wmnet}' [17:37:43] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1046.eqiad.wmnet}' [17:38:53] 06cloud-services-team, 10Toolforge, 10Tools, 13Patch-For-Review: Update toolserver.org redirects to use toolforge.org - https://phabricator.wikimedia.org/T271862#11207128 (10taavi) a:03taavi [17:39:19] FIRING: [2x] NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1043 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [17:42:07] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1046.eqiad.wmnet}' [17:42:09] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1047.eqiad.wmnet}' [17:42:59] 06cloud-services-team, 10Cloud-VPS, 07Documentation: Improve documentation on SSH host fingerprints - https://phabricator.wikimedia.org/T193648#11207158 (10taavi) 05Open→03Resolved [17:44:09] 10cloud-services-team (FY2025/26-Q1), 10Toolforge (Toolforge iteration 24), 13Patch-For-Review: [builds-builder] Add support for Heroku's "24" builder stack based on Ubuntu 2024.04 noble - https://phabricator.wikimedia.org/T380127#11207163 (10Don-vip) @dcaro Is it possible to test this new heroku stack? I'm... [17:44:19] RESOLVED: [2x] NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1045 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [17:46:03] 06cloud-services-team, 06Infrastructure-Foundations, 07Security: Disable insecure rsa-ssh public key signature algorithm - https://phabricator.wikimedia.org/T318345#11207167 (10taavi) 05Open→03Resolved a:03taavi This is done with the latest set of bastions. [17:47:32] 06cloud-services-team, 10Cloud-VPS, 07Upstream: neutron agents losing RabbitMQ connectivity don't crash properly - https://phabricator.wikimedia.org/T311149#11207186 (10taavi) 05Open→03Resolved We haven't seen this in a while so I'm assuming it's been fixed upstream. [17:49:09] 06cloud-services-team, 10Cloud-VPS: cloudvirt1025-1030 overheating issues - https://phabricator.wikimedia.org/T289159#11207195 (10taavi) 05Open→03Invalid these hosts no longer exist [17:49:28] 06cloud-services-team: Create a standup bot for wikimedia-cloud-private - https://phabricator.wikimedia.org/T288251#11207199 (10taavi) 05Open→03Declined Assuming not. [17:50:53] 06cloud-services-team, 10Toolforge: Monitor Toolforge external kubernetes certificate expiry - https://phabricator.wikimedia.org/T284310#11207208 (10taavi) →14Duplicate dup:03T366579 [17:50:54] 06cloud-services-team, 14Toolforge (Toolforge iteration 16): [infra,k8s,monitoring] Add an alert to warn when the prometheus k8s cert is about to expire - https://phabricator.wikimedia.org/T366579#11207210 (10taavi) [17:52:53] 06cloud-services-team, 10Cloud-VPS: ceph: device health metrics are failing to get scraped - https://phabricator.wikimedia.org/T281350#11207220 (10taavi) 05Open→03Resolved Based on the error in the task I'm assuming https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/55b07929c6df727853b5... [17:53:14] 06cloud-services-team, 10Toolforge: Toolforge: clarify usefullness of 'deb-tools.wmflabs.org' and refresh it if so - https://phabricator.wikimedia.org/T278436#11207224 (10taavi) 05Open→03Resolved This is useful and now deb.svc.toolforge.org. [17:53:55] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [17:54:34] 06cloud-services-team, 10Technical Blog: import posts from 'Clouds & Unicorns' into Wikimedia Tech Blog - https://phabricator.wikimedia.org/T278017#11207229 (10taavi) 05Open→03Declined Meh, this doesn't seem particularly relevant. [17:56:09] 06cloud-services-team, 10Cloud-VPS: CloudVPS DNS: double check if PTR records for floating IPs and instances are right - https://phabricator.wikimedia.org/T266805#11207237 (10taavi) 05Open→03Invalid The sync script is working as expected there. [17:56:23] 06cloud-services-team, 10Cloud-VPS: [ceph][icinga] Use health details when checking the alarm and include the info in icinga alerts - https://phabricator.wikimedia.org/T268885#11207239 (10taavi) 05Open→03Declined Assuming not. [17:58:14] PROBLEM - Host cloudcephosd1037 is DOWN: PING CRITICAL - Packet loss = 100% [17:58:14] 06cloud-services-team, 10Cloud-VPS: bad failure cases for wmcs custom puppet enc - https://phabricator.wikimedia.org/T262350#11207243 (10taavi) 05Open→03Resolved [18:00:42] RECOVERY - Host cloudcephosd1037 is UP: PING OK - Packet loss = 0%, RTA = 0.31 ms [18:00:53] 06cloud-services-team, 10Toolforge: [tools] Backup/provide HA for the puppetmaster private repo - https://phabricator.wikimedia.org/T145883#11207256 (10taavi) 05Open→03Resolved We have backups of the Cinder volume. [18:02:02] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.reactivate (exit_code=0) [18:03:19] 06cloud-services-team, 10Toolforge: Toolforge: refresh grafana dashboard - https://phabricator.wikimedia.org/T213567#11207287 (10taavi) 05Open→03Invalid This does not seem relevant anymore. [18:03:52] 06cloud-services-team, 10observability: Improve Dell server hardware monitoring - https://phabricator.wikimedia.org/T243533#11207292 (10taavi) →14Duplicate dup:03T253810 [18:04:02] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1047.eqiad.wmnet}' [18:04:03] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1048.eqiad.wmnet}' [18:04:39] 06cloud-services-team, 10Data-Services: comment and actor view challenges for Cloud Services - https://phabricator.wikimedia.org/T215445#11207305 (10taavi) 05Open→03Resolved As far as I can tell there is no plan to change the current subview setup. So closing to avoid this lingering on various workboar... [18:05:42] 06cloud-services-team, 10Data-Services: Denormalize user_groups to contain actor information - https://phabricator.wikimedia.org/T238497#11207311 (10taavi) 05Open→03Declined I'm declining since maintaining additional "real" columns would be very complicated for the relatively little benefit they give. [18:05:49] FIRING: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1047 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [18:06:14] 06cloud-services-team, 10Toolforge, 07Documentation: Document procedure for controlled cluster restart - https://phabricator.wikimedia.org/T216688#11207314 (10taavi) 05Open→03Invalid There is no specific order that needs following. [18:07:12] 06cloud-services-team: Action items and work for retro 20190703 - https://phabricator.wikimedia.org/T227211#11207318 (10taavi) 05Open→03Invalid This does not seem like a useful tracking task after 6 years :-) [18:10:49] RESOLVED: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1047 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [18:28:31] (03update) 10raymond-ndibe: [status] make job status an enum, with clearly defined states [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/208 (https://phabricator.wikimedia.org/T401172) [18:31:59] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1048.eqiad.wmnet}' [18:32:01] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1049.eqiad.wmnet}' [18:33:49] FIRING: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1048 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [18:36:00] 06cloud-services-team, 10Toolforge: [components-api] rebuilds un-changed images - https://phabricator.wikimedia.org/T403167#11207430 (10Raymond_Ndibe) >>! In T403167#11184588, @DamianZaremba wrote: > Hi @Raymond_Ndibe, > > Essentially what you describe is how you get into this state. > > I included it as an... [18:38:49] RESOLVED: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1048 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [18:51:38] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1049.eqiad.wmnet}' [18:52:16] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [18:53:05] !log andrew@cloudcumin1001 admin END (ERROR) - Cookbook wmcs.ceph.osd.reactivate (exit_code=97) [18:53:10] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [18:53:49] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.reactivate (exit_code=0) [18:53:49] FIRING: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1049 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [18:54:23] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [18:54:27] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.reactivate (exit_code=0) [18:55:57] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1050.eqiad.wmnet}' [18:58:49] RESOLVED: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1049 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [19:19:31] PROBLEM - Host cloudvirt1050 is DOWN: PING CRITICAL - Packet loss = 100% [19:21:25] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1050.eqiad.wmnet}' [19:21:26] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1051.eqiad.wmnet}' [19:21:49] FIRING: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1050 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [19:21:59] RECOVERY - Host cloudvirt1050 is UP: PING OK - Packet loss = 0%, RTA = 0.34 ms [19:26:49] RESOLVED: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1050 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [19:38:36] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [19:39:17] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.ceph.osd.reactivate (exit_code=99) [19:39:28] FIRING: [4x] PuppetAgentNoResources: No Puppet resources found on instance tools-harbor-2 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:42:08] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [19:42:39] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.reactivate (exit_code=0) [19:43:20] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1051.eqiad.wmnet}' [19:43:21] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1052.eqiad.wmnet}' [19:44:28] FIRING: [27x] PuppetAgentNoResources: No Puppet resources found on instance tools-bastion-15 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:45:49] FIRING: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1051 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [19:49:28] FIRING: [42x] PuppetAgentNoResources: No Puppet resources found on instance tools-acme-chief-3 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:50:49] RESOLVED: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1051 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [19:54:28] FIRING: [56x] PuppetAgentNoResources: No Puppet resources found on instance tools-acme-chief-3 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:59:28] FIRING: [76x] PuppetAgentNoResources: No Puppet resources found on instance tools-acme-chief-3 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [20:02:53] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1052.eqiad.wmnet}' [20:02:55] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1053.eqiad.wmnet}' [20:04:28] FIRING: [98x] PuppetAgentNoResources: No Puppet resources found on instance tools-acme-chief-3 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [20:09:28] FIRING: [113x] PuppetAgentNoResources: No Puppet resources found on instance tools-acme-chief-3 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [20:14:28] FIRING: [113x] PuppetAgentNoResources: No Puppet resources found on instance tools-acme-chief-3 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [20:19:28] FIRING: [113x] PuppetAgentNoResources: No Puppet resources found on instance tools-acme-chief-3 on project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [20:27:01] (03PS6) 10Krinkle: Replace divs with semantic elements [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/1091830 (https://phabricator.wikimedia.org/T227631) (owner: 10SomeRandomDeveloper) [20:28:13] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1053.eqiad.wmnet}' [20:28:15] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1054.eqiad.wmnet}' [20:29:19] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.ceph.osd.reactivate [20:29:28] (03PS7) 10Krinkle: Replace divs with semantic elements [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/1091830 (https://phabricator.wikimedia.org/T227631) (owner: 10SomeRandomDeveloper) [20:29:49] FIRING: NeutronAgentDown: Neutron neutron-openvswitch-agent on cloudvirt1053 is down - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Troubleshooting#Networking_failures - https://grafana.wikimedia.org/d/wKnDJf97z/wmcs-neutron-eqiad1 - https://alerts.wikimedia.org/?q=alertname%3DNeutronAgentDown [20:29:53] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.ceph.osd.reactivate (exit_code=0) [20:30:03] (03CR) 10Krinkle: [C:03+2] "I've added a cache invalidation to the stylesheet URL since these are backwards-incompatible changes. That way, people will not encounter " [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/1091830 (https://phabricator.wikimedia.org/T227631) (owner: 10SomeRandomDeveloper) [20:30:22] (03Abandoned) 10Krinkle: Replaced
by
,
by