[00:05:25] FIRING: SystemdUnitFailed: kernel-purge.service on ganeti1039:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:05:40] FIRING: SystemdUnitFailed: kernel-purge.service on ganeti1039:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:27:59] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN: Setup VRRP on both routers for the new subnets - https://phabricator.wikimedia.org/T427393#11970653 (10ayounsi) `--move-vlan` is only made to migrate core DCs from legacy to new per rack vlans. Let me know if its worth spending... [08:05:40] FIRING: SystemdUnitFailed: kernel-purge.service on ganeti1039:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:07:42] moritzm: ^ [08:50:07] 10netops, 06Infrastructure-Foundations, 06SRE: Don't announce OSPF routes in unicast BGP on Nokia SR-Linux - https://phabricator.wikimedia.org/T423430#11971143 (10ayounsi) Once this is fixed we can remove `|ibgp` from the [[ https://gerrit.wikimedia.org/r/c/operations/alerts/+/1295805 | RejectingBGPPrefixes... [09:34:25] 10SRE-tools, 06DBA, 10Spicerack: Provide downtime duration information in sre.mysql cookbooks - https://phabricator.wikimedia.org/T427780#11971342 (10Marostegui) @elukey any input on this? [09:34:34] 10SRE-tools, 06DBA, 06Infrastructure-Foundations, 10Spicerack: Provide downtime duration information in sre.mysql cookbooks - https://phabricator.wikimedia.org/T427780#11971346 (10Marostegui) p:05Triage→03Medium [09:35:27] 10SRE-tools, 06DBA, 06Infrastructure-Foundations, 10Spicerack: Provide downtime duration information in sre.mysql cookbooks - https://phabricator.wikimedia.org/T427780#11971353 (10Marostegui) [09:45:25] RESOLVED: SystemdUnitFailed: kernel-purge.service on ganeti1039:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:25:09] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN: Setup VRRP on both routers for the new subnets - https://phabricator.wikimedia.org/T427393#11971500 (10cmooney) >>! In T427393#11970653, @ayounsi wrote: > `--move-vlan` is only made to migrate core DCs from legacy to new per rac... [12:05:25] 10netops, 06Discovery-Search, 06Infrastructure-Foundations, 06Machine-Learning-Team, and 3 others: codfw: rack A4 maintenance - https://phabricator.wikimedia.org/T427357#11971733 (10ops-monitoring-bot) Draining ganeti2027.codfw.wmnet of running VMs [12:08:34] 10netops, 06Discovery-Search, 06Infrastructure-Foundations, 06Machine-Learning-Team, and 3 others: codfw: rack A4 maintenance - https://phabricator.wikimedia.org/T427357#11971736 (10ops-monitoring-bot) VM kubestagemaster2005.codfw.wmnet switching disk type to drbd [12:26:26] 10netops, 06Discovery-Search, 06Infrastructure-Foundations, 06Machine-Learning-Team, and 3 others: codfw: rack A4 maintenance - https://phabricator.wikimedia.org/T427357#11971797 (10ops-monitoring-bot) Draining ganeti2027.codfw.wmnet of running VMs [12:27:29] 10netops, 06Discovery-Search, 06Infrastructure-Foundations, 06Machine-Learning-Team, and 3 others: codfw: rack A4 maintenance - https://phabricator.wikimedia.org/T427357#11971798 (10ops-monitoring-bot) VM kubestagemaster2005.codfw.wmnet switching disk type to plain [12:29:14] 10netops, 06Discovery-Search, 06Infrastructure-Foundations, 06Machine-Learning-Team, and 3 others: codfw: rack A4 maintenance - https://phabricator.wikimedia.org/T427357#11971800 (10ops-monitoring-bot) Draining ganeti2027.codfw.wmnet of running VMs [14:20:04] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations: bird bfd session with 172.20.1.1 down - Bad packet from 172.20.1.1 - unknown session id - https://phabricator.wikimedia.org/T427202#11972440 (10ayounsi) p:05Triage→03Low [14:20:17] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO: Unrack old switches (asw2-22/23-ulsfo) - https://phabricator.wikimedia.org/T427283#11972441 (10ayounsi) p:05Triage→03Low [14:20:41] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN: Setup VRRP on both routers for the new subnets - https://phabricator.wikimedia.org/T427393#11972454 (10ayounsi) p:05Triage→03Medium [14:21:24] 10Mail, 06Infrastructure-Foundations: Email recipient rate limiting for Postfix - https://phabricator.wikimedia.org/T427002#11972458 (10LSobanski) p:05Triage→03Medium [14:21:58] 10Mail, 06Infrastructure-Foundations: Email recipient rate limiting for Postfix - https://phabricator.wikimedia.org/T427002#11972461 (10LSobanski) Lower priority than {T427001} as providing the right level of logging detail will be trickier here compared to MW. [14:35:55] 10SRE-tools, 06Infrastructure-Foundations, 06SRE: Decom cookbook should only warn about unexpected matches in Puppet - https://phabricator.wikimedia.org/T297516#11972594 (10LSobanski) This looks resolved. @RLazarus please reopen if you think otherwise. [14:38:45] 10netops, 06SRE, 06Traffic-Icebox: experiment with reenabling compression between applayer's TLS terminators and edge caches - https://phabricator.wikimedia.org/T263288#11972619 (10LSobanski) Untagging IF. [14:39:44] 10SRE-tools, 06Infrastructure-Foundations, 06SRE: reimage cookbook should exit cleanly if no puppet role is applied to a node - https://phabricator.wikimedia.org/T338990#11972623 (10LSobanski) p:05Medium→03Low [15:35:47] 10CAS-SSO, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations: "Application Not Authorized to Use CAS" error when attempting to authenticate to IDP - https://phabricator.wikimedia.org/T427826 (10bd808) 03NEW [15:39:32] 10CAS-SSO, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations: "Application Not Authorized to Use CAS" error when attempting to authenticate to IDP - https://phabricator.wikimedia.org/T427826#11972970 (10Manhgay2323) oke có gì tôi sẽ sửa lại lỗi này. Cảm ơn bạn [16:03:30] 10netops, 06Discovery-Search, 06Infrastructure-Foundations, 06Machine-Learning-Team, and 3 others: codfw: rack A4 maintenance - https://phabricator.wikimedia.org/T427357#11973104 (10ops-monitoring-bot) Draining ganeti2045.codfw.wmnet of running VMs [16:04:17] 10CAS-SSO, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations: "Application Not Authorized to Use CAS" error when attempting to authenticate to IDP - https://phabricator.wikimedia.org/T427826#11973107 (10taavi) 05Open→03Resolved a:03taavi `lang=irc this is very unsatisfying... [16:05:38] 10netops, 06Discovery-Search, 06Infrastructure-Foundations, 06Machine-Learning-Team, and 3 others: codfw: rack A4 maintenance - https://phabricator.wikimedia.org/T427357#11973112 (10ops-monitoring-bot) VM aux-k8s-etcd2003.codfw.wmnet switching disk type to drbd [16:15:24] 10netops, 06Discovery-Search, 06Infrastructure-Foundations, 06Machine-Learning-Team, and 3 others: codfw: rack A4 maintenance - https://phabricator.wikimedia.org/T427357#11973137 (10MoritzMuehlenhoff) [17:00:40] 10netops, 06Discovery-Search, 06Infrastructure-Foundations, 06Machine-Learning-Team, and 3 others: codfw: rack A4 maintenance - https://phabricator.wikimedia.org/T427357#11973321 (10ops-monitoring-bot) VM dse-k8s-etcd2001.codfw.wmnet switching disk type to drbd [17:59:30] 10netops, 06Discovery-Search, 06Infrastructure-Foundations, 06Machine-Learning-Team, and 3 others: codfw: rack A4 maintenance - https://phabricator.wikimedia.org/T427357#11973526 (10ops-monitoring-bot) Draining ganeti2045.codfw.wmnet of running VMs [18:01:38] 10netops, 06Discovery-Search, 06Infrastructure-Foundations, 06Machine-Learning-Team, and 3 others: codfw: rack A4 maintenance - https://phabricator.wikimedia.org/T427357#11973540 (10ops-monitoring-bot) VM aux-k8s-etcd2003.codfw.wmnet switching disk type to plain [18:03:04] 10netops, 06Discovery-Search, 06Infrastructure-Foundations, 06Machine-Learning-Team, and 3 others: codfw: rack A4 maintenance - https://phabricator.wikimedia.org/T427357#11973544 (10ops-monitoring-bot) VM dse-k8s-etcd2001.codfw.wmnet switching disk type to plain [18:05:53] 10netops, 06Discovery-Search, 06Infrastructure-Foundations, 06Machine-Learning-Team, and 3 others: codfw: rack A4 maintenance - https://phabricator.wikimedia.org/T427357#11973560 (10ops-monitoring-bot) Draining ganeti2045.codfw.wmnet of running VMs [23:14:25] FIRING: SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed