[00:17:13] FIRING: JobUnavailable: Reduced availability for job redis_arclamp in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [00:21:47] FIRING: RedisInstanceDown: Redis instance down arclamp2001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_arclamp&var-instance=arclamp2001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [00:45:27] !log ryankemper@cumin2003 START - Cookbook sre.discovery.service-route pool wdqs-main in eqiad: maintenance [00:46:43] !log [WDQS] T435443 Restore eqiad wdqs-main; wdqs was unable to keep up with traffic with only one datacenter. sadly this will continue to be the case until wdqsv2 is ready to switch backend architecture [00:46:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:46:47] T435443: Day 1 - Source datacenter depooling (Tue. Sept. 22nd - 2026) - https://phabricator.wikimedia.org/T435443 [00:49:32] FIRING: JobQueueLowTrafficConsumerWidespreadHighLatency: ... [00:49:32] Processing delay times for low-traffic consumer rules are unusually high - https://wikitech.wikimedia.org/wiki/MediaWiki_JobQueue/Operations#JobQueueLowTrafficConsumerWidespreadHighLatency - https://grafana.wikimedia.org/d/fe130675-0c2d-4991-9dec-f54cf6a9c4d8/jobqueue-low-traffic-jobs?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DJobQueueLowTrafficConsumerWidespreadHighLatency [00:50:33] !log ryankemper@cumin2003 END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wdqs-main in eqiad: maintenance [01:02:50] (03CR) 10Ryan Kemper: "oops forgot to ever send these comments" [puppet] - 10https://gerrit.wikimedia.org/r/1318976 (https://phabricator.wikimedia.org/T276088) (owner: 10Ryan Kemper) [01:09:50] (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1344120 [01:09:50] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1344120 (owner: 10TrainBranchBot) [01:21:56] (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1344120 (owner: 10TrainBranchBot) [02:00:05] Deploy window Automatic deployment of MediaWiki to pretrain wikis - see mw:Pretrain (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260923T0200) [02:00:42] !log mwpresync@deploy1003 Started scap build-images: Publishing wmf/next image [02:06:41] FIRING: [3x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:08:22] !log mwpresync@deploy1003 Finished scap build-images: Publishing wmf/next image (duration: 07m 39s) [02:12:13] FIRING: [3x] JobUnavailable: Reduced availability for job redis_arclamp in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:17:13] FIRING: [3x] JobUnavailable: Reduced availability for job redis_arclamp in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [02:46:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [03:10:02] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T438865#12353285 (10phaultfinder) [03:25:51] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users and deployment for lwatson - https://phabricator.wikimedia.org/T438926 (10lwatson) 03NEW [03:35:16] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users and deployment for lwatson - https://phabricator.wikimedia.org/T438926#12353310 (10lwatson) Hey @HSwan-WMF, I'm requesting analytics and deployment access and it requires your approval. Please comment here in the request task that y... [04:14:47] (03PS1) 10Andrew Bogott: Puppetize keystone-uwsgi.ini [puppet] - 10https://gerrit.wikimedia.org/r/1344127 (https://phabricator.wikimedia.org/T421911) [04:15:08] (03CR) 10Andrew Bogott: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1344127 (https://phabricator.wikimedia.org/T421911) (owner: 10Andrew Bogott) [04:16:21] (03CR) 10Andrew Bogott: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1344127 (https://phabricator.wikimedia.org/T421911) (owner: 10Andrew Bogott) [04:18:56] (03CR) 10Andrew Bogott: [C:03+2] Puppetize keystone-uwsgi.ini [puppet] - 10https://gerrit.wikimedia.org/r/1344127 (https://phabricator.wikimedia.org/T421911) (owner: 10Andrew Bogott) [04:21:47] FIRING: RedisInstanceDown: Redis instance down arclamp2001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_arclamp&var-instance=arclamp2001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [04:49:32] FIRING: JobQueueLowTrafficConsumerWidespreadHighLatency: ... [04:49:32] Processing delay times for low-traffic consumer rules are unusually high - https://wikitech.wikimedia.org/wiki/MediaWiki_JobQueue/Operations#JobQueueLowTrafficConsumerWidespreadHighLatency - https://grafana.wikimedia.org/d/fe130675-0c2d-4991-9dec-f54cf6a9c4d8/jobqueue-low-traffic-jobs?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DJobQueueLowTrafficConsumerWidespreadHighLatency [05:16:48] (03PS1) 10Marostegui: installserver: Do not reimage db1287 [puppet] - 10https://gerrit.wikimedia.org/r/1344130 [05:21:50] (03CR) 10Marostegui: [C:03+2] installserver: Do not reimage db1287 [puppet] - 10https://gerrit.wikimedia.org/r/1344130 (owner: 10Marostegui) [05:51:07] (03CR) 10Trueg: "very nice." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344104 (https://phabricator.wikimedia.org/T438789) (owner: 10Lerickson) [06:00:06] Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260923T0600) [06:06:41] FIRING: [3x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:17:28] FIRING: JobUnavailable: Reduced availability for job redis_arclamp in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [06:24:06] 10SRE-swift-storage, 10Ceph, 06Infrastructure-Foundations: Move the majority of the Registry's docker image prefixes to a new s3 bucket - https://phabricator.wikimedia.org/T435499#12353517 (10elukey) Hey @Dzahn! It should be fixed now! For the record this is what I've done (from https://wikitech.wikimed... [06:27:27] (03PS1) 10Filippo Giunchedi: hieradata: remove 'wmcs' cluster [puppet] - 10https://gerrit.wikimedia.org/r/1344134 (https://phabricator.wikimedia.org/T437272) [06:28:53] (03CR) 10Elukey: [C:03+1] stdlib upgrade: Remove use of compat types [puppet] - 10https://gerrit.wikimedia.org/r/1344101 (https://phabricator.wikimedia.org/T438912) (owner: 10JHathaway) [06:29:56] (03CR) 10Muehlenhoff: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1344101 (https://phabricator.wikimedia.org/T438912) (owner: 10JHathaway) [06:35:49] (03CR) 10Muehlenhoff: [C:03+1] "Looks good, one commit inline (but feel free to ignore)" [puppet] - 10https://gerrit.wikimedia.org/r/1344004 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [06:43:53] (03PS2) 10Slyngshede: D:prometheus::trafficserver_exporter: Metrics for cache evacuation [puppet] - 10https://gerrit.wikimedia.org/r/1343309 (https://phabricator.wikimedia.org/T438627) [06:43:55] (03CR) 10Muehlenhoff: [C:03+1] "Looks good, one nit inline (feel free to ignore)" [puppet] - 10https://gerrit.wikimedia.org/r/1344005 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey) [06:44:43] (03CR) 10Slyngshede: D:prometheus::trafficserver_exporter: Metrics for cache evacuation (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/1343309 (https://phabricator.wikimedia.org/T438627) (owner: 10Slyngshede) [06:46:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:00:05] Amir1, urbanecm, and awight: gettimeofday() says it's time for UTC morning backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260923T0700) [07:00:05] No Gerrit patches in the queue for this window AFAICS. [07:01:40] (03CR) 10Muehlenhoff: [C:03+2] use_linux612_on_bookworm: Bump kernel to 6.12.107 [puppet] - 10https://gerrit.wikimedia.org/r/1343856 (owner: 10Muehlenhoff) [07:01:55] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users, growthbook-customelevatedaccess for Anil - https://phabricator.wikimedia.org/T437611#12353561 (10tappof) [07:03:42] (03PS1) 10Awight: Configure beta cluster to use the SiteMatrixLookup [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344139 [07:14:45] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T438865#12353571 (10phaultfinder) [07:16:10] (03PS2) 10Awight: Configure beta cluster to use the SiteMatrixLookup [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344139 (https://phabricator.wikimedia.org/T432369) [07:27:13] FIRING: [2x] JobUnavailable: Reduced availability for job atlas_exporter in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [07:29:52] !log installing python-idna security updates [07:29:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:34:55] (03PS1) 10Tiziano Fogli: Revert^2 "kafka-logging: add kafka-logging100[7-8] to eqiad cluster" [puppet] - 10https://gerrit.wikimedia.org/r/1344145 [07:36:34] (03CR) 10Ayounsi: [C:03+2] move-vlan: handle 'systemctl restart networking' better [cookbooks] - 10https://gerrit.wikimedia.org/r/1344018 (owner: 10Ayounsi) [07:40:49] (03PS3) 10Mamouri: Enable $wgCiteDefaultRefDirAuto on fawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1325997 (https://phabricator.wikimedia.org/T432544) [07:40:51] (03Merged) 10jenkins-bot: move-vlan: handle 'systemctl restart networking' better [cookbooks] - 10https://gerrit.wikimedia.org/r/1344018 (owner: 10Ayounsi) [07:47:15] jouncebot: nowandnext [07:47:15] For the next 0 hour(s) and 12 minute(s): UTC morning backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260923T0700) [07:47:15] In 2 hour(s) and 12 minute(s): MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260923T1000) [07:48:42] (03CR) 10Tiziano Fogli: [C:03+2] Revert^2 "kafka-logging: add kafka-logging100[7-8] to eqiad cluster" [puppet] - 10https://gerrit.wikimedia.org/r/1344145 (owner: 10Tiziano Fogli) [07:56:20] (03PS3) 10Jelto: service::catalog: Set ipip_encapsulation for k8s-ingress-wikikube in codfw [puppet] - 10https://gerrit.wikimedia.org/r/1311444 (https://phabricator.wikimedia.org/T420436) [07:56:20] (03CR) 10Jelto: "this question is resolved and tested with the other services" [puppet] - 10https://gerrit.wikimedia.org/r/1311444 (https://phabricator.wikimedia.org/T420436) (owner: 10Jelto) [07:56:27] (03PS1) 10Jelto: service::catalog: Set ipip_encapsulation for k8s-ingress-wikikube in eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1344197 (https://phabricator.wikimedia.org/T420436) [07:57:49] 06SRE, 06Infrastructure-Foundations: Integrate Trixie 13.6 point update - https://phabricator.wikimedia.org/T434866#12353643 (10MoritzMuehlenhoff) [07:58:35] !log tappof@deploy1003 helmfile [eqiad] START helmfile.d/admin 'sync'. [07:58:50] !log tappof@deploy1003 helmfile [eqiad] DONE helmfile.d/admin 'sync'. [07:59:26] !log installing giflib security updates [07:59:27] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:59:31] !log tappof@deploy1003 helmfile [codfw] START helmfile.d/admin 'sync'. [08:00:07] !log tappof@deploy1003 helmfile [codfw] DONE helmfile.d/admin 'sync'. [08:04:11] (03CR) 10Mamouri: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1325997 (https://phabricator.wikimedia.org/T432544) (owner: 10Mamouri) [08:04:18] !log tappof@deploy1003 Started scap sync-world: T432444 - Provision kafka-logging100[6-8] [08:04:22] T432444: Provision kafka-logging100[6-8] and kafka-logging200[6-8] - https://phabricator.wikimedia.org/T432444 [08:05:47] !log installing grub2 bugfix updates on Bookworm hosts [08:05:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:13:53] !log tappof@deploy1003 Finished scap sync-world: T432444 - Provision kafka-logging100[6-8] (duration: 12m 52s) [08:13:57] T432444: Provision kafka-logging100[6-8] and kafka-logging200[6-8] - https://phabricator.wikimedia.org/T432444 [08:14:54] (03PS1) 10Kevin Bazira: ml-services: point the tts-section-generator to use the tts isvc endpoint in eqiad [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344200 (https://phabricator.wikimedia.org/T438647) [08:17:52] (03CR) 10Ozge: [C:03+1] ml-services: point the tts-section-generator to use the tts isvc endpoint in eqiad [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344200 (https://phabricator.wikimedia.org/T438647) (owner: 10Kevin Bazira) [08:18:47] (03CR) 10Kevin Bazira: [C:03+2] ml-services: point the tts-section-generator to use the tts isvc endpoint in eqiad [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344200 (https://phabricator.wikimedia.org/T438647) (owner: 10Kevin Bazira) [08:19:12] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users (Level 3) for SEgt-WMF - https://phabricator.wikimedia.org/T438767#12353699 (10tappof) [08:21:19] (03Merged) 10jenkins-bot: ml-services: point the tts-section-generator to use the tts isvc endpoint in eqiad [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344200 (https://phabricator.wikimedia.org/T438647) (owner: 10Kevin Bazira) [08:21:47] FIRING: RedisInstanceDown: Redis instance down arclamp2001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_arclamp&var-instance=arclamp2001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [08:23:49] 06SRE, 06Infrastructure-Foundations, 07Upstream: Ignore Valid-Until for WMF container images - https://phabricator.wikimedia.org/T438866#12353717 (10Aklapper) [08:24:46] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T438865#12353720 (10phaultfinder) [08:24:50] !log kevinbazira@deploy1003 helmfile [ml-serve-eqiad] 'sync' command on namespace 'tts-section-generator' for release 'main' . [08:25:30] !log kevinbazira@deploy1003 helmfile [ml-serve-codfw] 'sync' command on namespace 'tts-section-generator' for release 'main' . [08:27:22] !log brouberol@cumin1004 START - Cookbook sre.ceph.remove-osd [08:27:23] !log brouberol@cumin1004 END (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) [08:31:46] 06SRE, 06Infrastructure-Foundations: Integrate Bookworm 12.14 point update - https://phabricator.wikimedia.org/T426759#12353747 (10MoritzMuehlenhoff) [08:33:13] 06SRE, 06Infrastructure-Foundations: Integrate Trixie 13.6 point update - https://phabricator.wikimedia.org/T434866#12353758 (10MoritzMuehlenhoff) [08:35:09] 06SRE, 06Infrastructure-Foundations, 10Puppet-Infrastructure, 13Patch-For-Review: Upgrade stdlib to remove legacy facts - https://phabricator.wikimedia.org/T438912#12353764 (10LSobanski) 05Open→03In progress p:05Triage→03Medium [08:36:07] 06SRE, 10Wikimedia-Mailing-lists: Change owner of Wikiversity-l mailing list - https://phabricator.wikimedia.org/T438820#12353768 (10Cormaggio) Thanks - that's what I was hoping to do - I do have access - I wanted to know if there was someone who generally manages inactive lists, or who would be willing to man... [08:36:15] FIRING: MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-web - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?panelId=18&fullscreen&orgId=1&var-datasource=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [08:38:47] (03PS5) 10Daniel Kertesz: cache::haproxy: implement support for persistent stats [puppet] - 10https://gerrit.wikimedia.org/r/1343039 (https://phabricator.wikimedia.org/T343000) [08:41:15] RESOLVED: MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-web - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?panelId=18&fullscreen&orgId=1&var-datasource=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [08:41:18] !log brouberol@cumin1004 DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) [08:41:42] (03CR) 10Daniel Kertesz: "yeah, having to mess with `ExecStop` didn't felt right from the beginning; I'm glad we were able to find a better approach" [puppet] - 10https://gerrit.wikimedia.org/r/1343039 (https://phabricator.wikimedia.org/T343000) (owner: 10Daniel Kertesz) [08:44:03] !log brouberol@cumin1004 DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) [08:44:23] !log brouberol@cumin1004 DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) [08:45:10] !log brouberol@cumin1004 DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) [08:45:21] 06SRE, 10Wikimedia-Mailing-lists: Change owner of Wikiversity-l mailing list - https://phabricator.wikimedia.org/T438820#12353804 (10Aklapper) If there was someone, then you'd more likely find them by asking on the list itself, I don't think anyone will suddenly find this Phab ticket. [08:46:42] (03CR) 10Volans: "Post-merge thought" [cookbooks] - 10https://gerrit.wikimedia.org/r/1344018 (owner: 10Ayounsi) [08:48:13] (03CR) 10Cathal Mooney: [C:03+1] "LGTM!" [cookbooks] - 10https://gerrit.wikimedia.org/r/1343995 (https://phabricator.wikimedia.org/T327300) (owner: 10Ayounsi) [08:48:37] (03CR) 10Dpogorzelski: "so my route block doesn't seem to have any effect and is not needed. I would then abandon this change in favor of https://gerrit.wikimedia" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342612 (https://phabricator.wikimedia.org/T437706) (owner: 10Dpogorzelski) [08:49:32] FIRING: JobQueueLowTrafficConsumerWidespreadHighLatency: ... [08:49:33] Processing delay times for low-traffic consumer rules are unusually high - https://wikitech.wikimedia.org/wiki/MediaWiki_JobQueue/Operations#JobQueueLowTrafficConsumerWidespreadHighLatency - https://grafana.wikimedia.org/d/fe130675-0c2d-4991-9dec-f54cf6a9c4d8/jobqueue-low-traffic-jobs?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DJobQueueLowTrafficConsumerWidespreadHighLatency [08:50:45] (03CR) 10Cathal Mooney: [C:03+1] "Seems well formed so lgtm. I can't say for sure if the actions are right in each case but they seem to be the reverse of depool at least." [puppet] - 10https://gerrit.wikimedia.org/r/1344065 (https://phabricator.wikimedia.org/T327300) (owner: 10Ayounsi) [08:53:50] (03CR) 10Atsuko: [C:03+1] dse-k8s-eqiad: Add a second ipv4 pool to calico [deployment-charts] - 10https://gerrit.wikimedia.org/r/1343998 (https://phabricator.wikimedia.org/T430658) (owner: 10Btullis) [09:03:51] (03PS4) 10Muehlenhoff: profile::dbbackups::transfer: Default db transfers to false [puppet] - 10https://gerrit.wikimedia.org/r/1341121 (https://phabricator.wikimedia.org/T427897) [09:06:06] (03CR) 10Marco Fossati: [C:03+1] ReaderExperiments: Enable on testwiki (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344048 (owner: 10Eric Gardner) [09:06:51] 06SRE, 06Infrastructure-Foundations: Integrate Bookworm 12.15 point update - https://phabricator.wikimedia.org/T434631#12353860 (10MoritzMuehlenhoff) [09:07:30] (03CR) 10Muehlenhoff: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1341121 (https://phabricator.wikimedia.org/T427897) (owner: 10Muehlenhoff) [09:07:51] (03PS2) 10Muehlenhoff: Remove Python 2 exception from Hadoop nodes [puppet] - 10https://gerrit.wikimedia.org/r/1338126 [09:08:06] (03CR) 10Ayounsi: [C:03+2] move-vlan: handle 'systemctl restart networking' better (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/1344018 (owner: 10Ayounsi) [09:08:08] (03CR) 10Marco Fossati: [C:03+1] ReaderExperiments: Set the preferred-sources debug flag on testwiki (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344049 (https://phabricator.wikimedia.org/T436692) (owner: 10Eric Gardner) [09:08:44] (03PS2) 10Muehlenhoff: Uninstall intel-microcode on VMs [puppet] - 10https://gerrit.wikimedia.org/r/1201687 [09:09:17] (03CR) 10Ayounsi: [C:03+2] Add hiera facts for the "pool" action of the rack depool cookbook [puppet] - 10https://gerrit.wikimedia.org/r/1344065 (https://phabricator.wikimedia.org/T327300) (owner: 10Ayounsi) [09:12:46] (03CR) 10Ayounsi: [C:03+2] depool-rack: add policy "manual" for manual pool/depool [cookbooks] - 10https://gerrit.wikimedia.org/r/1343995 (https://phabricator.wikimedia.org/T327300) (owner: 10Ayounsi) [09:16:02] (03Merged) 10jenkins-bot: depool-rack: add policy "manual" for manual pool/depool [cookbooks] - 10https://gerrit.wikimedia.org/r/1343995 (https://phabricator.wikimedia.org/T327300) (owner: 10Ayounsi) [09:23:33] FIRING: KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=eqiad%20prometheus%2Fk8s&var-instance=wikikube-worker1152.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [09:24:02] (03CR) 10Marostegui: profile::dbbackups::transfer: Default db transfers to false (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1341121 (https://phabricator.wikimedia.org/T427897) (owner: 10Muehlenhoff) [09:25:31] (03PS5) 10Muehlenhoff: profile::dbbackups::transfer: Default db transfers to false [puppet] - 10https://gerrit.wikimedia.org/r/1341121 (https://phabricator.wikimedia.org/T427897) [09:25:51] (03PS1) 10Ayounsi: rack depool hiera, add policy: manual and remaining server_pool keys [puppet] - 10https://gerrit.wikimedia.org/r/1344205 (https://phabricator.wikimedia.org/T327300) [09:25:51] (03CR) 10Muehlenhoff: profile::dbbackups::transfer: Default db transfers to false (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1341121 (https://phabricator.wikimedia.org/T427897) (owner: 10Muehlenhoff) [09:26:05] (03CR) 10Muehlenhoff: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1201687 (owner: 10Muehlenhoff) [09:26:52] (03CR) 10Marostegui: [C:03+1] profile::dbbackups::transfer: Default db transfers to false [puppet] - 10https://gerrit.wikimedia.org/r/1341121 (https://phabricator.wikimedia.org/T427897) (owner: 10Muehlenhoff) [09:27:58] (03PS6) 10Daniel Kertesz: cache::haproxy: implement support for persistent stats [puppet] - 10https://gerrit.wikimedia.org/r/1343039 (https://phabricator.wikimedia.org/T343000) [09:29:21] (03CR) 10Daniel Kertesz: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1343039 (https://phabricator.wikimedia.org/T343000) (owner: 10Daniel Kertesz) [09:33:27] (03PS3) 10Muehlenhoff: Uninstall intel-microcode on VMs [puppet] - 10https://gerrit.wikimedia.org/r/1201687 [09:34:19] !log brouberol@cumin1004 DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) [09:34:23] (03CR) 10Muehlenhoff: [C:03+2] profile::dbbackups::transfer: Default db transfers to false [puppet] - 10https://gerrit.wikimedia.org/r/1341121 (https://phabricator.wikimedia.org/T427897) (owner: 10Muehlenhoff) [09:34:50] !incidents [09:34:50] 8357 (RESOLVED) PHPFPMTooBusy sre (mw-web main codfw) [09:34:52] cezmunsta: ^ [09:38:06] (03CR) 10Muehlenhoff: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1201687 (owner: 10Muehlenhoff) [09:38:36] !log brouberol@cumin1004 DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) [09:39:22] (03PS7) 10Daniel Kertesz: cache::haproxy: implement support for persistent stats [puppet] - 10https://gerrit.wikimedia.org/r/1343039 (https://phabricator.wikimedia.org/T343000) [09:39:26] 06SRE, 10Citoid, 10observability, 10Observability-Logging, and 2 others: Citoid is logging all request / response headers as separate fields - https://phabricator.wikimedia.org/T239713#12353962 (10Mvolz) 05Open→03Resolved [09:40:42] (03CR) 10Cathal Mooney: [C:03+1] "Did a basic check on the bigger packages and those jumping major versions. I didn't spot anything that ought to cause us problems in that" [software/netbox-deploy] - 10https://gerrit.wikimedia.org/r/1342301 (owner: 10Ayounsi) [09:40:48] 06SRE, 10Citoid, 10observability, 10Observability-Logging, and 2 others: Citoid is logging all request / response headers as separate fields - https://phabricator.wikimedia.org/T239713#12353969 (10Mvolz) Not sure when this got fixed but doesn't seem to happen anymore. Still not on ECS but we'll get the... [09:40:57] (03CR) 10Daniel Kertesz: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1343039 (https://phabricator.wikimedia.org/T343000) (owner: 10Daniel Kertesz) [09:42:55] (03CR) 10Muehlenhoff: [C:03+2] Uninstall intel-microcode on VMs [puppet] - 10https://gerrit.wikimedia.org/r/1201687 (owner: 10Muehlenhoff) [09:43:54] !log btullis@cumin1004 START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{dse-k8s-wdqs100[1-3].eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) [09:43:58] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1001.eqiad.wmnet [09:44:32] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1001.eqiad.wmnet [09:45:29] (03PS1) 10Blake: mw-web: upsize for single-DC serving. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344212 (https://phabricator.wikimedia.org/T438896) [09:50:14] !log brouberol@cumin1004 DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) [09:51:11] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1001.eqiad.wmnet [09:51:12] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1001.eqiad.wmnet [09:51:18] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1002.eqiad.wmnet [09:51:52] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1002.eqiad.wmnet [09:51:55] !log brouberol@cumin1004 DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) [09:52:45] !log brouberol@cumin1004 DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) [09:53:51] !log brouberol@cumin1004 DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) [09:54:05] !log brouberol@cumin1004 DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) [09:54:45] !log brouberol@cumin1004 DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) [09:55:27] !log brouberol@cumin1004 DONE (PASS) - Cookbook sre.ceph.remove-osd (exit_code=0) [09:57:00] (03CR) 10Ayounsi: [V:03+2 C:03+2] Add netbox-bgp and update wheels [software/netbox-deploy] - 10https://gerrit.wikimedia.org/r/1342301 (owner: 10Ayounsi) [09:57:27] !log brouberol@cumin1004 DONE (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) [09:57:43] !incidents [09:57:43] 8357 (RESOLVED) PHPFPMTooBusy sre (mw-web main codfw) [09:58:00] (03PS8) 10Daniel Kertesz: cache::haproxy: implement support for persistent stats [puppet] - 10https://gerrit.wikimedia.org/r/1343039 (https://phabricator.wikimedia.org/T343000) [09:58:26] (03PS17) 10Brouberol: ceph: add a cookbook to remove an OSD [cookbooks] - 10https://gerrit.wikimedia.org/r/1344201 (https://phabricator.wikimedia.org/T438823) [09:58:47] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1002.eqiad.wmnet [09:58:48] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1002.eqiad.wmnet [09:58:54] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1003.eqiad.wmnet [09:59:00] !log ayounsi@cumin1004 START - Cookbook sre.deploy.python-code netbox to netbox-dev2003.codfw.wmnet with reason: Add netbox-bgp and update wheelson netbox-next - ayounsi@cumin1004 [09:59:29] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1003.eqiad.wmnet [09:59:30] (03PS18) 10Brouberol: ceph: add a cookbook to remove an OSD [cookbooks] - 10https://gerrit.wikimedia.org/r/1344201 (https://phabricator.wikimedia.org/T438823) [10:00:05] Deploy window MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260923T1000) [10:01:40] (03PS19) 10Brouberol: ceph: add a cookbook to remove an OSD [cookbooks] - 10https://gerrit.wikimedia.org/r/1344201 (https://phabricator.wikimedia.org/T438823) [10:02:54] (03PS1) 10Ameisenigel: Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344216 [10:04:01] (03PS2) 10Ameisenigel: Enable Extension:Translate and Extension:TranslationNotifications on co.wikimedia.org [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344216 (https://phabricator.wikimedia.org/T386776) [10:04:28] !log ayounsi@cumin1004 END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2003.codfw.wmnet with reason: Add netbox-bgp and update wheelson netbox-next - ayounsi@cumin1004 [10:06:24] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1003.eqiad.wmnet [10:06:26] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1003.eqiad.wmnet [10:06:26] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{dse-k8s-wdqs100[1-3].eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) [10:06:41] FIRING: [3x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:08:11] (03PS3) 10Awight: Configure beta cluster to use the SiteMatrixLookup [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344139 (https://phabricator.wikimedia.org/T432369) [10:12:08] (03CR) 10JMeybohm: [C:03+1] mw-web: upsize for single-DC serving. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344212 (https://phabricator.wikimedia.org/T438896) (owner: 10Blake) [10:18:12] (03CR) 10Blake: [C:03+2] mw-web: upsize for single-DC serving. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344212 (https://phabricator.wikimedia.org/T438896) (owner: 10Blake) [10:20:47] (03Merged) 10jenkins-bot: mw-web: upsize for single-DC serving. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344212 (https://phabricator.wikimedia.org/T438896) (owner: 10Blake) [10:22:07] !log blake@deploy1003 Started scap sync-world: Upsize mw-web T438896 [10:22:12] T438896: Investigate high saturation in codfw during switchover resulting from increased thumb.wm.o cache pressure - https://phabricator.wikimedia.org/T438896 [10:22:45] (03CR) 10JMeybohm: [C:03+1] service::catalog: Set ipip_encapsulation for k8s-ingress-wikikube in codfw [puppet] - 10https://gerrit.wikimedia.org/r/1311444 (https://phabricator.wikimedia.org/T420436) (owner: 10Jelto) [10:22:49] (03CR) 10JMeybohm: [C:03+1] service::catalog: Set ipip_encapsulation for k8s-ingress-wikikube in eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1344197 (https://phabricator.wikimedia.org/T420436) (owner: 10Jelto) [10:24:56] (03CR) 10Mvolz: [C:03+2] citoid: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1343010 (owner: 10PipelineBot) [10:25:40] !log blake@deploy1003 Finished scap sync-world: Upsize mw-web T438896 (duration: 04m 20s) [10:27:19] (03Merged) 10jenkins-bot: citoid: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1343010 (owner: 10PipelineBot) [10:31:28] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/pageview-trending-relative: apply [10:31:38] !log javiermonton@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/pageview-trending-relative: apply [10:36:30] (03PS14) 10Arnaudb: mesh: add opt-in websocket support in configuration 1.16.1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338751 (https://phabricator.wikimedia.org/T436657) [10:36:30] (03CR) 10Arnaudb: "this change is now `-0/+41`. I tested as much as I could on both relation chains. The tests have been done on a local kind cluster. This c" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338751 (https://phabricator.wikimedia.org/T436657) (owner: 10Arnaudb) [10:36:38] (03PS8) 10Arnaudb: mesh: Copy mesh.configuration 1.16.0 to 1.16.1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1338959 (https://phabricator.wikimedia.org/T436657) [10:38:17] !log sudo confctl --quiet --object-type discovery select 'dnsdisc=mw-web-ro' set/ttl=10 - T438896 [10:38:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [10:38:21] T438896: Investigate high saturation in codfw during switchover resulting from increased thumb.wm.o cache pressure - https://phabricator.wikimedia.org/T438896 [10:40:58] (03PS1) 10Daniel Kertesz: hiera: enable HAProxy's persistent stats on two magru nodes [puppet] - 10https://gerrit.wikimedia.org/r/1344222 (https://phabricator.wikimedia.org/T343000) [10:45:03] (03CR) 10Daniel Kertesz: [C:04-1] "-1: CR depends on https://gerrit.wikimedia.org/r/c/operations/puppet/+/1343039" [puppet] - 10https://gerrit.wikimedia.org/r/1344222 (https://phabricator.wikimedia.org/T343000) (owner: 10Daniel Kertesz) [10:46:40] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:50:18] (03CR) 10Vgutierrez: [C:03+1] "nice job!" [puppet] - 10https://gerrit.wikimedia.org/r/1343039 (https://phabricator.wikimedia.org/T343000) (owner: 10Daniel Kertesz) [10:55:18] 06SRE, 06Data-Persistence, 10Wikifeeds, 06Content-Platform-Team (Work In Progress), 07Essential-Work: Provision Cassandra keyspace for WikiFeeds cache - https://phabricator.wikimedia.org/T435450#12354199 (10OSleger-WMF) a:05OSleger-WMF→03Eevans [10:55:46] (03PS1) 10Hnowlan: data-platform: remove icinga proc checks in favour of systemd checks [puppet] - 10https://gerrit.wikimedia.org/r/1344224 (https://phabricator.wikimedia.org/T357099) [10:55:48] (03PS1) 10Hnowlan: bacula: remove icinga check_procs reviews in favour of SystemdUnitFailed [puppet] - 10https://gerrit.wikimedia.org/r/1344225 (https://phabricator.wikimedia.org/T357099) [10:55:50] (03PS1) 10Hnowlan: collab-services: remove check_procs alerts in favour of SystemdUnitFailed [puppet] - 10https://gerrit.wikimedia.org/r/1344226 (https://phabricator.wikimedia.org/T357099) [10:55:53] (03PS1) 10Hnowlan: traffic: replace check_procs based alerts in favour of SystemdUnitFailed [puppet] - 10https://gerrit.wikimedia.org/r/1344227 (https://phabricator.wikimedia.org/T357099) [10:55:56] (03PS1) 10Hnowlan: query_service: remove icinga check_procs in favour of systemd check [puppet] - 10https://gerrit.wikimedia.org/r/1344228 (https://phabricator.wikimedia.org/T357099) [10:55:58] (03PS1) 10Hnowlan: jenkins: remove check_procs icinga check in favour of systemd alert [puppet] - 10https://gerrit.wikimedia.org/r/1344229 (https://phabricator.wikimedia.org/T357099) [10:56:01] (03PS1) 10Hnowlan: data persistence: remove check_procs icinga checks, use systemd alert [puppet] - 10https://gerrit.wikimedia.org/r/1344230 (https://phabricator.wikimedia.org/T357099) [10:56:03] (03PS1) 10Hnowlan: haproxy: remove check_procs check in favour of native systemd alerting [puppet] - 10https://gerrit.wikimedia.org/r/1344231 (https://phabricator.wikimedia.org/T357099) [10:56:05] (03PS1) 10Hnowlan: acme_chief: remove check_procs alerts in favour of SystemdUnitFailed [puppet] - 10https://gerrit.wikimedia.org/r/1344232 (https://phabricator.wikimedia.org/T357099) [10:56:08] (03PS1) 10Hnowlan: varnishkafka: remove check_proces check in favour of systemd alert [puppet] - 10https://gerrit.wikimedia.org/r/1344233 (https://phabricator.wikimedia.org/T357099) [10:56:12] (03PS2) 10Hnowlan: bacula: remove icinga check_procs alerts in favour of SystemdUnitFailed [puppet] - 10https://gerrit.wikimedia.org/r/1344225 (https://phabricator.wikimedia.org/T357099) [10:59:26] (03PS2) 10Majavah: P:wmcs::etcd: Rename nginx upstream [puppet] - 10https://gerrit.wikimedia.org/r/1343961 (https://phabricator.wikimedia.org/T410721) [10:59:26] (03PS2) 10Majavah: P:etcd::tlsproxy: Move hardcoded ACL exceptions to Hiera [puppet] - 10https://gerrit.wikimedia.org/r/1343962 (https://phabricator.wikimedia.org/T410721) [10:59:26] (03PS4) 10Majavah: P:etcd::tlsproxy: Add support for acme-chief certificates [puppet] - 10https://gerrit.wikimedia.org/r/1343963 (https://phabricator.wikimedia.org/T410721) [10:59:27] (03PS4) 10Majavah: P:etcd::tlsproxy: Support client cert auth to upstream [puppet] - 10https://gerrit.wikimedia.org/r/1343964 (https://phabricator.wikimedia.org/T410721) [11:00:05] mvolz: I seem to be stuck in Groundhog week. Sigh. Time for (yet another) Services – Citoid / Zotero deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260923T1100). [11:09:29] (03CR) 10Majavah: [C:03+2] P:wmcs::etcd: Rename nginx upstream [puppet] - 10https://gerrit.wikimedia.org/r/1343961 (https://phabricator.wikimedia.org/T410721) (owner: 10Majavah) [11:12:47] (03PS1) 10Muehlenhoff: Assign server IDs for new LDAP replicas [puppet] - 10https://gerrit.wikimedia.org/r/1344236 (https://phabricator.wikimedia.org/T331699) [11:13:36] (03PS1) 10Majavah: O:wmcs::cloudinfra_etcd: Add P:etcd::tlsproxy [puppet] - 10https://gerrit.wikimedia.org/r/1344237 (https://phabricator.wikimedia.org/T410721) [11:21:46] (03PS1) 10Majavah: cloudinfra: Add fake etcd secrets [labs/private] - 10https://gerrit.wikimedia.org/r/1344239 [11:22:00] (03CR) 10Majavah: [V:03+2 C:03+2] cloudinfra: Add fake etcd secrets [labs/private] - 10https://gerrit.wikimedia.org/r/1344239 (owner: 10Majavah) [11:23:27] (03PS1) 10Dpogorzelski: New upstream version 1.31.0.10 [debs/amd-k8s-device-plugin] - 10https://gerrit.wikimedia.org/r/1344240 [11:23:27] (03PS1) 10Dpogorzelski: Update upstream source from tag 'upstream/1.31.0.10' [debs/amd-k8s-device-plugin] - 10https://gerrit.wikimedia.org/r/1344241 [11:23:28] (03PS1) 10Dpogorzelski: Release upstream version 1.31.0.10 [debs/amd-k8s-device-plugin] - 10https://gerrit.wikimedia.org/r/1344242 (https://phabricator.wikimedia.org/T436928) [11:23:42] (03CR) 10AOkoth: [C:03+2] devtools: setup new deploy host [puppet] - 10https://gerrit.wikimedia.org/r/1344075 (https://phabricator.wikimedia.org/T429494) (owner: 10AOkoth) [11:25:21] !log mvolz@deploy1003 helmfile [staging] START helmfile.d/services/citoid: apply [11:25:41] !log mvolz@deploy1003 helmfile [staging] DONE helmfile.d/services/citoid: apply [11:25:57] jouncebot: nowandnext [11:25:57] For the next 0 hour(s) and 34 minute(s): Services – Citoid / Zotero (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260923T1100) [11:25:57] In 6 hour(s) and 34 minute(s): MediaWiki train - Utc-7 Version (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260923T1800) [11:27:00] Going to use scap to deploy private code [11:27:07] !log mvolz@deploy1003 helmfile [codfw] START helmfile.d/services/citoid: apply [11:27:28] FIRING: JobUnavailable: Reduced availability for job redis_arclamp in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [11:27:54] (03PS3) 10Majavah: P:etcd::tlsproxy: Move hardcoded ACL exceptions to Hiera [puppet] - 10https://gerrit.wikimedia.org/r/1343962 (https://phabricator.wikimedia.org/T410721) [11:27:54] (03PS5) 10Majavah: P:etcd::tlsproxy: Add support for acme-chief certificates [puppet] - 10https://gerrit.wikimedia.org/r/1343963 (https://phabricator.wikimedia.org/T410721) [11:27:55] (03PS5) 10Majavah: P:etcd::tlsproxy: Support client cert auth to upstream [puppet] - 10https://gerrit.wikimedia.org/r/1343964 (https://phabricator.wikimedia.org/T410721) [11:27:57] !log mvolz@deploy1003 helmfile [codfw] DONE helmfile.d/services/citoid: apply [11:27:59] (03PS2) 10Majavah: O:wmcs::cloudinfra_etcd: Add P:etcd::tlsproxy [puppet] - 10https://gerrit.wikimedia.org/r/1344237 (https://phabricator.wikimedia.org/T410721) [11:28:03] (03PS1) 10Majavah: P:etcd::tlsproxy: Fix compatibility with Trixie nginx [puppet] - 10https://gerrit.wikimedia.org/r/1344243 (https://phabricator.wikimedia.org/T410721) [11:28:33] !log mvolz@deploy1003 helmfile [eqiad] START helmfile.d/services/citoid: apply [11:31:10] !log mvolz@deploy1003 helmfile [eqiad] DONE helmfile.d/services/citoid: apply [11:34:52] (03PS1) 10Majavah: P:wmcs::etcd: Allow customizing advertised port [puppet] - 10https://gerrit.wikimedia.org/r/1344245 (https://phabricator.wikimedia.org/T410721) [11:36:11] (03CR) 10Volans: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1344243 (https://phabricator.wikimedia.org/T410721) (owner: 10Majavah) [11:36:51] (03CR) 10Volans: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1344237 (https://phabricator.wikimedia.org/T410721) (owner: 10Majavah) [11:37:38] Running scap now [11:37:54] (03CR) 10Volans: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1344245 (https://phabricator.wikimedia.org/T410721) (owner: 10Majavah) [11:37:58] (03CR) 10Majavah: [V:03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9474/co" [puppet] - 10https://gerrit.wikimedia.org/r/1344243 (https://phabricator.wikimedia.org/T410721) (owner: 10Majavah) [11:39:05] (03PS2) 10Dpogorzelski: New upstream version 1.31.0.10 [debs/amd-k8s-device-plugin] - 10https://gerrit.wikimedia.org/r/1344240 [11:39:50] When running scap I got a warning `from scap import cli ImportError: cannot import name 'cli' from 'scap' (unknown location)` [11:40:00] Appears to have happened in other scap runs yesterday [11:40:07] (03CR) 10Hnowlan: [C:03+2] team-sre: add infrastructure-foundations tag [alerts] - 10https://gerrit.wikimedia.org/r/1319832 (https://phabricator.wikimedia.org/T432376) (owner: 10Hnowlan) [11:41:55] (03CR) 10Majavah: [C:03+2] P:etcd::tlsproxy: Move hardcoded ACL exceptions to Hiera [puppet] - 10https://gerrit.wikimedia.org/r/1343962 (https://phabricator.wikimedia.org/T410721) (owner: 10Majavah) [11:42:04] Dreamy_Jazz: a new version went out yesterday, might be related [11:42:09] (03CR) 10Majavah: [C:03+2] P:etcd::tlsproxy: Add support for acme-chief certificates [puppet] - 10https://gerrit.wikimedia.org/r/1343963 (https://phabricator.wikimedia.org/T410721) (owner: 10Majavah) [11:42:23] (03CR) 10Majavah: [C:03+2] P:etcd::tlsproxy: Support client cert auth to upstream [puppet] - 10https://gerrit.wikimedia.org/r/1343964 (https://phabricator.wikimedia.org/T410721) (owner: 10Majavah) [11:42:40] (03CR) 10Majavah: [V:03+1 C:03+2] P:etcd::tlsproxy: Fix compatibility with Trixie nginx [puppet] - 10https://gerrit.wikimedia.org/r/1344243 (https://phabricator.wikimedia.org/T410721) (owner: 10Majavah) [11:43:09] (03Merged) 10jenkins-bot: team-sre: add infrastructure-foundations tag [alerts] - 10https://gerrit.wikimedia.org/r/1319832 (https://phabricator.wikimedia.org/T432376) (owner: 10Hnowlan) [11:43:18] (03CR) 10Hnowlan: "I think we'll need to change the data-persistence receiver to message #wikimedia-operations before merging this." [alerts] - 10https://gerrit.wikimedia.org/r/1319830 (https://phabricator.wikimedia.org/T432376) (owner: 10Hnowlan) [11:44:43] 10ops-eqiad, 06DC-Ops: hw troubleshooting: disk errors for cephosd1002.eqiad.wmnet /c0/ - https://phabricator.wikimedia.org/T438952 (10BTullis) 03NEW [11:45:15] 10ops-eqiad, 06DC-Ops: hw troubleshooting: disk errors for cephosd1002.eqiad.wmnet /c0/ - https://phabricator.wikimedia.org/T438952#12354560 (10BTullis) [11:47:26] (03CR) 10Majavah: [C:03+2] O:wmcs::cloudinfra_etcd: Add P:etcd::tlsproxy [puppet] - 10https://gerrit.wikimedia.org/r/1344237 (https://phabricator.wikimedia.org/T410721) (owner: 10Majavah) [11:48:08] 10ops-eqiad, 06DC-Ops: hw troubleshooting: disk errors for cephosd1002.eqiad.wmnet /c0/e23/s4 - https://phabricator.wikimedia.org/T438952#12354566 (10BTullis) [11:49:46] 10ops-eqiad, 06DC-Ops: hw troubleshooting: disk errors for cephosd1002.eqiad.wmnet /c0/e23/s4 - https://phabricator.wikimedia.org/T438952#12354567 (10BTullis) I think I enabled blinking for the failed drive. ` btullis@cephosd1002:~$ sudo perccli64 /c0/e23/s4 start locate CLI Version = 007.1910.0000.0000 Oct 08... [11:51:16] !log Deployed patch for T438729 [11:51:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:51:21] T438729: Update private code to enable the vandalism classifier - https://phabricator.wikimedia.org/T438729 [11:51:30] (03PS1) 10Awight: Fixes failing edge when page is missing and entity usage remain. Updating ReallyDoQuery to function like an inner join. [extensions/Wikibase] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344246 (https://phabricator.wikimedia.org/T437687) [11:51:51] (03CR) 10Awight: [C:03+1] Fixes failing edge when page is missing and entity usage remain. Updating ReallyDoQuery to function like an inner join. [extensions/Wikibase] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344246 (https://phabricator.wikimedia.org/T437687) (owner: 10Awight) [11:57:10] jouncebot: nowandnext [11:57:10] For the next 0 hour(s) and 2 minute(s): Services – Citoid / Zotero (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260923T1100) [11:57:10] In 6 hour(s) and 2 minute(s): MediaWiki train - Utc-7 Version (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260923T1800) [11:57:30] (03CR) 10TrainBranchBot: [C:03+2] "Approved by kharlan@deploy1003 using scap backport" [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1343982 (https://phabricator.wikimedia.org/T438467) (owner: 10Kosta Harlan) [11:59:42] (03Merged) 10jenkins-bot: AbuseReview: Add warning indicating alpha test to vandalism queue [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1343982 (https://phabricator.wikimedia.org/T438467) (owner: 10Kosta Harlan) [12:00:49] (03CR) 10Btullis: "Could we add a `--cluster` option or similar and limit it to the DPE clusters for now, please? just having a `--ceph-host` and allowing so" [cookbooks] - 10https://gerrit.wikimedia.org/r/1344201 (https://phabricator.wikimedia.org/T438823) (owner: 10Brouberol) [12:01:00] !log kharlan@deploy1003 Started scap sync-world: Backport for [[gerrit:1343982|AbuseReview: Add warning indicating alpha test to vandalism queue (T438467)]] [12:01:05] T438467: AbuseReview: Enable an alpha testing mode for vandalism detection - https://phabricator.wikimedia.org/T438467 [12:01:35] (03CR) 10Majavah: [C:03+2] P:wmcs::etcd: Allow customizing advertised port [puppet] - 10https://gerrit.wikimedia.org/r/1344245 (https://phabricator.wikimedia.org/T410721) (owner: 10Majavah) [12:02:31] (03CR) 10Mamouri: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1325997 (https://phabricator.wikimedia.org/T432544) (owner: 10Mamouri) [12:04:39] (03PS1) 10Filippo Giunchedi: team-wmcs: adjust new cluster names [alerts] - 10https://gerrit.wikimedia.org/r/1344249 (https://phabricator.wikimedia.org/T437272) [12:06:36] (03CR) 10Majavah: [C:03+1] team-wmcs: adjust new cluster names [alerts] - 10https://gerrit.wikimedia.org/r/1344249 (https://phabricator.wikimedia.org/T437272) (owner: 10Filippo Giunchedi) [12:06:43] (03CR) 10Klausman: ceph: add a cookbook to remove an OSD (032 comments) [cookbooks] - 10https://gerrit.wikimedia.org/r/1344201 (https://phabricator.wikimedia.org/T438823) (owner: 10Brouberol) [12:06:52] (03CR) 10Majavah: [C:03+1] hieradata: remove 'wmcs' cluster [puppet] - 10https://gerrit.wikimedia.org/r/1344134 (https://phabricator.wikimedia.org/T437272) (owner: 10Filippo Giunchedi) [12:08:26] (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, September 24 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deplo" [extensions/Wikibase] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344246 (https://phabricator.wikimedia.org/T437687) (owner: 10Awight) [12:08:28] (03CR) 10Filippo Giunchedi: [C:03+2] team-wmcs: adjust new cluster names [alerts] - 10https://gerrit.wikimedia.org/r/1344249 (https://phabricator.wikimedia.org/T437272) (owner: 10Filippo Giunchedi) [12:08:40] (03CR) 10Filippo Giunchedi: [C:03+2] hieradata: remove 'wmcs' cluster [puppet] - 10https://gerrit.wikimedia.org/r/1344134 (https://phabricator.wikimedia.org/T437272) (owner: 10Filippo Giunchedi) [12:08:50] (03CR) 10Brouberol: "Yep, fair!" [cookbooks] - 10https://gerrit.wikimedia.org/r/1344201 (https://phabricator.wikimedia.org/T438823) (owner: 10Brouberol) [12:09:43] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv4 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95133212 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [12:15:11] !log cdanis@cumin1004 START - Cookbook sre.dns.admin DNS admin: pool eqiad [reason: no reason specified, no task ID specified] [12:15:16] !log cdanis@cumin1004 END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool eqiad [reason: no reason specified, no task ID specified] [12:21:47] !log kharlan@deploy1003 kharlan: Backport for [[gerrit:1343982|AbuseReview: Add warning indicating alpha test to vandalism queue (T438467)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [12:21:47] FIRING: RedisInstanceDown: Redis instance down arclamp2001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_arclamp&var-instance=arclamp2001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [12:21:50] T438467: AbuseReview: Enable an alpha testing mode for vandalism detection - https://phabricator.wikimedia.org/T438467 [12:22:47] !log kharlan@deploy1003 kharlan: Continuing with deployment [12:23:04] (03CR) 10Lerickson: "Thanks! I already sent https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1343643 for that one so I think ideally I'd just su" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344104 (https://phabricator.wikimedia.org/T438789) (owner: 10Lerickson) [12:23:24] (03PS1) 10Majavah: P:wmcs::etcd::backup: Cleanup older backups [puppet] - 10https://gerrit.wikimedia.org/r/1344255 (https://phabricator.wikimedia.org/T438731) [12:27:40] (03CR) 10Cathal Mooney: [C:03+1] "Based on the description and current assignments seems sensible" [puppet] - 10https://gerrit.wikimedia.org/r/1344236 (https://phabricator.wikimedia.org/T331699) (owner: 10Muehlenhoff) [12:28:57] (03CR) 10Muehlenhoff: [C:03+2] Assign server IDs for new LDAP replicas [puppet] - 10https://gerrit.wikimedia.org/r/1344236 (https://phabricator.wikimedia.org/T331699) (owner: 10Muehlenhoff) [12:29:10] (03PS1) 10Muehlenhoff: Only install debuerreotype on the server if base images are built [puppet] - 10https://gerrit.wikimedia.org/r/1344257 (https://phabricator.wikimedia.org/T438866) [12:29:33] !log cwilliams@cumin1004 START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section test-s4 [12:29:43] RESOLVED: [4x] RipeAtlasAnchorUnreachable: ipv4 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95133212 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [12:30:31] (03CR) 10Trueg: [C:03+1] "Done" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1343643 (https://phabricator.wikimedia.org/T438789) (owner: 10Lerickson) [12:30:46] !log brouberol@cumin1004 START - Cookbook sre.ceph.remove-osd [12:30:53] (03CR) 10Muehlenhoff: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1344257 (https://phabricator.wikimedia.org/T438866) (owner: 10Muehlenhoff) [12:30:54] !log brouberol@cumin1004 END (PASS) - Cookbook sre.ceph.remove-osd (exit_code=0) [12:31:17] (03PS20) 10Brouberol: ceph: add a cookbook to remove an OSD [cookbooks] - 10https://gerrit.wikimedia.org/r/1344201 (https://phabricator.wikimedia.org/T438823) [12:31:29] (03CR) 10Trueg: [C:03+1] "Thanks, I forgot about that one. How about adding a "Depends-On" to reference the other CR?" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344104 (https://phabricator.wikimedia.org/T438789) (owner: 10Lerickson) [12:31:41] (03CR) 10Klausman: [C:03+1] New upstream version 1.31.0.10 [debs/amd-k8s-device-plugin] - 10https://gerrit.wikimedia.org/r/1344240 (owner: 10Dpogorzelski) [12:31:57] (03PS21) 10Brouberol: ceph: add a cookbook to remove an OSD [cookbooks] - 10https://gerrit.wikimedia.org/r/1344201 (https://phabricator.wikimedia.org/T438823) [12:32:01] !log brouberol@cumin1004 START - Cookbook sre.ceph.remove-osd [12:32:02] !log brouberol@cumin1004 END (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) [12:32:32] (03PS22) 10Brouberol: ceph: add a cookbook to remove an OSD [cookbooks] - 10https://gerrit.wikimedia.org/r/1344201 (https://phabricator.wikimedia.org/T438823) [12:32:37] !log brouberol@cumin1004 START - Cookbook sre.ceph.remove-osd [12:32:41] !log brouberol@cumin1004 END (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) [12:32:44] (03PS1) 10Muehlenhoff: Assign openldap::replica_mdb role to ldap-replica200[78] [puppet] - 10https://gerrit.wikimedia.org/r/1344259 (https://phabricator.wikimedia.org/T331699) [12:33:23] cwilliams@cumin1004 prepare (PID 391856) is awaiting input [12:33:31] (03PS23) 10Brouberol: ceph: add a cookbook to remove an OSD [cookbooks] - 10https://gerrit.wikimedia.org/r/1344201 (https://phabricator.wikimedia.org/T438823) [12:33:36] !log brouberol@cumin1004 START - Cookbook sre.ceph.remove-osd [12:33:49] !log brouberol@cumin1004 END (PASS) - Cookbook sre.ceph.remove-osd (exit_code=0) [12:34:34] !log kharlan@deploy1003 Finished scap sync-world: Backport for [[gerrit:1343982|AbuseReview: Add warning indicating alpha test to vandalism queue (T438467)]] (duration: 33m 33s) [12:34:37] T438467: AbuseReview: Enable an alpha testing mode for vandalism detection - https://phabricator.wikimedia.org/T438467 [12:36:03] (03CR) 10CI reject: [V:04-1] ceph: add a cookbook to remove an OSD [cookbooks] - 10https://gerrit.wikimedia.org/r/1344201 (https://phabricator.wikimedia.org/T438823) (owner: 10Brouberol) [12:36:22] (03CR) 10Brouberol: ceph: add a cookbook to remove an OSD (032 comments) [cookbooks] - 10https://gerrit.wikimedia.org/r/1344201 (https://phabricator.wikimedia.org/T438823) (owner: 10Brouberol) [12:37:14] (03PS24) 10Brouberol: ceph: add a cookbook to remove an OSD [cookbooks] - 10https://gerrit.wikimedia.org/r/1344201 (https://phabricator.wikimedia.org/T438823) [12:37:31] (03PS25) 10Brouberol: ceph: add a cookbook to remove an OSD [cookbooks] - 10https://gerrit.wikimedia.org/r/1344201 (https://phabricator.wikimedia.org/T438823) [12:38:55] (03PS26) 10Brouberol: ceph: add a cookbook to remove an OSD [cookbooks] - 10https://gerrit.wikimedia.org/r/1344201 (https://phabricator.wikimedia.org/T438823) [12:38:58] !log brouberol@cumin1004 START - Cookbook sre.ceph.remove-osd [12:39:06] !log brouberol@cumin1004 END (PASS) - Cookbook sre.ceph.remove-osd (exit_code=0) [12:39:10] FIRING: BFDdown: BFD session down between cr2-eqsin and fe80::6687:8807:ef2:7018 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqsin:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [12:39:47] !log cwilliams@cumin1004 END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section test-s4 [12:40:46] (03CR) 10Gmodena: [C:03+1] Remove a default value that never took effect anyway. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1343643 (https://phabricator.wikimedia.org/T438789) (owner: 10Lerickson) [12:41:43] (03CR) 10CI reject: [V:04-1] ceph: add a cookbook to remove an OSD [cookbooks] - 10https://gerrit.wikimedia.org/r/1344201 (https://phabricator.wikimedia.org/T438823) (owner: 10Brouberol) [12:42:34] (03PS27) 10Brouberol: ceph: add a cookbook to remove an OSD [cookbooks] - 10https://gerrit.wikimedia.org/r/1344201 (https://phabricator.wikimedia.org/T438823) [12:42:51] (03CR) 10Gmodena: Enable wdqs-proxy metrics on port 9100. (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344104 (https://phabricator.wikimedia.org/T438789) (owner: 10Lerickson) [12:43:10] (03CR) 10Dpogorzelski: [C:03+2] New upstream version 1.31.0.10 [debs/amd-k8s-device-plugin] - 10https://gerrit.wikimedia.org/r/1344240 (owner: 10Dpogorzelski) [12:43:18] (03CR) 10Dpogorzelski: [V:03+2 C:03+2] New upstream version 1.31.0.10 [debs/amd-k8s-device-plugin] - 10https://gerrit.wikimedia.org/r/1344240 (owner: 10Dpogorzelski) [12:43:27] FIRING: ProbeDown: Service thanos-query:443 has failed probes (http_thanos-query_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#thanos-query:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [12:44:02] (03CR) 10Dpogorzelski: [C:03+2] Update upstream source from tag 'upstream/1.31.0.10' [debs/amd-k8s-device-plugin] - 10https://gerrit.wikimedia.org/r/1344241 (owner: 10Dpogorzelski) [12:44:06] (03CR) 10Dpogorzelski: [V:03+2 C:03+2] Update upstream source from tag 'upstream/1.31.0.10' [debs/amd-k8s-device-plugin] - 10https://gerrit.wikimedia.org/r/1344241 (owner: 10Dpogorzelski) [12:44:10] RESOLVED: BFDdown: BFD session down between cr2-eqsin and fe80::6687:8807:ef2:7018 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqsin:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [12:44:29] (03PS1) 10Brouberol: ceph::osd: don't use puppet to delete an absented OSD [puppet] - 10https://gerrit.wikimedia.org/r/1344261 (https://phabricator.wikimedia.org/T438823) [12:44:57] !log cmooney@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr1-eqiad,ssw1-d[1,8]-eqiad with reason: re-rack ssw1-a1-eqiad [12:45:08] 10ops-eqiad, 06SRE, 06DC-Ops, 06Infrastructure-Foundations, 10netops: eqiad row A/B switch refresh prep - https://phabricator.wikimedia.org/T418012#12354779 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=7687fe60-8be3-4891-bb62-41d39b74f427) set by cmooney@cumin1004 for 2:00:00 on 3... [12:45:14] (03Abandoned) 10Dpogorzelski: Update upstream source from tag 'upstream/1.31.0.10' [debs/amd-k8s-device-plugin] - 10https://gerrit.wikimedia.org/r/1344241 (owner: 10Dpogorzelski) [12:45:22] !log add seanleong-wmde to deployment-prep [12:45:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:45:28] (03CR) 10Lerickson: "It doesn't really depend on it, though, since that other one is just cleaning up something that was a no-op!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344104 (https://phabricator.wikimedia.org/T438789) (owner: 10Lerickson) [12:45:30] (03CR) 10Dpogorzelski: [C:03+2] Release upstream version 1.31.0.10 [debs/amd-k8s-device-plugin] - 10https://gerrit.wikimedia.org/r/1344242 (https://phabricator.wikimedia.org/T436928) (owner: 10Dpogorzelski) [12:45:33] (03CR) 10Dpogorzelski: [V:03+2 C:03+2] Release upstream version 1.31.0.10 [debs/amd-k8s-device-plugin] - 10https://gerrit.wikimedia.org/r/1344242 (https://phabricator.wikimedia.org/T436928) (owner: 10Dpogorzelski) [12:45:38] (03CR) 10CI reject: [V:04-1] ceph::osd: don't use puppet to delete an absented OSD [puppet] - 10https://gerrit.wikimedia.org/r/1344261 (https://phabricator.wikimedia.org/T438823) (owner: 10Brouberol) [12:47:13] (03PS2) 10Brouberol: ceph::osd: don't use puppet to delete an absented OSD [puppet] - 10https://gerrit.wikimedia.org/r/1344261 (https://phabricator.wikimedia.org/T438823) [12:48:01] (03CR) 10Muehlenhoff: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1344259 (https://phabricator.wikimedia.org/T331699) (owner: 10Muehlenhoff) [12:48:18] (03CR) 10Gmodena: Enable wdqs-proxy metrics on port 9100. (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344104 (https://phabricator.wikimedia.org/T438789) (owner: 10Lerickson) [12:48:22] (03CR) 10CI reject: [V:04-1] ceph::osd: don't use puppet to delete an absented OSD [puppet] - 10https://gerrit.wikimedia.org/r/1344261 (https://phabricator.wikimedia.org/T438823) (owner: 10Brouberol) [12:48:27] RESOLVED: ProbeDown: Service thanos-query:443 has failed probes (http_thanos-query_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#thanos-query:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [12:49:10] (03PS3) 10Brouberol: ceph::osd: don't use puppet to delete an absented OSD [puppet] - 10https://gerrit.wikimedia.org/r/1344261 (https://phabricator.wikimedia.org/T438823) [12:49:33] FIRING: JobQueueLowTrafficConsumerWidespreadHighLatency: ... [12:49:33] Processing delay times for low-traffic consumer rules are unusually high - https://wikitech.wikimedia.org/wiki/MediaWiki_JobQueue/Operations#JobQueueLowTrafficConsumerWidespreadHighLatency - https://grafana.wikimedia.org/d/fe130675-0c2d-4991-9dec-f54cf6a9c4d8/jobqueue-low-traffic-jobs?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DJobQueueLowTrafficConsumerWidespreadHighLatency [12:50:45] (03PS6) 10Arnaudb: aux-k8s: add the aphlict namespace and its certificate [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341855 (https://phabricator.wikimedia.org/T436657) [12:50:47] (03PS2) 10Dpogorzelski: Release upstream version 1.31.0.10 [debs/amd-k8s-device-plugin] - 10https://gerrit.wikimedia.org/r/1344242 (https://phabricator.wikimedia.org/T436928) [12:51:01] (03PS7) 10Arnaudb: aphlict: deploy to both aux-k8s clusters [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341857 (https://phabricator.wikimedia.org/T436657) [12:51:04] (03CR) 10Dpogorzelski: [C:03+2] Release upstream version 1.31.0.10 [debs/amd-k8s-device-plugin] - 10https://gerrit.wikimedia.org/r/1344242 (https://phabricator.wikimedia.org/T436928) (owner: 10Dpogorzelski) [12:51:07] (03CR) 10Dpogorzelski: [V:03+2 C:03+2] Release upstream version 1.31.0.10 [debs/amd-k8s-device-plugin] - 10https://gerrit.wikimedia.org/r/1344242 (https://phabricator.wikimedia.org/T436928) (owner: 10Dpogorzelski) [12:52:28] (03PS10) 10Arnaudb: admin_ng: restrict the aphlict admin endpoint to the Phabricator servers [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341190 (https://phabricator.wikimedia.org/T436657) [12:52:28] (03CR) 10Arnaudb: "as I mentioned in https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1338751/comments/d62a9226_24fc904a" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341190 (https://phabricator.wikimedia.org/T436657) (owner: 10Arnaudb) [12:55:45] (03PS28) 10Brouberol: ceph: add a cookbook to remove an OSD [cookbooks] - 10https://gerrit.wikimedia.org/r/1344201 (https://phabricator.wikimedia.org/T438823) [12:55:50] !log brouberol@cumin1004 START - Cookbook sre.ceph.remove-osd [12:55:52] (03PS8) 10Arnaudb: aphlict: add chart for the Phabricator notification relay [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341856 (https://phabricator.wikimedia.org/T436657) [12:55:59] !log brouberol@cumin1004 END (PASS) - Cookbook sre.ceph.remove-osd (exit_code=0) [12:56:10] !log brouberol@cumin1004 START - Cookbook sre.ceph.remove-osd [12:56:11] !log brouberol@cumin1004 END (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) [12:56:16] !log manually run puppet agent [12:56:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:56:23] (sorry! wrong channel) [12:57:20] (03PS29) 10Brouberol: ceph: add a cookbook to remove an OSD [cookbooks] - 10https://gerrit.wikimedia.org/r/1344201 (https://phabricator.wikimedia.org/T438823) [12:57:24] !log brouberol@cumin1004 START - Cookbook sre.ceph.remove-osd [12:57:25] !log brouberol@cumin1004 END (FAIL) - Cookbook sre.ceph.remove-osd (exit_code=99) [13:01:09] (03PS7) 10Arnaudb: aux-k8s: add the aphlict namespace and its certificate [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341855 (https://phabricator.wikimedia.org/T436657) [13:01:09] (03PS8) 10Arnaudb: aphlict: deploy to both aux-k8s clusters [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341857 (https://phabricator.wikimedia.org/T436657) [13:01:09] (03PS11) 10Arnaudb: admin_ng: restrict the aphlict admin endpoint to the Phabricator servers [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341190 (https://phabricator.wikimedia.org/T436657) [13:01:41] 06SRE, 10observability, 06Traffic, 13Patch-For-Review: HAProxy metrics go down on config reload - https://phabricator.wikimedia.org/T343000#12354871 (10Vgutierrez) 05Stalled→03In progress [13:02:36] (03PS9) 10Arnaudb: aphlict: deploy to both aux-k8s clusters [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341857 (https://phabricator.wikimedia.org/T436657) [13:02:44] (03PS12) 10Arnaudb: admin_ng: restrict the aphlict admin endpoint to the Phabricator servers [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341190 (https://phabricator.wikimedia.org/T436657) [13:02:56] (03PS1) 10Ayounsi: Netbox: move the PLUGINS config to Hiera [puppet] - 10https://gerrit.wikimedia.org/r/1344269 [13:03:36] (03PS2) 10Lerickson: Enable wdqs-proxy metrics on port 9100. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344104 (https://phabricator.wikimedia.org/T438789) [13:04:04] (03PS30) 10Brouberol: ceph: add a cookbook to remove an OSD [cookbooks] - 10https://gerrit.wikimedia.org/r/1344201 (https://phabricator.wikimedia.org/T438823) [13:04:04] !log cwilliams@cumin1004 START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section test-s4 [13:05:25] (03CR) 10CI reject: [V:04-1] Netbox: move the PLUGINS config to Hiera [puppet] - 10https://gerrit.wikimedia.org/r/1344269 (owner: 10Ayounsi) [13:06:25] RESOLVED: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:06:30] (03PS2) 10Ayounsi: Netbox: move the PLUGINS config to Hiera [puppet] - 10https://gerrit.wikimedia.org/r/1344269 [13:07:15] cwilliams@cumin1004 finalize (PID 396246) is awaiting input [13:07:32] (03PS31) 10Brouberol: ceph: add a cookbook to remove an OSD [cookbooks] - 10https://gerrit.wikimedia.org/r/1344201 (https://phabricator.wikimedia.org/T438823) [13:07:33] (03PS10) 10Arnaudb: aphlict: deploy to both aux-k8s clusters [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341857 (https://phabricator.wikimedia.org/T436657) [13:07:33] (03PS13) 10Arnaudb: admin_ng: restrict the aphlict admin endpoint to the Phabricator servers [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341190 (https://phabricator.wikimedia.org/T436657) [13:08:17] (03CR) 10Ayounsi: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1344269 (owner: 10Ayounsi) [13:09:19] !log cwilliams@cumin1004 END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section test-s4 [13:10:08] (03PS14) 10Arnaudb: admin_ng: restrict the aphlict admin endpoint to the Phabricator servers [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341190 (https://phabricator.wikimedia.org/T436657) [13:10:09] (03CR) 10CI reject: [V:04-1] Netbox: move the PLUGINS config to Hiera [puppet] - 10https://gerrit.wikimedia.org/r/1344269 (owner: 10Ayounsi) [13:10:32] !log cwilliams@cumin1004 START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4 [13:12:53] (03CR) 10Herron: [C:03+1] "nice!" [alerts] - 10https://gerrit.wikimedia.org/r/1337633 (https://phabricator.wikimedia.org/T436049) (owner: 10Hnowlan) [13:13:04] (03CR) 10Lerickson: Enable wdqs-proxy metrics on port 9100. (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344104 (https://phabricator.wikimedia.org/T438789) (owner: 10Lerickson) [13:13:08] (03PS3) 10Ayounsi: Netbox: move the PLUGINS config to Hiera [puppet] - 10https://gerrit.wikimedia.org/r/1344269 [13:13:21] !log cwilliams@cumin1004 END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4 [13:13:45] (03CR) 10Lerickson: [C:03+2] Remove a default value that never took effect anyway. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1343643 (https://phabricator.wikimedia.org/T438789) (owner: 10Lerickson) [13:14:35] (03CR) 10Ayounsi: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1344269 (owner: 10Ayounsi) [13:16:21] (03Merged) 10jenkins-bot: Remove a default value that never took effect anyway. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1343643 (https://phabricator.wikimedia.org/T438789) (owner: 10Lerickson) [13:18:45] !log sukhe@cumin1004 START - Cookbook sre.hosts.reimage for host sretest2013.codfw.wmnet with OS trixie [13:18:53] 10ops-codfw, 06SRE, 06DC-Ops: Q1:rack/setup/install sretest2013 (configF test host) - https://phabricator.wikimedia.org/T436691#12354907 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1004 for host sretest2013.codfw.wmnet with OS trixie [13:21:13] 10ops-eqiad, 06SRE, 06DC-Ops: hw troubleshooting: disk errors for cephosd1002.eqiad.wmnet /c0/e23/s4 - https://phabricator.wikimedia.org/T438952#12354916 (10Jclark-ctr) @btullis this can be swapped at anytime correct? [13:23:48] FIRING: KubernetesCalicoDown: wikikube-worker1152.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=eqiad%20prometheus%2Fk8s&var-instance=wikikube-worker1152.eqiad.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [13:24:26] !log jelto@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on wikikube-worker1152.eqiad.wmnet with reason: hardware/networking issues [13:24:28] !log cwilliams@cumin1004 START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4 [13:24:45] 10ops-eqiad, 06SRE, 06DC-Ops, 10Prod-Kubernetes, and 3 others: wikikube-worker1152.eqiad.wmnet networking issue - https://phabricator.wikimedia.org/T438819#12354922 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=8e373ae7-e423-4139-8181-76cfabd08483) set by jelto@cumin1004 for 2 days, 0... [13:25:29] !log cwilliams@cumin1004 END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4 [13:25:37] (03PS4) 10Awight: Configure beta cluster to use the SiteMatrixLookup [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344139 (https://phabricator.wikimedia.org/T432369) [13:25:50] (03PS4) 10Ayounsi: Netbox: move the PLUGINS config to Hiera [puppet] - 10https://gerrit.wikimedia.org/r/1344269 [13:26:05] (03CR) 10TrainBranchBot: [C:03+2] "Approved by awight@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344139 (https://phabricator.wikimedia.org/T432369) (owner: 10Awight) [13:26:20] (03CR) 10Ayounsi: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1344269 (owner: 10Ayounsi) [13:27:30] (03Merged) 10jenkins-bot: Configure beta cluster to use the SiteMatrixLookup [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344139 (https://phabricator.wikimedia.org/T432369) (owner: 10Awight) [13:28:01] (03PS8) 10Arnaudb: aux-k8s: add the aphlict namespace and its certificate [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341855 (https://phabricator.wikimedia.org/T436657) [13:28:01] (03PS11) 10Arnaudb: aphlict: deploy to both aux-k8s clusters [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341857 (https://phabricator.wikimedia.org/T436657) [13:28:01] (03PS15) 10Arnaudb: admin_ng: restrict the aphlict admin endpoint to the Phabricator servers [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341190 (https://phabricator.wikimedia.org/T436657) [13:29:26] !log btullis@cumin1004 START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{dse-k8s-wdqs-test1001.eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) [13:29:29] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test1001.eqiad.wmnet [13:30:04] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test1001.eqiad.wmnet [13:30:39] (03CR) 10Klausman: [C:03+1] ceph: add a cookbook to remove an OSD (032 comments) [cookbooks] - 10https://gerrit.wikimedia.org/r/1344201 (https://phabricator.wikimedia.org/T438823) (owner: 10Brouberol) [13:32:50] 10ops-eqiad, 06SRE, 06DC-Ops: hw troubleshooting: disk errors for cephosd1002.eqiad.wmnet /c0/e23/s4 - https://phabricator.wikimedia.org/T438952#12354956 (10Jclark-ctr) a:03Jclark-ctr @BTullis we do not have this drive available in spares this is a 18tb SAS drive model ST18000NM006J @RobH @wiki_willy t... [13:33:21] (03CR) 10Klausman: [C:03+1] ceph: add a cookbook to remove an OSD (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/1344201 (https://phabricator.wikimedia.org/T438823) (owner: 10Brouberol) [13:34:45] (03CR) 10Btullis: [C:03+1] "Nice. Thanks." [cookbooks] - 10https://gerrit.wikimedia.org/r/1344201 (https://phabricator.wikimedia.org/T438823) (owner: 10Brouberol) [13:35:54] !log cdobbins@cumin1004 START - Cookbook sre.hosts.reimage for host ncredir1002.eqiad.wmnet with OS trixie [13:36:03] (03CR) 10Gmodena: [C:03+1] Enable wdqs-proxy metrics on port 9100. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344104 (https://phabricator.wikimedia.org/T438789) (owner: 10Lerickson) [13:36:09] (03CR) 10CDanis: [C:03+1] Netbox: move the PLUGINS config to Hiera [puppet] - 10https://gerrit.wikimedia.org/r/1344269 (owner: 10Ayounsi) [13:37:11] 10ops-eqiad, 06SRE, 06DC-Ops: hw troubleshooting: disk errors for cephosd1002.eqiad.wmnet /c0/e23/s4 - https://phabricator.wikimedia.org/T438952#12354980 (10BTullis) Oh, OK. I didn't realise that the warranty had expired. We could survive without it, but a replacement would be nice. [13:37:35] (03CR) 10Ayounsi: [C:03+2] Netbox: move the PLUGINS config to Hiera [puppet] - 10https://gerrit.wikimedia.org/r/1344269 (owner: 10Ayounsi) [13:37:39] (03CR) 10Gmodena: [C:03+1] Enable wdqs-proxy metrics on port 9100. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344104 (https://phabricator.wikimedia.org/T438789) (owner: 10Lerickson) [13:37:50] !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test1001.eqiad.wmnet [13:37:51] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test1001.eqiad.wmnet [13:37:51] !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{dse-k8s-wdqs-test1001.eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad) [13:38:53] !log vgutierrez@cumin1004 START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_magru and not P{cp[7010,7016].*} and A:cp - 3.2.23 upgrade (T438828) [13:38:55] T438828: Upgrade HAProxy to 3.2.23 on cp hosts - https://phabricator.wikimedia.org/T438828 [13:39:42] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T438865#12354986 (10phaultfinder) [13:40:58] (03PS1) 10Dreamy Jazz: WikimediaAntiAbuse: Enable likely vandalism classifier on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344281 (https://phabricator.wikimedia.org/T438860) [13:42:02] (03PS1) 10Jforrester: wikifunctions: Upgrade evaluators from 2026-09-16-212211 to 2026-09-23-002803 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344282 (https://phabricator.wikimedia.org/T402539) [13:42:17] (03PS1) 10Jforrester: wikifunctions: Upgrade orchestrator from 2026-09-16-192603 to 2026-09-23-042024 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344283 (https://phabricator.wikimedia.org/T396943) [13:45:45] 06SRE, 10SRE-swift-storage: rclone cannot handle objects with 0x201B / SINGLE HIGH-REVERSED-9 QUOTATION MARK / ‛ in the name - https://phabricator.wikimedia.org/T438961 (10MatthewVernon) 03NEW [13:47:05] (03CR) 10Kosta Harlan: [C:03+1] WikimediaAntiAbuse: Enable likely vandalism classifier on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344281 (https://phabricator.wikimedia.org/T438860) (owner: 10Dreamy Jazz) [13:52:00] FIRING: PingLossPercent: Blackbox probe packet loss 4.583% from codfw to magru - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=magru - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [13:53:18] !log cdobbins@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir1002.eqiad.wmnet with reason: host reimage [13:56:29] !log sukhe@cumin1004 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2013.codfw.wmnet with OS trixie [13:56:36] 10ops-codfw, 06SRE, 06DC-Ops: Q1:rack/setup/install sretest2013 (configF test host) - https://phabricator.wikimedia.org/T436691#12355129 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1004 for host sretest2013.codfw.wmnet with OS trixie executed with errors: - sretest20... [13:58:26] !log cdobbins@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir1002.eqiad.wmnet with reason: host reimage [13:59:48] (03CR) 10Volans: [C:04-1] "This is misusing completely the dry-run concept, needs clarifications" [cookbooks] - 10https://gerrit.wikimedia.org/r/1344201 (https://phabricator.wikimedia.org/T438823) (owner: 10Brouberol) [14:05:50] (03CR) 10Genoveva Galarza: [C:03+2] wikifunctions: Upgrade evaluators from 2026-09-16-212211 to 2026-09-23-002803 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344282 (https://phabricator.wikimedia.org/T402539) (owner: 10Jforrester) [14:06:41] FIRING: [3x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:06:54] (03CR) 10JHathaway: [C:03+2] stdlib upgrade: Remove use of compat types [puppet] - 10https://gerrit.wikimedia.org/r/1344101 (https://phabricator.wikimedia.org/T438912) (owner: 10JHathaway) [14:07:47] !log ebernhardson@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-semantic-search: apply [14:07:55] !log ebernhardson@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search: apply [14:08:11] (03Merged) 10jenkins-bot: wikifunctions: Upgrade evaluators from 2026-09-16-212211 to 2026-09-23-002803 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344282 (https://phabricator.wikimedia.org/T402539) (owner: 10Jforrester) [14:08:35] (03PS1) 10PipelineBot: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344291 [14:09:24] !log gengh@deploy1003 helmfile [staging] START helmfile.d/services/wikifunctions: apply [14:10:14] !log gengh@deploy1003 helmfile [staging] DONE helmfile.d/services/wikifunctions: apply [14:13:16] (03PS1) 10Urbanecm: feat(AddLink): Do not resuggest an already reviewed page [extensions/GrowthExperiments] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344292 (https://phabricator.wikimedia.org/T429417) [14:13:23] jouncebot: nowandnext [14:13:23] No deployments scheduled for the next 3 hour(s) and 46 minute(s) [14:13:23] In 3 hour(s) and 46 minute(s): MediaWiki train - Utc-7 Version (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260923T1800) [14:13:42] (03PS1) 10Urbanecm: feat(AddLink): Do not resuggest an already reviewed page [extensions/GrowthExperiments] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1344293 (https://phabricator.wikimedia.org/T429417) [14:13:49] (03CR) 10Urbanecm: [C:03+2] feat(AddLink): Do not resuggest an already reviewed page [extensions/GrowthExperiments] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344292 (https://phabricator.wikimedia.org/T429417) (owner: 10Urbanecm) [14:13:53] (03CR) 10Urbanecm: [C:03+2] feat(AddLink): Do not resuggest an already reviewed page [extensions/GrowthExperiments] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1344293 (https://phabricator.wikimedia.org/T429417) (owner: 10Urbanecm) [14:14:41] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T438865#12355193 (10phaultfinder) [14:17:38] !log cdobbins@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir1002.eqiad.wmnet with OS trixie [14:18:50] (03PS1) 10Genoveva Galarza: Revert "wikifunctions: Upgrade evaluators from 2026-09-16-212211 to 2026-09-23-002803" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344295 [14:19:27] (03CR) 10Genoveva Galarza: [C:03+2] Revert "wikifunctions: Upgrade evaluators from 2026-09-16-212211 to 2026-09-23-002803" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344295 (owner: 10Genoveva Galarza) [14:21:46] (03Merged) 10jenkins-bot: feat(AddLink): Do not resuggest an already reviewed page [extensions/GrowthExperiments] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344292 (https://phabricator.wikimedia.org/T429417) (owner: 10Urbanecm) [14:21:49] (03Merged) 10jenkins-bot: feat(AddLink): Do not resuggest an already reviewed page [extensions/GrowthExperiments] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1344293 (https://phabricator.wikimedia.org/T429417) (owner: 10Urbanecm) [14:21:58] (03Merged) 10jenkins-bot: Revert "wikifunctions: Upgrade evaluators from 2026-09-16-212211 to 2026-09-23-002803" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344295 (owner: 10Genoveva Galarza) [14:22:00] RESOLVED: PingLossPercent: Blackbox probe packet loss 3.125% from codfw to magru - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://grafana.wikimedia.org/d/c4eb2910-ee2b-459b-8df6-369803085a1b/?from=now-3h&to=now&var-site=magru - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [14:23:04] !log urbanecm@deploy1003 Started scap sync-world: Backport for [[gerrit:1344292|feat(AddLink): Do not resuggest an already reviewed page (T429417)]], [[gerrit:1344293|feat(AddLink): Do not resuggest an already reviewed page (T429417)]] [14:23:08] T429417: Add Link should not make a recommendation if one has already been accepted or declined (Limit an article to only one Add a Link suggestion) - https://phabricator.wikimedia.org/T429417 [14:23:26] !log gengh@deploy1003 helmfile [staging] START helmfile.d/services/wikifunctions: apply [14:23:53] 10SRE-swift-storage, 10Ceph, 06Infrastructure-Foundations: Move the majority of the Registry's docker image prefixes to a new s3 bucket - https://phabricator.wikimedia.org/T435499#12355228 (10Dzahn) Thank you very much @elukey [14:24:12] !log gengh@deploy1003 helmfile [staging] DONE helmfile.d/services/wikifunctions: apply [14:24:30] (03CR) 10Btullis: [C:03+2] dse-k8s-eqiad: Add a second ipv4 pool to calico [deployment-charts] - 10https://gerrit.wikimedia.org/r/1343998 (https://phabricator.wikimedia.org/T430658) (owner: 10Btullis) [14:24:36] (03CR) 10Genoveva Galarza: [C:03+2] wikifunctions: Upgrade orchestrator from 2026-09-16-192603 to 2026-09-23-042024 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344283 (https://phabricator.wikimedia.org/T396943) (owner: 10Jforrester) [14:26:07] !log gengh@deploy1003 helmfile [codfw] START helmfile.d/services/wikifunctions: apply [14:26:14] !log gengh@deploy1003 helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply [14:28:19] !log cdobbins@cumin1004 conftool action : set/pooled=yes; selector: name=ncredir1002.* [14:28:37] !log sukhe@cumin1004 START - Cookbook sre.hosts.reimage for host sretest2013.codfw.wmnet with OS trixie [14:28:44] 10ops-codfw, 06SRE, 06DC-Ops: Q1:rack/setup/install sretest2013 (configF test host) - https://phabricator.wikimedia.org/T436691#12355238 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1004 for host sretest2013.codfw.wmnet with OS trixie [14:29:45] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T438865#12355242 (10phaultfinder) [14:31:36] (03PS1) 10Jforrester: wikifunctions: Upgrade evaluators from 2026-09-16-212211 to 2026-09-22-135346 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344297 (https://phabricator.wikimedia.org/T402539) [14:35:22] (03PS1) 10Krinkle: ProductionServices: Simplify indirect parsercache-dbs stub [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344298 (https://phabricator.wikimedia.org/T362786) [14:35:55] (03Merged) 10jenkins-bot: dse-k8s-eqiad: Add a second ipv4 pool to calico [deployment-charts] - 10https://gerrit.wikimedia.org/r/1343998 (https://phabricator.wikimedia.org/T430658) (owner: 10Btullis) [14:35:57] (03Merged) 10jenkins-bot: wikifunctions: Upgrade orchestrator from 2026-09-16-192603 to 2026-09-23-042024 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344283 (https://phabricator.wikimedia.org/T396943) (owner: 10Jforrester) [14:36:14] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_magru and not P{cp[7010,7016].*} and A:cp - 3.2.23 upgrade (T438828) [14:36:18] T438828: Upgrade HAProxy to 3.2.23 on cp hosts - https://phabricator.wikimedia.org/T438828 [14:36:18] !log gengh@deploy1003 helmfile [staging] START helmfile.d/services/wikifunctions: apply [14:36:48] !log gengh@deploy1003 helmfile [staging] DONE helmfile.d/services/wikifunctions: apply [14:37:16] !log gengh@deploy1003 helmfile [codfw] START helmfile.d/services/wikifunctions: apply [14:37:28] !log vgutierrez@cumin1004 START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_magru and not P{cp[7010,7016].*} and A:cp - 3.2.23 upgrade (T438828) [14:37:39] !log urbanecm@deploy1003 Finished scap sync-world: Backport for [[gerrit:1344292|feat(AddLink): Do not resuggest an already reviewed page (T429417)]], [[gerrit:1344293|feat(AddLink): Do not resuggest an already reviewed page (T429417)]] (duration: 14m 34s) [14:37:42] T429417: Add Link should not make a recommendation if one has already been accepted or declined (Limit an article to only one Add a Link suggestion) - https://phabricator.wikimedia.org/T429417 [14:37:56] !log gengh@deploy1003 helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply [14:38:08] !log gengh@deploy1003 helmfile [eqiad] START helmfile.d/services/wikifunctions: apply [14:38:43] !log gengh@deploy1003 helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply [14:39:25] (03CR) 10Genoveva Galarza: [C:03+2] wikifunctions: Upgrade evaluators from 2026-09-16-212211 to 2026-09-22-135346 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344297 (https://phabricator.wikimedia.org/T402539) (owner: 10Jforrester) [14:39:29] !log upload debuerreotype 0.15-1.1+wmf13u1 to component/main from trixie-wikimedia T438866 [14:39:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:39:32] T438866: Ignore Valid-Until for WMF container images - https://phabricator.wikimedia.org/T438866 [14:40:07] !log cwilliams@cumin1004 START - Cookbook sre.switchdc.databases.finalize for the switch from codfw to eqiad for section test-s4 [14:40:57] !log cwilliams@cumin1004 END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from codfw to eqiad for section test-s4 [14:41:48] (03Merged) 10jenkins-bot: wikifunctions: Upgrade evaluators from 2026-09-16-212211 to 2026-09-22-135346 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344297 (https://phabricator.wikimedia.org/T402539) (owner: 10Jforrester) [14:42:07] !log cdobbins@cumin1004 START - Cookbook sre.hosts.reimage for host ncredir7004.magru.wmnet with OS trixie [14:42:18] !log aqu@deploy1003 Started deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] [14:42:20] !log cwilliams@cumin1004 START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4 [14:42:30] !log aqu@deploy1003 Finished deploy [analytics/refinery@58c9356] (thin): Regular analytics weekly train THIN [analytics/refinery@58c93566] (duration: 00m 12s) [14:42:42] !log gengh@deploy1003 helmfile [staging] START helmfile.d/services/wikifunctions: apply [14:43:33] !log gengh@deploy1003 helmfile [staging] DONE helmfile.d/services/wikifunctions: apply [14:43:58] !log cwilliams@cumin1004 END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4 [14:44:28] !log cwilliams@cumin1004 START - Cookbook sre.switchdc.databases.prepare for the switch from eqiad to codfw for section test-s4 [14:44:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T438865#12355342 (10phaultfinder) [14:44:42] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [14:45:18] !log btullis@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [14:45:25] !log gengh@deploy1003 helmfile [codfw] START helmfile.d/services/wikifunctions: apply [14:45:28] !log cwilliams@cumin1004 END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from eqiad to codfw for section test-s4 [14:46:26] !log cwilliams@cumin1004 START - Cookbook sre.switchdc.databases.finalize for the switch from eqiad to codfw for section test-s4 [14:46:26] (03CR) 10Lerickson: "Done" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344104 (https://phabricator.wikimedia.org/T438789) (owner: 10Lerickson) [14:47:00] !log cwilliams@cumin1004 END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from eqiad to codfw for section test-s4 [14:47:19] !log gengh@deploy1003 helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply [14:47:33] !log gengh@deploy1003 helmfile [eqiad] START helmfile.d/services/wikifunctions: apply [14:47:51] (03PS4) 10CWilliams: switchdc.databases.prepare: Retry checks for replication threads [cookbooks] - 10https://gerrit.wikimedia.org/r/1344031 (https://phabricator.wikimedia.org/T438833) [14:49:04] (03CR) 10CWilliams: "Test ouput is on the ticket. The only issues are due to the rw/ro status, which doesn't seem to be part of the DB cookbooks for the switch" [cookbooks] - 10https://gerrit.wikimedia.org/r/1344031 (https://phabricator.wikimedia.org/T438833) (owner: 10CWilliams) [14:53:04] !log cwilliams@cumin1004 START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4 [14:53:25] !log gengh@deploy1003 helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply [14:53:53] !log cwilliams@cumin1004 END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4 [14:54:12] !log jayme@cumin1004 START - Cookbook sre.discovery.service-route check mw-web-ro: maintenance [14:54:12] !log jayme@cumin1004 END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check mw-web-ro: maintenance [14:54:32] !log jayme@cumin1004 START - Cookbook sre.discovery.service-route depool mw-web-ro in eqiad: maintenance [14:54:39] !log cwilliams@cumin1004 START - Cookbook sre.switchdc.databases.finalize for the switch from codfw to eqiad for section test-s4 [14:55:09] !log cwilliams@cumin1004 END (PASS) - Cookbook sre.switchdc.databases.finalize (exit_code=0) for the switch from codfw to eqiad for section test-s4 [14:55:27] !log jayme@cumin1004 END (FAIL) - Cookbook sre.discovery.service-route (exit_code=99) depool mw-web-ro in eqiad: maintenance [14:55:50] !log jayme@cumin1004 START - Cookbook sre.discovery.service-route check mw-web-ro: maintenance [14:55:50] !log jayme@cumin1004 END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) check mw-web-ro: maintenance [14:55:51] (03PS3) 10Muehlenhoff: aptrepo: Remove buster-wikimedia [puppet] - 10https://gerrit.wikimedia.org/r/1247618 [14:55:54] (03PS2) 10Ebernhardson: opensearch-semantic-search-ssd: new release on local SSDs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344063 (https://phabricator.wikimedia.org/T438058) [14:55:55] (03CR) 10Ebernhardson: opensearch-semantic-search-ssd: new release on local SSDs (035 comments) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344063 (https://phabricator.wikimedia.org/T438058) (owner: 10Ebernhardson) [14:56:13] (03CR) 10Jgiannelos: [C:03+2] wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344291 (owner: 10PipelineBot) [14:56:23] !log sukhe@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2013.codfw.wmnet with reason: host reimage [14:56:57] (03CR) 10Jgiannelos: [V:03+2 C:03+2] wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344291 (owner: 10PipelineBot) [14:57:07] (03CR) 10Muehlenhoff: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1247618 (owner: 10Muehlenhoff) [14:57:39] !log jgiannelos@deploy1003 helmfile [staging] START helmfile.d/services/wikifeeds: apply [14:57:42] !log jgiannelos@deploy1003 helmfile [staging] DONE helmfile.d/services/wikifeeds: apply [14:57:44] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users and deployment for lwatson - https://phabricator.wikimedia.org/T438926#12355416 (10tappof) [14:57:48] !log jgiannelos@deploy1003 helmfile [codfw] START helmfile.d/services/wikifeeds: apply [14:57:54] !log jgiannelos@deploy1003 helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply [14:58:00] !log jgiannelos@deploy1003 helmfile [eqiad] START helmfile.d/services/wikifeeds: apply [14:58:00] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users and deployment for lwatson - https://phabricator.wikimedia.org/T438926#12355419 (10tappof) out-of-band verification in progress [14:58:03] !log jgiannelos@deploy1003 helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply [14:58:47] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users and deployment for lwatson - https://phabricator.wikimedia.org/T438926#12355427 (10tappof) @thcipriani could you please take a look at the request and, if everything looks good, approve it? [14:59:44] 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users and deployment for lwatson - https://phabricator.wikimedia.org/T438926#12355433 (10tappof) [15:00:48] !log jgiannelos@deploy1003 helmfile [staging] START helmfile.d/services/wikifeeds: apply [15:00:55] !log dancy@deploy1003 Installing scap version "4.292.0" for 3 host(s) [15:01:10] !log jgiannelos@deploy1003 helmfile [staging] DONE helmfile.d/services/wikifeeds: apply [15:01:12] hmm...this doesn't seem good https://www.irccloud.com/pastebin/ONVpUsRF/ [15:01:15] !log jgiannelos@deploy1003 helmfile [codfw] START helmfile.d/services/wikifeeds: apply [15:01:20] dancy: you seem installing new scap version, should i rerun? [15:01:42] !log jayme@deploy1003 conftool action : set/pooled=false; selector: dnsdisc=mw-web-next-ro,name=eqiad [15:01:43] !log jgiannelos@deploy1003 helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply [15:01:53] !log jgiannelos@deploy1003 helmfile [eqiad] START helmfile.d/services/wikifeeds: apply [15:01:55] (03PS1) 10CWilliams: mysql: add A:db-core-test query_parts [software/spicerack] - 10https://gerrit.wikimedia.org/r/1344300 (https://phabricator.wikimedia.org/T438833) [15:01:57] urbanecm: Sorry about that! I didn't see that you were running something. The scap upgrade will be done in a couple of minutes. I'll ping you [15:02:21] !log jgiannelos@deploy1003 helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply [15:02:43] !log sukhe@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2013.codfw.wmnet with reason: host reimage [15:02:51] (03CR) 10CWilliams: "Test output on the ticket showing that test-s4 could be used." [software/spicerack] - 10https://gerrit.wikimedia.org/r/1344300 (https://phabricator.wikimedia.org/T438833) (owner: 10CWilliams) [15:02:56] (03PS1) 10Lerickson: Bump wdqs-proxy 0.9.0 -> 0.10.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344301 (https://phabricator.wikimedia.org/T438789) [15:02:57] !log dancy@deploy1003 Installation of scap version "4.292.0" completed for 3 hosts [15:03:04] urbanecm: I'm done. [15:03:21] (03PS3) 10Lerickson: Enable wdqs-proxy metrics on port 9100. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344104 (https://phabricator.wikimedia.org/T438789) [15:03:26] dancy: my scap finished half an hour ago, i just noticed it printed that as a warning in the middle [15:03:31] <_joe_> uh please wait before doing a deployment [15:03:52] _joe_: waiting [15:04:08] (03PS34) 10Brouberol: ceph: add a cookbook to remove an OSD [cookbooks] - 10https://gerrit.wikimedia.org/r/1344201 (https://phabricator.wikimedia.org/T438823) [15:04:15] <_joe_> yeah our bad, we're testing capacity for the switchover that we botched today [15:04:19] urbanecm: Ah, I see. If you have something else to deploy, let's see how things look then. [15:04:34] i'm not sure my first deploy went through given the message i got [15:04:36] (03CR) 10Lerickson: "... But now that I think of it, it DOES depend on the proxy bump (or at least only makes sense after the proxy is bumped), so I will add a" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344104 (https://phabricator.wikimedia.org/T438789) (owner: 10Lerickson) [15:04:44] but i'll wait before touching anything else [15:04:52] urbanecm: Did you use spiderpig? If so, drop a link to the job log [15:04:54] (03CR) 10Brouberol: ceph: add a cookbook to remove an OSD (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/1344201 (https://phabricator.wikimedia.org/T438823) (owner: 10Brouberol) [15:05:01] (03CR) 10Muehlenhoff: [C:03+2] aptrepo: Remove buster-wikimedia [puppet] - 10https://gerrit.wikimedia.org/r/1247618 (owner: 10Muehlenhoff) [15:05:04] dancy: no, plain scap. i can copy the full logs. [15:06:05] (03CR) 10CI reject: [V:04-1] ceph: add a cookbook to remove an OSD [cookbooks] - 10https://gerrit.wikimedia.org/r/1344201 (https://phabricator.wikimedia.org/T438823) (owner: 10Brouberol) [15:06:28] (03CR) 10Brouberol: ceph: add a cookbook to remove an OSD (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/1344201 (https://phabricator.wikimedia.org/T438823) (owner: 10Brouberol) [15:06:35] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_magru and not P{cp[7010,7016].*} and A:cp - 3.2.23 upgrade (T438828) [15:06:38] T438828: Upgrade HAProxy to 3.2.23 on cp hosts - https://phabricator.wikimedia.org/T438828 [15:06:41] (03PS35) 10Brouberol: ceph: add a cookbook to remove an OSD [cookbooks] - 10https://gerrit.wikimedia.org/r/1344201 (https://phabricator.wikimedia.org/T438823) [15:07:00] (03CR) 10Trueg: [C:03+1] Bump wdqs-proxy 0.9.0 -> 0.10.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344301 (https://phabricator.wikimedia.org/T438789) (owner: 10Lerickson) [15:07:01] (03CR) 10CI reject: [V:04-1] mysql: add A:db-core-test query_parts [software/spicerack] - 10https://gerrit.wikimedia.org/r/1344300 (https://phabricator.wikimedia.org/T438833) (owner: 10CWilliams) [15:07:11] (03PS36) 10Brouberol: ceph: add a cookbook to remove an OSD [cookbooks] - 10https://gerrit.wikimedia.org/r/1344201 (https://phabricator.wikimedia.org/T438823) [15:07:11] (03PS1) 10Majavah: P:wmcs::etcd: Add profile to manage cloudinfra client firewall [puppet] - 10https://gerrit.wikimedia.org/r/1344303 (https://phabricator.wikimedia.org/T410721) [15:07:20] (03PS2) 10Majavah: P:wmcs::etcd: Add profile to manage cloudinfra client firewall [puppet] - 10https://gerrit.wikimedia.org/r/1344303 (https://phabricator.wikimedia.org/T410721) [15:07:29] dancy: https://phabricator.wikimedia.org/P96513 [15:07:58] (03CR) 10Volans: "Important question inline" [software/spicerack] - 10https://gerrit.wikimedia.org/r/1344300 (https://phabricator.wikimedia.org/T438833) (owner: 10CWilliams) [15:08:00] (03PS1) 10Tiziano Fogli: admin/data: grant access to segt (analytics_privatedata_users l3) [puppet] - 10https://gerrit.wikimedia.org/r/1344304 (https://phabricator.wikimedia.org/T438767) [15:08:14] urbanecm: Your deployment was good. No worries. [15:08:15] (03CR) 10Majavah: [V:03+1] "PCC SUCCESS (CORE_DIFF 1): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/9475/co" [puppet] - 10https://gerrit.wikimedia.org/r/1344303 (https://phabricator.wikimedia.org/T410721) (owner: 10Majavah) [15:08:27] thanks for confirming [15:08:35] (03PS37) 10Brouberol: ceph: add a cookbook to remove an OSD [cookbooks] - 10https://gerrit.wikimedia.org/r/1344201 (https://phabricator.wikimedia.org/T438823) [15:09:14] (03CR) 10CI reject: [V:04-1] admin/data: grant access to segt (analytics_privatedata_users l3) [puppet] - 10https://gerrit.wikimedia.org/r/1344304 (https://phabricator.wikimedia.org/T438767) (owner: 10Tiziano Fogli) [15:09:18] _joe_: lemme know when it's good for me to test a scap sync-world. [15:10:10] (03Abandoned) 10Jgiannelos: wikifeeds: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344097 (owner: 10PipelineBot) [15:10:18] (03PS38) 10Brouberol: ceph: add a cookbook to remove an OSD [cookbooks] - 10https://gerrit.wikimedia.org/r/1344201 (https://phabricator.wikimedia.org/T438823) [15:10:30] (03PS2) 10Tiziano Fogli: admin/data: grant access to segt (analytics_privatedata_users l3) [puppet] - 10https://gerrit.wikimedia.org/r/1344304 (https://phabricator.wikimedia.org/T438767) [15:11:15] FIRING: MediaWikiLatencyExceeded: p75 latency high: codfw mw-web releases routed via main (k8s) 1.054s - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=55&var-dc=codfw%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [15:11:53] (03CR) 10Jgiannelos: [C:03+2] mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342653 (owner: 10PipelineBot) [15:12:02] !log removed buster-wikimedia and all related components from apt.wikimedia.org following the merge of https://gerrit.wikimedia.org/r/c/operations/puppet/+/1247618 [15:12:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:12:48] !log jayme@deploy1003 conftool action : set/pooled=true; selector: dnsdisc=mw-web-next-ro,name=eqiad [15:12:52] !log jayme@deploy1003 conftool action : set/pooled=true; selector: dnsdisc=mw-web-ro,name=eqiad [15:14:15] (03Merged) 10jenkins-bot: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1342653 (owner: 10PipelineBot) [15:14:37] !log cdobbins@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir7004.magru.wmnet with reason: host reimage [15:15:05] (03PS1) 10Tiziano Fogli: admin/data: grant access to ashhan (analytics_privatedata_users l3) [puppet] - 10https://gerrit.wikimedia.org/r/1344305 (https://phabricator.wikimedia.org/T438538) [15:15:15] FIRING: [3x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?panelId=18&fullscreen&orgId=1&var-datasource=codfw%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [15:16:15] RESOLVED: MediaWikiLatencyExceeded: p75 latency high: codfw mw-web releases routed via main (k8s) 1.012s - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=55&var-dc=codfw%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [15:17:04] (03PS1) 10Muehlenhoff: Remove buster from the debian module [puppet] - 10https://gerrit.wikimedia.org/r/1344306 [15:17:21] (03CR) 10Filippo Giunchedi: [C:03+1] "Nice! LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1343670 (https://phabricator.wikimedia.org/T436275) (owner: 10Andrew Bogott) [15:18:18] !log cdobbins@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir7004.magru.wmnet with reason: host reimage [15:18:19] <_joe_> dancy: yeah sorry, it's gonna take some time [15:18:29] No problem. I got all day [15:19:01] (03PS39) 10Brouberol: ceph: add a cookbook to remove an OSD [cookbooks] - 10https://gerrit.wikimedia.org/r/1344201 (https://phabricator.wikimedia.org/T438823) [15:19:39] (03PS1) 10Muehlenhoff: Use a more modern distros in comments/examples [puppet] - 10https://gerrit.wikimedia.org/r/1344309 [15:19:41] !log jgiannelos@deploy1003 helmfile [staging] START helmfile.d/services/mobileapps: apply [15:20:00] !log jgiannelos@deploy1003 helmfile [staging] DONE helmfile.d/services/mobileapps: apply [15:20:15] RESOLVED: [6x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [15:20:29] !log jgiannelos@deploy1003 helmfile [codfw] START helmfile.d/services/mobileapps: apply [15:20:43] (03CR) 10CI reject: [V:04-1] Remove buster from the debian module [puppet] - 10https://gerrit.wikimedia.org/r/1344306 (owner: 10Muehlenhoff) [15:20:54] (03PS1) 10Muehlenhoff: uwsgi: Remove support for buster [puppet] - 10https://gerrit.wikimedia.org/r/1344310 [15:21:12] !log jgiannelos@deploy1003 helmfile [codfw] DONE helmfile.d/services/mobileapps: apply [15:21:20] !log jgiannelos@deploy1003 helmfile [eqiad] START helmfile.d/services/mobileapps: apply [15:21:51] !log sukhe@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2013.codfw.wmnet with OS trixie [15:22:02] 10ops-codfw, 06SRE, 06DC-Ops: Q1:rack/setup/install sretest2013 (configF test host) - https://phabricator.wikimedia.org/T436691#12355563 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1004 for host sretest2013.codfw.wmnet with OS trixie completed: - sretest2013 (**PASS*... [15:22:19] (03PS1) 10Ssingh: Revert^2 "cp2059: add this host as a text node" [puppet] - 10https://gerrit.wikimedia.org/r/1344314 [15:22:28] (03PS1) 10Ssingh: Revert^2 "sretest2013: remove manual references to this host" [puppet] - 10https://gerrit.wikimedia.org/r/1344315 [15:23:12] !log vgutierrez@cumin1004 START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_ulsfo and A:cp - 3.2.23 upgrade (T438828) [15:23:16] T438828: Upgrade HAProxy to 3.2.23 on cp hosts - https://phabricator.wikimedia.org/T438828 [15:23:33] !log jgiannelos@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply [15:26:34] (03CR) 10Ssingh: [C:03+2] Revert^2 "sretest2013: remove manual references to this host" [puppet] - 10https://gerrit.wikimedia.org/r/1344315 (owner: 10Ssingh) [15:26:40] (03CR) 10Ssingh: [C:03+2] Revert^2 "cp2059: add this host as a text node" [puppet] - 10https://gerrit.wikimedia.org/r/1344314 (owner: 10Ssingh) [15:27:01] (03PS40) 10Brouberol: ceph: add a cookbook to remove an OSD [cookbooks] - 10https://gerrit.wikimedia.org/r/1344201 (https://phabricator.wikimedia.org/T438823) [15:27:28] FIRING: JobUnavailable: Reduced availability for job redis_arclamp in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [15:32:06] !log rclone copy --no-update-modtime --checksum --config /etc/swift/rclone.conf 'eqiad:wikipedia-commons-local-public.41/4/41/ĞAVĀMI‛_al-ḤIKĀYĀT_VA_LAVĀMI‛_al-RIVĀYĀT._Sadīd_al-Dīn_Muḥ._b._Muḥ._b._Yaḥyà_‛Awfī_Buhārī_Ḥanafī._-_btv1b525129105_(033_of_524).jpg' codfw:wikipedia-commons-local-public.41/4/41 T438961 [15:32:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:32:08] T438961: rclone cannot handle objects with 0x201B / SINGLE HIGH-REVERSED-9 QUOTATION MARK / ‛ in the name - https://phabricator.wikimedia.org/T438961 [15:32:15] (03CR) 10Muehlenhoff: [C:03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/1344304 (https://phabricator.wikimedia.org/T438767) (owner: 10Tiziano Fogli) [15:32:45] 10ops-eqiad, 06SRE, 06Data-Platform-SRE, 06DC-Ops: QFY2627 :rack/setup/install ml-serve1016 - https://phabricator.wikimedia.org/T438161#12355685 (10klausman) >>! In T438161#12352808, @wiki_willy wrote: > Hi @klausman - to follow up, do you need 25g on this one because it's a MI350X instead of a MI300X? Ju... [15:33:26] (03CR) 10Muehlenhoff: [C:03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/1344305 (https://phabricator.wikimedia.org/T438538) (owner: 10Tiziano Fogli) [15:33:43] !log sukhe@cumin1004 START - Cookbook sre.hosts.rename from sretest2013 to cp2059 [15:33:57] 06SRE, 06Commons, 10MediaWiki-File-management, 06Traffic, and 2 others: Varnish serving outdated version at original/non-thumb URL of overwritten file upload - https://phabricator.wikimedia.org/T425216#12355691 (10AFBorchert) I run into this problem when trying to reupload [[ https://commons.wikimedia.org/... [15:34:06] !log sukhe@cumin1004 START - Cookbook sre.dns.netbox [15:34:47] !log dzahn@cumin2003 START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie [15:34:57] (03CR) 10Muehlenhoff: [C:03+2] Use a more modern distros in comments/examples [puppet] - 10https://gerrit.wikimedia.org/r/1344309 (owner: 10Muehlenhoff) [15:35:00] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [15:35:16] !log zuul1005 - reimage - should not have had nftables on it before T438786 [15:35:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:35:18] T438786: SystemdUnitFailed - zuul1005 - https://phabricator.wikimedia.org/T438786 [15:35:20] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [15:35:23] !log rclone copy --no-update-modtime --checksum --config /etc/swift/rclone.conf 'eqiad:wikipedia-commons-local-public.4d/4/4d/Kamāl_al-Dīn_Ḥusayn_b._ʿAlī_Bayhaqī_Sabzavārī_Vā‛iẓ_Kāšifī_._Anvār-i_Suhaylī_-_btv1b10515885n_(142_of_580).jpg' codfw:wikipedia-commons-local-public.4d/4/4d T438961 [15:35:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:36:01] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [15:36:23] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [15:37:00] (03CR) 10JHathaway: [C:03+1] uwsgi: Remove support for buster [puppet] - 10https://gerrit.wikimedia.org/r/1344310 (owner: 10Muehlenhoff) [15:37:59] !log sukhe@cumin1004 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming sretest2013 to cp2059 - sukhe@cumin1004" [15:38:22] !log sukhe@cumin1004 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming sretest2013 to cp2059 - sukhe@cumin1004" [15:38:22] !log sukhe@cumin1004 END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [15:38:22] !log sukhe@cumin1004 START - Cookbook sre.dns.wipe-cache cp2059 on all recursors [15:38:23] !log rclone copy --no-update-modtime --checksum --config /etc/swift/rclone.conf 'eqiad:wikipedia-commons-local-public.a9/a/a9/Ğāmi‛_al-tavārīḫ._Rašīd_al-Dīn_Fazl-ullāh_Hamadānī_-_btv1b8427170s_(182_of_597).jpg' codfw:wikipedia-commons-local-public.a9/a/a9/ T438961 [15:38:26] (03PS2) 10Dzahn: planet: add WWU feed to en.planet [puppet] - 10https://gerrit.wikimedia.org/r/1344039 [15:38:26] !log sukhe@cumin1004 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cp2059 on all recursors [15:38:26] !log sukhe@cumin1004 START - Cookbook sre.network.configure-switch-interfaces for host cp2059 [15:38:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:38:29] T438961: rclone cannot handle objects with 0x201B / SINGLE HIGH-REVERSED-9 QUOTATION MARK / ‛ in the name - https://phabricator.wikimedia.org/T438961 [15:38:40] (03CR) 10Dzahn: [C:03+2] planet: add WWU feed to en.planet [puppet] - 10https://gerrit.wikimedia.org/r/1344039 (owner: 10Dzahn) [15:38:42] !log sukhe@cumin1004 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cp2059 [15:39:18] !log sukhe@cumin1004 END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from sretest2013 to cp2059 [15:39:26] 10ops-codfw, 06SRE, 06DC-Ops: Q1:rack/setup/install sretest2013 (configF test host) - https://phabricator.wikimedia.org/T436691#12355740 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by sukhe@cumin1004 from sretest2013 to cp2059 completed: - sretest2013 (**PASS**) - ✔️ Downtimed ho... [15:41:38] !log rclone copy --no-update-modtime --checksum --config /etc/swift/rclone.conf 'eqiad:wikipedia-commons-local-public.c7/c/c7/Kamāl_al-Dīn_Ḥusayn_b._ʿAlī_Bayhaqī_Sabzavārī_Vā‛iẓ_Kāšifī_._Anvār-i_Suhaylī_-_btv1b10515885n_(248_of_580).jpg' codfw:wikipedia-commons-local-public.c7/c/c7 T438961 [15:41:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:41:41] (03PS1) 10Majavah: O:wmcs::novaproxy: Add confd profile [puppet] - 10https://gerrit.wikimedia.org/r/1344318 (https://phabricator.wikimedia.org/T438971) [15:41:58] !log homer "lsw1-e4-codfw.*" commit 'pending from cookbook' [15:41:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:42:11] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [15:42:22] !log cdobbins@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir7004.magru.wmnet with OS trixie [15:42:51] (03CR) 10Klausman: [C:03+1] ceph: add a cookbook to remove an OSD (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/1344201 (https://phabricator.wikimedia.org/T438823) (owner: 10Brouberol) [15:43:34] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c5-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T438872#12355761 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm [15:43:48] !log sukhe@cumin1004 START - Cookbook sre.hosts.reimage for host cp2059.codfw.wmnet with OS trixie [15:43:55] 10ops-codfw, 06SRE, 06DC-Ops: Q1:rack/setup/install sretest2013 (configF test host) - https://phabricator.wikimedia.org/T436691#12355763 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1004 for host cp2059.codfw.wmnet with OS trixie [15:43:59] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-c4-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T438868#12355764 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm [15:46:07] (03CR) 10Volans: ceph: add a cookbook to remove an OSD (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/1344201 (https://phabricator.wikimedia.org/T438823) (owner: 10Brouberol) [15:46:10] !log dancy@deploy1003 Started scap sync-world: testing [15:46:35] 06SRE, 06SRE Observability, 13Patch-For-Review: Chanserv Feature Request - Add the manager oncall in the topic of -sre-private and -operations - https://phabricator.wikimedia.org/T438657#12355798 (10hnowlan) a:03herron [15:47:22] 06SRE, 10SRE-swift-storage: rclone cannot handle objects with 0x201B / SINGLE HIGH-REVERSED-9 QUOTATION MARK / ‛ in the name - https://phabricator.wikimedia.org/T438961#12355813 (10MatthewVernon) I replicated the 4 sad items from eqiad to codfw, thus: # Edit `/etc/swift/rclone.conf` to add `encoding = Raw` to... [15:47:48] 10SRE-SLO, 06SRE Observability, 07Upstream: Thanos (store|query-frontend) memcached cache in bad status - https://phabricator.wikimedia.org/T411273#12355818 (10hnowlan) [15:48:15] 10ops-eqiad, 06SRE, 06Collaboration-Services, 06DC-Ops: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12355824 (10Dzahn) a:05VRiley-WMF→03Dzahn Taking over the ticket for a few follow-up changes. [15:48:26] 10ops-eqiad, 06SRE, 06Collaboration-Services, 06DC-Ops: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12355827 (10Dzahn) 05Resolved→03Open [15:48:32] (03PS2) 10Dzahn: zuul: remove zuul1001 from main zuul node names [puppet] - 10https://gerrit.wikimedia.org/r/1343649 (https://phabricator.wikimedia.org/T427353) [15:49:42] (03CR) 10Dzahn: [C:03+2] "zuul1004 is replacing it entirely" [puppet] - 10https://gerrit.wikimedia.org/r/1343649 (https://phabricator.wikimedia.org/T427353) (owner: 10Dzahn) [15:50:18] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply [15:50:36] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [15:51:25] (03CR) 10Btullis: [C:03+2] opensearch-semantic-search-ssd: new release on local SSDs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344063 (https://phabricator.wikimedia.org/T438058) (owner: 10Ebernhardson) [15:51:30] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply [15:52:31] !log sukhe@cumin1004 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2059.codfw.wmnet with OS trixie [15:52:37] 10ops-codfw, 06SRE, 06DC-Ops: Q1:rack/setup/install sretest2013 (configF test host) - https://phabricator.wikimedia.org/T436691#12355874 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1004 for host cp2059.codfw.wmnet with OS trixie executed with errors: - cp2059 (**FAIL... [15:52:43] !log sukhe@cumin1004 START - Cookbook sre.hosts.reimage for host cp2059.codfw.wmnet with OS trixie [15:52:51] 10ops-codfw, 06SRE, 06DC-Ops: Q1:rack/setup/install sretest2013 (configF test host) - https://phabricator.wikimedia.org/T436691#12355876 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1004 for host cp2059.codfw.wmnet with OS trixie [15:53:17] !log dancy@deploy1003 Finished scap sync-world: testing (duration: 07m 06s) [15:54:04] (03Merged) 10jenkins-bot: opensearch-semantic-search-ssd: new release on local SSDs [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344063 (https://phabricator.wikimedia.org/T438058) (owner: 10Ebernhardson) [15:54:10] !log cdobbins@cumin1004 conftool action : set/pooled=yes; selector: name=ncredir7004.* [15:54:54] !log installing cjose security updates [15:54:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:55:54] !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-upload_ulsfo and A:cp - 3.2.23 upgrade (T438828) [15:55:57] T438828: Upgrade HAProxy to 3.2.23 on cp hosts - https://phabricator.wikimedia.org/T438828 [15:56:12] (03CR) 10Btullis: [C:03+1] "Looks good to me." [cookbooks] - 10https://gerrit.wikimedia.org/r/1344201 (https://phabricator.wikimedia.org/T438823) (owner: 10Brouberol) [16:00:06] PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 140976576 and 16 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [16:01:06] RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 2302168 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [16:02:43] (03PS1) 10Andrew Bogott: openstack::patch: accept patches as strings as well as files [puppet] - 10https://gerrit.wikimedia.org/r/1344324 (https://phabricator.wikimedia.org/T421911) [16:02:46] (03PS1) 10Andrew Bogott: Patch openstack uwsgi ini files to support directing api logs to logstash [puppet] - 10https://gerrit.wikimedia.org/r/1344325 (https://phabricator.wikimedia.org/T421911) [16:03:34] 06SRE, 06Infrastructure-Foundations: Integrate Trixie 13.6 point update - https://phabricator.wikimedia.org/T434866#12355966 (10MoritzMuehlenhoff) [16:04:04] (03CR) 10CI reject: [V:04-1] Patch openstack uwsgi ini files to support directing api logs to logstash [puppet] - 10https://gerrit.wikimedia.org/r/1344325 (https://phabricator.wikimedia.org/T421911) (owner: 10Andrew Bogott) [16:04:51] (03PS2) 10Andrew Bogott: Patch openstack uwsgi ini files to support directing api logs to logstash [puppet] - 10https://gerrit.wikimedia.org/r/1344325 (https://phabricator.wikimedia.org/T421911) [16:05:03] (03CR) 10Dzahn: "@andrewbogott any opinion on this?" [puppet] - 10https://gerrit.wikimedia.org/r/1342354 (https://phabricator.wikimedia.org/T435503) (owner: 10Dzahn) [16:05:57] (03CR) 10CI reject: [V:04-1] Patch openstack uwsgi ini files to support directing api logs to logstash [puppet] - 10https://gerrit.wikimedia.org/r/1344325 (https://phabricator.wikimedia.org/T421911) (owner: 10Andrew Bogott) [16:06:41] (03CR) 10Andrew Bogott: [C:03+1] "Sorry this didn't show up on my radar! Yes, let's try it in codfw1dev" [puppet] - 10https://gerrit.wikimedia.org/r/1342354 (https://phabricator.wikimedia.org/T435503) (owner: 10Dzahn) [16:06:49] (03CR) 10Dzahn: "can I merge this? aware I might get the placeholder page cached but also know how I can purge it later." [puppet] - 10https://gerrit.wikimedia.org/r/1341338 (https://phabricator.wikimedia.org/T437635) (owner: 10Dzahn) [16:08:17] (03CR) 10Dzahn: [C:03+2] "thanks! this should only have an effect after a service restart. then it will start logging to /var/log/zookeeper/zookeeper.log" [puppet] - 10https://gerrit.wikimedia.org/r/1342354 (https://phabricator.wikimedia.org/T435503) (owner: 10Dzahn) [16:09:12] (03PS3) 10Andrew Bogott: Patch openstack uwsgi ini files to support directing api logs to logstash [puppet] - 10https://gerrit.wikimedia.org/r/1344325 (https://phabricator.wikimedia.org/T421911) [16:09:38] 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T438865#12356000 (10phaultfinder) [16:10:17] (03CR) 10CI reject: [V:04-1] Patch openstack uwsgi ini files to support directing api logs to logstash [puppet] - 10https://gerrit.wikimedia.org/r/1344325 (https://phabricator.wikimedia.org/T421911) (owner: 10Andrew Bogott) [16:10:32] !log sukhe@cumin1004 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2059.codfw.wmnet with OS trixie [16:10:41] 10ops-codfw, 06SRE, 06DC-Ops: Q1:rack/setup/install sretest2013 (configF test host) - https://phabricator.wikimedia.org/T436691#12356003 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1004 for host cp2059.codfw.wmnet with OS trixie executed with errors: - cp2059 (**FAIL... [16:10:54] !log sukhe@cumin1004 START - Cookbook sre.hosts.reimage for host cp2059.codfw.wmnet with OS trixie [16:11:01] 10ops-codfw, 06SRE, 06DC-Ops: Q1:rack/setup/install sretest2013 (configF test host) - https://phabricator.wikimedia.org/T436691#12356009 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1004 for host cp2059.codfw.wmnet with OS trixie [16:11:08] (03PS4) 10Andrew Bogott: Patch openstack uwsgi ini files to support directing api logs to logstash [puppet] - 10https://gerrit.wikimedia.org/r/1344325 (https://phabricator.wikimedia.org/T421911) [16:12:13] FIRING: [3x] JobUnavailable: Reduced availability for job redis_arclamp in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:13:35] (03PS1) 10Btullis: dse-k8s-eqiad: put dse-k8s-worker1015 back into service [puppet] - 10https://gerrit.wikimedia.org/r/1344327 (https://phabricator.wikimedia.org/T430658) [16:13:59] (03CR) 10Andrew Bogott: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1344325 (https://phabricator.wikimedia.org/T421911) (owner: 10Andrew Bogott) [16:15:31] (03PS2) 10Btullis: dse-k8s-eqiad: put dse-k8s-worker1015 back into service [puppet] - 10https://gerrit.wikimedia.org/r/1344327 (https://phabricator.wikimedia.org/T430658) [16:17:13] FIRING: [3x] JobUnavailable: Reduced availability for job redis_arclamp in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [16:18:24] (03PS5) 10Andrew Bogott: Patch openstack uwsgi ini files to support directing api logs to logstash [puppet] - 10https://gerrit.wikimedia.org/r/1344325 (https://phabricator.wikimedia.org/T421911) [16:21:47] FIRING: RedisInstanceDown: Redis instance down arclamp2001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_arclamp&var-instance=arclamp2001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [16:25:48] (03PS2) 10Andrew Bogott: openstack::patch: accept patches as strings as well as files [puppet] - 10https://gerrit.wikimedia.org/r/1344324 (https://phabricator.wikimedia.org/T421911) [16:25:48] (03PS6) 10Andrew Bogott: Patch openstack uwsgi ini files to support directing api logs to logstash [puppet] - 10https://gerrit.wikimedia.org/r/1344325 (https://phabricator.wikimedia.org/T421911) [16:26:46] (03CR) 10CI reject: [V:04-1] openstack::patch: accept patches as strings as well as files [puppet] - 10https://gerrit.wikimedia.org/r/1344324 (https://phabricator.wikimedia.org/T421911) (owner: 10Andrew Bogott) [16:27:11] dzahn@cumin2003 reimage (PID 302414) is awaiting input [16:28:59] (03PS3) 10Andrew Bogott: openstack::patch: accept patches as strings as well as files [puppet] - 10https://gerrit.wikimedia.org/r/1344324 (https://phabricator.wikimedia.org/T421911) [16:28:59] (03PS7) 10Andrew Bogott: Patch openstack uwsgi ini files to support directing api logs to logstash [puppet] - 10https://gerrit.wikimedia.org/r/1344325 (https://phabricator.wikimedia.org/T421911) [16:29:25] !log cdobbins@cumin1004 START - Cookbook sre.hosts.reimage for host ncredir6001.drmrs.wmnet with OS trixie [16:30:31] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [16:31:19] (03CR) 10Andrew Bogott: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1344324 (https://phabricator.wikimedia.org/T421911) (owner: 10Andrew Bogott) [16:33:39] 10ops-eqiad, 06SRE, 06Collaboration-Services, 06DC-Ops: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12356134 (10Dzahn) zuul1005 can not be reimaged :/ ` ==> Unable to verify that the host rebooted into the new OS, it might still be in the Debian installer, ple... [16:33:39] (03CR) 10Andrew Bogott: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1344325 (https://phabricator.wikimedia.org/T421911) (owner: 10Andrew Bogott) [16:34:19] !log dzahn@cumin2003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie [16:35:20] 10ops-eqiad, 06SRE, 06Collaboration-Services, 06DC-Ops: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12356153 (10Dzahn) after the failure of the cookbook it can also not be repeated. ` RuntimeError: Host zuul1005.eqiad.wmnet was not found in PuppetDB but --new... [16:35:36] !log dzahn@cumin2003 START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie [16:37:08] (03PS8) 10Andrew Bogott: Patch openstack uwsgi ini files to support directing api logs to logstash [puppet] - 10https://gerrit.wikimedia.org/r/1344325 (https://phabricator.wikimedia.org/T421911) [16:37:16] (03CR) 10Andrew Bogott: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1344325 (https://phabricator.wikimedia.org/T421911) (owner: 10Andrew Bogott) [16:37:16] (03CR) 10Ameisenigel: "Please create required tables with extensions/WikimediaMaintenance/maintenance/createExtensionTables.php" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344216 (https://phabricator.wikimedia.org/T386776) (owner: 10Ameisenigel) [16:38:05] (03PS1) 10Jgiannelos: prv: Enable parsoid rendering for 4 wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344329 (https://phabricator.wikimedia.org/T438998) [16:38:09] (03CR) 10CI reject: [V:04-1] Patch openstack uwsgi ini files to support directing api logs to logstash [puppet] - 10https://gerrit.wikimedia.org/r/1344325 (https://phabricator.wikimedia.org/T421911) (owner: 10Andrew Bogott) [16:39:21] (03PS9) 10Andrew Bogott: Patch openstack uwsgi ini files to support directing api logs to logstash [puppet] - 10https://gerrit.wikimedia.org/r/1344325 (https://phabricator.wikimedia.org/T421911) [16:39:59] !log sukhe@cumin1004 END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp2059.codfw.wmnet with OS trixie [16:40:05] 10ops-codfw, 06SRE, 06DC-Ops: Q1:rack/setup/install sretest2013 (configF test host) - https://phabricator.wikimedia.org/T436691#12356212 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1004 for host cp2059.codfw.wmnet with OS trixie executed with errors: - cp2059 (**FAIL... [16:42:49] FIRING: HelmReleaseBadStatus: Helm release wdqs-next/scholarly-external on k8s-dse@codfw in state pending-upgrade - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [16:43:52] (03CR) 10Andrew Bogott: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1344325 (https://phabricator.wikimedia.org/T421911) (owner: 10Andrew Bogott) [16:43:53] (03PS1) 10Ssingh: install_server: add cp2059 to set LBA_FORMAT_NUMBER=1 [puppet] - 10https://gerrit.wikimedia.org/r/1344330 (https://phabricator.wikimedia.org/T436691) [16:45:37] (03CR) 10BBlack: [C:03+1] install_server: add cp2059 to set LBA_FORMAT_NUMBER=1 [puppet] - 10https://gerrit.wikimedia.org/r/1344330 (https://phabricator.wikimedia.org/T436691) (owner: 10Ssingh) [16:46:05] (03CR) 10Ssingh: [C:03+2] install_server: add cp2059 to set LBA_FORMAT_NUMBER=1 [puppet] - 10https://gerrit.wikimedia.org/r/1344330 (https://phabricator.wikimedia.org/T436691) (owner: 10Ssingh) [16:48:31] !log sukhe@cumin1004 START - Cookbook sre.hosts.reimage for host cp2059.codfw.wmnet with OS trixie [16:48:50] 10ops-codfw, 06SRE, 06DC-Ops, 13Patch-For-Review: Q1:rack/setup/install sretest2013 (configF test host) - https://phabricator.wikimedia.org/T436691#12356346 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1004 for host cp2059.codfw.wmnet with OS trixie [16:49:29] (03CR) 10Andrea Denisse: [C:03+1] "LGTM, thank you!!" [alerts] - 10https://gerrit.wikimedia.org/r/1337633 (https://phabricator.wikimedia.org/T436049) (owner: 10Hnowlan) [16:49:33] FIRING: JobQueueLowTrafficConsumerWidespreadHighLatency: ... [16:49:33] Processing delay times for low-traffic consumer rules are unusually high - https://wikitech.wikimedia.org/wiki/MediaWiki_JobQueue/Operations#JobQueueLowTrafficConsumerWidespreadHighLatency - https://grafana.wikimedia.org/d/fe130675-0c2d-4991-9dec-f54cf6a9c4d8/jobqueue-low-traffic-jobs?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DJobQueueLowTrafficConsumerWidespreadHighLatency [16:51:12] !log sukhe@cumin1004 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp2059.codfw.wmnet with OS trixie [16:51:24] 10ops-codfw, 06SRE, 06DC-Ops, 13Patch-For-Review: Q1:rack/setup/install sretest2013 (configF test host) - https://phabricator.wikimedia.org/T436691#12356364 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1004 for host cp2059.codfw.wmnet with OS trixie executed with er... [16:51:34] !log sukhe@cumin1004 START - Cookbook sre.hosts.reimage for host cp2059.codfw.wmnet with OS trixie [16:51:46] 10ops-codfw, 06SRE, 06DC-Ops, 13Patch-For-Review: Q1:rack/setup/install sretest2013 (configF test host) - https://phabricator.wikimedia.org/T436691#12356370 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin1004 for host cp2059.codfw.wmnet with OS trixie [16:54:33] (03PS10) 10Andrew Bogott: Patch openstack uwsgi ini files to support directing api logs to logstash [puppet] - 10https://gerrit.wikimedia.org/r/1344325 (https://phabricator.wikimedia.org/T421911) [16:55:40] 10SRE-SLO, 06SRE Observability: Support SLOs based on Test Kitchen instruments - https://phabricator.wikimedia.org/T438274#12356404 (10hnowlan) [16:55:41] (03CR) 10CI reject: [V:04-1] Patch openstack uwsgi ini files to support directing api logs to logstash [puppet] - 10https://gerrit.wikimedia.org/r/1344325 (https://phabricator.wikimedia.org/T421911) (owner: 10Andrew Bogott) [16:57:12] (03CR) 10CWilliams: mysql: add A:db-core-test query_parts (031 comment) [software/spicerack] - 10https://gerrit.wikimedia.org/r/1344300 (https://phabricator.wikimedia.org/T438833) (owner: 10CWilliams) [16:58:13] (03PS4) 10Andrew Bogott: openstack::patch: accept patches as strings as well as files [puppet] - 10https://gerrit.wikimedia.org/r/1344324 (https://phabricator.wikimedia.org/T421911) [16:58:13] (03PS11) 10Andrew Bogott: Patch openstack uwsgi ini files to support directing api logs to logstash [puppet] - 10https://gerrit.wikimedia.org/r/1344325 (https://phabricator.wikimedia.org/T421911) [16:58:14] (03PS1) 10Andrew Bogott: Revert "Puppetize keystone-uwsgi.ini" [puppet] - 10https://gerrit.wikimedia.org/r/1344333 (https://phabricator.wikimedia.org/T421911) [16:59:02] (03PS12) 10Andrew Bogott: Patch openstack uwsgi ini files to support directing api logs to logstash [puppet] - 10https://gerrit.wikimedia.org/r/1344325 (https://phabricator.wikimedia.org/T421911) [16:59:14] !log cdobbins@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir6001.drmrs.wmnet with reason: host reimage [16:59:38] (03CR) 10CI reject: [V:04-1] Patch openstack uwsgi ini files to support directing api logs to logstash [puppet] - 10https://gerrit.wikimedia.org/r/1344325 (https://phabricator.wikimedia.org/T421911) (owner: 10Andrew Bogott) [17:02:13] (03CR) 10Andrew Bogott: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1344325 (https://phabricator.wikimedia.org/T421911) (owner: 10Andrew Bogott) [17:02:20] !log cdobbins@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir6001.drmrs.wmnet with reason: host reimage [17:05:01] andrewbogott: it's working. on cloudcontrol2005-dev. systemctl restart zookeeper. let there be logs. tail -f /var/log/zookeeper/zookeeper.log [17:06:03] (03CR) 10Dzahn: [C:03+2] "verified on cloudcontrol2005-dev:" [puppet] - 10https://gerrit.wikimedia.org/r/1342354 (https://phabricator.wikimedia.org/T435503) (owner: 10Dzahn) [17:06:07] (03CR) 10BCornwall: [C:03+1] druid: (step 2) remove coordinator LVS DNS records [dns] - 10https://gerrit.wikimedia.org/r/1342403 (https://phabricator.wikimedia.org/T432080) (owner: 10Ryan Kemper) [17:06:59] Nice! [17:07:36] PROBLEM - Host cp2049 is DOWN: PING CRITICAL - Packet loss = 100% [17:07:42] PROBLEM - Host mr1-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:07:58] PROBLEM - Host ps1-c4-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:07:58] PROBLEM - Host ps1-c5-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:07:58] PROBLEM - Host ps1-c6-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:07:58] PROBLEM - Host ps1-d6-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:07:58] PROBLEM - Host ps1-d7-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:07:58] !log cloudcontrol2005-dev, cloudcontrol2006-dev, cloudcontrol2010-dev: restart zookeeper, enabled logging (/var/log/zookeeper/zookeeper.log) after gerrit:1342354 [17:07:59] PROBLEM - Host ps1-d8-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:07:59] PROBLEM - Host ps1-e2-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:08:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:08:00] PROBLEM - Host ps1-e3-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:08:04] PROBLEM - Host ps1-a2-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:08:04] PROBLEM - Host ps1-a1-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:08:04] PROBLEM - Host ps1-a3-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:08:04] PROBLEM - Host ps1-a4-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:08:04] PROBLEM - Host ps1-a5-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:08:05] PROBLEM - Host ps1-a6-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:08:05] PROBLEM - Host ps1-a8-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:08:06] PROBLEM - Host ps1-b6-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:08:06] PROBLEM - Host ps1-c7-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:08:07] PROBLEM - Host ps1-c3-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:08:07] PROBLEM - Host ps1-c8-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:08:08] PROBLEM - Host ps1-d4-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:08:08] PROBLEM - Host ps1-d1-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:08:09] PROBLEM - Host ps1-d5-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:08:09] PROBLEM - Host ps1-e1-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:08:10] PROBLEM - Host ps1-d3-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:08:10] PROBLEM - Host ps1-e4-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:08:11] PROBLEM - Host ps1-d2-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:08:11] PROBLEM - Host ps1-e5-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:08:12] PROBLEM - Host ps1-b3-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:08:12] PROBLEM - Host ps1-c2-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:08:12] andrewbogott: done on the 3 dev host. that's it for now [17:08:13] PROBLEM - Host ps1-f1-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:08:13] PROBLEM - Host ps1-f2-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:08:14] PROBLEM - Host ps1-f3-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:08:14] PROBLEM - Host ps1-f4-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:08:15] PROBLEM - Host ps1-f5-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:08:22] PROBLEM - Juniper alarms on cr2-codfw is CRITICAL: JNX_ALARMS CRITICAL - 6 red alarms, 0 yellow alarms https://wikitech.wikimedia.org/wiki/Network_monitoring%23Juniper_alarm [17:08:22] PROBLEM - Juniper alarms on cr1-codfw is CRITICAL: JNX_ALARMS CRITICAL - 6 red alarms, 0 yellow alarms https://wikitech.wikimedia.org/wiki/Network_monitoring%23Juniper_alarm [17:08:31] uhhh [17:09:06] PROBLEM - Host ps1-a7-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:09:06] PROBLEM - Host ps1-b1-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:09:06] PROBLEM - Host ps1-b2-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:09:10] PROBLEM - Host ps1-b4-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:09:10] PROBLEM - Host ps1-b5-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:09:10] PROBLEM - Host ps1-b7-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:09:10] PROBLEM - Host ps1-b8-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:09:10] PROBLEM - Host ps1-c1-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:09:54] PROBLEM - Host mr1-codfw IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [17:09:55] FIRING: [4x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:fxp0 () - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [17:10:36] seems like we lost mr1-codfw? [17:10:39] FIRING: [3x] NetworkDeviceAlarmActive: Alarm active on cr1-codfw - https://wikitech.wikimedia.org/wiki/Network_monitoring#Juniper_alarm - https://alerts.wikimedia.org/?q=alertname%3DNetworkDeviceAlarmActive [17:10:44] !log sukhe@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on cp2059.codfw.wmnet with reason: host reimage [17:11:29] !log sukhe@puppetserver1001 conftool action : set/pooled=no; selector: name=cp2049 [17:11:35] !log sukhe@puppetserver1001 conftool action : set/pooled=no; selector: name=cp2049.codfw.wmnet [17:12:03] 10ops-eqiad, 06SRE, 06Data-Platform-SRE, 06DC-Ops: QFY2627 :rack/setup/install ml-serve1016 - https://phabricator.wikimedia.org/T438161#12356501 (10Jclark-ctr) 05Open→03Resolved @klausman thank you for update we will leave 10g installed for right now. We can order and replace with 25g later if i... [17:12:22] FIRING: [49x] CertAlmostExpired: gNMI TLS certificate for cloudsw1-b1-codfw.mgmt.codfw.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [17:13:27] FIRING: [2x] ProbeDown: Ripe Atlas anchor ripe-atlas-codfw:80 is not returning HTTP 200 OK on port 80 - https://wikitech.wikimedia.org/wiki/RIPE_Atlas#HTTP_checks_failing - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [17:14:31] !log sukhe@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp2059.codfw.wmnet with reason: host reimage [17:16:10] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply [17:17:13] FIRING: [2x] JobUnavailable: Reduced availability for job pdu_sentry4 in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [17:17:49] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply [17:17:49] RESOLVED: HelmReleaseBadStatus: Helm release wdqs-next/scholarly-external on k8s-dse@codfw in state pending-upgrade - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus [17:18:27] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply [17:21:03] PROBLEM - Host doh2002 is DOWN: PING CRITICAL - Packet loss = 100% [17:21:03] PROBLEM - Host dragonfly-supernode2001 is DOWN: PING CRITICAL - Packet loss = 100% [17:21:11] ineresting [17:21:17] PROBLEM - Host dse-k8s-ctrl2001 is DOWN: PING CRITICAL - Packet loss = 100% [17:21:17] with a t [17:21:22] PROBLEM - Host dse-k8s-etcd2001 is DOWN: PING CRITICAL - Packet loss = 100% [17:21:22] PROBLEM - Host dse-k8s-etcd2002 is DOWN: PING CRITICAL - Packet loss = 100% [17:21:22] PROBLEM - Host durum2001 is DOWN: PING CRITICAL - Packet loss = 100% [17:21:32] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply [17:21:43] (03CR) 10BCornwall: [C:03+1] traffserver/cache: add attribution.wikimedia.org map and default caching [puppet] - 10https://gerrit.wikimedia.org/r/1341338 (https://phabricator.wikimedia.org/T437635) (owner: 10Dzahn) [17:21:43] PROBLEM - Host cloudcumin2001 is DOWN: PING CRITICAL - Packet loss = 100% [17:21:43] PROBLEM - Host cloudsw1-b1-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:21:43] PROBLEM - Host cloudsw1-b1-codfw IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [17:21:45] PROBLEM - Host kubestagemaster2005 is DOWN: PING CRITICAL - Packet loss = 100% [17:21:51] PROBLEM - Host ldap-replica2005 is DOWN: PING CRITICAL - Packet loss = 100% [17:21:55] PROBLEM - Host lsw1-b5-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:21:55] PROBLEM - Host lsw1-b3-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:21:55] PROBLEM - Host lsw1-c3-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:21:55] PROBLEM - Host lsw1-c1-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:21:55] PROBLEM - Host lsw1-c5-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:21:56] PROBLEM - Host build2004 is DOWN: PING CRITICAL - Packet loss = 100% [17:21:56] PROBLEM - Host build2002 is DOWN: PING CRITICAL - Packet loss = 100% [17:21:56] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply [17:21:57] PROBLEM - Host aux-k8s-etcd2005 is DOWN: PING CRITICAL - Packet loss = 100% [17:21:57] PROBLEM - Host aux-k8s-worker2003 is DOWN: PING CRITICAL - Packet loss = 100% [17:21:58] PROBLEM - Host registry2004 is DOWN: PING CRITICAL - Packet loss = 100% [17:21:59] PROBLEM - Host ceph-admin2001 is DOWN: PING CRITICAL - Packet loss = 100% [17:21:59] PROBLEM - Host install2005 is DOWN: PING CRITICAL - Packet loss = 100% [17:21:59] PROBLEM - Host doc2003 is DOWN: PING CRITICAL - Packet loss = 100% [17:22:00] PROBLEM - Host wikikube-worker-exp2001 is DOWN: PING CRITICAL - Packet loss = 100% [17:22:03] PROBLEM - Host idp2005 is DOWN: PING CRITICAL - Packet loss = 100% [17:22:03] PROBLEM - Host ldap-rw2001 is DOWN: PING CRITICAL - Packet loss = 100% [17:22:05] PROBLEM - Host ml-staging-etcd2002 is DOWN: PING CRITICAL - Packet loss = 100% [17:22:05] PROBLEM - Host ml-etcd2002 is DOWN: PING CRITICAL - Packet loss = 100% [17:22:05] PROBLEM - Host ml-staging-etcd2003 is DOWN: PING CRITICAL - Packet loss = 100% [17:22:07] PROBLEM - Host poolcounter2006 is DOWN: PING CRITICAL - Packet loss = 100% [17:22:07] PROBLEM - Host logstash2024 is DOWN: PING CRITICAL - Packet loss = 100% [17:22:07] PROBLEM - Host logstash2032 is DOWN: PING CRITICAL - Packet loss = 100% [17:22:07] PROBLEM - Host lsw1-b2-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:22:07] PROBLEM - Host lsw1-b4-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:22:08] PROBLEM - Host lsw1-c2-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:22:09] PROBLEM - Host netflow2003 is DOWN: PING CRITICAL - Packet loss = 100% [17:22:09] PROBLEM - Host rdb-lock2001 is DOWN: PING CRITICAL - Packet loss = 100% [17:22:09] PROBLEM - Host lsw1-b6-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:22:10] PROBLEM - Host serpens is DOWN: PING CRITICAL - Packet loss = 100% [17:22:10] PROBLEM - Host tcp-proxy2001 is DOWN: PING CRITICAL - Packet loss = 100% [17:22:11] PROBLEM - Host schema2003 is DOWN: PING CRITICAL - Packet loss = 100% [17:22:11] PROBLEM - Host lsw1-c4-codfw is DOWN: PING CRITICAL - Packet loss = 100% [17:22:12] PROBLEM - Host zuul2003 is DOWN: PING CRITICAL - Packet loss = 100% [17:22:21] PROBLEM - Host ml-serve-ctrl2001 is DOWN: PING CRITICAL - Packet loss = 100% [17:22:25] PROBLEM - Host people2004 is DOWN: PING CRITICAL - Packet loss = 100% [17:22:25] PROBLEM - Host mx-in2001 is DOWN: PING CRITICAL - Packet loss = 100% [17:22:25] PROBLEM - Host urldownloader2005 is DOWN: PING CRITICAL - Packet loss = 100% [17:22:47] PROBLEM - Etcd cluster health on ml-staging-etcd2001 is CRITICAL: The etcd server is unhealthy https://wikitech.wikimedia.org/wiki/Etcd [17:22:57] FIRING: [2x] ProbeDown: Service text-https:443 has failed probes (http_text-https_ip4) #page - https://wikitech.wikimedia.org/wiki/Runbook#text-https:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [17:22:59] PROBLEM - ElasticSearch health check for shards on 9443 on search.svc.codfw.wmnet is CRITICAL: CRITICAL - elasticsearch inactive shards 1718 threshold =0.2 breach: cluster_name: production-search-omega-codfw, status: yellow, timed_out: False, number_of_nodes: 15, number_of_data_nodes: 15, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1732, active_shards: 3477, relocating_shards: 0, initializing_shards: [17:22:59] signed_shards: 1683, delayed_unassigned_shards: 0, number_of_pending_tasks: 14, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 890, active_shards_percent_as_number: 66.92974013474495 https://wikitech.wikimedia.org/wiki/Search%23Administration [17:23:05] PROBLEM - Etcd cluster health on conf2006 is CRITICAL: The etcd server is unhealthy https://wikitech.wikimedia.org/wiki/Etcd [17:23:15] PROBLEM - Etcd cluster health on dse-k8s-etcd2003 is CRITICAL: The etcd server is unhealthy https://wikitech.wikimedia.org/wiki/Etcd [17:23:16] !log sukhe@cumin1004 START - Cookbook sre.dns.admin DNS admin: depool codfw [reason: no reason specified, no task ID specified] [17:23:19] PROBLEM - Host lsw1-c3-codfw IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [17:23:22] !log sukhe@cumin1004 END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool codfw [reason: no reason specified, no task ID specified] [17:23:25] !log cdanis@cumin1004 START - Cookbook sre.dns.admin DNS admin: depool codfw [reason: no reason specified, no task ID specified] [17:23:27] RESOLVED: [2x] ProbeDown: Ripe Atlas anchor ripe-atlas-codfw:80 is not returning HTTP 200 OK on port 80 - https://wikitech.wikimedia.org/wiki/RIPE_Atlas#HTTP_checks_failing - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [17:23:27] PROBLEM - MariaDB Replica IO: pc2 on pc1022 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl2024@pc2022.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on pc2022.codfw.wmnet (110 Connection timed out) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:23:27] PROBLEM - MariaDB Replica IO: pc5 on pc1015 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl2024@pc2015.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on pc2015.codfw.wmnet (110 Connection timed out) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:23:28] !log cdanis@cumin1004 END (FAIL) - Cookbook sre.dns.admin (exit_code=99) DNS admin: depool codfw [reason: no reason specified, no task ID specified] [17:23:30] PROBLEM - MariaDB Replica IO: s2 #page on db2175 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl2024@db2204.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on db2204.codfw.wmnet (110 Connection timed out) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:23:30] PROBLEM - MariaDB Replica IO: s2 on db2197 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl2024@db2204.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on db2204.codfw.wmnet (110 Connection timed out) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:23:30] PROBLEM - MariaDB Replica IO: x1 on db2197 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl2024@db2191.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on db2191.codfw.wmnet (110 Connection timed out) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:23:30] PROBLEM - MariaDB Replica IO: backup1-codfw on db2183 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl2024@db2184.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on db2184.codfw.wmnet (110 Connection timed out) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:23:30] PROBLEM - MariaDB Replica IO: s3 on db2239 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl2024@db2205.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on db2205.codfw.wmnet (110 Connection timed out) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:23:31] PROBLEM - MariaDB Replica IO: s2 #page on db2189 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl2024@db2204.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on db2204.codfw.wmnet (110 Connection timed out) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:23:32] PROBLEM - MariaDB Replica IO: s8 #page on db2195 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl2024@db2161.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on db2161.codfw.wmnet (110 Connection timed out) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:23:33] PROBLEM - MariaDB Replica IO: s7 #page on db2220 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl2024@db2159.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on db2159.codfw.wmnet (110 Connection timed out) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:23:34] PROBLEM - MariaDB Replica IO: x1 #page on db2215 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl2024@db2191.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on db2191.codfw.wmnet (110 Connection timed out) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:23:34] PROBLEM - PyBal backends health check on lvs2014 is CRITICAL: PYBAL CRITICAL - CRITICAL - textlb6_80: Servers cp2046.codfw.wmnet are marked down but pooled: restbase-backend_7233: Servers restbase2028.codfw.wmnet are marked down but pooled: sophroid_4252: Servers aux-k8s-worker2003.codfw.wmnet, aux-k8s-worker2005.codfw.wmnet, aux-k8s-worker2007.codfw.wmnet, aux-k8s-worker2004.codfw.wmnet are marked down but pooled: textlb_80: Servers cp20 [17:23:34] .wmnet are marked down but pooled: restbase-https_7443: Servers restbase2025.codfw.wmnet are marked down but pooled: textlb_443: Servers cp2047.codfw.wmnet are marked down but pooled: textlb6_443: Servers cp2045.codfw.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [17:23:34] PROBLEM - Host lsw1-c1-codfw IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [17:23:34] PROBLEM - Host lsw1-c5-codfw IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [17:23:35] PROBLEM - Host lsw1-c4-codfw IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [17:23:35] PROBLEM - Host lsw1-c2-codfw IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [17:23:45] PROBLEM - MariaDB Replica IO: s8 #page on db2163 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl2024@db2161.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on db2161.codfw.wmnet (113 No route to host) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:23:46] PROBLEM - MariaDB Replica IO: s1 #page on db2153 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl2024@db2203.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on db2203.codfw.wmnet (110 Connection timed out) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:23:46] PROBLEM - MariaDB Replica IO: s8 #page on db2167 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl2024@db2161.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on db2161.codfw.wmnet (110 Connection timed out) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:23:47] PROBLEM - MariaDB Replica IO: s8 #page on db2154 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl2024@db2161.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on db2161.codfw.wmnet (110 Connection timed out) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:23:48] PROBLEM - MariaDB Replica IO: s1 #page on db2170 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl2024@db2203.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on db2203.codfw.wmnet (110 Connection timed out) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:23:49] PROBLEM - MariaDB Replica IO: s1 #page on db2173 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl2024@db2203.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on db2203.codfw.wmnet (110 Connection timed out) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:23:50] PROBLEM - MariaDB Replica IO: s1 #page on db2176 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl2024@db2203.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on db2203.codfw.wmnet (110 Connection timed out) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:23:51] PROBLEM - MariaDB Replica IO: s3 #page on db2156 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl2024@db2205.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on db2205.codfw.wmnet (110 Connection timed out) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:23:51] PROBLEM - MariaDB Replica IO: s8 on db2198 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl2024@db2161.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on db2161.codfw.wmnet (110 Connection timed out) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:23:52] PROBLEM - MariaDB Replica IO: s8 #page on db2164 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl2024@db2161.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on db2161.codfw.wmnet (113 No route to host) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:23:52] PROBLEM - MariaDB Replica IO: s7 on db2198 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl2024@db2159.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on db2159.codfw.wmnet (110 Connection timed out) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:23:53] PROBLEM - MariaDB Replica IO: s7 #page on db2168 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl2024@db2159.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on db2159.codfw.wmnet (110 Connection timed out) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:23:53] PROBLEM - MariaDB Replica IO: s1 #page on db2174 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl2024@db2203.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on db2203.codfw.wmnet (110 Connection timed out) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:23:53] PROBLEM - MariaDB Replica IO: x1 on db2201 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl2024@db2191.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on db2191.codfw.wmnet (110 Connection timed out) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:23:54] PROBLEM - MariaDB Replica IO: s8 #page on db2181 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl2024@db2161.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on db2161.codfw.wmnet (110 Connection timed out) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:23:55] PROBLEM - MariaDB Replica IO: s7 #page on db2218 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl2024@db2159.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on db2159.codfw.wmnet (110 Connection timed out) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:23:56] PROBLEM - MariaDB Replica IO: s3 #page on db2194 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl2024@db2205.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on db2205.codfw.wmnet (110 Connection timed out) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:23:57] PROBLEM - MariaDB Replica IO: x1 #page on db2196 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl2024@db2191.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on db2191.codfw.wmnet (110 Connection timed out) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:23:57] PROBLEM - MariaDB Replica IO: s7 on db2200 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl2024@db2159.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on db2159.codfw.wmnet (110 Connection timed out) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:24:00] PROBLEM - MariaDB Replica IO: s2 #page on db2225 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl2024@db2204.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on db2204.codfw.wmnet (110 Connection timed out) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:24:02] PROBLEM - MariaDB Replica IO: s7 #page on db2222 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl2024@db2159.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on db2159.codfw.wmnet (110 Connection timed out) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:24:06] PROBLEM - MariaDB Replica IO: s7 #page on db2182 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl2024@db2159.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on db2159.codfw.wmnet (110 Connection timed out) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:24:06] PROBLEM - MariaDB Replica IO: s7 #page on db2221 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl2024@db2159.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on db2159.codfw.wmnet (110 Connection timed out) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:24:06] PROBLEM - PyBal backends health check on lvs2011 is CRITICAL: PYBAL CRITICAL - CRITICAL - textlb6_80: Servers cp2047.codfw.wmnet are marked down but pooled: textlb_443: Servers cp2047.codfw.wmnet are marked down but pooled: textlb6_443: Servers cp2047.codfw.wmnet are marked down but pooled: textlb_80: Servers cp2045.codfw.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [17:24:06] PROBLEM - OpenSearch health check for shards on 9400 on cirrussearch2106 is CRITICAL: CRITICAL - elasticsearch inactive shards 1567 threshold =0.15 breach: cluster_name: production-search-omega-codfw, status: yellow, timed_out: False, number_of_nodes: 15, number_of_data_nodes: 15, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1732, active_shards: 3628, relocating_shards: 0, initializing_shards: 18, unas [17:24:06] hards: 1549, delayed_unassigned_shards: 0, number_of_pending_tasks: 4, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 370, active_shards_percent_as_number: 69.83638113570741 https://wikitech.wikimedia.org/wiki/Search%23Administration [17:24:26] !ack [17:24:26] 8358 (ACKED) [2x] ProbeDown sre (text-https:443 probes/service magru) [17:24:27] 8359 (ACKED) db2175 (paged)/MariaDB Replica IO: s2 (paged) [17:24:27] 8360 (ACKED) db2189 (paged)/MariaDB Replica IO: s2 (paged) [17:24:27] 8361 (ACKED) db2195 (paged)/MariaDB Replica IO: s8 (paged) [17:24:27] 8362 (ACKED) db2220 (paged)/MariaDB Replica IO: s7 (paged) [17:24:27] !ack [17:24:28] 8363 (ACKED) db2215 (paged)/MariaDB Replica IO: x1 (paged) [17:24:28] 8364 (ACKED) db2163 (paged)/MariaDB Replica IO: s8 (paged) [17:24:28] 8365 (ACKED) db2153 (paged)/MariaDB Replica IO: s1 (paged) [17:24:29] 8366 (ACKED) db2167 (paged)/MariaDB Replica IO: s8 (paged) [17:24:29] 8367 (ACKED) db2154 (paged)/MariaDB Replica IO: s8 (paged) [17:24:30] 8368 (ACKED) db2170 (paged)/MariaDB Replica IO: s1 (paged) [17:24:30] 8369 (ACKED) db2173 (paged)/MariaDB Replica IO: s1 (paged) [17:24:31] 8370 (ACKED) db2176 (paged)/MariaDB Replica IO: s1 (paged) [17:24:31] 8371 (ACKED) db2156 (paged)/MariaDB Replica IO: s3 (paged) [17:24:32] 8372 (ACKED) db2164 (paged)/MariaDB Replica IO: s8 (paged) [17:24:32] 8373 (ACKED) db2168 (paged)/MariaDB Replica IO: s7 (paged) [17:24:33] 8374 (ACKED) db2174 (paged)/MariaDB Replica IO: s1 (paged) [17:24:33] 8375 (ACKED) db2181 (paged)/MariaDB Replica IO: s8 (paged) [17:24:33] sukhe@cumin1004: Failed to log message to wiki. Somebody should check the error logs. [17:24:34] 8376 (ACKED) db2218 (paged)/MariaDB Replica IO: s7 (paged) [17:24:34] 8377 (ACKED) db2194 (paged)/MariaDB Replica IO: s3 (paged) [17:24:35] 8378 (ACKED) db2196 (paged)/MariaDB Replica IO: x1 (paged) [17:24:35] 8379 (ACKED) db2225 (paged)/MariaDB Replica IO: s2 (paged) [17:24:36] 8380 (ACKED) db2222 (paged)/MariaDB Replica IO: s7 (paged) [17:24:36] 8381 (ACKED) db2182 (paged)/MariaDB Replica IO: s7 (paged) [17:24:37] All incidents are already acked. [17:24:37] 8382 (ACKED) db2221 (paged)/MariaDB Replica IO: s7 (paged) [17:24:38] 8383 (ACKED) db2216 (paged)/MariaDB Replica IO: s1 (paged) [17:24:38] 8384 (ACKED) db2212 (paged)/MariaDB Replica IO: s1 (paged) [17:24:39] 8385 (ACKED) [2x] PHPFPMTooBusy sre (main) [17:24:39] 8386 (ACKED) es2047 (paged)/MariaDB Replica IO: es6 (paged) [17:24:40] 8387 (ACKED) db2238 (paged)/MariaDB Replica IO: s2 (paged) [17:24:40] 8388 (ACKED) db2249 (paged)/MariaDB Replica IO: x1 (paged) [17:25:00] RECOVERY - OpenSearch health check for shards on 9600 on cirrussearch2108 is OK: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 18, number_of_data_nodes: 18, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 4673, relocating_shards: 0, initializing_shards: 32, unassigned_shards: 499, de [17:25:00] assigned_shards: 0, number_of_pending_tasks: 16, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 7943, active_shards_percent_as_number: 89.79631053036125 https://wikitech.wikimedia.org/wiki/Search%23Administration [17:25:01] PROBLEM - OpenSearch health check for shards on 9200 on cirrussearch2114 is CRITICAL: CRITICAL - elasticsearch inactive shards 1426 threshold =0.15 breach: cluster_name: production-search-codfw, status: red, timed_out: False, number_of_nodes: 33, number_of_data_nodes: 33, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1252, active_shards: 2348, relocating_shards: 0, initializing_shards: 159, unassigned_s [17:25:01] 267, delayed_unassigned_shards: 0, number_of_pending_tasks: 9, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 332, active_shards_percent_as_number: 62.21515633280339 https://wikitech.wikimedia.org/wiki/Search%23Administration [17:25:01] RECOVERY - OpenSearch health check for shards on 9600 on cirrussearch2085 is OK: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 18, number_of_data_nodes: 18, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 4674, relocating_shards: 0, initializing_shards: 32, unassigned_shards: 498, de [17:25:01] assigned_shards: 0, number_of_pending_tasks: 22, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 8709, active_shards_percent_as_number: 89.81552651806302 https://wikitech.wikimedia.org/wiki/Search%23Administration [17:25:01] RECOVERY - OpenSearch health check for shards on 9600 on cirrussearch2076 is OK: OK - elasticsearch status production-search-psi-codfw: cluster_name: production-search-psi-codfw, status: yellow, timed_out: False, number_of_nodes: 18, number_of_data_nodes: 18, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1739, active_shards: 4674, relocating_shards: 0, initializing_shards: 32, unassigned_shards: 498, de [17:25:34] FIRING: [2x] PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-api-ext releases routed via main at eqiad: 0% idle #page - https://bit.ly/wmf-fpmsat - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [17:25:38] (03PS1) 10Ahmon Dancy: scap-master-sync: Sync releases only if the target directory exists [puppet] - 10https://gerrit.wikimedia.org/r/1344336 (https://phabricator.wikimedia.org/T435393) [17:25:39] FIRING: PoolcounterFullQueues: Full queues for poolcounter1006:9106 poolcounter - https://www.mediawiki.org/wiki/PoolCounter#Request_tracing_in_production - https://grafana.wikimedia.org/d/aIcYxuxZk/poolcounter?orgId=1&viewPanel=6&from=now-1h&to=now&var-dc=eqiad%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DPoolcounterFullQueues [17:26:26] !log cdobbins@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir6001.drmrs.wmnet with OS trixie [17:27:21] RESOLVED: [4x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:fxp0 () - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [17:27:49] FIRING: VarnishUnavailable: varnish-text has reduced HTTP availability #page - https://wikitech.wikimedia.org/wiki/Varnish#Diagnosing_Varnish_alerts - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DVarnishUnavailable [17:27:53] 10SRE-swift-storage, 10Ceph, 06Infrastructure-Foundations, 06Release-Engineering-Team: Test the Docker Registry's GC with Releng/MediaWiki images - https://phabricator.wikimedia.org/T437297#12356575 (10elukey) I ran the GC in dry-run mode and it completed in ~67 minutes, that is a nicer result than the fir... [17:27:53] FIRING: HaproxyUnavailable: HAProxy (cache_text) has reduced HTTP availability #page - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyUnavailable [17:27:57] !incidents [17:27:58] 8358 (ACKED) [2x] ProbeDown sre (text-https:443 probes/service magru) [17:27:58] 8359 (ACKED) db2175 (paged)/MariaDB Replica IO: s2 (paged) [17:27:58] 8360 (ACKED) db2189 (paged)/MariaDB Replica IO: s2 (paged) [17:27:58] 8361 (ACKED) db2195 (paged)/MariaDB Replica IO: s8 (paged) [17:27:59] FIRING: [3x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?panelId=18&fullscreen&orgId=1&var-datasource=eqiad%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [17:27:59] 8362 (ACKED) db2220 (paged)/MariaDB Replica IO: s7 (paged) [17:27:59] 8363 (ACKED) db2215 (paged)/MariaDB Replica IO: x1 (paged) [17:27:59] 8364 (ACKED) db2163 (paged)/MariaDB Replica IO: s8 (paged) [17:28:00] 8365 (ACKED) db2153 (paged)/MariaDB Replica IO: s1 (paged) [17:28:00] 8366 (ACKED) db2167 (paged)/MariaDB Replica IO: s8 (paged) [17:28:01] 8367 (ACKED) db2154 (paged)/MariaDB Replica IO: s8 (paged) [17:28:01] 8368 (ACKED) db2170 (paged)/MariaDB Replica IO: s1 (paged) [17:28:02] 8369 (ACKED) db2173 (paged)/MariaDB Replica IO: s1 (paged) [17:28:02] 8370 (ACKED) db2176 (paged)/MariaDB Replica IO: s1 (paged) [17:28:03] 8371 (ACKED) db2156 (paged)/MariaDB Replica IO: s3 (paged) [17:28:03] 8372 (ACKED) db2164 (paged)/MariaDB Replica IO: s8 (paged) [17:28:04] 8373 (ACKED) db2168 (paged)/MariaDB Replica IO: s7 (paged) [17:28:04] 8374 (ACKED) db2174 (paged)/MariaDB Replica IO: s1 (paged) [17:28:05] 8375 (ACKED) db2181 (paged)/MariaDB Replica IO: s8 (paged) [17:28:05] 8376 (ACKED) db2218 (paged)/MariaDB Replica IO: s7 (paged) [17:28:06] 8377 (ACKED) db2194 (paged)/MariaDB Replica IO: s3 (paged) [17:28:06] 8378 (ACKED) db2196 (paged)/MariaDB Replica IO: x1 (paged) [17:28:07] 8379 (ACKED) db2225 (paged)/MariaDB Replica IO: s2 (paged) [17:28:07] 8380 (ACKED) db2222 (paged)/MariaDB Replica IO: s7 (paged) [17:28:08] RESOLVED: [3x] NetworkDeviceAlarmActive: Alarm active on cr1-codfw - https://wikitech.wikimedia.org/wiki/Network_monitoring#Juniper_alarm - https://alerts.wikimedia.org/?q=alertname%3DNetworkDeviceAlarmActive [17:28:08] 8381 (ACKED) db2182 (paged)/MariaDB Replica IO: s7 (paged) [17:28:08] 8382 (ACKED) db2221 (paged)/MariaDB Replica IO: s7 (paged) [17:28:09] 8383 (ACKED) db2216 (paged)/MariaDB Replica IO: s1 (paged) [17:28:09] 8384 (ACKED) db2212 (paged)/MariaDB Replica IO: s1 (paged) [17:28:10] 8385 (ACKED) [2x] PHPFPMTooBusy sre (main) [17:28:10] 8386 (ACKED) es2047 (paged)/MariaDB Replica IO: es6 (paged) [17:28:11] 8387 (ACKED) db2238 (paged)/MariaDB Replica IO: s2 (paged) [17:28:11] 8388 (ACKED) db2249 (paged)/MariaDB Replica IO: x1 (paged) [17:28:12] 8389 (UNACKED) VarnishUnavailable global sre (varnish-text thanos-rule@main) [17:28:12] 8390 (UNACKED) HaproxyUnavailable cache_text global sre (thanos-rule@main) [17:28:13] !ack [17:28:13] 8389 (ACKED) VarnishUnavailable global sre (varnish-text thanos-rule@main) [17:28:14] 8390 (ACKED) HaproxyUnavailable cache_text global sre (thanos-rule@main) [17:28:33] !log rzl@cumin1004 START - Cookbook sre.discovery.datacenter pool all active/active services in eqiad: maintenance - T439010 [17:28:34] PROBLEM - Check if active EventStreams endpoint is delivering messages. on alert1002 is CRITICAL: CRITICAL: No EventStreams message was consumed from https://stream.wikimedia.org/v2/stream/recentchange within 10 seconds. https://wikitech.wikimedia.org/wiki/Event_Platform/EventStreams/Administration [17:28:40] <_joe_> !ack [17:28:40] All incidents are already acked. [17:28:40] FIRING: [2x] MediaWikiLatencyExceeded: p75 latency high: eqiad mw-api-ext releases routed via main (k8s) 2.5s - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [17:28:45] FIRING: [4x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:28:58] FIRING: [3x] CalicoKubeControllersDown: Calico Kubernetes Controllers not running - https://wikitech.wikimedia.org/wiki/Calico#Kube_Controllers - TODO - https://alerts.wikimedia.org/?q=alertname%3DCalicoKubeControllersDown [17:29:03] RESOLVED: RedisInstanceDown: Redis instance down arclamp2001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_arclamp&var-instance=arclamp2001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [17:29:11] FIRING: [2x] CalicoTyphaDown: Too few (1) calico-typha replicas running - https://wikitech.wikimedia.org/wiki/Calico#Typha" - TODO - https://alerts.wikimedia.org/?q=alertname%3DCalicoTyphaDown [17:29:17] FIRING: CirrusProducerFlinkJobNotRunning: cirrus_streaming_updater_producer in codfw (k8s) is not running - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?var-datasource=codfw+prometheus%2Fk8s&var-namespace=cirrus-streaming-updater&var-helm_release=producer - https://alerts.wikimedia.org/?q=alertname%3DCirrusProducerFlinkJobNotRunning [17:29:28] FIRING: SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://wikifeeds.svc.eqiad.wmnet:4101 - https://grafana.wikimedia.org/d/_77ik484k/openapi-swagger-endpoint-state?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [17:29:34] RESOLVED: [2x] JobUnavailable: Reduced availability for job pdu_sentry4 in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [17:29:45] RESOLVED: [49x] CertAlmostExpired: gNMI TLS certificate for cloudsw1-b1-codfw.mgmt.codfw.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [17:30:01] FIRING: [11x] ProbeDown: Service mw-api-int:4446 has failed probes (http_mw-api-int_ip4) #page - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [17:30:06] FIRING: [8x] ProbeDown: Service mw-api-ext:4447 has failed probes (http_mw-api-ext_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [17:30:10] FIRING: [119x] KubernetesCalicoDown: aux-k8s-ctrl2002.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [17:30:12] FIRING: CirrusSearchUpdaterKafkaMessagesInTooLow: ... [17:30:16] The summed message update rate of `(eqiad|codfw).cirrussearch.update_pipeline.update.v1` is too low - https://wikitech.wikimedia.org/wiki/Search/OpenSearch/Administration#Alerts/Dashboards - ... [17:30:22] https://grafana.wikimedia.org/d/000000234/kafka-by-topic?orgId=1&var-dc=eqiad%2520prometheus%252Fops&var-kafka_cluster=main-eqiad&var-kafka_broker=All&from=now-1h&to=now&refresh=5m&var-topic=codfw.cirrussearch.update_pipeline.update.v1&var-topic=eqiad.cirrussearch.update_pipeline.update.v1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchUpdaterKafkaMessagesInTooLow [17:30:30] (03PS1) 10Majavah: idp: Failover to eqiad [dns] - 10https://gerrit.wikimedia.org/r/1344337 [17:30:35] PROBLEM - MariaDB Replica Lag: s8 #page on db2154 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 614.61 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:30:36] PROBLEM - MariaDB Replica Lag: s1 #page on db2176 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 614.62 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:30:37] PROBLEM - MariaDB Replica Lag: s1 #page on db2174 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 614.63 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:30:38] PROBLEM - MariaDB Replica Lag: s1 #page on db2212 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 614.70 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:30:38] PROBLEM - MariaDB Replica Lag: backup1-codfw on db2183 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 614.72 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:30:38] PROBLEM - MariaDB Replica Lag: s7 #page on db2218 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 614.81 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:30:39] PROBLEM - MariaDB Replica Lag: s7 #page on db2220 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 614.90 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:30:40] PROBLEM - MariaDB Replica Lag: s2 #page on db2238 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 614.90 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:30:42] !log sukhe@puppetserver1001 conftool action : set/pooled=no; selector: cluster=dnsbox,dc=codfw [17:30:44] PROBLEM - MariaDB Replica Lag: pc5 on pc1015 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 624.47 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:30:49] PROBLEM - MariaDB Replica Lag: s8 #page on db2164 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 628.24 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:30:50] PROBLEM - MariaDB Replica Lag: s1 #page on db2170 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 628.30 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:30:51] PROBLEM - MariaDB Replica Lag: s8 #page on db2163 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 628.29 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:30:52] PROBLEM - MariaDB Replica Lag: s3 #page on db2156 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 628.31 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:30:53] PROBLEM - MariaDB Replica Lag: s1 #page on db2153 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 628.34 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:30:53] PROBLEM - MariaDB Replica Lag: s8 #page on db2181 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 628.35 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:30:54] PROBLEM - MariaDB Replica Lag: s7 #page on db2168 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 628.36 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:30:55] PROBLEM - MariaDB Replica Lag: s1 #page on db2173 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 628.40 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:30:56] PROBLEM - MariaDB Replica Lag: s8 #page on db2167 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 628.40 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:30:57] PROBLEM - MariaDB Replica Lag: s2 #page on db2175 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 628.40 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:31:08] PROBLEM - MariaDB Replica Lag: s7 #page on db2182 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 647.96 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:31:09] PROBLEM - MariaDB Replica Lag: x1 #page on db2196 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 648.00 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:31:10] PROBLEM - MariaDB Replica Lag: s2 #page on db2189 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 648.02 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:31:10] PROBLEM - MariaDB Replica Lag: s3 #page on db2194 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 648.06 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:31:11] PROBLEM - MariaDB Replica Lag: s8 #page on db2195 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 648.10 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:31:15] PROBLEM - MariaDB Replica Lag: ms1 on db1267 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 655.47 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:31:16] !ack [17:31:17] 8391 (ACKED) db2154 (paged)/MariaDB Replica Lag: s8 (paged) [17:31:17] 8392 (ACKED) db2176 (paged)/MariaDB Replica Lag: s1 (paged) [17:31:17] 8393 (ACKED) db2174 (paged)/MariaDB Replica Lag: s1 (paged) [17:31:17] 8394 (ACKED) db2218 (paged)/MariaDB Replica Lag: s7 (paged) [17:31:17] FIRING: [2x] PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-api-ext releases routed via main at codfw: 16.07% idle - https://bit.ly/wmf-fpmsat - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [17:31:18] 8395 (ACKED) db2212 (paged)/MariaDB Replica Lag: s1 (paged) [17:31:18] 8396 (ACKED) db2220 (paged)/MariaDB Replica Lag: s7 (paged) [17:31:18] PROBLEM - MariaDB Replica Lag: x1 #page on db2215 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 658.25 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:31:18] 8397 (ACKED) db2238 (paged)/MariaDB Replica Lag: s2 (paged) [17:31:19] 8398 (ACKED) db2164 (paged)/MariaDB Replica Lag: s8 (paged) [17:31:19] PROBLEM - MariaDB Replica Lag: s7 #page on db2221 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 658.28 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:31:19] 8399 (ACKED) db2170 (paged)/MariaDB Replica Lag: s1 (paged) [17:31:20] PROBLEM - MariaDB Replica Lag: s1 #page on db2216 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 658.32 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:31:20] 8400 (ACKED) db2163 (paged)/MariaDB Replica Lag: s8 (paged) [17:31:20] 8401 (ACKED) db2153 (paged)/MariaDB Replica Lag: s1 (paged) [17:31:21] 8402 (ACKED) db2156 (paged)/MariaDB Replica Lag: s3 (paged) [17:31:21] 8403 (ACKED) db2181 (paged)/MariaDB Replica Lag: s8 (paged) [17:31:22] FIRING: [4x] PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-api-ext releases routed via main at eqiad: 0% idle #page - https://bit.ly/wmf-fpmsat - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [17:31:22] 8404 (ACKED) db2168 (paged)/MariaDB Replica Lag: s7 (paged) [17:31:22] 8405 (ACKED) db2173 (paged)/MariaDB Replica Lag: s1 (paged) [17:31:23] 8406 (ACKED) db2167 (paged)/MariaDB Replica Lag: s8 (paged) [17:31:23] 8407 (ACKED) db2175 (paged)/MariaDB Replica Lag: s2 (paged) [17:31:24] 8408 (ACKED) db2182 (paged)/MariaDB Replica Lag: s7 (paged) [17:31:24] 8409 (ACKED) db2196 (paged)/MariaDB Replica Lag: x1 (paged) [17:31:24] PROBLEM - MariaDB Replica Lag: pc2 on pc1022 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 664.68 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:31:25] 8410 (ACKED) db2189 (paged)/MariaDB Replica Lag: s2 (paged) [17:31:25] 8411 (ACKED) db2194 (paged)/MariaDB Replica Lag: s3 (paged) [17:31:25] PROBLEM - MariaDB Replica Lag: s7 #page on db2222 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 665.06 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:31:26] 8412 (ACKED) db2195 (paged)/MariaDB Replica Lag: s8 (paged) [17:31:26] PROBLEM - MariaDB Replica Lag: es6 #page on es2047 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 665.09 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:31:27] PROBLEM - MariaDB Replica Lag: x1 #page on db2249 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 665.35 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:31:27] FIRING: [2x] CirrusSearchFullTextLatencyTooHigh: CirrusSearch full_text 95th percentiles latency is too high (mw@codfw to dnsdisc) - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://grafana.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchFullTextLatencyTooHigh [17:31:28] PROBLEM - MariaDB Replica Lag: s2 #page on db2225 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 665.68 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [17:31:38] RESOLVED: PoolcounterFullQueues: Full queues for poolcounter1006:9106 poolcounter - https://www.mediawiki.org/wiki/PoolCounter#Request_tracing_in_production - https://grafana.wikimedia.org/d/aIcYxuxZk/poolcounter?orgId=1&viewPanel=6&from=now-1h&to=now&var-dc=eqiad%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DPoolcounterFullQueues [17:32:11] !log rzl@cumin1003 conftool action : set/pooled=true; selector: name=eqiad [17:32:43] !ack [17:32:44] 8413 (ACKED) db2215 (paged)/MariaDB Replica Lag: x1 (paged) [17:32:44] 8414 (ACKED) db2221 (paged)/MariaDB Replica Lag: s7 (paged) [17:32:44] 8415 (ACKED) db2216 (paged)/MariaDB Replica Lag: s1 (paged) [17:32:44] 8416 (ACKED) db2222 (paged)/MariaDB Replica Lag: s7 (paged) [17:32:44] 8417 (ACKED) es2047 (paged)/MariaDB Replica Lag: es6 (paged) [17:32:45] 8418 (ACKED) db2249 (paged)/MariaDB Replica Lag: x1 (paged) [17:32:45] 8419 (ACKED) db2225 (paged)/MariaDB Replica Lag: s2 (paged) [17:32:45] 8420 (ACKED) [2x] RESTGatewayBackendErrorsHigh sre (mobileapps_cluster rest-gateway) [17:32:46] FIRING: [4x] MediaWikiLatencyExceeded: p75 latency high: eqiad mw-api-ext releases routed via main (k8s) 2.5s - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [17:32:48] rzl@cumin1004 datacenter (PID 440866) is awaiting input [17:33:03] FIRING: [5x] CalicoKubeControllersDown: Calico Kubernetes Controllers not running - https://wikitech.wikimedia.org/wiki/Calico#Kube_Controllers - TODO - https://alerts.wikimedia.org/?q=alertname%3DCalicoKubeControllersDown [17:33:11] RESOLVED: [2x] CalicoTyphaDown: Too few (1) calico-typha replicas running - https://wikitech.wikimedia.org/wiki/Calico#Typha" - TODO - https://alerts.wikimedia.org/?q=alertname%3DCalicoTyphaDown [17:33:16] FIRING: [2x] CirrusStreamingUpdaterFlinkJobUnstable: cirrus_streaming_updater_consumer_search_codfw in codfw (k8s) is unstable - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterFlinkJobUnstable [17:33:20] !log rzl@cumin1003 conftool action : set/pooled=false; selector: name=codfw [17:33:26] RESOLVED: CirrusProducerFlinkJobNotRunning: cirrus_streaming_updater_producer in codfw (k8s) is not running - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?var-datasource=codfw+prometheus%2Fk8s&var-namespace=cirrus-streaming-updater&var-helm_release=producer - https://alerts.wikimedia.org/?q=alertname%3DCirrusProducerFlinkJobNotRunning [17:33:36] FIRING: [2x] SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://grafana.wikimedia.org/d/_77ik484k/openapi-swagger-endpoint-state?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [17:33:40] !ack [17:33:41] All incidents are already acked. [17:33:52] FIRING: [2x] RESTGatewayBackendErrorsHigh: rest-gateway: high 5xx errors from mobileapps_cluster in codfw #page - https://wikitech.wikimedia.org/wiki/API_Gateway#How_to_debug_it - https://alerts.wikimedia.org/?q=alertname%3DRESTGatewayBackendErrorsHigh [17:33:57] PROBLEM - OpenSearch health check for shards on 9400 on cirrussearch2114 is CRITICAL: CRITICAL - elasticsearch inactive shards 1204 threshold =0.15 breach: cluster_name: production-search-omega-codfw, status: yellow, timed_out: False, number_of_nodes: 15, number_of_data_nodes: 15, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1732, active_shards: 3991, relocating_shards: 0, initializing_shards: 4, unass [17:33:57] ards: 1200, delayed_unassigned_shards: 0, number_of_pending_tasks: 2, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 261, active_shards_percent_as_number: 76.82386910490857 https://wikitech.wikimedia.org/wiki/Search%23Administration [17:34:10] FIRING: WidespreadPuppetFailure: Puppet has failed in magru - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [17:34:11] !log rzl@cumin1004 END (FAIL) - Cookbook sre.discovery.datacenter (exit_code=93) pool all active/active services in eqiad: maintenance - T439010 [17:34:21] FIRING: [12x] ProbeDown: Service mw-api-int:4446 has failed probes (http_mw-api-int_ip4) #page - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [17:34:26] FIRING: [8x] ProbeDown: Service mw-api-ext:4447 has failed probes (http_mw-api-ext_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [17:34:30] FIRING: [119x] KubernetesCalicoDown: aux-k8s-ctrl2002.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [17:34:52] FIRING: CirrusStreamingUpdaterRateTooLow: CirrusSearch update rate from flink-app-consumer-search is critically low - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterRateTooLow [17:35:43] <_joe_> !ack [17:35:44] All incidents are already acked. [17:36:20] FIRING: [4x] MediaWikiLatencyExceeded: p75 latency high: eqiad mw-api-ext releases routed via main (k8s) 2.458s - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [17:36:51] RESOLVED: [5x] SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [17:36:51] FIRING: [2x] ATSBackendErrorsHigh: ATS: elevated 5xx error ratio from restbase.discovery.wmnet in eqiad #page - https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Debugging - https://alerts.wikimedia.org/?q=alertname%3DATSBackendErrorsHigh [17:37:01] FIRING: [2x] RESTGatewayBackendErrorsHigh: rest-gateway: high 5xx errors from mobileapps_cluster in codfw #page - https://wikitech.wikimedia.org/wiki/API_Gateway#How_to_debug_it - https://alerts.wikimedia.org/?q=alertname%3DRESTGatewayBackendErrorsHigh [17:37:02] !log vriley@cumin1004 START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie [17:37:07] <_joe_> !ack [17:37:08] 8421 (ACKED) [2x] ATSBackendErrorsHigh cache_text sre (restbase.discovery.wmnet) [17:37:12] 10ops-eqiad, 06SRE, 06Collaboration-Services, 06DC-Ops: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12356602 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1004 for host zuul1005.eqiad.wmnet with OS trixie [17:37:19] RECOVERY - Host ps1-f1-codfw is UP: PING OK - Packet loss = 0%, RTA = 32.46 ms [17:37:19] RECOVERY - Host ps1-e5-codfw is UP: PING OK - Packet loss = 0%, RTA = 32.92 ms [17:37:21] PROBLEM - ps1-e5-codfw-infeed-load-tower-B-phase-Z on ps1-e5-codfw is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [17:37:21] PROBLEM - ps1-e5-codfw-infeed-load-tower-B-phase-X on ps1-e5-codfw is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [17:37:21] PROBLEM - ps1-f1-codfw-infeed-load-tower-A-phase-Y on ps1-f1-codfw is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [17:37:21] PROBLEM - ps1-e5-codfw-infeed-load-tower-B-phase-Y on ps1-e5-codfw is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [17:37:21] RECOVERY - Host ps1-d8-codfw is UP: PING OK - Packet loss = 0%, RTA = 32.89 ms [17:37:22] RECOVERY - Host ps1-e1-codfw is UP: PING OK - Packet loss = 0%, RTA = 33.37 ms [17:37:22] RECOVERY - Host ps1-e3-codfw is UP: PING OK - Packet loss = 0%, RTA = 32.31 ms [17:37:23] RECOVERY - Host ps1-f5-codfw is UP: PING OK - Packet loss = 0%, RTA = 32.52 ms [17:37:23] RECOVERY - Host ps1-f2-codfw is UP: PING OK - Packet loss = 0%, RTA = 32.93 ms [17:37:24] RECOVERY - Host ps1-e2-codfw is UP: PING OK - Packet loss = 0%, RTA = 33.24 ms [17:37:24] RECOVERY - Host mr1-codfw is UP: PING OK - Packet loss = 0%, RTA = 31.03 ms [17:37:25] RECOVERY - Host ps1-c7-codfw is UP: PING OK - Packet loss = 0%, RTA = 32.43 ms [17:37:25] RECOVERY - Host ps1-c8-codfw is UP: PING OK - Packet loss = 0%, RTA = 33.02 ms [17:37:26] RECOVERY - ps1-e5-codfw-infeed-load-tower-B-phase-Z on ps1-e5-codfw is OK: SNMP OK - ps1-e5-codfw-infeed-load-tower-B-phase-Z 260 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [17:37:26] RECOVERY - ps1-e5-codfw-infeed-load-tower-B-phase-X on ps1-e5-codfw is OK: SNMP OK - ps1-e5-codfw-infeed-load-tower-B-phase-X 199 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [17:37:27] RECOVERY - ps1-f1-codfw-infeed-load-tower-A-phase-Y on ps1-f1-codfw is OK: SNMP OK - ps1-f1-codfw-infeed-load-tower-A-phase-Y 324 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [17:37:27] RECOVERY - ps1-e5-codfw-infeed-load-tower-B-phase-Y on ps1-e5-codfw is OK: SNMP OK - ps1-e5-codfw-infeed-load-tower-B-phase-Y 163 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [17:37:28] PROBLEM - BFD status on ssw1-d1-codfw.mgmt is CRITICAL: Down: 5 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [17:37:28] PROBLEM - BFD status on ssw1-d8-codfw.mgmt is CRITICAL: Down: 5 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [17:37:29] RECOVERY - Host ps1-f3-codfw is UP: PING OK - Packet loss = 0%, RTA = 31.50 ms [17:37:29] RECOVERY - Host ps1-c6-codfw is UP: PING OK - Packet loss = 0%, RTA = 31.90 ms [17:37:30] RECOVERY - Host ps1-d4-codfw is UP: PING OK - Packet loss = 0%, RTA = 32.97 ms [17:37:30] RECOVERY - Host ps1-d5-codfw is UP: PING OK - Packet loss = 0%, RTA = 32.85 ms [17:37:31] RECOVERY - Host ps1-d3-codfw is UP: PING OK - Packet loss = 0%, RTA = 31.14 ms [17:37:33] PROBLEM - Host cloudsw1-b1-codfw.mgmt is DOWN: PING CRITICAL - Packet loss = 100% [17:37:33] PROBLEM - Host lsw1-a2-codfw.mgmt is DOWN: PING CRITICAL - Packet loss = 100% [17:37:33] PROBLEM - Host lsw1-a3-codfw.mgmt is DOWN: PING CRITICAL - Packet loss = 100% [17:37:33] PROBLEM - Host lsw1-a4-codfw.mgmt is DOWN: PING CRITICAL - Packet loss = 100% [17:37:33] PROBLEM - Host lsw1-a5-codfw.mgmt is DOWN: PING CRITICAL - Packet loss = 100% [17:37:45] FIRING: [2x] WidespreadPuppetFailure: Puppet has failed in eqsin - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [17:37:56] !ack [17:37:57] All incidents are already acked. [17:37:57] RESOLVED: [12x] ProbeDown: Service mw-api-int:4446 has failed probes (http_mw-api-int_ip4) #page - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [17:38:21] RECOVERY - Host ps1-e4-codfw is UP: PING OK - Packet loss = 0%, RTA = 31.19 ms [17:38:25] RECOVERY - Host ps1-d6-codfw is UP: PING OK - Packet loss = 0%, RTA = 31.08 ms [17:38:25] RECOVERY - Host ps1-d2-codfw is UP: PING OK - Packet loss = 0%, RTA = 31.01 ms [17:38:25] RECOVERY - Host ps1-d1-codfw is UP: PING OK - Packet loss = 0%, RTA = 30.87 ms [17:38:27] RESOLVED: [14x] ProbeDown: Service mw-api-ext:4447 has failed probes (http_mw-api-ext_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [17:38:38] RESOLVED: CirrusSearchUpdaterKafkaMessagesInTooLow: ... [17:38:38] The summed message update rate of `(eqiad|codfw).cirrussearch.update_pipeline.update.v1` is too low - https://wikitech.wikimedia.org/wiki/Search/OpenSearch/Administration#Alerts/Dashboards - ... [17:38:38] https://grafana.wikimedia.org/d/000000234/kafka-by-topic?orgId=1&var-dc=eqiad%2520prometheus%252Fops&var-kafka_cluster=main-eqiad&var-kafka_broker=All&from=now-1h&to=now&refresh=5m&var-topic=codfw.cirrussearch.update_pipeline.update.v1&var-topic=eqiad.cirrussearch.update_pipeline.update.v1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchUpdaterKafkaMessagesInTooLow [17:39:15] FIRING: [5x] PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-api-ext releases routed via main at codfw: 16.07% idle - https://bit.ly/wmf-fpmsat - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [17:39:15] RESOLVED: [4x] PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-api-ext releases routed via main at eqiad: 2.458% idle #page - https://bit.ly/wmf-fpmsat - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy [17:39:17] RECOVERY - ElasticSearch health check for shards on 9443 on search.svc.codfw.wmnet is OK: OK - elasticsearch status production-search-omega-codfw: cluster_name: production-search-omega-codfw, status: yellow, timed_out: False, number_of_nodes: 15, number_of_data_nodes: 15, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1732, active_shards: 4168, relocating_shards: 0, initializing_shards: 5, unassigned_sha [17:39:17] 2, delayed_unassigned_shards: 0, number_of_pending_tasks: 0, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 80.23099133782483 https://wikitech.wikimedia.org/wiki/Search%23Administration [17:39:18] RESOLVED: JobQueueLowTrafficConsumerWidespreadHighLatency: ... [17:39:23] Processing delay times for low-traffic consumer rules are unusually high - https://wikitech.wikimedia.org/wiki/MediaWiki_JobQueue/Operations#JobQueueLowTrafficConsumerWidespreadHighLatency - https://grafana.wikimedia.org/d/fe130675-0c2d-4991-9dec-f54cf6a9c4d8/jobqueue-low-traffic-jobs?var-site=eqiad - https://alerts.wikimedia.org/?q=alertname%3DJobQueueLowTrafficConsumerWidespreadHighLatency [17:39:29] FIRING: [2x] CirrusSearchFullTextLatencyTooHigh: CirrusSearch full_text 95th percentiles latency is too high (mw@codfw to dnsdisc) - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://grafana.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchFullTextLatencyTooHigh [17:39:44] FIRING: [3x] CirrusStreamingUpdaterRateTooLow: CirrusSearch update rate from flink-app-consumer-cloudelastic is critically low - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterRateTooLow [17:40:12] RESOLVED: VarnishUnavailable: varnish-text has reduced HTTP availability #page - https://wikitech.wikimedia.org/wiki/Varnish#Diagnosing_Varnish_alerts - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=3 - https://alerts.wikimedia.org/?q=alertname%3DVarnishUnavailable [17:40:13] RESOLVED: HaproxyUnavailable: HAProxy (cache_text) has reduced HTTP availability #page - https://wikitech.wikimedia.org/wiki/HAProxy#HAProxy_for_edge_caching - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyUnavailable [17:40:15] RESOLVED: [3x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?panelId=18&fullscreen&orgId=1&var-datasource=eqiad%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [17:40:31] RECOVERY - Host cp2049 is UP: PING OK - Packet loss = 0%, RTA = 32.96 ms [17:40:55] RECOVERY - Host mr1-codfw IPv6 is UP: PING OK - Packet loss = 0%, RTA = 32.34 ms [17:41:04] FIRING: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [17:41:09] PROBLEM - HAProxy HTTPS wikiworkshop.org ECDSA on cp2049 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection refused https://wikitech.wikimedia.org/wiki/HTTPS [17:41:09] PROBLEM - HAProxy HTTPS wikipedia.org ECDSA on cp2049 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection refused https://wikitech.wikimedia.org/wiki/HTTPS [17:41:09] PROBLEM - HAProxy HTTPS wikipedia25.org ECDSA on cp2049 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection refused https://wikitech.wikimedia.org/wiki/HTTPS [17:41:31] PROBLEM - haproxy process on cp2049 is CRITICAL: PROCS CRITICAL: 0 processes with command name haproxy https://wikitech.wikimedia.org/wiki/HAProxy [17:41:46] FIRING: [4x] MediaWikiLatencyExceeded: p75 latency high: eqiad mw-api-ext releases routed via main (k8s) 2.458s - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [17:41:51] FIRING: [11x] ATSBackendErrorsHigh: ATS: elevated 5xx error ratio from eventgate-logging-external.discovery.wmnet in eqiad #page - https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Debugging - https://alerts.wikimedia.org/?q=alertname%3DATSBackendErrorsHigh [17:41:55] (03PS1) 10Elukey: role::docker_registry: set eqiad read-write [puppet] - 10https://gerrit.wikimedia.org/r/1344338 [17:42:37] (03CR) 10Ssingh: [C:03+1] idp: Failover to eqiad [dns] - 10https://gerrit.wikimedia.org/r/1344337 (owner: 10Majavah) [17:42:46] (03CR) 10Giuseppe Lavagetto: [C:03+1] role::docker_registry: set eqiad read-write [puppet] - 10https://gerrit.wikimedia.org/r/1344338 (owner: 10Elukey) [17:42:58] (03CR) 10Majavah: [V:03+2 C:03+2] idp: Failover to eqiad [dns] - 10https://gerrit.wikimedia.org/r/1344337 (owner: 10Majavah) [17:43:23] !log taavi@dns1004 START - running authdns-update [17:43:30] (03CR) 10CDanis: [C:03+1] role::docker_registry: set eqiad read-write [puppet] - 10https://gerrit.wikimedia.org/r/1344338 (owner: 10Elukey) [17:43:37] <_joe_> !incidents [17:43:39] 8359 (ACKED) db2175 (paged)/MariaDB Replica IO: s2 (paged) [17:43:39] 8360 (ACKED) db2189 (paged)/MariaDB Replica IO: s2 (paged) [17:43:39] 8361 (ACKED) db2195 (paged)/MariaDB Replica IO: s8 (paged) [17:43:39] 8362 (ACKED) db2220 (paged)/MariaDB Replica IO: s7 (paged) [17:43:40] 8363 (ACKED) db2215 (paged)/MariaDB Replica IO: x1 (paged) [17:43:40] 8364 (ACKED) db2163 (paged)/MariaDB Replica IO: s8 (paged) [17:43:40] 8365 (ACKED) db2153 (paged)/MariaDB Replica IO: s1 (paged) [17:43:41] 8366 (ACKED) db2167 (paged)/MariaDB Replica IO: s8 (paged) [17:43:41] 8367 (ACKED) db2154 (paged)/MariaDB Replica IO: s8 (paged) [17:43:41] (03CR) 10Elukey: [C:03+2] role::docker_registry: set eqiad read-write [puppet] - 10https://gerrit.wikimedia.org/r/1344338 (owner: 10Elukey) [17:43:42] 8368 (ACKED) db2170 (paged)/MariaDB Replica IO: s1 (paged) [17:43:42] 8369 (ACKED) db2173 (paged)/MariaDB Replica IO: s1 (paged) [17:43:43] 8370 (ACKED) db2176 (paged)/MariaDB Replica IO: s1 (paged) [17:43:43] 8371 (ACKED) db2156 (paged)/MariaDB Replica IO: s3 (paged) [17:43:44] 8372 (ACKED) db2164 (paged)/MariaDB Replica IO: s8 (paged) [17:43:44] 8373 (ACKED) db2168 (paged)/MariaDB Replica IO: s7 (paged) [17:43:45] 8374 (ACKED) db2174 (paged)/MariaDB Replica IO: s1 (paged) [17:43:45] 8375 (ACKED) db2181 (paged)/MariaDB Replica IO: s8 (paged) [17:43:46] 8376 (ACKED) db2218 (paged)/MariaDB Replica IO: s7 (paged) [17:43:46] 8377 (ACKED) db2194 (paged)/MariaDB Replica IO: s3 (paged) [17:43:52] <_joe_> le sigh [17:44:01] <_joe_> sorry sirenbot [17:44:09] RECOVERY - HAProxy HTTPS wikiworkshop.org ECDSA on cp2049 is OK: SSL OK - Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2026-11-09 03:18:39 +0000 (expires in 46 days) https://wikitech.wikimedia.org/wiki/HTTPS [17:44:09] RECOVERY - HAProxy HTTPS wikipedia.org ECDSA on cp2049 is OK: SSL OK - Certificate *.wikipedia.org contains all required SANs:Certificate *.wikipedia.org (ECDSA) valid until 2026-11-03 19:15:40 +0000 (expires in 41 days) https://wikitech.wikimedia.org/wiki/HTTPS [17:44:09] RECOVERY - HAProxy HTTPS wikipedia25.org ECDSA on cp2049 is OK: SSL OK - Certificate wikipedia25.org contains all required SANs:Certificate wikipedia25.org (ECDSA) valid until 2026-12-03 05:05:02 +0000 (expires in 70 days) https://wikitech.wikimedia.org/wiki/HTTPS [17:44:18] (03CR) 10CDanis: [C:03+1] idp: Failover to eqiad [dns] - 10https://gerrit.wikimedia.org/r/1344337 (owner: 10Majavah) [17:44:20] RESOLVED: [2x] CirrusSearchFullTextLatencyTooHigh: CirrusSearch full_text 95th percentiles latency is too high (mw@codfw to dnsdisc) - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://grafana.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchFullTextLatencyTooHigh [17:44:21] Splunk is not showing the new pages to me. [17:44:29] RECOVERY - haproxy process on cp2049 is OK: PROCS OK: 2 processes with command name haproxy https://wikitech.wikimedia.org/wiki/HAProxy [17:44:51] !ack [17:45:40] Acked in Splunk. [17:45:44] FIRING: CirrusConsumerRerenderFetchErrorRate: cirrus_streaming_updater_consumer_search_codfw in codfw (k8s): ... [17:45:44] fetch error (rerenders) rate too high - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?var-datasource=codfw+prometheus%2Fk8s&var-namespace=cirrus-streaming-updater&var-helm_release=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusConsumerRerenderFetchErrorRate [17:46:02] !log taavi@dns1004 END - running authdns-update [17:46:04] RESOLVED: MediaWikiElevatedUnknownLogins: Elevated number of login successes (source unknown) via mw-api-ext - TODO - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?from=now-6h&orgId=1&to=now&viewPanel=26 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiElevatedUnknownLogins [17:46:42] FIRING: [5x] SystemdUnitFailed: prometheus-node-textfile-export_service_type.service on cumin1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:46:43] !ack [17:46:44] All incidents are already acked. [17:46:55] FIRING: [11x] ATSBackendErrorsHigh: ATS: elevated 5xx error ratio from eventgate-logging-external.discovery.wmnet in eqiad #page - https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Debugging - https://alerts.wikimedia.org/?q=alertname%3DATSBackendErrorsHigh [17:47:27] (03PS2) 10Eric Gardner: ReaderExperiments: Enable on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344048 [17:47:31] (03CR) 10Eric Gardner: ReaderExperiments: Enable on testwiki (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344048 (owner: 10Eric Gardner) [17:47:33] FIRING: [5x] KubernetesAPILatency: High Kubernetes API latency (GET leases) on k8s-mlstaging@codfw - https://wikitech.wikimedia.org/wiki/Kubernetes - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [17:47:40] (03PS2) 10Eric Gardner: ReaderExperiments: Set the preferred-sources debug flag on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344049 (https://phabricator.wikimedia.org/T436692) [17:48:07] RECOVERY - OpenSearch health check for shards on 9400 on cirrussearch2105 is OK: OK - elasticsearch status production-search-omega-codfw: cluster_name: production-search-omega-codfw, status: yellow, timed_out: False, number_of_nodes: 15, number_of_data_nodes: 15, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1732, active_shards: 4427, relocating_shards: 0, initializing_shards: 5, unassigned_shards: 763, [17:48:07] _unassigned_shards: 0, number_of_pending_tasks: 1, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 0, active_shards_percent_as_number: 85.21655437921079 https://wikitech.wikimedia.org/wiki/Search%23Administration [17:48:07] RECOVERY - OpenSearch health check for shards on 9400 on cirrussearch2104 is OK: OK - elasticsearch status production-search-omega-codfw: cluster_name: production-search-omega-codfw, status: yellow, timed_out: False, number_of_nodes: 15, number_of_data_nodes: 15, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1732, active_shards: 4427, relocating_shards: 0, initializing_shards: 5, unassigned_shards: 763, [17:48:07] _unassigned_shards: 0, number_of_pending_tasks: 2, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 39, active_shards_percent_as_number: 85.21655437921079 https://wikitech.wikimedia.org/wiki/Search%23Administration [17:48:07] RECOVERY - OpenSearch health check for shards on 9400 on cirrussearch2106 is OK: OK - elasticsearch status production-search-omega-codfw: cluster_name: production-search-omega-codfw, status: yellow, timed_out: False, number_of_nodes: 15, number_of_data_nodes: 15, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1732, active_shards: 4427, relocating_shards: 0, initializing_shards: 5, unassigned_shards: 763, [17:48:42] FIRING: JobUnavailable: Reduced availability for job netbox_global in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [17:48:57] RECOVERY - OpenSearch health check for shards on 9400 on cirrussearch2114 is OK: OK - elasticsearch status production-search-omega-codfw: cluster_name: production-search-omega-codfw, status: yellow, timed_out: False, number_of_nodes: 15, number_of_data_nodes: 15, discovered_master: True, discovered_cluster_manager: True, active_primary_shards: 1732, active_shards: 4634, relocating_shards: 0, initializing_shards: 1, unassigned_shards: 560, [17:48:57] _unassigned_shards: 0, number_of_pending_tasks: 2, number_of_in_flight_fetch: 0, task_max_waiting_in_queue_millis: 409, active_shards_percent_as_number: 89.20115495668912 https://wikitech.wikimedia.org/wiki/Search%23Administration [17:49:44] FIRING: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [17:49:50] CirrusSearch consumer-search@codfw is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=codfw&var-search_cluster=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [17:50:01] RESOLVED: [3x] CirrusStreamingUpdaterRateTooLow: CirrusSearch update rate from flink-app-consumer-cloudelastic is critically low - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterRateTooLow [17:50:04] (03PS3) 10Eric Gardner: ReaderExperiments: Set the preferred-sources debug flag on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344049 (https://phabricator.wikimedia.org/T436692) [17:50:26] (03PS1) 10Majavah: Failover Puppet to eqiad [dns] - 10https://gerrit.wikimedia.org/r/1344339 [17:50:29] (03CR) 10Eric Gardner: ReaderExperiments: Set the preferred-sources debug flag on testwiki (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344049 (https://phabricator.wikimedia.org/T436692) (owner: 10Eric Gardner) [17:50:37] (03PS2) 10Eric Gardner: ReaderExperiments: Drop the stale ShareHighlight config var [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344050 (https://phabricator.wikimedia.org/T424764) [17:50:47] (03CR) 10Ssingh: [C:03+1] Failover Puppet to eqiad [dns] - 10https://gerrit.wikimedia.org/r/1344339 (owner: 10Majavah) [17:51:02] (03CR) 10CI reject: [V:04-1] ReaderExperiments: Set the preferred-sources debug flag on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344049 (https://phabricator.wikimedia.org/T436692) (owner: 10Eric Gardner) [17:51:26] FIRING: [9x] SystemdUnitFailed: prometheus-node-textfile-export_service_type.service on cumin1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:51:44] (03CR) 10Majavah: [C:03+2] Failover Puppet to eqiad [dns] - 10https://gerrit.wikimedia.org/r/1344339 (owner: 10Majavah) [17:51:44] FIRING: [2x] CirrusStreamingUpdaterFlinkJobUnstable: cirrus_streaming_updater_consumer_search_codfw in codfw (k8s) is unstable - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterFlinkJobUnstable [17:51:55] !log taavi@dns1004 START - running authdns-update [17:52:33] FIRING: [40x] KubernetesAPILatency: High Kubernetes API latency (LIST adminnetworkpolicies) on k8s-dse@codfw - https://wikitech.wikimedia.org/wiki/Kubernetes - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [17:53:05] PROBLEM - check if authdns-update was run after a change was merged to operations/dns.git on dns2006 is CRITICAL: Local zone files are NOT in sync with operations/dns.git (SHA: local is 7adfce1efb8da45729c95741ebf8aab09efff421, dns.git is 00e9adda52528e25fcbfa66fc17475513601288d) https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run [17:53:05] PROBLEM - check if authdns-update was run after a change was merged to operations/dns.git on dns2004 is CRITICAL: Local zone files are NOT in sync with operations/dns.git (SHA: local is 7adfce1efb8da45729c95741ebf8aab09efff421, dns.git is 00e9adda52528e25fcbfa66fc17475513601288d) https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run [17:53:10] !log vriley@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on zuul1005.eqiad.wmnet with reason: host reimage [17:53:42] RESOLVED: JobUnavailable: Reduced availability for job netbox_global in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [17:53:44] FIRING: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [17:53:44] CirrusSearch consumer-search@codfw is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=codfw&var-search_cluster=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [17:54:17] !log taavi@dns1004 END - running authdns-update [17:55:12] FIRING: KafkaMirrorMakerAvgMsgConsumeRate: Kafka MirrorMaker main-codfw-to-main-eqiad average message consume rate in last 30m - https://wikitech.wikimedia.org/wiki/Kafka/Administration#MirrorMaker - https://grafana.wikimedia.org/d/000000521/kafka-mirrormaker?var-mirror_name=main-codfw-to-main-eqiad - https://alerts.wikimedia.org/?q=alertname%3DKafkaMirrorMakerAvgMsgConsumeRate [17:55:12] FIRING: KafkaMirrorMakerAvgMsgProduceRate: Kafka MirrorMaker main-codfw-to-main-eqiad average message produce rate in last 30m - https://wikitech.wikimedia.org/wiki/Kafka/Administration#MirrorMaker - https://grafana.wikimedia.org/d/000000521/kafka-mirrormaker?var-mirror_name=main-codfw-to-main-eqiad - https://alerts.wikimedia.org/?q=alertname%3DKafkaMirrorMakerAvgMsgProduceRate [17:55:57] (03PS1) 10CDanis: codfw etcd down [dns] - 10https://gerrit.wikimedia.org/r/1344342 [17:56:20] FIRING: CirrusSearchBackendMemoryIssue: CirrusSearch backend failed 41.04121666666667 times in the last 10 minutes due to memory usage (mw@codfw to dnsdisc) - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://grafana.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchBackendMemoryIssue [17:56:26] FIRING: [9x] SystemdUnitFailed: prometheus-node-textfile-export_service_type.service on cumin1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:56:31] jeena, jnuche: for avoidance of doubt, do not roll the train, major incident in progress [17:56:41] Thanks rzl [17:56:57] thank you :) [17:57:22] (03PS2) 10CDanis: codfw etcd down [dns] - 10https://gerrit.wikimedia.org/r/1344342 [17:57:23] jedgggeinetrfjgucnergjncev [17:57:33] RESOLVED: [38x] KubernetesAPILatency: High Kubernetes API latency (LIST adminnetworkpolicies) on k8s-dse@codfw - https://wikitech.wikimedia.org/wiki/Kubernetes - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [17:57:36] sorry ignore that^ [17:57:44] FIRING: CirrusStreamingUpdaterRateTooLow: CirrusSearch update rate from flink-app-consumer-search is critically low - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterRateTooLow [17:57:45] FIRING: [3x] WidespreadPuppetFailure: Puppet has failed in eqsin - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [17:57:54] !log vriley@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on zuul1005.eqiad.wmnet with reason: host reimage [17:58:10] (03CR) 10Giuseppe Lavagetto: [C:03+1] codfw etcd down [dns] - 10https://gerrit.wikimedia.org/r/1344342 (owner: 10CDanis) [17:58:19] (03CR) 10CDanis: [C:03+2] codfw etcd down [dns] - 10https://gerrit.wikimedia.org/r/1344342 (owner: 10CDanis) [17:58:23] (03CR) 10CDanis: [V:03+2 C:03+2] codfw etcd down [dns] - 10https://gerrit.wikimedia.org/r/1344342 (owner: 10CDanis) [17:58:24] RECOVERY - Check if active EventStreams endpoint is delivering messages. on alert1002 is OK: OK: An EventStreams message was consumed from https://stream.wikimedia.org/v2/stream/recentchange within 10 seconds. https://wikitech.wikimedia.org/wiki/Event_Platform/EventStreams/Administration [17:58:40] !log cdanis@dns1005 START - running authdns-update [18:00:05] jeena and jnuche: MediaWiki train - Utc-7 Version (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260923T1800). Please do the needful. [18:00:30] (03PS1) 10Eric Gardner: CentralNotice: Exclude the preferred sources experiment from banners [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344343 (https://phabricator.wikimedia.org/T436692) [18:01:05] !log cdanis@dns1005 END - running authdns-update [18:01:15] FIRING: [2x] MediaWikiLatencyExceeded: p75 latency high: codfw mw-web releases routed via main (k8s) 2.5s - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [18:01:20] RESOLVED: CirrusSearchBackendMemoryIssue: CirrusSearch backend failed 36.93556597222222 times in the last 10 minutes due to memory usage (mw@codfw to dnsdisc) - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://grafana.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchBackendMemoryIssue [18:01:26] FIRING: [9x] SystemdUnitFailed: prometheus-node-textfile-export_service_type.service on cumin1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:01:39] (03CR) 10Federico Ceratto: switchdc.databases.prepare: Retry checks for replication threads (032 comments) [cookbooks] - 10https://gerrit.wikimedia.org/r/1344031 (https://phabricator.wikimedia.org/T438833) (owner: 10CWilliams) [18:02:47] !log cdanis@cumin1004 START - Cookbook sre.dns.wipe-cache _etcd-client-ssl._tcp.ulsfo.wmnet on all recursors [18:02:57] !log cdanis@cumin1004 END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) _etcd-client-ssl._tcp.ulsfo.wmnet on all recursors [18:03:19] !log cdanis@cumin1004 START - Cookbook sre.dns.wipe-cache _etcd-client-ssl._tcp.eqsin.wmnet on all recursors [18:03:34] !log cdanis@cumin1004 END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) _etcd-client-ssl._tcp.eqsin.wmnet on all recursors [18:03:44] RESOLVED: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [18:03:44] CirrusSearch consumer-search@codfw is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=codfw&var-search_cluster=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [18:04:44] RESOLVED: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [18:04:44] CirrusSearch consumer-search@codfw is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=codfw&var-search_cluster=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [18:05:31] !log cdanis@cumin1004 START - Cookbook sre.dns.wipe-cache _etcd-client-ssl._tcp.eqsin.wmnet on all recursors [18:05:41] !log cdanis@cumin1004 END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) _etcd-client-ssl._tcp.eqsin.wmnet on all recursors [18:06:15] FIRING: [2x] MediaWikiLatencyExceeded: p75 latency high: codfw mw-web releases routed via main (k8s) 1.859s - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [18:08:38] !log sukhe@cumin1004 START - Cookbook sre.dns.wipe-cache idp.wikimedia.org on all recursors [18:08:48] !log sukhe@cumin1004 END (FAIL) - Cookbook sre.dns.wipe-cache (exit_code=99) idp.wikimedia.org on all recursors [18:10:27] FIRING: [2x] KafkaMirrorMakerAvgMsgConsumeRate: Kafka MirrorMaker main-eqiad-to-main-codfw average message consume rate in last 30m - https://wikitech.wikimedia.org/wiki/Kafka/Administration#MirrorMaker - https://alerts.wikimedia.org/?q=alertname%3DKafkaMirrorMakerAvgMsgConsumeRate [18:10:59] PROBLEM - Host crm2001 is DOWN: PING CRITICAL - Packet loss = 100% [18:11:42] PROBLEM - Host debmonitor2003 is DOWN: PING CRITICAL - Packet loss = 100% [18:11:42] PROBLEM - Host lsw1-a4-codfw is DOWN: PING CRITICAL - Packet loss = 100% [18:11:42] PROBLEM - Host lsw1-a6-codfw is DOWN: PING CRITICAL - Packet loss = 100% [18:11:48] PROBLEM - Host doh2001 is DOWN: PING CRITICAL - Packet loss = 100% [18:11:48] PROBLEM - Host lsw1-a2-codfw is DOWN: PING CRITICAL - Packet loss = 100% [18:11:48] PROBLEM - Host lsw1-a3-codfw is DOWN: PING CRITICAL - Packet loss = 100% [18:11:51] FIRING: [5x] ATSBackendErrorsHigh: ATS: elevated 5xx error ratio from eventgate-logging-external.discovery.wmnet in ulsfo #page - https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Debugging - https://alerts.wikimedia.org/?q=alertname%3DATSBackendErrorsHigh [18:11:59] PROBLEM - Host dse-k8s-ctrl2002 is DOWN: PING CRITICAL - Packet loss = 100% [18:12:00] PROBLEM - Host dse-k8s-etcd2003 is DOWN: PING CRITICAL - Packet loss = 100% [18:12:00] PROBLEM - Host lsw1-a5-codfw is DOWN: PING CRITICAL - Packet loss = 100% [18:12:00] PROBLEM - Host lsw1-a7-codfw is DOWN: PING CRITICAL - Packet loss = 100% [18:12:00] PROBLEM - Host search.svc.codfw.wmnet is DOWN: PING CRITICAL - Packet loss = 100% [18:12:00] PROBLEM - Host ns1-v4 is DOWN: PING CRITICAL - Packet loss = 100% [18:12:01] PROBLEM - Host lsw1-d3-codfw is DOWN: PING CRITICAL - Packet loss = 100% [18:12:01] PROBLEM - Host lsw1-e1-codfw is DOWN: PING CRITICAL - Packet loss = 100% [18:12:01] PROBLEM - Host lsw1-d4-codfw is DOWN: PING CRITICAL - Packet loss = 100% [18:12:02] PROBLEM - Host grafana2001 is DOWN: PING CRITICAL - Packet loss = 100% [18:12:02] PROBLEM - Host etherpad2002 is DOWN: PING CRITICAL - Packet loss = 100% [18:12:26] PROBLEM - Host chartmuseum2001 is DOWN: PING CRITICAL - Packet loss = 100% [18:12:26] PROBLEM - Host registry2005 is DOWN: PING CRITICAL - Packet loss = 100% [18:12:26] PROBLEM - Host webperf2003 is DOWN: PING CRITICAL - Packet loss = 100% [18:12:26] PROBLEM - Host ncredir2002 is DOWN: PING CRITICAL - Packet loss = 100% [18:12:26] PROBLEM - OSPF status on cr3-ulsfo is CRITICAL: OSPFv2: 3/4 UP : OSPFv3: 3/4 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [18:12:27] PROBLEM - OSPF status on cr2-drmrs is CRITICAL: OSPFv2: 3/4 UP : OSPFv3: 3/4 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [18:12:27] PROBLEM - OSPF status on cr1-drmrs is CRITICAL: OSPFv2: 3/4 UP : OSPFv3: 3/4 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [18:12:28] PROBLEM - OSPF status on cr1-magru is CRITICAL: OSPFv2: 4/5 UP : OSPFv3: 4/5 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [18:12:28] PROBLEM - Host idm2001 is DOWN: PING CRITICAL - Packet loss = 100% [18:12:29] PROBLEM - Host irc2003 is DOWN: PING CRITICAL - Packet loss = 100% [18:12:29] PROBLEM - Host logstash2025 is DOWN: PING CRITICAL - Packet loss = 100% [18:12:30] PROBLEM - Host logstash2031 is DOWN: PING CRITICAL - Packet loss = 100% [18:12:30] PROBLEM - Host logstash2030 is DOWN: PING CRITICAL - Packet loss = 100% [18:12:31] PROBLEM - Host aux-k8s-etcd2004 is DOWN: PING CRITICAL - Packet loss = 100% [18:12:31] PROBLEM - Host aux-k8s-etcd2003 is DOWN: PING CRITICAL - Packet loss = 100% [18:12:32] PROBLEM - MariaDB Replica IO: pc3 on pc1023 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error connecting to master repl2024@pc2023.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on pc2023.codfw.wmnet (110 Connection timed out) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:12:32] PROBLEM - MariaDB Replica IO: pc4 on pc1024 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error connecting to master repl2024@pc2024.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on pc2024.codfw.wmnet (110 Connection timed out) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:12:33] PROBLEM - Host pybal-test2003 is DOWN: PING CRITICAL - Packet loss = 100% [18:12:33] PROBLEM - Host releases2003 is DOWN: PING CRITICAL - Packet loss = 100% [18:12:34] PROBLEM - Host poolcounter2005 is DOWN: PING CRITICAL - Packet loss = 100% [18:12:34] PROBLEM - Host rdb-lock2002 is DOWN: PING CRITICAL - Packet loss = 100% [18:12:35] PROBLEM - Host ml-etcd2003 is DOWN: PING CRITICAL - Packet loss = 100% [18:12:35] PROBLEM - Host stewards2001 is DOWN: PING CRITICAL - Packet loss = 100% [18:12:36] PROBLEM - Host rdb-lock2003 is DOWN: PING CRITICAL - Packet loss = 100% [18:12:36] PROBLEM - Host ml-staging-etcd2001 is DOWN: PING CRITICAL - Packet loss = 100% [18:12:37] PROBLEM - Host schema2004 is DOWN: PING CRITICAL - Packet loss = 100% [18:12:37] PROBLEM - Host planet2003 is DOWN: PING CRITICAL - Packet loss = 100% [18:12:38] PROBLEM - Host urldownloader2003 is DOWN: PING CRITICAL - Packet loss = 100% [18:12:38] PROBLEM - OSPF status on cr2-esams is CRITICAL: OSPFv2: 3/4 UP : OSPFv3: 3/4 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [18:12:39] PROBLEM - OSPF status on cr1-esams is CRITICAL: OSPFv2: 3/4 UP : OSPFv3: 3/4 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [18:12:39] PROBLEM - OSPF status on cr4-ulsfo is CRITICAL: OSPFv2: 3/4 UP : OSPFv3: 3/4 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [18:12:40] PROBLEM - MariaDB Replica IO: pc1 on pc1021 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error connecting to master repl2024@pc2021.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on pc2021.codfw.wmnet (110 Connection timed out) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:12:40] <_joe_> !ack [18:12:40] All incidents are already acked. [18:12:43] PROBLEM - Host ml-serve-ctrl2002 is DOWN: PING CRITICAL - Packet loss = 100% [18:12:43] PROBLEM - Host ml-etcd2001 is DOWN: PING CRITICAL - Packet loss = 100% [18:12:45] PROBLEM - Host ps1-e1-codfw is DOWN: PING CRITICAL - Packet loss = 100% [18:12:46] PROBLEM - Host ps1-e2-codfw is DOWN: PING CRITICAL - Packet loss = 100% [18:12:46] PROBLEM - Host ps1-e3-codfw is DOWN: PING CRITICAL - Packet loss = 100% [18:12:46] PROBLEM - Host ps1-e4-codfw is DOWN: PING CRITICAL - Packet loss = 100% [18:12:46] PROBLEM - Host ps1-e5-codfw is DOWN: PING CRITICAL - Packet loss = 100% [18:12:46] PROBLEM - Host ps1-d3-codfw is DOWN: PING CRITICAL - Packet loss = 100% [18:12:46] PROBLEM - Host ps1-f1-codfw is DOWN: PING CRITICAL - Packet loss = 100% [18:12:47] PROBLEM - Host ps1-c8-codfw is DOWN: PING CRITICAL - Packet loss = 100% [18:12:47] PROBLEM - Host ps1-d8-codfw is DOWN: PING CRITICAL - Packet loss = 100% [18:12:48] PROBLEM - Host ps1-d6-codfw is DOWN: PING CRITICAL - Packet loss = 100% [18:12:48] PROBLEM - Host ps1-d5-codfw is DOWN: PING CRITICAL - Packet loss = 100% [18:12:49] PROBLEM - Host ps1-d1-codfw is DOWN: PING CRITICAL - Packet loss = 100% [18:12:49] PROBLEM - Host ps1-d2-codfw is DOWN: PING CRITICAL - Packet loss = 100% [18:12:50] PROBLEM - Host ps1-c7-codfw is DOWN: PING CRITICAL - Packet loss = 100% [18:12:50] PROBLEM - Host ps1-c6-codfw is DOWN: PING CRITICAL - Packet loss = 100% [18:12:50] FIRING: GerritHAProxyServiceUnavailable: Gerrit tcp-proxy (HAProxy) service gerrit_ssh is DOWN in eqiad - https://wikitech.wikimedia.org/wiki/Gerrit/Operations#GerritHAProxyServiceUnavailable - grafana.wikimedia.org/d/459365f6-df37-48d6-8142-82b22c1875e7/gerrit-tcp-proxy?viewPanel=panel-15 - https://alerts.wikimedia.org/?q=alertname%3DGerritHAProxyServiceUnavailable [18:12:50] FIRING: [2x] GerritHAProxyBackendUnavailable: Gerrit backend is unavilable for tcp-proxy (HAProxy) gerrit_ssh - https://wikitech.wikimedia.org/wiki/Gerrit/Operations#GerritHAProxyBackendUnavailable - grafana.wikimedia.org/d/459365f6-df37-48d6-8142-82b22c1875e7/gerrit-tcp-proxy?viewPanel=panel-15 - https://alerts.wikimedia.org/?q=alertname%3DGerritHAProxyBackendUnavailable [18:12:51] PROBLEM - Host ps1-d4-codfw is DOWN: PING CRITICAL - Packet loss = 100% [18:12:51] PROBLEM - Host ps1-d7-codfw is DOWN: PING CRITICAL - Packet loss = 100% [18:12:52] PROBLEM - Host ps1-f2-codfw is DOWN: PING CRITICAL - Packet loss = 100% [18:12:52] PROBLEM - MariaDB Replica IO: pc6 on pc1016 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error connecting to master repl2024@pc2016.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on pc2016.codfw.wmnet (110 Connection timed out) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:12:53] PROBLEM - Host mr1-codfw IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [18:12:55] PROBLEM - OSPF status on cr2-eqiad is CRITICAL: OSPFv2: 7/8 UP : OSPFv3: 7/8 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [18:13:04] PROBLEM - OSPF status on cr2-eqdfw is CRITICAL: OSPFv2: 1/3 UP : OSPFv3: 1/3 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [18:13:05] PROBLEM - OSPF status on cr2-eqsin is CRITICAL: OSPFv2: 3/4 UP : OSPFv3: 3/4 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [18:13:06] PROBLEM - OSPF status on cr3-eqsin is CRITICAL: OSPFv2: 3/4 UP : OSPFv3: 3/4 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [18:13:10] FIRING: [7x] ProbeDown: Service people1005:30443 has failed probes (http_15_wikipedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#people1005:30443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [18:13:11] PROBLEM - Host ps1-f3-codfw is DOWN: PING CRITICAL - Packet loss = 100% [18:13:13] PROBLEM - Host ps1-f5-codfw is DOWN: PING CRITICAL - Packet loss = 100% [18:13:14] PROBLEM - Host pfw1-codfw is DOWN: PING CRITICAL - Packet loss = 100% [18:13:27] PROBLEM - MariaDB Replica IO: ms3 on db1268 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl2024@db2252.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on db2252.codfw.wmnet (110 Connection timed out) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:13:29] FIRING: BFDdown: BFD session down between cr1-drmrs and 185.15.58.154 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr1-drmrs:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [18:13:39] PROBLEM - Host lsw1-a2-codfw IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [18:13:39] PROBLEM - Host lsw1-a3-codfw IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [18:13:39] PROBLEM - Host lsw1-a4-codfw IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [18:13:39] PROBLEM - Host ssw1-a1-codfw IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [18:13:51] PROBLEM - Host lsw1-a5-codfw IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [18:13:51] PROBLEM - Host lsw1-a7-codfw IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [18:13:51] PROBLEM - Host lsw1-a6-codfw IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [18:13:59] FIRING: [112x] KubernetesCalicoDown: aux-k8s-worker2003.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [18:13:59] PROBLEM - Host lsw1-c6-codfw IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [18:13:59] PROBLEM - Host lsw1-c7-codfw IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [18:13:59] PROBLEM - Host lsw1-d2-codfw IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [18:13:59] PROBLEM - Host lsw1-d4-codfw IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [18:13:59] PROBLEM - Host lsw1-d1-codfw IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [18:14:00] PROBLEM - Host lsw1-d3-codfw IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [18:14:00] PROBLEM - Host lsw1-d7-codfw IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [18:14:01] PROBLEM - Host lsw1-e1-codfw IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [18:14:01] PROBLEM - Host lsw1-d6-codfw IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [18:14:02] PROBLEM - Host lsw1-d8-codfw IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [18:14:02] PROBLEM - Host lsw1-e2-codfw IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [18:14:03] PROBLEM - Host lsw1-e3-codfw IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [18:14:03] PROBLEM - Host lsw1-d5-codfw IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [18:14:03] FIRING: [3x] CoreBGPDown: Core BGP session down between cr1-magru and cr1-codfw (195.200.68.138) - group Confed_codfw - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [18:14:04] PROBLEM - Host lsw1-e5-codfw IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [18:14:04] PROBLEM - Host lsw1-e4-codfw IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [18:14:05] PROBLEM - Host lsw1-f1-codfw IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [18:14:05] PROBLEM - Host lsw1-f2-codfw IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [18:14:06] PROBLEM - Host lsw1-f4-codfw IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [18:14:06] PROBLEM - Host lsw1-f3-codfw IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [18:14:09] PROBLEM - Host apt2002 is DOWN: PING CRITICAL - Packet loss = 100% [18:14:09] PROBLEM - Host ssw1-a1-codfw is DOWN: PING CRITICAL - Packet loss = 100% [18:14:09] PROBLEM - Host aux-k8s-worker2004 is DOWN: PING CRITICAL - Packet loss = 100% [18:14:56] PROBLEM - Host lsw1-a8-codfw IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [18:14:59] !log vriley@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host zuul1005.eqiad.wmnet with OS trixie [18:15:00] PROBLEM - Host ssw1-e1-codfw IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [18:15:00] PROBLEM - Host ssw1-d8-codfw is DOWN: PING CRITICAL - Packet loss = 100% [18:15:00] PROBLEM - Host ssw1-e1-codfw is DOWN: PING CRITICAL - Packet loss = 100% [18:15:00] PROBLEM - Host ssw1-d8-codfw IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [18:15:00] PROBLEM - Host ssw1-d1-codfw IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [18:15:01] PROBLEM - Host ssw1-d1-codfw is DOWN: PING CRITICAL - Packet loss = 100% [18:15:03] PROBLEM - Host ssw1-f1-codfw is DOWN: PING CRITICAL - Packet loss = 100% [18:15:03] PROBLEM - Host ssw1-f1-codfw IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [18:15:09] 10ops-eqiad, 06SRE, 06Collaboration-Services, 06DC-Ops: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12356746 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by vriley@cumin1004 for host zuul1005.eqiad.wmnet with OS trixie completed: - zuul100... [18:15:21] PROBLEM - Host lsw1-b7-codfw IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [18:15:21] PROBLEM - Host lsw1-b8-codfw IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [18:16:12] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1018.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [18:16:12] PROBLEM - Host aphlict2001 is DOWN: PING CRITICAL - Packet loss = 100% [18:16:12] PROBLEM - Host apifeatureusage2001 is DOWN: PING CRITICAL - Packet loss = 100% [18:16:12] PROBLEM - Host aux-k8s-ctrl2002 is DOWN: PING CRITICAL - Packet loss = 100% [18:16:12] PROBLEM - Host aux-k8s-worker2005 is DOWN: PING CRITICAL - Packet loss = 100% [18:16:13] PROBLEM - Host aux-k8s-ctrl2003 is DOWN: PING CRITICAL - Packet loss = 100% [18:16:32] FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr1-eqiad:et-1/1/2 (Transport: cr1-codfw:et-1/0/2 (Arelion, IC-374549)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [18:16:32] PROBLEM - Host ssw1-a8-codfw is DOWN: PING CRITICAL - Packet loss = 100% [18:16:32] PROBLEM - Host ssw1-a8-codfw IPv6 is DOWN: PING CRITICAL - Packet loss = 100% [18:16:33] PROBLEM - Host zuul2001 is DOWN: PING CRITICAL - Packet loss = 100% [18:16:33] PROBLEM - Host zuul2002 is DOWN: PING CRITICAL - Packet loss = 100% [18:16:51] FIRING: [2x] KafkaMirrorMakerAvgMsgConsumeRate: Kafka MirrorMaker main-eqiad-to-main-codfw average message consume rate in last 30m - https://wikitech.wikimedia.org/wiki/Kafka/Administration#MirrorMaker - https://alerts.wikimedia.org/?q=alertname%3DKafkaMirrorMakerAvgMsgConsumeRate [18:16:57] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [18:17:09] FIRING: PfwCoreBGPDown: Fundraising Firewall core BGP session down between pfw1-eqiad and (null) (10.195.0.249) - group VPN - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=eqiad&var-device=pfw1-eqiad:9804&var-bgp_group=VPN&var-bgp_neighbor=(null) - https://alerts.wikimedia.org/?q=alertname%3DPfwCoreBGPDown [18:17:31] RESOLVED: CirrusConsumerRerenderFetchErrorRate: cirrus_streaming_updater_consumer_search_codfw in codfw (k8s): ... [18:17:37] fetch error (rerenders) rate too high - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?var-datasource=codfw+prometheus%2Fk8s&var-namespace=cirrus-streaming-updater&var-helm_release=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusConsumerRerenderFetchErrorRate [18:18:14] RESOLVED: MediaWikiLatencyExceeded: p75 latency high: codfw mw-web releases routed via main (k8s) 2.157s - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=55&var-dc=codfw%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [18:18:23] FIRING: [6x] SystemdUnitFailed: prometheus-node-textfile-export_service_type.service on cumin1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:18:31] RESOLVED: [2x] CalicoKubeControllersDown: Calico Kubernetes Controllers not running - https://wikitech.wikimedia.org/wiki/Calico#Kube_Controllers - TODO - https://alerts.wikimedia.org/?q=alertname%3DCalicoKubeControllersDown [18:18:41] RESOLVED: CirrusStreamingUpdaterFlinkJobUnstable: cirrus_streaming_updater_consumer_search_codfw in codfw (k8s) is unstable - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?var-datasource=codfw+prometheus%2Fk8s&var-namespace=cirrus-streaming-updater&var-helm_release=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterFlinkJobUnstable [18:18:44] Gerrit is broken; Error: 502, Broken pipe at 2026-09-23 18:17:10 GMT [18:18:51] RESOLVED: [5x] ATSBackendErrorsHigh: ATS: elevated 5xx error ratio from eventgate-logging-external.discovery.wmnet in ulsfo #page - https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server#Debugging - https://alerts.wikimedia.org/?q=alertname%3DATSBackendErrorsHigh [18:18:59] RESOLVED: RESTGatewayBackendErrorsHigh: rest-gateway: high 5xx errors from mobileapps_cluster in codfw #page - https://wikitech.wikimedia.org/wiki/API_Gateway#How_to_debug_it - https://grafana.wikimedia.org/d/UOH-5IDMz/api-and-rest-gateway?orgId=1&refresh=30s&viewPanel=57&var-datasource=codfw%20prometheus/k8s&var-instance=rest-gateway - https://alerts.wikimedia.org/?q=alertname%3DRESTGatewayBackendErrorsHigh [18:19:05] Southparkfan: codfw dc lost power, gerrit will take lower prioroity for now [18:19:12] <_joe_> Southparkfan: we have bigger fishes to fry atm [18:19:15] <_joe_> sorry :) [18:19:25] RESOLVED: CirrusStreamingUpdaterRateTooLow: CirrusSearch update rate from flink-app-consumer-search is critically low - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterRateTooLow [18:19:28] Yeah, just saw the rest of the messages [18:19:35] FIRING: [5x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [18:19:37] Alright [18:19:42] FIRING: [15x] ProbeDown: Service people1005:30443 has failed probes (http_15_wikipedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#people1005:30443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [18:19:59] FIRING: [10x] BFDdown: BFD session down between cr1-drmrs and 185.15.58.154 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [18:20:00] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1018.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1020.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [18:20:07] RESOLVED: [84x] KubernetesCalicoDown: dse-k8s-ctrl2001.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [18:20:11] FIRING: [8x] CoreBGPDown: Core BGP session down between cr1-drmrs and cr1-codfw (185.15.58.154) - group Confed_codfw - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [18:20:37] PROBLEM - MariaDB Replica Lag: ms3 on db1268 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 626.16 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:20:43] PROBLEM - MariaDB Replica Lag: pc4 on pc1024 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 632.89 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:20:43] PROBLEM - MariaDB Replica Lag: pc3 on pc1023 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 632.97 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:20:46] PROBLEM - MariaDB Replica Lag: pc6 on pc1016 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 634.46 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:20:57] PROBLEM - MariaDB Replica Lag: pc1 on pc1021 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 645.45 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:20:59] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [18:21:26] FIRING: [4x] SystemdUnitFailed: prometheus-node-textfile-export_service_type.service on cumin1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:21:46] FIRING: [2x] JobUnavailable: Reduced availability for job gerrit in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [18:22:45] FIRING: [6x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [18:23:02] RESOLVED: [15x] ProbeDown: Service people1005:30443 has failed probes (http_15_wikipedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#people1005:30443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [18:28:52] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [18:29:06] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [18:30:25] RECOVERY - Host lsw1-d4-codfw is UP: PING OK - Packet loss = 0%, RTA = 183.78 ms [18:30:25] RECOVERY - Host lsw1-c6-codfw is UP: PING OK - Packet loss = 0%, RTA = 183.24 ms [18:30:25] RECOVERY - Host durum2002 is UP: PING OK - Packet loss = 0%, RTA = 176.99 ms [18:30:25] RECOVERY - Host dse-k8s-ctrl2002 is UP: PING OK - Packet loss = 0%, RTA = 177.45 ms [18:30:25] RECOVERY - Host hcaptcha-proxy2002 is UP: PING OK - Packet loss = 0%, RTA = 177.42 ms [18:30:25] FIRING: WidespreadPuppetFailure: Puppet has failed in eqiad - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [18:30:27] RECOVERY - Host lsw1-b3-codfw is UP: PING OK - Packet loss = 0%, RTA = 149.96 ms [18:30:27] RECOVERY - Host lsw1-c5-codfw is UP: PING OK - Packet loss = 0%, RTA = 152.83 ms [18:30:27] RECOVERY - Host lsw1-c2-codfw is UP: PING OK - Packet loss = 0%, RTA = 119.90 ms [18:30:27] RECOVERY - Host lsw1-b5-codfw is UP: PING OK - Packet loss = 0%, RTA = 147.15 ms [18:30:29] FIRING: [2x] RedisMemoryFull: Redis memory full on gitlab1003:9121 - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_gitlab - https://alerts.wikimedia.org/?q=alertname%3DRedisMemoryFull [18:30:33] FIRING: CalicoKubeControllersDown: Calico Kubernetes Controllers not running - https://wikitech.wikimedia.org/wiki/Calico#Kube_Controllers - TODO - https://alerts.wikimedia.org/?q=alertname%3DCalicoKubeControllersDown [18:30:41] FIRING: CalicoTyphaDown: Too few (1) calico-typha replicas running - https://wikitech.wikimedia.org/wiki/Calico#Typha" - TODO - https://alerts.wikimedia.org/?q=alertname%3DCalicoTyphaDown [18:30:52] FIRING: CirrusProducerFlinkJobNotRunning: cirrus_streaming_updater_producer in codfw (k8s) is not running - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?var-datasource=codfw+prometheus%2Fk8s&var-namespace=cirrus-streaming-updater&var-helm_release=producer - https://alerts.wikimedia.org/?q=alertname%3DCirrusProducerFlinkJobNotRunning [18:30:53] !log taavi@cumin1004 START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit1003.wikimedia.org [18:31:00] FIRING: [6x] KubernetesAPILatency: High Kubernetes API latency (GET bgpconfigurations) on k8s@codfw - https://wikitech.wikimedia.org/wiki/Kubernetes - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [18:31:05] FIRING: WebrequestSampledDown: Benthos metrics for webrequest_sampled are not reported from eqiad and codfw - https://wikitech.wikimedia.org/wiki/Benthos#Benthos_on_centrallog - https://grafana.wikimedia.org/d/V0TSK7O4z/benthos?var-port=4151 - https://alerts.wikimedia.org/?q=alertname%3DWebrequestSampledDown [18:31:46] FIRING: [10x] ProbeDown: Service docker-registry:443 has failed probes (http_docker-registry_ip4) #page - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [18:31:51] FIRING: [11x] ProbeDown: Service docker-registry:443 has failed probes (http_docker-registry_ip4) #page - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [18:31:55] FIRING: [7x] KubernetesCalicoDown: wikikube-worker2033.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [18:32:00] FIRING: [2x] ProbeDown: Service netbox2003:443 has failed probes (http_netbox_exports_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#netbox2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [18:32:15] FIRING: [2x] ProbeDown: Service idp2005:443 has failed probes (http_idp_wikimedia_org_ip4) - https://wikitech.wikimedia.org/wiki/CAS-SSO#Alerting - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [18:32:50] FIRING: ProbeDown: Service gerrit2003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [18:32:54] FIRING: EtcdReplicationDown: etcd replication down on conf2005:8000 #page - https://wikitech.wikimedia.org/wiki/Etcd/Main_cluster#Replication - TODO - https://alerts.wikimedia.org/?q=alertname%3DEtcdReplicationDown [18:34:12] RECOVERY - Host idm2001 is UP: PING OK - Packet loss = 0%, RTA = 33.39 ms [18:34:12] RECOVERY - Host irc2003 is UP: PING OK - Packet loss = 0%, RTA = 32.00 ms [18:34:12] RECOVERY - Host ncredir2002 is UP: PING OK - Packet loss = 0%, RTA = 33.20 ms [18:34:12] RECOVERY - Host kafkamon2003 is UP: PING OK - Packet loss = 0%, RTA = 33.09 ms [18:34:12] RECOVERY - OSPF status on cr2-eqdfw is OK: OSPFv2: 3/3 UP : OSPFv3: 3/3 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status [18:34:16] PROBLEM - HAProxy HTTPS wikipedia.org ECDSA on cp2044 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection refused https://wikitech.wikimedia.org/wiki/HTTPS [18:34:16] PROBLEM - HAProxy HTTPS wikipedia25.org ECDSA on cp2044 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection refused https://wikitech.wikimedia.org/wiki/HTTPS [18:34:18] PROBLEM - HAProxy HTTPS wikiworkshop.org ECDSA on cp2044 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection refused https://wikitech.wikimedia.org/wiki/HTTPS [18:34:18] PROBLEM - HAProxy HTTPS wikiworkshop.org ECDSA on cp2043 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection refused https://wikitech.wikimedia.org/wiki/HTTPS [18:34:18] PROBLEM - HAProxy HTTPS wikipedia.org ECDSA on cp2043 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection refused https://wikitech.wikimedia.org/wiki/HTTPS [18:34:18] PROBLEM - HAProxy HTTPS wikipedia25.org ECDSA on cp2043 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection refused https://wikitech.wikimedia.org/wiki/HTTPS [18:34:18] PROBLEM - mysqld processes on db2205 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [18:34:19] RECOVERY - Host cr1-codfw is UP: PING OK - Packet loss = 0%, RTA = 31.48 ms [18:34:24] PROBLEM - Check unit status of sync-puppet-volatile on puppetserver2001 is CRITICAL: CRITICAL: Status of the systemd unit sync-puppet-volatile https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [18:34:24] RECOVERY - Host build2004 is UP: PING OK - Packet loss = 0%, RTA = 33.93 ms [18:34:24] RECOVERY - Host build2002 is UP: PING OK - Packet loss = 0%, RTA = 32.44 ms [18:34:24] RECOVERY - Host install2005 is UP: PING OK - Packet loss = 0%, RTA = 32.03 ms [18:34:24] RECOVERY - Host people2004 is UP: PING OK - Packet loss = 0%, RTA = 31.95 ms [18:34:24] RECOVERY - Host mx-in2001 is UP: PING OK - Packet loss = 0%, RTA = 31.89 ms [18:35:30] PROBLEM - Check unit status of httpbb_kubernetes_mw-api-ext-next_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-api-ext-next_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [18:35:30] PROBLEM - PyBal connections to etcd on lvs2013 is CRITICAL: CRITICAL: 0 connections established with conf2004.codfw.wmnet:4001 (min=79) https://wikitech.wikimedia.org/wiki/PyBal [18:35:30] PROBLEM - PyBal connections to etcd on lvs2012 is CRITICAL: CRITICAL: 0 connections established with conf2004.codfw.wmnet:4001 (min=6) https://wikitech.wikimedia.org/wiki/PyBal [18:35:30] PROBLEM - MariaDB Events es3 on es2050 is CRITICAL: CRITICAL - Failed to query events: ERROR 2002 (HY000): Cant connect to local server through socket /run/mysqld/mysqld.sock (2) https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [18:35:30] PROBLEM - pt-heartbeat-wikimedia process on db2229 is CRITICAL: PROCS CRITICAL: 0 processes with args pt-heartbeat-wikimedia https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23pt-heartbeat [18:35:30] PROBLEM - mysqld processes on es2050 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [18:35:30] PROBLEM - mysqld processes on es2046 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [18:35:30] PROBLEM - mysqld processes on es2045 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [18:35:30] RECOVERY - Host urldownloader2005 is UP: PING OK - Packet loss = 0%, RTA = 32.10 ms [18:35:30] PROBLEM - MariaDB Events s6 on db2229 is CRITICAL: CRITICAL - Failed to query events: ERROR 2002 (HY000): Cant connect to local server through socket /run/mysqld/mysqld.sock (2) https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [18:35:31] PROBLEM - MariaDB Replica SQL: s2 #page on db2225 is CRITICAL: CRITICAL slave_sql_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:35:31] PROBLEM - MariaDB read only s2 on db2197 is CRITICAL: Could not connect to localhost:3312 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [18:35:32] PROBLEM - MariaDB read only s6 on db2224 is CRITICAL: Could not connect to localhost:3306 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [18:35:32] PROBLEM - MariaDB Event Scheduler s2 on db2225 is CRITICAL: Could not connect to localhost:3306 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [18:35:41] !ack [18:35:42] 8429 (ACKED) [10x] ProbeDown sre (probes/service codfw) [18:35:42] 8430 (ACKED) db2251 (paged)/mysqld processes (paged) [18:35:42] 8431 (ACKED) db2231 (paged)/MariaDB Replica SQL: x1 (paged) [18:35:43] 8432 (ACKED) db2227 (paged)/MariaDB Replica IO: s3 (paged) [18:35:43] 8433 (ACKED) db2227 (paged)/MariaDB Replica SQL: s3 (paged) [18:35:43] 8434 (ACKED) es2035 (paged)/MariaDB Replica SQL: es6 (paged) [18:35:43] 8435 (ACKED) es2035 (paged)/MariaDB Replica Lag: es6 (paged) [18:35:44] 8436 (ACKED) db2227 (paged)/MariaDB Replica Lag: s3 (paged) [18:35:44] 8437 (ACKED) db2178 (paged)/MariaDB Replica Lag: s5 (paged) [18:35:45] 8438 (ACKED) EtcdReplicationDown etcd sre (conf2005:8000 etcdmirror codfw) [18:35:45] 8439 (ACKED) db2243 (paged)/MariaDB Replica Lag: x3 (paged) [18:35:46] 8440 (ACKED) db2177 (paged)/MariaDB Replica IO: s3 (paged) [18:35:46] 8441 (ACKED) db2184/mysqld processes (paged) [18:35:47] 8442 (ACKED) db2243 (paged)/MariaDB Replica IO: x3 (paged) [18:35:47] 8443 (ACKED) db2191 (paged)/MariaDB Replica Lag: x1 (paged) [18:35:48] 8444 (ACKED) db2205 (paged)/MariaDB Replica IO: s3 (paged) [18:35:48] 8445 (ACKED) db2190 (paged)/MariaDB Replica IO: s3 (paged) [18:35:49] 8446 (ACKED) db2243 (paged)/MariaDB Replica SQL: x3 (paged) [18:35:49] 8447 (ACKED) db2188 (paged)/MariaDB Replica IO: s1 (paged) [18:35:50] 8448 (ACKED) db2203 (paged)/MariaDB Replica Lag: s1 (paged) [18:35:50] 8449 (ACKED) db2191 (paged)/MariaDB Replica IO: x1 (paged) [18:36:36] !log taavi@cumin1004 START - Cookbook sre.gerrit.read-only-toggle from gerrit2003.wikimedia.org [18:36:39] !log taavi@cumin1004 END (PASS) - Cookbook sre.gerrit.read-only-toggle (exit_code=0) from gerrit2003.wikimedia.org [18:36:43] !log taavi@cumin1004 START - Cookbook sre.gerrit.read-only-toggle from gerrit1003.wikimedia.org [18:36:45] !log taavi@cumin1004 END (PASS) - Cookbook sre.gerrit.read-only-toggle (exit_code=0) from gerrit1003.wikimedia.org [18:37:05] PROBLEM - Kafka Broker Server #page on kafka-main2006 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args Kafka /etc/kafka/server.properties https://wikitech.wikimedia.org/wiki/Kafka/Administration [18:37:06] PROBLEM - mysqld processes on db2223 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [18:37:08] PROBLEM - MariaDB Events s2 on db2189 is CRITICAL: CRITICAL - Failed to query events: ERROR 2002 (HY000): Cant connect to local server through socket /run/mysqld/mysqld.sock (2) https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [18:37:08] PROBLEM - mysqld processes on db2229 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [18:37:08] PROBLEM - mysqld processes on db2228 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [18:37:09] PROBLEM - MariaDB Replica SQL: s6 #page on db2229 is CRITICAL: CRITICAL slave_sql_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:37:10] PROBLEM - MariaDB Replica SQL: s2 #page on db2189 is CRITICAL: CRITICAL slave_sql_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:37:10] PROBLEM - mysqld processes on db2222 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [18:37:11] PROBLEM - MariaDB Replica IO: s6 #page on db2193 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl2024@db2229.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on db2229.codfw.wmnet (111 Connection refused) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:37:12] PROBLEM - MariaDB Replica SQL: s5 #page on db2228 is CRITICAL: CRITICAL slave_sql_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:37:13] PROBLEM - MariaDB Replica IO: s5 #page on db2228 is CRITICAL: CRITICAL slave_io_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:37:13] PROBLEM - mysqld processes on db2224 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [18:37:13] RECOVERY - Host ps1-a2-codfw is UP: PING OK - Packet loss = 0%, RTA = 32.53 ms [18:37:13] PROBLEM - Check whether ferm is active by checking the default input chain on aux-k8s-worker2006 is CRITICAL: ERROR ferm input drop default policy not set, ferm might not have been started correctly https://wikitech.wikimedia.org/wiki/Monitoring/check_ferm [18:37:28] PROBLEM - OpenSearch health check for shards on 9400 on cirrussearch2061 is CRITICAL: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4) https://wikitech.wikimedia.org/wiki/Search%23Administration [18:37:29] PROBLEM - OpenSearch health check for shards on 9600 on cirrussearch2079 is CRITICAL: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4) https://wikitech.wikimedia.org/wiki/Search%23Administration [18:37:29] PROBLEM - OpenSearch health check for shards on 9600 on cirrussearch2076 is CRITICAL: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4) https://wikitech.wikimedia.org/wiki/Search%23Administration [18:37:29] PROBLEM - OpenSearch health check for shards on 9600 on cirrussearch2069 is CRITICAL: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4) https://wikitech.wikimedia.org/wiki/Search%23Administration [18:37:29] PROBLEM - OpenSearch health check for shards on 9600 on cirrussearch2062 is CRITICAL: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4) https://wikitech.wikimedia.org/wiki/Search%23Administration [18:37:29] PROBLEM - OpenSearch health check for shards on 9400 on cirrussearch2073 is CRITICAL: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4) https://wikitech.wikimedia.org/wiki/Search%23Administration [18:37:29] PROBLEM - OpenSearch health check for shards on 9600 on cirrussearch2080 is CRITICAL: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4) https://wikitech.wikimedia.org/wiki/Search%23Administration [18:37:30] PROBLEM - OpenSearch health check for shards on 9600 on cirrussearch2075 is CRITICAL: CRITICAL - elasticsearch http://localhost:9600/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9600): Read timed out. (read timeout=4) https://wikitech.wikimedia.org/wiki/Search%23Administration [18:37:30] PROBLEM - OpenSearch health check for shards on 9400 on cirrussearch2074 is CRITICAL: CRITICAL - elasticsearch http://localhost:9400/_cluster/health error while fetching: HTTPConnectionPool(host=localhost, port=9400): Read timed out. (read timeout=4) https://wikitech.wikimedia.org/wiki/Search%23Administration [18:37:31] RECOVERY - Host lsw1-a2-codfw.mgmt is UP: PING OK - Packet loss = 0%, RTA = 30.67 ms [18:37:31] RECOVERY - check if authdns-update was run after a change was merged to operations/dns.git on dns2005 is OK: Local zone files and operations/dns.git are in sync https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run [18:37:32] RECOVERY - Host ps1-a1-codfw is UP: PING OK - Packet loss = 0%, RTA = 31.06 ms [18:37:32] RECOVERY - Blazegraph Port for wdqs-categories on wdqs2013 is OK: TCP OK - 0.000 second response time on 127.0.0.1 port 9990 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [18:37:33] RECOVERY - Blazegraph Port for wdqs-blazegraph on wdqs2013 is OK: TCP OK - 0.000 second response time on 127.0.0.1 port 9999 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [18:37:33] RECOVERY - Blazegraph process -wdqs-blazegraph- on wdqs2023 is OK: PROCS OK: 1 process with UID = 498 (blazegraph), regex args ^java .* --port 9999 .* blazegraph-service-.*war https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [18:37:34] RECOVERY - Blazegraph process -wdqs-categories- on wdqs2013 is OK: PROCS OK: 1 process with UID = 498 (blazegraph), regex args ^java .* --port 9990 .* blazegraph-service-.*war https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [18:37:35] RECOVERY - Kafka Broker Server on kafka-logging2001 is OK: PROCS OK: 1 process with command name java, args Kafka /etc/kafka/server.properties https://wikitech.wikimedia.org/wiki/Kafka/Administration [18:37:35] PROBLEM - MariaDB Replica IO: pc8 on pc2018 is CRITICAL: CRITICAL slave_io_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:37:36] PROBLEM - MariaDB Replica SQL: pc1 on pc2021 is CRITICAL: CRITICAL slave_sql_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:37:36] PROBLEM - MariaDB Events es5 on es2045 is CRITICAL: CRITICAL - Failed to query events: ERROR 2002 (HY000): Cant connect to local server through socket /run/mysqld/mysqld.sock (2) https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [18:37:37] PROBLEM - mysqld processes on pc2018 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [18:37:38] PROBLEM - Check unit status of push_cross_cluster_settings_9400 on cirrussearch2092 is CRITICAL: CRITICAL: Status of the systemd unit push_cross_cluster_settings_9400 https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [18:37:41] PROBLEM - Check unit status of push_cross_cluster_settings_9400 on cirrussearch2100 is CRITICAL: CRITICAL: Status of the systemd unit push_cross_cluster_settings_9400 https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [18:37:56] RECOVERY - Host ps1-c5-codfw is UP: PING OK - Packet loss = 0%, RTA = 31.09 ms [18:37:57] PROBLEM - mysqld processes #page on pc2017 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [18:37:57] PROBLEM - MariaDB Replica IO: pc7 on pc2017 is CRITICAL: CRITICAL slave_io_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:37:58] PROBLEM - MariaDB Replica SQL: pc8 on pc2018 is CRITICAL: CRITICAL slave_sql_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:37:58] PROBLEM - MariaDB Replica SQL: pc7 on pc2017 is CRITICAL: CRITICAL slave_sql_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:37:58] RECOVERY - Host lsw1-c4-codfw.mgmt is UP: PING OK - Packet loss = 0%, RTA = 453.82 ms [18:37:58] RECOVERY - Host ssw1-a1-codfw.mgmt is UP: PING OK - Packet loss = 0%, RTA = 30.99 ms [18:37:59] PROBLEM - Check whether ferm is active by checking the default input chain on aqs2005 is CRITICAL: ERROR ferm input drop default policy not set, ferm might not have been started correctly https://wikitech.wikimedia.org/wiki/Monitoring/check_ferm [18:37:59] RECOVERY - Host ps1-c4-codfw is UP: PING OK - Packet loss = 0%, RTA = 30.92 ms [18:37:59] RECOVERY - Host lsw1-c3-codfw.mgmt is UP: PING OK - Packet loss = 0%, RTA = 30.92 ms [18:38:01] RECOVERY - Host ps1-c3-codfw is UP: PING OK - Packet loss = 0%, RTA = 31.13 ms [18:38:01] RECOVERY - Host ps1-c2-codfw is UP: PING OK - Packet loss = 0%, RTA = 30.70 ms [18:38:01] RECOVERY - Blazegraph process -wdqs-categories- on wdqs2011 is OK: PROCS OK: 1 process with UID = 498 (blazegraph), regex args ^java .* --port 9990 .* blazegraph-service-.*war https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [18:38:01] RECOVERY - HAProxy HTTPS wikiworkshop.org ECDSA on cp2047 is OK: SSL OK - Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2026-11-09 03:18:39 +0000 (expires in 46 days) https://wikitech.wikimedia.org/wiki/HTTPS [18:38:02] RECOVERY - HAProxy HTTPS wikipedia25.org ECDSA on cp2047 is OK: SSL OK - Certificate wikipedia25.org contains all required SANs:Certificate wikipedia25.org (ECDSA) valid until 2026-12-03 05:05:02 +0000 (expires in 70 days) https://wikitech.wikimedia.org/wiki/HTTPS [18:38:02] RECOVERY - HAProxy HTTPS wikipedia.org ECDSA on cp2047 is OK: SSL OK - Certificate *.wikipedia.org contains all required SANs:Certificate *.wikipedia.org (ECDSA) valid until 2026-11-03 19:15:40 +0000 (expires in 41 days) https://wikitech.wikimedia.org/wiki/HTTPS [18:38:03] RECOVERY - HAProxy HTTPS wikipedia25.org ECDSA on cp2046 is OK: SSL OK - Certificate wikipedia25.org contains all required SANs:Certificate wikipedia25.org (ECDSA) valid until 2026-12-03 05:05:02 +0000 (expires in 70 days) https://wikitech.wikimedia.org/wiki/HTTPS [18:38:03] RECOVERY - HAProxy HTTPS measure-eqiad.wikimedia.org ECDSA on cp2048 is OK: SSL OK - Certificate measure-eqiad.wikimedia.org contains all required SANs:Certificate measure-eqiad.wikimedia.org (ECDSA) valid until 2026-12-02 14:04:24 +0000 (expires in 69 days) https://wikitech.wikimedia.org/wiki/HTTPS [18:38:04] RECOVERY - HAProxy HTTPS upload.wikimedia.org ECDSA on cp2048 is OK: SSL OK - Certificate upload.wikimedia.org contains all required SANs:Certificate upload.wikimedia.org (ECDSA) valid until 2026-11-09 05:19:14 +0000 (expires in 46 days) https://wikitech.wikimedia.org/wiki/HTTPS [18:38:04] RECOVERY - Kafka Broker Server on kafka-logging2003 is OK: PROCS OK: 1 process with command name java, args Kafka /etc/kafka/server.properties https://wikitech.wikimedia.org/wiki/Kafka/Administration [18:38:05] RECOVERY - Kafka broker TLS certificate validity on kafka-logging2001 is OK: SSL OK - Certificate kafka-logging2001.codfw.wmnet valid until 2027-05-04 14:44:00 +0000 (expires in 222 days) https://wikitech.wikimedia.org/wiki/Kafka/Administration%23Renew_TLS_certificate [18:38:05] RECOVERY - pybal on lvs2011 is OK: PROCS OK: 1 process with UID = 0 (root), args /usr/sbin/pybal https://wikitech.wikimedia.org/wiki/PyBal [18:38:06] RECOVERY - HAProxy HTTPS wikiworkshop.org ECDSA on cp2046 is OK: SSL OK - Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2026-11-09 03:18:39 +0000 (expires in 46 days) https://wikitech.wikimedia.org/wiki/HTTPS [18:38:06] PROBLEM - Blazegraph process -wdqs-blazegraph- on wdqs2007 is CRITICAL: PROCS CRITICAL: 0 processes with UID = 498 (blazegraph), regex args ^java .* --port 9999 .* blazegraph-service-.*war https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [18:38:07] PROBLEM - mysqld processes on db2198 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [18:38:07] PROBLEM - MariaDB Replica SQL: s5 on db2201 is CRITICAL: CRITICAL slave_sql_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:38:08] PROBLEM - mysqld processes on db2201 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [18:38:08] PROBLEM - mysqld processes on db2200 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [18:38:09] PROBLEM - MariaDB Replica SQL: s7 on db2198 is CRITICAL: CRITICAL slave_sql_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:38:09] PROBLEM - pt-heartbeat-wikimedia process on db2157 is CRITICAL: PROCS CRITICAL: 0 processes with args pt-heartbeat-wikimedia https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23pt-heartbeat [18:38:10] PROBLEM - pt-heartbeat-wikimedia process on pc2021 is CRITICAL: PROCS CRITICAL: 0 processes with args pt-heartbeat-wikimedia https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23pt-heartbeat [18:38:10] PROBLEM - mysqld processes #page on pc2021 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [18:38:11] PROBLEM - mysqld processes on db2176 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [18:38:11] PROBLEM - MariaDB Replica SQL: s7 on db2200 is CRITICAL: CRITICAL slave_sql_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:38:12] PROBLEM - mysqld processes on db2175 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [18:38:12] PROBLEM - MariaDB Replica IO: x3 on db2200 is CRITICAL: CRITICAL slave_io_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:38:13] PROBLEM - MariaDB Replica SQL: s8 on db2198 is CRITICAL: CRITICAL slave_sql_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:38:13] PROBLEM - mysqld processes on es2052 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [18:38:14] PROBLEM - MariaDB Replica IO: s5 #page on db2171 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master repl2024@db2157.codfw.wmnet:3306 - retry-time: 60 maximum-retries: 100000 message: Cant connect to server on db2157.codfw.wmnet (111 Connection refused) https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:38:14] PROBLEM - Check unit status of httpbb_kubernetes_mw-api-ext_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-api-ext_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [18:38:15] PROBLEM - MariaDB Replica SQL: s1 on db2250 is CRITICAL: CRITICAL slave_sql_state could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:38:15] RECOVERY - HAProxy HTTPS wikiworkshop.org ECDSA on cp2043 is OK: SSL OK - Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2026-11-09 03:18:39 +0000 (expires in 46 days) https://wikitech.wikimedia.org/wiki/HTTPS [18:38:16] RECOVERY - Host lsw1-c5-codfw.mgmt is UP: PING OK - Packet loss = 0%, RTA = 30.84 ms [18:38:16] RECOVERY - HAProxy HTTPS wikipedia25.org ECDSA on cp2043 is OK: SSL OK - Certificate wikipedia25.org contains all required SANs:Certificate wikipedia25.org (ECDSA) valid until 2026-12-03 05:05:02 +0000 (expires in 70 days) https://wikitech.wikimedia.org/wiki/HTTPS [18:38:17] RECOVERY - HAProxy HTTPS wikipedia.org ECDSA on cp2043 is OK: SSL OK - Certificate *.wikipedia.org contains all required SANs:Certificate *.wikipedia.org (ECDSA) valid until 2026-11-03 19:15:40 +0000 (expires in 41 days) https://wikitech.wikimedia.org/wiki/HTTPS [18:38:17] RECOVERY - Blazegraph process -wdqs-blazegraph- on wdqs2011 is OK: PROCS OK: 1 process with UID = 498 (blazegraph), regex args ^java .* --port 9999 .* blazegraph-service-.*war https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [18:38:18] RECOVERY - Host ps1-b1-codfw is UP: PING OK - Packet loss = 0%, RTA = 30.79 ms [18:38:18] RECOVERY - pybal on lvs2013 is OK: PROCS OK: 1 process with UID = 0 (root), args /usr/sbin/pybal https://wikitech.wikimedia.org/wiki/PyBal [18:38:19] RECOVERY - haproxy process on cp2046 is OK: PROCS OK: 2 processes with command name haproxy https://wikitech.wikimedia.org/wiki/HAProxy [18:38:19] RECOVERY - PyBal backends health check on lvs2011 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [18:38:20] RECOVERY - HAProxy HTTPS wikipedia.org ECDSA on cp2046 is OK: SSL OK - Certificate *.wikipedia.org contains all required SANs:Certificate *.wikipedia.org (ECDSA) valid until 2026-11-03 19:15:40 +0000 (expires in 41 days) https://wikitech.wikimedia.org/wiki/HTTPS [18:38:20] RECOVERY - Kafka Broker Server #page on kafka-main2006 is OK: PROCS OK: 1 process with command name java, args Kafka /etc/kafka/server.properties https://wikitech.wikimedia.org/wiki/Kafka/Administration [18:38:21] PROBLEM - Check unit status of mwscript-cleanup on deploy2002 is CRITICAL: CRITICAL: Status of the systemd unit mwscript-cleanup https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [18:38:21] PROBLEM - Check unit status of httpbb_kubernetes_mw-wikifunctions_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-wikifunctions_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [18:38:24] RECOVERY - Host lsw1-c2-codfw.mgmt is UP: PING OK - Packet loss = 0%, RTA = 31.07 ms [18:38:28] RECOVERY - Host cloudsw1-b1-codfw.mgmt is UP: PING OK - Packet loss = 0%, RTA = 30.62 ms [18:38:28] RECOVERY - Host ps1-b2-codfw is UP: PING OK - Packet loss = 0%, RTA = 31.99 ms [18:38:29] PROBLEM - Check unit status of httpbb_kubernetes_mw-api-int_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-api-int_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [18:38:36] RECOVERY - Blazegraph Port for wdqs-blazegraph on wdqs2023 is OK: TCP OK - 0.000 second response time on 127.0.0.1 port 9999 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [18:38:36] RECOVERY - haproxy process on cp2043 is OK: PROCS OK: 2 processes with command name haproxy https://wikitech.wikimedia.org/wiki/HAProxy [18:38:36] RECOVERY - Blazegraph Port for wdqs-categories on wdqs2011 is OK: TCP OK - 0.000 second response time on 127.0.0.1 port 9990 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [18:38:36] RECOVERY - freshclam running on vrts2002 is OK: PROCS OK: 1 process with UID = 110 (clamav), command name freshclam https://wikitech.wikimedia.org/wiki/VRT_System%23ClamAV [18:38:37] RECOVERY - Blazegraph Port for wdqs-blazegraph on wdqs2011 is OK: TCP OK - 0.000 second response time on 127.0.0.1 port 9999 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [18:38:40] RECOVERY - Host lsw1-b2-codfw.mgmt is UP: PING OK - Packet loss = 0%, RTA = 30.63 ms [18:38:40] RECOVERY - Kafka broker TLS certificate validity on kafka-logging2003 is OK: SSL OK - Certificate kafka-logging2003.codfw.wmnet valid until 2027-05-01 18:11:00 +0000 (expires in 219 days) https://wikitech.wikimedia.org/wiki/Kafka/Administration%23Renew_TLS_certificate [18:38:40] RECOVERY - haproxy process on cp2047 is OK: PROCS OK: 2 processes with command name haproxy https://wikitech.wikimedia.org/wiki/HAProxy [18:38:40] RECOVERY - haproxy process on cp2048 is OK: PROCS OK: 2 processes with command name haproxy https://wikitech.wikimedia.org/wiki/HAProxy [18:38:40] PROBLEM - Check whether ferm is active by checking the default input chain on ms-be2066 is CRITICAL: ERROR ferm input drop default policy not set, ferm might not have been started correctly https://wikitech.wikimedia.org/wiki/Monitoring/check_ferm [18:38:42] RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 218024 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [18:38:43] RECOVERY - Juniper alarms on cr1-codfw is OK: JNX_ALARMS OK - 0 red alarms, 0 yellow alarms https://wikitech.wikimedia.org/wiki/Network_monitoring%23Juniper_alarm [18:38:43] !log cdanis@cumin1004 START - Cookbook sre.dns.wipe-cache _etcd-client-ssl._tcp.eqsin.wmnet _etcd-client-ssl._tcp.ulsfo.wmnet _etcd-client-ssl._tcp.codfw.wmnet on all recursors [18:38:47] !log cdanis@cumin1004 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) _etcd-client-ssl._tcp.eqsin.wmnet _etcd-client-ssl._tcp.ulsfo.wmnet _etcd-client-ssl._tcp.codfw.wmnet on all recursors [18:38:59] RECOVERY - Blazegraph Port for wdqs-blazegraph on wdqs2024 is OK: TCP OK - 0.000 second response time on 127.0.0.1 port 9999 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [18:39:01] RECOVERY - Kafka broker TLS certificate validity on kafka-logging2004 is OK: SSL OK - Certificate kafka-logging2004.codfw.wmnet valid until 2027-05-01 15:38:00 +0000 (expires in 219 days) https://wikitech.wikimedia.org/wiki/Kafka/Administration%23Renew_TLS_certificate [18:39:03] PROBLEM - Check whether ferm is active by checking the default input chain on restbase2030 is CRITICAL: ERROR ferm input drop default policy not set, ferm might not have been started correctly https://wikitech.wikimedia.org/wiki/Monitoring/check_ferm [18:39:03] RECOVERY - Blazegraph process -wdqs-blazegraph- on wdqs2007 is OK: PROCS OK: 1 process with UID = 498 (blazegraph), regex args ^java .* --port 9999 .* blazegraph-service-.*war https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [18:39:04] !ack [18:39:06] RECOVERY - Kafka Broker Server #page on kafka-main2007 is OK: PROCS OK: 1 process with command name java, args Kafka /etc/kafka/server.properties https://wikitech.wikimedia.org/wiki/Kafka/Administration [18:39:11] RECOVERY - HAProxy HTTPS wikipedia25.org ECDSA on cp2044 is OK: SSL OK - Certificate wikipedia25.org contains all required SANs:Certificate wikipedia25.org (ECDSA) valid until 2026-12-03 05:05:02 +0000 (expires in 70 days) https://wikitech.wikimedia.org/wiki/HTTPS [18:39:12] RECOVERY - HAProxy HTTPS wikipedia.org ECDSA on cp2044 is OK: SSL OK - Certificate *.wikipedia.org contains all required SANs:Certificate *.wikipedia.org (ECDSA) valid until 2026-11-03 19:15:40 +0000 (expires in 41 days) https://wikitech.wikimedia.org/wiki/HTTPS [18:39:12] RECOVERY - HAProxy HTTPS wikiworkshop.org ECDSA on cp2044 is OK: SSL OK - Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2026-11-09 03:18:39 +0000 (expires in 46 days) https://wikitech.wikimedia.org/wiki/HTTPS [18:39:12] RECOVERY - Kafka broker TLS certificate validity on kafka-main2007 is OK: SSL OK - Certificate kafka-main2007.codfw.wmnet valid until 2027-06-04 15:25:00 +0000 (expires in 253 days) https://wikitech.wikimedia.org/wiki/Kafka/Administration%23Renew_TLS_certificate [18:39:12] RECOVERY - Blazegraph process -wdqs-blazegraph- on wdqs2024 is OK: PROCS OK: 1 process with UID = 498 (blazegraph), regex args ^java .* --port 9999 .* blazegraph-service-.*war https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [18:39:12] RECOVERY - haproxy process on cp2044 is OK: PROCS OK: 2 processes with command name haproxy https://wikitech.wikimedia.org/wiki/HAProxy [18:39:14] RECOVERY - Blazegraph process -wdqs-categories- on wdqs2010 is OK: PROCS OK: 1 process with UID = 498 (blazegraph), regex args ^java .* --port 9990 .* blazegraph-service-.*war https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [18:39:15] RECOVERY - Kafka Broker Server #page on kafka-main2008 is OK: PROCS OK: 1 process with command name java, args Kafka /etc/kafka/server.properties https://wikitech.wikimedia.org/wiki/Kafka/Administration [18:39:15] RECOVERY - Blazegraph process -wdqs-categories- on wdqs2014 is OK: PROCS OK: 1 process with UID = 498 (blazegraph), regex args ^java .* --port 9990 .* blazegraph-service-.*war https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [18:39:15] RECOVERY - Blazegraph process -wdqs-blazegraph- on wdqs2010 is OK: PROCS OK: 1 process with UID = 498 (blazegraph), regex args ^java .* --port 9999 .* blazegraph-service-.*war https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [18:39:15] RECOVERY - pybal on lvs2012 is OK: PROCS OK: 1 process with UID = 0 (root), args /usr/sbin/pybal https://wikitech.wikimedia.org/wiki/PyBal [18:39:16] RECOVERY - Kafka Broker Server on kafka-logging2002 is OK: PROCS OK: 1 process with command name java, args Kafka /etc/kafka/server.properties https://wikitech.wikimedia.org/wiki/Kafka/Administration [18:39:22] PROBLEM - Check unit status of httpbb_kubernetes_mw-web_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-web_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [18:39:26] RECOVERY - Host ps1-b8-codfw is UP: PING OK - Packet loss = 0%, RTA = 31.20 ms [18:39:28] !ack [18:39:32] RECOVERY - Host ps1-a6-codfw is UP: PING OK - Packet loss = 0%, RTA = 30.91 ms [18:39:36] RECOVERY - Blazegraph Port for wdqs-blazegraph on wdqs2007 is OK: TCP OK - 0.000 second response time on 127.0.0.1 port 9999 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [18:39:36] RECOVERY - Blazegraph Port for wdqs-blazegraph on wdqs2010 is OK: TCP OK - 0.000 second response time on 127.0.0.1 port 9999 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [18:39:36] RECOVERY - PyBal backends health check on lvs2012 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [18:39:37] RECOVERY - Host ps1-a7-codfw is UP: PING OK - Packet loss = 0%, RTA = 30.89 ms [18:39:37] RECOVERY - Blazegraph Port for wdqs-blazegraph on wdqs2016 is OK: TCP OK - 0.000 second response time on 127.0.0.1 port 9999 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [18:39:38] RECOVERY - Blazegraph Port for wdqs-categories on wdqs2010 is OK: TCP OK - 0.000 second response time on 127.0.0.1 port 9990 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [18:39:39] RECOVERY - Blazegraph Port for wdqs-blazegraph on wdqs2014 is OK: TCP OK - 0.000 second response time on 127.0.0.1 port 9999 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [18:39:39] RECOVERY - Blazegraph process -wdqs-blazegraph- on wdqs2016 is OK: PROCS OK: 1 process with UID = 498 (blazegraph), regex args ^java .* --port 9999 .* blazegraph-service-.*war https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [18:39:40] RECOVERY - Blazegraph process -wdqs-blazegraph- on wdqs2014 is OK: PROCS OK: 1 process with UID = 498 (blazegraph), regex args ^java .* --port 9999 .* blazegraph-service-.*war https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [18:39:40] RECOVERY - Kafka Broker Server on kafka-logging2004 is OK: PROCS OK: 1 process with command name java, args Kafka /etc/kafka/server.properties https://wikitech.wikimedia.org/wiki/Kafka/Administration [18:39:41] RECOVERY - Blazegraph Port for wdqs-categories on wdqs2014 is OK: TCP OK - 0.000 second response time on 127.0.0.1 port 9990 https://wikitech.wikimedia.org/wiki/Wikidata_query_service/Runbook [18:39:42] RECOVERY - haproxy process on cp2045 is OK: PROCS OK: 2 processes with command name haproxy https://wikitech.wikimedia.org/wiki/HAProxy [18:39:42] RECOVERY - Host ps1-a5-codfw is UP: PING OK - Packet loss = 0%, RTA = 31.08 ms [18:39:43] RECOVERY - Host ps1-b7-codfw is UP: PING OK - Packet loss = 0%, RTA = 30.83 ms [18:39:44] RECOVERY - Host ssw1-a8-codfw.mgmt is UP: PING OK - Packet loss = 0%, RTA = 30.78 ms [18:39:45] RECOVERY - Host lsw1-a5-codfw.mgmt is UP: PING OK - Packet loss = 0%, RTA = 30.71 ms [18:39:46] PROBLEM - Check whether ferm is active by checking the default input chain on restbase2036 is CRITICAL: ERROR ferm input drop default policy not set, ferm might not have been started correctly https://wikitech.wikimedia.org/wiki/Monitoring/check_ferm [18:39:46] PROBLEM - Host ns1-v6 is DOWN: PING CRITICAL - Packet loss = 100% [18:39:46] RECOVERY - Host ps1-a8-codfw is UP: PING OK - Packet loss = 0%, RTA = 31.24 ms [18:39:46] RECOVERY - Host lsw1-b7-codfw.mgmt is UP: PING OK - Packet loss = 0%, RTA = 30.73 ms [18:39:46] RECOVERY - check if authdns-update was run after a change was merged to operations/dns.git on dns2004 is OK: The check was skipped as the host is not pooled for authdns-update https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run [18:39:47] RECOVERY - check if authdns-update was run after a change was merged to operations/dns.git on dns2006 is OK: The check was skipped as the host is not pooled for authdns-update https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run [18:39:48] RECOVERY - Host lsw1-b8-codfw.mgmt is UP: PING OK - Packet loss = 0%, RTA = 30.67 ms [18:39:48] RECOVERY - Host lsw1-a6-codfw.mgmt is UP: PING OK - Packet loss = 0%, RTA = 30.59 ms [18:39:48] RECOVERY - Host lsw1-a8-codfw.mgmt is UP: PING OK - Packet loss = 0%, RTA = 30.75 ms [18:39:49] RECOVERY - Host lsw1-a7-codfw.mgmt is UP: PING OK - Packet loss = 0%, RTA = 30.58 ms [18:39:52] RECOVERY - Host ps1-a4-codfw is UP: PING OK - Packet loss = 0%, RTA = 31.56 ms [18:40:07] RECOVERY - HAProxy HTTPS wikipedia25.org ECDSA on cp2045 is OK: SSL OK - Certificate wikipedia25.org contains all required SANs:Certificate wikipedia25.org (ECDSA) valid until 2026-12-03 05:05:02 +0000 (expires in 70 days) https://wikitech.wikimedia.org/wiki/HTTPS [18:40:07] RECOVERY - HAProxy HTTPS wikipedia.org ECDSA on cp2045 is OK: SSL OK - Certificate *.wikipedia.org contains all required SANs:Certificate *.wikipedia.org (ECDSA) valid until 2026-11-03 19:15:40 +0000 (expires in 41 days) https://wikitech.wikimedia.org/wiki/HTTPS [18:40:07] RECOVERY - HAProxy HTTPS wikiworkshop.org ECDSA on cp2045 is OK: SSL OK - Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2026-11-09 03:18:39 +0000 (expires in 46 days) https://wikitech.wikimedia.org/wiki/HTTPS [18:40:08] RECOVERY - Kafka broker TLS certificate validity on kafka-main2008 is OK: SSL OK - Certificate kafka-main2008.codfw.wmnet valid until 2027-06-09 19:00:00 +0000 (expires in 259 days) https://wikitech.wikimedia.org/wiki/Kafka/Administration%23Renew_TLS_certificate [18:40:09] RECOVERY - Kafka broker TLS certificate validity on kafka-logging2002 is OK: SSL OK - Certificate kafka-logging2002.codfw.wmnet valid until 2027-05-01 19:59:00 +0000 (expires in 220 days) https://wikitech.wikimedia.org/wiki/Kafka/Administration%23Renew_TLS_certificate [18:40:09] PROBLEM - Check unit status of httpbb_kubernetes_mw-web-next_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-web-next_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [18:40:16] RECOVERY - PyBal connections to etcd on lvs2013 is OK: OK: 79 connections established with conf2004.codfw.wmnet:4001 (min=79) https://wikitech.wikimedia.org/wiki/PyBal [18:40:16] RECOVERY - PyBal connections to etcd on lvs2012 is OK: OK: 6 connections established with conf2004.codfw.wmnet:4001 (min=6) https://wikitech.wikimedia.org/wiki/PyBal [18:40:22] RECOVERY - Host lsw1-a4-codfw.mgmt is UP: PING OK - Packet loss = 0%, RTA = 30.79 ms [18:40:23] !log taavi@cumin1004 END (PASS) - Cookbook sre.gerrit.localbackup (exit_code=0) Prepare local backup on: gerrit1003.wikimedia.org [18:40:30] RECOVERY - Juniper alarms on cr2-codfw is OK: JNX_ALARMS OK - 0 red alarms, 0 yellow alarms https://wikitech.wikimedia.org/wiki/Network_monitoring%23Juniper_alarm [18:40:33] !log dzahn@cumin2003 START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit2003.wikimedia.org [18:40:34] !log dzahn@cumin2003 END (FAIL) - Cookbook sre.gerrit.localbackup (exit_code=99) Prepare local backup on: gerrit2003.wikimedia.org [18:40:40] !log dzahn@cumin2003 START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit2003.wikimedia.org [18:40:40] !log dzahn@cumin2003 END (FAIL) - Cookbook sre.gerrit.localbackup (exit_code=99) Prepare local backup on: gerrit2003.wikimedia.org [18:42:00] PROBLEM - Host ns1-v4 is DOWN: PING CRITICAL - Packet loss = 100% [18:42:08] RECOVERY - Host re0.cr1-codfw.mgmt is UP: PING OK - Packet loss = 0%, RTA = 30.66 ms [18:42:08] RECOVERY - Host re0.cr2-codfw.mgmt is UP: PING OK - Packet loss = 0%, RTA = 30.67 ms [18:42:08] RECOVERY - Host scs-a1-codfw is UP: PING OK - Packet loss = 0%, RTA = 30.72 ms [18:42:12] PROBLEM - Host ssw1-a8-codfw.mgmt is DOWN: PING CRITICAL - Packet loss = 100% [18:42:44] !log dzahn@cumin2003 START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit2003.wikimedia.org [18:42:44] !log dzahn@cumin2003 END (FAIL) - Cookbook sre.gerrit.localbackup (exit_code=99) Prepare local backup on: gerrit2003.wikimedia.org [18:42:55] !log sukhe@puppetserver1001 conftool action : set/pooled=yes; selector: dc=codfw,cluster=dnsbox,service=authdns-update [18:43:02] !log taavi@cumin1004 START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit2003.wikimedia.org [18:43:09] !log sukhe@dns1004 START - running authdns-update [18:43:36] PROBLEM - MariaDB Replica Lag: pc1 on pc2021 is CRITICAL: CRITICAL slave_sql_lag could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:43:37] PROBLEM - MariaDB Replica Lag: s6 #page on db2224 is CRITICAL: CRITICAL slave_sql_lag could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:43:41] PROBLEM - MariaDB Replica Lag: x3 #page on db2244 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 2010.05 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:43:42] PROBLEM - MariaDB Replica Lag: x3 #page on db2242 is CRITICAL: CRITICAL slave_sql_lag could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:43:42] PROBLEM - MariaDB Replica Lag: s5 #page on db2228 is CRITICAL: CRITICAL slave_sql_lag could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:43:43] PROBLEM - MariaDB Replica Lag: x3 #page on db2241 is CRITICAL: CRITICAL slave_sql_lag could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:43:51] RECOVERY - Check unit status of push_cross_cluster_settings_9200 on cirrussearch2093 is OK: OK: Status of the systemd unit push_cross_cluster_settings_9200 https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [18:43:51] RECOVERY - Check unit status of push_cross_cluster_settings_9200 on cirrussearch2081 is OK: OK: Status of the systemd unit push_cross_cluster_settings_9200 https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [18:44:14] PROBLEM - MariaDB Replica Lag: x3 #page on db2187 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 2042.98 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:44:18] PROBLEM - MariaDB Replica Lag: s5 #page on db2213 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 2045.57 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:44:19] PROBLEM - MariaDB Replica Lag: s5 #page on db2223 is CRITICAL: CRITICAL slave_sql_lag could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:44:19] PROBLEM - MariaDB Replica Lag: s5 #page on db2211 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 2045.62 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:44:20] PROBLEM - MariaDB Replica Lag: s5 #page on db2171 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 2047.00 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:44:21] PROBLEM - MariaDB Replica Lag: s6 #page on db2158 is CRITICAL: CRITICAL slave_sql_lag could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:44:22] PROBLEM - MariaDB Replica Lag: s4 #page on db2155 is CRITICAL: CRITICAL slave_sql_lag could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:44:23] PROBLEM - MariaDB Replica Lag: s6 #page on db2180 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 2047.56 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:44:24] PROBLEM - MariaDB Replica Lag: s5 #page on db2157 is CRITICAL: CRITICAL slave_sql_lag could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:44:26] RECOVERY - Check unit status of sync-puppet-volatile on puppetserver2001 is OK: OK: Status of the systemd unit sync-puppet-volatile https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [18:44:29] PROBLEM - MariaDB Replica Lag: s6 #page on db2229 is CRITICAL: CRITICAL slave_sql_lag could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:44:30] PROBLEM - MariaDB Replica Lag: s6 #page on db2214 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 2058.06 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:44:31] PROBLEM - MariaDB Replica Lag: s6 #page on db2193 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 2058.10 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:44:32] PROBLEM - MariaDB Replica Lag: s6 #page on db2217 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 2058.14 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:44:34] PROBLEM - Check whether ferm is active by checking the default input chain on sessionstore2005 is CRITICAL: ERROR ferm input drop default policy not set, ferm might not have been started correctly https://wikitech.wikimedia.org/wiki/Monitoring/check_ferm [18:44:35] !ack [18:44:36] 8547 (ACKED) db2180 (paged)/MariaDB Replica IO: s6 (paged) [18:44:36] 8548 (ACKED) db2214 (paged)/MariaDB Replica IO: s6 (paged) [18:44:36] 8549 (ACKED) db2156 (paged)/MariaDB Replica SQL: s3 (paged) [18:44:37] 8550 (ACKED) db2244 (paged)/MariaDB Replica IO: x3 (paged) [18:44:37] 8551 (ACKED) db2213 (paged)/MariaDB Replica IO: s5 (paged) [18:44:37] 8552 (ACKED) db2211 (paged)/MariaDB Replica IO: s5 (paged) [18:44:37] 8553 (ACKED) db2241 (paged)/MariaDB Replica IO: x3 (paged) [18:44:38] 8554 (ACKED) db2196 (paged)/MariaDB Replica SQL: x1 (paged) [18:44:38] 8555 (ACKED) db2242 (paged)/MariaDB Replica IO: x3 (paged) [18:44:39] 8556 (ACKED) db2241 (paged)/MariaDB Replica SQL: x3 (paged) [18:44:39] 8557 (ACKED) db2157 (paged)/MariaDB Replica IO: s5 (paged) [18:44:40] 8558 (ACKED) db2158 (paged)/MariaDB Replica SQL: s6 (paged) [18:44:40] 8559 (ACKED) db2154 (paged)/MariaDB Replica SQL: s8 (paged) [18:44:41] 8560 (ACKED) db2153 (paged)/MariaDB Replica SQL: s1 (paged) [18:44:41] 8561 (ACKED) db2175 (paged)/MariaDB Replica SQL: s2 (paged) [18:44:42] 8562 (ACKED) db2158 (paged)/MariaDB Replica IO: s6 (paged) [18:44:42] 8563 (ACKED) db2155 (paged)/MariaDB Replica SQL: s4 (paged) [18:44:43] 8564 (ACKED) db2155 (paged)/MariaDB Replica IO: s4 (paged) [18:44:43] 8565 (ACKED) db2164 (paged)/MariaDB Replica SQL: s8 (paged) [18:44:44] 8566 (ACKED) db2163 (paged)/MariaDB Replica SQL: s8 (paged) [18:44:44] 8567 (ACKED) db2157 (paged)/MariaDB Replica SQL: s5 (paged) [18:44:46] PROBLEM - MariaDB Replica Lag: pc7 on pc2017 is CRITICAL: CRITICAL slave_sql_lag could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:44:46] PROBLEM - MariaDB Replica Lag: pc8 on pc2018 is CRITICAL: CRITICAL slave_sql_lag could not connect https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:44:48] RECOVERY - Host ssw1-a8-codfw.mgmt is UP: PING OK - Packet loss = 0%, RTA = 30.97 ms [18:45:27] !log sukhe@dns1004 END - running authdns-update [18:45:42] RECOVERY - BFD status on ssw1-d8-codfw.mgmt is OK: UP: 18 AdminDown: 0 Down: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [18:45:42] RECOVERY - BFD status on ssw1-d1-codfw.mgmt is OK: UP: 18 AdminDown: 0 Down: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23BFD_status [18:45:52] RECOVERY - Check unit status of push_cross_cluster_settings_9400 on cirrussearch2073 is OK: OK: Status of the systemd unit push_cross_cluster_settings_9400 https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [18:45:52] RECOVERY - Check unit status of push_cross_cluster_settings_9600 on cirrussearch2076 is OK: OK: Status of the systemd unit push_cross_cluster_settings_9600 https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [18:45:52] RECOVERY - Check unit status of push_cross_cluster_settings_9200 on cirrussearch2111 is OK: OK: Status of the systemd unit push_cross_cluster_settings_9200 https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [18:46:02] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1021.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1016.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [18:46:04] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1018.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [18:46:11] !ack [18:46:11] All incidents are already acked. [18:47:02] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [18:47:04] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [18:47:36] RECOVERY - Check unit status of push_cross_cluster_settings_9400 on cirrussearch2092 is OK: OK: Status of the systemd unit push_cross_cluster_settings_9400 https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [18:47:36] RECOVERY - Check unit status of push_cross_cluster_settings_9400 on cirrussearch2100 is OK: OK: Status of the systemd unit push_cross_cluster_settings_9400 https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state [18:48:14] RECOVERY - PyBal backends health check on lvs2014 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [18:48:38] RECOVERY - PyBal backends health check on lvs2013 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [18:49:56] RECOVERY - Host ssw1-a8-codfw IPv6 is UP: PING OK - Packet loss = 0%, RTA = 33.43 ms [18:49:56] RECOVERY - Host ssw1-a8-codfw is UP: PING OK - Packet loss = 0%, RTA = 36.29 ms [18:50:08] !log taavi@cumin1004 END (PASS) - Cookbook sre.gerrit.localbackup (exit_code=0) Prepare local backup on: gerrit2003.wikimedia.org [18:50:49] RECOVERY - mysqld processes #page on pc2015 is OK: PROCS OK: 1 process with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [18:51:04] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1021.eqiad.wmnet, wdqs1020.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [18:51:07] RECOVERY - MariaDB read only pc5 #page on pc2015 is OK: Version 10.11.16-MariaDB-log, Uptime 60s, read_only: False, event_scheduler: True, 2868.04 QPS, connection latency: 0.024890s, query latency: 0.000937s https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [18:51:07] RECOVERY - MariaDB Event Scheduler pc5 on pc2015 is OK: Version 10.11.16-MariaDB-log, Uptime 60s, read_only: False, event_scheduler: True, 2702.27 QPS, connection latency: 0.020890s, query latency: 0.000919s https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [18:51:13] !ack [18:51:13] All incidents are already acked. [18:51:18] RECOVERY - MariaDB Replica SQL: pc5 on pc2015 is OK: OK slave_sql_state Slave_SQL_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:51:20] RECOVERY - MariaDB Replica IO: pc5 on pc2015 is OK: OK slave_io_state Slave_IO_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:52:13] RECOVERY - mysqld processes #page on pc2021 is OK: PROCS OK: 1 process with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [18:52:14] RECOVERY - MariaDB Replica IO: pc1 on pc2021 is OK: OK slave_io_state Slave_IO_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:52:19] RECOVERY - mysqld processes #page on pc2022 is OK: PROCS OK: 1 process with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [18:52:38] RECOVERY - MariaDB Event Scheduler pc1 on pc2021 is OK: Version 10.11.18-MariaDB-log, Uptime 56s, read_only: False, event_scheduler: True, 4460.16 QPS, connection latency: 0.011146s, query latency: 0.000230s https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [18:52:39] RECOVERY - MariaDB read only pc1 #page on pc2021 is OK: Version 10.11.18-MariaDB-log, Uptime 56s, read_only: False, event_scheduler: True, 4472.18 QPS, connection latency: 0.010744s, query latency: 0.000207s https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [18:52:42] RECOVERY - MariaDB Replica SQL: pc1 on pc2021 is OK: OK slave_sql_state Slave_SQL_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:52:48] RECOVERY - mysqld processes on es2035 is OK: PROCS OK: 1 process with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [18:52:51] RECOVERY - MariaDB read only pc2 #page on pc2022 is OK: Version 10.11.16-MariaDB-log, Uptime 42s, read_only: False, event_scheduler: True, 3391.10 QPS, connection latency: 0.013240s, query latency: 0.000263s https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [18:52:51] RECOVERY - MariaDB Event Scheduler pc2 on pc2022 is OK: Version 10.11.16-MariaDB-log, Uptime 42s, read_only: False, event_scheduler: True, 3386.52 QPS, connection latency: 0.009854s, query latency: 0.000221s https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [18:52:51] PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 124418008 and 10 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [18:53:04] RECOVERY - MariaDB Events es6 on es2035 is OK: OK - All 4 events in ops database are ENABLED https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [18:53:10] RECOVERY - MariaDB read only es6 on es2035 is OK: Version 10.11.16-MariaDB-log, Uptime 41s, read_only: True, event_scheduler: True, 7.52 QPS, connection latency: 0.025355s, query latency: 3.375049s https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [18:53:10] RECOVERY - MariaDB Event Scheduler es6 on es2035 is OK: Version 10.11.16-MariaDB-log, Uptime 41s, read_only: True, event_scheduler: True, 7.71 QPS, connection latency: 0.018794s, query latency: 3.360952s https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [18:53:18] RECOVERY - MariaDB Replica SQL: pc2 on pc2022 is OK: OK slave_sql_state Slave_SQL_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:53:18] RECOVERY - MariaDB Replica IO: pc2 on pc2022 is OK: OK slave_io_state Slave_IO_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:53:42] RECOVERY - MariaDB Events es6 on es2036 is OK: OK - All 4 events in ops database are ENABLED https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [18:53:42] RECOVERY - mysqld processes on pc2018 is OK: PROCS OK: 1 process with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [18:53:44] RECOVERY - MariaDB Event Scheduler pc7 on pc2017 is OK: Version 10.11.16-MariaDB-log, Uptime 41s, read_only: False, event_scheduler: True, 2553.76 QPS, connection latency: 0.019779s, query latency: 0.000954s https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [18:53:45] RECOVERY - MariaDB read only pc7 #page on pc2017 is OK: Version 10.11.16-MariaDB-log, Uptime 41s, read_only: False, event_scheduler: True, 2550.23 QPS, connection latency: 0.021641s, query latency: 0.000772s https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [18:53:48] RECOVERY - mysqld processes on es2036 is OK: PROCS OK: 1 process with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [18:53:49] RECOVERY - mysqld processes #page on pc2017 is OK: PROCS OK: 1 process with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [18:53:50] RECOVERY - MariaDB Replica IO: es6 #page on es2035 is OK: OK slave_io_state Slave_IO_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:53:50] RECOVERY - MariaDB Replica SQL: pc7 on pc2017 is OK: OK slave_sql_state not a slave https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:53:50] RECOVERY - MariaDB Replica Lag: pc7 on pc2017 is OK: OK slave_sql_lag not a slave https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:53:50] RECOVERY - MariaDB Replica IO: pc7 on pc2017 is OK: OK slave_io_state not a slave https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:54:04] RECOVERY - mysqld processes on es2037 is OK: PROCS OK: 1 process with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [18:54:06] RECOVERY - MariaDB Event Scheduler es6 on es2037 is OK: Version 10.11.16-MariaDB-log, Uptime 58s, read_only: True, event_scheduler: True, 12.18 QPS, connection latency: 0.016351s, query latency: 0.000679s https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [18:54:06] RECOVERY - MariaDB read only es6 on es2036 is OK: Version 10.11.16-MariaDB-log, Uptime 62s, read_only: True, event_scheduler: True, 24.16 QPS, connection latency: 0.020215s, query latency: 0.000688s https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [18:54:06] RECOVERY - MariaDB Event Scheduler es6 on es2036 is OK: Version 10.11.16-MariaDB-log, Uptime 62s, read_only: True, event_scheduler: True, 28.34 QPS, connection latency: 0.021149s, query latency: 0.001028s https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [18:54:12] RECOVERY - pt-heartbeat-wikimedia process on pc2021 is OK: PROCS OK: 1 process with args pt-heartbeat-wikimedia https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23pt-heartbeat [18:54:16] RECOVERY - MariaDB Events es6 on es2037 is OK: OK - All 2 events in ops database are ENABLED https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [18:54:17] RECOVERY - mysqld processes #page on db2251 is OK: PROCS OK: 1 process with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [18:54:19] RECOVERY - MariaDB Replica SQL: es6 #page on es2035 is OK: OK slave_sql_state Slave_SQL_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:54:20] RECOVERY - pt-heartbeat-wikimedia process on pc2022 is OK: PROCS OK: 1 process with args pt-heartbeat-wikimedia https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23pt-heartbeat [18:54:22] !ack [18:54:23] All incidents are already acked. [18:54:38] RECOVERY - MariaDB read only pc8 on pc2018 is OK: Version 10.11.16-MariaDB-log, Uptime 63s, read_only: False, event_scheduler: True, 991.80 QPS, connection latency: 0.019650s, query latency: 0.000967s https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [18:54:39] RECOVERY - MariaDB Replica SQL: es6 #page on es2036 is OK: OK slave_sql_state Slave_SQL_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:54:43] RECOVERY - MariaDB read only es6 #page on es2037 is OK: Version 10.11.16-MariaDB-log, Uptime 95s, read_only: True, event_scheduler: True, 12.38 QPS, connection latency: 0.020585s, query latency: 0.000642s https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [18:54:44] RECOVERY - MariaDB Replica IO: es6 #page on es2047 is OK: OK slave_io_state Slave_IO_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:54:44] RECOVERY - MariaDB Replica IO: pc8 on pc2018 is OK: OK slave_io_state not a slave https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:54:44] RECOVERY - MariaDB Event Scheduler pc8 on pc2018 is OK: Version 10.11.16-MariaDB-log, Uptime 69s, read_only: False, event_scheduler: True, 1081.57 QPS, connection latency: 0.031786s, query latency: 0.000973s https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [18:54:49] RECOVERY - MariaDB Replica IO: es6 #page on es2037 is OK: OK slave_io_state Slave_IO_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:54:49] RECOVERY - MariaDB Replica SQL: pc8 on pc2018 is OK: OK slave_sql_state not a slave https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:54:49] RECOVERY - MariaDB Replica Lag: pc8 on pc2018 is OK: OK slave_sql_lag not a slave https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:54:51] RECOVERY - MariaDB read only ms1 #page on db2251 is OK: Version 10.11.18-MariaDB-log, Uptime 39s, read_only: False, event_scheduler: True, 8783.16 QPS, connection latency: 0.011377s, query latency: 0.000212s https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [18:54:51] RECOVERY - MariaDB Event Scheduler ms1 on db2251 is OK: Version 10.11.18-MariaDB-log, Uptime 39s, read_only: False, event_scheduler: True, 8937.53 QPS, connection latency: 0.009861s, query latency: 0.000302s https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [18:55:04] RECOVERY - MariaDB Replica IO: ms1 on db2251 is OK: OK slave_io_state Slave_IO_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:55:16] RECOVERY - MariaDB Replica Lag: ms1 on db2251 is OK: OK slave_sql_lag Replication lag: 0.25 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:55:16] RECOVERY - MariaDB Replica SQL: ms1 on db2251 is OK: OK slave_sql_state Slave_SQL_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:55:20] RECOVERY - mysqld processes on db2204 is OK: PROCS OK: 1 process with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [18:55:20] RECOVERY - mysqld processes on db2203 is OK: PROCS OK: 1 process with command name mysqld https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting [18:55:21] RECOVERY - MariaDB Replica IO: s1 #page on db2173 is OK: OK slave_io_state Slave_IO_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:55:22] RECOVERY - MariaDB Replica IO: s1 #page on db2170 is OK: OK slave_io_state Slave_IO_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:55:23] PROBLEM - MariaDB Replica SQL: es6 #page on es2037 is CRITICAL: CRITICAL slave_sql_state Slave_SQL_Running: No, Errno: 1062, Errmsg: Could not execute Write_rows_v1 event on table commonswiki.blobs_cluster32: Duplicate entry 41860891 for key PRIMARY, Error_code: 1062: handler error HA_ERR_FOUND_DUPP_KEY: the events master log es1037-bin.005975, end_log_pos 208686070 https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Re [18:55:23] RECOVERY - MariaDB Replica IO: es6 #page on es2036 is OK: OK slave_io_state Slave_IO_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:55:24] RECOVERY - MariaDB Replica IO: s1 #page on db2203 is OK: OK slave_io_state Slave_IO_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:55:25] RECOVERY - MariaDB Replica IO: s1 #page on db2174 is OK: OK slave_io_state Slave_IO_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:55:39] RECOVERY - MariaDB Replica IO: s1 #page on db2216 is OK: OK slave_io_state Slave_IO_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:55:43] RECOVERY - MariaDB Replica IO: s1 #page on db2212 is OK: OK slave_io_state Slave_IO_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:55:48] RECOVERY - MariaDB Events s1 on db2203 is OK: OK - All 2 events in ops database are ENABLED https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [18:55:48] RECOVERY - MariaDB Events s2 on db2204 is OK: OK - All 2 events in ops database are ENABLED https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [18:55:50] RECOVERY - MariaDB Event Scheduler s2 on db2204 is OK: Version 10.11.16-MariaDB-log, Uptime 35s, read_only: True, event_scheduler: True, 12.16 QPS, connection latency: 0.020838s, query latency: 0.001080s https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [18:55:50] RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 3098536 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring [18:56:04] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [18:56:07] RECOVERY - MariaDB read only s1 #page on db2203 is OK: Version 10.11.16-MariaDB-log, Uptime 81s, read_only: True, event_scheduler: True, 45.07 QPS, connection latency: 0.016491s, query latency: 0.000902s https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [18:56:08] RECOVERY - MariaDB read only s2 #page on db2204 is OK: Version 10.11.16-MariaDB-log, Uptime 50s, read_only: True, event_scheduler: True, 24.33 QPS, connection latency: 0.023028s, query latency: 0.000747s https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Master_comes_back_in_read_only [18:56:08] RECOVERY - MariaDB Event Scheduler s1 on db2203 is OK: Version 10.11.16-MariaDB-log, Uptime 81s, read_only: True, event_scheduler: True, 45.07 QPS, connection latency: 0.025524s, query latency: 0.001095s https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Event_Scheduler [18:56:19] RECOVERY - MariaDB Replica Lag: es6 #page on es2035 is OK: OK slave_sql_lag Replication lag: 0.00 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:56:23] RECOVERY - MariaDB Replica Lag: es6 #page on es2036 is OK: OK slave_sql_lag Replication lag: 0.00 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:56:37] RECOVERY - MariaDB Replica IO: s2 #page on db2204 is OK: OK slave_io_state Slave_IO_Running: Yes https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:56:40] RECOVERY - pt-heartbeat-wikimedia process on es2037 is OK: PROCS OK: 1 process with args pt-heartbeat-wikimedia https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23pt-heartbeat [18:56:43] RECOVERY - MariaDB Replica Lag: es6 #page on es2047 is OK: OK slave_sql_lag Replication lag: 0.00 seconds https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response [18:58:13] RESOLVED: [2x] JobUnavailable: Reduced availability for job gerrit in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [18:58:13] !log taavi@cumin1004 START - Cookbook sre.dns.wipe-cache gerrit.discovery.wmnet on all recursors [18:58:17] !log taavi@cumin1004 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) gerrit.discovery.wmnet on all recursors [18:58:20] RECOVERY - pt-heartbeat-wikimedia process on db2205 is OK: PROCS OK: 1 process with args pt-heartbeat-wikimedia https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23pt-heartbeat [18:58:34] FIRING: MediaWikiLatencyExceeded: p75 latency high: codfw mw-api-int releases routed via main (k8s) 2.5s - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=55&var-dc=codfw%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-api-int&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [18:59:41] FIRING: [10x] CertAlmostExpired: gNMI TLS certificate for lsw1-a2-codfw.mgmt.codfw.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [18:59:42] !log sukhe@cumin1004 DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 6:00:00 on 980 hosts with reason: power is still coming back on [18:59:42] RECOVERY - pt-heartbeat-wikimedia process on pc2015 is OK: PROCS OK: 1 process with args pt-heartbeat-wikimedia https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23pt-heartbeat [18:59:53] FIRING: KeyholderUnarmed: 1 unarmed Keyholder key(s) on netmon2002:9100 - https://wikitech.wikimedia.org/wiki/Keyholder - TODO - https://alerts.wikimedia.org/?q=alertname%3DKeyholderUnarmed [19:00:02] FIRING: RedisInstanceDown: Redis instance down rdb-lock2001:16378 redis_lock - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_lock - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_lock&var-instance=rdb-lock2001:16378 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [19:00:07] FIRING: RedisInstanceDown: Redis instance down rdb2013:16380 redis_misc - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_misc - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_misc&var-instance=rdb2013:16380 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [19:01:18] 10ops-eqiad, 06SRE, 06Collaboration-Services, 06DC-Ops: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12356894 (10VRiley-WMF) I was able to get Zuul1005 to reimage. Check it out and see if that works for you [19:02:20] FIRING: [7x] WidespreadPuppetFailure: Puppet has failed in codfw - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [19:02:32] RESOLVED: GerritHAProxyServiceUnavailable: Gerrit tcp-proxy (HAProxy) service gerrit_ssh is DOWN in eqiad - https://wikitech.wikimedia.org/wiki/Gerrit/Operations#GerritHAProxyServiceUnavailable - grafana.wikimedia.org/d/459365f6-df37-48d6-8142-82b22c1875e7/gerrit-tcp-proxy?viewPanel=panel-15 - https://alerts.wikimedia.org/?q=alertname%3DGerritHAProxyServiceUnavailable [19:02:43] RESOLVED: [2x] GerritHAProxyBackendUnavailable: Gerrit backend is unavilable for tcp-proxy (HAProxy) gerrit_ssh - https://wikitech.wikimedia.org/wiki/Gerrit/Operations#GerritHAProxyBackendUnavailable - grafana.wikimedia.org/d/459365f6-df37-48d6-8142-82b22c1875e7/gerrit-tcp-proxy?viewPanel=panel-15 - https://alerts.wikimedia.org/?q=alertname%3DGerritHAProxyBackendUnavailable [19:02:53] FIRING: [11x] SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://grafana.wikimedia.org/d/_77ik484k/openapi-swagger-endpoint-state?var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [19:03:30] FIRING: [10x] BFDdown: BFD session down between cr1-drmrs and 185.15.58.154 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [19:03:39] FIRING: [36x] ProbeDown: Service doc1004.eqiad.wmnet:443 has failed probes (http_doc1004_eqiad_wmnet_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:04:08] FIRING: [209x] ProbeDown: Service aqs2001-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:04:13] FIRING: RedisInstanceDown: Redis instance down arclamp2001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_arclamp&var-instance=arclamp2001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [19:05:02] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1019.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [19:05:04] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1021.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1020.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [19:05:18] FIRING: [323x] KubernetesCalicoDown: aux-k8s-worker2002.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [19:05:30] RESOLVED: [8x] CoreBGPDown: Core BGP session down between cr1-drmrs and cr1-codfw (185.15.58.154) - group Confed_codfw - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [19:05:59] FIRING: [2x] SwitchCoreInterfaceDown: Switch core interface down - ssw1-d8-codfw:et-0/0/30 (Core: ssw1-a8-codfw:et-0/0/30) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [19:06:04] !ack [19:06:04] All incidents are already acked. [19:06:25] FIRING: [7x] CoreBGPDown: Core BGP session down between cr1-magru and cr1-codfw (195.200.68.138) - group Confed_codfw - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [19:07:02] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [19:07:04] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [19:07:06] FIRING: [5x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:fxp0 () - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [19:07:15] FIRING: [2x] KafkaMirrorMakerAvgMsgConsumeRate: Kafka MirrorMaker main-eqiad-to-main-codfw average message consume rate in last 30m - https://wikitech.wikimedia.org/wiki/Kafka/Administration#MirrorMaker - https://alerts.wikimedia.org/?q=alertname%3DKafkaMirrorMakerAvgMsgConsumeRate [19:07:21] RESOLVED: PfwCoreBGPDown: Fundraising Firewall core BGP session down between pfw1-eqiad and (null) (10.195.0.249) - group VPN - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=eqiad&var-device=pfw1-eqiad:9804&var-bgp_group=VPN&var-bgp_neighbor=(null) - https://alerts.wikimedia.org/?q=alertname%3DPfwCoreBGPDown [19:07:39] FIRING: SLOBudgetBurn: TrafficServer backend combined SLO is below 99.7% target for cache_text in codfw - https://alerts.wikimedia.org/?q=alertname%3DSLOBudgetBurn [19:08:06] FIRING: [6x] CalicoKubeControllersDown: Calico Kubernetes Controllers not running - https://wikitech.wikimedia.org/wiki/Calico#Kube_Controllers - TODO - https://alerts.wikimedia.org/?q=alertname%3DCalicoKubeControllersDown [19:08:07] !log sukhe@puppetserver1001 conftool action : set/pooled=no; selector: dc=codfw,cluster=dnsbox,service=authdns-update [19:08:10] FIRING: [2x] CalicoTyphaDown: Too few (1) calico-typha replicas running - https://wikitech.wikimedia.org/wiki/Calico#Typha" - TODO - https://alerts.wikimedia.org/?q=alertname%3DCalicoTyphaDown [19:08:14] RESOLVED: [17x] KubernetesAPILatency: High Kubernetes API latency (GET bgpconfigurations) on k8s@codfw - https://wikitech.wikimedia.org/wiki/Kubernetes - https://alerts.wikimedia.org/?q=alertname%3DKubernetesAPILatency [19:08:38] RESOLVED: WebrequestSampledDown: Benthos metrics for webrequest_sampled are not reported from eqiad and codfw - https://wikitech.wikimedia.org/wiki/Benthos#Benthos_on_centrallog - https://grafana.wikimedia.org/d/V0TSK7O4z/benthos?var-port=4151 - https://alerts.wikimedia.org/?q=alertname%3DWebrequestSampledDown [19:09:09] FIRING: [28x] ProbeDown: Service aux-k8s-ctrl2003:6443 has failed probes (http_aux_k8s_codfw_kube_apiserver_ip4) #page - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:09:22] FIRING: [36x] ProbeDown: Service doc1004.eqiad.wmnet:443 has failed probes (http_doc1004_eqiad_wmnet_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:09:25] !ack [19:09:26] All incidents are already acked. [19:09:28] FIRING: [12x] CoreBGPDown: Core BGP session down between cr1-drmrs and cr1-codfw (185.15.58.154) - group Confed_codfw - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [19:09:32] FIRING: [7x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:fxp0 () - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [19:09:36] FIRING: [2x] NetworkDeviceAlarmActive: Alarm active on cr2-codfw - https://wikitech.wikimedia.org/wiki/Network_monitoring#Juniper_alarm - https://alerts.wikimedia.org/?q=alertname%3DNetworkDeviceAlarmActive [19:09:41] FIRING: [323x] KubernetesCalicoDown: aux-k8s-worker2002.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [19:09:44] FIRING: [212x] ProbeDown: Service aqs2001-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:10:02] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [19:10:02] RESOLVED: [4x] ProbeDown: Service idp2005:443 has failed probes (http_idp_wikimedia_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:10:27] FIRING: [20x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:2:208:80:153:48 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [19:10:41] RESOLVED: [2x] ProbeDown: Service gerrit2003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:10:49] !log cdobbins@cumin1004 conftool action : set/pooled=yes; selector: name=ncredir6001.* [19:10:50] FIRING: [28x] ProbeDown: Service aux-k8s-ctrl2003:6443 has failed probes (http_aux_k8s_codfw_kube_apiserver_ip4) #page - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:10:50] !log taavi@cumin1004 START - Cookbook sre.gerrit.read-only-toggle from gerrit1003.wikimedia.org [19:10:59] !log taavi@cumin1004 END (PASS) - Cookbook sre.gerrit.read-only-toggle (exit_code=0) from gerrit1003.wikimedia.org [19:11:02] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [19:11:56] FIRING: [6x] PingLossPercent: Blackbox probe packet loss 43.83% from codfw to drmrs - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [19:14:04] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [19:14:49] !log taavi@dns1004 START - running authdns-update [19:15:02] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1021.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1012.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [19:16:02] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [19:16:04] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [19:17:15] !log taavi@dns1004 END - running authdns-update [19:20:02] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [19:20:04] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [19:20:13] !log sukhe@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on 979 hosts with reason: power is still coming back on [19:21:02] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [19:21:04] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [19:24:04] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1021.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1020.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [19:27:40] RECOVERY - Host dbprov2004 is UP: PING OK - Packet loss = 0%, RTA = 31.66 ms [19:29:02] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1016.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [19:32:02] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [19:32:14] FIRING: [60x] SystemdUnitFailed: ferm.service on aux-k8s-worker2006:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:32:40] FIRING: MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-web - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?panelId=18&fullscreen&orgId=1&var-datasource=eqiad%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [19:32:54] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply [19:33:03] !log rebooting arclamp2001.codfw.wmnet [19:33:04] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [19:33:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:33:50] !log lerickson@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply [19:34:17] !log sukhe@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp2059.codfw.wmnet with OS trixie [19:34:23] 10ops-codfw, 06SRE, 06DC-Ops: Q1:rack/setup/install sretest2013 (configF test host) - https://phabricator.wikimedia.org/T436691#12357017 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin1004 for host cp2059.codfw.wmnet with OS trixie completed: - cp2059 (**WARN**) - Rem... [19:34:48] FIRING: SystemdUnitFailed: kubelet.service on ml-serve2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:34:53] FIRING: RedisReplicaDown: Redis replica down rdb2014:16380 redis_misc - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_misc - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_misc&var-instance=rdb2014:16380 - https://alerts.wikimedia.org/?q=alertname%3DRedisReplicaDown [19:35:24] RESOLVED: MediaWikiLatencyExceeded: p75 latency high: codfw mw-api-int releases routed via main (k8s) 2.5s - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook#Average_latency_exceeded - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=55&var-dc=codfw%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-api-int&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiLatencyExceeded [19:36:04] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1020.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1012.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [19:36:05] (03PS1) 10Dzahn: switch gerrit-replica discovery record to gerrit2003 [dns] - 10https://gerrit.wikimedia.org/r/1344363 [19:36:40] (03PS2) 10Dzahn: switch gerrit-replica discovery record to gerrit2003 [dns] - 10https://gerrit.wikimedia.org/r/1344363 [19:36:42] RECOVERY - Host arclamp2001 is UP: PING OK - Packet loss = 0%, RTA = 31.76 ms [19:37:04] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [19:37:06] (03CR) 10LSobanski: [C:03+1] switch gerrit-replica discovery record to gerrit2003 [dns] - 10https://gerrit.wikimedia.org/r/1344363 (owner: 10Dzahn) [19:37:06] (03PS3) 10Scott French: wmnet: move remaining codfw-associated etcd client SRV records to eqiad [dns] - 10https://gerrit.wikimedia.org/r/1344348 [19:37:11] FIRING: [24x] CertAlmostExpired: gNMI TLS certificate for cloudsw1-b1-codfw.mgmt.codfw.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [19:37:20] FIRING: [16x] JobUnavailable: Reduced availability for job blazegraph in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [19:37:39] FIRING: [2x] KeyholderUnarmed: 2 unarmed Keyholder key(s) on cumin2003:9100 - https://wikitech.wikimedia.org/wiki/Keyholder - TODO - https://alerts.wikimedia.org/?q=alertname%3DKeyholderUnarmed [19:37:44] RESOLVED: RedisInstanceDown: Redis instance down rdb-lock2001:16378 redis_lock - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_lock - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_lock&var-instance=rdb-lock2001:16378 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown [19:39:46] FIRING: [37x] CirrusSearchNodeIndexingNotIncreasing: OpenSearch instance cirrussearch2062-production-search-codfw is not indexing - https://wikitech.wikimedia.org/wiki/Search/OpenSearch/Administration#Alerts/Dashboards - https://grafana.wikimedia.org/d/JLK3I_siz/elasticsearch-indexing?orgId=1&from=now-3d&to=now&viewPanel=57 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchNodeIndexingNotIncreasing [19:40:01] FIRING: KubernetesRsyslogDown: rsyslog on ml-serve2002:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=ml-serve2002 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [19:40:05] FIRING: [83x] KubernetesRsyslogDown: rsyslog on wikikube-worker2014:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [19:40:18] FIRING: [15x] SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://grafana.wikimedia.org/d/_77ik484k/openapi-swagger-endpoint-state?var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [19:40:32] FIRING: [15x] SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://grafana.wikimedia.org/d/_77ik484k/openapi-swagger-endpoint-state?var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [19:40:55] FIRING: [17x] JobUnavailable: Reduced availability for job blazegraph in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [19:41:08] FIRING: [34x] ProbeDown: Service doc1004.eqiad.wmnet:443 has failed probes (http_doc1004_eqiad_wmnet_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:41:28] FIRING: [213x] ProbeDown: Service aqs2001-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:42:33] FIRING: [323x] KubernetesCalicoDown: aux-k8s-worker2002.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [19:42:57] FIRING: [5x] SwitchCoreInterfaceDown: Switch core interface down - ssw1-d1-codfw:et-0/0/31 (Core: cr1-codfw:et-1/1/2) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [19:43:30] FIRING: [16x] CoreBGPDown: Core BGP session down between cr1-drmrs and cr1-codfw (185.15.58.154) - group Confed_codfw - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [19:43:41] FIRING: CirrusSearchMoreLikeLatencyTooHigh: CirrusSearch more_like 95th percentiles latency is too high (mw@codfw to eqiad) - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://grafana.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchMoreLikeLatencyTooHigh [19:43:58] FIRING: [7x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:fxp0 () - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [19:44:07] FIRING: [2x] KafkaMirrorMakerAvgMsgConsumeRate: Kafka MirrorMaker main-eqiad-to-main-codfw average message consume rate in last 30m - https://wikitech.wikimedia.org/wiki/Kafka/Administration#MirrorMaker - https://alerts.wikimedia.org/?q=alertname%3DKafkaMirrorMakerAvgMsgConsumeRate [19:44:11] RESOLVED: SLOBudgetBurn: TrafficServer backend combined SLO is below 99.7% target for cache_text in codfw - https://alerts.wikimedia.org/?q=alertname%3DSLOBudgetBurn [19:44:32] The alerts are silenced, not sure why they're alerting... [19:44:33] RESOLVED: CirrusProducerFlinkJobNotRunning: cirrus_streaming_updater_producer in codfw (k8s) is not running - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?var-datasource=codfw+prometheus%2Fk8s&var-namespace=cirrus-streaming-updater&var-helm_release=producer - https://alerts.wikimedia.org/?q=alertname%3DCirrusProducerFlinkJobNotRunning [19:44:46] FIRING: [27x] ProbeDown: Service aux-k8s-ctrl2003:6443 has failed probes (http_aux_k8s_codfw_kube_apiserver_ip4) #page - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:44:55] FIRING: [29x] ProbeDown: Service doc1004.eqiad.wmnet:443 has failed probes (http_doc1004_eqiad_wmnet_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:45:03] FIRING: [7x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:fxp0 () - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [19:45:16] FIRING: [323x] KubernetesCalicoDown: aux-k8s-worker2002.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [19:45:22] FIRING: [219x] ProbeDown: Service aqs2001-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:45:33] FIRING: CirrusProducerFlinkJobNotRunning: cirrus_streaming_updater_producer in codfw (k8s) is not running - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?var-datasource=codfw+prometheus%2Fk8s&var-namespace=cirrus-streaming-updater&var-helm_release=producer - https://alerts.wikimedia.org/?q=alertname%3DCirrusProducerFlinkJobNotRunning [19:46:00] RESOLVED: EtcdReplicationDown: etcd replication down on conf2005:8000 #page - https://wikitech.wikimedia.org/wiki/Etcd/Main_cluster#Replication - TODO - https://alerts.wikimedia.org/?q=alertname%3DEtcdReplicationDown [19:46:05] RESOLVED: [2x] KafkaMirrorMakerAvgMsgConsumeRate: Kafka MirrorMaker main-eqiad-to-main-codfw average message consume rate in last 30m - https://wikitech.wikimedia.org/wiki/Kafka/Administration#MirrorMaker - https://alerts.wikimedia.org/?q=alertname%3DKafkaMirrorMakerAvgMsgConsumeRate [19:46:10] RESOLVED: KafkaMirrorMakerAvgMsgProduceRate: Kafka MirrorMaker main-codfw-to-main-eqiad average message produce rate in last 30m - https://wikitech.wikimedia.org/wiki/Kafka/Administration#MirrorMaker - https://grafana.wikimedia.org/d/000000521/kafka-mirrormaker?var-mirror_name=main-codfw-to-main-eqiad - https://alerts.wikimedia.org/?q=alertname%3DKafkaMirrorMakerAvgMsgProduceRate [19:46:57] !log bounce ferm on aux-k8s-worker2006 [19:50:32] (03CR) 10Dzahn: "recheck" [dns] - 10https://gerrit.wikimedia.org/r/1344363 (owner: 10Dzahn) [19:52:17] !log cdobbins@cumin1004 START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie [19:52:38] FIRING: [61x] SystemdUnitFailed: ferm.service on aux-k8s-worker2006:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:52:56] FIRING: [2x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?panelId=18&fullscreen&orgId=1&var-datasource=eqiad%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [19:53:32] FIRING: [2x] NetworkDeviceAlarmActive: Alarm active on cr2-codfw - https://wikitech.wikimedia.org/wiki/Network_monitoring#Juniper_alarm - https://alerts.wikimedia.org/?q=alertname%3DNetworkDeviceAlarmActive [19:53:41] RESOLVED: SystemdUnitFailed: kubelet.service on ml-serve2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:53:54] FIRING: [61x] SystemdUnitFailed: ferm.service on aux-k8s-worker2006:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:54:03] FIRING: [2x] CirrusStreamingUpdaterFlinkJobUnstable: cirrus_streaming_updater_consumer_search_codfw in codfw (k8s) is unstable - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterFlinkJobUnstable [19:54:39] FIRING: RESTGatewayBackendErrorsHigh: rest-gateway: high 5xx errors from mobileapps_cluster in codfw #page - https://wikitech.wikimedia.org/wiki/API_Gateway#How_to_debug_it - https://grafana.wikimedia.org/d/UOH-5IDMz/api-and-rest-gateway?orgId=1&refresh=30s&viewPanel=57&var-datasource=codfw%20prometheus/k8s&var-instance=rest-gateway - https://alerts.wikimedia.org/?q=alertname%3DRESTGatewayBackendErrorsHigh [19:54:49] RESOLVED: CirrusProducerFlinkJobNotRunning: cirrus_streaming_updater_producer in codfw (k8s) is not running - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?var-datasource=codfw+prometheus%2Fk8s&var-namespace=cirrus-streaming-updater&var-helm_release=producer - https://alerts.wikimedia.org/?q=alertname%3DCirrusProducerFlinkJobNotRunning [19:55:02] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1021.eqiad.wmnet, wdqs1020.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [19:55:04] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1021.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1012.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [19:55:23] FIRING: [24x] CertAlmostExpired: gNMI TLS certificate for cloudsw1-b1-codfw.mgmt.codfw.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [19:55:28] FIRING: [17x] JobUnavailable: Reduced availability for job blazegraph in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [19:55:41] FIRING: [41x] CirrusSearchNodeIndexingNotIncreasing: OpenSearch instance cirrussearch2061-production-search-codfw is not indexing - https://wikitech.wikimedia.org/wiki/Search/OpenSearch/Administration#Alerts/Dashboards - https://grafana.wikimedia.org/d/JLK3I_siz/elasticsearch-indexing?orgId=1&from=now-3d&to=now&viewPanel=57 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchNodeIndexingNotIncreasing [19:55:56] RESOLVED: KubernetesRsyslogDown: rsyslog on ml-serve2002:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=ml-serve2002 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [19:56:00] FIRING: [89x] KubernetesRsyslogDown: rsyslog on wikikube-worker2014:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [19:56:04] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [19:56:18] RESOLVED: [15x] SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://grafana.wikimedia.org/d/_77ik484k/openapi-swagger-endpoint-state?var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures [19:56:43] RESOLVED: [11x] ProbeDown: Service doc1004.eqiad.wmnet:443 has failed probes (http_doc1004_eqiad_wmnet_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:56:57] FIRING: [217x] ProbeDown: Service aqs2001-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:57:02] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [19:57:06] FIRING: [6x] SwitchCoreInterfaceDown: Switch core interface down - lsw1-b4-codfw:et-0/0/54 (Core: ssw1-a8-codfw:et-0/0/10) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [19:57:10] FIRING: [18x] CoreBGPDown: Core BGP session down between cr1-drmrs and cr1-codfw (185.15.58.154) - group Confed_codfw - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [19:57:44] FIRING: [6x] CalicoKubeControllersDown: Calico Kubernetes Controllers not running - https://wikitech.wikimedia.org/wiki/Calico#Kube_Controllers - TODO - https://alerts.wikimedia.org/?q=alertname%3DCalicoKubeControllersDown [19:57:48] RESOLVED: [2x] CalicoTyphaDown: Too few (1) calico-typha replicas running - https://wikitech.wikimedia.org/wiki/Calico#Typha" - TODO - https://alerts.wikimedia.org/?q=alertname%3DCalicoTyphaDown [19:57:57] RESOLVED: [27x] ProbeDown: Service aux-k8s-ctrl2003:6443 has failed probes (http_aux_k8s_codfw_kube_apiserver_ip4) #page - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:58:14] FIRING: [19x] CoreBGPDown: Core BGP session down between cr1-drmrs and cr1-codfw (185.15.58.154) - group Confed_codfw - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [19:58:18] RESOLVED: [2x] NetworkDeviceAlarmActive: Alarm active on cr2-codfw - https://wikitech.wikimedia.org/wiki/Network_monitoring#Juniper_alarm - https://alerts.wikimedia.org/?q=alertname%3DNetworkDeviceAlarmActive [19:58:24] FIRING: [323x] KubernetesCalicoDown: aux-k8s-worker2002.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [19:58:27] FIRING: [217x] ProbeDown: Service aqs2001-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:59:04] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1020.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [19:59:46] FIRING: [61x] SystemdUnitFailed: ferm.service on aux-k8s-worker2006:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:59:52] RESOLVED: [2x] MediaWikiHighErrorRate: Elevated rate of MediaWiki errors - kube-mw-api-ext - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?panelId=18&fullscreen&orgId=1&var-datasource=eqiad%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiHighErrorRate [20:00:05] RoanKattouw, urbanecm, TheresNoTime, kindrobot, and cjming: That opportune time for a UTC late backport window deploy is upon us again. Don't be afraid. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260923T2000). [20:00:05] No Gerrit patches in the queue for this window AFAICS. [20:00:10] RESOLVED: [2x] CirrusStreamingUpdaterFlinkJobUnstable: cirrus_streaming_updater_consumer_search_codfw in codfw (k8s) is unstable - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterFlinkJobUnstable [20:00:14] FIRING: [2x] CirrusStreamingUpdaterFlinkJobUnstable: cirrus_streaming_updater_consumer_search_codfw in codfw (k8s) is unstable - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterFlinkJobUnstable [20:00:31] RESOLVED: RESTGatewayBackendErrorsHigh: rest-gateway: high 5xx errors from mobileapps_cluster in codfw #page - https://wikitech.wikimedia.org/wiki/API_Gateway#How_to_debug_it - https://grafana.wikimedia.org/d/UOH-5IDMz/api-and-rest-gateway?orgId=1&refresh=30s&viewPanel=57&var-datasource=codfw%20prometheus/k8s&var-instance=rest-gateway - https://alerts.wikimedia.org/?q=alertname%3DRESTGatewayBackendErrorsHigh [20:00:35] RESOLVED: [2x] CirrusStreamingUpdaterFlinkJobUnstable: cirrus_streaming_updater_consumer_search_codfw in codfw (k8s) is unstable - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterFlinkJobUnstable [20:00:45] FIRING: [24x] CertAlmostExpired: gNMI TLS certificate for cloudsw1-b1-codfw.mgmt.codfw.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [20:00:49] FIRING: [89x] KubernetesRsyslogDown: rsyslog on wikikube-worker2014:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [20:00:58] FIRING: [217x] ProbeDown: Service aqs2001-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:01:03] FIRING: [323x] KubernetesCalicoDown: aux-k8s-worker2002.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [20:01:12] RESOLVED: [9x] SwitchCoreInterfaceDown: Switch core interface down - lsw1-a4-codfw:et-0/0/54 (Core: ssw1-a8-codfw:et-0/0/3) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Switch_interface_down - https://alerts.wikimedia.org/?q=alertname%3DSwitchCoreInterfaceDown [20:01:16] RESOLVED: [22x] CoreBGPDown: Core BGP session down between cr1-drmrs and cr1-codfw (185.15.58.154) - group Confed_codfw - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown [20:01:22] RESOLVED: CirrusSearchMoreLikeLatencyTooHigh: CirrusSearch more_like 95th percentiles latency is too high (mw@codfw to eqiad) - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://grafana.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchMoreLikeLatencyTooHigh [20:01:38] RESOLVED: [7x] CoreRouterInterfaceDown: Core router interface down - cr1-codfw:fxp0 () - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown [20:01:57] RESOLVED: [6x] CalicoKubeControllersDown: Calico Kubernetes Controllers not running - https://wikitech.wikimedia.org/wiki/Calico#Kube_Controllers - TODO - https://alerts.wikimedia.org/?q=alertname%3DCalicoKubeControllersDown [20:02:01] FIRING: [216x] ProbeDown: Service aqs2001-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:02:45] FIRING: [61x] SystemdUnitFailed: ferm.service on aux-k8s-worker2006:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:03:18] FIRING: CirrusStreamingUpdaterRateTooLow: CirrusSearch update rate from flink-app-consumer-search is critically low - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterRateTooLow [20:03:23] FIRING: [7x] WidespreadPuppetFailure: Puppet has failed in codfw - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [20:03:29] FIRING: [15x] JobUnavailable: Reduced availability for job blazegraph in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [20:03:44] (03CR) 10Dzahn: [V:03+2 C:03+2] switch gerrit-replica discovery record to gerrit2003 [dns] - 10https://gerrit.wikimedia.org/r/1344363 (owner: 10Dzahn) [20:03:46] FIRING: [172x] ProbeDown: Service aqs2001-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:03:59] !log dzahn@dns1004 START - running authdns-update [20:04:15] FIRING: [7x] WidespreadPuppetFailure: Puppet has failed in codfw - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [20:04:23] (03PS1) 10Hashar: gerrit: make replication.config absence obvious [puppet] - 10https://gerrit.wikimedia.org/r/1344369 [20:04:30] FIRING: [99x] KubernetesCalicoDown: kubestage2001.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [20:04:33] FIRING: [170x] ProbeDown: Service aqs2001-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:05:16] (03CR) 10Hashar: "I think that will make it clearer on which hosts replication.config is present or absent ;)" [puppet] - 10https://gerrit.wikimedia.org/r/1344369 (owner: 10Hashar) [20:06:26] !log dzahn@dns1004 END - running authdns-update [20:07:07] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [20:07:16] FIRING: [15x] JobUnavailable: Reduced availability for job blazegraph in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [20:07:22] RESOLVED: [81x] KubernetesRsyslogDown: rsyslog on wikikube-worker2014:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown [20:07:27] FIRING: [7x] WidespreadPuppetFailure: Puppet has failed in codfw - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [20:07:35] FIRING: [10x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:2:208:80:153:48 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [20:08:01] RESOLVED: [65x] KubernetesCalicoDown: kubestage2001.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown [20:08:27] RESOLVED: [7x] WidespreadPuppetFailure: Puppet has failed in codfw - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure [20:08:34] FIRING: [10x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:2:208:80:153:48 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [20:08:38] RESOLVED: [6x] PingLossPercent: Blackbox probe packet loss 6.25% from codfw to drmrs - https://wikitech.wikimedia.org/wiki/Network_monitoring#PingLossPercent - https://alerts.wikimedia.org/?q=alertname%3DPingLossPercent [20:09:21] FIRING: SLOBudgetBurn: Search update lag is below 95% target in codfw - https://alerts.wikimedia.org/?q=alertname%3DSLOBudgetBurn [20:09:33] FIRING: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [20:09:39] CirrusSearch consumer-search@codfw is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=codfw&var-search_cluster=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [20:09:59] FIRING: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [20:10:05] CirrusSearch consumer-search@codfw is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=codfw&var-search_cluster=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [20:10:07] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [20:10:27] FIRING: [10x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:2:208:80:153:48 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [20:10:58] (03PS1) 10DLynch: editcheck-headless: Delay and relax the liveness probe [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344371 (https://phabricator.wikimedia.org/T436689) [20:12:07] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [20:12:35] RESOLVED: [10x] BFDdown: BFD session down between cr1-codfw and 2620:0:860:2:208:80:153:48 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown [20:12:59] Is it okay to do any backports in this deployment window, or does the incident fallout mean it's better avoided? [20:13:18] yeah, do not scap now please :) [20:14:14] o7 [20:15:05] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [20:16:06] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [20:20:06] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1012.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [20:20:29] (03CR) 10Eric Gardner: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344049 (https://phabricator.wikimedia.org/T436692) (owner: 10Eric Gardner) [20:21:06] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1018.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [20:21:48] FIRING: GnmiInterfaceCountersDrop: ... [20:21:54] lsw1-b5-codfw is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?orgId=1&from=now-24h&to=now&var-site=%24__all&var-instance=lsw1-b5-codfw:9804&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [20:22:04] Kemayo: CI is also broken so we can't do backports anyway [20:22:06] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [20:22:06] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [20:22:10] Like, when you +2 a patch, nothing happens [20:22:41] RESOLVED: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [20:22:47] CirrusSearch consumer-search@codfw is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=codfw&var-search_cluster=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [20:23:03] FIRING: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [20:23:08] RoanKattouw: nice. [20:23:09] CirrusSearch consumer-search@codfw is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=codfw&var-search_cluster=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [20:23:16] RESOLVED: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [20:23:24] CirrusSearch consumer-search@codfw is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=codfw&var-search_cluster=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [20:23:36] Kemayo: RoanKattouw CI is experiencing problems because of a datacenter power outage earlier today, team is working on it! [20:23:49] Yeah I figured this was all fallout from the same thing [20:23:51] FIRING: ConfdResourceFailed: confd resource _etc_haproxy_conf.d_tls.cfg.toml has errors - https://wikitech.wikimedia.org/wiki/Confd#Monitoring - https://grafana.wikimedia.org/d/OUJF1VI4k/confd - https://alerts.wikimedia.org/?q=alertname%3DConfdResourceFailed [20:24:27] FIRING: [51x] SystemdUnitFailed: prometheus-node-textfile-export_service_type.service on cumin1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:24:39] RESOLVED: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [20:24:39] (03CR) 10Dzahn: "recheck" [puppet] - 10https://gerrit.wikimedia.org/r/1344369 (owner: 10Hashar) [20:24:45] CirrusSearch consumer-search@codfw is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=codfw&var-search_cluster=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [20:25:06] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1021.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [20:25:06] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [20:25:31] FIRING: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [20:25:37] CirrusSearch consumer-search@codfw is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=codfw&var-search_cluster=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [20:25:41] (03CR) 10Dzahn: "recheck" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341335 (https://phabricator.wikimedia.org/T437635) (owner: 10Dzahn) [20:25:49] FIRING: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [20:25:55] CirrusSearch consumer-search@codfw is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=codfw&var-search_cluster=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [20:26:51] FIRING: [4x] JobUnavailable: Reduced availability for job mysql-dbstore in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [20:27:02] RESOLVED: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [20:27:06] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [20:27:08] CirrusSearch consumer-search@codfw is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=codfw&var-search_cluster=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [20:27:11] RESOLVED: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [20:27:12] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply [20:27:17] CirrusSearch consumer-search@codfw is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=codfw&var-search_cluster=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [20:27:24] !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply [20:27:31] FIRING: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [20:27:36] CirrusSearch consumer-search@codfw is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=codfw&var-search_cluster=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [20:27:39] FIRING: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [20:27:42] Kemayo: RoanKattouw CI should be functional again [20:27:45] CirrusSearch consumer-search@codfw is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=codfw&var-search_cluster=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [20:27:57] Seems to be working for me! [20:27:59] Thanks! [20:28:06] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [20:28:07] (03CR) 10Hashar: "recheck because CI/Zuul was disconnected due to codfw" [puppet] - 10https://gerrit.wikimedia.org/r/1344354 (https://phabricator.wikimedia.org/T439010) (owner: 10BCornwall) [20:28:31] RESOLVED: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [20:28:37] CirrusSearch consumer-search@codfw is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=codfw&var-search_cluster=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [20:28:49] RESOLVED: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [20:28:55] CirrusSearch consumer-search@codfw is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=codfw&var-search_cluster=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [20:29:01] FIRING: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [20:29:07] CirrusSearch consumer-search@codfw is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=codfw&var-search_cluster=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [20:29:29] FIRING: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [20:29:35] CirrusSearch consumer-search@codfw is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=codfw&var-search_cluster=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [20:29:48] (03CR) 10Eric Gardner: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344049 (https://phabricator.wikimedia.org/T436692) (owner: 10Eric Gardner) [20:29:52] RESOLVED: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [20:29:58] CirrusSearch consumer-search@codfw is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=codfw&var-search_cluster=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [20:30:01] RESOLVED: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [20:30:06] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1013.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [20:30:07] CirrusSearch consumer-search@codfw is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=codfw&var-search_cluster=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [20:30:17] jouncebot: nowandnext [20:30:17] For the next 0 hour(s) and 29 minute(s): UTC late backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260923T2000) [20:30:17] In 0 hour(s) and 29 minute(s): Wikifunctions Services UTC Late (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260923T2100) [20:30:26] FIRING: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [20:30:31] CirrusSearch consumer-search@codfw is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=codfw&var-search_cluster=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [20:30:43] FIRING: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [20:30:44] RoanKattouw: Kemayo: CI is fixed now but I don't think the site is yet healthy enough to do any backports [20:30:48] !log rzl@deploy1003 Locking from deployment [ALL REPOSITORIES]: incident recovery in progress T439010 [20:30:49] CirrusSearch consumer-search@codfw is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=codfw&var-search_cluster=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [20:31:06] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1020.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [20:31:07] ^ tried to silence that alert, hopefully it takes. meanwhile im still working on codfw cirrus [20:31:17] wdqs we will let do its thing [20:31:47] RESOLVED: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [20:31:53] CirrusSearch consumer-search@codfw is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=codfw&var-search_cluster=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [20:32:04] RESOLVED: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [20:32:10] CirrusSearch consumer-search@codfw is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=codfw&var-search_cluster=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [20:32:45] FIRING: [2x] Not accepting/receiving prefixes from anycast BGP peer: Alert for device cr1-codfw.wikimedia.org - Not accepting/receiving prefixes from anycast BGP peer - https://alerts.wikimedia.org/?q=alertname%3DNot+accepting%2Freceiving+prefixes+from+anycast+BGP+peer [20:33:05] FIRING: GerritDiskSpaceExhaustionIncoming: Gerrit disk space runway on gerrit1003:/srv is too low - https://wikitech.wikimedia.org/wiki/Gerrit/Operations#TODO - https://grafana.wikimedia.org/d/rYdddlPWk/node-exporter-collaboration-services?var-job=node&var-nodename=gerrit1003&var-node=gerrit1003%3A9100&refresh=1m - https://alerts.wikimedia.org/?q=alertname%3DGerritDiskSpaceExhaustionIncoming [20:33:06] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [20:33:24] FIRING: CirrusStreamingUpdaterSetWeightedTagsTooLow: ... [20:33:30] CirrusSearch consumer-search@codfw is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=codfw&var-search_cluster=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow [20:33:35] FIRING: CirrusStreamingUpdaterClearWeightedTagsTooLow: ... [20:33:40] CirrusSearch consumer-search@codfw is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=codfw&var-search_cluster=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow [20:34:03] FIRING: [3x] JobUnavailable: Reduced availability for job mysql-dbstore in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [20:34:20] FIRING: [3x] JobUnavailable: Reduced availability for job mysql-dbstore in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [20:34:37] (03CR) 10Brouberol: [C:03+2] ceph: add a cookbook to remove an OSD [cookbooks] - 10https://gerrit.wikimedia.org/r/1344201 (https://phabricator.wikimedia.org/T438823) (owner: 10Brouberol) [20:36:46] RESOLVED: GerritDiskSpaceExhaustionIncoming: Gerrit disk space runway on gerrit1003:/srv is too low - https://wikitech.wikimedia.org/wiki/Gerrit/Operations#TODO - https://grafana.wikimedia.org/d/rYdddlPWk/node-exporter-collaboration-services?var-job=node&var-nodename=gerrit1003&var-node=gerrit1003%3A9100&refresh=1m - https://alerts.wikimedia.org/?q=alertname%3DGerritDiskSpaceExhaustionIncoming [20:37:06] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [20:37:39] (03CR) 10Dzahn: [C:03+1] gerrit: make replication.config absence obvious [puppet] - 10https://gerrit.wikimedia.org/r/1344369 (owner: 10Hashar) [20:38:29] !log cdobbins@cumin1004 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie [20:39:37] !log cdobbins@cumin1004 START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie [20:41:33] !log [Cirrus] Been restarting all impacted codfw opensearch hosts one at a time (they didn't rejoin the cluster naturally) [20:41:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:41:58] (03PS1) 10Catrope: HookHandler: Guard against recovery code expiry being null [extensions/OATHAuth] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344388 (https://phabricator.wikimedia.org/T438593) [20:42:19] (03PS1) 10Catrope: HookHandler: Guard against recovery code expiry being null [extensions/OATHAuth] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1344389 (https://phabricator.wikimedia.org/T438593) [20:42:38] (03CR) 10Scott French: "recheck" [dns] - 10https://gerrit.wikimedia.org/r/1344348 (owner: 10Scott French) [20:42:48] (Just lining these up, I won't deploy them until SRE gives the all clear) [20:45:44] FIRING: CirrusStreamingUpdaterUnknownErrors: CirrusSearch consumer-search@codfw is failing write requests because of unknown errors - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterUnknownErrors [20:46:14] (03CR) 10Scott French: [C:03+2] wmnet: move remaining codfw-associated etcd client SRV records to eqiad [dns] - 10https://gerrit.wikimedia.org/r/1344348 (owner: 10Scott French) [20:46:42] !log swfrench@dns1004 START - running authdns-update [20:47:24] (03CR) 10Ssingh: [C:03+1] Switch acme-chief primaries to eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1344354 (https://phabricator.wikimedia.org/T439010) (owner: 10BCornwall) [20:47:27] FIRING: [3x] JobUnavailable: Reduced availability for job atlas_exporter in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [20:49:21] !log swfrench@dns1004 END - running authdns-update [20:54:13] RoanKattouw: what issues are you seeing (or what is the cause of caution?) - as I saw green light in a different channel [20:54:23] Fyi for backporters, train is still on group 0 [20:54:48] Reuven said in Slack 20 minutes ago: We're still recovering but not yet ready to resume deployments, so I've taken the scap lock for now. [20:54:58] I haven't heard anything to the contrary since [20:55:00] !log [Cirrus] Bump codfw cirrussearch shard recoveries from 5 to 10; cluster not serving live traffic so I'm hoping we have headroom to recover faster [20:55:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:55:08] oh so you are dependent on the scap lock - I see - understood [20:55:38] (03PS1) 10Scott French: wmnet: move codfw, eqsin, ulsfo etcd (RO) client SRV records to codfw [dns] - 10https://gerrit.wikimedia.org/r/1344391 [20:55:39] yup scap is still on hold [20:55:44] RESOLVED: CirrusStreamingUpdaterUnknownErrors: CirrusSearch consumer-search@codfw is failing write requests because of unknown errors - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterUnknownErrors [20:56:06] Yes that's right. Gerrit and CI are working so I did the first few steps of the process, but now I have to wait until scap becomes available (which is fine, do what you need to do, the Indonesian user who needs me to fix this bug is probably asleep at this hour anyway) [20:56:06] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1016.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [20:56:06] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1021.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1016.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [20:56:32] RoanKattouw: dang, the user is missing out on all the fun [20:57:06] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [20:57:06] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [20:57:09] log [Cirrus] cirrussearch codfw back to yellow status. active shard pct = 94.51% [20:57:27] !log [Cirrus] cirrussearch codfw back to yellow status. active shard pct = 94.51% [20:57:27] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:00:05] Deploy window Wikifunctions Services UTC Late (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260923T2100) [21:01:14] (03PS13) 10Andrew Bogott: Patch openstack uwsgi ini files to support directing api logs to logstash [puppet] - 10https://gerrit.wikimedia.org/r/1344325 (https://phabricator.wikimedia.org/T421911) [21:03:29] (03CR) 10Andrew Bogott: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1344325 (https://phabricator.wikimedia.org/T421911) (owner: 10Andrew Bogott) [21:06:06] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1021.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [21:07:06] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [21:07:48] !log marostegui@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2160,2232].codfw.wmnet with reason: needs fixing [21:07:58] !log marostegui@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2160,2233].codfw.wmnet with reason: needs fixing [21:08:07] !log marostegui@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2160,2234].codfw.wmnet with reason: needs fixing [21:08:18] !log marostegui@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db[2160,2235].codfw.wmnet with reason: needs fixing [21:18:43] !log reset-failed then restart ceph-mon on moss-be2003 [21:18:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:18:55] !log ceph mgr fail on apus-be2005 [21:18:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:21:59] !log cdobbins@cumin1004 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie [21:22:06] I failed to put my "scap lock" in a tmux, so I need to unlock it briefly but don't nobody get excited :) [21:22:18] !log rzl@deploy1003 Unlocked for deployment [ALL REPOSITORIES]: incident recovery in progress T439010 (duration: 51m 29s) [21:22:18] !log rzl@deploy1003 Locking from deployment [ALL REPOSITORIES]: incident recovery in progress T439010 [21:28:35] 10ops-eqiad, 06SRE, 06Collaboration-Services, 06DC-Ops: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12357301 (10Dzahn) @VRiley-WMF Thank you very much. Looks good to me:) Though I just realized a mistake I made previously for this specific case. So I have to do... [21:29:51] (03PS1) 10Dzahn: site: switch physical zuul hosts to insetup role without nftables [puppet] - 10https://gerrit.wikimedia.org/r/1344395 (https://phabricator.wikimedia.org/T427353) [21:34:21] (03CR) 10Dzahn: [C:03+2] site: switch physical zuul hosts to insetup role without nftables [puppet] - 10https://gerrit.wikimedia.org/r/1344395 (https://phabricator.wikimedia.org/T427353) (owner: 10Dzahn) [21:34:42] (03CR) 10Jdlrobson: [C:03+1] "FYI: I don't think this works on beta cluster FWIW" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344343 (https://phabricator.wikimedia.org/T436692) (owner: 10Eric Gardner) [21:41:22] rzl: so from the looks of the lock, guessing pushing train forward is a no-go today, correct? [21:45:08] thcipriani: he is out but I think that is a yes [21:46:41] ack [21:46:58] thanks! [21:49:13] !log dzahn@cumin2003 START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie [21:50:06] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [21:51:06] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [21:51:24] 10ops-eqiad, 06SRE, 06DC-Ops: hw troubleshooting: disk errors for cephosd1002.eqiad.wmnet /c0/e23/s4 - https://phabricator.wikimedia.org/T438952#12357353 (10wiki_willy) @RobH - can you request quote for the disk? Since we're in the middle of figuring out whether we should extend out the server life cycle fo... [21:52:27] RESOLVED: JobUnavailable: Reduced availability for job mysql-misc in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [21:54:37] thcipriani: yeah, I think odds for tomorrow are pretty good, but if you're okay waiting until then I think we'd prefer it [21:55:10] rzl: thanks and sure! [22:00:05] Deploy window Readers deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260923T2200) [22:07:59] 06SRE, 10SRE-Access-Requests: Requesting access to deployment for dpislaru - https://phabricator.wikimedia.org/T438827#12357398 (10thcipriani) Reason for access makes sense for `deployment` group membership. Approved! --- @DPislaru-WMF some additional info for you: - For MediaWiki, you'll also want access... [22:08:11] Are we likely to do two train pushes tomorrow, or push the group 2 deploy back to Friday? [22:08:45] !log [Cirrus] updater still failing in codfw cirrussearch; i've restarted the directly-impacted hosts but not the others. some bulk updates appear to be getting rejected, going to do some targeted restarts and assess impact before considering a broader operation. first up is `cirrussearch2071.codfw.wmnet` which is not the sole holder of any shards therefore should not plunge the cluster into red status [22:08:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:08:58] !log [Cirrus] (to be clear the cluster is not serving live traffic, but if I can avoid red I will) [22:08:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:10:03] (I ask solely because my PM wants to demo something at wikicon NA, and if group 2 won't be out until Friday I'll need to warn him to redirect where he's demoing.) [22:11:54] Kemayo: I plan to do two deploys tomorrow unless the incident isn't resolved [22:12:08] jeena: Thanks! [22:12:52] [aside for future reference though: all kinds of things can go wrong and block or revert deploys. when lining up a demo with a fixed date, probably better to plan more gap between deployment + demo] [22:13:09] Kemayo: I wonder if your demo could work on patchdemo though? [22:13:22] Schedules gonna schedule. 🤷🏻 [22:14:08] jeena: Yeah, that'd be the fallback. It's mostly that it'll look nicer with a "and this is available on enwiki right now" presentation. [22:14:18] 👍 [22:17:06] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1018.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1012.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [22:20:06] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [22:21:02] (wdqs will complain until codfw is back serving again, nothing to do there. staying on cirrussearch) [22:21:06] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [22:23:36] !log [Cirrus] Alright, I'm getting increasingly convinced that there's no way to restore healthy cluster state without inevitably having to restart sole-shard-holder hosts, which will put the cluster into red status. going to start with just `cirrussearch2105`; I expected red status. silencing alerts first so I don't blow out the channel [22:23:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:24:00] !log [Cirrus] s/expected/expect [22:24:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:24:06] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [22:28:12] !log ryankemper@cumin2003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2105.codfw.wmnet with reason: Codfw chi recovery canary on 2105; temporary service interruption expected (T439010) [22:29:11] !log [Cirrus] proceeding with manual restart of cirrussearch2105; red status expected, hopefully brief but we'll see [22:29:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:32:06] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [22:33:06] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [22:37:14] (03Abandoned) 10Jdlrobson: [DONOTMERGE - demoing deployment process only] Refactor vetags to not be a single saveField [extensions/VisualEditor] (wmf/1.47.0-wmf.19) - 10https://gerrit.wikimedia.org/r/1342046 (https://phabricator.wikimedia.org/T437736) (owner: 10Jdlrobson) [22:41:06] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1020.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [22:41:06] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1011.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [22:41:30] dzahn@cumin2003 reimage (PID 42833) is awaiting input [22:42:06] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [22:42:06] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [22:42:11] (03PS2) 10Jasmine: docroot/wikimedia.org: update assetlink.json for credential sharing on root domain [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1335409 (https://phabricator.wikimedia.org/T427929) [22:43:44] FIRING: RipeAtlasAnchorUnreachable: ipv6 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95140317 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [22:44:58] !log ryankemper@cumin2003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2106.codfw.wmnet with reason: Codfw chi survivor recovery on 2106; temporary red expected (T439010) [22:45:06] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1018.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [22:45:06] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [22:46:24] FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:47:06] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [22:47:06] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [22:47:23] FIRING: [3x] CertAlmostExpired: gNMI TLS certificate for lsw1-e2-codfw.mgmt.codfw.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired [22:47:37] (03PS1) 10Southparkfan: Beta Cluster: Use different Redis instance for LockManager [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344405 (https://phabricator.wikimedia.org/T436480) [22:48:44] FIRING: [2x] RipeAtlasAnchorUnreachable: ipv4 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95133212 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [22:52:44] FIRING: CirrusStreamingUpdaterRateTooLow: CirrusSearch update rate from flink-app-consumer-search is critically low - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterRateTooLow [22:53:44] RESOLVED: [2x] RipeAtlasAnchorUnreachable: ipv4 ping to magru RIPE Atlas anchor: failures over threshold for measurement 95133212 - https://wikitech.wikimedia.org/wiki/Network_monitoring#Atlas_alerts - https://grafana.wikimedia.org/d/K1qm1j-Wz/ripe-atlas?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DRipeAtlasAnchorUnreachable [22:59:06] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1018.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [22:59:30] !log ryankemper@cumin2003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2114.codfw.wmnet with reason: Codfw survivor recovery on 2114; sequential chi and omega restarts (T439010) [23:01:05] !log [Cirrus] Doing cirrussearch2114 next [23:01:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:01:06] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1018.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [23:01:32] FIRING: SLOBudgetBurn: Search update lag is below 95% target in codfw - https://alerts.wikimedia.org/?q=alertname%3DSLOBudgetBurn [23:02:06] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [23:02:06] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [23:05:06] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1012.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [23:06:06] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1014.eqiad.wmnet, wdqs1016.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [23:07:06] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [23:07:06] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [23:08:48] !log ryankemper@cumin2003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2086.codfw.wmnet with reason: Codfw survivor recovery on 2086; sequential chi and omega restarts (T439010) [23:09:26] !log [Cirrus] rolling cirrussearch2086 next [23:09:27] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:10:06] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1018.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [23:10:23] FIRING: GnmiInterfaceCountersDrop: ... [23:10:23] lsw1-b5-codfw is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?orgId=1&from=now-24h&to=now&var-site=%24__all&var-instance=lsw1-b5-codfw:9804&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop [23:10:37] (03PS11) 10Jasmine: mesh: move envoy drain configuration from mesh.extra_env to mesh.admin [deployment-charts] - 10https://gerrit.wikimedia.org/r/1335958 (https://phabricator.wikimedia.org/T427024) [23:12:06] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [23:16:06] PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1011.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [23:16:26] FIRING: SystemdUnitFailed: prometheus-node-textfile-export_service_type.service on cumin1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [23:17:06] RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [23:20:57] !log ryankemper@cumin2003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2072.codfw.wmnet with reason: Codfw survivor recovery on 2072; sequential chi and psi restarts (T439010) [23:23:32] (updating the text of the scap lock) [23:23:55] !log rzl@deploy1003 Unlocked for deployment [ALL REPOSITORIES]: incident recovery in progress T439010 (duration: 121m 40s) [23:23:55] !log rzl@deploy1003 Locking from deployment [ALL REPOSITORIES]: No deployments please, as we're still cleaning up from the codfw power incident T439010. Thursday UTC morning at the earliest, but please ask SRE oncall. [23:26:06] PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1013.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal [23:27:06] RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal [23:27:44] !log ryankemper@cumin2003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2085.codfw.wmnet with reason: Codfw survivor recovery on 2085; sequential chi and psi restarts (T439010) [23:30:05] (03Abandoned) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1270244 (owner: 10TrainBranchBot) [23:31:44] FIRING: CirrusStreamingUpdaterUnknownErrors: CirrusSearch consumer-search@codfw is failing write requests because of unknown errors - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterUnknownErrors [23:34:06] !log ryankemper@cumin2003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2107.codfw.wmnet with reason: Codfw survivor recovery on 2107; sequential chi and psi restarts (T439010) [23:35:01] !log import varnish 7.1.1-2~bpo13+wmf3 into trixie-wikimedia (T438293) [23:35:02] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:38:15] !log brett@cumin1004 START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P{cp7001.magru.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 () [23:38:44] !log ryankemper@cumin2003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2108.codfw.wmnet with reason: Codfw survivor recovery on 2108; sequential chi and psi restarts (T439010) [23:39:02] (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1344407 [23:39:02] (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1344407 (owner: 10TrainBranchBot) [23:41:44] RESOLVED: CirrusStreamingUpdaterUnknownErrors: CirrusSearch consumer-search@codfw is failing write requests because of unknown errors - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterUnknownErrors [23:46:57] ^ update pipeline is still broken fwiw, this alert does not indicate that things are better now [23:46:59] (03PS3) 10Dduvall: buildx: New driver for building images via Buildx/BuildKit [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1342799 (https://phabricator.wikimedia.org/T434958) [23:48:23] !log [Cirrus] Every host except 2084, which is the current elected chi master, has now been restarted, and shard recoveries healed accordingly. AFAICT we will not be able to revive the updater until we restart this host. Pausing for a few mins to mull things over and get my bearings though, because this restart would be higher-touch than the previous ones [23:48:24] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:48:42] !log brett@cumin1004 END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P{cp7001.magru.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 () [23:48:48] (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1344407 (owner: 10TrainBranchBot) [23:49:51] !log brett@cumin1004 START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P{cp7011.magru.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 () [23:50:35] (03CR) 10CI reject: [V:04-1] buildx: New driver for building images via Buildx/BuildKit [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1342799 (https://phabricator.wikimedia.org/T434958) (owner: 10Dduvall) [23:50:53] (03CR) 10BCornwall: [C:03+2] Switch acme-chief primaries to eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1344354 (https://phabricator.wikimedia.org/T439010) (owner: 10BCornwall) [23:54:11] !log brett@cumin1004 END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on P{cp7011.magru.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 () [23:54:48] ^That's what I get for doing two things at once - puppet's disabled, so it failed a run after upgrading [23:56:47] !log Switching acme-chief primary from codfw to eqiad - T439010 [23:56:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:58:54] !log dzahn@cumin2003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie