[00:00:19] <logmsgbot>	 !log brett@cumin1004 START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P{cp7011.magru.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
[00:05:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1012.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[00:05:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[00:05:40] <logmsgbot>	 !log brett@cumin1004 END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P{cp7011.magru.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
[00:06:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[00:07:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[00:10:01] <logmsgbot>	 !log brett@puppetserver1001 conftool action : set/pooled=yes; selector: name=cp7011.*
[00:11:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1021.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[00:11:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[00:12:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[00:12:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[00:12:36] <wikibugs>	 (03PS4) 10Dduvall: buildx: New driver for building images via Buildx/BuildKit [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1342799 (https://phabricator.wikimedia.org/T434958)
[00:16:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1016.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[00:16:31] <jinxer-wm>	 FIRING: [2x] Not accepting/receiving prefixes from anycast BGP peer: Alert for device cr1-codfw.wikimedia.org - Not accepting/receiving prefixes from anycast BGP peer   - https://alerts.wikimedia.org/?q=alertname%3DNot+accepting%2Freceiving+prefixes+from+anycast+BGP+peer
[00:17:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[00:17:20] <wikibugs>	 (03CR) 10CI reject: [V:04-1] buildx: New driver for building images via Buildx/BuildKit [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1342799 (https://phabricator.wikimedia.org/T434958) (owner: 10Dduvall)
[00:20:03] <wikibugs>	 (03PS5) 10Dduvall: buildx: New driver for building images via Buildx/BuildKit [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1342799 (https://phabricator.wikimedia.org/T434958)
[00:20:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1021.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[00:21:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[00:35:06] <logmsgbot>	 !log ryankemper@cumin2003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on 55 hosts with reason: Codfw chi elected-manager recovery on 2084; expected brief failover and red state (T439010)
[00:38:17] <ryankemper>	 !log [Cirrus] Preparing to restart cirrussearch2084 (active cluster manager). With luck, this should restore updater availability (and general cluster green status, after some reshuffling)
[00:38:18] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[00:40:08] <ryankemper>	 !log [Cirrus] Great news, we briefly dipped red (same as previous restarts) but went back to yellow almost immediately. AFAICT election went fine, still checking though
[00:40:09] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[00:40:44] <jinxer-wm>	 RESOLVED: CirrusStreamingUpdaterClearWeightedTagsTooLow: ...
[00:40:50] <jinxer-wm>	 CirrusSearch consumer-search@codfw is clearing too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=codfw&var-search_cluster=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterClearWeightedTagsTooLow
[00:41:02] <jinxer-wm>	 RESOLVED: CirrusStreamingUpdaterSetWeightedTagsTooLow: ...
[00:41:08] <jinxer-wm>	 CirrusSearch consumer-search@codfw is setting too few weighted tags - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/fe251f4f-f6cf-4010-8d78-5f482255b16f/cirrussearch-update-pipeline-weighted-tags?orgId=1&var-tag_prefix=All&var-search_cluster_site=codfw&var-search_cluster=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterSetWeightedTagsTooLow
[00:41:25] <ryankemper>	 hell yes, that's what we want to see
[00:44:44] <jinxer-wm>	 FIRING: CirrusStreamingUpdaterUnknownErrors: CirrusSearch consumer-search@codfw is failing write requests because of unknown errors - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterUnknownErrors
[00:54:44] <jinxer-wm>	 RESOLVED: CirrusStreamingUpdaterUnknownErrors: CirrusSearch consumer-search@codfw is failing write requests because of unknown errors - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterUnknownErrors
[00:58:12] <wikibugs>	 (03PS1) 10BCornwall: fixup! Switch acme-chief primaries to eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1344410
[00:59:51] <wikibugs>	 (03CR) 10BCornwall: [C:03+2] fixup! Switch acme-chief primaries to eqiad [puppet] - 10https://gerrit.wikimedia.org/r/1344410 (owner: 10BCornwall)
[01:00:02] <wikibugs>	 (03Abandoned) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1270254 (owner: 10TrainBranchBot)
[01:01:49] <icinga-wm>	 PROBLEM - HAProxy HTTPS wikipedia.org ECDSA on cp2059 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection refused https://wikitech.wikimedia.org/wiki/HTTPS
[01:01:49] <icinga-wm>	 PROBLEM - HAProxy HTTPS wikipedia25.org ECDSA on cp2059 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection refused https://wikitech.wikimedia.org/wiki/HTTPS
[01:01:50] <icinga-wm>	 PROBLEM - HAProxy HTTPS wikiworkshop.org ECDSA on cp2059 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection refused https://wikitech.wikimedia.org/wiki/HTTPS
[01:03:07] <brett>	 ^looking
[01:03:59] <ryankemper>	 !log [Cirrus] grr, I'd missed some hosts. restarting the last few dangling ones, we're really close to back to green, prob 3-ish more hosts
[01:04:00] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[01:04:27] <logmsgbot>	 !log ryankemper@cumin2003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2104.codfw.wmnet with reason: Codfw survivor recovery on 2104; temporary chi red expected (T439010)
[01:04:54] <icinga-wm>	 PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 301842952 and 41 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[01:05:05] <ryankemper>	 oh, lovely
[01:07:44] <jinxer-wm>	 RESOLVED: CirrusStreamingUpdaterRateTooLow: CirrusSearch update rate from flink-app-consumer-search is critically low - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/jKqki4MSk/cirrus-streaming-updater - https://alerts.wikimedia.org/?q=alertname%3DCirrusStreamingUpdaterRateTooLow
[01:07:56] <icinga-wm>	 RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 6065208 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[01:08:57] <wikibugs>	 (03PS1) 10TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1344411
[01:08:57] <wikibugs>	 (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1344411 (owner: 10TrainBranchBot)
[01:10:18] <logmsgbot>	 !log ryankemper@cumin2003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2109.codfw.wmnet with reason: Codfw survivor recovery on 2109; temporary chi red expected (T439010)
[01:10:34] <brett>	 considering the check was never healthy and it's depooled, I'm going to assume it's known broken as sukhe was bringing it up. I'm silencing this for 24h
[01:11:50] <rzl>	 yeah, he was working on that today right before everything got exciting and I assume never got a chance to come back to it
[01:11:52] <logmsgbot>	 !log brett@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on cp2059.codfw.wmnet with reason: failing services but not in service yet
[01:14:31] <logmsgbot>	 !log ryankemper@cumin2003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cirrussearch2115.codfw.wmnet with reason: Codfw survivor recovery on 2115; temporary chi red expected (T439010)
[01:16:12] <ryankemper>	 !log [Cirrus] With the restart of `cirrussearch2115`, the codfw cluster has officially reached green status!!! Still working on full verification, but we're almost done here
[01:16:13] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[01:17:36] <wikibugs>	 (03Merged) 10jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - 10https://gerrit.wikimedia.org/r/1344411 (owner: 10TrainBranchBot)
[01:19:13] <ryankemper>	 !log [Cirrus] Reverted `node_concurrent_recoveries` to 5 from 10, now that we're back to green
[01:19:14] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[01:20:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1012.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[01:20:40] <jinxer-wm>	 FIRING: [2x] ProbeDown: Service restbase2029:7231 has failed probes (http_restbase_discovery_wmnet_ip4) - https://wikitech.wikimedia.org/wiki/RESTBase - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[01:21:10] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: prometheus-node-textfile-export_service_type.service on cumin1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[01:21:31] <jinxer-wm>	 FIRING: RedisReplicaDown: Redis replica down rdb2014:16380 redis_misc - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_misc - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_misc&var-instance=rdb2014:16380 - https://alerts.wikimedia.org/?q=alertname%3DRedisReplicaDown
[01:21:48] <wikibugs>	 10ops-codfw, 06DC-Ops: Power Supply - Status - issue on cirrussearch2080:9290 - https://phabricator.wikimedia.org/T439054 (10phaultfinder) 03NEW
[01:21:49] <logmsgbot>	 !log ryankemper@cumin2003 START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster search_codfw: Restart codfw following today's power incident to ensure we return to our full expected state - ryankemper@cumin2003 - T439010
[01:21:49] <wikibugs>	 10ops-codfw, 06DC-Ops: Power Supply - Status - issue on wikikube-ctrl2001:9290 - https://phabricator.wikimedia.org/T439055 (10phaultfinder) 03NEW
[01:21:56] <jinxer-wm>	 FIRING: GitlabPackagePullerFailedOnPrepare: Package puller has some run errors while preparing projects. - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DGitlabPackagePullerFailedOnPrepare
[01:22:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[01:22:39] <jinxer-wm>	 FIRING: [2x] CirrusSearchNodeIndexingNotIncreasing: OpenSearch instance cirrussearch2100-production-search-codfw is not indexing - https://wikitech.wikimedia.org/wiki/Search/OpenSearch/Administration#Alerts/Dashboards - https://grafana.wikimedia.org/d/JLK3I_siz/elasticsearch-indexing?orgId=1&from=now-3d&to=now&viewPanel=57 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchNodeIndexingNotIncreasing
[01:22:41] <rzl>	 something about those "Power Supply - Status - issue" tickets is deeply funny
[01:22:46] <rzl>	 yeah, I know, buddy. I know.
[01:23:01] <jinxer-wm>	 FIRING: RedisInstanceDown: Redis instance down rdb2013:16380 redis_misc - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_misc - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_misc&var-instance=rdb2013:16380 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown
[01:24:44] <wikibugs>	 10ops-codfw, 06DC-Ops: Power Supply - Status - issue on cloudbackup2003:9290 - https://phabricator.wikimedia.org/T439058 (10phaultfinder) 03NEW
[01:24:44] <wikibugs>	 10ops-codfw, 06DC-Ops: Power Supply - Status - issue on logstash2036:9290 - https://phabricator.wikimedia.org/T439057 (10phaultfinder) 03NEW
[01:25:28] <rzl>	 I can't actually tell if those reflect new issues or not, but the actual hosts seem okay atm
[01:26:11] <rzl>	 and they're going to DC-Ops, which is the right place, so I'm not going to worry about it rn
[01:26:20] <rzl>	 Tomorrow's Problem
[01:27:39] <jinxer-wm>	 FIRING: [2x] CirrusSearchNodeIndexingNotIncreasing: OpenSearch instance cirrussearch2100-production-search-codfw is not indexing - https://wikitech.wikimedia.org/wiki/Search/OpenSearch/Administration#Alerts/Dashboards - https://grafana.wikimedia.org/d/JLK3I_siz/elasticsearch-indexing?orgId=1&from=now-3d&to=now&viewPanel=57 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchNodeIndexingNotIncreasing
[01:30:56] <icinga-wm>	 PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 266300704 and 18 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[01:34:56] <icinga-wm>	 RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 0 and 1 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[01:35:44] <jinxer-wm>	 FIRING: CirrusConsumerFetchErrorRate: cirrus_streaming_updater_consumer_search_codfw in codfw (k8s): fetch error rate too high - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?var-datasource=codfw+prometheus%2Fk8s&var-namespace=cirrus-streaming-updater&var-helm_release=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusConsumerFetchErrorRate
[01:40:44] <jinxer-wm>	 RESOLVED: CirrusConsumerFetchErrorRate: cirrus_streaming_updater_consumer_search_codfw in codfw (k8s): fetch error rate too high - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?var-datasource=codfw+prometheus%2Fk8s&var-namespace=cirrus-streaming-updater&var-helm_release=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusConsumerFetchErrorRate
[02:00:05] <jouncebot>	 Deploy window Automatic deployment of MediaWiki to pretrain wikis - see mw:Pretrain (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260924T0200)
[02:02:44] <jinxer-wm>	 FIRING: CirrusConsumerFetchErrorRate: cirrus_streaming_updater_consumer_search_codfw in codfw (k8s): fetch error rate too high - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?var-datasource=codfw+prometheus%2Fk8s&var-namespace=cirrus-streaming-updater&var-helm_release=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusConsumerFetchErrorRate
[02:06:10] <jinxer-wm>	 FIRING: [3x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[02:07:44] <jinxer-wm>	 RESOLVED: CirrusConsumerFetchErrorRate: cirrus_streaming_updater_consumer_search_codfw in codfw (k8s): fetch error rate too high - https://wikitech.wikimedia.org/wiki/Search#Streaming_Updater - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?var-datasource=codfw+prometheus%2Fk8s&var-namespace=cirrus-streaming-updater&var-helm_release=consumer-search - https://alerts.wikimedia.org/?q=alertname%3DCirrusConsumerFetchErrorRate
[02:10:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1018.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[02:11:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[02:11:42] <jinxer-wm>	 FIRING: JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[02:16:42] <jinxer-wm>	 FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[02:21:42] <jinxer-wm>	 RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[02:27:39] <jinxer-wm>	 RESOLVED: CirrusSearchNodeIndexingNotIncreasing: OpenSearch instance cirrussearch2100-production-search-codfw is not indexing - https://wikitech.wikimedia.org/wiki/Search/OpenSearch/Administration#Alerts/Dashboards - https://grafana.wikimedia.org/d/JLK3I_siz/elasticsearch-indexing?orgId=1&from=now-3d&to=now&viewPanel=57 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchNodeIndexingNotIncreasing
[02:28:27] <jinxer-wm>	 FIRING: [3x] ProbeDown: Service centrallog2002:6514 has failed probes (tcp_rsyslog_receiver_ip6)  - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[02:46:24] <jinxer-wm>	 FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[02:47:23] <jinxer-wm>	 FIRING: [3x] CertAlmostExpired: gNMI TLS certificate for lsw1-e2-codfw.mgmt.codfw.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired
[03:01:32] <jinxer-wm>	 FIRING: SLOBudgetBurn: Search update lag is below 95% target in codfw   - https://alerts.wikimedia.org/?q=alertname%3DSLOBudgetBurn
[03:10:23] <jinxer-wm>	 FIRING: GnmiInterfaceCountersDrop: ...
[03:10:23] <jinxer-wm>	 lsw1-b5-codfw is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?orgId=1&from=now-24h&to=now&var-site=%24__all&var-instance=lsw1-b5-codfw:9804&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop
[03:11:48] <jinxer-wm>	 FIRING: PuppetFailure: Puppet has failed on cloudidp2001-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[03:13:58] <icinga-wm>	 PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 205015224 and 23 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[03:15:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[03:15:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[03:15:58] <icinga-wm>	 RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 960 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[03:16:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[03:16:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[03:21:48] <jinxer-wm>	 RESOLVED: PuppetFailure: Puppet has failed on cloudidp2001-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[04:16:32] <jinxer-wm>	 FIRING: [2x] Not accepting/receiving prefixes from anycast BGP peer: Alert for device cr1-codfw.wikimedia.org - Not accepting/receiving prefixes from anycast BGP peer   - https://alerts.wikimedia.org/?q=alertname%3DNot+accepting%2Freceiving+prefixes+from+anycast+BGP+peer
[04:22:58] <icinga-wm>	 PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 751407728 and 53 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[04:23:58] <icinga-wm>	 RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 3376896 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[04:49:58] <icinga-wm>	 PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 33269600 and 5 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[04:50:58] <icinga-wm>	 RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 3208704 and 1 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[04:53:58] <icinga-wm>	 PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 352197248 and 22 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[04:54:58] <icinga-wm>	 RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 72 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[05:02:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1016.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[05:05:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1012.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[05:07:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[05:07:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[05:07:34] <logmsgbot>	 !log ryankemper@cumin2003 END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster search_codfw: Restart codfw following today's power incident to ensure we return to our full expected state - ryankemper@cumin2003 - T439010
[05:10:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1018.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[05:11:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1018.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[05:11:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[05:12:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[05:17:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1019.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[05:21:31] <jinxer-wm>	 FIRING: RedisReplicaDown: Redis replica down rdb2014:16380 redis_misc - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_misc - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_misc&var-instance=rdb2014:16380 - https://alerts.wikimedia.org/?q=alertname%3DRedisReplicaDown
[05:21:56] <jinxer-wm>	 FIRING: GitlabPackagePullerFailedOnPrepare: Package puller has some run errors while preparing projects. - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DGitlabPackagePullerFailedOnPrepare
[05:22:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[05:23:01] <jinxer-wm>	 FIRING: RedisInstanceDown: Redis instance down rdb2013:16380 redis_misc - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_misc - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_misc&var-instance=rdb2013:16380 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown
[05:25:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1018.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[05:25:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1018.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[05:27:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[05:29:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[05:34:58] <icinga-wm>	 PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 37069792 and 6 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[05:36:58] <icinga-wm>	 RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 3180832 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[05:40:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1021.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[05:40:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1021.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[05:41:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[05:41:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[05:41:41] <wikibugs>	 (03PS1) 10AKhatun: Add article-feature-counts alerts [alerts] - 10https://gerrit.wikimedia.org/r/1344530 (https://phabricator.wikimedia.org/T438384)
[05:43:58] <icinga-wm>	 PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 694682672 and 36 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[05:45:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1019.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[05:45:31] <wikibugs>	 (03PS2) 10AKhatun: article-feature-counts: add deployment chart files [deployment-charts] - 10https://gerrit.wikimedia.org/r/1343044 (https://phabricator.wikimedia.org/T437000)
[05:45:58] <icinga-wm>	 RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 19336 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[05:46:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1020.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[05:47:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[05:47:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[05:48:15] <wikibugs>	 (03CR) 10Marostegui: switchdc.databases.prepare: Retry checks for replication threads (033 comments) [cookbooks] - 10https://gerrit.wikimedia.org/r/1344031 (https://phabricator.wikimedia.org/T438833) (owner: 10CWilliams)
[05:50:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1020.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[05:51:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[05:54:15] <jinxer-wm>	 FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[05:55:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[05:55:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1018.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1012.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[05:57:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[05:57:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[05:58:36] <wikibugs>	 (03PS1) 10Jelto: aptrepo: update gitlab-ce and gitlab-runner to 19.2 [puppet] - 10https://gerrit.wikimedia.org/r/1344531 (https://phabricator.wikimedia.org/T439037)
[05:59:15] <jinxer-wm>	 RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[05:59:21] <wikibugs>	 (03PS2) 10Jelto: aptrepo: update gitlab-ce and gitlab-runner to 19.2 [puppet] - 10https://gerrit.wikimedia.org/r/1344531 (https://phabricator.wikimedia.org/T439037)
[06:00:05] <jouncebot>	 Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260924T0600)
[06:00:05] <jouncebot>	 marostegui, cezmunsta, and federico3: How many deployers does it take to do Primary database switchover deploy? (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260924T0600).
[06:00:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1013.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[06:00:23] <jinxer-wm>	 RESOLVED: GnmiInterfaceCountersDrop: ...
[06:00:23] <jinxer-wm>	 lsw1-b5-codfw is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?orgId=1&from=now-24h&to=now&var-site=%24__all&var-instance=lsw1-b5-codfw:9804&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop
[06:01:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[06:01:15] <jinxer-wm>	 FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook  - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[06:03:58] <icinga-wm>	 PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 651443120 and 82 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[06:05:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[06:06:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[06:06:15] <jinxer-wm>	 FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook  - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[06:06:26] <jinxer-wm>	 FIRING: [3x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[06:06:49] <logmsgbot>	 !log ayounsi@cumin1004 START - Cookbook sre.network.tls for network device lsw1-e5-codfw
[06:07:45] <logmsgbot>	 !log ayounsi@cumin1004 END (FAIL) - Cookbook sre.network.tls (exit_code=99) for network device lsw1-e5-codfw
[06:07:58] <icinga-wm>	 PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 302432880 and 37 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[06:08:33] <wikibugs>	 (03CR) 10Jelto: [C:03+2] aptrepo: update gitlab-ce and gitlab-runner to 19.2 [puppet] - 10https://gerrit.wikimedia.org/r/1344531 (https://phabricator.wikimedia.org/T439037) (owner: 10Jelto)
[06:09:58] <icinga-wm>	 RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 2181600 and 15 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[06:11:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1019.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[06:11:15] <jinxer-wm>	 FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook  - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[06:12:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[06:13:01] <jinxer-wm>	 RESOLVED: RedisInstanceDown: Redis instance down rdb2013:16380 redis_misc - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_misc - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_misc&var-instance=rdb2013:16380 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown
[06:14:01] <jinxer-wm>	 FIRING: RedisInstanceDown: Redis instance down rdb2013:16380 redis_misc - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_misc - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_misc&var-instance=rdb2013:16380 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown
[06:15:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1021.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[06:16:15] <jinxer-wm>	 RESOLVED: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook  - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[06:20:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[06:22:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[06:22:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[06:25:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1012.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[06:26:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1019.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[06:27:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[06:27:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[06:27:30] <wikibugs>	 10ops-codfw, 06DC-Ops: Unresponsive management for dse-k8s-wdqs2001.mgmt:22 - https://phabricator.wikimedia.org/T439068 (10phaultfinder) 03NEW
[06:27:58] <icinga-wm>	 PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 219903128 and 21 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[06:28:15] <jinxer-wm>	 FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.28% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy
[06:28:16] <wikibugs>	 (03PS1) 10Marostegui: installserver: Do not reimage db1288 [puppet] - 10https://gerrit.wikimedia.org/r/1344533
[06:28:25] <wikibugs>	 10ops-codfw, 06DC-Ops: Unresponsive management for apus-be2006.mgmt:22 - https://phabricator.wikimedia.org/T439070 (10phaultfinder) 03NEW
[06:28:26] <wikibugs>	 10ops-codfw, 06DC-Ops: Unresponsive management for wikikube-worker2342.mgmt:22 - https://phabricator.wikimedia.org/T439069 (10phaultfinder) 03NEW
[06:28:27] <jinxer-wm>	 FIRING: [2x] ProbeDown: Service restbase2029:7231 has failed probes (http_restbase_discovery_wmnet_ip4) - https://wikitech.wikimedia.org/wiki/RESTBase - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[06:28:28] <wikibugs>	 10ops-codfw, 06DC-Ops: Unresponsive management for wikikube-worker2335.mgmt:22 - https://phabricator.wikimedia.org/T439076 (10phaultfinder) 03NEW
[06:28:29] <wikibugs>	 10ops-codfw, 06DC-Ops: Unresponsive management for ms-be2095.mgmt:22 - https://phabricator.wikimedia.org/T439072 (10phaultfinder) 03NEW
[06:28:30] <wikibugs>	 10ops-codfw, 06DC-Ops: Unresponsive management for wikikube-worker2344.mgmt:22 - https://phabricator.wikimedia.org/T439075 (10phaultfinder) 03NEW
[06:28:31] <wikibugs>	 10ops-codfw, 06DC-Ops: Unresponsive management for wikikube-worker2373.mgmt:22 - https://phabricator.wikimedia.org/T439077 (10phaultfinder) 03NEW
[06:28:33] <wikibugs>	 10ops-codfw, 06DC-Ops: Unresponsive management for wikikube-worker2366.mgmt:22 - https://phabricator.wikimedia.org/T439071 (10phaultfinder) 03NEW
[06:28:37] <wikibugs>	 10ops-codfw, 06DC-Ops: Unresponsive management for wikikube-worker2334.mgmt:22 - https://phabricator.wikimedia.org/T439073 (10phaultfinder) 03NEW
[06:28:41] <wikibugs>	 10ops-codfw, 06DC-Ops: Unresponsive management for wikikube-worker2360.mgmt:22 - https://phabricator.wikimedia.org/T439074 (10phaultfinder) 03NEW
[06:29:31] <wikibugs>	 10ops-codfw, 06DC-Ops: Unresponsive management for backup2017.mgmt:22 - https://phabricator.wikimedia.org/T439080 (10phaultfinder) 03NEW
[06:29:33] <wikibugs>	 10ops-codfw, 06DC-Ops: Unresponsive management for mwlog2003.mgmt:22 - https://phabricator.wikimedia.org/T439079 (10phaultfinder) 03NEW
[06:29:34] <wikibugs>	 10ops-codfw, 06DC-Ops: Unresponsive management for wikikube-worker2359.mgmt:22 - https://phabricator.wikimedia.org/T439081 (10phaultfinder) 03NEW
[06:29:35] <wikibugs>	 10ops-codfw, 06SRE, 06DC-Ops: Unresponsive management for wikikube-worker2336.mgmt:22 - https://phabricator.wikimedia.org/T439082 (10phaultfinder) 03NEW
[06:29:37] <wikibugs>	 10ops-codfw, 06SRE, 06DC-Ops: Unresponsive management for ms-fe2023.mgmt:22 - https://phabricator.wikimedia.org/T439085 (10phaultfinder) 03NEW
[06:29:38] <wikibugs>	 10ops-codfw, 06SRE, 06DC-Ops: Unresponsive management for wikikube-worker2355.mgmt:22 - https://phabricator.wikimedia.org/T439084 (10phaultfinder) 03NEW
[06:29:40] <wikibugs>	 10ops-codfw, 06SRE, 06DC-Ops: Unresponsive management for ms-fe2024.mgmt:22 - https://phabricator.wikimedia.org/T439083 (10phaultfinder) 03NEW
[06:29:42] <wikibugs>	 10ops-codfw, 06SRE, 06DC-Ops: Unresponsive management for wikikube-worker2365.mgmt:22 - https://phabricator.wikimedia.org/T439088 (10phaultfinder) 03NEW
[06:29:46] <wikibugs>	 10ops-codfw, 06SRE, 06DC-Ops: Unresponsive management for wikikube-worker2356.mgmt:22 - https://phabricator.wikimedia.org/T439087 (10phaultfinder) 03NEW
[06:29:50] <wikibugs>	 10ops-codfw, 06SRE, 06DC-Ops: Unresponsive management for backup2018.mgmt:22 - https://phabricator.wikimedia.org/T439086 (10phaultfinder) 03NEW
[06:30:23] <wikibugs>	 10ops-codfw, 06DC-Ops: Unresponsive management for wikikube-worker2343.mgmt:22 - https://phabricator.wikimedia.org/T439090 (10phaultfinder) 03NEW
[06:30:24] <wikibugs>	 10ops-codfw, 06DC-Ops: Unresponsive management for wikikube-worker2374.mgmt:22 - https://phabricator.wikimedia.org/T439091 (10phaultfinder) 03NEW
[06:30:25] <wikibugs>	 10ops-codfw, 06DC-Ops: Unresponsive management for wikikube-worker2354.mgmt:22 - https://phabricator.wikimedia.org/T439089 (10phaultfinder) 03NEW
[06:30:58] <icinga-wm>	 RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 3173760 and 1 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[06:31:29] <wikibugs>	 10ops-codfw, 06DC-Ops: Unresponsive management for ms-be2098.mgmt:22 - https://phabricator.wikimedia.org/T439092 (10phaultfinder) 03NEW
[06:32:02] <wikibugs>	 (03CR) 10Marostegui: [C:03+2] installserver: Do not reimage db1288 [puppet] - 10https://gerrit.wikimedia.org/r/1344533 (owner: 10Marostegui)
[06:32:15] <jinxer-wm>	 FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[06:34:58] <icinga-wm>	 PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 562714240 and 74 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[06:36:58] <icinga-wm>	 RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 0 and 1 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[06:37:00] <kart_>	 Deploying MinT in staging.
[06:37:15] <jinxer-wm>	 RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[06:38:29] <logmsgbot>	 !log kartik@deploy1003 helmfile [staging] START helmfile.d/services/machinetranslation: apply
[06:39:15] <jinxer-wm>	 FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[06:39:33] <kart_>	 !log staging: Update machinetranslation/MinT to 2026-09-21-112314-production
[06:39:35] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[06:41:31] <logmsgbot>	 !log kartik@deploy1003 helmfile [staging] DONE helmfile.d/services/machinetranslation: apply
[06:41:50] <kart_>	 !log staging: Update machinetranslation/MinT to 2026-09-21-112314-production (T437213)
[06:41:51] <wikibugs>	 10ops-eqiad, 06DC-Ops: Interface eno1 on wikikube-worker1160 has the wrong speed - https://phabricator.wikimedia.org/T439094 (10phaultfinder) 03NEW
[06:41:52] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[06:41:52] <stashbot>	 T437213: Review and update LPL service base images – 2026Q3 - https://phabricator.wikimedia.org/T437213
[06:43:15] <jinxer-wm>	 RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.64% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy
[06:44:15] <jinxer-wm>	 RESOLVED: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook  - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[06:46:25] <jinxer-wm>	 FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[06:47:15] <jinxer-wm>	 FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.34% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy
[06:47:23] <jinxer-wm>	 FIRING: [3x] CertAlmostExpired: gNMI TLS certificate for lsw1-e2-codfw.mgmt.codfw.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired
[06:51:14] <logmsgbot>	 !log jelto@cumin1004 START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: version upgrade
[06:52:15] <jinxer-wm>	 RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.34% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy
[06:59:15] <jinxer-wm>	 FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 22.07% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy
[06:59:47] <wikibugs>	 (03CR) 10Arnaudb: [C:03+1] "looks good to me!" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1341335 (https://phabricator.wikimedia.org/T437635) (owner: 10Dzahn)
[07:00:05] <jouncebot>	 Amir1, urbanecm, and awight: May I have your attention please! UTC morning backport window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260924T0700)
[07:00:05] <jouncebot>	 MichaelG_WMF, mamouri, and nicksbro_wmde: A patch you scheduled for UTC morning backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker.
[07:00:28] <wikibugs>	 (03CR) 10Arnaudb: [C:03+1] aptrepo: update gitlab-ce and gitlab-runner to 19.2 [puppet] - 10https://gerrit.wikimedia.org/r/1344531 (https://phabricator.wikimedia.org/T439037) (owner: 10Jelto)
[07:01:32] <jinxer-wm>	 FIRING: SLOBudgetBurn: Search update lag is below 95% target in codfw   - https://alerts.wikimedia.org/?q=alertname%3DSLOBudgetBurn
[07:02:17] <logmsgbot>	 !log jelto@cumin1004 END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: version upgrade
[07:02:44] <awight>	 MichaelG_WMF: Would you prefer to self-deploy, or may I?
[07:02:52] <awight>	 mamouri: ping?
[07:03:15] <jinxer-wm>	 FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[07:03:51] <awight>	 (I will start with nicksbro's patch until I hear back...)
[07:06:04] <logmsgbot>	 !log jelto@cumin1004 START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: version upgrade
[07:06:35] <MichaelG_WMF>	 awight: hi!
[07:06:43] <MichaelG_WMF>	 sorry for being slightly late
[07:06:43] <wikibugs>	 10ops-codfw, 06DC-Ops: arclamp2001 has a broken DIMM - https://phabricator.wikimedia.org/T439095 (10MoritzMuehlenhoff) 03NEW
[07:06:52] <MichaelG_WMF>	 it would be great if you could do it!
[07:06:56] <awight>	 MichaelG_WMF: sorry, I just found that deployments are blocked for the moment: T439010
[07:06:58] <MichaelG_WMF>	 I can test 👍
[07:07:08] * MichaelG_WMF is looking
[07:08:15] <jinxer-wm>	 RESOLVED: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook  - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[07:08:25] <awight>	 mamouri: fyi deployments are blocked at the moment...
[07:08:47] <mamouri>	 Hi team, sorry I am late. I had a technical issue. I am here to verify enabling $wgCiteDefaultRefDirAuto deployment on fawiki today
[07:09:43] <wikibugs>	 10ops-eqiad, 06DC-Ops: Alert for device ps1-c6-eqiad.mgmt.eqiad.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T439096 (10phaultfinder) 03NEW
[07:09:49] <mamouri>	 awight: got it, thanks — no rush on my end
[07:10:21] <MichaelG_WMF>	 my config change is also not urgent yet. Can do at the late window today or on Monday
[07:10:48] <awight>	 ty!
[07:11:58] <icinga-wm>	 PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 81792448 and 2 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[07:12:07] <mamouri>	 By the way, my change is just turning a flag on. If I am not mistaken I need a +1 on. This is my first time testing a change and I don't have enough experience. Can someone more familiar tell me if I still need a +1? Thanks
[07:12:58] <icinga-wm>	 RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 0 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[07:14:16] <awight>	 mamouri: +1 code review is always helpful but IMO not strictly necessary.  It's just a sign that someone close to the change has taken a look at the patch...
[07:15:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1020.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[07:15:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1020.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[07:16:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[07:16:15] <jinxer-wm>	 FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[07:16:32] <jinxer-wm>	 RESOLVED: SLOBudgetBurn: Search update lag is below 95% target in codfw   - https://alerts.wikimedia.org/?q=alertname%3DSLOBudgetBurn
[07:16:54] <logmsgbot>	 !log jelto@cumin1004 END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: version upgrade
[07:17:17] <mamouri>	 awight: thank for the explanation. I already had a +1, but because I rebased, I have lost it, but my change is literaly one line and it is enabling a flag in wikipedia farsi which only make dir: auto to citations that are dir: ltr by default
[07:19:39] <logmsgbot>	 !log jelto@cumin1004 START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: version upgrade
[07:19:44] <jinxer-wm>	 FIRING: KubernetesDeploymentUnavailableReplicas: ...
[07:19:44] <jinxer-wm>	 Deployment mw-web.eqiad.main in mw-web at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s&var-namespace=mw-web&var-deployment=mw-web.eqiad.main - ...
[07:19:44] <jinxer-wm>	 https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas
[07:20:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1018.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1019.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[07:21:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[07:21:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[07:21:15] <jinxer-wm>	 RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[07:24:15] <jinxer-wm>	 RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 23.47% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy
[07:26:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1021.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[07:26:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[07:31:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[07:31:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[07:32:15] <jinxer-wm>	 FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook  - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[07:34:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1021.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[07:35:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[07:37:05] <logmsgbot>	 jelto@cumin1004 jelto: The backup on gitlab1004 is complete, ready to proceed with upgrade.
[07:37:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[07:37:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[07:37:15] <jinxer-wm>	 FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook  - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[07:41:48] <jinxer-wm>	 FIRING: PuppetFailure: Puppet has failed on cloudidp2001-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[07:42:15] <jinxer-wm>	 FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook  - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[07:42:42] <marostegui>	 !incidents
[07:42:42] <sirenbot>	 8359 (ACKED)  db2175 (paged)/MariaDB Replica IO: s2 (paged)
[07:42:42] <sirenbot>	 8360 (ACKED)  db2189 (paged)/MariaDB Replica IO: s2 (paged)
[07:42:42] <sirenbot>	 8361 (ACKED)  db2195 (paged)/MariaDB Replica IO: s8 (paged)
[07:42:43] <sirenbot>	 8362 (ACKED)  db2220 (paged)/MariaDB Replica IO: s7 (paged)
[07:42:43] <sirenbot>	 8363 (ACKED)  db2215 (paged)/MariaDB Replica IO: x1 (paged)
[07:42:43] <sirenbot>	 8364 (ACKED)  db2163 (paged)/MariaDB Replica IO: s8 (paged)
[07:42:43] <sirenbot>	 8365 (ACKED)  db2153 (paged)/MariaDB Replica IO: s1 (paged)
[07:42:44] <sirenbot>	 8366 (ACKED)  db2167 (paged)/MariaDB Replica IO: s8 (paged)
[07:42:44] <sirenbot>	 8367 (ACKED)  db2154 (paged)/MariaDB Replica IO: s8 (paged)
[07:42:45] <sirenbot>	 8370 (ACKED)  db2176 (paged)/MariaDB Replica IO: s1 (paged)
[07:42:45] <sirenbot>	 8371 (ACKED)  db2156 (paged)/MariaDB Replica IO: s3 (paged)
[07:42:46] <sirenbot>	 8372 (ACKED)  db2164 (paged)/MariaDB Replica IO: s8 (paged)
[07:42:46] <sirenbot>	 8373 (ACKED)  db2168 (paged)/MariaDB Replica IO: s7 (paged)
[07:42:47] <sirenbot>	 8375 (ACKED)  db2181 (paged)/MariaDB Replica IO: s8 (paged)
[07:42:47] <sirenbot>	 8376 (ACKED)  db2218 (paged)/MariaDB Replica IO: s7 (paged)
[07:42:48] <sirenbot>	 8377 (ACKED)  db2194 (paged)/MariaDB Replica IO: s3 (paged)
[07:42:48] <sirenbot>	 8378 (ACKED)  db2196 (paged)/MariaDB Replica IO: x1 (paged)
[07:42:49] <sirenbot>	 8379 (ACKED)  db2225 (paged)/MariaDB Replica IO: s2 (paged)
[07:42:49] <sirenbot>	 8380 (ACKED)  db2222 (paged)/MariaDB Replica IO: s7 (paged)
[07:42:50] <sirenbot>	 8381 (ACKED)  db2182 (paged)/MariaDB Replica IO: s7 (paged)
[07:42:50] <sirenbot>	 8382 (ACKED)  db2221 (paged)/MariaDB Replica IO: s7 (paged)
[07:42:58] <icinga-wm>	 PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 259267032 and 38 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[07:43:58] <icinga-wm>	 RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 2912320 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[07:44:00] <icinga-wm>	 PROBLEM - Gitlab HTTPS healthcheck on gitlab.wikimedia.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 2353 bytes in 0.018 second response time https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring
[07:45:00] <icinga-wm>	 RECOVERY - Gitlab HTTPS healthcheck on gitlab.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 30612 bytes in 0.402 second response time https://wikitech.wikimedia.org/wiki/GitLab%23Monitoring
[07:45:19] <wikibugs>	 (03PS1) 10Dpogorzelski: ml-serve: repartition ml-serve1013 and 1014 to 16x96GB [puppet] - 10https://gerrit.wikimedia.org/r/1344593 (https://phabricator.wikimedia.org/T436928)
[07:47:15] <jinxer-wm>	 FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook  - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[07:48:11] <logmsgbot>	 !log jelto@cumin1004 END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: version upgrade
[07:49:02] <wikibugs>	 (03CR) 10CWilliams: switchdc.databases.prepare: Retry checks for replication threads (034 comments) [cookbooks] - 10https://gerrit.wikimedia.org/r/1344031 (https://phabricator.wikimedia.org/T438833) (owner: 10CWilliams)
[07:50:07] <wikibugs>	 (03PS5) 10CWilliams: switchdc.databases.prepare: Retry checks for replication threads [cookbooks] - 10https://gerrit.wikimedia.org/r/1344031 (https://phabricator.wikimedia.org/T438833)
[07:51:07] <wikibugs>	 (03CR) 10CWilliams: switchdc.databases.prepare: Retry checks for replication threads (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/1344031 (https://phabricator.wikimedia.org/T438833) (owner: 10CWilliams)
[07:51:48] <jinxer-wm>	 RESOLVED: PuppetFailure: Puppet has failed on cloudidp2001-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[07:52:12] <logmsgbot>	 !log ayounsi@cumin1004 START - Cookbook sre.dns.admin DNS admin: depool drmrs [reason: switch upgrade, T437984]
[07:52:15] <jinxer-wm>	 RESOLVED: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook  - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[07:52:16] <stashbot>	 T437984: Junos file descriptors exhaustion - https://phabricator.wikimedia.org/T437984
[07:52:16] <logmsgbot>	 !log ayounsi@cumin1004 END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool drmrs [reason: switch upgrade, T437984]
[07:53:49] <logmsgbot>	 !log ayounsi@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 20 hosts with reason: Switches upgrade
[07:59:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1020.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1012.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[08:00:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1021.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[08:02:52] <logmsgbot>	 !log ayounsi@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on asw1-b12-drmrs,asw1-b12-drmrs IPv6,asw1-b12-drmrs.mgmt with reason: Switch upgrade
[08:05:15] <jinxer-wm>	 FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[08:06:02] <wikibugs>	 (03CR) 10Klausman: [C:03+1] ml-serve: repartition ml-serve1013 and 1014 to 16x96GB [puppet] - 10https://gerrit.wikimedia.org/r/1344593 (https://phabricator.wikimedia.org/T436928) (owner: 10Dpogorzelski)
[08:06:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[08:06:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[08:06:59] <logmsgbot>	 !log jelto@cumin1004 START - Cookbook sre.gitlab.reboot-runner rolling reboot on A:gitlab-runner
[08:07:04] <wikibugs>	 (03CR) 10Marostegui: switchdc.databases.prepare: Retry checks for replication threads (032 comments) [cookbooks] - 10https://gerrit.wikimedia.org/r/1344031 (https://phabricator.wikimedia.org/T438833) (owner: 10CWilliams)
[08:09:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1018.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1012.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[08:09:46] <wikibugs>	 (03PS1) 10Michael Große: EarlyOnboarding: record account setup motivation in `action_context` [extensions/WikimediaEvents] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344594 (https://phabricator.wikimedia.org/T437339)
[08:10:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1020.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[08:10:15] <jinxer-wm>	 RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[08:11:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[08:12:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[08:12:47] <wikibugs>	 (03CR) 10Btullis: [C:03+2] dse-k8s-eqiad: put dse-k8s-worker1015 back into service [puppet] - 10https://gerrit.wikimedia.org/r/1344327 (https://phabricator.wikimedia.org/T430658) (owner: 10Btullis)
[08:13:07] <logmsgbot>	 !log ayounsi@cumin1004 START - Cookbook sre.network.depool-rack with action 'depool' for drmrs rack B12
[08:13:14] <wikibugs>	 (03CR) 10CI reject: [V:04-1] EarlyOnboarding: record account setup motivation in `action_context` [extensions/WikimediaEvents] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344594 (https://phabricator.wikimedia.org/T437339) (owner: 10Michael Große)
[08:13:50] <wikibugs>	 (03CR) 10Michael Große: "A trailing comma 🤦" [extensions/WikimediaEvents] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344594 (https://phabricator.wikimedia.org/T437339) (owner: 10Michael Große)
[08:14:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1021.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[08:15:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1020.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[08:16:07] <logmsgbot>	 ayounsi@cumin1004 depool-rack (PID 584827) is awaiting input
[08:16:32] <jinxer-wm>	 FIRING: [2x] Not accepting/receiving prefixes from anycast BGP peer: Alert for device cr1-codfw.wikimedia.org - Not accepting/receiving prefixes from anycast BGP peer   - https://alerts.wikimedia.org/?q=alertname%3DNot+accepting%2Freceiving+prefixes+from+anycast+BGP+peer
[08:17:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[08:17:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[08:18:49] <wikibugs>	 (03PS2) 10Michael Große: EarlyOnboarding: record account setup motivation in `action_context` [extensions/WikimediaEvents] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344594 (https://phabricator.wikimedia.org/T437339)
[08:19:38] <wikibugs>	 06SRE, 06Infrastructure-Foundations: Integrate Bookworm 12.15 point update - https://phabricator.wikimedia.org/T434631#12358515 (10MoritzMuehlenhoff)
[08:20:28] <logmsgbot>	 !log ayounsi@cumin1004 END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'depool' for drmrs rack B12
[08:22:07] <XioNoX>	 !log asw1-b12-drmrs> request system reboot - T437984
[08:22:10] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[08:22:11] <stashbot>	 T437984: Junos file descriptors exhaustion - https://phabricator.wikimedia.org/T437984
[08:25:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1016.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[08:25:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1016.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[08:27:07] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[08:27:39] <jinxer-wm>	 FIRING: [4x] CoreBGPDown: Core BGP session down between cr1-drmrs and asw1-b12-drmrs (185.15.58.143) - group Switch - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status  - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown
[08:28:05] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[08:29:56] <jinxer-wm>	 FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr1-drmrs:et-0/0/1 (Core: asw1-b12-drmrs:et-0/0/48) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down  - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown
[08:30:07] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1019.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[08:30:41] <wikibugs>	 (03PS1) 10Arnaudb: gerrit: hold GerritDiskSpaceExhaustionIncoming for 30m [alerts] - 10https://gerrit.wikimedia.org/r/1344595 (https://phabricator.wikimedia.org/T439039)
[08:31:00] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{dse-k8s-worker1015.eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
[08:31:03] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1015.eqiad.wmnet
[08:31:04] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1015.eqiad.wmnet
[08:31:10] <jinxer-wm>	 FIRING: [4x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[08:31:15] <jinxer-wm>	 FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[08:32:05] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[08:32:33] <wikibugs>	 (03PS1) 10Btullis: dse-k8s-eqiad: put dse-k8s-worker1016 back into service [puppet] - 10https://gerrit.wikimedia.org/r/1344597 (https://phabricator.wikimedia.org/T430658)
[08:32:48] <wikibugs>	 (03PS2) 10Btullis: dse-k8s-eqiad: put dse-k8s-worker1016 back into service [puppet] - 10https://gerrit.wikimedia.org/r/1344597 (https://phabricator.wikimedia.org/T430658)
[08:34:41] <jinxer-wm>	 FIRING: [7x] JobUnavailable: Reduced availability for job fifo_log_demux in ops@drmrs - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[08:36:15] <jinxer-wm>	 RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[08:37:04] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1015.eqiad.wmnet
[08:37:05] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[08:37:07] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[08:37:45] <jinxer-wm>	 FIRING: WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure
[08:37:54] <logmsgbot>	 !log btullis@cumin1004 END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) pool for host dse-k8s-worker1015.eqiad.wmnet
[08:37:54] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{dse-k8s-worker1015.eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
[08:37:59] <icinga-wm>	 PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 59290544 and 3 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[08:38:27] <jinxer-wm>	 FIRING: [6x] ProbeDown: Service restbase2029:7231 has failed probes (http_restbase_discovery_wmnet_ip4)   - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[08:38:59] <icinga-wm>	 RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 3212464 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[08:39:23] <wikibugs>	 (03PS1) 10Arnaudb: gerrit: rotate httpd logs of all vhosts [puppet] - 10https://gerrit.wikimedia.org/r/1344598 (https://phabricator.wikimedia.org/T439104)
[08:39:41] <jinxer-wm>	 RESOLVED: [26x] JobUnavailable: Reduced availability for job benthos in ops@drmrs - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[08:39:56] <jinxer-wm>	 RESOLVED: [2x] CoreRouterInterfaceDown: Core router interface down - cr1-drmrs:et-0/0/1 (Core: asw1-b12-drmrs:et-0/0/48) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down  - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown
[08:40:07] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1018.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1012.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[08:40:08] <wikibugs>	 06SRE, 06Commons, 10MediaWiki-File-management, 06Traffic, and 2 others: Varnish serving outdated version at original/non-thumb URL of overwritten file upload - https://phabricator.wikimedia.org/T425216#12358708 (10Tgr) >>! In T425216#12351266, @BBlack wrote: > To not copy any more application-specific busi...
[08:40:48] <wikibugs>	 (03PS2) 10Jgiannelos: prv: Enable parsoid rendering for 5 wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344329 (https://phabricator.wikimedia.org/T438998)
[08:41:05] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1018.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[08:41:10] <jinxer-wm>	 FIRING: [4x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[08:41:18] <wikibugs>	 (03PS3) 10Gmodena: wdqs: add wikidata prefixes config [deployment-charts] - 10https://gerrit.wikimedia.org/r/1343616 (https://phabricator.wikimedia.org/T438476)
[08:42:05] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[08:42:07] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[08:42:39] <jinxer-wm>	 RESOLVED: [4x] CoreBGPDown: Core BGP session down between cr1-drmrs and asw1-b12-drmrs (185.15.58.143) - group Switch - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status  - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown
[08:45:00] <logmsgbot>	 !log ayounsi@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on asw1-b13-drmrs,asw1-b13-drmrs IPv6,asw1-b13-drmrs.mgmt with reason: Switch upgrade
[08:45:24] <logmsgbot>	 !log ayounsi@cumin1004 START - Cookbook sre.hosts.remove-downtime for asw1-b12-drmrs,asw1-b12-drmrs IPv6,asw1-b12-drmrs.mgmt
[08:45:26] <logmsgbot>	 !log ayounsi@cumin1004 END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for asw1-b12-drmrs,asw1-b12-drmrs IPv6,asw1-b12-drmrs.mgmt
[08:46:32] <moritzm>	 !log uploaded debuerreotype 0.15-1.1+wmf13u1 to component/main from trixie-wikimedia T438866
[08:46:35] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[08:46:36] <stashbot>	 T438866: Ignore Valid-Until for WMF container images - https://phabricator.wikimedia.org/T438866
[08:46:39] <logmsgbot>	 !log ayounsi@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 19 hosts with reason: Switches upgrade
[08:47:15] <jinxer-wm>	 FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook  - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[08:47:54] <logmsgbot>	 !log ayounsi@cumin1004 START - Cookbook sre.network.depool-rack with action 'pool' for drmrs rack B12
[08:48:01] <wikibugs>	 (03CR) 10Marostegui: [C:03+1] "Given that the DC has been post-poned I think we can merge this (and revert if needed)." [puppet] - 10https://gerrit.wikimedia.org/r/1343590 (https://phabricator.wikimedia.org/T407329) (owner: 10Hnowlan)
[08:49:17] <jinxer-wm>	 FIRING: ProbeDown: Service wdqs1016:443 has failed probes (http_wdqs_main_external_search_sparql_endpoint_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#wdqs1016:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[08:49:21] <logmsgbot>	 !log ayounsi@cumin1004 END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'pool' for drmrs rack B12
[08:50:24] <wikibugs>	 (03CR) 10Dpogorzelski: [C:03+2] ml-serve: repartition ml-serve1013 and 1014 to 16x96GB [puppet] - 10https://gerrit.wikimedia.org/r/1344593 (https://phabricator.wikimedia.org/T436928) (owner: 10Dpogorzelski)
[08:52:02] <wikibugs>	 (03CR) 10Klausman: [C:03+1] dse-k8s-eqiad: put dse-k8s-worker1016 back into service [puppet] - 10https://gerrit.wikimedia.org/r/1344597 (https://phabricator.wikimedia.org/T430658) (owner: 10Btullis)
[08:52:15] <jinxer-wm>	 RESOLVED: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook  - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[08:52:29] <wikibugs>	 (03CR) 10Btullis: [C:03+2] dse-k8s-eqiad: put dse-k8s-worker1016 back into service [puppet] - 10https://gerrit.wikimedia.org/r/1344597 (https://phabricator.wikimedia.org/T430658) (owner: 10Btullis)
[08:53:40] <wikibugs>	 06SRE, 06Infrastructure-Foundations, 13Patch-For-Review, 07Upstream: Ignore Valid-Until for WMF container images - https://phabricator.wikimedia.org/T438866#12358803 (10MoritzMuehlenhoff) a:03MoritzMuehlenhoff I've updated build2004 to a package which includes Tim's patch. The next weekly rebuild will ha...
[08:53:51] <jinxer-wm>	 FIRING: CoreOutboundSaturation: Core link outbound traffic above 90% capacity - asw1-by27-esams:ae0 (Core: ...
[08:53:51] <jinxer-wm>	 cr1-esams) #page - https://wikitech.wikimedia.org/wiki/Network_monitoring#Primary_outbound_port_utilization_over_90% - https://grafana.wikimedia.org/d/d968a627-b6f6-47fc-9316-e058854a4945/throughput-network-device-interfaces?var-site=esams+prometheus%2Fops&var-device=asw1-by27-esams:9804&var-interface=ae0 - https://alerts.wikimedia.org/?q=alertname%3DCoreOutboundSaturation
[08:54:15] <jinxer-wm>	 FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook  - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[08:54:39] <logmsgbot>	 !log filippo@cumin1004 START - Cookbook sre.hosts.provision for host cloudvirt1078.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
[08:54:51] <wikibugs>	 07sre-alert-triage, 06Infrastructure-Foundations: Alert in need of triage: SystemdUnitFailed (instance idm-test1001:9100) - https://phabricator.wikimedia.org/T381947#12358805 (10SLyngshede-WMF) 05Open→03Resolved
[08:54:59] <icinga-wm>	 PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 52637984 and 23 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[08:55:48] <logmsgbot>	 !log filippo@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudvirt1078.eqiad.wmnet with reason: provision
[08:56:06] <logmsgbot>	 !log ayounsi@cumin1004 START - Cookbook sre.network.depool-rack with action 'depool' for drmrs rack B13
[08:56:07] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1016.eqiad.wmnet, wdqs1020.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[08:56:07] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1021.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1012.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[08:57:05] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[08:57:07] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[08:58:54] <moritzm>	 !log installing node-lodash security updates
[08:58:55] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[08:59:15] <jinxer-wm>	 FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook  - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[08:59:17] <jinxer-wm>	 FIRING: [2x] ProbeDown: Service wdqs1016:443 has failed probes (http_wdqs_main_external_search_sparql_endpoint_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#wdqs1016:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[08:59:45] <logmsgbot>	 !log filippo@cumin1004 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudvirt1078.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART
[09:00:05] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[09:00:07] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1012.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[09:00:13] <logmsgbot>	 !log ayounsi@cumin1004 END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'depool' for drmrs rack B13
[09:00:41] <logmsgbot>	 !log jelto@cumin1004 END (PASS) - Cookbook sre.gitlab.reboot-runner (exit_code=0) rolling reboot on A:gitlab-runner
[09:01:05] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[09:01:20] <XioNoX>	 !log asw1-b13-drmrs> request system reboot - T437984
[09:01:20] <wikibugs>	 (03CR) 10Muehlenhoff: [C:03+2] uwsgi: Remove support for buster [puppet] - 10https://gerrit.wikimedia.org/r/1344310 (owner: 10Muehlenhoff)
[09:01:23] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[09:01:23] <stashbot>	 T437984: Junos file descriptors exhaustion - https://phabricator.wikimedia.org/T437984
[09:02:07] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[09:02:42] <wikibugs>	 (03Abandoned) 10Muehlenhoff: build-bare-slim: Avoid snapshot.debian.org by using --non-debian (WIP) [puppet] - 10https://gerrit.wikimedia.org/r/1341110 (https://phabricator.wikimedia.org/T437829) (owner: 10Muehlenhoff)
[09:03:43] <icinga-wm>	 PROBLEM - Host ml-serve1013 is DOWN: PING CRITICAL - Packet loss = 100%
[09:03:51] <jinxer-wm>	 RESOLVED: CoreOutboundSaturation: Core link outbound traffic above 90% capacity - asw1-by27-esams:ae0 (Core: ...
[09:03:51] <jinxer-wm>	 cr1-esams) #page - https://wikitech.wikimedia.org/wiki/Network_monitoring#Primary_outbound_port_utilization_over_90% - https://grafana.wikimedia.org/d/d968a627-b6f6-47fc-9316-e058854a4945/throughput-network-device-interfaces?var-site=esams+prometheus%2Fops&var-device=asw1-by27-esams:9804&var-interface=ae0 - https://alerts.wikimedia.org/?q=alertname%3DCoreOutboundSaturation
[09:04:15] <jinxer-wm>	 FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook  - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[09:04:17] <jinxer-wm>	 RESOLVED: [2x] ProbeDown: Service wdqs1016:443 has failed probes (http_wdqs_main_external_search_sparql_endpoint_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#wdqs1016:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[09:04:22] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1016.eqiad.wmnet
[09:04:59] <icinga-wm>	 PROBLEM - Router interfaces on mr1-drmrs is CRITICAL: CRITICAL: host 185.15.58.130, interfaces up: 34, down: 1, dormant: 0, excluded: 0, unused: 0: https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down
[09:05:43] <logmsgbot>	 !log vgutierrez@cumin1004 START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_ulsfo and A:cp - 3.2.23 upgrade (T438828)
[09:05:48] <stashbot>	 T438828: Upgrade HAProxy to 3.2.23 on cp hosts - https://phabricator.wikimedia.org/T438828
[09:06:07] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1018.eqiad.wmnet, wdqs1012.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[09:06:11] <icinga-wm>	 RECOVERY - Host ml-serve1013 is UP: PING OK - Packet loss = 0%, RTA = 0.29 ms
[09:06:51] <wikibugs>	 06SRE, 06Infrastructure-Foundations, 10Observability-Alerting, 13Patch-For-Review: get-raid-status-perccli not working as expected - https://phabricator.wikimedia.org/T354254#12358827 (10SLyngshede-WMF) 05Open→03Declined No longer in use
[09:07:07] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[09:07:45] <jinxer-wm>	 RESOLVED: WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure
[09:08:01] <wikibugs>	 10SRE-tools, 10Icinga, 06Infrastructure-Foundations: get-raid-status-perccli should allow for commands to return non-zero exit code - https://phabricator.wikimedia.org/T320998#12358833 (10SLyngshede-WMF) 05Open→03Declined Script no longer exists.
[09:09:15] <jinxer-wm>	 RESOLVED: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook  - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[09:10:15] <jinxer-wm>	 FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.92% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy
[09:10:30] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1016.eqiad.wmnet
[09:12:05] <Dreamy_Jazz>	 jouncebot: nowandnext
[09:12:05] <jouncebot>	 No deployments scheduled for the next 2 hour(s) and 47 minute(s)
[09:12:05] <jouncebot>	 In 2 hour(s) and 47 minute(s): Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260924T1200)
[09:12:31] <wikibugs>	 (03PS1) 10Arthur taylor: wikidata-query-gui: bump to latest version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344605 (https://phabricator.wikimedia.org/T427589)
[09:13:59] <icinga-wm>	 RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 83312 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[09:14:13] <wikibugs>	 (03CR) 10Hasan Akgün (WMDE): [C:03+1] wikidata-query-gui: bump to latest version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344605 (https://phabricator.wikimedia.org/T427589) (owner: 10Arthur taylor)
[09:15:12] <jinxer-wm>	 FIRING: [12x] JobUnavailable: Reduced availability for job benthos in ops@drmrs - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[09:15:12] <jinxer-wm>	 FIRING: [28x] JobUnavailable: Reduced availability for job benthos in ops@drmrs - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[09:15:19] <icinga-wm>	 PROBLEM - Host 2a02:ec80:600:2:185:15:58:37 is DOWN: PING CRITICAL - Packet loss = 100%
[09:15:20] <jinxer-wm>	 RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 24.92% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy
[09:15:35] <icinga-wm>	 RECOVERY - Host 2a02:ec80:600:2:185:15:58:37 is UP: PING OK - Packet loss = 0%, RTA = 82.00 ms
[09:16:01] <icinga-wm>	 RECOVERY - Router interfaces on mr1-drmrs is OK: OK: host 185.15.58.130, interfaces up: 35, down: 0, dormant: 0, excluded: 0, unused: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down
[09:16:05] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[09:16:10] <jinxer-wm>	 FIRING: [4x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[09:16:19] <codders>	 jouncebot: nowandnext
[09:16:19] <jouncebot>	 No deployments scheduled for the next 2 hour(s) and 43 minute(s)
[09:16:19] <jouncebot>	 In 2 hour(s) and 43 minute(s): Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260924T1200)
[09:17:05] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[09:17:23] <jinxer-wm>	 FIRING: [5x] CertAlmostExpired: gNMI TLS certificate for cr1-drmrs.wikimedia.org is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired  - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired
[09:17:32] <wikibugs>	 (03CR) 10Arthur taylor: [C:03+2] wikidata-query-gui: bump to latest version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344605 (https://phabricator.wikimedia.org/T427589) (owner: 10Arthur taylor)
[09:17:37] <Dreamy_Jazz>	 Scap appears still locked, unsure if Thursday morning UTC is still on the cards for unlocking
[09:17:45] <jinxer-wm>	 FIRING: WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure
[09:18:27] <jinxer-wm>	 FIRING: [10x] ProbeDown: Service restbase2029:7231 has failed probes (http_restbase_discovery_wmnet_ip4)   - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[09:18:36] <hnowlan>	 Dreamy_Jazz: we're just assessing whether that can be unlocked
[09:18:42] <Dreamy_Jazz>	 Thanks
[09:20:07] <wikibugs>	 (03Merged) 10jenkins-bot: wikidata-query-gui: bump to latest version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344605 (https://phabricator.wikimedia.org/T427589) (owner: 10Arthur taylor)
[09:20:12] <jinxer-wm>	 RESOLVED: [31x] JobUnavailable: Reduced availability for job benthos in ops@drmrs - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[09:21:28] <logmsgbot>	 !log arthurtaylor@deploy1003 helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply
[09:21:31] <jinxer-wm>	 FIRING: RedisReplicaDown: Redis replica down rdb2014:16380 redis_misc - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_misc - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_misc&var-instance=rdb2014:16380 - https://alerts.wikimedia.org/?q=alertname%3DRedisReplicaDown
[09:21:56] <jinxer-wm>	 FIRING: GitlabPackagePullerFailedOnPrepare: Package puller has some run errors while preparing projects. - TODO - TODO - https://alerts.wikimedia.org/?q=alertname%3DGitlabPackagePullerFailedOnPrepare
[09:21:57] <logmsgbot>	 !log arthurtaylor@deploy1003 helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply
[09:22:12] <logmsgbot>	 !log arthurtaylor@deploy1003 helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply
[09:22:37] <logmsgbot>	 !log arthurtaylor@deploy1003 helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply
[09:22:49] <logmsgbot>	 !log arthurtaylor@deploy1003 helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply
[09:23:11] <logmsgbot>	 !log arthurtaylor@deploy1003 helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply
[09:24:23] <wikibugs>	 (03PS2) 10Muehlenhoff: Remove buster from the debian module [puppet] - 10https://gerrit.wikimedia.org/r/1344306
[09:24:39] <wikibugs>	 10ops-eqiad, 06DC-Ops: Alert for device ps1-f6-eqiad.mgmt.eqiad.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T439113 (10phaultfinder) 03NEW
[09:25:07] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1021.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[09:25:27] <logmsgbot>	 !log ayounsi@cumin1004 START - Cookbook sre.network.depool-rack with action 'pool' for drmrs rack B13
[09:26:05] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1021.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[09:26:28] <logmsgbot>	 !log ayounsi@cumin1004 END (PASS) - Cookbook sre.network.depool-rack (exit_code=0) with action 'pool' for drmrs rack B13
[09:27:05] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[09:27:07] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[09:27:10] <logmsgbot>	 !log ayounsi@cumin1004 START - Cookbook sre.dns.admin DNS admin: pool drmrs [reason: switch upgrade, T437984]
[09:27:12] <logmsgbot>	 !log ayounsi@cumin1004 END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool drmrs [reason: switch upgrade, T437984]
[09:27:13] <stashbot>	 T437984: Junos file descriptors exhaustion - https://phabricator.wikimedia.org/T437984
[09:27:28] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Remove buster from the debian module [puppet] - 10https://gerrit.wikimedia.org/r/1344306 (owner: 10Muehlenhoff)
[09:28:24] <jinxer-wm>	 FIRING: SystemdUnitFailed: amd-smi-gpu-partition.service on ml-serve1014:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[09:29:38] <wikibugs>	 10ops-eqiad, 06SRE, 06DC-Ops: Alert for device ps1-c6-eqiad.mgmt.eqiad.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T439096#12358939 (10phaultfinder)
[09:31:05] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1018.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[09:31:07] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1018.eqiad.wmnet, wdqs1019.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[09:31:10] <jinxer-wm>	 FIRING: [4x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[09:32:05] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[09:32:07] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[09:32:13] <wikibugs>	 (03PS1) 10Santiago Faci: Test Kitchen UI: Deploy v2.0.1 release to staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344607 (https://phabricator.wikimedia.org/T421814)
[09:33:24] <jinxer-wm>	 RESOLVED: SystemdUnitFailed: amd-smi-gpu-partition.service on ml-serve1014:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[09:34:15] <wikibugs>	 (03PS1) 10Santiago Faci: Test Kitchen UI: Deploy v2.0.1 release to production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344609 (https://phabricator.wikimedia.org/T421814)
[09:36:05] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1016.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[09:37:05] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[09:40:06] <wikibugs>	 (03PS1) 10Santiago Faci: Deploy GrowthBook 5.1.0 to staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344610 (https://phabricator.wikimedia.org/T438954)
[09:40:29] <logmsgbot>	 !log vgutierrez@puppetserver1001 conftool action : set/pooled=no; selector: name=cirrussearch1120.eqiad.wmnet
[09:41:05] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[09:41:11] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1020.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[09:41:48] <jinxer-wm>	 FIRING: PuppetFailure: Puppet has failed on cloudidp2001-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[09:42:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[09:42:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[09:42:45] <jinxer-wm>	 RESOLVED: WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure
[09:43:22] <icinga-wm>	 PROBLEM - Host dse-k8s-worker1016 is DOWN: PING CRITICAL - Packet loss = 100%
[09:44:50] <icinga-wm>	 RECOVERY - Host dse-k8s-worker1016 is UP: PING OK - Packet loss = 0%, RTA = 0.55 ms
[09:46:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1011.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[09:46:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1012.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[09:46:15] <jinxer-wm>	 FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[09:46:27] <wikibugs>	 06SRE, 06Infrastructure-Foundations: Integrate Trixie 13.7 point update - https://phabricator.wikimedia.org/T437715#12359072 (10MoritzMuehlenhoff)
[09:46:54] <moritzm>	 !log installing bind9 security updates (client-side tools/libs only)
[09:46:55] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[09:47:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[09:48:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[09:49:44] <wikibugs>	 (03CR) 10Majavah: [C:03+2] P:etcd::tlsproxy: Support client cert auth to upstream (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1343964 (https://phabricator.wikimedia.org/T410721) (owner: 10Majavah)
[09:51:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1012.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[09:51:15] <jinxer-wm>	 RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[09:51:48] <jinxer-wm>	 RESOLVED: PuppetFailure: Puppet has failed on cloudidp2001-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[09:52:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[09:53:20] <icinga-wm>	 PROBLEM - Host dse-k8s-worker1016 is DOWN: PING CRITICAL - Packet loss = 100%
[09:54:50] <icinga-wm>	 RECOVERY - Host dse-k8s-worker1016 is UP: PING OK - Packet loss = 0%, RTA = 0.37 ms
[09:55:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[09:55:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1018.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[09:55:08] <wikibugs>	 06SRE, 10SRE-Access-Requests: Requesting access to deployment for dpislaru - https://phabricator.wikimedia.org/T438827#12359099 (10tappof)
[09:56:56] <wikibugs>	 (03CR) 10Tiziano Fogli: [C:03+2] admin/data: grant access to ashhan (analytics_privatedata_users l3) [puppet] - 10https://gerrit.wikimedia.org/r/1344305 (https://phabricator.wikimedia.org/T438538) (owner: 10Tiziano Fogli)
[09:57:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[09:57:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[09:58:24] <wikibugs>	 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics-privatedata-users for AHan-WMF - https://phabricator.wikimedia.org/T438538#12359109 (10tappof) 05Open→03Resolved a:03tappof Patch merged. Access granted.
[09:59:17] <jinxer-wm>	 FIRING: ProbeDown: Service wdqs1013:443 has failed probes (http_wdqs_main_external_search_sparql_endpoint_ip6) - https://wikitech.wikimedia.org/wiki/Runbook#wdqs1013:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[10:00:33] <wikibugs>	 (03CR) 10Phuedx: [C:03+1] Deploy GrowthBook 5.1.0 to staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344610 (https://phabricator.wikimedia.org/T438954) (owner: 10Santiago Faci)
[10:03:03] <moritzm>	 !log installing apr-util security updates
[10:03:04] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:03:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[10:04:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1018.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[10:05:56] <wikibugs>	 (03PS3) 10Tiziano Fogli: admin/data: grant access to segt (analytics_privatedata_users l3) [puppet] - 10https://gerrit.wikimedia.org/r/1344304 (https://phabricator.wikimedia.org/T438767)
[10:07:08] <wikibugs>	 (03PS1) 10Atsuko: airflow3: track next-latest docker image version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344615 (https://phabricator.wikimedia.org/T434896)
[10:07:22] <icinga-wm>	 PROBLEM - Host dse-k8s-worker1015 is DOWN: PING CRITICAL - Packet loss = 100%
[10:07:53] <wikibugs>	 (03CR) 10Tiziano Fogli: [C:03+2] admin/data: grant access to segt (analytics_privatedata_users l3) [puppet] - 10https://gerrit.wikimedia.org/r/1344304 (https://phabricator.wikimedia.org/T438767) (owner: 10Tiziano Fogli)
[10:08:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[10:08:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[10:08:13] <logmsgbot>	 !log vgutierrez@cumin1004 START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-upload_eqsin and A:cp - 3.2.23 upgrade (T438828)
[10:08:17] <stashbot>	 T438828: Upgrade HAProxy to 3.2.23 on cp hosts - https://phabricator.wikimedia.org/T438828
[10:08:37] <wikibugs>	 (03CR) 10Santiago Faci: [C:03+2] Deploy GrowthBook 5.1.0 to staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344610 (https://phabricator.wikimedia.org/T438954) (owner: 10Santiago Faci)
[10:08:50] <icinga-wm>	 RECOVERY - Host dse-k8s-worker1015 is UP: PING OK - Packet loss = 0%, RTA = 0.31 ms
[10:09:22] <wikibugs>	 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics-privatedata-users (Level 3) for SEgt-WMF - https://phabricator.wikimedia.org/T438767#12359136 (10tappof) 05Open→03Resolved a:03tappof Patch merged. Access granted.
[10:09:36] <icinga-wm>	 PROBLEM - Postfix SMTP on crm2001 is CRITICAL: CRITICAL - Certificate crm2001.codfw.wmnet expires in 15 day(s) (Sat 10 Oct 2026 10:09:00 AM GMT +0000). https://wikitech.wikimedia.org/wiki/Mail%23Troubleshooting
[10:11:01] <wikibugs>	 (03Merged) 10jenkins-bot: Deploy GrowthBook 5.1.0 to staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344610 (https://phabricator.wikimedia.org/T438954) (owner: 10Santiago Faci)
[10:11:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1012.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[10:11:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1012.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[10:11:55] <logmsgbot>	 !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-text_ulsfo and A:cp - 3.2.23 upgrade (T438828)
[10:12:04] <logmsgbot>	 !log sfaci@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply
[10:12:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[10:12:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[10:13:18] <wikibugs>	 (03PS1) 10Tiziano Fogli: admin/data: grant access to dpislaru (deployment) [puppet] - 10https://gerrit.wikimedia.org/r/1344616 (https://phabricator.wikimedia.org/T438827)
[10:14:01] <jinxer-wm>	 FIRING: RedisInstanceDown: Redis instance down rdb2013:16380 redis_misc - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_misc - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_misc&var-instance=rdb2013:16380 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown
[10:14:17] <jinxer-wm>	 RESOLVED: ProbeDown: Service wdqs1013:443 has failed probes (http_wdqs_main_external_search_sparql_endpoint_ip6) - https://wikitech.wikimedia.org/wiki/Runbook#wdqs1013:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[10:14:23] <wikibugs>	 (03CR) 10Brouberol: [C:03+1] airflow3: track next-latest docker image version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344615 (https://phabricator.wikimedia.org/T434896) (owner: 10Atsuko)
[10:15:46] <wikibugs>	 (03PS3) 10Muehlenhoff: Remove buster from the debian module [puppet] - 10https://gerrit.wikimedia.org/r/1344306
[10:16:15] <jinxer-wm>	 FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[10:16:16] <wikibugs>	 (03CR) 10Atsuko: "change image pull policy to always only for airflow 3" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344615 (https://phabricator.wikimedia.org/T434896) (owner: 10Atsuko)
[10:16:56] <wikibugs>	 (03PS1) 10Marco Fossati: Instrument five-arm image carousel retest [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344617 (https://phabricator.wikimedia.org/T431362)
[10:17:25] <logmsgbot>	 !log hnowlan@deploy1003 Forcefully removing global lock: Unlocking scap after restoration of power in codfw
[10:17:32] <logmsgbot>	 !log rzl@deploy1003 Unlocked for deployment [ALL REPOSITORIES]: No deployments please, as we're still cleaning up from the codfw power incident T439010. Thursday UTC morning at the earliest, but please ask SRE oncall. (duration: 653m 55s)
[10:17:39] <wikibugs>	 (03PS1) 10Samtar: CommonSettings: Load ModeratorToolkit extension [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1343958 (https://phabricator.wikimedia.org/T438334)
[10:17:43] <wikibugs>	 (03PS1) 10Samtar: InitialiseSettings: Enable ModeratorToolkit on test.wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1343957 (https://phabricator.wikimedia.org/T438334)
[10:17:50] <hnowlan>	 Dreamy_Jazz: lock as been removed
[10:17:51] <wikibugs>	 (03PS2) 10Samtar: InitialiseSettings: Enable ModeratorToolkit on test.wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1343957 (https://phabricator.wikimedia.org/T438334)
[10:17:56] <Dreamy_Jazz>	 Thanks!
[10:17:59] <wikibugs>	 (03PS2) 10Samtar: CommonSettings: Load ModeratorToolkit extension [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1343958 (https://phabricator.wikimedia.org/T438334)
[10:18:00] <hnowlan>	 please be mindful and careful with your changes 
[10:18:10] <Dreamy_Jazz>	 jouncebot: nowandnext
[10:18:10] <jouncebot>	 No deployments scheduled for the next 1 hour(s) and 41 minute(s)
[10:18:10] <jouncebot>	 In 1 hour(s) and 41 minute(s): Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260924T1200)
[10:18:18] <wikibugs>	 (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, September 24 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#dep" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344617 (https://phabricator.wikimedia.org/T431362) (owner: 10Marco Fossati)
[10:18:19] <Dreamy_Jazz>	 Going to proceed with a mediawiki config change
[10:18:32] <wikibugs>	 (03CR) 10Volans: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1344303 (https://phabricator.wikimedia.org/T410721) (owner: 10Majavah)
[10:18:47] <wikibugs>	 (03PS1) 10Marco Fossati: Wire image carousel retest instrumentation [extensions/ReaderExperiments] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344619 (https://phabricator.wikimedia.org/T431362)
[10:18:51] <wikibugs>	 (03CR) 10TrainBranchBot: [C:03+2] "Approved by dreamyjazz@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344281 (https://phabricator.wikimedia.org/T438860) (owner: 10Dreamy Jazz)
[10:18:54] <Lucas_WMDE>	 FWIW awight also tried to deploy a backport this morning according to scap
[10:18:59] <wikibugs>	 (03CR) 10Volans: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1344318 (https://phabricator.wikimedia.org/T438971) (owner: 10Majavah)
[10:19:05] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Remove buster from the debian module [puppet] - 10https://gerrit.wikimedia.org/r/1344306 (owner: 10Muehlenhoff)
[10:19:11] <Dreamy_Jazz>	 Yeah seems like during the morning UTC backport window
[10:19:14] <Lucas_WMDE>	 I’ll let them know that it’s unlocked (and being mindful and careful etc.)
[10:19:25] <Dreamy_Jazz>	 👍
[10:20:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1016.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1012.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[10:20:48] <wikibugs>	 (03Merged) 10jenkins-bot: WikimediaAntiAbuse: Enable likely vandalism classifier on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344281 (https://phabricator.wikimedia.org/T438860) (owner: 10Dreamy Jazz)
[10:21:15] <jinxer-wm>	 FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook  - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[10:21:28] <logmsgbot>	 !log dreamyjazz@deploy1003 Started scap sync-world: Backport for [[gerrit:1344281|WikimediaAntiAbuse: Enable likely vandalism classifier on testwiki (T438860)]]
[10:21:31] <stashbot>	 T438860: Enable the vandalism classifier alpha test on testwiki - https://phabricator.wikimedia.org/T438860
[10:22:14] <logmsgbot>	 !log sfaci@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply
[10:23:23] <logmsgbot>	 !log filippo@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudvirt1079.eqiad.wmnet with reason: provision
[10:23:39] <jinxer-wm>	 FIRING: TransitBGPDown: Transit BGP session down between cr4-ulsfo and NTT (129.250.204.5) - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=ulsfo&var-device=cr4-ulsfo:9804&var-bgp_group=Transit4&var-bgp_neighbor=NTT - https://alerts.wikimedia.org/?q=alertname%3DTransitBGPDown
[10:24:32] <wikibugs>	 (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, September 24 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#dep" [extensions/ReaderExperiments] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344619 (https://phabricator.wikimedia.org/T431362) (owner: 10Marco Fossati)
[10:24:51] <wikibugs>	 10ops-eqiad, 06SRE, 06DC-Ops: Interface eno1 on wikikube-worker1160 has the wrong speed - https://phabricator.wikimedia.org/T439094#12359244 (10Jclark-ctr) a:03Jclark-ctr
[10:24:56] <jinxer-wm>	 FIRING: CoreRouterInterfaceDown: Core router interface down - cr4-ulsfo:xe-0/1/0 (Transit: NTT (234631)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr4-ulsfo:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown
[10:25:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[10:25:20] <wikibugs>	 (03CR) 10Btullis: [C:03+1] ceph::osd: don't use puppet to delete an absented OSD [puppet] - 10https://gerrit.wikimedia.org/r/1344261 (https://phabricator.wikimedia.org/T438823) (owner: 10Brouberol)
[10:25:29] <wikibugs>	 (03CR) 10Muehlenhoff: [C:03+1] "Looks good" [puppet] - 10https://gerrit.wikimedia.org/r/1344616 (https://phabricator.wikimedia.org/T438827) (owner: 10Tiziano Fogli)
[10:25:38] <wikibugs>	 (03CR) 10Brouberol: [C:03+2] ceph::osd: don't use puppet to delete an absented OSD [puppet] - 10https://gerrit.wikimedia.org/r/1344261 (https://phabricator.wikimedia.org/T438823) (owner: 10Brouberol)
[10:25:42] <logmsgbot>	 !log dreamyjazz@deploy1003 dreamyjazz: Backport for [[gerrit:1344281|WikimediaAntiAbuse: Enable likely vandalism classifier on testwiki (T438860)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
[10:25:55] <wikibugs>	 10ops-eqiad, 06SRE, 06DC-Ops: Alert for device ps1-c6-eqiad.mgmt.eqiad.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T439096#12359250 (10Jclark-ctr) p:05Triage→03Medium a:03Jclark-ctr
[10:26:01] <wikibugs>	 10ops-eqiad, 06SRE, 06DC-Ops: Alert for device ps1-c6-eqiad.mgmt.eqiad.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T439096#12359254 (10Jclark-ctr) 05Open→03Resolved
[10:26:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[10:26:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[10:26:11] <wikibugs>	 (03CR) 10Tiziano Fogli: [C:03+2] admin/data: grant access to dpislaru (deployment) [puppet] - 10https://gerrit.wikimedia.org/r/1344616 (https://phabricator.wikimedia.org/T438827) (owner: 10Tiziano Fogli)
[10:26:15] <jinxer-wm>	 RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[10:26:36] <wikibugs>	 10ops-eqiad, 06SRE, 06DC-Ops: Alert for device ps1-f6-eqiad.mgmt.eqiad.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T439113#12359257 (10Jclark-ctr) 05Open→03Resolved a:03Jclark-ctr
[10:26:38] <logmsgbot>	 !log trueg@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
[10:26:38] <logmsgbot>	 !log dreamyjazz@deploy1003 dreamyjazz: Continuing with deployment
[10:26:54] <wikibugs>	 (03PS1) 10Dreamy Jazz: AbuseReview: Let specific users and suppressors see vandalism tag [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344621 (https://phabricator.wikimedia.org/T438860)
[10:27:10] <wikibugs>	 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to deployment for dpislaru - https://phabricator.wikimedia.org/T438827#12359262 (10tappof) 05Open→03Resolved Patch merged. Access granted.
[10:27:15] <jinxer-wm>	 FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[10:27:52] <Dreamy_Jazz>	 Let me know if anyone else wants to use scap
[10:28:02] <Dreamy_Jazz>	 I want to deploy a wmf.21 backport after this
[10:28:13] <Dreamy_Jazz>	 So if anyone else is needing scap sooner please ping me
[10:28:27] <Dreamy_Jazz>	 So far deployment of config patch is looking fine
[10:28:39] <jinxer-wm>	 FIRING: [2x] TransitBGPDown: Transit BGP session down between cr4-ulsfo and NTT (129.250.204.5) - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status  - https://alerts.wikimedia.org/?q=alertname%3DTransitBGPDown
[10:29:01] <logmsgbot>	 !log trueg@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
[10:31:21] <wikibugs>	 (03CR) 10Btullis: [C:03+1] airflow3: track next-latest docker image version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344615 (https://phabricator.wikimedia.org/T434896) (owner: 10Atsuko)
[10:31:26] <Dreamy_Jazz>	 Saw a spike in DB connection errors in the logstash errors channel, appears part of the general post power outage issues
[10:31:30] <jinxer-wm>	 RESOLVED: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook  - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[10:31:39] <wikibugs>	 (03CR) 10Atsuko: [C:04-1] "that escalated quickly" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344615 (https://phabricator.wikimedia.org/T434896) (owner: 10Atsuko)
[10:31:44] <Dreamy_Jazz>	 Spike has gone down again
[10:32:02] <logmsgbot>	 !log dreamyjazz@deploy1003 Finished scap sync-world: Backport for [[gerrit:1344281|WikimediaAntiAbuse: Enable likely vandalism classifier on testwiki (T438860)]] (duration: 10m 34s)
[10:32:05] <stashbot>	 T438860: Enable the vandalism classifier alpha test on testwiki - https://phabricator.wikimedia.org/T438860
[10:32:14] <icinga-wm>	 PROBLEM - Check unit status of httpbb_kubernetes_mw-api-int_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-api-int_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
[10:33:40] <wikibugs>	 (03CR) 10TrainBranchBot: [C:03+2] "Approved by dreamyjazz@deploy1003 using scap backport" [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344621 (https://phabricator.wikimedia.org/T438860) (owner: 10Dreamy Jazz)
[10:33:42] <hnowlan>	 significant spike in db writes there
[10:34:19] <logmsgbot>	 !log sfaci@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply
[10:34:58] <wikibugs>	 (03Merged) 10jenkins-bot: AbuseReview: Let specific users and suppressors see vandalism tag [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344621 (https://phabricator.wikimedia.org/T438860) (owner: 10Dreamy Jazz)
[10:35:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[10:35:21] <logmsgbot>	 !log dreamyjazz@deploy1003 Started scap sync-world: Backport for [[gerrit:1344621|AbuseReview: Let specific users and suppressors see vandalism tag (T438860)]]
[10:35:58] <icinga-wm>	 PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 66397672 and 9 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[10:36:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[10:36:26] <logmsgbot>	 !log filippo@cumin1004 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on cloudvirt1080.eqiad.wmnet with reason: provision
[10:36:58] <icinga-wm>	 RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 2408832 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[10:37:04] <wikibugs>	 (03PS2) 10Atsuko: airflow3: track next-latest docker image version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344615 (https://phabricator.wikimedia.org/T434896)
[10:38:45] <jinxer-wm>	 FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[10:39:36] <logmsgbot>	 !log dreamyjazz@deploy1003 dreamyjazz: Backport for [[gerrit:1344621|AbuseReview: Let specific users and suppressors see vandalism tag (T438860)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
[10:39:40] <stashbot>	 T438860: Enable the vandalism classifier alpha test on testwiki - https://phabricator.wikimedia.org/T438860
[10:39:45] <wikibugs>	 (03CR) 10Btullis: [C:03+1] airflow3: track next-latest docker image version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344615 (https://phabricator.wikimedia.org/T434896) (owner: 10Atsuko)
[10:40:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1021.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1012.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[10:40:14] <logmsgbot>	 !log dreamyjazz@deploy1003 dreamyjazz: Continuing with deployment
[10:40:37] <wikibugs>	 (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, September 29 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#depl" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1343957 (https://phabricator.wikimedia.org/T438334) (owner: 10Samtar)
[10:40:45] <wikibugs>	 (03CR) 10Muehlenhoff: [C:03+2] Assign openldap::replica_mdb role to ldap-replica200[78] [puppet] - 10https://gerrit.wikimedia.org/r/1344259 (https://phabricator.wikimedia.org/T331699) (owner: 10Muehlenhoff)
[10:40:54] <wikibugs>	 (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, September 29 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#depl" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1343958 (https://phabricator.wikimedia.org/T438334) (owner: 10Samtar)
[10:41:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[10:41:30] <jinxer-wm>	 FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook  - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[10:42:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[10:42:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[10:42:20] <wikibugs>	 (03CR) 10Hashar: "hello," [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1305800 (owner: 101Veertje)
[10:42:34] <logmsgbot>	 !log vgutierrez@cumin1004 END (FAIL) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=1) rolling upgrade of HAProxy on A:cp-upload_eqsin and A:cp - 3.2.23 upgrade (T438828)
[10:42:37] <stashbot>	 T438828: Upgrade HAProxy to 3.2.23 on cp hosts - https://phabricator.wikimedia.org/T438828
[10:43:07] <wikibugs>	 (03CR) 10Atsuko: [C:03+2] airflow3: track next-latest docker image version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344615 (https://phabricator.wikimedia.org/T434896) (owner: 10Atsuko)
[10:43:45] <jinxer-wm>	 FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook  - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[10:44:34] <logmsgbot>	 !log sfaci@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply
[10:44:58] <icinga-wm>	 PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 484490760 and 28 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[10:45:22] <jinxer-wm>	 FIRING: GnmiInterfaceCountersDrop: ...
[10:45:22] <jinxer-wm>	 asw1-b13-drmrs is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?orgId=1&from=now-24h&to=now&var-site=%24__all&var-instance=asw1-b13-drmrs:9804&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop
[10:45:31] <logmsgbot>	 !log dreamyjazz@deploy1003 Finished scap sync-world: Backport for [[gerrit:1344621|AbuseReview: Let specific users and suppressors see vandalism tag (T438860)]] (duration: 10m 09s)
[10:45:36] <stashbot>	 T438860: Enable the vandalism classifier alpha test on testwiki - https://phabricator.wikimedia.org/T438860
[10:45:49] <Dreamy_Jazz>	 Finished with scap for now
[10:46:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1018.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[10:46:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1018.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[10:46:25] <jinxer-wm>	 FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[10:46:30] <jinxer-wm>	 FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook  - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[10:46:37] <wikibugs>	 (03Merged) 10jenkins-bot: airflow3: track next-latest docker image version [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344615 (https://phabricator.wikimedia.org/T434896) (owner: 10Atsuko)
[10:46:58] <wikibugs>	 (03PS1) 10Brouberol: ceph::osd: disable the OSD service at deletion [cookbooks] - 10https://gerrit.wikimedia.org/r/1344626 (https://phabricator.wikimedia.org/T438823)
[10:46:58] <icinga-wm>	 RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 936632 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[10:47:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[10:47:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[10:48:45] <jinxer-wm>	 RESOLVED: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook  - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[10:49:53] <wikibugs>	 (03CR) 10Btullis: ceph::osd: disable the OSD service at deletion (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/1344626 (https://phabricator.wikimedia.org/T438823) (owner: 10Brouberol)
[10:50:35] <hnowlan>	 !log deleting stuck mw-web pods in eqiad
[10:50:36] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[10:50:38] <wikibugs>	 (03PS1) 10Marco Fossati: ThumbExtractor: trim nbsp and dangling colons from caption text [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344627 (https://phabricator.wikimedia.org/T435672)
[10:50:50] <wikibugs>	 10ops-eqiad, 06DC-Ops: Alert for device ps1-c6-eqiad.mgmt.eqiad.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T439120 (10phaultfinder) 03NEW
[10:50:55] <wikibugs>	 (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, September 24 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#dep" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344627 (https://phabricator.wikimedia.org/T435672) (owner: 10Marco Fossati)
[10:51:36] <wikibugs>	 (03PS1) 10Btullis: dse-k8s-eqiad: Disable the smaller, original ipv4 ippool [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344628 (https://phabricator.wikimedia.org/T430658)
[10:55:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[10:57:52] <wikibugs>	 (03PS1) 10Marco Fossati: ThumbExtractor: exclude lead infobox images from the carousel [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344630 (https://phabricator.wikimedia.org/T438907)
[10:58:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[10:58:09] <wikibugs>	 (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, September 24 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#dep" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344630 (https://phabricator.wikimedia.org/T438907) (owner: 10Marco Fossati)
[10:58:15] <jinxer-wm>	 FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook  - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[10:59:36] <wikibugs>	 10ops-eqiad, 06DC-Ops: Alert for device ps1-a4-eqiad.mgmt.eqiad.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T439122 (10phaultfinder) 03NEW
[11:01:50] <wikibugs>	 (03CR) 10Atsuko: [C:03+1] dse-k8s-eqiad: Disable the smaller, original ipv4 ippool [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344628 (https://phabricator.wikimedia.org/T430658) (owner: 10Btullis)
[11:03:15] <jinxer-wm>	 FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook  - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[11:03:33] <wikibugs>	 (03PS1) 10Btullis: clouddumps: Allow the new IP Pool in dse-k8s-eqiad to ssh to clouddumps [puppet] - 10https://gerrit.wikimedia.org/r/1344631 (https://phabricator.wikimedia.org/T430658)
[11:04:07] <logmsgbot>	 !log vgutierrez@cumin1004 START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P{cp[5031,5032].*} and A:cp - 3.2.23 upgrade (T438828)
[11:04:08] <wikibugs>	 (03CR) 10CI reject: [V:04-1] clouddumps: Allow the new IP Pool in dse-k8s-eqiad to ssh to clouddumps [puppet] - 10https://gerrit.wikimedia.org/r/1344631 (https://phabricator.wikimedia.org/T430658) (owner: 10Btullis)
[11:04:11] <stashbot>	 T438828: Upgrade HAProxy to 3.2.23 on cp hosts - https://phabricator.wikimedia.org/T438828
[11:04:11] <wikibugs>	 (03CR) 10CWilliams: "+1" [puppet] - 10https://gerrit.wikimedia.org/r/1343590 (https://phabricator.wikimedia.org/T407329) (owner: 10Hnowlan)
[11:04:32] <wikibugs>	 (03PS2) 10Btullis: clouddumps: Allow the new IP Pool in dse-k8s-eqiad to ssh to clouddumps [puppet] - 10https://gerrit.wikimedia.org/r/1344631 (https://phabricator.wikimedia.org/T430658)
[11:04:56] <jinxer-wm>	 RESOLVED: CoreRouterInterfaceDown: Core router interface down - cr4-ulsfo:xe-0/1/0 (Transit: NTT (234631)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr4-ulsfo:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown
[11:05:05] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[11:05:07] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[11:05:13] <wikibugs>	 (03CR) 10CI reject: [V:04-1] clouddumps: Allow the new IP Pool in dse-k8s-eqiad to ssh to clouddumps [puppet] - 10https://gerrit.wikimedia.org/r/1344631 (https://phabricator.wikimedia.org/T430658) (owner: 10Btullis)
[11:06:19] <wikibugs>	 (03PS1) 10Muehlenhoff: Assign openldap::replica_mdb role to ldap-replica200[78] [puppet] - 10https://gerrit.wikimedia.org/r/1344632 (https://phabricator.wikimedia.org/T331699)
[11:07:11] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[11:08:15] <jinxer-wm>	 RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[11:08:59] <icinga-wm>	 PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 406539720 and 53 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[11:11:07] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1019.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[11:11:59] <icinga-wm>	 RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 563952 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[11:12:20] <wikibugs>	 (03CR) 10Btullis: [C:03+2] dse-k8s-eqiad: Disable the smaller, original ipv4 ippool [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344628 (https://phabricator.wikimedia.org/T430658) (owner: 10Btullis)
[11:12:40] <wikibugs>	 (03PS3) 10Btullis: clouddumps: Allow the new IP Pool in dse-k8s-eqiad to ssh to clouddumps [puppet] - 10https://gerrit.wikimedia.org/r/1344631 (https://phabricator.wikimedia.org/T430658)
[11:13:05] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[11:13:07] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[11:13:15] <jinxer-wm>	 FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[11:13:35] <hnowlan>	 !log restarted restbase on restbase2029
[11:13:36] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[11:13:39] <wikibugs>	 (03CR) 10Btullis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1344631 (https://phabricator.wikimedia.org/T430658) (owner: 10Btullis)
[11:14:56] <jinxer-wm>	 FIRING: CoreRouterInterfaceDown: Core router interface down - cr4-ulsfo:xe-0/1/0 (Transit: NTT (234631)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr4-ulsfo:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown
[11:15:40] <jinxer-wm>	 RESOLVED: [2x] ProbeDown: Service restbase2029:7231 has failed probes (http_restbase_discovery_wmnet_ip4) - https://wikitech.wikimedia.org/wiki/RESTBase - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[11:16:05] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1020.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[11:16:05] <wikibugs>	 (03PS4) 10Btullis: clouddumps: Allow the new IP Pool in dse-k8s-eqiad to ssh to clouddumps [puppet] - 10https://gerrit.wikimedia.org/r/1344631 (https://phabricator.wikimedia.org/T430658)
[11:16:07] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1020.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[11:16:41] <wikibugs>	 (03CR) 10CI reject: [V:04-1] clouddumps: Allow the new IP Pool in dse-k8s-eqiad to ssh to clouddumps [puppet] - 10https://gerrit.wikimedia.org/r/1344631 (https://phabricator.wikimedia.org/T430658) (owner: 10Btullis)
[11:18:15] <jinxer-wm>	 RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[11:19:44] <jinxer-wm>	 FIRING: KubernetesDeploymentUnavailableReplicas: ...
[11:19:44] <jinxer-wm>	 Deployment mw-web.eqiad.main in mw-web at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s&var-namespace=mw-web&var-deployment=mw-web.eqiad.main - ...
[11:19:44] <jinxer-wm>	 https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas
[11:19:56] <jinxer-wm>	 RESOLVED: CoreRouterInterfaceDown: Core router interface down - cr4-ulsfo:xe-0/1/0 (Transit: NTT (234631)) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr4-ulsfo:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown
[11:21:05] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[11:21:05] <logmsgbot>	 !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P{cp[5031,5032].*} and A:cp - 3.2.23 upgrade (T438828)
[11:21:07] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[11:21:09] <stashbot>	 T438828: Upgrade HAProxy to 3.2.23 on cp hosts - https://phabricator.wikimedia.org/T438828
[11:21:22] <wikibugs>	 10ops-eqiad, 06DC-Ops: Alert for device ps1-c6-eqiad.mgmt.eqiad.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T439120#12359478 (10Jclark-ctr) a:03Jclark-ctr
[11:22:15] <jinxer-wm>	 FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[11:22:29] <wikibugs>	 (03Merged) 10jenkins-bot: dse-k8s-eqiad: Disable the smaller, original ipv4 ippool [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344628 (https://phabricator.wikimedia.org/T430658) (owner: 10Btullis)
[11:22:46] <wikibugs>	 10ops-eqiad, 06DC-Ops: Alert for device ps1-a4-eqiad.mgmt.eqiad.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T439122#12359482 (10Jclark-ctr) a:03Jclark-ctr
[11:23:39] <jinxer-wm>	 RESOLVED: [2x] TransitBGPDown: Transit BGP session down between cr4-ulsfo and NTT (129.250.204.5) - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status  - https://alerts.wikimedia.org/?q=alertname%3DTransitBGPDown
[11:24:05] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[11:25:07] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[11:27:05] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[11:27:07] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[11:27:15] <jinxer-wm>	 RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[11:30:07] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1020.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[11:31:05] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1021.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1016.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[11:32:05] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[11:32:07] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[11:32:13] <icinga-wm>	 RECOVERY - Check unit status of httpbb_kubernetes_mw-api-int_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-api-int_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
[11:34:02] <wikibugs>	 (03PS1) 10Elukey: docker_registry: rework read-only [puppet] - 10https://gerrit.wikimedia.org/r/1344634 (https://phabricator.wikimedia.org/T437297)
[11:34:43] <wikibugs>	 (03PS2) 10Elukey: docker_registry: rework read-only [puppet] - 10https://gerrit.wikimedia.org/r/1344634 (https://phabricator.wikimedia.org/T437297)
[11:35:53] <wikibugs>	 10ops-eqiad, 06SRE, 06DC-Ops: Alert for device ps1-a4-eqiad.mgmt.eqiad.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T439122#12359526 (10phaultfinder)
[11:36:05] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1021.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1020.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[11:36:07] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1016.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[11:37:12] <wikibugs>	 (03CR) 10Elukey: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1344634 (https://phabricator.wikimedia.org/T437297) (owner: 10Elukey)
[11:37:15] <jinxer-wm>	 FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook  - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[11:37:50] <wikibugs>	 (03CR) 10CI reject: [V:04-1] docker_registry: rework read-only [puppet] - 10https://gerrit.wikimedia.org/r/1344634 (https://phabricator.wikimedia.org/T437297) (owner: 10Elukey)
[11:38:05] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[11:39:06] <wikibugs>	 (03PS3) 10Elukey: docker_registry: rework read-only [puppet] - 10https://gerrit.wikimedia.org/r/1344634 (https://phabricator.wikimedia.org/T437297)
[11:41:05] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1014.eqiad.wmnet, wdqs1012.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[11:43:05] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[11:43:07] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[11:43:17] <jinxer-wm>	 FIRING: ProbeDown: Service wdqs1016:443 has failed probes (http_wdqs_main_external_search_sparql_endpoint_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#wdqs1016:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[11:47:15] <jinxer-wm>	 RESOLVED: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook  - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[11:50:05] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[11:50:07] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[11:51:05] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[11:51:07] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[11:51:53] <wikibugs>	 (03PS5) 10Btullis: clouddumps: Allow the new IP Pool in dse-k8s-eqiad to ssh to clouddumps [puppet] - 10https://gerrit.wikimedia.org/r/1344631 (https://phabricator.wikimedia.org/T430658)
[11:53:25] <wikibugs>	 (03CR) 10Muehlenhoff: [C:03+2] Assign openldap::replica_mdb role to ldap-replica200[78] [puppet] - 10https://gerrit.wikimedia.org/r/1344632 (https://phabricator.wikimedia.org/T331699) (owner: 10Muehlenhoff)
[11:54:05] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1012.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[11:54:07] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[11:55:59] <icinga-wm>	 PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 148925576 and 19 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[11:57:05] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[11:57:07] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[11:57:15] <jinxer-wm>	 FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[11:57:59] <icinga-wm>	 PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 183403664 and 23 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[11:58:59] <icinga-wm>	 RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 3482096 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[12:00:05] <jouncebot>	 Deploy window Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260924T1200)
[12:00:07] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1018.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1016.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[12:01:07] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[12:02:15] <jinxer-wm>	 RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[12:03:56] <logmsgbot>	 !log btullis@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
[12:04:07] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[12:04:32] <logmsgbot>	 !log btullis@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
[12:06:05] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1012.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[12:08:05] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[12:08:07] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[12:08:17] <jinxer-wm>	 FIRING: [2x] ProbeDown: Service wdqs1011:443 has failed probes (http_wdqs_main_external_search_sparql_endpoint_ip6)  - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[12:09:05] <wikibugs>	 (03PS4) 10Elukey: docker_registry: rework read-only [puppet] - 10https://gerrit.wikimedia.org/r/1344634 (https://phabricator.wikimedia.org/T437297)
[12:09:05] <wikibugs>	 (03PS1) 10Elukey: profile::docker_registry: remove debug logging for restricted [puppet] - 10https://gerrit.wikimedia.org/r/1344652
[12:10:35] <wikibugs>	 (03PS2) 10Elukey: profile::pki::multirootca: disable mTLS requirements [puppet] - 10https://gerrit.wikimedia.org/r/1344004 (https://phabricator.wikimedia.org/T436809)
[12:11:05] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1016.eqiad.wmnet, wdqs1020.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[12:11:07] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1020.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[12:11:16] <wikibugs>	 (03CR) 10Elukey: profile::pki::multirootca: disable mTLS requirements (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1344004 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey)
[12:12:03] <wikibugs>	 (03PS3) 10Elukey: profile::pki::client: disable client auth via mTLS [puppet] - 10https://gerrit.wikimedia.org/r/1344005 (https://phabricator.wikimedia.org/T436809)
[12:16:32] <jinxer-wm>	 FIRING: [2x] Not accepting/receiving prefixes from anycast BGP peer: Alert for device cr1-codfw.wikimedia.org - Not accepting/receiving prefixes from anycast BGP peer   - https://alerts.wikimedia.org/?q=alertname%3DNot+accepting%2Freceiving+prefixes+from+anycast+BGP+peer
[12:17:05] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[12:17:07] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[12:18:14] <wikibugs>	 (03CR) 10Elukey: [C:03+2] profile::docker_registry: remove debug logging for restricted [puppet] - 10https://gerrit.wikimedia.org/r/1344652 (owner: 10Elukey)
[12:21:07] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1020.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1019.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[12:22:31] <wikibugs>	 (03CR) 10Elukey: [C:03+2] profile::pki::multirootca: disable mTLS requirements [puppet] - 10https://gerrit.wikimedia.org/r/1344004 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey)
[12:23:17] <jinxer-wm>	 FIRING: [3x] ProbeDown: Service wdqs1011:443 has failed probes (http_wdqs_main_external_search_sparql_endpoint_ip4)  - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[12:25:05] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[12:28:05] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[12:28:07] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[12:28:17] <jinxer-wm>	 FIRING: [5x] ProbeDown: Service wdqs1011:443 has failed probes (http_wdqs_main_external_search_sparql_endpoint_ip4)  - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[12:30:52] <wikibugs>	 (03CR) 10CWilliams: switchdc.databases.prepare: Retry checks for replication threads (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/1344031 (https://phabricator.wikimedia.org/T438833) (owner: 10CWilliams)
[12:32:30] <wikibugs>	 (03PS4) 10Elukey: profile::pki::client: disable client auth via mTLS [puppet] - 10https://gerrit.wikimedia.org/r/1344005 (https://phabricator.wikimedia.org/T436809)
[12:32:39] <wikibugs>	 (03CR) 10Elukey: profile::pki::client: disable client auth via mTLS (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1344005 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey)
[12:33:17] <jinxer-wm>	 FIRING: [7x] ProbeDown: Service wdqs1011:443 has failed probes (http_wdqs_main_external_search_sparql_endpoint_ip4)  - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[12:34:07] <urbanecm>	 jouncebot: nowandnext
[12:34:07] <jouncebot>	 For the next 0 hour(s) and 25 minute(s): Mobileapps/RESTBase/Wikifeeds (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260924T1200)
[12:34:07] <jouncebot>	 In 0 hour(s) and 25 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260924T1300)
[12:34:12] <logmsgbot>	 !log urbanecm@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-experimental: apply
[12:34:46] <logmsgbot>	 !log urbanecm@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-experimental: apply
[12:35:48] <jinxer-wm>	 FIRING: PuppetFailure: Puppet has failed on ml-serve1015:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[12:36:07] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[12:36:17] <wikibugs>	 (03CR) 10Huji: [C:03+1] Enable $wgCiteDefaultRefDirAuto on fawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1325997 (https://phabricator.wikimedia.org/T432544) (owner: 10Mamouri)
[12:37:05] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1021.eqiad.wmnet, wdqs1011.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[12:37:35] <wikibugs>	 (03PS4) 10Blake: mw-web: set an anti-affinity preference for thumbor nodes [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344653
[12:38:05] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[12:38:07] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[12:39:39] <wikibugs>	 (03CR) 10VolkerE: [C:03+1] EarlyOnboarding: record account setup motivation in `action_context` [extensions/WikimediaEvents] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344594 (https://phabricator.wikimedia.org/T437339) (owner: 10Michael Große)
[12:40:27] <wikibugs>	 (03CR) 10Marostegui: switchdc.databases.prepare: Retry checks for replication threads (031 comment) [cookbooks] - 10https://gerrit.wikimedia.org/r/1344031 (https://phabricator.wikimedia.org/T438833) (owner: 10CWilliams)
[12:41:07] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1013.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[12:41:11] <wikibugs>	 (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, September 24 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#dep" [extensions/Wikibase] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344246 (https://phabricator.wikimedia.org/T437687) (owner: 10Awight)
[12:42:07] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[12:43:17] <jinxer-wm>	 FIRING: [3x] ProbeDown: Service wdqs1013:443 has failed probes (http_wdqs_main_external_search_sparql_endpoint_ip4)  - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[12:43:27] <logmsgbot>	 !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply
[12:43:30] <logmsgbot>	 !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply
[12:45:07] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[12:46:07] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[12:46:30] <wikibugs>	 10ops-eqiad, 06SRE, 10Continuous-Integration-Infrastructure, 06DC-Ops, 06Release-Engineering-Team: Check and fix BIOS on new Supermicro cloudvirt* systems - https://phabricator.wikimedia.org/T435537#12359827 (10fgiunchedi) 05Open→03Resolved Fleet is now all at the same settings  ` # cumin 'cloudv...
[12:48:18] <wikibugs>	 (03PS1) 10Muehlenhoff: Use the new MDB/Trixie servers for tests [puppet] - 10https://gerrit.wikimedia.org/r/1344656 (https://phabricator.wikimedia.org/T331699)
[12:48:44] <wikibugs>	 (03PS2) 10Muehlenhoff: Use the new MDB/Trixie servers for tests [puppet] - 10https://gerrit.wikimedia.org/r/1344656 (https://phabricator.wikimedia.org/T331699)
[12:49:56] <wikibugs>	 10ops-eqiad, 06SRE, 06DC-Ops: Alert for device ps1-c6-eqiad.mgmt.eqiad.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T439120#12359871 (10Jclark-ctr)   ps1-c6-eqiad.mgmt.eqiad.wmnet #1: Sensor: Phase, AA:L1-L2, Active Power Value: 1.671 kW (power) Thresholds: High: 1650
[12:49:59] <icinga-wm>	 PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 562442648 and 38 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[12:51:07] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1021.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[12:51:59] <icinga-wm>	 RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 76312 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[12:52:15] <wikibugs>	 (03PS1) 10Elukey: admin_ng: move staging clusters to cfssl/pki port 443 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344657 (https://phabricator.wikimedia.org/T436809)
[12:52:32] <wikibugs>	 10ops-eqiad, 06SRE, 06DC-Ops: Alert for device ps1-c6-eqiad.mgmt.eqiad.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T439120#12359878 (10Jclark-ctr) 05Open→03Resolved
[12:52:41] <logmsgbot>	 !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply
[12:52:43] <logmsgbot>	 !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply
[12:53:07] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[12:53:17] <jinxer-wm>	 FIRING: [3x] ProbeDown: Service wdqs1013:443 has failed probes (http_wdqs_main_external_search_sparql_endpoint_ip4)  - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[12:53:19] <wikibugs>	 10ops-eqiad, 06SRE, 06DC-Ops: Alert for device ps1-a4-eqiad.mgmt.eqiad.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T439122#12359880 (10Jclark-ctr) 1: Sensor: Line, BA:L2, Current Value: 12.08 A (current) Thresholds: High: 12
[12:56:06] <wikibugs>	 (03CR) 10Muehlenhoff: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1344656 (https://phabricator.wikimedia.org/T331699) (owner: 10Muehlenhoff)
[12:56:07] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1014.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[12:56:09] <wikibugs>	 10ops-eqiad, 06SRE, 06DC-Ops: Alert for device ps1-a4-eqiad.mgmt.eqiad.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T439122#12359890 (10Jclark-ctr) 05Open→03Resolved
[12:57:07] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[12:58:01] <logmsgbot>	 !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply
[12:58:05] <logmsgbot>	 !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply
[12:58:17] <jinxer-wm>	 FIRING: [3x] ProbeDown: Service wdqs1013:443 has failed probes (http_wdqs_main_external_search_sparql_endpoint_ip4)  - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[12:59:23] <logmsgbot>	 !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply
[12:59:26] <logmsgbot>	 !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply
[12:59:42] <wikibugs>	 10ops-eqiad, 06SRE, 06DC-Ops: Interface eno1 on wikikube-worker1160 has the wrong speed - https://phabricator.wikimedia.org/T439094#12359908 (10Jclark-ctr) replaced cable
[13:00:05] <jouncebot>	 Lucas_WMDE, urbanecm, and TheresNoTime: UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260924T1300). Please do the needful.
[13:00:05] <jouncebot>	 matthiasmullie and awight: A patch you scheduled for UTC afternoon backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker.
[13:00:07] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[13:00:10] <Lucas_WMDE>	 o/
[13:00:10] <matthiasmullie>	 o/
[13:00:22] <awight>	 :-D
[13:00:23] <seanleong-wmde>	 o/
[13:00:33] <seanleong-wmde>	 \o7
[13:00:57] <Lucas_WMDE>	 o_O the MultimediaViewer/ReaderExperiments backports changed ownership
[13:01:07] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[13:01:34] <Lucas_WMDE>	 matthiasmullie: go ahead, I think ^^
[13:01:48] <matthiasmullie>	 AIght!
[13:01:59] <logmsgbot>	 !log cdobbins@cumin1004 conftool action : set/pooled=yes; selector: name=cp2059.*
[13:02:13] <logmsgbot>	 !log vgutierrez@cumin1004 START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-text_eqsin and A:cp - 3.2.23 upgrade (T438828)
[13:02:16] <stashbot>	 T438828: Upgrade HAProxy to 3.2.23 on cp hosts - https://phabricator.wikimedia.org/T438828
[13:02:17] * Lucas_WMDE frowns at the many ConvertibleTimestamp errors in logspam-^atch
[13:02:19] <Lucas_WMDE>	 *-watch
[13:02:50] <logmsgbot>	 !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply
[13:02:52] <logmsgbot>	 !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply
[13:02:53] <wikibugs>	 (03CR) 10TrainBranchBot: [C:03+2] "Approved by mlitn@deploy1003 using scap backport" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344617 (https://phabricator.wikimedia.org/T431362) (owner: 10Marco Fossati)
[13:02:53] <wikibugs>	 (03CR) 10TrainBranchBot: [C:03+2] "Approved by mlitn@deploy1003 using scap backport" [extensions/ReaderExperiments] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344619 (https://phabricator.wikimedia.org/T431362) (owner: 10Marco Fossati)
[13:02:53] <wikibugs>	 (03CR) 10TrainBranchBot: [C:03+2] "Approved by mlitn@deploy1003 using scap backport" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344627 (https://phabricator.wikimedia.org/T435672) (owner: 10Marco Fossati)
[13:02:54] <wikibugs>	 (03CR) 10TrainBranchBot: [C:03+2] "Approved by mlitn@deploy1003 using scap backport" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344630 (https://phabricator.wikimedia.org/T438907) (owner: 10Marco Fossati)
[13:03:17] <jinxer-wm>	 FIRING: [4x] ProbeDown: Service wdqs1013:443 has failed probes (http_wdqs_main_external_search_sparql_endpoint_ip4)  - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[13:03:31] <Lucas_WMDE>	 (looks like the fix from https://phabricator.wikimedia.org/T436409 isn’t rolled out yet, if ReleaseTaggerBot is right about the version)
[13:03:59] <awight>	 Lucas_WMDE: wmf.21 did not go out to group1 yet
[13:04:37] * Lucas_WMDE nods
[13:04:44] <jinxer-wm>	 RESOLVED: KubernetesDeploymentUnavailableReplicas: ...
[13:04:44] <jinxer-wm>	 Deployment mw-web.eqiad.main in mw-web at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s&var-namespace=mw-web&var-deployment=mw-web.eqiad.main - ...
[13:04:44] <jinxer-wm>	 https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas
[13:05:01] <Lucas_WMDE>	 (for an extra bonus, I had misparsed the 09-22 date as wmf.22 and was wondering why the week number was off by one more than expected 🤦)
[13:05:05] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1021.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1018.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[13:05:07] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[13:05:13] <Lucas_WMDE>	 but yeah that’ll do it
[13:05:26] <wikibugs>	 (03Merged) 10jenkins-bot: Instrument five-arm image carousel retest [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344617 (https://phabricator.wikimedia.org/T431362) (owner: 10Marco Fossati)
[13:05:50] <wikibugs>	 (03Merged) 10jenkins-bot: Wire image carousel retest instrumentation [extensions/ReaderExperiments] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344619 (https://phabricator.wikimedia.org/T431362) (owner: 10Marco Fossati)
[13:05:52] <wikibugs>	 (03Merged) 10jenkins-bot: ThumbExtractor: trim nbsp and dangling colons from caption text [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344627 (https://phabricator.wikimedia.org/T435672) (owner: 10Marco Fossati)
[13:05:54] <wikibugs>	 (03Merged) 10jenkins-bot: ThumbExtractor: exclude lead infobox images from the carousel [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344630 (https://phabricator.wikimedia.org/T438907) (owner: 10Marco Fossati)
[13:05:55] <wikibugs>	 (03CR) 10Slyngshede: [C:03+1] "Looks correct." [puppet] - 10https://gerrit.wikimedia.org/r/1344656 (https://phabricator.wikimedia.org/T331699) (owner: 10Muehlenhoff)
[13:06:03] <logmsgbot>	 !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply
[13:06:07] <logmsgbot>	 !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply
[13:06:10] <awight>	 Hopefully tonight's train goes to both group1 and group2... :sweat: glad I am taking the bus instead
[13:06:21] <logmsgbot>	 !log mlitn@deploy1003 Started scap sync-world: Backport for [[gerrit:1344617|Instrument five-arm image carousel retest (T431362)]], [[gerrit:1344619|Wire image carousel retest instrumentation (T431362)]], [[gerrit:1344627|ThumbExtractor: trim nbsp and dangling colons from caption text (T435672)]], [[gerrit:1344630|ThumbExtractor: exclude lead infobox images from the carousel (T438907)]]
[13:06:25] <jinxer-wm>	 RESOLVED: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[13:06:28] <stashbot>	 T431362: Image Carousel A/B/C/D/E retest - https://phabricator.wikimedia.org/T431362
[13:06:29] <stashbot>	 T435672: [Image Browsing] Code output in carousel captions - https://phabricator.wikimedia.org/T435672
[13:06:29] <stashbot>	 T438907: [Image Browsing] Exclude infobox images from carousel - https://phabricator.wikimedia.org/T438907
[13:07:07] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[13:07:22] <wikibugs>	 10ops-eqiad, 06SRE, 06DC-Ops: krb1002 booting up extremely slow - https://phabricator.wikimedia.org/T435354#12359947 (10VRiley-WMF) 05Open→03Resolved Closing this for now, will put the system in an inventory status
[13:07:46] <logmsgbot>	 !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply
[13:07:49] <logmsgbot>	 !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/postgresql-growthbook-next: apply
[13:08:23] <logmsgbot>	 !log cdobbins@cumin1004 START - Cookbook sre.hosts.reimage for host ncredir4004.ulsfo.wmnet with OS trixie
[13:09:23] <wikibugs>	 (03PS1) 10Michael Große: fix(AccountSetup): ensure TestKitchen knows about new user in CentralAuth redirect [extensions/GrowthExperiments] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344662 (https://phabricator.wikimedia.org/T436872)
[13:10:07] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[13:10:32] <logmsgbot>	 !log mlitn@deploy1003 mfossati, mlitn: Backport for [[gerrit:1344617|Instrument five-arm image carousel retest (T431362)]], [[gerrit:1344619|Wire image carousel retest instrumentation (T431362)]], [[gerrit:1344627|ThumbExtractor: trim nbsp and dangling colons from caption text (T435672)]], [[gerrit:1344630|ThumbExtractor: exclude lead infobox images from the carousel (T438907)]] synced to the testservers (see https://wiki
[13:10:32] <logmsgbot>	 tech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
[13:11:05] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[13:11:32] <MichaelG_WMF>	 would there be capacity to backport another small change? If not, then it is also fine if it happens later today
[13:12:07] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[13:12:43] <Lucas_WMDE>	 MichaelG_WMF: probably, the first four backports are currently being deployed simultaneously
[13:12:55] <wikibugs>	 10ops-eqiad, 06SRE, 06DC-Ops: krb1002 booting up extremely slow - https://phabricator.wikimedia.org/T435354#12359970 (10MoritzMuehlenhoff) >>! In T435354#12359947, @VRiley-WMF wrote: > Closing this for now, will put the system in an inventory status  See https://phabricator.wikimedia.org/T438229#12350803...
[13:13:01] <wikibugs>	 (03PS1) 10Muehlenhoff: Apply hiera config to correct idp-test host [puppet] - 10https://gerrit.wikimedia.org/r/1344664
[13:13:01] <MichaelG_WMF>	 great, I'll add mine to calendar!
[13:13:01] <wikibugs>	 (03PS1) 10Muehlenhoff: idp-test: Configure to use the new Trixie/MDB LDAP servers for tests [puppet] - 10https://gerrit.wikimedia.org/r/1344665 (https://phabricator.wikimedia.org/T331699)
[13:13:17] <jinxer-wm>	 FIRING: [4x] ProbeDown: Service wdqs1013:443 has failed probes (http_wdqs_main_external_search_sparql_endpoint_ip4)  - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[13:13:24] <wikibugs>	 (03CR) 10Muehlenhoff: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1344665 (https://phabricator.wikimedia.org/T331699) (owner: 10Muehlenhoff)
[13:13:26] <logmsgbot>	 !log mlitn@deploy1003 mfossati, mlitn: Continuing with deployment
[13:15:05] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[13:15:07] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[13:17:05] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[13:17:07] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[13:17:23] <jinxer-wm>	 FIRING: [3x] CertAlmostExpired: gNMI TLS certificate for lsw1-e2-codfw.mgmt.codfw.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired
[13:17:53] <MichaelG_WMF>	 Lucas_WMDE: I added my change. Testing it might take a moment because I may have to create multiple new accounts (50/50 chance to be sorted into treatment)
[13:18:08] <Lucas_WMDE>	 ack
[13:18:47] <logmsgbot>	 !log mlitn@deploy1003 Finished scap sync-world: Backport for [[gerrit:1344617|Instrument five-arm image carousel retest (T431362)]], [[gerrit:1344619|Wire image carousel retest instrumentation (T431362)]], [[gerrit:1344627|ThumbExtractor: trim nbsp and dangling colons from caption text (T435672)]], [[gerrit:1344630|ThumbExtractor: exclude lead infobox images from the carousel (T438907)]] (duration: 12m 25s)
[13:18:53] <stashbot>	 T431362: Image Carousel A/B/C/D/E retest - https://phabricator.wikimedia.org/T431362
[13:18:53] <stashbot>	 T435672: [Image Browsing] Code output in carousel captions - https://phabricator.wikimedia.org/T435672
[13:18:54] <stashbot>	 T438907: [Image Browsing] Exclude infobox images from carousel - https://phabricator.wikimedia.org/T438907
[13:19:14] <wikibugs>	 10ops-eqiad, 06SRE, 06DC-Ops: krb1002 booting up extremely slow - https://phabricator.wikimedia.org/T435354#12359990 (10VRiley-WMF) 05Resolved→03Open
[13:19:51] <matthiasmullie>	 Done; awight you're up!
[13:20:07] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1018.eqiad.wmnet, wdqs1014.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[13:20:08] <awight>	 matthiasmullie: ack, ty!
[13:20:35] <wikibugs>	 (03CR) 10TrainBranchBot: [C:03+2] "Approved by awight@deploy1003 using scap backport" [extensions/Wikibase] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344246 (https://phabricator.wikimedia.org/T437687) (owner: 10Awight)
[13:21:07] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[13:21:31] <jinxer-wm>	 FIRING: RedisReplicaDown: Redis replica down rdb2014:16380 redis_misc - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_misc - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_misc&var-instance=rdb2014:16380 - https://alerts.wikimedia.org/?q=alertname%3DRedisReplicaDown
[13:23:05] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1018.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1019.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[13:23:10] <jinxer-wm>	 FIRING: BFDdown: BFD session down between cr2-eqsin and fe80::6687:8807:ef2:7018 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqsin:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown
[13:23:17] <jinxer-wm>	 FIRING: [4x] ProbeDown: Service wdqs1013:443 has failed probes (http_wdqs_main_external_search_sparql_endpoint_ip4)  - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[13:23:51] <wikibugs>	 10SRE-SLO, 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): page_change SLO windows - https://phabricator.wikimedia.org/T438054#12360023 (10elukey) @APizzata-WMF got it thanks! @RLazarus will review this use case and propose a way forward soon!
[13:24:07] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1021.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[13:24:15] <logmsgbot>	 !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply
[13:24:28] <wikibugs>	 (03CR) 10Elukey: [C:03+2] admin_ng: move staging clusters to cfssl/pki port 443 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344657 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey)
[13:25:02] <wikibugs>	 (03CR) 10Ottomata: [C:03+1] "LGTM. Could you add a comment about the naming of the class?  We intend to rename this class since it no longer is about only event schema" [puppet] - 10https://gerrit.wikimedia.org/r/1343092 (https://phabricator.wikimedia.org/T438208) (owner: 10Aghirelli)
[13:25:38] <logmsgbot>	 !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply
[13:25:44] <logmsgbot>	 !log elukey@deploy1003 helmfile [staging-codfw] START helmfile.d/admin 'sync'.
[13:25:49] <logmsgbot>	 !log elukey@deploy1003 helmfile [staging-codfw] DONE helmfile.d/admin 'sync'.
[13:26:17] <logmsgbot>	 !log elukey@deploy1003 helmfile [staging-eqiad] START helmfile.d/admin 'sync'.
[13:26:20] <logmsgbot>	 !log elukey@deploy1003 helmfile [staging-eqiad] DONE helmfile.d/admin 'sync'.
[13:26:29] <logmsgbot>	 !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/growthbook-next: apply
[13:26:54] <logmsgbot>	 !log brouberol@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/growthboo-next: apply
[13:27:05] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[13:27:07] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[13:28:10] <jinxer-wm>	 RESOLVED: BFDdown: BFD session down between cr2-eqsin and fe80::6687:8807:ef2:7018 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqsin:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown
[13:28:17] <jinxer-wm>	 FIRING: [4x] ProbeDown: Service wdqs1013:443 has failed probes (http_wdqs_main_external_search_sparql_endpoint_ip4)  - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[13:29:25] <wikibugs>	 (03CR) 10Slyngshede: [C:03+1] "Once done we can remove the hosts files." [puppet] - 10https://gerrit.wikimedia.org/r/1344665 (https://phabricator.wikimedia.org/T331699) (owner: 10Muehlenhoff)
[13:29:33] <wikibugs>	 (03Merged) 10jenkins-bot: Fixes failing edge when page is missing and entity usage remain. Updating ReallyDoQuery to function like an inner join. [extensions/Wikibase] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344246 (https://phabricator.wikimedia.org/T437687) (owner: 10Awight)
[13:29:52] <logmsgbot>	 !log awight@deploy1003 Started scap sync-world: Backport for [[gerrit:1344246|Fixes failing edge when page is missing and entity usage remain. Updating ReallyDoQuery to function like an inner join. (T437687)]]
[13:29:56] <stashbot>	 T437687: TypeError: MediaWiki\Title\NamespaceInfo::exists(): Argument #1 ($index) must be of type int, null given, called in /srv/mediawiki/php-1.47.0-wmf.19/includes/Title/Title.php on line 644 - https://phabricator.wikimedia.org/T437687
[13:30:07] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[13:30:39] <jinxer-wm>	 FIRING: CoreBGPDown: Core BGP session down between cr1-eqiad and cr2-eqsin (103.102.166.147) - group Confed_eqsin - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=eqiad&var-device=cr1-eqiad:9804&var-bgp_group=Confed_eqsin&var-bgp_neighbor=cr2-eqsin - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown
[13:31:05] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1019.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[13:31:26] <jinxer-wm>	 FIRING: [3x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[13:33:05] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[13:33:51] <logmsgbot>	 !log awight@deploy1003 awight: Backport for [[gerrit:1344246|Fixes failing edge when page is missing and entity usage remain. Updating ReallyDoQuery to function like an inner join. (T437687)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
[13:34:23] <logmsgbot>	 !log cdobbins@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir4004.ulsfo.wmnet with reason: host reimage
[13:35:08] <wikibugs>	 06SRE, 06Commons, 10MediaWiki-File-management, 06Traffic, and 2 others: Varnish serving outdated version at original/non-thumb URL of overwritten file upload - https://phabricator.wikimedia.org/T425216#12360080 (10Krinkle) >>! In T425216#12351266, @BBlack wrote: > I was under the impression (implicitly, du...
[13:35:11] <logmsgbot>	 !log awight@deploy1003 awight: Continuing with deployment
[13:35:27] <moritzm>	 !log installing nghttp2 security updates
[13:35:29] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:35:39] <jinxer-wm>	 RESOLVED: CoreBGPDown: Core BGP session down between cr1-eqiad and cr2-eqsin (103.102.166.147) - group Confed_eqsin - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=eqiad&var-device=cr1-eqiad:9804&var-bgp_group=Confed_eqsin&var-bgp_neighbor=cr2-eqsin - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown
[13:36:05] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1013.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[13:36:19] <wikibugs>	 (03PS1) 10Ayounsi: Add flag to deprecate Juniper's hostkey-algorithm [homer/public] - 10https://gerrit.wikimedia.org/r/1344672 (https://phabricator.wikimedia.org/T439136)
[13:37:05] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[13:37:07] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[13:37:33] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Add flag to deprecate Juniper's hostkey-algorithm [homer/public] - 10https://gerrit.wikimedia.org/r/1344672 (https://phabricator.wikimedia.org/T439136) (owner: 10Ayounsi)
[13:37:59] <wikibugs>	 07sre-alert-triage, 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: Alert in need of triage: PKICertificateExpiry (instance pki2002:9100) - https://phabricator.wikimedia.org/T438703#12360095 (10Gehel)
[13:38:00] <wikibugs>	 07sre-alert-triage, 06Data-Platform-SRE (2026-09-18 - 2026-10-09), 07Essential-Work: Alert in need of triage: PKICertificateExpiry (instance pki2002:9100) - https://phabricator.wikimedia.org/T438703#12360096 (10Gehel) p:05Triage→03High
[13:39:01] <icinga-wm>	 PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 62174952 and 8 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[13:39:25] <logmsgbot>	 !log cdobbins@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir4004.ulsfo.wmnet with reason: host reimage
[13:39:29] <wikibugs>	 10ops-eqiad, 06SRE, 06DC-Ops, 06Machine-Learning-Team, and 2 others: Q2:rack/setup/install ml-serve1017 - https://phabricator.wikimedia.org/T438253#12360126 (10Gehel)
[13:39:37] <wikibugs>	 10ops-eqiad, 06SRE, 06DC-Ops, 06Machine-Learning-Team, and 2 others: Q2:rack/setup/install ml-serve1017 - https://phabricator.wikimedia.org/T438253#12360125 (10Gehel)
[13:39:45] <wikibugs>	 (03PS2) 10Ayounsi: Add flag to deprecate Juniper's hostkey-algorithm [homer/public] - 10https://gerrit.wikimedia.org/r/1344672 (https://phabricator.wikimedia.org/T439136)
[13:39:45] <logmsgbot>	 !log lerickson@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
[13:40:05] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[13:40:07] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[13:40:30] <logmsgbot>	 !log awight@deploy1003 Finished scap sync-world: Backport for [[gerrit:1344246|Fixes failing edge when page is missing and entity usage remain. Updating ReallyDoQuery to function like an inner join. (T437687)]] (duration: 10m 38s)
[13:40:33] <stashbot>	 T437687: TypeError: MediaWiki\Title\NamespaceInfo::exists(): Argument #1 ($index) must be of type int, null given, called in /srv/mediawiki/php-1.47.0-wmf.19/includes/Title/Title.php on line 644 - https://phabricator.wikimedia.org/T437687
[13:41:05] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[13:41:15] * MichaelG_WMF is still around
[13:41:29] <seanleong-wmde>	 Hi MichaelG_WMF, we're done. Feel free to go ahead!
[13:42:01] <icinga-wm>	 RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 57752 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[13:42:35] <MichaelG_WMF>	 seanleong-wmde: Cool, thanks! Lucas_WMDE: do you have capacity to do the deployment? It's fine if not, then I can look for someone internally
[13:44:21] <wikibugs>	 (03CR) 10Cathal Mooney: [C:03+1] Add flag to deprecate Juniper's hostkey-algorithm [homer/public] - 10https://gerrit.wikimedia.org/r/1344672 (https://phabricator.wikimedia.org/T439136) (owner: 10Ayounsi)
[13:46:01] <wikibugs>	 (03CR) 10Ayounsi: [C:03+2] Add flag to deprecate Juniper's hostkey-algorithm [homer/public] - 10https://gerrit.wikimedia.org/r/1344672 (https://phabricator.wikimedia.org/T439136) (owner: 10Ayounsi)
[13:46:07] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[13:46:08] <urbanecm>	 MichaelG_WMF: i can deploy for you
[13:46:24] <MichaelG_WMF>	 urbanecm: Thanks!
[13:46:28] <wikibugs>	 (03CR) 10Urbanecm: [C:03+2] fix(AccountSetup): ensure TestKitchen knows about new user in CentralAuth redirect [extensions/GrowthExperiments] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344662 (https://phabricator.wikimedia.org/T436872) (owner: 10Michael Große)
[13:46:37] <wikibugs>	 (03PS1) 10Kosta Harlan: AbuseReview: Inidicate if the queue hides recent edits [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344677 (https://phabricator.wikimedia.org/T438235)
[13:46:51] <wikibugs>	 (03PS1) 10Elukey: Remove exclude_from_switchover flag for the Docker Registry [puppet] - 10https://gerrit.wikimedia.org/r/1344678
[13:47:05] <Lucas_WMDE>	 sorry, I was afk for a bit
[13:47:07] <Lucas_WMDE>	 thanks urbanecm!
[13:47:09] <urbanecm>	 no worries Lucas_WMDE 
[13:47:33] <wikibugs>	 (03CR) 10TrainBranchBot: [C:03+2] "Approved by urbanecm@deploy1003 using scap backport" [extensions/GrowthExperiments] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344662 (https://phabricator.wikimedia.org/T436872) (owner: 10Michael Große)
[13:47:36] <wikibugs>	 (03Merged) 10jenkins-bot: Add flag to deprecate Juniper's hostkey-algorithm [homer/public] - 10https://gerrit.wikimedia.org/r/1344672 (https://phabricator.wikimedia.org/T439136) (owner: 10Ayounsi)
[13:50:21] <wikibugs>	 06SRE, 10SRE-Access-Requests: Requesting access to deployment for Tsevener - https://phabricator.wikimedia.org/T439138 (10Tsevener) 03NEW
[13:50:52] <wikibugs>	 (03CR) 10Elukey: Move the pki service entry to service_setup (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1338937 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey)
[13:51:49] <jinxer-wm>	 FIRING: HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@codfw in state pending-upgrade - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus
[13:52:27] <wikibugs>	 (03PS3) 10Elukey: Move the pki service entry to service_setup [puppet] - 10https://gerrit.wikimedia.org/r/1338937 (https://phabricator.wikimedia.org/T436809)
[13:52:44] <wikibugs>	 (03PS4) 10Elukey: role::pki: add lvs configurations [puppet] - 10https://gerrit.wikimedia.org/r/1338936 (https://phabricator.wikimedia.org/T436809)
[13:54:08] <kostajh>	 urbanecm: when you're done, I have another patch (or maybe 2) 
[13:54:14] <urbanecm>	 i'll ping you when done?
[13:54:19] <wikibugs>	 (03Merged) 10jenkins-bot: fix(AccountSetup): ensure TestKitchen knows about new user in CentralAuth redirect [extensions/GrowthExperiments] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344662 (https://phabricator.wikimedia.org/T436872) (owner: 10Michael Große)
[13:54:41] <logmsgbot>	 !log urbanecm@deploy1003 Started scap sync-world: Backport for [[gerrit:1344662|fix(AccountSetup): ensure TestKitchen knows about new user in CentralAuth redirect (T436872)]]
[13:54:45] <stashbot>	 T436872: Account Setup experiment: test end-to-end flow and verify translations on pilot wikis (DE1.3.1) - https://phabricator.wikimedia.org/T436872
[13:55:08] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{dse-k8s-wdqs100[1-3].eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
[13:55:08] <logmsgbot>	 !log lerickson@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
[13:55:12] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1001.eqiad.wmnet
[13:55:19] <wikibugs>	 (03PS4) 10Elukey: Move the pki service entry to service_setup [puppet] - 10https://gerrit.wikimedia.org/r/1338937 (https://phabricator.wikimedia.org/T436809)
[13:55:19] <wikibugs>	 (03PS5) 10Elukey: role::pki: add lvs configurations [puppet] - 10https://gerrit.wikimedia.org/r/1338936 (https://phabricator.wikimedia.org/T436809)
[13:55:19] <wikibugs>	 (03PS2) 10Elukey: Move pki.discovery.wmnet to lvs_setup state [puppet] - 10https://gerrit.wikimedia.org/r/1342214 (https://phabricator.wikimedia.org/T436809)
[13:55:20] <logmsgbot>	 !log lerickson@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
[13:55:47] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1001.eqiad.wmnet
[13:56:46] * MichaelG_WMF is standing by for testing. But it might take a moment to test this because I may have to create multiple accounts until I have one that is enrolled into treatment.
[13:57:48] <wikibugs>	 (03PS2) 10Clare Ming: ext.wikimediaEvents.testKitchen: Add withContext helper [extensions/WikimediaEvents] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344406 (https://phabricator.wikimedia.org/T438898)
[13:58:02] <wikibugs>	 (03CR) 10JHathaway: [C:03+1] Move the pki service entry to service_setup [puppet] - 10https://gerrit.wikimedia.org/r/1338937 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey)
[13:58:17] <jinxer-wm>	 FIRING: [2x] ProbeDown: Service wdqs1013:443 has failed probes (http_wdqs_main_external_search_sparql_endpoint_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#wdqs1013:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[13:58:49] <logmsgbot>	 !log urbanecm@deploy1003 migr, urbanecm: Backport for [[gerrit:1344662|fix(AccountSetup): ensure TestKitchen knows about new user in CentralAuth redirect (T436872)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
[13:58:57] <urbanecm>	 MichaelG_WMF: please test
[13:59:17] <wikibugs>	 (03PS6) 10Elukey: role::pki: add lvs configurations [puppet] - 10https://gerrit.wikimedia.org/r/1338936 (https://phabricator.wikimedia.org/T436809)
[13:59:17] <wikibugs>	 (03PS3) 10Elukey: Move pki.discovery.wmnet to lvs_setup state [puppet] - 10https://gerrit.wikimedia.org/r/1342214 (https://phabricator.wikimedia.org/T436809)
[13:59:35] <MichaelG_WMF>	 urbanecm: the k8s-mwdebug is picking the right one automatically, right?
[13:59:39] <urbanecm>	 correct
[13:59:45] <MichaelG_WMF>	 great. testing...
[13:59:51] <wikibugs>	 (03CR) 10Elukey: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1338936 (https://phabricator.wikimedia.org/T436809) (owner: 10Elukey)
[14:00:01] <icinga-wm>	 PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 187240336 and 25 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[14:00:07] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1021.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[14:00:20] <logmsgbot>	 !log cdobbins@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir4004.ulsfo.wmnet with OS trixie
[14:01:04] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1001.eqiad.wmnet
[14:01:05] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1001.eqiad.wmnet
[14:01:11] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1002.eqiad.wmnet
[14:01:25] <MichaelG_WMF>	 urbanecm: I got lucky and the first account was treatment right away. It works as expected 👍. I think we can move forward.
[14:01:38] <urbanecm>	 perfect
[14:01:39] <logmsgbot>	 !log urbanecm@deploy1003 migr, urbanecm: Continuing with deployment
[14:01:43] <urbanecm>	 proceeding
[14:01:45] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1002.eqiad.wmnet
[14:02:07] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[14:03:30] <logmsgbot>	 !log lerickson@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
[14:05:05] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[14:05:07] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[14:05:11] <logmsgbot>	 !log lerickson@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
[14:05:43] <wikibugs>	 (03PS1) 10Ayounsi: Remove "hostkey_algorithm" in all sites except eqiad/codfw [homer/public] - 10https://gerrit.wikimedia.org/r/1344682 (https://phabricator.wikimedia.org/T439136)
[14:05:58] <logmsgbot>	 !log lerickson@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
[14:06:49] <jinxer-wm>	 RESOLVED: HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@codfw in state pending-upgrade - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=codfw&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus
[14:07:07] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[14:07:07] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[14:07:09] <logmsgbot>	 !log urbanecm@deploy1003 Finished scap sync-world: Backport for [[gerrit:1344662|fix(AccountSetup): ensure TestKitchen knows about new user in CentralAuth redirect (T436872)]] (duration: 12m 27s)
[14:07:17] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1002.eqiad.wmnet
[14:07:18] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1002.eqiad.wmnet
[14:07:22] <stashbot>	 T436872: Account Setup experiment: test end-to-end flow and verify translations on pilot wikis (DE1.3.1) - https://phabricator.wikimedia.org/T436872
[14:07:23] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs1003.eqiad.wmnet
[14:07:48] <logmsgbot>	 !log cdobbins@cumin1004 conftool action : set/pooled=yes; selector: name=ncredir4004.*
[14:07:58] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs1003.eqiad.wmnet
[14:08:13] <logmsgbot>	 !log lerickson@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
[14:09:02] <logmsgbot>	 !log cdobbins@cumin1004 START - Cookbook sre.hosts.reimage for host ncredir3005.esams.wmnet with OS trixie
[14:09:04] <logmsgbot>	 !log lerickson@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
[14:10:01] <icinga-wm>	 PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 70808152 and 35 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[14:10:07] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1021.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1012.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[14:11:05] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1019.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[14:11:14] <logmsgbot>	 !log vgutierrez@cumin1004 END (FAIL) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=1) rolling upgrade of HAProxy on A:cp-text_eqsin and A:cp - 3.2.23 upgrade (T438828)
[14:11:17] <stashbot>	 T438828: Upgrade HAProxy to 3.2.23 on cp hosts - https://phabricator.wikimedia.org/T438828
[14:11:44] <wikibugs>	 (03PS1) 10Kosta Harlan: AbuseReview: Add local CheckUsers to vandalism alpha test [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344684 (https://phabricator.wikimedia.org/T438467)
[14:12:01] <icinga-wm>	 RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 106832 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[14:12:05] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[14:12:07] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[14:13:16] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs1003.eqiad.wmnet
[14:13:17] <jinxer-wm>	 RESOLVED: ProbeDown: Service wdqs1013:443 has failed probes (http_wdqs_main_external_search_sparql_endpoint_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#wdqs1013:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[14:13:17] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs1003.eqiad.wmnet
[14:13:17] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{dse-k8s-wdqs100[1-3].eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
[14:14:01] <jinxer-wm>	 FIRING: RedisInstanceDown: Redis instance down rdb2013:16380 redis_misc - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_misc - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_misc&var-instance=rdb2013:16380 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown
[14:16:20] <wikibugs>	 (03PS1) 10Dpogorzelski: amd_gpu: set the compute partition mode through sysfs [puppet] - 10https://gerrit.wikimedia.org/r/1344685 (https://phabricator.wikimedia.org/T436928)
[14:16:44] <wikibugs>	 (03CR) 10MVernon: [C:03+1] "Happy with this from an apus POV." [puppet] - 10https://gerrit.wikimedia.org/r/1344678 (owner: 10Elukey)
[14:17:21] <moritzm>	 !log installing Bird security updates
[14:17:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:19:07] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1021.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1020.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[14:19:45] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{dse-k8s-wdqs-test1001.eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
[14:19:49] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-wdqs-test1001.eqiad.wmnet
[14:20:05] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1014.eqiad.wmnet, wdqs1016.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[14:20:23] <kostajh>	 urbanecm: still deploying? 
[14:20:23] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-wdqs-test1001.eqiad.wmnet
[14:21:01] <logmsgbot>	 !log vgutierrez@cumin1004 START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P{cp[5025,5026].*} and A:cp - 3.2.23 upgrade (T438828)
[14:21:04] <stashbot>	 T438828: Upgrade HAProxy to 3.2.23 on cp hosts - https://phabricator.wikimedia.org/T438828
[14:21:38] <logmsgbot>	 !log btullis@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/admin 'apply'.
[14:22:05] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[14:22:07] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[14:22:29] <elukey>	 !log elukey@rdb2013:/srv/redis/appendonlydir$ sudo -u redis redis-check-aof --fix rdb2013-6380.aof.22039.incr.aof
[14:22:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:22:34] <logmsgbot>	 !log btullis@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/admin 'apply'.
[14:22:53] <urbanecm>	 kostajh: oh, sorry, no i'm done
[14:22:54] <urbanecm>	 go ahead please
[14:22:58] <kostajh>	 thx
[14:23:03] <kostajh>	 jouncebot: nowandnext
[14:23:03] <jouncebot>	 No deployments scheduled for the next 0 hour(s) and 6 minute(s)
[14:23:03] <jouncebot>	 In 0 hour(s) and 6 minute(s): Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260924T1430)
[14:23:43] <wikibugs>	 (03CR) 10TrainBranchBot: [C:03+2] "Approved by kharlan@deploy1003 using scap backport" [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344684 (https://phabricator.wikimedia.org/T438467) (owner: 10Kosta Harlan)
[14:23:43] <wikibugs>	 (03CR) 10TrainBranchBot: [C:03+2] "Approved by kharlan@deploy1003 using scap backport" [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344677 (https://phabricator.wikimedia.org/T438235) (owner: 10Kosta Harlan)
[14:24:01] <jinxer-wm>	 RESOLVED: RedisInstanceDown: Redis instance down rdb2013:16380 redis_misc - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_misc - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_misc&var-instance=rdb2013:16380 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown
[14:24:24] <wikibugs>	 06SRE, 10SRE-Access-Requests: Requesting access to Deployment for GreyOlson/GOlson-WMF - https://phabricator.wikimedia.org/T439143 (10GOlson-WMF) 03NEW
[14:25:05] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[14:25:07] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[14:25:27] <wikibugs>	 (03Merged) 10jenkins-bot: AbuseReview: Add local CheckUsers to vandalism alpha test [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344684 (https://phabricator.wikimedia.org/T438467) (owner: 10Kosta Harlan)
[14:26:25] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-wdqs-test1001.eqiad.wmnet
[14:26:26] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-wdqs-test1001.eqiad.wmnet
[14:26:26] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.reboot-nodes (exit_code=0) rolling reboot on P{dse-k8s-wdqs-test1001.eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
[14:26:31] <jinxer-wm>	 RESOLVED: RedisReplicaDown: Redis replica down rdb2014:16380 redis_misc - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_misc - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_misc&var-instance=rdb2014:16380 - https://alerts.wikimedia.org/?q=alertname%3DRedisReplicaDown
[14:26:35] <wikibugs>	 (03Merged) 10jenkins-bot: AbuseReview: Inidicate if the queue hides recent edits [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344677 (https://phabricator.wikimedia.org/T438235) (owner: 10Kosta Harlan)
[14:27:01] <logmsgbot>	 !log kharlan@deploy1003 Started scap sync-world: Backport for [[gerrit:1344684|AbuseReview: Add local CheckUsers to vandalism alpha test (T438467)]], [[gerrit:1344677|AbuseReview: Inidicate if the queue hides recent edits (T438235)]]
[14:27:05] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[14:27:06] <stashbot>	 T438467: AbuseReview: Enable an alpha testing mode for vandalism detection - https://phabricator.wikimedia.org/T438467
[14:27:06] <stashbot>	 T438235: AbuseReview: Implement filter for time delay view of edits - https://phabricator.wikimedia.org/T438235
[14:27:24] <wikibugs>	 (03CR) 10Daniel Kertesz: [C:03+2] cache::haproxy: implement support for persistent stats [puppet] - 10https://gerrit.wikimedia.org/r/1343039 (https://phabricator.wikimedia.org/T343000) (owner: 10Daniel Kertesz)
[14:29:52] <wikibugs>	 (03CR) 10Daniel Kertesz: hiera: enable HAProxy's persistent stats on two magru nodes [puppet] - 10https://gerrit.wikimedia.org/r/1344222 (https://phabricator.wikimedia.org/T343000) (owner: 10Daniel Kertesz)
[14:30:05] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[14:30:05] <jouncebot>	 Deploy window Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260924T1430)
[14:30:17] <jinxer-wm>	 FIRING: [3x] ProbeDown: Service wdqs1011:443 has failed probes (http_wdqs_main_external_search_sparql_endpoint_ip4)  - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[14:30:17] <wikibugs>	 (03CR) 10Fabfur: [C:03+1] "LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/1344222 (https://phabricator.wikimedia.org/T343000) (owner: 10Daniel Kertesz)
[14:30:43] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.reboot-nodes rolling reboot on P{dse-k8s-worker1*.eqiad.wmnet} and (A:dse-k8s-master-eqiad or A:dse-k8s-worker-eqiad)
[14:30:47] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1001.eqiad.wmnet
[14:31:18] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1001.eqiad.wmnet
[14:32:05] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[14:32:07] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[14:32:41] <dkertesz>	 !log depooling cp7001|7011 to apply https://gerrit.wikimedia.org/r/c/operations/puppet/+/1344222 (context: https://phabricator.wikimedia.org/T343000)
[14:32:43] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:33:59] <logmsgbot>	 !log dkertesz@cumin1004 conftool action : set/pooled=no; selector: name=cp7001.*
[14:34:53] <logmsgbot>	 !log dkertesz@cumin1004 conftool action : set/pooled=no; selector: name=cp7011.*
[14:35:17] <jinxer-wm>	 FIRING: [3x] ProbeDown: Service wdqs1011:443 has failed probes (http_wdqs_main_external_search_sparql_endpoint_ip4)  - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[14:36:05] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[14:37:05] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[14:37:49] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1001.eqiad.wmnet
[14:37:50] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1001.eqiad.wmnet
[14:37:58] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1002.eqiad.wmnet
[14:38:22] <wikibugs>	 (03CR) 10Daniel Kertesz: [C:03+2] hiera: enable HAProxy's persistent stats on two magru nodes [puppet] - 10https://gerrit.wikimedia.org/r/1344222 (https://phabricator.wikimedia.org/T343000) (owner: 10Daniel Kertesz)
[14:39:01] <wikibugs>	 (03PS1) 10Btullis: dse-k8s-eqiad: Put dse-k8s-worker1017 back into service [puppet] - 10https://gerrit.wikimedia.org/r/1344690 (https://phabricator.wikimedia.org/T429773)
[14:39:29] <logmsgbot>	 !log cdobbins@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir3005.esams.wmnet with reason: host reimage
[14:39:42] <logmsgbot>	 !log vgutierrez@cumin1004 END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P{cp[5025,5026].*} and A:cp - 3.2.23 upgrade (T438828)
[14:39:44] <stashbot>	 T438828: Upgrade HAProxy to 3.2.23 on cp hosts - https://phabricator.wikimedia.org/T438828
[14:39:50] <logmsgbot>	 !log btullis@puppetserver1001 conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker1015.eqiad.wmnet
[14:39:55] <logmsgbot>	 !log btullis@puppetserver1001 conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker1016.eqiad.wmnet
[14:40:02] <logmsgbot>	 !log btullis@puppetserver1001 conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker1015.eqiad.wmnet
[14:40:03] <logmsgbot>	 !log btullis@cumin1004 END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1002.eqiad.wmnet
[14:40:05] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1020.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1019.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[14:40:07] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[14:40:07] <logmsgbot>	 !log btullis@puppetserver1001 conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=codfw,name=dse-k8s-worker1016.eqiad.wmnet
[14:40:36] <logmsgbot>	 !log btullis@puppetserver1001 conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1015.eqiad.wmnet
[14:40:42] <logmsgbot>	 !log btullis@puppetserver1001 conftool action : set/weight=10; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1016.eqiad.wmnet
[14:40:51] <logmsgbot>	 !log btullis@puppetserver1001 conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1015.eqiad.wmnet
[14:40:56] <logmsgbot>	 !log btullis@puppetserver1001 conftool action : set/pooled=yes; selector: service=kubesvc,cluster=dse-k8s,dc=eqiad,name=dse-k8s-worker1016.eqiad.wmnet
[14:41:05] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[14:42:07] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[14:42:24] <wikibugs>	 (03PS2) 10Muehlenhoff: Remove wmflib::wmf_php_version [puppet] - 10https://gerrit.wikimedia.org/r/1342215
[14:42:54] <wikibugs>	 (03PS3) 10Muehlenhoff: Rename puppetmaster::puppetdb to puppetdb [puppet] - 10https://gerrit.wikimedia.org/r/1331470
[14:43:04] <logmsgbot>	 !log cdobbins@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir3005.esams.wmnet with reason: host reimage
[14:45:07] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1020.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[14:45:17] <jinxer-wm>	 RESOLVED: ProbeDown: Service wdqs1016:443 has failed probes (http_wdqs_main_external_search_sparql_endpoint_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#wdqs1016:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[14:45:23] <jinxer-wm>	 FIRING: GnmiInterfaceCountersDrop: ...
[14:45:23] <jinxer-wm>	 asw1-b13-drmrs is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?orgId=1&from=now-24h&to=now&var-site=%24__all&var-instance=asw1-b13-drmrs:9804&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop
[14:46:05] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1021.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[14:46:22] <logmsgbot>	 !log kharlan@deploy1003 kharlan: Backport for [[gerrit:1344684|AbuseReview: Add local CheckUsers to vandalism alpha test (T438467)]], [[gerrit:1344677|AbuseReview: Inidicate if the queue hides recent edits (T438235)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
[14:46:27] <stashbot>	 T438467: AbuseReview: Enable an alpha testing mode for vandalism detection - https://phabricator.wikimedia.org/T438467
[14:46:28] <stashbot>	 T438235: AbuseReview: Implement filter for time delay view of edits - https://phabricator.wikimedia.org/T438235
[14:47:05] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[14:47:07] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[14:47:26] <logmsgbot>	 !log kharlan@deploy1003 kharlan: Continuing with deployment
[14:49:48] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1002.eqiad.wmnet
[14:49:49] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1002.eqiad.wmnet
[14:49:55] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1003.eqiad.wmnet
[14:50:05] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1014.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[14:50:35] <logmsgbot>	 !log lerickson@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
[14:51:22] <dkertesz>	 !log repooling cp7001|7011 after successful testing (T343000)
[14:51:25] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[14:51:26] <logmsgbot>	 !log lerickson@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
[14:51:26] <stashbot>	 T343000: HAProxy metrics go down on config reload - https://phabricator.wikimedia.org/T343000
[14:52:05] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[14:53:06] <logmsgbot>	 !log lerickson@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
[14:53:33] <logmsgbot>	 !log lerickson@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
[14:53:53] <logmsgbot>	 !log lerickson@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
[14:54:18] <logmsgbot>	 !log lerickson@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
[14:54:33] <logmsgbot>	 !log dkertesz@cumin1004 conftool action : set/pooled=yes; selector: name=cp7001.*
[14:54:39] <logmsgbot>	 !log dkertesz@cumin1004 conftool action : set/pooled=yes; selector: name=cp7011.*
[14:54:39] <wikibugs>	 (03CR) 10Atsuko: [C:03+1] dse-k8s-eqiad: Put dse-k8s-worker1017 back into service [puppet] - 10https://gerrit.wikimedia.org/r/1344690 (https://phabricator.wikimedia.org/T429773) (owner: 10Btullis)
[14:56:05] <icinga-wm>	 PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 94785624 and 10 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[14:56:59] <wikibugs>	 (03PS1) 10Kosta Harlan: Sync wmf/1.47.0-wmf.20 with wmf/1.47.0-wmf.21 for vandalism alpha [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1344693 (https://phabricator.wikimedia.org/T438467)
[14:57:02] <logmsgbot>	 !log trueg@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
[14:57:05] <icinga-wm>	 RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 1024 and 2 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[14:58:57] <wikibugs>	 (03CR) 10Muehlenhoff: [C:03+2] Apply hiera config to correct idp-test host [puppet] - 10https://gerrit.wikimedia.org/r/1344664 (owner: 10Muehlenhoff)
[14:59:21] <logmsgbot>	 !log kharlan@deploy1003 Finished scap sync-world: Backport for [[gerrit:1344684|AbuseReview: Add local CheckUsers to vandalism alpha test (T438467)]], [[gerrit:1344677|AbuseReview: Inidicate if the queue hides recent edits (T438235)]] (duration: 32m 20s)
[14:59:26] <stashbot>	 T438467: AbuseReview: Enable an alpha testing mode for vandalism detection - https://phabricator.wikimedia.org/T438467
[14:59:27] <stashbot>	 T438235: AbuseReview: Implement filter for time delay view of edits - https://phabricator.wikimedia.org/T438235
[14:59:38] <wikibugs>	 (03CR) 10EarlyWarningBot: "[Failed command](https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php83/115151/consoleFull): `composer run --timeout=0 phpuni" [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1344693 (https://phabricator.wikimedia.org/T438467) (owner: 10Kosta Harlan)
[14:59:45] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Sync wmf/1.47.0-wmf.20 with wmf/1.47.0-wmf.21 for vandalism alpha [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1344693 (https://phabricator.wikimedia.org/T438467) (owner: 10Kosta Harlan)
[15:00:01] <wikibugs>	 (03CR) 10Muehlenhoff: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1331470 (owner: 10Muehlenhoff)
[15:00:02] <wikibugs>	 06SRE, 10observability, 06Traffic, 13Patch-For-Review: HAProxy metrics go down on config reload - https://phabricator.wikimedia.org/T343000#12360510 (10dkertesz)
[15:02:01] <wikibugs>	 06SRE, 10observability, 06Traffic, 13Patch-For-Review: HAProxy metrics go down on config reload - https://phabricator.wikimedia.org/T343000#12360520 (10dkertesz) I have merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/1343039 which introduces support for persistent stats in haproxy; to test that...
[15:02:30] <wikibugs>	 (03CR) 10Btullis: [C:03+2] dse-k8s-eqiad: Put dse-k8s-worker1017 back into service [puppet] - 10https://gerrit.wikimedia.org/r/1344690 (https://phabricator.wikimedia.org/T429773) (owner: 10Btullis)
[15:03:39] <logmsgbot>	 !log vgutierrez@puppetserver1001 conftool action : set/pooled=yes; selector: name=dns2.*,service=authdns-update
[15:03:56] <wikibugs>	 06SRE, 10observability, 06Traffic, 13Patch-For-Review: HAProxy metrics go down on config reload - https://phabricator.wikimedia.org/T343000#12360525 (10dkertesz)
[15:04:14] <wikibugs>	 10ops-eqiad, 06SRE, 06DC-Ops: Interface eno1 on wikikube-worker1160 has the wrong speed - https://phabricator.wikimedia.org/T439094#12360526 (10Jclark-ctr) 05Open→03Resolved
[15:04:25] <wikibugs>	 (03PS1) 10Kosta Harlan: EventMapper::fetchByPage: Allow filtering by type [extensions/Echo] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1344694 (https://phabricator.wikimedia.org/T438031)
[15:04:36] <kostajh>	 going with another backport
[15:04:39] <logmsgbot>	 !log vgutierrez@dns1004 START - running authdns-update
[15:04:51] <wikibugs>	 (03CR) 10TrainBranchBot: [C:03+2] "Approved by kharlan@deploy1003 using scap backport" [extensions/Echo] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1344694 (https://phabricator.wikimedia.org/T438031) (owner: 10Kosta Harlan)
[15:05:14] <hnowlan>	 kostajh: please hold for a minute 
[15:05:36] <hnowlan>	 we are bringing DNS back up in codfw
[15:06:07] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1018.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[15:06:08] <kostajh>	 hnowlan: ok
[15:06:40] <wikibugs>	 (03CR) 10Kosta Harlan: [C:04-2] EventMapper::fetchByPage: Allow filtering by type [extensions/Echo] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1344694 (https://phabricator.wikimedia.org/T438031) (owner: 10Kosta Harlan)
[15:06:59] <logmsgbot>	 !log vgutierrez@dns1004 END - running authdns-update
[15:07:07] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[15:09:36] <wikibugs>	 (03CR) 10Kosta Harlan: EventMapper::fetchByPage: Allow filtering by type [extensions/Echo] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1344694 (https://phabricator.wikimedia.org/T438031) (owner: 10Kosta Harlan)
[15:09:49] <jinxer-wm>	 FIRING: HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@eqiad in state pending-upgrade - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus
[15:09:57] <wikibugs>	 (03PS1) 10Kosta Harlan: EventMapper::fetchByPage: Allow filtering by type [extensions/Echo] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1344694 (https://phabricator.wikimedia.org/T438031)
[15:10:05] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[15:11:08] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[15:12:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[15:12:10] <logmsgbot>	 !log vgutierrez@puppetserver1001 conftool action : set/pooled=yes; selector: dc=codfw,cluster=dnsbox
[15:12:15] <jinxer-wm>	 FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=codfw%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[15:13:08] <icinga-wm>	 RECOVERY - Host ns1-v6 is UP: PING OK - Packet loss = 0%, RTA = 31.74 ms
[15:13:08] <icinga-wm>	 RECOVERY - Host ns1-v4 is UP: PING OK - Packet loss = 0%, RTA = 30.48 ms
[15:14:08] <icinga-wm>	 PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 442252440 and 29 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[15:15:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1018.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[15:16:08] <icinga-wm>	 RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 2926928 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[15:16:08] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[15:16:25] <hnowlan>	 kostajh: think you're okay to proceed
[15:16:32] <jinxer-wm>	 RESOLVED: [2x] Not accepting/receiving prefixes from anycast BGP peer: Device cr1-codfw.wikimedia.org recovered from Not accepting/receiving prefixes from anycast BGP peer   - https://alerts.wikimedia.org/?q=alertname%3DNot+accepting%2Freceiving+prefixes+from+anycast+BGP+peer
[15:17:01] <Dreamy_Jazz>	 Kosta had to step out, I'm going to do it
[15:17:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[15:17:15] <jinxer-wm>	 RESOLVED: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook  - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[15:17:40] <wikibugs>	 (03CR) 10TrainBranchBot: [C:03+2] "Approved by dreamyjazz@deploy1003 using scap backport" [extensions/Echo] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1344694 (https://phabricator.wikimedia.org/T438031) (owner: 10Kosta Harlan)
[15:18:02] <logmsgbot>	 !log cdobbins@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir3005.esams.wmnet with OS trixie
[15:18:10] <Dreamy_Jazz>	 jouncebot: nowandnext
[15:18:11] <jouncebot>	 No deployments scheduled for the next 0 hour(s) and 41 minute(s)
[15:18:11] <jouncebot>	 In 0 hour(s) and 41 minute(s): Puppet request window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260924T1600)
[15:18:20] <wikibugs>	 06SRE, 10SRE-Access-Requests: Requesting access to deployment for Tsevener - https://phabricator.wikimedia.org/T439138#12360573 (10Seddon) Approved from myside!
[15:18:56] <wikibugs>	 (03PS2) 10Dreamy Jazz: Sync wmf/1.47.0-wmf.20 with wmf/1.47.0-wmf.21 for vandalism alpha [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1344693 (https://phabricator.wikimedia.org/T438467) (owner: 10Kosta Harlan)
[15:19:41] <wikibugs>	 (03Merged) 10jenkins-bot: EventMapper::fetchByPage: Allow filtering by type [extensions/Echo] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1344694 (https://phabricator.wikimedia.org/T438031) (owner: 10Kosta Harlan)
[15:19:44] <wikibugs>	 (03CR) 10TrainBranchBot: [C:03+2] "Approved by dreamyjazz@deploy1003 using scap backport" [extensions/Echo] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1344694 (https://phabricator.wikimedia.org/T438031) (owner: 10Kosta Harlan)
[15:20:08] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1021.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1012.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[15:20:10] <logmsgbot>	 !log dreamyjazz@deploy1003 Started scap sync-world: Backport for [[gerrit:1344694|EventMapper::fetchByPage: Allow filtering by type (T438031)]]
[15:20:11] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1003.eqiad.wmnet
[15:20:14] <stashbot>	 T438031: EchoPersonalInfoFlagNotificationModerator::moderate can select more than 100,000 database rows in a single query - https://phabricator.wikimedia.org/T438031
[15:21:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1019.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[15:22:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[15:23:08] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[15:24:31] <logmsgbot>	 !log dreamyjazz@deploy1003 kharlan, dreamyjazz: Backport for [[gerrit:1344694|EventMapper::fetchByPage: Allow filtering by type (T438031)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
[15:24:49] <jinxer-wm>	 RESOLVED: HelmReleaseBadStatus: Helm release wdqs-next/main-external on k8s-dse@eqiad in state pending-upgrade - https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Rolling_back_in_an_emergency - https://grafana.wikimedia.org/d/UT4GtK3nz?var-site=eqiad&var-cluster=k8s-dse&var-namespace=wdqs-next - https://alerts.wikimedia.org/?q=alertname%3DHelmReleaseBadStatus
[15:25:12] <logmsgbot>	 !log dreamyjazz@deploy1003 kharlan, dreamyjazz: Continuing with deployment
[15:26:02] <icinga-wm>	 RECOVERY - Host ps1-f4-codfw is UP: PING OK - Packet loss = 0%, RTA = 34.85 ms
[15:26:04] <icinga-wm>	 PROBLEM - ps1-f4-codfw-infeed-load-tower-A-phase-Y on ps1-f4-codfw is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook
[15:26:04] <icinga-wm>	 PROBLEM - ps1-f4-codfw-infeed-load-tower-B-phase-Z on ps1-f4-codfw is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook
[15:26:04] <icinga-wm>	 PROBLEM - ps1-f4-codfw-infeed-load-tower-B-phase-X on ps1-f4-codfw is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook
[15:26:04] <icinga-wm>	 PROBLEM - ps1-f4-codfw-infeed-load-tower-A-phase-Z on ps1-f4-codfw is CRITICAL: CRITICAL - Plugin timed out while executing system call https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook
[15:26:08] <icinga-wm>	 RECOVERY - ps1-f4-codfw-infeed-load-tower-A-phase-Z on ps1-f4-codfw is OK: SNMP OK - ps1-f4-codfw-infeed-load-tower-A-phase-Z 140 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook
[15:26:08] <icinga-wm>	 RECOVERY - ps1-f4-codfw-infeed-load-tower-B-phase-Z on ps1-f4-codfw is OK: SNMP OK - ps1-f4-codfw-infeed-load-tower-B-phase-Z 147 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook
[15:26:08] <icinga-wm>	 RECOVERY - ps1-f4-codfw-infeed-load-tower-A-phase-Y on ps1-f4-codfw is OK: SNMP OK - ps1-f4-codfw-infeed-load-tower-A-phase-Y 221 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook
[15:26:08] <icinga-wm>	 RECOVERY - ps1-f4-codfw-infeed-load-tower-B-phase-X on ps1-f4-codfw is OK: SNMP OK - ps1-f4-codfw-infeed-load-tower-B-phase-X 306 https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook
[15:26:35] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1003.eqiad.wmnet
[15:26:36] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1003.eqiad.wmnet
[15:26:42] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1004.eqiad.wmnet
[15:26:48] <wikibugs>	 10ops-codfw, 06SRE, 06DC-Ops: Unresponsive management for wikikube-worker2374.mgmt:22 - https://phabricator.wikimedia.org/T439091#12360624 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm rebooted msw in f4
[15:27:10] <wikibugs>	 10ops-codfw, 06SRE, 06DC-Ops: Unresponsive management for ms-be2098.mgmt:22 - https://phabricator.wikimedia.org/T439092#12360629 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm rebooted msw in f4
[15:27:12] <logmsgbot>	 !log trueg@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
[15:27:23] <jinxer-wm>	 FIRING: [3x] CertAlmostExpired: gNMI TLS certificate for lsw1-e2-codfw.mgmt.codfw.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired
[15:27:23] <logmsgbot>	 !log trueg@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
[15:29:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[15:29:08] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[15:29:09] <logmsgbot>	 !log trueg@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
[15:29:15] <jinxer-wm>	 FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[15:29:34] <icinga-wm>	 RECOVERY - Host lsw1-f4-codfw.mgmt is UP: PING OK - Packet loss = 0%, RTA = 31.72 ms
[15:30:07] <wikibugs>	 (03PS1) 10Trueg: Bump wdqs-streaming-consumer version to 0.3.1 on wdqs-next main eqiad [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344706 (https://phabricator.wikimedia.org/T430316)
[15:30:29] <logmsgbot>	 !log ebernhardson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
[15:30:33] <logmsgbot>	 !log ebernhardson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
[15:31:06] <wikibugs>	 (03PS1) 10Vgutierrez: wikimediafoundation.org: Restore pki.goog CAA record [dns] - 10https://gerrit.wikimedia.org/r/1344707 (https://phabricator.wikimedia.org/T428093)
[15:32:01] <icinga-wm>	 RECOVERY - Host lsw1-e5-codfw.mgmt is UP: PING OK - Packet loss = 0%, RTA = 32.95 ms
[15:32:05] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[15:32:09] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[15:32:39] <wikibugs>	 (03CR) 10CDanis: [C:03+1] wikimediafoundation.org: Restore pki.goog CAA record [dns] - 10https://gerrit.wikimedia.org/r/1344707 (https://phabricator.wikimedia.org/T428093) (owner: 10Vgutierrez)
[15:32:43] <logmsgbot>	 !log dreamyjazz@deploy1003 Finished scap sync-world: Backport for [[gerrit:1344694|EventMapper::fetchByPage: Allow filtering by type (T438031)]] (duration: 12m 33s)
[15:32:47] <stashbot>	 T438031: EchoPersonalInfoFlagNotificationModerator::moderate can select more than 100,000 database rows in a single query - https://phabricator.wikimedia.org/T438031
[15:33:02] <Dreamy_Jazz>	 Going again for backports
[15:33:03] <wikibugs>	 (03CR) 10Vgutierrez: [C:03+2] wikimediafoundation.org: Restore pki.goog CAA record [dns] - 10https://gerrit.wikimedia.org/r/1344707 (https://phabricator.wikimedia.org/T428093) (owner: 10Vgutierrez)
[15:33:09] <icinga-wm>	 RECOVERY - Host ps1-e5-codfw is UP: PING OK - Packet loss = 0%, RTA = 32.37 ms
[15:33:12] <wikibugs>	 (03CR) 10Lerickson: [C:03+1] Bump wdqs-streaming-consumer version to 0.3.1 on wdqs-next main eqiad [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344706 (https://phabricator.wikimedia.org/T430316) (owner: 10Trueg)
[15:33:30] <logmsgbot>	 !log vgutierrez@dns1004 START - running authdns-update
[15:33:59] <wikibugs>	 (03CR) 10Trueg: [C:03+2] Bump wdqs-streaming-consumer version to 0.3.1 on wdqs-next main eqiad [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344706 (https://phabricator.wikimedia.org/T430316) (owner: 10Trueg)
[15:34:15] <jinxer-wm>	 RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[15:35:31] <wikibugs>	 (03PS1) 10Dreamy Jazz: AbuseReview: Enable on enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344711 (https://phabricator.wikimedia.org/T439149)
[15:35:59] <icinga-wm>	 RECOVERY - Host ps1-e2-codfw is UP: PING OK - Packet loss = 0%, RTA = 30.93 ms
[15:35:59] <logmsgbot>	 !log vgutierrez@dns1004 END - running authdns-update
[15:36:23] <wikibugs>	 (03Merged) 10jenkins-bot: Bump wdqs-streaming-consumer version to 0.3.1 on wdqs-next main eqiad [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344706 (https://phabricator.wikimedia.org/T430316) (owner: 10Trueg)
[15:36:43] <wikibugs>	 (03PS1) 10Urbanecm: ReassignMentees: Actually deduplicate reassignMenteesJob [extensions/GrowthExperiments] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344712 (https://phabricator.wikimedia.org/T418194)
[15:36:58] <wikibugs>	 (03PS1) 10Urbanecm: ReassignMentees: Actually deduplicate reassignMenteesJob [extensions/GrowthExperiments] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1344713 (https://phabricator.wikimedia.org/T418194)
[15:37:02] <wikibugs>	 10ops-codfw, 06SRE, 06DC-Ops: Unresponsive management for wikikube-worker2355.mgmt:22 - https://phabricator.wikimedia.org/T439084#12360701 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm rebooted msw in f4
[15:37:07] <icinga-wm>	 RECOVERY - Host lsw1-e2-codfw.mgmt is UP: PING OK - Packet loss = 0%, RTA = 32.90 ms
[15:37:23] <jinxer-wm>	 RESOLVED: [3x] CertAlmostExpired: gNMI TLS certificate for lsw1-e2-codfw.mgmt.codfw.wmnet is going to expire in 0s - https://wikitech.wikimedia.org/wiki/Network_monitoring#CertAlmostExpired - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic?var-site=codfw - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired
[15:37:53] <wikibugs>	 10ops-codfw, 06SRE, 06DC-Ops: Unresponsive management for backup2018.mgmt:22 - https://phabricator.wikimedia.org/T439086#12360708 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm rebooted msw in F4
[15:38:09] <wikibugs>	 (03CR) 10TrainBranchBot: [C:03+2] "Approved by dreamyjazz@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344711 (https://phabricator.wikimedia.org/T439149) (owner: 10Dreamy Jazz)
[15:38:09] <wikibugs>	 (03CR) 10TrainBranchBot: [C:03+2] "Approved by dreamyjazz@deploy1003 using scap backport" [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1344693 (https://phabricator.wikimedia.org/T438467) (owner: 10Kosta Harlan)
[15:38:50] <wikibugs>	 10ops-codfw, 06SRE, 06DC-Ops: Unresponsive management for ms-fe2024.mgmt:22 - https://phabricator.wikimedia.org/T439083#12360715 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm rebooted msw in F4
[15:38:55] <Dreamy_Jazz>	 jouncebot: nowandnext
[15:38:55] <jouncebot>	 No deployments scheduled for the next 0 hour(s) and 21 minute(s)
[15:38:55] <jouncebot>	 In 0 hour(s) and 21 minute(s): Puppet request window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260924T1600)
[15:39:23] <wikibugs>	 10ops-codfw, 06SRE, 06DC-Ops: Unresponsive management for wikikube-worker2354.mgmt:22 - https://phabricator.wikimedia.org/T439089#12360723 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm rebooted msw in F4
[15:39:30] <wikibugs>	 (03Merged) 10jenkins-bot: AbuseReview: Enable on enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344711 (https://phabricator.wikimedia.org/T439149) (owner: 10Dreamy Jazz)
[15:40:11] <wikibugs>	 10ops-codfw, 06SRE, 06DC-Ops: Unresponsive management for ms-fe2023.mgmt:22 - https://phabricator.wikimedia.org/T439085#12360732 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm rebooted msw in E5
[15:40:12] <wikibugs>	 (03Merged) 10jenkins-bot: Sync wmf/1.47.0-wmf.20 with wmf/1.47.0-wmf.21 for vandalism alpha [extensions/WikimediaAntiAbuse] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1344693 (https://phabricator.wikimedia.org/T438467) (owner: 10Kosta Harlan)
[15:40:37] <logmsgbot>	 !log dreamyjazz@deploy1003 Started scap sync-world: Backport for [[gerrit:1344711|AbuseReview: Enable on enwiki (T439149)]], [[gerrit:1344693|Sync wmf/1.47.0-wmf.20 with wmf/1.47.0-wmf.21 for vandalism alpha (T438467)]]
[15:40:43] <stashbot>	 T439149: AbuseReview: Enable an alpha testing mode for vandalism detection on enwiki - https://phabricator.wikimedia.org/T439149
[15:40:43] <stashbot>	 T438467: AbuseReview: Enable an alpha testing mode for vandalism detection - https://phabricator.wikimedia.org/T438467
[15:40:45] <wikibugs>	 10ops-codfw, 06SRE, 06DC-Ops: Unresponsive management for wikikube-worker2359.mgmt:22 - https://phabricator.wikimedia.org/T439081#12360738 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm rebooted msw in E2
[15:41:27] <wikibugs>	 10ops-codfw, 06SRE, 06DC-Ops: Unresponsive management for backup2017.mgmt:22 - https://phabricator.wikimedia.org/T439080#12360750 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm rebooted the msw in E2
[15:42:15] <wikibugs>	 10ops-codfw, 06SRE, 06DC-Ops: Unresponsive management for wikikube-worker2343.mgmt:22 - https://phabricator.wikimedia.org/T439090#12360754 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm rebooted msw in E5
[15:42:54] <wikibugs>	 10ops-codfw, 06SRE, 06DC-Ops: Unresponsive management for wikikube-worker2365.mgmt:22 - https://phabricator.wikimedia.org/T439088#12360758 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm rebooted msw in E5
[15:44:05] <wikibugs>	 10ops-codfw, 06SRE, 06DC-Ops: Unresponsive management for wikikube-worker2356.mgmt:22 - https://phabricator.wikimedia.org/T439087#12360764 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm rebooted msw in F4
[15:45:25] <urbanecm>	 Dreamy_Jazz: how are your deployments going?
[15:45:46] <Dreamy_Jazz>	 It has i18n rebuild so going slowly :D
[15:45:52] <urbanecm>	 i see :D
[15:45:55] <urbanecm>	 I'd like to squeeze https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/1344712 and https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/1344713 in if possible
[15:45:56] <wikibugs>	 (03PS2) 10DLynch: EditCheck: add some statsv tracking of check/suggestion actions [extensions/VisualEditor] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344714 (https://phabricator.wikimedia.org/T438916)
[15:45:59] <urbanecm>	 but wcs i'll do them next week
[15:46:04] <Dreamy_Jazz>	 jouncebot: nowandnext
[15:46:04] <jouncebot>	 No deployments scheduled for the next 0 hour(s) and 13 minute(s)
[15:46:04] <jouncebot>	 In 0 hour(s) and 13 minute(s): Puppet request window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260924T1600)
[15:46:07] <icinga-wm>	 PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 555388360 and 65 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[15:46:08] <wikibugs>	 (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, September 24 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployca" [extensions/VisualEditor] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344714 (https://phabricator.wikimedia.org/T438916) (owner: 10DLynch)
[15:46:21] <Dreamy_Jazz>	 Nothing in the puppet window, so should be fine to deploy there
[15:46:59] <Dreamy_Jazz>	 I'll ping you when I'm done
[15:47:21] <wikibugs>	 (03PS1) 10Ebernhardson: semantic ssd test: Drop to 3 masters, fix affinity [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344715 (https://phabricator.wikimedia.org/T438058)
[15:47:35] <urbanecm>	 sounds good
[15:47:59] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.hosts.reboot-single for host dse-k8s-worker1017.eqiad.wmnet
[15:48:15] <jinxer-wm>	 FIRING: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[15:48:59] <wikibugs>	 (03PS1) 10Marco Fossati: ReaderExperiments: add dewiki and svwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344716 (https://phabricator.wikimedia.org/T438072)
[15:50:07] <icinga-wm>	 RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 39512 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[15:50:09] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1020.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[15:51:05] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1021.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[15:51:09] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[15:51:25] <swfrench-wmf>	 !log begin rolling restarts of confds in eqsin, codfw, ulsfo to reflect etcd SRV record changes
[15:51:26] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[15:51:51] <logmsgbot>	 !log cdobbins@cumin1004 conftool action : set/pooled=yes; selector: name=ncredir3005.*
[15:52:05] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[15:52:44] <logmsgbot>	 !log cdobbins@cumin1004 START - Cookbook sre.hosts.reimage for host ncredir1001.eqiad.wmnet with OS trixie
[15:53:14] <wikibugs>	 (03CR) 10Kosta Harlan: [C:03+1] AbuseReview: Enable on enwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344711 (https://phabricator.wikimedia.org/T439149) (owner: 10Dreamy Jazz)
[15:53:15] <jinxer-wm>	 RESOLVED: MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts?var-datasource=eqiad%20prometheus/ops&viewPanel=19 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[15:53:55] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host dse-k8s-worker1017.eqiad.wmnet
[15:55:05] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[15:55:09] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[15:56:56] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1004.eqiad.wmnet
[15:57:28] <logmsgbot>	 !log ebernhardson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
[15:57:32] <logmsgbot>	 !log ebernhardson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
[15:59:57] <logmsgbot>	 !log dreamyjazz@deploy1003 kharlan, dreamyjazz: Backport for [[gerrit:1344711|AbuseReview: Enable on enwiki (T439149)]], [[gerrit:1344693|Sync wmf/1.47.0-wmf.20 with wmf/1.47.0-wmf.21 for vandalism alpha (T438467)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
[16:00:02] <stashbot>	 T439149: AbuseReview: Enable an alpha testing mode for vandalism detection on enwiki - https://phabricator.wikimedia.org/T439149
[16:00:03] <stashbot>	 T438467: AbuseReview: Enable an alpha testing mode for vandalism detection - https://phabricator.wikimedia.org/T438467
[16:00:04] <jouncebot>	 jhathaway and rzl: It is that lovely time of the day again! You are hereby commanded to deploy Puppet request window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260924T1600).
[16:00:04] <jouncebot>	 No Gerrit patches in the queue for this window AFAICS.
[16:00:30] <Dreamy_Jazz>	 I'm currently deploying, though looks like no patches in this window so should be okay?
[16:00:32] <wikibugs>	 (03CR) 10Btullis: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1344631 (https://phabricator.wikimedia.org/T430658) (owner: 10Btullis)
[16:00:36] <wikibugs>	 (03PS5) 10Blake: mw-web: set an anti-affinity preference for thumbor nodes [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344653 (https://phabricator.wikimedia.org/T420223)
[16:00:58] <rzl>	 nothing planned for the puppet window afaik, yeah
[16:01:26] <jhathaway>	 nod, thanks rzl 
[16:01:26] <logmsgbot>	 !log dreamyjazz@deploy1003 kharlan, dreamyjazz: Continuing with deployment
[16:01:31] <logmsgbot>	 !log swfrench@cumin1004 START - Cookbook sre.loadbalancer.upgrade restart A:liberica-ulsfo
[16:01:38] <wikibugs>	 (03PS1) 10Andrew Bogott: cloud-vps eqiad1: Enable zookeeper logs [puppet] - 10https://gerrit.wikimedia.org/r/1344721 (https://phabricator.wikimedia.org/T435503)
[16:01:51] <logmsgbot>	 !log hnowlan@cumin1004 START - Cookbook sre.switchdc.mediawiki.00-optional-warmup-caches for datacenter switchover from eqiad to codfw
[16:02:16] <wikibugs>	 10ops-codfw, 06SRE, 06DC-Ops: Unresponsive management for wikikube-worker2336.mgmt:22 - https://phabricator.wikimedia.org/T439082#12360857 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm rebooted msw in E2
[16:02:22] <wikibugs>	 10ops-codfw, 06SRE, 06DC-Ops: Unresponsive management for mwlog2003.mgmt:22 - https://phabricator.wikimedia.org/T439079#12360861 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm rebooted msw in F4
[16:02:26] <wikibugs>	 (03CR) 10Andrew Bogott: "Because more logs are better!" [puppet] - 10https://gerrit.wikimedia.org/r/1344721 (https://phabricator.wikimedia.org/T435503) (owner: 10Andrew Bogott)
[16:02:32] <wikibugs>	 10ops-codfw, 06SRE, 06DC-Ops: Unresponsive management for wikikube-worker2373.mgmt:22 - https://phabricator.wikimedia.org/T439077#12360868 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm rebooted msw in F4
[16:02:39] <wikibugs>	 10ops-codfw, 06SRE, 06DC-Ops: Unresponsive management for wikikube-worker2335.mgmt:22 - https://phabricator.wikimedia.org/T439076#12360874 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm rebooted msw in E2
[16:02:56] <wikibugs>	 10ops-codfw, 06SRE, 06DC-Ops: Unresponsive management for wikikube-worker2344.mgmt:22 - https://phabricator.wikimedia.org/T439075#12360880 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm rebooted msw in E5
[16:03:12] <wikibugs>	 10ops-codfw, 06SRE, 06DC-Ops: Unresponsive management for wikikube-worker2360.mgmt:22 - https://phabricator.wikimedia.org/T439074#12360887 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm rebooted msw in E2
[16:03:15] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1004.eqiad.wmnet
[16:03:16] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1004.eqiad.wmnet
[16:03:19] <logmsgbot>	 !log swfrench@cumin1004 END (PASS) - Cookbook sre.loadbalancer.upgrade (exit_code=0) restart A:liberica-ulsfo
[16:03:21] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1005.eqiad.wmnet
[16:03:34] <wikibugs>	 10ops-codfw, 06SRE, 06DC-Ops: Unresponsive management for wikikube-worker2334.mgmt:22 - https://phabricator.wikimedia.org/T439073#12360891 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm rebooted msw in E2
[16:03:51] <wikibugs>	 10ops-codfw, 06SRE, 06DC-Ops: Unresponsive management for ms-be2095.mgmt:22 - https://phabricator.wikimedia.org/T439072#12360895 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm rebooted msw in E5
[16:04:07] <wikibugs>	 10ops-codfw, 06SRE, 06DC-Ops: Unresponsive management for wikikube-worker2366.mgmt:22 - https://phabricator.wikimedia.org/T439071#12360899 (10Jhancock.wm) rebooted msw in E5
[16:04:24] <wikibugs>	 10ops-codfw, 06SRE, 06DC-Ops: Unresponsive management for wikikube-worker2366.mgmt:22 - https://phabricator.wikimedia.org/T439071#12360900 (10Jhancock.wm) rebooted msw in E5
[16:04:54] <wikibugs>	 10ops-codfw, 06SRE, 06DC-Ops: Unresponsive management for wikikube-worker2366.mgmt:22 - https://phabricator.wikimedia.org/T439071#12360903 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm
[16:04:59] <wikibugs>	 (03CR) 10Atsuko: [C:03+1] clouddumps: Allow the new IP Pool in dse-k8s-eqiad to ssh to clouddumps [puppet] - 10https://gerrit.wikimedia.org/r/1344631 (https://phabricator.wikimedia.org/T430658) (owner: 10Btullis)
[16:05:11] <wikibugs>	 10ops-codfw, 06SRE, 06DC-Ops: Unresponsive management for apus-be2006.mgmt:22 - https://phabricator.wikimedia.org/T439070#12360906 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm rebooted msw in F4
[16:05:31] <wikibugs>	 (03CR) 10Brouberol: [C:03+1] clouddumps: Allow the new IP Pool in dse-k8s-eqiad to ssh to clouddumps [puppet] - 10https://gerrit.wikimedia.org/r/1344631 (https://phabricator.wikimedia.org/T430658) (owner: 10Btullis)
[16:05:33] <wikibugs>	 10ops-codfw, 06SRE, 06DC-Ops: Unresponsive management for wikikube-worker2342.mgmt:22 - https://phabricator.wikimedia.org/T439069#12360910 (10Jhancock.wm) rebooted msw in E5
[16:05:47] <wikibugs>	 10ops-codfw, 06SRE, 06DC-Ops: Unresponsive management for wikikube-worker2342.mgmt:22 - https://phabricator.wikimedia.org/T439069#12360911 (10Jhancock.wm) rebooted msw in E5
[16:06:04] <wikibugs>	 10ops-codfw, 06SRE, 06DC-Ops: Unresponsive management for wikikube-worker2342.mgmt:22 - https://phabricator.wikimedia.org/T439069#12360912 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm
[16:06:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[16:06:08] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[16:06:10] <icinga-wm>	 PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 272571824 and 22 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[16:06:20] <wikibugs>	 10ops-codfw, 06SRE, 06DC-Ops: Unresponsive management for dse-k8s-wdqs2001.mgmt:22 - https://phabricator.wikimedia.org/T439068#12360915 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm rebooted msw in E5
[16:06:41] <Dreamy_Jazz>	 urbanecm: I believe SRE will need the scap lock after me, so yours may need to wait longer
[16:06:54] <urbanecm>	 sounds good
[16:06:56] <urbanecm>	 i'll wait
[16:08:05] <logmsgbot>	 !log cdobbins@cumin1004 START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir1001.eqiad.wmnet with reason: host reimage
[16:09:27] <wikibugs>	 (03CR) 10Btullis: [C:03+2] clouddumps: Allow the new IP Pool in dse-k8s-eqiad to ssh to clouddumps [puppet] - 10https://gerrit.wikimedia.org/r/1344631 (https://phabricator.wikimedia.org/T430658) (owner: 10Btullis)
[16:11:42] <jinxer-wm>	 FIRING: JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[16:14:29] <logmsgbot>	 !log dreamyjazz@deploy1003 Finished scap sync-world: Backport for [[gerrit:1344711|AbuseReview: Enable on enwiki (T439149)]], [[gerrit:1344693|Sync wmf/1.47.0-wmf.20 with wmf/1.47.0-wmf.21 for vandalism alpha (T438467)]] (duration: 33m 52s)
[16:14:35] <stashbot>	 T439149: AbuseReview: Enable an alpha testing mode for vandalism detection on enwiki - https://phabricator.wikimedia.org/T439149
[16:14:36] <stashbot>	 T438467: AbuseReview: Enable an alpha testing mode for vandalism detection - https://phabricator.wikimedia.org/T438467
[16:14:38] <Dreamy_Jazz>	 I'm done with scap
[16:14:41] <logmsgbot>	 !log cdobbins@cumin1004 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir1001.eqiad.wmnet with reason: host reimage
[16:15:12] <icinga-wm>	 RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 543936 and 1 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[16:15:31] <logmsgbot>	 !log taavi@deploy1003 Locking from deployment [ALL REPOSITORIES]: locked for re-pooling codfw for read traffic, contact SRE for equestions
[16:16:42] <jinxer-wm>	 FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[16:18:46] <wikibugs>	 (03CR) 10Clare Ming: [C:03+2] Test Kitchen UI: Deploy v2.0.1 release to staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344607 (https://phabricator.wikimedia.org/T421814) (owner: 10Santiago Faci)
[16:19:56] <wikibugs>	 10ops-codfw, 06SRE, 06DC-Ops: Power Supply - Status - issue on cloudbackup2003:9290 - https://phabricator.wikimedia.org/T439058#12360953 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm reseated psu/cable
[16:19:58] <logmsgbot>	 !log hnowlan@cumin1004 END (FAIL) - Cookbook sre.switchdc.mediawiki.00-optional-warmup-caches (exit_code=99) for datacenter switchover from eqiad to codfw
[16:20:38] <logmsgbot>	 !log hnowlan@cumin1004 START - Cookbook sre.switchdc.mediawiki.00-optional-warmup-caches for datacenter switchover from eqiad to codfw
[16:21:02] <wikibugs>	 (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, September 24 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployca" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344716 (https://phabricator.wikimedia.org/T438072) (owner: 10Marco Fossati)
[16:21:09] <wikibugs>	 (03Merged) 10jenkins-bot: Test Kitchen UI: Deploy v2.0.1 release to staging [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344607 (https://phabricator.wikimedia.org/T421814) (owner: 10Santiago Faci)
[16:21:42] <jinxer-wm>	 RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[16:23:12] <wikibugs>	 10ops-codfw, 06SRE, 06DC-Ops: Power Supply - Status - issue on logstash2036:9290 - https://phabricator.wikimedia.org/T439057#12360969 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm reseated psu/cable
[16:23:19] <logmsgbot>	 !log sfaci@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply
[16:23:42] <wikibugs>	 10ops-codfw, 06SRE, 06DC-Ops: Power Supply - Status - issue on wikikube-ctrl2001:9290 - https://phabricator.wikimedia.org/T439055#12360974 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm reseated psu/cable
[16:23:49] <logmsgbot>	 !log sfaci@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply
[16:23:56] <wikibugs>	 10ops-codfw, 06SRE, 06DC-Ops: Power Supply - Status - issue on cirrussearch2080:9290 - https://phabricator.wikimedia.org/T439054#12360978 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm reseated psu/cable
[16:24:47] <wikibugs>	 (03CR) 10Clare Ming: [C:03+2] Test Kitchen UI: Deploy v2.0.1 release to production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344609 (https://phabricator.wikimedia.org/T421814) (owner: 10Santiago Faci)
[16:25:08] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1021.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1020.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[16:25:57] <wikibugs>	 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-b2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T438869#12360993 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm
[16:26:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[16:26:47] <wikibugs>	 10ops-codfw, 06SRE, 06DC-Ops: Alert for device ps1-a2-codfw.mgmt.codfw.wmnet - PDU sensor over limit - https://phabricator.wikimedia.org/T438865#12360998 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm
[16:27:07] <wikibugs>	 (03Merged) 10jenkins-bot: Test Kitchen UI: Deploy v2.0.1 release to production [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344609 (https://phabricator.wikimedia.org/T421814) (owner: 10Santiago Faci)
[16:27:10] <wikibugs>	 (03PS6) 10Dduvall: buildx: New driver for building images via Buildx/BuildKit [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1342799 (https://phabricator.wikimedia.org/T434958)
[16:27:58] <wikibugs>	 (03CR) 10Dduvall: "@glavagetto@wikimedia.org the latest patchset should be cruft-free and ready for review." [docker-images/docker-pkg] - 10https://gerrit.wikimedia.org/r/1342799 (https://phabricator.wikimedia.org/T434958) (owner: 10Dduvall)
[16:30:11] <wikibugs>	 (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, September 24 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployca" [extensions/WikimediaEvents] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344406 (https://phabricator.wikimedia.org/T438898) (owner: 10Clare Ming)
[16:31:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[16:31:06] <wikibugs>	 (03CR) 10Dzahn: [C:03+1] "ah, cool. for some reason I had not seen the eqiad role afair" [puppet] - 10https://gerrit.wikimedia.org/r/1344721 (https://phabricator.wikimedia.org/T435503) (owner: 10Andrew Bogott)
[16:31:08] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[16:31:15] <jinxer-wm>	 FIRING: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook  - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[16:32:12] <icinga-wm>	 PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 322566520 and 43 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[16:33:07] <logmsgbot>	 !log cdobbins@cumin1004 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir1001.eqiad.wmnet with OS trixie
[16:33:20] <logmsgbot>	 !log cdanis@cumin1004 conftool action : set/pooled=true; selector: name=codfw,dnsdisc=mw-api-int-ro
[16:33:23] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1005.eqiad.wmnet
[16:35:17] <jinxer-wm>	 FIRING: ProbeDown: Service wdqs1012:443 has failed probes (http_wdqs_main_external_search_sparql_endpoint_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#wdqs1012:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[16:35:48] <jinxer-wm>	 FIRING: PuppetFailure: Puppet has failed on ml-serve1015:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[16:36:15] <jinxer-wm>	 RESOLVED: [2x] MediaWikiMemcachedHighErrorRate: MediaWiki memcached error rate is elevated globally - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook  - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiMemcachedHighErrorRate
[16:37:36] <logmsgbot>	 !log cdanis@cumin1004 conftool action : set/pooled=true; selector: name=codfw,dnsdisc=mw-web-ro
[16:38:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1012.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[16:39:08] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[16:39:12] <icinga-wm>	 RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 49160 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[16:40:08] <logmsgbot>	 !log dzahn@cumin2003 START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie
[16:40:14] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1005.eqiad.wmnet
[16:40:15] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1005.eqiad.wmnet
[16:40:21] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1006.eqiad.wmnet
[16:40:50] <icinga-wm>	 PROBLEM - Host arclamp2001 is DOWN: PING CRITICAL - Packet loss = 100%
[16:42:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[16:42:08] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[16:45:18] <icinga-wm>	 RECOVERY - Host arclamp2001 is UP: PING OK - Packet loss = 0%, RTA = 31.69 ms
[16:45:31] <jinxer-wm>	 FIRING: RedisInstanceDown: Redis instance down arclamp2001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_arclamp&var-instance=arclamp2001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown
[16:47:00] <icinga-wm>	 PROBLEM - Host wikikube-worker1067 is DOWN: PING CRITICAL - Packet loss = 100%
[16:47:38] <icinga-wm>	 RECOVERY - Host wikikube-worker1067 is UP: PING OK - Packet loss = 0%, RTA = 0.35 ms
[16:48:00] <wikibugs>	 (03CR) 10Matthias Mullie: [C:03+1] ReaderExperiments: add dewiki and svwiki (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344716 (https://phabricator.wikimedia.org/T438072) (owner: 10Marco Fossati)
[16:49:01] <wikibugs>	 10ops-codfw, 06SRE, 06DC-Ops: arclamp2001 has a broken DIMM - https://phabricator.wikimedia.org/T439095#12361095 (10Jhancock.wm) 05Open→03Resolved a:03Jhancock.wm replaced dimm from spare stock. dimm appears healthy and pings.
[16:49:37] <logmsgbot>	 !log cdanis@cumin1004 conftool action : set/pooled=true; selector: name=codfw,dnsdisc=mw-web-next-ro
[16:50:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1011.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[16:50:31] <jinxer-wm>	 RESOLVED: RedisInstanceDown: Redis instance down arclamp2001:9121 redis_arclamp - https://wikitech.wikimedia.org/wiki/Redis#Cluster_redis_arclamp - https://grafana.wikimedia.org/d/000000174/redis?orgId=1&var-site=codfw&var-job=redis_arclamp&var-instance=arclamp2001:9121 - https://alerts.wikimedia.org/?q=alertname%3DRedisInstanceDown
[16:51:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[16:54:47] <logmsgbot>	 !log ebernhardson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
[16:54:49] <logmsgbot>	 !log ebernhardson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
[16:55:42] <logmsgbot>	 !log ebernhardson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
[16:55:47] <logmsgbot>	 !log ebernhardson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
[16:59:42] <wikibugs>	 (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, September 24 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployca" [extensions/OATHAuth] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344388 (https://phabricator.wikimedia.org/T438593) (owner: 10Catrope)
[16:59:52] <wikibugs>	 (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, September 24 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployca" [extensions/OATHAuth] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1344389 (https://phabricator.wikimedia.org/T438593) (owner: 10Catrope)
[17:00:04] <jouncebot>	 bd808: Time to snap out of that daydream and deploy Cloud Services/Technical Documentation weekly deploy (Toolhub, Developer portal, Striker). (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260924T1700).
[17:00:05] <jouncebot>	 Deploy window MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260924T1700)
[17:01:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1018.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1012.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[17:01:08] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1012.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[17:02:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[17:02:08] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[17:02:12] <icinga-wm>	 PROBLEM - Postgres Replication Lag on puppetdb2003 is CRITICAL: POSTGRES_HOT_STANDBY_DELAY CRITICAL: DB puppetdb (host:localhost) 314252816 and 6 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[17:03:12] <icinga-wm>	 RECOVERY - Postgres Replication Lag on puppetdb2003 is OK: POSTGRES_HOT_STANDBY_DELAY OK: DB puppetdb (host:localhost) 35824 and 0 seconds https://wikitech.wikimedia.org/wiki/Postgres%23Monitoring
[17:03:39] <wikibugs>	 (03PS2) 10Ebernhardson: semantic ssd test: Drop to 3 masters, fix affinity [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344715 (https://phabricator.wikimedia.org/T438058)
[17:03:39] <wikibugs>	 (03CR) 10Ebernhardson: semantic ssd test: Drop to 3 masters, fix affinity (031 comment) [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344715 (https://phabricator.wikimedia.org/T438058) (owner: 10Ebernhardson)
[17:04:12] <wikibugs>	 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users and deployment for lwatson - https://phabricator.wikimedia.org/T438926#12361145 (10HSwan-WMF) Manager approval here!
[17:05:45] <logmsgbot>	 !log cdobbins@cumin1004 conftool action : set/pooled=yes; selector: name=ncredir1001.*
[17:09:56] <wikibugs>	 (03CR) 10Ladsgroup: "I'm totally pro trying this out but as one of the maintainers of thumbor, I'd like to know what's wrong with it. Also we added envoy to th" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344653 (https://phabricator.wikimedia.org/T420223) (owner: 10Blake)
[17:10:23] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1006.eqiad.wmnet
[17:14:12] <wikibugs>	 (03PS2) 10Lerickson: Bump wdqs-proxy 0.9.0 -> 0.10.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344301 (https://phabricator.wikimedia.org/T438789)
[17:14:44] <wikibugs>	 (03PS3) 10Lerickson: Bump wdqs-proxy 0.9.0 -> 0.11.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344301 (https://phabricator.wikimedia.org/T438789)
[17:15:08] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1021.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[17:16:29] <wikibugs>	 (03CR) 10Dzahn: [C:03+2] traffserver/cache: add attribution.wikimedia.org map and default caching [puppet] - 10https://gerrit.wikimedia.org/r/1341338 (https://phabricator.wikimedia.org/T437635) (owner: 10Dzahn)
[17:17:02] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1006.eqiad.wmnet
[17:17:03] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1006.eqiad.wmnet
[17:17:08] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[17:17:08] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1007.eqiad.wmnet
[17:17:24] <logmsgbot>	 !log cdanis@cumin1004 conftool action : set/pooled=true; selector: name=codfw,dnsdisc=urldownloader
[17:17:42] <wikibugs>	 (03CR) 10Dzahn: [C:03+1] "ah!:) simple and effective" [puppet] - 10https://gerrit.wikimedia.org/r/1344598 (https://phabricator.wikimedia.org/T439104) (owner: 10Arnaudb)
[17:17:43] <wikibugs>	 (03PS1) 10Hnowlan: services_proxy: drop eventgate timeout to 11s [puppet] - 10https://gerrit.wikimedia.org/r/1344739 (https://phabricator.wikimedia.org/T438896)
[17:17:50] <wikibugs>	 (03CR) 10Dzahn: [C:03+2] gerrit: rotate httpd logs of all vhosts [puppet] - 10https://gerrit.wikimedia.org/r/1344598 (https://phabricator.wikimedia.org/T439104) (owner: 10Arnaudb)
[17:18:31] <logmsgbot>	 !log cdanis@cumin1004 conftool action : set/pooled=true; selector: name=codfw,dnsdisc=shellbox.*
[17:19:18] <logmsgbot>	 !log cdanis@cumin1004 conftool action : set/pooled=true; selector: name=codfw,dnsdisc=thumbor
[17:20:10] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1018.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1019.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[17:22:10] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[17:22:23] <logmsgbot>	 !log cdobbins@cumin1004 START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie
[17:23:08] <wikibugs>	 (03PS1) 10Matthias Mullie: Image Browsing carousel: taps outside the preview dialog should close it [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344740 (https://phabricator.wikimedia.org/T439006)
[17:23:40] <jinxer-wm>	 FIRING: [2x] KubernetesRsyslogDown: rsyslog on wikikube-worker2170:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues  - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown
[17:25:10] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1019.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[17:25:17] <jinxer-wm>	 RESOLVED: ProbeDown: Service wdqs1012:443 has failed probes (http_wdqs_main_external_search_sparql_endpoint_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#wdqs1012:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[17:26:10] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[17:26:57] <wikibugs>	 (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, September 24 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployca" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344329 (https://phabricator.wikimedia.org/T438998) (owner: 10Jgiannelos)
[17:29:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[17:29:45] <jinxer-wm>	 FIRING: WidespreadPuppetFailure: Puppet has failed in eqsin - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure
[17:30:10] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1018.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1012.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[17:30:24] <logmsgbot>	 !log cdanis@cumin1004 conftool action : set/pooled=true; selector: name=codfw,dnsdisc=apertium|schema|termbox|proton|cxserver
[17:30:42] <wikibugs>	 (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, September 24 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployca" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344740 (https://phabricator.wikimedia.org/T439006) (owner: 10Matthias Mullie)
[17:31:26] <jinxer-wm>	 FIRING: [3x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[17:31:33] <logmsgbot>	 dzahn@cumin2003 reimage (PID 278761) is awaiting input
[17:32:02] <wikibugs>	 (03CR) 10Dzahn: [C:03+1] "compiled it and noticed on gerrit2002 (but not gerrit1003) it changes the owner from gerrit to root:" [puppet] - 10https://gerrit.wikimedia.org/r/1344369 (owner: 10Hashar)
[17:32:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[17:32:10] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[17:32:22] <logmsgbot>	 !log cdanis@cumin1004 conftool action : set/pooled=true; selector: name=codfw,dnsdisc=citoid|zotero
[17:32:30] <wikibugs>	 (03CR) 10RLazarus: [C:03+1] tcpircbot: replace newlines with spaces in PRIVMSG [puppet] - 10https://gerrit.wikimedia.org/r/1342704 (https://phabricator.wikimedia.org/T438269) (owner: 10Tiziano Fogli)
[17:33:32] <wikibugs>	 (03CR) 10Dzahn: [C:03+1] "oh well, it's just a bit confusing output. it should say the file is removed. instead it shows it as if the file is there just empty and o" [puppet] - 10https://gerrit.wikimedia.org/r/1344369 (owner: 10Hashar)
[17:34:02] <logmsgbot>	 !log dzahn@cumin2003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie
[17:37:28] <logmsgbot>	 !log brett@cumin1004 START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_magru and not P{cp7001.magru.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
[17:37:50] <logmsgbot>	 !log brett@cumin1004 START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_magru and not P{cp7011.magru.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
[17:38:04] <logmsgbot>	 !log dzahn@cumin2003 START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie
[17:38:19] <wikibugs>	 10ops-eqiad, 06SRE, 06Collaboration-Services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12361291 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by dzahn@cumin2003 for host zuul1005.eqiad.wmnet with OS tr...
[17:40:59] <wikibugs>	 (03CR) 10Dzahn: [C:03+2] gerrit: make replication.config absence obvious [puppet] - 10https://gerrit.wikimedia.org/r/1344369 (owner: 10Hashar)
[17:42:02] <logmsgbot>	 !log cdanis@cumin1004 conftool action : set/pooled=true; selector: name=codfw,dnsdisc=config-master|device-analytics|echostore|helm-charts|k8s-ingress-wikikube-ro|linkrecommendation|mathoid|restbase|restbase-async|rest-gateway-ro|mobileapps|mwdebug.*|push-notifications|recommendation-api|releases|wikifeeds
[17:44:22] <wikibugs>	 (03CR) 10Dzahn: [C:03+2] "I did one "systemctl start logrotate" after the change was applied on gerrit2003 to verify it's not broken." [puppet] - 10https://gerrit.wikimedia.org/r/1344598 (https://phabricator.wikimedia.org/T439104) (owner: 10Arnaudb)
[17:44:51] <logmsgbot>	 !log cdanis@cumin1004 conftool action : set/pooled=true; selector: name=codfw,dnsdisc=swift
[17:47:11] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1007.eqiad.wmnet
[17:47:16] <logmsgbot>	 !log cdanis@cumin1004 conftool action : set/pooled=true; selector: name=codfw,dnsdisc=mwdebug.*
[17:47:52] <logmsgbot>	 !log cdanis@cumin1004 conftool action : set/pooled=true; selector: name=codfw,dnsdisc=apus
[17:49:21] <logmsgbot>	 !log cdanis@cumin1004 conftool action : set/pooled=true; selector: name=codfw,dnsdisc=mw-.*-ro
[17:50:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[17:50:10] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[17:51:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[17:51:10] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[17:52:07] <logmsgbot>	 !log swfrench@cumin1004 conftool action : set/pooled=false; selector: dnsdisc=mwdebug.*,name=codfw
[17:52:14] <logmsgbot>	 !log cdanis@cumin1004 conftool action : set/pooled=false; selector: name=codfw,dnsdisc=mwdebug.*
[17:54:04] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1007.eqiad.wmnet
[17:54:05] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1007.eqiad.wmnet
[17:54:11] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1008.eqiad.wmnet
[17:54:43] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1008.eqiad.wmnet
[17:54:45] <jinxer-wm>	 RESOLVED: WidespreadPuppetFailure: Puppet has failed in eqsin - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure
[17:55:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[17:55:10] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[17:56:07] <wikibugs>	 10SRE-SLO, 06Data-Engineering (Q1 FS26/27 July 1st - September 30th): page_change SLO windows - https://phabricator.wikimedia.org/T438054#12361311 (10Ahoelzl) a:03APizzata-WMF
[17:57:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[17:57:09] <wikibugs>	 (03CR) 10Dzahn: [C:03+2] "verified noop on replica and spare (2002,1003) and main (1003) as well" [puppet] - 10https://gerrit.wikimedia.org/r/1344369 (owner: 10Hashar)
[17:57:10] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[17:57:45] <wikibugs>	 (03CR) 10Dzahn: [C:03+1] gerrit: hold GerritDiskSpaceExhaustionIncoming for 30m [alerts] - 10https://gerrit.wikimedia.org/r/1344595 (https://phabricator.wikimedia.org/T439039) (owner: 10Arnaudb)
[17:58:59] <logmsgbot>	 !log hnowlan@cumin1004 END (FAIL) - Cookbook sre.switchdc.mediawiki.00-optional-warmup-caches (exit_code=99) for datacenter switchover from eqiad to codfw
[17:59:20] <jinxer-wm>	 FIRING: CirrusSearchMoreLikeLatencyTooHigh: CirrusSearch more_like 95th percentiles latency is too high (mw@codfw to eqiad) - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://grafana.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchMoreLikeLatencyTooHigh
[17:59:38] <wikibugs>	 (03CR) 10Lerickson: "Self +2 because the image is a better version of the one that was +1'd yesterday, and my usual +1'ers are done for the day today. I deploy" [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344301 (https://phabricator.wikimedia.org/T438789) (owner: 10Lerickson)
[17:59:40] <logmsgbot>	 !log cdanis@cumin1004 START - Cookbook sre.dns.admin DNS admin: pool codfw [reason: no reason specified, no task ID specified]
[17:59:42] <wikibugs>	 (03CR) 10Lerickson: [C:03+2] Bump wdqs-proxy 0.9.0 -> 0.11.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344301 (https://phabricator.wikimedia.org/T438789) (owner: 10Lerickson)
[17:59:47] <logmsgbot>	 !log cdanis@cumin1004 END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool codfw [reason: no reason specified, no task ID specified]
[18:00:05] <jouncebot>	 jeena and jnuche: I seem to be stuck in Groundhog week. Sigh. Time for (yet another) MediaWiki train - Utc-7 Version deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260924T1800).
[18:00:10] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[18:00:19] <wikibugs>	 (03CR) 10Dzahn: [C:03+2] scap-master-sync: Sync releases only if the target directory exists [puppet] - 10https://gerrit.wikimedia.org/r/1344336 (https://phabricator.wikimedia.org/T435393) (owner: 10Ahmon Dancy)
[18:00:39] <jeena>	 Standing by for go-ahead to deploy train
[18:00:56] <cdanis>	 jeena: thx! just a few more minutes
[18:01:06] <jeena>	 👍
[18:01:22] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1008.eqiad.wmnet
[18:01:23] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1008.eqiad.wmnet
[18:01:28] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1009.eqiad.wmnet
[18:02:00] <logmsgbot>	 !log ebernhardson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
[18:02:00] <wikibugs>	 (03Merged) 10jenkins-bot: Bump wdqs-proxy 0.9.0 -> 0.11.0 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344301 (https://phabricator.wikimedia.org/T438789) (owner: 10Lerickson)
[18:02:01] <logmsgbot>	 !log ebernhardson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
[18:02:10] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[18:04:20] <jinxer-wm>	 FIRING: CirrusSearchCompletionLatencyTooHigh: CirrusSearch comp_suggest 95th percentiles latency is too high (mw@codfw to dnsdisc) - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://grafana.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchCompletionLatencyTooHigh
[18:04:26] <wikibugs>	 (03PS2) 10Kimberly Sarabia: ReaderExperiments: add dewiki and svwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344716 (https://phabricator.wikimedia.org/T438072) (owner: 10Marco Fossati)
[18:04:42] <logmsgbot>	 !log cdobbins@cumin1004 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie
[18:04:45] <wikibugs>	 (03CR) 10Dzahn: "@abran@wikimedia.org @hashar@free.fr Speaking just about gerrit. I agree with the general statement that we don't need to monitor for the " [puppet] - 10https://gerrit.wikimedia.org/r/1344226 (https://phabricator.wikimedia.org/T357099) (owner: 10Hnowlan)
[18:04:54] <logmsgbot>	 !log taavi@deploy1003 Unlocked for deployment [ALL REPOSITORIES]: locked for re-pooling codfw for read traffic, contact SRE for equestions (duration: 109m 23s)
[18:05:10] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[18:05:16] <cdanis>	 jeena: 🚀 
[18:06:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[18:06:23] <jeena>	 thanks cdanis !
[18:06:27] <logmsgbot>	 !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
[18:06:48] <logmsgbot>	 !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
[18:06:56] <wikibugs>	 (03PS1) 10TrainBranchBot: group1 to 1.47.0-wmf.21 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344747 (https://phabricator.wikimedia.org/T438217)
[18:06:58] <wikibugs>	 (03CR) 10TrainBranchBot: [C:03+2] "Initiated by jhuneidi@deploy1003" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344747 (https://phabricator.wikimedia.org/T438217) (owner: 10TrainBranchBot)
[18:07:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[18:07:17] <wikibugs>	 (03CR) 10Dzahn: "for mailman and vrts I am not so sure. if those processes do not run if we will be notified in other ways. Wondering if @aokoth@wikimedia." [puppet] - 10https://gerrit.wikimedia.org/r/1344226 (https://phabricator.wikimedia.org/T357099) (owner: 10Hnowlan)
[18:07:52] <wikibugs>	 (03Merged) 10jenkins-bot: group1 to 1.47.0-wmf.21 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344747 (https://phabricator.wikimedia.org/T438217) (owner: 10TrainBranchBot)
[18:08:16] <logmsgbot>	 !log lerickson@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
[18:09:20] <jinxer-wm>	 FIRING: [2x] CirrusSearchMoreLikeLatencyTooHigh: CirrusSearch more_like 95th percentiles latency is too high (mw@codfw to eqiad) - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://grafana.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchMoreLikeLatencyTooHigh
[18:09:26] <logmsgbot>	 !log lerickson@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
[18:09:31] <jinxer-wm>	 FIRING: [2x] CirrusSearchCompletionLatencyTooHigh: CirrusSearch comp_suggest 95th percentiles latency is too high (mw@codfw to dnsdisc) - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://grafana.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchCompletionLatencyTooHigh
[18:10:08] <wikibugs>	 06SRE, 06Commons, 10MediaWiki-File-management, 06Traffic, and 2 others: Varnish serving outdated version at original/non-thumb URL of overwritten file upload - https://phabricator.wikimedia.org/T425216#12361336 (10Verdy_p) Still not solved, and signaled with more details on https://commons.wikimedia.org/wi...
[18:10:08] <logmsgbot>	 !log lerickson@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
[18:11:06] <logmsgbot>	 !log brett@cumin1004 START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_ulsfo - 7.1.1-2~bpo13+wmf3 ()
[18:11:10] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[18:11:16] <logmsgbot>	 !log brett@cumin1004 START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_ulsfo - 7.1.1-2~bpo13+wmf3 ()
[18:11:29] <logmsgbot>	 !log lerickson@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
[18:12:44] <logmsgbot>	 !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
[18:13:08] <logmsgbot>	 !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
[18:14:15] <wikibugs>	 (03CR) 10Dzahn: "re: gearman TCP check. This looks ok. Though on the "Ipv4" only part I am not sure. When I try from alert1002 to that port I can connect o" [puppet] - 10https://gerrit.wikimedia.org/r/1343955 (https://phabricator.wikimedia.org/T384939) (owner: 10Hnowlan)
[18:15:26] <wikibugs>	 (03CR) 10Dzahn: "re: zuul-merge proc check. As far as I remember this was also about checking there are exactly 2 and not more than 2 or fewer than 2 proce" [puppet] - 10https://gerrit.wikimedia.org/r/1343955 (https://phabricator.wikimedia.org/T384939) (owner: 10Hnowlan)
[18:15:48] <ryankemper>	 !log [WDQS] Preparing to repool codfw WDQS shortly; it's been operating single DC so this second DC should restore proper service availability
[18:15:49] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[18:16:50] <logmsgbot>	 !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
[18:17:17] <logmsgbot>	 !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
[18:18:21] <logmsgbot>	 !log lerickson@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
[18:18:53] <logmsgbot>	 !log lerickson@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
[18:19:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1021.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[18:19:13] <logmsgbot>	 !log lerickson@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
[18:19:20] <logmsgbot>	 !log jhuneidi@deploy1003 rebuilt and synchronized wikiversions files: group1 to 1.47.0-wmf.21  refs T438217
[18:19:20] <jinxer-wm>	 FIRING: [2x] CirrusSearchMoreLikeLatencyTooHigh: CirrusSearch more_like 95th percentiles latency is too high (mw@codfw to eqiad) - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://grafana.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchMoreLikeLatencyTooHigh
[18:19:23] <stashbot>	 T438217: 1.47.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T438217
[18:19:34] <logmsgbot>	 !log lerickson@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
[18:20:10] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1016.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[18:20:41] <logmsgbot>	 !log ryankemper@cumin2003 START - Cookbook sre.discovery.service-route pool wdqs-main in codfw: maintenance
[18:20:58] <wikibugs>	 (03CR) 10Lerickson: [C:03+2] Enable wdqs-proxy metrics on port 9100. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344104 (https://phabricator.wikimedia.org/T438789) (owner: 10Lerickson)
[18:21:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[18:21:10] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[18:23:30] <wikibugs>	 (03Merged) 10jenkins-bot: Enable wdqs-proxy metrics on port 9100. [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344104 (https://phabricator.wikimedia.org/T438789) (owner: 10Lerickson)
[18:24:20] <jinxer-wm>	 FIRING: [2x] CirrusSearchCompletionLatencyTooHigh: CirrusSearch comp_suggest 95th percentiles latency is too high (mw@codfw to dnsdisc) - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://grafana.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchCompletionLatencyTooHigh
[18:25:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1011.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[18:25:32] <logmsgbot>	 !log cdobbins@cumin1004 START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie
[18:25:46] <logmsgbot>	 !log ryankemper@cumin2003 END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wdqs-main in codfw: maintenance
[18:26:06] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[18:27:29] <wikibugs>	 (03CR) 10Matthias Mullie: [C:03+1] ReaderExperiments: add dewiki and svwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344716 (https://phabricator.wikimedia.org/T438072) (owner: 10Marco Fossati)
[18:28:01] <logmsgbot>	 !log ryankemper@cumin2003 START - Cookbook sre.discovery.service-route pool wdqs-scholarly in codfw: maintenance
[18:29:38] <wikibugs>	 (03CR) 10Dzahn: [C:03+1] "if you can please join #wikimedia-observability and ask about this" [puppet] - 10https://gerrit.wikimedia.org/r/1341351 (https://phabricator.wikimedia.org/T435393) (owner: 10Southparkfan)
[18:29:52] <logmsgbot>	 dzahn@cumin2003 reimage (PID 290398) is awaiting input
[18:29:59] <logmsgbot>	 !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs-next: apply
[18:30:15] <logmsgbot>	 !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs-next: apply
[18:30:44] <wikibugs>	 (03PS1) 10Ebernhardson: Revert "cirrus: Send more_like traffic to eqiad" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344750
[18:31:06] <wikibugs>	 (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, September 24 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployca" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344750 (owner: 10Ebernhardson)
[18:31:22] <icinga-wm>	 RECOVERY - Check unit status of mwscript-cleanup on deploy2002 is OK: OK: Status of the systemd unit mwscript-cleanup https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
[18:31:35] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1009.eqiad.wmnet
[18:31:40] <wikibugs>	 (03PS1) 10Matthias Mullie: Cap the dialog viewport [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344752 (https://phabricator.wikimedia.org/T439007)
[18:33:07] <logmsgbot>	 !log ryankemper@cumin2003 END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wdqs-scholarly in codfw: maintenance
[18:33:21] <logmsgbot>	 !log lerickson@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
[18:33:35] <logmsgbot>	 !log ryankemper@cumin2003 START - Cookbook sre.discovery.service-route pool wcqs in codfw: maintenance
[18:34:01] <wikibugs>	 (03PS1) 10Daniel Kinzler: REST: restore PageContentHelper::checkAccess (fix live breakage) [core] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344753
[18:34:32] <logmsgbot>	 !log lerickson@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
[18:34:59] <logmsgbot>	 !log lerickson@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs-next: apply
[18:36:27] <logmsgbot>	 !log lerickson@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs-next: apply
[18:37:17] <logmsgbot>	 !log herron@puppetserver1001 conftool action : set/pooled=true; selector: dnsdisc=thanos-web.*,name=codfw
[18:38:40] <logmsgbot>	 !log ryankemper@cumin2003 END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wcqs in codfw: maintenance
[18:39:05] <logmsgbot>	 !log ryankemper@cumin2003 START - Cookbook sre.discovery.service-route pool wdqs-internal-main in codfw: maintenance
[18:39:17] <logmsgbot>	 !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
[18:39:33] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1009.eqiad.wmnet
[18:39:34] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1009.eqiad.wmnet
[18:39:39] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1010.eqiad.wmnet
[18:39:46] <logmsgbot>	 !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
[18:39:58] <logmsgbot>	 !log herron@puppetserver1001 conftool action : set/pooled=true; selector: dnsdisc=thanos-query,name=codfw
[18:40:16] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1010.eqiad.wmnet
[18:42:19] <logmsgbot>	 !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/services/wdqs: apply
[18:42:42] <logmsgbot>	 !log lerickson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/services/wdqs: apply
[18:42:52] <logmsgbot>	 !log herron@puppetserver1001 conftool action : set/pooled=true; selector: dnsdisc=thanos-swift,name=codfw
[18:43:13] <wikibugs>	 (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Thursday, September 24 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployca" [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344752 (https://phabricator.wikimedia.org/T439007) (owner: 10Matthias Mullie)
[18:44:10] <logmsgbot>	 !log ryankemper@cumin2003 END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wdqs-internal-main in codfw: maintenance
[18:44:20] <jinxer-wm>	 RESOLVED: CirrusSearchCompletionLatencyTooHigh: CirrusSearch comp_suggest 95th percentiles latency is too high (mw@codfw to dnsdisc) - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://grafana.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchCompletionLatencyTooHigh
[18:44:21] <logmsgbot>	 !log ryankemper@cumin2003 START - Cookbook sre.discovery.service-route pool wdqs-internal-scholarly in codfw: maintenance
[18:45:23] <jinxer-wm>	 FIRING: GnmiInterfaceCountersDrop: ...
[18:45:23] <jinxer-wm>	 asw1-b13-drmrs is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?orgId=1&from=now-24h&to=now&var-site=%24__all&var-instance=asw1-b13-drmrs:9804&viewPanel=panel-19 - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop
[18:47:12] <wikibugs>	 (03CR) 10TrainBranchBot: [C:03+2] "Approved by jhuneidi@deploy1003 using scap backport" [core] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344753 (owner: 10Daniel Kinzler)
[18:48:39] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1010.eqiad.wmnet
[18:48:40] <jinxer-wm>	 FIRING: [2x] KubernetesRsyslogDown: rsyslog on wikikube-worker2170:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues  - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown
[18:48:40] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1010.eqiad.wmnet
[18:48:45] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1011.eqiad.wmnet
[18:49:18] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1011.eqiad.wmnet
[18:49:27] <logmsgbot>	 !log ryankemper@cumin2003 END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool wdqs-internal-scholarly in codfw: maintenance
[18:52:05] <wikibugs>	 (03Merged) 10jenkins-bot: REST: restore PageContentHelper::checkAccess (fix live breakage) [core] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344753 (owner: 10Daniel Kinzler)
[18:52:30] <logmsgbot>	 !log jhuneidi@deploy1003 Started scap sync-world: Backport for [[gerrit:1344753|REST: restore PageContentHelper::checkAccess (fix live breakage)]]
[18:55:17] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1011.eqiad.wmnet
[18:55:18] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1011.eqiad.wmnet
[18:55:23] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1012.eqiad.wmnet
[18:56:50] <logmsgbot>	 !log jhuneidi@deploy1003 daniel, jhuneidi: Backport for [[gerrit:1344753|REST: restore PageContentHelper::checkAccess (fix live breakage)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
[18:57:28] <logmsgbot>	 !log jhuneidi@deploy1003 daniel, jhuneidi: Continuing with deployment
[19:02:06] <icinga-wm>	 PROBLEM - Host wikikube-worker2280 is DOWN: PING CRITICAL - Packet loss = 100%
[19:02:45] <logmsgbot>	 !log jhuneidi@deploy1003 Finished scap sync-world: Backport for [[gerrit:1344753|REST: restore PageContentHelper::checkAccess (fix live breakage)]] (duration: 10m 15s)
[19:03:20] <jinxer-wm>	 FIRING: CirrusSearchCompletionLatencyTooHigh: CirrusSearch comp_suggest 95th percentiles latency is too high (mw@codfw to dnsdisc) - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://grafana.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchCompletionLatencyTooHigh
[19:03:40] <jinxer-wm>	 RESOLVED: [2x] KubernetesRsyslogDown: rsyslog on wikikube-worker2170:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues  - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown
[19:04:22] <ryankemper>	 not sure why cirrus compl altency is firing, it's not pooled yet. might just be noise because it's not serving so only a few bad latencies could trigger, but taking a look
[19:05:21] <cdanis>	 ryankemper: that'
[19:05:23] <jinxer-wm>	 FIRING: [23x] GnmiInterfaceCountersDrop: asw1-b13-drmrs is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop  - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop
[19:05:23] <cdanis>	 oops
[19:05:37] <ryankemper>	 oh i might be thinking about it wrong, it's probably firing because it's routing to eqiad rn
[19:07:18] <cdanis>	 is it?  I think codfw is getting traffic now, from the dashboard
[19:07:27] <cdanis>	 but probably all their caches are cold and needs to warm up
[19:07:53] <logmsgbot>	 !log cdobbins@cumin1004 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie
[19:07:58] <cdanis>	 oh wait
[19:08:03] <cdanis>	 I see
[19:08:06] <cdanis>	 that's *mw* being repooled in codfw
[19:08:08] <cdanis>	 so yeah that's plausible
[19:08:33] <jinxer-wm>	 FIRING: KubernetesCalicoDown: wikikube-worker2280.codfw.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations - https://grafana.wikimedia.org/d/G8zPL7-Wz/?var-dc=codfw%20prometheus%2Fk8s&var-instance=wikikube-worker2280.codfw.wmnet - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown
[19:10:23] <jinxer-wm>	 FIRING: [23x] GnmiInterfaceCountersDrop: asw1-b13-drmrs is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop  - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop
[19:11:48] <logmsgbot>	 !log cdanis@cumin1004 conftool action : set/pooled=true; selector: name=codfw,dnsdisc=(kartotherian|tegola-vector-tiles)
[19:11:51] <ryankemper>	 !log [Cirrus] Repooling codfw, chi first followed by the small clusters
[19:11:52] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[19:12:10] <logmsgbot>	 !log ryankemper@cumin2003 START - Cookbook sre.discovery.service-route pool search in codfw: maintenance
[19:12:40] <wikibugs>	 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users and deployment for lwatson - https://phabricator.wikimedia.org/T438926#12361554 (10Dzahn) a:03thcipriani
[19:13:20] <jinxer-wm>	 RESOLVED: CirrusSearchCompletionLatencyTooHigh: CirrusSearch comp_suggest 95th percentiles latency is too high (mw@codfw to dnsdisc) - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://grafana.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchCompletionLatencyTooHigh
[19:13:44] <logmsgbot>	 !log cdobbins@cumin1004 START - Cookbook sre.hosts.reimage for host ncredir5004.eqsin.wmnet with OS trixie
[19:17:10] <logmsgbot>	 !log cdanis@cumin1004 conftool action : set/ttl=300; selector: name=codfw
[19:17:16] <logmsgbot>	 !log ryankemper@cumin2003 END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) pool search in codfw: maintenance
[19:17:50] <ryankemper>	 !log [Cirrus] codfw chi (big cluster) repooled; metrics are already improving, I see poolcounter rejections dropping significantly
[19:17:51] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[19:18:11] <logmsgbot>	 !log ryankemper@cumin2003 START - Cookbook sre.discovery.service-route pool search-psi in codfw: maintenance
[19:18:47] <logmsgbot>	 !log dzahn@cumin2003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host zuul1005.eqiad.wmnet with OS trixie
[19:19:01] <wikibugs>	 10ops-eqiad, 06SRE, 06Collaboration-Services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12361581 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by dzahn@cumin2003 for host zuul1005.eqiad.wmnet with OS trixie...
[19:19:26] <wikibugs>	 (03CR) 10Matthias Mullie: "This one can be abandoned - has been rolled into https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1344716, which @ksarabia@w" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344048 (owner: 10Eric Gardner)
[19:20:18] <logmsgbot>	 !log ebernhardson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
[19:20:21] <logmsgbot>	 !log ebernhardson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
[19:23:14] <logmsgbot>	 !log ryankemper@cumin2003 END (FAIL) - Cookbook sre.discovery.service-route (exit_code=99) pool search-psi in codfw: maintenance
[19:24:15] <ryankemper>	 !log [Cirrus] `dns.resolver.NoAnswer: The DNS response does not contain an answer to the question: search-psi.svc.eqiad.wmnet` checking briefly if this is real failure or just some TTL wonkiness
[19:24:16] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[19:24:20] <jinxer-wm>	 RESOLVED: CirrusSearchMoreLikeLatencyTooHigh: CirrusSearch more_like 95th percentiles latency is too high (mw@codfw to eqiad) - https://wikitech.wikimedia.org/wiki/Search#Health/Activity_Monitoring - https://grafana.wikimedia.org/d/dc04b9f2-b8d5-4ab6-9482-5d9a75728951/elasticsearch-percentiles?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DCirrusSearchMoreLikeLatencyTooHigh
[19:25:27] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1012.eqiad.wmnet
[19:26:40] <ryankemper>	 !log [Cirrus] nevermind, that's just the cookbook assuming the DNS record should exist, which it doesn't because chi/psi/omega all share `search.svc.$DC.wmnet`
[19:26:41] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[19:27:52] <logmsgbot>	 !log cdanis@cumin1004 conftool action : set/pooled=true; selector: name=codfw,dnsdisc=k8s-ingress-aux-ro
[19:28:27] <logmsgbot>	 !log ryankemper@cumin2003 START - Cookbook sre.discovery.service-route pool search-omega in codfw: maintenance
[19:29:24] <logmsgbot>	 !log ebernhardson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
[19:29:25] <logmsgbot>	 !log ebernhardson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
[19:30:38] <logmsgbot>	 dzahn@cumin2003 reimage (PID 312115) is awaiting input
[19:31:24] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1012.eqiad.wmnet
[19:31:26] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1012.eqiad.wmnet
[19:31:32] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1013.eqiad.wmnet
[19:32:21] <wikibugs>	 (03CR) 10BCornwall: [V:03+1 C:03+1] "tested working on clouddumps" [puppet] - 10https://gerrit.wikimedia.org/r/1333244 (https://phabricator.wikimedia.org/T433672) (owner: 10CDobbins)
[19:33:29] <logmsgbot>	 !log ryankemper@cumin2003 END (FAIL) - Cookbook sre.discovery.service-route (exit_code=99) pool search-omega in codfw: maintenance
[19:33:42] <ryankemper>	 (failure expected, pool worked fine)
[19:34:12] <logmsgbot>	 !log ebernhardson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
[19:34:15] <logmsgbot>	 !log ebernhardson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
[19:36:11] <ryankemper>	 !log [Cirrus] All cirrus pools are serving again. Actively monitoring while the system returns to equilibrium, but all initial indications are that things are as they should be
[19:36:12] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[19:37:57] <wikibugs>	 (03PS11) 10CDobbins: prometheus: fix prometheus-ferm-mss.py [puppet] - 10https://gerrit.wikimedia.org/r/1333244 (https://phabricator.wikimedia.org/T433672)
[19:38:15] <jeena>	 cdanis: would you like me to wait a bit longer to go to group 2 or should I go ahead now?
[19:39:58] <wikibugs>	 (03CR) 10CDobbins: prometheus: fix prometheus-ferm-mss.py (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/1333244 (https://phabricator.wikimedia.org/T433672) (owner: 10CDobbins)
[19:40:23] <jinxer-wm>	 FIRING: [2x] GnmiInterfaceCountersDrop: asw1-b12-drmrs is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop  - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop
[19:46:15] <logmsgbot>	 !log vriley@cumin1004 START - Cookbook sre.hosts.reimage for host zuul1005.eqiad.wmnet with OS trixie
[19:46:33] <wikibugs>	 (03PS1) 10JHathaway: puppetdb: increase threshold for postgres checks [puppet] - 10https://gerrit.wikimedia.org/r/1344770 (https://phabricator.wikimedia.org/T439010)
[19:46:35] <wikibugs>	 10ops-eqiad, 06SRE, 06Collaboration-Services, 06DC-Ops, 13Patch-For-Review: Repurpose ganeti102[3456] for Zuul migration - https://phabricator.wikimedia.org/T427353#12361653 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by vriley@cumin1004 for host zuul1005.eqiad.wmnet with OS t...
[19:46:47] <wikibugs>	 (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1344770 (https://phabricator.wikimedia.org/T439010) (owner: 10JHathaway)
[19:49:21] <wikibugs>	 (03CR) 10BCornwall: [V:03+1 C:03+1] prometheus: fix prometheus-ferm-mss.py [puppet] - 10https://gerrit.wikimedia.org/r/1333244 (https://phabricator.wikimedia.org/T433672) (owner: 10CDobbins)
[19:49:42] <wikibugs>	 (03CR) 10CI reject: [V:04-1] puppetdb: increase threshold for postgres checks [puppet] - 10https://gerrit.wikimedia.org/r/1344770 (https://phabricator.wikimedia.org/T439010) (owner: 10JHathaway)
[19:50:27] <logmsgbot>	 !log brett@cumin1004 END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_magru and not P{cp7011.magru.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
[19:52:11] <wikibugs>	 (03PS2) 10Andrew Bogott: Revert "Puppetize keystone-uwsgi.ini" [puppet] - 10https://gerrit.wikimedia.org/r/1344333 (https://phabricator.wikimedia.org/T421911)
[19:52:12] <wikibugs>	 (03PS14) 10Andrew Bogott: openstack apis: Add additional uwsgi ini file to support logstash logging [puppet] - 10https://gerrit.wikimedia.org/r/1344325 (https://phabricator.wikimedia.org/T421911)
[19:53:04] <wikibugs>	 (03PS1) 10VolkerE: feat(AccountSetup), styles: Increase thumbnail size in ReadingRecommendations [extensions/GrowthExperiments] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344771 (https://phabricator.wikimedia.org/T436332)
[19:53:30] <wikibugs>	 (03CR) 10CI reject: [V:04-1] openstack apis: Add additional uwsgi ini file to support logstash logging [puppet] - 10https://gerrit.wikimedia.org/r/1344325 (https://phabricator.wikimedia.org/T421911) (owner: 10Andrew Bogott)
[19:56:05] <logmsgbot>	 !log cdobbins@cumin1004 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ncredir5004.eqsin.wmnet with OS trixie
[19:56:42] <logmsgbot>	 !log cjming@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen: apply
[19:57:02] <icinga-wm>	 PROBLEM - grafana-rw.wikimedia.org tls expiry on grafana1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration
[19:57:08] <icinga-wm>	 PROBLEM - grafana-next-rw.wikimedia.org requires authentication on grafana1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration
[19:57:21] <logmsgbot>	 !log cjming@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen: apply
[19:57:32] <icinga-wm>	 PROBLEM - grafana-next-rw.wikimedia.org tls expiry on grafana1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration
[19:57:34] <icinga-wm>	 PROBLEM - SSH on grafana1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring
[19:57:34] <icinga-wm>	 PROBLEM - grafana-rw.wikimedia.org requires authentication on grafana1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration
[19:57:57] <wikibugs>	 (03PS15) 10Andrew Bogott: openstack apis: Add additional uwsgi ini file to support logstash logging [puppet] - 10https://gerrit.wikimedia.org/r/1344325 (https://phabricator.wikimedia.org/T421911)
[19:58:04] <icinga-wm>	 RECOVERY - grafana-next-rw.wikimedia.org requires authentication on grafana1002 is OK: HTTP OK: Status line output matched HTTP/1.1 302 - 602 bytes in 7.322 second response time https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration
[19:58:22] <icinga-wm>	 RECOVERY - grafana-next-rw.wikimedia.org tls expiry on grafana1002 is OK: OK - Certificate grafana.discovery.wmnet will expire on Sat 10 Oct 2026 09:13:00 AM GMT +0000. https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration
[19:58:23] <wikibugs>	 (03PS1) 10VolkerE: feat(AccountSetup), styles: Apply correct background and button styles [extensions/GrowthExperiments] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344772 (https://phabricator.wikimedia.org/T436332)
[19:58:24] <icinga-wm>	 RECOVERY - SSH on grafana1002 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u10 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring
[19:58:24] <icinga-wm>	 RECOVERY - grafana-rw.wikimedia.org requires authentication on grafana1002 is OK: HTTP OK: Status line output matched HTTP/1.1 302 - 592 bytes in 0.063 second response time https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration
[19:58:52] <icinga-wm>	 RECOVERY - grafana-rw.wikimedia.org tls expiry on grafana1002 is OK: OK - Certificate grafana.discovery.wmnet will expire on Sat 10 Oct 2026 09:13:00 AM GMT +0000. https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration
[19:59:05] <wikibugs>	 (03CR) 10CI reject: [V:04-1] openstack apis: Add additional uwsgi ini file to support logstash logging [puppet] - 10https://gerrit.wikimedia.org/r/1344325 (https://phabricator.wikimedia.org/T421911) (owner: 10Andrew Bogott)
[19:59:32] <wikibugs>	 (03PS2) 10JHathaway: puppetdb: increase threshold for postgres checks [puppet] - 10https://gerrit.wikimedia.org/r/1344770 (https://phabricator.wikimedia.org/T439010)
[19:59:37] <wikibugs>	 (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1344770 (https://phabricator.wikimedia.org/T439010) (owner: 10JHathaway)
[20:00:05] <jouncebot>	 RoanKattouw, urbanecm, TheresNoTime, kindrobot, and cjming: How many deployers does it take to do UTC late backport window deploy? (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260924T2000).
[20:00:05] <jouncebot>	 kemayo, kimberly_sarabia, cjming, RoanKattouw, arlolra, and ebernhardson: A patch you scheduled for UTC late backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker.
[20:00:21] <RoanKattouw>	 I can deploy but I will need a few minutes before I'm ready
[20:00:32] <kimberly_sarabia>	 Hello. I'm here
[20:00:34] <Kemayo>	 I can deploy mind just fine.
[20:00:37] <Kemayo>	 *mine
[20:00:42] <RoanKattouw>	 Please go ahead Kemayo 
[20:00:42] <ebernhardson>	 \o
[20:00:45] <cjming>	  o/
[20:00:46] <arlolra>	 \o
[20:01:05] <wikibugs>	 (03CR) 10TrainBranchBot: [C:03+2] "Approved by kemayo@deploy1003 using scap backport" [extensions/VisualEditor] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344714 (https://phabricator.wikimedia.org/T438916) (owner: 10DLynch)
[20:01:34] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1013.eqiad.wmnet
[20:02:43] <wikibugs>	 (03PS16) 10Andrew Bogott: openstack apis: Add additional uwsgi ini file to support logstash logging [puppet] - 10https://gerrit.wikimedia.org/r/1344325 (https://phabricator.wikimedia.org/T421911)
[20:03:00] <wikibugs>	 (03CR) 10JHathaway: "ready for review" [puppet] - 10https://gerrit.wikimedia.org/r/1344770 (https://phabricator.wikimedia.org/T439010) (owner: 10JHathaway)
[20:03:32] <cdanis>	 jeena: I think it's fine after backports are done, or, if any of the backports are made redundant by rolling .21 to group2
[20:03:40] <cdanis>	 (maybe all of them are?)
[20:03:53] <jeena>	 let's see
[20:04:11] <jeena>	 I'm fine waiting for backports to go ahead
[20:04:28] <cdanis>	 sure, also trying to save the backporters some time
[20:06:21] <jeena>	 I think there's only one going to wmf.20 that we could skip
[20:07:03] <Kemayo>	 RoanKattouw's the only one with a .20 listed, at least.
[20:07:55] <wikibugs>	 (03Merged) 10jenkins-bot: EditCheck: add some statsv tracking of check/suggestion actions [extensions/VisualEditor] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344714 (https://phabricator.wikimedia.org/T438916) (owner: 10DLynch)
[20:08:18] <logmsgbot>	 !log kemayo@deploy1003 Started scap sync-world: Backport for [[gerrit:1344714|EditCheck: add some statsv tracking of check/suggestion actions (T438916)]]
[20:08:21] <stashbot>	 T438916: Make Suggestion Mode usage data publicly available - https://phabricator.wikimedia.org/T438916
[20:09:36] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1013.eqiad.wmnet
[20:09:37] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1013.eqiad.wmnet
[20:09:43] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1014.eqiad.wmnet
[20:09:51] <wikibugs>	 (03CR) 10Andrew Bogott: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1344325 (https://phabricator.wikimedia.org/T421911) (owner: 10Andrew Bogott)
[20:12:03] <wikibugs>	 (03PS1) 10SBassett: Content Security Policy: add stricter report-only policy [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344774 (https://phabricator.wikimedia.org/T419612)
[20:12:25] <RoanKattouw>	 jeena: I can skip my .20 backport if you're about to do the train after this
[20:12:31] <logmsgbot>	 !log kemayo@deploy1003 kemayo: Backport for [[gerrit:1344714|EditCheck: add some statsv tracking of check/suggestion actions (T438916)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
[20:12:36] <wikibugs>	 (03PS17) 10Andrew Bogott: openstack apis: Add additional uwsgi ini file to support logstash logging [puppet] - 10https://gerrit.wikimedia.org/r/1344325 (https://phabricator.wikimedia.org/T421911)
[20:12:45] <jeena>	 RoanKattouw: I am, so you don't need to backport it
[20:12:51] <RoanKattouw>	 Great
[20:13:18] <wikibugs>	 (03Abandoned) 10Catrope: HookHandler: Guard against recovery code expiry being null [extensions/OATHAuth] (wmf/1.47.0-wmf.20) - 10https://gerrit.wikimedia.org/r/1344389 (https://phabricator.wikimedia.org/T438593) (owner: 10Catrope)
[20:13:28] <wikibugs>	 (03PS18) 10Andrew Bogott: openstack apis: Add additional uwsgi ini file to support logstash logging [puppet] - 10https://gerrit.wikimedia.org/r/1344325 (https://phabricator.wikimedia.org/T421911)
[20:14:25] <logmsgbot>	 !log kemayo@deploy1003 kemayo: Continuing with deployment
[20:15:23] <Kemayo>	 (Sorry, testing took slightly longer than I meant it to, because I realized that the test article I had loaded up actually didn't trigger what I needed.)
[20:15:28] <wikibugs>	 (03CR) 10Andrew Bogott: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1344325 (https://phabricator.wikimedia.org/T421911) (owner: 10Andrew Bogott)
[20:17:29] <wikibugs>	 (03PS19) 10Andrew Bogott: openstack apis: Add additional uwsgi ini file to support logstash logging [puppet] - 10https://gerrit.wikimedia.org/r/1344325 (https://phabricator.wikimedia.org/T421911)
[20:19:42] <logmsgbot>	 !log kemayo@deploy1003 Finished scap sync-world: Backport for [[gerrit:1344714|EditCheck: add some statsv tracking of check/suggestion actions (T438916)]] (duration: 11m 23s)
[20:19:45] <stashbot>	 T438916: Make Suggestion Mode usage data publicly available - https://phabricator.wikimedia.org/T438916
[20:19:53] <Kemayo>	 RoanKattouw: Okay, I'm done.
[20:22:00] <wikibugs>	 (03PS1) 10PipelineBot: mobileapps: pipeline bot promote [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344776
[20:22:03] <RoanKattouw>	 Great, I'll get started in a minute
[20:25:33] <logmsgbot>	 !log brett@cumin1004 END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on A:cp-upload_ulsfo - 7.1.1-2~bpo13+wmf3 ()
[20:25:34] <logmsgbot>	 !log brett@cumin1004 END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on A:cp-text_ulsfo - 7.1.1-2~bpo13+wmf3 ()
[20:27:18] <icinga-wm>	 PROBLEM - Check unit status of httpbb_kubernetes_mw-web_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-web_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
[20:30:52] <wikibugs>	 (03CR) 10VolkerE: [C:03+1] feat(AccountSetup), styles: Apply correct background and button styles [extensions/GrowthExperiments] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344772 (https://phabricator.wikimedia.org/T436332) (owner: 10VolkerE)
[20:31:08] <wikibugs>	 (03CR) 10VolkerE: [C:03+1] feat(AccountSetup), styles: Increase thumbnail size in ReadingRecommendations [extensions/GrowthExperiments] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344771 (https://phabricator.wikimedia.org/T436332) (owner: 10VolkerE)
[20:34:14] <logmsgbot>	 !log ebernhardson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
[20:34:18] <logmsgbot>	 !log ebernhardson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
[20:35:48] <jinxer-wm>	 FIRING: PuppetFailure: Puppet has failed on ml-serve1015:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
[20:35:55] <wikibugs>	 (03CR) 10TrainBranchBot: [C:03+2] "Approved by catrope@deploy1003 using scap backport" [extensions/OATHAuth] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344388 (https://phabricator.wikimedia.org/T438593) (owner: 10Catrope)
[20:36:07] <logmsgbot>	 vriley@cumin1004 reimage (PID 690552) is awaiting input
[20:38:45] <wikibugs>	 (03Merged) 10jenkins-bot: HookHandler: Guard against recovery code expiry being null [extensions/OATHAuth] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344388 (https://phabricator.wikimedia.org/T438593) (owner: 10Catrope)
[20:39:08] <logmsgbot>	 !log catrope@deploy1003 Started scap sync-world: Backport for [[gerrit:1344388|HookHandler: Guard against recovery code expiry being null (T438593)]]
[20:39:12] <stashbot>	 T438593: TypeError: MediaWiki\Message\Message::dateParam(): Argument #1 ($date) must be of type string, null given, called in /srv/mediawiki/php-1.47.0-wmf.20/includes/Language/Message/Message.php on line 729 - https://phabricator.wikimedia.org/T438593
[20:39:49] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1014.eqiad.wmnet
[20:40:40] <jinxer-wm>	 FIRING: ProbeDown: Service titan1002:443 has failed probes (http_thanos_wikimedia_org_ip6) - https://wikitech.wikimedia.org/wiki/Runbook#titan1002:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[20:41:45] <wikibugs>	 (03CR) 10SBassett: [C:04-1] "Hold for config deployment." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344774 (https://phabricator.wikimedia.org/T419612) (owner: 10SBassett)
[20:42:40] <wikibugs>	 (03CR) 10ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, September 29 UTC late backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344774 (https://phabricator.wikimedia.org/T419612) (owner: 10SBassett)
[20:43:21] <logmsgbot>	 !log catrope@deploy1003 catrope: Backport for [[gerrit:1344388|HookHandler: Guard against recovery code expiry being null (T438593)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
[20:43:27] <jinxer-wm>	 RESOLVED: [2x] ProbeDown: Service titan1002:443 has failed probes (http_thanos_wikimedia_org_ip6)   - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[20:43:37] <logmsgbot>	 !log brett@cumin1004 START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P{cp404[1-6].ulsfo.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
[20:44:11] <logmsgbot>	 !log catrope@deploy1003 catrope: Continuing with deployment
[20:44:31] <logmsgbot>	 !log brett@cumin1004 START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P{cp405[1-2].ulsfo.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
[20:46:06] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[20:46:10] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1021.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[20:47:27] <sbassett>	 jouncebot: now
[20:47:27] <jouncebot>	 For the next 0 hour(s) and 12 minute(s): UTC late backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260924T2000)
[20:47:55] <logmsgbot>	 !log lerickson@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
[20:48:19] <logmsgbot>	 !log lerickson@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
[20:48:44] <logmsgbot>	 !log lerickson@deploy1003 helmfile [dse-k8s-codfw] START helmfile.d/services/wdqs: apply
[20:48:47] <sbassett>	 Hey all - wanted to see how late backport window was going.  We have a sec patch we’d like to get deployed within the next hour or so, if possible.
[20:49:05] <logmsgbot>	 !log lerickson@deploy1003 helmfile [dse-k8s-codfw] DONE helmfile.d/services/wdqs: apply
[20:49:07] <ebernhardson>	 there is more, but i'll drop my patch from it. I can't stay too late
[20:49:28] <logmsgbot>	 !log catrope@deploy1003 Finished scap sync-world: Backport for [[gerrit:1344388|HookHandler: Guard against recovery code expiry being null (T438593)]] (duration: 10m 19s)
[20:49:36] <stashbot>	 T438593: TypeError: MediaWiki\Message\Message::dateParam(): Argument #1 ($date) must be of type string, null given, called in /srv/mediawiki/php-1.47.0-wmf.20/includes/Language/Message/Message.php on line 729 - https://phabricator.wikimedia.org/T438593
[20:49:58] <RoanKattouw>	 ebernhardson: I can do yours right now if that works for you
[20:50:00] <cjming>	 i need to get mine out today but i can do it later if that's helpful
[20:50:17] <ebernhardson>	 RoanKattouw: sure, mine is pretty safe
[20:50:24] <RoanKattouw>	 And I'll do some tricks to speed things up
[20:50:40] <kimberly_sarabia>	 i dont mind going last. im flexible
[20:50:40] <arlolra>	 mine can be combined
[20:50:44] <ebernhardson>	 thanks!
[20:50:46] <logmsgbot>	 !log brett@cumin1004 END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on P{cp405[1-2].ulsfo.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
[20:51:08] <RoanKattouw>	 OK great, I'll do Erik's and Arlo's together then 
[20:51:24] <wikibugs>	 (03CR) 10TrainBranchBot: [C:03+2] "Approved by catrope@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344750 (owner: 10Ebernhardson)
[20:51:24] <wikibugs>	 (03CR) 10TrainBranchBot: [C:03+2] "Approved by catrope@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344329 (https://phabricator.wikimedia.org/T438998) (owner: 10Jgiannelos)
[20:51:51] <wikibugs>	 (03CR) 10Catrope: [C:03+2] Image Browsing carousel: taps outside the preview dialog should close it [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344740 (https://phabricator.wikimedia.org/T439006) (owner: 10Matthias Mullie)
[20:51:57] <wikibugs>	 (03CR) 10Catrope: [C:03+2] Cap the dialog viewport [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344752 (https://phabricator.wikimedia.org/T439007) (owner: 10Matthias Mullie)
[20:52:07] <sbassett>	 RoanKattouw: I’m just trying to get this https://gerrit.wikimedia.org/r/1344102 deployed via spiderpig asap.  If it can ride along with yours, that’d be convenient...
[20:52:17] <wikibugs>	 (03PS4) 10Eric Gardner: ReaderExperiments: Set the preferred-sources debug flag on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344049 (https://phabricator.wikimedia.org/T436692)
[20:52:27] <wikibugs>	 (03PS3) 10Eric Gardner: ReaderExperiments: Drop the stale ShareHighlight config var [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344050 (https://phabricator.wikimedia.org/T424764)
[20:52:46] <RoanKattouw>	 sbassett: Sounds good, I'll have that ride along with the next group
[20:52:47] <wikibugs>	 (03Merged) 10jenkins-bot: Revert "cirrus: Send more_like traffic to eqiad" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344750 (owner: 10Ebernhardson)
[20:52:51] <wikibugs>	 (03Merged) 10jenkins-bot: prv: Enable parsoid rendering for 5 wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344329 (https://phabricator.wikimedia.org/T438998) (owner: 10Jgiannelos)
[20:53:15] <logmsgbot>	 !log catrope@deploy1003 Started scap sync-world: Backport for [[gerrit:1344750|Revert "cirrus: Send more_like traffic to eqiad"]], [[gerrit:1344329|prv: Enable parsoid rendering for 5 wikis (T438998)]]
[20:53:18] <stashbot>	 T438998: Rollout parsoid read views on wikisource - week of 19 Sep - https://phabricator.wikimedia.org/T438998
[20:54:32] <RoanKattouw>	 cjming: Is there another change that's going to be backported that uses the withContext helper introduced by your change? Maybe I'm missing something, but your change on its own doesn't seem to do anything
[20:55:05] <logmsgbot>	 !log ebernhardson@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
[20:55:07] <logmsgbot>	 !log ebernhardson@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search-ssd: apply
[20:55:19] <cjming>	 RoanKattouw: ya it's a dependency for https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikimediaEvents/+/1344594
[20:55:50] <wikibugs>	 (03Merged) 10jenkins-bot: Image Browsing carousel: taps outside the preview dialog should close it [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344740 (https://phabricator.wikimedia.org/T439006) (owner: 10Matthias Mullie)
[20:55:52] <wikibugs>	 (03Merged) 10jenkins-bot: Cap the dialog viewport [extensions/MultimediaViewer] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344752 (https://phabricator.wikimedia.org/T439007) (owner: 10Matthias Mullie)
[20:55:56] <cjming>	 which i think ^^ will be backported Monday maybe?
[20:56:14] <sbassett>	 RoanKattouw: Thanks, fixing some CI issues, should be merging to master shortly.
[20:56:31] <RoanKattouw>	 cjming: It's not currently scheduled for backport but that's good enough for me
[20:56:41] <wikibugs>	 (03CR) 10Catrope: [C:03+2] ext.wikimediaEvents.testKitchen: Add withContext helper [extensions/WikimediaEvents] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344406 (https://phabricator.wikimedia.org/T438898) (owner: 10Clare Ming)
[20:56:42] <cjming>	 ty ty
[20:56:49] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1014.eqiad.wmnet
[20:56:50] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1014.eqiad.wmnet
[20:56:56] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1015.eqiad.wmnet
[20:57:13] <logmsgbot>	 !log brett@cumin1004 START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P{cp405[1-2].ulsfo.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
[20:57:24] <logmsgbot>	 !log catrope@deploy1003 catrope, ebernhardson, jgiannelos: Backport for [[gerrit:1344750|Revert "cirrus: Send more_like traffic to eqiad"]], [[gerrit:1344329|prv: Enable parsoid rendering for 5 wikis (T438998)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
[20:57:57] <RoanKattouw>	 arlolra: ebernhardson: Please test your changes (to the extent possible) and lmk when I can continue
[20:58:22] <ebernhardson>	 RoanKattouw: looks good
[20:58:29] <arlolra>	 RoanKattouw: lgtm
[20:58:33] <jinxer-wm>	 FIRING: [2x] KubernetesCalicoDown: dse-k8s-worker1014.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations  - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown
[20:58:34] <wikibugs>	 (03PS2) 10SBassett: Content Security Policy: add stricter report-only policy [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344774 (https://phabricator.wikimedia.org/T419612)
[20:58:35] <logmsgbot>	 !log catrope@deploy1003 catrope, ebernhardson, jgiannelos: Continuing with deployment
[21:00:05] <jouncebot>	 Deploy window Readers deployment window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260924T2100)
[21:02:07] <wikibugs>	 (03Merged) 10jenkins-bot: ext.wikimediaEvents.testKitchen: Add withContext helper [extensions/WikimediaEvents] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344406 (https://phabricator.wikimedia.org/T438898) (owner: 10Clare Ming)
[21:02:29] <logmsgbot>	 !log brett@puppetserver1001 conftool action : set/pooled=yes; selector: name=cp4041.*
[21:03:55] <logmsgbot>	 !log brett@puppetserver1001 conftool action : set/pooled=yes; selector: name=cp4051.*
[21:04:00] <logmsgbot>	 !log catrope@deploy1003 Finished scap sync-world: Backport for [[gerrit:1344750|Revert "cirrus: Send more_like traffic to eqiad"]], [[gerrit:1344329|prv: Enable parsoid rendering for 5 wikis (T438998)]] (duration: 10m 45s)
[21:04:03] <stashbot>	 T438998: Rollout parsoid read views on wikisource - week of 19 Sep - https://phabricator.wikimedia.org/T438998
[21:05:06] <RoanKattouw>	 sbassett's patch is unfortunately still in CI purgatory (I just tried to fix the phpcs issues) so I will do kimberly_sarabia's set of patches first, then after that I will do cjming's and hopefully sbassett's at the same time
[21:05:28] <kimberly_sarabia>	 copy that
[21:05:30] <wikibugs>	 (03CR) 10Dbrant: [C:03+1] docroot/wikimedia.org: update assetlink.json for credential sharing on root domain [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1335409 (https://phabricator.wikimedia.org/T427929) (owner: 10Jasmine)
[21:05:40] <wikibugs>	 (03CR) 10TrainBranchBot: [C:03+2] "Approved by catrope@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344716 (https://phabricator.wikimedia.org/T438072) (owner: 10Marco Fossati)
[21:05:43] <arlolra>	 Thanks Roan
[21:05:45] <sbassett>	 RoanKattouw: Randall should have the CI issues addressed in the latest PS, tests are running now, so far so good.
[21:05:46] * cjming bows to RoanKattouw
[21:06:13] <RoanKattouw>	 sbassett: I saw one of the interim test results showing they weren't addressed, so I addressed them myself, but now CI needs to rerun
[21:06:44] <wikibugs>	 (03Merged) 10jenkins-bot: ReaderExperiments: add dewiki and svwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344716 (https://phabricator.wikimedia.org/T438072) (owner: 10Marco Fossati)
[21:06:58] <EricGardner>	 Jon, Jan, and I are going to deploy some patches in the readers window once current work is done
[21:07:16] <jeena>	 I do need to deploy to group2 at some point
[21:07:36] <RoanKattouw>	 OK never mind, Spiderpig won't let me separate kimberly_sarabia's from cjming's because I already merged cjming's, so I'm doing those together
[21:07:37] <wikibugs>	 (03PS3) 10Jdlrobson: Enable ReadingList CTA on Minerva for our test wikis (inc beta cluster) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344118 (https://phabricator.wikimedia.org/T438779)
[21:08:05] <kimberly_sarabia>	 np. ty
[21:08:31] <logmsgbot>	 !log catrope@deploy1003 Started scap sync-world: Backport for [[gerrit:1344406|ext.wikimediaEvents.testKitchen: Add withContext helper (T438898)]], [[gerrit:1344716|ReaderExperiments: add dewiki and svwiki (T438072)]], [[gerrit:1344740|Image Browsing carousel: taps outside the preview dialog should close it (T439006)]], [[gerrit:1344752|Cap the dialog viewport (T439007)]]
[21:08:39] <stashbot>	 T438898: Add wrapper helpers to add more context to analytics events - https://phabricator.wikimedia.org/T438898
[21:08:40] <stashbot>	 T438072: [Image Browsing] Launch Image Carousel A/B/C/D/E experiment - https://phabricator.wikimedia.org/T438072
[21:08:40] <stashbot>	 T439006: Image Browsing: tapping outside the carousel preview dialog should dismiss it - https://phabricator.wikimedia.org/T439006
[21:08:41] <stashbot>	 T439007: [Image Browsing] too small preview dialog - https://phabricator.wikimedia.org/T439007
[21:09:02] <rzl>	 thanks all for your patience with us having to block out deployments for so long while we were firefighting yesterday, I know that's what caused the congestion now
[21:09:15] <EricGardner>	 Kim are you deploying something now too? We can wait or try to do it at the same time
[21:09:27] <RoanKattouw>	 Maybe we could: 1) let the deploy I just started finish; 2) deploy sbassett's patch if it's ready at that time (otherwise skip it), 3) advance the train to wmf.21, 4) do EricGardner's deploys?
[21:09:38] <RoanKattouw>	 Kim's patches are already in flight
[21:09:58] <jeena>	 EricGardner: are you okay waiting for me to finish up train after this?
[21:10:13] <jeena>	 rzl thanks for all your hard work!
[21:11:11] <EricGardner>	 jeena that's fine, maybe we can come back in 20-30 min in that case
[21:11:23] <EricGardner>	 we just have config changes
[21:11:33] <jeena>	 okay, thank you!!
[21:12:38] <logmsgbot>	 !log catrope@deploy1003 cjming, mfossati, catrope, mlitn: Backport for [[gerrit:1344406|ext.wikimediaEvents.testKitchen: Add withContext helper (T438898)]], [[gerrit:1344716|ReaderExperiments: add dewiki and svwiki (T438072)]], [[gerrit:1344740|Image Browsing carousel: taps outside the preview dialog should close it (T439006)]], [[gerrit:1344752|Cap the dialog viewport (T439007)]] synced to the testservers (see https://wi
[21:12:38] <EricGardner>	 We'll check back at 2:30 and see where things are at
[21:12:39] <logmsgbot>	 kitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
[21:13:06] <jeena>	 👍
[21:13:12] <RoanKattouw>	 kimberly_sarabia, cjming : Please test your changes (to the extent possible)
[21:13:34] <RoanKattouw>	 (Kim's seem like they should definitely be testable, Claire's I'm not as sure, maybe in the JS console)
[21:13:40] <RoanKattouw>	 *Clare's
[21:13:48] <cjming>	 ya testing now
[21:14:51] <wikibugs>	 (03PS1) 10Catrope: Catch newline character in UserMailer to prevent it from allowing bad actors to create an additional header [core] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344795 (https://phabricator.wikimedia.org/T434545)
[21:15:00] <wikibugs>	 (03CR) 10Catrope: [C:03+2] Catch newline character in UserMailer to prevent it from allowing bad actors to create an additional header [core] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344795 (https://phabricator.wikimedia.org/T434545) (owner: 10Catrope)
[21:18:05] <cjming>	 all good - please sync - ty!!
[21:19:26] <kimberly_sarabia>	 LGTM
[21:22:02] <wikibugs>	 (03PS1) 10JHathaway: postfix: upgrade module [puppet] - 10https://gerrit.wikimedia.org/r/1344797 (https://phabricator.wikimedia.org/T438912)
[21:22:17] <wikibugs>	 (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1344797 (https://phabricator.wikimedia.org/T438912) (owner: 10JHathaway)
[21:22:42] <logmsgbot>	 !log catrope@deploy1003 cjming, mfossati, catrope, mlitn: Continuing with deployment
[21:24:09] <wikibugs>	 (03Merged) 10jenkins-bot: Catch newline character in UserMailer to prevent it from allowing bad actors to create an additional header [core] (wmf/1.47.0-wmf.21) - 10https://gerrit.wikimedia.org/r/1344795 (https://phabricator.wikimedia.org/T434545) (owner: 10Catrope)
[21:26:57] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1015.eqiad.wmnet
[21:27:18] <icinga-wm>	 RECOVERY - Check unit status of httpbb_kubernetes_mw-web_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-web_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
[21:27:58] <logmsgbot>	 !log catrope@deploy1003 Finished scap sync-world: Backport for [[gerrit:1344406|ext.wikimediaEvents.testKitchen: Add withContext helper (T438898)]], [[gerrit:1344716|ReaderExperiments: add dewiki and svwiki (T438072)]], [[gerrit:1344740|Image Browsing carousel: taps outside the preview dialog should close it (T439006)]], [[gerrit:1344752|Cap the dialog viewport (T439007)]] (duration: 19m 27s)
[21:28:08] <stashbot>	 T438898: Add wrapper helpers to add more context to analytics events - https://phabricator.wikimedia.org/T438898
[21:28:08] <stashbot>	 T438072: [Image Browsing] Launch Image Carousel A/B/C/D/E experiment - https://phabricator.wikimedia.org/T438072
[21:28:09] <stashbot>	 T439006: Image Browsing: tapping outside the carousel preview dialog should dismiss it - https://phabricator.wikimedia.org/T439006
[21:28:09] <stashbot>	 T439007: [Image Browsing] too small preview dialog - https://phabricator.wikimedia.org/T439007
[21:28:40] <wikibugs>	 (03CR) 10Scott French: [C:03+1] "Thanks, Luca!" [puppet] - 10https://gerrit.wikimedia.org/r/1344678 (owner: 10Elukey)
[21:28:43] <wikibugs>	 (03CR) 10JHathaway: [C:03+2] postfix: upgrade module [puppet] - 10https://gerrit.wikimedia.org/r/1344797 (https://phabricator.wikimedia.org/T438912) (owner: 10JHathaway)
[21:28:48] <RoanKattouw>	 Now deploying sbassett's patch, and then it'll be time for jeena to do the train
[21:28:53] <logmsgbot>	 !log catrope@deploy1003 Started scap sync-world: Backport for [[gerrit:1344795|Catch newline character in UserMailer to prevent it from allowing bad actors to create an additional header (T434545)]]
[21:29:10] <jeena>	 Thanks RoanKattouw!
[21:31:26] <jinxer-wm>	 FIRING: [3x] SystemdUnitFailed: docker-reporter-kubernetes-aux_eqiad-images.service on build2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[21:32:11] <EricGardner>	 Would anyone object if we do our readers config deployments at 3pm today (22:00 UTC)? Presumably the train will have left the station by then
[21:32:46] <jeena>	 It doesn't look like there is anything on the deployment calendar until much later
[21:33:05] <logmsgbot>	 !log catrope@deploy1003 catrope: Backport for [[gerrit:1344795|Catch newline character in UserMailer to prevent it from allowing bad actors to create an additional header (T434545)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
[21:33:14] <icinga-wm>	 PROBLEM - Check unit status of httpbb_kubernetes_mw-api-int_hourly on cumin2003 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-api-int_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
[21:34:03] <RoanKattouw>	 sbassett: Any testing you want to do, or should I just continue with the deployment?
[21:34:04] <logmsgbot>	 !log brett@cumin1004 END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on P{cp405[1-2].ulsfo.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
[21:34:57] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1015.eqiad.wmnet
[21:34:58] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1015.eqiad.wmnet
[21:35:04] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1016.eqiad.wmnet
[21:35:33] <wikibugs>	 (03CR) 10Scott French: [C:03+1] "Thanks, Blake! Apologies for the delayed review - this LGTM." [deployment-charts] - 10https://gerrit.wikimedia.org/r/1333837 (https://phabricator.wikimedia.org/T417800) (owner: 10Blake)
[21:35:35] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1016.eqiad.wmnet
[21:37:38] <cjming>	 RoanKattouw: tysm!
[21:38:04] <wikibugs>	 (03CR) 10Scott French: [C:03+1] "Thanks, Moritz! There are still non-k8s PHP workloads (production deployment hosts, and deployment-prep), but this is very clearly dead co" [puppet] - 10https://gerrit.wikimedia.org/r/1342215 (owner: 10Muehlenhoff)
[21:40:40] <RoanKattouw>	 Going to proceed with the deploymnet
[21:40:43] <logmsgbot>	 !log catrope@deploy1003 catrope: Continuing with deployment
[21:41:19] <wikibugs>	 (03CR) 10Scott French: "Ah, thank you for that, Daniel!" [puppet] - 10https://gerrit.wikimedia.org/r/1322828 (https://phabricator.wikimedia.org/T424266) (owner: 10Scott French)
[21:42:08] <wikibugs>	 (03PS1) 10DLynch: ingress: Copy istio 1.0.3 to 1.0.4 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344800 (https://phabricator.wikimedia.org/T436689)
[21:42:10] <wikibugs>	 (03PS1) 10DLynch: ingress.istio: Add useClientProtocol [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344801 (https://phabricator.wikimedia.org/T436689)
[21:42:13] <wikibugs>	 (03PS1) 10DLynch: python-webapp: Update ingress.istio to 1.0.4 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344802 (https://phabricator.wikimedia.org/T436689)
[21:42:15] <wikibugs>	 (03PS1) 10DLynch: editcheck-headless: Forward gRPC as HTTP/2 through the ingress [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344803 (https://phabricator.wikimedia.org/T436689)
[21:42:20] <logmsgbot>	 !log brett@cumin1004 START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P{cp4052.ulsfo.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
[21:42:51] <wikibugs>	 (03PS1) 10DLynch: mesh: Copy service 1.1.0 to 1.1.1 and configuration 1.15.3 to 1.15.4 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344804 (https://phabricator.wikimedia.org/T436689)
[21:42:53] <wikibugs>	 (03PS1) 10DLynch: mesh: Add alpn_protocols and public_port_app_protocol [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344805 (https://phabricator.wikimedia.org/T436689)
[21:42:56] <wikibugs>	 (03PS1) 10DLynch: python-webapp: Update mesh.configuration to 1.15.4 and service to 1.1.1 [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344806 (https://phabricator.wikimedia.org/T436689)
[21:42:58] <wikibugs>	 (03PS1) 10DLynch: editcheck-headless: Use HTTP/2 through the mesh for gRPC [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344807 (https://phabricator.wikimedia.org/T436689)
[21:43:38] <wikibugs>	 (03PS1) 10DLynch: python-webapp: Add grpc_ingress for a plaintext gRPC port [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344808 (https://phabricator.wikimedia.org/T436689)
[21:43:41] <wikibugs>	 (03PS1) 10DLynch: editcheck-headless: Route the ingress to the plaintext gRPC port [deployment-charts] - 10https://gerrit.wikimedia.org/r/1344809 (https://phabricator.wikimedia.org/T436689)
[21:45:24] <jinxer-wm>	 FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[21:45:58] <logmsgbot>	 !log catrope@deploy1003 Finished scap sync-world: Backport for [[gerrit:1344795|Catch newline character in UserMailer to prevent it from allowing bad actors to create an additional header (T434545)]] (duration: 17m 05s)
[21:46:08] <RoanKattouw>	 I'm done
[21:46:10] <RoanKattouw>	 jeena: All yours
[21:46:11] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1016.eqiad.wmnet
[21:46:12] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1016.eqiad.wmnet
[21:46:17] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1018.eqiad.wmnet
[21:46:31] <jeena>	 thank you :)
[21:46:51] <wikibugs>	 (03PS1) 10TrainBranchBot: group2 to 1.47.0-wmf.21 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344810 (https://phabricator.wikimedia.org/T438217)
[21:46:53] <wikibugs>	 (03CR) 10TrainBranchBot: [C:03+2] "Initiated by jhuneidi@deploy1003" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344810 (https://phabricator.wikimedia.org/T438217) (owner: 10TrainBranchBot)
[21:46:55] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1018.eqiad.wmnet
[21:47:52] <wikibugs>	 (03Merged) 10jenkins-bot: group2 to 1.47.0-wmf.21 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344810 (https://phabricator.wikimedia.org/T438217) (owner: 10TrainBranchBot)
[21:48:16] <logmsgbot>	 !log brett@cumin1004 END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P{cp4052.ulsfo.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
[21:48:33] <jinxer-wm>	 FIRING: [2x] KubernetesCalicoDown: dse-k8s-worker1016.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations  - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown
[21:52:18] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[21:52:18] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[21:53:05] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1018.eqiad.wmnet
[21:53:06] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1018.eqiad.wmnet
[21:53:12] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1019.eqiad.wmnet
[21:53:45] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1019.eqiad.wmnet
[21:55:05] <wikibugs>	 (03PS1) 10JHathaway: stdlib: remove has_key usage [puppet] - 10https://gerrit.wikimedia.org/r/1344811 (https://phabricator.wikimedia.org/T438912)
[21:55:18] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1020 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[21:55:32] <wikibugs>	 (03CR) 10JHathaway: "check experimental" [puppet] - 10https://gerrit.wikimedia.org/r/1344811 (https://phabricator.wikimedia.org/T438912) (owner: 10JHathaway)
[21:56:18] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[21:57:22] <logmsgbot>	 !log jhuneidi@deploy1003 rebuilt and synchronized wikiversions files: group2 to 1.47.0-wmf.21  refs T438217
[21:57:25] <stashbot>	 T438217: 1.47.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T438217
[21:57:49] <jeena>	 All done, thanks for waiting EricGardner 
[21:58:03] <EricGardner>	 No prob!
[21:58:24] <logmsgbot>	 !log brett@puppetserver1001 conftool action : set/pooled=yes; selector: name=cp4052.*
[21:59:53] <EricGardner>	 Starting with some readers config patches now
[22:00:07] <wikibugs>	 (03CR) 10TrainBranchBot: [C:03+2] "Approved by egardner@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344048 (owner: 10Eric Gardner)
[22:00:07] <wikibugs>	 (03CR) 10TrainBranchBot: [C:03+2] "Approved by egardner@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344049 (https://phabricator.wikimedia.org/T436692) (owner: 10Eric Gardner)
[22:00:07] <wikibugs>	 (03CR) 10TrainBranchBot: [C:03+2] "Approved by egardner@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344050 (https://phabricator.wikimedia.org/T424764) (owner: 10Eric Gardner)
[22:00:08] <wikibugs>	 (03CR) 10TrainBranchBot: [C:03+2] "Approved by egardner@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344118 (https://phabricator.wikimedia.org/T438779) (owner: 10Jdlrobson)
[22:00:09] <wikibugs>	 (03CR) 10TrainBranchBot: [C:03+2] "Approved by egardner@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1343560 (https://phabricator.wikimedia.org/T437339) (owner: 10VolkerE)
[22:00:14] <icinga-wm>	 PROBLEM - Host wikikube-worker1016 is DOWN: PING CRITICAL - Packet loss = 75%, RTA = 3215.29 ms
[22:00:18] <wikibugs>	 (03CR) 10CI reject: [V:04-1] ReaderExperiments: Enable on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344048 (owner: 10Eric Gardner)
[22:00:19] <wikibugs>	 (03CR) 10CI reject: [V:04-1] ReaderExperiments: Set the preferred-sources debug flag on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344049 (https://phabricator.wikimedia.org/T436692) (owner: 10Eric Gardner)
[22:00:20] <wikibugs>	 (03CR) 10CI reject: [V:04-1] ReaderExperiments: Drop the stale ShareHighlight config var [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344050 (https://phabricator.wikimedia.org/T424764) (owner: 10Eric Gardner)
[22:00:21] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1019.eqiad.wmnet
[22:00:22] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1019.eqiad.wmnet
[22:00:27] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1020.eqiad.wmnet
[22:01:11] <wikibugs>	 (03Merged) 10jenkins-bot: Enable ReadingList CTA on Minerva for our test wikis (inc beta cluster) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344118 (https://phabricator.wikimedia.org/T438779) (owner: 10Jdlrobson)
[22:01:12] <icinga-wm>	 RECOVERY - Host wikikube-worker1016 is UP: PING OK - Packet loss = 0%, RTA = 3.38 ms
[22:01:13] <wikibugs>	 (03Merged) 10jenkins-bot: Revert "Enable Reading Recommendations experiment on test wiki" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1343560 (https://phabricator.wikimedia.org/T437339) (owner: 10VolkerE)
[22:02:44] <logmsgbot>	 !log btullis@cumin1004 END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1020.eqiad.wmnet
[22:04:36] <logmsgbot>	 !log brett@cumin1004 END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-text_magru and not P{cp7001.magru.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
[22:04:55] <wikibugs>	 (03PS4) 10Eric Gardner: ReaderExperiments: Enable on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344048
[22:06:26] <wikibugs>	 (03Abandoned) 10Eric Gardner: ReaderExperiments: Enable on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344048 (owner: 10Eric Gardner)
[22:07:04] <wikibugs>	 (03PS5) 10Eric Gardner: ReaderExperiments: Set the preferred-sources debug flag on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344049 (https://phabricator.wikimedia.org/T436692)
[22:07:21] <wikibugs>	 (03PS4) 10Eric Gardner: ReaderExperiments: Drop the stale ShareHighlight config var [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344050 (https://phabricator.wikimedia.org/T424764)
[22:08:45] <wikibugs>	 (03CR) 10TrainBranchBot: "Approved by egardner@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344049 (https://phabricator.wikimedia.org/T436692) (owner: 10Eric Gardner)
[22:08:45] <wikibugs>	 (03CR) 10TrainBranchBot: "Approved by egardner@deploy1003 using scap backport" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344050 (https://phabricator.wikimedia.org/T424764) (owner: 10Eric Gardner)
[22:09:31] <wikibugs>	 (03Merged) 10jenkins-bot: ReaderExperiments: Set the preferred-sources debug flag on testwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344049 (https://phabricator.wikimedia.org/T436692) (owner: 10Eric Gardner)
[22:09:48] <wikibugs>	 (03Merged) 10jenkins-bot: ReaderExperiments: Drop the stale ShareHighlight config var [mediawiki-config] - 10https://gerrit.wikimedia.org/r/1344050 (https://phabricator.wikimedia.org/T424764) (owner: 10Eric Gardner)
[22:10:02] <logmsgbot>	 !log egardner@deploy1003 Started scap sync-world: Backport for [[gerrit:1344049|ReaderExperiments: Set the preferred-sources debug flag on testwiki (T436692)]], [[gerrit:1344050|ReaderExperiments: Drop the stale ShareHighlight config var (T424764)]], [[gerrit:1344118|Enable ReadingList CTA on Minerva for our test wikis (inc beta cluster) (T438779)]], [[gerrit:1343560|Revert "Enable Reading Recommendations experiment on te
[22:10:02] <logmsgbot>	 st wiki" (T437339)]]
[22:10:09] <stashbot>	 T436692: Implement Preferred Sources experiment - https://phabricator.wikimedia.org/T436692
[22:10:09] <stashbot>	 T424764: [Share Highlights] Launch the experiment - https://phabricator.wikimedia.org/T424764
[22:10:11] <stashbot>	 T438779: Mobile CTA: Turn on the anonymous mobile bookmark experience for all Wikipedia users - https://phabricator.wikimedia.org/T438779
[22:10:11] <stashbot>	 T437339: Recommended reading exp: Instrument user interaction on module - https://phabricator.wikimedia.org/T437339
[22:14:04] <logmsgbot>	 !log egardner@deploy1003 volker-e, egardner, jdlrobson: Backport for [[gerrit:1344049|ReaderExperiments: Set the preferred-sources debug flag on testwiki (T436692)]], [[gerrit:1344050|ReaderExperiments: Drop the stale ShareHighlight config var (T424764)]], [[gerrit:1344118|Enable ReadingList CTA on Minerva for our test wikis (inc beta cluster) (T438779)]], [[gerrit:1343560|Revert "Enable Reading Recommendations experiment
[22:14:04] <logmsgbot>	 on test wiki" (T437339)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
[22:19:34] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1020.eqiad.wmnet
[22:19:35] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1020.eqiad.wmnet
[22:19:40] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1021.eqiad.wmnet
[22:20:40] <jinxer-wm>	 FIRING: [2x] KubernetesCalicoDown: dse-k8s-worker1020.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations  - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown
[22:21:18] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[22:21:44] <logmsgbot>	 !log btullis@cumin1004 END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1021.eqiad.wmnet
[22:22:16] <logmsgbot>	 !log egardner@deploy1003 volker-e, egardner, jdlrobson: Continuing with deployment
[22:25:18] <icinga-wm>	 PROBLEM - PyBal backends health check on lvs1019 is CRITICAL: PYBAL CRITICAL - CRITICAL - wdqs-main_443: Servers wdqs1017.eqiad.wmnet, wdqs1021.eqiad.wmnet, wdqs1020.eqiad.wmnet, wdqs1011.eqiad.wmnet, wdqs1013.eqiad.wmnet, wdqs1014.eqiad.wmnet, wdqs1018.eqiad.wmnet, wdqs1016.eqiad.wmnet, wdqs1019.eqiad.wmnet, wdqs1012.eqiad.wmnet, wdqs1022.eqiad.wmnet are marked down but pooled https://wikitech.wikimedia.org/wiki/PyBal
[22:26:50] <ryankemper>	 ^ with dcs restored this happening means someone needs to be blocked, looking
[22:27:32] <logmsgbot>	 !log egardner@deploy1003 Finished scap sync-world: Backport for [[gerrit:1344049|ReaderExperiments: Set the preferred-sources debug flag on testwiki (T436692)]], [[gerrit:1344050|ReaderExperiments: Drop the stale ShareHighlight config var (T424764)]], [[gerrit:1344118|Enable ReadingList CTA on Minerva for our test wikis (inc beta cluster) (T438779)]], [[gerrit:1343560|Revert "Enable Reading Recommendations experiment on t
[22:27:33] <logmsgbot>	 est wiki" (T437339)]] (duration: 17m 30s)
[22:27:39] <stashbot>	 T436692: Implement Preferred Sources experiment - https://phabricator.wikimedia.org/T436692
[22:27:39] <stashbot>	 T424764: [Share Highlights] Launch the experiment - https://phabricator.wikimedia.org/T424764
[22:27:39] <stashbot>	 T438779: Mobile CTA: Turn on the anonymous mobile bookmark experience for all Wikipedia users - https://phabricator.wikimedia.org/T438779
[22:27:40] <stashbot>	 T437339: Recommended reading exp: Instrument user interaction on module - https://phabricator.wikimedia.org/T437339
[22:28:02] <EricGardner>	 Ok, we are done!
[22:28:50] <ryankemper>	 !log [WDQS] Expanding match in https://requestctl.wikimedia.org/pattern/ua/rocks to test a likely block candidate
[22:28:51] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[22:32:18] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1020 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[22:32:18] <icinga-wm>	 RECOVERY - PyBal backends health check on lvs1019 is OK: PYBAL OK - All pools are healthy https://wikitech.wikimedia.org/wiki/PyBal
[22:33:14] <icinga-wm>	 RECOVERY - Check unit status of httpbb_kubernetes_mw-api-int_hourly on cumin2003 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-api-int_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
[22:41:17] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1021.eqiad.wmnet
[22:41:18] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1021.eqiad.wmnet
[22:41:23] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1022.eqiad.wmnet
[22:43:33] <jinxer-wm>	 FIRING: [2x] KubernetesCalicoDown: dse-k8s-worker1021.eqiad.wmnet is not running calico-node Pod - https://wikitech.wikimedia.org/wiki/Calico#Operations  - https://alerts.wikimedia.org/?q=alertname%3DKubernetesCalicoDown
[23:00:13] <wikibugs>	 10ops-eqiad, 06DC-Ops: Possible Netbox incorrection on device type - https://phabricator.wikimedia.org/T438963#12362203 (10wiki_willy)
[23:00:34] <wikibugs>	 10ops-eqiad, 06DC-Ops: Possible Netbox incorrection on device type - https://phabricator.wikimedia.org/T438963#12362205 (10wiki_willy) Adding ops-eqiad and tagging @VRiley-WMF and @Jclark-ctr
[23:11:41] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1022.eqiad.wmnet
[23:18:44] <icinga-wm>	 PROBLEM - Host lsw1-b7-codfw.mgmt is DOWN: PING CRITICAL - Packet loss = 100%
[23:20:12] <icinga-wm>	 RECOVERY - Host lsw1-b7-codfw.mgmt is UP: PING OK - Packet loss = 0%, RTA = 32.04 ms
[23:20:34] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1022.eqiad.wmnet
[23:20:35] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1022.eqiad.wmnet
[23:20:40] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1023.eqiad.wmnet
[23:25:25] <wikibugs>	 (03PS1) 10Ryan Kemper: wdqs: bump thread restart threshold to 1000 [puppet] - 10https://gerrit.wikimedia.org/r/1344822 (https://phabricator.wikimedia.org/T242453)
[23:27:15] <wikibugs>	 (03CR) 10Ryan Kemper: "check puppet" [puppet] - 10https://gerrit.wikimedia.org/r/1344822 (https://phabricator.wikimedia.org/T242453) (owner: 10Ryan Kemper)
[23:32:16] <logmsgbot>	 !log brett@cumin1004 END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on P{cp404[1-6].ulsfo.wmnet} and A:cp - 7.1.1-2~bpo13+wmf3 ()
[23:38:57] <wikibugs>	 (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1344824
[23:38:57] <wikibugs>	 (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1344824 (owner: 10TrainBranchBot)
[23:40:23] <jinxer-wm>	 FIRING: [2x] GnmiInterfaceCountersDrop: asw1-b12-drmrs is exporting less than half the gNMI interface counters it had 24h ago - https://wikitech.wikimedia.org/wiki/Network_monitoring#GnmiInterfaceCountersDrop  - https://alerts.wikimedia.org/?q=alertname%3DGnmiInterfaceCountersDrop
[23:48:50] <wikibugs>	 (03CR) 10Ryan Kemper: [C:03+2] wdqs: bump thread restart threshold to 1000 [puppet] - 10https://gerrit.wikimedia.org/r/1344822 (https://phabricator.wikimedia.org/T242453) (owner: 10Ryan Kemper)
[23:50:45] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1023.eqiad.wmnet
[23:51:50] <wikibugs>	 (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1344824 (owner: 10TrainBranchBot)
[23:57:24] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1023.eqiad.wmnet
[23:57:25] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1023.eqiad.wmnet
[23:57:30] <logmsgbot>	 !log btullis@cumin1004 START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1024.eqiad.wmnet
[23:58:05] <logmsgbot>	 !log btullis@cumin1004 END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1024.eqiad.wmnet