|
2026-04-17 00:03:09
|
<wikibugs>
|
'SRE-swift-storage, ''MediaWiki-File-management: Stuck-hidden file - https://phabricator.wikimedia.org/T423065#11832147 (''Pppery) Still happening: https://en.wikipedia.org/wiki/User_talk:Pppery#File%3AHambonesMeditations.jpg'
|
|
2026-04-17 00:03:11
|
<logmsgbot>
|
!log jasmine@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
|
|
2026-04-17 00:03:54
|
<logmsgbot>
|
!log jasmine@cumin1003 START - Cookbook sre.hosts.provision for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
|
|
2026-04-17 00:10:34
|
<logmsgbot>
|
!log jasmine@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host wikikube-ctrl2006.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART
|
|
2026-04-17 00:19:08
|
<wikibugs>
|
('CR) ''Dzahn: [C:''+1] "ah yea, I wanted to do the --name thing anyways" [puppet] - ''https://gerrit.wikimedia.org/r/1272961 (https://phabricator.wikimedia.org/T406384) (owner: ''Dduvall)'
|
|
2026-04-17 00:19:20
|
<wikibugs>
|
('CR) ''Dzahn: [C:''+2] zuul: Name service containers and remove them when stopped [puppet] - ''https://gerrit.wikimedia.org/r/1272961 (https://phabricator.wikimedia.org/T406384) (owner: ''Dduvall)'
|
|
2026-04-17 00:21:57
|
<jinxer-wm>
|
FIRING: ProbeDown: Service text:80 has failed probes (http_text_ip6) #page - https://wikitech.wikimedia.org/wiki/Runbook#text:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
|
|
2026-04-17 00:22:36
|
<jasmine_>
|
!incidents
|
|
2026-04-17 00:22:36
|
<sirenbot>
|
7846 (UNACKED) ProbeDown sre (2001:df2:e500:ed1a::1 ip6 text:80 probes/service http_text_ip6 eqsin)
|
|
2026-04-17 00:22:36
|
<sirenbot>
|
7845 (RESOLVED) TransitPeeringTransportOutSaturation network sre (cr1-eqiad:9804 Transit: NTT (234630) {#3475} xe-3/1/6 gnmi eqiad)
|
|
2026-04-17 00:22:37
|
<sirenbot>
|
7844 (RESOLVED) TransitPeeringTransportOutSaturation network sre (cr1-eqiad:9804 Transit: Lumen (442550281) {#3867} xe-3/3/2 gnmi eqiad)
|
|
2026-04-17 00:22:37
|
<sirenbot>
|
7843 (RESOLVED) TransitPeeringTransportOutSaturation network sre (cr1-codfw:9804 Transport: cr4-ulsfo:xe-0/1/1 (Lumen, 442550294) {#12252_12295-1} xe-1/1/1:0 gnmi codfw)
|
|
2026-04-17 00:22:48
|
<jasmine_>
|
!ack 7846
|
|
2026-04-17 00:22:48
|
<sirenbot>
|
7846 (ACKED) ProbeDown sre (2001:df2:e500:ed1a::1 ip6 text:80 probes/service http_text_ip6 eqsin)
|
|
2026-04-17 00:24:17
|
<sukhe>
|
port 80?
|
|
2026-04-17 00:24:20
|
<sukhe>
|
that's interesting
|
|
2026-04-17 00:26:57
|
<jinxer-wm>
|
RESOLVED: ProbeDown: Service text:80 has failed probes (http_text_ip6) #page - https://wikitech.wikimedia.org/wiki/Runbook#text:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
|
|
2026-04-17 00:26:58
|
<wikibugs>
|
('PS2) ''Dzahn: zuul: Provide tenant configuration [puppet] - ''https://gerrit.wikimedia.org/r/1272970 (https://phabricator.wikimedia.org/T406384) (owner: ''Dduvall)'
|
|
2026-04-17 00:27:27
|
<wikibugs>
|
('CR) ''Dzahn: "it wanted a license for the config file:) added one" [puppet] - ''https://gerrit.wikimedia.org/r/1272970 (https://phabricator.wikimedia.org/T406384) (owner: ''Dduvall)'
|
|
2026-04-17 00:32:19
|
<wikibugs>
|
'SRE-swift-storage, ''MediaWiki-File-management: Stuck-hidden file - https://phabricator.wikimedia.org/T423065#11832210 (''Ladsgroup) Can't say for sure but maybe that's because of a missing update in the new file schema? See line 192 on the new side https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1273030/1...'
|
|
2026-04-17 00:33:27
|
<jinxer-wm>
|
FIRING: [4x] SystemdUnitFailed: wmf_auto_restart_prometheus-blazegraph-exporter-wdqs-blazegraph.service on wdqs1012:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
|
|
2026-04-17 00:34:05
|
<wikibugs>
|
('CR) ''Dzahn: [C:''+1] "smart-reconfigure is smart :)" [puppet] - ''https://gerrit.wikimedia.org/r/1272970 (https://phabricator.wikimedia.org/T406384) (owner: ''Dduvall)'
|
|
2026-04-17 00:36:42
|
<wikibugs>
|
('CR) ''Dzahn: [V:''+1 C:''+2] "https://puppet-compiler.wmflabs.org/output/1272970/8435/zuul1001.eqiad.wmnet/index.html"; [puppet] - ''https://gerrit.wikimedia.org/r/1272970 (https://phabricator.wikimedia.org/T406384) (owner: ''Dduvall)'
|
|
2026-04-17 00:41:40
|
<jinxer-wm>
|
FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps2011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
|
|
2026-04-17 00:44:57
|
<jinxer-wm>
|
FIRING: CertAlmostExpired: Certificate for service ssw1-e1-codfw.mgmt.codfw.wmnet:32767 is about to expire - https://wikitech.wikimedia.org/wiki/TLS/Runbook#ssw1-e1-codfw.mgmt.codfw.wmnet:32767 - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired
|
|
2026-04-17 00:47:13
|
<wikibugs>
|
('CR) ''Dzahn: [V:''+1 C:''+2] "it had an issue on the very first puppet run but on second run it looked alright." [puppet] - ''https://gerrit.wikimedia.org/r/1272970 (https://phabricator.wikimedia.org/T406384) (owner: ''Dduvall)'
|
|
2026-04-17 00:47:24
|
<wikibugs>
|
('CR) ''Dzahn: [V:''+1 C:''+2] "please do not worry about "ERROR zuul.GerritConnection.ssh: IsADirectoryError: [Errno 21] Is a directory: '/var/ssh/zuul'" I will fix th" [puppet] - ''https://gerrit.wikimedia.org/r/1272970 (https://phabricator.wikimedia.org/T406384) (owner: ''Dduvall)'
|
|
2026-04-17 00:48:07
|
<jinxer-wm>
|
FIRING: ProbeDown: Service aqs1010-a:9042 has failed probes (tcp_cassandra_a_cql_ip4) - https://wikitech.wikimedia.org/wiki/TLS/Runbook#aqs1010-a:9042 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
|
|
2026-04-17 00:51:32
|
<wikibugs>
|
('CR) ''Dzahn: [C:''+2] "restarted zuul-nodepool on zuul1001 and they all have the names now:" [puppet] - ''https://gerrit.wikimedia.org/r/1272961 (https://phabricator.wikimedia.org/T406384) (owner: ''Dduvall)'
|
|
2026-04-17 00:53:07
|
<jinxer-wm>
|
FIRING: [2x] ProbeDown: Service aqs1010-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
|
|
2026-04-17 00:54:57
|
<jinxer-wm>
|
FIRING: [2x] CertAlmostExpired: Certificate for service lsw1-e1-codfw.mgmt.codfw.wmnet:32767 is about to expire - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired
|
|
2026-04-17 00:57:08
|
<wikibugs>
|
'ops-codfw, ''SRE, ''Data-Persistence, ''DC-Ops: Q4:rack/setup/install 4 new db hosts in codfw - https://phabricator.wikimedia.org/T418911#11832236 (''Jhancock.wm) a:''Jhancock.wm'
|
|
2026-04-17 00:57:56
|
<wikibugs>
|
'ops-codfw, ''SRE, ''Data-Persistence, ''DC-Ops: Q3:rack/setup/install pc202[1-4] - https://phabricator.wikimedia.org/T418907#11832239 (''Jhancock.wm) a:''Jhancock.wm'
|
|
2026-04-17 01:03:52
|
<icinga-wm>
|
PROBLEM - OSPF status on cr2-eqdfw is CRITICAL: OSPFv2: 7/7 UP : OSPFv3: 6/7 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status
|
|
2026-04-17 01:04:52
|
<icinga-wm>
|
RECOVERY - OSPF status on cr2-eqdfw is OK: OSPFv2: 7/7 UP : OSPFv3: 7/7 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status
|
|
2026-04-17 01:09:57
|
<wikibugs>
|
('PS1) ''TrainBranchBot: Branch commit for wmf/next [core] (wmf/next) - ''https://gerrit.wikimedia.org/r/1273066'
|
|
2026-04-17 01:09:57
|
<wikibugs>
|
('CR) ''TrainBranchBot: [C:''+2] Branch commit for wmf/next [core] (wmf/next) - ''https://gerrit.wikimedia.org/r/1273066 (owner: ''TrainBranchBot)'
|
|
2026-04-17 01:15:14
|
<icinga-wm>
|
PROBLEM - OSPF status on cr2-esams is CRITICAL: OSPFv2: 3/3 UP : OSPFv3: 2/3 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status
|
|
2026-04-17 01:16:14
|
<icinga-wm>
|
RECOVERY - OSPF status on cr2-esams is OK: OSPFv2: 3/3 UP : OSPFv3: 3/3 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status
|
|
2026-04-17 01:21:22
|
<wikibugs>
|
('Merged) ''jenkins-bot: Branch commit for wmf/next [core] (wmf/next) - ''https://gerrit.wikimedia.org/r/1273066 (owner: ''TrainBranchBot)'
|
|
2026-04-17 01:44:57
|
<jinxer-wm>
|
FIRING: [3x] CertAlmostExpired: Certificate for service lsw1-e1-codfw.mgmt.codfw.wmnet:32767 is about to expire - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired
|
|
2026-04-17 01:45:35
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1261 (T419635)', diff saved to https://phabricator.wikimedia.org/P91004 and previous config saved to /var/cache/conftool/dbconfig/20260417-014534-fceratto.json
|
|
2026-04-17 01:45:39
|
<stashbot>
|
T419635: Drop il_to column from imagelinks table in wmf production - https://phabricator.wikimedia.org/T419635
|
|
2026-04-17 01:55:43
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1261', diff saved to https://phabricator.wikimedia.org/P91005 and previous config saved to /var/cache/conftool/dbconfig/20260417-015542-fceratto.json
|
|
2026-04-17 01:59:09
|
<wikibugs>
|
'SRE, ''Traffic: Investigate port 80 page in text@esams for Ipv6 - https://phabricator.wikimedia.org/T423667 (''jasmine_) ''NEW'
|
|
2026-04-17 02:01:16
|
<logmsgbot>
|
!log mwpresync@deploy1003 Started scap build-images: Publishing wmf/next image
|
|
2026-04-17 02:03:36
|
<wikibugs>
|
'SRE: Investigate port 80 page in text@esams for Ipv6 - https://phabricator.wikimedia.org/T423667#11832332 (''jasmine_)'
|
|
2026-04-17 02:05:51
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1261', diff saved to https://phabricator.wikimedia.org/P91006 and previous config saved to /var/cache/conftool/dbconfig/20260417-020550-fceratto.json
|
|
2026-04-17 02:07:42
|
<logmsgbot>
|
!log mwpresync@deploy1003 Finished scap build-images: Publishing wmf/next image (duration: 06m 25s)
|
|
2026-04-17 02:09:17
|
<jinxer-wm>
|
FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
|
|
2026-04-17 02:15:59
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1261 (T419635)', diff saved to https://phabricator.wikimedia.org/P91007 and previous config saved to /var/cache/conftool/dbconfig/20260417-021558-fceratto.json
|
|
2026-04-17 02:16:03
|
<stashbot>
|
T419635: Drop il_to column from imagelinks table in wmf production - https://phabricator.wikimedia.org/T419635
|
|
2026-04-17 02:16:16
|
<logmsgbot>
|
!log fceratto@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1262.eqiad.wmnet with reason: Maintenance
|
|
2026-04-17 02:16:25
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Depooling db1262 (T419635)', diff saved to https://phabricator.wikimedia.org/P91008 and previous config saved to /var/cache/conftool/dbconfig/20260417-021624-fceratto.json
|
|
2026-04-17 02:34:17
|
<jinxer-wm>
|
RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
|
|
2026-04-17 03:25:44
|
<jinxer-wm>
|
FIRING: KubernetesDeploymentUnavailableReplicas: ...
|
|
2026-04-17 03:25:44
|
<jinxer-wm>
|
Deployment linkrecommendation-internal in linkrecommendation at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s&var-namespace=linkrecommendation&var-deployment=linkrecommendation-internal - ...
|
|
2026-04-17 03:25:44
|
<jinxer-wm>
|
https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas
|
|
2026-04-17 04:04:03
|
<jinxer-wm>
|
FIRING: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures
|
|
2026-04-17 04:14:55
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1262 (T419635)', diff saved to https://phabricator.wikimedia.org/P91009 and previous config saved to /var/cache/conftool/dbconfig/20260417-041454-fceratto.json
|
|
2026-04-17 04:14:59
|
<stashbot>
|
T419635: Drop il_to column from imagelinks table in wmf production - https://phabricator.wikimedia.org/T419635
|
|
2026-04-17 04:25:03
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1262', diff saved to https://phabricator.wikimedia.org/P91010 and previous config saved to /var/cache/conftool/dbconfig/20260417-042502-fceratto.json
|
|
2026-04-17 04:33:42
|
<jinxer-wm>
|
FIRING: [3x] SystemdUnitFailed: wmf_auto_restart_prometheus-blazegraph-exporter-wdqs-blazegraph.service on wdqs1012:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
|
|
2026-04-17 04:35:10
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1262', diff saved to https://phabricator.wikimedia.org/P91011 and previous config saved to /var/cache/conftool/dbconfig/20260417-043510-fceratto.json
|
|
2026-04-17 04:41:40
|
<jinxer-wm>
|
FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps2011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
|
|
2026-04-17 04:45:19
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1262 (T419635)', diff saved to https://phabricator.wikimedia.org/P91012 and previous config saved to /var/cache/conftool/dbconfig/20260417-044518-fceratto.json
|
|
2026-04-17 04:45:23
|
<stashbot>
|
T419635: Drop il_to column from imagelinks table in wmf production - https://phabricator.wikimedia.org/T419635
|
|
2026-04-17 04:45:36
|
<logmsgbot>
|
!log fceratto@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1263.eqiad.wmnet with reason: Maintenance
|
|
2026-04-17 04:45:44
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Depooling db1263 (T419635)', diff saved to https://phabricator.wikimedia.org/P91013 and previous config saved to /var/cache/conftool/dbconfig/20260417-044543-fceratto.json
|
|
2026-04-17 04:49:03
|
<jinxer-wm>
|
RESOLVED: MediaWikiEditFailures: Elevated MediaWiki edit failures (session_loss) for cluster - https://wikitech.wikimedia.org/wiki/Application_servers/Runbook - https://grafana.wikimedia.org/d/000000208/edit-count?orgId=1&viewPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DMediaWikiEditFailures
|
|
2026-04-17 04:49:07
|
<wikibugs>
|
'ops-eqiad, ''SRE, ''DBA, ''DC-Ops: Degraded RAID on pc1011 - https://phabricator.wikimedia.org/T423630#11832436 (''Marostegui) p:''Triage→''Medium Any used disk that we can use for this host?
Thanks!'
|
|
2026-04-17 04:53:07
|
<jinxer-wm>
|
FIRING: [2x] ProbeDown: Service aqs1010-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
|
|
2026-04-17 05:08:27
|
<jinxer-wm>
|
FIRING: [3x] SystemdUnitFailed: wmf_auto_restart_prometheus-blazegraph-exporter-wdqs-blazegraph.service on wdqs1012:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
|
|
2026-04-17 05:12:19
|
<wikibugs>
|
('PS1) ''Marostegui: db2158: Disable notifications [puppet] - ''https://gerrit.wikimedia.org/r/1273316'
|
|
2026-04-17 05:12:59
|
<wikibugs>
|
('CR) ''Marostegui: [C:''+2] db2158: Disable notifications [puppet] - ''https://gerrit.wikimedia.org/r/1273316 (owner: ''Marostegui)'
|
|
2026-04-17 05:13:18
|
<logmsgbot>
|
!log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2158.codfw.wmnet with reason: Reimage to Trixie
|
|
2026-04-17 05:13:24
|
<logmsgbot>
|
!log marostegui@cumin1003 START - Cookbook sre.mysql.depool depool db2158: Reimage to Trixie
|
|
2026-04-17 05:13:42
|
<logmsgbot>
|
!log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db2158: Reimage to Trixie
|
|
2026-04-17 05:16:25
|
<jinxer-wm>
|
FIRING: [2x] SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
|
|
2026-04-17 05:16:37
|
<logmsgbot>
|
!log marostegui@cumin1003 START - Cookbook sre.hosts.reimage for host db2158.codfw.wmnet with OS trixie
|
|
2026-04-17 05:33:57
|
<logmsgbot>
|
!log marostegui@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on db2158.codfw.wmnet with reason: host reimage
|
|
2026-04-17 05:37:36
|
<wikibugs>
|
('PS1) ''Marostegui: Revert "db2158: Disable notifications" [puppet] - ''https://gerrit.wikimedia.org/r/1273343'
|
|
2026-04-17 05:39:14
|
<logmsgbot>
|
!log marostegui@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2158.codfw.wmnet with reason: host reimage
|
|
2026-04-17 05:45:12
|
<jinxer-wm>
|
FIRING: [3x] CertAlmostExpired: Certificate for service lsw1-e1-codfw.mgmt.codfw.wmnet:32767 is about to expire - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired
|
|
2026-04-17 05:50:00
|
<wikibugs>
|
('CR) ''Marostegui: [C:''+2] Revert "db2158: Disable notifications" [puppet] - ''https://gerrit.wikimedia.org/r/1273343 (owner: ''Marostegui)'
|
|
2026-04-17 05:56:00
|
<wikibugs>
|
'SRE, ''Infrastructure-Foundations: Integrate Trixie 13.4 point update - https://phabricator.wikimedia.org/T420240#11832482 (''MoritzMuehlenhoff)'
|
|
2026-04-17 05:58:18
|
<wikibugs>
|
'SRE, ''Infrastructure-Foundations, ''Release-Engineering-Team (Radar): Sunsetting mirrors.wikimedia.org - https://phabricator.wikimedia.org/T416707#11832487 (''LSobanski) @bd808 In addition to the good point raised by @A_smart_kitten above the general intent here is to reduce complexity. Leaving a dependen...'
|
|
2026-04-17 06:00:05
|
<jouncebot>
|
Deploy window MediaWiki infrastructure (UTC early) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260417T0600)
|
|
2026-04-17 06:00:51
|
<logmsgbot>
|
!log marostegui@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2158.codfw.wmnet with OS trixie
|
|
2026-04-17 06:03:22
|
<wikibugs>
|
('PS3) ''Muehlenhoff: profile::zookeeper::firewall: Also allow passing a list of hosts [puppet] - ''https://gerrit.wikimedia.org/r/1272766'
|
|
2026-04-17 06:03:40
|
<wikibugs>
|
('PS4) ''Muehlenhoff: profile::zookeeper::firewall: Also allow passing a list of hosts [puppet] - ''https://gerrit.wikimedia.org/r/1272766'
|
|
2026-04-17 06:04:20
|
<logmsgbot>
|
!log marostegui@cumin1003 START - Cookbook sre.mysql.pool pool db2158: repool after maintenance
|
|
2026-04-17 06:05:55
|
<wikibugs>
|
('PS1) ''Marostegui: db1201: Disable notifications [puppet] - ''https://gerrit.wikimedia.org/r/1273388'
|
|
2026-04-17 06:06:05
|
<wikibugs>
|
('CR) ''Muehlenhoff: "check experimental" [puppet] - ''https://gerrit.wikimedia.org/r/1272766 (owner: ''Muehlenhoff)'
|
|
2026-04-17 06:06:40
|
<wikibugs>
|
('CR) ''Marostegui: [C:''+2] db1201: Disable notifications [puppet] - ''https://gerrit.wikimedia.org/r/1273388 (owner: ''Marostegui)'
|
|
2026-04-17 06:06:53
|
<logmsgbot>
|
!log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db1201.eqiad.wmnet with reason: Reimage to Trixie
|
|
2026-04-17 06:06:58
|
<logmsgbot>
|
!log marostegui@cumin1003 START - Cookbook sre.mysql.depool depool db1201: Reimage to Trixie
|
|
2026-04-17 06:07:26
|
<logmsgbot>
|
!log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.depool (exit_code=0) depool db1201: Reimage to Trixie
|
|
2026-04-17 06:08:21
|
<logmsgbot>
|
!log marostegui@cumin1003 START - Cookbook sre.hosts.reimage for host db1201.eqiad.wmnet with OS trixie
|
|
2026-04-17 06:12:45
|
<wikibugs>
|
('PS5) ''Muehlenhoff: profile::zookeeper::firewall: Also allow passing a list of hosts [puppet] - ''https://gerrit.wikimedia.org/r/1272766'
|
|
2026-04-17 06:14:15
|
<jinxer-wm>
|
FIRING: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 20.04% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy
|
|
2026-04-17 06:16:24
|
<wikibugs>
|
'SRE-swift-storage, ''Ceph, ''Data-Persistence, ''DBA: Data persistance: Re-IP eqiad private baremetal hosts to new per-rack vlans/subnets - https://phabricator.wikimedia.org/T421719#11832500 (''Marostegui)'
|
|
2026-04-17 06:16:57
|
<wikibugs>
|
('CR) ''Muehlenhoff: "check experimental" [puppet] - ''https://gerrit.wikimedia.org/r/1272766 (owner: ''Muehlenhoff)'
|
|
2026-04-17 06:17:02
|
<wikibugs>
|
'SRE-swift-storage, ''Ceph, ''Data-Persistence, ''DBA: Data persistance: Re-IP eqiad private baremetal hosts to new per-rack vlans/subnets - https://phabricator.wikimedia.org/T421719#11832504 (''Marostegui)'
|
|
2026-04-17 06:22:01
|
<logmsgbot>
|
!log marostegui@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on db1201.eqiad.wmnet with reason: host reimage
|
|
2026-04-17 06:24:15
|
<jinxer-wm>
|
RESOLVED: PHPFPMTooBusy: Not enough idle PHP-FPM workers for Mediawiki mw-web releases routed via main at eqiad: 20.66% idle - https://bit.ly/wmf-fpmsat - https://grafana.wikimedia.org/d/U7JT--knk/mw-on-k8s?orgId=1&viewPanel=84&var-dc=eqiad%20prometheus/k8s&var-service=mediawiki&var-namespace=mw-web&var-container_name=All&var-release=main - https://alerts.wikimedia.org/?q=alertname%3DPHPFPMTooBusy
|
|
2026-04-17 06:24:47
|
<logmsgbot>
|
!log marostegui@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1201.eqiad.wmnet with reason: host reimage
|
|
2026-04-17 06:25:06
|
<wikibugs>
|
'SRE, ''Maps, ''Traffic: Allow Wikimedia Maps usage on <domain> - https://phabricator.wikimedia.org/T423672 (''Abijithkumar2025) ''NEW Closing this task as invalid due to missing information.'
|
|
2026-04-17 06:26:01
|
<wikibugs>
|
'SRE, ''Maps, ''Traffic: Allow Wikimedia Maps usage on <domain> - https://phabricator.wikimedia.org/T423672#11832518 (''Abijithkumar2025) ''Invalid→''Open'
|
|
2026-04-17 06:26:53
|
<wikibugs>
|
('PS1) ''Marostegui: Revert "db1201: Disable notifications" [puppet] - ''https://gerrit.wikimedia.org/r/1273408'
|
|
2026-04-17 06:27:32
|
<wikibugs>
|
('CR) ''Marostegui: [C:''+2] Revert "db1201: Disable notifications" [puppet] - ''https://gerrit.wikimedia.org/r/1273408 (owner: ''Marostegui)'
|
|
2026-04-17 06:29:19
|
<wikibugs>
|
('PS1) ''Muehlenhoff: Remove bast1003 from list of bastions [puppet] - ''https://gerrit.wikimedia.org/r/1273413'
|
|
2026-04-17 06:40:07
|
<wikibugs>
|
'SRE, ''Maps, ''Traffic: Allow Wikimedia Maps usage on <domain> - https://phabricator.wikimedia.org/T423672#11832538 (''Aklapper) ''Open→''Invalid @Abijithkumar2025: Again: Please do not create empty tasks but fill out all
fields.'
|
|
2026-04-17 06:40:23
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1263 (T419635)', diff saved to https://phabricator.wikimedia.org/P91019 and previous config saved to /var/cache/conftool/dbconfig/20260417-064023-fceratto.json
|
|
2026-04-17 06:40:28
|
<stashbot>
|
T419635: Drop il_to column from imagelinks table in wmf production - https://phabricator.wikimedia.org/T419635
|
|
2026-04-17 06:46:47
|
<logmsgbot>
|
!log marostegui@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1201.eqiad.wmnet with OS trixie
|
|
2026-04-17 06:48:28
|
<logmsgbot>
|
!log marostegui@cumin1003 START - Cookbook sre.mysql.pool pool db1201: after reimage to trixie
|
|
2026-04-17 06:49:46
|
<logmsgbot>
|
!log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db2158: repool after maintenance
|
|
2026-04-17 06:50:32
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1263', diff saved to https://phabricator.wikimedia.org/P91022 and previous config saved to /var/cache/conftool/dbconfig/20260417-065031-fceratto.json
|
|
2026-04-17 06:52:01
|
<wikibugs>
|
('PS1) ''Muehlenhoff: Switch Cloud VPS to deb.debian.org [puppet] - ''https://gerrit.wikimedia.org/r/1273441 (https://phabricator.wikimedia.org/T416707)'
|
|
2026-04-17 06:52:17
|
<wikibugs>
|
('PS2) ''Muehlenhoff: Switch Cloud VPS to deb.debian.org [puppet] - ''https://gerrit.wikimedia.org/r/1273441 (https://phabricator.wikimedia.org/T416707)'
|
|
2026-04-17 06:56:50
|
<wikibugs>
|
('PS1) ''Jelto: gerrit: migrate data gerrit1003 to /srv/gerrit [puppet] - ''https://gerrit.wikimedia.org/r/1273449 (https://phabricator.wikimedia.org/T333143)'
|
|
2026-04-17 06:58:32
|
<wikibugs>
|
('PS1) ''Muehlenhoff: Switch the base images to deb.debian.org [puppet] - ''https://gerrit.wikimedia.org/r/1273453 (https://phabricator.wikimedia.org/T423622)'
|
|
2026-04-17 07:00:02
|
<wikibugs>
|
('CR) ''Jelto: [V:''+1] "PCC SUCCESS (CORE_DIFF 1 NOOP 2): https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/label=puppet7-compiler-node/"; [puppet] - ''https://gerrit.wikimedia.org/r/1273449 (https://phabricator.wikimedia.org/T333143) (owner: ''Jelto)'
|
|
2026-04-17 07:00:05
|
<jouncebot>
|
Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260417T0700)
|
|
2026-04-17 07:00:40
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1263', diff saved to https://phabricator.wikimedia.org/P91023 and previous config saved to /var/cache/conftool/dbconfig/20260417-070039-fceratto.json
|
|
2026-04-17 07:04:33
|
<wikibugs>
|
('CR) ''Arnaudb: [C:''+1] "lgtm!" [puppet] - ''https://gerrit.wikimedia.org/r/1273449 (https://phabricator.wikimedia.org/T333143) (owner: ''Jelto)'
|
|
2026-04-17 07:05:07
|
<wikibugs>
|
'SRE, ''Infrastructure-Foundations, ''Patch-For-Review, ''Release-Engineering-Team (Radar), ''User-notice: Sunsetting mirrors.wikimedia.org - https://phabricator.wikimedia.org/T416707#11832582 (''Nemoralis)'
|
|
2026-04-17 07:10:48
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1263 (T419635)', diff saved to https://phabricator.wikimedia.org/P91025 and previous config saved to /var/cache/conftool/dbconfig/20260417-071048-fceratto.json
|
|
2026-04-17 07:10:52
|
<stashbot>
|
T419635: Drop il_to column from imagelinks table in wmf production - https://phabricator.wikimedia.org/T419635
|
|
2026-04-17 07:11:05
|
<logmsgbot>
|
!log fceratto@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
|
|
2026-04-17 07:25:16
|
<wikibugs>
|
'SRE, ''Ganeti, ''Infrastructure-Foundations: Migrating eqsin to routed Ganeti - https://phabricator.wikimedia.org/T421863#11832624 (''MoritzMuehlenhoff)'
|
|
2026-04-17 07:25:59
|
<jinxer-wm>
|
FIRING: KubernetesDeploymentUnavailableReplicas: ...
|
|
2026-04-17 07:25:59
|
<jinxer-wm>
|
Deployment linkrecommendation-internal in linkrecommendation at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s&var-namespace=linkrecommendation&var-deployment=linkrecommendation-internal - ...
|
|
2026-04-17 07:25:59
|
<jinxer-wm>
|
https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas
|
|
2026-04-17 07:32:15
|
<wikibugs>
|
('CR) ''Elukey: [C:''+1] Remove Puppet 5 support from Spicerack [software/spicerack] - ''https://gerrit.wikimedia.org/r/1240877 (https://phabricator.wikimedia.org/T365798) (owner: ''Muehlenhoff)'
|
|
2026-04-17 07:33:53
|
<logmsgbot>
|
!log marostegui@cumin1003 END (PASS) - Cookbook sre.mysql.pool (exit_code=0) pool db1201: after reimage to trixie
|
|
2026-04-17 07:44:59
|
<wikibugs>
|
('CR) ''Brouberol: [C:''+1] "Nothing to add on top of what otto said! Great work!" [docker-images/production-images] - ''https://gerrit.wikimedia.org/r/1272726 (https://phabricator.wikimedia.org/T418525) (owner: ''Atsuko)'
|
|
2026-04-17 07:53:29
|
<wikibugs>
|
('PS1) ''Daniel Kinzler: rest gateway: add more known cg-nat addresses [deployment-charts] - ''https://gerrit.wikimedia.org/r/1273510'
|
|
2026-04-17 07:53:47
|
<wikibugs>
|
('PS1) ''Ayounsi: eqsin: add routed ganeti customer [homer/public] - ''https://gerrit.wikimedia.org/r/1273511 (https://phabricator.wikimedia.org/T421863)'
|
|
2026-04-17 07:54:09
|
<wikibugs>
|
('CR) ''Elukey: ipmi: rework how to use a different user (''1 comment) [software/spicerack] - ''https://gerrit.wikimedia.org/r/1271631 (https://phabricator.wikimedia.org/T418929) (owner: ''Elukey)'
|
|
2026-04-17 07:59:13
|
<wikibugs>
|
('CR) ''Ayounsi: [C:''+2] eqsin: add routed ganeti customer [homer/public] - ''https://gerrit.wikimedia.org/r/1273511 (https://phabricator.wikimedia.org/T421863) (owner: ''Ayounsi)'
|
|
2026-04-17 07:59:46
|
<wikibugs>
|
('CR) ''Muehlenhoff: "LGTM" [homer/public] - ''https://gerrit.wikimedia.org/r/1273511 (https://phabricator.wikimedia.org/T421863) (owner: ''Ayounsi)'
|
|
2026-04-17 08:00:50
|
<wikibugs>
|
('Merged) ''jenkins-bot: eqsin: add routed ganeti customer [homer/public] - ''https://gerrit.wikimedia.org/r/1273511 (https://phabricator.wikimedia.org/T421863) (owner: ''Ayounsi)'
|
|
2026-04-17 08:09:10
|
<jinxer-wm>
|
FIRING: [2x] GanetiBGPDown: BGP session down between ganeti5007 and cr2-eqsin - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPDown - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPDown
|
|
2026-04-17 08:09:25
|
<wikibugs>
|
('CR) ''Majavah: [C:''+1] Switch Cloud VPS to deb.debian.org [puppet] - ''https://gerrit.wikimedia.org/r/1273441 (https://phabricator.wikimedia.org/T416707) (owner: ''Muehlenhoff)'
|
|
2026-04-17 08:10:53
|
<wikibugs>
|
('CR) ''Majavah: Openstack: use debian.net repo rather than the wmf-hosted repo (''1 comment) [puppet] - ''https://gerrit.wikimedia.org/r/1272837 (https://phabricator.wikimedia.org/T423598) (owner: ''Andrew Bogott)'
|
|
2026-04-17 08:14:10
|
<jinxer-wm>
|
FIRING: [4x] GanetiBGPDown: BGP session down between ganeti5007 and cr2-eqsin - group Ganeti4 - https://wikitech.wikimedia.org/wiki/Ganeti#GanetiBGPDown - https://alerts.wikimedia.org/?q=alertname%3DGanetiBGPDown
|
|
2026-04-17 08:17:22
|
<wikibugs>
|
'SRE, ''Infrastructure-Foundations: Failing Trixie VM installations on routed Ganeti - https://phabricator.wikimedia.org/T422596#11832695 (''MoritzMuehlenhoff) This turned out to be also reproducible on classic Ganeti and after some painful debugging given the very early failure (approx 1.5 seconds after Linu...'
|
|
2026-04-17 08:18:02
|
<wikibugs>
|
'SRE, ''Infrastructure-Foundations: Failing Trixie VM installations on routed Ganeti - https://phabricator.wikimedia.org/T422596#11832697 (''MoritzMuehlenhoff)'
|
|
2026-04-17 08:21:40
|
<wikibugs>
|
('CR) ''Muehlenhoff: [C:''+2] Remove Puppet 5 support from Spicerack [software/spicerack] - ''https://gerrit.wikimedia.org/r/1240877 (https://phabricator.wikimedia.org/T365798) (owner: ''Muehlenhoff)'
|
|
2026-04-17 08:27:00
|
<wikibugs>
|
('PS4) ''Atsuko: flink: Install flink in blubber-compatible venv [docker-images/production-images] - ''https://gerrit.wikimedia.org/r/1272726 (https://phabricator.wikimedia.org/T418525)'
|
|
2026-04-17 08:33:22
|
<wikibugs>
|
('CR) ''DCausse: "I908966a32fc264c5084719337812ca303bac509c might make this patch useless since it removes space-discount" [mediawiki-config] - ''https://gerrit.wikimedia.org/r/1267130 (https://phabricator.wikimedia.org/T420427) (owner: ''DCausse)'
|
|
2026-04-17 08:33:56
|
<wikibugs>
|
('Abandoned) ''DCausse: search: add space-discount for wikidata custom prefix search profiles [mediawiki-config] - ''https://gerrit.wikimedia.org/r/1267130 (https://phabricator.wikimedia.org/T420427) (owner: ''DCausse)'
|
|
2026-04-17 08:37:58
|
<jinxer-wm>
|
FIRING: [4x] ProbeDown: Service text-https:443 has failed probes (http_text-https_ip4) #page - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
|
|
2026-04-17 08:37:58
|
<logmsgbot>
|
!log dpogorzelski@deploy1003 helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'.
|
|
2026-04-17 08:38:07
|
<icinga-wm>
|
PROBLEM - SSH on cp3072 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring
|
|
2026-04-17 08:38:07
|
<icinga-wm>
|
PROBLEM - SSH on cp3070 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring
|
|
2026-04-17 08:38:07
|
<icinga-wm>
|
PROBLEM - SSH on cp3068 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring
|
|
2026-04-17 08:38:07
|
<icinga-wm>
|
PROBLEM - SSH on cp3067 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring
|
|
2026-04-17 08:38:07
|
<icinga-wm>
|
PROBLEM - SSH on cp3073 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring
|
|
2026-04-17 08:38:07
|
<icinga-wm>
|
PROBLEM - SSH on cp3066 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring
|
|
2026-04-17 08:38:09
|
<icinga-wm>
|
PROBLEM - SSH on cp3069 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring
|
|
2026-04-17 08:38:09
|
<icinga-wm>
|
PROBLEM - SSH on cp3071 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring
|
|
2026-04-17 08:38:13
|
<icinga-wm>
|
PROBLEM - Ensure traffic_manager binds on 3128 and responds to HTTP requests on cp3072 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server
|
|
2026-04-17 08:38:13
|
<icinga-wm>
|
PROBLEM - Ensure traffic_manager binds on 3128 and responds to HTTP requests on cp3070 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server
|
|
2026-04-17 08:38:13
|
<icinga-wm>
|
PROBLEM - Ensure traffic_manager binds on 3128 and responds to HTTP requests on cp3071 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server
|
|
2026-04-17 08:38:13
|
<icinga-wm>
|
PROBLEM - Ensure traffic_manager binds on 3128 and responds to HTTP requests on cp3066 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server
|
|
2026-04-17 08:38:13
|
<icinga-wm>
|
PROBLEM - Ensure traffic_manager binds on 3128 and responds to HTTP requests on cp3068 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server
|
|
2026-04-17 08:38:13
|
<icinga-wm>
|
PROBLEM - Ensure traffic_manager binds on 3128 and responds to HTTP requests on cp3073 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server
|
|
2026-04-17 08:38:13
|
<icinga-wm>
|
PROBLEM - Ensure traffic_manager binds on 3128 and responds to HTTP requests on cp3069 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server
|
|
2026-04-17 08:38:27
|
<icinga-wm>
|
PROBLEM - Ensure traffic_exporter for the backend instance binds on port 9122 and responds to HTTP requests on cp3070 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server
|
|
2026-04-17 08:38:27
|
<icinga-wm>
|
PROBLEM - Ensure traffic_exporter for the backend instance binds on port 9122 and responds to HTTP requests on cp3071 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server
|
|
2026-04-17 08:38:27
|
<icinga-wm>
|
PROBLEM - Ensure traffic_exporter for the backend instance binds on port 9122 and responds to HTTP requests on cp3072 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server
|
|
2026-04-17 08:38:27
|
<icinga-wm>
|
PROBLEM - Ensure traffic_exporter for the backend instance binds on port 9122 and responds to HTTP requests on cp3067 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server
|
|
2026-04-17 08:38:27
|
<icinga-wm>
|
PROBLEM - Ensure traffic_exporter for the backend instance binds on port 9122 and responds to HTTP requests on cp3066 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server
|
|
2026-04-17 08:38:27
|
<icinga-wm>
|
PROBLEM - Ensure traffic_exporter for the backend instance binds on port 9122 and responds to HTTP requests on cp3068 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server
|
|
2026-04-17 08:38:27
|
<icinga-wm>
|
PROBLEM - Ensure traffic_exporter for the backend instance binds on port 9122 and responds to HTTP requests on cp3069 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server
|
|
2026-04-17 08:38:30
|
<jelto>
|
I cant reach any of our services (gerrit, grafana) from Germany
|
|
2026-04-17 08:38:48
|
<arnaudb>
|
works from France
|
|
2026-04-17 08:38:48
|
<bjensen>
|
!ack
|
|
2026-04-17 08:38:49
|
<sirenbot>
|
7847 (ACKED) [4x] ProbeDown sre (probes/service esams)
|
|
2026-04-17 08:38:52
|
<jelto>
|
esams has problems?
|
|
2026-04-17 08:38:59
|
<icinga-wm>
|
RECOVERY - SSH on cp3067 is OK: SSH OK - OpenSSH_10.0p2 Debian-7+deb13u1 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring
|
|
2026-04-17 08:39:17
|
<icinga-wm>
|
RECOVERY - Ensure traffic_exporter for the backend instance binds on port 9122 and responds to HTTP requests on cp3067 is OK: HTTP OK: HTTP/1.0 200 OK - 37121 bytes in 0.287 second response time https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server
|
|
2026-04-17 08:39:45
|
<logmsgbot>
|
!log dpogorzelski@deploy1003 helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'.
|
|
2026-04-17 08:39:49
|
<marostegui>
|
Maybe what XioNoX mentioned about gtt?
|
|
2026-04-17 08:39:52
|
<logmsgbot>
|
!log dpogorzelski@deploy1003 helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'.
|
|
2026-04-17 08:39:57
|
<icinga-wm>
|
RECOVERY - SSH on cp3073 is OK: SSH OK - OpenSSH_10.0p2 Debian-7+deb13u1 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring
|
|
2026-04-17 08:40:03
|
<icinga-wm>
|
RECOVERY - Ensure traffic_manager binds on 3128 and responds to HTTP requests on cp3073 is OK: HTTP OK: HTTP/1.1 200 OK - 50385 bytes in 0.332 second response time https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server
|
|
2026-04-17 08:40:05
|
<icinga-wm>
|
RECOVERY - Ensure traffic_manager binds on 3128 and responds to HTTP requests on cp3072 is OK: HTTP OK: HTTP/1.1 200 OK - 50371 bytes in 1.841 second response time https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server
|
|
2026-04-17 08:40:18
|
<bjensen>
|
hm, having a tough time opening wikitech/grafana
|
|
2026-04-17 08:40:29
|
<bjensen>
|
from Ireland
|
|
2026-04-17 08:40:30
|
<jinxer-wm>
|
FIRING: [8x] LibericaUnhealthyRealserverPooled: Liberica service gerrit-httpslb6_443 has 4 unhealthy realservers pooled on lvs3008:3003 - https://wikitech.wikimedia.org/wiki/Liberica#LibericaUnhealthyRealserverPooled - https://alerts.wikimedia.org/?q=alertname%3DLibericaUnhealthyRealserverPooled
|
|
2026-04-17 08:40:40
|
<marostegui>
|
Works from Spain
|
|
2026-04-17 08:41:30
|
<jelto>
|
with tunnelencabulator it works again
|
|
2026-04-17 08:41:46
|
<marostegui>
|
Should we depool esams?
|
|
2026-04-17 08:41:50
|
<marostegui>
|
But it does work for me
|
|
2026-04-17 08:41:51
|
<jinxer-wm>
|
FIRING: SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://en.wikipedia.org/api/rest_v1 - https://grafana.wikimedia.org/d/_77ik484k/openapi-swagger-endpoint-state?var-site=esams - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures
|
|
2026-04-17 08:42:02
|
<jelto>
|
I think that was a traffic spike: https://grafana.wikimedia.org/d/O_OXJyTVk/home-w-wiki-status?orgId=1&from=now-3h&to=now&timezone=utc&refresh=5m&viewPanel=panel-2
|
|
2026-04-17 08:42:14
|
<logmsgbot>
|
!log dpogorzelski@deploy1003 helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'.
|
|
2026-04-17 08:42:15
|
<arnaudb>
|
oof
|
|
2026-04-17 08:42:27
|
<icinga-wm>
|
PROBLEM - Ensure traffic_exporter for the backend instance binds on port 9122 and responds to HTTP requests on cp3067 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server
|
|
2026-04-17 08:42:27
|
<marostegui>
|
Seems to be recovering?
|
|
2026-04-17 08:42:35
|
<logmsgbot>
|
!log dpogorzelski@deploy1003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'edit-check' for release 'main' .
|
|
2026-04-17 08:42:58
|
<jinxer-wm>
|
FIRING: NELHigh: Elevated Network Error Logging events (tcp.timed_out) #page - https://wikitech.wikimedia.org/wiki/Network_monitoring#NEL_alerts - https://logstash.wikimedia.org/goto/5c8f4ca1413eda33128e5c5a35da7e28 - https://alerts.wikimedia.org/?q=alertname%3DNELHigh
|
|
2026-04-17 08:42:58
|
<jinxer-wm>
|
FIRING: [4x] NELByCountryHigh: Elevated Network Error Logging events (tcp.timed_out from DE) - https://wikitech.wikimedia.org/wiki/Network_monitoring#NEL_alerts - https://logstash.wikimedia.org/goto/5c8f4ca1413eda33128e5c5a35da7e28 - https://alerts.wikimedia.org/?q=alertname%3DNELByCountryHigh
|
|
2026-04-17 08:43:07
|
<icinga-wm>
|
PROBLEM - SSH on cp3067 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/SSH/monitoring
|
|
2026-04-17 08:43:13
|
<icinga-wm>
|
PROBLEM - Ensure traffic_manager binds on 3128 and responds to HTTP requests on cp3072 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server
|
|
2026-04-17 08:43:13
|
<icinga-wm>
|
PROBLEM - Ensure traffic_manager binds on 3128 and responds to HTTP requests on cp3067 is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server
|
|
2026-04-17 08:43:27
|
<bjensen>
|
i suppose that makes sense
|
|
2026-04-17 08:43:45
|
<marostegui>
|
bjensen: what?
|
|
2026-04-17 08:43:58
|
<bjensen>
|
the by-country alert
|
|
2026-04-17 08:44:00
|
<wikibugs>
|
('CR) ''Ayounsi: [C:''+1] "Lgtm, I also see that there is a mention of bast1003 in `modules/wmflib/spec/functions/ipresolve_spec.rb`" [puppet] - ''https://gerrit.wikimedia.org/r/1273413 (owner: ''Muehlenhoff)'
|
|
2026-04-17 08:44:05
|
<logmsgbot>
|
!log dpogorzelski@deploy1003 helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'.
|
|
2026-04-17 08:44:17
|
<jinxer-wm>
|
FIRING: [4x] JobUnavailable: Reduced availability for job probes/swagger in ops@esams - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
|
|
2026-04-17 08:45:03
|
<bjensen>
|
is depooling esams likely to intensify the problem?
|
|
2026-04-17 08:45:05
|
<icinga-wm>
|
RECOVERY - SSH on cp3072 is OK: SSH OK - OpenSSH_10.0p2 Debian-7+deb13u1 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring
|
|
2026-04-17 08:45:05
|
<icinga-wm>
|
RECOVERY - Ensure traffic_manager binds on 3128 and responds to HTTP requests on cp3072 is OK: HTTP OK: HTTP/1.1 200 OK - 50375 bytes in 2.672 second response time https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server
|
|
2026-04-17 08:45:17
|
<icinga-wm>
|
RECOVERY - Ensure traffic_exporter for the backend instance binds on port 9122 and responds to HTTP requests on cp3072 is OK: HTTP OK: HTTP/1.0 200 OK - 37130 bytes in 0.291 second response time https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server
|
|
2026-04-17 08:45:53
|
<logmsgbot>
|
!log dpogorzelski@deploy1003 helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'.
|
|
2026-04-17 08:45:57
|
<icinga-wm>
|
RECOVERY - SSH on cp3067 is OK: SSH OK - OpenSSH_10.0p2 Debian-7+deb13u1 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring
|
|
2026-04-17 08:46:03
|
<icinga-wm>
|
RECOVERY - Ensure traffic_manager binds on 3128 and responds to HTTP requests on cp3067 is OK: HTTP OK: HTTP/1.1 200 OK - 50372 bytes in 0.329 second response time https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server
|
|
2026-04-17 08:46:05
|
<logmsgbot>
|
!log dpogorzelski@deploy1003 helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'.
|
|
2026-04-17 08:46:17
|
<icinga-wm>
|
RECOVERY - Ensure traffic_exporter for the backend instance binds on port 9122 and responds to HTTP requests on cp3067 is OK: HTTP OK: HTTP/1.0 200 OK - 37121 bytes in 0.288 second response time https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server
|
|
2026-04-17 08:46:53
|
<wikibugs>
|
('PS1) ''Slyngshede: data: align config [puppet] - ''https://gerrit.wikimedia.org/r/1273658'
|
|
2026-04-17 08:46:57
|
<icinga-wm>
|
RECOVERY - SSH on cp3066 is OK: SSH OK - OpenSSH_10.0p2 Debian-7+deb13u1 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring
|
|
2026-04-17 08:47:03
|
<icinga-wm>
|
RECOVERY - Ensure traffic_manager binds on 3128 and responds to HTTP requests on cp3066 is OK: HTTP OK: HTTP/1.1 200 OK - 50378 bytes in 0.329 second response time https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server
|
|
2026-04-17 08:47:17
|
<icinga-wm>
|
RECOVERY - Ensure traffic_exporter for the backend instance binds on port 9122 and responds to HTTP requests on cp3066 is OK: HTTP OK: HTTP/1.0 200 OK - 37129 bytes in 0.288 second response time https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server
|
|
2026-04-17 08:47:57
|
<jinxer-wm>
|
FIRING: [4x] ProbeDown: Service text-https:443 has failed probes (http_text-https_ip4) #page - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
|
|
2026-04-17 08:47:58
|
<jinxer-wm>
|
FIRING: [2x] NELHigh: Elevated Network Error Logging events (tcp.address_unreachable) #page - https://wikitech.wikimedia.org/wiki/Network_monitoring#NEL_alerts - https://logstash.wikimedia.org/goto/5c8f4ca1413eda33128e5c5a35da7e28 - https://alerts.wikimedia.org/?q=alertname%3DNELHigh
|
|
2026-04-17 08:47:59
|
<jinxer-wm>
|
FIRING: [6x] NELByCountryHigh: Elevated Network Error Logging events (tcp.address_unreachable from DE) - https://wikitech.wikimedia.org/wiki/Network_monitoring#NEL_alerts - https://logstash.wikimedia.org/goto/5c8f4ca1413eda33128e5c5a35da7e28 - https://alerts.wikimedia.org/?q=alertname%3DNELByCountryHigh
|
|
2026-04-17 08:48:06
|
<marostegui>
|
!ack
|
|
2026-04-17 08:48:07
|
<sirenbot>
|
All incidents are already acked.
|
|
2026-04-17 08:48:26
|
<bjensen>
|
!incidents
|
|
2026-04-17 08:48:26
|
<sirenbot>
|
7847 (ACKED) [4x] ProbeDown sre (probes/service esams)
|
|
2026-04-17 08:48:26
|
<sirenbot>
|
7848 (ACKED) NELHigh sre (thanos-rule@main tcp.timed_out)
|
|
2026-04-17 08:48:26
|
<sirenbot>
|
7846 (RESOLVED) ProbeDown sre (2001:df2:e500:ed1a::1 ip6 text:80 probes/service http_text_ip6 eqsin)
|
|
2026-04-17 08:48:27
|
<sirenbot>
|
7845 (RESOLVED) TransitPeeringTransportOutSaturation network sre (cr1-eqiad:9804 Transit: NTT (234630) {#3475} xe-3/1/6 gnmi eqiad)
|
|
2026-04-17 08:48:27
|
<logmsgbot>
|
!log dpogorzelski@deploy1003 helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'.
|
|
2026-04-17 08:48:27
|
<sirenbot>
|
7844 (RESOLVED) TransitPeeringTransportOutSaturation network sre (cr1-eqiad:9804 Transit: Lumen (442550281) {#3867} xe-3/3/2 gnmi eqiad)
|
|
2026-04-17 08:48:27
|
<sirenbot>
|
7843 (RESOLVED) TransitPeeringTransportOutSaturation network sre (cr1-codfw:9804 Transport: cr4-ulsfo:xe-0/1/1 (Lumen, 442550294) {#12252_12295-1} xe-1/1/1:0 gnmi codfw)
|
|
2026-04-17 08:49:17
|
<jinxer-wm>
|
FIRING: [4x] JobUnavailable: Reduced availability for job probes/swagger in ops@esams - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
|
|
2026-04-17 08:50:30
|
<jinxer-wm>
|
FIRING: [8x] LibericaUnhealthyRealserverPooled: Liberica service gerrit-httpslb6_443 has 3 unhealthy realservers pooled on lvs3008:3003 - https://wikitech.wikimedia.org/wiki/Liberica#LibericaUnhealthyRealserverPooled - https://alerts.wikimedia.org/?q=alertname%3DLibericaUnhealthyRealserverPooled
|
|
2026-04-17 08:50:42
|
<logmsgbot>
|
!log dpogorzelski@deploy1003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'edit-check' for release 'main' .
|
|
2026-04-17 08:51:16
|
<logmsgbot>
|
!log cmooney@cumin1003 START - Cookbook sre.dns.admin DNS admin: depool esams [reason: no reason specified, no task ID specified]
|
|
2026-04-17 08:51:19
|
<logmsgbot>
|
!log cmooney@cumin1003 END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool esams [reason: no reason specified, no task ID specified]
|
|
2026-04-17 08:51:42
|
<topranks>
|
!log depool esams due to connectivity issues
|
|
2026-04-17 08:51:43
|
<stashbot>
|
Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
|
|
2026-04-17 08:51:51
|
<jinxer-wm>
|
RESOLVED: SwaggerProbeHasFailures: Not all openapi/swagger endpoints returned healthy - https://wikitech.wikimedia.org/wiki/Runbook#https://en.wikipedia.org/api/rest_v1 - https://grafana.wikimedia.org/d/_77ik484k/openapi-swagger-endpoint-state?var-site=esams - https://alerts.wikimedia.org/?q=alertname%3DSwaggerProbeHasFailures
|
|
2026-04-17 08:52:57
|
<jinxer-wm>
|
FIRING: [4x] ProbeDown: Service text-https:443 has failed probes (http_text-https_ip4) #page - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
|
|
2026-04-17 08:53:07
|
<jinxer-wm>
|
FIRING: [4x] ProbeDown: Service aqs1010-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
|
|
2026-04-17 08:53:25
|
<icinga-wm>
|
RECOVERY - Ensure traffic_exporter for the backend instance binds on port 9122 and responds to HTTP requests on cp3070 is OK: HTTP OK: HTTP/1.0 200 OK - 37119 bytes in 7.573 second response time https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server
|
|
2026-04-17 08:53:50
|
<wikibugs>
|
('PS2) ''Dpogorzelski: knative-serving: update chart to 1.21.1 [deployment-charts] - ''https://gerrit.wikimedia.org/r/1271709'
|
|
2026-04-17 08:53:57
|
<icinga-wm>
|
RECOVERY - SSH on cp3070 is OK: SSH OK - OpenSSH_10.0p2 Debian-7+deb13u1 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring
|
|
2026-04-17 08:54:03
|
<icinga-wm>
|
RECOVERY - Ensure traffic_manager binds on 3128 and responds to HTTP requests on cp3070 is OK: HTTP OK: HTTP/1.1 200 OK - 50370 bytes in 0.329 second response time https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server
|
|
2026-04-17 08:54:17
|
<jinxer-wm>
|
RESOLVED: [4x] JobUnavailable: Reduced availability for job probes/swagger in ops@esams - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
|
|
2026-04-17 08:55:30
|
<jinxer-wm>
|
FIRING: [8x] LibericaUnhealthyRealserverPooled: Liberica service gerrit-httpslb6_443 has 3 unhealthy realservers pooled on lvs3008:3003 - https://wikitech.wikimedia.org/wiki/Liberica#LibericaUnhealthyRealserverPooled - https://alerts.wikimedia.org/?q=alertname%3DLibericaUnhealthyRealserverPooled
|
|
2026-04-17 08:56:13
|
<wikibugs>
|
('PS1) ''Dpogorzelski: kserve-resources: fix securityContext propagation [deployment-charts] - ''https://gerrit.wikimedia.org/r/1273664 (https://phabricator.wikimedia.org/T423149)'
|
|
2026-04-17 08:57:54
|
<wikibugs>
|
('CR) ''CI reject: [V:''-1] kserve-resources: fix securityContext propagation [deployment-charts] - ''https://gerrit.wikimedia.org/r/1273664 (https://phabricator.wikimedia.org/T423149) (owner: ''Dpogorzelski)'
|
|
2026-04-17 08:57:58
|
<jinxer-wm>
|
RESOLVED: [2x] NELHigh: Elevated Network Error Logging events (tcp.address_unreachable) #page - https://wikitech.wikimedia.org/wiki/Network_monitoring#NEL_alerts - https://logstash.wikimedia.org/goto/5c8f4ca1413eda33128e5c5a35da7e28 - https://alerts.wikimedia.org/?q=alertname%3DNELHigh
|
|
2026-04-17 08:57:59
|
<jinxer-wm>
|
RESOLVED: [6x] NELByCountryHigh: Elevated Network Error Logging events (tcp.address_unreachable from DE) - https://wikitech.wikimedia.org/wiki/Network_monitoring#NEL_alerts - https://logstash.wikimedia.org/goto/5c8f4ca1413eda33128e5c5a35da7e28 - https://alerts.wikimedia.org/?q=alertname%3DNELByCountryHigh
|
|
2026-04-17 09:05:09
|
<icinga-wm>
|
RECOVERY - Ensure traffic_manager binds on 3128 and responds to HTTP requests on cp3069 is OK: HTTP OK: HTTP/1.1 200 OK - 50375 bytes in 6.214 second response time https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server
|
|
2026-04-17 09:05:17
|
<icinga-wm>
|
RECOVERY - Ensure traffic_exporter for the backend instance binds on port 9122 and responds to HTTP requests on cp3069 is OK: HTTP OK: HTTP/1.0 200 OK - 37126 bytes in 0.290 second response time https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server
|
|
2026-04-17 09:05:59
|
<icinga-wm>
|
RECOVERY - SSH on cp3069 is OK: SSH OK - OpenSSH_10.0p2 Debian-7+deb13u1 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring
|
|
2026-04-17 09:07:50
|
<wikibugs>
|
('CR) ''Elukey: "Really sorry for all the extra work that I caused with the suggestion of removing this, at this point we need to just add the workloadType" [deployment-charts] - ''https://gerrit.wikimedia.org/r/1273664 (https://phabricator.wikimedia.org/T423149) (owner: ''Dpogorzelski)'
|
|
2026-04-17 09:08:00
|
<wikibugs>
|
('CR) ''Atsuko: flink: Install flink in blubber-compatible venv (''1 comment) [docker-images/production-images] - ''https://gerrit.wikimedia.org/r/1272726 (https://phabricator.wikimedia.org/T418525) (owner: ''Atsuko)'
|
|
2026-04-17 09:08:42
|
<jinxer-wm>
|
FIRING: [2x] SystemdUnitFailed: wmf_auto_restart_prometheus-blazegraph-exporter-wdqs-blazegraph.service on wdqs1012:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
|
|
2026-04-17 09:10:30
|
<jinxer-wm>
|
RESOLVED: [4x] LibericaUnhealthyRealserverPooled: Liberica service text-httpslb6_443 has 3 unhealthy realservers pooled on lvs3008:3003 - https://wikitech.wikimedia.org/wiki/Liberica#LibericaUnhealthyRealserverPooled - https://alerts.wikimedia.org/?q=alertname%3DLibericaUnhealthyRealserverPooled
|
|
2026-04-17 09:12:41
|
<wikibugs>
|
('CR) ''Elukey: "Even if it makes no sense, the current CRD allows extra fields, it doesn't really validate them. So it should be ok now." [deployment-charts] - ''https://gerrit.wikimedia.org/r/1273664 (https://phabricator.wikimedia.org/T423149) (owner: ''Dpogorzelski)'
|
|
2026-04-17 09:13:39
|
<wikibugs>
|
('CR) ''Muehlenhoff: [C:''+1] "Looks good" [puppet] - ''https://gerrit.wikimedia.org/r/1273658 (owner: ''Slyngshede)'
|
|
2026-04-17 09:13:46
|
<wikibugs>
|
('PS1) ''Jcrespo: backup: Ignore /srv/docker from srv-deployment backups, move cluster mgmt [puppet] - ''https://gerrit.wikimedia.org/r/1273676 (https://phabricator.wikimedia.org/T423619)'
|
|
2026-04-17 09:14:01
|
<wikibugs>
|
('PS2) ''Jcrespo: backup: Ignore /srv/docker from srv-deployment backups, move cluster mgmt [puppet] - ''https://gerrit.wikimedia.org/r/1273676 (https://phabricator.wikimedia.org/T423619)'
|
|
2026-04-17 09:14:10
|
<wikibugs>
|
('CR) ''Jcrespo: "check experimental" [puppet] - ''https://gerrit.wikimedia.org/r/1273676 (https://phabricator.wikimedia.org/T423619) (owner: ''Jcrespo)'
|
|
2026-04-17 09:14:12
|
<marostegui>
|
!incidents
|
|
2026-04-17 09:14:12
|
<sirenbot>
|
7847 (ACKED) [4x] ProbeDown sre (probes/service esams)
|
|
2026-04-17 09:14:12
|
<sirenbot>
|
7848 (RESOLVED) NELHigh sre (thanos-rule@main tcp.timed_out)
|
|
2026-04-17 09:14:13
|
<sirenbot>
|
7846 (RESOLVED) ProbeDown sre (2001:df2:e500:ed1a::1 ip6 text:80 probes/service http_text_ip6 eqsin)
|
|
2026-04-17 09:14:13
|
<sirenbot>
|
7845 (RESOLVED) TransitPeeringTransportOutSaturation network sre (cr1-eqiad:9804 Transit: NTT (234630) {#3475} xe-3/1/6 gnmi eqiad)
|
|
2026-04-17 09:14:13
|
<sirenbot>
|
7844 (RESOLVED) TransitPeeringTransportOutSaturation network sre (cr1-eqiad:9804 Transit: Lumen (442550281) {#3867} xe-3/3/2 gnmi eqiad)
|
|
2026-04-17 09:16:25
|
<jinxer-wm>
|
FIRING: [2x] SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
|
|
2026-04-17 09:16:51
|
<wikibugs>
|
('CR) ''Brouberol: [C:''+1] flink: Install flink in blubber-compatible venv [docker-images/production-images] - ''https://gerrit.wikimedia.org/r/1272726 (https://phabricator.wikimedia.org/T418525) (owner: ''Atsuko)'
|
|
2026-04-17 09:21:00
|
<wikibugs>
|
('PS1) ''Jelto: gerrit: migrate gerrit2003 data to /srv/gerrit [puppet] - ''https://gerrit.wikimedia.org/r/1273683 (https://phabricator.wikimedia.org/T333143)'
|
|
2026-04-17 09:21:27
|
<wikibugs>
|
('CR) ''Jelto: [C:''-1] "should not be merged yet" [puppet] - ''https://gerrit.wikimedia.org/r/1273683 (https://phabricator.wikimedia.org/T333143) (owner: ''Jelto)'
|
|
2026-04-17 09:21:34
|
<wikibugs>
|
('CR) ''Atsuko: [C:''+2] flink: Install flink in blubber-compatible venv [docker-images/production-images] - ''https://gerrit.wikimedia.org/r/1272726 (https://phabricator.wikimedia.org/T418525) (owner: ''Atsuko)'
|
|
2026-04-17 09:25:04
|
<wikibugs>
|
'SRE-tools, ''Infrastructure-Foundations, ''Data-Platform-SRE (2026-03-27 - 2026-04-17): debmonitor-client crashes for growthbook image - https://phabricator.wikimedia.org/T423413#11832785 (''brouberol) a:''brouberol'
|
|
2026-04-17 09:25:34
|
<wikibugs>
|
('CR) ''Atsuko: [V:''+2 C:''+2] flink: Install flink in blubber-compatible venv [docker-images/production-images] - ''https://gerrit.wikimedia.org/r/1272726 (https://phabricator.wikimedia.org/T418525) (owner: ''Atsuko)'
|
|
2026-04-17 09:26:13
|
<wikibugs>
|
'SRE-tools, ''Infrastructure-Foundations, ''Data-Platform-SRE (2026-03-27 - 2026-04-17): debmonitor-client crashes for growthbook image - https://phabricator.wikimedia.org/T423413#11832790 (''brouberol) @elukey I'd appreciated guidance as to how to build the new `docker-report` deb package, and deploy it. T...'
|
|
2026-04-17 09:29:35
|
<wikibugs>
|
('CR) ''Jcrespo: "Please have a look and send amends/comments." [puppet] - ''https://gerrit.wikimedia.org/r/1273676 (https://phabricator.wikimedia.org/T423619) (owner: ''Jcrespo)'
|
|
2026-04-17 09:32:47
|
<icinga-wm>
|
PROBLEM - Host cp3068 is DOWN: PING CRITICAL - Packet loss = 100%
|
|
2026-04-17 09:34:35
|
<icinga-wm>
|
RECOVERY - Host cp3068 is UP: PING OK - Packet loss = 0%, RTA = 80.14 ms
|
|
2026-04-17 09:34:37
|
<icinga-wm>
|
PROBLEM - haproxy process on cp3068 is CRITICAL: PROCS CRITICAL: 0 processes with command name haproxy https://wikitech.wikimedia.org/wiki/HAProxy
|
|
2026-04-17 09:34:57
|
<icinga-wm>
|
RECOVERY - SSH on cp3068 is OK: SSH OK - OpenSSH_10.0p2 Debian-7+deb13u1 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring
|
|
2026-04-17 09:35:07
|
<icinga-wm>
|
RECOVERY - Ensure traffic_manager binds on 3128 and responds to HTTP requests on cp3068 is OK: HTTP OK: HTTP/1.1 200 OK - 48162 bytes in 0.324 second response time https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server
|
|
2026-04-17 09:35:17
|
<icinga-wm>
|
RECOVERY - Ensure traffic_exporter for the backend instance binds on port 9122 and responds to HTTP requests on cp3068 is OK: HTTP OK: HTTP/1.0 200 OK - 36142 bytes in 0.280 second response time https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server
|
|
2026-04-17 09:35:20
|
<logmsgbot>
|
!log ayounsi@cumin1003 START - Cookbook sre.network.tls for network device lsw1-e3-codfw
|
|
2026-04-17 09:35:21
|
<icinga-wm>
|
PROBLEM - HAProxy HTTPS wikiworkshop.org ECDSA on cp3068 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection refused https://wikitech.wikimedia.org/wiki/HTTPS
|
|
2026-04-17 09:35:21
|
<icinga-wm>
|
PROBLEM - HAProxy HTTPS wikipedia25.org ECDSA on cp3068 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection refused https://wikitech.wikimedia.org/wiki/HTTPS
|
|
2026-04-17 09:35:21
|
<icinga-wm>
|
PROBLEM - HAProxy HTTPS wikipedia.org ECDSA on cp3068 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection refused https://wikitech.wikimedia.org/wiki/HTTPS
|
|
2026-04-17 09:35:27
|
<logmsgbot>
|
!log ayounsi@cumin1003 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e3-codfw
|
|
2026-04-17 09:35:34
|
<logmsgbot>
|
!log ayounsi@cumin1003 START - Cookbook sre.network.tls for network device lsw1-e1-codfw
|
|
2026-04-17 09:35:41
|
<logmsgbot>
|
!log ayounsi@cumin1003 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-e1-codfw
|
|
2026-04-17 09:35:49
|
<logmsgbot>
|
!log ayounsi@cumin1003 START - Cookbook sre.network.tls for network device ssw1-e1-codfw
|
|
2026-04-17 09:35:56
|
<logmsgbot>
|
!log ayounsi@cumin1003 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device ssw1-e1-codfw
|
|
2026-04-17 09:36:04
|
<logmsgbot>
|
!log ayounsi@cumin1003 START - Cookbook sre.network.tls for network device ssw1-f1-codfw
|
|
2026-04-17 09:36:11
|
<logmsgbot>
|
!log ayounsi@cumin1003 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device ssw1-f1-codfw
|
|
2026-04-17 09:36:20
|
<logmsgbot>
|
!log ayounsi@cumin1003 START - Cookbook sre.network.tls for network device lsw1-f3-codfw
|
|
2026-04-17 09:36:27
|
<logmsgbot>
|
!log ayounsi@cumin1003 END (PASS) - Cookbook sre.network.tls (exit_code=0) for network device lsw1-f3-codfw
|
|
2026-04-17 09:38:21
|
<icinga-wm>
|
RECOVERY - HAProxy HTTPS wikiworkshop.org ECDSA on cp3068 is OK: SSL OK - Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2026-05-13 04:44:41 +0000 (expires in 25 days) https://wikitech.wikimedia.org/wiki/HTTPS
|
|
2026-04-17 09:38:21
|
<icinga-wm>
|
RECOVERY - HAProxy HTTPS wikipedia.org ECDSA on cp3068 is OK: SSL OK - Certificate *.wikipedia.org contains all required SANs:Certificate *.wikipedia.org (ECDSA) valid until 2026-07-06 20:52:29 +0000 (expires in 80 days) https://wikitech.wikimedia.org/wiki/HTTPS
|
|
2026-04-17 09:38:21
|
<icinga-wm>
|
RECOVERY - HAProxy HTTPS wikipedia25.org ECDSA on cp3068 is OK: SSL OK - Certificate wikipedia25.org contains all required SANs:Certificate wikipedia25.org (ECDSA) valid until 2026-06-06 06:58:50 +0000 (expires in 49 days) https://wikitech.wikimedia.org/wiki/HTTPS
|
|
2026-04-17 09:38:38
|
<wikibugs>
|
('PS2) ''Jcrespo: dbbackups: Backup only regularly clusters 32 & 33, the read-write ones [puppet] - ''https://gerrit.wikimedia.org/r/1271730 (https://phabricator.wikimedia.org/T421729)'
|
|
2026-04-17 09:38:38
|
<wikibugs>
|
('PS3) ''Jcrespo: backup: Ignore /srv/docker from srv-deployment backups, move cluster mgmt [puppet] - ''https://gerrit.wikimedia.org/r/1273676 (https://phabricator.wikimedia.org/T423619)'
|
|
2026-04-17 09:38:38
|
<wikibugs>
|
('PS1) ''Jcrespo: backup: Remove references to ips of hosts about to be decommissioned [puppet] - ''https://gerrit.wikimedia.org/r/1273699 (https://phabricator.wikimedia.org/T422851)'
|
|
2026-04-17 09:38:39
|
<icinga-wm>
|
RECOVERY - haproxy process on cp3068 is OK: PROCS OK: 2 processes with command name haproxy https://wikitech.wikimedia.org/wiki/HAProxy
|
|
2026-04-17 09:38:51
|
<wikibugs>
|
'SRE, ''SRE-Access-Requests, ''Data-Engineering: Requesting access to analytics_privatedata_users and SQL Lab for AnnieKim_WMDE - https://phabricator.wikimedia.org/T420500#11832838 (''Martyn.ranyard) I approve this, but I think @Scott_French or someone else on that team still needs to add you to the gro...'
|
|
2026-04-17 09:39:13
|
<wikibugs>
|
('PS2) ''Jcrespo: backup: Remove references to ips of hosts about to be decommissioned [puppet] - ''https://gerrit.wikimedia.org/r/1273699 (https://phabricator.wikimedia.org/T422851)'
|
|
2026-04-17 09:39:26
|
<wikibugs>
|
('PS4) ''Jcrespo: backup: Ignore /srv/docker from srv-deployment backups, move cluster mgmt [puppet] - ''https://gerrit.wikimedia.org/r/1273676 (https://phabricator.wikimedia.org/T423619)'
|
|
2026-04-17 09:39:57
|
<jinxer-wm>
|
RESOLVED: [3x] CertAlmostExpired: Certificate for service lsw1-e1-codfw.mgmt.codfw.wmnet:32767 is about to expire - TODO - https://alerts.wikimedia.org/?q=alertname%3DCertAlmostExpired
|
|
2026-04-17 09:43:59
|
<icinga-wm>
|
PROBLEM - Host cp3071 is DOWN: PING CRITICAL - Packet loss = 100%
|
|
2026-04-17 09:44:34
|
<moritzm>
|
!log initialise eqsin02 Ganeti cluster T421863
|
|
2026-04-17 09:44:37
|
<stashbot>
|
Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
|
|
2026-04-17 09:44:37
|
<icinga-wm>
|
RECOVERY - Host cp3071 is UP: PING OK - Packet loss = 0%, RTA = 80.19 ms
|
|
2026-04-17 09:44:37
|
<icinga-wm>
|
PROBLEM - haproxy process on cp3071 is CRITICAL: PROCS CRITICAL: 0 processes with command name haproxy https://wikitech.wikimedia.org/wiki/HAProxy
|
|
2026-04-17 09:44:37
|
<icinga-wm>
|
PROBLEM - statsv Varnishkafka log producer on cp3071 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka
|
|
2026-04-17 09:44:38
|
<stashbot>
|
T421863: Migrating eqsin to routed Ganeti - https://phabricator.wikimedia.org/T421863
|
|
2026-04-17 09:44:59
|
<icinga-wm>
|
RECOVERY - SSH on cp3071 is OK: SSH OK - OpenSSH_10.0p2 Debian-7+deb13u1 (protocol 2.0) https://wikitech.wikimedia.org/wiki/SSH/monitoring
|
|
2026-04-17 09:45:05
|
<icinga-wm>
|
RECOVERY - Ensure traffic_manager binds on 3128 and responds to HTTP requests on cp3071 is OK: HTTP OK: HTTP/1.1 200 OK - 48144 bytes in 0.323 second response time https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server
|
|
2026-04-17 09:45:17
|
<icinga-wm>
|
RECOVERY - Ensure traffic_exporter for the backend instance binds on port 9122 and responds to HTTP requests on cp3071 is OK: HTTP OK: HTTP/1.0 200 OK - 36141 bytes in 0.281 second response time https://wikitech.wikimedia.org/wiki/Apache_Traffic_Server
|
|
2026-04-17 09:45:29
|
<icinga-wm>
|
PROBLEM - HAProxy HTTPS wikiworkshop.org ECDSA on cp3071 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection refused https://wikitech.wikimedia.org/wiki/HTTPS
|
|
2026-04-17 09:45:29
|
<icinga-wm>
|
PROBLEM - HAProxy HTTPS wikipedia25.org ECDSA on cp3071 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection refused https://wikitech.wikimedia.org/wiki/HTTPS
|
|
2026-04-17 09:45:29
|
<icinga-wm>
|
PROBLEM - HAProxy HTTPS wikipedia.org ECDSA on cp3071 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection refused https://wikitech.wikimedia.org/wiki/HTTPS
|
|
2026-04-17 09:45:37
|
<icinga-wm>
|
RECOVERY - statsv Varnishkafka log producer on cp3071 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka
|
|
2026-04-17 09:47:57
|
<jinxer-wm>
|
RESOLVED: [2x] ProbeDown: Service text:80 has failed probes (http_text_ip4) #page - https://wikitech.wikimedia.org/wiki/Runbook#text:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
|
|
2026-04-17 09:48:05
|
<XioNoX>
|
nice
|
|
2026-04-17 09:48:07
|
<jinxer-wm>
|
FIRING: [4x] ProbeDown: Service aqs1010-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
|
|
2026-04-17 09:48:29
|
<icinga-wm>
|
RECOVERY - HAProxy HTTPS wikiworkshop.org ECDSA on cp3071 is OK: SSL OK - Certificate wikiworkshop.org contains all required SANs:Certificate wikiworkshop.org (ECDSA) valid until 2026-05-13 04:44:41 +0000 (expires in 25 days) https://wikitech.wikimedia.org/wiki/HTTPS
|
|
2026-04-17 09:48:29
|
<icinga-wm>
|
RECOVERY - HAProxy HTTPS wikipedia25.org ECDSA on cp3071 is OK: SSL OK - Certificate wikipedia25.org contains all required SANs:Certificate wikipedia25.org (ECDSA) valid until 2026-06-06 06:58:50 +0000 (expires in 49 days) https://wikitech.wikimedia.org/wiki/HTTPS
|
|
2026-04-17 09:48:29
|
<icinga-wm>
|
RECOVERY - HAProxy HTTPS wikipedia.org ECDSA on cp3071 is OK: SSL OK - Certificate *.wikipedia.org contains all required SANs:Certificate *.wikipedia.org (ECDSA) valid until 2026-07-06 20:52:29 +0000 (expires in 80 days) https://wikitech.wikimedia.org/wiki/HTTPS
|
|
2026-04-17 09:48:39
|
<icinga-wm>
|
RECOVERY - haproxy process on cp3071 is OK: PROCS OK: 2 processes with command name haproxy https://wikitech.wikimedia.org/wiki/HAProxy
|
|
2026-04-17 09:48:42
|
<marostegui>
|
!incidents
|
|
2026-04-17 09:48:43
|
<sirenbot>
|
7847 (RESOLVED) [4x] ProbeDown sre (probes/service esams)
|
|
2026-04-17 09:48:43
|
<sirenbot>
|
7848 (RESOLVED) NELHigh sre (thanos-rule@main tcp.timed_out)
|
|
2026-04-17 09:48:43
|
<sirenbot>
|
7846 (RESOLVED) ProbeDown sre (2001:df2:e500:ed1a::1 ip6 text:80 probes/service http_text_ip6 eqsin)
|
|
2026-04-17 09:48:43
|
<sirenbot>
|
7845 (RESOLVED) TransitPeeringTransportOutSaturation network sre (cr1-eqiad:9804 Transit: NTT (234630) {#3475} xe-3/1/6 gnmi eqiad)
|
|
2026-04-17 09:48:43
|
<sirenbot>
|
7844 (RESOLVED) TransitPeeringTransportOutSaturation network sre (cr1-eqiad:9804 Transit: Lumen (442550281) {#3867} xe-3/3/2 gnmi eqiad)
|
|
2026-04-17 09:50:02
|
<wikibugs>
|
('PS1) ''Muehlenhoff: Netbox: Add eqsin02 to Ganeti sync [puppet] - ''https://gerrit.wikimedia.org/r/1273700 (https://phabricator.wikimedia.org/T421863)'
|
|
2026-04-17 09:50:03
|
<wikibugs>
|
('CR) ''Jcrespo: "I believe you were the ones to introduce these real ips, making sure you are ok with this change." [puppet] - ''https://gerrit.wikimedia.org/r/1273699 (https://phabricator.wikimedia.org/T422851) (owner: ''Jcrespo)'
|
|
2026-04-17 09:52:01
|
<icinga-wm>
|
PROBLEM - Check unit status of httpbb_kubernetes_mw-web-next_hourly on cumin2002 is CRITICAL: CRITICAL: Status of the systemd unit httpbb_kubernetes_mw-web-next_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
|
|
2026-04-17 09:53:51
|
<logmsgbot>
|
!log marostegui@cumin1003 START - Cookbook sre.dns.admin DNS admin: pool esams [reason: no reason specified, no task ID specified]
|
|
2026-04-17 09:53:58
|
<logmsgbot>
|
!log marostegui@cumin1003 END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool esams [reason: no reason specified, no task ID specified]
|
|
2026-04-17 09:54:05
|
<marostegui>
|
!log pool esams
|
|
2026-04-17 09:54:07
|
<stashbot>
|
Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
|
|
2026-04-17 09:55:06
|
<logmsgbot>
|
!log jynus@cumin1003 START - Cookbook sre.hosts.decommission for hosts ms-backup1001.eqiad.wmnet
|
|
2026-04-17 09:56:36
|
<wikibugs>
|
('CR) ''Ayounsi: [C:''+1] Netbox: Add eqsin02 to Ganeti sync [puppet] - ''https://gerrit.wikimedia.org/r/1273700 (https://phabricator.wikimedia.org/T421863) (owner: ''Muehlenhoff)'
|
|
2026-04-17 09:58:57
|
<wikibugs>
|
('CR) ''Muehlenhoff: [C:''+2] Netbox: Add eqsin02 to Ganeti sync [puppet] - ''https://gerrit.wikimedia.org/r/1273700 (https://phabricator.wikimedia.org/T421863) (owner: ''Muehlenhoff)'
|
|
2026-04-17 10:00:56
|
<logmsgbot>
|
!log jynus@cumin1003 START - Cookbook sre.dns.netbox
|
|
2026-04-17 10:02:08
|
<wikibugs>
|
'ops-eqiad, ''SRE, ''DC-Ops, ''Data-Platform-SRE (2026-03-27 - 2026-04-17): Degraded RAID on an-worker1205 - https://phabricator.wikimedia.org/T422317#11832942 (''brouberol) @Jclark-ctr Feel free to replace the drive whenever convenient for you. Thank you!'
|
|
2026-04-17 10:02:19
|
<wikibugs>
|
'SRE, ''cloud-services-team, ''Infrastructure-Foundations, ''Toolforge: Adjust WMCS Gitlab CI/CD repo to stop using mirrors.wikimedia.org - https://phabricator.wikimedia.org/T423596#11832943 (''A_smart_kitten) (per T423596#11829501 & as the files seem potentially toolforge-related FWICS; feel free to reta...'
|
|
2026-04-17 10:03:19
|
<wikibugs>
|
('PS1) ''Jcrespo: backup: Remove old references to ms-backup1001 & ms-backup1002 [puppet] - ''https://gerrit.wikimedia.org/r/1273704 (https://phabricator.wikimedia.org/T422851)'
|
|
2026-04-17 10:03:44
|
<logmsgbot>
|
!log fceratto@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1156.eqiad.wmnet with reason: Maintenance
|
|
2026-04-17 10:03:53
|
<logmsgbot>
|
!log fceratto@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1014,1018].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
|
|
2026-04-17 10:03:55
|
<wikibugs>
|
('PS6) ''Jelto: gerrit: migrate gerrit_site away from root partition [puppet] - ''https://gerrit.wikimedia.org/r/1270774 (https://phabricator.wikimedia.org/T423027) (owner: ''Arnaudb)'
|
|
2026-04-17 10:04:01
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Depooling db1156 (T419961)', diff saved to https://phabricator.wikimedia.org/P91029 and previous config saved to /var/cache/conftool/dbconfig/20260417-100401-fceratto.json
|
|
2026-04-17 10:05:56
|
<wikibugs>
|
('CR) ''Jelto: [C:''-1] "@abran@wikimedia.org I rebased the change, maybe you can check if it also makes sense to you? -1 for now, should be merged after the migra" [puppet] - ''https://gerrit.wikimedia.org/r/1270774 (https://phabricator.wikimedia.org/T423027) (owner: ''Arnaudb)'
|
|
2026-04-17 10:06:39
|
<logmsgbot>
|
jynus@cumin1003 decommission (PID 2051147) is awaiting input
|
|
2026-04-17 10:11:16
|
<jynus>
|
thank you mr logmsgbot
|
|
2026-04-17 10:11:50
|
<logmsgbot>
|
!log jynus@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ms-backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1003"
|
|
2026-04-17 10:12:33
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1156 (T419961)', diff saved to https://phabricator.wikimedia.org/P91030 and previous config saved to /var/cache/conftool/dbconfig/20260417-101233-fceratto.json
|
|
2026-04-17 10:12:47
|
<jynus>
|
there is a new ganeti_cluster: codfw_test for netbox
|
|
2026-04-17 10:12:58
|
<logmsgbot>
|
!log jynus@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ms-backup1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1003"
|
|
2026-04-17 10:12:58
|
<logmsgbot>
|
!log jynus@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
|
|
2026-04-17 10:12:59
|
<logmsgbot>
|
!log jynus@cumin1003 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ms-backup1001.eqiad.wmnet
|
|
2026-04-17 10:13:56
|
<logmsgbot>
|
!log jynus@cumin1003 START - Cookbook sre.hosts.decommission for hosts ms-backup1002.eqiad.wmnet
|
|
2026-04-17 10:14:25
|
<jinxer-wm>
|
FIRING: SystemdUnitFailed: netbox_ganeti_eqsin02_sync.service on netbox1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
|
|
2026-04-17 10:15:09
|
<wikibugs>
|
'SRE, ''Ganeti, ''Infrastructure-Foundations, ''Patch-For-Review: Migrating eqsin to routed Ganeti - https://phabricator.wikimedia.org/T421863#11832984 (''MoritzMuehlenhoff)'
|
|
2026-04-17 10:20:08
|
<logmsgbot>
|
!log jynus@cumin1003 START - Cookbook sre.dns.netbox
|
|
2026-04-17 10:22:41
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P91031 and previous config saved to /var/cache/conftool/dbconfig/20260417-102241-fceratto.json
|
|
2026-04-17 10:23:17
|
<wikibugs>
|
('PS2) ''Muehlenhoff: Remove bast1003 from list of bastions [puppet] - ''https://gerrit.wikimedia.org/r/1273413'
|
|
2026-04-17 10:25:47
|
<logmsgbot>
|
jynus@cumin1003 decommission (PID 2069153) is awaiting input
|
|
2026-04-17 10:31:08
|
<logmsgbot>
|
!log jynus@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ms-backup1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1003"
|
|
2026-04-17 10:32:50
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P91032 and previous config saved to /var/cache/conftool/dbconfig/20260417-103249-fceratto.json
|
|
2026-04-17 10:34:13
|
<logmsgbot>
|
jynus@cumin1003 decommission (PID 2069153) is awaiting input
|
|
2026-04-17 10:36:13
|
<wikibugs>
|
'SRE-tools, ''bacula, ''Data-Persistence-Backup, ''Infrastructure-Foundations, and 2 others: netbox2003 backups (maybe others?) are missconfigured or failing to find the configured directory - https://phabricator.wikimedia.org/T423689 (''jcrespo) ''NEW'
|
|
2026-04-17 10:36:46
|
<wikibugs>
|
'SRE, ''SRE-swift-storage, ''SRE Observability: Thanos backends filling their root filesystems overnight - https://phabricator.wikimedia.org/T423690 (''MatthewVernon) ''NEW'
|
|
2026-04-17 10:36:51
|
<wikibugs>
|
'SRE, ''SRE-swift-storage, ''SRE Observability: Thanos backends filling their root filesystems overnight - https://phabricator.wikimedia.org/T423690#11833110 (''MatthewVernon) p:''Triage→''High'
|
|
2026-04-17 10:36:53
|
<wikibugs>
|
'SRE-tools, ''bacula, ''Data-Persistence-Backup, ''Infrastructure-Foundations, and 2 others: netbox2003 backups (maybe others?) are missconfigured or failing to find the configured directory - https://phabricator.wikimedia.org/T423689#11833112 (''jcrespo) Let me know if this box or any other requires inve...'
|
|
2026-04-17 10:37:16
|
<logmsgbot>
|
!log jynus@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ms-backup1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1003"
|
|
2026-04-17 10:37:16
|
<logmsgbot>
|
!log jynus@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
|
|
2026-04-17 10:37:17
|
<logmsgbot>
|
!log jynus@cumin1003 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ms-backup1002.eqiad.wmnet
|
|
2026-04-17 10:37:24
|
<wikibugs>
|
('CR) ''Clément Goubert: [C:''+1] backup: Ignore /srv/docker from srv-deployment backups, move cluster mgmt [puppet] - ''https://gerrit.wikimedia.org/r/1273676 (https://phabricator.wikimedia.org/T423619) (owner: ''Jcrespo)'
|
|
2026-04-17 10:37:48
|
<wikibugs>
|
('CR) ''Jcrespo: [C:''+2] backup: Remove old references to ms-backup1001 & ms-backup1002 [puppet] - ''https://gerrit.wikimedia.org/r/1273704 (https://phabricator.wikimedia.org/T422851) (owner: ''Jcrespo)'
|
|
2026-04-17 10:38:08
|
<wikibugs>
|
('PS2) ''Clément Goubert: gateway-check: Add matchers for liftwing and recommendation-api-ng [puppet] - ''https://gerrit.wikimedia.org/r/1271804 (https://phabricator.wikimedia.org/T422804)'
|
|
2026-04-17 10:38:56
|
<wikibugs>
|
'SRE-tools, ''bacula, ''Data-Persistence-Backup, ''Infrastructure-Foundations, and 2 others: netbox2003 backups (maybe others?) are missconfigured or failing to find the backup directory - https://phabricator.wikimedia.org/T423689#11833117 (''jcrespo)'
|
|
2026-04-17 10:40:45
|
<wikibugs>
|
'SRE, ''SRE-swift-storage, ''Ceph, ''ServiceOps new, and 2 others: scap can’t deploy (blob upload unknown) after apus.discovery.wmnet is repooled in codfw - https://phabricator.wikimedia.org/T422166#11833133 (''Blake) That makes a lot of sense; I've updated the docs with those edits, thanks! I've also lin...'
|
|
2026-04-17 10:41:48
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''decommission-hardware: decommission ms-backup1001 & ms-backup1002 - https://phabricator.wikimedia.org/T422851#11833134 (''jcrespo) a:''jcrespo→''None'
|
|
2026-04-17 10:42:01
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''decommission-hardware: decommission ms-backup1001 & ms-backup1002 - https://phabricator.wikimedia.org/T422851#11833138 (''jcrespo) This is ready for dc ops.'
|
|
2026-04-17 10:42:27
|
<wikibugs>
|
('CR) ''Clément Goubert: [C:''+1] opensearch on k8s: Add semantic-search and ipoid to services proxy [puppet] - ''https://gerrit.wikimedia.org/r/1264739 (https://phabricator.wikimedia.org/T421293) (owner: ''Bking)'
|
|
2026-04-17 10:42:58
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1156 (T419961)', diff saved to https://phabricator.wikimedia.org/P91033 and previous config saved to /var/cache/conftool/dbconfig/20260417-104257-fceratto.json
|
|
2026-04-17 10:43:01
|
<wikibugs>
|
('CR) ''Clément Goubert: [C:''+1] Switch the base images to deb.debian.org [puppet] - ''https://gerrit.wikimedia.org/r/1273453 (https://phabricator.wikimedia.org/T423622) (owner: ''Muehlenhoff)'
|
|
2026-04-17 10:43:06
|
<logmsgbot>
|
!log jynus@cumin1003 START - Cookbook sre.hosts.decommission for hosts ms-backup2001.codfw.wmnet
|
|
2026-04-17 10:43:19
|
<logmsgbot>
|
!log fceratto@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1162.eqiad.wmnet with reason: Maintenance
|
|
2026-04-17 10:43:28
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Depooling db1162 (T419961)', diff saved to https://phabricator.wikimedia.org/P91034 and previous config saved to /var/cache/conftool/dbconfig/20260417-104327-fceratto.json
|
|
2026-04-17 10:43:32
|
<wikibugs>
|
('CR) ''Clément Goubert: [C:''+1] opensearch on k8s: Activate semantic-search and ipoid in services proxy [puppet] - ''https://gerrit.wikimedia.org/r/1272909 (https://phabricator.wikimedia.org/T421293) (owner: ''Bking)'
|
|
2026-04-17 10:44:49
|
<wikibugs>
|
('PS2) ''Effie Mouzeli: mcrouter: update to 1.3.5 [deployment-charts] - ''https://gerrit.wikimedia.org/r/1272785 (https://phabricator.wikimedia.org/T421360)'
|
|
2026-04-17 10:44:59
|
<wikibugs>
|
('CR) ''Muehlenhoff: [C:''+2] Switch the base images to deb.debian.org [puppet] - ''https://gerrit.wikimedia.org/r/1273453 (https://phabricator.wikimedia.org/T423622) (owner: ''Muehlenhoff)'
|
|
2026-04-17 10:45:41
|
<icinga-wm>
|
PROBLEM - Check if Pybal has been restarted after pybal.conf was changed on lvs2014 is CRITICAL: CRITICAL: Service pybal.service has not been restarted after /etc/pybal/pybal.conf was changed (gt 1h). https://wikitech.wikimedia.org/wiki/PyBal%23Pybal_service_has_not_been_restarted
|
|
2026-04-17 10:47:04
|
<wikibugs>
|
'SRE, ''Infrastructure-Foundations, ''Patch-For-Review, ''Release-Engineering-Team (Radar): New base images without mirrors.wikimedia.org - https://phabricator.wikimedia.org/T423622#11833147 (''MoritzMuehlenhoff) >>! In T423622#11830895, @thcipriani wrote: > Updated task description to clarify, yes, the S...'
|
|
2026-04-17 10:47:15
|
<wikibugs>
|
'SRE, ''Infrastructure-Foundations, ''Patch-For-Review, ''Release-Engineering-Team (Radar): New base images without mirrors.wikimedia.org - https://phabricator.wikimedia.org/T423622#11833149 (''MoritzMuehlenhoff) p:''Triage→''High'
|
|
2026-04-17 10:48:36
|
<logmsgbot>
|
!log jynus@cumin1003 START - Cookbook sre.dns.netbox
|
|
2026-04-17 10:50:21
|
<icinga-wm>
|
PROBLEM - Check if Pybal has been restarted after pybal.conf was changed on lvs1019 is CRITICAL: CRITICAL: Service pybal.service has not been restarted after /etc/pybal/pybal.conf was changed (gt 1h). https://wikitech.wikimedia.org/wiki/PyBal%23Pybal_service_has_not_been_restarted
|
|
2026-04-17 10:50:21
|
<icinga-wm>
|
PROBLEM - Check if Pybal has been restarted after pybal.conf was changed on lvs2013 is CRITICAL: CRITICAL: Service pybal.service has not been restarted after /etc/pybal/pybal.conf was changed (gt 1h). https://wikitech.wikimedia.org/wiki/PyBal%23Pybal_service_has_not_been_restarted
|
|
2026-04-17 10:51:18
|
<wikibugs>
|
('PS1) ''Jcrespo: bacula: Reenable copy jobs to the offsite pool [puppet] - ''https://gerrit.wikimedia.org/r/1273738 (https://phabricator.wikimedia.org/T313582)'
|
|
2026-04-17 10:51:31
|
<wikibugs>
|
('PS2) ''Jcrespo: bacula: Reenable copy jobs to the offsite pool [puppet] - ''https://gerrit.wikimedia.org/r/1273738 (https://phabricator.wikimedia.org/T313582)'
|
|
2026-04-17 10:52:01
|
<icinga-wm>
|
RECOVERY - Check unit status of httpbb_kubernetes_mw-web-next_hourly on cumin2002 is OK: OK: Status of the systemd unit httpbb_kubernetes_mw-web-next_hourly https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
|
|
2026-04-17 10:52:27
|
<wikibugs>
|
('CR) ''Jcrespo: "check experimental" [puppet] - ''https://gerrit.wikimedia.org/r/1273738 (https://phabricator.wikimedia.org/T313582) (owner: ''Jcrespo)'
|
|
2026-04-17 10:52:35
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1162 (T419961)', diff saved to https://phabricator.wikimedia.org/P91035 and previous config saved to /var/cache/conftool/dbconfig/20260417-105234-fceratto.json
|
|
2026-04-17 10:53:28
|
<logmsgbot>
|
!log jynus@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ms-backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1003"
|
|
2026-04-17 10:54:42
|
<logmsgbot>
|
!log jynus@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ms-backup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1003"
|
|
2026-04-17 10:54:42
|
<logmsgbot>
|
!log jynus@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
|
|
2026-04-17 10:54:42
|
<wikibugs>
|
('PS1) ''Effie Mouzeli: (WIP) update mcrouter module to 1.3.5 [deployment-charts] - ''https://gerrit.wikimedia.org/r/1273739'
|
|
2026-04-17 10:54:43
|
<logmsgbot>
|
!log jynus@cumin1003 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ms-backup2001.codfw.wmnet
|
|
2026-04-17 10:55:21
|
<logmsgbot>
|
!log jynus@cumin1003 START - Cookbook sre.hosts.decommission for hosts ms-backup2002.codfw.wmnet
|
|
2026-04-17 10:55:40
|
<icinga-wm>
|
PROBLEM - Check if Pybal has been restarted after pybal.conf was changed on lvs1020 is CRITICAL: CRITICAL: Service pybal.service has not been restarted after /etc/pybal/pybal.conf was changed (gt 1h). https://wikitech.wikimedia.org/wiki/PyBal%23Pybal_service_has_not_been_restarted
|
|
2026-04-17 10:57:04
|
<wikibugs>
|
('PS3) ''Jcrespo: bacula: Reenable copy jobs to the offsite pool [puppet] - ''https://gerrit.wikimedia.org/r/1273738 (https://phabricator.wikimedia.org/T313582)'
|
|
2026-04-17 10:57:10
|
<wikibugs>
|
('CR) ''Jcrespo: "check experimental" [puppet] - ''https://gerrit.wikimedia.org/r/1273738 (https://phabricator.wikimedia.org/T313582) (owner: ''Jcrespo)'
|
|
2026-04-17 11:00:05
|
<jouncebot>
|
Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260417T0700)
|
|
2026-04-17 11:00:05
|
<jouncebot>
|
jelto, arnoldokoth, mutante, and arnaudb: May I have your attention please! GitLab version upgrades. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260417T1100)
|
|
2026-04-17 11:02:05
|
<wikibugs>
|
('CR) ''Jcrespo: [C:''+2] bacula: Reenable copy jobs to the offsite pool [puppet] - ''https://gerrit.wikimedia.org/r/1273738 (https://phabricator.wikimedia.org/T313582) (owner: ''Jcrespo)'
|
|
2026-04-17 11:02:43
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P91036 and previous config saved to /var/cache/conftool/dbconfig/20260417-110242-fceratto.json
|
|
2026-04-17 11:03:00
|
<logmsgbot>
|
!log jynus@cumin1003 START - Cookbook sre.dns.netbox
|
|
2026-04-17 11:08:18
|
<logmsgbot>
|
!log jynus@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ms-backup2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1003"
|
|
2026-04-17 11:11:24
|
<logmsgbot>
|
jynus@cumin1003 decommission (PID 2112382) is awaiting input
|
|
2026-04-17 11:11:31
|
<logmsgbot>
|
!log jynus@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
|
|
2026-04-17 11:11:31
|
<logmsgbot>
|
!log jynus@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ms-backup2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin1003"
|
|
2026-04-17 11:11:32
|
<logmsgbot>
|
!log jynus@cumin1003 END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ms-backup2002.codfw.wmnet
|
|
2026-04-17 11:12:05
|
<wikibugs>
|
('PS1) ''Jcrespo: backup: Fix wrong storage config for offsite [puppet] - ''https://gerrit.wikimedia.org/r/1273740 (https://phabricator.wikimedia.org/T313582)'
|
|
2026-04-17 11:12:32
|
<wikibugs>
|
('PS2) ''Jcrespo: backup: Fix wrong storage config for offsite [puppet] - ''https://gerrit.wikimedia.org/r/1273740 (https://phabricator.wikimedia.org/T313582)'
|
|
2026-04-17 11:12:51
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P91037 and previous config saved to /var/cache/conftool/dbconfig/20260417-111250-fceratto.json
|
|
2026-04-17 11:14:38
|
<wikibugs>
|
'SRE, ''Infrastructure-Foundations: Integrate Trixie 13.3 point update - https://phabricator.wikimedia.org/T414179#11833215 (''MoritzMuehlenhoff)'
|
|
2026-04-17 11:15:27
|
<wikibugs>
|
('CR) ''Jcrespo: "check experimental" [puppet] - ''https://gerrit.wikimedia.org/r/1273740 (https://phabricator.wikimedia.org/T313582) (owner: ''Jcrespo)'
|
|
2026-04-17 11:15:52
|
<icinga-wm>
|
PROBLEM - Backup freshness on backup1014 is CRITICAL: Stale: 1 (backup1013), Fresh: 138 jobs https://wikitech.wikimedia.org/wiki/Bacula%23Monitoring
|
|
2026-04-17 11:17:40
|
<wikibugs>
|
('CR) ''Jcrespo: [C:''+2] backup: Fix wrong storage config for offsite [puppet] - ''https://gerrit.wikimedia.org/r/1273740 (https://phabricator.wikimedia.org/T313582) (owner: ''Jcrespo)'
|
|
2026-04-17 11:19:35
|
<wikibugs>
|
('PS1) ''Muehlenhoff: sre.ganeti.makevm: Bump memory requirements for VMs to 2G [cookbooks] - ''https://gerrit.wikimedia.org/r/1273743 (https://phabricator.wikimedia.org/T422596)'
|
|
2026-04-17 11:23:04
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1162 (T419961)', diff saved to Unable to send diff to phaste and previous config saved to /var/cache/conftool/dbconfig/20260417-112259-fceratto.json
|
|
2026-04-17 11:23:25
|
<logmsgbot>
|
!log fceratto@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1182.eqiad.wmnet with reason: Maintenance
|
|
2026-04-17 11:23:34
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Depooling db1182 (T419961)', diff saved to https://phabricator.wikimedia.org/P91039 and previous config saved to /var/cache/conftool/dbconfig/20260417-112333-fceratto.json
|
|
2026-04-17 11:25:10
|
<wikibugs>
|
('PS1) ''Ayounsi: eqsin routed ganeti: use private IPs instead of loopbacks [puppet] - ''https://gerrit.wikimedia.org/r/1273744 (https://phabricator.wikimedia.org/T421863)'
|
|
2026-04-17 11:25:59
|
<jinxer-wm>
|
FIRING: KubernetesDeploymentUnavailableReplicas: ...
|
|
2026-04-17 11:25:59
|
<jinxer-wm>
|
Deployment linkrecommendation-internal in linkrecommendation at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s&var-namespace=linkrecommendation&var-deployment=linkrecommendation-internal - ...
|
|
2026-04-17 11:25:59
|
<jinxer-wm>
|
https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas
|
|
2026-04-17 11:30:56
|
<icinga-wm>
|
RECOVERY - MariaDB Replica Lag: s8 on dbstore1009 is OK: OK slave_sql_lag Replication lag: 0.36 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica
|
|
2026-04-17 11:31:49
|
<wikibugs>
|
'ops-eqiad, ''SRE, ''DC-Ops, ''decommission-hardware: decommission ms-backup1001 & ms-backup1002 - https://phabricator.wikimedia.org/T422851#11833242 (''Jclark-ctr) a:''Jclark-ctr'
|
|
2026-04-17 11:32:01
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1182 (T419961)', diff saved to https://phabricator.wikimedia.org/P91040 and previous config saved to /var/cache/conftool/dbconfig/20260417-113201-fceratto.json
|
|
2026-04-17 11:32:25
|
<wikibugs>
|
('CR) ''Muehlenhoff: [C:''+1] "Looks good" [puppet] - ''https://gerrit.wikimedia.org/r/1273744 (https://phabricator.wikimedia.org/T421863) (owner: ''Ayounsi)'
|
|
2026-04-17 11:32:48
|
<wikibugs>
|
'ops-eqiad, ''SRE, ''cloud-services-team, ''Data-Services, and 3 others: decommission clouddb1019.eqiad.wmnet - https://phabricator.wikimedia.org/T423151#11833255 (''Jclark-ctr)'
|
|
2026-04-17 11:32:50
|
<wikibugs>
|
'ops-eqiad, ''SRE, ''cloud-services-team, ''Data-Services, and 3 others: decommission clouddb1019.eqiad.wmnet - https://phabricator.wikimedia.org/T423151#11833256 (''Jclark-ctr) ''Open→''Resolved'
|
|
2026-04-17 11:34:58
|
<wikibugs>
|
('CR) ''Ayounsi: [C:''+2] eqsin routed ganeti: use private IPs instead of loopbacks [puppet] - ''https://gerrit.wikimedia.org/r/1273744 (https://phabricator.wikimedia.org/T421863) (owner: ''Ayounsi)'
|
|
2026-04-17 11:35:30
|
<icinga-wm>
|
PROBLEM - Host ganeti1031 is DOWN: PING CRITICAL - Packet loss = 100%
|
|
2026-04-17 11:35:32
|
<icinga-wm>
|
PROBLEM - Host netboxdb1003 is DOWN: PING CRITICAL - Packet loss = 100%
|
|
2026-04-17 11:35:34
|
<icinga-wm>
|
PROBLEM - Host logstash1023 is DOWN: PING CRITICAL - Packet loss = 100%
|
|
2026-04-17 11:36:04
|
<icinga-wm>
|
PROBLEM - Host apt1002 is DOWN: PING CRITICAL - Packet loss = 100%
|
|
2026-04-17 11:37:00
|
<icinga-wm>
|
RECOVERY - Host ganeti1031 is UP: PING OK - Packet loss = 0%, RTA = 0.29 ms
|
|
2026-04-17 11:37:30
|
<wikibugs>
|
('PS1) ''Jcrespo: mediabackup: Remove last references to ms-backup2001 & ms-backup2002 [puppet] - ''https://gerrit.wikimedia.org/r/1273745 (https://phabricator.wikimedia.org/T422852)'
|
|
2026-04-17 11:38:07
|
<jinxer-wm>
|
FIRING: [5x] ProbeDown: Service aqs1010-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
|
|
2026-04-17 11:38:33
|
<wikibugs>
|
('CR) ''Ayounsi: [C:''+1] sre.ganeti.makevm: Bump memory requirements for VMs to 2G [cookbooks] - ''https://gerrit.wikimedia.org/r/1273743 (https://phabricator.wikimedia.org/T422596) (owner: ''Muehlenhoff)'
|
|
2026-04-17 11:38:37
|
<wikibugs>
|
('CR) ''Jcrespo: [C:''+2] mediabackup: Remove last references to ms-backup2001 & ms-backup2002 [puppet] - ''https://gerrit.wikimedia.org/r/1273745 (https://phabricator.wikimedia.org/T422852) (owner: ''Jcrespo)'
|
|
2026-04-17 11:39:14
|
<wikibugs>
|
'SRE, ''Ganeti, ''Infrastructure-Foundations, ''Patch-For-Review: Migrating eqsin to routed Ganeti - https://phabricator.wikimedia.org/T421863#11833265 (''ayounsi)'
|
|
2026-04-17 11:39:17
|
<jinxer-wm>
|
FIRING: [2x] JobUnavailable: Reduced availability for job netbox_global in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
|
|
2026-04-17 11:40:32
|
<icinga-wm>
|
RECOVERY - Host apt1002 is UP: PING OK - Packet loss = 0%, RTA = 0.59 ms
|
|
2026-04-17 11:40:34
|
<wikibugs>
|
'ops-codfw, ''DC-Ops, ''decommission-hardware: decommission ms-backup2001 & ms-backup2002 - https://phabricator.wikimedia.org/T422852#11833266 (''jcrespo) a:''jcrespo→''None'
|
|
2026-04-17 11:40:36
|
<icinga-wm>
|
RECOVERY - Host logstash1023 is UP: PING OK - Packet loss = 0%, RTA = 0.34 ms
|
|
2026-04-17 11:40:36
|
<icinga-wm>
|
RECOVERY - Host netboxdb1003 is UP: PING WARNING - Packet loss = 50%, RTA = 0.84 ms
|
|
2026-04-17 11:40:47
|
<wikibugs>
|
'ops-codfw, ''DC-Ops, ''decommission-hardware: decommission ms-backup2001 & ms-backup2002 - https://phabricator.wikimedia.org/T422852#11833270 (''jcrespo) This is ready for dc ops.'
|
|
2026-04-17 11:41:53
|
<wikibugs>
|
('CR) ''Klausman: [C:''+1] "Just one clarification question, LGTM." [deployment-charts] - ''https://gerrit.wikimedia.org/r/1271709 (owner: ''Dpogorzelski)'
|
|
2026-04-17 11:42:10
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P91041 and previous config saved to /var/cache/conftool/dbconfig/20260417-114210-fceratto.json
|
|
2026-04-17 11:42:20
|
<wikibugs>
|
('CR) ''Klausman: [C:''+1] kserve-resources: fix securityContext propagation [deployment-charts] - ''https://gerrit.wikimedia.org/r/1273664 (https://phabricator.wikimedia.org/T423149) (owner: ''Dpogorzelski)'
|
|
2026-04-17 11:43:07
|
<jinxer-wm>
|
FIRING: [5x] ProbeDown: Service aqs1010-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
|
|
2026-04-17 11:44:17
|
<jinxer-wm>
|
RESOLVED: [2x] JobUnavailable: Reduced availability for job netbox_global in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
|
|
2026-04-17 11:44:25
|
<jinxer-wm>
|
FIRING: [7x] SystemdUnitFailed: netbox_ganeti_codfw_sync.service on netbox1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
|
|
2026-04-17 11:45:44
|
<XioNoX>
|
moritzm: any idea why ganeti1031 rebooted? :) ganeti1031:~$ uptime -> 11:45:17 up 8 min, 2 users, load average: 5.94, 4.17, 1.94
|
|
2026-04-17 11:45:55
|
<XioNoX>
|
it briefly took down Netbox's DB
|
|
2026-04-17 11:47:08
|
<wikibugs>
|
'ops-eqiad, ''SRE, ''DC-Ops, ''decommission-hardware: decommission ms-backup1001 & ms-backup1002 - https://phabricator.wikimedia.org/T422851#11833279 (''Jclark-ctr)'
|
|
2026-04-17 11:47:11
|
<wikibugs>
|
'ops-eqiad, ''SRE, ''DC-Ops, ''decommission-hardware: decommission ms-backup1001 & ms-backup1002 - https://phabricator.wikimedia.org/T422851#11833280 (''Jclark-ctr) ''Open→''Resolved'
|
|
2026-04-17 11:47:51
|
<moritzm>
|
possibly some hardware issue, can you connect to the mgmt? it stalls for me
|
|
2026-04-17 11:48:06
|
<XioNoX>
|
moritzm: I can ssh fine to the host
|
|
2026-04-17 11:48:17
|
<moritzm>
|
the host, yes, but also the mgmt?
|
|
2026-04-17 11:48:24
|
<XioNoX>
|
dunno, haven't tried :)
|
|
2026-04-17 11:48:37
|
<moritzm>
|
can you check? otherwise I'll make a dc ops task
|
|
2026-04-17 11:49:21
|
<moritzm>
|
the server is quite old, it'll be refreshed sometime next FY
|
|
2026-04-17 11:49:25
|
<jinxer-wm>
|
FIRING: [8x] SystemdUnitFailed: check_netbox_uncommitted_dns_changes.service on netbox1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
|
|
2026-04-17 11:49:39
|
<XioNoX>
|
moritzm: the server's mgmt doesn't reply to pings
|
|
2026-04-17 11:49:44
|
<XioNoX>
|
something is fried
|
|
2026-04-17 11:50:00
|
<wikibugs>
|
('CR) ''Muehlenhoff: [C:''+2] sre.ganeti.makevm: Bump memory requirements for VMs to 2G [cookbooks] - ''https://gerrit.wikimedia.org/r/1273743 (https://phabricator.wikimedia.org/T422596) (owner: ''Muehlenhoff)'
|
|
2026-04-17 11:50:21
|
<moritzm>
|
k, I'll open a DC ops task for this so that they can check it Monday
|
|
2026-04-17 11:52:19
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P91042 and previous config saved to /var/cache/conftool/dbconfig/20260417-115218-fceratto.json
|
|
2026-04-17 11:53:00
|
<logmsgbot>
|
!log jiji@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
|
|
2026-04-17 11:53:27
|
<logmsgbot>
|
!log jiji@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
|
|
2026-04-17 11:54:25
|
<jinxer-wm>
|
FIRING: [8x] SystemdUnitFailed: check_netbox_uncommitted_dns_changes.service on netbox1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
|
|
2026-04-17 11:54:51
|
<logmsgbot>
|
!log jiji@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
|
|
2026-04-17 11:55:19
|
<logmsgbot>
|
!log jiji@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
|
|
2026-04-17 11:56:28
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops: Unreachable mgmt on ganeti1031 - https://phabricator.wikimedia.org/T423697 (''MoritzMuehlenhoff) ''NEW'
|
|
2026-04-17 11:58:47
|
<wikibugs>
|
('CR) ''Elukey: [C:''+1] "\o/" [deployment-charts] - ''https://gerrit.wikimedia.org/r/1272785 (https://phabricator.wikimedia.org/T421360) (owner: ''Effie Mouzeli)'
|
|
2026-04-17 11:59:25
|
<jinxer-wm>
|
FIRING: [8x] SystemdUnitFailed: check_netbox_uncommitted_dns_changes.service on netbox1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
|
|
2026-04-17 12:02:17
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops: Unreachable mgmt on ganeti1031 - https://phabricator.wikimedia.org/T423697#11833311 (''Jclark-ctr) ''Open→''Resolved a:''Jclark-ctr Replaced power cable. The latch tab on the RJ45 end of the management cable was
broken, and the cable had fallen out.'
|
|
2026-04-17 12:02:27
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1182 (T419961)', diff saved to https://phabricator.wikimedia.org/P91043 and previous config saved to /var/cache/conftool/dbconfig/20260417-120226-fceratto.json
|
|
2026-04-17 12:02:48
|
<logmsgbot>
|
!log fceratto@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1188.eqiad.wmnet with reason: Maintenance
|
|
2026-04-17 12:02:56
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Depooling db1188 (T419961)', diff saved to https://phabricator.wikimedia.org/P91044 and previous config saved to /var/cache/conftool/dbconfig/20260417-120255-fceratto.json
|
|
2026-04-17 12:06:29
|
<wikibugs>
|
('CR) ''Elukey: [C:''+1] "Really nice! I think it is worth to add a comment about the kubernetes min version to the README for future references." [deployment-charts] - ''https://gerrit.wikimedia.org/r/1271709 (owner: ''Dpogorzelski)'
|
|
2026-04-17 12:06:46
|
<wikibugs>
|
('PS1) ''Tiziano Fogli: thanos/compact: avoid constant Puppet changes [puppet] - ''https://gerrit.wikimedia.org/r/1273762 (https://phabricator.wikimedia.org/T386911)'
|
|
2026-04-17 12:09:25
|
<jinxer-wm>
|
FIRING: [7x] SystemdUnitFailed: netbox_ganeti_codfw_sync.service on netbox1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
|
|
2026-04-17 12:10:56
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1188 (T419961)', diff saved to https://phabricator.wikimedia.org/P91045 and previous config saved to /var/cache/conftool/dbconfig/20260417-121056-fceratto.json
|
|
2026-04-17 12:12:00
|
<wikibugs>
|
('CR) ''Tiziano Fogli: [C:''+1] puppet: remove pyrra modules/profiles [puppet] - ''https://gerrit.wikimedia.org/r/1270996 (https://phabricator.wikimedia.org/T423307) (owner: ''Herron)'
|
|
2026-04-17 12:12:11
|
<wikibugs>
|
('CR) ''Tiziano Fogli: [C:''+1] pyrra: remove pyrra/slo/slos dns entries [dns] - ''https://gerrit.wikimedia.org/r/1270995 (https://phabricator.wikimedia.org/T423307) (owner: ''Herron)'
|
|
2026-04-17 12:12:23
|
<wikibugs>
|
('CR) ''Tiziano Fogli: [C:''+1] pyrra: remove configuration for web interface [puppet] - ''https://gerrit.wikimedia.org/r/1270992 (https://phabricator.wikimedia.org/T423307) (owner: ''Herron)'
|
|
2026-04-17 12:12:35
|
<wikibugs>
|
('CR) ''Tiziano Fogli: [C:''+1] pyrra: ensure absent on package and services [puppet] - ''https://gerrit.wikimedia.org/r/1270974 (https://phabricator.wikimedia.org/T423307) (owner: ''Herron)'
|
|
2026-04-17 12:13:44
|
<wikibugs>
|
('CR) ''Tiziano Fogli: [C:''+1] Avoid false positive alerts after Ganeti master failover [puppet] - ''https://gerrit.wikimedia.org/r/1272701 (owner: ''Muehlenhoff)'
|
|
2026-04-17 12:14:25
|
<jinxer-wm>
|
RESOLVED: [3x] SystemdUnitFailed: netbox_ganeti_eqsin02_sync.service on netbox1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
|
|
2026-04-17 12:16:09
|
<wikibugs>
|
'SRE-tools, ''Infrastructure-Foundations, ''Data-Platform-SRE (2026-03-27 - 2026-04-17): debmonitor-client crashes for growthbook image - https://phabricator.wikimedia.org/T423413#11833335 (''elukey) I already had the docker-report repo checked out in my home dir on build2002, so I pulled your changes and r...'
|
|
2026-04-17 12:17:13
|
<wikibugs>
|
('CR) ''A smart kitten: Enwikinews: disable lingering FlaggedRevs template processing (''1 comment) [mediawiki-config] - ''https://gerrit.wikimedia.org/r/1271839 (https://phabricator.wikimedia.org/T423512) (owner: ''Pppery)'
|
|
2026-04-17 12:21:04
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P91046 and previous config saved to /var/cache/conftool/dbconfig/20260417-122104-fceratto.json
|
|
2026-04-17 12:25:31
|
<wikibugs>
|
('CR) ''Tiziano Fogli: "The metric mediawiki_http_requests_duration_bucket comes from the ops instance. Ideally, if possible, the best place for a recording rule " [puppet] - ''https://gerrit.wikimedia.org/r/1270480 (https://phabricator.wikimedia.org/T249663) (owner: ''Hnowlan)'
|
|
2026-04-17 12:25:56
|
<wikibugs>
|
'SRE-tools, ''Infrastructure-Foundations, ''Data-Platform-SRE (2026-03-27 - 2026-04-17): debmonitor-client crashes for growthbook image - https://phabricator.wikimedia.org/T423413#11833361 (''elukey) The docker-report run now works, but it happened the same also tonight EU time (no growthbook error reported...'
|
|
2026-04-17 12:26:43
|
<wikibugs>
|
'ops-eqiad, ''SRE, ''DC-Ops: Unresponsive management for clouddb1019.mgmt:22 - https://phabricator.wikimedia.org/T423387#11833362 (''Jclark-ctr) ''Open→''Resolved'
|
|
2026-04-17 12:29:04
|
<icinga-wm>
|
PROBLEM - MariaDB Replica Lag: m1 on db2160 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 617.46 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica
|
|
2026-04-17 12:31:12
|
<marostegui>
|
jynus: ^ a big delete happened in bacula or something? looks like it on the replica, nothing important, but just checking
|
|
2026-04-17 12:31:12
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P91047 and previous config saved to /var/cache/conftool/dbconfig/20260417-123111-fceratto.json
|
|
2026-04-17 12:31:16
|
<wikibugs>
|
('CR) ''Kamila Součková: "recheck" [deployment-charts] - ''https://gerrit.wikimedia.org/r/1201804 (https://phabricator.wikimedia.org/T388969) (owner: ''Kamila Součková)'
|
|
2026-04-17 12:32:05
|
<jynus>
|
marostegui: yeah, I was doing some cleanup
|
|
2026-04-17 12:32:11
|
<marostegui>
|
jynus: cool, thanks! noted
|
|
2026-04-17 12:32:12
|
<jynus>
|
but didn't expect it was so large
|
|
2026-04-17 12:32:16
|
<marostegui>
|
nah no issue
|
|
2026-04-17 12:32:26
|
<marostegui>
|
it won't p4ge or anything like that
|
|
2026-04-17 12:33:01
|
<jynus>
|
I think there was some corruption due to old records being stale
|
|
2026-04-17 12:33:04
|
<jynus>
|
and I am fixing that
|
|
2026-04-17 12:33:45
|
<jynus>
|
let me know if it goes too bad
|
|
2026-04-17 12:36:04
|
<icinga-wm>
|
RECOVERY - MariaDB Replica Lag: m1 on db2160 is OK: OK slave_sql_lag Replication lag: 0.27 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica
|
|
2026-04-17 12:36:53
|
<marostegui>
|
jynus: ^ wohoo
|
|
2026-04-17 12:36:59
|
<wikibugs>
|
('PS1) ''Effie Mouzeli: mw-parsoid: bump php-fpm worker number [deployment-charts] - ''https://gerrit.wikimedia.org/r/1273766'
|
|
2026-04-17 12:37:46
|
<wikibugs>
|
('CR) ''Jgiannelos: [C:''+1] mw-parsoid: bump php-fpm worker number [deployment-charts] - ''https://gerrit.wikimedia.org/r/1273766 (owner: ''Effie Mouzeli)'
|
|
2026-04-17 12:38:19
|
<wikibugs>
|
('CR) ''Effie Mouzeli: [C:''+2] mw-parsoid: bump php-fpm worker number [deployment-charts] - ''https://gerrit.wikimedia.org/r/1273766 (owner: ''Effie Mouzeli)'
|
|
2026-04-17 12:40:28
|
<wikibugs>
|
('Merged) ''jenkins-bot: mw-parsoid: bump php-fpm worker number [deployment-charts] - ''https://gerrit.wikimedia.org/r/1273766 (owner: ''Effie Mouzeli)'
|
|
2026-04-17 12:41:15
|
<logmsgbot>
|
!log jiji@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
|
|
2026-04-17 12:41:21
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1188 (T419961)', diff saved to https://phabricator.wikimedia.org/P91048 and previous config saved to /var/cache/conftool/dbconfig/20260417-124120-fceratto.json
|
|
2026-04-17 12:41:41
|
<logmsgbot>
|
!log jiji@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
|
|
2026-04-17 12:41:42
|
<logmsgbot>
|
!log fceratto@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1197.eqiad.wmnet with reason: Maintenance
|
|
2026-04-17 12:41:50
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Depooling db1197 (T419961)', diff saved to https://phabricator.wikimedia.org/P91049 and previous config saved to /var/cache/conftool/dbconfig/20260417-124149-fceratto.json
|
|
2026-04-17 12:43:07
|
<jinxer-wm>
|
FIRING: [4x] ProbeDown: Service aqs1010-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
|
|
2026-04-17 12:45:32
|
<wikibugs>
|
('PS2) ''Ottomata: flink-app - default to setting metrics.internal.query-service.port [deployment-charts] - ''https://gerrit.wikimedia.org/r/1268071 (https://phabricator.wikimedia.org/T421216)'
|
|
2026-04-17 12:45:47
|
<wikibugs>
|
('CR) ''Ottomata: flink-app - default to setting metrics.internal.query-service.port (''2 comments) [deployment-charts] - ''https://gerrit.wikimedia.org/r/1268071 (https://phabricator.wikimedia.org/T421216) (owner: ''Ottomata)'
|
|
2026-04-17 12:46:45
|
<wikibugs>
|
('PS4) ''Arnaudb: gerrit: update sync-instances cookbook [cookbooks] - ''https://gerrit.wikimedia.org/r/1270863 (https://phabricator.wikimedia.org/T333143)'
|
|
2026-04-17 12:47:45
|
<jinxer-wm>
|
FIRING: WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure
|
|
2026-04-17 12:50:10
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1197 (T419961)', diff saved to https://phabricator.wikimedia.org/P91050 and previous config saved to /var/cache/conftool/dbconfig/20260417-125009-fceratto.json
|
|
2026-04-17 12:50:55
|
<wikibugs>
|
('PS3) ''A smart kitten: enwikinews: Move override for $wgFlaggedRevsHandleIncludes to InitialiseSettings.php [mediawiki-config] - ''https://gerrit.wikimedia.org/r/1273763 (https://phabricator.wikimedia.org/T423512)'
|
|
2026-04-17 12:51:38
|
<wikibugs>
|
('CR) ''Lucas Werkmeister (WMDE): [C:''+1] enwikinews: Move override for $wgFlaggedRevsHandleIncludes to InitialiseSettings.php [mediawiki-config] - ''https://gerrit.wikimedia.org/r/1273763 (https://phabricator.wikimedia.org/T423512) (owner: ''A smart kitten)'
|
|
2026-04-17 12:51:58
|
<wikibugs>
|
('CR) ''A smart kitten: "Thanks again for your investigations @lucas.werkmeister@wikimedia.de :)" [mediawiki-config] - ''https://gerrit.wikimedia.org/r/1273763 (https://phabricator.wikimedia.org/T423512) (owner: ''A smart kitten)'
|
|
2026-04-17 12:52:43
|
<wikibugs>
|
('PS1) ''Marostegui: mariadb: Productionize clouddb1032 [puppet] - ''https://gerrit.wikimedia.org/r/1273769 (https://phabricator.wikimedia.org/T409557)'
|
|
2026-04-17 12:52:49
|
<marostegui>
|
dhinus: ^
|
|
2026-04-17 12:53:11
|
<wikibugs>
|
('CR) ''Marostegui: "Initial patch to be able to start the cloning" [puppet] - ''https://gerrit.wikimedia.org/r/1273769 (https://phabricator.wikimedia.org/T409557) (owner: ''Marostegui)'
|
|
2026-04-17 12:53:21
|
<wikibugs>
|
('CR) ''A smart kitten: Enwikinews: disable lingering FlaggedRevs template processing (''1 comment) [mediawiki-config] - ''https://gerrit.wikimedia.org/r/1271839 (https://phabricator.wikimedia.org/T423512) (owner: ''Pppery)'
|
|
2026-04-17 12:53:37
|
<wikibugs>
|
('PS3) ''Dpogorzelski: knative-serving: update chart to 1.21.1 [deployment-charts] - ''https://gerrit.wikimedia.org/r/1271709'
|
|
2026-04-17 12:54:22
|
<wikibugs>
|
('CR) ''Dpogorzelski: knative-serving: update chart to 1.21.1 (''1 comment) [deployment-charts] - ''https://gerrit.wikimedia.org/r/1271709 (owner: ''Dpogorzelski)'
|
|
2026-04-17 12:54:54
|
<logmsgbot>
|
!log fceratto@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2150.codfw.wmnet with reason: Maintenance
|
|
2026-04-17 12:55:02
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Depooling db2150 (T419635)', diff saved to https://phabricator.wikimedia.org/P91051 and previous config saved to /var/cache/conftool/dbconfig/20260417-125501-fceratto.json
|
|
2026-04-17 12:55:06
|
<stashbot>
|
T419635: Drop il_to column from imagelinks table in wmf production - https://phabricator.wikimedia.org/T419635
|
|
2026-04-17 12:55:22
|
<dhinus>
|
marostegui: wait, clouddb1032 has not arrived yet, has it?
|
|
2026-04-17 12:55:49
|
<marostegui>
|
dhinus: you are right, I have no idea why I did 32 instead of 24
|
|
2026-04-17 12:55:51
|
<marostegui>
|
let me ammend
|
|
2026-04-17 12:55:56
|
<dhinus>
|
ah ok :D
|
|
2026-04-17 12:56:36
|
<dhinus>
|
btw when is the x4 split going to happen in prod?
|
|
2026-04-17 12:56:48
|
<marostegui>
|
dhinus: I am not sure yet
|
|
2026-04-17 12:56:49
|
<marostegui>
|
Amir1: ^
|
|
2026-04-17 12:57:14
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db2150 (T419635)', diff saved to https://phabricator.wikimedia.org/P91052 and previous config saved to /var/cache/conftool/dbconfig/20260417-125714-fceratto.json
|
|
2026-04-17 12:57:21
|
<dhinus>
|
let's move to -data-persistence :)
|
|
2026-04-17 12:58:21
|
<wikibugs>
|
('CR) ''Kamila Součková: "@jmeybohm@wikimedia.org it passed! I even ran it a 2nd time because I couldn't believe it!" [deployment-charts] - ''https://gerrit.wikimedia.org/r/1201804 (https://phabricator.wikimedia.org/T388969) (owner: ''Kamila Součková)'
|
|
2026-04-17 12:58:51
|
<wikibugs>
|
'SRE-tools, ''Infrastructure-Foundations, ''Data-Platform-SRE (2026-03-27 - 2026-04-17): debmonitor-client crashes for growthbook image - https://phabricator.wikimedia.org/T423413#11833429 (''brouberol) I had modified the script on disk with `PYTHONPATH=''` to run the whole command, to ensure our change wou...'
|
|
2026-04-17 12:59:17
|
<wikibugs>
|
'SRE-tools, ''Infrastructure-Foundations, ''Data-Platform-SRE (2026-03-27 - 2026-04-17): debmonitor-client crashes for growthbook image - https://phabricator.wikimedia.org/T423413#11833432 (''brouberol) Thank you @elukey for your assistance in the review, build and release process! I think we can now close...'
|
|
2026-04-17 13:00:18
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P91053 and previous config saved to /var/cache/conftool/dbconfig/20260417-130018-fceratto.json
|
|
2026-04-17 13:00:37
|
<logmsgbot>
|
!log jiji@cumin1003 START - Cookbook sre.ganeti.reboot-vm for VM wikikube-worker-exp1001.eqiad.wmnet
|
|
2026-04-17 13:01:22
|
<Lucas_WMDE>
|
fabfur, tappof: I would like to do an emergency deploy for https://gerrit.wikimedia.org/r/1273763 (unbreak some enwikinews things, see https://phabricator.wikimedia.org/T423512#11827523 for impact); I’m not sure if it qualifies as an emergency, but as I already tested the fix on mw-experimental I think the risk should also be fairly low. Are SRE
|
|
2026-04-17 13:01:22
|
<Lucas_WMDE>
|
okay with deployment? (cc thcipriani, dduvall, A_smart_kitten)
|
|
2026-04-17 13:01:23
|
<wikibugs>
|
('Abandoned) ''Marostegui: mariadb: Productionize clouddb1032 [puppet] - ''https://gerrit.wikimedia.org/r/1273769 (https://phabricator.wikimedia.org/T409557) (owner: ''Marostegui)'
|
|
2026-04-17 13:01:35
|
<wikibugs>
|
'SRE-tools, ''Infrastructure-Foundations, ''Data-Platform-SRE (2026-03-27 - 2026-04-17): debmonitor-client crashes for growthbook image - https://phabricator.wikimedia.org/T423413#11833440 (''elukey) ''In progress→''Resolved Thank you for
the code fixes!'
|
|
2026-04-17 13:01:47
|
<Lucas_WMDE>
|
(also, A_smart_kitten, do you know how to test the change via the web, beyond checking the value of the $wg variable? ^^)
|
|
2026-04-17 13:02:43
|
<Lucas_WMDE>
|
(I *think* I’m actually seeing latest template versions on the main page, but I might be missing something)
|
|
2026-04-17 13:02:45
|
<jinxer-wm>
|
FIRING: [4x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure
|
|
2026-04-17 13:03:08
|
<A_smart_kitten>
|
Lucas_WMDE: not currently, checking the variable on the shell side of things would currently be my idea for how to test this if a deployment happens. i'll read some discussions to see if i can find something that isn't currently working that might be testable
|
|
2026-04-17 13:03:09
|
<wikibugs>
|
('CR) ''Kamila Součková: [C:''+2] Revert "shellbox: Setup shellbox-icu72" [deployment-charts] - ''https://gerrit.wikimedia.org/r/1270557 (https://phabricator.wikimedia.org/T422546) (owner: ''Kamila Součková)'
|
|
2026-04-17 13:03:31
|
<Lucas_WMDE>
|
the edit summary at https://en.wikinews.org/w/index.php?title=Template:Lead_article_1&action=history looks like the main page issue may have been worked around
|
|
2026-04-17 13:03:36
|
<A_smart_kitten>
|
i believe the main page issue specifically was fixed by a deletion & recreation of the templates (or something like that), xref https://phabricator.wikimedia.org/T423512#11827469
|
|
2026-04-17 13:03:48
|
<Lucas_WMDE>
|
yeah
|
|
2026-04-17 13:05:10
|
<wikibugs>
|
('Merged) ''jenkins-bot: Revert "shellbox: Setup shellbox-icu72" [deployment-charts] - ''https://gerrit.wikimedia.org/r/1270557 (https://phabricator.wikimedia.org/T422546) (owner: ''Kamila Součková)'
|
|
2026-04-17 13:06:25
|
<jinxer-wm>
|
RESOLVED: [2x] SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
|
|
2026-04-17 13:07:02
|
<logmsgbot>
|
!log jiji@cumin1003 END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM wikikube-worker-exp1001.eqiad.wmnet
|
|
2026-04-17 13:07:23
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db2150', diff saved to https://phabricator.wikimedia.org/P91054 and previous config saved to /var/cache/conftool/dbconfig/20260417-130722-fceratto.json
|
|
2026-04-17 13:07:51
|
<jinxer-wm>
|
FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr2-codfw:et-0/1/4 (Transport: cr2-eqiad:et-1/1/5 (Lumen, 449169461) {#3909}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown
|
|
2026-04-17 13:08:16
|
<A_smart_kitten>
|
Lucas_WMDE: okay, may(TM) be testable with https://en.wikinews.org/wiki/Tour_de_France:_Alberto_Contador_wins_the_grand_tour -- currently all entries in that infobox display a date in february 2025, but https://en.wikinews.org/wiki/Template:Tour_de_France_2007 seems to have since been updated with new dates
|
|
2026-04-17 13:08:34
|
<A_smart_kitten>
|
(unless that is as a result of something else. in which case, who knows)
|
|
2026-04-17 13:08:42
|
<jinxer-wm>
|
FIRING: [2x] SystemdUnitFailed: wmf_auto_restart_prometheus-blazegraph-exporter-wdqs-blazegraph.service on wdqs1012:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
|
|
2026-04-17 13:09:20
|
<wikibugs>
|
('CR) ''Arnaudb: [C:''+1] "thanks for the rebase! that makes total sense." [puppet] - ''https://gerrit.wikimedia.org/r/1270774 (https://phabricator.wikimedia.org/T423027) (owner: ''Arnaudb)'
|
|
2026-04-17 13:09:41
|
<Lucas_WMDE>
|
A_smart_kitten: sounds good, let’s test that in mw-experimental
|
|
2026-04-17 13:09:56
|
<wikibugs>
|
'ops-eqiad, ''SRE, ''DC-Ops, ''Data-Platform-SRE (2026-03-27 - 2026-04-17): Degraded RAID on an-worker1205 - https://phabricator.wikimedia.org/T422317#11833474 (''Jclark-ctr) @brouberol Drive has been Swapped Thanks!'
|
|
2026-04-17 13:10:27
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P91055 and previous config saved to /var/cache/conftool/dbconfig/20260417-131026-fceratto.json
|
|
2026-04-17 13:10:41
|
<Lucas_WMDE>
|
mw-experimental-eqiad updated
|
|
2026-04-17 13:11:15
|
<Lucas_WMDE>
|
hm, still shows 2025-02-17 dates even after a purge with XWD…
|
|
2026-04-17 13:11:21
|
<wikibugs>
|
'ops-eqiad, ''SRE, ''DBA, ''DC-Ops: Degraded RAID on pc1011 - https://phabricator.wikimedia.org/T423630#11833478 (''Jclark-ctr) a:''Jclark-ctr Solid State Disk 0:1:6 Removed 6 1787.88 GB SATA'
|
|
2026-04-17 13:11:41
|
<A_smart_kitten>
|
Lucas_WMDE: i'm gonna try special:purge connected to experimental-eqiad
|
|
2026-04-17 13:11:43
|
<wikibugs>
|
'ops-eqiad, ''SRE, ''DBA, ''DC-Ops: Degraded RAID on pc1011 - https://phabricator.wikimedia.org/T423630#11833480 (''Marostegui) Thank you!'
|
|
2026-04-17 13:12:15
|
<A_smart_kitten>
|
WORKS (i think)
|
|
2026-04-17 13:12:30
|
<A_smart_kitten>
|
Lucas_WMDE: can you confirm if you see the new date now?
|
|
2026-04-17 13:12:32
|
<wikibugs>
|
('PS1) ''Marostegui: site.pp: Move clouddb1024 to analytics [puppet] - ''https://gerrit.wikimedia.org/r/1273777 (https://phabricator.wikimedia.org/T409557)'
|
|
2026-04-17 13:12:44
|
<Lucas_WMDE>
|
yup
|
|
2026-04-17 13:12:48
|
<Lucas_WMDE>
|
what did you do differently :o
|
|
2026-04-17 13:12:53
|
<Lucas_WMDE>
|
teach me your magic
|
|
2026-04-17 13:13:22
|
<Lucas_WMDE>
|
also TIL Special:Purge
|
|
2026-04-17 13:13:34
|
<A_smart_kitten>
|
just https://en.wikinews.org/wiki/special:purge/Tour_de_France:_Alberto_Contador_wins_the_grand_tour connected to k8s-mw-experimental-eqiad using WikimediaDebug i think
|
|
2026-04-17 13:13:44
|
<Lucas_WMDE>
|
I’m trying another purge without WikimediaDebug to see if it restores the breakage
|
|
2026-04-17 13:13:49
|
<Lucas_WMDE>
|
it does
|
|
2026-04-17 13:14:19
|
<Lucas_WMDE>
|
okay, and now I also got it back into the fixed state
|
|
2026-04-17 13:14:23
|
<Lucas_WMDE>
|
with WikimediaDebug and that backend
|
|
2026-04-17 13:14:28
|
<Lucas_WMDE>
|
but only after a few extra reloads after the purge
|
|
2026-04-17 13:14:28
|
<Lucas_WMDE>
|
o_O
|
|
2026-04-17 13:14:52
|
<wikibugs>
|
'ops-eqiad, ''SRE, ''DBA, ''DC-Ops: Degraded RAID on pc1011 - https://phabricator.wikimedia.org/T423630#11833487 (''Jclark-ctr) @Marostegui drive has been swapped'
|
|
2026-04-17 13:14:53
|
<Lucas_WMDE>
|
anyway, the change is testable, that’s good. so we would just need SRE approval for the deploy
|
|
2026-04-17 13:15:11
|
<A_smart_kitten>
|
ack
|
|
2026-04-17 13:16:33
|
<wikibugs>
|
'ops-eqiad, ''SRE, ''DBA, ''DC-Ops: Degraded RAID on pc1011 - https://phabricator.wikimedia.org/T423630#11833496 (''Marostegui) Rebuilding: ` Raw Size: 1.746 TB [0xdf8fe2b0 Sectors] Firmware state: =====> Rebuild <===== `'
|
|
2026-04-17 13:16:38
|
<A_smart_kitten>
|
at least we now seem to have confirmed that this might be the fix, whether or not a today-deploy is available :)
|
|
2026-04-17 13:17:31
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db2150', diff saved to https://phabricator.wikimedia.org/P91056 and previous config saved to /var/cache/conftool/dbconfig/20260417-131730-fceratto.json
|
|
2026-04-17 13:17:35
|
<wikibugs>
|
('PS2) ''Dpogorzelski: kserve-resources: fix securityContext propagation [deployment-charts] - ''https://gerrit.wikimedia.org/r/1273664 (https://phabricator.wikimedia.org/T423149)'
|
|
2026-04-17 13:18:39
|
<wikibugs>
|
('PS1) ''Ottomata: html-enrich - reduce checkpoint timeout [deployment-charts] - ''https://gerrit.wikimedia.org/r/1273779 (https://phabricator.wikimedia.org/T421216)'
|
|
2026-04-17 13:20:06
|
<tappof>
|
Lucas_WMDE: A_smart_kitten We're checking if it's okay to proceed. We'll let you know.
|
|
2026-04-17 13:20:35
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1197 (T419961)', diff saved to https://phabricator.wikimedia.org/P91057 and previous config saved to /var/cache/conftool/dbconfig/20260417-132034-fceratto.json
|
|
2026-04-17 13:20:56
|
<logmsgbot>
|
!log fceratto@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1225.eqiad.wmnet with reason: Maintenance
|
|
2026-04-17 13:21:04
|
<wikibugs>
|
('PS1) ''Dreamy Jazz: maintain-views: Hide blocks with bl_deleted set to 2 [puppet] - ''https://gerrit.wikimedia.org/r/1273781 (https://phabricator.wikimedia.org/T414188)'
|
|
2026-04-17 13:21:10
|
<Lucas_WMDE>
|
ack, thanks!
|
|
2026-04-17 13:22:02
|
<logmsgbot>
|
!log jiji@cumin1003 START - Cookbook sre.ganeti.reboot-vm for VM wikikube-worker-exp2001.codfw.wmnet
|
|
2026-04-17 13:22:45
|
<jinxer-wm>
|
FIRING: [4x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure
|
|
2026-04-17 13:24:25
|
<wikibugs>
|
('CR) ''FNegri: [C:''+1] site.pp: Move clouddb1024 to analytics [puppet] - ''https://gerrit.wikimedia.org/r/1273777 (https://phabricator.wikimedia.org/T409557) (owner: ''Marostegui)'
|
|
2026-04-17 13:24:59
|
<wikibugs>
|
('CR) ''Marostegui: [C:''+2] site.pp: Move clouddb1024 to analytics [puppet] - ''https://gerrit.wikimedia.org/r/1273777 (https://phabricator.wikimedia.org/T409557) (owner: ''Marostegui)'
|
|
2026-04-17 13:26:02
|
<logmsgbot>
|
!log jiji@cumin1003 END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM wikikube-worker-exp2001.codfw.wmnet
|
|
2026-04-17 13:26:20
|
<logmsgbot>
|
!log fceratto@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1229.eqiad.wmnet with reason: Maintenance
|
|
2026-04-17 13:26:29
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Depooling db1229 (T419961)', diff saved to https://phabricator.wikimedia.org/P91058 and previous config saved to /var/cache/conftool/dbconfig/20260417-132628-fceratto.json
|
|
2026-04-17 13:26:34
|
<logmsgbot>
|
!log jiji@deploy1003 helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
|
|
2026-04-17 13:27:16
|
<wikibugs>
|
'ops-eqiad, ''SRE, ''DC-Ops, ''Data-Platform-SRE (2026-03-27 - 2026-04-17): Degraded RAID on an-worker1205 - https://phabricator.wikimedia.org/T422317#11833527 (''brouberol) Thank you!'
|
|
2026-04-17 13:27:23
|
<logmsgbot>
|
!log jiji@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
|
|
2026-04-17 13:27:39
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db2150 (T419635)', diff saved to https://phabricator.wikimedia.org/P91059 and previous config saved to /var/cache/conftool/dbconfig/20260417-132738-fceratto.json
|
|
2026-04-17 13:27:43
|
<stashbot>
|
T419635: Drop il_to column from imagelinks table in wmf production - https://phabricator.wikimedia.org/T419635
|
|
2026-04-17 13:27:56
|
<logmsgbot>
|
!log fceratto@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2159.codfw.wmnet with reason: Maintenance
|
|
2026-04-17 13:28:03
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Depooling db2159 (T419635)', diff saved to https://phabricator.wikimedia.org/P91060 and previous config saved to /var/cache/conftool/dbconfig/20260417-132802-fceratto.json
|
|
2026-04-17 13:28:05
|
<sobanski>
|
Lucas_WMDE: re. "not sure if it qualifies as an emergency" would it be a big problem if we waited until Monday?
|
|
2026-04-17 13:29:01
|
<wikibugs>
|
('PS1) ''Marostegui: cloudb1024: Add s6 [puppet] - ''https://gerrit.wikimedia.org/r/1273785 (https://phabricator.wikimedia.org/T409557)'
|
|
2026-04-17 13:29:15
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db2159 (T419635)', diff saved to https://phabricator.wikimedia.org/P91061 and previous config saved to /var/cache/conftool/dbconfig/20260417-132914-fceratto.json
|
|
2026-04-17 13:29:23
|
<Lucas_WMDE>
|
sobanski: I don’t think so, though I don’t fully understand the impact of the task
|
|
2026-04-17 13:29:35
|
<wikibugs>
|
('CR) ''Marostegui: "Won't be submitted today, will wait for the day of the announcement." [puppet] - ''https://gerrit.wikimedia.org/r/1273785 (https://phabricator.wikimedia.org/T409557) (owner: ''Marostegui)'
|
|
2026-04-17 13:29:44
|
<Lucas_WMDE>
|
it sounds like they worked around the issue for the main page, but the rest of the wiki is still affected
|
|
2026-04-17 13:30:03
|
<Lucas_WMDE>
|
and they don’t have a ton of time left before the wiki closes, so I thought it would be nice if we could unbreak it before the weekend
|
|
2026-04-17 13:30:05
|
<wikibugs>
|
('CR) ''Marostegui: "@fnegri@wikimedia.org when could I stop 1015 to clone this one?" [puppet] - ''https://gerrit.wikimedia.org/r/1273785 (https://phabricator.wikimedia.org/T409557) (owner: ''Marostegui)'
|
|
2026-04-17 13:32:45
|
<jinxer-wm>
|
FIRING: [4x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure
|
|
2026-04-17 13:33:04
|
<wikibugs>
|
('PS1) ''Jforrester: ImageListPager: Make sure file and filerevision are in correct order [core] (wmf/1.46.0-wmf.24) - ''https://gerrit.wikimedia.org/r/1273787 (https://phabricator.wikimedia.org/T423654)'
|
|
2026-04-17 13:33:27
|
<jinxer-wm>
|
RESOLVED: [2x] SystemdUnitFailed: wmf_auto_restart_prometheus-blazegraph-exporter-wdqs-blazegraph.service on wdqs1012:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
|
|
2026-04-17 13:34:00
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1229 (T419961)', diff saved to https://phabricator.wikimedia.org/P91062 and previous config saved to /var/cache/conftool/dbconfig/20260417-133359-fceratto.json
|
|
2026-04-17 13:35:16
|
<wikibugs>
|
('CR) ''Ottomata: [C:''+2] html-enrich - reduce checkpoint timeout [deployment-charts] - ''https://gerrit.wikimedia.org/r/1273779 (https://phabricator.wikimedia.org/T421216) (owner: ''Ottomata)'
|
|
2026-04-17 13:37:16
|
<wikibugs>
|
('Merged) ''jenkins-bot: html-enrich - reduce checkpoint timeout [deployment-charts] - ''https://gerrit.wikimedia.org/r/1273779 (https://phabricator.wikimedia.org/T421216) (owner: ''Ottomata)'
|
|
2026-04-17 13:37:47
|
<wikibugs>
|
('Abandoned) ''Jcrespo: firewall: Update firewall definitions for mediabackups to Puppet 7 syntax [puppet] - ''https://gerrit.wikimedia.org/r/1211145 (https://phabricator.wikimedia.org/T349619) (owner: ''Jcrespo)'
|
|
2026-04-17 13:39:23
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P91063 and previous config saved to /var/cache/conftool/dbconfig/20260417-133923-fceratto.json
|
|
2026-04-17 13:39:44
|
<wikibugs>
|
'SRE, ''Ganeti, ''Infrastructure-Foundations: Migrating eqsin to routed Ganeti - https://phabricator.wikimedia.org/T421863#11833549 (''MoritzMuehlenhoff)'
|
|
2026-04-17 13:40:06
|
<wikibugs>
|
'SRE-tools, ''bacula, ''Data-Persistence-Backup, ''Infrastructure-Foundations, and 2 others: netbox2003 backups (maybe others?) are missconfigured or failing to find the backup directory - https://phabricator.wikimedia.org/T423689#11833550 (''ayounsi) We're not doing Netbox CSV dumps anymore. So you can r...'
|
|
2026-04-17 13:42:05
|
<wikibugs>
|
('PS1) ''Muehlenhoff: Depool puppetserver1002 [dns] - ''https://gerrit.wikimedia.org/r/1273788 (https://phabricator.wikimedia.org/T423282)'
|
|
2026-04-17 13:42:29
|
<inflatador>
|
!log bking@apt1002 sudo -E reprepro -C component/opensearch2 include trixie-wikimedia /home/bking/wmf-opensearch-search-plugins-2.19.5+5-trixie/wmf-opensearch-search-plugins_2.19.5+5_amd64.changes
|
|
2026-04-17 13:42:30
|
<stashbot>
|
Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
|
|
2026-04-17 13:42:48
|
<jinxer-wm>
|
FIRING: PuppetFailure: Puppet has failed on cirrussearch1113:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
|
|
2026-04-17 13:42:54
|
<icinga-wm>
|
PROBLEM - MariaDB Replica Lag: m1 on db2232 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 626.33 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica
|
|
2026-04-17 13:43:04
|
<icinga-wm>
|
PROBLEM - MariaDB Replica Lag: m1 on db2160 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 637.21 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica
|
|
2026-04-17 13:43:09
|
<icinga-wm>
|
PROBLEM - MariaDB Replica Lag: m1 on db1217 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 640.05 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica
|
|
2026-04-17 13:43:26
|
<wikibugs>
|
('CR) ''Muehlenhoff: [C:''+2] Depool puppetserver1002 [dns] - ''https://gerrit.wikimedia.org/r/1273788 (https://phabricator.wikimedia.org/T423282) (owner: ''Muehlenhoff)'
|
|
2026-04-17 13:43:33
|
<logmsgbot>
|
!log jmm@dns1004 START - running authdns-update
|
|
2026-04-17 13:44:08
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P91064 and previous config saved to /var/cache/conftool/dbconfig/20260417-134408-fceratto.json
|
|
2026-04-17 13:44:32
|
<wikibugs>
|
'SRE, ''Infrastructure-Foundations, ''Puppet-Infrastructure, ''Patch-For-Review: Timeouts on puppetserver1002 past reboot - https://phabricator.wikimedia.org/T423282#11833559 (''MoritzMuehlenhoff) ''Resolved→''Open This
started again and I've just depooled 1002 again.'
|
|
2026-04-17 13:44:59
|
<logmsgbot>
|
!log jmm@dns1004 END - running authdns-update
|
|
2026-04-17 13:45:11
|
<wikibugs>
|
('PS3) ''Dpogorzelski: kserve-resources: fix securityContext propagation [deployment-charts] - ''https://gerrit.wikimedia.org/r/1273664 (https://phabricator.wikimedia.org/T423149)'
|
|
2026-04-17 13:45:55
|
<wikibugs>
|
('PS1) ''Bking: Revert "opensearch: strip bundled plugins before WMF pkg" [puppet] - ''https://gerrit.wikimedia.org/r/1273789'
|
|
2026-04-17 13:46:05
|
<wikibugs>
|
('CR) ''Bking: [V:''+2 C:''+2] Revert "opensearch: strip bundled plugins before WMF pkg" [puppet] - ''https://gerrit.wikimedia.org/r/1273789 (owner: ''Bking)'
|
|
2026-04-17 13:46:54
|
<icinga-wm>
|
RECOVERY - MariaDB Replica Lag: m1 on db2232 is OK: OK slave_sql_lag Replication lag: 0.36 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica
|
|
2026-04-17 13:47:15
|
<wikibugs>
|
('CR) ''CI reject: [V:''-1] kserve-resources: fix securityContext propagation [deployment-charts] - ''https://gerrit.wikimedia.org/r/1273664 (https://phabricator.wikimedia.org/T423149) (owner: ''Dpogorzelski)'
|
|
2026-04-17 13:47:45
|
<jinxer-wm>
|
FIRING: [4x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure
|
|
2026-04-17 13:48:23
|
<fabfur>
|
^^ waiting for clients resolving new address
|
|
2026-04-17 13:49:31
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P91065 and previous config saved to /var/cache/conftool/dbconfig/20260417-134930-fceratto.json
|
|
2026-04-17 13:50:08
|
<icinga-wm>
|
RECOVERY - MariaDB Replica Lag: m1 on db1217 is OK: OK slave_sql_lag Replication lag: 0.02 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica
|
|
2026-04-17 13:50:31
|
<A_smart_kitten>
|
i've probably got to go at some point shortly (~5/10mins?) - if a deploy is given the go-ahead while i'm gone, Lucas_WMDE: you are free to deploy my patch while i'm gone if you feel comfortable doing so, if not then no worries & i'll schedule it for monday
|
|
2026-04-17 13:50:35
|
<wikibugs>
|
('CR) ''Dpogorzelski: [V:''+2 C:''+2] kserve-resources: fix securityContext propagation [deployment-charts] - ''https://gerrit.wikimedia.org/r/1273664 (https://phabricator.wikimedia.org/T423149) (owner: ''Dpogorzelski)'
|
|
2026-04-17 13:50:47
|
<Lucas_WMDE>
|
ack
|
|
2026-04-17 13:50:59
|
<Lucas_WMDE>
|
thanks!
|
|
2026-04-17 13:51:03
|
<A_smart_kitten>
|
np :)
|
|
2026-04-17 13:51:53
|
<wikibugs>
|
('PS3) ''Elukey: istio: revisit Prometheus buckets for Wikikube [deployment-charts] - ''https://gerrit.wikimedia.org/r/1269998 (https://phabricator.wikimedia.org/T392886)'
|
|
2026-04-17 13:52:29
|
<logmsgbot>
|
!log otto@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply
|
|
2026-04-17 13:52:33
|
<wikibugs>
|
('CR) ''Elukey: istio: revisit Prometheus buckets for Wikikube (''1 comment) [deployment-charts] - ''https://gerrit.wikimedia.org/r/1269998 (https://phabricator.wikimedia.org/T392886) (owner: ''Elukey)'
|
|
2026-04-17 13:52:34
|
<logmsgbot>
|
!log otto@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply
|
|
2026-04-17 13:52:45
|
<jinxer-wm>
|
FIRING: [4x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure
|
|
2026-04-17 13:54:17
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P91066 and previous config saved to /var/cache/conftool/dbconfig/20260417-135416-fceratto.json
|
|
2026-04-17 13:54:33
|
<fabfur>
|
!log restarting varnish on cp3066 to clear alerts
|
|
2026-04-17 13:54:35
|
<stashbot>
|
Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
|
|
2026-04-17 13:54:47
|
<logmsgbot>
|
!log fabfur@cumin1003 START - Cookbook sre.cdn.roll-restart-varnish rolling restart of Varnish on 1 hosts matching query P{cp3066.*}
|
|
2026-04-17 13:54:49
|
<wikibugs>
|
('PS1) ''Muehlenhoff: Remove puppetmaster::gitpuppet [puppet] - ''https://gerrit.wikimedia.org/r/1273790 (https://phabricator.wikimedia.org/T365798)'
|
|
2026-04-17 13:55:51
|
<wikibugs>
|
('CR) ''Dpogorzelski: [C:''+2] knative-serving: update chart to 1.21.1 [deployment-charts] - ''https://gerrit.wikimedia.org/r/1271709 (owner: ''Dpogorzelski)'
|
|
2026-04-17 13:55:55
|
<wikibugs>
|
('PS1) ''Jcrespo: netbox: Remove backups from netbox server, only leave postgres ones [puppet] - ''https://gerrit.wikimedia.org/r/1273791 (https://phabricator.wikimedia.org/T423689)'
|
|
2026-04-17 13:56:23
|
<wikibugs>
|
('CR) ''Andrew Bogott: [C:''-1] Openstack: use debian.net repo rather than the wmf-hosted repo (''1 comment) [puppet] - ''https://gerrit.wikimedia.org/r/1272837 (https://phabricator.wikimedia.org/T423598) (owner: ''Andrew Bogott)'
|
|
2026-04-17 13:56:55
|
<wikibugs>
|
'SRE-tools, ''bacula, ''Data-Persistence-Backup, ''Infrastructure-Foundations, and 3 others: netbox2003 backups (maybe others?) are missconfigured or failing to find the backup directory - https://phabricator.wikimedia.org/T423689#11833584 (''jcrespo) That was the only thing being backed up. ` bacula:...'
|
|
2026-04-17 13:57:05
|
<wikibugs>
|
('PS1) ''Muehlenhoff: Remove obsolete Hiera file [puppet] - ''https://gerrit.wikimedia.org/r/1273792 (https://phabricator.wikimedia.org/T365798)'
|
|
2026-04-17 13:57:19
|
<logmsgbot>
|
!log fabfur@cumin1003 END (PASS) - Cookbook sre.cdn.roll-restart-varnish (exit_code=0) rolling restart of Varnish on 1 hosts matching query P{cp3066.*}
|
|
2026-04-17 13:57:45
|
<jinxer-wm>
|
FIRING: [4x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure
|
|
2026-04-17 13:57:55
|
<wikibugs>
|
('PS4) ''Dpogorzelski: knative-serving: update chart to 1.21.1 [deployment-charts] - ''https://gerrit.wikimedia.org/r/1271709'
|
|
2026-04-17 13:58:19
|
<wikibugs>
|
('CR) ''Dpogorzelski: [V:''+2 C:''+2] knative-serving: update chart to 1.21.1 [deployment-charts] - ''https://gerrit.wikimedia.org/r/1271709 (owner: ''Dpogorzelski)'
|
|
2026-04-17 13:58:31
|
<wikibugs>
|
('PS4) ''Andrew Bogott: Openstack: use debian.net repo rather than the wmf-hosted repo [puppet] - ''https://gerrit.wikimedia.org/r/1272837 (https://phabricator.wikimedia.org/T423598)'
|
|
2026-04-17 13:58:31
|
<wikibugs>
|
('CR) ''Ayounsi: [C:''+1] "thanks!" [puppet] - ''https://gerrit.wikimedia.org/r/1273791 (https://phabricator.wikimedia.org/T423689) (owner: ''Jcrespo)'
|
|
2026-04-17 13:58:46
|
<logmsgbot>
|
!log fabfur@cumin1003 START - Cookbook sre.cdn.roll-restart-varnish rolling restart of Varnish on 1 hosts matching query P{cp3069.*}
|
|
2026-04-17 13:59:06
|
<wikibugs>
|
('CR) ''CI reject: [V:''-1] Openstack: use debian.net repo rather than the wmf-hosted repo [puppet] - ''https://gerrit.wikimedia.org/r/1272837 (https://phabricator.wikimedia.org/T423598) (owner: ''Andrew Bogott)'
|
|
2026-04-17 13:59:39
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db2159 (T419635)', diff saved to https://phabricator.wikimedia.org/P91067 and previous config saved to /var/cache/conftool/dbconfig/20260417-135938-fceratto.json
|
|
2026-04-17 13:59:45
|
<stashbot>
|
T419635: Drop il_to column from imagelinks table in wmf production - https://phabricator.wikimedia.org/T419635
|
|
2026-04-17 13:59:54
|
<logmsgbot>
|
!log dpogorzelski@deploy1003 helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'.
|
|
2026-04-17 13:59:56
|
<logmsgbot>
|
!log fceratto@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2168.codfw.wmnet with reason: Maintenance
|
|
2026-04-17 14:00:04
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Depooling db2168 (T419635)', diff saved to https://phabricator.wikimedia.org/P91068 and previous config saved to /var/cache/conftool/dbconfig/20260417-140003-fceratto.json
|
|
2026-04-17 14:00:20
|
<fabfur>
|
!log restart varnish on cp3069, cp3070, cp3072, cp3073 to clear alerts
|
|
2026-04-17 14:00:22
|
<stashbot>
|
Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
|
|
2026-04-17 14:00:28
|
<wikibugs>
|
'SRE-tools, ''bacula, ''Data-Persistence-Backup, ''Infrastructure-Foundations, and 3 others: netbox2003 backups (maybe others?) are missconfigured or failing to find the backup directory - https://phabricator.wikimedia.org/T423689#11833590 (''ayounsi) Yeah, Postgres is where all the data are. So +1 to not...'
|
|
2026-04-17 14:00:31
|
<logmsgbot>
|
!log fabfur@cumin1003 END (PASS) - Cookbook sre.cdn.roll-restart-varnish (exit_code=0) rolling restart of Varnish on 1 hosts matching query P{cp3069.*}
|
|
2026-04-17 14:01:11
|
<logmsgbot>
|
!log fabfur@cumin1003 START - Cookbook sre.cdn.roll-restart-varnish rolling restart of Varnish on 1 hosts matching query P{cp3070.*}
|
|
2026-04-17 14:01:15
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db2168 (T419635)', diff saved to https://phabricator.wikimedia.org/P91069 and previous config saved to /var/cache/conftool/dbconfig/20260417-140115-fceratto.json
|
|
2026-04-17 14:01:51
|
<wikibugs>
|
('CR) ''Jcrespo: "Will better deploy next week, even if it is a trivial patch (I may ask you to be around in case something goes wrong)." [puppet] - ''https://gerrit.wikimedia.org/r/1273791 (https://phabricator.wikimedia.org/T423689) (owner: ''Jcrespo)'
|
|
2026-04-17 14:01:55
|
<logmsgbot>
|
!log dpogorzelski@deploy1003 helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'.
|
|
2026-04-17 14:02:42
|
<logmsgbot>
|
!log fabfur@cumin1003 END (PASS) - Cookbook sre.cdn.roll-restart-varnish (exit_code=0) rolling restart of Varnish on 1 hosts matching query P{cp3070.*}
|
|
2026-04-17 14:02:43
|
<wikibugs>
|
'SRE-tools, ''bacula, ''Data-Persistence-Backup, ''Infrastructure-Foundations, and 3 others: netbox2003 backups (maybe others?) are missconfigured or failing to find the backup directory - https://phabricator.wikimedia.org/T423689#11833597 (''jcrespo) p:''Triage→''Medium a:''jcrespo'
|
|
2026-04-17 14:02:45
|
<jinxer-wm>
|
FIRING: [4x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure
|
|
2026-04-17 14:03:07
|
<logmsgbot>
|
!log fabfur@cumin1003 START - Cookbook sre.cdn.roll-restart-varnish rolling restart of Varnish on 1 hosts matching query P{cp3072.*}
|
|
2026-04-17 14:03:13
|
<logmsgbot>
|
!log dpogorzelski@deploy1003 helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'.
|
|
2026-04-17 14:04:20
|
<logmsgbot>
|
!log dpogorzelski@deploy1003 helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'.
|
|
2026-04-17 14:04:25
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1229 (T419961)', diff saved to https://phabricator.wikimedia.org/P91070 and previous config saved to /var/cache/conftool/dbconfig/20260417-140424-fceratto.json
|
|
2026-04-17 14:04:40
|
<logmsgbot>
|
!log fabfur@cumin1003 END (PASS) - Cookbook sre.cdn.roll-restart-varnish (exit_code=0) rolling restart of Varnish on 1 hosts matching query P{cp3072.*}
|
|
2026-04-17 14:04:46
|
<logmsgbot>
|
!log fceratto@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1233.eqiad.wmnet with reason: Maintenance
|
|
2026-04-17 14:04:48
|
<wikibugs>
|
'SRE, ''Infrastructure-Foundations: Failing Trixie VM installations on routed Ganeti - https://phabricator.wikimedia.org/T422596#11833600 (''jcrespo) On a trixie system I had an issue in which the host decided to kill big processes rather than becoming slow (something related to stall detection, not due to VM...'
|
|
2026-04-17 14:04:54
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Depooling db1233 (T419961)', diff saved to https://phabricator.wikimedia.org/P91071 and previous config saved to /var/cache/conftool/dbconfig/20260417-140454-fceratto.json
|
|
2026-04-17 14:05:01
|
<logmsgbot>
|
!log fabfur@cumin1003 START - Cookbook sre.cdn.roll-restart-varnish rolling restart of Varnish on 1 hosts matching query P{cp3073.*}
|
|
2026-04-17 14:05:06
|
<icinga-wm>
|
RECOVERY - MariaDB Replica Lag: m1 on db2160 is OK: OK slave_sql_lag Replication lag: 0.41 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica
|
|
2026-04-17 14:06:33
|
<logmsgbot>
|
!log eevans@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 30 days, 0:00:00 on aqs1011.eqiad.wmnet with reason: Bootstrapping — T412830
|
|
2026-04-17 14:06:37
|
<stashbot>
|
T412830: Hardware refresh of aqs101[0-2,4-5] w/ aqs102[3-7] - https://phabricator.wikimedia.org/T412830
|
|
2026-04-17 14:06:47
|
<logmsgbot>
|
!log fabfur@cumin1003 END (PASS) - Cookbook sre.cdn.roll-restart-varnish (exit_code=0) rolling restart of Varnish on 1 hosts matching query P{cp3073.*}
|
|
2026-04-17 14:09:25
|
<urandom>
|
!log decommissioning Cassandra, aqs1011 [a,b] — T412830
|
|
2026-04-17 14:09:28
|
<stashbot>
|
Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
|
|
2026-04-17 14:10:18
|
<logmsgbot>
|
!log dpogorzelski@deploy1003 helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'.
|
|
2026-04-17 14:11:23
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db2168', diff saved to https://phabricator.wikimedia.org/P91072 and previous config saved to /var/cache/conftool/dbconfig/20260417-141123-fceratto.json
|
|
2026-04-17 14:11:27
|
<logmsgbot>
|
!log dpogorzelski@deploy1003 helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'.
|
|
2026-04-17 14:12:22
|
<logmsgbot>
|
!log dpogorzelski@deploy1003 helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'.
|
|
2026-04-17 14:12:23
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1233 (T419961)', diff saved to https://phabricator.wikimedia.org/P91073 and previous config saved to /var/cache/conftool/dbconfig/20260417-141222-fceratto.json
|
|
2026-04-17 14:12:28
|
<logmsgbot>
|
!log dpogorzelski@deploy1003 helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'.
|
|
2026-04-17 14:12:37
|
<logmsgbot>
|
!log dpogorzelski@deploy1003 helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'.
|
|
2026-04-17 14:12:48
|
<jinxer-wm>
|
RESOLVED: PuppetFailure: Puppet has failed on cirrussearch1113:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure
|
|
2026-04-17 14:13:27
|
<logmsgbot>
|
!log dpogorzelski@deploy1003 helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'.
|
|
2026-04-17 14:14:13
|
<wikibugs>
|
('PS2) ''Jcrespo: netbox: Remove backups from netbox server, only leave postgres ones [puppet] - ''https://gerrit.wikimedia.org/r/1273791 (https://phabricator.wikimedia.org/T423689)'
|
|
2026-04-17 14:14:30
|
<logmsgbot>
|
!log dpogorzelski@deploy1003 helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'.
|
|
2026-04-17 14:16:16
|
<logmsgbot>
|
!log dpogorzelski@deploy1003 helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'.
|
|
2026-04-17 14:16:37
|
<logmsgbot>
|
!log dpogorzelski@deploy1003 helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'.
|
|
2026-04-17 14:17:45
|
<jinxer-wm>
|
FIRING: [4x] WidespreadPuppetFailure: Puppet has failed in drmrs - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure
|
|
2026-04-17 14:18:26
|
<logmsgbot>
|
!log dpogorzelski@deploy1003 helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'.
|
|
2026-04-17 14:18:33
|
<wikibugs>
|
('PS3) ''Jcrespo: netbox: Remove backups from netbox server, only leave postgres ones [puppet] - ''https://gerrit.wikimedia.org/r/1273791 (https://phabricator.wikimedia.org/T423689)'
|
|
2026-04-17 14:18:37
|
<wikibugs>
|
('CR) ''Jcrespo: "check experimental" [puppet] - ''https://gerrit.wikimedia.org/r/1273791 (https://phabricator.wikimedia.org/T423689) (owner: ''Jcrespo)'
|
|
2026-04-17 14:20:42
|
<wikibugs>
|
('PS4) ''Jcrespo: netbox: Remove backups from netbox server, only leave postgres ones [puppet] - ''https://gerrit.wikimedia.org/r/1273791 (https://phabricator.wikimedia.org/T423689)'
|
|
2026-04-17 14:20:51
|
<wikibugs>
|
('PS5) ''Jcrespo: netbox: Remove backups from netbox server, only leave postgres ones [puppet] - ''https://gerrit.wikimedia.org/r/1273791 (https://phabricator.wikimedia.org/T423689)'
|
|
2026-04-17 14:20:53
|
<wikibugs>
|
('CR) ''Jcrespo: "check experimental" [puppet] - ''https://gerrit.wikimedia.org/r/1273791 (https://phabricator.wikimedia.org/T423689) (owner: ''Jcrespo)'
|
|
2026-04-17 14:21:31
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db2168', diff saved to https://phabricator.wikimedia.org/P91074 and previous config saved to /var/cache/conftool/dbconfig/20260417-142130-fceratto.json
|
|
2026-04-17 14:22:31
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P91075 and previous config saved to /var/cache/conftool/dbconfig/20260417-142230-fceratto.json
|
|
2026-04-17 14:22:45
|
<jinxer-wm>
|
RESOLVED: [2x] WidespreadPuppetFailure: Puppet has failed in esams - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure
|
|
2026-04-17 14:24:21
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11833663 (''Jclark-ctr) a:''Jclark-ctr'
|
|
2026-04-17 14:25:37
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11833677 (''Jclark-ctr)'
|
|
2026-04-17 14:31:39
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db2168 (T419635)', diff saved to https://phabricator.wikimedia.org/P91076 and previous config saved to /var/cache/conftool/dbconfig/20260417-143139-fceratto.json
|
|
2026-04-17 14:31:44
|
<stashbot>
|
T419635: Drop il_to column from imagelinks table in wmf production - https://phabricator.wikimedia.org/T419635
|
|
2026-04-17 14:31:56
|
<logmsgbot>
|
!log fceratto@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2182.codfw.wmnet with reason: Maintenance
|
|
2026-04-17 14:32:04
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Depooling db2182 (T419635)', diff saved to https://phabricator.wikimedia.org/P91077 and previous config saved to /var/cache/conftool/dbconfig/20260417-143204-fceratto.json
|
|
2026-04-17 14:32:39
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P91078 and previous config saved to /var/cache/conftool/dbconfig/20260417-143238-fceratto.json
|
|
2026-04-17 14:32:55
|
<logmsgbot>
|
!log dpogorzelski@deploy1003 helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'.
|
|
2026-04-17 14:32:59
|
<logmsgbot>
|
!log dpogorzelski@deploy1003 helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'.
|
|
2026-04-17 14:33:05
|
<logmsgbot>
|
!log dpogorzelski@deploy1003 helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'.
|
|
2026-04-17 14:33:09
|
<logmsgbot>
|
!log dpogorzelski@deploy1003 helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'.
|
|
2026-04-17 14:33:39
|
<logmsgbot>
|
!log dpogorzelski@deploy1003 helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'.
|
|
2026-04-17 14:33:55
|
<wikibugs>
|
('PS2) ''Bartosz Dziewoński: Remove temporary `wgOAuth2UsePrefixedSub` feature flag [mediawiki-config] - ''https://gerrit.wikimedia.org/r/1270882 (https://phabricator.wikimedia.org/T417690) (owner: ''D3r1ck01)'
|
|
2026-04-17 14:34:01
|
<wikibugs>
|
('CR) ''Bartosz Dziewoński: [C:''+1] Remove temporary `wgOAuth2UsePrefixedSub` feature flag [mediawiki-config] - ''https://gerrit.wikimedia.org/r/1270882 (https://phabricator.wikimedia.org/T417690) (owner: ''D3r1ck01)'
|
|
2026-04-17 14:34:17
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db2182 (T419635)', diff saved to https://phabricator.wikimedia.org/P91079 and previous config saved to /var/cache/conftool/dbconfig/20260417-143416-fceratto.json
|
|
2026-04-17 14:36:00
|
<logmsgbot>
|
!log dpogorzelski@deploy1003 helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'.
|
|
2026-04-17 14:36:30
|
<logmsgbot>
|
!log dpogorzelski@deploy1003 helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'edit-check' for release 'main' .
|
|
2026-04-17 14:37:19
|
<wikibugs>
|
('CR) ''FNegri: "Is next Tuesday ok? Any day next week is fine." [puppet] - ''https://gerrit.wikimedia.org/r/1273785 (https://phabricator.wikimedia.org/T409557) (owner: ''Marostegui)'
|
|
2026-04-17 14:37:40
|
<wikibugs>
|
('PS5) ''Andrew Bogott: Openstack: use debian.net repo rather than the wmf-hosted repo [puppet] - ''https://gerrit.wikimedia.org/r/1272837 (https://phabricator.wikimedia.org/T423598)'
|
|
2026-04-17 14:37:40
|
<wikibugs>
|
('PS1) ''Andrew Bogott: Cloud-vps: use Flamingo config files for Horizon [puppet] - ''https://gerrit.wikimedia.org/r/1273831'
|
|
2026-04-17 14:37:40
|
<wikibugs>
|
('PS1) ''Andrew Bogott: Openstack: change spec tests to expect verison Flamingo [puppet] - ''https://gerrit.wikimedia.org/r/1273832'
|
|
2026-04-17 14:37:41
|
<wikibugs>
|
('PS1) ''Andrew Bogott: cloud-vps: switch VM openstack references to version Flamingo [puppet] - ''https://gerrit.wikimedia.org/r/1273833'
|
|
2026-04-17 14:37:42
|
<wikibugs>
|
('PS1) ''Andrew Bogott: Openstack: remove packages for version Dalmatian [puppet] - ''https://gerrit.wikimedia.org/r/1273834'
|
|
2026-04-17 14:37:44
|
<wikibugs>
|
('PS1) ''Andrew Bogott: Openstack: remove packages for version Epoxy [puppet] - ''https://gerrit.wikimedia.org/r/1273835'
|
|
2026-04-17 14:37:55
|
<wikibugs>
|
('CR) ''Marostegui: "Tuesday works for me yep. s6 is small so it shouldn't take long (famous last words?)" [puppet] - ''https://gerrit.wikimedia.org/r/1273785 (https://phabricator.wikimedia.org/T409557) (owner: ''Marostegui)'
|
|
2026-04-17 14:39:06
|
<wikibugs>
|
('CR) ''CI reject: [V:''-1] Openstack: use debian.net repo rather than the wmf-hosted repo [puppet] - ''https://gerrit.wikimedia.org/r/1272837 (https://phabricator.wikimedia.org/T423598) (owner: ''Andrew Bogott)'
|
|
2026-04-17 14:39:16
|
<wikibugs>
|
('PS15) ''Blake: kubernetes-generic: Add alerts for BGP failure scenarios. [alerts] - ''https://gerrit.wikimedia.org/r/1269994 (https://phabricator.wikimedia.org/T356877)'
|
|
2026-04-17 14:39:47
|
<wikibugs>
|
('CR) ''Blake: kubernetes-generic: Add alerts for BGP failure scenarios. (''3 comments) [alerts] - ''https://gerrit.wikimedia.org/r/1269994 (https://phabricator.wikimedia.org/T356877) (owner: ''Blake)'
|
|
2026-04-17 14:42:47
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1233 (T419961)', diff saved to https://phabricator.wikimedia.org/P91080 and previous config saved to /var/cache/conftool/dbconfig/20260417-144247-fceratto.json
|
|
2026-04-17 14:43:09
|
<wikibugs>
|
('CR) ''Andrew Bogott: "check experimental" [puppet] - ''https://gerrit.wikimedia.org/r/1273831 (owner: ''Andrew Bogott)'
|
|
2026-04-17 14:43:09
|
<logmsgbot>
|
!log fceratto@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1239.eqiad.wmnet with reason: Maintenance
|
|
2026-04-17 14:44:25
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P91081 and previous config saved to /var/cache/conftool/dbconfig/20260417-144424-fceratto.json
|
|
2026-04-17 14:48:11
|
<logmsgbot>
|
!log fceratto@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1254.eqiad.wmnet with reason: Maintenance
|
|
2026-04-17 14:48:20
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Depooling db1254 (T419961)', diff saved to https://phabricator.wikimedia.org/P91082 and previous config saved to /var/cache/conftool/dbconfig/20260417-144819-fceratto.json
|
|
2026-04-17 14:48:34
|
<logmsgbot>
|
!log otto@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply
|
|
2026-04-17 14:48:39
|
<logmsgbot>
|
!log otto@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply
|
|
2026-04-17 14:51:16
|
<logmsgbot>
|
!log otto@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply
|
|
2026-04-17 14:51:20
|
<logmsgbot>
|
!log otto@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich: apply
|
|
2026-04-17 14:52:00
|
<Lucas_WMDE>
|
sobanski: any update on that deploy request?
|
|
2026-04-17 14:52:20
|
<wikibugs>
|
('CR) ''JHathaway: "@jcrespo@wikimedia.org does it cause problems to leave them in test files?" [puppet] - ''https://gerrit.wikimedia.org/r/1273699 (https://phabricator.wikimedia.org/T422851) (owner: ''Jcrespo)'
|
|
2026-04-17 14:53:04
|
<icinga-wm>
|
PROBLEM - MariaDB Replica Lag: m1 on db2160 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 609.26 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica
|
|
2026-04-17 14:53:08
|
<icinga-wm>
|
PROBLEM - MariaDB Replica Lag: m1 on db1217 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 612.08 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica
|
|
2026-04-17 14:53:50
|
<wikibugs>
|
'SRE, ''collaboration-services, ''Infrastructure-Foundations, ''Wikimedia-Mailing-lists: lists.wikimedia.org subscription email rejected by DKIM - https://phabricator.wikimedia.org/T409137#11833798 (''Aklapper) @DamianZaremba: Could you please answer the last comment? Thanks in advance!'
|
|
2026-04-17 14:53:54
|
<icinga-wm>
|
PROBLEM - MariaDB Replica Lag: m1 on db2232 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 658.15 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica
|
|
2026-04-17 14:54:33
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P91083 and previous config saved to /var/cache/conftool/dbconfig/20260417-145432-fceratto.json
|
|
2026-04-17 14:54:54
|
<icinga-wm>
|
RECOVERY - MariaDB Replica Lag: m1 on db2232 is OK: OK slave_sql_lag Replication lag: 25.32 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica
|
|
2026-04-17 14:55:13
|
<wikibugs>
|
('CR) ''Andrew Bogott: [C:''+2] Cloud-vps: use Flamingo config files for Horizon [puppet] - ''https://gerrit.wikimedia.org/r/1273831 (owner: ''Andrew Bogott)'
|
|
2026-04-17 14:55:20
|
<wikibugs>
|
('CR) ''Andrew Bogott: [C:''+2] Openstack: change spec tests to expect verison Flamingo [puppet] - ''https://gerrit.wikimedia.org/r/1273832 (owner: ''Andrew Bogott)'
|
|
2026-04-17 14:55:25
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1254 (T419961)', diff saved to https://phabricator.wikimedia.org/P91084 and previous config saved to /var/cache/conftool/dbconfig/20260417-145524-fceratto.json
|
|
2026-04-17 14:56:41
|
<wikibugs>
|
('PS2) ''Andrew Bogott: cloud-vps: switch VM openstack references to version Flamingo [puppet] - ''https://gerrit.wikimedia.org/r/1273833'
|
|
2026-04-17 14:56:41
|
<wikibugs>
|
('PS2) ''Andrew Bogott: Openstack: remove packages for version Dalmatian [puppet] - ''https://gerrit.wikimedia.org/r/1273834'
|
|
2026-04-17 14:56:41
|
<wikibugs>
|
('PS2) ''Andrew Bogott: Openstack: remove packages for version Epoxy [puppet] - ''https://gerrit.wikimedia.org/r/1273835'
|
|
2026-04-17 14:56:42
|
<wikibugs>
|
('PS6) ''Andrew Bogott: Openstack: use debian.net repo rather than the wmf-hosted repo [puppet] - ''https://gerrit.wikimedia.org/r/1272837 (https://phabricator.wikimedia.org/T423598)'
|
|
2026-04-17 14:58:19
|
<wikibugs>
|
('CR) ''CI reject: [V:''-1] Openstack: use debian.net repo rather than the wmf-hosted repo [puppet] - ''https://gerrit.wikimedia.org/r/1272837 (https://phabricator.wikimedia.org/T423598) (owner: ''Andrew Bogott)'
|
|
2026-04-17 15:00:40
|
<jinxer-wm>
|
FIRING: KubernetesRsyslogDown: rsyslog on wikikube-worker1059:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=wikikube-worker1059 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown
|
|
2026-04-17 15:01:20
|
<wikibugs>
|
('PS7) ''Andrew Bogott: Openstack: use debian.net repo rather than the wmf-hosted repo [puppet] - ''https://gerrit.wikimedia.org/r/1272837 (https://phabricator.wikimedia.org/T423598)'
|
|
2026-04-17 15:02:08
|
<icinga-wm>
|
RECOVERY - MariaDB Replica Lag: m1 on db1217 is OK: OK slave_sql_lag Replication lag: 0.00 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica
|
|
2026-04-17 15:04:40
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db2182 (T419635)', diff saved to https://phabricator.wikimedia.org/P91085 and previous config saved to /var/cache/conftool/dbconfig/20260417-150440-fceratto.json
|
|
2026-04-17 15:04:47
|
<stashbot>
|
T419635: Drop il_to column from imagelinks table in wmf production - https://phabricator.wikimedia.org/T419635
|
|
2026-04-17 15:04:57
|
<logmsgbot>
|
!log fceratto@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2198.codfw.wmnet with reason: Maintenance
|
|
2026-04-17 15:05:26
|
<icinga-wm>
|
PROBLEM - Host mr1-magru.oob is DOWN: PING CRITICAL - Packet loss = 100%
|
|
2026-04-17 15:05:34
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P91086 and previous config saved to /var/cache/conftool/dbconfig/20260417-150532-fceratto.json
|
|
2026-04-17 15:05:40
|
<jinxer-wm>
|
RESOLVED: KubernetesRsyslogDown: rsyslog on wikikube-worker1059:9105 is missing kubernetes logs - https://wikitech.wikimedia.org/wiki/Kubernetes/Logging#Common_issues - https://grafana.wikimedia.org/d/OagQjQmnk?var-server=wikikube-worker1059 - https://alerts.wikimedia.org/?q=alertname%3DKubernetesRsyslogDown
|
|
2026-04-17 15:08:59
|
<Lucas_WMDE>
|
FTR, I’ve received a go-ahead for the emergency deploy mentioned earlier, so I’ll do that soon unless someone shouts
|
|
2026-04-17 15:10:28
|
<wikibugs>
|
('PS1) ''Aude: Enable ReadingLists beta feature for all Wikipedia wikis [mediawiki-config] - ''https://gerrit.wikimedia.org/r/1273842 (https://phabricator.wikimedia.org/T420881)'
|
|
2026-04-17 15:11:39
|
<wikibugs>
|
('CR) ''ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, April 20 UTC afternoon backport window](https://wikitech.wikimedia.org/wiki/Deployments#deploycal"; [mediawiki-config] - ''https://gerrit.wikimedia.org/r/1273842 (https://phabricator.wikimedia.org/T420881) (owner: ''Aude)'
|
|
2026-04-17 15:14:21
|
<wikibugs>
|
'ops-codfw, ''SRE, ''collaboration-services, ''DC-Ops: lists2001 has multiple bus errors - https://phabricator.wikimedia.org/T423159#11833878 (''LSobanski) lists2001 is the standby host so it should be safe to reboot. cc @ABran-WMF for confirmation.'
|
|
2026-04-17 15:14:38
|
<wikibugs>
|
'ops-codfw, ''SRE, ''collaboration-services, ''DC-Ops: lists2001 has multiple bus errors - https://phabricator.wikimedia.org/T423159#11833879 (''LSobanski) p:''Triage→''Low'
|
|
2026-04-17 15:15:40
|
<icinga-wm>
|
RECOVERY - Host mr1-magru.oob is UP: PING OK - Packet loss = 0%, RTA = 118.78 ms
|
|
2026-04-17 15:15:42
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1254', diff saved to https://phabricator.wikimedia.org/P91087 and previous config saved to /var/cache/conftool/dbconfig/20260417-151541-fceratto.json
|
|
2026-04-17 15:16:04
|
<icinga-wm>
|
RECOVERY - MariaDB Replica Lag: m1 on db2160 is OK: OK slave_sql_lag Replication lag: 0.25 seconds https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Depooling_a_replica
|
|
2026-04-17 15:16:56
|
<logmsgbot>
|
!log fceratto@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2200.codfw.wmnet with reason: Maintenance
|
|
2026-04-17 15:17:16
|
<logmsgbot>
|
!log fceratto@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2208.codfw.wmnet with reason: Maintenance
|
|
2026-04-17 15:17:24
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Depooling db2208 (T419635)', diff saved to https://phabricator.wikimedia.org/P91088 and previous config saved to /var/cache/conftool/dbconfig/20260417-151723-fceratto.json
|
|
2026-04-17 15:17:28
|
<stashbot>
|
T419635: Drop il_to column from imagelinks table in wmf production - https://phabricator.wikimedia.org/T419635
|
|
2026-04-17 15:17:38
|
<logmsgbot>
|
!log otto@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply
|
|
2026-04-17 15:17:43
|
<logmsgbot>
|
!log otto@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply
|
|
2026-04-17 15:18:25
|
<logmsgbot>
|
!log otto@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply
|
|
2026-04-17 15:18:30
|
<logmsgbot>
|
!log otto@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply
|
|
2026-04-17 15:18:52
|
<wikibugs>
|
('CR) ''TrainBranchBot: [C:''+2] "Approved by lucaswerkmeister-wmde@deploy1003 using scap backport" [mediawiki-config] - ''https://gerrit.wikimedia.org/r/1273763 (https://phabricator.wikimedia.org/T423512) (owner: ''A smart kitten)'
|
|
2026-04-17 15:19:36
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db2208 (T419635)', diff saved to https://phabricator.wikimedia.org/P91089 and previous config saved to /var/cache/conftool/dbconfig/20260417-151936-fceratto.json
|
|
2026-04-17 15:19:50
|
<wikibugs>
|
('Merged) ''jenkins-bot: enwikinews: Move override for $wgFlaggedRevsHandleIncludes to InitialiseSettings.php [mediawiki-config] - ''https://gerrit.wikimedia.org/r/1273763 (https://phabricator.wikimedia.org/T423512) (owner: ''A smart kitten)'
|
|
2026-04-17 15:20:42
|
<logmsgbot>
|
!log lucaswerkmeister-wmde@deploy1003 Started scap sync-world: Backport for [[gerrit:1273763|enwikinews: Move override for $wgFlaggedRevsHandleIncludes to InitialiseSettings.php (T423512)]]
|
|
2026-04-17 15:20:46
|
<stashbot>
|
T423512: Some pages on en.WN not transcluding specific templates properly - https://phabricator.wikimedia.org/T423512
|
|
2026-04-17 15:21:56
|
<Lucas_WMDE>
|
deploying ^
|
|
2026-04-17 15:22:07
|
<logmsgbot>
|
!log otto@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply
|
|
2026-04-17 15:22:12
|
<logmsgbot>
|
!log otto@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply
|
|
2026-04-17 15:22:26
|
<logmsgbot>
|
!log lucaswerkmeister-wmde@deploy1003 lucaswerkmeister-wmde, asmartkitten: Backport for [[gerrit:1273763|enwikinews: Move override for $wgFlaggedRevsHandleIncludes to InitialiseSettings.php (T423512)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
|
|
2026-04-17 15:22:45
|
<Lucas_WMDE>
|
testing
|
|
2026-04-17 15:23:03
|
<Lucas_WMDE>
|
needed a Ctrl+F5 again (idk why) but yes it seems to work \o/
|
|
2026-04-17 15:23:18
|
<logmsgbot>
|
!log otto@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply
|
|
2026-04-17 15:23:22
|
<logmsgbot>
|
!log otto@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply
|
|
2026-04-17 15:23:33
|
<logmsgbot>
|
!log lucaswerkmeister-wmde@deploy1003 lucaswerkmeister-wmde, asmartkitten: Continuing with sync
|
|
2026-04-17 15:24:19
|
<logmsgbot>
|
!log otto@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply
|
|
2026-04-17 15:24:24
|
<logmsgbot>
|
!log otto@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply
|
|
2026-04-17 15:25:44
|
<logmsgbot>
|
!log otto@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply
|
|
2026-04-17 15:25:48
|
<logmsgbot>
|
!log otto@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply
|
|
2026-04-17 15:25:50
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1254 (T419961)', diff saved to https://phabricator.wikimedia.org/P91090 and previous config saved to /var/cache/conftool/dbconfig/20260417-152549-fceratto.json
|
|
2026-04-17 15:25:59
|
<jinxer-wm>
|
FIRING: KubernetesDeploymentUnavailableReplicas: ...
|
|
2026-04-17 15:25:59
|
<jinxer-wm>
|
Deployment linkrecommendation-internal in linkrecommendation at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s&var-namespace=linkrecommendation&var-deployment=linkrecommendation-internal - ...
|
|
2026-04-17 15:25:59
|
<jinxer-wm>
|
https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas
|
|
2026-04-17 15:26:12
|
<logmsgbot>
|
!log fceratto@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1259.eqiad.wmnet with reason: Maintenance
|
|
2026-04-17 15:26:21
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Depooling db1259 (T419961)', diff saved to https://phabricator.wikimedia.org/P91091 and previous config saved to /var/cache/conftool/dbconfig/20260417-152620-fceratto.json
|
|
2026-04-17 15:27:33
|
<logmsgbot>
|
!log lucaswerkmeister-wmde@deploy1003 Finished scap sync-world: Backport for [[gerrit:1273763|enwikinews: Move override for $wgFlaggedRevsHandleIncludes to InitialiseSettings.php (T423512)]] (duration: 06m 51s)
|
|
2026-04-17 15:27:38
|
<stashbot>
|
T423512: Some pages on en.WN not transcluding specific templates properly - https://phabricator.wikimedia.org/T423512
|
|
2026-04-17 15:27:57
|
<wikibugs>
|
'SRE-OnFire, ''Release-Engineering-Team, ''Scap, ''serviceops-deprecated, ''Sustainability (Incident Followup): Should scap be able to update helmfile-defaults when -Dbuild_mw_container_image:False ? - https://phabricator.wikimedia.org/T390531#11833922 (''dancy) >>! In T390531#11831389, @dancy wrote...'
|
|
2026-04-17 15:29:45
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P91092 and previous config saved to /var/cache/conftool/dbconfig/20260417-152944-fceratto.json
|
|
2026-04-17 15:31:45
|
<Lucas_WMDE>
|
done deploying
|
|
2026-04-17 15:33:54
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1259 (T419961)', diff saved to https://phabricator.wikimedia.org/P91093 and previous config saved to /var/cache/conftool/dbconfig/20260417-153354-fceratto.json
|
|
2026-04-17 15:39:53
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P91094 and previous config saved to /var/cache/conftool/dbconfig/20260417-153953-fceratto.json
|
|
2026-04-17 15:41:09
|
<wikibugs>
|
('CR) ''FNegri: [C:''+2] mariadb: wiki-replicas: add missing grants [puppet] - ''https://gerrit.wikimedia.org/r/1270891 (https://phabricator.wikimedia.org/T422806) (owner: ''FNegri)'
|
|
2026-04-17 15:41:38
|
<wikibugs>
|
('CR) ''FNegri: [C:''+2] mariadb: wiki-replicas: add grants for %_maintain [puppet] - ''https://gerrit.wikimedia.org/r/1270465 (https://phabricator.wikimedia.org/T422806) (owner: ''FNegri)'
|
|
2026-04-17 15:41:42
|
<wikibugs>
|
('CR) ''FNegri: [C:''+2] mariadb: wiki-replicas: remove redundant grants [puppet] - ''https://gerrit.wikimedia.org/r/1270464 (https://phabricator.wikimedia.org/T422806) (owner: ''FNegri)'
|
|
2026-04-17 15:43:37
|
<wikibugs>
|
('PS8) ''Andrew Bogott: Openstack: use debian.net repo rather than the wmf-hosted repo [puppet] - ''https://gerrit.wikimedia.org/r/1272837 (https://phabricator.wikimedia.org/T423598)'
|
|
2026-04-17 15:44:03
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1259', diff saved to https://phabricator.wikimedia.org/P91095 and previous config saved to /var/cache/conftool/dbconfig/20260417-154402-fceratto.json
|
|
2026-04-17 15:44:57
|
<wikibugs>
|
('CR) ''CI reject: [V:''-1] Openstack: use debian.net repo rather than the wmf-hosted repo [puppet] - ''https://gerrit.wikimedia.org/r/1272837 (https://phabricator.wikimedia.org/T423598) (owner: ''Andrew Bogott)'
|
|
2026-04-17 15:48:10
|
<wikibugs>
|
('PS1) ''Effie Mouzeli: mw-parsoid: bump envoy workers [deployment-charts] - ''https://gerrit.wikimedia.org/r/1273855 (https://phabricator.wikimedia.org/T420336)'
|
|
2026-04-17 15:49:27
|
<wikibugs>
|
('CR) ''Jgiannelos: [C:''+1] mw-parsoid: bump envoy workers [deployment-charts] - ''https://gerrit.wikimedia.org/r/1273855 (https://phabricator.wikimedia.org/T420336) (owner: ''Effie Mouzeli)'
|
|
2026-04-17 15:49:56
|
<logmsgbot>
|
!log elukey@cumin1003 START - Cookbook sre.hosts.provision for host phab2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
|
|
2026-04-17 15:50:02
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db2208 (T419635)', diff saved to https://phabricator.wikimedia.org/P91096 and previous config saved to /var/cache/conftool/dbconfig/20260417-155001-fceratto.json
|
|
2026-04-17 15:50:06
|
<stashbot>
|
T419635: Drop il_to column from imagelinks table in wmf production - https://phabricator.wikimedia.org/T419635
|
|
2026-04-17 15:50:08
|
<logmsgbot>
|
!log fceratto@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2218.codfw.wmnet with reason: Maintenance
|
|
2026-04-17 15:50:16
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Depooling db2218 (T419635)', diff saved to https://phabricator.wikimedia.org/P91097 and previous config saved to /var/cache/conftool/dbconfig/20260417-155015-fceratto.json
|
|
2026-04-17 15:51:26
|
<wikibugs>
|
('PS2) ''Effie Mouzeli: mw-parsoid: bump envoy workers [deployment-charts] - ''https://gerrit.wikimedia.org/r/1273855 (https://phabricator.wikimedia.org/T420336)'
|
|
2026-04-17 15:52:26
|
<wikibugs>
|
('CR) ''Marostegui: "@fnegri@wikimedia.org this looks good right?" [puppet] - ''https://gerrit.wikimedia.org/r/1273785 (https://phabricator.wikimedia.org/T409557) (owner: ''Marostegui)'
|
|
2026-04-17 15:52:28
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db2218 (T419635)', diff saved to https://phabricator.wikimedia.org/P91098 and previous config saved to /var/cache/conftool/dbconfig/20260417-155228-fceratto.json
|
|
2026-04-17 15:52:41
|
<wikibugs>
|
('CR) ''Jgiannelos: [C:''+1] mw-parsoid: bump envoy workers [deployment-charts] - ''https://gerrit.wikimedia.org/r/1273855 (https://phabricator.wikimedia.org/T420336) (owner: ''Effie Mouzeli)'
|
|
2026-04-17 15:53:25
|
<logmsgbot>
|
!log elukey@cumin1003 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host phab2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
|
|
2026-04-17 15:53:47
|
<logmsgbot>
|
!log elukey@cumin1003 START - Cookbook sre.hosts.provision for host phab2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
|
|
2026-04-17 15:54:07
|
<logmsgbot>
|
!log elukey@cumin1003 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host phab2003.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
|
|
2026-04-17 15:54:11
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1259', diff saved to https://phabricator.wikimedia.org/P91099 and previous config saved to /var/cache/conftool/dbconfig/20260417-155410-fceratto.json
|
|
2026-04-17 15:56:22
|
<wikibugs>
|
('CR) ''Effie Mouzeli: [C:''+2] mw-parsoid: bump envoy workers [deployment-charts] - ''https://gerrit.wikimedia.org/r/1273855 (https://phabricator.wikimedia.org/T420336) (owner: ''Effie Mouzeli)'
|
|
2026-04-17 15:57:48
|
<wikibugs>
|
'ops-codfw, ''SRE, ''collaboration-services, ''DC-Ops: Q3:rack/setup/install phab2003 - https://phabricator.wikimedia.org/T418899#11834010 (''elukey) @VRiley-WMF sorry for the lag, completely lost this task! I tried today to run the provision script but I get stuck in the first check_connection: ` 2026-0...'
|
|
2026-04-17 15:58:30
|
<wikibugs>
|
('Merged) ''jenkins-bot: mw-parsoid: bump envoy workers [deployment-charts] - ''https://gerrit.wikimedia.org/r/1273855 (https://phabricator.wikimedia.org/T420336) (owner: ''Effie Mouzeli)'
|
|
2026-04-17 15:58:53
|
<logmsgbot>
|
!log jiji@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
|
|
2026-04-17 15:59:04
|
<logmsgbot>
|
!log jiji@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
|
|
2026-04-17 15:59:15
|
<logmsgbot>
|
!log jiji@deploy1003 helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
|
|
2026-04-17 15:59:44
|
<logmsgbot>
|
!log jiji@deploy1003 helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
|
|
2026-04-17 16:01:50
|
<logmsgbot>
|
!log jiji@deploy1003 helmfile [codfw] START helmfile.d/services/mw-parsoid: apply
|
|
2026-04-17 16:02:21
|
<logmsgbot>
|
!log jiji@deploy1003 helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply
|
|
2026-04-17 16:02:37
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db2218', diff saved to https://phabricator.wikimedia.org/P91100 and previous config saved to /var/cache/conftool/dbconfig/20260417-160236-fceratto.json
|
|
2026-04-17 16:04:19
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db1259 (T419961)', diff saved to https://phabricator.wikimedia.org/P91101 and previous config saved to /var/cache/conftool/dbconfig/20260417-160418-fceratto.json
|
|
2026-04-17 16:08:45
|
<wikibugs>
|
('CR) ''FNegri: [C:''+1] "Yep sorry, forgot to +1!" [puppet] - ''https://gerrit.wikimedia.org/r/1273785 (https://phabricator.wikimedia.org/T409557) (owner: ''Marostegui)'
|
|
2026-04-17 16:09:17
|
<jinxer-wm>
|
FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
|
|
2026-04-17 16:12:45
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db2218', diff saved to https://phabricator.wikimedia.org/P91102 and previous config saved to /var/cache/conftool/dbconfig/20260417-161245-fceratto.json
|
|
2026-04-17 16:15:30
|
<wikibugs>
|
('PS9) ''Andrew Bogott: Openstack: use debian.net repo rather than the wmf-hosted repo [puppet] - ''https://gerrit.wikimedia.org/r/1272837 (https://phabricator.wikimedia.org/T423598)'
|
|
2026-04-17 16:15:30
|
<wikibugs>
|
('PS1) ''Andrew Bogott: cloud-vps cloudcontrols: add profile::openstack::base::serverpackages [puppet] - ''https://gerrit.wikimedia.org/r/1273861'
|
|
2026-04-17 16:16:34
|
<wikibugs>
|
('CR) ''CI reject: [V:''-1] Openstack: use debian.net repo rather than the wmf-hosted repo [puppet] - ''https://gerrit.wikimedia.org/r/1272837 (https://phabricator.wikimedia.org/T423598) (owner: ''Andrew Bogott)'
|
|
2026-04-17 16:17:27
|
<wikibugs>
|
('CR) ''CI reject: [V:''-1] cloud-vps cloudcontrols: add profile::openstack::base::serverpackages [puppet] - ''https://gerrit.wikimedia.org/r/1273861 (owner: ''Andrew Bogott)'
|
|
2026-04-17 16:19:23
|
<wikibugs>
|
('PS2) ''Andrew Bogott: cloud-vps cloudcontrols: add profile::openstack::base::serverpackages [puppet] - ''https://gerrit.wikimedia.org/r/1273861'
|
|
2026-04-17 16:22:47
|
<wikibugs>
|
('PS10) ''Andrew Bogott: Openstack: use debian.net repo rather than the wmf-hosted repo [puppet] - ''https://gerrit.wikimedia.org/r/1272837 (https://phabricator.wikimedia.org/T423598)'
|
|
2026-04-17 16:22:54
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db2218 (T419635)', diff saved to https://phabricator.wikimedia.org/P91103 and previous config saved to /var/cache/conftool/dbconfig/20260417-162253-fceratto.json
|
|
2026-04-17 16:22:58
|
<stashbot>
|
T419635: Drop il_to column from imagelinks table in wmf production - https://phabricator.wikimedia.org/T419635
|
|
2026-04-17 16:23:00
|
<logmsgbot>
|
!log fceratto@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2221.codfw.wmnet with reason: Maintenance
|
|
2026-04-17 16:23:08
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Depooling db2221 (T419635)', diff saved to https://phabricator.wikimedia.org/P91104 and previous config saved to /var/cache/conftool/dbconfig/20260417-162307-fceratto.json
|
|
2026-04-17 16:23:15
|
<wikibugs>
|
('CR) ''Andrew Bogott: "check experimental" [puppet] - ''https://gerrit.wikimedia.org/r/1273861 (owner: ''Andrew Bogott)'
|
|
2026-04-17 16:23:48
|
<wikibugs>
|
('CR) ''CI reject: [V:''-1] Openstack: use debian.net repo rather than the wmf-hosted repo [puppet] - ''https://gerrit.wikimedia.org/r/1272837 (https://phabricator.wikimedia.org/T423598) (owner: ''Andrew Bogott)'
|
|
2026-04-17 16:25:19
|
<wikibugs>
|
'ops-eqiad, ''SRE, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Repurpose tools-k8s-ctrl[1001-1002],tools-k8s-worker[1001-1008] to wikikube-worker13[73-82] - https://phabricator.wikimedia.org/T423719 (''JMeybohm) ''NEW'
|
|
2026-04-17 16:25:21
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db2221 (T419635)', diff saved to https://phabricator.wikimedia.org/P91105 and previous config saved to /var/cache/conftool/dbconfig/20260417-162520-fceratto.json
|
|
2026-04-17 16:27:57
|
<wikibugs>
|
'ops-eqiad, ''SRE, ''DC-Ops: Q2:rack/setup/install Toolforge - https://phabricator.wikimedia.org/T410403#11834146 (''JMeybohm)'
|
|
2026-04-17 16:33:09
|
<wikibugs>
|
('PS3) ''Andrew Bogott: cloud-vps cloudcontrols: add profile::openstack::base::serverpackages [puppet] - ''https://gerrit.wikimedia.org/r/1273861'
|
|
2026-04-17 16:33:09
|
<wikibugs>
|
('PS11) ''Andrew Bogott: Openstack: use debian.net repo rather than the wmf-hosted repo [puppet] - ''https://gerrit.wikimedia.org/r/1272837 (https://phabricator.wikimedia.org/T423598)'
|
|
2026-04-17 16:34:17
|
<jinxer-wm>
|
RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
|
|
2026-04-17 16:34:38
|
<wikibugs>
|
('CR) ''CI reject: [V:''-1] Openstack: use debian.net repo rather than the wmf-hosted repo [puppet] - ''https://gerrit.wikimedia.org/r/1272837 (https://phabricator.wikimedia.org/T423598) (owner: ''Andrew Bogott)'
|
|
2026-04-17 16:35:29
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P91107 and previous config saved to /var/cache/conftool/dbconfig/20260417-163528-fceratto.json
|
|
2026-04-17 16:35:55
|
<wikibugs>
|
('PS4) ''Andrew Bogott: cloud-vps cloudcontrols: add profile::openstack::base::serverpackages [puppet] - ''https://gerrit.wikimedia.org/r/1273861'
|
|
2026-04-17 16:35:55
|
<wikibugs>
|
('PS12) ''Andrew Bogott: Openstack: use debian.net repo rather than the wmf-hosted repo [puppet] - ''https://gerrit.wikimedia.org/r/1272837 (https://phabricator.wikimedia.org/T423598)'
|
|
2026-04-17 16:36:46
|
<wikibugs>
|
'SRE, ''Infrastructure-Foundations, ''ServiceOps new, ''ServiceOps-Datastores: Upgrade kafka-logging to version 3.x - https://phabricator.wikimedia.org/T423723 (''herron) ''NEW'
|
|
2026-04-17 16:37:01
|
<wikibugs>
|
('PS5) ''Andrew Bogott: cloud-vps cloudcontrols: add profile::openstack::base::serverpackages [puppet] - ''https://gerrit.wikimedia.org/r/1273861'
|
|
2026-04-17 16:37:01
|
<wikibugs>
|
('PS13) ''Andrew Bogott: Openstack: use debian.net repo rather than the wmf-hosted repo [puppet] - ''https://gerrit.wikimedia.org/r/1272837 (https://phabricator.wikimedia.org/T423598)'
|
|
2026-04-17 16:37:20
|
<wikibugs>
|
('CR) ''Andrew Bogott: "check experimental" [puppet] - ''https://gerrit.wikimedia.org/r/1273861 (owner: ''Andrew Bogott)'
|
|
2026-04-17 16:37:57
|
<wikibugs>
|
'SRE, ''Infrastructure-Foundations, ''ServiceOps new, ''ServiceOps-Datastores: Upgrade kafka-logging to version 3.x - https://phabricator.wikimedia.org/T423723#11834206 (''herron) p:''Triage→''High'
|
|
2026-04-17 16:38:26
|
<wikibugs>
|
'SRE, ''Infrastructure-Foundations, ''ServiceOps new, ''ServiceOps-Datastores: Upgrade kafka-logging to version 3.x - https://phabricator.wikimedia.org/T423723#11834210 (''herron)'
|
|
2026-04-17 16:38:31
|
<wikibugs>
|
('CR) ''CI reject: [V:''-1] Openstack: use debian.net repo rather than the wmf-hosted repo [puppet] - ''https://gerrit.wikimedia.org/r/1272837 (https://phabricator.wikimedia.org/T423598) (owner: ''Andrew Bogott)'
|
|
2026-04-17 16:38:39
|
<jinxer-wm>
|
FIRING: [2x] TransitBGPDown: Transit BGP session down between cr3-ulsfo and Hurricane Electric (2001:504:30::ba00:6939:1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DTransitBGPDown
|
|
2026-04-17 16:39:02
|
<wikibugs>
|
'ops-eqsin, ''SRE, ''DC-Ops, ''Infrastructure-Foundations, ''netops: EQSIN:Switch refresh diagram and wiring - https://phabricator.wikimedia.org/T423724 (''Papaul) ''NEW'
|
|
2026-04-17 16:39:29
|
<wikibugs>
|
('CR) ''CI reject: [V:''-1] cloud-vps cloudcontrols: add profile::openstack::base::serverpackages [puppet] - ''https://gerrit.wikimedia.org/r/1273861 (owner: ''Andrew Bogott)'
|
|
2026-04-17 16:39:30
|
<wikibugs>
|
'ops-eqiad, ''SRE, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Repurpose tools-k8s-ctrl[1001-1002],tools-k8s-worker[1001-1008] to wikikube-worker13[73-82] - https://phabricator.wikimedia.org/T423719#11834227 (''wiki_willy) Thanks @Clement_Goubert! @VRiley-WMF & @Jclark-ctr -
this is the...'
|
|
2026-04-17 16:42:10
|
<wikibugs>
|
'SRE, ''Infrastructure-Foundations, ''ServiceOps new, ''ServiceOps-Datastores: Upgrade kafka-logging to version 3.x - https://phabricator.wikimedia.org/T423723#11834242 (''herron) @brouberol, do the steps in the description look ok to you, anything I missed?'
|
|
2026-04-17 16:42:40
|
<wikibugs>
|
'SRE, ''Infrastructure-Foundations, ''ServiceOps new, ''ServiceOps-Datastores: Upgrade kafka-logging to version 3.x - https://phabricator.wikimedia.org/T423723#11834243 (''herron) a:''herron'
|
|
2026-04-17 16:43:05
|
<wikibugs>
|
('PS6) ''Andrew Bogott: cloud-vps cloudcontrols: add profile::openstack::base::serverpackages [puppet] - ''https://gerrit.wikimedia.org/r/1273861'
|
|
2026-04-17 16:43:05
|
<wikibugs>
|
('PS14) ''Andrew Bogott: Openstack: use debian.net repo rather than the wmf-hosted repo [puppet] - ''https://gerrit.wikimedia.org/r/1272837 (https://phabricator.wikimedia.org/T423598)'
|
|
2026-04-17 16:43:23
|
<jinxer-wm>
|
FIRING: [4x] ProbeDown: Service aqs1010-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
|
|
2026-04-17 16:43:37
|
<wikibugs>
|
('CR) ''Andrew Bogott: "check experimental" [puppet] - ''https://gerrit.wikimedia.org/r/1273861 (owner: ''Andrew Bogott)'
|
|
2026-04-17 16:44:29
|
<wikibugs>
|
('CR) ''CI reject: [V:''-1] Openstack: use debian.net repo rather than the wmf-hosted repo [puppet] - ''https://gerrit.wikimedia.org/r/1272837 (https://phabricator.wikimedia.org/T423598) (owner: ''Andrew Bogott)'
|
|
2026-04-17 16:45:37
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db2221', diff saved to https://phabricator.wikimedia.org/P91108 and previous config saved to /var/cache/conftool/dbconfig/20260417-164536-fceratto.json
|
|
2026-04-17 16:52:13
|
<wikibugs>
|
'SRE-swift-storage, ''API Platform, ''Commons, ''MediaWiki-File-management, and 3 others: Commons: UploadChunkFileException: Error storing file: backend-fail-internal; local-swift-codfw - https://phabricator.wikimedia.org/T328872#11834286 (''CDanis) FYI: as of my Puppet patches above, you can now use an x...'
|
|
2026-04-17 16:55:45
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db2221 (T419635)', diff saved to https://phabricator.wikimedia.org/P91110 and previous config saved to /var/cache/conftool/dbconfig/20260417-165544-fceratto.json
|
|
2026-04-17 16:55:49
|
<stashbot>
|
T419635: Drop il_to column from imagelinks table in wmf production - https://phabricator.wikimedia.org/T419635
|
|
2026-04-17 16:55:52
|
<logmsgbot>
|
!log fceratto@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2222.codfw.wmnet with reason: Maintenance
|
|
2026-04-17 16:56:00
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Depooling db2222 (T419635)', diff saved to https://phabricator.wikimedia.org/P91111 and previous config saved to /var/cache/conftool/dbconfig/20260417-165559-fceratto.json
|
|
2026-04-17 16:58:03
|
<wikibugs>
|
('PS7) ''Andrew Bogott: cloud-vps cloudcontrols: add profile::openstack::base::serverpackages [puppet] - ''https://gerrit.wikimedia.org/r/1273861'
|
|
2026-04-17 16:58:03
|
<wikibugs>
|
('PS15) ''Andrew Bogott: Openstack: use debian.net repo rather than the wmf-hosted repo [puppet] - ''https://gerrit.wikimedia.org/r/1272837 (https://phabricator.wikimedia.org/T423598)'
|
|
2026-04-17 16:58:09
|
<wikibugs>
|
('CR) ''Andrew Bogott: "check experimental" [puppet] - ''https://gerrit.wikimedia.org/r/1273861 (owner: ''Andrew Bogott)'
|
|
2026-04-17 16:58:12
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db2222 (T419635)', diff saved to https://phabricator.wikimedia.org/P91112 and previous config saved to /var/cache/conftool/dbconfig/20260417-165811-fceratto.json
|
|
2026-04-17 16:58:39
|
<jinxer-wm>
|
RESOLVED: [2x] TransitBGPDown: Transit BGP session down between cr3-ulsfo and Hurricane Electric (2001:504:30::ba00:6939:1) - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DTransitBGPDown
|
|
2026-04-17 16:58:51
|
<wikibugs>
|
('CR) ''CI reject: [V:''-1] cloud-vps cloudcontrols: add profile::openstack::base::serverpackages [puppet] - ''https://gerrit.wikimedia.org/r/1273861 (owner: ''Andrew Bogott)'
|
|
2026-04-17 17:00:28
|
<wikibugs>
|
'ops-codfw, ''SRE, ''collaboration-services, ''DC-Ops: Q3:rack/setup/install phab2003 - https://phabricator.wikimedia.org/T418899#11834314 (''Dzahn) Once the setup issues are resolved we will implement Phabricator and replace phab2002 with it over at T423727.'
|
|
2026-04-17 17:03:52
|
<wikibugs>
|
('PS1) ''Bking: OpenSearch: Control which plugins we use via systemd PrivateMounts [puppet] - ''https://gerrit.wikimedia.org/r/1273887 (https://phabricator.wikimedia.org/T422860)'
|
|
2026-04-17 17:05:00
|
<wikibugs>
|
'SRE, ''SRE-Access-Requests, ''Data-Engineering: Requesting access to analytics_privatedata_users and SQL Lab for AnnieKim_WMDE - https://phabricator.wikimedia.org/T420500#11834353 (''Dzahn) ''Resolved→''Open a:''Scott_French→''None'
|
|
2026-04-17 17:05:02
|
<wikibugs>
|
'SRE-swift-storage, ''API Platform, ''Commons, ''MediaWiki-File-management, and 3 others: Commons: UploadChunkFileException: Error storing file: backend-fail-internal; local-swift-codfw - https://phabricator.wikimedia.org/T328872#11834355 (''CDanis) BTW -- here are two canned queries for distributed trace...'
|
|
2026-04-17 17:05:15
|
<wikibugs>
|
('CR) ''Bking: "check experimental" [puppet] - ''https://gerrit.wikimedia.org/r/1273887 (https://phabricator.wikimedia.org/T422860) (owner: ''Bking)'
|
|
2026-04-17 17:07:30
|
<icinga-wm>
|
PROBLEM - OSPF status on cr1-magru is CRITICAL: OSPFv2: 1/2 UP : OSPFv3: 1/2 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status
|
|
2026-04-17 17:08:06
|
<jinxer-wm>
|
FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr2-codfw:et-0/1/4 (Transport: cr2-eqiad:et-1/1/5 (Lumen, 449169461) {#3909}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown
|
|
2026-04-17 17:08:20
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db2222', diff saved to https://phabricator.wikimedia.org/P91113 and previous config saved to /var/cache/conftool/dbconfig/20260417-170819-fceratto.json
|
|
2026-04-17 17:09:39
|
<jinxer-wm>
|
FIRING: [2x] TransitBGPDown: Transit BGP session down between cr1-magru and Telxius (2001:1498:1:966:1::251) - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DTransitBGPDown
|
|
2026-04-17 17:10:11
|
<jinxer-wm>
|
FIRING: [2x] BFDdown: BFD session down between cr2-eqiad and 195.200.68.151 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown
|
|
2026-04-17 17:10:19
|
<wikibugs>
|
('PS2) ''Bking: OpenSearch: Control which plugins we use via systemd PrivateMounts [puppet] - ''https://gerrit.wikimedia.org/r/1273887 (https://phabricator.wikimedia.org/T422860)'
|
|
2026-04-17 17:10:22
|
<wikibugs>
|
('CR) ''CDanis: [V:''+1 C:''+1] "Amir or someone, please roll this out in my absence on Monday :D" [puppet] - ''https://gerrit.wikimedia.org/r/1272869 (owner: ''CDanis)'
|
|
2026-04-17 17:10:40
|
<jinxer-wm>
|
FIRING: [4x] CoreBGPDown: Core BGP session down between cr1-magru and cr2-eqiad (195.200.68.150) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown
|
|
2026-04-17 17:10:47
|
<wikibugs>
|
'SRE-OnFire: List cookbooks on wikitech - https://phabricator.wikimedia.org/T423730 (''jijiki) ''NEW'
|
|
2026-04-17 17:10:49
|
<wikibugs>
|
('PS1) ''Dzahn: etherpad: update warning message about truncation to end of May [puppet] - ''https://gerrit.wikimedia.org/r/1273889 (https://phabricator.wikimedia.org/T415237)'
|
|
2026-04-17 17:11:21
|
<wikibugs>
|
('CR) ''Pppery: etherpad: update warning message about truncation to end of May (''1 comment) [puppet] - ''https://gerrit.wikimedia.org/r/1273889 (https://phabricator.wikimedia.org/T415237) (owner: ''Dzahn)'
|
|
2026-04-17 17:11:34
|
<wikibugs>
|
('CR) ''Dzahn: [C:''+2] etherpad: update warning message about truncation to end of May [puppet] - ''https://gerrit.wikimedia.org/r/1273889 (https://phabricator.wikimedia.org/T415237) (owner: ''Dzahn)'
|
|
2026-04-17 17:11:54
|
<wikibugs>
|
('PS3) ''Bking: OpenSearch: Control which plugins we use via systemd PrivateMounts [puppet] - ''https://gerrit.wikimedia.org/r/1273887 (https://phabricator.wikimedia.org/T422860)'
|
|
2026-04-17 17:12:00
|
<wikibugs>
|
('CR) ''Bking: "check experimental" [puppet] - ''https://gerrit.wikimedia.org/r/1273887 (https://phabricator.wikimedia.org/T422860) (owner: ''Bking)'
|
|
2026-04-17 17:12:12
|
<wikibugs>
|
('PS2) ''Dzahn: etherpad: update warning message about truncation to end of May [puppet] - ''https://gerrit.wikimedia.org/r/1273889 (https://phabricator.wikimedia.org/T415237)'
|
|
2026-04-17 17:12:18
|
<wikibugs>
|
('CR) ''Dzahn: etherpad: update warning message about truncation to end of May (''1 comment) [puppet] - ''https://gerrit.wikimedia.org/r/1273889 (https://phabricator.wikimedia.org/T415237) (owner: ''Dzahn)'
|
|
2026-04-17 17:13:00
|
<wikibugs>
|
('CR) ''Dzahn: [C:''+2] etherpad: update warning message about truncation to end of May [puppet] - ''https://gerrit.wikimedia.org/r/1273889 (https://phabricator.wikimedia.org/T415237) (owner: ''Dzahn)'
|
|
2026-04-17 17:14:19
|
<wikibugs>
|
('PS8) ''Andrew Bogott: cloud-vps cloudcontrols: add profile::openstack::base::serverpackages [puppet] - ''https://gerrit.wikimedia.org/r/1273861'
|
|
2026-04-17 17:14:35
|
<wikibugs>
|
('CR) ''Andrew Bogott: "check experimental" [puppet] - ''https://gerrit.wikimedia.org/r/1272837 (https://phabricator.wikimedia.org/T423598) (owner: ''Andrew Bogott)'
|
|
2026-04-17 17:14:39
|
<jinxer-wm>
|
FIRING: [4x] TransitBGPDown: Transit BGP session down between cr1-magru and Telxius (2001:1498:1:966:1::251) - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DTransitBGPDown
|
|
2026-04-17 17:16:21
|
<wikibugs>
|
'SRE-SLO, ''ServiceOps new, ''Data-Platform-SRE (2026-03-27 - 2026-04-17), ''Essential-Work, and 2 others: IPoid: Define service level indicators and service level objectives - https://phabricator.wikimedia.org/T348935#11834387 (''BTullis) I have drafted some SLI definitions here: https://wikitech.wikimed...'
|
|
2026-04-17 17:18:28
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db2222', diff saved to https://phabricator.wikimedia.org/P91114 and previous config saved to /var/cache/conftool/dbconfig/20260417-171827-fceratto.json
|
|
2026-04-17 17:24:11
|
<wikibugs>
|
('PS4) ''Bking: OpenSearch: Control which plugins we use via systemd PrivateMounts [puppet] - ''https://gerrit.wikimedia.org/r/1273887 (https://phabricator.wikimedia.org/T422860)'
|
|
2026-04-17 17:25:05
|
<wikibugs>
|
'SRE-SLO, ''ServiceOps new, ''Data-Platform-SRE (2026-03-27 - 2026-04-17), ''Essential-Work, and 2 others: IPoid: Define service level indicators and service level objectives - https://phabricator.wikimedia.org/T348935#11834420 (''BTullis) It's worth noting that when I started working on the latency indic...'
|
|
2026-04-17 17:25:06
|
<wikibugs>
|
('CR) ''Bking: "check experimental" [puppet] - ''https://gerrit.wikimedia.org/r/1273887 (https://phabricator.wikimedia.org/T422860) (owner: ''Bking)'
|
|
2026-04-17 17:28:36
|
<logmsgbot>
|
!log fceratto@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db2222 (T419635)', diff saved to https://phabricator.wikimedia.org/P91116 and previous config saved to /var/cache/conftool/dbconfig/20260417-172835-fceratto.json
|
|
2026-04-17 17:28:40
|
<stashbot>
|
T419635: Drop il_to column from imagelinks table in wmf production - https://phabricator.wikimedia.org/T419635
|
|
2026-04-17 17:28:49
|
<wikibugs>
|
'SRE-SLO, ''ServiceOps new, ''Data-Platform-SRE (2026-03-27 - 2026-04-17), ''Essential-Work, and 2 others: IPoid: Define service level indicators and service level objectives - https://phabricator.wikimedia.org/T348935#11834430 (''BTullis) There are also some draft SLOs here: https://wikitech.wikimedia.or...'
|
|
2026-04-17 17:32:26
|
<wikibugs>
|
('PS9) ''Andrew Bogott: cloud-vps cloudcontrols: add profile::openstack::base::serverpackages [puppet] - ''https://gerrit.wikimedia.org/r/1273861'
|
|
2026-04-17 17:32:26
|
<wikibugs>
|
('PS16) ''Andrew Bogott: Openstack: use debian.net repo rather than the wmf-hosted repo [puppet] - ''https://gerrit.wikimedia.org/r/1272837 (https://phabricator.wikimedia.org/T423598)'
|
|
2026-04-17 17:34:02
|
<wikibugs>
|
('PS5) ''Bking: OpenSearch: Control which plugins we use via systemd PrivateMounts [puppet] - ''https://gerrit.wikimedia.org/r/1273887 (https://phabricator.wikimedia.org/T422860)'
|
|
2026-04-17 17:34:27
|
<wikibugs>
|
'SRE, ''Infrastructure-Foundations, ''ServiceOps new, ''ServiceOps-Datastores, ''Patch-For-Review: Upgrade kafka-logging to version 3.x - https://phabricator.wikimedia.org/T423723#11834455 (''brouberol) cc @elukey as he was the one working on the puppet side of things. It //does// look good
to me though!'
|
|
2026-04-17 17:34:28
|
<wikibugs>
|
('PS10) ''Andrew Bogott: cloud-vps: consolidate inclusion of openstack server/client packages [puppet] - ''https://gerrit.wikimedia.org/r/1273861'
|
|
2026-04-17 17:34:28
|
<wikibugs>
|
('PS17) ''Andrew Bogott: Openstack: use debian.net repo rather than the wmf-hosted repo [puppet] - ''https://gerrit.wikimedia.org/r/1272837 (https://phabricator.wikimedia.org/T423598)'
|
|
2026-04-17 17:34:44
|
<wikibugs>
|
('CR) ''Bking: "check experimental" [puppet] - ''https://gerrit.wikimedia.org/r/1273887 (https://phabricator.wikimedia.org/T422860) (owner: ''Bking)'
|
|
2026-04-17 17:35:51
|
<wikibugs>
|
('CR) ''CI reject: [V:''-1] cloud-vps: consolidate inclusion of openstack server/client packages [puppet] - ''https://gerrit.wikimedia.org/r/1273861 (owner: ''Andrew Bogott)'
|
|
2026-04-17 17:36:02
|
<wikibugs>
|
('PS1) ''Dzahn: etherpad: insert a warning message into the index HTML [puppet] - ''https://gerrit.wikimedia.org/r/1273902 (https://phabricator.wikimedia.org/T420793)'
|
|
2026-04-17 17:36:35
|
<wikibugs>
|
('CR) ''CI reject: [V:''-1] etherpad: insert a warning message into the index HTML [puppet] - ''https://gerrit.wikimedia.org/r/1273902 (https://phabricator.wikimedia.org/T420793) (owner: ''Dzahn)'
|
|
2026-04-17 17:36:49
|
<wikibugs>
|
('CR) ''Dzahn: "using some parts of the message we also use in new pad messages" [puppet] - ''https://gerrit.wikimedia.org/r/1273902 (https://phabricator.wikimedia.org/T420793) (owner: ''Dzahn)'
|
|
2026-04-17 17:37:13
|
<wikibugs>
|
('CR) ''Andrew Bogott: "check experimental" [puppet] - ''https://gerrit.wikimedia.org/r/1273861 (owner: ''Andrew Bogott)'
|
|
2026-04-17 17:37:22
|
<wikibugs>
|
('CR) ''Andrew Bogott: "check experimental" [puppet] - ''https://gerrit.wikimedia.org/r/1272837 (https://phabricator.wikimedia.org/T423598) (owner: ''Andrew Bogott)'
|
|
2026-04-17 17:37:36
|
<wikibugs>
|
('PS2) ''Dzahn: etherpad: insert a warning message into the index HTML [puppet] - ''https://gerrit.wikimedia.org/r/1273902 (https://phabricator.wikimedia.org/T420793)'
|
|
2026-04-17 17:40:40
|
<wikibugs>
|
('CR) ''Pppery: "I would stick a "Bug: T371591" link on this commit, as this finally does what that task actually requested." [puppet] - ''https://gerrit.wikimedia.org/r/1273902 (https://phabricator.wikimedia.org/T420793) (owner: ''Dzahn)'
|
|
2026-04-17 17:42:35
|
<wikibugs>
|
('PS3) ''Dzahn: etherpad: insert a warning message into the index HTML [puppet] - ''https://gerrit.wikimedia.org/r/1273902 (https://phabricator.wikimedia.org/T420793)'
|
|
2026-04-17 17:42:50
|
<wikibugs>
|
('CR) ''Dzahn: "ah, thanks! doing!" [puppet] - ''https://gerrit.wikimedia.org/r/1273902 (https://phabricator.wikimedia.org/T420793) (owner: ''Dzahn)'
|
|
2026-04-17 17:43:06
|
<wikibugs>
|
('CR) ''Pppery: "Also maybe stick a link to https://meta.wikimedia.org/wiki/Etherpad somewhere in the content somewhere. Otherwise looks good to me as far " [puppet] - ''https://gerrit.wikimedia.org/r/1273902 (https://phabricator.wikimedia.org/T420793) (owner: ''Dzahn)'
|
|
2026-04-17 17:43:23
|
<wikibugs>
|
('CR) ''Dzahn: [V:''+1] "https://puppet-compiler.wmflabs.org/output/1273902/8439/etherpad1004.eqiad.wmnet/index.html"; [puppet] - ''https://gerrit.wikimedia.org/r/1273902 (https://phabricator.wikimedia.org/T420793) (owner:
''Dzahn)'
|
|
2026-04-17 17:46:28
|
<wikibugs>
|
('CR) ''Dzahn: [V:''+1] "Ok. done! linked "WMF Etherpad" to the URL above." [puppet] - ''https://gerrit.wikimedia.org/r/1273902 (https://phabricator.wikimedia.org/T420793) (owner: ''Dzahn)'
|
|
2026-04-17 17:46:53
|
<wikibugs>
|
('PS4) ''Dzahn: etherpad: insert a warning message into the index HTML [puppet] - ''https://gerrit.wikimedia.org/r/1273902 (https://phabricator.wikimedia.org/T420793)'
|
|
2026-04-17 17:47:29
|
<wikibugs>
|
('CR) ''Pppery: [C:''+1] etherpad: insert a warning message into the index HTML [puppet] - ''https://gerrit.wikimedia.org/r/1273902 (https://phabricator.wikimedia.org/T420793) (owner: ''Dzahn)'
|
|
2026-04-17 17:47:42
|
<wikibugs>
|
('CR) ''Pppery: [C:''+1] "This got lost in PS4" [puppet] - ''https://gerrit.wikimedia.org/r/1273902 (https://phabricator.wikimedia.org/T420793) (owner: ''Dzahn)'
|
|
2026-04-17 17:48:04
|
<wikibugs>
|
('CR) ''Dzahn: [C:''+2] etherpad: insert a warning message into the index HTML [puppet] - ''https://gerrit.wikimedia.org/r/1273902 (https://phabricator.wikimedia.org/T420793) (owner: ''Dzahn)'
|
|
2026-04-17 17:49:27
|
<wikibugs>
|
('CR) ''Jdlrobson: [C:''+1] Enable ReadingLists beta feature for all Wikipedia wikis [mediawiki-config] - ''https://gerrit.wikimedia.org/r/1273842 (https://phabricator.wikimedia.org/T420881) (owner: ''Aude)'
|
|
2026-04-17 17:49:39
|
<wikibugs>
|
('PS5) ''Dzahn: etherpad: insert a warning message into the index HTML [puppet] - ''https://gerrit.wikimedia.org/r/1273902 (https://phabricator.wikimedia.org/T420793)'
|
|
2026-04-17 17:49:52
|
<wikibugs>
|
('PS6) ''Dzahn: etherpad: insert a warning message into the index HTML [puppet] - ''https://gerrit.wikimedia.org/r/1273902 (https://phabricator.wikimedia.org/T420793)'
|
|
2026-04-17 17:52:42
|
<wikibugs>
|
('PS6) ''Bking: OpenSearch: Control which plugins we use via systemd PrivateMounts [puppet] - ''https://gerrit.wikimedia.org/r/1273887 (https://phabricator.wikimedia.org/T422860)'
|
|
2026-04-17 17:52:59
|
<wikibugs>
|
('CR) ''Bking: "check experimental" [puppet] - ''https://gerrit.wikimedia.org/r/1273887 (https://phabricator.wikimedia.org/T422860) (owner: ''Bking)'
|
|
2026-04-17 17:53:13
|
<wikibugs>
|
('CR) ''CI reject: [V:''-1] OpenSearch: Control which plugins we use via systemd PrivateMounts [puppet] - ''https://gerrit.wikimedia.org/r/1273887 (https://phabricator.wikimedia.org/T422860) (owner: ''Bking)'
|
|
2026-04-17 17:55:42
|
<wikibugs>
|
('CR) ''Dzahn: [C:''+2] etherpad: insert a warning message into the index HTML [puppet] - ''https://gerrit.wikimedia.org/r/1273902 (https://phabricator.wikimedia.org/T420793) (owner: ''Dzahn)'
|
|
2026-04-17 17:59:03
|
<wikibugs>
|
('PS1) ''CDanis: mwscript-k8s: add --output-file flag [puppet] - ''https://gerrit.wikimedia.org/r/1273905'
|
|
2026-04-17 18:45:55
|
<wikibugs>
|
('CR) ''Dzahn: [C:''+1] gerrit: migrate data gerrit1003 to /srv/gerrit [puppet] - ''https://gerrit.wikimedia.org/r/1273449 (https://phabricator.wikimedia.org/T333143) (owner: ''Jelto)'
|
|
2026-04-17 18:47:57
|
<wikibugs>
|
('CR) ''Dzahn: gerrit: update sync-instances cookbook (''1 comment) [cookbooks] - ''https://gerrit.wikimedia.org/r/1270863 (https://phabricator.wikimedia.org/T333143) (owner: ''Arnaudb)'
|
|
2026-04-17 18:51:42
|
<wikibugs>
|
('CR) ''Dzahn: [C:''+1] "+1 as long as things happen in the right order" [puppet] - ''https://gerrit.wikimedia.org/r/1270774 (https://phabricator.wikimedia.org/T423027) (owner: ''Arnaudb)'
|
|
2026-04-17 18:52:34
|
<wikibugs>
|
('PS7) ''Bking: OpenSearch: Control which plugins we use via systemd PrivateMounts [puppet] - ''https://gerrit.wikimedia.org/r/1273887 (https://phabricator.wikimedia.org/T422860)'
|
|
2026-04-17 18:53:30
|
<wikibugs>
|
('CR) ''Bking: "check experimental" [puppet] - ''https://gerrit.wikimedia.org/r/1273887 (https://phabricator.wikimedia.org/T422860) (owner: ''Bking)'
|
|
2026-04-17 18:54:35
|
<wikibugs>
|
('CR) ''CI reject: [V:''-1] OpenSearch: Control which plugins we use via systemd PrivateMounts [puppet] - ''https://gerrit.wikimedia.org/r/1273887 (https://phabricator.wikimedia.org/T422860) (owner: ''Bking)'
|
|
2026-04-17 18:55:31
|
<wikibugs>
|
('PS8) ''Bking: OpenSearch: Control which plugins we use via systemd PrivateMounts [puppet] - ''https://gerrit.wikimedia.org/r/1273887 (https://phabricator.wikimedia.org/T422860)'
|
|
2026-04-17 18:57:25
|
<wikibugs>
|
('PS11) ''Andrew Bogott: cloud-vps: consolidate inclusion of openstack server/client packages [puppet] - ''https://gerrit.wikimedia.org/r/1273861'
|
|
2026-04-17 18:57:25
|
<wikibugs>
|
('PS18) ''Andrew Bogott: Openstack: use debian.net repo rather than the wmf-hosted repo [puppet] - ''https://gerrit.wikimedia.org/r/1272837 (https://phabricator.wikimedia.org/T423598)'
|
|
2026-04-17 18:57:39
|
<wikibugs>
|
('CR) ''CI reject: [V:''-1] OpenSearch: Control which plugins we use via systemd PrivateMounts [puppet] - ''https://gerrit.wikimedia.org/r/1273887 (https://phabricator.wikimedia.org/T422860) (owner: ''Bking)'
|
|
2026-04-17 18:58:36
|
<wikibugs>
|
('PS9) ''Bking: OpenSearch: Control which plugins we use via systemd PrivateMounts [puppet] - ''https://gerrit.wikimedia.org/r/1273887 (https://phabricator.wikimedia.org/T422860)'
|
|
2026-04-17 18:59:28
|
<wikibugs>
|
('CR) ''CI reject: [V:''-1] OpenSearch: Control which plugins we use via systemd PrivateMounts [puppet] - ''https://gerrit.wikimedia.org/r/1273887 (https://phabricator.wikimedia.org/T422860) (owner: ''Bking)'
|
|
2026-04-17 19:02:11
|
<wikibugs>
|
('PS10) ''Bking: OpenSearch: Control which plugins we use via systemd PrivateMounts [puppet] - ''https://gerrit.wikimedia.org/r/1273887 (https://phabricator.wikimedia.org/T422860)'
|
|
2026-04-17 19:02:18
|
<wikibugs>
|
('CR) ''Bking: "check experimental" [puppet] - ''https://gerrit.wikimedia.org/r/1273887 (https://phabricator.wikimedia.org/T422860) (owner: ''Bking)'
|
|
2026-04-17 19:04:06
|
<wikibugs>
|
('CR) ''Dzahn: [C:''+2] jenkins: allow disabling jenkins even on the manager host [puppet] - ''https://gerrit.wikimedia.org/r/1271017 (https://phabricator.wikimedia.org/T418109) (owner: ''Dzahn)'
|
|
2026-04-17 19:05:11
|
<jinxer-wm>
|
FIRING: [3x] BFDdown: BFD session down between cr1-eqiad and fe80::7a4f:9b00:d4e:7c0c - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown
|
|
2026-04-17 19:09:27
|
<wikibugs>
|
('CR) ''Dzahn: [C:''+2] "complete noop on all contint* servers - but allows disabling jenkins on legacy hosts" [puppet] - ''https://gerrit.wikimedia.org/r/1271017 (https://phabricator.wikimedia.org/T418109) (owner: ''Dzahn)'
|
|
2026-04-17 19:11:48
|
<wikibugs>
|
('PS1) ''Dzahn: contint: disable jenkins on legacy CI hosts [puppet] - ''https://gerrit.wikimedia.org/r/1273919 (https://phabricator.wikimedia.org/T418109)'
|
|
2026-04-17 19:12:28
|
<wikibugs>
|
('PS11) ''Bking: OpenSearch: Control which plugins we use via systemd PrivateMounts [puppet] - ''https://gerrit.wikimedia.org/r/1273887 (https://phabricator.wikimedia.org/T422860)'
|
|
2026-04-17 19:12:41
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.dns.netbox
|
|
2026-04-17 19:13:40
|
<wikibugs>
|
('CR) ''Bking: "check experimental" [puppet] - ''https://gerrit.wikimedia.org/r/1273887 (https://phabricator.wikimedia.org/T422860) (owner: ''Bking)'
|
|
2026-04-17 19:14:10
|
<wikibugs>
|
('PS1) ''CDobbins: data.yaml: remove legacy ssh key; add fido backup [puppet] - ''https://gerrit.wikimedia.org/r/1273925'
|
|
2026-04-17 19:15:38
|
<wikibugs>
|
('CR) ''Dzahn: [V:''+1] "https://puppet-compiler.wmflabs.org/output/1273919/8441/"; [puppet] - ''https://gerrit.wikimedia.org/r/1273919 (https://phabricator.wikimedia.org/T418109) (owner: ''Dzahn)'
|
|
2026-04-17 19:16:02
|
<wikibugs>
|
('PS7) ''CDanis: fundraising_data_import maintenance script wrapper & timer [puppet] - ''https://gerrit.wikimedia.org/r/1271028 (https://phabricator.wikimedia.org/T416948)'
|
|
2026-04-17 19:16:02
|
<wikibugs>
|
('PS1) ''CDanis: kubeadm: add kubectl wait-job plugin [puppet] - ''https://gerrit.wikimedia.org/r/1273926'
|
|
2026-04-17 19:16:45
|
<wikibugs>
|
('CR) ''CDobbins: [C:''+2] data.yaml: remove legacy ssh key; add fido backup [puppet] - ''https://gerrit.wikimedia.org/r/1273925 (owner: ''CDobbins)'
|
|
2026-04-17 19:16:55
|
<wikibugs>
|
('CR) ''CDanis: "This is admittedly untested (too many prereq patches in this stack), but should work :)" [puppet] - ''https://gerrit.wikimedia.org/r/1271028 (https://phabricator.wikimedia.org/T416948) (owner: ''CDanis)'
|
|
2026-04-17 19:17:17
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding phab1006 to eqiad - jclark@cumin1003"
|
|
2026-04-17 19:17:23
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding phab1006 to eqiad - jclark@cumin1003"
|
|
2026-04-17 19:17:23
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
|
|
2026-04-17 19:17:36
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.dns.netbox
|
|
2026-04-17 19:18:25
|
<wikibugs>
|
('CR) ''CI reject: [V:''-1] fundraising_data_import maintenance script wrapper & timer [puppet] - ''https://gerrit.wikimedia.org/r/1271028 (https://phabricator.wikimedia.org/T416948) (owner: ''CDanis)'
|
|
2026-04-17 19:18:43
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.provision for host mc1072.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 19:19:02
|
<icinga-wm>
|
PROBLEM - OSPF status on cr1-eqiad is CRITICAL: OSPFv2: 3/5 UP : OSPFv3: 3/5 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status
|
|
2026-04-17 19:19:08
|
<icinga-wm>
|
PROBLEM - OSPF status on cr2-eqdfw is CRITICAL: OSPFv2: 5/7 UP : OSPFv3: 4/7 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status
|
|
2026-04-17 19:19:30
|
<icinga-wm>
|
PROBLEM - OSPF status on cr2-esams is CRITICAL: OSPFv2: 1/3 UP : OSPFv3: 2/3 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status
|
|
2026-04-17 19:19:58
|
<icinga-wm>
|
RECOVERY - OSPF status on cr1-eqiad is OK: OSPFv2: 5/5 UP : OSPFv3: 5/5 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status
|
|
2026-04-17 19:20:08
|
<icinga-wm>
|
RECOVERY - OSPF status on cr2-eqdfw is OK: OSPFv2: 7/7 UP : OSPFv3: 7/7 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status
|
|
2026-04-17 19:20:11
|
<jinxer-wm>
|
FIRING: [4x] BFDdown: BFD session down between cr1-eqiad and fe80::7a4f:9b00:d4e:7c0c - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown
|
|
2026-04-17 19:20:30
|
<icinga-wm>
|
RECOVERY - OSPF status on cr2-esams is OK: OSPFv2: 3/3 UP : OSPFv3: 3/3 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status
|
|
2026-04-17 19:20:40
|
<jinxer-wm>
|
FIRING: [7x] CoreBGPDown: Core BGP session down between cr1-eqiad and cr2-esams (185.15.59.145) - group Confed_esams - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown
|
|
2026-04-17 19:21:11
|
<wikibugs>
|
('CR) ''Dzahn: [V:''+1] "https://puppet-compiler.wmflabs.org/output/1270580/8442/zuul1001.eqiad.wmnet/index.html"; [puppet] - ''https://gerrit.wikimedia.org/r/1270580 (https://phabricator.wikimedia.org/T422895) (owner: ''Dzahn)'
|
|
2026-04-17 19:21:26
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.provision for host mc1071.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 19:21:27
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding phab1006 to eqiad - jclark@cumin1003"
|
|
2026-04-17 19:21:33
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding phab1006 to eqiad - jclark@cumin1003"
|
|
2026-04-17 19:21:33
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
|
|
2026-04-17 19:21:40
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.provision for host mc1070.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 19:21:48
|
<wikibugs>
|
('PS8) ''CDanis: fundraising_data_import maintenance script wrapper & timer [puppet] - ''https://gerrit.wikimedia.org/r/1271028 (https://phabricator.wikimedia.org/T416948)'
|
|
2026-04-17 19:22:01
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.provision for host mc1069.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 19:22:07
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.provision for host mc1068.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 19:22:18
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.provision for host mc1067.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 19:22:34
|
<wikibugs>
|
('CR) ''Bking: [C:''+2] "self-merging, as PCC suggests only the test server will be affected." [puppet] - ''https://gerrit.wikimedia.org/r/1273887 (https://phabricator.wikimedia.org/T422860) (owner: ''Bking)'
|
|
2026-04-17 19:23:28
|
<wikibugs>
|
('CR) ''Dduvall: "So smart! 😊" [puppet] - ''https://gerrit.wikimedia.org/r/1272970 (https://phabricator.wikimedia.org/T406384) (owner: ''Dduvall)'
|
|
2026-04-17 19:24:41
|
<logmsgbot>
|
!log jclark@cumin1003 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host mc1072.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 19:25:11
|
<jinxer-wm>
|
FIRING: [5x] BFDdown: BFD session down between cr1-eqiad and fe80::ee38:7300:ce8:9c56 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown
|
|
2026-04-17 19:25:30
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.provision for host mc1066.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 19:25:40
|
<jinxer-wm>
|
FIRING: [8x] CoreBGPDown: Core BGP session down between cr1-eqiad and cr2-esams (185.15.59.145) - group Confed_esams - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown
|
|
2026-04-17 19:25:59
|
<jinxer-wm>
|
FIRING: KubernetesDeploymentUnavailableReplicas: ...
|
|
2026-04-17 19:25:59
|
<jinxer-wm>
|
Deployment linkrecommendation-internal in linkrecommendation at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s&var-namespace=linkrecommendation&var-deployment=linkrecommendation-internal - ...
|
|
2026-04-17 19:25:59
|
<jinxer-wm>
|
https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas
|
|
2026-04-17 19:28:04
|
<wikibugs>
|
('CR) ''Dzahn: [V:''+1 C:''+2] zuul: make gerrit ssh key configurable in Hiera and add it [puppet] - ''https://gerrit.wikimedia.org/r/1270580 (https://phabricator.wikimedia.org/T422895) (owner: ''Dzahn)'
|
|
2026-04-17 19:32:14
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mc1071.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 19:32:53
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mc1069.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 19:32:59
|
<logmsgbot>
|
!log otto@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply
|
|
2026-04-17 19:33:01
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mc1070.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 19:33:03
|
<logmsgbot>
|
!log otto@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply
|
|
2026-04-17 19:33:07
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mc1068.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 19:33:12
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mc1067.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 19:33:22
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.provision for host mc1064.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 19:33:24
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.provision for host mc1063.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 19:33:28
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.provision for host mc1065.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 19:33:37
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.provision for host mc1062.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 19:35:04
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.provision for host mc1061.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 19:35:11
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.provision for host mc1059.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 19:35:30
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.provision for host mc1060.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 19:35:32
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.provision for host mc1058.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 19:36:12
|
<logmsgbot>
|
!log jclark@cumin1003 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host mc1058.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 19:36:26
|
<wikibugs>
|
('CR) ''Dzahn: [V:''+1 C:''+2] "The error message we wanted to see. No more other problems and the key is there - it just has not been added on the Gerrit side: "paramiko" [puppet] - ''https://gerrit.wikimedia.org/r/1270580 (https://phabricator.wikimedia.org/T422895) (owner: ''Dzahn)'
|
|
2026-04-17 19:36:42
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mc1066.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 19:36:43
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.provision for host mc1057.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 19:37:02
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.provision for host mc1058.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 19:37:26
|
<logmsgbot>
|
!log jclark@cumin1003 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host mc1058.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 19:37:47
|
<wikibugs>
|
('PS1) ''Bking: opensearch: move var up so we can use it earlier [puppet] - ''https://gerrit.wikimedia.org/r/1273937 (https://phabricator.wikimedia.org/T422860)'
|
|
2026-04-17 19:38:21
|
<wikibugs>
|
('CR) ''Bking: "check experimental" [puppet] - ''https://gerrit.wikimedia.org/r/1273937 (https://phabricator.wikimedia.org/T422860) (owner: ''Bking)'
|
|
2026-04-17 19:38:43
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.provision for host mc1056.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 19:40:56
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mc1063.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 19:41:15
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.provision for host mc1055.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 19:42:03
|
<wikibugs>
|
('CR) ''Bking: [C:''+2] opensearch: move var up so we can use it earlier [puppet] - ''https://gerrit.wikimedia.org/r/1273937 (https://phabricator.wikimedia.org/T422860) (owner: ''Bking)'
|
|
2026-04-17 19:44:15
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mc1064.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 19:44:25
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mc1065.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 19:44:32
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mc1062.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 19:46:08
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mc1061.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 19:47:21
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mc1059.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 19:48:11
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.provision for host mc1058.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 19:48:44
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mc1057.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 19:49:04
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mc1060.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 19:49:31
|
<wikibugs>
|
('PS2) ''CDanis: deployment_server: add kubectl wait-job plugin [puppet] - ''https://gerrit.wikimedia.org/r/1273926'
|
|
2026-04-17 19:49:31
|
<wikibugs>
|
('PS9) ''CDanis: fundraising_data_import maintenance script wrapper & timer [puppet] - ''https://gerrit.wikimedia.org/r/1271028 (https://phabricator.wikimedia.org/T416948)'
|
|
2026-04-17 19:50:35
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mc1056.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 19:50:40
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.reimage for host mc1060.eqiad.wmnet with OS bookworm
|
|
2026-04-17 19:50:45
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11834773 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host mc1060.eqiad.wmnet with OS bookworm'
|
|
2026-04-17 19:53:31
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mc1055.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 19:55:17
|
<wikibugs>
|
('PS1) ''Bking: cloudelastic1012: remove the deliberately-introduced typo [puppet] - ''https://gerrit.wikimedia.org/r/1273943 (https://phabricator.wikimedia.org/T423327)'
|
|
2026-04-17 19:57:20
|
<wikibugs>
|
('CR) ''Bking: [C:''+2] cloudelastic1012: remove the deliberately-introduced typo [puppet] - ''https://gerrit.wikimedia.org/r/1273943 (https://phabricator.wikimedia.org/T423327) (owner: ''Bking)'
|
|
2026-04-17 19:59:04
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mc1058.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 20:03:24
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.reimage for host mc1055.eqiad.wmnet with OS bookworm
|
|
2026-04-17 20:03:35
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11834789 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host mc1055.eqiad.wmnet with OS bookworm'
|
|
2026-04-17 20:03:47
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.reimage for host mc1056.eqiad.wmnet with OS bookworm
|
|
2026-04-17 20:03:52
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11834790 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host mc1056.eqiad.wmnet with OS bookworm'
|
|
2026-04-17 20:04:08
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.reimage for host mc1057.eqiad.wmnet with OS bookworm
|
|
2026-04-17 20:04:13
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11834791 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host mc1057.eqiad.wmnet with OS bookworm'
|
|
2026-04-17 20:06:12
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on mc1060.eqiad.wmnet with reason: host reimage
|
|
2026-04-17 20:08:30
|
<jinxer-wm>
|
FIRING: Outbound discards: Alert for device asw2-b-eqiad.mgmt.eqiad.wmnet - Outbound discards - https://alerts.wikimedia.org/?q=alertname%3DOutbound+discards
|
|
2026-04-17 20:09:11
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.reimage for host mc1059.eqiad.wmnet with OS bookworm
|
|
2026-04-17 20:09:15
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.reimage for host mc1064.eqiad.wmnet with OS bookworm
|
|
2026-04-17 20:09:17
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11834792 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host mc1059.eqiad.wmnet with OS bookworm'
|
|
2026-04-17 20:09:20
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11834793 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host mc1064.eqiad.wmnet with OS bookworm'
|
|
2026-04-17 20:10:21
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.reimage for host mc1061.eqiad.wmnet with OS bookworm
|
|
2026-04-17 20:10:29
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1060.eqiad.wmnet with reason: host reimage
|
|
2026-04-17 20:10:30
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.reimage for host mc1062.eqiad.wmnet with OS bookworm
|
|
2026-04-17 20:10:32
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11834794 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host mc1061.eqiad.wmnet with OS bookworm'
|
|
2026-04-17 20:10:35
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11834795 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host mc1062.eqiad.wmnet with OS bookworm'
|
|
2026-04-17 20:13:21
|
<mutante>
|
!log planet1003, planet2003 - rebooting on ganeti level for T422596
|
|
2026-04-17 20:13:25
|
<stashbot>
|
Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
|
|
2026-04-17 20:13:25
|
<stashbot>
|
T422596: Failing Trixie VM installations on routed Ganeti - https://phabricator.wikimedia.org/T422596
|
|
2026-04-17 20:14:16
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on mc1055.eqiad.wmnet with reason: host reimage
|
|
2026-04-17 20:14:57
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on mc1057.eqiad.wmnet with reason: host reimage
|
|
2026-04-17 20:15:08
|
<logmsgbot>
|
!log jhancock@cumin2002 START - Cookbook sre.dns.netbox
|
|
2026-04-17 20:17:23
|
<logmsgbot>
|
!log jhancock@cumin2002 END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
|
|
2026-04-17 20:17:28
|
<logmsgbot>
|
!log jhancock@cumin2002 START - Cookbook sre.dns.netbox
|
|
2026-04-17 20:20:09
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on mc1064.eqiad.wmnet with reason: host reimage
|
|
2026-04-17 20:20:12
|
<wikibugs>
|
'SRE, ''Infrastructure-Foundations: Failing Trixie VM installations on routed Ganeti - https://phabricator.wikimedia.org/T422596#11834804 (''Dzahn) I did: ` [ganeti1046:~] $ sudo gnt-instance modify -B memory=2G planet1003.eqiad.wmnet .. [ganeti1046:~] $ sudo gnt-instance reboot planet1003.eqiad.wmnet .....'
|
|
2026-04-17 20:20:13
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on mc1059.eqiad.wmnet with reason: host reimage
|
|
2026-04-17 20:20:19
|
<wikibugs>
|
('PS10) ''Kamila Součková: mw-web: Remove the hard-coded k8s version [deployment-charts] - ''https://gerrit.wikimedia.org/r/1201804 (https://phabricator.wikimedia.org/T388969)'
|
|
2026-04-17 20:20:19
|
<wikibugs>
|
('PS1) ''Kamila Součková: Remove k8s version from all services [deployment-charts] - ''https://gerrit.wikimedia.org/r/1273967 (https://phabricator.wikimedia.org/T388969)'
|
|
2026-04-17 20:20:30
|
<wikibugs>
|
'SRE, ''collaboration-services, ''Infrastructure-Foundations: Failing Trixie VM installations on routed Ganeti - https://phabricator.wikimedia.org/T422596#11834806 (''Dzahn)'
|
|
2026-04-17 20:20:49
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1055.eqiad.wmnet with reason: host reimage
|
|
2026-04-17 20:20:54
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on mc1061.eqiad.wmnet with reason: host reimage
|
|
2026-04-17 20:21:14
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on mc1062.eqiad.wmnet with reason: host reimage
|
|
2026-04-17 20:21:51
|
<logmsgbot>
|
!log jhancock@cumin2002 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding db2250 to codfw - jhancock@cumin2002"
|
|
2026-04-17 20:21:56
|
<logmsgbot>
|
!log jhancock@cumin2002 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding db2250 to codfw - jhancock@cumin2002"
|
|
2026-04-17 20:21:56
|
<logmsgbot>
|
!log jhancock@cumin2002 END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
|
|
2026-04-17 20:22:07
|
<wikibugs>
|
('CR) ''CI reject: [V:''-1] mw-web: Remove the hard-coded k8s version [deployment-charts] - ''https://gerrit.wikimedia.org/r/1201804 (https://phabricator.wikimedia.org/T388969) (owner: ''Kamila Součková)'
|
|
2026-04-17 20:22:13
|
<logmsgbot>
|
!log jhancock@cumin2002 START - Cookbook sre.network.configure-switch-interfaces for host db2250
|
|
2026-04-17 20:22:17
|
<wikibugs>
|
('CR) ''CI reject: [V:''-1] Remove k8s version from all services [deployment-charts] - ''https://gerrit.wikimedia.org/r/1273967 (https://phabricator.wikimedia.org/T388969) (owner: ''Kamila Součková)'
|
|
2026-04-17 20:22:34
|
<logmsgbot>
|
!log jhancock@cumin2002 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host db2250
|
|
2026-04-17 20:22:37
|
<logmsgbot>
|
!log jhancock@cumin2002 START - Cookbook sre.network.configure-switch-interfaces for host db2251
|
|
2026-04-17 20:22:46
|
<logmsgbot>
|
!log jhancock@cumin2002 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host db2251
|
|
2026-04-17 20:22:57
|
<logmsgbot>
|
!log jhancock@cumin2002 START - Cookbook sre.network.configure-switch-interfaces for host db2252
|
|
2026-04-17 20:23:06
|
<logmsgbot>
|
!log jhancock@cumin2002 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host db2252
|
|
2026-04-17 20:23:10
|
<logmsgbot>
|
!log jhancock@cumin2002 START - Cookbook sre.network.configure-switch-interfaces for host db2253
|
|
2026-04-17 20:23:19
|
<logmsgbot>
|
!log jhancock@cumin2002 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host db2253
|
|
2026-04-17 20:24:44
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1064.eqiad.wmnet with reason: host reimage
|
|
2026-04-17 20:25:13
|
<logmsgbot>
|
!log jhancock@cumin2002 START - Cookbook sre.dns.netbox
|
|
2026-04-17 20:25:26
|
<wikibugs>
|
('PS2) ''Dzahn: ci: switch jenkins proxy target to new discovery name [puppet] - ''https://gerrit.wikimedia.org/r/1254308 (https://phabricator.wikimedia.org/T418521)'
|
|
2026-04-17 20:28:01
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
|
|
2026-04-17 20:28:33
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
|
|
2026-04-17 20:28:34
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1060.eqiad.wmnet with OS bookworm
|
|
2026-04-17 20:28:39
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11834825 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host mc1060.eqiad.wmnet with OS bookworm completed: - mc1...'
|
|
2026-04-17 20:28:44
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1061.eqiad.wmnet with reason: host reimage
|
|
2026-04-17 20:28:49
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.reimage for host mc1063.eqiad.wmnet with OS bookworm
|
|
2026-04-17 20:28:56
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11834826 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host mc1063.eqiad.wmnet with OS bookworm'
|
|
2026-04-17 20:29:01
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.reimage for host mc1058.eqiad.wmnet with OS bookworm
|
|
2026-04-17 20:29:05
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11834827 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host mc1058.eqiad.wmnet with OS bookworm'
|
|
2026-04-17 20:29:53
|
<wikibugs>
|
('CR) ''Kamila Součková: "OF COURSE SOMEBODY MERGES A COMPLETELY UNRELATED CHANGE THAT BREAKS CI THE MOMENT I FINALLY GET THIS MONTHS OLD THING TO NOT BREAK CI XD" [deployment-charts] - ''https://gerrit.wikimedia.org/r/1201804 (https://phabricator.wikimedia.org/T388969) (owner: ''Kamila Součková)'
|
|
2026-04-17 20:30:25
|
<jinxer-wm>
|
FIRING: SystemdUnitFailed: check_netbox_uncommitted_dns_changes.service on netbox1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
|
|
2026-04-17 20:31:36
|
<logmsgbot>
|
!log jhancock@cumin2002 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding pc2021 to codfw - jhancock@cumin2002"
|
|
2026-04-17 20:31:41
|
<logmsgbot>
|
!log jhancock@cumin2002 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding pc2021 to codfw - jhancock@cumin2002"
|
|
2026-04-17 20:31:42
|
<logmsgbot>
|
!log jhancock@cumin2002 END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
|
|
2026-04-17 20:31:48
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.reimage for host mc1065.eqiad.wmnet with OS bookworm
|
|
2026-04-17 20:31:53
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11834837 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host mc1065.eqiad.wmnet with OS bookworm'
|
|
2026-04-17 20:32:09
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1062.eqiad.wmnet with reason: host reimage
|
|
2026-04-17 20:33:56
|
<logmsgbot>
|
!log jhancock@cumin2002 START - Cookbook sre.network.configure-switch-interfaces for host pc2021
|
|
2026-04-17 20:34:06
|
<logmsgbot>
|
!log jhancock@cumin2002 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2021
|
|
2026-04-17 20:34:09
|
<logmsgbot>
|
!log jhancock@cumin2002 START - Cookbook sre.network.configure-switch-interfaces for host pc2022
|
|
2026-04-17 20:34:23
|
<logmsgbot>
|
!log jhancock@cumin2002 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2022
|
|
2026-04-17 20:34:26
|
<logmsgbot>
|
!log jhancock@cumin2002 START - Cookbook sre.network.configure-switch-interfaces for host pc2023
|
|
2026-04-17 20:34:36
|
<logmsgbot>
|
!log jhancock@cumin2002 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2023
|
|
2026-04-17 20:34:54
|
<logmsgbot>
|
!log jhancock@cumin2002 START - Cookbook sre.network.configure-switch-interfaces for host pc2024
|
|
2026-04-17 20:35:04
|
<logmsgbot>
|
!log jhancock@cumin2002 END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host pc2024
|
|
2026-04-17 20:35:25
|
<jinxer-wm>
|
RESOLVED: SystemdUnitFailed: check_netbox_uncommitted_dns_changes.service on netbox1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
|
|
2026-04-17 20:36:42
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1057.eqiad.wmnet with reason: host reimage
|
|
2026-04-17 20:36:48
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
|
|
2026-04-17 20:36:53
|
<icinga-wm>
|
RECOVERY - MegaRAID on pc1011 is OK: OK: optimal, 1 logical, 10 physical, WriteBack policy https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring
|
|
2026-04-17 20:37:07
|
<logmsgbot>
|
!log jhancock@cumin2002 START - Cookbook sre.hosts.provision for host pc2021.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
|
|
2026-04-17 20:37:23
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
|
|
2026-04-17 20:37:23
|
<logmsgbot>
|
!log jhancock@cumin2002 START - Cookbook sre.hosts.provision for host pc2022.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
|
|
2026-04-17 20:37:24
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1055.eqiad.wmnet with OS bookworm
|
|
2026-04-17 20:37:36
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11834858 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host mc1055.eqiad.wmnet with OS bookworm completed: - mc1...'
|
|
2026-04-17 20:37:43
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.reimage for host mc1066.eqiad.wmnet with OS bookworm
|
|
2026-04-17 20:37:45
|
<logmsgbot>
|
!log jhancock@cumin2002 START - Cookbook sre.hosts.provision for host pc2023.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
|
|
2026-04-17 20:37:49
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11834859 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host mc1066.eqiad.wmnet with OS bookworm'
|
|
2026-04-17 20:37:59
|
<wikibugs>
|
('PS1) ''Dzahn: etherpad: add 90 day deletion warning to default pad message [puppet] - ''https://gerrit.wikimedia.org/r/1273991 (https://phabricator.wikimedia.org/T421315)'
|
|
2026-04-17 20:38:01
|
<logmsgbot>
|
!log jhancock@cumin2002 START - Cookbook sre.hosts.provision for host pc2024.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
|
|
2026-04-17 20:38:55
|
<wikibugs>
|
('CR) ''Dzahn: [C:''+2] etherpad: add 90 day deletion warning to default pad message [puppet] - ''https://gerrit.wikimedia.org/r/1273991 (https://phabricator.wikimedia.org/T421315) (owner: ''Dzahn)'
|
|
2026-04-17 20:39:08
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on mc1063.eqiad.wmnet with reason: host reimage
|
|
2026-04-17 20:40:06
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on mc1058.eqiad.wmnet with reason: host reimage
|
|
2026-04-17 20:42:03
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1059.eqiad.wmnet with reason: host reimage
|
|
2026-04-17 20:42:59
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on mc1065.eqiad.wmnet with reason: host reimage
|
|
2026-04-17 20:43:00
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
|
|
2026-04-17 20:43:22
|
<wikibugs>
|
('CR) ''Catrope: Set CSP to enforce with currently-allow-listed domains on Beta (''1 comment) [mediawiki-config] - ''https://gerrit.wikimedia.org/r/1272895 (https://phabricator.wikimedia.org/T419612) (owner: ''SBassett)'
|
|
2026-04-17 20:43:23
|
<jinxer-wm>
|
FIRING: [4x] ProbeDown: Service aqs1010-a:7000 has failed probes (tcp_cassandra_a_ssl_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
|
|
2026-04-17 20:43:23
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
|
|
2026-04-17 20:43:24
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1064.eqiad.wmnet with OS bookworm
|
|
2026-04-17 20:43:28
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11834898 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host mc1064.eqiad.wmnet with OS bookworm completed: - mc1...'
|
|
2026-04-17 20:45:04
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1058.eqiad.wmnet with reason: host reimage
|
|
2026-04-17 20:45:05
|
<wikibugs>
|
'ops-eqiad, ''SRE, ''DBA, ''DC-Ops: Degraded RAID on pc1011 - https://phabricator.wikimedia.org/T423630#11834899 (''Jclark-ctr) ''Open→''Resolved Drive restored error
cleared'
|
|
2026-04-17 20:46:45
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
|
|
2026-04-17 20:47:26
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
|
|
2026-04-17 20:47:27
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1061.eqiad.wmnet with OS bookworm
|
|
2026-04-17 20:47:32
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11834904 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host mc1061.eqiad.wmnet with OS bookworm completed: - mc1...'
|
|
2026-04-17 20:47:35
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11834905 (''Jclark-ctr)'
|
|
2026-04-17 20:47:51
|
<logmsgbot>
|
!log jhancock@cumin2002 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pc2021.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
|
|
2026-04-17 20:48:33
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on mc1066.eqiad.wmnet with reason: host reimage
|
|
2026-04-17 20:48:34
|
<logmsgbot>
|
!log jhancock@cumin2002 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pc2023.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
|
|
2026-04-17 20:48:46
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11834906 (''Jclark-ctr)'
|
|
2026-04-17 20:48:50
|
<logmsgbot>
|
!log jhancock@cumin2002 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pc2024.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
|
|
2026-04-17 20:49:36
|
<wikibugs>
|
('PS1) ''Dzahn: etherpad: stop puppet from adding the same file_line multiple times [puppet] - ''https://gerrit.wikimedia.org/r/1274005 (https://phabricator.wikimedia.org/T420793)'
|
|
2026-04-17 20:50:07
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
|
|
2026-04-17 20:50:33
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1065.eqiad.wmnet with reason: host reimage
|
|
2026-04-17 20:50:34
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
|
|
2026-04-17 20:50:35
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1062.eqiad.wmnet with OS bookworm
|
|
2026-04-17 20:50:38
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11834911 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host mc1062.eqiad.wmnet with OS bookworm completed: - mc1...'
|
|
2026-04-17 20:51:40
|
<wikibugs>
|
('PS2) ''Dzahn: etherpad: stop puppet from adding the same file_line multiple times [puppet] - ''https://gerrit.wikimedia.org/r/1274005 (https://phabricator.wikimedia.org/T420793)'
|
|
2026-04-17 20:51:44
|
<logmsgbot>
|
!log jhancock@cumin2002 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pc2022.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
|
|
2026-04-17 20:53:36
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
|
|
2026-04-17 20:54:00
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
|
|
2026-04-17 20:54:01
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1057.eqiad.wmnet with OS bookworm
|
|
2026-04-17 20:54:06
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11834917 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host mc1057.eqiad.wmnet with OS bookworm completed: - mc1...'
|
|
2026-04-17 20:54:28
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11834918 (''Jclark-ctr)'
|
|
2026-04-17 20:54:30
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11834919 (''Jclark-ctr)'
|
|
2026-04-17 20:54:35
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1066.eqiad.wmnet with reason: host reimage
|
|
2026-04-17 20:55:01
|
<wikibugs>
|
'ops-eqiad, ''SRE, ''Data-Persistence, ''DC-Ops: Q4:rack/setup/install pc102[1-4] - https://phabricator.wikimedia.org/T418908#11834920 (''VRiley-WMF) a:''VRiley-WMF'
|
|
2026-04-17 20:55:10
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.reimage for host mc1067.eqiad.wmnet with OS bookworm
|
|
2026-04-17 20:55:14
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11834921 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host mc1067.eqiad.wmnet with OS bookworm'
|
|
2026-04-17 20:56:19
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.reimage for host mc1068.eqiad.wmnet with OS bookworm
|
|
2026-04-17 20:56:26
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11834922 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host mc1068.eqiad.wmnet with OS bookworm'
|
|
2026-04-17 20:59:20
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1063.eqiad.wmnet with reason: host reimage
|
|
2026-04-17 20:59:37
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
|
|
2026-04-17 20:59:49
|
<wikibugs>
|
('CR) ''Dzahn: [C:''+2] etherpad: stop puppet from adding the same file_line multiple times [puppet] - ''https://gerrit.wikimedia.org/r/1274005 (https://phabricator.wikimedia.org/T420793) (owner: ''Dzahn)'
|
|
2026-04-17 21:00:37
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
|
|
2026-04-17 21:00:40
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1059.eqiad.wmnet with OS bookworm
|
|
2026-04-17 21:00:49
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11834929 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host mc1059.eqiad.wmnet with OS bookworm completed: - mc1...'
|
|
2026-04-17 21:01:10
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11834930 (''Jclark-ctr)'
|
|
2026-04-17 21:02:02
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
|
|
2026-04-17 21:02:07
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.reimage for host mc1069.eqiad.wmnet with OS bookworm
|
|
2026-04-17 21:02:14
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11834936 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host mc1069.eqiad.wmnet with OS bookworm'
|
|
2026-04-17 21:02:21
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
|
|
2026-04-17 21:02:22
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1058.eqiad.wmnet with OS bookworm
|
|
2026-04-17 21:02:27
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11834937 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host mc1058.eqiad.wmnet with OS bookworm completed: - mc1...'
|
|
2026-04-17 21:02:44
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11834938 (''Jclark-ctr)'
|
|
2026-04-17 21:03:07
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.reimage for host mc1070.eqiad.wmnet with OS bookworm
|
|
2026-04-17 21:03:14
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11834939 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host mc1070.eqiad.wmnet with OS bookworm'
|
|
2026-04-17 21:03:18
|
<wikibugs>
|
('PS1) ''CDanis: More features and fixes for historical API [software/klaxon] - ''https://gerrit.wikimedia.org/r/1274025'
|
|
2026-04-17 21:03:18
|
<wikibugs>
|
('PS1) ''CDanis: Add SQLite-backed incident store, migrations, and weekly analytics CLI [software/klaxon] - ''https://gerrit.wikimedia.org/r/1274026'
|
|
2026-04-17 21:03:29
|
<logmsgbot>
|
!log jhancock@cumin2002 START - Cookbook sre.hosts.provision for host pc2021.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
|
|
2026-04-17 21:03:45
|
<logmsgbot>
|
!log jhancock@cumin2002 START - Cookbook sre.hosts.provision for host pc2022.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
|
|
2026-04-17 21:04:01
|
<logmsgbot>
|
!log jhancock@cumin2002 START - Cookbook sre.hosts.provision for host pc2023.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
|
|
2026-04-17 21:04:17
|
<logmsgbot>
|
!log jhancock@cumin2002 START - Cookbook sre.hosts.provision for host pc2024.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
|
|
2026-04-17 21:04:53
|
<wikibugs>
|
('CR) ''CI reject: [V:''-1] Add SQLite-backed incident store, migrations, and weekly analytics CLI [software/klaxon] - ''https://gerrit.wikimedia.org/r/1274026 (owner: ''CDanis)'
|
|
2026-04-17 21:05:19
|
<wikibugs>
|
('PS2) ''CDanis: Add SQLite-backed incident store, migrations, and weekly analytics CLI [software/klaxon] - ''https://gerrit.wikimedia.org/r/1274026'
|
|
2026-04-17 21:06:35
|
<wikibugs>
|
('CR) ''CI reject: [V:''-1] Add SQLite-backed incident store, migrations, and weekly analytics CLI [software/klaxon] - ''https://gerrit.wikimedia.org/r/1274026 (owner: ''CDanis)'
|
|
2026-04-17 21:06:55
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on mc1068.eqiad.wmnet with reason: host reimage
|
|
2026-04-17 21:07:33
|
<logmsgbot>
|
!log jhancock@cumin2002 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pc2021.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
|
|
2026-04-17 21:07:41
|
<logmsgbot>
|
!log jclark@cumin1003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host mc1065.eqiad.wmnet with OS bookworm
|
|
2026-04-17 21:07:51
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11834964 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host mc1065.eqiad.wmnet with OS bookworm executed with er...'
|
|
2026-04-17 21:08:06
|
<jinxer-wm>
|
FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - cr2-codfw:et-0/1/4 (Transport: cr2-eqiad:et-1/1/5 (Lumen, 449169461) {#3909}) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown
|
|
2026-04-17 21:08:36
|
<wikibugs>
|
('PS3) ''CDanis: WIP [DNM] Add SQLite-backed incident store and analytics CLI [software/klaxon] - ''https://gerrit.wikimedia.org/r/1274026'
|
|
2026-04-17 21:09:04
|
<icinga-wm>
|
PROBLEM - OSPF status on cr1-drmrs is CRITICAL: OSPFv2: 2/3 UP : OSPFv3: 2/3 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status
|
|
2026-04-17 21:09:49
|
<wikibugs>
|
('CR) ''CI reject: [V:''-1] WIP [DNM] Add SQLite-backed incident store and analytics CLI [software/klaxon] - ''https://gerrit.wikimedia.org/r/1274026 (owner: ''CDanis)'
|
|
2026-04-17 21:09:51
|
<logmsgbot>
|
!log jhancock@cumin2002 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pc2022.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
|
|
2026-04-17 21:09:58
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.reimage for host mc1065.eqiad.wmnet with OS bookworm
|
|
2026-04-17 21:10:05
|
<logmsgbot>
|
!log jhancock@cumin2002 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pc2023.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
|
|
2026-04-17 21:10:09
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11834967 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host mc1065.eqiad.wmnet with OS bookworm'
|
|
2026-04-17 21:10:20
|
<logmsgbot>
|
!log jhancock@cumin2002 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host pc2024.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
|
|
2026-04-17 21:10:49
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1068.eqiad.wmnet with reason: host reimage
|
|
2026-04-17 21:12:47
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
|
|
2026-04-17 21:12:51
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on mc1069.eqiad.wmnet with reason: host reimage
|
|
2026-04-17 21:13:12
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
|
|
2026-04-17 21:13:13
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1066.eqiad.wmnet with OS bookworm
|
|
2026-04-17 21:13:16
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11834985 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host mc1066.eqiad.wmnet with OS bookworm completed: - mc1...'
|
|
2026-04-17 21:13:38
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11834986 (''Jclark-ctr)'
|
|
2026-04-17 21:14:16
|
<logmsgbot>
|
!log jhancock@cumin2002 START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['pc2021']
|
|
2026-04-17 21:14:31
|
<logmsgbot>
|
!log jhancock@cumin2002 END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['pc2021']
|
|
2026-04-17 21:14:54
|
<jinxer-wm>
|
FIRING: [2x] TransitBGPDown: Transit BGP session down between cr1-magru and Telxius (2001:1498:1:966:1::251) - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DTransitBGPDown
|
|
2026-04-17 21:15:34
|
<logmsgbot>
|
!log jhancock@cumin2002 START - Cookbook sre.hosts.reimage for host pc2021.codfw.wmnet with OS trixie
|
|
2026-04-17 21:15:42
|
<wikibugs>
|
'ops-codfw, ''SRE, ''Data-Persistence, ''DC-Ops: Q3:rack/setup/install pc202[1-4] - https://phabricator.wikimedia.org/T418907#11835004 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host pc2021.codfw.wmnet with OS trixie'
|
|
2026-04-17 21:15:52
|
<logmsgbot>
|
!log jhancock@cumin2002 START - Cookbook sre.hosts.reimage for host pc2022.codfw.wmnet with OS trixie
|
|
2026-04-17 21:16:00
|
<wikibugs>
|
'ops-codfw, ''SRE, ''Data-Persistence, ''DC-Ops: Q3:rack/setup/install pc202[1-4] - https://phabricator.wikimedia.org/T418907#11835005 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host pc2022.codfw.wmnet with OS trixie'
|
|
2026-04-17 21:16:03
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
|
|
2026-04-17 21:16:10
|
<wikibugs>
|
('CR) ''Dzahn: [C:''-1] "21 needs to be a string" [puppet] - ''https://gerrit.wikimedia.org/r/1271032 (https://phabricator.wikimedia.org/T418109) (owner: ''Dzahn)'
|
|
2026-04-17 21:16:18
|
<logmsgbot>
|
!log jhancock@cumin2002 START - Cookbook sre.hosts.reimage for host pc2023.codfw.wmnet with OS trixie
|
|
2026-04-17 21:16:29
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
|
|
2026-04-17 21:16:29
|
<wikibugs>
|
'ops-codfw, ''SRE, ''Data-Persistence, ''DC-Ops: Q3:rack/setup/install pc202[1-4] - https://phabricator.wikimedia.org/T418907#11835006 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host pc2023.codfw.wmnet with OS trixie'
|
|
2026-04-17 21:16:30
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1063.eqiad.wmnet with OS bookworm
|
|
2026-04-17 21:16:31
|
<logmsgbot>
|
!log jhancock@cumin2002 START - Cookbook sre.hosts.reimage for host pc2024.codfw.wmnet with OS trixie
|
|
2026-04-17 21:16:34
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11835007 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host mc1063.eqiad.wmnet with OS bookworm completed: - mc1...'
|
|
2026-04-17 21:16:38
|
<wikibugs>
|
'ops-codfw, ''SRE, ''Data-Persistence, ''DC-Ops: Q3:rack/setup/install pc202[1-4] - https://phabricator.wikimedia.org/T418907#11835008 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jhancock@cumin2002 for host pc2024.codfw.wmnet with OS trixie'
|
|
2026-04-17 21:16:40
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11835009 (''Jclark-ctr)'
|
|
2026-04-17 21:17:17
|
<wikibugs>
|
('CR) ''SBassett: Set CSP to enforce with currently-allow-listed domains on Beta (''1 comment) [mediawiki-config] - ''https://gerrit.wikimedia.org/r/1272895 (https://phabricator.wikimedia.org/T419612) (owner: ''SBassett)'
|
|
2026-04-17 21:17:27
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11835011 (''Jclark-ctr)'
|
|
2026-04-17 21:18:40
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1069.eqiad.wmnet with reason: host reimage
|
|
2026-04-17 21:21:48
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on mc1065.eqiad.wmnet with reason: host reimage
|
|
2026-04-17 21:23:06
|
<wikibugs>
|
('CR) ''JHathaway: [C:''+1] ipmi: rework how to use a different user (''1 comment) [software/spicerack] - ''https://gerrit.wikimedia.org/r/1271631 (https://phabricator.wikimedia.org/T418929) (owner: ''Elukey)'
|
|
2026-04-17 21:23:36
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.reimage for host mc1071.eqiad.wmnet with OS bookworm
|
|
2026-04-17 21:23:42
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11835020 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host mc1071.eqiad.wmnet with OS bookworm'
|
|
2026-04-17 21:24:04
|
<logmsgbot>
|
!log jclark@cumin1003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host mc1056.eqiad.wmnet with OS bookworm
|
|
2026-04-17 21:24:09
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11835025 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host mc1056.eqiad.wmnet with OS bookworm executed with er...'
|
|
2026-04-17 21:27:26
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
|
|
2026-04-17 21:28:47
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1065.eqiad.wmnet with reason: host reimage
|
|
2026-04-17 21:30:32
|
<logmsgbot>
|
jclark@cumin1003 reimage (PID 2544631) is awaiting input
|
|
2026-04-17 21:32:15
|
<logmsgbot>
|
!log jhancock@cumin2002 START - Cookbook sre.hosts.downtime for 2:00:00 on pc2021.codfw.wmnet with reason: host reimage
|
|
2026-04-17 21:33:06
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.provision for host mc1056.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 21:34:29
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on mc1071.eqiad.wmnet with reason: host reimage
|
|
2026-04-17 21:34:56
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
|
|
2026-04-17 21:34:57
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1068.eqiad.wmnet with OS bookworm
|
|
2026-04-17 21:35:04
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11835039 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host mc1068.eqiad.wmnet with OS bookworm completed: - mc1...'
|
|
2026-04-17 21:35:27
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11835040 (''Jclark-ctr)'
|
|
2026-04-17 21:35:27
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
|
|
2026-04-17 21:35:30
|
<logmsgbot>
|
!log jhancock@cumin2002 START - Cookbook sre.hosts.downtime for 2:00:00 on pc2022.codfw.wmnet with reason: host reimage
|
|
2026-04-17 21:35:46
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
|
|
2026-04-17 21:35:47
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1069.eqiad.wmnet with OS bookworm
|
|
2026-04-17 21:35:51
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11835052 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host mc1069.eqiad.wmnet with OS bookworm completed: - mc1...'
|
|
2026-04-17 21:36:21
|
<logmsgbot>
|
!log jhancock@cumin2002 START - Cookbook sre.hosts.downtime for 2:00:00 on pc2023.codfw.wmnet with reason: host reimage
|
|
2026-04-17 21:36:21
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11835054 (''Jclark-ctr)'
|
|
2026-04-17 21:37:39
|
<logmsgbot>
|
!log jhancock@cumin2002 START - Cookbook sre.hosts.downtime for 2:00:00 on pc2024.codfw.wmnet with reason: host reimage
|
|
2026-04-17 21:39:21
|
<logmsgbot>
|
!log jclark@cumin1003 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host mc1056.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 21:39:36
|
<logmsgbot>
|
!log jhancock@cumin2002 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pc2021.codfw.wmnet with reason: host reimage
|
|
2026-04-17 21:41:08
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.provision for host mc1056.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 21:41:51
|
<wikibugs>
|
('CR) ''Jcrespo: "Absolutely not, but the decom script may warn you incorrectly about references left behind or searching could cause false positives for th" [puppet] - ''https://gerrit.wikimedia.org/r/1273699 (https://phabricator.wikimedia.org/T422851) (owner: ''Jcrespo)'
|
|
2026-04-17 21:42:27
|
<logmsgbot>
|
!log jclark@cumin1003 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host mc1056.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 21:42:36
|
<logmsgbot>
|
!log jhancock@cumin2002 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pc2023.codfw.wmnet with reason: host reimage
|
|
2026-04-17 21:45:06
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1065.eqiad.wmnet with OS bookworm
|
|
2026-04-17 21:45:10
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11835087 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host mc1065.eqiad.wmnet with OS bookworm completed: - mc1...'
|
|
2026-04-17 21:47:23
|
<logmsgbot>
|
!log jhancock@cumin2002 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pc2022.codfw.wmnet with reason: host reimage
|
|
2026-04-17 21:48:10
|
<wikibugs>
|
('PS1) ''Ryan Kemper: cloudelastic1012: override common_settings merge to first [puppet] - ''https://gerrit.wikimedia.org/r/1274061 (https://phabricator.wikimedia.org/T422860)'
|
|
2026-04-17 21:48:45
|
<wikibugs>
|
('CR) ''Ryan Kemper: "check experimental" [puppet] - ''https://gerrit.wikimedia.org/r/1274061 (https://phabricator.wikimedia.org/T422860) (owner: ''Ryan Kemper)'
|
|
2026-04-17 21:49:09
|
<wikibugs>
|
('CR) ''Bking: [C:''+1] cloudelastic1012: override common_settings merge to first [puppet] - ''https://gerrit.wikimedia.org/r/1274061 (https://phabricator.wikimedia.org/T422860) (owner: ''Ryan Kemper)'
|
|
2026-04-17 21:51:01
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1071.eqiad.wmnet with reason: host reimage
|
|
2026-04-17 21:51:19
|
<wikibugs>
|
('PS1) ''Dzahn: ci::docker: do not try to install docker-cli on bookworm [puppet] - ''https://gerrit.wikimedia.org/r/1274067 (https://phabricator.wikimedia.org/T418109)'
|
|
2026-04-17 21:52:11
|
<wikibugs>
|
('CR) ''Ryan Kemper: "PCC looks terrible, this overrode whole common_settings. fixing" [puppet] - ''https://gerrit.wikimedia.org/r/1274061 (https://phabricator.wikimedia.org/T422860) (owner: ''Ryan Kemper)'
|
|
2026-04-17 21:52:43
|
<logmsgbot>
|
jclark@cumin1003 provision (PID 2561685) is awaiting input
|
|
2026-04-17 21:53:06
|
<wikibugs>
|
('CR) ''Catrope: Set CSP to enforce with currently-allow-listed domains on Beta (''1 comment) [mediawiki-config] - ''https://gerrit.wikimedia.org/r/1272895 (https://phabricator.wikimedia.org/T419612) (owner: ''SBassett)'
|
|
2026-04-17 21:54:15
|
<logmsgbot>
|
!log jhancock@cumin2002 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002"
|
|
2026-04-17 21:54:45
|
<logmsgbot>
|
!log jhancock@cumin2002 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002"
|
|
2026-04-17 21:54:46
|
<logmsgbot>
|
!log jhancock@cumin2002 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host pc2021.codfw.wmnet with OS trixie
|
|
2026-04-17 21:54:55
|
<wikibugs>
|
'ops-codfw, ''SRE, ''Data-Persistence, ''DC-Ops: Q3:rack/setup/install pc202[1-4] - https://phabricator.wikimedia.org/T418907#11835103 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host pc2021.codfw.wmnet with OS trixie completed: - pc2021 (**PASS**) - Rem...'
|
|
2026-04-17 21:55:10
|
<wikibugs>
|
('PS1) ''Ryan Kemper: cloudelastic1012: full common_settings override for OS2 [puppet] - ''https://gerrit.wikimedia.org/r/1274075 (https://phabricator.wikimedia.org/T422860)'
|
|
2026-04-17 21:55:56
|
<wikibugs>
|
('Abandoned) ''Ryan Kemper: cloudelastic1012: full common_settings override for OS2 [puppet] - ''https://gerrit.wikimedia.org/r/1274075 (https://phabricator.wikimedia.org/T422860) (owner: ''Ryan Kemper)'
|
|
2026-04-17 21:56:40
|
<wikibugs>
|
('PS2) ''Ryan Kemper: cloudelastic1012: full common_settings override for OS2 [puppet] - ''https://gerrit.wikimedia.org/r/1274061 (https://phabricator.wikimedia.org/T422860)'
|
|
2026-04-17 21:56:56
|
<wikibugs>
|
('CR) ''Ryan Kemper: "check experimental" [puppet] - ''https://gerrit.wikimedia.org/r/1274061 (https://phabricator.wikimedia.org/T422860) (owner: ''Ryan Kemper)'
|
|
2026-04-17 21:57:52
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.provision for host mc1072.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 21:58:51
|
<logmsgbot>
|
!log jhancock@cumin2002 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002"
|
|
2026-04-17 21:59:25
|
<logmsgbot>
|
!log jhancock@cumin2002 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on pc2024.codfw.wmnet with reason: host reimage
|
|
2026-04-17 22:00:12
|
<wikibugs>
|
('CR) ''Ryan Kemper: [C:''+2] "PCC looks correct now; merging" [puppet] - ''https://gerrit.wikimedia.org/r/1274061 (https://phabricator.wikimedia.org/T422860) (owner: ''Ryan Kemper)'
|
|
2026-04-17 22:00:37
|
<wikibugs>
|
('PS2) ''Dzahn: ci::docker: only install docker-cli if on trixie or newer [puppet] - ''https://gerrit.wikimedia.org/r/1274067 (https://phabricator.wikimedia.org/T418109)'
|
|
2026-04-17 22:01:39
|
<logmsgbot>
|
!log jhancock@cumin2002 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002"
|
|
2026-04-17 22:01:40
|
<logmsgbot>
|
!log jhancock@cumin2002 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002"
|
|
2026-04-17 22:01:40
|
<logmsgbot>
|
!log jhancock@cumin2002 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host pc2023.codfw.wmnet with OS trixie
|
|
2026-04-17 22:01:52
|
<wikibugs>
|
'ops-codfw, ''SRE, ''Data-Persistence, ''DC-Ops: Q3:rack/setup/install pc202[1-4] - https://phabricator.wikimedia.org/T418907#11835127 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host pc2023.codfw.wmnet with OS trixie completed: - pc2023 (**PASS**) - Rem...'
|
|
2026-04-17 22:02:08
|
<logmsgbot>
|
!log jhancock@cumin2002 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002"
|
|
2026-04-17 22:02:10
|
<logmsgbot>
|
!log jhancock@cumin2002 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host pc2022.codfw.wmnet with OS trixie
|
|
2026-04-17 22:02:22
|
<wikibugs>
|
'ops-codfw, ''SRE, ''Data-Persistence, ''DC-Ops: Q3:rack/setup/install pc202[1-4] - https://phabricator.wikimedia.org/T418907#11835128 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host pc2022.codfw.wmnet with OS trixie completed: - pc2022 (**PASS**) - Rem...'
|
|
2026-04-17 22:02:50
|
<wikibugs>
|
('CR) ''Thcipriani: [C:''+1] "cherry pick seems to work on integration-docker-agent" [puppet] - ''https://gerrit.wikimedia.org/r/1274067 (https://phabricator.wikimedia.org/T418109) (owner: ''Dzahn)'
|
|
2026-04-17 22:06:19
|
<wikibugs>
|
('PS4) ''SBassett: Set CSP to enforce with currently-allow-listed domains on Beta [mediawiki-config] - ''https://gerrit.wikimedia.org/r/1272895 (https://phabricator.wikimedia.org/T419612)'
|
|
2026-04-17 22:06:44
|
<wikibugs>
|
('CR) ''SBassett: Set CSP to enforce with currently-allow-listed domains on Beta (''1 comment) [mediawiki-config] - ''https://gerrit.wikimedia.org/r/1272895 (https://phabricator.wikimedia.org/T419612) (owner: ''SBassett)'
|
|
2026-04-17 22:08:40
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mc1072.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 22:08:47
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
|
|
2026-04-17 22:09:02
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
|
|
2026-04-17 22:09:03
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1071.eqiad.wmnet with OS bookworm
|
|
2026-04-17 22:09:07
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11835132 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host mc1071.eqiad.wmnet with OS bookworm completed: - mc1...'
|
|
2026-04-17 22:09:31
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.reimage for host mc1072.eqiad.wmnet with OS bookworm
|
|
2026-04-17 22:09:40
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11835135 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host mc1072.eqiad.wmnet with OS bookworm'
|
|
2026-04-17 22:10:51
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11835136 (''Jclark-ctr)'
|
|
2026-04-17 22:13:12
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.reimage for host mc1056.eqiad.wmnet with OS bookworm
|
|
2026-04-17 22:13:16
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11835137 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host mc1056.eqiad.wmnet with OS bookworm'
|
|
2026-04-17 22:13:37
|
<logmsgbot>
|
!log jhancock@cumin2002 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002"
|
|
2026-04-17 22:15:22
|
<logmsgbot>
|
!log jclark@cumin1003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host mc1067.eqiad.wmnet with OS bookworm
|
|
2026-04-17 22:15:25
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11835141 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host mc1067.eqiad.wmnet with OS bookworm executed with er...'
|
|
2026-04-17 22:16:42
|
<logmsgbot>
|
jhancock@cumin2002 reimage (PID 2138794) is awaiting input
|
|
2026-04-17 22:16:50
|
<icinga-wm>
|
RECOVERY - OSPF status on cr1-magru is OK: OSPFv2: 2/2 UP : OSPFv3: 2/2 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status
|
|
2026-04-17 22:18:45
|
<jinxer-wm>
|
FIRING: WidespreadPuppetFailure: Puppet has failed in magru - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure
|
|
2026-04-17 22:19:39
|
<jinxer-wm>
|
RESOLVED: [2x] TransitBGPDown: Transit BGP session down between cr1-magru and Telxius (2001:1498:1:966:1::251) - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DTransitBGPDown
|
|
2026-04-17 22:19:48
|
<logmsgbot>
|
!log jhancock@cumin2002 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002"
|
|
2026-04-17 22:19:50
|
<logmsgbot>
|
!log jhancock@cumin2002 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host pc2024.codfw.wmnet with OS trixie
|
|
2026-04-17 22:19:57
|
<wikibugs>
|
'ops-codfw, ''SRE, ''Data-Persistence, ''DC-Ops: Q3:rack/setup/install pc202[1-4] - https://phabricator.wikimedia.org/T418907#11835154 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jhancock@cumin2002 for host pc2024.codfw.wmnet with OS trixie completed: - pc2024 (**PASS**) - Rem...'
|
|
2026-04-17 22:20:10
|
<jinxer-wm>
|
RESOLVED: [2x] BFDdown: BFD session down between cr2-eqiad and 195.200.68.151 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=cr2-eqiad:9804 - https://alerts.wikimedia.org/?q=alertname%3DBFDdown
|
|
2026-04-17 22:20:15
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on mc1072.eqiad.wmnet with reason: host reimage
|
|
2026-04-17 22:20:40
|
<jinxer-wm>
|
RESOLVED: [4x] CoreBGPDown: Core BGP session down between cr1-magru and cr2-eqiad (195.200.68.150) - group Confed_eqiad - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DCoreBGPDown
|
|
2026-04-17 22:20:44
|
<wikibugs>
|
'ops-codfw, ''SRE, ''Data-Persistence, ''DC-Ops: Q3:rack/setup/install pc202[1-4] - https://phabricator.wikimedia.org/T418907#11835155 (''Jhancock.wm) ''Open→''Resolved'
|
|
2026-04-17 22:21:05
|
<wikibugs>
|
'ops-codfw, ''SRE, ''Data-Persistence, ''DC-Ops: Q3:rack/setup/install pc202[1-4] - https://phabricator.wikimedia.org/T418907#11835158 (''Jhancock.wm) @Marostegui these are complete!'
|
|
2026-04-17 22:23:20
|
<logmsgbot>
|
!log jclark@cumin1003 END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host mc1070.eqiad.wmnet with OS bookworm
|
|
2026-04-17 22:23:28
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11835160 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host mc1070.eqiad.wmnet with OS bookworm executed with er...'
|
|
2026-04-17 22:24:38
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1072.eqiad.wmnet with reason: host reimage
|
|
2026-04-17 22:33:18
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on mc1056.eqiad.wmnet with reason: host reimage
|
|
2026-04-17 22:35:32
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.reimage for host mc1067.eqiad.wmnet with OS bookworm
|
|
2026-04-17 22:35:41
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11835182 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host mc1067.eqiad.wmnet with OS bookworm'
|
|
2026-04-17 22:36:51
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.provision for host mc1067.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 22:37:32
|
<logmsgbot>
|
!log jhancock@cumin2002 START - Cookbook sre.hosts.provision for host db2250.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
|
|
2026-04-17 22:38:03
|
<logmsgbot>
|
!log jhancock@cumin2002 START - Cookbook sre.hosts.provision for host db2251.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
|
|
2026-04-17 22:38:06
|
<logmsgbot>
|
!log jclark@cumin1003 END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host mc1067.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 22:38:19
|
<logmsgbot>
|
!log jhancock@cumin2002 START - Cookbook sre.hosts.provision for host db2252.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
|
|
2026-04-17 22:38:22
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.provision for host mc1067.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 22:38:35
|
<logmsgbot>
|
!log jhancock@cumin2002 START - Cookbook sre.hosts.provision for host db2253.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
|
|
2026-04-17 22:39:12
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1056.eqiad.wmnet with reason: host reimage
|
|
2026-04-17 22:40:34
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
|
|
2026-04-17 22:40:50
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
|
|
2026-04-17 22:40:51
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1072.eqiad.wmnet with OS bookworm
|
|
2026-04-17 22:40:58
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11835190 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host mc1072.eqiad.wmnet with OS bookworm completed: - mc1...'
|
|
2026-04-17 22:41:21
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.provision for host mc1070.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 22:42:50
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11835191 (''Jclark-ctr)'
|
|
2026-04-17 22:45:15
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mc1067.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 22:48:28
|
<logmsgbot>
|
!log jhancock@cumin2002 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db2250.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
|
|
2026-04-17 22:48:45
|
<jinxer-wm>
|
RESOLVED: WidespreadPuppetFailure: Puppet has failed in magru - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?orgId=1&viewPanel=6 - https://alerts.wikimedia.org/?q=alertname%3DWidespreadPuppetFailure
|
|
2026-04-17 22:49:04
|
<logmsgbot>
|
!log jhancock@cumin2002 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db2252.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
|
|
2026-04-17 22:49:17
|
<logmsgbot>
|
!log jhancock@cumin2002 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db2251.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
|
|
2026-04-17 22:49:22
|
<logmsgbot>
|
!log jhancock@cumin2002 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host db2253.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED
|
|
2026-04-17 22:50:35
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.reimage for host mc1070.eqiad.wmnet with OS bookworm
|
|
2026-04-17 22:50:40
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11835200 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jclark@cumin1003 for host mc1070.eqiad.wmnet with OS bookworm'
|
|
2026-04-17 22:51:05
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mc1070.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 22:54:10
|
<wikibugs>
|
('PS1) ''Ryan Kemper: cloudelastic1012: full common_settings override for OS2 [puppet] - ''https://gerrit.wikimedia.org/r/1274134 (https://phabricator.wikimedia.org/T422860)'
|
|
2026-04-17 22:54:43
|
<wikibugs>
|
('CR) ''Ryan Kemper: "check experimental" [puppet] - ''https://gerrit.wikimedia.org/r/1274134 (https://phabricator.wikimedia.org/T422860) (owner: ''Ryan Kemper)'
|
|
2026-04-17 22:56:20
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
|
|
2026-04-17 22:56:36
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
|
|
2026-04-17 22:56:37
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1056.eqiad.wmnet with OS bookworm
|
|
2026-04-17 22:56:41
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11835205 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host mc1056.eqiad.wmnet with OS bookworm completed: - mc1...'
|
|
2026-04-17 22:57:11
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11835206 (''Jclark-ctr)'
|
|
2026-04-17 22:57:21
|
<logmsgbot>
|
jclark@cumin1003 provision (PID 2567031) is awaiting input
|
|
2026-04-17 22:57:45
|
<wikibugs>
|
('CR) ''Ryan Kemper: "PCC shows -BindPaths=/usr/share/opensearch/plugins/ltr as desired; merging" [puppet] - ''https://gerrit.wikimedia.org/r/1274134 (https://phabricator.wikimedia.org/T422860) (owner: ''Ryan Kemper)'
|
|
2026-04-17 22:57:46
|
<wikibugs>
|
('CR) ''Ryan Kemper: [C:''+2] cloudelastic1012: full common_settings override for OS2 [puppet] - ''https://gerrit.wikimedia.org/r/1274134 (https://phabricator.wikimedia.org/T422860) (owner: ''Ryan Kemper)'
|
|
2026-04-17 23:00:27
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.provision for host mc1070.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 23:01:58
|
<wikibugs>
|
('PS1) ''Dzahn: etherpad: avoid adding warning message multiple times [puppet] - ''https://gerrit.wikimedia.org/r/1274138 (https://phabricator.wikimedia.org/T420793)'
|
|
2026-04-17 23:04:12
|
<wikibugs>
|
('CR) ''Dzahn: [C:''+2] etherpad: avoid adding warning message multiple times [puppet] - ''https://gerrit.wikimedia.org/r/1274138 (https://phabricator.wikimedia.org/T420793) (owner: ''Dzahn)'
|
|
2026-04-17 23:05:26
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on mc1067.eqiad.wmnet with reason: host reimage
|
|
2026-04-17 23:07:12
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mc1070.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART
|
|
2026-04-17 23:08:45
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1067.eqiad.wmnet with reason: host reimage
|
|
2026-04-17 23:10:23
|
<logmsgbot>
|
!log otto@deploy1003 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply
|
|
2026-04-17 23:10:27
|
<logmsgbot>
|
!log otto@deploy1003 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mw-page-html-content-change-enrich-next: apply
|
|
2026-04-17 23:21:33
|
<wikibugs>
|
('PS1) ''Dzahn: etherpad: remove code to insert warning message [puppet] - ''https://gerrit.wikimedia.org/r/1274151 (https://phabricator.wikimedia.org/T420793)'
|
|
2026-04-17 23:22:49
|
<wikibugs>
|
('PS2) ''Dzahn: etherpad: remove code to insert warning message [puppet] - ''https://gerrit.wikimedia.org/r/1274151 (https://phabricator.wikimedia.org/T420793)'
|
|
2026-04-17 23:23:10
|
<wikibugs>
|
('CR) ''Dzahn: [C:''+2] etherpad: remove code to insert warning message [puppet] - ''https://gerrit.wikimedia.org/r/1274151 (https://phabricator.wikimedia.org/T420793) (owner: ''Dzahn)'
|
|
2026-04-17 23:24:38
|
<wikibugs>
|
('CR) ''Dzahn: [C:''+2] "recheck" [puppet] - ''https://gerrit.wikimedia.org/r/1274151 (https://phabricator.wikimedia.org/T420793) (owner: ''Dzahn)'
|
|
2026-04-17 23:24:49
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on mc1070.eqiad.wmnet with reason: host reimage
|
|
2026-04-17 23:25:56
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
|
|
2026-04-17 23:25:59
|
<jinxer-wm>
|
FIRING: KubernetesDeploymentUnavailableReplicas: ...
|
|
2026-04-17 23:25:59
|
<jinxer-wm>
|
Deployment linkrecommendation-internal in linkrecommendation at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s&var-namespace=linkrecommendation&var-deployment=linkrecommendation-internal - ...
|
|
2026-04-17 23:25:59
|
<jinxer-wm>
|
https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas
|
|
2026-04-17 23:26:14
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
|
|
2026-04-17 23:26:15
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1067.eqiad.wmnet with OS bookworm
|
|
2026-04-17 23:26:25
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11835283 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host mc1067.eqiad.wmnet with OS bookworm completed: - mc1...'
|
|
2026-04-17 23:26:41
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11835284 (''Jclark-ctr)'
|
|
2026-04-17 23:31:42
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mc1070.eqiad.wmnet with reason: host reimage
|
|
2026-04-17 23:40:09
|
<wikibugs>
|
('PS1) ''TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - ''https://gerrit.wikimedia.org/r/1274153'
|
|
2026-04-17 23:40:09
|
<wikibugs>
|
('CR) ''TrainBranchBot: [C:''+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - ''https://gerrit.wikimedia.org/r/1274153 (owner: ''TrainBranchBot)'
|
|
2026-04-17 23:48:13
|
<logmsgbot>
|
!log jclark@cumin1003 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
|
|
2026-04-17 23:51:16
|
<wikibugs>
|
('Merged) ''jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - ''https://gerrit.wikimedia.org/r/1274153 (owner: ''TrainBranchBot)'
|
|
2026-04-17 23:51:19
|
<logmsgbot>
|
jclark@cumin1003 reimage (PID 2575711) is awaiting input
|
|
2026-04-17 23:52:24
|
<wikibugs>
|
('PS1) ''Ecarg: Wikifunctions: add helm values for function-evaluator in Rust [deployment-charts] - ''https://gerrit.wikimedia.org/r/1274165 (https://phabricator.wikimedia.org/T423627)'
|
|
2026-04-17 23:53:05
|
<wikibugs>
|
('CR) ''Ecarg: Wikifunctions: add helm values for function-evaluator in Rust (''1 comment) [deployment-charts] - ''https://gerrit.wikimedia.org/r/1274165 (https://phabricator.wikimedia.org/T423627) (owner: ''Ecarg)'
|
|
2026-04-17 23:55:06
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1003"
|
|
2026-04-17 23:55:07
|
<logmsgbot>
|
!log jclark@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mc1070.eqiad.wmnet with OS bookworm
|
|
2026-04-17 23:55:15
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11835321 (''ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jclark@cumin1003 for host mc1070.eqiad.wmnet with OS bookworm completed: - mc1...'
|
|
2026-04-17 23:55:31
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11835324 (''Jclark-ctr)'
|
|
2026-04-17 23:56:09
|
<wikibugs>
|
'ops-eqiad, ''DC-Ops, ''ServiceOps new, ''ServiceOps-Upgrades-Hardware: Q2:rack/setup/install mc1055-72 - https://phabricator.wikimedia.org/T412255#11835325 (''Jclark-ctr) ''Open→''Resolved
@jijiki install is finished'
|
