[00:02:59] <wikibugs>	 (03Merged) 10jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1035866 (owner: 10TrainBranchBot)
[00:34:45] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P63208 and previous config saved to /var/cache/conftool/dbconfig/20240526-003444-marostegui.json
[00:36:47] <jinxer-wm>	 FIRING: SystemdUnitFailed: logrotate.service on moss-be1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[00:49:53] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P63209 and previous config saved to /var/cache/conftool/dbconfig/20240526-004952-marostegui.json
[01:05:01] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2148 (T364299)', diff saved to https://phabricator.wikimedia.org/P63210 and previous config saved to /var/cache/conftool/dbconfig/20240526-010500-marostegui.json
[01:05:03] <logmsgbot>	 !log marostegui@cumin1002 START - Cookbook sre.hosts.downtime for 6:00:00 on db2175.codfw.wmnet with reason: Maintenance
[01:05:05] <stashbot>	 T364299: Make rc_id a bigint - https://phabricator.wikimedia.org/T364299
[01:05:16] <logmsgbot>	 !log marostegui@cumin1002 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2175.codfw.wmnet with reason: Maintenance
[01:05:26] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Depooling db2175 (T364299)', diff saved to https://phabricator.wikimedia.org/P63211 and previous config saved to /var/cache/conftool/dbconfig/20240526-010523-marostegui.json
[01:08:05] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[01:08:10] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[01:19:26] <jinxer-wm>	 FIRING: [12x] SystemdUnitFailed: httpbb_hourly_appserver.service on cumin1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[01:19:41] <jinxer-wm>	 FIRING: [12x] SystemdUnitFailed: httpbb_hourly_appserver.service on cumin1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[01:26:49] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2156 (T364069)', diff saved to https://phabricator.wikimedia.org/P63212 and previous config saved to /var/cache/conftool/dbconfig/20240526-012648-marostegui.json
[01:26:54] <stashbot>	 T364069: Rebuild pagelinks tables - https://phabricator.wikimedia.org/T364069
[01:37:10] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[01:37:14] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[01:41:57] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P63213 and previous config saved to /var/cache/conftool/dbconfig/20240526-014156-marostegui.json
[01:57:05] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P63214 and previous config saved to /var/cache/conftool/dbconfig/20240526-015704-marostegui.json
[02:12:15] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2156 (T364069)', diff saved to https://phabricator.wikimedia.org/P63215 and previous config saved to /var/cache/conftool/dbconfig/20240526-021213-marostegui.json
[02:12:17] <logmsgbot>	 !log marostegui@cumin1002 START - Cookbook sre.hosts.downtime for 12:00:00 on db2177.codfw.wmnet with reason: Maintenance
[02:12:20] <stashbot>	 T364069: Rebuild pagelinks tables - https://phabricator.wikimedia.org/T364069
[02:12:31] <logmsgbot>	 !log marostegui@cumin1002 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2177.codfw.wmnet with reason: Maintenance
[02:12:40] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Depooling db2177 (T364069)', diff saved to https://phabricator.wikimedia.org/P63216 and previous config saved to /var/cache/conftool/dbconfig/20240526-021238-marostegui.json
[02:17:13] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2175 (T364299)', diff saved to https://phabricator.wikimedia.org/P63217 and previous config saved to /var/cache/conftool/dbconfig/20240526-021711-marostegui.json
[02:17:18] <stashbot>	 T364299: Make rc_id a bigint - https://phabricator.wikimedia.org/T364299
[02:29:21] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[02:29:25] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[02:32:21] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P63218 and previous config saved to /var/cache/conftool/dbconfig/20240526-023220-marostegui.json
[02:36:47] <jinxer-wm>	 FIRING: JobUnavailable: Reduced availability for job sidekiq in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[02:43:22] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[02:43:26] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[02:47:29] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P63219 and previous config saved to /var/cache/conftool/dbconfig/20240526-024728-marostegui.json
[02:58:57] <jinxer-wm>	 RESOLVED: JobUnavailable: Reduced availability for job sidekiq in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[03:02:37] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2175 (T364299)', diff saved to https://phabricator.wikimedia.org/P63220 and previous config saved to /var/cache/conftool/dbconfig/20240526-030236-marostegui.json
[03:02:39] <logmsgbot>	 !log marostegui@cumin1002 START - Cookbook sre.hosts.downtime for 6:00:00 on db2189.codfw.wmnet with reason: Maintenance
[03:02:43] <stashbot>	 T364299: Make rc_id a bigint - https://phabricator.wikimedia.org/T364299
[03:02:52] <logmsgbot>	 !log marostegui@cumin1002 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2189.codfw.wmnet with reason: Maintenance
[03:03:00] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Depooling db2189 (T364299)', diff saved to https://phabricator.wikimedia.org/P63221 and previous config saved to /var/cache/conftool/dbconfig/20240526-030259-marostegui.json
[03:06:48] <jinxer-wm>	 FIRING: [4x] ProbeDown: Service ml-staging-ctrl2001:6443 has failed probes (http_ml_staging_codfw_kube_apiserver_ip4)  - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[03:19:26] <jinxer-wm>	 FIRING: [12x] SystemdUnitFailed: httpbb_hourly_appserver.service on cumin1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[03:24:26] <jinxer-wm>	 FIRING: [12x] SystemdUnitFailed: httpbb_hourly_appserver.service on cumin1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[03:26:47] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: logrotate.service on moss-be1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[03:35:40] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: kube-controller-manager.service on ml-staging-ctrl2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[04:08:33] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2189 (T364299)', diff saved to https://phabricator.wikimedia.org/P63222 and previous config saved to /var/cache/conftool/dbconfig/20240526-040833-marostegui.json
[04:08:38] <stashbot>	 T364299: Make rc_id a bigint - https://phabricator.wikimedia.org/T364299
[04:23:42] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P63223 and previous config saved to /var/cache/conftool/dbconfig/20240526-042341-marostegui.json
[04:24:01] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[04:24:05] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[04:25:35] <icinga-wm_>	 PROBLEM - snapshot of s2 in eqiad on backupmon1001 is CRITICAL: snapshot for s2 at eqiad (db1225) taken more than 3 days ago: Most recent backup 2024-05-23 04:04:11 https://wikitech.wikimedia.org/wiki/MariaDB/Backups%23Rerun_a_failed_backup
[04:38:50] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P63224 and previous config saved to /var/cache/conftool/dbconfig/20240526-043849-marostegui.json
[04:41:10] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[04:41:14] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[04:45:40] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2177 (T364069)', diff saved to https://phabricator.wikimedia.org/P63225 and previous config saved to /var/cache/conftool/dbconfig/20240526-044539-marostegui.json
[04:45:44] <stashbot>	 T364069: Rebuild pagelinks tables - https://phabricator.wikimedia.org/T364069
[04:53:58] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2189 (T364299)', diff saved to https://phabricator.wikimedia.org/P63226 and previous config saved to /var/cache/conftool/dbconfig/20240526-045357-marostegui.json
[04:54:00] <logmsgbot>	 !log marostegui@cumin1002 START - Cookbook sre.hosts.downtime for 6:00:00 on db2197.codfw.wmnet with reason: Maintenance
[04:54:03] <stashbot>	 T364299: Make rc_id a bigint - https://phabricator.wikimedia.org/T364299
[04:54:14] <logmsgbot>	 !log marostegui@cumin1002 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2197.codfw.wmnet with reason: Maintenance
[05:00:48] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P63227 and previous config saved to /var/cache/conftool/dbconfig/20240526-050047-marostegui.json
[05:15:56] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P63228 and previous config saved to /var/cache/conftool/dbconfig/20240526-051555-marostegui.json
[05:19:26] <jinxer-wm>	 FIRING: [12x] SystemdUnitFailed: httpbb_hourly_appserver.service on cumin1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[05:24:26] <jinxer-wm>	 FIRING: [12x] SystemdUnitFailed: httpbb_hourly_appserver.service on cumin1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[05:31:04] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2177 (T364069)', diff saved to https://phabricator.wikimedia.org/P63229 and previous config saved to /var/cache/conftool/dbconfig/20240526-053103-marostegui.json
[05:31:06] <logmsgbot>	 !log marostegui@cumin1002 START - Cookbook sre.hosts.downtime for 12:00:00 on db2190.codfw.wmnet with reason: Maintenance
[05:31:09] <stashbot>	 T364069: Rebuild pagelinks tables - https://phabricator.wikimedia.org/T364069
[05:31:19] <logmsgbot>	 !log marostegui@cumin1002 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2190.codfw.wmnet with reason: Maintenance
[05:31:28] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Depooling db2190 (T364069)', diff saved to https://phabricator.wikimedia.org/P63230 and previous config saved to /var/cache/conftool/dbconfig/20240526-053127-marostegui.json
[05:38:21] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[05:38:25] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[05:42:44] <logmsgbot>	 !log marostegui@cumin1002 START - Cookbook sre.hosts.downtime for 6:00:00 on db2204.codfw.wmnet with reason: Maintenance
[05:42:57] <logmsgbot>	 !log marostegui@cumin1002 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2204.codfw.wmnet with reason: Maintenance
[05:43:06] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Depooling db2204 (T364299)', diff saved to https://phabricator.wikimedia.org/P63231 and previous config saved to /var/cache/conftool/dbconfig/20240526-054305-marostegui.json
[05:43:10] <stashbot>	 T364299: Make rc_id a bigint - https://phabricator.wikimedia.org/T364299
[05:49:28] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[05:49:31] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[06:04:21] <jinxer-wm>	 FIRING: PoolcounterFullQueues: Full queues for poolcounter1004:9106 poolcounter - https://www.mediawiki.org/wiki/PoolCounter#Request_tracing_in_production - https://grafana.wikimedia.org/d/aIcYxuxZk/poolcounter?orgId=1&viewPanel=6&from=now-1h&to=now&var-dc=eqiad%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DPoolcounterFullQueues
[06:09:21] <jinxer-wm>	 RESOLVED: PoolcounterFullQueues: Full queues for poolcounter1004:9106 poolcounter - https://www.mediawiki.org/wiki/PoolCounter#Request_tracing_in_production - https://grafana.wikimedia.org/d/aIcYxuxZk/poolcounter?orgId=1&viewPanel=6&from=now-1h&to=now&var-dc=eqiad%20prometheus/ops - https://alerts.wikimedia.org/?q=alertname%3DPoolcounterFullQueues
[06:15:48] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[06:15:52] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[06:30:42] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[06:30:46] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[06:37:52] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2204 (T364299)', diff saved to https://phabricator.wikimedia.org/P63232 and previous config saved to /var/cache/conftool/dbconfig/20240526-063752-marostegui.json
[06:37:57] <stashbot>	 T364299: Make rc_id a bigint - https://phabricator.wikimedia.org/T364299
[06:41:11] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[06:41:15] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[06:53:00] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2204', diff saved to https://phabricator.wikimedia.org/P63233 and previous config saved to /var/cache/conftool/dbconfig/20240526-065259-marostegui.json
[06:56:04] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[06:56:09] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[06:58:23] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[06:58:27] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[07:00:05] <jouncebot>	 Deploy window No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20240526T0700)
[07:06:47] <jinxer-wm>	 FIRING: [4x] ProbeDown: Service ml-staging-ctrl2001:6443 has failed probes (http_ml_staging_codfw_kube_apiserver_ip4)  - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[07:08:09] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2204', diff saved to https://phabricator.wikimedia.org/P63234 and previous config saved to /var/cache/conftool/dbconfig/20240526-070808-marostegui.json
[07:09:06] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[07:09:10] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[07:23:16] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2204 (T364299)', diff saved to https://phabricator.wikimedia.org/P63235 and previous config saved to /var/cache/conftool/dbconfig/20240526-072316-marostegui.json
[07:23:21] <stashbot>	 T364299: Make rc_id a bigint - https://phabricator.wikimedia.org/T364299
[07:26:47] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: logrotate.service on moss-be1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[07:29:14] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[07:29:19] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[07:35:40] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: kube-controller-manager.service on ml-staging-ctrl2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[07:37:46] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2190 (T364069)', diff saved to https://phabricator.wikimedia.org/P63236 and previous config saved to /var/cache/conftool/dbconfig/20240526-073745-marostegui.json
[07:37:51] <stashbot>	 T364069: Rebuild pagelinks tables - https://phabricator.wikimedia.org/T364069
[07:40:09] <icinga-wm_>	 PROBLEM - BGP status on cr4-ulsfo is CRITICAL: BGP CRITICAL - AS64605/IPv6: Active - Anycast, AS64605/IPv4: Active - Anycast https://wikitech.wikimedia.org/wiki/Network_monitoring%23BGP_status
[07:44:06] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[07:44:10] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[07:46:09] <icinga-wm_>	 PROBLEM - BGP status on cr4-ulsfo is CRITICAL: BGP CRITICAL - AS64605/IPv4: Active - Anycast, AS64605/IPv6: Active - Anycast https://wikitech.wikimedia.org/wiki/Network_monitoring%23BGP_status
[07:51:22] <wikibugs>	 06SRE, 10DNS, 06Traffic, 10WikiLearn: DNS records for WikiLearn - https://phabricator.wikimedia.org/T365435#9833158 (10Asaf) Thank you. We will look into setting up email on learn.wiki with our vendor, and update the ticket when we learn if that's feasible.
[07:52:54] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2190', diff saved to https://phabricator.wikimedia.org/P63237 and previous config saved to /var/cache/conftool/dbconfig/20240526-075253-marostegui.json
[07:54:25] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[07:54:29] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[07:58:03] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[07:58:07] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[08:04:11] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[08:04:15] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[08:08:02] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2190', diff saved to https://phabricator.wikimedia.org/P63238 and previous config saved to /var/cache/conftool/dbconfig/20240526-080802-marostegui.json
[08:09:21] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[08:09:25] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[08:23:11] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2190 (T364069)', diff saved to https://phabricator.wikimedia.org/P63239 and previous config saved to /var/cache/conftool/dbconfig/20240526-082310-marostegui.json
[08:23:13] <logmsgbot>	 !log marostegui@cumin1002 START - Cookbook sre.hosts.downtime for 12:00:00 on db2194.codfw.wmnet with reason: Maintenance
[08:23:15] <stashbot>	 T364069: Rebuild pagelinks tables - https://phabricator.wikimedia.org/T364069
[08:23:26] <logmsgbot>	 !log marostegui@cumin1002 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2194.codfw.wmnet with reason: Maintenance
[08:23:34] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Depooling db2194 (T364069)', diff saved to https://phabricator.wikimedia.org/P63240 and previous config saved to /var/cache/conftool/dbconfig/20240526-082333-marostegui.json
[08:24:14] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[08:24:18] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[08:39:00] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[08:39:04] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[08:46:09] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[08:46:13] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[09:01:01] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[09:01:05] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[09:12:54] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[09:12:58] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[09:15:53] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[09:15:57] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[09:19:26] <jinxer-wm>	 FIRING: [12x] SystemdUnitFailed: httpbb_hourly_appserver.service on cumin1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[09:20:41] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[09:20:45] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[09:24:19] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[09:24:23] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[09:24:26] <jinxer-wm>	 FIRING: [12x] SystemdUnitFailed: httpbb_hourly_appserver.service on cumin1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[09:35:53] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[09:35:57] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[09:48:36] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[09:48:40] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[09:56:14] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[09:56:18] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[10:01:47] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[10:01:51] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[10:11:52] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[10:11:56] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[10:16:10] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[10:16:14] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[10:30:11] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2194 (T364069)', diff saved to https://phabricator.wikimedia.org/P63241 and previous config saved to /var/cache/conftool/dbconfig/20240526-103010-marostegui.json
[10:30:15] <stashbot>	 T364069: Rebuild pagelinks tables - https://phabricator.wikimedia.org/T364069
[10:31:09] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[10:31:13] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[10:34:47] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[10:34:51] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[10:45:04] <wikibugs>	 06SRE, 10Wikimedia-Mailing-lists: Make Chqaz admin of Wikija-g mailing list - https://phabricator.wikimedia.org/T365933#9833308 (10Aklapper)
[10:45:19] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2194', diff saved to https://phabricator.wikimedia.org/P63242 and previous config saved to /var/cache/conftool/dbconfig/20240526-104518-marostegui.json
[10:51:15] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[10:51:20] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[10:53:14] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[10:53:18] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[11:00:27] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2194', diff saved to https://phabricator.wikimedia.org/P63243 and previous config saved to /var/cache/conftool/dbconfig/20240526-110026-marostegui.json
[11:06:47] <jinxer-wm>	 FIRING: [4x] ProbeDown: Service ml-staging-ctrl2001:6443 has failed probes (http_ml_staging_codfw_kube_apiserver_ip4)  - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[11:15:35] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2194 (T364069)', diff saved to https://phabricator.wikimedia.org/P63244 and previous config saved to /var/cache/conftool/dbconfig/20240526-111534-marostegui.json
[11:15:37] <logmsgbot>	 !log marostegui@cumin1002 START - Cookbook sre.hosts.downtime for 12:00:00 on db2209.codfw.wmnet with reason: Maintenance
[11:15:40] <stashbot>	 T364069: Rebuild pagelinks tables - https://phabricator.wikimedia.org/T364069
[11:15:50] <logmsgbot>	 !log marostegui@cumin1002 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2209.codfw.wmnet with reason: Maintenance
[11:15:58] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Depooling db2209 (T364069)', diff saved to https://phabricator.wikimedia.org/P63245 and previous config saved to /var/cache/conftool/dbconfig/20240526-111558-marostegui.json
[11:19:41] <jinxer-wm>	 FIRING: [12x] SystemdUnitFailed: httpbb_hourly_appserver.service on cumin1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[11:20:22] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[11:20:27] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[11:22:43] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[11:22:48] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[11:24:26] <jinxer-wm>	 FIRING: [12x] SystemdUnitFailed: httpbb_hourly_appserver.service on cumin1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[11:24:42] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[11:24:46] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[11:26:47] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: logrotate.service on moss-be1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[11:35:31] <jinxer-wm>	 FIRING: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[11:35:40] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: kube-controller-manager.service on ml-staging-ctrl2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[11:40:31] <jinxer-wm>	 RESOLVED: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[12:01:31] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[12:01:35] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[12:16:30] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[12:16:34] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[12:19:26] <jinxer-wm>	 FIRING: [12x] SystemdUnitFailed: httpbb_hourly_appserver.service on cumin1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[12:24:26] <jinxer-wm>	 FIRING: [12x] SystemdUnitFailed: httpbb_hourly_appserver.service on cumin1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[12:46:59] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[12:47:02] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[12:48:57] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[12:49:01] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[12:50:55] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[12:51:00] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[12:57:38] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[12:57:42] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[13:05:21] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[13:05:25] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[13:14:25] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[13:14:29] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[13:17:27] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2209 (T364069)', diff saved to https://phabricator.wikimedia.org/P63246 and previous config saved to /var/cache/conftool/dbconfig/20240526-131726-marostegui.json
[13:17:31] <stashbot>	 T364069: Rebuild pagelinks tables - https://phabricator.wikimedia.org/T364069
[13:29:26] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[13:29:30] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[13:32:35] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2209', diff saved to https://phabricator.wikimedia.org/P63247 and previous config saved to /var/cache/conftool/dbconfig/20240526-133234-marostegui.json
[13:44:44] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[13:44:48] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[13:47:31] <jinxer-wm>	 FIRING: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[13:47:43] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2209', diff saved to https://phabricator.wikimedia.org/P63248 and previous config saved to /var/cache/conftool/dbconfig/20240526-134742-marostegui.json
[13:50:31] <icinga-wm_>	 PROBLEM - HTTPS on gerrit1003 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection refused https://phabricator.wikimedia.org/project/view/330/
[13:50:58] <jelto>	 !log restart apache2 on gerrit1003
[13:51:01] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[13:51:35] <icinga-wm_>	 RECOVERY - HTTPS on gerrit1003 is OK: SSL OK - Certificate gerrit.wikimedia.org valid until 2024-08-03 19:50:23 +0000 (expires in 69 days) https://phabricator.wikimedia.org/project/view/330/
[13:51:47] <jinxer-wm>	 FIRING: [4x] JobUnavailable: Reduced availability for job gerrit in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[13:52:32] <jinxer-wm>	 RESOLVED: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[13:53:57] <jinxer-wm>	 RESOLVED: [4x] JobUnavailable: Reduced availability for job gerrit in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[13:54:26] <jinxer-wm>	 FIRING: [13x] SystemdUnitFailed: helm-chartctl-package-all.service on chartmuseum2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[13:59:43] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[13:59:47] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[14:01:51] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[14:01:55] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[14:02:51] <logmsgbot>	 !log marostegui@cumin1002 dbctl commit (dc=all): 'Repooling after maintenance db2209 (T364069)', diff saved to https://phabricator.wikimedia.org/P63249 and previous config saved to /var/cache/conftool/dbconfig/20240526-140250-marostegui.json
[14:03:00] <stashbot>	 T364069: Rebuild pagelinks tables - https://phabricator.wikimedia.org/T364069
[14:04:29] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[14:04:33] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[14:13:09] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[14:13:13] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[14:19:26] <jinxer-wm>	 FIRING: [12x] SystemdUnitFailed: httpbb_hourly_appserver.service on cumin1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[14:21:38] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[14:21:42] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[14:24:26] <jinxer-wm>	 FIRING: [12x] SystemdUnitFailed: httpbb_hourly_appserver.service on cumin1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[14:34:59] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[14:35:03] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[14:36:31] <jinxer-wm>	 FIRING: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[14:36:47] <jinxer-wm>	 FIRING: JobUnavailable: Reduced availability for job sidekiq in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[14:37:08] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[14:37:12] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[14:41:31] <jinxer-wm>	 RESOLVED: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[14:42:06] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[14:42:10] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[14:56:47] <jinxer-wm>	 RESOLVED: JobUnavailable: Reduced availability for job sidekiq in ops@eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
[15:06:47] <jinxer-wm>	 FIRING: [4x] ProbeDown: Service ml-staging-ctrl2001:6443 has failed probes (http_ml_staging_codfw_kube_apiserver_ip4)  - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[15:19:26] <jinxer-wm>	 FIRING: [12x] SystemdUnitFailed: httpbb_hourly_appserver.service on cumin1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[15:24:26] <jinxer-wm>	 FIRING: [12x] SystemdUnitFailed: httpbb_hourly_appserver.service on cumin1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[15:26:45] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[15:26:47] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: logrotate.service on moss-be1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[15:26:49] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[15:35:40] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: kube-controller-manager.service on ml-staging-ctrl2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[15:38:53] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[15:38:58] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[15:41:59] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[15:42:03] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[15:45:51] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[15:45:56] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[15:56:07] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[15:56:12] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[16:02:21] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[16:02:25] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[16:25:10] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[16:25:14] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[16:27:08] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[16:27:12] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[16:31:48] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[16:31:52] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[16:39:25] <icinga-wm_>	 PROBLEM - BGP status on cr3-ulsfo is CRITICAL: BGP CRITICAL - AS64605/IPv4: Active - Anycast, AS64605/IPv6: Active - Anycast https://wikitech.wikimedia.org/wiki/Network_monitoring%23BGP_status
[16:46:48] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[16:46:52] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[16:51:36] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[16:51:40] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[16:58:36] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[16:58:40] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[17:06:51] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[17:06:55] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[17:16:27] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[17:16:31] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[17:23:04] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[17:23:08] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[17:29:39] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[17:29:43] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[17:34:38] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[17:34:42] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[17:38:52] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[17:38:57] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[17:40:51] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[17:40:55] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[17:50:10] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[17:50:14] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[17:58:59] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[17:59:03] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[18:02:58] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[18:03:02] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[18:06:16] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[18:06:20] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[18:17:54] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[18:17:58] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[18:20:49] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[18:20:53] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[18:22:58] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[18:23:02] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[18:25:40] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[18:25:44] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[18:38:35] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[18:38:40] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[18:41:51] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[18:41:55] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[18:44:59] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[18:45:03] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[18:55:12] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[18:55:17] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[18:57:11] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[18:57:15] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[18:58:59] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[18:59:03] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[19:01:37] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[19:01:41] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[19:06:47] <jinxer-wm>	 FIRING: [4x] ProbeDown: Service ml-staging-ctrl2001:6443 has failed probes (http_ml_staging_codfw_kube_apiserver_ip4)  - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[19:08:03] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[19:08:07] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[19:10:01] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[19:10:05] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[19:18:43] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[19:18:48] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[19:20:52] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[19:20:56] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[19:24:41] <jinxer-wm>	 FIRING: [12x] SystemdUnitFailed: httpbb_hourly_appserver.service on cumin1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[19:25:04] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[19:25:08] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[19:26:48] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: logrotate.service on moss-be1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[19:27:13] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[19:27:17] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[19:35:34] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[19:35:38] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[19:35:40] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: kube-controller-manager.service on ml-staging-ctrl2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[19:44:15] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[19:44:19] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[19:50:53] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[19:50:57] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[19:57:02] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[19:57:06] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[20:02:53] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[20:02:57] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[20:11:23] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[20:11:26] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[20:16:24] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[20:16:28] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[20:20:18] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[20:20:23] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[20:22:07] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[20:22:12] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[20:28:22] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[20:28:26] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[20:36:00] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[20:36:04] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[20:38:20] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[20:38:24] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[20:40:19] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[20:40:23] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[20:43:22] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[20:43:26] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[20:53:39] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[20:53:43] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[20:59:31] <jinxer-wm>	 FIRING: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[21:02:27] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[21:02:30] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[21:04:15] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[21:04:19] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[21:04:31] <jinxer-wm>	 RESOLVED: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[21:06:25] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[21:06:29] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[21:15:59] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[21:16:03] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[21:17:01] <jinxer-wm>	 FIRING: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[21:27:01] <jinxer-wm>	 RESOLVED: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[21:27:42] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[21:27:46] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[21:32:27] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[21:32:31] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[21:34:26] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[21:34:30] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[21:36:34] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[21:36:38] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[21:43:19] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[21:43:23] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[21:48:31] <jinxer-wm>	 FIRING: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[21:49:54] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[21:49:58] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[21:53:50] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[21:53:54] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[21:55:48] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[21:55:52] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[21:58:07] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[21:58:12] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[21:58:31] <jinxer-wm>	 RESOLVED: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[22:00:37] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[22:00:41] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[22:10:39] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[22:10:43] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[22:23:39] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[22:23:43] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[22:26:31] <jinxer-wm>	 FIRING: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[22:27:21] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[22:27:25] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[22:31:31] <jinxer-wm>	 RESOLVED: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[22:33:33] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[22:33:37] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[22:42:28] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[22:42:32] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[22:44:26] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[22:44:30] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[22:55:36] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[22:55:41] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[23:02:40] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[23:02:44] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[23:05:15] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[23:05:19] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[23:06:47] <jinxer-wm>	 FIRING: [4x] ProbeDown: Service ml-staging-ctrl2001:6443 has failed probes (http_ml_staging_codfw_kube_apiserver_ip4)  - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
[23:16:09] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[23:16:13] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[23:22:10] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[23:22:15] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[23:24:41] <jinxer-wm>	 FIRING: [12x] SystemdUnitFailed: httpbb_hourly_appserver.service on cumin1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[23:26:47] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: logrotate.service on moss-be1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[23:27:01] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[23:27:05] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[23:33:17] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[23:33:21] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[23:35:40] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: kube-controller-manager.service on ml-staging-ctrl2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[23:38:07] <wikibugs>	 (03PS1) 10TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1035867
[23:38:07] <wikibugs>	 (03CR) 10TrainBranchBot: [C:03+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - 10https://gerrit.wikimedia.org/r/1035867 (owner: 10TrainBranchBot)
[23:42:53] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[23:42:58] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[23:46:55] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[23:46:59] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[23:48:53] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[23:48:57] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[23:51:01] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[23:51:05] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
[23:57:20] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
[23:57:24] <logmsgbot>	 !log @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply