|
2026-02-16 00:40:51
|
<wikibugs>
|
('PS1) ''TrainBranchBot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - ''https://gerrit.wikimedia.org/r/1239558'
|
|
2026-02-16 00:40:51
|
<wikibugs>
|
('CR) ''TrainBranchBot: [C:''+2] Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - ''https://gerrit.wikimedia.org/r/1239558 (owner: ''TrainBranchBot)'
|
|
2026-02-16 00:53:04
|
<wikibugs>
|
('Merged) ''jenkins-bot: Branch commit for wmf/branch_cut_pretest [core] (wmf/branch_cut_pretest) - ''https://gerrit.wikimedia.org/r/1239558 (owner: ''TrainBranchBot)'
|
|
2026-02-16 01:46:03
|
<jinxer-wm>
|
FIRING: [5x] PuppetCertificateAboutToExpire: Puppet CA certificate _etcd-server-ssl._tcp.ml_etcd.codfw.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Puppet#Renew_agent_certificate - TODO - https://alerts.wikimedia.org/?q=alertname%3DPuppetCertificateAboutToExpire
|
|
2026-02-16 02:00:39
|
<logmsgbot>
|
!log mwpresync@deploy2002 Started scap build-images: Publishing wmf/next image
|
|
2026-02-16 02:09:19
|
<jinxer-wm>
|
FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
|
|
2026-02-16 02:13:26
|
<logmsgbot>
|
!log mwpresync@deploy2002 Finished scap build-images: Publishing wmf/next image (duration: 12m 46s)
|
|
2026-02-16 02:34:19
|
<jinxer-wm>
|
RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
|
|
2026-02-16 03:14:43
|
<jinxer-wm>
|
FIRING: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag
|
|
2026-02-16 03:18:13
|
<jinxer-wm>
|
FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - pfw1-codfw:reth2 (fasw1-f5 2x25G) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=pfw1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown
|
|
2026-02-16 05:46:03
|
<jinxer-wm>
|
FIRING: [5x] PuppetCertificateAboutToExpire: Puppet CA certificate _etcd-server-ssl._tcp.ml_etcd.codfw.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Puppet#Renew_agent_certificate - TODO - https://alerts.wikimedia.org/?q=alertname%3DPuppetCertificateAboutToExpire
|
|
2026-02-16 05:50:43
|
<wikibugs>
|
('CR) ''ArielGlenn: rest gateway: add tests for chart rendering (''3 comments) [deployment-charts] - ''https://gerrit.wikimedia.org/r/1225085 (owner: ''Daniel Kinzler)'
|
|
2026-02-16 06:18:47
|
<logmsgbot>
|
!log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1150.eqiad.wmnet with reason: Maintenance
|
|
2026-02-16 06:19:32
|
<logmsgbot>
|
!log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2147.codfw.wmnet with reason: Maintenance
|
|
2026-02-16 06:19:41
|
<logmsgbot>
|
!log marostegui@cumin1003 dbctl commit (dc=all): 'Depooling db2147 (T415786)', diff saved to https://phabricator.wikimedia.org/P88824 and previous config saved to /var/cache/conftool/dbconfig/20260216-061940-marostegui.json
|
|
2026-02-16 06:19:45
|
<stashbot>
|
T415786: Update imagelinks primary key on wmf production - https://phabricator.wikimedia.org/T415786
|
|
2026-02-16 06:21:44
|
<wikibugs>
|
('CR) ''Marostegui: [C:''+2] sre.mysql.sanitize-wiki: Fix --check-only still dropping private data [cookbooks] - ''https://gerrit.wikimedia.org/r/1239346 (https://phabricator.wikimedia.org/T415567) (owner: ''Marostegui)'
|
|
2026-02-16 06:35:40
|
<wikibugs>
|
('PS1) ''Marostegui: dbproxy1022: Disable notifications [puppet] - ''https://gerrit.wikimedia.org/r/1239565 (https://phabricator.wikimedia.org/T414656)'
|
|
2026-02-16 06:41:07
|
<wikibugs>
|
('CR) ''Marostegui: [C:''+2] dbproxy1022: Disable notifications [puppet] - ''https://gerrit.wikimedia.org/r/1239565 (https://phabricator.wikimedia.org/T414656) (owner: ''Marostegui)'
|
|
2026-02-16 06:45:32
|
<logmsgbot>
|
!log marostegui@cumin1003 START - Cookbook sre.hosts.reimage for host dbproxy1022.eqiad.wmnet with OS trixie
|
|
2026-02-16 06:55:19
|
<logmsgbot>
|
!log arnaudb@cumin1003 START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit1003.wikimedia.org
|
|
2026-02-16 06:55:54
|
<logmsgbot>
|
!log arnaudb@cumin1003 START - Cookbook sre.gerrit.sync-instances sync Gerrit data from gerrit1003.wikimedia.org to gerrit2003.wikimedia.org
|
|
2026-02-16 06:59:59
|
<logmsgbot>
|
!log arnaudb@cumin1003 END (PASS) - Cookbook sre.gerrit.localbackup (exit_code=0) Prepare local backup on: gerrit1003.wikimedia.org
|
|
2026-02-16 07:01:42
|
<logmsgbot>
|
!log marostegui@cumin1003 START - Cookbook sre.hosts.downtime for 2:00:00 on dbproxy1022.eqiad.wmnet with reason: host reimage
|
|
2026-02-16 07:07:51
|
<logmsgbot>
|
!log arnaudb@cumin1003 END (PASS) - Cookbook sre.gerrit.sync-instances (exit_code=0) sync Gerrit data from gerrit1003.wikimedia.org to gerrit2003.wikimedia.org
|
|
2026-02-16 07:07:53
|
<logmsgbot>
|
!log marostegui@cumin1003 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbproxy1022.eqiad.wmnet with reason: host reimage
|
|
2026-02-16 07:08:58
|
<logmsgbot>
|
!log arnaudb@cumin1003 START - Cookbook sre.gerrit.sync-instances sync Gerrit data from gerrit1003.wikimedia.org to gerrit2003.wikimedia.org
|
|
2026-02-16 07:12:41
|
<logmsgbot>
|
!log arnaudb@cumin1003 END (PASS) - Cookbook sre.gerrit.sync-instances (exit_code=0) sync Gerrit data from gerrit1003.wikimedia.org to gerrit2003.wikimedia.org
|
|
2026-02-16 07:13:42
|
<logmsgbot>
|
!log arnaudb@cumin1003 START - Cookbook sre.gerrit.sync-instances sync Gerrit data from gerrit1003.wikimedia.org to gerrit2003.wikimedia.org
|
|
2026-02-16 07:14:43
|
<jinxer-wm>
|
FIRING: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag
|
|
2026-02-16 07:16:06
|
<logmsgbot>
|
!log arnaudb@cumin1003 END (PASS) - Cookbook sre.gerrit.sync-instances (exit_code=0) sync Gerrit data from gerrit1003.wikimedia.org to gerrit2003.wikimedia.org
|
|
2026-02-16 07:18:13
|
<jinxer-wm>
|
FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - pfw1-codfw:reth2 (fasw1-f5 2x25G) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=pfw1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown
|
|
2026-02-16 07:31:05
|
<logmsgbot>
|
!log marostegui@cumin1003 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbproxy1022.eqiad.wmnet with OS trixie
|
|
2026-02-16 07:32:50
|
<wikibugs>
|
'SRE: offboarding Alex Kosiaris - https://phabricator.wikimedia.org/T417465#11618962 (''MoritzMuehlenhoff)'
|
|
2026-02-16 07:36:41
|
<wikibugs>
|
'SRE: offboarding Alex Kosiaris - https://phabricator.wikimedia.org/T417465#11618967 (''MoritzMuehlenhoff)'
|
|
2026-02-16 07:38:41
|
<wikibugs>
|
('PS1) ''Muehlenhoff: Remove Alex from Icinga [puppet] - ''https://gerrit.wikimedia.org/r/1239568 (https://phabricator.wikimedia.org/T417465)'
|
|
2026-02-16 07:44:02
|
<wikibugs>
|
('PS1) ''Marostegui: Revert "dbproxy1022: Disable notifications" [puppet] - ''https://gerrit.wikimedia.org/r/1239569'
|
|
2026-02-16 07:44:37
|
<wikibugs>
|
('CR) ''Muehlenhoff: [C:''+2] Remove Alex from Icinga [puppet] - ''https://gerrit.wikimedia.org/r/1239568 (https://phabricator.wikimedia.org/T417465) (owner: ''Muehlenhoff)'
|
|
2026-02-16 07:44:49
|
<wikibugs>
|
('CR) ''Marostegui: [C:''+2] Revert "dbproxy1022: Disable notifications" [puppet] - ''https://gerrit.wikimedia.org/r/1239569 (owner: ''Marostegui)'
|
|
2026-02-16 07:46:05
|
<wikibugs>
|
'SRE, ''Patch-For-Review: offboarding Alex Kosiaris - https://phabricator.wikimedia.org/T417465#11618974 (''MoritzMuehlenhoff)'
|
|
2026-02-16 07:47:30
|
<wikibugs>
|
('PS1) ''Ayounsi: MirrorHighLag - set to warning instead of critical [alerts] - ''https://gerrit.wikimedia.org/r/1239570'
|
|
2026-02-16 07:51:30
|
<wikibugs>
|
'SRE, ''Patch-For-Review: offboarding Alex Kosiaris - https://phabricator.wikimedia.org/T417465#11619004 (''MoritzMuehlenhoff)'
|
|
2026-02-16 07:52:15
|
<wikibugs>
|
('CR) ''Muehlenhoff: [C:''+1] "Looks good" [alerts] - ''https://gerrit.wikimedia.org/r/1239570 (owner: ''Ayounsi)'
|
|
2026-02-16 07:53:53
|
<wikibugs>
|
('PS1) ''WMDE-Fisch: Parsoid: Add safeguard when checking for reflist template [extensions/Cite] (wmf/1.46.0-wmf.15) - ''https://gerrit.wikimedia.org/r/1239573 (https://phabricator.wikimedia.org/T416630)'
|
|
2026-02-16 07:54:35
|
<wikibugs>
|
('PS1) ''Muehlenhoff: Remove Alex from routers [homer/public] - ''https://gerrit.wikimedia.org/r/1239574 (https://phabricator.wikimedia.org/T417465)'
|
|
2026-02-16 07:55:41
|
<Jhs>
|
!nowandnext
|
|
2026-02-16 07:55:50
|
<Jhs>
|
jouncebot: nowandnext
|
|
2026-02-16 07:55:50
|
<jouncebot>
|
For the next 0 hour(s) and 4 minute(s): No deploys all day! See Deployments/Emergencies if things are broken. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260215T0800)
|
|
2026-02-16 07:55:50
|
<jouncebot>
|
In 0 hour(s) and 4 minute(s): UTC morning backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260216T0800)
|
|
2026-02-16 07:56:28
|
<wikibugs>
|
('PS1) ''Jon Harald Søby: Enable PageImages for bnwikisource [mediawiki-config] - ''https://gerrit.wikimedia.org/r/1239571 (https://phabricator.wikimedia.org/T416800)'
|
|
2026-02-16 07:56:42
|
<wikibugs>
|
('CR) ''ScheduleDeploymentBot: "Scheduled for deployment in the [Monday, February 16 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployca"; [mediawiki-config] - ''https://gerrit.wikimedia.org/r/1239571 (https://phabricator.wikimedia.org/T416800) (owner: ''Jon Harald Søby)'
|
|
2026-02-16 07:57:22
|
<wikibugs>
|
('CR) ''Ayounsi: [C:''+2] MirrorHighLag - set to warning instead of critical [alerts] - ''https://gerrit.wikimedia.org/r/1239570 (owner: ''Ayounsi)'
|
|
2026-02-16 07:59:01
|
<wikibugs>
|
('Merged) ''jenkins-bot: MirrorHighLag - set to warning instead of critical [alerts] - ''https://gerrit.wikimedia.org/r/1239570 (owner: ''Ayounsi)'
|
|
2026-02-16 08:00:05
|
<jouncebot>
|
Amir1, Urbanecm, and awight: I seem to be stuck in Groundhog week. Sigh. Time for (yet another) UTC morning backport window deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260216T0800).
|
|
2026-02-16 08:00:05
|
<jouncebot>
|
Msz2001, hashar, kipfel, and Jhs: A patch you scheduled for UTC morning backport window is about to be deployed. Please be around during the process. Note: If you break AND fix the wikis, you will be rewarded with a sticker.
|
|
2026-02-16 08:00:09
|
<Msz2001>
|
o/
|
|
2026-02-16 08:00:26
|
<Msz2001>
|
I'm ready to start deploying
|
|
2026-02-16 08:00:46
|
<kipfel>
|
o/
|
|
2026-02-16 08:01:17
|
<wikibugs>
|
('CR) ''TrainBranchBot: [C:''+2] "Approved by mszwarc@deploy2002 using scap backport" [extensions/IPInfo] (wmf/1.46.0-wmf.15) - ''https://gerrit.wikimedia.org/r/1239337 (https://phabricator.wikimedia.org/T417250) (owner: ''Kosta Harlan)'
|
|
2026-02-16 08:02:29
|
<wikibugs>
|
('Merged) ''jenkins-bot: Add infobox case handling for Special:IPContributions [extensions/IPInfo] (wmf/1.46.0-wmf.15) - ''https://gerrit.wikimedia.org/r/1239337 (https://phabricator.wikimedia.org/T417250) (owner: ''Kosta Harlan)'
|
|
2026-02-16 08:03:08
|
<logmsgbot>
|
!log mszwarc@deploy2002 Started scap sync-world: Backport for [[gerrit:1239337|Add infobox case handling for Special:IPContributions (T417250)]]
|
|
2026-02-16 08:03:12
|
<stashbot>
|
T417250: "IP information for this address cannot be retrieved since no contributions have been made from it" on Special:IPContributions - https://phabricator.wikimedia.org/T417250
|
|
2026-02-16 08:03:40
|
<hashar>
|
o/
|
|
2026-02-16 08:04:35
|
<hashar>
|
Msz2001: can you self deploy your change?
|
|
2026-02-16 08:04:46
|
<Msz2001>
|
Yes, I'm doing it right now
|
|
2026-02-16 08:04:49
|
<hashar>
|
awesome
|
|
2026-02-16 08:07:20
|
<Jhs>
|
My patch enables PageImages in bnwikisource by community requst. That extension is enabled by default everywhere, except for Wikibooks and Wikisource, due to https://phabricator.wikimedia.org/T68455 from 2014, which is basically just one person saying "I'm not sure if this makes sense for these projects" and that's it. But since there are community requests from Wikisource to enable it, would it make more sense to just remove the "exception" for
|
|
2026-02-16 08:07:20
|
<Jhs>
|
Wikisource and Wikibooks for `wmgUsePageImages`? Right now, it being disabled for Wikisource and Wikibooks makes all search results (in typeahead sarch) have those default thumbnails, instead of showing relevant images where they could show relevant images.
|
|
2026-02-16 08:08:15
|
<hashar>
|
kipfel: your change look good, I ll deploy it. I would have been happy to have deployed it yesterday, I guess we will want a procedure for that somehow :)
|
|
2026-02-16 08:08:33
|
<wikibugs>
|
('PS1) ''Muehlenhoff: Remove LDAP access for runa [puppet] - ''https://gerrit.wikimedia.org/r/1239575'
|
|
2026-02-16 08:10:08
|
<kipfel>
|
hashar: thanks! its indeed a little bit late for today, since CNY is just tomorrow :)
|
|
2026-02-16 08:11:08
|
<hashar>
|
better late than never! :)
|
|
2026-02-16 08:11:17
|
<wikibugs>
|
('CR) ''Muehlenhoff: [C:''+2] Remove LDAP access for runa [puppet] - ''https://gerrit.wikimedia.org/r/1239575 (owner: ''Muehlenhoff)'
|
|
2026-02-16 08:12:14
|
<hashar>
|
Jhs: excellent thank you, you can paste your above comment on the task, I guess that will be heflpul for later
|
|
2026-02-16 08:12:39
|
<hashar>
|
and from what you say, I imagine we should open up a discussion to have PageImages enabled everywhere
|
|
2026-02-16 08:12:45
|
<hashar>
|
that'll simplify the config a bit
|
|
2026-02-16 08:13:54
|
<Jhs>
|
hashar, in the 2014 task, you mean? And then open a new task to enable it everywhere?
|
|
2026-02-16 08:14:18
|
<hashar>
|
if you feel like doing it yes :-]
|
|
2026-02-16 08:14:58
|
<Jhs>
|
aight
|
|
2026-02-16 08:14:59
|
<hashar>
|
what I get is that anytime a wikibook/wikisource wants to add it different people have to go through the whole process of a task, opening a discussion, waiting, sending a patch etc
|
|
2026-02-16 08:15:12
|
<Jhs>
|
yup
|
|
2026-02-16 08:15:43
|
<wikibugs>
|
('CR) ''Thiemo Kreuz (WMDE): [C:''+1] Parsoid: Add safeguard when checking for reflist template [extensions/Cite] (wmf/1.46.0-wmf.15) - ''https://gerrit.wikimedia.org/r/1239573 (https://phabricator.wikimedia.org/T416630) (owner: ''WMDE-Fisch)'
|
|
2026-02-16 08:15:55
|
<Jhs>
|
and now with Vector-22 as the default skin (which assumes PageImages being enabled in its search widget), the issue is more "in your face" than it was in 2014, obviously
|
|
2026-02-16 08:15:56
|
<hashar>
|
I don't know much about PageImages, worse case the wikis have barely any image and the extension has no usage
|
|
2026-02-16 08:16:04
|
<hashar>
|
which saves everyone else the trouble to having to enable it
|
|
2026-02-16 08:16:05
|
<wikibugs>
|
'SRE, ''Patch-For-Review: offboarding Alex Kosiaris - https://phabricator.wikimedia.org/T417465#11619033 (''MoritzMuehlenhoff)'
|
|
2026-02-16 08:16:12
|
<Jhs>
|
yeah, exactly
|
|
2026-02-16 08:16:26
|
<hashar>
|
and you seem to know all about the impact it has on the wikis
|
|
2026-02-16 08:16:55
|
<hashar>
|
so I am inclined to enable it anywhere. I am not sure who manages that at the WMF though
|
|
2026-02-16 08:17:09
|
<wikibugs>
|
('PS1) ''Muehlenhoff: Remove LDAP access for tadeleye [puppet] - ''https://gerrit.wikimedia.org/r/1239583'
|
|
2026-02-16 08:17:29
|
<wikibugs>
|
('CR) ''CI reject: [V:''-1] Remove LDAP access for tadeleye [puppet] - ''https://gerrit.wikimedia.org/r/1239583 (owner: ''Muehlenhoff)'
|
|
2026-02-16 08:18:32
|
<wikibugs>
|
('PS2) ''Muehlenhoff: Remove LDAP access for tadeleye [puppet] - ''https://gerrit.wikimedia.org/r/1239583'
|
|
2026-02-16 08:19:17
|
<jinxer-wm>
|
FIRING: [2x] ProbeDown: Service wdqs1013:443 has failed probes (http_wdqs_main_external_search_sparql_endpoint_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#wdqs1013:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
|
|
2026-02-16 08:21:01
|
<logmsgbot>
|
!log arnaudb@cumin1003 START - Cookbook sre.gerrit.sync-instances sync Gerrit data from gerrit1003.wikimedia.org to gerrit2003.wikimedia.org
|
|
2026-02-16 08:21:59
|
<hashar>
|
Msz2001: let me know when you are done and I will proceed the other patches
|
|
2026-02-16 08:22:16
|
<Msz2001>
|
Sure, it's syncing test servers now
|
|
2026-02-16 08:22:21
|
<hashar>
|
it will probably take a while
|
|
2026-02-16 08:22:37
|
<Msz2001>
|
Yeah, monday mornings take quite long time
|
|
2026-02-16 08:22:41
|
<hashar>
|
iirc the first deploy of the week implies syncing a full mw image whch is ... large
|
|
2026-02-16 08:22:43
|
<hashar>
|
yeah
|
|
2026-02-16 08:22:54
|
<hashar>
|
I need to raise it up so it can be acted upon
|
|
2026-02-16 08:24:01
|
<Jhs>
|
hashar, done at T417538. If there are no objections in the task, maybe I could add a patch for it in a week's time or so?
|
|
2026-02-16 08:24:02
|
<stashbot>
|
T417538: Enable PageImages by default for Wikisource and Wikibooks - https://phabricator.wikimedia.org/T417538
|
|
2026-02-16 08:24:27
|
<logmsgbot>
|
!log arnaudb@cumin1003 END (PASS) - Cookbook sre.gerrit.sync-instances (exit_code=0) sync Gerrit data from gerrit1003.wikimedia.org to gerrit2003.wikimedia.org
|
|
2026-02-16 08:24:28
|
<jinxer-wm>
|
RESOLVED: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag
|
|
2026-02-16 08:25:38
|
<hashar>
|
Jhs: do you know how to make wikisource & wikibooks communities to be made aware of this?
|
|
2026-02-16 08:25:56
|
<hashar>
|
we have a technology community newsletter iirc
|
|
2026-02-16 08:26:15
|
<logmsgbot>
|
!log mszwarc@deploy2002 mszwarc, kharlan: Backport for [[gerrit:1239337|Add infobox case handling for Special:IPContributions (T417250)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
|
|
2026-02-16 08:26:19
|
<stashbot>
|
T417250: "IP information for this address cannot be retrieved since no contributions have been made from it" on Special:IPContributions - https://phabricator.wikimedia.org/T417250
|
|
2026-02-16 08:26:19
|
<Jhs>
|
Not sure about Wikibooks, but there are some Telegram groups for Wikisource with representation from many communities (that's where this was first brought to my attention, in fact)
|
|
2026-02-16 08:26:19
|
<hashar>
|
but I don't know who maintains it or how to reach out to them. Though I can look it up
|
|
2026-02-16 08:26:42
|
<hashar>
|
I guess you can poke the telegram group to have some reply added on that task
|
|
2026-02-16 08:26:49
|
<Jhs>
|
👍
|
|
2026-02-16 08:26:52
|
<logmsgbot>
|
!log mszwarc@deploy2002 mszwarc, kharlan: Continuing with sync
|
|
2026-02-16 08:26:55
|
<hashar>
|
and I will ask inside the WMF if someone can look it up and handle the comm
|
|
2026-02-16 08:27:46
|
<hashar>
|
I have added that as a reminder for this afternoon :)
|
|
2026-02-16 08:27:54
|
<Jhs>
|
thanks!
|
|
2026-02-16 08:27:55
|
<hashar>
|
I got an other operation scheduled this morning
|
|
2026-02-16 08:28:28
|
<hashar>
|
and if you don't hear back from me, feel free to ping me on the task with `@hashar` and some message ;)
|
|
2026-02-16 08:28:43
|
<hashar>
|
I am doing too many thing and I easily forget about stuff
|
|
2026-02-16 08:28:45
|
<wikibugs>
|
'SRE, ''MediaWiki-Shell, ''serviceops: Support cgroup 2 - https://phabricator.wikimedia.org/T417502#11619071 (''MatthewVernon) [I think this is the correct subteam assignment, if not LMK and/or repoint to the correct team]'
|
|
2026-02-16 08:28:46
|
<Jhs>
|
hashar, hah, me too, am in the hospital right now
|
|
2026-02-16 08:28:51
|
<hashar>
|
oh
|
|
2026-02-16 08:29:08
|
<Jhs>
|
(well, i just heart like 30 mins ago that it's pushed back until tomorrow, so... i'm stuck here being bored)
|
|
2026-02-16 08:29:23
|
<hashar>
|
good
|
|
2026-02-16 08:29:34
|
<Jhs>
|
good luck with yours!
|
|
2026-02-16 08:30:38
|
<Jhs>
|
(just realized maybe you meant an operation like #wikimedia-operations operation, but good luck anyways :P )
|
|
2026-02-16 08:30:49
|
<wikibugs>
|
('CR) ''Muehlenhoff: [C:''+2] Remove LDAP access for tadeleye [puppet] - ''https://gerrit.wikimedia.org/r/1239583 (owner: ''Muehlenhoff)'
|
|
2026-02-16 08:33:22
|
<wikibugs>
|
'SRE, ''Infrastructure-Foundations, ''Patch-For-Review: offboarding Alex Kosiaris - https://phabricator.wikimedia.org/T417465#11619077 (''MatthewVernon)'
|
|
2026-02-16 08:36:37
|
<wikibugs>
|
'SRE, ''MinT, ''Prod-Kubernetes, ''ServiceOps new, and 3 others: Can't deploy machinetranslation due to exceeding resource quotas - https://phabricator.wikimedia.org/T411058#11619096 (''KartikMistry) >>! In T411058#11565014, @KartikMistry wrote: > I'm still debugging, and probably best way to check with
r...'
|
|
2026-02-16 08:39:46
|
<logmsgbot>
|
!log mszwarc@deploy2002 Finished scap sync-world: Backport for [[gerrit:1239337|Add infobox case handling for Special:IPContributions (T417250)]] (duration: 36m 38s)
|
|
2026-02-16 08:39:50
|
<stashbot>
|
T417250: "IP information for this address cannot be retrieved since no contributions have been made from it" on Special:IPContributions - https://phabricator.wikimedia.org/T417250
|
|
2026-02-16 08:40:03
|
<Msz2001>
|
hashar: My deploy finished
|
|
2026-02-16 08:40:07
|
<hashar>
|
awesome
|
|
2026-02-16 08:40:28
|
<hashar>
|
Jhs: kipfel: I am processing your changes now
|
|
2026-02-16 08:40:43
|
<kipfel>
|
i'm here
|
|
2026-02-16 08:41:15
|
<wikibugs>
|
('CR) ''Ayounsi: [C:''+2] Remove Alex from routers [homer/public] - ''https://gerrit.wikimedia.org/r/1239574 (https://phabricator.wikimedia.org/T417465) (owner: ''Muehlenhoff)'
|
|
2026-02-16 08:41:31
|
<wikibugs>
|
('CR) ''Daniel Kinzler: [C:''-1] "should not yet enable in prod" [deployment-charts] - ''https://gerrit.wikimedia.org/r/1225699 (https://phabricator.wikimedia.org/T413183) (owner: ''Daniel Kinzler)'
|
|
2026-02-16 08:42:42
|
<wikibugs>
|
('CR) ''TrainBranchBot: [C:''+2] "Approved by hashar@deploy2002 using scap backport" [extensions/WP25EasterEggs] (wmf/1.46.0-wmf.15) - ''https://gerrit.wikimedia.org/r/1239403 (https://phabricator.wikimedia.org/T417077) (owner: ''Jdrewniak)'
|
|
2026-02-16 08:42:42
|
<wikibugs>
|
('CR) ''TrainBranchBot: [C:''+2] "Approved by hashar@deploy2002 using scap backport" [extensions/WP25EasterEggs] (wmf/1.46.0-wmf.15) - ''https://gerrit.wikimedia.org/r/1239404 (owner: ''Jdrewniak)'
|
|
2026-02-16 08:42:43
|
<wikibugs>
|
('CR) ''TrainBranchBot: [C:''+2] "Approved by hashar@deploy2002 using scap backport" [extensions/WP25EasterEggs] (wmf/1.46.0-wmf.15) - ''https://gerrit.wikimedia.org/r/1239419 (https://phabricator.wikimedia.org/T410091) (owner: ''Jdrewniak)'
|
|
2026-02-16 08:42:43
|
<wikibugs>
|
('CR) ''TrainBranchBot: [C:''+2] "Approved by hashar@deploy2002 using scap backport" [extensions/WP25EasterEggs] (wmf/1.46.0-wmf.15) - ''https://gerrit.wikimedia.org/r/1239418 (https://phabricator.wikimedia.org/T417078) (owner: ''Jdrewniak)'
|
|
2026-02-16 08:42:44
|
<wikibugs>
|
('CR) ''TrainBranchBot: [C:''+2] "Approved by hashar@deploy2002 using scap backport" [mediawiki-config] - ''https://gerrit.wikimedia.org/r/1239400 (https://phabricator.wikimedia.org/T417110) (owner: ''Jdrewniak)'
|
|
2026-02-16 08:42:45
|
<wikibugs>
|
('CR) ''TrainBranchBot: [C:''+2] "Approved by hashar@deploy2002 using scap backport" [mediawiki-config] - ''https://gerrit.wikimedia.org/r/1239468 (https://phabricator.wikimedia.org/T417240) (owner: ''Stang)'
|
|
2026-02-16 08:42:49
|
<wikibugs>
|
('CR) ''TrainBranchBot: [C:''+2] "Approved by hashar@deploy2002 using scap backport" [mediawiki-config] - ''https://gerrit.wikimedia.org/r/1239571 (https://phabricator.wikimedia.org/T416800) (owner: ''Jon Harald Søby)'
|
|
2026-02-16 08:42:53
|
<hashar>
|
I am doing them all together in the interest of time
|
|
2026-02-16 08:42:57
|
<wikibugs>
|
('Merged) ''jenkins-bot: Remove Alex from routers [homer/public] - ''https://gerrit.wikimedia.org/r/1239574 (https://phabricator.wikimedia.org/T417465) (owner: ''Muehlenhoff)'
|
|
2026-02-16 08:42:57
|
<hashar>
|
arnaudb: ^ :)
|
|
2026-02-16 08:43:14
|
<arnaudb>
|
ack!
|
|
2026-02-16 08:43:17
|
<hashar>
|
arnaudb: I am deploying everything in one go which really is "just" 3 features being turned on
|
|
2026-02-16 08:43:28
|
<hashar>
|
a couple are to be verified by kipfel and Jhs
|
|
2026-02-16 08:43:39
|
<hashar>
|
the other is the WP25 extension and I'll do it
|
|
2026-02-16 08:43:49
|
<wikibugs>
|
('Merged) ''jenkins-bot: Setting $wgWp25EasterEggsEnable to true for Wikipedias. [mediawiki-config] - ''https://gerrit.wikimedia.org/r/1239400 (https://phabricator.wikimedia.org/T417110) (owner: ''Jdrewniak)'
|
|
2026-02-16 08:43:57
|
<wikibugs>
|
('Merged) ''jenkins-bot: zhwiki: Add 2026 CNY celebration logos [mediawiki-config] - ''https://gerrit.wikimedia.org/r/1239468 (https://phabricator.wikimedia.org/T417240) (owner: ''Stang)'
|
|
2026-02-16 08:44:00
|
<wikibugs>
|
('Merged) ''jenkins-bot: Enable PageImages for bnwikisource [mediawiki-config] - ''https://gerrit.wikimedia.org/r/1239571 (https://phabricator.wikimedia.org/T416800) (owner: ''Jon Harald Søby)'
|
|
2026-02-16 08:44:47
|
<wikibugs>
|
('CR) ''Muehlenhoff: "What is the exact failure, where did you see this? We don't run buster any more in cloud VPS and the last prod node is also going away. Th" [puppet] - ''https://gerrit.wikimedia.org/r/1239434 (https://phabricator.wikimedia.org/T401832) (owner: ''BCornwall)'
|
|
2026-02-16 08:47:12
|
<wikibugs>
|
('PS1) ''Muehlenhoff: Run Puppetboard spec tests on Bookworm [puppet] - ''https://gerrit.wikimedia.org/r/1239586'
|
|
2026-02-16 08:48:06
|
<wikibugs>
|
('CR) ''CI reject: [V:''-1] Run Puppetboard spec tests on Bookworm [puppet] - ''https://gerrit.wikimedia.org/r/1239586 (owner: ''Muehlenhoff)'
|
|
2026-02-16 08:48:49
|
<wikibugs>
|
('PS1) ''Muehlenhoff: Stop running the Gitlab spec tests on Buster [puppet] - ''https://gerrit.wikimedia.org/r/1239588'
|
|
2026-02-16 08:48:57
|
<wikibugs>
|
('CR) ''JMeybohm: "Ah, I did not realize the package_from_component resource name is the same in both cases...not sure what will happen tbh." [puppet] - ''https://gerrit.wikimedia.org/r/1239135 (https://phabricator.wikimedia.org/T416670) (owner: ''Elukey)'
|
|
2026-02-16 08:49:12
|
<Jhs>
|
hashar, is it ready to test on mwdebug yet?
|
|
2026-02-16 08:49:40
|
<wikibugs>
|
('CR) ''CI reject: [V:''-1] Stop running the Gitlab spec tests on Buster [puppet] - ''https://gerrit.wikimedia.org/r/1239588 (owner: ''Muehlenhoff)'
|
|
2026-02-16 08:49:53
|
<hashar>
|
soon
|
|
2026-02-16 08:50:04
|
<wikibugs>
|
('Merged) ''jenkins-bot: Add "Learn more" link below Baby Globe on Minerva [extensions/WP25EasterEggs] (wmf/1.46.0-wmf.15) - ''https://gerrit.wikimedia.org/r/1239403 (https://phabricator.wikimedia.org/T417077) (owner: ''Jdrewniak)'
|
|
2026-02-16 08:50:04
|
<hashar>
|
there are some patches still in CI
|
|
2026-02-16 08:50:07
|
<hashar>
|
which are about to complete
|
|
2026-02-16 08:50:27
|
<wikibugs>
|
('PS2) ''Muehlenhoff: Run Puppetboard spec tests on Bookworm [puppet] - ''https://gerrit.wikimedia.org/r/1239586'
|
|
2026-02-16 08:50:36
|
<wikibugs>
|
('PS2) ''Muehlenhoff: Stop running the Gitlab spec tests on Buster [puppet] - ''https://gerrit.wikimedia.org/r/1239588'
|
|
2026-02-16 08:50:37
|
<hashar>
|
(the whole CI tests are taking way too long but I am wokring on that with others :) )
|
|
2026-02-16 08:51:09
|
<wikibugs>
|
('CR) ''CI reject: [V:''-1] Run Puppetboard spec tests on Bookworm [puppet] - ''https://gerrit.wikimedia.org/r/1239586 (owner: ''Muehlenhoff)'
|
|
2026-02-16 08:51:18
|
<wikibugs>
|
('CR) ''CI reject: [V:''-1] Stop running the Gitlab spec tests on Buster [puppet] - ''https://gerrit.wikimedia.org/r/1239588 (owner: ''Muehlenhoff)'
|
|
2026-02-16 08:51:59
|
<wikibugs>
|
('Merged) ''jenkins-bot: Update Qids to initial public version [extensions/WP25EasterEggs] (wmf/1.46.0-wmf.15) - ''https://gerrit.wikimedia.org/r/1239404 (owner: ''Jdrewniak)'
|
|
2026-02-16 08:51:59
|
<wikibugs>
|
('Merged) ''jenkins-bot: Escape the unescaped i18n messages [extensions/WP25EasterEggs] (wmf/1.46.0-wmf.15) - ''https://gerrit.wikimedia.org/r/1239419 (https://phabricator.wikimedia.org/T410091) (owner: ''Jdrewniak)'
|
|
2026-02-16 08:52:00
|
<wikibugs>
|
('Merged) ''jenkins-bot: Do not show companion when visual editor is active [extensions/WP25EasterEggs] (wmf/1.46.0-wmf.15) - ''https://gerrit.wikimedia.org/r/1239418 (https://phabricator.wikimedia.org/T417078) (owner: ''Jdrewniak)'
|
|
2026-02-16 08:52:20
|
<wikibugs>
|
('PS11) ''Daniel Kinzler: rest gateway: implement per-policy shadow mode [deployment-charts] - ''https://gerrit.wikimedia.org/r/1225699 (https://phabricator.wikimedia.org/T413183)'
|
|
2026-02-16 08:52:21
|
<hashar>
|
they all merged
|
|
2026-02-16 08:52:24
|
<logmsgbot>
|
!log hashar@deploy2002 Started scap sync-world: Backport for [[gerrit:1239403|Add "Learn more" link below Baby Globe on Minerva (T417077)]], [[gerrit:1239404|Update Qids to initial public version]], [[gerrit:1239419|Escape the unescaped i18n messages (T410091)]], [[gerrit:1239418|Do not show companion when visual editor is active (T417078)]], [[gerrit:1239400|Setting $wgWp25EasterEggsEnable to true for Wikipedias. (T41711
|
|
2026-02-16 08:52:24
|
<logmsgbot>
|
0)]], [[gerrit:1239468|zhwiki: Add 2026 CNY celebration logos (T417240)]], [[gerrit:1239571|Enable PageImages for bnwikisource (T416800)]]
|
|
2026-02-16 08:52:33
|
<stashbot>
|
T417077: Add link to settings below Baby Globe on Minerva - https://phabricator.wikimedia.org/T417077
|
|
2026-02-16 08:52:33
|
<stashbot>
|
T410091: Security review for Extension:WP25EasterEggs - https://phabricator.wikimedia.org/T410091
|
|
2026-02-16 08:52:34
|
<stashbot>
|
T417078: Hide Baby Globe when the article editing opens without navigation - https://phabricator.wikimedia.org/T417078
|
|
2026-02-16 08:52:34
|
<wikibugs>
|
('PS3) ''Muehlenhoff: Stop running the Gitlab spec tests on Buster [puppet] - ''https://gerrit.wikimedia.org/r/1239588'
|
|
2026-02-16 08:52:34
|
<stashbot>
|
T41711: Login should only be required when uploading photos - https://phabricator.wikimedia.org/T41711
|
|
2026-02-16 08:52:35
|
<stashbot>
|
T417240: Requesting temporary logo change for zhwiki (CNY 2026) - https://phabricator.wikimedia.org/T417240
|
|
2026-02-16 08:52:35
|
<stashbot>
|
T416800: Enable PageImages extension in bnwikisource - https://phabricator.wikimedia.org/T416800
|
|
2026-02-16 08:53:08
|
<hashar>
|
and of course that had to trigger a localization update refresh
|
|
2026-02-16 08:53:16
|
<wikibugs>
|
('CR) ''CI reject: [V:''-1] Stop running the Gitlab spec tests on Buster [puppet] - ''https://gerrit.wikimedia.org/r/1239588 (owner: ''Muehlenhoff)'
|
|
2026-02-16 08:53:24
|
<hashar>
|
I should have gotten those code changes merged together with the other patch
|
|
2026-02-16 08:53:28
|
<hashar>
|
anyway
|
|
2026-02-16 08:54:00
|
<wikibugs>
|
'SRE, ''Infrastructure-Foundations, ''Patch-For-Review: offboarding Alex Kosiaris - https://phabricator.wikimedia.org/T417465#11619165 (''MoritzMuehlenhoff)'
|
|
2026-02-16 08:54:56
|
<wikibugs>
|
('PS4) ''Muehlenhoff: Stop running the Gitlab spec tests on Buster [puppet] - ''https://gerrit.wikimedia.org/r/1239588'
|
|
2026-02-16 08:55:20
|
<wikibugs>
|
'SRE, ''MediaWiki-Shell, ''ServiceOps new, ''ServiceOps-Mediawiki: Support cgroup 2 - https://phabricator.wikimedia.org/T417502#11619167 (''JMeybohm) Could you please elaborate a bit on what you're trying to do exactly and where? In production, mediawiki is running in containers on kubernetes. So it is li...'
|
|
2026-02-16 08:55:23
|
<hashar>
|
the new image is being pushed
|
|
2026-02-16 08:55:50
|
<wikibugs>
|
('CR) ''Elukey: "I think it is a good point, the name of the packages are different (confluent-kafka-2.11 vs confluent-kafka) so I believe both are install" [puppet] - ''https://gerrit.wikimedia.org/r/1239135 (https://phabricator.wikimedia.org/T416670) (owner: ''Elukey)'
|
|
2026-02-16 08:57:15
|
<wikibugs>
|
('PS3) ''Muehlenhoff: Run Puppetboard spec tests on Bookworm [puppet] - ''https://gerrit.wikimedia.org/r/1239586'
|
|
2026-02-16 08:59:57
|
<wikibugs>
|
('PS1) ''Muehlenhoff: Remove obsolete spec test [puppet] - ''https://gerrit.wikimedia.org/r/1239591'
|
|
2026-02-16 09:00:05
|
<jouncebot>
|
arnaudb, hashar, and sobanski: Deploy window Gerrit (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260216T0900)
|
|
2026-02-16 09:01:55
|
<arnaudb>
|
we might start a few minutes late
|
|
2026-02-16 09:07:06
|
<hashar>
|
09:06:02 [mediawiki-publish-83] Waiting 300 seconds for swift after full mediawiki image build (T390251)
|
|
2026-02-16 09:07:06
|
<stashbot>
|
T390251: docker-registry.wikimedia.org keeps serving bad blobs - https://phabricator.wikimedia.org/T390251
|
|
2026-02-16 09:07:07
|
<hashar>
|
:-(
|
|
2026-02-16 09:08:23
|
<wikibugs>
|
('CR) ''Vgutierrez: "check experimental" [puppet] - ''https://gerrit.wikimedia.org/r/1239376 (https://phabricator.wikimedia.org/T417291) (owner: ''Vgutierrez)'
|
|
2026-02-16 09:08:36
|
<hashar>
|
Jhs: kipfel: it is still in the pipes :-\
|
|
2026-02-16 09:08:42
|
<wikibugs>
|
('PS1) ''MVernon: Add hmonroy to analytics-privatedata-users, enable krb [puppet] - ''https://gerrit.wikimedia.org/r/1239595 (https://phabricator.wikimedia.org/T417459)'
|
|
2026-02-16 09:09:16
|
<Jhs>
|
yeah, i figured. no rush on my part though
|
|
2026-02-16 09:09:22
|
<wikibugs>
|
'SRE, ''SRE-Access-Requests, ''Patch-For-Review: Requesting access to analytics-privatedata-users for HMonroy - https://phabricator.wikimedia.org/T417459#11619201 (''MatthewVernon)'
|
|
2026-02-16 09:10:26
|
<wikibugs>
|
('PS1) ''Muehlenhoff: Mark WDQS spec tests to run on Bullseye [puppet] - ''https://gerrit.wikimedia.org/r/1239596'
|
|
2026-02-16 09:10:49
|
<wikibugs>
|
'SRE, ''Infrastructure-Foundations, ''netops: Update esams network pop diagrams - https://phabricator.wikimedia.org/T368084#11619205 (''ayounsi) Nice, latest version LGTM !'
|
|
2026-02-16 09:11:09
|
<wikibugs>
|
('CR) ''CI reject: [V:''-1] Mark WDQS spec tests to run on Bullseye [puppet] - ''https://gerrit.wikimedia.org/r/1239596 (owner: ''Muehlenhoff)'
|
|
2026-02-16 09:11:49
|
<hashar>
|
09:11:04 [root] Image builds completed
|
|
2026-02-16 09:13:53
|
<wikibugs>
|
('PS2) ''Muehlenhoff: Mark WDQS spec tests to run on Bullseye [puppet] - ''https://gerrit.wikimedia.org/r/1239596'
|
|
2026-02-16 09:15:30
|
<wikibugs>
|
('PS1) ''PipelineBot: wikifeeds: pipeline bot promote [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239598'
|
|
2026-02-16 09:15:39
|
<wikibugs>
|
('CR) ''Arnaudb: [C:''+1] Add hmonroy to analytics-privatedata-users, enable krb [puppet] - ''https://gerrit.wikimedia.org/r/1239595 (https://phabricator.wikimedia.org/T417459) (owner: ''MVernon)'
|
|
2026-02-16 09:16:13
|
<wikibugs>
|
('CR) ''Muehlenhoff: [C:''+1] "LGTM" [puppet] - ''https://gerrit.wikimedia.org/r/1239595 (https://phabricator.wikimedia.org/T417459) (owner: ''MVernon)'
|
|
2026-02-16 09:16:19
|
<logmsgbot>
|
!log hashar@deploy2002 jdrewniak, jhsoby, hashar, stang: Backport for [[gerrit:1239403|Add "Learn more" link below Baby Globe on Minerva (T417077)]], [[gerrit:1239404|Update Qids to initial public version]], [[gerrit:1239419|Escape the unescaped i18n messages (T410091)]], [[gerrit:1239418|Do not show companion when visual editor is active (T417078)]], [[gerrit:1239400|Setting $wgWp25EasterEggsEnable to true for Wikipedias
|
|
2026-02-16 09:16:20
|
<logmsgbot>
|
. (T417110)]], [[gerrit:1239468|zhwiki: Add 2026 CNY celebration logos (T417240)]], [[gerrit:1239571|Enable PageImages for bnwikisource (T416800)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
|
|
2026-02-16 09:16:26
|
<stashbot>
|
T417077: Add link to settings below Baby Globe on Minerva - https://phabricator.wikimedia.org/T417077
|
|
2026-02-16 09:16:26
|
<stashbot>
|
T410091: Security review for Extension:WP25EasterEggs - https://phabricator.wikimedia.org/T410091
|
|
2026-02-16 09:16:27
|
<stashbot>
|
T417078: Hide Baby Globe when the article editing opens without navigation - https://phabricator.wikimedia.org/T417078
|
|
2026-02-16 09:16:27
|
<stashbot>
|
T417110: WP25 Easter Egg deployment: enable $wgWp25EasterEggsEnable config flag - https://phabricator.wikimedia.org/T417110
|
|
2026-02-16 09:16:27
|
<stashbot>
|
T417240: Requesting temporary logo change for zhwiki (CNY 2026) - https://phabricator.wikimedia.org/T417240
|
|
2026-02-16 09:16:28
|
<stashbot>
|
T416800: Enable PageImages extension in bnwikisource - https://phabricator.wikimedia.org/T416800
|
|
2026-02-16 09:16:58
|
<hashar>
|
Jhs: kipfel: the code is availalbe on the tests servers!
|
|
2026-02-16 09:17:51
|
<wikibugs>
|
('CR) ''Elukey: [C:''+1] Remove obsolete spec test [puppet] - ''https://gerrit.wikimedia.org/r/1239591 (owner: ''Muehlenhoff)'
|
|
2026-02-16 09:18:19
|
<kipfel>
|
hashar: the logo looks good to me
|
|
2026-02-16 09:19:10
|
<wikibugs>
|
('PS1) ''Muehlenhoff: Run Bird spec tests on Bookworm/Trixie [puppet] - ''https://gerrit.wikimedia.org/r/1239599 (https://phabricator.wikimedia.org/T335765)'
|
|
2026-02-16 09:19:39
|
<wikibugs>
|
('CR) ''MVernon: [C:''+2] Add hmonroy to analytics-privatedata-users, enable krb [puppet] - ''https://gerrit.wikimedia.org/r/1239595 (https://phabricator.wikimedia.org/T417459) (owner: ''MVernon)'
|
|
2026-02-16 09:19:53
|
<wikibugs>
|
('CR) ''CI reject: [V:''-1] Run Bird spec tests on Bookworm/Trixie [puppet] - ''https://gerrit.wikimedia.org/r/1239599 (https://phabricator.wikimedia.org/T335765) (owner: ''Muehlenhoff)'
|
|
2026-02-16 09:20:43
|
<hashar>
|
the WP25 sounds good to me as well, I will check the log
|
|
2026-02-16 09:21:31
|
<wikibugs>
|
('PS1) ''Muehlenhoff: dnsdist: Run spec tests on Bookworm/Trixie [puppet] - ''https://gerrit.wikimedia.org/r/1239600 (https://phabricator.wikimedia.org/T335765)'
|
|
2026-02-16 09:22:13
|
<wikibugs>
|
('CR) ''CI reject: [V:''-1] dnsdist: Run spec tests on Bookworm/Trixie [puppet] - ''https://gerrit.wikimedia.org/r/1239600 (https://phabricator.wikimedia.org/T335765) (owner: ''Muehlenhoff)'
|
|
2026-02-16 09:22:42
|
<logmsgbot>
|
!log hashar@deploy2002 jdrewniak, jhsoby, hashar, stang: Continuing with sync
|
|
2026-02-16 09:22:44
|
<wikibugs>
|
'SRE, ''SRE-Access-Requests, ''Patch-For-Review: Requesting access to analytics-privatedata-users for HMonroy - https://phabricator.wikimedia.org/T417459#11619232 (''MatthewVernon) ''Open→''Resolved a:''MatthewVernon All done. @HMonroy you should have had an email with a temporary kerberos passwor...'
|
|
2026-02-16 09:22:58
|
<hashar>
|
I have confirmed PageImages is enabled on bnwikisource
|
|
2026-02-16 09:23:24
|
<wikibugs>
|
('PS1) ''Vgutierrez: cache::wmfuniq: Use same filesystem for tempfile to avoid cross-filesystem errors [puppet] - ''https://gerrit.wikimedia.org/r/1239602 (https://phabricator.wikimedia.org/T401832)'
|
|
2026-02-16 09:24:29
|
<wikibugs>
|
('PS1) ''Muehlenhoff: Run cloudlb spec tests on Bookworm [puppet] - ''https://gerrit.wikimedia.org/r/1239603'
|
|
2026-02-16 09:24:53
|
<hashar>
|
changes are now syncing to the whole infra
|
|
2026-02-16 09:25:15
|
<wikibugs>
|
('CR) ''CI reject: [V:''-1] cache::wmfuniq: Use same filesystem for tempfile to avoid cross-filesystem errors [puppet] - ''https://gerrit.wikimedia.org/r/1239602 (https://phabricator.wikimedia.org/T401832) (owner: ''Vgutierrez)'
|
|
2026-02-16 09:25:41
|
<wikibugs>
|
('CR) ''Fabfur: [C:''+1] "LGTM" [puppet] - ''https://gerrit.wikimedia.org/r/1239376 (https://phabricator.wikimedia.org/T417291) (owner: ''Vgutierrez)'
|
|
2026-02-16 09:27:15
|
<wikibugs>
|
('PS2) ''Muehlenhoff: dnsdist: Run spec tests on Bookworm/Trixie [puppet] - ''https://gerrit.wikimedia.org/r/1239600 (https://phabricator.wikimedia.org/T335765)'
|
|
2026-02-16 09:27:25
|
<wikibugs>
|
('CR) ''Ayounsi: "Chane lgtm, @ssingh@wikimedia.org ping me when it's time to deploy it." [homer/public] - ''https://gerrit.wikimedia.org/r/1238015 (https://phabricator.wikimedia.org/T81605) (owner: ''Cathal Mooney)'
|
|
2026-02-16 09:27:37
|
<wikibugs>
|
('PS2) ''Muehlenhoff: Run Bird spec tests on Bookworm/Trixie [puppet] - ''https://gerrit.wikimedia.org/r/1239599 (https://phabricator.wikimedia.org/T335765)'
|
|
2026-02-16 09:27:52
|
<wikibugs>
|
('CR) ''Muehlenhoff: [C:''+2] Remove obsolete spec test [puppet] - ''https://gerrit.wikimedia.org/r/1239591 (owner: ''Muehlenhoff)'
|
|
2026-02-16 09:32:03
|
<wikibugs>
|
('PS1) ''Federico Ceratto: mariadb: allow dborch1002 to db1215 connection [puppet] - ''https://gerrit.wikimedia.org/r/1239606 (https://phabricator.wikimedia.org/T416582)'
|
|
2026-02-16 09:32:03
|
<wikibugs>
|
('CR) ''Federico Ceratto: "The change on db1215 look ok, I'm not sure where the ACME change is being deployed https://puppet-compiler.wmflabs.org/output/1239606/8040"; [puppet] - ''https://gerrit.wikimedia.org/r/1239606 (https://phabricator.wikimedia.org/T416582) (owner: ''Federico Ceratto)'
|
|
2026-02-16 09:34:01
|
<wikibugs>
|
('PS2) ''Vgutierrez: cache::wmfuniq: Fix cross-filesystem tempfile error on Trixie [puppet] - ''https://gerrit.wikimedia.org/r/1239602 (https://phabricator.wikimedia.org/T401832)'
|
|
2026-02-16 09:34:13
|
<wikibugs>
|
('PS24) ''Daniel Kinzler: rest gateway: add tests for chart rendering [deployment-charts] - ''https://gerrit.wikimedia.org/r/1225085'
|
|
2026-02-16 09:34:27
|
<Jhs>
|
hashar, thanks! sorry, was busy there for a bit
|
|
2026-02-16 09:34:40
|
<hashar>
|
that is understandable no worries :)
|
|
2026-02-16 09:34:57
|
<hashar>
|
I have verified on the test server that PageImages shows in Special:Version of bnwikisource
|
|
2026-02-16 09:35:04
|
<hashar>
|
the change is about to go live in prod
|
|
2026-02-16 09:35:17
|
<logmsgbot>
|
!log hashar@deploy2002 Finished scap sync-world: Backport for [[gerrit:1239403|Add "Learn more" link below Baby Globe on Minerva (T417077)]], [[gerrit:1239404|Update Qids to initial public version]], [[gerrit:1239419|Escape the unescaped i18n messages (T410091)]], [[gerrit:1239418|Do not show companion when visual editor is active (T417078)]], [[gerrit:1239400|Setting $wgWp25EasterEggsEnable to true for Wikipedias. (T4171
|
|
2026-02-16 09:35:17
|
<logmsgbot>
|
10)]], [[gerrit:1239468|zhwiki: Add 2026 CNY celebration logos (T417240)]], [[gerrit:1239571|Enable PageImages for bnwikisource (T416800)]] (duration: 42m 52s)
|
|
2026-02-16 09:35:25
|
<stashbot>
|
T417077: Add link to settings below Baby Globe on Minerva - https://phabricator.wikimedia.org/T417077
|
|
2026-02-16 09:35:26
|
<stashbot>
|
T410091: Security review for Extension:WP25EasterEggs - https://phabricator.wikimedia.org/T410091
|
|
2026-02-16 09:35:26
|
<stashbot>
|
T417078: Hide Baby Globe when the article editing opens without navigation - https://phabricator.wikimedia.org/T417078
|
|
2026-02-16 09:35:27
|
<stashbot>
|
T417240: Requesting temporary logo change for zhwiki (CNY 2026) - https://phabricator.wikimedia.org/T417240
|
|
2026-02-16 09:35:27
|
<stashbot>
|
T4171: Query page to list protected pages - https://phabricator.wikimedia.org/T4171
|
|
2026-02-16 09:35:27
|
<stashbot>
|
T416800: Enable PageImages extension in bnwikisource - https://phabricator.wikimedia.org/T416800
|
|
2026-02-16 09:36:16
|
<wikibugs>
|
('CR) ''Daniel Kinzler: rest gateway: add tests for chart rendering (''2 comments) [deployment-charts] - ''https://gerrit.wikimedia.org/r/1225085 (owner: ''Daniel Kinzler)'
|
|
2026-02-16 09:36:19
|
<wikibugs>
|
('CR) ''Vgutierrez: [C:''+2] haproxy: Use lua5.4-maxminddb for haproxy 3.0 [puppet] - ''https://gerrit.wikimedia.org/r/1239376 (https://phabricator.wikimedia.org/T417291) (owner: ''Vgutierrez)'
|
|
2026-02-16 09:38:16
|
<wikibugs>
|
('CR) ''Jelto: [C:''+1] "lgtm, thanks for the cleanup" [puppet] - ''https://gerrit.wikimedia.org/r/1239588 (owner: ''Muehlenhoff)'
|
|
2026-02-16 09:41:21
|
<hashar>
|
Lucas_WMDE: codders: we are about to shut down Gerrit for a scheduled maintenance. That got posted on wikitech-l but I am not sure whether the info made it to WMDE people
|
|
2026-02-16 09:41:37
|
<Lucas_WMDE>
|
I forwarded it
|
|
2026-02-16 09:41:44
|
<hashar>
|
awesome :)
|
|
2026-02-16 09:43:44
|
<wikibugs>
|
('CR) ''Muehlenhoff: [C:''+2] Stop running the Gitlab spec tests on Buster [puppet] - ''https://gerrit.wikimedia.org/r/1239588 (owner: ''Muehlenhoff)'
|
|
2026-02-16 09:45:21
|
<wikibugs>
|
('CR) ''Slyngshede: [C:''+1] "It feels a little weird having a temp file in /etc, but for the purpose it's fine." [puppet] - ''https://gerrit.wikimedia.org/r/1239602 (https://phabricator.wikimedia.org/T401832) (owner: ''Vgutierrez)'
|
|
2026-02-16 09:46:03
|
<jinxer-wm>
|
FIRING: [5x] PuppetCertificateAboutToExpire: Puppet CA certificate _etcd-server-ssl._tcp.ml_etcd.codfw.wmnet is about to expire - https://wikitech.wikimedia.org/wiki/Puppet#Renew_agent_certificate - TODO - https://alerts.wikimedia.org/?q=alertname%3DPuppetCertificateAboutToExpire
|
|
2026-02-16 09:47:21
|
<wikibugs>
|
('CR) ''Ayounsi: [C:''+1] "lgtm, 1 nit." [software/netbox-extras] - ''https://gerrit.wikimedia.org/r/1238379 (https://phabricator.wikimedia.org/T403035) (owner: ''Cathal Mooney)'
|
|
2026-02-16 09:47:44
|
<wikibugs>
|
('CR) ''Fabfur: "I'm afraid this has been already implemented in https://gerrit.wikimedia.org/r/c/operations/puppet/+/1239376"; [puppet] - ''https://gerrit.wikimedia.org/r/1239463 (https://phabricator.wikimedia.org/T401832) (owner: ''BCornwall)'
|
|
2026-02-16 09:48:42
|
<arnaudb>
|
we'll start the switchover
|
|
2026-02-16 09:49:02
|
<wikibugs>
|
('CR) ''Arnaudb: [C:''+2] gerrit: Switchover gerrit1003 → gerrit2003 [puppet] - ''https://gerrit.wikimedia.org/r/1217133 (https://phabricator.wikimedia.org/T338470) (owner: ''Arnaudb)'
|
|
2026-02-16 09:49:08
|
<wikibugs>
|
('CR) ''Arnaudb: [C:''+2] gerrit: switchover from gerrit1003 to gerrit2003 [dns] - ''https://gerrit.wikimedia.org/r/1238708 (https://phabricator.wikimedia.org/T387833) (owner: ''Arnaudb)'
|
|
2026-02-16 09:49:11
|
<moritzm>
|
good luck :-)
|
|
2026-02-16 09:50:01
|
<logmsgbot>
|
!log arnaudb@cumin1003 START - Cookbook sre.gerrit.topology-check Validate Gerrit topology (source=gerrit1003, replica=gerrit2003)
|
|
2026-02-16 09:50:06
|
<logmsgbot>
|
!log arnaudb@cumin1003 END (PASS) - Cookbook sre.gerrit.topology-check (exit_code=0) Validate Gerrit topology (source=gerrit1003, replica=gerrit2003)
|
|
2026-02-16 09:50:11
|
<logmsgbot>
|
!log arnaudb@cumin1003 START - Cookbook sre.gerrit.failover from gerrit1003.wikimedia.org to gerrit2003.wikimedia.org
|
|
2026-02-16 09:50:18
|
<hashar>
|
:-]
|
|
2026-02-16 09:50:37
|
<wikibugs>
|
('CR) ''Vgutierrez: "dupe of Idee228cf20c011040052d9e4e6e1349de94893f9" [puppet] - ''https://gerrit.wikimedia.org/r/1239463 (https://phabricator.wikimedia.org/T401832) (owner: ''BCornwall)'
|
|
2026-02-16 09:50:40
|
<logmsgbot>
|
!log arnaudb@dns1004 START - running authdns-update
|
|
2026-02-16 09:54:41
|
<logmsgbot>
|
!log arnaudb@cumin1003 START - Cookbook sre.gerrit.read-only-toggle from gerrit1003.wikimedia.org
|
|
2026-02-16 09:55:49
|
<wikibugs>
|
('CR) ''Arnaudb: [C:''+2] "azeaze" [puppet] - ''https://gerrit.wikimedia.org/r/1217133 (https://phabricator.wikimedia.org/T338470) (owner: ''Arnaudb)'
|
|
2026-02-16 09:55:50
|
<wikibugs>
|
('CR) ''LSobanski: "Test comment, ignore." [puppet] - ''https://gerrit.wikimedia.org/r/1239087 (https://phabricator.wikimedia.org/T417263) (owner: ''Jelto)'
|
|
2026-02-16 09:56:23
|
<wikibugs>
|
('CR) ''Vgutierrez: [C:''+2] cache::wmfuniq: Fix cross-filesystem tempfile error on Trixie [puppet] - ''https://gerrit.wikimedia.org/r/1239602 (https://phabricator.wikimedia.org/T401832) (owner: ''Vgutierrez)'
|
|
2026-02-16 09:57:41
|
<logmsgbot>
|
arnaudb@cumin1003 failover (PID 380224) is awaiting input
|
|
2026-02-16 09:59:27
|
<logmsgbot>
|
!log arnaudb@cumin1003 END (PASS) - Cookbook sre.gerrit.read-only-toggle (exit_code=0) from gerrit1003.wikimedia.org
|
|
2026-02-16 09:59:49
|
<logmsgbot>
|
!log arnaudb@cumin1003 START - Cookbook sre.gerrit.read-only-toggle from gerrit2003.wikimedia.org
|
|
2026-02-16 09:59:59
|
<logmsgbot>
|
!log arnaudb@cumin1003 END (PASS) - Cookbook sre.gerrit.read-only-toggle (exit_code=0) from gerrit2003.wikimedia.org
|
|
2026-02-16 10:00:06
|
<logmsgbot>
|
!log arnaudb@cumin1003 START - Cookbook sre.gerrit.localbackup Prepare local backup on: gerrit1003.wikimedia.org
|
|
2026-02-16 10:00:19
|
<logmsgbot>
|
!log arnaudb@cumin1003 END (PASS) - Cookbook sre.gerrit.localbackup (exit_code=0) Prepare local backup on: gerrit1003.wikimedia.org
|
|
2026-02-16 10:01:15
|
<logmsgbot>
|
!log arnaudb@cumin1003 START - Cookbook sre.gerrit.sync-instances sync Gerrit data from gerrit1003.wikimedia.org to gerrit2003.wikimedia.org
|
|
2026-02-16 10:01:34
|
<icinga-wm>
|
PROBLEM - check if authdns-update was run after a change was merged to operations/dns.git on dns1006 is CRITICAL: Local zone files are NOT in sync with operations/dns.git (SHA: local is 8bbb3a70bd5909305fc75bf31af735281b6941a8, dns.git is 48db02e01b8393b9917cf95ff96c6db087401ddc) https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run
|
|
2026-02-16 10:01:34
|
<icinga-wm>
|
PROBLEM - check if authdns-update was run after a change was merged to operations/dns.git on dns1005 is CRITICAL: Local zone files are NOT in sync with operations/dns.git (SHA: local is 8bbb3a70bd5909305fc75bf31af735281b6941a8, dns.git is 48db02e01b8393b9917cf95ff96c6db087401ddc) https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run
|
|
2026-02-16 10:01:34
|
<icinga-wm>
|
PROBLEM - check if authdns-update was run after a change was merged to operations/dns.git on dns2005 is CRITICAL: Local zone files are NOT in sync with operations/dns.git (SHA: local is 8bbb3a70bd5909305fc75bf31af735281b6941a8, dns.git is 48db02e01b8393b9917cf95ff96c6db087401ddc) https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run
|
|
2026-02-16 10:01:34
|
<icinga-wm>
|
PROBLEM - check if authdns-update was run after a change was merged to operations/dns.git on dns2004 is CRITICAL: Local zone files are NOT in sync with operations/dns.git (SHA: local is 8bbb3a70bd5909305fc75bf31af735281b6941a8, dns.git is 48db02e01b8393b9917cf95ff96c6db087401ddc) https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run
|
|
2026-02-16 10:01:40
|
<icinga-wm>
|
PROBLEM - check if authdns-update was run after a change was merged to operations/dns.git on dns2006 is CRITICAL: Local zone files are NOT in sync with operations/dns.git (SHA: local is 8bbb3a70bd5909305fc75bf31af735281b6941a8, dns.git is 48db02e01b8393b9917cf95ff96c6db087401ddc) https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run
|
|
2026-02-16 10:01:40
|
<icinga-wm>
|
PROBLEM - check if authdns-update was run after a change was merged to operations/dns.git on dns4004 is CRITICAL: Local zone files are NOT in sync with operations/dns.git (SHA: local is 8bbb3a70bd5909305fc75bf31af735281b6941a8, dns.git is 48db02e01b8393b9917cf95ff96c6db087401ddc) https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run
|
|
2026-02-16 10:01:40
|
<icinga-wm>
|
PROBLEM - check if authdns-update was run after a change was merged to operations/dns.git on dns4003 is CRITICAL: Local zone files are NOT in sync with operations/dns.git (SHA: local is 8bbb3a70bd5909305fc75bf31af735281b6941a8, dns.git is 48db02e01b8393b9917cf95ff96c6db087401ddc) https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run
|
|
2026-02-16 10:01:42
|
<icinga-wm>
|
PROBLEM - check if authdns-update was run after a change was merged to operations/dns.git on dns3004 is CRITICAL: Local zone files are NOT in sync with operations/dns.git (SHA: local is 8bbb3a70bd5909305fc75bf31af735281b6941a8, dns.git is 48db02e01b8393b9917cf95ff96c6db087401ddc) https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run
|
|
2026-02-16 10:01:42
|
<icinga-wm>
|
PROBLEM - check if authdns-update was run after a change was merged to operations/dns.git on dns3003 is CRITICAL: Local zone files are NOT in sync with operations/dns.git (SHA: local is 8bbb3a70bd5909305fc75bf31af735281b6941a8, dns.git is 48db02e01b8393b9917cf95ff96c6db087401ddc) https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run
|
|
2026-02-16 10:01:42
|
<icinga-wm>
|
PROBLEM - check if authdns-update was run after a change was merged to operations/dns.git on dns6001 is CRITICAL: Local zone files are NOT in sync with operations/dns.git (SHA: local is 8bbb3a70bd5909305fc75bf31af735281b6941a8, dns.git is 48db02e01b8393b9917cf95ff96c6db087401ddc) https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run
|
|
2026-02-16 10:01:42
|
<icinga-wm>
|
PROBLEM - check if authdns-update was run after a change was merged to operations/dns.git on dns6002 is CRITICAL: Local zone files are NOT in sync with operations/dns.git (SHA: local is 8bbb3a70bd5909305fc75bf31af735281b6941a8, dns.git is 48db02e01b8393b9917cf95ff96c6db087401ddc) https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run
|
|
2026-02-16 10:01:42
|
<icinga-wm>
|
PROBLEM - check if authdns-update was run after a change was merged to operations/dns.git on dns7002 is CRITICAL: Local zone files are NOT in sync with operations/dns.git (SHA: local is 8bbb3a70bd5909305fc75bf31af735281b6941a8, dns.git is 48db02e01b8393b9917cf95ff96c6db087401ddc) https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run
|
|
2026-02-16 10:01:44
|
<icinga-wm>
|
PROBLEM - check if authdns-update was run after a change was merged to operations/dns.git on dns5003 is CRITICAL: Local zone files are NOT in sync with operations/dns.git (SHA: local is 8bbb3a70bd5909305fc75bf31af735281b6941a8, dns.git is 48db02e01b8393b9917cf95ff96c6db087401ddc) https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run
|
|
2026-02-16 10:01:44
|
<icinga-wm>
|
PROBLEM - check if authdns-update was run after a change was merged to operations/dns.git on dns5004 is CRITICAL: Local zone files are NOT in sync with operations/dns.git (SHA: local is 8bbb3a70bd5909305fc75bf31af735281b6941a8, dns.git is 48db02e01b8393b9917cf95ff96c6db087401ddc) https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run
|
|
2026-02-16 10:01:56
|
<icinga-wm>
|
PROBLEM - check if authdns-update was run after a change was merged to operations/dns.git on dns1004 is CRITICAL: Local zone files are NOT in sync with operations/dns.git (SHA: local is 8bbb3a70bd5909305fc75bf31af735281b6941a8, dns.git is 48db02e01b8393b9917cf95ff96c6db087401ddc) https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run
|
|
2026-02-16 10:03:38
|
<logmsgbot>
|
!log arnaudb@cumin1003 END (PASS) - Cookbook sre.gerrit.sync-instances (exit_code=0) sync Gerrit data from gerrit1003.wikimedia.org to gerrit2003.wikimedia.org
|
|
2026-02-16 10:04:06
|
<icinga-wm>
|
PROBLEM - check if authdns-update was run after a change was merged to operations/dns.git on dns7001 is CRITICAL: Local zone files are NOT in sync with operations/dns.git (SHA: local is 8bbb3a70bd5909305fc75bf31af735281b6941a8, dns.git is 48db02e01b8393b9917cf95ff96c6db087401ddc) https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run
|
|
2026-02-16 10:06:38
|
<logmsgbot>
|
arnaudb@cumin1003 failover (PID 380224) is awaiting input
|
|
2026-02-16 10:06:47
|
<arnaudb>
|
yep ↑
|
|
2026-02-16 10:06:55
|
<arnaudb>
|
we're in progress
|
|
2026-02-16 10:07:15
|
<vgutierrez>
|
:)
|
|
2026-02-16 10:08:58
|
<logmsgbot>
|
!log arnaudb@dns1004 END - running authdns-update
|
|
2026-02-16 10:09:01
|
<logmsgbot>
|
!log arnaudb@cumin1003 START - Cookbook sre.dns.wipe-cache gerrit.wikimedia.org gerrit-replica.wikimedia.org gerrit.discovery.wmnet on all recursors
|
|
2026-02-16 10:09:05
|
<icinga-wm>
|
RECOVERY - check if authdns-update was run after a change was merged to operations/dns.git on dns7001 is OK: Local zone files and operations/dns.git are in sync https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run
|
|
2026-02-16 10:09:05
|
<logmsgbot>
|
!log arnaudb@cumin1003 END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) gerrit.wikimedia.org gerrit-replica.wikimedia.org gerrit.discovery.wmnet on all recursors
|
|
2026-02-16 10:10:46
|
<jinxer-wm>
|
FIRING: [3x] GerritHAProxyBackendUnavailable: Gerrit backend is unavilable for tcp-proxy (HAProxy) gerrit_ssh - https://wikitech.wikimedia.org/wiki/Gerrit/Operations#GerritHAProxyBackendUnavailable - grafana.wikimedia.org/d/459365f6-df37-48d6-8142-82b22c1875e7/gerrit-tcp-proxy?viewPanel=panel-15 - https://alerts.wikimedia.org/?q=alertname%3DGerritHAProxyBackendUnavailable
|
|
2026-02-16 10:11:33
|
<icinga-wm>
|
RECOVERY - check if authdns-update was run after a change was merged to operations/dns.git on dns1006 is OK: Local zone files and operations/dns.git are in sync https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run
|
|
2026-02-16 10:11:33
|
<icinga-wm>
|
RECOVERY - check if authdns-update was run after a change was merged to operations/dns.git on dns1005 is OK: Local zone files and operations/dns.git are in sync https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run
|
|
2026-02-16 10:11:33
|
<icinga-wm>
|
RECOVERY - check if authdns-update was run after a change was merged to operations/dns.git on dns2004 is OK: Local zone files and operations/dns.git are in sync https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run
|
|
2026-02-16 10:11:33
|
<icinga-wm>
|
RECOVERY - check if authdns-update was run after a change was merged to operations/dns.git on dns2005 is OK: Local zone files and operations/dns.git are in sync https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run
|
|
2026-02-16 10:11:39
|
<icinga-wm>
|
RECOVERY - check if authdns-update was run after a change was merged to operations/dns.git on dns2006 is OK: Local zone files and operations/dns.git are in sync https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run
|
|
2026-02-16 10:11:39
|
<icinga-wm>
|
RECOVERY - check if authdns-update was run after a change was merged to operations/dns.git on dns3003 is OK: Local zone files and operations/dns.git are in sync https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run
|
|
2026-02-16 10:11:39
|
<icinga-wm>
|
RECOVERY - check if authdns-update was run after a change was merged to operations/dns.git on dns3004 is OK: Local zone files and operations/dns.git are in sync https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run
|
|
2026-02-16 10:11:41
|
<icinga-wm>
|
RECOVERY - check if authdns-update was run after a change was merged to operations/dns.git on dns6001 is OK: Local zone files and operations/dns.git are in sync https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run
|
|
2026-02-16 10:11:41
|
<icinga-wm>
|
RECOVERY - check if authdns-update was run after a change was merged to operations/dns.git on dns6002 is OK: Local zone files and operations/dns.git are in sync https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run
|
|
2026-02-16 10:11:41
|
<icinga-wm>
|
RECOVERY - check if authdns-update was run after a change was merged to operations/dns.git on dns4003 is OK: Local zone files and operations/dns.git are in sync https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run
|
|
2026-02-16 10:11:41
|
<icinga-wm>
|
RECOVERY - check if authdns-update was run after a change was merged to operations/dns.git on dns4004 is OK: Local zone files and operations/dns.git are in sync https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run
|
|
2026-02-16 10:11:41
|
<icinga-wm>
|
RECOVERY - check if authdns-update was run after a change was merged to operations/dns.git on dns7002 is OK: Local zone files and operations/dns.git are in sync https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run
|
|
2026-02-16 10:11:43
|
<icinga-wm>
|
RECOVERY - check if authdns-update was run after a change was merged to operations/dns.git on dns5004 is OK: Local zone files and operations/dns.git are in sync https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run
|
|
2026-02-16 10:11:43
|
<icinga-wm>
|
RECOVERY - check if authdns-update was run after a change was merged to operations/dns.git on dns5003 is OK: Local zone files and operations/dns.git are in sync https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run
|
|
2026-02-16 10:11:46
|
<jinxer-wm>
|
FIRING: [7x] GerritHAProxyServiceUnavailable: Gerrit tcp-proxy (HAProxy) service gerrit_ssh is DOWN in codfw - https://wikitech.wikimedia.org/wiki/Gerrit/Operations#GerritHAProxyServiceUnavailable - grafana.wikimedia.org/d/459365f6-df37-48d6-8142-82b22c1875e7/gerrit-tcp-proxy?viewPanel=panel-15 - https://alerts.wikimedia.org/?q=alertname%3DGerritHAProxyServiceUnavailable
|
|
2026-02-16 10:11:55
|
<icinga-wm>
|
RECOVERY - check if authdns-update was run after a change was merged to operations/dns.git on dns1004 is OK: Local zone files and operations/dns.git are in sync https://wikitech.wikimedia.org/wiki/DNS%23authdns_update_run
|
|
2026-02-16 10:11:56
|
<hashar>
|
^ those are due to Gerrit
|
|
2026-02-16 10:13:47
|
<logmsgbot>
|
!log arnaudb@cumin1003 START - Cookbook sre.gerrit.topology-check Validate Gerrit topology (source=gerrit2003, replica=gerrit1003)
|
|
2026-02-16 10:13:52
|
<logmsgbot>
|
!log arnaudb@cumin1003 END (PASS) - Cookbook sre.gerrit.topology-check (exit_code=0) Validate Gerrit topology (source=gerrit2003, replica=gerrit1003)
|
|
2026-02-16 10:15:46
|
<jinxer-wm>
|
RESOLVED: [14x] GerritHAProxyBackendUnavailable: Gerrit backend is unavilable for tcp-proxy (HAProxy) gerrit_ssh - https://wikitech.wikimedia.org/wiki/Gerrit/Operations#GerritHAProxyBackendUnavailable - grafana.wikimedia.org/d/459365f6-df37-48d6-8142-82b22c1875e7/gerrit-tcp-proxy?viewPanel=panel-15 - https://alerts.wikimedia.org/?q=alertname%3DGerritHAProxyBackendUnavailable
|
|
2026-02-16 10:15:59
|
<logmsgbot>
|
!log arnaudb@cumin1003 START - Cookbook sre.gerrit.read-only-toggle from gerrit2003.wikimedia.org
|
|
2026-02-16 10:16:11
|
<logmsgbot>
|
!log arnaudb@cumin1003 END (PASS) - Cookbook sre.gerrit.read-only-toggle (exit_code=0) from gerrit2003.wikimedia.org
|
|
2026-02-16 10:16:11
|
<logmsgbot>
|
!log arnaudb@cumin1003 START - Cookbook sre.gerrit.read-only-toggle from gerrit1003.wikimedia.org
|
|
2026-02-16 10:16:16
|
<logmsgbot>
|
!log arnaudb@cumin1003 END (PASS) - Cookbook sre.gerrit.read-only-toggle (exit_code=0) from gerrit1003.wikimedia.org
|
|
2026-02-16 10:16:46
|
<jinxer-wm>
|
RESOLVED: [7x] GerritHAProxyServiceUnavailable: Gerrit tcp-proxy (HAProxy) service gerrit_ssh is DOWN in codfw - https://wikitech.wikimedia.org/wiki/Gerrit/Operations#GerritHAProxyServiceUnavailable - grafana.wikimedia.org/d/459365f6-df37-48d6-8142-82b22c1875e7/gerrit-tcp-proxy?viewPanel=panel-15 - https://alerts.wikimedia.org/?q=alertname%3DGerritHAProxyServiceUnavailable
|
|
2026-02-16 10:17:44
|
<logmsgbot>
|
!log arnaudb@cumin1003 END (PASS) - Cookbook sre.gerrit.failover (exit_code=0) from gerrit1003.wikimedia.org to gerrit2003.wikimedia.org
|
|
2026-02-16 10:19:24
|
<wikibugs>
|
('CR) ''DCausse: opensearch-cluster: allow the definition of custom network policies (''1 comment) [deployment-charts] - ''https://gerrit.wikimedia.org/r/1238298 (https://phabricator.wikimedia.org/T414095) (owner: ''Brouberol)'
|
|
2026-02-16 10:20:23
|
<wikibugs>
|
'SRE, ''SRE-Access-Requests, ''Data-Platform-SRE: Requesting access to "Community Wishlist" dashboard for hmonroy on Superset - https://phabricator.wikimedia.org/T416721#11619620 (''MatthewVernon) @Gehel can I ping you for your thoughts on this ticket, please? It's a request to grant someone co-owner access...'
|
|
2026-02-16 10:20:26
|
<logmsgbot>
|
!log sfaci@deploy2002 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/test-kitchen-next: apply
|
|
2026-02-16 10:20:58
|
<logmsgbot>
|
!log sfaci@deploy2002 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/test-kitchen-next: apply
|
|
2026-02-16 10:21:01
|
<wikibugs>
|
('PS1) ''JMeybohm: Update/fix comment block for ipblock_source creation [software/hiddenparma/deploy] - ''https://gerrit.wikimedia.org/r/1239626 (https://phabricator.wikimedia.org/T412805)'
|
|
2026-02-16 10:21:23
|
<wikibugs>
|
('PS7) ''Arnaudb: gerrit: re-enable backups and monitoring on gerrit2003 [puppet] - ''https://gerrit.wikimedia.org/r/1217134 (https://phabricator.wikimedia.org/T387833)'
|
|
2026-02-16 10:21:48
|
<wikibugs>
|
('PS1) ''Jelto: admin:data: add backup yubikey for jelto [puppet] - ''https://gerrit.wikimedia.org/r/1239628'
|
|
2026-02-16 10:22:17
|
<wikibugs>
|
('CR) ''JMeybohm: [V:''+2 C:''+2] Update/fix comment block for ipblock_source creation [software/hiddenparma/deploy] - ''https://gerrit.wikimedia.org/r/1239626 (https://phabricator.wikimedia.org/T412805) (owner: ''JMeybohm)'
|
|
2026-02-16 10:23:33
|
<logmsgbot>
|
!log jayme@cumin1003 START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "[not really into teleological thinking] - jayme@cumin1003 - T412805"
|
|
2026-02-16 10:23:34
|
<logmsgbot>
|
!log jayme@cumin1003 START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: [not really into teleological thinking] - jayme@cumin1003 - T412805
|
|
2026-02-16 10:23:36
|
<stashbot>
|
T412805: Migrate ipblocks from fetch_external_clouds_vendors_nets.py to HIDDENPARMA - https://phabricator.wikimedia.org/T412805
|
|
2026-02-16 10:23:57
|
<wikibugs>
|
('CR) ''Arnaudb: [C:''+2] gerrit: re-enable backups and monitoring on gerrit2003 [puppet] - ''https://gerrit.wikimedia.org/r/1217134 (https://phabricator.wikimedia.org/T387833) (owner: ''Arnaudb)'
|
|
2026-02-16 10:24:23
|
<logmsgbot>
|
!log jayme@cumin1003 END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: [not really into teleological thinking] - jayme@cumin1003 - T412805
|
|
2026-02-16 10:24:25
|
<logmsgbot>
|
!log jayme@cumin1003 END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: "[not really into teleological thinking] - jayme@cumin1003 - T412805"
|
|
2026-02-16 10:24:34
|
<wikibugs>
|
'SRE, ''ServiceOps new, ''Patch-For-Review: Migrate ipblocks from fetch_external_clouds_vendors_nets.py to HIDDENPARMA - https://phabricator.wikimedia.org/T412805#11619634 (''ops-monitoring-bot) Deployed hiddenparma to alert[1002,2002].wikimedia.org with reason: [not really into teleological thinking] - jay...'
|
|
2026-02-16 10:25:39
|
<logmsgbot>
|
!log jayme@cumin1003 START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "[not really into teleological thinking] - jayme@cumin1003 - T412805"
|
|
2026-02-16 10:25:41
|
<logmsgbot>
|
!log jayme@cumin1003 START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: [not really into teleological thinking] - jayme@cumin1003 - T412805
|
|
2026-02-16 10:26:33
|
<logmsgbot>
|
!log jayme@cumin1003 END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: [not really into teleological thinking] - jayme@cumin1003 - T412805
|
|
2026-02-16 10:26:35
|
<logmsgbot>
|
!log jayme@cumin1003 END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: "[not really into teleological thinking] - jayme@cumin1003 - T412805"
|
|
2026-02-16 10:26:47
|
<wikibugs>
|
'SRE, ''ServiceOps new, ''Patch-For-Review: Migrate ipblocks from fetch_external_clouds_vendors_nets.py to HIDDENPARMA - https://phabricator.wikimedia.org/T412805#11619641 (''ops-monitoring-bot) Deployed hiddenparma to alert[1002,2002].wikimedia.org with reason: [not really into teleological thinking] - jay...'
|
|
2026-02-16 10:27:03
|
<wikibugs>
|
('PS2) ''JMeybohm: hiddenparma: Temporarily disable ipblock_source_no_ipblock_exists policy [puppet] - ''https://gerrit.wikimedia.org/r/1223649 (https://phabricator.wikimedia.org/T412805)'
|
|
2026-02-16 10:28:36
|
<wikibugs>
|
'SRE, ''Infrastructure-Foundations: offboarding Alex Kosiaris - https://phabricator.wikimedia.org/T417465#11619644 (''MoritzMuehlenhoff)'
|
|
2026-02-16 10:33:17
|
<wikibugs>
|
('PS1) ''Kevin Bazira: ml-services: update rr-wikidata prod image [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239630 (https://phabricator.wikimedia.org/T414060)'
|
|
2026-02-16 10:34:22
|
<wikibugs>
|
('PS1) ''Vgutierrez: cache::dp_key: Fix cross-filesystems tempfile error on Trixie [puppet] - ''https://gerrit.wikimedia.org/r/1239631 (https://phabricator.wikimedia.org/T401832)'
|
|
2026-02-16 10:35:01
|
<wikibugs>
|
('CR) ''Vgutierrez: "check experimental" [puppet] - ''https://gerrit.wikimedia.org/r/1239631 (https://phabricator.wikimedia.org/T401832) (owner: ''Vgutierrez)'
|
|
2026-02-16 10:37:44
|
<wikibugs>
|
('CR) ''Slyngshede: [C:''+1] "LGTM" [puppet] - ''https://gerrit.wikimedia.org/r/1239631 (https://phabricator.wikimedia.org/T401832) (owner: ''Vgutierrez)'
|
|
2026-02-16 10:41:15
|
<wikibugs>
|
('PS3) ''DCausse: cirrus: enable default_sort for completion on a set of wikis [mediawiki-config] - ''https://gerrit.wikimedia.org/r/1207758 (https://phabricator.wikimedia.org/T404858)'
|
|
2026-02-16 10:45:00
|
<icinga-wm>
|
PROBLEM - OSPF status on cr2-eqdfw is CRITICAL: OSPFv2: 7/7 UP : OSPFv3: 6/7 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status
|
|
2026-02-16 10:46:00
|
<icinga-wm>
|
RECOVERY - OSPF status on cr2-eqdfw is OK: OSPFv2: 7/7 UP : OSPFv3: 7/7 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status
|
|
2026-02-16 10:50:01
|
<logmsgbot>
|
!log jmm@cumin2002 START - Cookbook sre.hosts.decommission for hosts puppetmaster1001.eqiad.wmnet
|
|
2026-02-16 10:56:02
|
<logmsgbot>
|
!log jmm@cumin2002 START - Cookbook sre.dns.netbox
|
|
2026-02-16 10:58:10
|
<jinxer-wm>
|
FIRING: [3x] BFDdown: BFD session down between cr2-eqdfw and fe80::7a4f:9b00:174e:7c0c - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown
|
|
2026-02-16 11:00:05
|
<jouncebot>
|
Deploy window MediaWiki infrastructure (UTC mid-day) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260216T1100)
|
|
2026-02-16 11:01:12
|
<logmsgbot>
|
!log jmm@cumin2002 END (FAIL) - Cookbook sre.dns.netbox (exit_code=94)
|
|
2026-02-16 11:01:14
|
<logmsgbot>
|
!log jmm@cumin2002 END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts puppetmaster1001.eqiad.wmnet
|
|
2026-02-16 11:01:28
|
<wikibugs>
|
'SRE, ''Infrastructure-Foundations, ''Puppet-Infrastructure, ''Patch-For-Review: Shutdown of Puppet 5 servers - https://phabricator.wikimedia.org/T365798#11619780 (''ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by jmm@cumin2002 for hosts: `puppetmaster1001.eqiad.wmnet` - puppetmaster1001....'
|
|
2026-02-16 11:02:41
|
<wikibugs>
|
('CR) ''Vgutierrez: [C:''+2] cache::dp_key: Fix cross-filesystems tempfile error on Trixie [puppet] - ''https://gerrit.wikimedia.org/r/1239631 (https://phabricator.wikimedia.org/T401832) (owner: ''Vgutierrez)'
|
|
2026-02-16 11:03:10
|
<jinxer-wm>
|
RESOLVED: [4x] BFDdown: BFD session down between cr2-drmrs and fe80::5e5e:ab00:103d:83c7 - https://wikitech.wikimedia.org/wiki/Network_monitoring#BFD_status - https://alerts.wikimedia.org/?q=alertname%3DBFDdown
|
|
2026-02-16 11:03:12
|
<icinga-wm>
|
PROBLEM - Backup freshness on backup1014 is CRITICAL: Stale: 2 (gerrit2003, ...), Fresh: 138 jobs https://wikitech.wikimedia.org/wiki/Bacula%23Monitoring
|
|
2026-02-16 11:04:03
|
<hashar>
|
error: RPC failed; HTTP 502 curl 22 The requested URL returned error: 502
|
|
2026-02-16 11:04:03
|
<hashar>
|
fatal: expected flush after ref listing
|
|
2026-02-16 11:04:03
|
<hashar>
|
fatal: clone of 'https://gerrit.wikimedia.org/r/mediawiki/extensions/AddHTMLMetaAndTitle'; into submodule path '/home/hashar/extensions/AddHTMLMetaAndTitle' failed
|
|
2026-02-16 11:04:47
|
<wikibugs>
|
('PS1) ''Muehlenhoff: sre.puppet.sync-netbox-hiera: Remove support for Puppet 5 [cookbooks] - ''https://gerrit.wikimedia.org/r/1239638 (https://phabricator.wikimedia.org/T365798)'
|
|
2026-02-16 11:09:05
|
<wikibugs>
|
('CR) ''Muehlenhoff: [C:''+1] "Looks good and verified out of band" [puppet] - ''https://gerrit.wikimedia.org/r/1239628 (owner: ''Jelto)'
|
|
2026-02-16 11:18:13
|
<jinxer-wm>
|
FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - pfw1-codfw:reth2 (fasw1-f5 2x25G) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=pfw1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown
|
|
2026-02-16 11:19:22
|
<wikibugs>
|
('CR) ''Elukey: [C:''+1] sre.puppet.sync-netbox-hiera: Remove support for Puppet 5 [cookbooks] - ''https://gerrit.wikimedia.org/r/1239638 (https://phabricator.wikimedia.org/T365798) (owner: ''Muehlenhoff)'
|
|
2026-02-16 11:23:24
|
<wikibugs>
|
('CR) ''Muehlenhoff: [C:''+2] sre.puppet.sync-netbox-hiera: Remove support for Puppet 5 [cookbooks] - ''https://gerrit.wikimedia.org/r/1239638 (https://phabricator.wikimedia.org/T365798) (owner: ''Muehlenhoff)'
|
|
2026-02-16 11:23:29
|
<wikibugs>
|
('PS4) ''Effie Mouzeli: mw-parsoid: repurpose for parsoidtest use #4 [deployment-charts] - ''https://gerrit.wikimedia.org/r/1237472 (https://phabricator.wikimedia.org/T386246)'
|
|
2026-02-16 11:25:16
|
<wikibugs>
|
('PS1) ''MVernon: swift: add 4 new eqiad frontends ms-fe102[1-4] [puppet] - ''https://gerrit.wikimedia.org/r/1239643 (https://phabricator.wikimedia.org/T416245)'
|
|
2026-02-16 11:27:16
|
<logmsgbot>
|
!log jmm@cumin2002 START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "decom puppetmaster1001 - jmm@cumin2002"
|
|
2026-02-16 11:27:32
|
<logmsgbot>
|
!log jmm@cumin2002 END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "decom puppetmaster1001 - jmm@cumin2002"
|
|
2026-02-16 11:28:22
|
<logmsgbot>
|
!log jmm@cumin2002 START - Cookbook sre.dns.netbox
|
|
2026-02-16 11:28:44
|
<wikibugs>
|
('CR) ''Daniel Kinzler: rest gateway: implement per-policy shadow mode (''1 comment) [deployment-charts] - ''https://gerrit.wikimedia.org/r/1225699 (https://phabricator.wikimedia.org/T413183) (owner: ''Daniel Kinzler)'
|
|
2026-02-16 11:29:17
|
<wikibugs>
|
('CR) ''Gkyziridis: [C:''+1] "LGTM" [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239630 (https://phabricator.wikimedia.org/T414060) (owner: ''Kevin Bazira)'
|
|
2026-02-16 11:30:29
|
<wikibugs>
|
('CR) ''Kevin Bazira: [C:''+2] ml-services: update rr-wikidata prod image [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239630 (https://phabricator.wikimedia.org/T414060) (owner: ''Kevin Bazira)'
|
|
2026-02-16 11:31:17
|
<logmsgbot>
|
!log jmm@cumin2002 END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
|
|
2026-02-16 11:32:16
|
<wikibugs>
|
('PS1) ''MVernon: installserver: new ms-fe nodes are UEFI booted [puppet] - ''https://gerrit.wikimedia.org/r/1239645 (https://phabricator.wikimedia.org/T416245)'
|
|
2026-02-16 11:32:50
|
<wikibugs>
|
('Merged) ''jenkins-bot: ml-services: update rr-wikidata prod image [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239630 (https://phabricator.wikimedia.org/T414060) (owner: ''Kevin Bazira)'
|
|
2026-02-16 11:34:31
|
<logmsgbot>
|
!log kevinbazira@deploy2002 helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
|
|
2026-02-16 11:35:02
|
<wikibugs>
|
'SRE, ''decommission-hardware: decommission puppetmaster1001 - https://phabricator.wikimedia.org/T417580 (''MoritzMuehlenhoff) ''NEW'
|
|
2026-02-16 11:35:23
|
<wikibugs>
|
'SRE, ''decommission-hardware: decommission puppetmaster1001 - https://phabricator.wikimedia.org/T417580#11619930 (''MoritzMuehlenhoff)'
|
|
2026-02-16 11:39:42
|
<wikibugs>
|
('PS2) ''Effie Mouzeli: admin_ng: add ValidatingAdmissionPolicy to mw-parsoid #1 [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239174 (https://phabricator.wikimedia.org/T386246)'
|
|
2026-02-16 11:40:26
|
<wikibugs>
|
('PS2) ''Effie Mouzeli: validating-admission-policies: add /srv/parsoid-testing #0 [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239170 (https://phabricator.wikimedia.org/T386246)'
|
|
2026-02-16 11:40:59
|
<logmsgbot>
|
!log kevinbazira@deploy2002 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
|
|
2026-02-16 11:41:01
|
<wikibugs>
|
('PS3) ''Effie Mouzeli: validating-admission-policies: add /srv/parsoid-testing (vanilla) [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239169 (https://phabricator.wikimedia.org/T386246)'
|
|
2026-02-16 11:41:59
|
<wikibugs>
|
('PS3) ''Effie Mouzeli: admin_ng: add ValidatingAdmissionPolicy for mw-parsoid #1 [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239174 (https://phabricator.wikimedia.org/T386246)'
|
|
2026-02-16 11:42:21
|
<wikibugs>
|
('PS2) ''MVernon: swift: add 4 new eqiad frontends ms-fe102[1-4] [puppet] - ''https://gerrit.wikimedia.org/r/1239643 (https://phabricator.wikimedia.org/T416245)'
|
|
2026-02-16 11:46:01
|
<wikibugs>
|
('PS1) ''Muehlenhoff: Remove puppetmaster1001 from site.pp [puppet] - ''https://gerrit.wikimedia.org/r/1239646 (https://phabricator.wikimedia.org/T417580)'
|
|
2026-02-16 11:46:03
|
<wikibugs>
|
('PS1) ''Muehlenhoff: puppetdb: Drop firewall rule for access to Puppet 5 servers [puppet] - ''https://gerrit.wikimedia.org/r/1239647 (https://phabricator.wikimedia.org/T365798)'
|
|
2026-02-16 11:47:46
|
<wikibugs>
|
('PS1) ''Muehlenhoff: Remove now obsolete Cumin aliases for Buster and Puppet 5 [puppet] - ''https://gerrit.wikimedia.org/r/1239648 (https://phabricator.wikimedia.org/T365798)'
|
|
2026-02-16 11:48:45
|
<wikibugs>
|
('CR) ''Muehlenhoff: "check experimental" [puppet] - ''https://gerrit.wikimedia.org/r/1239647 (https://phabricator.wikimedia.org/T365798) (owner: ''Muehlenhoff)'
|
|
2026-02-16 11:52:59
|
<logmsgbot>
|
!log ammarpad@deploy2002 mwscript-k8s job started: extensions/Translate/scripts/moveTranslatableBundle.php --wiki=mediawikiwiki --reason 'Requested at [[phab:T417210]]' 'Writing systems/Syntax' 'Language Converter/Advanced syntax' Ammarpad # T417210
|
|
2026-02-16 11:53:03
|
<stashbot>
|
T417210: Request to move translatable page: Writing systems/Syntax - https://phabricator.wikimedia.org/T417210
|
|
2026-02-16 12:04:23
|
<wikibugs>
|
('PS1) ''Effie Mouzeli: service.yaml: switch mw-parsoid to lvs_setup #2 [puppet] - ''https://gerrit.wikimedia.org/r/1239651 (https://phabricator.wikimedia.org/T386246)'
|
|
2026-02-16 12:05:03
|
<wikibugs>
|
('PS5) ''Effie Mouzeli: kubernetes::mediawiki_experimental: add parsoid repo #2 [puppet] - ''https://gerrit.wikimedia.org/r/1238345 (https://phabricator.wikimedia.org/T386246)'
|
|
2026-02-16 12:05:46
|
<wikibugs>
|
('PS6) ''Effie Mouzeli: kubernetes::mediawiki_experimental: add parsoid repo #3 [puppet] - ''https://gerrit.wikimedia.org/r/1238345 (https://phabricator.wikimedia.org/T386246)'
|
|
2026-02-16 12:09:43
|
<wikibugs>
|
('PS3) ''Effie Mouzeli: deployment_server: add parsoid pinkllama release #4 [puppet] - ''https://gerrit.wikimedia.org/r/1238349 (https://phabricator.wikimedia.org/T386246)'
|
|
2026-02-16 12:11:08
|
<wikibugs>
|
('PS4) ''Effie Mouzeli: mediawiki: mount parsoid-testing via hostPath #5 [deployment-charts] - ''https://gerrit.wikimedia.org/r/1238355 (https://phabricator.wikimedia.org/T386246)'
|
|
2026-02-16 12:12:23
|
<wikibugs>
|
('PS5) ''Effie Mouzeli: mw-parsoid: repurpose for parsoidtest use #6 [deployment-charts] - ''https://gerrit.wikimedia.org/r/1237472 (https://phabricator.wikimedia.org/T386246)'
|
|
2026-02-16 12:13:55
|
<wikibugs>
|
('CR) ''Effie Mouzeli: "Thank you scott! Submitted I891f08c04e0936e5203698e5622b543f2c94dfc0" [deployment-charts] - ''https://gerrit.wikimedia.org/r/1237472 (https://phabricator.wikimedia.org/T386246) (owner: ''Effie Mouzeli)'
|
|
2026-02-16 12:15:11
|
<wikibugs>
|
('CR) ''Effie Mouzeli: "thank you scott! Folks have confirmed that nothings should end up here." [puppet] - ''https://gerrit.wikimedia.org/r/1238349 (https://phabricator.wikimedia.org/T386246) (owner: ''Effie Mouzeli)'
|
|
2026-02-16 12:18:45
|
<wikibugs>
|
'SRE, ''MediaWiki-Shell, ''ServiceOps new, ''ServiceOps-Mediawiki: Support cgroup 2 - https://phabricator.wikimedia.org/T417502#11619988 (''Paladox) Seems an upgrade to cgroups v2 is a must in trixie. No way to get existing behaviour to work. >>! In T417502#11619167, @JMeybohm wrote: > Could you please e...'
|
|
2026-02-16 12:19:32
|
<jinxer-wm>
|
FIRING: [2x] ProbeDown: Service wdqs1013:443 has failed probes (http_wdqs_main_external_search_sparql_endpoint_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#wdqs1013:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
|
|
2026-02-16 12:21:24
|
<wikibugs>
|
('CR) ''Marostegui: [C:''+1] swift: add 4 new eqiad frontends ms-fe102[1-4] [puppet] - ''https://gerrit.wikimedia.org/r/1239643 (https://phabricator.wikimedia.org/T416245) (owner: ''MVernon)'
|
|
2026-02-16 12:21:42
|
<wikibugs>
|
('CR) ''Marostegui: [C:''+1] mariadb: allow dborch1002 to db1215 connection [puppet] - ''https://gerrit.wikimedia.org/r/1239606 (https://phabricator.wikimedia.org/T416582) (owner: ''Federico Ceratto)'
|
|
2026-02-16 12:23:05
|
<wikibugs>
|
('PS5) ''Federico Ceratto: mysql: simple cookbook to update replication source [cookbooks] - ''https://gerrit.wikimedia.org/r/1238368 (https://phabricator.wikimedia.org/T373436)'
|
|
2026-02-16 12:23:53
|
<wikibugs>
|
('CR) ''Federico Ceratto: [C:''+2] mariadb: allow dborch1002 to db1215 connection [puppet] - ''https://gerrit.wikimedia.org/r/1239606 (https://phabricator.wikimedia.org/T416582) (owner: ''Federico Ceratto)'
|
|
2026-02-16 12:26:10
|
<logmsgbot>
|
!log fceratto@cumin1003 START - Cookbook sre.mysql.update-replication
|
|
2026-02-16 12:26:19
|
<logmsgbot>
|
!log fceratto@cumin1003 END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0)
|
|
2026-02-16 12:27:10
|
<wikibugs>
|
('CR) ''CI reject: [V:''-1] mysql: simple cookbook to update replication source [cookbooks] - ''https://gerrit.wikimedia.org/r/1238368 (https://phabricator.wikimedia.org/T373436) (owner: ''Federico Ceratto)'
|
|
2026-02-16 12:29:47
|
<wikibugs>
|
'SRE, ''MediaWiki-Shell, ''ServiceOps new, ''ServiceOps-Mediawiki, ''Shellbox: Support cgroup 2 - https://phabricator.wikimedia.org/T417502#11619999 (''Clement_Goubert) p:''Triage→''Low'
|
|
2026-02-16 12:31:03
|
<wikibugs>
|
('PS6) ''Federico Ceratto: mysql: update replication source [cookbooks] - ''https://gerrit.wikimedia.org/r/1238368 (https://phabricator.wikimedia.org/T373436)'
|
|
2026-02-16 12:33:25
|
<wikibugs>
|
('PS12) ''Daniel Kinzler: rest gateway: implement per-policy shadow mode [deployment-charts] - ''https://gerrit.wikimedia.org/r/1225699 (https://phabricator.wikimedia.org/T413183)'
|
|
2026-02-16 12:33:36
|
<wikibugs>
|
('PS7) ''Federico Ceratto: mysql: update replication source [cookbooks] - ''https://gerrit.wikimedia.org/r/1238368 (https://phabricator.wikimedia.org/T373436)'
|
|
2026-02-16 12:34:25
|
<logmsgbot>
|
!log fceratto@cumin1003 START - Cookbook sre.mysql.update-replication
|
|
2026-02-16 12:34:32
|
<logmsgbot>
|
!log fceratto@cumin1003 END (FAIL) - Cookbook sre.mysql.update-replication (exit_code=99)
|
|
2026-02-16 12:36:06
|
<wikibugs>
|
('PS8) ''Federico Ceratto: mysql: update replication source [cookbooks] - ''https://gerrit.wikimedia.org/r/1238368 (https://phabricator.wikimedia.org/T373436)'
|
|
2026-02-16 12:36:09
|
<logmsgbot>
|
!log fceratto@cumin1003 START - Cookbook sre.mysql.update-replication
|
|
2026-02-16 12:36:18
|
<logmsgbot>
|
!log fceratto@cumin1003 END (FAIL) - Cookbook sre.mysql.update-replication (exit_code=99)
|
|
2026-02-16 12:36:30
|
<logmsgbot>
|
!log fceratto@cumin1003 START - Cookbook sre.mysql.update-replication
|
|
2026-02-16 12:36:38
|
<logmsgbot>
|
!log fceratto@cumin1003 END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0)
|
|
2026-02-16 12:36:55
|
<logmsgbot>
|
!log fceratto@cumin1003 START - Cookbook sre.mysql.update-replication
|
|
2026-02-16 12:37:04
|
<logmsgbot>
|
!log fceratto@cumin1003 END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0)
|
|
2026-02-16 12:37:23
|
<logmsgbot>
|
!log fceratto@cumin1003 START - Cookbook sre.mysql.update-replication
|
|
2026-02-16 12:37:29
|
<logmsgbot>
|
!log fceratto@cumin1003 END (FAIL) - Cookbook sre.mysql.update-replication (exit_code=99)
|
|
2026-02-16 12:38:26
|
<logmsgbot>
|
!log fceratto@cumin1003 START - Cookbook sre.mysql.update-replication
|
|
2026-02-16 12:38:34
|
<logmsgbot>
|
!log fceratto@cumin1003 END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0)
|
|
2026-02-16 12:39:00
|
<wikibugs>
|
('PS1) ''Daniel Kinzler: rest-gateway: use MINUTE limits in staging [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239669'
|
|
2026-02-16 12:39:27
|
<wikibugs>
|
('CR) ''CI reject: [V:''-1] rest-gateway: use MINUTE limits in staging [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239669 (owner: ''Daniel Kinzler)'
|
|
2026-02-16 12:39:38
|
<logmsgbot>
|
!log fceratto@cumin1003 START - Cookbook sre.mysql.update-replication
|
|
2026-02-16 12:39:46
|
<logmsgbot>
|
!log fceratto@cumin1003 END (FAIL) - Cookbook sre.mysql.update-replication (exit_code=99)
|
|
2026-02-16 12:40:13
|
<logmsgbot>
|
!log fceratto@cumin1003 START - Cookbook sre.mysql.update-replication
|
|
2026-02-16 12:40:20
|
<logmsgbot>
|
!log fceratto@cumin1003 END (FAIL) - Cookbook sre.mysql.update-replication (exit_code=99)
|
|
2026-02-16 12:41:17
|
<logmsgbot>
|
!log fceratto@cumin1003 START - Cookbook sre.mysql.update-replication
|
|
2026-02-16 12:41:24
|
<logmsgbot>
|
!log fceratto@cumin1003 END (FAIL) - Cookbook sre.mysql.update-replication (exit_code=99)
|
|
2026-02-16 12:41:29
|
<wikibugs>
|
('CR) ''CI reject: [V:''-1] mysql: update replication source [cookbooks] - ''https://gerrit.wikimedia.org/r/1238368 (https://phabricator.wikimedia.org/T373436) (owner: ''Federico Ceratto)'
|
|
2026-02-16 12:42:47
|
<logmsgbot>
|
!log fceratto@cumin1003 START - Cookbook sre.mysql.update-replication
|
|
2026-02-16 12:42:54
|
<logmsgbot>
|
!log fceratto@cumin1003 END (FAIL) - Cookbook sre.mysql.update-replication (exit_code=99)
|
|
2026-02-16 12:43:33
|
<wikibugs>
|
('PS1) ''Brouberol: airflow-research: revert to storing all XCOMs in DB [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239670 (https://phabricator.wikimedia.org/T417190)'
|
|
2026-02-16 12:43:50
|
<logmsgbot>
|
!log fceratto@cumin1003 START - Cookbook sre.mysql.update-replication
|
|
2026-02-16 12:43:57
|
<logmsgbot>
|
!log fceratto@cumin1003 END (FAIL) - Cookbook sre.mysql.update-replication (exit_code=99)
|
|
2026-02-16 12:44:32
|
<logmsgbot>
|
!log fceratto@cumin1003 START - Cookbook sre.mysql.update-replication
|
|
2026-02-16 12:44:39
|
<logmsgbot>
|
!log fceratto@cumin1003 END (FAIL) - Cookbook sre.mysql.update-replication (exit_code=99)
|
|
2026-02-16 12:46:10
|
<wikibugs>
|
('PS9) ''Federico Ceratto: mysql: update replication source [cookbooks] - ''https://gerrit.wikimedia.org/r/1238368 (https://phabricator.wikimedia.org/T373436)'
|
|
2026-02-16 12:46:46
|
<logmsgbot>
|
!log fceratto@cumin1003 START - Cookbook sre.mysql.update-replication
|
|
2026-02-16 12:46:54
|
<logmsgbot>
|
!log fceratto@cumin1003 END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0)
|
|
2026-02-16 12:48:34
|
<wikibugs>
|
('PS1) ''Marostegui: wmnet: Failover m3-master [dns] - ''https://gerrit.wikimedia.org/r/1239671 (https://phabricator.wikimedia.org/T414656)'
|
|
2026-02-16 12:50:20
|
<wikibugs>
|
('PS1) ''Phuedx: Test Kitchen: Set event intake service name [mediawiki-config] - ''https://gerrit.wikimedia.org/r/1239672'
|
|
2026-02-16 12:51:15
|
<wikibugs>
|
('CR) ''CI reject: [V:''-1] mysql: update replication source [cookbooks] - ''https://gerrit.wikimedia.org/r/1238368 (https://phabricator.wikimedia.org/T373436) (owner: ''Federico Ceratto)'
|
|
2026-02-16 12:57:27
|
<wikibugs>
|
('PS1) ''Muehlenhoff: Remove buster_ssh_keys [puppet] - ''https://gerrit.wikimedia.org/r/1239673'
|
|
2026-02-16 13:02:59
|
<wikibugs>
|
('CR) ''Muehlenhoff: [C:''+2] Remove puppetmaster1001 from site.pp [puppet] - ''https://gerrit.wikimedia.org/r/1239646 (https://phabricator.wikimedia.org/T417580) (owner: ''Muehlenhoff)'
|
|
2026-02-16 13:07:51
|
<wikibugs>
|
('CR) ''Federico Ceratto: "Updated based on feedback" [cookbooks] - ''https://gerrit.wikimedia.org/r/1238368 (https://phabricator.wikimedia.org/T373436) (owner: ''Federico Ceratto)'
|
|
2026-02-16 13:08:50
|
<wikibugs>
|
('PS1) ''Ayounsi: AM: only send critical I/F alerts to the I/F IRC chan [puppet] - ''https://gerrit.wikimedia.org/r/1239674'
|
|
2026-02-16 13:10:47
|
<wikibugs>
|
('CR) ''MVernon: [C:''+2] swift: add 4 new eqiad frontends ms-fe102[1-4] [puppet] - ''https://gerrit.wikimedia.org/r/1239643 (https://phabricator.wikimedia.org/T416245) (owner: ''MVernon)'
|
|
2026-02-16 13:11:50
|
<wikibugs>
|
('CR) ''Jelto: [C:''+2] admin:data: add backup yubikey for jelto [puppet] - ''https://gerrit.wikimedia.org/r/1239628 (owner: ''Jelto)'
|
|
2026-02-16 13:15:41
|
<icinga-wm>
|
RECOVERY - Swift https backend on ms-fe1024 is OK: HTTP OK: HTTP/1.1 200 OK - 503 bytes in 7.193 second response time https://wikitech.wikimedia.org/wiki/Swift
|
|
2026-02-16 13:16:18
|
<wikibugs>
|
('CR) ''FNegri: [C:''+1] Remove buster_ssh_keys [puppet] - ''https://gerrit.wikimedia.org/r/1239673 (owner: ''Muehlenhoff)'
|
|
2026-02-16 13:16:37
|
<icinga-wm>
|
RECOVERY - Swift https backend on ms-fe1021 is OK: HTTP OK: HTTP/1.1 200 OK - 503 bytes in 3.383 second response time https://wikitech.wikimedia.org/wiki/Swift
|
|
2026-02-16 13:16:49
|
<icinga-wm>
|
RECOVERY - Swift https backend on ms-fe1022 is OK: HTTP OK: HTTP/1.1 200 OK - 501 bytes in 0.092 second response time https://wikitech.wikimedia.org/wiki/Swift
|
|
2026-02-16 13:17:59
|
<icinga-wm>
|
PROBLEM - Host ms-fe1021 is DOWN: PING CRITICAL - Packet loss = 100%
|
|
2026-02-16 13:18:13
|
<icinga-wm>
|
PROBLEM - Host ms-fe1024 is DOWN: PING CRITICAL - Packet loss = 100%
|
|
2026-02-16 13:18:21
|
<icinga-wm>
|
RECOVERY - orchestrator.wikimedia.org requires authentication on dborch1002 is OK: HTTP OK: Status line output matched HTTP/1.1 302 - 636 bytes in 0.018 second response time https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration
|
|
2026-02-16 13:18:21
|
<icinga-wm>
|
RECOVERY - orchestrator.wikimedia.org tls expiry on dborch1002 is OK: OK - Certificate orchestrator.wikimedia.org will expire on Sun 05 Apr 2026 07:22:46 AM GMT +0000. https://wikitech.wikimedia.org/wiki/CAS-SSO/Administration
|
|
2026-02-16 13:18:55
|
<icinga-wm>
|
PROBLEM - Host ms-fe1022 is DOWN: PING CRITICAL - Packet loss = 100%
|
|
2026-02-16 13:19:11
|
<icinga-wm>
|
PROBLEM - Host ms-fe1023 is DOWN: PING CRITICAL - Packet loss = 100%
|
|
2026-02-16 13:19:35
|
<icinga-wm>
|
RECOVERY - Swift https backend on ms-fe1023 is OK: HTTP OK: HTTP/1.1 200 OK - 502 bytes in 0.250 second response time https://wikitech.wikimedia.org/wiki/Swift
|
|
2026-02-16 13:19:37
|
<icinga-wm>
|
RECOVERY - Host ms-fe1024 is UP: PING OK - Packet loss = 0%, RTA = 6.54 ms
|
|
2026-02-16 13:19:37
|
<icinga-wm>
|
RECOVERY - Host ms-fe1023 is UP: PING OK - Packet loss = 0%, RTA = 6.85 ms
|
|
2026-02-16 13:19:37
|
<icinga-wm>
|
RECOVERY - Host ms-fe1021 is UP: PING OK - Packet loss = 0%, RTA = 0.30 ms
|
|
2026-02-16 13:19:51
|
<icinga-wm>
|
RECOVERY - Host ms-fe1022 is UP: PING OK - Packet loss = 0%, RTA = 0.26 ms
|
|
2026-02-16 13:20:00
|
<wikibugs>
|
('PS1) ''Muehlenhoff: Remove puppetmaster::frontend role [puppet] - ''https://gerrit.wikimedia.org/r/1239676 (https://phabricator.wikimedia.org/T365798)'
|
|
2026-02-16 13:20:41
|
<wikibugs>
|
('PS2) ''Muehlenhoff: Remove puppetmaster::frontend role [puppet] - ''https://gerrit.wikimedia.org/r/1239676 (https://phabricator.wikimedia.org/T365798)'
|
|
2026-02-16 13:22:07
|
<wikibugs>
|
('CR) ''JMeybohm: [C:''+2] Remove to be migrated ipblock sources fetch_external_*_nets.py [puppet] - ''https://gerrit.wikimedia.org/r/1223648 (https://phabricator.wikimedia.org/T412805) (owner: ''JMeybohm)'
|
|
2026-02-16 13:22:14
|
<wikibugs>
|
('CR) ''JMeybohm: [C:''+2] hiddenparma: Temporarily disable ipblock_source_no_ipblock_exists policy [puppet] - ''https://gerrit.wikimedia.org/r/1223649 (https://phabricator.wikimedia.org/T412805) (owner: ''JMeybohm)'
|
|
2026-02-16 13:23:20
|
<wikibugs>
|
('CR) ''Kamila Součková: [C:''+1] "LGTM from an SRE viewpoint, I left a few nits on the implementation but I don't insist on those." [deployment-charts] - ''https://gerrit.wikimedia.org/r/1225085 (owner: ''Daniel Kinzler)'
|
|
2026-02-16 13:23:33
|
<wikibugs>
|
('PS1) ''Volans: Drop support for Python 3.7 and 3.8 [software/pywmflib] - ''https://gerrit.wikimedia.org/r/1239678'
|
|
2026-02-16 13:23:33
|
<wikibugs>
|
('PS1) ''Volans: tests: remove fixture require_caplog [software/pywmflib] - ''https://gerrit.wikimedia.org/r/1239679'
|
|
2026-02-16 13:23:33
|
<wikibugs>
|
('PS1) ''Volans: type hints: use standard types as type hints [software/pywmflib] - ''https://gerrit.wikimedia.org/r/1239680'
|
|
2026-02-16 13:24:20
|
<wikibugs>
|
('PS5) ''Effie Mouzeli: mediawiki: mount parsoid-testing via hostPath #5 [deployment-charts] - ''https://gerrit.wikimedia.org/r/1238355 (https://phabricator.wikimedia.org/T386246)'
|
|
2026-02-16 13:26:17
|
<wikibugs>
|
('PS6) ''Effie Mouzeli: mediawiki: mount parsoid-testing via hostPath #5 [deployment-charts] - ''https://gerrit.wikimedia.org/r/1238355 (https://phabricator.wikimedia.org/T386246)'
|
|
2026-02-16 13:28:48
|
<wikibugs>
|
('CR) ''CI reject: [V:''-1] mediawiki: mount parsoid-testing via hostPath #5 [deployment-charts] - ''https://gerrit.wikimedia.org/r/1238355 (https://phabricator.wikimedia.org/T386246) (owner: ''Effie Mouzeli)'
|
|
2026-02-16 13:31:14
|
<wikibugs>
|
('PS7) ''Effie Mouzeli: mediawiki: mount parsoid-testing via hostPath #5 [deployment-charts] - ''https://gerrit.wikimedia.org/r/1238355 (https://phabricator.wikimedia.org/T386246)'
|
|
2026-02-16 13:35:59
|
<logmsgbot>
|
!log mvernon@cumin2002 START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on P{ms-fe[1009-1020].eqiad.wmnet} and (A:swift-fe or A:swift-fe-canary or A:swift-fe-codfw or A:swift-fe-eqiad)
|
|
2026-02-16 13:36:25
|
<wikibugs>
|
('CR) ''JMeybohm: "Totally! We'd have to have two separate package_from_component resources so one can be absent. The tricky part here is to avoid resource n" [puppet] - ''https://gerrit.wikimedia.org/r/1239135 (https://phabricator.wikimedia.org/T416670) (owner: ''Elukey)'
|
|
2026-02-16 13:38:26
|
<wikibugs>
|
('PS8) ''Effie Mouzeli: mediawiki: mount parsoid-testing via hostPath #5 [deployment-charts] - ''https://gerrit.wikimedia.org/r/1238355 (https://phabricator.wikimedia.org/T386246)'
|
|
2026-02-16 13:38:38
|
<wikibugs>
|
('PS1) ''Brouberol: chart: add maintenance metadata to all dpe charts [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239681 (https://phabricator.wikimedia.org/T416712)'
|
|
2026-02-16 13:39:07
|
<wikibugs>
|
('CR) ''Effie Mouzeli: mediawiki: mount parsoid-testing via hostPath #5 (''1 comment) [deployment-charts] - ''https://gerrit.wikimedia.org/r/1238355 (https://phabricator.wikimedia.org/T386246) (owner: ''Effie Mouzeli)'
|
|
2026-02-16 13:39:28
|
<wikibugs>
|
('PS4) ''Effie Mouzeli: validating-admission-policies: add /srv/parsoid-testing (vanilla) [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239169 (https://phabricator.wikimedia.org/T386246)'
|
|
2026-02-16 13:39:43
|
<wikibugs>
|
('PS3) ''Effie Mouzeli: validating-admission-policies: add /srv/parsoid-testing #0 [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239170 (https://phabricator.wikimedia.org/T386246)'
|
|
2026-02-16 13:42:05
|
<Amir1>
|
!log ladsgroup@deploy2002:~$ mwscript-k8s --dblist=all -- purgeUserOptions.php --login-age 5 echo-subscriptions-email-edit-thank
|
|
2026-02-16 13:42:07
|
<stashbot>
|
Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
|
|
2026-02-16 13:42:55
|
<logmsgbot>
|
!log mvernon@cumin2002 END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on P{ms-fe[1009-1020].eqiad.wmnet} and (A:swift-fe or A:swift-fe-canary or A:swift-fe-codfw or A:swift-fe-eqiad)
|
|
2026-02-16 13:46:11
|
<wikibugs>
|
('PS3) ''Elukey: profile::kafka::broker: support new confluent distributions [puppet] - ''https://gerrit.wikimedia.org/r/1239135 (https://phabricator.wikimedia.org/T416670)'
|
|
2026-02-16 13:48:26
|
<wikibugs>
|
('CR) ''JMeybohm: [C:''+2] Revert "hiddenparma: Temporarily disable ipblock_source_no_ipblock_exists policy" [puppet] - ''https://gerrit.wikimedia.org/r/1223650 (https://phabricator.wikimedia.org/T412805) (owner: ''JMeybohm)'
|
|
2026-02-16 13:50:18
|
<wikibugs>
|
'SRE, ''ServiceOps new, ''Patch-For-Review: Migrate ipblocks from fetch_external_clouds_vendors_nets.py to HIDDENPARMA - https://phabricator.wikimedia.org/T412805#11620132 (''JMeybohm) ''Open→''Resolved All compatible ipblock sources have
been migrated to hiddenparma'
|
|
2026-02-16 13:52:25
|
<jayme>
|
!log All compatible ipblock sources have been migrated from fetch_external_clouds_vendors_nets.py to hiddenparma - T412805
|
|
2026-02-16 13:52:29
|
<stashbot>
|
Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
|
|
2026-02-16 13:52:29
|
<stashbot>
|
T412805: Migrate ipblocks from fetch_external_clouds_vendors_nets.py to HIDDENPARMA - https://phabricator.wikimedia.org/T412805
|
|
2026-02-16 13:56:58
|
<wikibugs>
|
('PS1) ''Kevin Bazira: ml-services: bump up rr-wikidata workers [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239685 (https://phabricator.wikimedia.org/T414060)'
|
|
2026-02-16 13:57:21
|
<logmsgbot>
|
!log mvernon@cumin2002 conftool action : set/weight=40; selector: name=ms-fe2021.eqiad.wmnet
|
|
2026-02-16 13:57:31
|
<logmsgbot>
|
!log mvernon@cumin2002 conftool action : set/pooled=yes; selector: name=ms-fe2021.eqiad.wmnet
|
|
2026-02-16 13:58:15
|
<logmsgbot>
|
!log mvernon@cumin2002 conftool action : set/weight=40; selector: name=ms-fe1021.eqiad.wmnet
|
|
2026-02-16 13:58:29
|
<logmsgbot>
|
!log mvernon@cumin2002 conftool action : set/pooled=yes; selector: name=ms-fe1021.eqiad.wmnet
|
|
2026-02-16 13:58:37
|
<wikibugs>
|
('PS2) ''Kevin Bazira: ml-services: bump up rr-wikidata workers [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239685 (https://phabricator.wikimedia.org/T414060)'
|
|
2026-02-16 14:00:05
|
<jouncebot>
|
Lucas_WMDE, Urbanecm, and TheresNoTime: How many deployers does it take to do UTC afternoon backport window deploy? (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260216T1400).
|
|
2026-02-16 14:00:05
|
<jouncebot>
|
No Gerrit patches in the queue for this window AFAICS.
|
|
2026-02-16 14:00:12
|
<Lucas_WMDE>
|
good, because I’m in a meeting ^^
|
|
2026-02-16 14:00:51
|
<logmsgbot>
|
!log mvernon@cumin2002 conftool action : set/weight=40; selector: name=ms-fe1022.eqiad.wmnet
|
|
2026-02-16 14:00:59
|
<logmsgbot>
|
!log mvernon@cumin2002 conftool action : set/weight=40; selector: name=ms-fe1023.eqiad.wmnet
|
|
2026-02-16 14:01:10
|
<logmsgbot>
|
!log mvernon@cumin2002 conftool action : set/weight=40; selector: name=ms-fe1024.eqiad.wmnet
|
|
2026-02-16 14:01:31
|
<logmsgbot>
|
!log mvernon@cumin2002 conftool action : set/pooled=yes; selector: name=ms-fe1022.eqiad.wmnet
|
|
2026-02-16 14:01:39
|
<logmsgbot>
|
!log mvernon@cumin2002 conftool action : set/pooled=yes; selector: name=ms-fe1023.eqiad.wmnet
|
|
2026-02-16 14:01:45
|
<logmsgbot>
|
!log mvernon@cumin2002 conftool action : set/pooled=yes; selector: name=ms-fe1024.eqiad.wmnet
|
|
2026-02-16 14:05:43
|
<wikibugs>
|
('CR) ''Federico Ceratto: [C:''+1] "I'm seeing m3-master being switched from dbproxy1028 to dbproxy1026" [dns] - ''https://gerrit.wikimedia.org/r/1239671 (https://phabricator.wikimedia.org/T414656) (owner: ''Marostegui)'
|
|
2026-02-16 14:07:46
|
<wikibugs>
|
('PS5) ''Effie Mouzeli: validating-admission-policies: add /srv/parsoid-testing (vanilla) [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239169 (https://phabricator.wikimedia.org/T386246)'
|
|
2026-02-16 14:08:14
|
<wikibugs>
|
('PS1) ''Brouberol: spark/3.4: rebuild on top of Bookworm [docker-images/production-images] - ''https://gerrit.wikimedia.org/r/1239687 (https://phabricator.wikimedia.org/T416455)'
|
|
2026-02-16 14:08:30
|
<wikibugs>
|
('PS6) ''Effie Mouzeli: validating-admission-policies: add /srv/parsoid-testing (vanilla) [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239169 (https://phabricator.wikimedia.org/T386246)'
|
|
2026-02-16 14:10:33
|
<wikibugs>
|
('PS2) ''Brouberol: spark/3.4: rebuild on top of Bookworm [docker-images/production-images] - ''https://gerrit.wikimedia.org/r/1239687 (https://phabricator.wikimedia.org/T416455)'
|
|
2026-02-16 14:11:13
|
<wikibugs>
|
('CR) ''Elukey: "I think you are missing the Dockerfile change, or was it done previously? Moreover, just to be sure, we cannot go directly to Trixie since" [docker-images/production-images] - ''https://gerrit.wikimedia.org/r/1239687 (https://phabricator.wikimedia.org/T416455) (owner: ''Brouberol)'
|
|
2026-02-16 14:14:14
|
<wikibugs>
|
('PS1) ''Muehlenhoff: standard_packages: Remove support for buster [puppet] - ''https://gerrit.wikimedia.org/r/1239688'
|
|
2026-02-16 14:14:59
|
<wikibugs>
|
'SRE: Migrate RIPE ipblock sources to hiddenparma - https://phabricator.wikimedia.org/T417586 (''JMeybohm) ''NEW'
|
|
2026-02-16 14:15:01
|
<wikibugs>
|
'SRE: Migrate CSV ipblock sources to hiddenparma - https://phabricator.wikimedia.org/T417587 (''JMeybohm) ''NEW'
|
|
2026-02-16 14:15:31
|
<wikibugs>
|
'SRE, ''ServiceOps new, ''Epic: FY 25/26 WE 5.4.2: Known bots / clients - https://phabricator.wikimedia.org/T400100#11620242 (''JMeybohm) ''Open→''Resolved'
|
|
2026-02-16 14:16:42
|
<wikibugs>
|
'ops-eqiad, ''SRE, ''DC-Ops, ''decommission-hardware: decommission puppetmaster1001 - https://phabricator.wikimedia.org/T417580#11620247 (''MoritzMuehlenhoff)'
|
|
2026-02-16 14:18:12
|
<wikibugs>
|
('Abandoned) ''Effie Mouzeli: validating-admission-policies: add /srv/parsoid-testing #0 [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239170 (https://phabricator.wikimedia.org/T386246) (owner: ''Effie Mouzeli)'
|
|
2026-02-16 14:19:56
|
<wikibugs>
|
('CR) ''Muehlenhoff: [C:''+2] Remove puppetmaster::frontend role [puppet] - ''https://gerrit.wikimedia.org/r/1239676 (https://phabricator.wikimedia.org/T365798) (owner: ''Muehlenhoff)'
|
|
2026-02-16 14:20:30
|
<wikibugs>
|
('PS4) ''Elukey: profile::kafka::broker: support new confluent distributions [puppet] - ''https://gerrit.wikimedia.org/r/1239135 (https://phabricator.wikimedia.org/T416670)'
|
|
2026-02-16 14:20:30
|
<wikibugs>
|
('PS3) ''Elukey: role::kafka::test: prepare the cluster for the Kafka upgrade [puppet] - ''https://gerrit.wikimedia.org/r/1239142 (https://phabricator.wikimedia.org/T416670)'
|
|
2026-02-16 14:20:41
|
<wikibugs>
|
('PS1) ''Volans: wmcs: infra-tracing-nfs support non-k8s nodes [puppet] - ''https://gerrit.wikimedia.org/r/1239689 (https://phabricator.wikimedia.org/T415199)'
|
|
2026-02-16 14:21:20
|
<wikibugs>
|
('CR) ''Muehlenhoff: "That worked fine, but since we won't need it again, I'll abandon the patch" [cookbooks] - ''https://gerrit.wikimedia.org/r/1239314 (https://phabricator.wikimedia.org/T365798) (owner: ''Muehlenhoff)'
|
|
2026-02-16 14:21:23
|
<wikibugs>
|
('Abandoned) ''Muehlenhoff: sre.hosts.decommission: Hack to allow decommission of puppetmaster1001 [cookbooks] - ''https://gerrit.wikimedia.org/r/1239314 (https://phabricator.wikimedia.org/T365798) (owner: ''Muehlenhoff)'
|
|
2026-02-16 14:21:51
|
<wikibugs>
|
('PS5) ''Elukey: profile::kafka::broker: support new confluent distributions [puppet] - ''https://gerrit.wikimedia.org/r/1239135 (https://phabricator.wikimedia.org/T416670)'
|
|
2026-02-16 14:21:52
|
<wikibugs>
|
('PS4) ''Elukey: role::kafka::test: prepare the cluster for the Kafka upgrade [puppet] - ''https://gerrit.wikimedia.org/r/1239142 (https://phabricator.wikimedia.org/T416670)'
|
|
2026-02-16 14:24:16
|
<wikibugs>
|
('PS1) ''Muehlenhoff: production-m2.sql.erb: Update comment [puppet] - ''https://gerrit.wikimedia.org/r/1239690'
|
|
2026-02-16 14:24:37
|
<wikibugs>
|
('PS1) ''Jgiannelos: parsoid: Allow overriding special testing config for both host and servergroup [mediawiki-config] - ''https://gerrit.wikimedia.org/r/1239692 (https://phabricator.wikimedia.org/T386246)'
|
|
2026-02-16 14:24:43
|
<wikibugs>
|
('CR) ''Elukey: "check experimental" [puppet] - ''https://gerrit.wikimedia.org/r/1239135 (https://phabricator.wikimedia.org/T416670) (owner: ''Elukey)'
|
|
2026-02-16 14:24:50
|
<wikibugs>
|
('CR) ''Elukey: "check experimental" [puppet] - ''https://gerrit.wikimedia.org/r/1239142 (https://phabricator.wikimedia.org/T416670) (owner: ''Elukey)'
|
|
2026-02-16 14:26:07
|
<wikibugs>
|
('CR) ''Volans: "check experimental" [puppet] - ''https://gerrit.wikimedia.org/r/1239689 (https://phabricator.wikimedia.org/T415199) (owner: ''Volans)'
|
|
2026-02-16 14:26:19
|
<wikibugs>
|
('PS2) ''Jgiannelos: parsoid: Override special test config for parsoid testing env on k8s [mediawiki-config] - ''https://gerrit.wikimedia.org/r/1239692 (https://phabricator.wikimedia.org/T386246)'
|
|
2026-02-16 14:27:05
|
<wikibugs>
|
('PS3) ''Jgiannelos: parsoid: Override special test config for parsoid testing env on k8s [mediawiki-config] - ''https://gerrit.wikimedia.org/r/1239692 (https://phabricator.wikimedia.org/T386246)'
|
|
2026-02-16 14:27:29
|
<wikibugs>
|
('PS4) ''Jgiannelos: parsoid: Override test config for parsoid testing env on k8s [mediawiki-config] - ''https://gerrit.wikimedia.org/r/1239692 (https://phabricator.wikimedia.org/T386246)'
|
|
2026-02-16 14:28:17
|
<wikibugs>
|
('PS1) ''Effie Mouzeli: validating-admission-policies: add /srv/parsoid-testing #0 [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239695 (https://phabricator.wikimedia.org/T386246)'
|
|
2026-02-16 14:28:18
|
<wikibugs>
|
('PS1) ''Muehlenhoff: base::kernel: Unconditionally use the autoremove logic [puppet] - ''https://gerrit.wikimedia.org/r/1239696'
|
|
2026-02-16 14:28:26
|
<wikibugs>
|
('CR) ''Majavah: [C:''+1] Remove buster_ssh_keys [puppet] - ''https://gerrit.wikimedia.org/r/1239673 (owner: ''Muehlenhoff)'
|
|
2026-02-16 14:28:45
|
<wikibugs>
|
('CR) ''Majavah: [C:''+1] "the boxes are on trixie so this could go directly there, or I can submit a follow-up" [puppet] - ''https://gerrit.wikimedia.org/r/1239603 (owner: ''Muehlenhoff)'
|
|
2026-02-16 14:29:39
|
<wikibugs>
|
('CR) ''Volans: "Tested on an NFS worker on toolsbeta for regressions and on toolsbeta-bastion-7." [puppet] - ''https://gerrit.wikimedia.org/r/1239689 (https://phabricator.wikimedia.org/T415199) (owner: ''Volans)'
|
|
2026-02-16 14:30:34
|
<wikibugs>
|
('CR) ''CI reject: [V:''-1] base::kernel: Unconditionally use the autoremove logic [puppet] - ''https://gerrit.wikimedia.org/r/1239696 (owner: ''Muehlenhoff)'
|
|
2026-02-16 14:30:49
|
<wikibugs>
|
('PS4) ''Effie Mouzeli: admin_ng: add ValidatingAdmissionPolicy for mw-parsoid #1 [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239174 (https://phabricator.wikimedia.org/T386246)'
|
|
2026-02-16 14:31:32
|
<wikibugs>
|
('PS2) ''Muehlenhoff: base::kernel: Unconditionally use the autoremove logic [puppet] - ''https://gerrit.wikimedia.org/r/1239696'
|
|
2026-02-16 14:31:44
|
<wikibugs>
|
('CR) ''Brouberol: "re jvm8: yes that's right." [docker-images/production-images] - ''https://gerrit.wikimedia.org/r/1239687 (https://phabricator.wikimedia.org/T416455) (owner: ''Brouberol)'
|
|
2026-02-16 14:32:32
|
<wikibugs>
|
'SRE-swift-storage, ''Data-Persistence, ''MediaViewer, ''Thumbor, and 2 others: FY 25/26 WE 5.4.10 Standard Thumbnail Sizes Only - https://phabricator.wikimedia.org/T414805#11620283 (''Ladsgroup) FWIW: Out of 9K results in https://global-search.toolforge.org/?q=%5C%2Fthumb%5C%2F®ex=1&namespaces=8&titl...'
|
|
2026-02-16 14:32:34
|
<wikibugs>
|
('PS5) ''Effie Mouzeli: admin_ng: add ValidatingAdmissionPolicy for mw-parsoid #1 [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239174 (https://phabricator.wikimedia.org/T386246)'
|
|
2026-02-16 14:33:18
|
<wikibugs>
|
('PS6) ''Effie Mouzeli: admin_ng: add ValidatingAdmissionPolicy for mw-parsoid #1 [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239174 (https://phabricator.wikimedia.org/T386246)'
|
|
2026-02-16 14:33:43
|
<wikibugs>
|
('PS7) ''Effie Mouzeli: admin_ng: add ValidatingAdmissionPolicy for mw-parsoid #1 [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239174 (https://phabricator.wikimedia.org/T386246)'
|
|
2026-02-16 14:36:42
|
<wikibugs>
|
('CR) ''Marostegui: [C:''+1] production-m2.sql.erb: Update comment [puppet] - ''https://gerrit.wikimedia.org/r/1239690 (owner: ''Muehlenhoff)'
|
|
2026-02-16 14:36:58
|
<wikibugs>
|
('CR) ''Federico Ceratto: [C:''+2] installserver: new ms-fe nodes are UEFI booted [puppet] - ''https://gerrit.wikimedia.org/r/1239645 (https://phabricator.wikimedia.org/T416245) (owner: ''MVernon)'
|
|
2026-02-16 14:37:39
|
<wikibugs>
|
('CR) ''Federico Ceratto: [C:''+1] "LGTM" [puppet] - ''https://gerrit.wikimedia.org/r/1239645 (https://phabricator.wikimedia.org/T416245) (owner: ''MVernon)'
|
|
2026-02-16 14:37:58
|
<Thiemo_WMDE>
|
Lucas_WMDE Question, are you still available to do a backport or is it to late for today?
|
|
2026-02-16 14:38:14
|
<Lucas_WMDE>
|
I’m still in a meeting (haven’t been available the whole time)
|
|
2026-02-16 14:38:33
|
<Thiemo_WMDE>
|
Ok, don't worry.
|
|
2026-02-16 14:38:37
|
<wikibugs>
|
('CR) ''MVernon: [C:''+2] installserver: new ms-fe nodes are UEFI booted [puppet] - ''https://gerrit.wikimedia.org/r/1239645 (https://phabricator.wikimedia.org/T416245) (owner: ''MVernon)'
|
|
2026-02-16 14:38:59
|
<wikibugs>
|
('PS1) ''Fabfur: cache::upload: increase global request limit on upload (browser) [puppet] - ''https://gerrit.wikimedia.org/r/1239703 (https://phabricator.wikimedia.org/T406545)'
|
|
2026-02-16 14:40:48
|
<wikibugs>
|
('CR) ''Marostegui: [C:''+2] wmnet: Failover m3-master [dns] - ''https://gerrit.wikimedia.org/r/1239671 (https://phabricator.wikimedia.org/T414656) (owner: ''Marostegui)'
|
|
2026-02-16 14:40:56
|
<logmsgbot>
|
!log marostegui@dns1006 START - running authdns-update
|
|
2026-02-16 14:41:15
|
<wikibugs>
|
('CR) ''Elukey: "still no ready :)" [puppet] - ''https://gerrit.wikimedia.org/r/1239135 (https://phabricator.wikimedia.org/T416670) (owner: ''Elukey)'
|
|
2026-02-16 14:41:24
|
<wikibugs>
|
('PS5) ''Jgiannelos: parsoid: Override test config for parsoid testing env on k8s [mediawiki-config] - ''https://gerrit.wikimedia.org/r/1239692 (https://phabricator.wikimedia.org/T386246)'
|
|
2026-02-16 14:41:44
|
<wikibugs>
|
('CR) ''ScheduleDeploymentBot: "Scheduled for deployment in the [Tuesday, February 17 UTC morning backport window](https://wikitech.wikimedia.org/wiki/Deployments#deployc"; [extensions/Cite] (wmf/1.46.0-wmf.15) - ''https://gerrit.wikimedia.org/r/1239573 (https://phabricator.wikimedia.org/T416630) (owner: ''WMDE-Fisch)'
|
|
2026-02-16 14:41:48
|
<marostegui>
|
!log Failover m3 dbproxy (phabricator) T414656
|
|
2026-02-16 14:41:51
|
<stashbot>
|
Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
|
|
2026-02-16 14:41:52
|
<stashbot>
|
T414656: Migrate dbproxy* to Debian Trixie - https://phabricator.wikimedia.org/T414656
|
|
2026-02-16 14:42:12
|
<logmsgbot>
|
!log marostegui@dns1006 END - running authdns-update
|
|
2026-02-16 14:42:23
|
<wikibugs>
|
('PS6) ''Elukey: profile::kafka::broker: support new confluent distributions [puppet] - ''https://gerrit.wikimedia.org/r/1239135 (https://phabricator.wikimedia.org/T416670)'
|
|
2026-02-16 14:42:23
|
<wikibugs>
|
('PS5) ''Elukey: role::kafka::test: prepare the cluster for the Kafka upgrade [puppet] - ''https://gerrit.wikimedia.org/r/1239142 (https://phabricator.wikimedia.org/T416670)'
|
|
2026-02-16 14:42:40
|
<wikibugs>
|
('CR) ''Elukey: "check experimental" [puppet] - ''https://gerrit.wikimedia.org/r/1239135 (https://phabricator.wikimedia.org/T416670) (owner: ''Elukey)'
|
|
2026-02-16 14:43:09
|
<wikibugs>
|
('PS2) ''Volans: wmcs: infra-tracing-nfs support non-k8s nodes [puppet] - ''https://gerrit.wikimedia.org/r/1239689 (https://phabricator.wikimedia.org/T415199)'
|
|
2026-02-16 14:45:07
|
<wikibugs>
|
('PS8) ''Effie Mouzeli: admin_ng: add ValidatingAdmissionPolicy for mw-parsoid #1 [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239174 (https://phabricator.wikimedia.org/T386246)'
|
|
2026-02-16 14:45:22
|
<zabe>
|
jouncebot: nowandnext
|
|
2026-02-16 14:45:23
|
<jouncebot>
|
For the next 0 hour(s) and 14 minute(s): UTC afternoon backport window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260216T1400)
|
|
2026-02-16 14:45:23
|
<jouncebot>
|
In 0 hour(s) and 44 minute(s): Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260216T1530)
|
|
2026-02-16 14:45:48
|
<wikibugs>
|
('PS3) ''Brouberol: spark/3.4: rebuild on top of Bookworm [docker-images/production-images] - ''https://gerrit.wikimedia.org/r/1239687 (https://phabricator.wikimedia.org/T416455)'
|
|
2026-02-16 14:46:20
|
<wikibugs>
|
('CR) ''Brouberol: "Nvm, golang was only needed for the operator. I indeed needed to update the build image tag." [docker-images/production-images] - ''https://gerrit.wikimedia.org/r/1239687 (https://phabricator.wikimedia.org/T416455) (owner: ''Brouberol)'
|
|
2026-02-16 14:47:18
|
<wikibugs>
|
('CR) ''Zabe: [C:''+2] Start reading from il_target_id on commonswiki [mediawiki-config] - ''https://gerrit.wikimedia.org/r/1238010 (https://phabricator.wikimedia.org/T413669) (owner: ''Zabe)'
|
|
2026-02-16 14:47:50
|
<wikibugs>
|
('CR) ''Elukey: "IIUC the spark3.4 image uses a multi-stage build, so spark3.4-build is on bookworkm but it is the build variant, meanwhile the one that en" [docker-images/production-images] - ''https://gerrit.wikimedia.org/r/1239687 (https://phabricator.wikimedia.org/T416455) (owner: ''Brouberol)'
|
|
2026-02-16 14:48:21
|
<wikibugs>
|
('Merged) ''jenkins-bot: Start reading from il_target_id on commonswiki [mediawiki-config] - ''https://gerrit.wikimedia.org/r/1238010 (https://phabricator.wikimedia.org/T413669) (owner: ''Zabe)'
|
|
2026-02-16 14:48:58
|
<logmsgbot>
|
!log zabe@deploy2002 Started scap sync-world: Backport for [[gerrit:1238010|Start reading from il_target_id on commonswiki (T413669)]]
|
|
2026-02-16 14:49:02
|
<stashbot>
|
T413669: Set imagelinks migration to read new - https://phabricator.wikimedia.org/T413669
|
|
2026-02-16 14:49:48
|
<wikibugs>
|
('CR) ''Elukey: "check experimental" [puppet] - ''https://gerrit.wikimedia.org/r/1239142 (https://phabricator.wikimedia.org/T416670) (owner: ''Elukey)'
|
|
2026-02-16 14:50:40
|
<wikibugs>
|
('CR) ''Fabian Kaelin: [C:''+1] "Thanks!" [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239670 (https://phabricator.wikimedia.org/T417190) (owner: ''Brouberol)'
|
|
2026-02-16 14:51:13
|
<wikibugs>
|
('PS3) ''Volans: wmcs: infra-tracing-nfs support non-k8s nodes [puppet] - ''https://gerrit.wikimedia.org/r/1239689 (https://phabricator.wikimedia.org/T415199)'
|
|
2026-02-16 14:51:34
|
<wikibugs>
|
('PS1) ''Arnaudb: gerrit: rename failover cookbook to switchover [cookbooks] - ''https://gerrit.wikimedia.org/r/1239693 (https://phabricator.wikimedia.org/T387833)'
|
|
2026-02-16 14:51:34
|
<wikibugs>
|
('CR) ''Arnaudb: "This is the last item on the todolist for T387833" [cookbooks] - ''https://gerrit.wikimedia.org/r/1239693 (https://phabricator.wikimedia.org/T387833) (owner: ''Arnaudb)'
|
|
2026-02-16 14:51:36
|
<wikibugs>
|
('CR) ''Brouberol: [C:''+2] airflow-research: revert to storing all XCOMs in DB [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239670 (https://phabricator.wikimedia.org/T417190) (owner: ''Brouberol)'
|
|
2026-02-16 14:51:44
|
<wikibugs>
|
('CR) ''Arnaudb: [C:''+2] gerrit: rename failover cookbook to switchover [cookbooks] - ''https://gerrit.wikimedia.org/r/1239693 (https://phabricator.wikimedia.org/T387833) (owner: ''Arnaudb)'
|
|
2026-02-16 14:51:50
|
<wikibugs>
|
('PS6) ''Effie Mouzeli: mw-parsoid: repurpose for parsoidtest use #6 [deployment-charts] - ''https://gerrit.wikimedia.org/r/1237472 (https://phabricator.wikimedia.org/T386246)'
|
|
2026-02-16 14:52:45
|
<wikibugs>
|
('PS4) ''Brouberol: spark/3.4: rebuild on top of Bookworm [docker-images/production-images] - ''https://gerrit.wikimedia.org/r/1239687 (https://phabricator.wikimedia.org/T416455)'
|
|
2026-02-16 14:53:01
|
<logmsgbot>
|
!log zabe@deploy2002 zabe: Backport for [[gerrit:1238010|Start reading from il_target_id on commonswiki (T413669)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.
|
|
2026-02-16 14:53:29
|
<wikibugs>
|
('CR) ''Volans: "puppet compiler output at:" [puppet] - ''https://gerrit.wikimedia.org/r/1239689 (https://phabricator.wikimedia.org/T415199) (owner: ''Volans)'
|
|
2026-02-16 14:54:11
|
<logmsgbot>
|
!log zabe@deploy2002 zabe: Continuing with sync
|
|
2026-02-16 14:55:33
|
<wikibugs>
|
('CR) ''Gehel: [C:''+1] "I haven't reviewed each file. The change seems low risk, appropriate, and we can iterate if we need to correct. So from my side: you can m" [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239681 (https://phabricator.wikimedia.org/T416712) (owner: ''Brouberol)'
|
|
2026-02-16 14:56:45
|
<wikibugs>
|
('Merged) ''jenkins-bot: gerrit: rename failover cookbook to switchover [cookbooks] - ''https://gerrit.wikimedia.org/r/1239693 (https://phabricator.wikimedia.org/T387833) (owner: ''Arnaudb)'
|
|
2026-02-16 14:58:38
|
<wikibugs>
|
('PS1) ''Effie Mouzeli: restbase::production: remove mw-parsoid listener [puppet] - ''https://gerrit.wikimedia.org/r/1239709 (https://phabricator.wikimedia.org/T386246)'
|
|
2026-02-16 14:58:45
|
<wikibugs>
|
('CR) ''Filippo Giunchedi: "LGTM modulo typo" [puppet] - ''https://gerrit.wikimedia.org/r/1239689 (https://phabricator.wikimedia.org/T415199) (owner: ''Volans)'
|
|
2026-02-16 15:00:30
|
<logmsgbot>
|
!log zabe@deploy2002 Finished scap sync-world: Backport for [[gerrit:1238010|Start reading from il_target_id on commonswiki (T413669)]] (duration: 11m 31s)
|
|
2026-02-16 15:00:33
|
<stashbot>
|
T413669: Set imagelinks migration to read new - https://phabricator.wikimedia.org/T413669
|
|
2026-02-16 15:01:01
|
<wikibugs>
|
('PS7) ''Elukey: profile::kafka::broker: support new confluent distributions [puppet] - ''https://gerrit.wikimedia.org/r/1239135 (https://phabricator.wikimedia.org/T416670)'
|
|
2026-02-16 15:01:01
|
<wikibugs>
|
('PS6) ''Elukey: role::kafka::test: prepare the cluster for the Kafka upgrade [puppet] - ''https://gerrit.wikimedia.org/r/1239142 (https://phabricator.wikimedia.org/T416670)'
|
|
2026-02-16 15:01:03
|
<wikibugs>
|
('CR) ''Effie Mouzeli: mw-parsoid: repurpose for parsoidtest use #6 (''1 comment) [deployment-charts] - ''https://gerrit.wikimedia.org/r/1237472 (https://phabricator.wikimedia.org/T386246) (owner: ''Effie Mouzeli)'
|
|
2026-02-16 15:01:16
|
<wikibugs>
|
('CR) ''Elukey: "check experimental" [puppet] - ''https://gerrit.wikimedia.org/r/1239135 (https://phabricator.wikimedia.org/T416670) (owner: ''Elukey)'
|
|
2026-02-16 15:01:22
|
<wikibugs>
|
('CR) ''Elukey: "check experimental" [puppet] - ''https://gerrit.wikimedia.org/r/1239142 (https://phabricator.wikimedia.org/T416670) (owner: ''Elukey)'
|
|
2026-02-16 15:03:08
|
<wikibugs>
|
('CR) ''Gkyziridis: [C:''+1] "Thank you for deploying!" [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239685 (https://phabricator.wikimedia.org/T414060) (owner: ''Kevin Bazira)'
|
|
2026-02-16 15:03:34
|
<wikibugs>
|
('CR) ''Kevin Bazira: [C:''+2] ml-services: bump up rr-wikidata workers [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239685 (https://phabricator.wikimedia.org/T414060) (owner: ''Kevin Bazira)'
|
|
2026-02-16 15:04:34
|
<wikibugs>
|
('PS2) ''Effie Mouzeli: validating-admission-policies: add /srv/parsoid-testing #0 [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239695 (https://phabricator.wikimedia.org/T386246)'
|
|
2026-02-16 15:06:01
|
<wikibugs>
|
('Merged) ''jenkins-bot: ml-services: bump up rr-wikidata workers [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239685 (https://phabricator.wikimedia.org/T414060) (owner: ''Kevin Bazira)'
|
|
2026-02-16 15:06:01
|
<wikibugs>
|
('CR) ''Elukey: "I reworked the patch a little bit, lemme know :)" [puppet] - ''https://gerrit.wikimedia.org/r/1239135 (https://phabricator.wikimedia.org/T416670) (owner: ''Elukey)'
|
|
2026-02-16 15:06:41
|
<wikibugs>
|
('PS9) ''Effie Mouzeli: admin_ng: add ValidatingAdmissionPolicy for mw-parsoid #1 [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239174 (https://phabricator.wikimedia.org/T386246)'
|
|
2026-02-16 15:07:02
|
<logmsgbot>
|
!log kevinbazira@deploy2002 helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
|
|
2026-02-16 15:07:18
|
<wikibugs>
|
('CR) ''Brouberol: [C:''+2] chart: add maintenance metadata to all dpe charts [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239681 (https://phabricator.wikimedia.org/T416712) (owner: ''Brouberol)'
|
|
2026-02-16 15:09:03
|
<wikibugs>
|
('PS7) ''Effie Mouzeli: mw-parsoid: repurpose for parsoidtest use #6 [deployment-charts] - ''https://gerrit.wikimedia.org/r/1237472 (https://phabricator.wikimedia.org/T386246)'
|
|
2026-02-16 15:15:27
|
<wikibugs>
|
'SRE, ''SRE-Access-Requests, ''Data-Platform-SRE (2026-02-13 - 2026-03-06): Requesting access to "Community Wishlist" dashboard for hmonroy on Superset - https://phabricator.wikimedia.org/T416721#11620423 (''Gehel) >>! In T416721#11619620, @MatthewVernon wrote: > @Gehel can I ping you for your thoughts on t...'
|
|
2026-02-16 15:18:13
|
<jinxer-wm>
|
FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - pfw1-codfw:reth2 (fasw1-f5 2x25G) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=pfw1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown
|
|
2026-02-16 15:21:42
|
<wikibugs>
|
'SRE-swift-storage: Cleanup old swift-cert - https://phabricator.wikimedia.org/T414973#11620431 (''MatthewVernon) ''Open→''Resolved Done - I checked on the frontends that these keys weren't used, removed them from private puppet, and then re-ran puppet on a frontend to check for no changes.'
|
|
2026-02-16 15:21:47
|
<wikibugs>
|
'SRE, ''SRE-Access-Requests, ''Data-Platform-SRE (2026-02-13 - 2026-03-06): Requesting access to "Community Wishlist" dashboard for hmonroy on Superset - https://phabricator.wikimedia.org/T416721#11620434 (''brouberol) I've set @HMonroy as co-owner of https://superset.wikimedia.org/superset/dashboard/686/'
|
|
2026-02-16 15:22:05
|
<wikibugs>
|
('PS1) ''Federico Ceratto: acme_chief: remove old dborch1001 node [puppet] - ''https://gerrit.wikimedia.org/r/1239720 (https://phabricator.wikimedia.org/T416582)'
|
|
2026-02-16 15:23:01
|
<wikibugs>
|
('PS1) ''Kevin Bazira: ml-services: scale down rr-wikidata pod memory [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239721 (https://phabricator.wikimedia.org/T414060)'
|
|
2026-02-16 15:23:39
|
<wikibugs>
|
('CR) ''Marostegui: [C:''+1] acme_chief: remove old dborch1001 node [puppet] - ''https://gerrit.wikimedia.org/r/1239720 (https://phabricator.wikimedia.org/T416582) (owner: ''Federico Ceratto)'
|
|
2026-02-16 15:23:49
|
<wikibugs>
|
'SRE, ''SRE-Access-Requests, ''Data-Platform-SRE (2026-02-13 - 2026-03-06): Requesting access to "Community Wishlist" dashboard for hmonroy on Superset - https://phabricator.wikimedia.org/T416721#11620445 (''KSiebert) Can you also make it publicly visible to all staff?'
|
|
2026-02-16 15:23:56
|
<wikibugs>
|
'SRE, ''Infrastructure-Foundations: decom cookbook used Junos commands on a Nokia switch - https://phabricator.wikimedia.org/T417428#11620446 (''ayounsi) p:''Triage→''High'
|
|
2026-02-16 15:24:18
|
<wikibugs>
|
'sre-alert-triage, ''Infrastructure-Foundations, ''netops: Alert in need of triage: PeeringBGPDown (instance cr1-drmrs:9804) - https://phabricator.wikimedia.org/T416987#11620447 (''ayounsi) p:''Triage→''Low a:''ayounsi'
|
|
2026-02-16 15:25:01
|
<wikibugs>
|
('CR) ''Federico Ceratto: [C:''+2] acme_chief: remove old dborch1001 node [puppet] - ''https://gerrit.wikimedia.org/r/1239720 (https://phabricator.wikimedia.org/T416582) (owner: ''Federico Ceratto)'
|
|
2026-02-16 15:26:45
|
<wikibugs>
|
('PS1) ''Effie Mouzeli: mw-on-k8s: do not alert for mw-experimental and mw-parsoid [alerts] - ''https://gerrit.wikimedia.org/r/1239724'
|
|
2026-02-16 15:26:59
|
<wikibugs>
|
'SRE, ''SRE-Access-Requests, ''Data-Platform-SRE (2026-02-13 - 2026-03-06): Requesting access to "Community Wishlist" dashboard for hmonroy on Superset - https://phabricator.wikimedia.org/T416721#11620453 (''Gehel) >>! In T416721#11620445, @KSiebert wrote: > Can you also make it publicly visible to all staf...'
|
|
2026-02-16 15:27:21
|
<wikibugs>
|
'SRE, ''Incident Tooling: Migrate RIPE ipblock sources to hiddenparma - https://phabricator.wikimedia.org/T417586#11620454 (''MatthewVernon)'
|
|
2026-02-16 15:27:34
|
<wikibugs>
|
'SRE, ''Incident Tooling: Migrate CSV ipblock sources to hiddenparma - https://phabricator.wikimedia.org/T417587#11620455 (''MatthewVernon)'
|
|
2026-02-16 15:28:10
|
<wikibugs>
|
('CR) ''Gkyziridis: [C:''+1] "LGTM" [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239721 (https://phabricator.wikimedia.org/T414060) (owner: ''Kevin Bazira)'
|
|
2026-02-16 15:28:39
|
<wikibugs>
|
('PS1) ''Federico Ceratto: orchestrator: switch orchestrator.w.o to dborch1002 [dns] - ''https://gerrit.wikimedia.org/r/1239725 (https://phabricator.wikimedia.org/T416582)'
|
|
2026-02-16 15:28:57
|
<wikibugs>
|
('CR) ''Kevin Bazira: [C:''+2] ml-services: scale down rr-wikidata pod memory [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239721 (https://phabricator.wikimedia.org/T414060) (owner: ''Kevin Bazira)'
|
|
2026-02-16 15:29:36
|
<wikibugs>
|
'SRE, ''SRE-Access-Requests, ''Data-Platform-SRE (2026-02-13 - 2026-03-06): Requesting access to "Community Wishlist" dashboard for hmonroy on Superset - https://phabricator.wikimedia.org/T416721#11620460 (''MatthewVernon) FWIW, it is (I think!) now visible to staff with sufficient analytics-privatedata rig...'
|
|
2026-02-16 15:30:05
|
<jouncebot>
|
Deploy window Test Kitchen Experiment Deployment Window (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260216T1530)
|
|
2026-02-16 15:30:49
|
<wikibugs>
|
('Merged) ''jenkins-bot: ml-services: scale down rr-wikidata pod memory [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239721 (https://phabricator.wikimedia.org/T414060) (owner: ''Kevin Bazira)'
|
|
2026-02-16 15:31:43
|
<wikibugs>
|
('CR) ''Marostegui: [C:''+1] orchestrator: switch orchestrator.w.o to dborch1002 [dns] - ''https://gerrit.wikimedia.org/r/1239725 (https://phabricator.wikimedia.org/T416582) (owner: ''Federico Ceratto)'
|
|
2026-02-16 15:32:06
|
<wikibugs>
|
'ops-eqiad, ''SRE, ''DC-Ops, ''Data-Platform-SRE (2026-02-13 - 2026-03-06): Q3:rack/setup/install dse-k8s-worker10[20-23] - https://phabricator.wikimedia.org/T414216#11620494 (''Gehel)'
|
|
2026-02-16 15:32:22
|
<wikibugs>
|
'ops-eqiad, ''SRE, ''DC-Ops, ''Data-Platform-SRE (2026-02-13 - 2026-03-06), ''Essential-Work: Q2:rack/setup/install wdqs1033-1035 - https://phabricator.wikimedia.org/T411731#11620500 (''Gehel)'
|
|
2026-02-16 15:32:30
|
<logmsgbot>
|
!log kevinbazira@deploy2002 helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
|
|
2026-02-16 15:32:34
|
<wikibugs>
|
('CR) ''Federico Ceratto: [C:''+2] orchestrator: switch orchestrator.w.o to dborch1002 [dns] - ''https://gerrit.wikimedia.org/r/1239725 (https://phabricator.wikimedia.org/T416582) (owner: ''Federico Ceratto)'
|
|
2026-02-16 15:33:34
|
<wikibugs>
|
'SRE-SLO, ''observability, ''Wikidata, ''Wikidata-Query-Service, and 3 others: Update WDQS SLOs to reflect graph split changes - https://phabricator.wikimedia.org/T393966#11620531 (''Gehel)'
|
|
2026-02-16 15:33:55
|
<logmsgbot>
|
!log fceratto@dns1004 START - running authdns-update
|
|
2026-02-16 15:34:06
|
<wikibugs>
|
'SRE, ''Data-Platform-SRE (2026-02-13 - 2026-03-06), ''Patch-For-Review: October 2025 Bullseye reboots: Data Platform Engineering-owned hosts - https://phabricator.wikimedia.org/T411568#11620542 (''Gehel)'
|
|
2026-02-16 15:34:36
|
<wikibugs>
|
'ops-eqiad, ''SRE, ''DC-Ops, ''Data-Platform-SRE (2026-02-13 - 2026-03-06): Unusually high disk errors on the an-worker nodes since upgrading the disks - https://phabricator.wikimedia.org/T415002#11620555 (''Gehel)'
|
|
2026-02-16 15:34:46
|
<wikibugs>
|
'ops-eqiad, ''SRE, ''DC-Ops, ''Data-Platform-SRE (2026-02-13 - 2026-03-06): Degraded RAID on an-worker1204 - https://phabricator.wikimedia.org/T414861#11620561 (''Gehel)'
|
|
2026-02-16 15:35:12
|
<logmsgbot>
|
!log fceratto@dns1004 END - running authdns-update
|
|
2026-02-16 15:35:44
|
<wikibugs>
|
'SRE, ''Infrastructure-Foundations, ''netops, ''Data-Platform-SRE (2026-02-13 - 2026-03-06), ''Essential-Work: Socket leaking on some dse-k8s row C & D hosts - https://phabricator.wikimedia.org/T414460#11620591 (''Gehel)'
|
|
2026-02-16 15:36:21
|
<wikibugs>
|
'sre-alert-triage, ''Data-Platform-SRE (2026-02-13 - 2026-03-06): Alert in need of triage: KubernetesAPIErrorRate - https://phabricator.wikimedia.org/T414970#11620597 (''Gehel)'
|
|
2026-02-16 15:36:26
|
<wikibugs>
|
'sre-alert-triage, ''Data-Platform-SRE (2026-02-13 - 2026-03-06): Alert in need of triage: KubernetesAPIErrorRate - https://phabricator.wikimedia.org/T414413#11620599 (''Gehel)'
|
|
2026-02-16 15:37:24
|
<icinga-wm>
|
PROBLEM - orchestrator TCP port on dborch1001 is CRITICAL: connect to address 127.0.0.1 and port 3000: Connection refused https://wikitech.wikimedia.org/wiki/Orchestrator
|
|
2026-02-16 15:37:24
|
<icinga-wm>
|
PROBLEM - orchestrator process on dborch1001 is CRITICAL: PROCS CRITICAL: 0 processes with regex args orchestrator http https://wikitech.wikimedia.org/wiki/Orchestrator
|
|
2026-02-16 15:39:02
|
<logmsgbot>
|
!log brouberol@deploy2002 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset-next: apply
|
|
2026-02-16 15:39:25
|
<wikibugs>
|
'ops-eqiad, ''SRE, ''DC-Ops, ''Data-Platform-SRE (2026-02-13 - 2026-03-06): Follow-up: Degraded Disk Not Yet Added to RAID (an-worker1175, an-worker1199) - https://phabricator.wikimedia.org/T416166#11620657 (''Gehel)'
|
|
2026-02-16 15:39:58
|
<logmsgbot>
|
!log brouberol@deploy2002 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset-next: apply
|
|
2026-02-16 15:42:53
|
<wikibugs>
|
'SRE, ''Infrastructure-Foundations: offboarding Alex Kosiaris - https://phabricator.wikimedia.org/T417465#11620706 (''MoritzMuehlenhoff) p:''Triage→''Medium'
|
|
2026-02-16 15:42:57
|
<wikibugs>
|
('CR) ''Elukey: [C:''+1] spark/3.4: rebuild on top of Bookworm [docker-images/production-images] - ''https://gerrit.wikimedia.org/r/1239687 (https://phabricator.wikimedia.org/T416455) (owner: ''Brouberol)'
|
|
2026-02-16 15:45:03
|
<wikibugs>
|
('Abandoned) ''BCornwall: cache::haproxy: Only use lua5.3 mmdb on haproxy28 [puppet] - ''https://gerrit.wikimedia.org/r/1239463 (https://phabricator.wikimedia.org/T401832) (owner: ''BCornwall)'
|
|
2026-02-16 15:48:41
|
<wikibugs>
|
('CR) ''JMeybohm: [C:''+1] validating-admission-policies: add /srv/parsoid-testing #0 [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239695 (https://phabricator.wikimedia.org/T386246) (owner: ''Effie Mouzeli)'
|
|
2026-02-16 15:48:48
|
<wikibugs>
|
('CR) ''JMeybohm: [C:''+1] validating-admission-policies: add /srv/parsoid-testing (vanilla) [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239169 (https://phabricator.wikimedia.org/T386246) (owner: ''Effie Mouzeli)'
|
|
2026-02-16 15:48:50
|
<wikibugs>
|
('PS1) ''Brouberol: superset: update PYTHONPATH to reflect recent change to bookworm [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239730 (https://phabricator.wikimedia.org/T416455)'
|
|
2026-02-16 15:48:52
|
<wikibugs>
|
('PS1) ''Brouberol: superset: release new bookworm-based image [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239731 (https://phabricator.wikimedia.org/T416455)'
|
|
2026-02-16 15:49:56
|
<wikibugs>
|
('CR) ''Brouberol: [C:''+2] spark/3.4: rebuild on top of Bookworm [docker-images/production-images] - ''https://gerrit.wikimedia.org/r/1239687 (https://phabricator.wikimedia.org/T416455) (owner: ''Brouberol)'
|
|
2026-02-16 15:49:59
|
<wikibugs>
|
('CR) ''Brouberol: [V:''+2 C:''+2] spark/3.4: rebuild on top of Bookworm [docker-images/production-images] - ''https://gerrit.wikimedia.org/r/1239687 (https://phabricator.wikimedia.org/T416455) (owner: ''Brouberol)'
|
|
2026-02-16 15:50:16
|
<wikibugs>
|
('CR) ''JMeybohm: [C:''+1] admin_ng: add ValidatingAdmissionPolicy for mw-parsoid #1 [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239174 (https://phabricator.wikimedia.org/T386246) (owner: ''Effie Mouzeli)'
|
|
2026-02-16 15:52:33
|
<wikibugs>
|
('Abandoned) ''BCornwall: wmfuniq_experiment_fetcher: Use shutil for conf mv [puppet] - ''https://gerrit.wikimedia.org/r/1239458 (https://phabricator.wikimedia.org/T417476) (owner: ''BCornwall)'
|
|
2026-02-16 16:01:24
|
<icinga-wm>
|
RECOVERY - orchestrator TCP port on dborch1001 is OK: TCP OK - 0.001 second response time on 127.0.0.1 port 3000 https://wikitech.wikimedia.org/wiki/Orchestrator
|
|
2026-02-16 16:01:24
|
<icinga-wm>
|
RECOVERY - orchestrator process on dborch1001 is OK: PROCS OK: 1 process with regex args orchestrator http https://wikitech.wikimedia.org/wiki/Orchestrator
|
|
2026-02-16 16:09:19
|
<jinxer-wm>
|
FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
|
|
2026-02-16 16:10:04
|
<wikibugs>
|
('CR) ''Muehlenhoff: "check experimental" [puppet] - ''https://gerrit.wikimedia.org/r/1239696 (owner: ''Muehlenhoff)'
|
|
2026-02-16 16:19:32
|
<jinxer-wm>
|
FIRING: [2x] ProbeDown: Service wdqs1013:443 has failed probes (http_wdqs_main_external_search_sparql_endpoint_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#wdqs1013:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
|
|
2026-02-16 16:29:03
|
<wikibugs>
|
('CR) ''Gehel: [C:''+1] "LGTM" [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239730 (https://phabricator.wikimedia.org/T416455) (owner: ''Brouberol)'
|
|
2026-02-16 16:30:05
|
<jouncebot>
|
jan_drewniak: Wikimedia Portals Update (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260216T1630). Please do the needful.
|
|
2026-02-16 16:31:25
|
<jinxer-wm>
|
FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
|
|
2026-02-16 16:31:27
|
<wikibugs>
|
('PS4) ''Volans: wmcs: infra-tracing-nfs support non-k8s nodes [puppet] - ''https://gerrit.wikimedia.org/r/1239689 (https://phabricator.wikimedia.org/T415199)'
|
|
2026-02-16 16:32:05
|
<wikibugs>
|
('CR) ''Volans: wmcs: infra-tracing-nfs support non-k8s nodes (''1 comment) [puppet] - ''https://gerrit.wikimedia.org/r/1239689 (https://phabricator.wikimedia.org/T415199) (owner: ''Volans)'
|
|
2026-02-16 16:34:19
|
<jinxer-wm>
|
FIRING: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
|
|
2026-02-16 16:35:07
|
<wikibugs>
|
('PS1) ''MVernon: codfw swift: remove drained ms-be20[57-61] for decom [puppet] - ''https://gerrit.wikimedia.org/r/1239734 (https://phabricator.wikimedia.org/T404771)'
|
|
2026-02-16 16:35:08
|
<wikibugs>
|
('PS1) ''MVernon: hiera: remove ms-be20[57-61] for decom [puppet] - ''https://gerrit.wikimedia.org/r/1239735 (https://phabricator.wikimedia.org/T404771)'
|
|
2026-02-16 16:35:15
|
<jinxer-wm>
|
RESOLVED: [2x] JobUnavailable: Reduced availability for job sidekiq in ops@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable
|
|
2026-02-16 16:52:01
|
<vgutierrez>
|
!log depool cp7001 - T417536
|
|
2026-02-16 16:52:04
|
<stashbot>
|
Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
|
|
2026-02-16 16:52:07
|
<stashbot>
|
T417536: Investigate gerrit 5xx responses - https://phabricator.wikimedia.org/T417536
|
|
2026-02-16 16:59:07
|
<wikibugs>
|
('PS1) ''Kevin Bazira: ml-services: reduce rr-wikidata memory to comply with LimitRange [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239738 (https://phabricator.wikimedia.org/T414060)'
|
|
2026-02-16 17:00:42
|
<wikibugs>
|
('PS1) ''MVernon: admin: remove mvernon software-only ssh pubkey [puppet] - ''https://gerrit.wikimedia.org/r/1239739 (https://phabricator.wikimedia.org/T412796)'
|
|
2026-02-16 17:03:06
|
<icinga-wm>
|
PROBLEM - Host wikikube-worker1268 is DOWN: PING CRITICAL - Packet loss = 100%
|
|
2026-02-16 17:03:34
|
<icinga-wm>
|
RECOVERY - Host wikikube-worker1268 is UP: PING OK - Packet loss = 0%, RTA = 0.32 ms
|
|
2026-02-16 17:15:16
|
<wikibugs>
|
('CR) ''Muehlenhoff: [C:''+1] "Looks good!" [puppet] - ''https://gerrit.wikimedia.org/r/1239739 (https://phabricator.wikimedia.org/T412796) (owner: ''MVernon)'
|
|
2026-02-16 17:16:56
|
<wikibugs>
|
('CR) ''Gkyziridis: [C:''+1] "Thnx for fixing this!" [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239738 (https://phabricator.wikimedia.org/T414060) (owner: ''Kevin Bazira)'
|
|
2026-02-16 17:18:45
|
<wikibugs>
|
('CR) ''MVernon: [C:''+2] admin: remove mvernon software-only ssh pubkey [puppet] - ''https://gerrit.wikimedia.org/r/1239739 (https://phabricator.wikimedia.org/T412796) (owner: ''MVernon)'
|
|
2026-02-16 17:19:54
|
<wikibugs>
|
('PS1) ''Muehlenhoff: profile::puppet::agent: Remove support for Buster [puppet] - ''https://gerrit.wikimedia.org/r/1239749 (https://phabricator.wikimedia.org/T365798)'
|
|
2026-02-16 17:20:19
|
<wikibugs>
|
'SRE, ''Data-Persistence, ''Patch-For-Review: Add FIDO ssh key for mvernon - https://phabricator.wikimedia.org/T412796#11620869 (''MatthewVernon) ''Stalled→''Resolved'
|
|
2026-02-16 17:23:16
|
<wikibugs>
|
('PS2) ''Muehlenhoff: Run cloudlb spec tests on Trixie [puppet] - ''https://gerrit.wikimedia.org/r/1239603'
|
|
2026-02-16 17:23:41
|
<wikibugs>
|
('CR) ''Kevin Bazira: [C:''+2] ml-services: reduce rr-wikidata memory to comply with LimitRange [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239738 (https://phabricator.wikimedia.org/T414060) (owner: ''Kevin Bazira)'
|
|
2026-02-16 17:24:33
|
<wikibugs>
|
('CR) ''Muehlenhoff: "check experimental" [puppet] - ''https://gerrit.wikimedia.org/r/1239749 (https://phabricator.wikimedia.org/T365798) (owner: ''Muehlenhoff)'
|
|
2026-02-16 17:25:15
|
<vgutierrez>
|
!log repool cp7001 - T417536
|
|
2026-02-16 17:25:18
|
<stashbot>
|
Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
|
|
2026-02-16 17:25:19
|
<stashbot>
|
T417536: Investigate gerrit 5xx responses - https://phabricator.wikimedia.org/T417536
|
|
2026-02-16 17:26:05
|
<wikibugs>
|
('Merged) ''jenkins-bot: ml-services: reduce rr-wikidata memory to comply with LimitRange [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239738 (https://phabricator.wikimedia.org/T414060) (owner: ''Kevin Bazira)'
|
|
2026-02-16 17:28:29
|
<logmsgbot>
|
!log kevinbazira@deploy2002 helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
|
|
2026-02-16 17:31:37
|
<logmsgbot>
|
!log kevinbazira@deploy2002 helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
|
|
2026-02-16 18:00:04
|
<jouncebot>
|
Deploy window MediaWiki infrastructure (UTC late) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260216T1800)
|
|
2026-02-16 18:00:05
|
<jouncebot>
|
ryankemper: #bothumor Q:Why did functions stop calling each other? A:They had arguments. Rise for Wikidata Query Service weekly deploy . (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260216T1800).
|
|
2026-02-16 18:08:09
|
<logmsgbot>
|
!log fceratto@cumin1003 START - Cookbook sre.mysql.update-replication
|
|
2026-02-16 18:08:19
|
<logmsgbot>
|
!log fceratto@cumin1003 END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0)
|
|
2026-02-16 18:08:41
|
<logmsgbot>
|
!log fceratto@cumin1003 START - Cookbook sre.mysql.update-replication
|
|
2026-02-16 18:08:49
|
<logmsgbot>
|
!log fceratto@cumin1003 END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0)
|
|
2026-02-16 18:09:24
|
<logmsgbot>
|
!log fceratto@cumin1003 START - Cookbook sre.mysql.update-replication
|
|
2026-02-16 18:09:32
|
<logmsgbot>
|
!log fceratto@cumin1003 END (PASS) - Cookbook sre.mysql.update-replication (exit_code=0)
|
|
2026-02-16 19:07:35
|
<zabe>
|
!log zabe@deploy2002:~$ mwscript extensions/TimedMediaHandler/maintenance/migrateTranscodeStates.php testwiki --force # T415064
|
|
2026-02-16 19:07:37
|
<stashbot>
|
Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
|
|
2026-02-16 19:07:38
|
<stashbot>
|
T415064: Backfill new status and touched columns - https://phabricator.wikimedia.org/T415064
|
|
2026-02-16 19:18:13
|
<jinxer-wm>
|
FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - pfw1-codfw:reth2 (fasw1-f5 2x25G) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=pfw1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown
|
|
2026-02-16 19:19:25
|
<wikibugs>
|
('CR) ''Brouberol: [C:''+2] superset: update PYTHONPATH to reflect recent change to bookworm [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239730 (https://phabricator.wikimedia.org/T416455) (owner: ''Brouberol)'
|
|
2026-02-16 19:19:30
|
<wikibugs>
|
('CR) ''Brouberol: [C:''+2] superset: release new bookworm-based image [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239731 (https://phabricator.wikimedia.org/T416455) (owner: ''Brouberol)'
|
|
2026-02-16 19:21:42
|
<wikibugs>
|
('Merged) ''jenkins-bot: superset: update PYTHONPATH to reflect recent change to bookworm [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239730 (https://phabricator.wikimedia.org/T416455) (owner: ''Brouberol)'
|
|
2026-02-16 19:21:50
|
<wikibugs>
|
('Merged) ''jenkins-bot: superset: release new bookworm-based image [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239731 (https://phabricator.wikimedia.org/T416455) (owner: ''Brouberol)'
|
|
2026-02-16 19:23:41
|
<wikibugs>
|
('PS1) ''Brouberol: spark-history: update the image to the newwest bookworm-base tag [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239760 (https://phabricator.wikimedia.org/T416455)'
|
|
2026-02-16 19:24:32
|
<logmsgbot>
|
!log brouberol@deploy2002 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset-next: apply
|
|
2026-02-16 19:26:02
|
<logmsgbot>
|
!log brouberol@deploy2002 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset-next: apply
|
|
2026-02-16 19:26:58
|
<logmsgbot>
|
!log brouberol@deploy2002 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset-next: apply
|
|
2026-02-16 19:27:25
|
<logmsgbot>
|
!log brouberol@deploy2002 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset-next: apply
|
|
2026-02-16 19:27:51
|
<logmsgbot>
|
!log brouberol@deploy2002 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset: apply
|
|
2026-02-16 19:28:21
|
<logmsgbot>
|
!log brouberol@deploy2002 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset: apply
|
|
2026-02-16 19:30:16
|
<wikibugs>
|
('CR) ''Brouberol: "recheck" [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239760 (https://phabricator.wikimedia.org/T416455) (owner: ''Brouberol)'
|
|
2026-02-16 19:32:44
|
<wikibugs>
|
('PS1) ''Brouberol: superset: add brouberol to the list of maintainers [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239761'
|
|
2026-02-16 19:33:08
|
<wikibugs>
|
('PS1) ''Zabe: Add small comment pointing to ForeignDBViaLBRepo above file migration [mediawiki-config] - ''https://gerrit.wikimedia.org/r/1239762 (https://phabricator.wikimedia.org/T416548)'
|
|
2026-02-16 19:33:40
|
<wikibugs>
|
('CR) ''Zabe: "Uploaded https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1239762"; [mediawiki-config] - ''https://gerrit.wikimedia.org/r/1239497 (https://phabricator.wikimedia.org/T416548) (owner: ''Zabe)'
|
|
2026-02-16 19:35:40
|
<wikibugs>
|
('CR) ''Brouberol: [C:''+2] spark-history: update the image to the newwest bookworm-base tag [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239760 (https://phabricator.wikimedia.org/T416455) (owner: ''Brouberol)'
|
|
2026-02-16 19:36:01
|
<wikibugs>
|
('CR) ''Brouberol: [C:''+2] superset: add brouberol to the list of maintainers [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239761 (owner: ''Brouberol)'
|
|
2026-02-16 19:36:42
|
<logmsgbot>
|
!log brouberol@deploy2002 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/spark-history: apply
|
|
2026-02-16 19:37:31
|
<logmsgbot>
|
!log brouberol@deploy2002 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/spark-history: apply
|
|
2026-02-16 19:38:16
|
<logmsgbot>
|
!log brouberol@deploy2002 helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/spark-history: apply
|
|
2026-02-16 19:39:06
|
<logmsgbot>
|
!log brouberol@deploy2002 helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/spark-history: apply
|
|
2026-02-16 19:40:19
|
<logmsgbot>
|
!log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1160.eqiad.wmnet with reason: Maintenance
|
|
2026-02-16 19:40:27
|
<logmsgbot>
|
!log marostegui@cumin1003 dbctl commit (dc=all): 'Depooling db1160 (T415786)', diff saved to https://phabricator.wikimedia.org/P88829 and previous config saved to /var/cache/conftool/dbconfig/20260216-194026-marostegui.json
|
|
2026-02-16 19:40:31
|
<stashbot>
|
T415786: Update imagelinks primary key on wmf production - https://phabricator.wikimedia.org/T415786
|
|
2026-02-16 19:42:37
|
<wikibugs>
|
'SRE-Access-Requests: Requesting update of Raymond Ndibe's SSH key to Yubikey-backed key - https://phabricator.wikimedia.org/T417594 (''Raymond_Ndibe) ''NEW'
|
|
2026-02-16 19:43:23
|
<wikibugs>
|
'SRE-Access-Requests: Requesting update of Raymond Ndibe's SSH key to Yubikey-backed key - https://phabricator.wikimedia.org/T417594#11621042 (''Raymond_Ndibe) a:''Raymond_Ndibe→''None'
|
|
2026-02-16 20:19:32
|
<jinxer-wm>
|
FIRING: [2x] ProbeDown: Service wdqs1013:443 has failed probes (http_wdqs_main_external_search_sparql_endpoint_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#wdqs1013:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
|
|
2026-02-16 20:31:40
|
<jinxer-wm>
|
FIRING: SystemdUnitFailed: send_tile_invalidations.service on maps1011:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
|
|
2026-02-16 20:53:29
|
<wikibugs>
|
('CR) ''Daniel Kinzler: "recheck" [deployment-charts] - ''https://gerrit.wikimedia.org/r/1239669 (owner: ''Daniel Kinzler)'
|
|
2026-02-16 20:57:19
|
<cscott>
|
gerrit seems to be pretty uphappy
|
|
2026-02-16 20:57:24
|
<cscott>
|
*unhappy
|
|
2026-02-16 20:57:41
|
<wikibugs>
|
('PS25) ''Daniel Kinzler: rest gateway: add tests for chart rendering [deployment-charts] - ''https://gerrit.wikimedia.org/r/1225085'
|
|
2026-02-16 21:00:05
|
<jouncebot>
|
RoanKattouw, Urbanecm, TheresNoTime, kindrobot, and cjming: I, the Bot under the Fountain, call upon thee, The Deployer, to do UTC late backport window deploy. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260216T2100).
|
|
2026-02-16 21:00:05
|
<jouncebot>
|
No Gerrit patches in the queue for this window AFAICS.
|
|
2026-02-16 21:13:25
|
<jinxer-wm>
|
FIRING: SystemdUnitFailed: wmf_auto_restart_rsyslog.service on ml-serve2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
|
|
2026-02-16 21:41:28
|
<logmsgbot>
|
!log marostegui@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db2147 (T415786)', diff saved to https://phabricator.wikimedia.org/P88830 and previous config saved to /var/cache/conftool/dbconfig/20260216-214127-marostegui.json
|
|
2026-02-16 21:41:32
|
<stashbot>
|
T415786: Update imagelinks primary key on wmf production - https://phabricator.wikimedia.org/T415786
|
|
2026-02-16 21:56:36
|
<logmsgbot>
|
!log marostegui@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P88831 and previous config saved to /var/cache/conftool/dbconfig/20260216-215635-marostegui.json
|
|
2026-02-16 22:00:05
|
<jouncebot>
|
Reedy, sbassett, Maryum, and manfredi: #bothumor When your hammer is PHP, everything starts looking like a thumb. Rise for Weekly Security deployment window. (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20260216T2200).
|
|
2026-02-16 22:11:44
|
<logmsgbot>
|
!log marostegui@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P88832 and previous config saved to /var/cache/conftool/dbconfig/20260216-221143-marostegui.json
|
|
2026-02-16 22:26:52
|
<logmsgbot>
|
!log marostegui@cumin1003 dbctl commit (dc=all): 'Repooling after maintenance db2147 (T415786)', diff saved to https://phabricator.wikimedia.org/P88833 and previous config saved to /var/cache/conftool/dbconfig/20260216-222651-marostegui.json
|
|
2026-02-16 22:26:56
|
<stashbot>
|
T415786: Update imagelinks primary key on wmf production - https://phabricator.wikimedia.org/T415786
|
|
2026-02-16 22:27:08
|
<logmsgbot>
|
!log marostegui@cumin1003 DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2155.codfw.wmnet with reason: Maintenance
|
|
2026-02-16 22:27:18
|
<logmsgbot>
|
!log marostegui@cumin1003 dbctl commit (dc=all): 'Depooling db2155 (T415786)', diff saved to https://phabricator.wikimedia.org/P88834 and previous config saved to /var/cache/conftool/dbconfig/20260216-222716-marostegui.json
|
|
2026-02-16 22:49:37
|
<icinga-wm>
|
PROBLEM - Host wikikube-worker1019 is DOWN: PING CRITICAL - Packet loss = 50%, RTA = 2061.85 ms
|
|
2026-02-16 22:50:13
|
<icinga-wm>
|
RECOVERY - Host wikikube-worker1019 is UP: PING OK - Packet loss = 0%, RTA = 0.31 ms
|
|
2026-02-16 23:18:13
|
<jinxer-wm>
|
FIRING: [2x] CoreRouterInterfaceDown: Core router interface down - pfw1-codfw:reth2 (fasw1-f5 2x25G) - https://wikitech.wikimedia.org/wiki/Network_monitoring#Router_interface_down - https://grafana.wikimedia.org/d/fb403d62-5f03-434a-9dff-bd02b9fff504/network-device-overview?var-instance=pfw1-codfw:9804 - https://alerts.wikimedia.org/?q=alertname%3DCoreRouterInterfaceDown
|