|
2026-02-23 08:07:24
|
<wikibugs>
|
('CR) ''Hashar: [C:''+2] Add TestKitchen dependency to WikimediaEvents [integration/config] - ''https://gerrit.wikimedia.org/r/1241869 (https://phabricator.wikimedia.org/T417068) (owner: ''Kareid)'
|
|
2026-02-23 08:09:10
|
<wikibugs>
|
('Merged) ''jenkins-bot: Add TestKitchen dependency to WikimediaEvents [integration/config] - ''https://gerrit.wikimedia.org/r/1241869 (https://phabricator.wikimedia.org/T417068) (owner: ''Kareid)'
|
|
2026-02-23 08:16:43
|
<wikibugs>
|
'Gerrit, ''collaboration-services: gerrit: replication monitoring improvement - https://phabricator.wikimedia.org/T418084 (''ABran-WMF) ''NEW'
|
|
2026-02-23 08:27:49
|
<wikibugs>
|
'Gerrit, ''collaboration-services, ''Patch-For-Review, ''Puppet: Gerrit git replication should not break when Puppet changes its config - https://phabricator.wikimedia.org/T416929#11639182 (''hashar) The short fix is to [[ https://gerrit.wikimedia.org/r/c/operations/puppet/+/1238043 | disable configuratio...'
|
|
2026-02-23 08:41:54
|
<wikibugs>
|
('CR) ''Hashar: [C:''+2] build: Updating dependencies [integration/docroot] - ''https://gerrit.wikimedia.org/r/1241071 (owner: ''Libraryupgrader)'
|
|
2026-02-23 08:46:36
|
<wikibugs>
|
'Release-Engineering-Team, ''Scap: git pull on a scap deployment repository deletes scap sync tags - https://phabricator.wikimedia.org/T418085 (''hashar) ''NEW'
|
|
2026-02-23 09:03:36
|
<wikibugs>
|
('CR) ''Hashar: [C:''+2] Zuul: [mediawiki/extensions/CollabPads] Use quibble-bluespice [integration/config] - ''https://gerrit.wikimedia.org/r/1241059 (owner: ''Umherirrender)'
|
|
2026-02-23 09:05:17
|
<wikibugs>
|
('Merged) ''jenkins-bot: Zuul: [mediawiki/extensions/CollabPads] Use quibble-bluespice [integration/config] - ''https://gerrit.wikimedia.org/r/1241059 (owner: ''Umherirrender)'
|
|
2026-02-23 09:06:34
|
<wikibugs>
|
'Gerrit, ''collaboration-services, ''Patch-For-Review: gerrit: replication monitoring improvement - https://phabricator.wikimedia.org/T418084#11639274 (''ABran-WMF) p:''Triage→''Medium'
|
|
2026-02-23 09:07:08
|
<wikibugs>
|
('CR) ''Hashar: [C:''+2] "Deployed" [integration/config] - ''https://gerrit.wikimedia.org/r/1241059 (owner: ''Umherirrender)'
|
|
2026-02-23 09:26:46
|
<wikibugs>
|
'Release-Engineering-Team, ''translatewiki.net: MediaWiki translation sync broken - https://phabricator.wikimedia.org/T418087 (''Nikerabbit) ''NEW'
|
|
2026-02-23 09:27:22
|
<wmf-insecte>
|
Project beta-scap-sync-world build #246533: 'FAILURE in 2 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/246533/'
|
|
2026-02-23 09:37:31
|
<wmf-insecte>
|
Yippee, build fixed!
|
|
2026-02-23 09:37:31
|
<wmf-insecte>
|
Project beta-scap-sync-world build #246534: 'FIXED in 2 min 12 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/246534/'
|
|
2026-02-23 09:38:00
|
<wikibugs>
|
'Release-Engineering-Team, ''translatewiki.net: MediaWiki translation sync broken - https://phabricator.wikimedia.org/T418087#11639429 (''Joe) Do you happen to know what is the user-agent used by the script? Or point me to the source code for it.'
|
|
2026-02-23 09:46:15
|
<wikibugs>
|
'Release-Engineering-Team, ''translatewiki.net: MediaWiki translation sync broken - https://phabricator.wikimedia.org/T418087#11639503 (''Joe) Ah seen the script (thanks @Nikerabbit); first thing to do is make the user-agent compliant with the wikimedia User-Agent policy, so it should contain an email address...'
|
|
2026-02-23 09:46:34
|
<wikibugs>
|
'Gerrit, ''Release-Engineering-Team, ''collaboration-services, ''Patch-For-Review: Rename gerrit2 unix user to gerrit and assign a fixed uid - https://phabricator.wikimedia.org/T338470#11639506 (''ABran-WMF)'
|
|
2026-02-23 10:07:37
|
<wikibugs>
|
'Gerrit, ''Release-Engineering-Team, ''collaboration-services: gerrit: gerrit-replica behind CDN - https://phabricator.wikimedia.org/T418108 (''ABran-WMF) ''NEW'
|
|
2026-02-23 10:07:50
|
<wikibugs>
|
'Gerrit, ''Release-Engineering-Team, ''collaboration-services: gerrit: gerrit-replica behind CDN - https://phabricator.wikimedia.org/T418108#11639691 (''ABran-WMF)'
|
|
2026-02-23 10:07:53
|
<wikibugs>
|
'Gerrit, ''Release-Engineering-Team, ''collaboration-services, ''Traffic, ''Patch-For-Review: ATS: align ATS and Gerrit Apache timeouts to reenable connection re-use - https://phabricator.wikimedia.org/T417998#11639690 (''ABran-WMF)'
|
|
2026-02-23 10:09:14
|
<wikibugs>
|
'Gerrit, ''Release-Engineering-Team, ''collaboration-services, ''Traffic, ''Patch-For-Review: ATS: align ATS and Gerrit Apache timeouts to reenable connection re-use - https://phabricator.wikimedia.org/T417998#11639695 (''ABran-WMF) ''Open→''Stalled p:''Triage→''Medium This is currently blocked...'
|
|
2026-02-23 10:09:34
|
<wikibugs>
|
'Gerrit, ''Release-Engineering-Team, ''collaboration-services, ''Traffic, ''Patch-For-Review: ATS: align ATS and Gerrit Apache timeouts to reenable connection re-use - https://phabricator.wikimedia.org/T417998#11639703 (''ABran-WMF) a:''ABran-WMF'
|
|
2026-02-23 10:09:51
|
<wikibugs>
|
'Gerrit, ''Release-Engineering-Team, ''collaboration-services: gerrit: gerrit-replica behind CDN - https://phabricator.wikimedia.org/T418108#11639705 (''ABran-WMF) ''Open→''In progress p:''Triage→''High'
|
|
2026-02-23 10:20:56
|
<wikibugs>
|
'Continuous-Integration-Infrastructure, ''Jenkins, ''collaboration-services: Update Jenkins hosts from Java 17 to Java 21 - https://phabricator.wikimedia.org/T418109 (''jnuche) ''NEW'
|
|
2026-02-23 10:51:17
|
<wikibugs>
|
'Gerrit, ''collaboration-services, ''Patch-For-Review: gerrit: replication monitoring improvement - https://phabricator.wikimedia.org/T418084#11639819 (''hashar) From my comment on https://gerrit.wikimedia.org/r/c/operations/puppet/+/1238315/comments/a98b94a7_a77584ef , the Gerrit replication plugin
exposes...'
|
|
2026-02-23 11:19:39
|
<wikibugs>
|
'Gerrit, ''collaboration-services, ''Patch-For-Review: gerrit: replication monitoring improvement - https://phabricator.wikimedia.org/T418084#11639934 (''hashar) I have also added the per replicas retries (whatever it means) counts and rate: {F72279291 width=500}'
|
|
2026-02-23 11:30:28
|
<wmcs-alerts>
|
FIRING: PuppetAgentFailure: Puppet agent failure detected on instance deployment-puppetserver-1 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure
|
|
2026-02-23 11:30:38
|
<wikibugs>
|
'Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T418120 (''wmcs-alerts) ''NEW'
|
|
2026-02-23 11:32:20
|
<wikibugs>
|
'Gerrit-Privilege-Requests, ''Release-Engineering-Team, ''Security-Team, ''SRE, ''SRE-Access-Requests: Request membership in wmf-deployment group for Rsilvola - https://phabricator.wikimedia.org/T418004#11640007 (''Rsilvola)'
|
|
2026-02-23 11:32:56
|
<wikibugs>
|
'Gerrit-Privilege-Requests, ''Release-Engineering-Team, ''Security-Team, ''SRE, ''SRE-Access-Requests: Request membership in wmf-deployment group for Rsilvola - https://phabricator.wikimedia.org/T418004#11640023 (''Rsilvola)
''Open→''Declined Hello @Dzahn, Much of this was already filled in [T4...'
|
|
2026-02-23 12:04:40
|
<wikibugs>
|
'Release-Engineering-Team, ''translatewiki.net: MediaWiki translation sync broken - https://phabricator.wikimedia.org/T418087#11640145 (''Joe) p:''Triage→''Medium a:''Joe'
|
|
2026-02-23 12:05:03
|
<wmf-insecte>
|
maintenance-disconnect-full-disks build 783601 integration-agent-docker-1057 (/: 37%, /srv: 97%, /var/lib/docker: 28%): OFFLINE due to disk space
|
|
2026-02-23 12:05:11
|
<wmf-insecte>
|
Project beta-code-update-eqiad build #588867: 'FAILURE in 14 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/588867/'
|
|
2026-02-23 12:10:02
|
<wmf-insecte>
|
maintenance-disconnect-full-disks build 783602 integration-agent-docker-1057 (/: 37%, /srv: 83%, /var/lib/docker: 27%): RECOVERY disk space OK
|
|
2026-02-23 12:15:17
|
<wmf-insecte>
|
Yippee, build fixed!
|
|
2026-02-23 12:15:18
|
<wmf-insecte>
|
Project beta-code-update-eqiad build #588868: 'FIXED in 2 min 17 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/588868/'
|
|
2026-02-23 12:22:58
|
<wikibugs>
|
'Phabricator, ''Release-Engineering-Team (Priority Backlog 📥): Decrease number of open Phab tickets with assignee field set for more than two years (aka cookie licking) (Q2/2026 edition) - https://phabricator.wikimedia.org/T418127 (''Aklapper) ''NEW p:''Triage→''Low'
|
|
2026-02-23 12:23:01
|
<wikibugs>
|
'Phabricator, ''Release-Engineering-Team (Doing 😎): Decrease number of open Phab tickets with assignee field set for more than two years (aka cookie licking) (Q4/2025 edition) - https://phabricator.wikimedia.org/T397713#11640303 (''Aklapper) ''Stalled→''Resolved'
|
|
2026-02-23 12:25:42
|
<wikibugs>
|
('open) ''aklapper: Uninstall Countdown application [repos/phabricator/deployment] (wmf/stable) - ''https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/95 (https://phabricator.wikimedia.org/T418033)'
|
|
2026-02-23 12:26:19
|
<wikibugs>
|
'Phabricator, ''Patch-For-Review: Uninstall Countdown (Phabricator application) - https://phabricator.wikimedia.org/T418033#11640312 (''Aklapper)'
|
|
2026-02-23 13:13:21
|
<wikibugs>
|
'Release-Engineering-Team, ''translatewiki.net: MediaWiki translation sync broken - https://phabricator.wikimedia.org/T418087#11640435 (''Joe) ''Open→''Resolved Added an exception for translatewiki so they can keep syncing from gerrit at the previous rate, given that never created an issue for
us.'
|
|
2026-02-23 13:14:47
|
<wikibugs>
|
'Diffusion, ''Phabricator, ''Release-Engineering-Team (Priority Backlog 📥): Disable "diffusion.allow-http-auth" - https://phabricator.wikimedia.org/T418045#11640440 (''Aklapper)'
|
|
2026-02-23 13:16:12
|
<wikibugs>
|
('open) ''aklapper: Remove / disable diffusion.allow-http-auth [repos/phabricator/deployment] (wmf/stable) - ''https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/96 (https://phabricator.wikimedia.org/T418045)'
|
|
2026-02-23 13:16:56
|
<wikibugs>
|
'Gerrit, ''collaboration-services, ''Patch-For-Review: gerrit: replication monitoring improvement - https://phabricator.wikimedia.org/T418084#11640451 (''ABran-WMF) Thanks @hashar for the graphs! The issue we had was happening between `12:xx` and `18:xx` and I found nothing reflecting that in these graphs....'
|
|
2026-02-23 13:40:27
|
<wikibugs>
|
'Gerrit, ''collaboration-services, ''Patch-For-Review: gerrit: replication monitoring improvement - https://phabricator.wikimedia.org/T418084#11640533 (''ABran-WMF) I think I found something interesting on the dashboard [[ https://gerrit.wikimedia.org/r/c/operations/puppet/+/1238315/comments/a98b94a7_a77584...'
|
|
2026-02-23 13:45:02
|
<wmf-insecte>
|
maintenance-disconnect-full-disks build 783621 integration-agent-docker-1054 (/: 51%, /srv: 98%, /var/lib/docker: 22%): OFFLINE due to disk space
|
|
2026-02-23 13:50:03
|
<wmf-insecte>
|
maintenance-disconnect-full-disks build 783622 integration-agent-docker-1054 (/: 51%, /srv: 66%, /var/lib/docker: 21%): RECOVERY disk space OK
|
|
2026-02-23 13:50:03
|
<wmf-insecte>
|
maintenance-disconnect-full-disks build 783622 integration-agent-docker-1051 (/: 36%, /srv: 97%, /var/lib/docker: 24%): OFFLINE due to disk space
|
|
2026-02-23 13:55:03
|
<wmf-insecte>
|
maintenance-disconnect-full-disks build 783623 integration-agent-docker-1051 (/: 36%, /srv: 56%, /var/lib/docker: 24%): RECOVERY disk space OK
|
|
2026-02-23 14:00:09
|
<wikibugs>
|
'Gerrit, ''collaboration-services, ''Patch-For-Review: gerrit: replication monitoring improvement - https://phabricator.wikimedia.org/T418084#11640604 (''ABran-WMF)'
|
|
2026-02-23 14:37:17
|
<Krinkle>
|
https://etherpad-backup.toolforge.org/p/-1-to-100-Team
|
|
2026-02-23 14:37:27
|
<Krinkle>
|
" [Chad] - Testing *is* faster, we fixed that ~2 weeks ago. Tests should only take 10-15mins for the full suite now, instead of 2-3 hours."
|
|
2026-02-23 14:37:55
|
<Krinkle>
|
Puts things in perspective :)
|
|
2026-02-23 14:46:48
|
<wikibugs>
|
'Phabricator, ''Release-Engineering-Team (Doing 😎), ''Tool-gitlab-account-approval: Mark glaab Phabricator account as bot in database - https://phabricator.wikimedia.org/T407690#11640711 (''Aklapper) ''Open→''Resolved a:''Aklapper Done. FYI Conduit logs: https://phabricator.wikimedia.org/conduit/...'
|
|
2026-02-23 14:50:34
|
<wikibugs>
|
'Phabricator, ''Release-Engineering-Team (Doing 😎): Mark Pywikibugs Phabricator account as bot in database - https://phabricator.wikimedia.org/T410571#11640720 (''Aklapper) ''Open→''Resolved a:''Aklapper Done. FYI Conduit logs: https://phabricator.wikimedia.org/conduit/log/query/3QErnZRvzCYM/#R'
|
|
2026-02-23 14:52:04
|
<wikibugs>
|
'Gerrit, ''collaboration-services: gerrit: nft_throttling_denylists triggers NodeTextfileStale alert - https://phabricator.wikimedia.org/T418139 (''ABran-WMF) ''NEW'
|
|
2026-02-23 14:53:17
|
<wikibugs>
|
'Gerrit, ''collaboration-services, ''Patch-For-Review: gerrit: nft_throttling_denylists triggers NodeTextfileStale alert - https://phabricator.wikimedia.org/T418139#11640750 (''ABran-WMF) ''Open→''In progress p:''Triage→''Low'
|
|
2026-02-23 14:53:26
|
<wikibugs>
|
'Phabricator, ''Release-Engineering-Team (Doing 😎), ''ReleaseTaggerBot: Mark ReleaseTaggerBot Phabricator account as bot in database - https://phabricator.wikimedia.org/T329748#11640753 (''Aklapper) ''Open→''Resolved a:''Aklapper Done. FYI hourly Conduit logs: https://phabricator.wikimedia.org/co...'
|
|
2026-02-23 15:01:57
|
<wikibugs>
|
'Gerrit, ''Release-Engineering-Team, ''collaboration-services, ''Patch-For-Review: gerrit: gerrit-replica behind CDN - https://phabricator.wikimedia.org/T418108#11640773 (''ABran-WMF)'
|
|
2026-02-23 15:03:22
|
<wikibugs>
|
'Gerrit, ''collaboration-services, ''Traffic, ''ci-test-error (WMF-deployed Build Failure), ''Patch-For-Review: ATS causes git fetches from Gerrit to fail with 502 responses - https://phabricator.wikimedia.org/T417536#11640775 (''ABran-WMF)'
|
|
2026-02-23 15:03:23
|
<wikibugs>
|
'Gerrit, ''Release-Engineering-Team, ''collaboration-services, ''Patch-For-Review: gerrit: gerrit-replica behind CDN - https://phabricator.wikimedia.org/T418108#11640776 (''ABran-WMF)'
|
|
2026-02-23 15:03:24
|
<wikibugs>
|
'Phabricator, ''Release-Engineering-Team (Priority Backlog 📥): Mark DeploymentCalendarTool Phabricator account as bot in database - https://phabricator.wikimedia.org/T418140 (''Aklapper) ''NEW p:''Triage→''Low'
|
|
2026-02-23 15:03:24
|
<wikibugs>
|
'Gerrit, ''Release-Engineering-Team, ''collaboration-services, ''Traffic, ''Patch-For-Review: ATS: align ATS and Gerrit Apache timeouts to reenable connection re-use - https://phabricator.wikimedia.org/T417998#11640777 (''ABran-WMF)'
|
|
2026-02-23 15:03:31
|
<wikibugs>
|
'Phabricator, ''Release-Engineering-Team (Priority Backlog 📥): Mark Wmdephabbot Phabricator account as bot in database - https://phabricator.wikimedia.org/T418141 (''Aklapper) ''NEW p:''Triage→''Low'
|
|
2026-02-23 15:32:05
|
<wikibugs>
|
'Continuous-Integration-Config, ''Continuous-Integration-Infrastructure, ''Release-Engineering-Team, ''Zuul, and 3 others: Make puppet-compiler execution run with higher priority, not like other 'experimental' jobs - https://phabricator.wikimedia.org/T414621#11640983 (''LSobanski) p:''Triage→''Medium'
|
|
2026-02-23 15:32:30
|
<wikibugs>
|
('CR) ''Jforrester: "Yes, apparently it's still in active development in that repo as well as pulled into thumbor (?) somewhere." [integration/config] - ''https://gerrit.wikimedia.org/r/1240998 (owner: ''Jforrester)'
|
|
2026-02-23 15:33:25
|
<wikibugs>
|
'Continuous-Integration-Infrastructure, ''Release-Engineering-Team, ''Infrastructure-Foundations: Upgrade releng/operations-puppet CI image from Bullseye to Bookworm - https://phabricator.wikimedia.org/T417976#11640984 (''LSobanski)'
|
|
2026-02-23 15:33:43
|
<wikibugs>
|
'Continuous-Integration-Infrastructure, ''Release-Engineering-Team, ''Infrastructure-Foundations: Upgrade releng/operations-puppet CI image from Bullseye to Bookworm - https://phabricator.wikimedia.org/T417976#11640985 (''LSobanski) p:''Triage→''Medium'
|
|
2026-02-23 15:50:40
|
<wikibugs>
|
'Continuous-Integration-Infrastructure, ''Release-Engineering-Team (Seen), ''Infrastructure-Foundations: Make it possible to run the mediawiki testsuite against a staging repo of apt.wikimedia.org - https://phabricator.wikimedia.org/T157038#11641071 (''MoritzMuehlenhoff) ''Open→''Declined This is no...'
|
|
2026-02-23 16:17:41
|
<wikibugs>
|
'Gerrit, ''collaboration-services, ''Patch-For-Review: gerrit: nft_throttling_denylists triggers NodeTextfileStale alert - https://phabricator.wikimedia.org/T418139#11641285 (''ABran-WMF)'
|
|
2026-02-23 16:24:40
|
<wikibugs>
|
('merge) ''jforrester: Remove leading newline from the "what" parameters in the calendar [repos/releng/release] - ''https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/235 (owner: ''matmarex)'
|
|
2026-02-23 16:25:51
|
<wikibugs>
|
('CR) ''Jforrester: [C:''+2] Zuul: [3d2png] Add basic Node CI at version 20 [integration/config] - ''https://gerrit.wikimedia.org/r/1240998 (owner: ''Jforrester)'
|
|
2026-02-23 16:28:10
|
<wikibugs>
|
('Merged) ''jenkins-bot: Zuul: [3d2png] Add basic Node CI at version 20 [integration/config] - ''https://gerrit.wikimedia.org/r/1240998 (owner: ''Jforrester)'
|
|
2026-02-23 16:29:02
|
<James_F>
|
!log Zuul: [3d2png] Add basic Node CI at version 20
|
|
2026-02-23 16:29:03
|
<stashbot>
|
Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
|
|
2026-02-23 16:51:13
|
<wikibugs>
|
'Continuous-Integration-Infrastructure (Zuul upgrade), ''Gerrit, ''Release-Engineering-Team, ''collaboration-services: Fix up Gerrit sshd.idleTimeout - https://phabricator.wikimedia.org/T417996#11641444 (''LSobanski) p:''Triage→''Medium'
|
|
2026-02-23 16:53:19
|
<wikibugs>
|
'Continuous-Integration-Infrastructure (Zuul upgrade), ''Gerrit, ''Release-Engineering-Team, ''collaboration-services: Fix up Gerrit sshd.idleTimeout - https://phabricator.wikimedia.org/T417996#11641455 (''LSobanski) Is https://gerrit.wikimedia.org/r/c/operations/puppet/+/1240243 related?'
|
|
2026-02-23 16:58:57
|
<wikibugs>
|
'Gerrit, ''Release-Engineering-Team, ''collaboration-services: Create an OpenSearch dashboard for Gerrit sshd logs - https://phabricator.wikimedia.org/T417753#11641502 (''LSobanski) p:''Triage→''Medium'
|
|
2026-02-23 17:10:24
|
<wikibugs>
|
('CR) ''Hnowlan: "In a less-than-ideal flow, when thumbor images are built, 3d2png's `master` is also installed to use the system nodejs (currently bookworm" [integration/config] - ''https://gerrit.wikimedia.org/r/1240998 (owner: ''Jforrester)'
|
|
2026-02-23 17:29:26
|
<wikibugs>
|
('open) ''dancy: Remove redundant php_entrypoint.sh [repos/releng/dev-images] - ''https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/91'
|
|
2026-02-23 17:29:28
|
<wikibugs>
|
('update) ''dancy: Remove redundant php_entrypoint.sh [repos/releng/dev-images] - ''https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/91'
|
|
2026-02-23 17:36:32
|
<wikibugs>
|
('update) ''dancy: Use direct reference to php_entrypoint.sh instead of symlink [repos/releng/dev-images] - ''https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/91'
|
|
2026-02-23 17:36:35
|
<wikibugs>
|
('update) ''dancy: Use direct reference to php_entrypoint.sh instead of symlink [repos/releng/dev-images] - ''https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/91'
|
|
2026-02-23 17:45:57
|
<wikibugs>
|
('close) ''dancy: Use direct reference to php_entrypoint.sh instead of symlink [repos/releng/dev-images] - ''https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/91'
|
|
2026-02-23 17:45:58
|
<wikibugs>
|
('update) ''dancy: Use direct reference to php_entrypoint.sh instead of symlink [repos/releng/dev-images] - ''https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/91'
|
|
2026-02-23 18:19:05
|
<wikibugs>
|
('merge) ''aklapper: Hide unneeded "SSH Public Keys" personal settings panel [repos/phabricator/phabricator] (wmf/stable) - ''https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/113 (https://phabricator.wikimedia.org/T418044)'
|
|
2026-02-23 18:19:40
|
<wikibugs>
|
'Phabricator (phabricator-next), ''Release-Engineering-Team (Doing 😎): Hide unneeded "SSH Public Keys" personal settings panel - https://phabricator.wikimedia.org/T418044#11641855 (''Aklapper)'
|
|
2026-02-23 18:20:03
|
<wmf-insecte>
|
maintenance-disconnect-full-disks build 783676 integration-agent-docker-1041 (/: 37%, /srv: 97%, /var/lib/docker: 23%): OFFLINE due to disk space
|
|
2026-02-23 18:20:03
|
<wmf-insecte>
|
maintenance-disconnect-full-disks build 783676 integration-agent-docker-1051 (/: 36%, /srv: 97%, /var/lib/docker: 26%): OFFLINE due to disk space
|
|
2026-02-23 18:25:03
|
<wmf-insecte>
|
maintenance-disconnect-full-disks build 783677 integration-agent-docker-1041 (/: 37%, /srv: 84%, /var/lib/docker: 22%): RECOVERY disk space OK
|
|
2026-02-23 18:25:03
|
<wmf-insecte>
|
maintenance-disconnect-full-disks build 783677 integration-agent-docker-1051 (/: 36%, /srv: 69%, /var/lib/docker: 26%): RECOVERY disk space OK
|
|
2026-02-23 18:45:15
|
<MatmaRex>
|
we don't have database backups for the beta cluster, do we?
|
|
2026-02-23 18:45:43
|
<MatmaRex>
|
(i didn't break anything too badly)
|
|
2026-02-23 18:47:15
|
<bd808>
|
MatmaRex: we don't as far as I know. There was once a deliberately lagged replica but I think that may be gone these days.
|
|
2026-02-23 18:49:00
|
<MatmaRex>
|
alright. fyi, i ran a maintenance script: https://phabricator.wikimedia.org/P88982 and i think the output is lying to me. i'd like to find out what exactly it did, but i guess we'll never know
|
|
2026-02-23 19:07:20
|
<wikibugs>
|
'Gerrit-Privilege-Requests, ''Release-Engineering-Team, ''Security-Team, ''SRE, ''SRE-Access-Requests: Request membership in wmf-deployment group for Rsilvola - https://phabricator.wikimedia.org/T418004#11642002 (''Rsilvola) ''Declined→''Open (reopening)'
|
|
2026-02-23 19:31:36
|
<wikibugs>
|
'Release-Engineering-Team (Doing), ''MediaWiki-extensions-InterwikiSorting, ''MW-1.46-notes (1.46.0-wmf.16; 2026-02-17), ''Patch-For-Review, and 2 others: Undeploy the InterwikiSorting extension from Wikipedia production - https://phabricator.wikimedia.org/T253764#11642138 (''Iniquity) One of the
main...'
|
|
2026-02-23 20:26:45
|
<hashar>
|
!log Deleted "replication-upstream" Grafana dashboard in favor of a copy/new "replication" one. https://grafana.wikimedia.org/d/RFLS1GsWk/replication-upstream , replaced it by https://grafana.wikimedia.org/d/d4a4da73-c27f-4ce6-a9e5-ab84dd7a4ebb/replication
|
|
2026-02-23 20:26:46
|
<stashbot>
|
Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
|
|
2026-02-23 20:27:25
|
<wikibugs>
|
'Release-Engineering-Team, ''dev-images, ''Catalyst (Radar): dev-images: bookworm-php-sury children fail to build on sury key - https://phabricator.wikimedia.org/T417711#11642333 (''dancy) a:''dancy I'm running into this problem today so I'll take this ticket.'
|
|
2026-02-23 20:27:30
|
<wikibugs>
|
'Release-Engineering-Team, ''dev-images, ''Catalyst (Radar): dev-images: bookworm-php-sury children fail to build on sury key - https://phabricator.wikimedia.org/T417711#11642335 (''dancy) ''Open→''In progress p:''Triage→''Medium'
|
|
2026-02-23 20:50:36
|
<wikibugs>
|
('open) ''dancy: bookworm-php-sury: Update sury-php GPG key [repos/releng/dev-images] - ''https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/92 (https://phabricator.wikimedia.org/T417711)'
|
|
2026-02-23 20:50:37
|
<wikibugs>
|
('update) ''dancy: bookworm-php-sury: Update sury-php GPG key [repos/releng/dev-images] - ''https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/92 (https://phabricator.wikimedia.org/T417711)'
|
|
2026-02-23 20:52:26
|
<wikibugs>
|
('update) ''dancy: bookworm-php-sury: Update sury-php GPG key [repos/releng/dev-images] - ''https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/92 (https://phabricator.wikimedia.org/T417711)'
|
|
2026-02-23 21:16:39
|
<wikibugs>
|
('update) ''dancy: bookworm-php-sury: Update sury-php GPG key [repos/releng/dev-images] - ''https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/92 (https://phabricator.wikimedia.org/T417711)'
|
|
2026-02-23 21:16:39
|
<wikibugs>
|
('open) ''dancy: php_entrypoint.sh: Run php-fpm under tini [repos/releng/dev-images] - ''https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/93'
|
|
2026-02-23 21:16:39
|
<wikibugs>
|
('update) ''dancy: php_entrypoint.sh: Run php-fpm under tini [repos/releng/dev-images] - ''https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/93'
|
|
2026-02-23 21:16:46
|
<wikibugs>
|
('update) ''dancy: php_entrypoint.sh: Run php-fpm under tini [repos/releng/dev-images] - ''https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/93'
|
|
2026-02-23 21:16:46
|
<wikibugs>
|
('update) ''dancy: bookworm-php-sury: Update sury-php GPG key [repos/releng/dev-images] - ''https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/92 (https://phabricator.wikimedia.org/T417711)'
|
|
2026-02-23 21:19:00
|
<wikibugs>
|
('update) ''dancy: php_entrypoint.sh: Run php-fpm under tini [repos/releng/dev-images] - ''https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/93'
|
|
2026-02-23 21:24:00
|
<wikibugs>
|
('CR) ''Dmaza: [C:''+1] "What would be the ideal flow?" [integration/config] - ''https://gerrit.wikimedia.org/r/1240998 (owner: ''Jforrester)'
|
|
2026-02-23 21:42:25
|
<wikibugs>
|
('approved) ''jhuneidi: bookworm-php-sury: Update sury-php GPG key [repos/releng/dev-images] - ''https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/92 (https://phabricator.wikimedia.org/T417711) (owner: ''dancy)'
|
|
2026-02-23 21:42:54
|
<wikibugs>
|
('update) ''dancy: php_entrypoint.sh: Run php-fpm under tini [repos/releng/dev-images] - ''https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/93'
|
|
2026-02-23 21:42:54
|
<wikibugs>
|
('merge) ''dancy: bookworm-php-sury: Update sury-php GPG key [repos/releng/dev-images] - ''https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/92 (https://phabricator.wikimedia.org/T417711)'
|
|
2026-02-23 21:54:25
|
<Kemayo>
|
Now I've gotten my mitigation patch for this deployed and have time to look at secondary issues... anyone here in a position to have an idea what's going on with https://api.wikimedia.org/service/lw/inference/v1/models/edit-check:predict 404ing?
|
|
2026-02-23 22:12:03
|
<bd808>
|
!log Unblock 191.80.192.0/18 (T418132)
|
|
2026-02-23 22:12:04
|
<stashbot>
|
Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
|
|
2026-02-23 22:18:29
|
<wikibugs>
|
('update) ''dduvall: digitalocean: Separate management of cluster and in-cluster resources [repos/releng/gitlab-cloud-runner] - ''https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/552 (https://phabricator.wikimedia.org/T416260)'
|
|
2026-02-23 22:18:32
|
<wikibugs>
|
('update) ''dduvall: digitalocean: Separate management of cluster and in-cluster resources [repos/releng/gitlab-cloud-runner] - ''https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/552 (https://phabricator.wikimedia.org/T416260)'
|
|
2026-02-23 22:23:26
|
<wikibugs>
|
'Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T418120#11642781 (''bd808) `lang=shell-session,counterexample bd808@deployment-puppetserver-1.deployment-prep.eqiad1:~$ sudo -i puppet agent -tv Info:...'
|
|
2026-02-23 22:35:44
|
<MatmaRex>
|
sorry, it's me again… is the beta cluster able to send emails? specifically, emails from the Special:ConfirmEmail feature? i am trying to change my email address there and haven't gotten a confirmation email
|
|
2026-02-23 22:41:48
|
<wikibugs>
|
'Gerrit, ''collaboration-services, ''Patch-For-Review: gerrit: replication monitoring improvement - https://phabricator.wikimedia.org/T418084#11642836 (''hashar) Awesome! For reference the metric is: ` events_ref_replication_scheduled_total - events_ref_replicated_total ` I could not explain those two down...'
|
|
2026-02-23 22:42:31
|
<dancy>
|
MatmaRex: I believe so
|
|
2026-02-23 22:42:49
|
<wikibugs>
|
('open) ''jhuneidi: Draft: catalyst-*: Refresh for gpg key update [repos/releng/dev-images] - ''https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/94 (https://phabricator.wikimedia.org/T417711)'
|
|
2026-02-23 22:43:04
|
<dancy>
|
!log Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/92
|
|
2026-02-23 22:43:05
|
<stashbot>
|
Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
|
|
2026-02-23 22:44:19
|
<bd808>
|
MatmaRex: in theory yes, but T212327 might still be a thing.
|
|
2026-02-23 22:44:20
|
<stashbot>
|
T212327: Beta Cluster mailer not sending emails to @wikimedia.org addresses - https://phabricator.wikimedia.org/T212327
|
|
2026-02-23 22:44:48
|
<MatmaRex>
|
yeah, i just found that… T285527 and T291679 too
|
|
2026-02-23 22:44:49
|
<stashbot>
|
T285527: Unable to confirm email address on beta cluster - https://phabricator.wikimedia.org/T285527
|
|
2026-02-23 22:44:49
|
<stashbot>
|
T291679: "Sender address rejected: Domain not found" for emails sent from the beta cluster - https://phabricator.wikimedia.org/T291679
|
|
2026-02-23 22:45:01
|
<MatmaRex>
|
fwiw i used a @fastmail.com address
|
|
2026-02-23 22:51:25
|
<bd808>
|
MatmaRex: I don't see any stuck mails outbound to fastmail on deployment-mx04. The message might not have gotten that far though. Did you look for mediawiki logs?
|
|
2026-02-23 22:52:50
|
<MatmaRex>
|
yup, haven't found anything relevant, except "Changing email address for Yatu from A to B"
|
|
2026-02-23 22:53:14
|
<wikibugs>
|
'Gerrit, ''collaboration-services, ''Patch-For-Review: gerrit: replication monitoring improvement - https://phabricator.wikimedia.org/T418084#11642858 (''hashar) Another thing I looked at tonight is the delay/latency graph. They show the values for different quantiles which remains when there is no replicat...'
|
|
2026-02-23 22:53:30
|
<MatmaRex>
|
this guy seems smart and proposed a solution (back in 2021): https://phabricator.wikimedia.org/T291679#7474212
|
|
2026-02-23 22:55:01
|
<MatmaRex>
|
i guess i could do what he says? it probably won't break anything worse than it is. i have to say i don't know what i'm doing though
|
|
2026-02-23 22:58:17
|
<bd808>
|
There is a floating IP setup in the Deployment-prep Cloud VPS project with the description "For MX server".
|
|
2026-02-23 22:58:33
|
<bd808>
|
looks closer
|
|
2026-02-23 22:59:37
|
<bd808>
|
huh. not mapped to any instance and the Horizon UI doesn't give me any logging about the history of that object
|
|
2026-02-23 23:00:00
|
<wikibugs>
|
'Gerrit, ''collaboration-services, ''Patch-For-Review: gerrit: replication monitoring improvement - https://phabricator.wikimedia.org/T418084#11642872 (''hashar) Also over the week-end I felt like we should have a view of the replication plugin WorkQueue, it does not emit metrics which I think was forgotten...'
|
|
2026-02-23 23:00:38
|
<bd808>
|
it does reverse to "mail.beta.wmflabs.org", "wikimedia.beta.wmflabs.org", and "wikimedia.beta.wmcloud.org"
|
|
2026-02-23 23:01:13
|
<bd808>
|
Beta is a hot mess and I don't know what to do about that
|
|
2026-02-23 23:06:38
|
<hashar>
|
bd808: Beta got used to validate/pretest an extension that deals with emails and or for a mx migration maybe?
|
|
2026-02-23 23:07:02
|
<jeena>
|
!log Updated development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/92
|
|
2026-02-23 23:07:03
|
<stashbot>
|
Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
|
|
2026-02-23 23:07:22
|
<MatmaRex>
|
i'll try doing the thing chlod suggested and see if it does anything
|
|
2026-02-23 23:07:25
|
<hashar>
|
Puppet has a bunch of history for ""deployment-mx""
|
|
2026-02-23 23:07:55
|
<hashar>
|
and I think anything email related can most probably be withdrawn in favor of using whatever email relay is available inside of WMCS
|
|
2026-02-23 23:08:06
|
<bd808>
|
hashar: I think mostly the config rotted over time and nobody at all tries to fix it because, well, it's beta and beta is broken
|
|
2026-02-23 23:08:35
|
<hashar>
|
well sometime it is helpful :]
|
|
2026-02-23 23:08:55
|
<bd808>
|
"just use another MX" is the sort of thing everyone says and nobody tries
|
|
2026-02-23 23:08:57
|
<hashar>
|
I once found a bug in our Varnish VCL because beta upgraded automatically
|
|
2026-02-23 23:09:30
|
<bd808>
|
is in the middle of writing up an existing Puppet bug for Beta
|
|
2026-02-23 23:09:32
|
<hashar>
|
I think the deployment-mx was made because some wanted to reproduced every single bit of infra we had
|
|
2026-02-23 23:09:40
|
<hashar>
|
or maybe there was a use case for testing something involving the mx
|
|
2026-02-23 23:09:52
|
<bd808>
|
I know it was made because Cloud VPS did not have a shared MX
|
|
2026-02-23 23:10:02
|
<hashar>
|
AH!
|
|
2026-02-23 23:10:11
|
<hashar>
|
that sounds like a reasonable reason to set one up
|
|
2026-02-23 23:10:25
|
<bd808>
|
sahred MX servers are hard
|
|
2026-02-23 23:10:37
|
<hashar>
|
nods
|
|
2026-02-23 23:10:47
|
<hashar>
|
I am SO happy to not have to manage an email server anymore
|
|
2026-02-23 23:11:08
|
<hashar>
|
or a bind dns server :]
|
|
2026-02-23 23:11:38
|
<bd808>
|
I have bind and sendmail books from the 90s on my shelf here...
|
|
2026-02-23 23:12:34
|
<hashar>
|
smiles
|
|
2026-02-23 23:12:41
|
<hashar>
|
we are getting old
|
|
2026-02-23 23:13:16
|
<hashar>
|
but I managed to mess up with some Grafana dashboard / PromQL and digging into metrics this evening
|
|
2026-02-23 23:13:23
|
<hashar>
|
so I am quite happy :] I am not THAT old afterall!
|
|
2026-02-23 23:13:30
|
<hashar>
|
https://grafana-rw.wikimedia.org/d/d4a4da73-c27f-4ce6-a9e5-ab84dd7a4ebb/replication !
|
|
2026-02-23 23:14:31
|
<MatmaRex>
|
well, i did this: https://horizon.wikimedia.org/ngdetails/OS::Designate::RecordSet/4482e09c-3d25-447f-b8e2-5aa2a105b60e/99124be1-c471-4c74-b2f9-b4288ae3447c it doesn't seem to have helped
|
|
2026-02-23 23:16:00
|
<wikibugs>
|
'Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-puppetserver-1 in project deployment-prep - https://phabricator.wikimedia.org/T418120#11642920 (''bd808) `lang=shell-session bd808@mbp03:~/projects/wmf/operations/puppet$ git log --after=2026-02-21 $(git grep -l analytics-sre) 94...'
|
|
2026-02-23 23:26:11
|
<bd808>
|
MatmaRex: I think we should delete that rather than leaving it around to possibly confuse folks in the future. Pointing it at an outbound MX would I think only help if the outside world was just desperate for any type=MX result too.
|
|
2026-02-23 23:26:57
|
<MatmaRex>
|
bd808: i was just about to comment on the task with a link, to see if anyone there is interested in helping debug it
|
|
2026-02-23 23:26:57
|
<bd808>
|
is working on getting puppet to work on the puppetserver at the moment
|
|
2026-02-23 23:28:45
|
<bd808>
|
MatmaRex: ok. maybe add a note to the record that points back to the phab task then?
|
|
2026-02-23 23:29:01
|
<bd808>
|
and I see you did that/ never mind
|
|
2026-02-23 23:29:01
|
<MatmaRex>
|
i did
|
|
2026-02-23 23:29:04
|
<MatmaRex>
|
unless i messed it up
|
|
2026-02-23 23:29:07
|
<wikibugs>
|
'Beta-Cluster-Infrastructure: "Sender address rejected: Domain not found" for emails sent from the beta cluster - https://phabricator.wikimedia.org/T291679#11642952 (''matmarex) Hi, I found this task, as I am also unable to confirm my email address on the beta cluster. >>! In T291679#7474212, @Chlod wrote: >...'
|
|
2026-02-23 23:29:07
|
<MatmaRex>
|
ah :)
|
|
2026-02-23 23:29:16
|
<MatmaRex>
|
i wasn't sure if putting a URL in that field would break anything
|
|
2026-02-23 23:29:39
|
<MatmaRex>
|
but i did that now
|
|
2026-02-23 23:29:45
|
<bd808>
|
the full url should be fine. that's just a database field and not really DNS anything
|
|
2026-02-23 23:31:49
|
<bd808>
|
I see there is an SPF record for wikimedia.beta.wmcloud.org that points to the unused floating IP. That's probably not helping things either.
|
|
2026-02-23 23:40:00
|
<wikibugs>
|
'Beta-Cluster-Infrastructure: "Sender address rejected: Domain not found" for emails sent from the beta cluster - https://phabricator.wikimedia.org/T291679#11642973 (''LucasWerkmeister) Well, I just tried sending an email to myself via Special:EmailUser and it worked fine. So from my side this seems to have wor...'
|
|
2026-02-23 23:55:37
|
<wikibugs>
|
'Beta-Cluster-Infrastructure: "Sender address rejected: Domain not found" for emails sent from the beta cluster - https://phabricator.wikimedia.org/T291679#11642995 (''matmarex) It depends on the recipient address. I can receive the emails at an @gmail.com address (and I could receive them earlier today as well...'
|