|
2025-08-19 00:02:45
|
<wikibugs>
|
'Project-Admins: Create project tag for Wikinews volunteer developed software - https://phabricator.wikimedia.org/T402150#11096576 (''Gryllida) I already provided links and a description. "Project tag for volunteers to coordinate development of wiki software, including mediawiki, its extensions, and third party...'
|
|
2025-08-19 00:03:23
|
<wikibugs>
|
'Project-Admins: Create project tag for Wikinews volunteer developed software - https://phabricator.wikimedia.org/T402150#11096577 (''Gryllida) "The only thing blocking this at the moment is the project description." 😢'
|
|
2025-08-19 00:14:26
|
<wikibugs>
|
'Project-Admins: Create project tag for Wikinews volunteer developed software - https://phabricator.wikimedia.org/T402150#11096597 (''thcipriani) ''Open→''Resolved a:''thcipriani >>! In T402150#11096576, @Gryllida wrote: > I already provided links and a description. "Project tag for volunteers to
coor...'
|
|
2025-08-19 01:16:13
|
<wikibugs>
|
('open) ''swfrench: Draft: make-container-image: introduce flavour metadata [repos/releng/release] - ''https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/207 (https://phabricator.wikimedia.org/T401721)'
|
|
2025-08-19 01:25:45
|
<wikibugs>
|
'Scap, ''serviceops, ''Patch-For-Review: Provide MediaWiki app image PHP version in helm values - https://phabricator.wikimedia.org/T401721#11096756 (''Scott_French) Alright, after a bit of thought, I believe the simplest option is use the build report. As noted in [[ https://gitlab.wikimedia.org/repos/rele...'
|
|
2025-08-19 01:32:17
|
<wikibugs>
|
'GitLab (Project Migration), ''Community-Tech, ''translatewiki.net, ''WS Export, ''Patch-For-Review: Migrate ws-export repo from GitHub to GitLab - https://phabricator.wikimedia.org/T395398#11096758 (''Samwilson)'
|
|
2025-08-19 01:32:23
|
<wikibugs>
|
'GitLab (Project Migration), ''Community-Tech, ''translatewiki.net, ''WS Export, ''Patch-For-Review: Migrate ws-export repo from GitHub to GitLab - https://phabricator.wikimedia.org/T395398#11096759 (''Samwilson) Translatewiki.net is now updating the new repo, e.g. https://gitlab.wikimedia.org/toolforge...'
|
|
2025-08-19 02:59:11
|
<wikibugs>
|
'GitLab (Account Approval), ''Release-Engineering-Team: Requesting GitLab account activation for awesomeaasim - https://phabricator.wikimedia.org/T402100#11096807 (''bd808) The shell account name from your Developer account is the gitlab username. https://ldap.toolforge.org/user/awesomeaasim'
|
|
2025-08-19 05:58:18
|
<wikibugs>
|
('PS1) ''Jeena Huneidi: Update wikilambda daily selenium job [integration/config] - ''https://gerrit.wikimedia.org/r/1179783 (https://phabricator.wikimedia.org/T388056)'
|
|
2025-08-19 05:59:54
|
<wikibugs>
|
('CR) ''Jeena Huneidi: "I added the parameters but in the jenkins UI I see them twice and the second ones are empty. I don't know how to override the existing par" [integration/config] - ''https://gerrit.wikimedia.org/r/1179783 (https://phabricator.wikimedia.org/T388056) (owner: ''Jeena Huneidi)'
|
|
2025-08-19 06:14:22
|
<wmf-insecte>
|
Project beta-code-update-eqiad build #561786: 'FAILURE in 1 min 21 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/561786/'
|
|
2025-08-19 06:25:15
|
<wmf-insecte>
|
Yippee, build fixed!
|
|
2025-08-19 06:25:16
|
<wmf-insecte>
|
Project beta-code-update-eqiad build #561787: 'FIXED in 2 min 14 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/561787/'
|
|
2025-08-19 07:13:04
|
<wikibugs>
|
('CR) ''Hashar: "The empty parameters are added by:" [integration/config] - ''https://gerrit.wikimedia.org/r/1179783 (https://phabricator.wikimedia.org/T388056) (owner: ''Jeena Huneidi)'
|
|
2025-08-19 08:09:39
|
<wikibugs>
|
'Scap: SecurityPatchBot: Notify task when patch fails to apply during train deployment - https://phabricator.wikimedia.org/T402262 (''jnuche) ''NEW'
|
|
2025-08-19 10:06:44
|
<wikibugs>
|
'GitLab (Project Migration), ''Community-Tech, ''translatewiki.net, ''WS Export, ''Patch-For-Review: Migrate ws-export repo from GitHub to GitLab - https://phabricator.wikimedia.org/T395398#11097429 (''Samwilson) ''Open→''In progress'
|
|
2025-08-19 10:32:02
|
<wikibugs>
|
'Continuous-Integration-Config, ''MediaWiki-Core-Tests, ''MW-1.45-notes (1.45.0-wmf.16; 2025-08-26), ''Patch-For-Review, ''Technical-Debt: Drop deprecated phpunit.php and suite.xml - https://phabricator.wikimedia.org/T395470#11097580 (''Daimona)'
|
|
2025-08-19 11:02:51
|
<wikibugs>
|
('update) ''sfaci: Added Metrics Platform Experimentation Lab deployment window to deployments calendar [repos/releng/release] - ''https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/186 (https://phabricator.wikimedia.org/T396742)'
|
|
2025-08-19 11:05:05
|
<wikibugs>
|
('update) ''sfaci: Added Metrics Platform Experimentation Lab deployment window to deployments calendar [repos/releng/release] - ''https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/186 (https://phabricator.wikimedia.org/T396742)'
|
|
2025-08-19 11:17:53
|
<wikibugs>
|
'GitLab (Project Migration), ''Community-Tech, ''translatewiki.net, ''WS Export, ''Patch-For-Review: Migrate ws-export repo from GitHub to GitLab - https://phabricator.wikimedia.org/T395398#11097865 (''Samwilson)'
|
|
2025-08-19 11:28:09
|
<wikibugs>
|
'Scap: SecurityPatchBot: Notify task when patch fails to apply during train deployment - https://phabricator.wikimedia.org/T402262#11097908 (''jnuche)'
|
|
2025-08-19 11:30:04
|
<wmf-insecte>
|
maintenance-disconnect-full-disks build 729476 integration-agent-docker-1054 (/: 40%, /srv: 97%, /var/lib/docker: 31%): OFFLINE due to disk space
|
|
2025-08-19 11:35:04
|
<wmf-insecte>
|
maintenance-disconnect-full-disks build 729477 integration-agent-docker-1054 (/: 40%, /srv: 27%, /var/lib/docker: 30%): RECOVERY disk space OK
|
|
2025-08-19 11:49:46
|
<mszabo>
|
dancy: thanks for the review!
|
|
2025-08-19 11:52:03
|
<wikibugs>
|
'Phabricator, ''Release-Engineering-Team (Priority Backlog 📥): Allow admins to rename personal Herald rules - https://phabricator.wikimedia.org/T386703#11097996 (''Aklapper) ''Declined→''Open p:''Triage→''Low a:''Aklapper I'm going to reopen because once again came back to this thought today as it i...'
|
|
2025-08-19 12:01:08
|
<wikibugs>
|
'Release-Engineering-Team (Priority Backlog 📥), ''Release, ''Train Deployments: 1.45.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T396376#11098036 (''Urbanecm_WMF)'
|
|
2025-08-19 12:12:04
|
<wikibugs>
|
'Phabricator, ''Release-Engineering-Team (Priority Backlog 📥): Allow admins to rename personal Herald rules - https://phabricator.wikimedia.org/T386703#11098052 (''Aklapper) Playing locally with a "Personal Rule" created by a non-admin account, ` diff --git a/src/applications/herald/storage/HeraldRule.php b/s...'
|
|
2025-08-19 12:17:09
|
<wmf-insecte>
|
Yippee, build fixed!
|
|
2025-08-19 12:17:09
|
<wmf-insecte>
|
Project beta-scap-sync-world build #220169: 'FIXED in 1 min 58 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/220169/'
|
|
2025-08-19 12:20:12
|
<wikibugs>
|
'Project-Admins: Create project tag for Wikinews volunteer developed software - https://phabricator.wikimedia.org/T402150#11098065 (''Gryllida) Thank you @thcipriani 😊 Please could you help with access or point me to some documentation, {F65786491}'
|
|
2025-08-19 12:23:59
|
<wikibugs>
|
'Project-Admins: Create project tag for Wikinews volunteer developed software - https://phabricator.wikimedia.org/T402150#11098066 (''Aklapper) Hi, see https://www.mediawiki.org/wiki/Phabricator/Project_management#Boards : "Editing a project and its columns requires membership in [Trusted-Contributors](http...'
|
|
2025-08-19 12:27:21
|
<wikibugs>
|
'Project-Admins: Create project tag for Wikinews volunteer developed software - https://phabricator.wikimedia.org/T402150#11098068 (''Aklapper) I realize I should probably just post the usual text after project creation here. :) Let me do that: Requested public project has been created: https://phabricator...'
|
|
2025-08-19 12:31:13
|
<wikibugs>
|
('open) ''aklapper: Allow admins to rename / edit personal Herald rules [repos/phabricator/phabricator] (wmf/stable) - ''https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/94 (https://phabricator.wikimedia.org/T386703)'
|
|
2025-08-19 13:12:51
|
<wikibugs>
|
'Phabricator, ''Release-Engineering-Team (Doing 😎): Herald rule to add Connection team tag to maintained projects - https://phabricator.wikimedia.org/T402240#11098244 (''Aklapper) ''Open→''Resolved a:''Aklapper Hi, done in H466. I also
boldly added #Campaign-Registration: * When any of these condit...'
|
|
2025-08-19 13:13:06
|
<wikibugs>
|
'Phabricator, ''Release-Engineering-Team (Doing 😎), ''Connection-Team: Herald rule to add Connection team tag to maintained projects - https://phabricator.wikimedia.org/T402240#11098248 (''Aklapper)'
|
|
2025-08-19 13:36:05
|
<Lucas_WMDE>
|
!log lucaswerkmeister-wmde@deployment-puppetserver-1:/srv/git/operations/puppet$ pgit revert 20ea82337d # T401592
|
|
2025-08-19 13:36:07
|
<stashbot>
|
T401592: Property suggestor not appearing on Beta - https://phabricator.wikimedia.org/T401592
|
|
2025-08-19 13:36:07
|
<stashbot>
|
Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
|
|
2025-08-19 13:38:52
|
<wikibugs>
|
('open) ''jnuche: patches: enhance patch failure notifications [repos/releng/scap] - ''https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/986 (https://phabricator.wikimedia.org/T402262)'
|
|
2025-08-19 13:39:39
|
<Lucas_WMDE>
|
hm, `sudo puppetserver-deploy-code` doesn’t work
|
|
2025-08-19 13:39:45
|
<Lucas_WMDE>
|
(on deployment-puppetserver-1)
|
|
2025-08-19 13:40:02
|
<Lucas_WMDE>
|
“WARN: git repository /srv/git/operations/puppet does not exist or is unreachable at this moment!” due to a “dubious ownership” error
|
|
2025-08-19 13:41:18
|
<wikibugs>
|
('update) ''jnuche: patches: enhance patch failure notifications [repos/releng/scap] - ''https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/986 (https://phabricator.wikimedia.org/T402262)'
|
|
2025-08-19 13:43:21
|
<hashar>
|
Lucas_WMDE: I know nothing about that script sorry :/
|
|
2025-08-19 13:43:26
|
<Lucas_WMDE>
|
(`sudo -u gitpuppet puppetserver-deploy-code` also doesn’t work fwiw)
|
|
2025-08-19 13:46:49
|
<Lucas_WMDE>
|
huh, according to https://gerrit.wikimedia.org/r/c/operations/puppet/+/1060919 “we're running puppetserver-deploy-code as gitpuppet”
|
|
2025-08-19 13:46:58
|
<Lucas_WMDE>
|
why does that give me an error then
|
|
2025-08-19 13:46:59
|
<Lucas_WMDE>
|
“Error: /var/cache/g10k exists, but is not writable! Exiting!”
|
|
2025-08-19 13:47:21
|
<Lucas_WMDE>
|
did I break it by running it as root first (per the docs at https://wikitech.wikimedia.org/wiki/Help:Project_puppetserver#Deploying_changes ?)
|
|
2025-08-19 13:47:53
|
<Lucas_WMDE>
|
no, it looks like that directory has existed since 2024
|
|
2025-08-19 13:48:37
|
<wikibugs>
|
'Phabricator, ''Release-Engineering-Team (Doing 😎), ''Connection-Team: Herald rule to add Connection team tag to maintained projects - https://phabricator.wikimedia.org/T402240#11098388 (''Daimona) Looks correct to me, thank you!'
|
|
2025-08-19 13:49:54
|
<Lucas_WMDE>
|
anyone around who knows how to make (or rather revert) puppet changes in beta correctly?
|
|
2025-08-19 13:56:03
|
<hashar>
|
Lucas_WMDE: I think that wrapper was added by t.aavi. But you should be able to use raw git commands in /srv/git/operations/puppet
|
|
2025-08-19 13:56:23
|
<Lucas_WMDE>
|
that’s what I did (using the pgit wrapper so it’s the right user)
|
|
2025-08-19 13:56:39
|
<Lucas_WMDE>
|
I’m mainly not sure how to get it “deployed” after that
|
|
2025-08-19 13:56:49
|
<hashar>
|
yeah let me try
|
|
2025-08-19 13:57:11
|
<taavi>
|
I think I originally wrote `pgit`, but definitely not `puppetserver-deploy-code`
|
|
2025-08-19 13:57:31
|
<Lucas_WMDE>
|
if I SSH into another deployment-prep instance I can see my revert as the “last puppet commit” in the banner, so that part looks okay
|
|
2025-08-19 13:57:35
|
<taavi>
|
anyway, that script is a total mess, but I think the deploy script is supposed to be run as root
|
|
2025-08-19 13:57:44
|
<hashar>
|
I have reset --hard HEAD^^
|
|
2025-08-19 13:57:59
|
<hashar>
|
which dropped:
|
|
2025-08-19 13:57:59
|
<hashar>
|
33d05192cd (HEAD -> production) Revert "mediawiki: Remove unused wikidata.org vhost and fix redirect in beta"
|
|
2025-08-19 13:57:59
|
<hashar>
|
20ea82337d (tag: snapshot-202508191327) mediawiki: Remove unused wikidata.org vhost and fix redirect in beta
|
|
2025-08-19 13:58:45
|
<hashar>
|
+ g10k_envdir=/home/gitpuppet/.puppet/code/environments_staging
|
|
2025-08-19 13:58:45
|
<hashar>
|
+ g10k -quiet -branch production -config /etc/puppet/g10k.conf
|
|
2025-08-19 13:58:45
|
<hashar>
|
checkDirAndCreate(): Error: /var/cache/g10k exists, but is not writable! Exiting!
|
|
2025-08-19 13:58:48
|
<hashar>
|
whistles
|
|
2025-08-19 13:58:58
|
<hashar>
|
who knows what `g10k` is
|
|
2025-08-19 13:59:14
|
<Lucas_WMDE>
|
ok, so that’s as far as I got, except you’ve erased the Git record of what I did…
|
|
2025-08-19 13:59:25
|
<hashar>
|
which record?
|
|
2025-08-19 13:59:37
|
<Lucas_WMDE>
|
the revert commit, with a message explaining why I reverted it
|
|
2025-08-19 13:59:57
|
<hashar>
|
well I reset back to that state
|
|
2025-08-19 14:00:37
|
<hashar>
|
rerr
|
|
2025-08-19 14:00:45
|
<hashar>
|
Lucas_WMDE: I can reset to the state as I pasted above
|
|
2025-08-19 14:01:42
|
<hashar>
|
but if your patch broke something you can simply remove it from the stack of patches
|
|
2025-08-19 14:01:45
|
<hashar>
|
instead of doing a revert
|
|
2025-08-19 14:01:52
|
<Lucas_WMDE>
|
the patch I reverted wasn’t mine
|
|
2025-08-19 14:02:18
|
<hashar>
|
ah yeah that was by Timo
|
|
2025-08-19 14:02:24
|
<Lucas_WMDE>
|
I had told Krinkle a few hours ago on https://gerrit.wikimedia.org/r/c/operations/puppet/+/1179719 that it seemed to break mobile beta wikidata, and got no reply, so I thought I’d try reverting it
|
|
2025-08-19 14:02:40
|
<Lucas_WMDE>
|
and do it as a git revert instead of a reset so that there would be an explanation in the git log, besides the comments on the Gerrit change
|
|
2025-08-19 14:02:42
|
<hashar>
|
well I guess we can drop both patches and mentions the patch got removed from beta
|
|
2025-08-19 14:04:15
|
<Lucas_WMDE>
|
well, it looks like mobile beta wikidata is working again now
|
|
2025-08-19 14:04:34
|
<Lucas_WMDE>
|
I suspect we just needed to wait half an hour for puppet to apply changes
|
|
2025-08-19 14:04:38
|
<hashar>
|
!log deployment-prep: sudo systemctl start puppet-git-sync-upstream.service
|
|
2025-08-19 14:04:39
|
<stashbot>
|
Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
|
|
2025-08-19 14:04:41
|
<Lucas_WMDE>
|
but maybe the reset vs revert made the difference, idk
|
|
2025-08-19 14:04:49
|
<Lucas_WMDE>
|
or that…
|
|
2025-08-19 14:04:51
|
<hashar>
|
who knows
|
|
2025-08-19 14:04:59
|
<hashar>
|
then we can run puppet on the instance that needs it
|
|
2025-08-19 14:06:03
|
<hashar>
|
hmm dpeloyment-mediawiki14 had puppet running:
|
|
2025-08-19 14:06:05
|
<hashar>
|
The last Puppet run was at Tue Aug 19 13:58:03 UTC 2025 (7 minutes ago).
|
|
2025-08-19 14:06:05
|
<hashar>
|
Last Puppet commit: (33d05192cd) Lucas Werkmeister - Revert "mediawiki: Remove unused wikidata.org vhost and fix redirect in beta"
|
|
2025-08-19 14:06:35
|
<hashar>
|
same for deployment-mediawiki13
|
|
2025-08-19 14:07:32
|
<Lucas_WMDE>
|
that’s what I mentioned above already, I think?
|
|
2025-08-19 14:07:34
|
<Lucas_WMDE>
|
“if I SSH into another deployment-prep instance I can see my revert as the “last puppet commit” in the banner”
|
|
2025-08-19 14:08:01
|
<Lucas_WMDE>
|
presumably that’ll go away soon now, to reflect the git reset
|
|
2025-08-19 14:10:09
|
<Lucas_WMDE>
|
is done touching puppet for now
|
|
2025-08-19 14:25:57
|
<wikibugs>
|
'Phabricator (phabricator-next), ''Release-Engineering-Team, ''collaboration-services: Deploy Phabricator/Phorge 2025-08-19 - https://phabricator.wikimedia.org/T402309 (''brennen) ''NEW'
|
|
2025-08-19 14:26:53
|
<wikibugs>
|
('open) ''brennen: update submodules for 2025-08-19 deploy [repos/phabricator/deployment] (wmf/stable) - ''https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/77 (https://phabricator.wikimedia.org/T402309)'
|
|
2025-08-19 14:27:32
|
<hashar>
|
brennen: hi, I have fixed up the puppet patch to bump the php APC limit from 32MB to 256MB https://gerrit.wikimedia.org/r/c/operations/puppet/+/1175916
|
|
2025-08-19 14:27:59
|
<hashar>
|
maybe you want it applied as before/after/as part of the upgrade
|
|
2025-08-19 14:29:41
|
<Krinkle>
|
Lucas_WMDE: I think periodic rebase made (or would have made) the revert dissappear anyhow :)
|
|
2025-08-19 14:29:50
|
<Krinkle>
|
Re-applied now with an attempted fix
|
|
2025-08-19 14:30:10
|
<Lucas_WMDE>
|
thanks!
|
|
2025-08-19 14:30:22
|
<wikibugs>
|
('open) ''swfrench: Draft: kubernetes: adopt new report structure and set php.version [repos/releng/scap] - ''https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/987 (https://phabricator.wikimedia.org/T401721)'
|
|
2025-08-19 14:31:01
|
<Lucas_WMDE>
|
what I had in mind was that you’d clean up the git history when you got back fwiw, I just didn’t know when that would be ^^
|
|
2025-08-19 14:31:32
|
<wikibugs>
|
('merge) ''brennen: update submodules for 2025-08-19 deploy [repos/phabricator/deployment] (wmf/stable) - ''https://gitlab.wikimedia.org/repos/phabricator/deployment/-/merge_requests/77 (https://phabricator.wikimedia.org/T402309)'
|
|
2025-08-19 14:32:36
|
<wikibugs>
|
'Scap, ''Essential-Work, ''Patch-For-Review: SecurityPatchBot: Notify task when patch fails to apply during train deployment - https://phabricator.wikimedia.org/T402262#11098651 (''jnuche)'
|
|
2025-08-19 14:34:21
|
<Krinkle>
|
Lucas_WMDE: np yeah, sounds good :)
|
|
2025-08-19 14:34:28
|
<Lucas_WMDE>
|
hm, now I get “We can’t connect to the server at m.wikidata.beta.wmcloud.org.”
|
|
2025-08-19 14:35:35
|
<Lucas_WMDE>
|
maybe I caught the server in the middle of a restart or something
|
|
2025-08-19 14:36:37
|
<Lucas_WMDE>
|
and now it seems to be timing out instead
|
|
2025-08-19 14:37:28
|
<wmcs-alerts>
|
FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
|
|
2025-08-19 14:37:35
|
<wikibugs>
|
'Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-cache-text08 is down - https://phabricator.wikimedia.org/T402311 (''wmcs-alerts) ''NEW'
|
|
2025-08-19 14:37:37
|
<Krinkle>
|
Yeah, it's still running puppet
|
|
2025-08-19 14:37:40
|
<Lucas_WMDE>
|
ok
|
|
2025-08-19 14:38:08
|
<Krinkle>
|
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[load-new-vcl-file-frontend]/returns: subprocess.CalledProcessError: Command '['/usr/bin/varnishadm', '-n', 'frontend', 'vcl.label', 'wikimedia_misc', 'vcl-a1f9de39-9991-434d-9249-bbb27c9d2c52']' returned non-zero exit status 2.
|
|
2025-08-19 14:38:18
|
<Krinkle>
|
Not sure what it's doing , it's seemingly just spinning its wheels
|
|
2025-08-19 14:38:32
|
<Krinkle>
|
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[load-new-vcl-file-frontend]/returns: Rejected 400
|
|
2025-08-19 14:38:32
|
<Krinkle>
|
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[load-new-vcl-file-frontend]/returns: CLI communication error (hdr)
|
|
2025-08-19 14:38:32
|
<Krinkle>
|
Notice: /Stage[main]/Profile::Cache::Varnish::Frontend/Varnish::Instance[text-frontend]/Exec[load-new-vcl-file-frontend]/returns: Executing: "/usr/bin/varnishadm -n frontend vcl.load vcl-a1f9de39-9991-434d-9249-bbb27c9d2c52 /etc/varnish/wikimedia_misc-frontend.vcl"
|
|
2025-08-19 14:38:38
|
<Krinkle>
|
Once it exists I'll run it again
|
|
2025-08-19 14:39:18
|
<wikibugs>
|
'Release-Engineering-Team (Priority Backlog 📥), ''Release, ''Train Deployments: 1.45.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T396376#11098703 (''jnuche)'
|
|
2025-08-19 14:39:39
|
<wikibugs>
|
'Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-cache-text08 is down - https://phabricator.wikimedia.org/T402311#11098711 (''Lucas_Werkmeister_WMDE) Possibly related to ongoing work at T401592 / [Gerrit](https://gerrit.wikimedia.org/r/c/operations/puppet/+/1179719). (Or the timing coul...'
|
|
2025-08-19 14:40:01
|
<Krinkle>
|
It's not even responding to ctrl-C
|
|
2025-08-19 14:41:56
|
<Krinkle>
|
https://grafana.wmcloud.org/d/0g9N-7pVz/cloud-vps-project-board?var-project=deployment-prep&orgId=1&from=now-1h&to=now&timezone=browser&var-instance=deployment-cache-text08
|
|
2025-08-19 14:42:28
|
<wmcs-alerts>
|
RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
|
|
2025-08-19 14:42:37
|
<wikibugs>
|
'Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-cache-text08 is down - https://phabricator.wikimedia.org/T402311#11098727 (''wmcs-alerts)'
|
|
2025-08-19 14:43:34
|
<Lucas_WMDE>
|
the whole VM is overloaded? (or was?)
|
|
2025-08-19 14:43:47
|
<Krinkle>
|
no
|
|
2025-08-19 14:43:50
|
<Krinkle>
|
I was expecting it to be
|
|
2025-08-19 14:43:52
|
<Krinkle>
|
but it wasn't
|
|
2025-08-19 14:44:04
|
<Krinkle>
|
re-running puppet now to see if it sorts stuff out or not
|
|
2025-08-19 14:44:17
|
<Krinkle>
|
well...
|
|
2025-08-19 14:44:21
|
<Lucas_WMDE>
|
I thought the gap in Grafana meant it was too busy to even respond to Prometheus
|
|
2025-08-19 14:44:25
|
<Krinkle>
|
I suppose if it was unresponsive for a while, it wouldn't... ^
|
|
2025-08-19 14:44:26
|
<Krinkle>
|
yeah
|
|
2025-08-19 14:44:30
|
<Lucas_WMDE>
|
(until it managed to report load values in the double digits)
|
|
2025-08-19 14:44:32
|
<Lucas_WMDE>
|
ok ^^
|
|
2025-08-19 14:45:29
|
<Krinkle>
|
maybe whatever state our VCL code is in as of $when means Varnish can't do live reloads anymore
|
|
2025-08-19 14:45:36
|
<Krinkle>
|
given the size of the VM
|
|
2025-08-19 14:45:55
|
<Krinkle>
|
it's probably going to tail spin again during this run
|
|
2025-08-19 14:46:40
|
<Lucas_WMDE>
|
knows nothing about VCL
|
|
2025-08-19 14:47:27
|
<Lucas_WMDE>
|
but Grafana looks like it’s very busy again yeah
|
|
2025-08-19 14:49:54
|
<Krinkle>
|
well, it finished running on its own eventually
|
|
2025-08-19 14:50:13
|
<Krinkle>
|
seems to work fine
|
|
2025-08-19 14:50:37
|
<Krinkle>
|
Lucas_WMDE: let me know if it (beta wikidata) works correclty for you as well!
|
|
2025-08-19 14:50:44
|
<Lucas_WMDE>
|
trying it now
|
|
2025-08-19 14:51:57
|
<Lucas_WMDE>
|
seems to work fine \o/
|
|
2025-08-19 14:52:10
|
<Lucas_WMDE>
|
(except that firefox refuses to reset the zoom level to 100% o_O but that’s certainly unrelated ^^)
|
|
2025-08-19 14:52:28
|
<Lucas_WMDE>
|
both on mobile and when using either m. or www. on desktop
|
|
2025-08-19 14:52:29
|
<Lucas_WMDE>
|
\o/
|
|
2025-08-19 14:55:26
|
<Lucas_WMDE>
|
thanks a lot Krinkle!
|
|
2025-08-19 14:55:42
|
<Lucas_WMDE>
|
(Grafana also looks alright again)
|
|
2025-08-19 14:57:06
|
<wikibugs>
|
'Release-Engineering-Team (Priority Backlog 📥), ''Release, ''Train Deployments: 1.45.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T396376#11098822 (''jnuche)'
|
|
2025-08-19 14:58:23
|
<wikibugs>
|
'Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-cache-text08 is down - https://phabricator.wikimedia.org/T402311#11098827 (''Lucas_Werkmeister_WMDE) ''Open→''Resolved Gonna boldly close this – the instance was temporarily overloaded during some VCL work (see [Grafana](https://g...'
|
|
2025-08-19 15:07:12
|
<wikibugs>
|
'Phabricator (2025-08-19), ''Release-Engineering-Team, ''collaboration-services, ''Patch-For-Review: Deploy Phabricator/Phorge 2025-08-19 - https://phabricator.wikimedia.org/T402309#11098874 (''Aklapper) Included: * {T399604} / https://gerrit.wikimedia.org/r/c/phabricator/translations/+/1175157 * Some sma...'
|
|
2025-08-19 15:15:38
|
<wikibugs>
|
'Release-Engineering-Team (Doing 😎): Improve garbage collection of unused MediaWiki images on deployment host - https://phabricator.wikimedia.org/T387927#11098914 (''dancy) Noting that as currently written, `scap clean-images` can only be executed by people in the `docker` group. We should improve that to make...'
|
|
2025-08-19 15:25:40
|
<wikibugs>
|
('approved) ''dancy: patches: enhance patch failure notifications [repos/releng/scap] - ''https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/986 (https://phabricator.wikimedia.org/T402262) (owner: ''jnuche)'
|
|
2025-08-19 16:02:07
|
<wikibugs>
|
'Release-Engineering-Team, ''Content-Transform-Team, ''Quibble, ''Catalyst (PatchDemo), and 3 others: Not adding trailing / to autoload patches breaks Parsoid CI - https://phabricator.wikimedia.org/T400299#11099148 (''hashar) @cscott CI should be all set now :) Your patch //[[ https://gerrit.wikimedia.or...'
|
|
2025-08-19 16:21:14
|
<wikibugs>
|
'Phabricator, ''Release-Engineering-Team (Doing 😎), ''Connection-Team: Herald rule to add Connection team tag to maintained projects - https://phabricator.wikimedia.org/T402240#11099242 (''Daimona) ''Resolved→''Open Apologies, I just noticed a serious issue with the current rule: whenever a
task is tagg...'
|
|
2025-08-19 16:23:54
|
<wikibugs>
|
'Phabricator, ''Release-Engineering-Team (Doing 😎), ''Connection-Team: Herald rule to add Connection team tag to maintained projects - https://phabricator.wikimedia.org/T402240#11099255 (''Aklapper) ''Open→''Resolved Done.'
|
|
2025-08-19 16:25:59
|
<wikibugs>
|
('update) ''swfrench: make-container-image: introduce flavour metadata [repos/releng/release] - ''https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/207 (https://phabricator.wikimedia.org/T401721)'
|
|
2025-08-19 16:26:04
|
<wikibugs>
|
('update) ''swfrench: make-container-image: introduce flavour metadata [repos/releng/release] - ''https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/207 (https://phabricator.wikimedia.org/T401721)'
|
|
2025-08-19 16:26:52
|
<wikibugs>
|
'GitLab (Upstream pit of despair 🕳️), ''Release-Engineering-Team (Seen): Gitlab "Review changes" button not working - https://phabricator.wikimedia.org/T400091#11099287 (''dancy) Upstream has closed https://gitlab.com/gitlab-org/gitlab/-/issues/553219 as "cannot reproduce". I have confirmed that the "Review...'
|
|
2025-08-19 16:27:23
|
<wikibugs>
|
'GitLab (Upstream pit of despair 🕳️), ''Release-Engineering-Team (Seen): Gitlab "Review changes" button not working - https://phabricator.wikimedia.org/T400091#11099290 (''dancy) @Jelto Adding you to make you aware of this bug that probably won't be fixed until we upgrade our Gitlab instance.'
|
|
2025-08-19 16:27:38
|
<wikibugs>
|
('approved) ''dancy: make-container-image: introduce flavour metadata [repos/releng/release] - ''https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/207 (https://phabricator.wikimedia.org/T401721) (owner: ''swfrench)'
|
|
2025-08-19 17:13:10
|
<wikibugs>
|
'Phabricator, ''Release-Engineering-Team, ''collaboration-services, ''Patch-For-Review: Phorge setup check caching is misbehaving, leading to many duck-sound=quack requests - https://phabricator.wikimedia.org/T401157#11099479 (''Dzahn) After deploying the config change (and restarting php7.4-fpm) we can n...'
|
|
2025-08-19 17:13:32
|
<wikibugs>
|
('merge) ''swfrench: make-container-image: introduce flavour metadata [repos/releng/release] - ''https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/207 (https://phabricator.wikimedia.org/T401721)'
|
|
2025-08-19 17:20:19
|
<wikibugs>
|
'Scap, ''serviceops, ''Patch-For-Review: Provide MediaWiki app image PHP version in helm values - https://phabricator.wikimedia.org/T401721#11099540 (''Scott_French) Alright, [[ https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/207 | release!207 ]] is live, and the updated build report
look...'
|
|
2025-08-19 17:21:15
|
<wikibugs>
|
('update) ''swfrench: kubernetes: adopt new report structure and set php.version [repos/releng/scap] - ''https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/987 (https://phabricator.wikimedia.org/T401721)'
|
|
2025-08-19 17:30:57
|
<wikibugs>
|
'Continuous-Integration-Infrastructure (Zuul upgrade), ''Release-Engineering-Team (Priority Backlog 📥), ''collaboration-services, ''Essential-Work: Puppetize systemd unit for zuul's nodepool - https://phabricator.wikimedia.org/T401614#11099578 (''Dzahn) Hello, so regarding the mount points, I ensured thes...'
|
|
2025-08-19 17:42:57
|
<wikibugs>
|
('update) ''sfaci: Added Metrics Platform Experimentation Lab deployment window to deployments calendar [repos/releng/release] - ''https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/186 (https://phabricator.wikimedia.org/T396742)'
|
|
2025-08-19 17:58:02
|
<wikibugs>
|
'Beta-Cluster-Infrastructure, ''Patch-For-Review: Beta cluster IP block page should not point to noc@wikimedia.org - https://phabricator.wikimedia.org/T393404#11099686 (''bd808) ''Open→''Resolved'
|
|
2025-08-19 18:01:33
|
<wikibugs>
|
'Continuous-Integration-Infrastructure (Zuul upgrade), ''Release-Engineering-Team (Priority Backlog 📥), ''collaboration-services, ''Essential-Work, ''Patch-For-Review: Puppetize systemd unit for zuul's nodepool - https://phabricator.wikimedia.org/T401614#11099704 (''bd808)
a:''bd808→''Dzahn Unlickin...'
|
|
2025-08-19 18:09:51
|
<wikibugs>
|
('update) ''dancy: kubernetes: adopt new report structure and set php.version [repos/releng/scap] - ''https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/987 (https://phabricator.wikimedia.org/T401721) (owner: ''swfrench)'
|
|
2025-08-19 18:09:58
|
<wikibugs>
|
('merge) ''dancy: kubernetes: adopt new report structure and set php.version [repos/releng/scap] - ''https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/987 (https://phabricator.wikimedia.org/T401721) (owner: ''swfrench)'
|
|
2025-08-19 18:10:45
|
<wikibugs>
|
('open) ''dancy: Release 4.206.0 [repos/releng/scap] - ''https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/988'
|
|
2025-08-19 18:11:43
|
<wikibugs>
|
'Continuous-Integration-Infrastructure (Zuul upgrade), ''Release-Engineering-Team (Priority Backlog 📥), ''collaboration-services, ''Essential-Work, ''Patch-For-Review: Puppetize systemd unit for zuul's nodepool - https://phabricator.wikimedia.org/T401614#11099734 (''bd808) >>! In
T401614#11099578, @Dzah...'
|
|
2025-08-19 18:13:10
|
<wikibugs>
|
('merge) ''dancy: Release 4.206.0 [repos/releng/scap] - ''https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/988'
|
|
2025-08-19 18:15:37
|
<wikibugs>
|
'Release-Engineering-Team, ''collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T402331 (''phaultfinder) ''NEW'
|
|
2025-08-19 18:15:38
|
<jinxer-wm>
|
FIRING: ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip6) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
|
|
2025-08-19 18:20:31
|
<jinxer-wm>
|
RESOLVED: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown
|
|
2025-08-19 18:23:55
|
<ejegg>
|
hi releng folks, we're seeing an odd failure on gate-and-submit for the DonationInterface extension
|
|
2025-08-19 18:24:07
|
<ejegg>
|
I think this is the relevant line:
|
|
2025-08-19 18:24:10
|
<ejegg>
|
https://integration.wikimedia.org/ci/job/mwgate-composer-php82/1539/console
|
|
2025-08-19 18:24:21
|
<ejegg>
|
14:20:02 + git init --initial-branch=master
|
|
2025-08-19 18:24:22
|
<ejegg>
|
14:20:02 /src/.git: Permission denied
|
|
2025-08-19 18:24:33
|
<ejegg>
|
any suggestions?
|
|
2025-08-19 18:24:53
|
<mutante>
|
I hit rebuild to see if that happens a second time.
|
|
2025-08-19 18:25:07
|
<mutante>
|
https://integration.wikimedia.org/ci/job/mwgate-composer-php82/1540/console
|
|
2025-08-19 18:25:46
|
<mutante>
|
looks like it's fine now? permission denied on /srv/.git that goes away sounds like unlucky timing / race ?
|
|
2025-08-19 18:26:59
|
<wikibugs>
|
'Continuous-Integration-Infrastructure (Zuul upgrade), ''Release-Engineering-Team (Priority Backlog 📥), ''collaboration-services, ''Essential-Work, ''Patch-For-Review: Puppetize systemd unit for zuul's nodepool - https://phabricator.wikimedia.org/T401614#11099815 (''bd808) >>! In
T401614#11099578, @Dzah...'
|
|
2025-08-19 18:27:42
|
<ejegg>
|
thanks mutante
|
|
2025-08-19 18:29:35
|
<mutante>
|
ejegg: did you get a V+1 on the change now from jenkins bot? if not you can comment "recheck" to trigger that
|
|
2025-08-19 18:31:11
|
<ejegg>
|
thanks, got it merged now!
|
|
2025-08-19 18:34:02
|
<mutante>
|
ok, cool
|
|
2025-08-19 18:46:11
|
<wikibugs>
|
'Scap, ''serviceops: Provide MediaWiki app image PHP version in helm values - https://phabricator.wikimedia.org/T401721#11099918 (''Scott_French) This is now live and exports `php.version` as expected, e.g. ` swfrench@deploy1003:~$ cat /etc/helmfile-defaults/mediawiki/release/mw-web-main.yaml docker: regi...'
|
|
2025-08-19 21:37:55
|
<wikibugs>
|
'GitLab (Account Approval), ''Release-Engineering-Team: Requesting GitLab account activation for SaintWolf - https://phabricator.wikimedia.org/T399836#11100432 (''Aklapper) ''Open→''Stalled @SaintWolf That sounds like a great project! Did you manage to get the code that you need? If not, could you provide...'
|
|
2025-08-19 21:39:42
|
<wikibugs>
|
'GitLab (Account Approval), ''Release-Engineering-Team: Requesting GitLab account activation for Meetmendapara - https://phabricator.wikimedia.org/T398769#11100441 (''Aklapper) ''Open→''Declined Unfortunately closing this Phabricator task as no further information has been provided. @Meetmendapara:
A...'
|
|
2025-08-19 21:55:03
|
<wikibugs>
|
'Release-Engineering-Team: Add a staging kubernetes cluster to train-dev - https://phabricator.wikimedia.org/T402350 (''dancy) ''NEW'
|
|
2025-08-19 21:56:47
|
<wikibugs>
|
'Release-Engineering-Team (Priority Backlog 📥), ''Epic, ''OKR-Work: [WE6.1.1] Move image build to deployment server and update for backports - https://phabricator.wikimedia.org/T398868#11100503 (''bd808)'
|
|
2025-08-19 21:56:48
|
<wikibugs>
|
'Release-Engineering-Team: Add a staging kubernetes cluster to train-dev - https://phabricator.wikimedia.org/T402350#11100504 (''bd808)'
|
|
2025-08-19 22:05:33
|
<wikibugs>
|
'Release-Engineering-Team (Priority Backlog 📥): Add a staging kubernetes cluster to train-dev - https://phabricator.wikimedia.org/T402350#11100541 (''bd808)'
|
|
2025-08-19 22:38:28
|
<wmcs-alerts>
|
FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
|
|
2025-08-19 22:38:37
|
<wikibugs>
|
'Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-cache-text08 is down - https://phabricator.wikimedia.org/T402355 (''wmcs-alerts) ''NEW'
|
|
2025-08-19 22:43:28
|
<wmcs-alerts>
|
RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
|
|
2025-08-19 22:43:39
|
<wikibugs>
|
'Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-cache-text08 is down - https://phabricator.wikimedia.org/T402355#11100632 (''wmcs-alerts)'
|
|
2025-08-19 23:12:27
|
<wikibugs>
|
'Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-cache-text08 is down - https://phabricator.wikimedia.org/T402355#11100674 (''bd808) ''Open→''Invalid Flapping check? {T402311}'
|
|
2025-08-19 23:53:41
|
<wikibugs>
|
'Release-Engineering-Team (Priority Backlog 📥), ''Release, ''Train Deployments: 1.45.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T396377#11100705 (''SecurityPatchBot)'
|