[08:33:14] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Joe) [08:34:30] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Joe) [08:36:11] jnuche: good morning, What is the purpose of the MediaWiki daily branch `wmf/branch_cut_pretest`? :) [08:36:43] 10GitLab (Auth & Access): Access to Gitlab project - https://phabricator.wikimedia.org/T337007 (10Albertoleoncio) [08:36:48] I know it exists from some past actions but never went to find what it is used for :D [08:37:31] hashar: morning, it runs the branch cut process nightly to try to catch issues ahead of time [08:37:40] don't worry about last night error, it was a hiccup [08:38:00] I'm double-checking right now and will send out an email once the last run finishes [08:39:27] jnuche: how does it catch issues? Is that deployed somewhere , triggering a build generation? [08:40:19] or is that to trigger a CI run on a daily basis? [08:41:09] in which case I would expect CI job to always pass cause the combination of commits has already been tested together in order for them to be merged in `master`. So if something fails, it would be interesting to investigate the failure and backtrack how a faulty patch merged in :] [08:41:31] anyway the more tests the merrier [08:41:31] it runs a CI job analogous to the branch cut, but with a specific separate branch just for the purpose of testing [08:41:45] for example, this is what's running now: https://releases-jenkins.wikimedia.org/job/Automatic%20branch%20cut%20pretest/68/console [08:41:53] ah [08:44:24] fun thing is that the `wmf/branch_cut_test` is not recognized by Zuul and that change is tested as if it got send to master [08:46:59] not sure what that implies, are you saying the tests are running with the contents of master and not the branch? [08:47:07] so Zuul runs the job for the mediawiki tarball environment and a bunch of others we don't care for wmf branches such as php 8.0 [08:47:56] ok, I see, not super efficient [08:48:04] or more precisely the `wmf/branch_cut_test` is not consumed by the `gate-and-submit-wmf` Zuul pipeline which has a branch filter of `wmf/\d\.\d{2}\.\d-wmf\.\d{1,2}` [08:48:36] and it ends up accepted by the general `gate-and-submit` intended for the `master` branch which has the opposite filter [08:48:37] 10Release-Engineering-Team, 10Scap: Honor flavor fields in MW K8s deployment config files - https://phabricator.wikimedia.org/T318538 (10Joe) 05Openβ†’03Resolved p:05Triageβ†’03Medium a:03Joe @jnuche I'm closing this task as resolved as AIUI we do this since months. [08:48:39] 10Release-Engineering-Team, 10Scap, 10MW-on-K8s: Scap Mediawiki K8s deployments - https://phabricator.wikimedia.org/T318536 (10Joe) [08:48:39] so yeah over testing [08:49:19] but what is tested in `gate-and-submit` for `master` branch is a super set of what is tested in `gate-and-submit-wmf` for the `wmf/` branches. [08:49:43] I guess the Zuul regex could be relaxed to filter branch solely on the prefix `^wmf/` [08:49:55] 10Release-Engineering-Team, 10Scap: Verify images before MW K8s deployments - https://phabricator.wikimedia.org/T318223 (10Joe) @jnuche is this task still valid? I think it's not really an issue given how scap works now, including the pre-download of images on the k8s nodes. [08:49:59] anyway, it does not hurt [08:50:33] alternatively, we could add `wmf/branch_cut_pretest` explicitly to the regex of `gate-and-submit-wmf` [08:53:27] that too yes [08:53:42] and in all case if there is a failure that would be good to know, theorically it should not happen [08:53:59] then I am not entirely sure how the jobs tests the wmf branch which have submodules [08:54:37] ie the patch targets mediawiki/core @ wmf/branch_cut_pretest but I don't think they process submodules, instead they checkout the wmf branches from each extension [08:54:58] but they should be the same since the submodules are automatically bumped by Gerrit to always point to the tip of the branch of each repos [08:55:01] so it is the same [08:55:20] and I think that job will never fail (unless the CI images are changed maybe) [09:26:22] 10GitLab (Auth & Access), 10User-bd808: Access to Gitlab project - https://phabricator.wikimedia.org/T337007 (10valerio.bozzolan) 05Openβ†’03Resolved a:03bd808 Many thaaanks to Bryan who resolved this thing in IRC πŸŽ‰ Now Alberto is Owner if the related repo https://gitlab.wikimedia.org/toolforge-repos/al... [09:40:56] 10Project-Admins: Create project tag for Wikimania Hackathon 2023 - https://phabricator.wikimedia.org/T337016 (10Robertsky) [09:42:27] 10Phabricator: Avoid the "Action with no effect" warning when removing the auto assigned user - https://phabricator.wikimedia.org/T337017 (10WMDE-Fisch) [09:50:11] 10Project-Admins: Create project tag for Wikimania Hackathon 2023 - https://phabricator.wikimedia.org/T337016 (10Aklapper) 05Openβ†’03Resolved a:03Aklapper Requested public project has been created: #wikimania-hackathon-2023 (In case you need to edit the project or project workboard itself at some point and... [09:51:49] 10Project-Admins: Create a #openstack-magnum - https://phabricator.wikimedia.org/T336941 (10Aklapper) @rook: Sure. Please see https://www.mediawiki.org/wiki/Phabricator/Creating_and_renaming_projects and use the template link under "Request a single project" to provide all required information. [09:52:04] 10Project-Admins: Create a #wmcs-superset tag - https://phabricator.wikimedia.org/T336940 (10Aklapper) @rook: Sure. Please see https://www.mediawiki.org/wiki/Phabricator/Creating_and_renaming_projects and use the template link under "Request a single project" to provide all required information. [10:03:28] 10Phabricator: Remove "Action Has No Effect" warning dialog when removing auto assigned user - https://phabricator.wikimedia.org/T337017 (10Aklapper) [10:06:50] 10Phabricator: Remove "Action Has No Effect" warning dialog when removing auto assigned user - https://phabricator.wikimedia.org/T337017 (10Aklapper) p:05Triageβ†’03Medium Makes sense to me. That's `src/applications/transactions/response/PhabricatorApplicationTransactionNoEffectResponse.php` in https://gitlab.... [10:50:08] 10GitLab (Infrastructure), 10SRE, 10ops-eqiad, 10serviceops-collab: gitlab-runner1003 is not coming back online - https://phabricator.wikimedia.org/T336737 (10Jelto) 05Openβ†’03Resolved @Jclark-ctr thanks a lot for the quick response! Error is gone! I'm closing this task. [10:51:31] maintenance-disconnect-full-disks build 492473 integration-agent-docker-1028 (/: 29%, /srv: 97%, /var/lib/docker: 51%): OFFLINE due to disk space [10:56:04] maintenance-disconnect-full-disks build 492474 integration-agent-docker-1028 (/: 29%, /srv: 48%, /var/lib/docker: 49%): RECOVERY disk space OK [11:01:59] 10GitLab (CI & Job Runners), 10serviceops-collab: GitLab CI: Could not resolve host: gitlab.wikimedia.org - https://phabricator.wikimedia.org/T327989 (10Jelto) 05Openβ†’03Resolved a:03Jelto This should be related to maintenance on runner1030 in T327949#8564035 during that time and should be resolved. I'm... [11:29:13] 10Project-Admins: Create project tag for wmcs-superset - https://phabricator.wikimedia.org/T337024 (10rook) [11:30:24] 10Project-Admins: Create project tag for wmcs-superset - https://phabricator.wikimedia.org/T337024 (10rook) [11:30:41] 10Project-Admins: Create a #wmcs-superset tag - https://phabricator.wikimedia.org/T336940 (10rook) [11:30:59] 10Project-Admins: Create project tag for wmcs-superset - https://phabricator.wikimedia.org/T337024 (10rook) [11:31:01] 10Project-Admins: Create a #wmcs-superset tag - https://phabricator.wikimedia.org/T336940 (10rook) [11:37:09] hoo: WMDE-Fisch: do you have a way to reach to Wikidata/Wikibase people? There is some exception that was going on apparently apparently due to a double redirect which caused resolution error: Unresolved redirect from Q16887380 to Q16889133 [11:37:31] hey hashar [11:37:37] I can have a quick look [11:37:48] You're not in Athens, are you? [11:37:51] a user reported it might be caused by a double redirect [11:37:53] nop :/ [11:38:18] we had the error on wmf.8 and wmf.9 and I am pretty sure the error is entirely due to the redirect [11:38:49] I think I will update the wikis again to wmf.9 cause I don't think that specific one is any related [11:39:43] Can definitely poke someone here. In a session atm though. [11:39:59] no worries :] [11:40:02] I see where the problem is coming from [11:40:21] and it's even something I did… but a few years already [11:40:24] I don't even need a fix, just a confirmation about the investigation I did this morning that is indeed related to the redirect [11:40:27] we probably only got unlucky now [11:40:35] and Pilkne on the task suggested it is due to the double redir [11:40:52] the root cause can be fixed later I guess [11:42:12] hashar: btw, i wanted to ask, should my bot that updates infoboxes on extension pages use gerrit-replica ? [11:42:26] It regularly git pull's basically every extension [11:44:20] bawolff: most probably yes, but there is no need to change it right now [11:45:13] bawolff: I think the primary already has all extensions and skins in cache so it serves everything directly from the cache/memory which is probably not an issue at all [11:45:49] in the future, I might get rid of the gerrit-replica in favor of having a high availability setup, but I am afraid that might require a lot of work [11:46:45] hoo: if you have any explanation may you post it on the train blocker task at https://phabricator.wikimedia.org/T336952 ? And I guess after that I will do the train deployment in an hour or so [11:56:51] 10Project-Admins: Create a #openstack-magnum - https://phabricator.wikimedia.org/T336941 (10rook) I think it would be a sub project of cloud-vps. Would that be a different process? [11:56:59] thcipriani: hello! l.egoktm and I are at the hackathon and would like to deploy https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/921326/ even though it is a Friday, is that ok? we will fix any breakage we might cause [12:05:24] hashar: Uploaded a patch and posted a short explanation [12:07:55] hoo: that is great thank you! [12:08:08] hoo: and the issue has been there for a while correct? [12:08:40] Yeah, for ages… double redirects are probably so rare that we just never (not very often) hit it before [12:09:04] 10Release-Engineering-Team, 10Scap, 10MW-on-K8s: Scap Mediawiki K8s deployments - https://phabricator.wikimedia.org/T318536 (10jnuche) [12:09:06] 10Release-Engineering-Team, 10Scap: Verify images before MW K8s deployments - https://phabricator.wikimedia.org/T318223 (10jnuche) 05Openβ†’03Declined Yeah, I don't think this provides value anymore. Declining. [12:09:24] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10MW-on-K8s, 10serviceops: mwdebug: people in the "deployment" group should be able to launch 'experimental' instances for testing purposes - https://phabricator.wikimedia.org/T324003 (10Joe) An alternative idea from @akosiaris which is also very interesting:... [12:10:19] hoo: great thank you so much. I will roll the train after my lunch [12:10:30] hoo: and happy hackathon if you are there [12:10:33] GWToolset is broken, so I'm planning to turn it off in a little bit: https://phabricator.wikimedia.org/T270911 [12:10:35] WMDE-Fisch: solved! :] [12:10:46] I am, thanks! :) [12:12:00] taavi: legoktm: I don't know the policy about changing stuff on Friday, I guess given it is done during the hackathon is something goes havoc you can shout in the room to get stuff investigated/fixed and tentatively have anyone needed to help? [12:12:20] I plan to run the train in ~ 50 minutes, but I guess feel free to push those changes [12:12:52] though I can't vet what RealMe might cause, and it looks GWToolset is no more used anywhere (it was a narrow case for uploading files en masse) [12:13:04] looks like both are fairly isolated [12:15:00] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Joe) [12:17:07] hashar: yeah, basically :) just wanted to make sure releng was aware [12:21:45] taavi: legoktm: feel free to send me the reviews and I will +1 if you need a formal approval [12:22:05] then I am pushing the train in 40 minutes, so they are probably best done after [12:22:21] hashar: https://gerrit.wikimedia.org/r/921252 - we'll just do the initial undeployment now and do the cleanup next week (just in case we missed something) [12:22:53] we want to make one i18n tweak for RealMe, taavi is writing the patch right now, and we'll backport it into the branch so it is included in the i18n cache when you do the train deploy [12:23:44] 10Project-Admins: Create a #openstack-magnum - https://phabricator.wikimedia.org/T336941 (10Aklapper) No, it would not be a different process, but we need required information [12:25:33] legoktm: +1 :) [12:28:07] ty! [12:28:17] also nvm on the i18n patch for RealMe, we'll do that ourselves later [12:28:42] 10Project-Admins: Create a #wmcs-superset tag - https://phabricator.wikimedia.org/T336940 (10Aklapper) Thanks. https://phabricator.wikimedia.org/project/profile/5683/ currently talks about "Wikimedia's Superset instance(s)." What should that be changed to, so users can find the right project tag? [12:31:19] maintenance-disconnect-full-disks build 492493 integration-agent-docker-1032 (/: 29%, /srv: 95%, /var/lib/docker: 66%): OFFLINE due to disk space [12:31:25] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10MW-on-K8s, 10serviceops: mwdebug: people in the "deployment" group should be able to launch 'experimental' instances for testing purposes - https://phabricator.wikimedia.org/T324003 (10Joe) So the steps to do what Alex proposed would be: * Add support for i... [12:35:58] maintenance-disconnect-full-disks build 492494 integration-agent-docker-1032 (/: 29%, /srv: 39%, /var/lib/docker: 61%): RECOVERY disk space OK [12:58:53] legoktm: +1ed all your GWToolset removal patches [13:02:08] taavi: and I +1ed the RealMe deploy [13:02:17] I am going to roll the MediaWiki train from #wikimedia-operations [13:04:04] 10Release-Engineering-Team (They Live πŸ•ΆοΈπŸ§Ÿ), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.41.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T330215 (10hashar) [13:04:33] 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10Aklapper) @Seddon: done [13:04:53] 10Phabricator: Allow anonymous people to View "Slowvotes" polls - https://phabricator.wikimedia.org/T337031 (10valerio.bozzolan) p:05Triageβ†’03Low [13:05:34] 10Release-Engineering-Team (They Live πŸ•ΆοΈπŸ§Ÿ), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.41.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T330215 (10hashar) We have ruled out {T336952} being a blocker. The issue is Scribunto / Wikibase mishandling a double redir... [13:24:08] hashar: let us know when you're done with the train please and we'll start deploying stuff [13:25:04] I am checking the logs and metrics [14:10:52] maintenance-disconnect-full-disks build 492513 integration-agent-docker-1034 (/: 29%, /srv: 96%, /var/lib/docker: 46%): OFFLINE due to disk space [14:13:58] 10Phabricator: Phabricator file deletion request - https://phabricator.wikimedia.org/T336928 (10Aklapper) 05Openβ†’03Resolved a:03Aklapper Done! [14:16:36] maintenance-disconnect-full-disks build 492514 integration-agent-docker-1034 (/: 29%, /srv: 16%, /var/lib/docker: 45%): RECOVERY disk space OK [14:17:10] 10Phabricator: Allow anonymous people to View "Slowvotes" polls - https://phabricator.wikimedia.org/T337031 (10Aklapper) There is no separate setting for //creating// a Slowvote item versus //seeing and voting// in a Slowvote item. So I'd prefer not to do this. [15:01:09] 10Phabricator: Allow anonymous people to View "Slowvotes" polls - https://phabricator.wikimedia.org/T337031 (10waldyrious) @valerio.bozzolan is this separation of settings something that could be implemented in Phorge? [15:12:05] taavi: thanks for the heads up, happy hacking [16:00:46] I am off for the week-end [17:43:28] 10Project-Admins: Create a #openstack-magnum - https://phabricator.wikimedia.org/T336941 (10rook) [18:22:00] 10Phabricator, 10Release-Engineering-Team: Make Taavi a Phab admin - https://phabricator.wikimedia.org/T337065 (10taavi) [22:24:12] 10Phabricator: Allow anonymous people to View "Slowvotes" polls - https://phabricator.wikimedia.org/T337031 (10valerio.bozzolan) 05Openβ†’03Stalled [22:24:37] 10Phabricator: Allow anonymous people to View "Slowvotes" polls - https://phabricator.wikimedia.org/T337031 (10valerio.bozzolan) >>! In T337031#8865162, @waldyrious wrote: > @valerio.bozzolan is this separation of settings something that could be implemented in Phorge? Good idea. Feel free to Subscribe here too... [22:37:33] 10Release-Engineering-Team (They Live πŸ•ΆοΈπŸ§Ÿ), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.41.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T330215 (10brennen) 05Openβ†’03Resolved