[03:44:59] (03CR) 10Legoktm: [C: 03+2] "Sure. Please document this at " [integration/commit-message-validator] - 10https://gerrit.wikimedia.org/r/862233 (owner: 10Daniel Kinzler) [03:45:43] (03CR) 10Legoktm: [C: 04-2] "I think we want to keep this check. If there's an issue with the Gerrit hook, that should be fixed." [integration/commit-message-validator] - 10https://gerrit.wikimedia.org/r/863238 (https://phabricator.wikimedia.org/T324316) (owner: 10David Caro) [03:46:00] (03Merged) 10jenkins-bot: Support Needed-By as a backlink to Depends-On [integration/commit-message-validator] - 10https://gerrit.wikimedia.org/r/862233 (owner: 10Daniel Kinzler) [09:20:15] 10Continuous-Integration-Config, 10Quality-and-Test-Engineering-Team (QTE), 10phpunit-patch-coverage: Most (all?) phpunit-patch-coverage jobs failing with "String could not be parsed as XML" instead of actually running - https://phabricator.wikimedia.org/T323139 (10pwangai) a:03pwangai [09:31:24] (03CR) 10Hashar: "Please note `Depends-On` is specific to Zuul and is the only way to enforce a dependency." [integration/commit-message-validator] - 10https://gerrit.wikimedia.org/r/862233 (owner: 10Daniel Kinzler) [10:06:46] (03CR) 10Hashar: [C: 03+2] Zuul: [mediawiki/extensions/LDAPAuthentication2] Run noselenium tests [integration/config] - 10https://gerrit.wikimedia.org/r/864079 (owner: 10Umherirrender) [10:08:51] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/LDAPAuthentication2] Run noselenium tests [integration/config] - 10https://gerrit.wikimedia.org/r/864079 (owner: 10Umherirrender) [10:11:19] (03CR) 10Hashar: [C: 03+2] "deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/864079 (owner: 10Umherirrender) [10:35:12] RECOVERY - Host contint1001 is UP: PING OK - Packet loss = 0%, RTA = 0.24 ms [10:35:23] 10Release-Engineering-Team (Seen), 10serviceops: switch contint prod server back from contint2001 to contint1001 - https://phabricator.wikimedia.org/T256422 (10fgiunchedi) FWIW while looking into sth unrelated I found contint1001 crashed today since two days [11:44:35] 10Continuous-Integration-Config: test-prio job for operations/puppet/+/863332/2 stuck - https://phabricator.wikimedia.org/T324394 (10Umherirrender) zuul was restarted and the job is gone > 11:07 hashar: Restarted Zuul to clear a stuck ssh connection with Gerrit - T309376 [12:34:17] 10Continuous-Integration-Infrastructure, 10SRE, 10serviceops-collab: contint1002 service implementation tracking - https://phabricator.wikimedia.org/T313832 (10MoritzMuehlenhoff) Switching to contint1002 would also be a good opportunity to migrate to Bullseye (which per https://wikitech.wikimedia.org/wiki/Op... [12:49:57] 10Continuous-Integration-Config: test-prio job for operations/puppet/+/863332/2 stuck - https://phabricator.wikimedia.org/T324394 (10TheresNoTime) [12:49:59] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: gerrit-bot holding open SSH sessions - https://phabricator.wikimedia.org/T309376 (10TheresNoTime) [12:50:37] 10Continuous-Integration-Config: test-prio job for operations/puppet/+/863332/2 stuck - https://phabricator.wikimedia.org/T324394 (10TheresNoTime) 05Open→03Resolved a:03hashar [12:50:39] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: gerrit-bot holding open SSH sessions - https://phabricator.wikimedia.org/T309376 (10TheresNoTime) [13:20:39] hashar: there seems to be some npm CI cache issue, https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/863258 complains about an undefined variable, but that variable is removed in the patch [13:28:56] kostajh: I guess eslint cache fails to detect a file got deleted? [13:30:27] hashar: not sure :/ [13:31:42] oh no [13:31:54] kostajh: maybe cause the variable is used in the target branch [13:32:05] if the patch is lagging behind, it is tested with the latest tip of the branhc [13:32:09] so potentially it needs to be rebaesd [13:33:10] yeah [13:33:18] `git grep -n isUnactiveOrDisabled origin/master` shows a few matches [13:36:14] kostajh: yes it needs to be rebased. I can reproduce locally by retrieving the change and rebasing it [13:36:32] modules/ext.growthExperiments.Homepage.NewImpact/App.vue:67: [13:36:33] if ( !isUnactiveOrDisabled && !mw.user.options.get( 'growthexperiments-tour-newimpact-discovery' ) && renderMode === 'desktop' ) { [13:36:45] ^^^^^^^^^^^^ [13:47:43] 10Continuous-Integration-Infrastructure, 10Gerrit, 10Release-Engineering-Team (Seen), 10Zuul, 10Patch-For-Review: Display Zuul status of jobs for a change on Gerrit UI - https://phabricator.wikimedia.org/T214068 (10hashar) I have posted to wikitech-l how one can try out the plugin locally: https://lists.... [13:53:48] 10Continuous-Integration-Config: test-prio job for operations/puppet/+/863332/2 stuck - https://phabricator.wikimedia.org/T324394 (10hashar) Indeed I have restarted Zuul after I got ping on IRC about some merge failures occurring. It is indeed the same as T309376. As for why the job got stuck, I don't have a re... [13:58:35] hashar: aha. d'oh. thanks [14:10:33] :) [14:18:51] could someone point me towards the docs for how `scap backport` uses mw-on-k8s? :) [14:37:44] ^ related there was some discussion in #wikimedia-operations earlier about how `scap backport` is taking considerably longer than it used to due to docker image building, and some notice to deployers would be appreciated! [14:38:27] admittedly it was "just" the initial build [14:39:31] 10Beta-Cluster-Infrastructure, 10Growth-Team, 10Growth-Team-Filtering, 10PageTriage, and 5 others: Set wgPageTriageEnableEnglishWikipediaFeatures to False on the Beta Cluster - https://phabricator.wikimedia.org/T321922 (10TheresNoTime) a:03Stang [14:43:44] it was fairly slow in the UTC morning window [14:51:05] TheresNoTime: scap syncing operations are now deploying to mw-on-k8s, we don't have docs specific to that at the moment [14:51:19] kostajh: I just saw building and pushing the image took 6m, sorry about that, in our tests with manual backports it was faster :( [14:51:35] I don't really know why the push took so long when the delta for backports is relatively small, we will need to look into it [14:51:49] also apologies for not having given a heads-up, I will send out an email in a bit [14:53:25] Is awesome to see this happening though! [14:57:25] yeah :D [15:01:33] jnuche: no worries! yeah a short announcement would be great. and +1 to what TheresNoTime said [15:44:53] 10Gerrit, 10Sustainability (Incident Followup): Investigate Gerrit h2 cache being way too large - https://phabricator.wikimedia.org/T323754 (10hashar) After looking at the H2 Database source code 'see below) we might be able to set `MAX_COMPACT_TIME` via a system setting: `-Dh2.maxCompactTime`. It is not expl... [16:02:25] I'm having some fun trying to write a helm chart... [16:05:03] Kind of stuck on one quirk at the moment, using the new generator. I'm just including a configMap volume but found that I needed templates to define the volume, so it had to be moved out of values.yaml [16:05:39] Blindly guessing, I tried putting the config volume into templates/deployment.yaml but now I don't see that this file gets included at all... [16:06:01] I suppose I'll hardcode the volume name just to get past this blockage. [16:09:40] 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃), 10Scap: Ensure efficient Gitlab CI operations for scap - https://phabricator.wikimedia.org/T323140 (10dancy) [16:09:52] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃): Try DigitalOcean registry for buildkit caching - https://phabricator.wikimedia.org/T323148 (10dancy) 05Open→03Declined Cancelling this idea. The D.O. registry will not give us the required level of control. [16:10:20] 10GitLab (Infrastructure), 10serviceops-collab, 10Cloud-VPS (Quota-requests), 10User-dcaro: Request quota increase for devtools Cloud VPS project - https://phabricator.wikimedia.org/T323986 (10LSobanski) [16:34:09] 10Continuous-Integration-Config, 10MediaWiki-Core-Tests, 10Browser-Tests, 10User-zeljkofilipin: Jenkins job should fail when all (selenium) tests are skipped - https://phabricator.wikimedia.org/T324480 (10zeljkofilipin) [16:42:45] 10Phabricator, 10Release-Engineering-Team, 10Diffusion-Repository-Administrators, 10serviceops-collab: Audit Diffusion-Repository-Administrators group membership and rights - https://phabricator.wikimedia.org/T324171 (10LSobanski) [16:43:19] 10Phabricator, 10Release-Engineering-Team, 10Diffusion-Repository-Administrators, 10serviceops-collab: Audit Diffusion-Repository-Administrators group membership and rights - https://phabricator.wikimedia.org/T324171 (10LSobanski) p:05Triage→03Medium [16:46:18] 10Gerrit, 10Release-Engineering-Team (Seen), 10SRE, 10serviceops-collab: Create Gerrit Administrator right policy - https://phabricator.wikimedia.org/T218686 (10LSobanski) p:05Medium→03Low [16:53:12] 10Diffusion, 10Phabricator, 10serviceops-collab, 10Patch-For-Review: Redirect https://phabricator.wikimedia.org/r/ to https://gerrit.wikimedia.org/g/ - https://phabricator.wikimedia.org/T324311 (10LSobanski) a:05Aklapper→03Dzahn [17:00:03] 10Release-Engineering-Team, 10SRE, 10serviceops-collab: Redirect revisions from svn.wikimedia.org to https://static-codereview.wikimedia.org - https://phabricator.wikimedia.org/T119846 (10LSobanski) [17:00:23] 10Diffusion, 10Release-Engineering-Team, 10SRE, 10serviceops-collab: svn.wikimedia.org redirects to Diffusion main page, hence hard to find e.g. "flexbisonparse" - https://phabricator.wikimedia.org/T140594 (10LSobanski) [17:00:47] 10Release-Engineering-Team, 10SRE, 10serviceops-collab: Redirect revisions from svn.wikimedia.org to https://static-codereview.wikimedia.org - https://phabricator.wikimedia.org/T119846 (10LSobanski) p:05Low→03Lowest [17:10:31] 10Continuous-Integration-Config, 10MediaWiki-Core-Tests, 10Browser-Tests, 10User-zeljkofilipin: Jenkins job should fail when all (selenium) tests are skipped - https://phabricator.wikimedia.org/T324480 (10zeljkofilipin) [18:10:18] (03CR) 10Ladsgroup: jjb: Make wikimedia-portals-build job rebase (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/856033 (owner: 10Ladsgroup) [18:34:18] 10Release-Engineering-Team (Seen), 10serviceops, 10serviceops-collab: switch contint prod server back from contint2001 to contint1001 - https://phabricator.wikimedia.org/T256422 (10Dzahn) [18:48:58] 10GitLab (Project Migration), 10Release-Engineering-Team: Create new GitLab project group: abstract-wiki - https://phabricator.wikimedia.org/T323957 (10brennen) 05Open→03Resolved a:03brennen Created group at https://gitlab.wikimedia.org/admin/groups/repos/abstract-wiki - added folks from team list with e... [19:20:17] 10Continuous-Integration-Infrastructure, 10Jenkins, 10SRE, 10SRE-Access-Requests, 10serviceops-collab: New Keyholder identity for RelEng Jenkins service - https://phabricator.wikimedia.org/T324014 (10Dzahn) [19:20:24] 10Continuous-Integration-Infrastructure, 10Jenkins, 10SRE, 10SRE-Access-Requests, 10serviceops-collab: New Keyholder identity for RelEng Jenkins service - https://phabricator.wikimedia.org/T324014 (10Dzahn) a:03Dzahn [19:22:55] 10Continuous-Integration-Infrastructure, 10Jenkins, 10SRE, 10SRE-Access-Requests, 10serviceops-collab: New Keyholder identity for RelEng Jenkins service - https://phabricator.wikimedia.org/T324014 (10Dzahn) p:05Triage→03Medium [19:36:25] brennen: phab1001 is about to be shut down forever. I made a final copy of /srv/deployment, /etc/ and /root (heh) and copied those to /srv/homes on phab1004. Any concerns or comments what else to do before I actually destroy it? [19:40:21] mutante: good by me - no concerns i can think of. [19:40:32] 10GitLab (Project Migration), 10Wikimedia-GitHub, 10Epic, 10User-AKlapper: Migrate active Wikimedia repositories in GitHub to GitLab - https://phabricator.wikimedia.org/T305039 (10nshahquinn-wmf) [19:40:35] 10GitLab (Project Migration), 10Product-Analytics, 10Wmfdata-Python: Move Wmfdata-Python from Github to Gitlab - https://phabricator.wikimedia.org/T304544 (10nshahquinn-wmf) [19:41:42] brennen: :) thanks! [19:47:13] 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃), 10Scap, 10Developer Productivity: Add support for "gerrit/r/:changenum" URLs to scap backport command - https://phabricator.wikimedia.org/T323320 (10dancy) 05Open→03Resolved Included in scap 4.30.0 which has been deployed. [20:47:12] 10Phabricator, 10decommission-hardware, 10serviceops-collab, 10Patch-For-Review: decommission phab1001.eqiad.wmnet - https://phabricator.wikimedia.org/T323418 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by dzahn@cumin2002 for hosts: `phab1001.eqiad.wmnet` - phab1001.eqiad.wmnet (**WARN... [21:50:35] 10Phabricator, 10decommission-hardware, 10ops-eqiad, 10serviceops-collab: decommission phab1001.eqiad.wmnet - https://phabricator.wikimedia.org/T323418 (10Dzahn) [21:51:07] 10Phabricator, 10decommission-hardware, 10ops-eqiad, 10serviceops-collab: decommission phab1001.eqiad.wmnet - https://phabricator.wikimedia.org/T323418 (10Dzahn) [21:51:49] 10Phabricator, 10decommission-hardware, 10ops-eqiad, 10serviceops-collab: decommission phab1001.eqiad.wmnet - https://phabricator.wikimedia.org/T323418 (10Dzahn) a:05Dzahn→03Jclark-ctr https://netbox.wikimedia.org/dcim/devices/1557/ has been permanently shut down [22:03:56] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 10serviceops-collab, 10Patch-For-Review: move phabricator to new hardware generation - https://phabricator.wikimedia.org/T280597 (10Dzahn) 05In progress→03Resolved [22:04:46] 10Project-Admins: Create project tag for RealMe - https://phabricator.wikimedia.org/T324518 (10taavi) [22:11:12] 10GitLab, 10serviceops-collab: Optimize Gitlab Backups - https://phabricator.wikimedia.org/T324506 (10Dzahn) [22:31:32] (03CR) 10Ladsgroup: jjb: Make wikimedia-portals-build job rebase (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/856033 (owner: 10Ladsgroup) [22:39:33] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃), 10User-brennen: Sandbox task for gitlab-phabricator comment integration - https://phabricator.wikimedia.org/T324164 (10brennen) Commit [[https://gitlab.wikimedia.org/brennen/test/-/commit/3909673995e55586fcd908... [23:19:37] 10Project-Admins: Create project tag for RealMe - https://phabricator.wikimedia.org/T324518 (10Aklapper) 05Open→03Resolved a:03Aklapper Requested public project #Realme has been created. Interested people are welcome to join the project as {icon users} members, and to [watch the project](https://www.media... [23:56:50] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team (GitLab III: GitLab in LA 🪃), 10User-brennen: Sandbox task for gitlab-phabricator comment integration - https://phabricator.wikimedia.org/T324164 (10brennen) Commit [[https://gitlab.wikimedia.org/brennen/test/-/commit/436323ecac8910c1252f51...