[02:30:33] maintenance-disconnect-full-disks build 502159 integration-agent-docker-1032 (/: 29%, /srv: 96%, /var/lib/docker: 51%): OFFLINE due to disk space [02:35:31] maintenance-disconnect-full-disks build 502160 integration-agent-docker-1032 (/: 29%, /srv: 91%, /var/lib/docker: 51%): RECOVERY disk space OK [04:45:30] maintenance-disconnect-full-disks build 502186 integration-agent-docker-1032 (/: 29%, /srv: 100%, /var/lib/docker: 47%): OFFLINE due to disk space [05:00:30] maintenance-disconnect-full-disks build 502189 integration-agent-docker-1032 (/: 29%, /srv: 84%, /var/lib/docker: 46%): RECOVERY disk space OK [05:49:12] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Cloud-VPS (Quota-requests): Rebuild WMCS insegration instances to larger flavor - https://phabricator.wikimedia.org/T340070 (10hashar) The previous flavor g3.cores8.ram24.disk20.ephemeral60.4xiops was created as: From T299704#7652833: >... [05:57:40] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Quibble, 10castor: Figure out how to garbage collect the npm cache - https://phabricator.wikimedia.org/T340092 (10hashar) [07:19:04] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Quibble, 10castor: Figure out how to garbage collect the npm cache - https://phabricator.wikimedia.org/T340092 (10kostajh) > Which looks like maybe we should have Quibble to garbage collect for us as a tear down command when the build i... [07:51:57] 10GitLab (Project Migration), 10Release-Engineering-Team (They Live 🕶️🧟), 10Infrastructure-Foundations, 10serviceops-collab: New LDAP user to trigger Jenkins downstream jobs - https://phabricator.wikimedia.org/T338950 (10jnuche) User is not showing up yet in queries: ` jnuche@mwmaint1002:~$ ldapsearch -x... [07:55:40] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Cloud-VPS (Quota-requests): Rebuild WMCS integration instances to larger flavor - https://phabricator.wikimedia.org/T340070 (10hashar) [07:57:16] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Cloud-VPS (Quota-requests): Rebuild WMCS integration instances to larger flavor - https://phabricator.wikimedia.org/T340070 (10hashar) Holding on that, after some deeper investigation, the root cause for the instance overflowing disk is t... [08:02:39] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Quibble, 10castor: Figure out how to garbage collect the npm cache - https://phabricator.wikimedia.org/T340092 (10hashar) I am trying to understand how `npm cache verify` does the garbage collection. The cache is a content addressable o... [08:39:18] 10GitLab (Project Migration), 10Release-Engineering-Team (They Live 🕶️🧟), 10Infrastructure-Foundations, 10serviceops-collab: New LDAP user to trigger Jenkins downstream jobs - https://phabricator.wikimedia.org/T338950 (10jbond) CC @SLyngshede-WMF [08:44:29] 10GitLab (Project Migration), 10Release-Engineering-Team (They Live 🕶️🧟), 10Infrastructure-Foundations, 10serviceops-collab: New LDAP user to trigger Jenkins downstream jobs - https://phabricator.wikimedia.org/T338950 (10SLyngshede-WMF) I see the error at least, let me just see if I made some mistake durin... [08:54:28] 10GitLab (Project Migration), 10Release-Engineering-Team (They Live 🕶️🧟), 10Infrastructure-Foundations, 10serviceops-collab: New LDAP user to trigger Jenkins downstream jobs - https://phabricator.wikimedia.org/T338950 (10SLyngshede-WMF) Okay, so the issue is that it's created with releng+jenkins-rel-api@li... [08:57:37] 10GitLab (Project Migration), 10Release-Engineering-Team (They Live 🕶️🧟), 10Infrastructure-Foundations, 10serviceops-collab: New LDAP user to trigger Jenkins downstream jobs - https://phabricator.wikimedia.org/T338950 (10SLyngshede-WMF) I've deleted the failed signup, to free up the username which will oth... [09:16:36] 10GitLab (Project Migration), 10Release-Engineering-Team (They Live 🕶️🧟), 10Infrastructure-Foundations, 10serviceops-collab: New LDAP user to trigger Jenkins downstream jobs - https://phabricator.wikimedia.org/T338950 (10jnuche) @SLyngshede-WMF you are right about the email, I had used `releng@lists.wikime... [09:18:35] 10GitLab (Project Migration), 10Release-Engineering-Team (They Live 🕶️🧟), 10Infrastructure-Foundations, 10serviceops-collab: New LDAP user to trigger Jenkins downstream jobs - https://phabricator.wikimedia.org/T338950 (10SLyngshede-WMF) @jnuche Thank you for helping us testing out the user creation, it nic... [09:25:28] 10GitLab (Project Migration), 10Release-Engineering-Team (They Live 🕶️🧟), 10Infrastructure-Foundations, 10serviceops-collab: New LDAP user to trigger Jenkins downstream jobs - https://phabricator.wikimedia.org/T338950 (10jnuche) [09:37:43] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Quibble, 10castor: Figure out how to garbage collect the npm cache - https://phabricator.wikimedia.org/T340092 (10hashar) Grepping through `npm/_cacache/index-v5`, each files has a list of keys stored and there are a bunch of duplicates... [09:52:41] 10GitLab (Project Migration), 10Release-Engineering-Team (They Live 🕶️🧟), 10Infrastructure-Foundations, 10serviceops-collab: New LDAP user to trigger Jenkins downstream jobs - https://phabricator.wikimedia.org/T338950 (10jnuche) 05Open→03Resolved a:03jnuche [09:52:44] 10GitLab (Project Migration), 10Release-Engineering-Team (They Live 🕶️🧟), 10serviceops-collab: Provide mechanism to publish to doc.wikimedia.org from GitLab CI - https://phabricator.wikimedia.org/T336168 (10jnuche) [09:52:51] 10GitLab (Project Migration), 10Release-Engineering-Team (They Live 🕶️🧟), 10Infrastructure-Foundations, 10serviceops-collab: New LDAP user to trigger Jenkins downstream jobs - https://phabricator.wikimedia.org/T338950 (10jnuche) a:05jnuche→03None [10:00:42] 10GitLab (Project Migration), 10Release-Engineering-Team (They Live 🕶️🧟), 10serviceops-collab: Provide mechanism to publish to doc.wikimedia.org from GitLab CI - https://phabricator.wikimedia.org/T336168 (10MoritzMuehlenhoff) [10:01:00] 10GitLab (Project Migration), 10Release-Engineering-Team (They Live 🕶️🧟), 10Infrastructure-Foundations, 10serviceops-collab: New LDAP user to trigger Jenkins downstream jobs - https://phabricator.wikimedia.org/T338950 (10MoritzMuehlenhoff) 05Resolved→03Open We still need to move the user to the ou=sysu... [10:19:43] (03PS1) 10Hashar: jjb: gc quibble jobs npm cache before saving it [integration/config] - 10https://gerrit.wikimedia.org/r/932195 (https://phabricator.wikimedia.org/T340092) [10:25:31] maintenance-disconnect-full-disks build 502254 integration-agent-docker-1030 (/: 29%, /srv: 95%, /var/lib/docker: 48%): OFFLINE due to disk space [10:26:37] 10GitLab (Project Migration), 10Release-Engineering-Team (They Live 🕶️🧟), 10Infrastructure-Foundations, 10serviceops-collab: New LDAP user to trigger Jenkins downstream jobs - https://phabricator.wikimedia.org/T338950 (10hashar) Looks like `ou=sysusers,,dc=wikimedia,dc=org` is a new feature and exceeds the... [10:30:32] maintenance-disconnect-full-disks build 502255 integration-agent-docker-1030 (/: 29%, /srv: 62%, /var/lib/docker: 47%): RECOVERY disk space OK [10:53:30] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Quibble, 10Patch-For-Review, 10castor: Figure out how to garbage collect the npm cache - https://phabricator.wikimedia.org/T340092 (10hashar) I went to try with https://gerrit.wikimedia.org/r/mediawiki/extensions/TimedMediaHandler doi... [10:56:36] (03CR) 10Hashar: "I need to pick up a job for testing. The cache in castor will need to be compared before and after a save." [integration/config] - 10https://gerrit.wikimedia.org/r/932195 (https://phabricator.wikimedia.org/T340092) (owner: 10Hashar) [11:19:13] 10GitLab (Project Migration), 10Release-Engineering-Team (They Live 🕶️🧟), 10Infrastructure-Foundations, 10serviceops-collab: New LDAP user to trigger Jenkins downstream jobs - https://phabricator.wikimedia.org/T338950 (10MoritzMuehlenhoff) >>! In T338950#8955323, @hashar wrote: > Looks like `ou=sysusers,,d... [11:20:10] 10GitLab (Project Migration), 10Release-Engineering-Team (They Live 🕶️🧟), 10serviceops-collab: Provide mechanism to publish to doc.wikimedia.org from GitLab CI - https://phabricator.wikimedia.org/T336168 (10MoritzMuehlenhoff) [11:20:25] 10GitLab (Project Migration), 10Release-Engineering-Team (They Live 🕶️🧟), 10Infrastructure-Foundations, 10serviceops-collab: New LDAP user to trigger Jenkins downstream jobs - https://phabricator.wikimedia.org/T338950 (10MoritzMuehlenhoff) 05Open→03Resolved a:03MoritzMuehlenhoff The user has been mov... [11:24:15] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Cloud-VPS (Quota-requests): Rebuild WMCS integration instances to larger flavor - https://phabricator.wikimedia.org/T340070 (10hashar) [11:24:19] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Quibble, 10Patch-For-Review, 10castor: Figure out how to garbage collect the npm cache - https://phabricator.wikimedia.org/T340092 (10hashar) [11:25:02] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Cloud-VPS (Quota-requests): Rebuild WMCS integration instances to larger flavor - https://phabricator.wikimedia.org/T340070 (10hashar) 05Open→03Stalled Stalled pending outcome of running `npm cache verify` to garbage collect the huge... [11:40:20] 10GitLab (Pipeline Services Migration🐤), 10Release-Engineering-Team, 10MediaWiki-Docker, 10dev-images, and 3 others: MySQL/MariaDB images for development environments - https://phabricator.wikimedia.org/T238925 (10kostajh) [14:22:26] 10GitLab (Pipeline Services Migration🐤), 10Release-Engineering-Team, 10MediaWiki-Docker, 10dev-images, and 3 others: MySQL/MariaDB images for development environments - https://phabricator.wikimedia.org/T238925 (10thcipriani) >>! In T238925#8939278, @kostajh wrote: > Does #release-engineering-team have sug... [14:32:54] btw, is it intentional that the “(don’t) keep this build forever” button in Jenkins is now much less prominent / visible than it used to be? [14:56:07] 10GitLab (Pipeline Services Migration🐤), 10Release-Engineering-Team, 10MediaWiki-Docker, 10dev-images, and 3 others: MySQL/MariaDB images for development environments - https://phabricator.wikimedia.org/T238925 (10hashar) I have done some work on spawning a MySQL database for Quibble and thus MediaWiki. Ma... [14:56:21] 10GitLab (Pipeline Services Migration🐤), 10Release-Engineering-Team, 10MediaWiki-Docker, 10dev-images, and 3 others: MySQL/MariaDB images for development environments - https://phabricator.wikimedia.org/T238925 (10hashar) Else go with the official MariaDB image. I have lost track but I am pretty sure we ca... [15:11:10] Lucas_WMDE: we have upgraded Jenkins last week which came with a bunch of UI changes [15:11:54] and yeah the light gray background of the button against a white background lacks contrast :] [15:12:13] it does, but I thought perhaps it’s intentional to stop people from clicking it too much ;) [15:12:36] * Lucas_WMDE has previously accidentally deleted a build that was being kept permanently, by clicking the button that wasn’t in the state I thought it was in [15:13:01] but I don’t even know if that’s a standard jenkins feature or a custom extension of ours [15:13:20] `--button-background: rgba(175,176,207,.1)` [15:20:31] that is `hsla(240, 25%, 75%, 0.1)` which is #afafcf with 10% opacity [15:20:35] and it was darker previously for sure [15:21:28] Lucas_WMDE: https://github.com/jenkinsci/jenkins/commit/cb232e8a7c330166243ae87c4bcfb651c83d7060#diff-f8fe658d4393c0c5d8f2d078430d99018bf8165b671e9d916065c8a092d638b8R316-R320 :] [15:22:28] which is https://github.com/jenkinsci/jenkins/pull/6799 where someone complains that previously they were blue indicating a clickable action [15:24:02] and their design is a plugin which can be browseable on the weekly demo instance: https://weekly.ci.jenkins.io/design-library/Buttons/ [15:24:25] so yeah I guess the keep this build forever should have the class `jenkins-button--primary` applied [15:25:11] yeah that looks much more familiar [15:25:15] (if I try it in dev tools) [15:26:29] hm, though it was made explicitly non-primary in https://github.com/jenkinsci/jenkins/commit/5870838f690eaa06762630eceb21c40d3df0ceb8#diff-146545849e9be8a5d41dfe0df7c6aa9d2b43582f8013ff7dfb82b87bc69cede5 [15:26:55] https://github.com/jenkinsci/jenkins/pull/7203/commits/b7b37d54d975a908ccfb0ca2c57bfe9b1fbe01e4 [15:28:47] nice dig :] [15:29:28] maybe that one got screwed up due to mass replace [15:30:56] should I try to file an issue or just send them a lil pull request? [15:31:01] * Lucas_WMDE hasn’t interacted with Jenkins maintainers before [15:31:14] if you feel adventure you can propose a PR against https://github.com/jenkinsci/jenkins/ which revert that b7b37d54d975a908ccfb0ca2c57bfe9b1fbe01e4 commit [15:31:24] possibly quoting https://github.com/jenkinsci/jenkins/pull/6799#issuecomment-1217526980 which complains about the blue disappearance [15:31:38] their issue tracker is Jira :] [15:31:41] so I would go with a PR [15:31:45] :D :D :D [15:31:48] yup [15:31:50] * Lucas_WMDE stays away from jira [15:32:01] I don't think they require any CLA [15:32:17] there is a long PR check list though but for a css I don't think any item would apply [15:32:39] so I guess revert, send to a branch of your fork, hit PR with some justification "actions should be blue per design guide" [15:32:54] and then you officially become a java (sic) developer [15:33:16] I think the change was unintentional, I don't see why it should not be blue [15:35:43] funny thing they have `jenkins-button--primary` (blue) and `jenkins-button--tertiary` (transparent, light gray on hover) but there is no `jenkins-button--secondary` :-] [15:36:01] o_O [15:36:37] ~~~there hasn’t been a secondary button state since the collapse of the soviet union~~~ [15:36:38] but https://weekly.ci.jenkins.io/design-library/Buttons/ is nice :] [15:36:51] maybe we could have a MediaWiki extension showing the design rules [15:38:23] “We do not require Jira issues for minor improvements.” yay [15:39:43] \o/ [15:45:05] hashar: https://github.com/janfaracik/jenkins/pull/134 [15:45:09] oh shit [15:45:19] I was wondering “why is the PR number so low” [15:45:29] because I opened it against the wrong fork >.< [15:47:46] https://github.com/jenkinsci/jenkins/pull/8178 [15:49:19] HAHA [15:49:22] wrong fork! [15:50:01] https://images.squarespace-cdn.com/content/v1/5e0ab605317b09689f136846/1614022825838-EB67I5YKFTBK2VQK6HVH/tumblr_ee0643258b187ab7f9ec3088fd17de7f_670c9eaa_500.gif [15:53:28] Lucas_WMDE: well done! [15:54:26] https://gitpod.io/#https://github.com/jenkinsci/jenkins/pull/8178 Open a cloud development environment for the repository jenkinsci/jenkins/pull/8178 [15:54:29] that looks like our patchdemo [15:54:52] neat [15:55:17] but apparently I’d need to sign up first [15:55:32] oh no it is a dev environment with all dependencies, tools provided [15:55:37] so click, ready [15:56:12] locally I use `mvn hpi:run` which, after it downloads half of the internet, get me a running junkins on http://127.0.0.1:8080/ [15:56:18] s/junkins/jenkins/ (typo) [15:56:27] freudian? ;) [15:56:43] ? [15:56:55] freudian typo, insulting jenkins as junk, maybe ;) [15:57:24] ah yeah maybe [15:57:42] it is not that bad really [15:57:56] a few years ago (say like 10/8 years ago) Jenkins definitely had large troubles here and there [15:58:36] but it has been fairly stable for more than a few years and upgrades are usually straightforward (save for plugins dependency managements but well ... that is dependency management) [16:04:35] * hashar out for virtual offsite [16:20:03] 10Release-Engineering-Team (Radar), 10Foundational Technology Requests: Deploy to Wikimedia Production from GitLab - https://phabricator.wikimedia.org/T315253 (10Jdforrester-WMF) Is this "Provide an integration with GitLab so that users can make the currently-manual steps (`deployment-charts` bump/merge plus `... [17:05:09] (03CR) 10Hashar: [C: 03+2] zuul: Add WikiEditor dependency for Cite [integration/config] - 10https://gerrit.wikimedia.org/r/932001 (owner: 10Jon Harald Søby) [17:06:23] (03Merged) 10jenkins-bot: zuul: Add WikiEditor dependency for Cite [integration/config] - 10https://gerrit.wikimedia.org/r/932001 (owner: 10Jon Harald Søby) [17:08:39] (03CR) 10Hashar: [C: 03+2] "Deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/932001 (owner: 10Jon Harald Søby) [17:24:47] (03CR) 10Jon Harald Søby: zuul: Add WikiEditor dependency for Cite (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/932001 (owner: 10Jon Harald Søby) [17:36:59] xpost from -serviceops as I guess it's beta-related: question about T340087 if someone in the know has a moment — solely out of curiosity, how would a new deb/build of wikidiff2 be deployed to the beta cluster? [17:36:59] T340087: Deploy wikidiff2 1.14.0 - https://phabricator.wikimedia.org/T340087 [18:10:40] 10GitLab (Upstream pit of despair 🕳️), 10Upstream: GitLab truncates commit messages over 1k of text - https://phabricator.wikimedia.org/T330790 (10Aklapper) This seems to be fixed nowadays? At least we could not reproduce today in a meeting [19:09:44] TheresNoTime: very late response (you may have already gotten a smarter one by now) after it's uploaded to wikimedia's apt repos via https://wikitech.wikimedia.org/wiki/Reprepro we should be able to apt install --upgrade php-wikidiff2 (although there may be some nuance that makes this hard, like debian version, etc) [20:18:55] thcipriani: TheresNoTime: the beta cluster has Debian unattended upgrades system enabled on all instances [20:19:05] via hiera and from cloud/instance-puppet.git: [20:19:15] deployment-prep/_.roles:- apt::unattendedupgrades [20:19:54] which should kick in automatic apt update && apt upgrade every day at something like 4 or 5 am utc [20:20:13] else what tyler said :) [21:14:36] 10Phabricator, 10serviceops-collab: consider moving aphlict admin port to https / envoy - https://phabricator.wikimedia.org/T340169 (10Dzahn) [22:48:41] 10Release-Engineering-Team (Radar), 10Foundational Technology Requests: Deploy to Wikimedia Production from GitLab - https://phabricator.wikimedia.org/T315253 (10thcipriani) 05Open→03Resolved >>! In T315253#8956451, @Jdforrester-WMF wrote: > Is this "Provide an integration with GitLab so that users can mak...