[00:04:04] 10Phabricator, 10Striker, 10cloud-services-team (Kanban): Unable to mirror repository from git.legoktm.com into diffusion - https://phabricator.wikimedia.org/T143969 (10Legoktm) 05Open→03Declined Yep :/ [07:51:49] (03CR) 10Hashar: [C: 03+2] "That was done yesterday" [integration/config] - 10https://gerrit.wikimedia.org/r/849118 (owner: 10Hashar) [07:53:39] (03Merged) 10jenkins-bot: jjb: update to Quibble 1.4.7 [integration/config] - 10https://gerrit.wikimedia.org/r/849118 (owner: 10Hashar) [07:54:40] kostajh: good morning, I am around for dropping QUIBBLE_APACHE=1 for GrowthExperiments api-testing ( https://gerrit.wikimedia.org/r/c/integration/config/+/843421 ) [07:55:11] I am not sure which patch to try it out against. Please ping me when you have some spare time [08:43:04] (03PS1) 10AikoChou: inference-services: add revertrisk pipelines [integration/config] - 10https://gerrit.wikimedia.org/r/849480 (https://phabricator.wikimedia.org/T321594) [08:55:18] (03CR) 10Hashar: [C: 03+2] "I have deployed the jobs:" [integration/config] - 10https://gerrit.wikimedia.org/r/849480 (https://phabricator.wikimedia.org/T321594) (owner: 10AikoChou) [08:58:15] (03Merged) 10jenkins-bot: inference-services: add revertrisk pipelines [integration/config] - 10https://gerrit.wikimedia.org/r/849480 (https://phabricator.wikimedia.org/T321594) (owner: 10AikoChou) [08:58:31] !log Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/849480 # T321594 [08:58:33] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:58:34] T321594: Deploy revert-risk-model to production - https://phabricator.wikimedia.org/T321594 [08:58:50] hashar: <3 [08:58:55] grr [08:59:03] (for the inference-services patch) [08:59:09] reloaded now ;) [08:59:14] I made a mistake :D [08:59:37] elukey: you are welcome! [09:01:04] elukey: I did a `recheck` on https://gerrit.wikimedia.org/r/c/machinelearning/liftwing/inference-services/+/849478 [09:01:47] something something is running at https://integration.wikimedia.org/ci/blue/organizations/jenkins/inference-services-pipeline-revertrisk/detail/inference-services-pipeline-revertrisk/1/pipeline [09:05:57] 10Beta-Cluster-Infrastructure, 10Traffic, 10Beta-Cluster-reproducible: Thumbnails on beta cluster return 503 Service Unavailable - https://phabricator.wikimedia.org/T321654 (10AlexisJazz) [09:13:37] (03PS1) 10Hashar: zuul: add Dinar Qurbanov to allow list [integration/config] - 10https://gerrit.wikimedia.org/r/849489 (https://phabricator.wikimedia.org/T27537) [09:13:50] (03CR) 10Hashar: [C: 03+2] zuul: add Dinar Qurbanov to allow list [integration/config] - 10https://gerrit.wikimedia.org/r/849489 (https://phabricator.wikimedia.org/T27537) (owner: 10Hashar) [09:16:14] (03Merged) 10jenkins-bot: zuul: add Dinar Qurbanov to allow list [integration/config] - 10https://gerrit.wikimedia.org/r/849489 (https://phabricator.wikimedia.org/T27537) (owner: 10Hashar) [09:29:47] 10Beta-Cluster-Infrastructure, 10SRE, 10Traffic, 10Beta-Cluster-reproducible: Thumbnails on beta cluster return 503 Service Unavailable - https://phabricator.wikimedia.org/T321654 (10TheresNoTime) If someone could purge `https://upload.wikimedia.beta.wmflabs.org/wikipedia/en/thumb/1/13/Bert_Self-portrait2.... [09:41:29] 10Beta-Cluster-Infrastructure, 10SRE, 10Traffic, 10Beta-Cluster-reproducible: Thumbnails on beta cluster return 503 Service Unavailable - https://phabricator.wikimedia.org/T321654 (10Vgutierrez) I've followed the steps mentioned by @TheresNoTime but sadly it didn't help at all. Please consider that varnish... [09:46:49] 10Beta-Cluster-Infrastructure, 10SRE, 10Beta-Cluster-reproducible: Thumbnails on beta cluster return 503 Service Unavailable - https://phabricator.wikimedia.org/T321654 (10Vgutierrez) Removing Traffic since haproxy/varnish/ATS isn't at fault here. [09:51:44] 10Beta-Cluster-Infrastructure, 10SRE, 10Beta-Cluster-reproducible: Thumbnails on beta cluster return 503 Service Unavailable - https://phabricator.wikimedia.org/T321654 (10TheresNoTime) Thanks @Vgutierrez, I'll take a closer look in a moment, but just noting from `deployment-ms-be05`: ` Oct 26 09:49:16 deplo... [09:53:05] 10Beta-Cluster-Infrastructure, 10SRE, 10Beta-Cluster-reproducible: Thumbnails on beta cluster return 503 Service Unavailable - https://phabricator.wikimedia.org/T321654 (10Vgutierrez) @TheresNoTime ats-be in deployment-cache-upload07 forwards the requests to deployment-ms-fe03.deployment-prep.eqiad.wmflabs... [09:59:29] 10Beta-Cluster-Infrastructure, 10SRE, 10Beta-Cluster-reproducible: Thumbnails on beta cluster return 503 Service Unavailable - https://phabricator.wikimedia.org/T321654 (10AlexisJazz) "If someone could purge.. that should fix it" Even if it worked, I'd rather not do that whenever I want to see a thumbnail o... [10:02:09] 10Beta-Cluster-Infrastructure, 10SRE, 10Beta-Cluster-reproducible: Thumbnails on beta cluster return 503 Service Unavailable - https://phabricator.wikimedia.org/T321654 (10RhinosF1) >>! In T321654#8344934, @AlexisJazz wrote: > "If someone could purge.. that should fix it" > > Even if it worked, I'd rather n... [10:03:34] 10Beta-Cluster-Infrastructure, 10SRE, 10Beta-Cluster-reproducible: Thumbnails on beta cluster return 503 Service Unavailable - https://phabricator.wikimedia.org/T321654 (10TheresNoTime) ` samtar@deployment-ms-fe03:~$ swift list wikipedia-en-local-thumb.13 1/13/Bert_Self-portrait2.png/150px-Bert_Self-portrait... [10:38:09] (03CR) 10Kosta Harlan: [C: 03+1] jjb: Deployed version of Quibble has QUIBBLE_APACHE flag (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/843421 (https://phabricator.wikimedia.org/T320935) (owner: 10Kosta Harlan) [10:43:19] hashar: hi, sorry I just saw your message from the morning [10:44:25] hashar: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/849509 would be a patch to check for the QUIBBLE_APACHE change [10:46:43] kostajh: great ;) [10:48:32] (03PS1) 10Hashar: Upgrade jjb to 3.12.0 [integration/config] - 10https://gerrit.wikimedia.org/r/849504 [10:52:12] (03CR) 10Hashar: [C: 03+2] "Noop as expected" [integration/config] - 10https://gerrit.wikimedia.org/r/849504 (owner: 10Hashar) [10:54:29] (03Merged) 10jenkins-bot: Upgrade jjb to 3.12.0 [integration/config] - 10https://gerrit.wikimedia.org/r/849504 (owner: 10Hashar) [10:56:55] 10Continuous-Integration-Config, 10MediaWiki-extensions-General: Please add mediawiki-i18n-check-docker to repository mediawiki/extensions/PageAuthors - https://phabricator.wikimedia.org/T321668 (10Nikerabbit) [11:17:32] 10Phabricator: Changing Username (Self; to kayxspre) - https://phabricator.wikimedia.org/T321673 (10Vg_thWIKI) [12:44:22] 10Continuous-Integration-Config, 10MediaWiki-extensions-General: Please add mediawiki-i18n-check-docker to repository mediawiki/extensions/PageAuthors - https://phabricator.wikimedia.org/T321668 (10Sophivorus) Hi! I'd like to help but I'm not sure to what file I have to add the mediawiki-i18n-check-docker dire... [13:07:28] 10Phabricator: Changing Username (Self; to kayxspre) - https://phabricator.wikimedia.org/T321673 (10Aklapper) 05Open→03Resolved a:03Aklapper Sure, done! [13:43:00] (03PS1) 10Hashar: Add .gitreview [releng/phatality] - 10https://gerrit.wikimedia.org/r/849549 [13:43:02] (03PS1) 10Hashar: Dockerfile: remove artifacts after apt install [releng/phatality] - 10https://gerrit.wikimedia.org/r/849550 [13:43:04] (03PS1) 10Hashar: Dockerfile: use node:14 and use a shallow clone [releng/phatality] - 10https://gerrit.wikimedia.org/r/849551 [13:43:06] (03PS1) 10Hashar: Dockerfile: update to dashboards 2.2.0 [releng/phatality] - 10https://gerrit.wikimedia.org/r/849552 [13:43:09] (03PS1) 10Hashar: Dockerfile: remove CACHE_BUSTER and use a COPY instead [releng/phatality] - 10https://gerrit.wikimedia.org/r/849553 [13:43:14] jnuche: dancy: some tweaks to releng/phatality :-] [13:43:34] it was based on node:10 base image which is very very old and ships git 2.11 [13:44:58] (03CR) 10Jaime Nuche: [C: 03+2] Add .gitreview [releng/phatality] - 10https://gerrit.wikimedia.org/r/849549 (owner: 10Hashar) [13:45:46] (03CR) 10Jaime Nuche: [C: 03+2] Dockerfile: remove artifacts after apt install [releng/phatality] - 10https://gerrit.wikimedia.org/r/849550 (owner: 10Hashar) [13:45:59] (03CR) 10CI reject: [V: 04-1] Dockerfile: remove artifacts after apt install [releng/phatality] - 10https://gerrit.wikimedia.org/r/849550 (owner: 10Hashar) [13:46:14] oh there is some ci fun [13:46:44] OH [13:46:51] cause CI is unable to submit I think [13:48:12] jnuche: https://gerrit.wikimedia.org/r/c/releng/phatality/+/849516 ;) [13:48:42] I should move that repo [13:48:44] hmm [13:51:25] https://gerrit.wikimedia.org/r/c/releng/+/849517 :) [13:54:29] !log gerrit: changed all releng/* repositories to inherit rights from releng.git and granted rights to CI to submit change from there [13:54:36] thanks for sharing that, I didn't know that's how access in our repos CI is configured [13:54:53] (03CR) 10Hashar: [C: 04-1] "The build fails:" [releng/phatality] - 10https://gerrit.wikimedia.org/r/849551 (owner: 10Hashar) [13:59:17] I screwed up something in permissions apparently [14:01:31] (03CR) 10Hashar: [C: 03+2] Add .gitreview [releng/phatality] - 10https://gerrit.wikimedia.org/r/849549 (owner: 10Hashar) [14:01:59] (03Merged) 10jenkins-bot: Add .gitreview [releng/phatality] - 10https://gerrit.wikimedia.org/r/849549 (owner: 10Hashar) [14:02:01] (03Merged) 10jenkins-bot: Dockerfile: remove artifacts after apt install [releng/phatality] - 10https://gerrit.wikimedia.org/r/849550 (owner: 10Hashar) [14:03:47] (03CR) 10Jaime Nuche: Dockerfile: use node:14 and use a shallow clone (031 comment) [releng/phatality] - 10https://gerrit.wikimedia.org/r/849551 (owner: 10Hashar) [14:08:45] jnuche: I will reorder those patches [14:09:37] (03Abandoned) 10Hashar: Dockerfile: update to dashboards 2.2.0 [releng/phatality] - 10https://gerrit.wikimedia.org/r/849552 (owner: 10Hashar) [14:20:04] Unpacking git (1:2.20.1-2+deb10u4) over (1:2.20.1-2+deb10u3) ... [14:20:05] :) [14:20:30] looks like those node* images are not routinely rebuild to catch up with debian updates [14:22:36] git 2.20, kinda old :) [14:22:53] (03PS2) 10Hashar: Dockerfile: dashboards 2.2.0 and node 14.20.0 [releng/phatality] - 10https://gerrit.wikimedia.org/r/849551 [14:22:55] (03PS1) 10Hashar: Dockerfile: use a shallow clone [releng/phatality] - 10https://gerrit.wikimedia.org/r/849565 [14:23:35] (03PS2) 10Hashar: Dockerfile: remove CACHE_BUSTER and use a COPY instead [releng/phatality] - 10https://gerrit.wikimedia.org/r/849553 [14:25:14] jnuche: ^ I am building it locally [14:26:54] OHHH [14:27:03] so looking at the README there is a section about using scap deploy [14:27:18] and on the deployment server the HEAD is pointing at * d8e2adc - (HEAD -> master, tag: scap/sync/2022-04-27/0001) phatality-1.2.0.zip: Remove phatalityId concept, search for task title instead (6 months ago) [14:28:48] with some zip file having to be created under ./deploy [14:29:19] yeah, cwhite just replied to the ticket mentioning that: https://phabricator.wikimedia.org/T304440#8346034 [14:29:24] so what I am thinking is cwhite change that bumps the open search dashboard is not sufficient [14:29:40] we need to deploy the plugin using scap [14:29:46] yeah [14:29:50] I am building it locally [14:30:01] it is scary [14:31:57] build-zip-using-docker eventually fails https://phabricator.wikimedia.org/P36572 [14:34:18] (03PS3) 10Hashar: Dockerfile: remove CACHE_BUSTER and use a COPY instead [releng/phatality] - 10https://gerrit.wikimedia.org/r/849553 [14:35:44] (03CR) 10Hashar: Dockerfile: remove CACHE_BUSTER and use a COPY instead (031 comment) [releng/phatality] - 10https://gerrit.wikimedia.org/r/849553 (owner: 10Hashar) [14:39:25] (03PS1) 10Hashar: Update deployment zip for dashboard 2.2.0 [releng/phatality] - 10https://gerrit.wikimedia.org/r/849590 (https://phabricator.wikimedia.org/T340440) [14:39:32] jnuche: sorry I went a frenzy rush, should have paired that probably [14:41:12] that'd have been cool :) [14:42:11] Hello [14:43:24] 👋 [14:43:35] there are surely some things broken I can't start it :( [14:45:48] ah yeah it works [14:45:57] but requires an opensearch instance to point to [14:46:48] which leads me to lean at https://hub.docker.com/r/opensearchproject/opensearch and using docker compose [14:46:54] but well at this point my brain is fried [14:47:56] dancy: have you tried before to build the Phatality plugin locally? [14:49:13] * dancy thinks. [14:49:25] hashar: I can manually deploy the new zip to beta as a test if you like? [14:49:27] I have a vague recollection of doing so.. and if I recall correctly it wasn't easy [14:49:48] lemme see what I have [14:49:51] cwhite: that would be great yet. It will take us a bit of a time to properly review the series of change I have send. [14:50:24] cwhite: I think the zip file can be downloaded from `https://gerrit.wikimedia.org/r/changes/releng%2Fphatality~849590/revisions/1/files/deploy%2Fphatality-2.2.0.zip/download` [14:50:38] (and today I have learned one can download a binary file from Gerrit web UI) [14:51:17] thanks, I'll ping here when it's up on beta-logs [14:51:52] (03CR) 10Hashar: "When you have created the Dockerfile it was based on the node10 image which ships with an outdated version of git 2.11. That version is in" [releng/phatality] - 10https://gerrit.wikimedia.org/r/849565 (owner: 10Hashar) [14:52:07] jnuche: Looks like I created `build-zip-using-docker` to perform the build. [14:53:06] dancy: yep, we saw that, seems like hashar managed to create a zip in the end though [14:53:31] I didn't get a chance to read all the scrollback. Is there an outstanding problem? [14:53:56] (03CR) 10Hashar: "I am not sure why there was a CACHE_BUSTER based on $(date). Looks like doing a COPY of the Phatality repo is sufficient to invalidate a " [releng/phatality] - 10https://gerrit.wikimedia.org/r/849553 (owner: 10Hashar) [14:54:29] the short story is our OpenSearch got upgraded from 1.2.0 to 2.2.0 [14:54:30] The cache buster is there to ensure that the git clone operation runs every time. [14:54:35] that new version requires phatality to be rebuild [14:54:43] and bump from node10 to node14.20.0 (not 14.20.1) [14:54:46] hashar: I think gerrit may have zipped the zip file. I extracted what was downloaded at that gerrit link and installed the plugin on beta-logs. [14:55:00] + sometime to discover the build stack and eventually I got some untested zip files ;-] [14:55:25] * hashar fires up https://beta-logs.wmcloud.org/ [14:55:35] I am sorry I forgot about Phatality :-\ [14:56:03] looks like it works on beta! [14:56:24] dancy: (thanks for looking) [14:58:16] cwhite: thank you. I am assuming once we (releng) are ready, we can scap deploy the new zip file and eventually the proper scripts will take care of updating the production one [14:59:31] hashar: yes, feel free to deploy to codfw. Note that 2.2.0 is only installed on codfw. Phatality will fail to install in eqiad. [15:05:21] cwhite: so kibana at logstash.wikimedia.org/ currently loads from codfw correct? [15:05:31] while the instance on eqiad is ketp at the old version but not serving anything? [15:05:40] correct [15:05:48] thanks sir! [15:43:35] jnuche: dancy: should I process with the Phatality changes? [15:44:08] I'll start reviewing now (but will probably be interrupted by the releng meeting) [15:46:19] the CACHE_BUSTER one can probably be skipped and done later ;) [15:46:38] (03CR) 10Ahmon Dancy: [C: 03+2] Dockerfile: dashboards 2.2.0 and node 14.20.0 (031 comment) [releng/phatality] - 10https://gerrit.wikimedia.org/r/849551 (owner: 10Hashar) [15:47:03] (03Merged) 10jenkins-bot: Dockerfile: dashboards 2.2.0 and node 14.20.0 [releng/phatality] - 10https://gerrit.wikimedia.org/r/849551 (owner: 10Hashar) [15:48:54] (03CR) 10Ahmon Dancy: "Thanks for the improvement! Just a comment on the commit message text." [releng/phatality] - 10https://gerrit.wikimedia.org/r/849565 (owner: 10Hashar) [15:49:00] hashar ^ [15:49:03] ;) [15:54:03] (03PS2) 10Hashar: Dockerfile: use a shallow clone [releng/phatality] - 10https://gerrit.wikimedia.org/r/849565 [15:54:26] (03CR) 10Hashar: Dockerfile: use a shallow clone (031 comment) [releng/phatality] - 10https://gerrit.wikimedia.org/r/849565 (owner: 10Hashar) [15:55:53] (03CR) 10Jaime Nuche: Dockerfile: dashboards 2.2.0 and node 14.20.0 (031 comment) [releng/phatality] - 10https://gerrit.wikimedia.org/r/849551 (owner: 10Hashar) [15:56:55] hmm [15:57:17] so with node:10 git version 2.11.0 I can do a shallow clone from a tag [15:57:25] (03CR) 10Jaime Nuche: [C: 03+2] "Tested the clone locally, tag gets picked correctly." [releng/phatality] - 10https://gerrit.wikimedia.org/r/849565 (owner: 10Hashar) [15:57:40] so I have no idea why a non shallow clone was done :-\ [15:57:49] (03Merged) 10jenkins-bot: Dockerfile: use a shallow clone [releng/phatality] - 10https://gerrit.wikimedia.org/r/849565 (owner: 10Hashar) [15:58:49] I had some kind of problem originally... can't remember what happened exactly [15:59:31] anyway it is a shallow one now ;) [16:00:17] Testing now [16:07:19] (03CR) 10Ahmon Dancy: [C: 03+2] Dockerfile: remove CACHE_BUSTER and use a COPY instead (031 comment) [releng/phatality] - 10https://gerrit.wikimedia.org/r/849553 (owner: 10Hashar) [16:07:43] (03CR) 10Ahmon Dancy: [C: 03+2] Update deployment zip for dashboard 2.2.0 [releng/phatality] - 10https://gerrit.wikimedia.org/r/849590 (https://phabricator.wikimedia.org/T340440) (owner: 10Hashar) [16:15:03] (03PS4) 10Hashar: Dockerfile: remove CACHE_BUSTER and use a COPY instead [releng/phatality] - 10https://gerrit.wikimedia.org/r/849553 [16:15:08] (03PS2) 10Hashar: Update deployment zip for dashboard 2.2.0 [releng/phatality] - 10https://gerrit.wikimedia.org/r/849590 (https://phabricator.wikimedia.org/T340440) [16:15:39] (03CR) 10Hashar: [C: 03+2] "Had to rebase due to parent being changed" [releng/phatality] - 10https://gerrit.wikimedia.org/r/849553 (owner: 10Hashar) [16:16:04] (03Merged) 10jenkins-bot: Dockerfile: remove CACHE_BUSTER and use a COPY instead [releng/phatality] - 10https://gerrit.wikimedia.org/r/849553 (owner: 10Hashar) [16:16:27] (03PS3) 10Hashar: Update deployment zip for dashboard 2.2.0 [releng/phatality] - 10https://gerrit.wikimedia.org/r/849590 (https://phabricator.wikimedia.org/T340440) [16:17:10] (03CR) 10Hashar: [C: 03+2] "Had to rebase cause some previous changes got rebased meanwhile :-\" [releng/phatality] - 10https://gerrit.wikimedia.org/r/849590 (https://phabricator.wikimedia.org/T340440) (owner: 10Hashar) [16:17:37] (03Merged) 10jenkins-bot: Update deployment zip for dashboard 2.2.0 [releng/phatality] - 10https://gerrit.wikimedia.org/r/849590 (https://phabricator.wikimedia.org/T340440) (owner: 10Hashar) [16:17:52] thanks ;) [16:22:03] I have used the wrong task number apparently [17:01:29] I am going to deploy the Phatality plugin for 2.2.0 of opensearch to codfw ( poke cwhite ) [17:02:15] and of course that fails ;D [17:02:39] :( [17:04:39] Unable to remove plugin because of error: "ENOTEMPTY: directory not empty, rmdir '/usr/share/opensearch-dashboards/plugins/phatality'" [17:04:39] Attempting to transfer from file:///srv/deployment/releng/phatality/deploy/phatality-2.2.0.zip [17:10:04] I am afraid that is an upstream bug :-\ [17:17:39] * cwhite looking [17:26:15] the code seems to be https://github.com/opensearch-project/OpenSearch-Dashboards/blob/main/src/cli_plugin/remove/remove.js [17:26:27] which does ` del.sync(settings.pluginPath, { force: true });` [17:26:32] and that hasn't been touched in years [17:26:48] I guess something got broken somewhere in https://www.npmjs.com/package/del [17:27:08] welcome to nodejs in which deleting a directory obviously requires a third party dependency ;) [17:27:10] there are remnants left behind for some reason. I'm looking to remove them manually [17:28:53] hashar: want to try running the deployment again? [17:29:07] sure! [17:29:54] this time only targetting logstash2023.codfw.wmnet [17:30:39] I have also send a patch to use `set -e` in the upgrade-phatality script in order to have it abort if the removal failed ( https://gerrit.wikimedia.org/r/c/operations/puppet/+/849631/ ) [17:30:44] cwhite: that has worked! [17:31:41] and I have successfully deployed on all hosts [17:32:34] looks like it's working to me :) [17:33:03] have you changed anything? [17:33:26] I manually removed the lingering phatality files from the plugin directory [17:33:48] not sure why they were there in the first place [17:34:11] so potentially if I redeploy I would get the issue again? [17:34:33] not sure. want to try it? [17:35:06] yeah trying with scap deploy --force -l 'logstash2023.codfw.wmnet' [17:35:23] worked [17:35:25] hmm [17:35:51] who knows what wrong magic got triggered somewhere in the nodejs module [17:36:33] cwhite: jnuche: dancy: Phatality works again in Kibana \o/ {{SUCCESS}} [17:36:39] thank you so much Cole [17:36:44] Woot! Nice work everyone! [17:37:02] <3 [17:37:59] I am so happy that got fixed so fast ;) [17:38:43] as to why https://github.com/opensearch-project/OpenSearch-Dashboards/blob/main/src/cli_plugin/remove/remove.js#L49 fails [17:38:55] that is unknown, I am too tired to dig that hole [17:39:28] I am off, that is been a long day [17:42:22] Have a good night! [17:54:15] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for Sergio Gimeno - https://phabricator.wikimedia.org/T320722 (10Sgs) @thcipriani Yes, see you in some hours! Apologies for the prior miss-scheduling. [17:54:56] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for Sergio Gimeno - https://phabricator.wikimedia.org/T320722 (10Sgs) [18:05:56] (03PS1) 10Jbond: tox-buster: Add python3.9 to tox-buster base image [integration/config] - 10https://gerrit.wikimedia.org/r/849635 [18:07:16] (03PS2) 10Jbond: tox-buster: Add python3.9 to tox-buster base image [integration/config] - 10https://gerrit.wikimedia.org/r/849635 (https://phabricator.wikimedia.org/T289222) [18:07:58] jbond: oh that is great [18:08:51] 10Continuous-Integration-Infrastructure, 10Patch-For-Review: Add Python 3.9 to Wikimedia CI - https://phabricator.wikimedia.org/T289222 (10jbond) > Or drop support for python 3.4, 3.5 and 3.6 entirely and get the new minimum supported version to be 3.7 shipped by Buster but there bunch of repositories still re... [18:09:00] (03CR) 10Hashar: "For docker-pkg to recognize this as a new image you also need to:" [integration/config] - 10https://gerrit.wikimedia.org/r/849635 (https://phabricator.wikimedia.org/T289222) (owner: 10Jbond) [18:16:13] (03PS3) 10Jbond: tox-buster: Add python3.9 to tox-buster base image [integration/config] - 10https://gerrit.wikimedia.org/r/849635 (https://phabricator.wikimedia.org/T289222) [18:16:29] (03CR) 10Jbond: "added a changelog entry" [integration/config] - 10https://gerrit.wikimedia.org/r/849635 (https://phabricator.wikimedia.org/T289222) (owner: 10Jbond) [18:17:06] (03CR) 10Hashar: [C: 03+2] tox-buster: Add python3.9 to tox-buster base image [integration/config] - 10https://gerrit.wikimedia.org/r/849635 (https://phabricator.wikimedia.org/T289222) (owner: 10Jbond) [18:17:47] (03CR) 10Jforrester: "You'll want to cascade this to the other tox-buster images." [integration/config] - 10https://gerrit.wikimedia.org/r/849635 (https://phabricator.wikimedia.org/T289222) (owner: 10Jbond) [18:18:11] hashar: tox-pywikibot etc. depend on tox-buster so need version bumps. [18:18:14] ah very true sorry [18:18:19] I keep forgetting about that [18:18:24] I'll do it. [18:18:38] 💌 [18:20:29] (03PS4) 10Jforrester: tox-buster: Add python3.9 to tox-buster base image [integration/config] - 10https://gerrit.wikimedia.org/r/849635 (https://phabricator.wikimedia.org/T289222) (owner: 10Jbond) [18:20:33] ^^ done [18:20:42] Didn't build-test though. [18:21:11] should be fine ™ [18:21:25] should we bump the version of the child images rather than using -sXX ? [18:21:30] What Could Possibly Go Wrong?™ [18:21:32] it doesn't really matter [18:21:43] (03PS1) 10Umherirrender: Zuul: [mediawiki/extensions/PageCheckout] Add [integration/config] - 10https://gerrit.wikimedia.org/r/849638 (https://phabricator.wikimedia.org/T321668) [18:21:47] I just did what `docker-pkg update` does. [18:21:55] roger [18:21:56] 10Continuous-Integration-Config, 10MediaWiki-extensions-General, 10Patch-For-Review: Please add mediawiki-i18n-check-docker to repository mediawiki/extensions/PageAuthors - https://phabricator.wikimedia.org/T321668 (10Umherirrender) a:03Umherirrender The extension does not have any jenkins support at all. [18:22:06] (03CR) 10Hashar: [C: 03+2] tox-buster: Add python3.9 to tox-buster base image [integration/config] - 10https://gerrit.wikimedia.org/r/849635 (https://phabricator.wikimedia.org/T289222) (owner: 10Jbond) [18:22:21] (03PS2) 10Umherirrender: Zuul: [mediawiki/extensions/PageAuthors] Add [integration/config] - 10https://gerrit.wikimedia.org/r/849638 (https://phabricator.wikimedia.org/T321668) [18:24:11] (03Merged) 10jenkins-bot: tox-buster: Add python3.9 to tox-buster base image [integration/config] - 10https://gerrit.wikimedia.org/r/849635 (https://phabricator.wikimedia.org/T289222) (owner: 10Jbond) [18:24:54] (03CR) 10Hashar: [C: 03+2] Zuul: [mediawiki/extensions/PageAuthors] Add [integration/config] - 10https://gerrit.wikimedia.org/r/849638 (https://phabricator.wikimedia.org/T321668) (owner: 10Umherirrender) [18:26:44] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/PageAuthors] Add [integration/config] - 10https://gerrit.wikimedia.org/r/849638 (https://phabricator.wikimedia.org/T321668) (owner: 10Umherirrender) [18:28:41] !log Reloaded Zuul for https://gerrit.wikimedia.org/r/849638 # T321668 [18:28:43] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:28:43] T321668: Please add mediawiki-i18n-check-docker to repository mediawiki/extensions/PageAuthors - https://phabricator.wikimedia.org/T321668 [18:34:28] (03CR) 10Hashar: [C: 03+2] "== Step 1: building images ==" [integration/config] - 10https://gerrit.wikimedia.org/r/849635 (https://phabricator.wikimedia.org/T289222) (owner: 10Jbond) [18:34:55] the poetry one failed to build [18:36:57] 10Continuous-Integration-Infrastructure: Fails to build releng/tox-poetry-buster:0.0.1-s1 - https://phabricator.wikimedia.org/T321724 (10hashar) [18:44:38] 10Continuous-Integration-Infrastructure, 10Upstream: Fails to build releng/tox-poetry-buster:0.0.1-s1 - https://phabricator.wikimedia.org/T321724 (10hashar) [18:45:12] 10Continuous-Integration-Infrastructure, 10Upstream: Fails to build releng/tox-poetry-buster:0.0.1-s1 - https://phabricator.wikimedia.org/T321724 (10hashar) [18:48:13] (03PS1) 10Hashar: dockerfiles: run get-poetry.py with Python 3 [integration/config] - 10https://gerrit.wikimedia.org/r/849639 (https://phabricator.wikimedia.org/T321724) [18:48:41] (03CR) 10Hashar: [C: 03+2] "Lets try" [integration/config] - 10https://gerrit.wikimedia.org/r/849639 (https://phabricator.wikimedia.org/T321724) (owner: 10Hashar) [18:50:28] (03Merged) 10jenkins-bot: dockerfiles: run get-poetry.py with Python 3 [integration/config] - 10https://gerrit.wikimedia.org/r/849639 (https://phabricator.wikimedia.org/T321724) (owner: 10Hashar) [18:51:22] 10Continuous-Integration-Config, 10MediaWiki-extensions-General: Please add mediawiki-i18n-check-docker to repository mediawiki/extensions/PageAuthors - https://phabricator.wikimedia.org/T321668 (10Umherirrender) 05Open→03Resolved [18:51:33] 10Continuous-Integration-Infrastructure: Add Python 3.9 to Wikimedia CI - https://phabricator.wikimedia.org/T289222 (10hashar) [18:51:35] 10Continuous-Integration-Infrastructure, 10Patch-For-Review, 10Upstream: Fails to build releng/tox-poetry-buster:0.0.1-s1 - https://phabricator.wikimedia.org/T321724 (10hashar) 05Open→03Resolved a:03hashar Had to run `get-poetry.py` with `python3` [18:52:02] (03CR) 10Hashar: [C: 03+2] "get-poetry.py had to be run with python3 T321724" [integration/config] - 10https://gerrit.wikimedia.org/r/849635 (https://phabricator.wikimedia.org/T289222) (owner: 10Jbond) [18:55:17] my meeting is finishing, will check to update those jobs tommorrow [18:56:26] 10Continuous-Integration-Infrastructure: Add Python 3.9 to Wikimedia CI - https://phabricator.wikimedia.org/T289222 (10hashar) Thanks for the python3.9 package! I have build the CI images. Will switch the Jenkins jobs to it tomorrow [19:31:55] (03PS1) 10Jforrester: jjb: Update Python jobs to ones with py39 [integration/config] - 10https://gerrit.wikimedia.org/r/849647 (https://phabricator.wikimedia.org/T289222) [20:48:06] 10GitLab (CI & Job Runners), 10Data Pipelines: Allow a shared, protected runner for the data-engineering group in GitLab - https://phabricator.wikimedia.org/T295045 (10xcollazo) Just passing by to +1 this idea. @BTullis mentioned above the `airflow-dags` project, but it would also immediately benefit the `con... [21:51:59] 10Phabricator: Herald rule to auto-set point value of tasks entering Web team board - https://phabricator.wikimedia.org/T321749 (10LGoto) [22:09:42] (03PS1) 10Subramanya Sastry: Tweak eswiki CSS based on what is in Common.css on the wiki [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/849662 [22:17:48] 10Release-Engineering-Team (GitLab II: Wrath of Kahn 👾), 10Scap, 10serviceops-collab: Set up equivalent of the scap-tox-publish job in scap's .gitlab-ci.yml - https://phabricator.wikimedia.org/T321629 (10dancy) @Jelto Is there an existing list of trusted gitlab runner hostnames or IP addresses in puppet that... [22:20:47] 10Release-Engineering-Team (GitLab II: Wrath of Kahn 👾), 10Scap, 10serviceops-collab: Set up equivalent of the scap-tox-publish job in scap's .gitlab-ci.yml - https://phabricator.wikimedia.org/T321629 (10Dzahn) Recently I wanted to do something similar and tried to use an anchor/alias to avoid repeating host... [22:24:16] 10Release-Engineering-Team (GitLab II: Wrath of Kahn 👾), 10Scap, 10serviceops-collab: Set up equivalent of the scap-tox-publish job in scap's .gitlab-ci.yml - https://phabricator.wikimedia.org/T321629 (10Dzahn) @dancy Looking at the existing values you listed and if I can come up with a patch. [22:32:03] hey releng, any idea what is going on here? https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/849139 I updated a comment and then it's been getting the instant -1's since [22:33:31] cstone: I'll take a look [22:33:40] thanks dancy ! [22:34:22] (03PS2) 10Subramanya Sastry: Use sup.reference instead of sup.mw-ref (courtesy T265930) [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/848554 [22:34:24] (03PS2) 10Subramanya Sastry: Tweak eswiki CSS based on what is in Common.css on the wiki [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/849662 [22:34:31] (03CR) 10CI reject: [V: 04-1] Tweak eswiki CSS based on what is in Common.css on the wiki [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/849662 (owner: 10Subramanya Sastry) [22:53:30] cstone: Tests passed this time. I don't know why it is behaving weirdly though [22:53:40] no worries thanks so much dancy ! [22:59:04] 10Continuous-Integration-Config: php-composer-security-docker currently failing with "curl: command not found" error this time for php-7.4 - https://phabricator.wikimedia.org/T321753 (10Dwisehaupt) [23:04:29] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: CI fails with 'This change or one of its cross-repo dependencies was unable to be automatically merged' for a lot of repos - https://phabricator.wikimedia.org/T308943 (10dancy) 05Resolved→03Open [23:04:31] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: gerrit-bot holding open SSH sessions - https://phabricator.wikimedia.org/T309376 (10dancy) [23:04:53] cstone: It looks like https://phabricator.wikimedia.org/T308943 I will try the actions that hash.ar previously did [23:10:09] !sal Restarted Zuul CI server due to stall ssh connections which went against the max per user connection limit in Gerrit #T308943 [23:10:09] https://tools.wmflabs.org/sal/releng [23:10:09] T308943: CI fails with 'This change or one of its cross-repo dependencies was unable to be automatically merged' for a lot of repos - https://phabricator.wikimedia.org/T308943 [23:12:12] !log Restarted Zuul CI server due to stall ssh connections which went against the max per user connection limit in Gerrit #T308943 [23:12:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:12:52] (03PS1) 10Dwisehaupt: docker: add curl for composer-security-check [integration/config] - 10https://gerrit.wikimedia.org/r/849712 (https://phabricator.wikimedia.org/T321753) [23:16:19] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: CI fails with 'This change or one of its cross-repo dependencies was unable to be automatically merged' for a lot of repos - https://phabricator.wikimedia.org/T308943 (10dancy) 05Open→03Resolved Same thing happened today. Resolved by res... [23:16:21] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: gerrit-bot holding open SSH sessions - https://phabricator.wikimedia.org/T309376 (10dancy) [23:18:18] cstone: Should be fixed now [23:18:56] thanks dancy !!