[05:35:21] 10Beta-Cluster-Infrastructure, 10serviceops, 10wikidiff2, 10Better-Diffs-2023, 10Community-Tech (CommTech-Kanban): Install wikidiff2 1.14.0 deb on deployment-prep & test - https://phabricator.wikimedia.org/T340542 (10tstarling) >>! In T340542#8973199, @dom_walden wrote: > Or compare: > * https://en.wikti... [08:00:48] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Patch-For-Review, 10Release, 10Train Deployments: 1.41.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T340244 (10hashar) >>! In T340244#8983322, @RhinosF1 wrote: > Previous week train has been reopened Indeed wmf.15 got reopened due to T... [08:28:41] 10Release-Engineering-Team (Seen), 10Data-Persistence (work done), 10Security-Team, 10SecTeam-Processed, 10Security: Determine CI best practices for service which connects to MySQL - https://phabricator.wikimedia.org/T308789 (10kostajh) [08:36:27] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Patch-For-Review, 10Release, 10Train Deployments: 1.41.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T340244 (10jnuche) [09:45:05] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Cloud-VPS (Quota-requests), 10Patch-For-Review: Rebuild WMCS integration instances to larger flavor - https://phabricator.wikimedia.org/T340070 (10hashar) [09:48:19] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Cloud-VPS (Quota-requests), 10Patch-For-Review: Rebuild WMCS integration instances to larger flavor - https://phabricator.wikimedia.org/T340070 (10hashar) **Summary** The existing instances can't be switched to a flavor with a larger e... [09:57:56] 10GitLab (Misc), 10Release-Engineering-Team (Seen), 10User-aborrero: Gerritlab: Stacked pull requests - https://phabricator.wikimedia.org/T300819 (10aborrero) [10:04:32] 10GitLab (Misc), 10Release-Engineering-Team (Seen), 10User-aborrero: Gerritlab: Stacked pull requests - https://phabricator.wikimedia.org/T300819 (10aborrero) My team has adopted gitlab for a number of repositories already. There are weekly mentions to the stacked patchset workflow now. I'm curious about th... [10:30:48] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Cloud-VPS (Quota-requests), 10Patch-For-Review: Rebuild WMCS integration instances to larger flavor - https://phabricator.wikimedia.org/T340070 (10hashar) [10:47:13] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Fix Puppet agent provisioning on Jenkins agent instances - https://phabricator.wikimedia.org/T341051 (10hashar) [10:47:34] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Fix Puppet agent provisioning on Jenkins agent instances - https://phabricator.wikimedia.org/T341051 (10hashar) [10:47:37] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Cloud-VPS (Quota-requests), 10Patch-For-Review: Rebuild WMCS integration instances to larger flavor - https://phabricator.wikimedia.org/T340070 (10hashar) [10:50:18] 10Beta-Cluster-Infrastructure, 10serviceops, 10wikidiff2, 10Better-Diffs-2023, 10Community-Tech (CommTech-Kanban): Install wikidiff2 1.14.1 deb on deployment-prep & test - https://phabricator.wikimedia.org/T340542 (10TheresNoTime) [10:51:39] 10Beta-Cluster-Infrastructure, 10serviceops, 10wikidiff2, 10Better-Diffs-2023, 10Community-Tech (CommTech-Kanban): Install wikidiff2 1.14.1 deb on deployment-prep & test - https://phabricator.wikimedia.org/T340542 (10TheresNoTime) @MoritzMuehlenhoff (see above for context) β€” we've released `1.14.1` to fi... [11:09:19] 10Phabricator (Upstream), 10TimedMediaHandler, 10Browser-Support-Apple-Safari: Safari is very slow to load pages with many audio players - https://phabricator.wikimedia.org/T341055 (10TheDJ) [11:09:30] 10Phabricator (Upstream), 10TimedMediaHandler, 10Browser-Support-Apple-Safari: Safari is very slow to load pages with many audio players - https://phabricator.wikimedia.org/T341055 (10TheDJ) p:05Triageβ†’03Low [12:16:48] 10Continuous-Integration-Config, 10MediaWiki-extensions-WikibaseClient, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, and 2 others: Re-start Wikibase test coverage reporting - https://phabricator.wikimedia.org/T288396 (10hashar) T288396#8979619 is a very nice dig :) > If you omit the `$ext/tests/... [12:27:32] (03PS1) 10Hashar: dockerfiles: use extension base path for coverage report [integration/config] - 10https://gerrit.wikimedia.org/r/935410 (https://phabricator.wikimedia.org/T288396) [12:48:39] (03CR) 10Lucas Werkmeister (WMDE): [C: 03+1] dockerfiles: use extension base path for coverage report [integration/config] - 10https://gerrit.wikimedia.org/r/935410 (https://phabricator.wikimedia.org/T288396) (owner: 10Hashar) [12:56:39] 10Release-Engineering-Team (They Live πŸ•ΆοΈπŸ§Ÿ), 10Developer-Advocacy (Jul-Sep 2023): phabricator: quarterly_metrics.sh: Improve Bitergia instructions - https://phabricator.wikimedia.org/T341064 (10Aklapper) 05Openβ†’03In progress p:05Triageβ†’03Low [13:34:37] 10Beta-Cluster-Infrastructure, 10User-DannyS712: Beta Cluster mailer not sending emails - https://phabricator.wikimedia.org/T212327 (10taavi) [13:49:30] !log integration: pooled new Jenkins agents `integration-agent-docker-1040` and `integration-agent-docker-1041` with larger partitions for Docker and Jenkins workspace - T340070 [13:49:33] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:49:33] T340070: Rebuild WMCS integration instances to larger flavor - https://phabricator.wikimedia.org/T340070 [14:11:02] 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10dcaro) Hi! I have to take over some project management duties for toolforge/toolforge-build-service and as part of it I will have to create iteration subprojects [[ https://phabricat... [14:15:16] 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10taavi) >>! In T706#8988292, @dcaro wrote: > Can I get project creation superpowers please? :) Done! I'd add the usual disclaimer here but looks like you already read through the inst... [14:38:42] 10Continuous-Integration-Infrastructure, 10Jenkins, 10serviceops-collab: Do not expose LDAP users on release Jenkins UI - https://phabricator.wikimedia.org/T341074 (10jnuche) [14:42:15] !log Adding new columns for the CampaignEvents extension in beta wikishared # T340694 [14:42:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:42:18] T340694: Create new participant questions columns in beta DB - https://phabricator.wikimedia.org/T340694 [14:43:05] thcipriani: dancy: looking at beta cherry picks, good ol' 2017 is still asking for resolution on Scap UID's in Beta. https://gerrit.wikimedia.org/r/c/operations/puppet/+/361796 Maybe this is easier to solve in 2023 in a way that works for prod/beta alike and/or re-written in a way that encodes the difference within puppet? [14:46:04] 10Beta-Cluster-Infrastructure, 10Dumps-Generation: mediawiki scripts fail on new buster image in deployment-prep - https://phabricator.wikimedia.org/T273089 (10Krinkle) Has this been resolved meanwhile? I imagine the snapshot hosts in Beta are not different from any other host in Beta connecting to etcd, but i... [14:46:55] 10Beta-Cluster-Infrastructure, 10CampaignEvents, 10Campaign-Registration, 10Campaign-Tools (Campaign-Tools-Current-Sprint): Create new participant questions columns in beta DB - https://phabricator.wikimedia.org/T340694 (10Daimona) 05Openβ†’03Resolved a:03Daimona [14:47:30] 10GitLab (Pipeline Services Migration🐀), 10Release-Engineering-Team (They Live πŸ•ΆοΈπŸ§Ÿ), 10serviceops-collab: Provide mechanism to publish to doc.wikimedia.org from GitLab CI - https://phabricator.wikimedia.org/T336168 (10jnuche) [14:48:24] 10Continuous-Integration-Infrastructure, 10Jenkins, 10serviceops-collab: Do not expose LDAP users on release Jenkins UI - https://phabricator.wikimedia.org/T341074 (10jnuche) [14:48:27] !log integration: pooled new Jenkins agents 1042, 1043, 1044, 1045, 1046 - T340070 [14:48:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:48:30] T340070: Rebuild WMCS integration instances to larger flavor - https://phabricator.wikimedia.org/T340070 [14:55:39] hashar: nice work! How is the automation around agent creation these days? Apart from the cloud-init issue mentioned on-task, is everythign more or less automated and as-documented? [14:56:05] Krinkle: jnuche did the configuration management for the release jenkins [14:56:23] then beside that I am following the runbook at https://wikitech.wikimedia.org/wiki/Nova_Resource:Integration/Setup :) [14:56:41] until we manage to migrate the CI Jenkins to the same config management as releases Jenkins [14:57:05] t.aavi suggested using terraform to manage the fleet of instance but that is yet another thing to learn/add [14:58:24] Ah right, there is a few steps in the Jenkins admin GUI. Is that done via puppet for the release Jenkins? Or in some other way? [14:59:07] Jenkins has a yaml based DSL to configure an instance https://github.com/jenkinsci/configuration-as-code-plugin/ [14:59:35] and the config for the release jenkins is at https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/tree/master/conf/releasing [14:59:43] plugins.txt is a lock file of plugins/versions to install [15:00:10] as hashar said [15:00:14] yaml is here: https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/blob/master/conf/releasing/casc/jenkins.yaml [15:00:28] and https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/blob/master/conf/releasing/casc/jenkins.yaml#L97-112 for the node definition [15:00:39] it is quite magic [15:04:42] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Patch-For-Review: Fix Puppet agent provisioning on Jenkins agent instances - https://phabricator.wikimedia.org/T341051 (10hashar) With https://gerrit.wikimedia.org/r/c/operations/puppet/+/935405 applied: ` Notice: /Stage[main]/Labs_lvm/Ex... [15:11:44] 10Continuous-Integration-Infrastructure, 10Patch-For-Review: Add Python 3.9 to Wikimedia CI - https://phabricator.wikimedia.org/T289222 (10Xqt) Isn't this done already? An can we have Python 3.10 for tests? [15:15:34] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Direct 0.5% of all traffic to mw-on-k8s - https://phabricator.wikimedia.org/T341078 (10Clement_Goubert) [15:15:52] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Clement_Goubert) [15:16:02] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Direct 0.5% of all traffic to mw-on-k8s - https://phabricator.wikimedia.org/T341078 (10Clement_Goubert) 05Openβ†’03In progress p:05Triageβ†’03High [15:16:07] !log Create missing xhguiadmin and xhgui users on deployment-db11. Matching settings from prod https://gerrit.wikimedia.org/g/operations/puppet/+/4f545a1765b21e8c0f843268d09fb101cb8c7530/modules/profile/templates/mariadb/grants/production-m2.sql.erb#241 and passwords from beta https://gerrit.wikimedia.org/g/cloud/instance-puppet/+/f5cf0f02aaaf78bdbd3b4399a4ab57405abfcab8/deployment-prep/deployment-xhgui.yaml [15:16:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:17:11] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Clement_Goubert) [15:18:17] 10Beta-Cluster-Infrastructure, 10Dumps-Generation: mediawiki scripts fail on new buster image in deployment-prep - https://phabricator.wikimedia.org/T273089 (10ArielGlenn) I've not spun up any more buster images, and the next one I create will likely be bullseye. Maybe someone else has done so though. [15:42:21] !log integration: Reloading Jenkins CI configuration from disk after manually adding a dozen of agents [15:42:22] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:46:08] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Cloud-VPS (Quota-requests): Rebuild WMCS integration instances to larger flavor - https://phabricator.wikimedia.org/T340070 (10hashar) [15:51:23] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Cloud-VPS (Quota-requests): Rebuild WMCS integration instances to larger flavor - https://phabricator.wikimedia.org/T340070 (10hashar) 05Openβ†’03Resolved I have rebuild all instances with the flavor `g3.cores8.ram24.disk20.ephemeral90.... [15:56:16] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Patch-For-Review: Fix Puppet agent provisioning on Jenkins agent instances - https://phabricator.wikimedia.org/T341051 (10hashar) a:03hashar [15:56:37] I think that is all for today [16:05:13] 10phabricator maintenance bot: Make it possible to add reviewers automatically to patches uploaded by Maintenance bot - https://phabricator.wikimedia.org/T340796 (10Aklapper) > To ensure patches uploaded by the Gerrit maintenance bot Could you link to an example please? Also, how is this related to #phabricator_... [20:26:02] !log Remove deployment-xhgui03 host, ref https://gerrit.wikimedia.org/r/c/operations/puppet/+/935512 and https://gerrit.wikimedia.org/r/c/performance/docroot/+/935519 for T340713. Host no longer used. XHGui frontend now runs alongside excimer on webperf hosts. [20:26:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:09:55] 10Continuous-Integration-Infrastructure, 10Jenkins, 10serviceops-collab, 10Patch-For-Review: Do not expose LDAP users on release Jenkins UI - https://phabricator.wikimedia.org/T341074 (10Peachey88) > but not exposing that information is still an improvement. Why? you haven't identified a issue this is re... [22:34:53] hi folks. we've got a docker image merged to dev-images:main that we'd like to deploy to the wmf registry. I can see in the project docs that we use a script via`fab deploy_devimages` but that's not doing anything when run locally. Is there something we're missing? [22:35:16] this is the merged image update - https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/31 [22:35:53] oh I think I know what's up [22:36:23] the fab script is actually a shell script in the project [22:36:39] I mistakenly installed Fabric after running `fab` and being told I needed Fabric installed [22:36:47] ok let's try the actual script [23:04:29] ok the script works but I'm being prompted for a pw which I'm not sure which one it needs