[02:06:47] (03PS6) 10Vaughn Walters: jjb: [catalyst-daily] Export deploy.env so selenium-test can access MW_SERVER [integration/config] - 10https://gerrit.wikimedia.org/r/1305214 (https://phabricator.wikimedia.org/T429629) [04:38:33] FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance deployment-changeprop-1 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [04:38:34] (03CR) 10Hashar: jjb: [catalyst-daily] Export deploy.env so selenium-test can access MW_SERVER (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1305214 (https://phabricator.wikimedia.org/T429629) (owner: 10Vaughn Walters) [05:04:42] (03PS7) 10Hashar: jjb: [catalyst-daily-core] export Catalyst env [integration/config] - 10https://gerrit.wikimedia.org/r/1305214 (https://phabricator.wikimedia.org/T429629) (owner: 10Vaughn Walters) [05:06:14] (03CR) 10CI reject: [V:04-1] jjb: [catalyst-daily-core] export Catalyst env [integration/config] - 10https://gerrit.wikimedia.org/r/1305214 (https://phabricator.wikimedia.org/T429629) (owner: 10Vaughn Walters) [05:10:14] (03PS8) 10Hashar: jjb: [catalyst-daily-core] export Catalyst env [integration/config] - 10https://gerrit.wikimedia.org/r/1305214 (https://phabricator.wikimedia.org/T429629) (owner: 10Vaughn Walters) [05:11:29] (03PS9) 10Hashar: jjb: [catalyst-daily-core] export Catalyst env [integration/config] - 10https://gerrit.wikimedia.org/r/1305214 (https://phabricator.wikimedia.org/T429629) (owner: 10Vaughn Walters) [05:11:51] (03CR) 10Hashar: jjb: [catalyst-daily-core] export Catalyst env (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1305214 (https://phabricator.wikimedia.org/T429629) (owner: 10Vaughn Walters) [05:13:43] 06Release-Engineering-Team, 07Puppet: registry-homepage-builder.py doesn't sort images as expected - https://phabricator.wikimedia.org/T388287#12048852 (10hashar) The `build-homepage` service is indeed failing https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status?orgId=1&from=now-5m&to=now&timezone=utc&var-... [05:17:15] (03CR) 10Hashar: [C:03+2] "I have updated the job ( https://integration.wikimedia.org/ci/job/catalyst-daily-core/ ), manually ran it and this time it passed! Well d" [integration/config] - 10https://gerrit.wikimedia.org/r/1305214 (https://phabricator.wikimedia.org/T429629) (owner: 10Vaughn Walters) [05:18:55] (03Merged) 10jenkins-bot: jjb: [catalyst-daily-core] export Catalyst env [integration/config] - 10https://gerrit.wikimedia.org/r/1305214 (https://phabricator.wikimedia.org/T429629) (owner: 10Vaughn Walters) [05:37:07] 10Phabricator, 06Release-Engineering-Team (Doing 😎), 06ServiceOps new: Custom task form for Production Readiness Checklist process (SCROLL) - https://phabricator.wikimedia.org/T429692#12048890 (10Aklapper) a:03Aklapper Morn, I created the [Form](https://www.mediawiki.org/wiki/Phabricator/Help/Forms) https:... [05:50:53] 10Phabricator (Upstream): Result paging: (AphrontParameterQueryException) Expected a numeric scalar or null for %Ld conversion. Query: x.id IN (%Ld) - https://phabricator.wikimedia.org/T430016 (10Aklapper) 03NEW p:05Triage→03Low [06:05:41] 10Continuous-Integration-Infrastructure, 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, and 2 others: Fetches from Gerrit aborted due to: GnuTLS recv error (-54): Error in the pull function - https://phabricator.wikimedia.org/T420865#12048937 (10ABran-WMF) >>! In T420865#12046290, @hashar... [06:06:40] 10Continuous-Integration-Infrastructure, 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, and 2 others: Fetches from Gerrit aborted due to: GnuTLS recv error (-54): Error in the pull function - https://phabricator.wikimedia.org/T420865#12048938 (10ABran-WMF) 05Open→03Resolved [06:06:42] 10Phabricator (Upstream), 07Upstream: Result paging: (AphrontParameterQueryException) Expected a numeric scalar or null for %Ld conversion. Query: x.id IN (%Ld) - https://phabricator.wikimedia.org/T430016#12048950 (10Aklapper) [06:50:56] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Dumbledore - https://phabricator.wikimedia.org/T430019 (10Dumbledore) 03NEW [06:52:52] 10GitLab (Account Approval), 06Release-Engineering-Team (Doing 😎): Requesting GitLab account activation for Dumbledore - https://phabricator.wikimedia.org/T430019#12049013 (10Aklapper) 05Open→03Resolved a:03Aklapper Done. Happy hacking! [06:55:54] (03update) 10lenap: group-management: add retry to gitlab helper readonly methods [repos/releng/gitlab-settings] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/merge_requests/82 (https://phabricator.wikimedia.org/T423984) [06:57:30] (03update) 10lenap: group-management: add retry to gitlab helper _get method [repos/releng/gitlab-settings] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-settings/-/merge_requests/82 (https://phabricator.wikimedia.org/T423984) [06:59:03] 06Release-Engineering-Team, 07Puppet: registry-homepage-builder.py doesn't sort images as expected - https://phabricator.wikimedia.org/T388287#12049032 (10elukey) ` Jun 24 06:24:07 registry2004 registry-homepage-builder[3522966]: INFO:root:Fetching the image catalog for localhost:5004 Jun 24 06:24:07 registry2... [07:25:09] 06Release-Engineering-Team (Doing 😎), 10Development-Metrics: Improve data quality in Development Metrics database (June 2026) - https://phabricator.wikimedia.org/T429739#12049065 (10Aklapper) [08:28:19] (03PS3) 10Abijeet Patro: Add ULS as a dependency for CX [integration/config] - 10https://gerrit.wikimedia.org/r/1305348 [08:28:28] (03PS4) 10Abijeet Patro: Add ULS as a phan dependency for CX [integration/config] - 10https://gerrit.wikimedia.org/r/1305348 [08:43:26] (03PS5) 10Abijeet Patro: zuul: Add ULS as a phan dependency for CX [integration/config] - 10https://gerrit.wikimedia.org/r/1305348 [08:45:48] (03CR) 10Hashar: [C:03+2] zuul: Add ULS as a phan dependency for CX [integration/config] - 10https://gerrit.wikimedia.org/r/1305348 (owner: 10Abijeet Patro) [08:46:53] 06Release-Engineering-Team (Doing 😎), 10Development-Metrics: Improve data quality in Development Metrics database (June 2026) - https://phabricator.wikimedia.org/T429739#12049270 (10Aklapper) [08:47:24] (03Merged) 10jenkins-bot: zuul: Add ULS as a phan dependency for CX [integration/config] - 10https://gerrit.wikimedia.org/r/1305348 (owner: 10Abijeet Patro) [08:48:53] (03CR) 10Hashar: [C:03+2] "Deployed and I have `recheck` the two changes:" [integration/config] - 10https://gerrit.wikimedia.org/r/1305348 (owner: 10Abijeet Patro) [08:53:26] 10GitLab, 06Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12049273 (10ABran-WMF) >>! In T425441#12047493, @dcaro wrote: > Fyi. With the new host, the [[ https://docs.gitlab.com/cli/ | gitlab client (glab) ]] sto... [09:01:52] 10GitLab, 06Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12049283 (10ABran-WMF) @dcaro after a quick verification of my config, it seems that there was a yaml key `host` that was set to `gitlab.com` despite a f... [09:02:46] 10GitLab, 06Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12049286 (10dcaro) A bit more detail on my specific error in case it helps, with the settings changed as the help says modifying the global git config: `... [09:05:15] 10GitLab, 06Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12049300 (10ABran-WMF) >>! In T425441#12049286, @dcaro wrote: > With that it works without having to add extra remotes to git. Neat! I'll update the doc... [09:23:20] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Remove buildkit helper image docker/dockerfile-copy from build pipeline - https://phabricator.wikimedia.org/T321316#12049374 (10Jelto) [09:27:52] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Remove buildkit helper image docker/dockerfile-copy from build pipeline - https://phabricator.wikimedia.org/T321316#12049385 (10Jelto) [09:42:15] 10Phabricator, 06Release-Engineering-Team (Priority Backlog 📥): Phab Scap template vs actual value out of sync for numerous values - https://phabricator.wikimedia.org/T398460#12049458 (10Aklapper) Removed `differential.allow-self-accept` from our config file in https://gitlab.wikimedia.org/repos/phabricator/de... [09:43:11] Hey y'all? Can I tap one of y'all for a bit of help with some mediawiki shenanigans afterwards? I've never done anything to beta and I'd like to test https://phabricator.wikimedia.org/T418494 there before I do it on prod. Is the procedure to merge https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1304845 to master (just +2 it), test the db changes there then I can move on to backporting [09:43:13] to prod? [09:45:39] 10Phabricator, 06Release-Engineering-Team (Priority Backlog 📥), 13Patch-For-Review: Notify Phab WMF users to link their SUL account (who currently only link LDAP) - https://phabricator.wikimedia.org/T407064#12049478 (10Aklapper) 05Stalled→03Open [09:45:54] 10GitLab, 06Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12049480 (10CWilliams-WMF) I added the global rewrite as per the notice and then received "Permission denied (publickey)" instead of success. Removing th... [09:45:54] (03update) 10aklapper: Notify Phab WMF users to link their SUL account [repos/phabricator/phabricator] (wmf/stable) - 10https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/104 (https://phabricator.wikimedia.org/T407064) [10:19:22] 10Phabricator, 06Release-Engineering-Team (Priority Backlog 📥): Phab Scap template vs actual value out of sync for numerous values - https://phabricator.wikimedia.org/T398460#12049622 (10Aklapper) Deleted three more custom database values which were moved to the config file yesterday: ` aklapper@phab1004:/srv/... [10:51:11] 10GitLab, 06Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12049705 (10ABran-WMF) >>! In T425441#12049480, @CWilliams-WMF wrote: > What is the fix for that? Thanks for reporting the installed version! Have you... [11:22:37] 10Gerrit, 06collaboration-services: Gerrit backups are growing - https://phabricator.wikimedia.org/T411583#12049793 (10jcrespo) 05Resolved→03Open p:05Medium→03Unbreak! Gerrit backups are again out of control, and growing at +12Tib in 18 days, at that rate, backups will run out of space in a couple of m... [11:36:27] (03PS1) 10Hslater: Zuul: [mediawiki/extensions/AtMentions] Re-enable master jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1305386 [11:39:04] 10Phabricator, 06Data-Platform-SRE, 10Test Kitchen, 07OKR-Work: Add phabricator to the kubernetes service mesh - https://phabricator.wikimedia.org/T430024#12049839 (10Aklapper) [11:39:43] 10Phabricator, 06Data-Platform-SRE, 10Test Kitchen, 07OKR-Work: Add phabricator to the kubernetes service mesh - https://phabricator.wikimedia.org/T430024#12049841 (10Aklapper) [11:39:44] 10Phabricator-Bot-Requests, 06Release-Engineering-Team (Doing 😎): Phabricator Bot Request: Test Kitchen Experiments Status Reporter - https://phabricator.wikimedia.org/T429377#12049842 (10Aklapper) [11:53:24] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: Remove buildkit helper image docker/dockerfile-copy from build pipeline - https://phabricator.wikimedia.org/T321316#12049867 (10Jelto) [12:10:30] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Sanskardubeydev - https://phabricator.wikimedia.org/T430034 (10Sanskardubeydev) 03NEW [12:21:55] 10GitLab, 06Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12049941 (10CWilliams-WMF) > Thanks for reporting the installed version! Have you tried to run wmf-update-ssh-config after the update? I have just tried... [12:24:04] 10GitLab (Infrastructure), 06collaboration-services: Make sure GitLab does not exceed apus object storage quotas - https://phabricator.wikimedia.org/T413871#12049943 (10Jelto) p:05Triage→03Medium current quota usage: ` jelto@gitlab1004:~$ s3cmd du -H s3://gitlab-packages 270G 8501 objects s3://gitlab-... [12:40:48] (03CR) 10Zaidusyy: "I tried to submit this upstream to Google Gerrit, but my Google account is bugged and giving me a 403 Permission Denied error even after s" [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1305218 (https://phabricator.wikimedia.org/T429901) (owner: 10Zaidusyy) [12:50:34] (03PS2) 10Arnaudb: jjb: trace git HTTPS transport on gated quibble jobs to debug gerrit resets [integration/config] - 10https://gerrit.wikimedia.org/r/1305393 (https://phabricator.wikimedia.org/T429749) [12:52:56] 10Continuous-Integration-Infrastructure, 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, and 2 others: Gerrit: network errors in CI - https://phabricator.wikimedia.org/T429749#12050055 (10ABran-WMF) @hashar I tried to reuse [[ https://gerrit.wikimedia.org/r/c/integration/config/+/1283846 |... [12:55:23] (03CR) 10Reedy: "CC paladox" [software/gerrit] (wmf/stable-3.10) - 10https://gerrit.wikimedia.org/r/1305218 (https://phabricator.wikimedia.org/T429901) (owner: 10Zaidusyy) [12:57:06] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Sanskardubeydev - https://phabricator.wikimedia.org/T430034#12050105 (10Aklapper) 05Open→03Invalid Approval happened already automatically: https://wikitech.wikimedia.org/wiki/Tool:Gitlab-account-approval/Log [13:22:22] 10GitLab, 06Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12050233 (10ABran-WMF) Thanks, this is very helpful. Your `~/.ssh/config` is up to date (the `Host gitlab.wikimedia.org gitlab-ssh.wikimedia.org` block i... [13:30:46] 06Release-Engineering-Team, 07Puppet: registry-homepage-builder.py doesn't sort images as expected - https://phabricator.wikimedia.org/T388287#12050271 (10hashar) The pages got generated. I went to purge the example page: ` $ mwscript purgeList --wiki=aawiki https://docker-registry.wikimedia.org/releng/node22-... [13:30:57] 06Release-Engineering-Team, 07Puppet: registry-homepage-builder.py doesn't sort images as expected - https://phabricator.wikimedia.org/T388287#12050274 (10hashar) 05Open→03Resolved [13:43:18] 06Release-Engineering-Team (Radar), 06Infrastructure-Foundations: registry-homepage-builder.py doesn't sort images as expected - https://phabricator.wikimedia.org/T388287#12050396 (10hashar) [14:16:42] 10GitLab, 06Release-Engineering-Team (Radar), 06collaboration-services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12050647 (10CWilliams-WMF) Well, well... `sh ❯ ssh -T git@gitlab.wikimedia.org Welcome to GitLab, @cwilliams! ❯ ssh -T git@gitlab-ssh.wikimedia.org Wel... [15:12:08] !log sudo systemctl restart php8.3-fpm on deployment-jobrunner05 (Attempting to resolve logspam) [15:12:09] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:19:10] 10Beta-Cluster-Infrastructure: Error during startup of php8.3-fpm on deployment-jobrunner05.deployment-prep - https://phabricator.wikimedia.org/T430062 (10dancy) 03NEW [15:29:47] 10Phabricator, 10Test Kitchen, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 07OKR-Work, 13Patch-For-Review: Add phabricator to the kubernetes service mesh - https://phabricator.wikimedia.org/T430024#12050942 (10brouberol) a:03brouberol [15:29:49] 10Phabricator, 10Test Kitchen, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 07OKR-Work, 13Patch-For-Review: Add phabricator to the kubernetes service mesh - https://phabricator.wikimedia.org/T430024#12050945 (10brouberol) 05Open→03In progress [15:29:51] 10Phabricator, 10Test Kitchen, 06Data-Platform-SRE (2026-06-05 - 2026-06-26), 07OKR-Work, 13Patch-For-Review: Add phabricator to the kubernetes service mesh - https://phabricator.wikimedia.org/T430024#12050944 (10brouberol) [15:42:07] (03CR) 10Vaughn Walters: "Thx Antoine ... and your code comment actually makes sense as opposed to mine! 😄. Also TIL about YAML line folding (`>-`)" [integration/config] - 10https://gerrit.wikimedia.org/r/1305214 (https://phabricator.wikimedia.org/T429629) (owner: 10Vaughn Walters) [15:58:04] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance deployment-changeprop-1 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [16:10:59] (03open) 10hashar: Default to user `zuul` [repos/releng/zuul/zuul] - 10https://gitlab.wikimedia.org/repos/releng/zuul/zuul/-/merge_requests/20 [16:30:22] (03open) 10dancy: config.py: Fix default mw_web_clusters [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1209 [16:48:36] (03merge) 10dancy: config.py: Fix default mw_web_clusters [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1209 [17:06:46] 10Beta-Cluster-Infrastructure: Enable opcache revalidation on deployment-prep jobrunners - https://phabricator.wikimedia.org/T430075 (10dancy) 03NEW [17:17:39] 10Beta-Cluster-Infrastructure: Enable opcache revalidation on deployment-prep jobrunners - https://phabricator.wikimedia.org/T430075#12051456 (10dancy) [17:19:18] 10Beta-Cluster-Infrastructure: Properly handle mediawiki code/config updates on deployment-prep jobrunners - https://phabricator.wikimedia.org/T430075#12051464 (10dancy) p:05Triage→03Medium [17:26:18] !log Set `profile::puppetserver::autosign: /usr/local/sbin/validatecloudvpsfqdn.py` in hiera config for deployment-puppetserver prefix (T429413) [17:26:21] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:26:21] T429413: Eliminate sudo rm -rf /var/lib/puppet/ssl step in new deployment-prep WMCS project (and others) - https://phabricator.wikimedia.org/T429413 [18:03:50] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for buxtang - https://phabricator.wikimedia.org/T430079 (10Buxtang) 03NEW [18:05:31] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for buxtang - https://phabricator.wikimedia.org/T430079#12051648 (10Buxtang) seeking to take this task https://phabricator.wikimedia.org/T429229 [18:07:42] 10GitLab (Account Approval), 06Release-Engineering-Team, 07Essential-Work: Requesting GitLab account activation for buxtang - https://phabricator.wikimedia.org/T430079#12051656 (10brennen) 05Open→03Resolved p:05Triage→03Low a:03brennen [18:08:50] 10GitLab (Account Approval), 06Release-Engineering-Team, 07Essential-Work: Requesting GitLab account activation for buxtang - https://phabricator.wikimedia.org/T430079#12051664 (10brennen) Hi @Buxtang - I've approved your GitLab account. Happy hacking! (Not added to Trusted-Contributors since there's no... [18:11:49] 10GitLab (Account Approval), 06Release-Engineering-Team, 07Essential-Work: Requesting GitLab account activation for buxtang - https://phabricator.wikimedia.org/T430079#12051684 (10Buxtang) Thanks! [18:34:39] 10GitLab (CI & Job Runners), 06Release-Engineering-Team, 07Essential-Work: Buildkit v0.31.1 released - https://phabricator.wikimedia.org/T429988#12051759 (10dancy) [18:38:43] 10Beta-Cluster-Infrastructure: Last Puppet run was over 24 hours ago on instance deployment-changeprop-1 in project deployment-prep - https://phabricator.wikimedia.org/T429712#12051777 (10dancy) →14Duplicate dup:03T429662 [18:38:44] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-changeprop-1 on project deployment-prep - https://phabricator.wikimedia.org/T429662#12051779 (10dancy) [18:39:21] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-changeprop-1 on project deployment-prep - https://phabricator.wikimedia.org/T429662#12051785 (10dancy) The tail end of `run-puppet-agent` output: ` Error: Failed to apply catalog: No space left on device @ dir_s_mkdir - /var/lib/pupp... [18:39:45] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-changeprop-1 on project deployment-prep - https://phabricator.wikimedia.org/T429662#12051787 (10dancy) ` dancy@deployment-changeprop-1:~$ df -t ext4 -h Filesystem Size Used Avail Use% Mounted on /dev/sda1 20G 20G 0... [18:43:10] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-changeprop-1 on project deployment-prep - https://phabricator.wikimedia.org/T429662#12051793 (10dancy) The biggest consumer is ` -rw-r----- 1 root root 9.0G Jun 24 18:42 /var/lib/docker/containers/c5a95725142c1168d1dca1c5a6bd3bf4ec5... [18:50:40] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-changeprop-1 on project deployment-prep - https://phabricator.wikimedia.org/T429662#12051803 (10dancy) 05Open→03Resolved a:03dancy I chose to resolve this by deleting the two running Docker containers and making puppet recr... [18:53:47] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: setup 2 contint machines for jenkins - https://phabricator.wikimedia.org/T418521#12051810 (10Dzahn) @hashar regarding the test with netcat: This is actually an IPv4 vs IPv6 issue: `nc -v -l... [18:54:34] 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Properly handle mediawiki code/config updates on deployment-prep jobrunners - https://phabricator.wikimedia.org/T430075#12051811 (10dancy) 05Open→03Resolved a:03dancy [18:56:16] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: setup 2 contint machines for jenkins - https://phabricator.wikimedia.org/T418521#12051817 (10Dzahn) regardless; let me configure the `upstream_server` to use localhost instead: https://gerrit... [19:03:04] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance deployment-changeprop-1 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:03:33] RESOLVED: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance deployment-changeprop-1 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [19:14:51] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: setup 2 contint machines for jenkins - https://phabricator.wikimedia.org/T418521#12051855 (10Dzahn) https://puppet-compiler.wmflabs.org/output/1305488/8778/contint1003.wikimedia.org/index.html... [19:18:18] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06collaboration-services, 13Patch-For-Review: setup 2 contint machines for jenkins - https://phabricator.wikimedia.org/T418521#12051861 (10Dzahn) @hashar The change above has been made. Can we start jenkins to test again and with the real... [20:48:36] https://global-search.toolforge.org/gives me a 500 error when I try to log in. Known? [20:51:56] if its a oauth1 user, it appears there was a bit of a surprise breaking change [20:52:15] bawollfs comment in -tech [20:53:48] is there a phab task for that? maybe i could connect the pieces to help musikanimal and ebernhardson fix it up? [20:58:10] this was the change that was referenced https://gerrit.wikimedia.org/r/c/mediawiki/extensions/OAuth/+/1304678 [20:58:18] but i havnt seen any tasks popup [21:36:15] (03PS1) 10Alex Paskulin: Remove API Portal skin and extension [integration/config] - 10https://gerrit.wikimedia.org/r/1305507 (https://phabricator.wikimedia.org/T429373) [21:45:51] p858snake|cloud: thanks, that was very helpful [21:48:18] 06Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.47.0-wmf.8 deployment blockers - https://phabricator.wikimedia.org/T423917#12052161 (10cscott) [21:51:08] bawolff was the real vip by commenting in a channel i saw [21:51:35] Credit goes to the commons users who was yelling at me on discord :P [21:51:57] i'm assuming this merits an UBN backport? [21:52:46] I bumped the priority of the bug, but tgr and thiemo were the original reviewers, and they are likely into their evening activities Europe time. [21:56:40] it's getting close to the end of my day as well, and I'd love to hand this off to a Californian [22:06:56] cscott: Are you just looking for a +2? Because i'm happy to +2 it on the principle that controversial changes should be reverted and discussed [22:07:55] tgr|away just answered on wmf slack, and said "i'll have a fix in a few minutes" so I'm going to let him take it from here. [22:08:28] bawolff: i was mostly looking for someone i could hand off responsibility too because I don't like leaving the wikis broken, but i need to get AFK and make dinner for my family [22:09:05] fair. I'd be happy to +2 something, but i don't really want responsibility for it either