[00:23:21] 10Continuous-Integration-Config, 10MediaWiki-Installer, 06MediaWiki-Platform-Team, 10MediaWiki-extensions-CentralAuth, 07ci-test-error: Admin account created by the installer isn't made global by CentralAuth (at least in CI) - https://phabricator.wikimedia.org/T358985#9613867 (10Daimona) >>! In T358985#9... [00:23:36] 10Continuous-Integration-Config, 10MediaWiki-Installer, 06MediaWiki-Platform-Team, 10MediaWiki-extensions-CentralAuth, 07ci-test-error: Admin account created by the installer isn't made global by CentralAuth - https://phabricator.wikimedia.org/T358985#9613868 (10Daimona) [07:16:02] 10Phabricator (Upstream), 07Upstream: Public log for Phabricator user account role change (e.g. deactivation, admin rights) - https://phabricator.wikimedia.org/T102576#9614269 (10Bugreporter) It would at least be better if account disabling is shown in Main Page Activity Feed and Recent Activity of the admin u... [08:16:49] 10Release-Engineering-Team (Now this 🫠), 05Release, 05Train Deployments: 1.42.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T354439#9614313 (10ovasileva) [08:38:32] 10Release-Engineering-Team (Now this 🫠), 05Release, 05Train Deployments: 1.42.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T354439#9614334 (10jnuche) 05Open→03Resolved [09:07:07] 10Release-Engineering-Team (Now this 🫠), 06Security-Team, 13Patch-For-Review, 07SecTeam-Processed: Notify MediaWiki security tasks as soon as an uploaded patch fails to apply - https://phabricator.wikimedia.org/T350065#9614350 (10CodeReviewBot) jnuche opened https://gitlab.wikimedia.org/repos/releng/releas... [09:08:00] 10Release-Engineering-Team (Now this 🫠), 06Security-Team, 13Patch-For-Review, 07SecTeam-Processed: Notify MediaWiki security tasks as soon as an uploaded patch fails to apply - https://phabricator.wikimedia.org/T350065#9614352 (10CodeReviewBot) jnuche merged https://gitlab.wikimedia.org/repos/releng/releas... [09:50:01] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.42.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T354440#9614406 (10SecurityPatchBot) [10:16:37] 10Release-Engineering-Team (Now this 🫠), 06Security-Team, 13Patch-For-Review, 07SecTeam-Processed: Notify MediaWiki security tasks as soon as an uploaded patch fails to apply - https://phabricator.wikimedia.org/T350065#9614419 (10CodeReviewBot) jnuche opened https://gitlab.wikimedia.org/repos/releng/scap/-... [10:20:38] 10Release-Engineering-Team (Now this 🫠), 06Security-Team, 13Patch-For-Review, 07SecTeam-Processed: Notify MediaWiki security tasks as soon as an uploaded patch fails to apply - https://phabricator.wikimedia.org/T350065#9614433 (10CodeReviewBot) jnuche merged https://gitlab.wikimedia.org/repos/releng/scap/-... [13:47:33] 10Release-Engineering-Team (Now this 🫠), 06Security-Team, 07SecTeam-Processed: Notify MediaWiki security tasks as soon as an uploaded patch fails to apply - https://phabricator.wikimedia.org/T350065#9614933 (10Jdforrester-WMF) This is brilliant, thank you for this! [13:50:38] 10Release-Engineering-Team (Radar), 10Release Pipeline, 06SRE, 06serviceops, and 2 others: Remove obsoleted docker images - https://phabricator.wikimedia.org/T242604#9614971 (10akosiaris) [13:53:43] 10Phabricator (Upstream), 10Release-Engineering-Team (Radar), 06Infrastructure-Foundations, 06collaboration-services, and 2 others: Ensure that phabricator.wikimedia.org adheres to Google's sender guidelines - https://phabricator.wikimedia.org/T355691#9615008 (10eoghan) 05Open→03Resolved We've ticked a... [13:54:03] 10Continuous-Integration-Infrastructure, 06SRE, 06collaboration-services, 10vm-requests, 13Patch-For-Review: Ganeti VM for contint migration - https://phabricator.wikimedia.org/T358237#9615011 (10jnuche) >>! In T358237#9611992, @Dzahn wrote: > zuul has now succesfully been deployed to this machine by @jn... [13:55:32] 06Release-Engineering-Team, 10Release Pipeline (Blubber): Allow installing Poetry without installing dependencies with it - https://phabricator.wikimedia.org/T359630 (10taavi) 03NEW [14:10:02] 10GitLab (Infrastructure), 06collaboration-services: Evaluate a high available GitLab architecture - https://phabricator.wikimedia.org/T323201#9615154 (10Jelto) >>! In T323201#9523904, @Joe wrote: > Gitlab needs regular maintenance windows, at least once a month, if not more often and they usually last around... [15:47:15] 10Release-Engineering-Team (Now this 🫠), 06Security-Team, 07SecTeam-Processed: Notify MediaWiki security tasks as soon as an uploaded patch fails to apply - https://phabricator.wikimedia.org/T350065#9615506 (10dancy) >>! In T350065#9614931, @Jdforrester-WMF wrote: > This is brilliant, thank you for this! Ag... [16:15:50] 06Release-Engineering-Team, 10Scap: Get rid of the /srv/mediawiki/php symbolic link - https://phabricator.wikimedia.org/T359643 (10dancy) 03NEW [16:17:00] 06Release-Engineering-Team, 10Scap: Get rid of the /srv/mediawiki/php symbolic link - https://phabricator.wikimedia.org/T359643#9615596 (10dancy) @Krinkle I would like your input on this please. [16:51:34] (Queue (Jenkins jobs + Zuul functions) alert) firing: - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [16:56:34] (Queue (Jenkins jobs + Zuul functions) alert) resolved: - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [16:56:49] (Queue (Jenkins jobs + Zuul functions) alert) firing: - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [16:59:59] PROBLEM - Work requests waiting in Zuul Gearman server on contint2002 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [400.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [17:01:49] (Queue (Jenkins jobs + Zuul functions) alert) resolved: - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [17:01:59] RECOVERY - Work requests waiting in Zuul Gearman server on contint2002 is OK: OK: Less than 100.00% above the threshold [200.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [17:02:00] ^ I _think_ it's temporary ? [17:02:02] ah :) [17:02:15] big spike.. but a spike [17:08:21] 06Release-Engineering-Team, 10Scap: Get rid of the /srv/mediawiki/php symbolic link - https://phabricator.wikimedia.org/T359643#9615753 (10Krinkle) Historically, this symlink was part of the `/h/w/c/p` shortcut. See also [wikitech:Hwcp](https://wikitech.wikimedia.org/wiki/Obsolete:Hwcp). It powered various th... [17:08:29] 06Release-Engineering-Team, 10Scap, 10MediaWiki-Platform-Team (Radar): Get rid of the /srv/mediawiki/php symbolic link - https://phabricator.wikimedia.org/T359643#9615754 (10Krinkle) [17:16:40] (Queue (Jenkins jobs + Zuul functions) alert) firing: - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [17:25:01] PROBLEM - Work requests waiting in Zuul Gearman server on contint2002 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [400.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [17:36:40] (Queue (Jenkins jobs + Zuul functions) alert) resolved: - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [17:58:03] RECOVERY - Work requests waiting in Zuul Gearman server on contint2002 is OK: OK: Less than 100.00% above the threshold [200.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [18:13:25] !log Starting gitlab cloud runners k8s cluster upgrade (T359594) [18:13:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:13:29] T359594: Upgrade production gitlab-cloud-runners to kubernetes 1.29 - https://phabricator.wikimedia.org/T359594 [19:00:17] 06Release-Engineering-Team: gitlab-cloud-runners k8s cluster bootstrapping problem - https://phabricator.wikimedia.org/T359661#9616098 (10dancy) [19:08:00] 06Release-Engineering-Team: gitlab-cloud-runners k8s cluster bootstrapping problem - https://phabricator.wikimedia.org/T359661#9616121 (10dancy) [19:21:14] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Now this 🫠), 13Patch-For-Review: Upgrade production gitlab-cloud-runners to kubernetes 1.29 - https://phabricator.wikimedia.org/T359594#9616160 (10Sandeeps) 05Open→03In progress [19:28:57] 10Release-Engineering-Team (Seen), 10Scap: Run Swagger checks in Scap before exposing to prod MW traffic - https://phabricator.wikimedia.org/T239376#9616184 (10thcipriani) >>! In T239376#9612482, @dancy wrote: >>>! In T239376#9612439, @Krinkle wrote: >> * The Swagger spec is now unused, and can be removed from... [19:37:18] 06Release-Engineering-Team, 13Patch-For-Review: gitlab-cloud-runners k8s cluster bootstrapping problem - https://phabricator.wikimedia.org/T359661#9616210 (10CodeReviewBot) dancy opened https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/352 Add depends_on settings to improve bootst... [19:49:54] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Now this 🫠), 13Patch-For-Review: Upgrade production gitlab-cloud-runners to kubernetes 1.29 - https://phabricator.wikimedia.org/T359594#9616251 (10CodeReviewBot) sandeeps merged https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requ... [20:43:00] 06Release-Engineering-Team, 10Scap, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: Get rid of the /srv/mediawiki/php symbolic link - https://phabricator.wikimedia.org/T359643#9616447 (10dancy) @Krinkle Would it be feasible to bring `COPYING` and `CREDITS` under the care of `/w/static.php` (or somet... [21:21:56] 06Release-Engineering-Team, 10Scap, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: Get rid of the /srv/mediawiki/php symbolic link - https://phabricator.wikimedia.org/T359643#9616500 (10Krinkle) From a quick glance, static.php already allows anything to be served more or less. As long as 1) the pat... [21:32:21] !log Added ecarg (WMF staff member) as a member of deployment-prep so they can help with Wikifunctions Beta Cluster maintenance. [21:32:23] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:18:05] 06Release-Engineering-Team, 10Scap, 10MediaWiki-libs-Mime, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: Get rid of the /srv/mediawiki/php symbolic link - https://phabricator.wikimedia.org/T359643#9616664 (10Krinkle) [22:19:48] 06Release-Engineering-Team, 13Patch-For-Review: gitlab-cloud-runners k8s cluster bootstrapping problem - https://phabricator.wikimedia.org/T359661#9616665 (10CodeReviewBot) dancy merged https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/352 Add depends_on settings to improve bootst... [22:38:35] 10Release-Engineering-Team (Now this 🫠): gitlab-cloud-runners k8s cluster bootstrapping problem - https://phabricator.wikimedia.org/T359661#9616689 (10dancy) 05Open→03In progress p:05Triage→03Medium [22:39:00] 10Release-Engineering-Team (Now this 🫠): gitlab-cloud-runners k8s cluster bootstrapping problem - https://phabricator.wikimedia.org/T359661#9616692 (10dancy) Need to retest staging cluster destroy/rebuild. [23:08:11] 10Release-Engineering-Team (Seen), 10Scap: Run Swagger checks in Scap before exposing to prod MW traffic - https://phabricator.wikimedia.org/T239376#9616730 (10Krinkle) I'd use [kafkacat](https://wikitech.wikimedia.org/wiki/Kafkacat) to consume [webrequest_text](https://wikitech.wikimedia.org/wiki/Analytics/Da... [23:14:44] thcipriani: fyi ^ since the task is resolved [23:17:01] Krinkle: thanks! ...I definitely do not have access to a stats host. [23:17:11] ah.. [23:18:17] * Krinkle patiently runs command [23:27:29] thanks and sorry Krinkle didn't know offhand this would be one of the hats I don't own [23:30:58] https://github.com/wikimedia/operations-puppet/blob/fd84206c394687307c3294efdc5cfa6158967366/modules/admin/data/data.yaml#L381C41-L425C103