[00:59:30] 10Phabricator, 10Trust-and-Safety, 10cloud-services-team, 10wikitech.wikimedia.org: Reset 2FA for Developer account 'Rosalie Perside (WMDE)' and Phabricator account @Rosalie_WMDE - https://phabricator.wikimedia.org/T329179 (10bd808) @Rosalie_WMDE, please follow the steps at https://wikitech.wikimedia.org/w... [03:18:29] (03PS1) 10Jforrester: Zuul: [wikimedia/fundraising/crm/civicrm] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/888117 (https://phabricator.wikimedia.org/T324732) [03:19:53] (03CR) 10Jforrester: [C: 03+2] Zuul: [wikimedia/fundraising/crm/civicrm] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/888117 (https://phabricator.wikimedia.org/T324732) (owner: 10Jforrester) [03:21:02] (03Merged) 10jenkins-bot: Zuul: [wikimedia/fundraising/crm/civicrm] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/888117 (https://phabricator.wikimedia.org/T324732) (owner: 10Jforrester) [03:21:30] !log Zuul: [wikimedia/fundraising/crm/civicrm] Mark as archived for T324732 [03:21:33] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [03:21:33] T324732: Archive the fundraising/crm/civicrm repo - https://phabricator.wikimedia.org/T324732 [03:28:54] 10Release-Engineering-Team, 10Fundraising-Backlog, 10Projects-Cleanup, 10Wikimedia-Fundraising-CiviCRM: Decommission Fundraising's crm/civicrm git repo - https://phabricator.wikimedia.org/T314995 (10Jdforrester-WMF) [03:30:07] 10Release-Engineering-Team, 10Fundraising-Backlog, 10Projects-Cleanup, 10Wikimedia-Fundraising-CiviCRM: Decommission Fundraising's crm/civicrm git repo - https://phabricator.wikimedia.org/T314995 (10Jdforrester-WMF) [03:30:16] 10Release-Engineering-Team, 10Fundraising-Backlog, 10Projects-Cleanup, 10Wikimedia-Fundraising-CiviCRM: Decommission Fundraising's crm/civicrm git repo - https://phabricator.wikimedia.org/T314995 (10Jdforrester-WMF) 05Open→03Resolved [04:33:07] 10Continuous-Integration-Infrastructure, 10OOUI, 10Regression: OOUI PHP demos page is broken (again) - https://phabricator.wikimedia.org/T322357 (10OriginalAuthority) Is there any update on when this is going to be fixed? Not having this documentation severly hinders extension development for those not famil... [07:52:15] 10Continuous-Integration-Config, 10Growth-Team, 10GrowthExperiments, 10ci-test-error: phpbench job for GrowthExperiments fails due to missing dependency - https://phabricator.wikimedia.org/T329280 (10kostajh) >>! In T329280#8603092, @Tgr wrote: > Possibly a duplicate of {T232413}? I don't think so, becaus... [08:00:57] 10Continuous-Integration-Config, 10Growth-Team, 10GrowthExperiments, 10ci-test-error: phpbench job for GrowthExperiments fails due to missing dependency - https://phabricator.wikimedia.org/T329280 (10Tgr) > I don't think so, because that is for phan, while this is for the new-ish (T291549) phpbench job. O... [08:02:10] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10MediaWiki-Installer, 10Wikimedia-production-error (ARCHIVED -- Shared Build Failure): mwext-php72-phan-docker job fails on WikimediaMessages: A dependency error was encountered while installing... - https://phabricator.wikimedia.org/T232413 [09:20:10] (03PS1) 10Ayounsi: Add CI to homer/deploy [integration/config] - 10https://gerrit.wikimedia.org/r/888171 (https://phabricator.wikimedia.org/T277440) [09:37:01] 10GitLab (Infrastructure), 10serviceops-collab: Migrate gitlab-test instance to bullseye - https://phabricator.wikimedia.org/T318521 (10Jelto) GitLab test instance under `gitlab.devtools.wmcloud.org` works again and data from the buster instance were transferred to the new bullseye instance. I used roughly th... [11:03:00] 10Project-Admins: Create project tag for Internships - https://phabricator.wikimedia.org/T329355 (10KSiebert) [11:20:39] 10Project-Admins: Create project tag for Internships - https://phabricator.wikimedia.org/T329355 (10Peachey88) > This project tag is to help interns to identify projects that are suitable for them. How would these tasks differ from #good_first_task? [11:29:43] 10Project-Admins: Create project tag for Internships - https://phabricator.wikimedia.org/T329355 (10KSiebert) [11:36:15] (03CR) 10Sergio Gimeno: zuul: Add GrowthExperiments to extension-javascript-documentation (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/888022 (https://phabricator.wikimedia.org/T329034) (owner: 10Kosta Harlan) [11:39:05] 10Project-Admins: Create project tag for Internships - https://phabricator.wikimedia.org/T329355 (10KSiebert) As we will want to be able to assess the contributions that interns have made it will be helpful to have a separate tag. [11:43:25] 10Project-Admins: Create project tag for Internships - https://phabricator.wikimedia.org/T329355 (10Aklapper) Hmm, what does "assess mean"? Maybe you could filter on the usernames of the interns? If a tag was somehow required, then this rather sounds like a `WMF-Interns-2023` group tag to me, as `#Internships` c... [11:51:53] 10GitLab (CI & Job Runners), 10serviceops-collab, 10Patch-For-Review: Use dedicated volume for /var/lib/docker on Trusted Runners - https://phabricator.wikimedia.org/T329035 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by eoghan@cumin2002 for host gitlab-runner2002.codfw.wmnet with... [12:26:58] 10GitLab (CI & Job Runners), 10serviceops-collab, 10Patch-For-Review: Use dedicated volume for /var/lib/docker on Trusted Runners - https://phabricator.wikimedia.org/T329035 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by eoghan@cumin2002 for host gitlab-runner2002.codfw.wmnet with OS... [12:39:08] 10Continuous-Integration-Config, 10Release-Engineering-Team (Radar), 10Analytics-Radar, 10ChangeProp, and 6 others: Run EventBus tests in MediaWiki core CI - https://phabricator.wikimedia.org/T257583 (10EChetty) [12:40:34] 10Release-Engineering-Team (Radar), 10Analytics, 10Data-Engineering-Planning, 10Event-Platform Value Stream: Stop using puppet + git pull for auto deployment of schema repos - https://phabricator.wikimedia.org/T274901 (10EChetty) [12:45:28] 10GitLab (CI & Job Runners), 10serviceops-collab: Use dedicated volume for /var/lib/docker on Trusted Runners - https://phabricator.wikimedia.org/T329035 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by eoghan@cumin2002 for host gitlab-runner2003.codfw.wmnet with OS bullseye [13:18:12] 10GitLab (CI & Job Runners), 10serviceops-collab: Use dedicated volume for /var/lib/docker on Trusted Runners - https://phabricator.wikimedia.org/T329035 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by eoghan@cumin2002 for host gitlab-runner2003.codfw.wmnet with OS bullseye completed: -... [13:21:33] 10GitLab (CI & Job Runners), 10serviceops-collab: Use dedicated volume for /var/lib/docker on Trusted Runners - https://phabricator.wikimedia.org/T329035 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by eoghan@cumin2002 for host gitlab-runner2004.codfw.wmnet with OS bullseye [13:56:15] 10GitLab (CI & Job Runners), 10serviceops-collab: Use dedicated volume for /var/lib/docker on Trusted Runners - https://phabricator.wikimedia.org/T329035 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by eoghan@cumin2002 for host gitlab-runner2004.codfw.wmnet with OS bullseye completed: -... [14:00:19] 10GitLab (CI & Job Runners), 10serviceops-collab: Use dedicated volume for /var/lib/docker on Trusted Runners - https://phabricator.wikimedia.org/T329035 (10eoghan) [14:00:47] 10Release-Engineering-Team (Radar), 10Analytics, 10Data-Engineering-Planning, 10Event-Platform Value Stream: Stop using puppet + git pull for auto deployment of schema repos - https://phabricator.wikimedia.org/T274901 (10awight) I would suggest going in a slightly different direction than described in the... [14:03:39] 10Project-Admins: Create project tag for Internships - https://phabricator.wikimedia.org/T329355 (10KSiebert) Leadership wants to be able to measure which internships have succeeded and how much time they got done during their time to be able to decide who will receive fulltime roles. [14:04:27] 10GitLab (Infrastructure), 10serviceops-collab: Migrate gitlab-test instance to bullseye - https://phabricator.wikimedia.org/T318521 (10Jelto) I've done some cleanup in the puppet code (both hiera and removing buster dependencies) and I updated https://wikitech.wikimedia.org/wiki/GitLab/Test_Instance as much a... [14:56:05] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MediaWiki-Stakeholders-Group, 10Epic, 10MW-1.40-release: Expand the set of bundled extensions and skins in MediaWiki 1.40 - https://phabricator.wikimedia.org/T317146 (10Dreamy_Jazz) [14:56:58] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MediaWiki-Stakeholders-Group, 10Epic, 10MW-1.40-release: Expand the set of bundled extensions and skins in MediaWiki 1.40 - https://phabricator.wikimedia.org/T317146 (10Dreamy_Jazz) [15:01:48] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MediaWiki-Stakeholders-Group, 10Epic, 10MW-1.40-release: Expand the set of bundled extensions and skins in MediaWiki 1.40 - https://phabricator.wikimedia.org/T317146 (10Dreamy_Jazz) [15:02:19] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MediaWiki-Stakeholders-Group, 10Epic, 10MW-1.40-release: Expand the set of bundled extensions and skins in MediaWiki 1.40 - https://phabricator.wikimedia.org/T317146 (10Dreamy_Jazz) [15:04:14] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MediaWiki-Stakeholders-Group, 10Epic, 10MW-1.40-release: Expand the set of bundled extensions and skins in MediaWiki 1.40 - https://phabricator.wikimedia.org/T317146 (10Dreamy_Jazz) [15:04:42] dancy: thcipriani: sorry for asking so late, but, would either of you be available Monday 17:00 UTC for SRE's incident review ritual to potentially discuss https://wikitech.wikimedia.org/wiki/Incidents/2023-01-17_MediaWiki ? [15:05:23] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MediaWiki-Stakeholders-Group, 10Epic, 10MW-1.40-release: Expand the set of bundled extensions and skins in MediaWiki 1.40 - https://phabricator.wikimedia.org/T317146 (10Dreamy_Jazz) [15:34:34] cdanis: Works for me [15:38:59] 10Project-Admins: Create project tag for WMF Internships - https://phabricator.wikimedia.org/T329355 (10Aklapper) [15:40:04] 10Project-Admins: Create project tag for WMF Internships - https://phabricator.wikimedia.org/T329355 (10Aklapper) 05Open→03Resolved a:03Aklapper Requested public project #wmf-internships-2023 has been created: https://phabricator.wikimedia.org/project/view/6390/ (In case you need to edit the project or pr... [15:53:28] cdanis: sure, I could make that work [15:54:23] thcipriani: dancy: thanks so much! added you both to the invite. of particular interest will be why the canary checks didn't save us from an outage here -- will you have a little time to take a look at that in advance? [15:54:51] I will make an attempt to collect information today. [15:54:55] <3 [15:55:07] cdanis: dancy: the reason is that this specific issue was only happening between the file sync and the php-fpm restart. with canary sync, those happen back-to-back, but not with the normal sync [15:56:23] ^ yeah, I think that's the theory. Canary servers didn't get any traffic until after php-fpm restart [15:57:04] and since that happens faster because of a smaller pool, there was no explosion there [15:57:24] how to mitigate that in future is a question [16:03:42] as always, thank you taavi [16:04:28] thcipriani: a related question on my mind is, how would this have looked different in a m7i-on-k8s world [16:05:24] 10Project-Admins, 10Data-Engineering, 10PM: Archive Analytics tag - https://phabricator.wikimedia.org/T298671 (10JArguello-WMF) Hi! The 218 open tasks should all go to #Data-Engineering-Icebox. The 116 open in #analytics-radar do not belong to Data Engineering, the team was keeping track of them, but was not... [16:06:47] I will always defer to dancy, but right now my thinking is this wouldn't happen in k8s because we'd never be running new code without a restart, but rather we'd be deploying a whole new container...I think :) [16:07:30] ^ dancy does this seem right to you? Or is there nuance I'm missing? [16:07:37] Right. There's never a case in the k8s world where there's a combination of old and new code in the same container. [16:10:21] that was my guess but I'm glad to hear it confirmed [17:20:20] 10GitLab (CI & Job Runners), 10serviceops-collab: Use dedicated volume for /var/lib/docker on Trusted Runners - https://phabricator.wikimedia.org/T329035 (10eoghan) 05Open→03Resolved I think this can be closed, we're using `/var/lib/docker` for all gitlab-runner hosts now, and I've updated the thresholds f... [17:20:23] 10GitLab (CI & Job Runners), 10serviceops-collab: add disk space usage to grafana dashboard for gitlab-runners - https://phabricator.wikimedia.org/T327435 (10eoghan) [17:47:24] 10Project-Admins, 10Data-Engineering, 10PM: Archive Analytics tag - https://phabricator.wikimedia.org/T298671 (10Aklapper) Thanks for the reply! > The 116 open in #analytics-radar do not belong to Data Engineering, the team was keeping track of them, but was not the responsible party, therefore, they should... [17:56:19] GitHub just re-invented Zuul :) https://github.blog/changelog/2023-02-08-pull-request-merge-queue-public-beta/ [17:57:08] * bd808 assumes there will be a crippled version in GitLab CE in 6 months or so [18:41:13] Hey all - would be nice to get some eyes on this failing security-ish CI job: https://phabricator.wikimedia.org/T329266#8605965. Thanks. [19:06:27] 10Phabricator, 10Release-Engineering-Team, 10serviceops-collab: phd on phab2002 does not start or should be masked - https://phabricator.wikimedia.org/T329285 (10Dzahn) This is a bit strange because we did this quite some time ago. Here is the evidence for it: role/codfw/phabricator.yaml:profile::phabricat... [19:07:51] 10Phabricator, 10Release-Engineering-Team, 10serviceops-collab: phd on phab2002 does not start or should be masked - https://phabricator.wikimedia.org/T329285 (10Dzahn) We have the keys in Hiera but no code uses "phd_service_ensure" anymore. It must have been removed / broken in some code refactoring. [19:13:47] 10Phabricator, 10Release-Engineering-Team, 10serviceops-collab: phd on phab2002 does not start or should be masked - https://phabricator.wikimedia.org/T329285 (10Dzahn) [phab2002:~] $ systemctl status phd ` Active: inactive (dead) (Result: exit-code) since Thu 2023-02-09 13:27:02 UTC; 1 day 5h ago Pro... [19:19:46] bd808: i think that's https://docs.gitlab.com/ee/ci/pipelines/merge_trains.html in GitLab which is *sigh* a premium feature [19:20:06] as is https://docs.gitlab.com/ee/ci/pipelines/merged_results_pipelines.html which seems like part of merge trains [19:28:27] 10Phabricator, 10Release-Engineering-Team, 10serviceops-collab: phd on phab2002 does not start or should be masked - https://phabricator.wikimedia.org/T329285 (10Dzahn) How did you find out phd refuses to start? Was it maybe manually started/restarted as part of deploying the change? Was there a scap deploy... [19:33:03] (03CR) 10Bartosz Dziewoński: [C: 03+1] "(I can't approve in this repository)" [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/887808 (owner: 10Subramanya Sastry) [19:33:16] sbassett: looking [19:38:50] (03CR) 10Subramanya Sastry: [C: 03+2] "I am going to merge this. This is not production code and is just a copy of CSS generated from the other script." [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/887808 (owner: 10Subramanya Sastry) [19:39:28] (03Merged) 10jenkins-bot: Update CSS to reflect bug fixes in the Cite CSS generating script [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/887808 (owner: 10Subramanya Sastry) [20:27:46] dduvall: ugh. they really do love to put any remotely useful feature behind the paywall. [20:28:01] they do [20:28:53] want to start a chaos club explicitly focused on implementing pay features of open core projects as actual FLOSS? [20:30:02] I would be down with that, but first we would have to upstream a modular design to replace their near-zero extension point RoR mess :/ [21:30:39] 10Continuous-Integration-Config, 10MediaWiki-Codesniffer, 10MediaWiki-Core-Tests: Consider enforcing usage of MediaWiki(Unit|Integration)TestCase in MW PHPUnit Test - https://phabricator.wikimedia.org/T319536 (10Umherirrender) [21:49:25] (03CR) 10Jforrester: [C: 03+2] Add CI to homer/deploy [integration/config] - 10https://gerrit.wikimedia.org/r/888171 (https://phabricator.wikimedia.org/T277440) (owner: 10Ayounsi) [21:49:40] (03PS2) 10Jforrester: Zuul: [operations/software/homer/deploy] Add CI [integration/config] - 10https://gerrit.wikimedia.org/r/888171 (https://phabricator.wikimedia.org/T277440) (owner: 10Ayounsi) [21:49:43] (03CR) 10Jforrester: "…" [integration/config] - 10https://gerrit.wikimedia.org/r/888171 (https://phabricator.wikimedia.org/T277440) (owner: 10Ayounsi) [21:49:46] (03PS1) 10Dduvall: jjb: Avoid "detected dubious ownership" git failure in mediawiki-i18n-check-docker [integration/config] - 10https://gerrit.wikimedia.org/r/888270 (https://phabricator.wikimedia.org/T329266) [21:50:55] (03Merged) 10jenkins-bot: Zuul: [operations/software/homer/deploy] Add CI [integration/config] - 10https://gerrit.wikimedia.org/r/888171 (https://phabricator.wikimedia.org/T277440) (owner: 10Ayounsi) [21:51:02] (03CR) 10CI reject: [V: 04-1] jjb: Avoid "detected dubious ownership" git failure in mediawiki-i18n-check-docker [integration/config] - 10https://gerrit.wikimedia.org/r/888270 (https://phabricator.wikimedia.org/T329266) (owner: 10Dduvall) [21:52:16] (03PS2) 10Dduvall: jjb: Avoid "dubious ownership" git failure in mediawiki-i18n-check-docker [integration/config] - 10https://gerrit.wikimedia.org/r/888270 (https://phabricator.wikimedia.org/T329266) [21:52:25] !log Zuul: [operations/software/homer/deploy] Add CI for T277440 [21:52:27] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:52:27] T277440: Add CI to homer-deploy repo - https://phabricator.wikimedia.org/T277440 [22:03:20] (03CR) 10Dduvall: [C: 03+2] "Note this is already deployed (see task), so I'm going to self merge." [integration/config] - 10https://gerrit.wikimedia.org/r/888270 (https://phabricator.wikimedia.org/T329266) (owner: 10Dduvall) [22:04:27] (03Merged) 10jenkins-bot: jjb: Avoid "dubious ownership" git failure in mediawiki-i18n-check-docker [integration/config] - 10https://gerrit.wikimedia.org/r/888270 (https://phabricator.wikimedia.org/T329266) (owner: 10Dduvall) [22:09:34] 10Phabricator, 10Release-Engineering-Team, 10serviceops-collab, 10Patch-For-Review: phd on phab2002 does not start or should be masked - https://phabricator.wikimedia.org/T329285 (10Dzahn) @hashar While I could not really explain why it was ever started (current theory is it wasn't but "status" remembered...