[00:31:03] (03PS16) 10Cwhite: Add document data model and support ECS [releng/phatality] - 10https://gerrit.wikimedia.org/r/832004 (https://phabricator.wikimedia.org/T314098) [06:38:15] 10Phabricator Antivandalism Extension, 10DBA, 10Patch-For-Review: Switchover m3 master db1164 -> db1159 - https://phabricator.wikimedia.org/T329141 (10Marostegui) [06:41:15] 10Phabricator Antivandalism Extension, 10DBA, 10Patch-For-Review: Switchover m3 master db1164 -> db1159 - https://phabricator.wikimedia.org/T329141 (10Marostegui) [06:43:28] 10Phabricator Antivandalism Extension, 10DBA, 10Patch-For-Review: Switchover m3 master db1164 -> db1159 - https://phabricator.wikimedia.org/T329141 (10Marostegui) [07:01:21] 10Phabricator Antivandalism Extension, 10DBA, 10Patch-For-Review: Switchover m3 master db1164 -> db1159 - https://phabricator.wikimedia.org/T329141 (10Marostegui) test [07:01:57] 10Phabricator Antivandalism Extension, 10DBA, 10Patch-For-Review: Switchover m3 master db1164 -> db1159 - https://phabricator.wikimedia.org/T329141 (10Marostegui) [07:02:35] 10Phabricator Antivandalism Extension, 10DBA, 10Patch-For-Review: Switchover m3 master db1164 -> db1159 - https://phabricator.wikimedia.org/T329141 (10Marostegui) 05Open→03Resolved This was done [07:03:01] 10Phabricator Antivandalism Extension, 10DBA, 10Patch-For-Review: Switchover m3 master db1164 -> db1159 - https://phabricator.wikimedia.org/T329141 (10Marostegui) [07:59:04] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10MediaWiki-Installer, 10Wikimedia-production-error (ARCHIVED -- Shared Build Failure): mwext-php72-phan-docker job fails on WikimediaMessages: A dependency error was encountered while installing... - https://phabricator.wikimedia.org/T232413 [07:59:55] 10Phabricator: Prevent adding new tasks in #Cloud-Services umbrella tag - https://phabricator.wikimedia.org/T329027 (10Aklapper) The only two ideas I have are either archiving #Cloud-Services (which will make it not less likely but not impossible to appear as a autocomplete proposal, but it feels wrong to disabl... [08:48:18] 10Phabricator Antivandalism Extension, 10DBA: Switchover m3 master db1164 -> db1159 - https://phabricator.wikimedia.org/T329141 (10Marostegui) [08:48:54] 10Phabricator, 10DBA: Switchover m3 master db1164 -> db1159 - https://phabricator.wikimedia.org/T329141 (10Marostegui) [09:30:54] 10Phabricator: Rename Phabricator account "Ddwaal-WMF" to "Ddwaal". - https://phabricator.wikimedia.org/T326329 (10Aklapper) 05Open→03Resolved a:03Aklapper Sorry, I somehow missed this ticket. Done now. [09:33:15] \o [09:33:41] I had to remove the Depends-On in https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/887848/ for the patch to work with scap backport, because it complained about duplicate change-ids. Is there a better way? [09:55:47] kostajh: there is a task about it (I have seen it yesterday) https://phabricator.wikimedia.org/T323277 [09:55:54] it might have an answer/workaround/solution [09:56:46] aha [09:56:59] thx [10:41:34] kostajh: I think the issue is one sends a change to the master branch with `Change-Id: C`, it is then cherry-picked to the wmf branch and that cherry pick has the same `Change-Id: C` [10:42:16] when querying for cross repositories dependencies for that Change-Id: C, the end points finds two change for that id (the one from master, the other that got cherry picked) and bails out with an error [10:42:26] scap backport doesn't know what to do with it and gives up [10:43:27] the lookup should be done against the change number rather than the change id [10:43:42] which probably leads to some other side effects :-] [10:55:23] 10Phabricator, 10Trust-and-Safety, 10cloud-services-team, 10wikitech.wikimedia.org: Reset 2FA for Developer account 'Rosalie Perside (WMDE)' and Phabricator account @Rosalie_WMDE - https://phabricator.wikimedia.org/T329179 (10Aklapper) a:05Rosalie_WMDE→03None [10:55:30] 10Phabricator, 10Trust-and-Safety, 10cloud-services-team, 10wikitech.wikimedia.org: Reset 2FA for Developer account 'Rosalie Perside (WMDE)' and Phabricator account @Rosalie_WMDE - https://phabricator.wikimedia.org/T329179 (10Aklapper) Thanks. Stripped in Phabricator: ` aklapper@phab1004:~$ sudo /srv/phab/... [10:55:43] 10GitLab (CI & Job Runners), 10serviceops-collab, 10Patch-For-Review: Use dedicated volume for /var/lib/docker on Trusted Runners - https://phabricator.wikimedia.org/T329035 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by eoghan@cumin1001 for host gitlab-runner1003.eqiad.wmnet with... [10:57:16] (03CR) 10Hashar: [C: 03+2] Zuul: [mediawiki/extensions/Kartographer] Enable Sonar Codehealth Checks [integration/config] - 10https://gerrit.wikimedia.org/r/887296 (https://phabricator.wikimedia.org/T321837) (owner: 10Pwangai) [10:57:21] (03CR) 10Hashar: [C: 03+2] Zuul: [mediawiki/extensions/VisualEditor] Enable Sonar Codehealth Checks [integration/config] - 10https://gerrit.wikimedia.org/r/886921 (https://phabricator.wikimedia.org/T321837) (owner: 10Pwangai) [10:58:36] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/Kartographer] Enable Sonar Codehealth Checks [integration/config] - 10https://gerrit.wikimedia.org/r/887296 (https://phabricator.wikimedia.org/T321837) (owner: 10Pwangai) [10:58:44] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/VisualEditor] Enable Sonar Codehealth Checks [integration/config] - 10https://gerrit.wikimedia.org/r/886921 (https://phabricator.wikimedia.org/T321837) (owner: 10Pwangai) [11:03:34] (03CR) 10Hashar: [C: 03+2] "deployed!" [integration/config] - 10https://gerrit.wikimedia.org/r/887296 (https://phabricator.wikimedia.org/T321837) (owner: 10Pwangai) [11:03:36] (03CR) 10Hashar: [C: 03+2] "deployed!" [integration/config] - 10https://gerrit.wikimedia.org/r/886921 (https://phabricator.wikimedia.org/T321837) (owner: 10Pwangai) [11:10:54] Hi! Is there anything I can do to get the bullseye tox CI images unblocked? (https://gerrit.wikimedia.org/r/c/integration/config/+/868624), we are starting to have to pin versions of things specially to allow running on buster ones [11:31:13] 10GitLab (CI & Job Runners), 10serviceops-collab: Use dedicated volume for /var/lib/docker on Trusted Runners - https://phabricator.wikimedia.org/T329035 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by eoghan@cumin1001 for host gitlab-runner1003.eqiad.wmnet with OS bullseye completed: -... [12:01:58] I don't know how to add reviewers to this, https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/11 [12:11:01] 10Continuous-Integration-Config, 10Growth-Team, 10GrowthExperiments, 10ci-test-error: phpbench job for GrowthExperiments fails due to missing dependency - https://phabricator.wikimedia.org/T329280 (10kostajh) [12:11:27] 10Continuous-Integration-Config, 10Growth-Team, 10GrowthExperiments, 10ci-test-error: phpbench job for GrowthExperiments fails due to missing dependency - https://phabricator.wikimedia.org/T329280 (10kostajh) [12:20:06] (03CR) 10Hashar: [C: 04-1] "I don't think it makes much sense to create templates for MediaWiki services. I am going to treat mediawiki/services/parsoid as a library " [integration/config] - 10https://gerrit.wikimedia.org/r/884843 (https://phabricator.wikimedia.org/T321837) (owner: 10Hashar) [13:12:33] 10GitLab (CI & Job Runners), 10Release-Engineering-Team, 10mwbot-rs, 10mwcli: GitLab CI jobs failing with "You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" - https://phabricator.wikimedia.org/T329216 (10Addshore) [13:13:55] 10GitLab (CI & Job Runners), 10Release-Engineering-Team, 10mwbot-rs, 10mwcli: GitLab CI jobs failing with "You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" - https://phabricator.wikimedia.org/T329216 (10Addshore)... [13:24:41] 10GitLab (CI & Job Runners), 10Release-Engineering-Team, 10mwbot-rs, 10mwcli: GitLab CI jobs failing with "You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" - https://phabricator.wikimedia.org/T329216 (10Addshore)... [13:33:28] 10Phabricator, 10Release-Engineering-Team, 10serviceops-collab: phd on phab2002 does not start or should be masked - https://phabricator.wikimedia.org/T329285 (10hashar) [13:34:06] 10Phabricator, 10serviceops-collab: Phabricator phd.service refuses to start on machine boot - https://phabricator.wikimedia.org/T326146 (10hashar) 05Open→03Resolved a:03hashar The series of patch proved to fix the issue on devtools. I have applied the patch with @jbond on `phab1004.eqiad.wmnet`. Next ti... [13:35:06] (03PS1) 10Hashar: Add patch coverage job for parsoid [integration/config] - 10https://gerrit.wikimedia.org/r/888001 [13:38:48] Amir1: I added d.ancy and then b.rennen before remembering that GitLab only lets you specify one. [13:39:04] :((( [13:39:14] (Sorry for the noise, both.) [13:41:45] 10GitLab (CI & Job Runners), 10Release-Engineering-Team, 10mwbot-rs, 10mwcli: GitLab CI jobs failing with "You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" - https://phabricator.wikimedia.org/T329216 (10Addshore)... [13:50:15] hi - is CI having issues? I get error: RPC failed; HTTP 500 curl 22 The requested URL returned error: 500 Server Error repeatedly @ pywikibot repo [13:53:39] maybe it's because of the subproject commit update to a non-existing one: https://gerrit.wikimedia.org/r/q/bf6405318e10a38c8972d366d330762f74aec553 [14:05:41] 10GitLab (CI & Job Runners), 10serviceops-collab, 10Patch-For-Review: Use dedicated volume for /var/lib/docker on Trusted Runners - https://phabricator.wikimedia.org/T329035 (10eoghan) [14:07:54] 10GitLab (CI & Job Runners), 10observability, 10serviceops-collab: node exporter for gitlab-runner hosts is missing metrics for /var/lib/docker - https://phabricator.wikimedia.org/T329286 (10Jelto) [14:18:40] (03CR) 10Hashar: [C: 03+2] "Lets try, I have poked #content-transformers team about it" [integration/config] - 10https://gerrit.wikimedia.org/r/888001 (owner: 10Hashar) [14:19:55] (03Merged) 10jenkins-bot: Add patch coverage job for parsoid [integration/config] - 10https://gerrit.wikimedia.org/r/888001 (owner: 10Hashar) [14:31:30] (03CR) 10Hashar: [C: 03+2] "Tested on https://gerrit.wikimedia.org/r/c/mediawiki/services/parsoid/+/888015 ;)" [integration/config] - 10https://gerrit.wikimedia.org/r/888001 (owner: 10Hashar) [14:35:41] ma: that is probably an issue with the proposed change? [14:36:03] hashar: I think it's CI not finding that commit update? [14:36:09] you can always file a task about it and mark it with #ci-test-error [14:36:32] no idea really, it depends on the change that is causing the issue [14:36:47] hashar: I though I pasted the link: https://gerrit.wikimedia.org/r/c/pywikibot/core/+/887948 [14:36:51] Pardonez moi [14:36:53] maybe someone did a local submodule update and did a `git add` of a local change [14:37:16] the commit do not exist so logically CI can't find it [14:37:37] I suggested them to remove the submodule update and see if CI can pass then [14:37:43] yeah https://gerrit.wikimedia.org/r/c/pywikibot/core/+/887948/1/scripts/i18n [14:37:45] pwb i18n has its own repo [14:37:54] so not sure why they added that update there [14:38:09] and that commit is not in pywikibot/i18n repo as far as I can tell [14:38:21] I would guess the person did a `git commit -a` [14:38:40] while they have some local commit in their local `./scripts/i18n` [14:39:05] so that bf6405318e10a38c8972d366d330762f74aec553 sha1 points to something which is only available to the patch uploader [14:39:32] yep [14:39:37] I told 'em on the patch [14:41:09] +1 [14:42:39] Merci bien, veuillez agreér etc :) [14:44:13] ahah [14:44:14] merci ma! [14:52:15] (03PS1) 10Kosta Harlan: zuul: Add GrowthExperiments to extension-javascript-documentation [integration/config] - 10https://gerrit.wikimedia.org/r/888022 (https://phabricator.wikimedia.org/T329034) [14:54:00] (03CR) 10Kosta Harlan: "Hi Antoine," [integration/config] - 10https://gerrit.wikimedia.org/r/888022 (https://phabricator.wikimedia.org/T329034) (owner: 10Kosta Harlan) [15:18:12] (03CR) 10Hashar: zuul: Add GrowthExperiments to extension-javascript-documentation (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/888022 (https://phabricator.wikimedia.org/T329034) (owner: 10Kosta Harlan) [15:23:59] o/ is it safe to delete revisions in the scap project-name-cache/revs folder? (I'm running out of space on a server) [15:44:45] dcausse: there should be a symlink to one of those revs for the currentlydeployed revision [15:45:09] the rest are copy of the previous deployments to ease rollback (we can thus just change the symlink from the currently deployed revision to the new/old revision [15:45:14] so yeah I think it is safe [15:45:47] 10Phabricator, 10Release-Engineering-Team: Move phabricator/deployments repository from Differential to Gerrit - https://phabricator.wikimedia.org/T321380 (10Aklapper) > | https://phabricator.wikimedia.org/diffusion/PHEX/phab-extensions.git | https://gerrit.wikimedia.org/r/phabricator/extensions | Note that th... [15:46:25] hashar: thanks! [15:47:33] it should keep up to 5 revisions by default, can be tuned in scap.cfg by setting `cache_revs = 3` for example [15:48:04] so tentatively you can change that setting, do a deploy and scap will happilly delete old revisions when it finalizes the deployment [15:49:49] ok will do that but I need to clean this up first, the deploy fails on out of space :/ [16:02:49] 10GitLab (CI & Job Runners), 10serviceops-collab, 10Patch-For-Review: Use dedicated volume for /var/lib/docker on Trusted Runners - https://phabricator.wikimedia.org/T329035 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by eoghan@cumin1001 for host gitlab-runner1004.eqiad.wmnet with... [16:36:39] taavi: re wikifunctions.beta.wmflabs.org alert in -operations ... could be feasible that wikifunctions.beta.wmflabs.org is missing from the unified cert issued by deployment-acme-chief-03? [16:45:52] hmmmm nope [16:46:17] it's there and the cert is renewed as expected [16:46:20] root@deployment-acme-chief03:/var/lib/acme-chief/certs/unified/live# openssl x509 -dates -noout -in ec-prime256v1.crt [16:46:20] notBefore=Jan 17 11:47:43 2023 GMT [16:46:20] notAfter=Apr 17 11:47:42 2023 GMT [16:46:49] vgutierrez: Do we need to restart the docker service or something to pick it up, perhaps? [16:47:07] deployment-cache-text07 is a text HAProxy/Varnish/ATS cache server (cache::text) [16:47:07] The last Puppet run was at Thu Jan 12 19:09:03 UTC 2023 (40177 minutes ago). [16:47:15] puppet is broken in deployment-cache-text07 [16:47:53] * taavi tries to act surprised [16:47:55] Ah. [16:48:00] taavi: Indeed. :-( [16:48:33] 10GitLab (CI & Job Runners), 10serviceops-collab: Use dedicated volume for /var/lib/docker on Trusted Runners - https://phabricator.wikimedia.org/T329035 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by eoghan@cumin1001 for host gitlab-runner1004.eqiad.wmnet with OS bullseye completed: -... [16:48:42] Also we should either auto-renew more than 10 days out, or have the alert not trigger for a few days before the auto-renew happens. [16:49:58] James_F: auto renew happens 1 month before the cert expires [16:50:08] Oh, hmm, so the alerts were real. [16:50:16] yes [16:50:23] the new cert didn't reach the web servers [16:50:29] Right. [16:50:29] (cause puppet is broken there) [16:50:34] fixing it right now... [16:50:37] s/there/everywhere/ [16:51:47] fwiw, I still maintain that using production alerting infra to monitor beta is 'wrong', but I don't really have any better options available :/ [16:51:57] Yeah. :-( [16:52:24] taavi: Also mostly we're using in beta to prove that the monitoring works before copying it to point to the real service once we spin it up. [16:52:51] At which point we'll probably kill the beta monitoring, or at least scale it back. [16:56:00] $ openssl s_client -connect wikifunctions.beta.wmflabs.org:443 2>/dev/null notBefore=Jan 17 11:47:43 2023 GMT [16:56:00] notAfter=Apr 17 11:47:42 2023 GMT [16:56:10] looking good now [16:56:36] (alert resolved as well) [16:57:20] vgutierrez: Thank you! [16:57:31] I guess it'll likely break again in three months. [17:06:08] 10GitLab (CI & Job Runners), 10serviceops-collab: Use dedicated volume for /var/lib/docker on Trusted Runners - https://phabricator.wikimedia.org/T329035 (10eoghan) [17:07:47] James_F: it’s being breaking every 3 months for about 2.5 years [17:40:17] RhinosF1: :-( [17:55:15] 10GitLab (Infrastructure), 10serviceops-collab: Migrate gitlab-test instance to bullseye - https://phabricator.wikimedia.org/T318521 (10Jelto) a:05Dzahn→03Jelto I removed the `gitlab-bulleye.devtools.wmcloud.org` DNS entry which was pointing to `185.15.56.117` (and confusing me quite a bit). I re-run the... [18:07:00] (03CR) 10Ahmon Dancy: [C: 03+2] Add document data model and support ECS [releng/phatality] - 10https://gerrit.wikimedia.org/r/832004 (https://phabricator.wikimedia.org/T314098) (owner: 10Cwhite) [18:07:21] (03CR) 10Ahmon Dancy: [C: 03+2] Emphasize "too long" warning label [releng/phatality] - 10https://gerrit.wikimedia.org/r/814009 (owner: 10Krinkle) [18:07:27] (03Merged) 10jenkins-bot: Add document data model and support ECS [releng/phatality] - 10https://gerrit.wikimedia.org/r/832004 (https://phabricator.wikimedia.org/T314098) (owner: 10Cwhite) [18:07:35] (03CR) 10Ahmon Dancy: [C: 03+2] Switch kbnDocViewerTable -> osdDocViewerTable [releng/phatality] - 10https://gerrit.wikimedia.org/r/814010 (owner: 10Krinkle) [18:07:52] (03Merged) 10jenkins-bot: Emphasize "too long" warning label [releng/phatality] - 10https://gerrit.wikimedia.org/r/814009 (owner: 10Krinkle) [18:08:01] (03Merged) 10jenkins-bot: Switch kbnDocViewerTable -> osdDocViewerTable [releng/phatality] - 10https://gerrit.wikimedia.org/r/814010 (owner: 10Krinkle) [18:12:42] James_F: until you lot set up that alert and inadvertently cared, there was a huge set of tasks filed every 3 months when it expired [18:12:54] Because no one noticed it was going until it blew up [18:13:11] RhinosF1: I think I noticed through manual browsing at least once or twice. [18:13:43] You probably filed a few of the tasks in that long list then :) [18:56:36] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MediaWiki-Stakeholders-Group, 10Epic, 10MW-1.40-release: Expand the set of bundled extensions and skins in MediaWiki 1.40 - https://phabricator.wikimedia.org/T317146 (10Jdforrester-WMF) [18:56:41] 10MediaWiki-Releasing, 10MediaWiki-extensions-LoginNotify, 10MW-1.40-release, 10Patch-For-Review: Bundle LoginNotify extension with MediaWiki - https://phabricator.wikimedia.org/T191746 (10Jdforrester-WMF) [18:56:48] 10MediaWiki-Releasing, 10Growth-Team, 10Growth-Team-Filtering, 10Thanks, and 2 others: Bundle Thanks extension with MediaWiki - https://phabricator.wikimedia.org/T191739 (10Jdforrester-WMF) [18:57:02] 10MediaWiki-Releasing, 10Growth-Team, 10Growth-Team-Filtering, 10Notifications, and 2 others: Bundle Echo extension with MediaWiki - https://phabricator.wikimedia.org/T191738 (10Jdforrester-WMF) 05Open→03Resolved a:03Jdforrester-WMF [18:57:08] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MediaWiki-Stakeholders-Group, 10Epic, 10MW-1.40-release: Expand the set of bundled extensions and skins in MediaWiki 1.40 - https://phabricator.wikimedia.org/T317146 (10Jdforrester-WMF) [18:57:26] 10MediaWiki-Releasing, 10Growth-Team, 10Growth-Team-Filtering, 10Thanks, and 2 others: Bundle Thanks extension with MediaWiki - https://phabricator.wikimedia.org/T191739 (10Jdforrester-WMF) 05Open→03Resolved a:03Jdforrester-WMF [18:57:30] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MediaWiki-Stakeholders-Group, 10Epic, 10MW-1.40-release: Expand the set of bundled extensions and skins in MediaWiki 1.40 - https://phabricator.wikimedia.org/T317146 (10Jdforrester-WMF) [18:57:42] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MediaWiki-Stakeholders-Group, 10Epic, 10MW-1.40-release: Expand the set of bundled extensions and skins in MediaWiki 1.40 - https://phabricator.wikimedia.org/T317146 (10Jdforrester-WMF) [18:58:01] 10MediaWiki-Releasing, 10MediaWiki-extensions-LoginNotify, 10MW-1.40-release, 10Patch-For-Review: Bundle LoginNotify extension with MediaWiki - https://phabricator.wikimedia.org/T191746 (10Jdforrester-WMF) 05Open→03Resolved a:03Jdforrester-WMF [20:30:42] 10Project-Admins, 10Data-Engineering, 10PM: Archive Analytics tag - https://phabricator.wikimedia.org/T298671 (10Aklapper) This task got opened a year ago. @odimitrijevic, @JArguello-WMF: Could this please see decisions? * There are 218 open #Analytics tasks in https://phabricator.wikimedia.org/maniphest/qu... [22:08:50] 10Continuous-Integration-Config, 10Growth-Team, 10GrowthExperiments, 10ci-test-error: phpbench job for GrowthExperiments fails due to missing dependency - https://phabricator.wikimedia.org/T329280 (10Tgr) Possibly a duplicate of {T232413}?