[00:05:01] 10Phabricator, 10Release-Engineering-Team (The Decommission Mission 💀), 10serviceops, 10serviceops-collab: Email tool maintainers about git-ssh deprecation on phabricator - https://phabricator.wikimedia.org/T313359 (10Dzahn) https://etherpad.wikimedia.org/p/1K-YSmk7PCIDjWlEQfMM [00:06:43] brennen: ACK, thank you. it's no big deal, just got that calendar notification [00:07:21] there are 445 striker repos, name starting with tool- [00:07:28] but 225 of them have 0 commits [00:07:50] I listed those and if we can just delete those there are 200 left [00:08:19] and if we crowd source it just a little bit..then we can just click through in the web UI [00:08:22] to get the owners [00:08:46] I also tried direct mysql but that even goes across multiple databases..not just tables [00:08:57] because phab has like 60 DBs [00:09:09] https://phabricator.wikimedia.org/P31558 [00:09:22] https://etherpad.wikimedia.org/p/1K-YSmk7PCIDjWlEQfMM [00:09:45] https://phabricator.wikimedia.org/T313359#8092964 [00:09:46] laters [05:00:06] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 10MediaWiki-SettingsBuilder, 10ci-test-error: beta-update-databases-eqiad failing due to invalid MediaWiki configuration parameters - https://phabricator.wikimedia.org/T313128 (10tstarling) I think --skip-config-validation should be us... [08:16:47] 10Phabricator, 10Release-Engineering-Team (The Decommission Mission 💀), 10serviceops, 10serviceops-collab: Email tool maintainers about git-ssh deprecation on phabricator - https://phabricator.wikimedia.org/T313359 (10hashar) Phabricator keeps an history of push actions which can be seen at https://phabric... [08:21:07] 10GitLab (Project Migration), 10Release-Engineering-Team: Create new GitLab project group: structured-data - https://phabricator.wikimedia.org/T312037 (10mfossati) Thanks a lot, @brennen ! [08:21:12] hashar: I can try and generate a list of people wish push rights this morning if needed [08:21:18] from the api [08:25:14] RhinosF1: I guess it can help yes ;) [08:25:29] I went to look solely at people using ssh for pushes which is a rather small set [08:35:18] hashar: I have a fairly empty few days so an excuse to get python out and mess around is good [08:58:44] hashar: conduit api is broken [09:21:46] hashar: phab has lost me [09:22:10] I do exactly what it does in the conduit web interface in python and it fails [09:25:55] now i got it [09:51:57] hashar: running now [09:52:49] 10Release-Engineering-Team, 10Scap: Scap pool counter error while backporting - https://phabricator.wikimedia.org/T310835 (10Joe) >>! In T310835#8085147, @dancy wrote: > I looked through operations/puppet/modules/conftool/files/safe-service-restart.py and I don't see any code for falling back to an ungraceful... [09:53:51] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10Patch-For-Review, 10User-brennen: Authenticate trusted runners for registry access against GitLab using temporary JSON Web Token - https://phabricator.wikimedia.org/T308501 (10Joe) I'll report here the results of a meeting I had w... [09:58:25] 10GitLab (Project Migration), 10serviceops: import subversion repos from Phabricator into Gitlab - https://phabricator.wikimedia.org/T308061 (10Aklapper) > I would rather keep the old subversion repos hosted on Phab I would not. I'd personally like to see repo hosting in Phab to be shut down. It's hard to writ... [09:59:57] 10Phabricator, 10Release-Engineering-Team (The Decommission Mission 💀), 10serviceops, 10serviceops-collab: Email tool maintainers about git-ssh deprecation on phabricator - https://phabricator.wikimedia.org/T313359 (10RhinosF1) I have quickly made a script to generate the list of all for the etherpad. It's... [10:09:01] 10Phabricator, 10Release-Engineering-Team (The Decommission Mission 💀), 10serviceops, 10serviceops-collab: Email tool maintainers about git-ssh deprecation on phabricator - https://phabricator.wikimedia.org/T313359 (10hashar) rPHTO is for Phabricator https://phabricator.wikimedia.org/diffusion/PHTO/manage/... [10:18:09] hashar: that's not a tool- anyway so it shouldn't actually be in there [10:18:15] phab search issue [10:21:38] hashar: okay, script working now, updating etherpad in a minute [10:22:00] just piping it to a file so i can copy easier than VSC [10:31:00] 10Phabricator, 10Release-Engineering-Team (The Decommission Mission 💀), 10serviceops, 10serviceops-collab: Email tool maintainers about git-ssh deprecation on phabricator - https://phabricator.wikimedia.org/T313359 (10RhinosF1) Etherpad is updated. Please also see {Tx} [10:33:08] 10Phabricator, 10Release-Engineering-Team (The Decommission Mission 💀), 10serviceops, 10serviceops-collab: Email tool maintainers about git-ssh deprecation on phabricator - https://phabricator.wikimedia.org/T313359 (10RhinosF1) [10:33:57] hashar: can you do https://phabricator.wikimedia.org/T313487 [12:40:44] 10Project-Admins, 10Product-Analytics: Create #pageviews-anomaly tag - https://phabricator.wikimedia.org/T214721 (10Aklapper) The issue with this tag is that there is not always some followup. While people file tickets and probably assuming that someone will look at their tickets. Tickets should probably end u... [13:08:17] A few days ago the CampaignEvents extension was enabled on meta, and it adds a new "Event:" namespace. Someone pointed out that the search does not return any results for that namespace, compare https://meta.wikimedia.beta.wmflabs.org/wiki/Special:AllPages?from=&to=&namespace=1728 and https://meta.wikimedia.beta.wmflabs.org/w/index.php?search=pizza&title=Special:Search&profile=advanced&fulltext=1&ns1728=1 . This does not happen [13:08:17] locally or on patchdemo. Is there something going on with beta? [13:09:56] Daimona: is always broken [13:10:27] Cool! Is there a task about it? [13:11:22] How do you file a task for a genericly always broken thing? :P [13:13:06] phabricator.wikimedia.org -> Create task -> Enter "BeTa IS bRoKen!!11!" -> save -> let it rot :D [13:13:15] But yeah, I guess it doesn't really matter [13:13:37] Just good to know that it's normal, thanks :) [13:14:56] Daimona: beta is very good at not working [15:05:04] Daimona: I think elasticsearch isn't working on beta (that's one issue at least) [15:06:11] dwalden Have you verified broken Elasticsearch on beta? Happy to help get it working if so (I'm an SRE on the Search team) [15:06:52] inflatador: Just looking on logstash now, and we see lots of errors: [15:06:54] "Unexpected connection error communicating with Elasticsearch" [15:07:02] So I am assuming it is [15:07:37] dwalden ACK, taking a look at the beta ES cluster now [15:07:54] Thank you [15:08:35] Ah, that would explain it! Thanks :) [15:09:25] * Krinkle is in train triage meeting [16:02:16] dwalden created a task to investigate beta cluster here: https://phabricator.wikimedia.org/T313521 [16:03:37] 10Beta-Cluster-Infrastructure, 10Discovery-Search: Verify (and/or fix) Elasticsearch beta cluster problems - https://phabricator.wikimedia.org/T313521 (10RhinosF1) [16:09:51] 10Project-Admins, 10Lingua Libre: Add milestone "2022 Hackathon" - https://phabricator.wikimedia.org/T313518 (10Aklapper) 05Open→03Resolved Hi, I've created https://phabricator.wikimedia.org/project/profile/6052/ (For future reference, please tag #Project-Admins and no need to assign stuff to me - several... [16:20:06] dancy: yeah my java fun looks like `class DynamicItem does not have a @Inject annotated constructor or a no-arg constructor.` :D [16:26:39] That sounds great [16:27:08] 10Phabricator, 10Release-Engineering-Team (The Decommission Mission 💀), 10serviceops, 10serviceops-collab: Email tool maintainers about git-ssh deprecation on phabricator - https://phabricator.wikimedia.org/T313359 (10Dzahn) Thank you very much to @RhinosF1 for https://www.irccloud.com/pastebin/Rrnxda31/sc... [16:27:48] 10Phabricator, 10Release-Engineering-Team (The Decommission Mission 💀), 10serviceops, 10serviceops-collab: Email tool maintainers about git-ssh deprecation on phabricator - https://phabricator.wikimedia.org/T313359 (10Dzahn) [16:40:32] (03PS1) 10Ahmon Dancy: deploy-promote: Terminate line after jenkins has merged the patch [tools/scap] - 10https://gerrit.wikimedia.org/r/816015 [17:26:37] 10Scap, 10Performance-Team, 10serviceops: MW wmf-config tmp cache stays outdated after Scap deploy (opcache revalidation is off) - https://phabricator.wikimedia.org/T311788 (10dancy) >>! In T311788#8042839, @Krinkle wrote: > If we want to preserve the same working logic as before, we'd have to increase the t... [17:28:59] 10Scap, 10Performance-Team, 10serviceops: MW wmf-config tmp cache stays outdated after Scap deploy (opcache revalidation is off) - https://phabricator.wikimedia.org/T311788 (10Krinkle) @dancy As long as it means incurring cache misses instead of serving stale - Sounds good to me! We can afford a minute of fo... [17:39:40] 10Phabricator, 10Release-Engineering-Team (The Decommission Mission 💀), 10User-brennen: Deploy Phabricator with scap - https://phabricator.wikimedia.org/T313259 (10brennen) 05Open→03In progress Notes so far: - Deploy targets live in `scap/phabricator-targets` in the repo - `/srv/phab` is a symlink to `/... [17:45:54] 10Scap, 10Performance-Team, 10serviceops, 10Patch-For-Review: MW wmf-config tmp cache stays outdated after Scap deploy (opcache revalidation is off) - https://phabricator.wikimedia.org/T311788 (10dancy) I landed on a grace period of 3 minutes because the php-fpm-restart phase of scap currently takes about... [19:04:38] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 10MediaWiki-SettingsBuilder, 10ci-test-error: beta-update-databases-eqiad failing due to invalid MediaWiki configuration parameters - https://phabricator.wikimedia.org/T313128 (10daniel) >>! In T313128#8093164, @tstarling wrote: > I th... [19:04:50] * Krinkle is cooling the water [19:31:28] 10Release-Engineering-Team (Radar), 10MediaWiki-Debug-Logger, 10MediaWiki-General, 10SRE, and 2 others: Post-send work someimes fatals with "Errro: The UdpSocket to 127.0.0.1:10514 has been closed" (esp mwdebug hosts) - https://phabricator.wikimedia.org/T214734 (10Krinkle) [19:31:36] 10Release-Engineering-Team (Radar), 10MediaWiki-Debug-Logger, 10MediaWiki-General, 10SRE, and 2 others: Post-send work sometimes fatals with "Errro: The UdpSocket to 127.0.0.1:10514 has been closed" (esp mwdebug hosts) - https://phabricator.wikimedia.org/T214734 (10Krinkle) [19:31:42] 10Release-Engineering-Team (Radar), 10MediaWiki-Debug-Logger, 10MediaWiki-General, 10SRE, and 2 others: Post-send work sometimes fatals at "Error: The UdpSocket to 127.0.0.1:10514 has been closed" (esp. mwdebug hosts) - https://phabricator.wikimedia.org/T214734 (10Krinkle) [19:31:50] 10Release-Engineering-Team (Radar), 10MediaWiki-Debug-Logger, 10MediaWiki-General, 10SRE, and 2 others: Post-send work sometimes fatals at "Error: The UdpSocket to 127.0.0.1:10514 has been closed" (esp. mwdebug hosts) - https://phabricator.wikimedia.org/T214734 (10Krinkle) [19:41:29] hmm, so `scap deploy` fails to phab2001 with a "Too many authentication failures" [19:42:29] i'm seeing it tries a /etc/keyholder.d/phabricator, but guessing maybe that's not authorized on phab2001... [19:56:26] * dancy takes a look [19:56:51] hm. I can't log into that host. [19:57:12] guessing you're not in phabricator-roots, which we probably ought to fix. [19:59:22] confirmed not in there. I'll make a commit. [20:03:29] ::thumbs up emoji:: [20:06:02] 10Release-Engineering-Team, 10SRE-Access-Requests: Add dancy to phabricator-roots - https://phabricator.wikimedia.org/T313551 (10dancy) [20:08:24] 10Release-Engineering-Team (The Decommission Mission 💀), 10SRE-Access-Requests, 10Patch-For-Review: Add dancy to phabricator-roots - https://phabricator.wikimedia.org/T313551 (10dancy) [20:08:49] 10Release-Engineering-Team (Radar), 10MediaWiki-Debug-Logger, 10MediaWiki-General, 10SRE, and 2 others: Post-send work sometimes fatals at "Error: The UdpSocket to 127.0.0.1:10514 has been closed" (esp. mwdebug hosts) - https://phabricator.wikimedia.org/T214734 (10Krinkle) >>! In T214734#7838777, @Krinkle... [20:10:10] 10Release-Engineering-Team (The Decommission Mission 💀), 10SRE-Access-Requests, 10Patch-For-Review: Add dancy to phabricator-roots - https://phabricator.wikimedia.org/T313551 (10dancy) Noting that my manager @thcipriani is on vacation right now. [20:14:00] 10Continuous-Integration-Config, 10ProofreadPage: Enable PHP 8 CI for ProofreadPage - https://phabricator.wikimedia.org/T312971 (10Reedy) Just be aware of the dependancy tree: `lang=yaml 'ProofreadPage': ['LabeledSectionTransclusion', 'Scribunto', 'VisualEditor'], ` And then we have... [20:23:57] 10Release-Engineering-Team (The Decommission Mission 💀), 10SRE-Access-Requests, 10Patch-For-Review: Add dancy to phabricator-roots - https://phabricator.wikimedia.org/T313551 (10hashar) [20:24:30] 10Continuous-Integration-Config, 10PostgreSQL, 10SQLite: Make postgres/sqlite CI jobs voting in gate-submit for wmf-deployed extensions with abstract schema files - https://phabricator.wikimedia.org/T313138 (10Umherirrender) Getting the Flow tests working with postgres is a big challenge, beginning with sche... [20:35:07] 10Release-Engineering-Team (The Decommission Mission 💀), 10SRE, 10SRE-Access-Requests, 10Patch-For-Review: Add dancy to phabricator-roots - https://phabricator.wikimedia.org/T313551 (10brennen) @kchapman could you sign off as skip-level manager? [20:46:59] (03PS1) 10Ahmon Dancy: scap backport: Handle gerrit_push_user properly [tools/scap] - 10https://gerrit.wikimedia.org/r/816041 [20:48:57] (03PS2) 10Ahmon Dancy: scap backport: Handle gerrit_push_user properly [tools/scap] - 10https://gerrit.wikimedia.org/r/816041 [20:52:39] 10Gerrit: Gerrit ssh replication should use the global ssh known_hosts - https://phabricator.wikimedia.org/T313553 (10hashar) [21:08:10] (03CR) 10Jeena Huneidi: "LGTM" [tools/scap] - 10https://gerrit.wikimedia.org/r/816041 (owner: 10Ahmon Dancy) [21:08:27] (03CR) 10Jeena Huneidi: [C: 03+2] scap backport: Handle gerrit_push_user properly [tools/scap] - 10https://gerrit.wikimedia.org/r/816041 (owner: 10Ahmon Dancy) [21:09:17] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 10MediaWiki-SettingsBuilder, 10ci-test-error: beta-update-databases-eqiad failing due to invalid MediaWiki configuration parameters - https://phabricator.wikimedia.org/T313128 (10RhinosF1) We spoke about adding it config CI and @Jdforr... [21:23:06] (03CR) 10Hashar: Send events to Wikimedia EventGate (031 comment) [software/gerrit/plugins/events-wikimedia] - 10https://gerrit.wikimedia.org/r/814807 (owner: 10Hashar) [21:26:21] (03Merged) 10jenkins-bot: scap backport: Handle gerrit_push_user properly [tools/scap] - 10https://gerrit.wikimedia.org/r/816041 (owner: 10Ahmon Dancy) [21:55:45] !log Upgrading scap to 4.11.2-1+0~20220720160115.349~1.gbpd4a6cb in beta cluster [21:55:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:56:45] (03PS1) 10Ahmon Dancy: Release 4.11.3-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/816050 [21:56:47] (03CR) 10Ahmon Dancy: [C: 03+2] Release 4.11.3-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/816050 (owner: 10Ahmon Dancy) [22:01:25] (03Merged) 10jenkins-bot: Release 4.11.3-1 [tools/scap] - 10https://gerrit.wikimedia.org/r/816050 (owner: 10Ahmon Dancy) [22:07:30] 10Scap, 10Performance-Team, 10serviceops: MW wmf-config tmp cache stays outdated after Scap deploy (opcache revalidation is off) - https://phabricator.wikimedia.org/T311788 (10dancy) 05Open→03Resolved [22:07:36] 10Release-Engineering-Team (Radar), 10Scap, 10Patch-For-Review, 10User-jijiki: Update Scap to perform rolling restart for all MW deploy - https://phabricator.wikimedia.org/T266055 (10dancy) [22:21:20] 10Continuous-Integration-Config, 10PHP 8.0 support: Make PHP 8.0 voting on MW master - https://phabricator.wikimedia.org/T300463 (10Bawolff) (perhaps a stupid question): I see that mediawiki-quibble-composer-mysql is passing ( https://integration.wikimedia.org/ci/job/mediawiki-quibble-composer-mysql-php80-dock... [22:40:48] 10GitLab (Project Migration), 10serviceops: import subversion repos from Phabricator into Gitlab - https://phabricator.wikimedia.org/T308061 (10demon) >>! In T308061#8092652, @hashar wrote: > This way they can still be easily be browsed But are they being actively browsed beyond a stray link to them? Or is th... [22:45:35] 10Continuous-Integration-Config, 10PHP 8.0 support: Make PHP 8.0 voting on MW master - https://phabricator.wikimedia.org/T300463 (10Reedy) >>! In T300463#8096279, @Bawolff wrote: > (perhaps a stupid question): I see that mediawiki-quibble-composer-mysql is passing ( https://integration.wikimedia.org/ci/job/med... [22:55:41] 10Continuous-Integration-Config, 10PHP 8.0 support: Make PHP 8.0 voting on MW master - https://phabricator.wikimedia.org/T300463 (10Reedy) Looks like `mediawiki-quibble-composertest-php80-docker` passes on release branches (because of stripped vendor). I think promoting that at the same time (for release bran... [22:56:57] 10Continuous-Integration-Config, 10PHP 8.0 support: Make PHP 8.0 voting on MW master - https://phabricator.wikimedia.org/T300463 (10Reedy) >>! In T300463#8096390, @Reedy wrote: > Looks like `mediawiki-quibble-composertest-php80-docker` passes on release branches (because of stripped vendor). > > I think promo... [22:57:13] 10Continuous-Integration-Config, 10MediaWiki-extensions-OAuth, 10PHP 8.0 support: Bump lcobucci/jwt & league/uri-components for php8 - https://phabricator.wikimedia.org/T313563 (10Bawolff) [22:57:36] 10Continuous-Integration-Config, 10PHP 8.0 support: Make PHP 8.0 voting on MW master - https://phabricator.wikimedia.org/T300463 (10Reedy) 05Stalled→03Open [23:01:44] 10GitLab (Project Migration), 10serviceops: import subversion repos from Phabricator into Gitlab - https://phabricator.wikimedia.org/T308061 (10bd808) For what it is worth, {rTSVN} is ancient now (read-only since 2014) and archived at https://archive.org/details/toolserver-svn. I don't think we need it anywher... [23:08:51] 10GitLab (Project Migration), 10serviceops: import subversion repos from Phabricator into Gitlab - https://phabricator.wikimedia.org/T308061 (10bd808) >>! In T308061#8096422, @bd808 wrote: > I think we can update the mw.o template to point to pages in https://static-codereview.wikimedia.org/MediaWiki/ instead... [23:50:15] 10Continuous-Integration-Config, 10PHP 8.0 support: Make PHP 8.0 voting on MW master - https://phabricator.wikimedia.org/T300463 (10Bawolff) I guess we could have a file in vendor.git with overrides for php 8, and if quibble detects its on php8, it runs `composer require $LISTOFOVERRIDES`