[00:02:07] <wikibugs>	 (03PS2) 10Jeena Huneidi: scap backport: deploy to mwdebug [tools/scap] - 10https://gerrit.wikimedia.org/r/803370 (https://phabricator.wikimedia.org/T308476)
[00:42:54] <wikibugs>	 10Release-Engineering-Team, 10Scap: Scap's php-fpm restart step "left" counter may be counter-intuitive - https://phabricator.wikimedia.org/T310242 (10Krinkle)
[01:35:18] <wikibugs>	 (03PS1) 10Seddon: Add SearchVue repo zuul config [integration/config] - 10https://gerrit.wikimedia.org/r/804015 (https://phabricator.wikimedia.org/T309932)
[01:37:06] <wikibugs>	 (03CR) 10CI reject: [V: 04-1] Add SearchVue repo zuul config [integration/config] - 10https://gerrit.wikimedia.org/r/804015 (https://phabricator.wikimedia.org/T309932) (owner: 10Seddon)
[01:51:22] <wikibugs>	 (03PS2) 10Seddon: Add SearchVue repo zuul config [integration/config] - 10https://gerrit.wikimedia.org/r/804015 (https://phabricator.wikimedia.org/T309932)
[04:28:42] <wikibugs>	 (03CR) 10Kosta Harlan: [C: 03+1] "let’s try it!" [integration/config] - 10https://gerrit.wikimedia.org/r/803990 (owner: 10C. Scott Ananian)
[05:23:31] <icinga-wm>	 PROBLEM - Check systemd state on doc1001 is CRITICAL: CRITICAL - degraded: The following units failed: rsync-doc-doc2001.codfw.wmnet.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
[06:18:47] <icinga-wm>	 RECOVERY - Check systemd state on doc1001 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
[08:16:01] <wikibugs>	 10Project-Admins: Request for new SDAW-SearchVue Project Tag - https://phabricator.wikimedia.org/T309934 (10Peachey88) Is there any particular reason the project tag is getting called SDAW-SearchVue, Instead of SearchVue or MediaWiki-extensions-SearchVue to match the name of the extension?
[09:23:34] <wikibugs>	 10Release-Engineering-Team (Next), 10Scap, 10Patch-For-Review: Bootstrap new scap targets - https://phabricator.wikimedia.org/T309713 (10jnuche) 05In progress→03Resolved
[09:23:40] <wikibugs>	 10Release-Engineering-Team (Priority Backlog 📥), 10Scap, 10Infrastructure-Foundations, 10serviceops, 10Patch-For-Review: Use scap to deploy itself to scap targets - https://phabricator.wikimedia.org/T303559 (10jnuche)
[09:44:40] <wikibugs>	 10Release-Engineering-Team, 10serviceops, 10Patch-For-Review: PendingDeprecationWarning on update_version.py - https://phabricator.wikimedia.org/T310133 (10JMeybohm)
[09:45:45] <wikibugs>	 10Release-Engineering-Team (Radar), 10Scap, 10serviceops: Deploy Scap version 4.7.1 - https://phabricator.wikimedia.org/T306998 (10JMeybohm) 05Open→03Resolved a:03JMeybohm
[10:09:45] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Math: Math CI tests fail - https://phabricator.wikimedia.org/T310255 (10Physikerwelt)
[10:15:35] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Math, 10ci-test-error (WMF-deployed Build Failure): Math CI tests fail - https://phabricator.wikimedia.org/T310255 (10Lucas_Werkmeister_WMDE) p:05Triage→03Unbreak! This is also affecting Wikibase changes, e.g. [this build](https://integration.wikimedia.org/ci/jo...
[10:20:37] <wikibugs>	 10Project-Admins: Request for new SDAW-SearchVue Project Tag - https://phabricator.wikimedia.org/T309934 (10Aklapper) 05Open→03Resolved a:03Aklapper Requested public project #sdaw-searchvue has been created: https://phabricator.wikimedia.org/project/view/5979/  (In case you need to edit the project or proj...
[10:21:15] <wikibugs>	 10Project-Admins: Request for new SDAW-SearchVue Project Tag - https://phabricator.wikimedia.org/T309934 (10Aklapper) (MediaWiki-extensions-SearchVue is an alternative hashtag)
[10:22:07] <duesen>	 hashar: quick question: do we run api-testing for all extensions in core CI? 
[10:25:02] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Math, 10ci-test-error (WMF-deployed Build Failure): Math CI tests fail - https://phabricator.wikimedia.org/T310255 (10kostajh) >>! In T310255#7991473, @Lucas_Werkmeister_WMDE wrote: > Could be related to T90875?  It's possible, but I am not sure why it didn't fail e...
[10:25:34] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Math, 10ci-test-error (WMF-deployed Build Failure): Math CI tests fail - https://phabricator.wikimedia.org/T310255 (10kostajh)
[10:25:50] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Math, 10MediaWiki-extensions-WikibaseClient, 10ci-test-error (WMF-deployed Build Failure): Math CI tests fail - https://phabricator.wikimedia.org/T310255 (10kostajh)
[10:26:35] <kostajh>	 duesen: hash.ar is out today
[10:26:50] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Math, 10MediaWiki-extensions-WikibaseClient, 10ci-test-error (WMF-deployed Build Failure): Wikibase\Client\Tests\Unit\Api\ApiFormatReferenceTest#testInvalidReferenceJson failure - https://phabricator.wikimedia.org/T310255 (10kostajh)
[10:27:02] <kostajh>	 duesen: but the answer is "no", afaik. see also T303255
[10:27:03] <stashbot>	 T303255: Should core apitests job run extension/skin api tests as well? - https://phabricator.wikimedia.org/T303255
[10:35:13] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Math, 10MediaWiki-extensions-WikibaseClient, 10Wikidata, and 2 others: Wikibase\Client\Tests\Unit\Api\ApiFormatReferenceTest#testInvalidReferenceJson failure - https://phabricator.wikimedia.org/T310255 (10Lucas_Werkmeister_WMDE) I can’t reproduce this yet, `vendor...
[10:43:47] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Math, 10MediaWiki-extensions-CentralAuth, 10MediaWiki-extensions-WikibaseClient, and 3 others: Wikibase\Client\Tests\Unit\Api\ApiFormatReferenceTest#testInvalidReferenceJson failure - https://phabricator.wikimedia.org/T310255 (10Lucas_Werkmeister_WMDE) Okay, I can...
[10:45:27] <duesen>	 kostajh: ok thank you!
[10:53:20] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Math, 10MediaWiki-extensions-CentralAuth, 10MediaWiki-extensions-WikibaseClient, and 3 others: Wikibase\Client\Tests\Unit\Api\ApiFormatReferenceTest#testInvalidReferenceJson failure - https://phabricator.wikimedia.org/T310255 (10Lucas_Werkmeister_WMDE) Okay, Centr...
[11:01:05] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Math, 10MediaWiki-extensions-CentralAuth, 10MediaWiki-extensions-WikibaseClient, and 3 others: Wikibase\Client\Tests\Unit\Api\ApiFormatReferenceTest#testInvalidReferenceJson failure - https://phabricator.wikimedia.org/T310255 (10kostajh) >>! In T310255#7991592, @L...
[11:01:44] <wikibugs>	 10Continuous-Integration-Config, 10Quibble, 10Platform Team Initiatives (API Integration Tests): Core apitests job should run extension/skin api tests as well - https://phabricator.wikimedia.org/T303255 (10kostajh)
[11:14:37] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Math, 10MediaWiki-extensions-CentralAuth, 10MediaWiki-extensions-WikibaseClient, and 3 others: Wikibase\Client\Tests\Unit\Api\ApiFormatReferenceTest#testInvalidReferenceJson failure - https://phabricator.wikimedia.org/T310255 (10Lucas_Werkmeister_WMDE) > I think w...
[11:17:03] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Math, 10MediaWiki-extensions-CentralAuth, 10MediaWiki-extensions-WikibaseClient, and 3 others: Wikibase\Client\Tests\Unit\Api\ApiFormatReferenceTest#testInvalidReferenceJson failure - https://phabricator.wikimedia.org/T310255 (10Lucas_Werkmeister_WMDE) Prior to 72...
[11:23:49] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Math, 10MediaWiki-extensions-CentralAuth, 10MediaWiki-extensions-WikibaseClient, and 3 others: Wikibase\Client\Tests\Unit\Api\ApiFormatReferenceTest#testInvalidReferenceJson failure - https://phabricator.wikimedia.org/T310255 (10Lucas_Werkmeister_WMDE) >>! In T310...
[11:30:36] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Math, 10MediaWiki-extensions-CentralAuth, 10MediaWiki-extensions-WikibaseClient, and 3 others: Wikibase\Client\Tests\Unit\Api\ApiFormatReferenceTest#testInvalidReferenceJson failure - https://phabricator.wikimedia.org/T310255 (10Lucas_Werkmeister_WMDE) Wait, no. `...
[11:39:20] <wikibugs>	 10GitLab (Infrastructure), 10serviceops: Reduce usage of public IPv4 addresses on GitLab hosts - https://phabricator.wikimedia.org/T310265 (10Jelto)
[11:39:40] <wikibugs>	 10GitLab (Infrastructure), 10serviceops: Reduce usage of public IPv4 addresses on GitLab hosts - https://phabricator.wikimedia.org/T310265 (10Jelto) p:05Triage→03Medium
[12:26:46] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Math, 10MediaWiki-extensions-CentralAuth, 10MediaWiki-extensions-WikibaseClient, and 3 others: Wikibase\Client\Tests\Unit\Api\ApiFormatReferenceTest#testInvalidReferenceJson failure - https://phabricator.wikimedia.org/T310255 (10Nikerabbit) We also observed the Ap...
[12:28:58] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Math, 10MediaWiki-extensions-CentralAuth, 10MediaWiki-extensions-WikibaseClient, and 3 others: Wikibase\Client\Tests\Unit\Api\ApiFormatReferenceTest#testInvalidReferenceJson failure - https://phabricator.wikimedia.org/T310255 (10kostajh) >>! In T310255#7991676, @L...
[12:29:25] <kostajh>	 Lucas_WMDE: are you able to reproduce the ApiStructureTest failure locally?
[12:38:06] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Math, 10MediaWiki-extensions-CentralAuth, 10MediaWiki-extensions-WikibaseClient, and 3 others: Wikibase\Client\Tests\Unit\Api\ApiFormatReferenceTest#testInvalidReferenceJson failure - https://phabricator.wikimedia.org/T310255 (10kostajh) >>! In T310255#7991842, @k...
[12:45:09] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Math, 10MediaWiki-extensions-CentralAuth, 10MediaWiki-extensions-WikibaseClient, and 3 others: Wikibase\Client\Tests\Unit\Api\ApiFormatReferenceTest#testInvalidReferenceJson failure - https://phabricator.wikimedia.org/T310255 (10kostajh) >>! In T310255#7991664, @L...
[12:54:02] <wmf-insecte>	 Project beta-scap-sync-world build #54764: 04FAILURE in 9 min 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/54764/
[12:57:07] <wmf-insecte>	 Yippee, build fixed!
[12:57:08] <wmf-insecte>	 Project beta-scap-sync-world build #54765: 09FIXED in 1 min 8 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/54765/
[13:10:57] <Lucas_WMDE>	 kostajh: yes
[13:12:11] <kostajh>	 hmm. I can't :\
[13:12:42] <Lucas_WMDE>	 oh wait, you asked about the structure test
[13:12:46] <Lucas_WMDE>	 let me try that
[13:12:50] <Lucas_WMDE>	 I think the one I reproduced was the Wikibase one
[13:15:03] <Lucas_WMDE>	 kostajh: I can still reproduce it
[13:15:44] <Lucas_WMDE>	 with wfLoadExtension('CentralAuth') in my LocalSettings
[13:15:49] <Lucas_WMDE>	 not sure what else I have in there that would be relevant
[13:16:47] <Lucas_WMDE>	 I can still reproduce the test failure when I move the wfLoadExtension( 'CentralAuth' ) immediately below the “end of automatically generated settings” and put a `return;` after it
[13:16:57] <Lucas_WMDE>	 which should mean the rest of my LocalSettings isn’t effective, I think?
[13:19:48] <kostajh>	 hm, I have some other CentralAuth config that is probably the reason I cannot trigger it
[13:27:27] <kostajh>	 Lucas_WMDE: not sure what I'm missing. I have:
[13:27:36] <kostajh>	 https://www.irccloud.com/pastebin/MMoW1A9Q/
[13:31:49] <Lucas_WMDE>	 I can reproduce the warning with that…
[13:32:37] <kostajh>	 Lucas_WMDE: not sure if https://gerrit.wikimedia.org/r/c/mediawiki/core/+/804319 would temporarily get us past this hurdle. 
[13:32:54] <Lucas_WMDE>	 I don’t think it would
[13:33:05] <Lucas_WMDE>	 because the tables in that database still need to be created
[13:33:13] <Lucas_WMDE>	 and IIUC CentralAuth doesn’t hook that up with update.php
[13:33:19] <kostajh>	 :(
[13:33:36] <Lucas_WMDE>	 it only gets done in the unit test hook handler, or in a real install, I think the admin is supposed to do it manually
[13:40:18] <kostajh>	 Lucas_WMDE: OK, well I reproduced by dropping the `centralauth` database I have in my wiki. (duh) And it seems that the patch should work
[13:40:29] <kostajh>	 for that particular test, anyway
[13:40:35] <Lucas_WMDE>	 ah, right, that makes sense
[13:40:49] <Lucas_WMDE>	 I dropped that table from my wiki
[13:41:22] <kostajh>	 Lucas_WMDE: ugh, no, slightly different error now https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php72-noselenium-docker/157045/console
[13:41:33] <kostajh>	 `Cannot access the database: Access denied for user 'wikiuser'@'localhost' to database 'my_wiki'`
[13:41:41] <Lucas_WMDE>	 huh
[13:41:42] <kostajh>	 but locally it works
[13:44:03] <kostajh>	 I guess it should be trying to connect to `localhost:/workspace/db/quibble-mysql-5hjinx97/socket`
[13:45:41] <Lucas_WMDE>	 is that not what it’s trying to connect to?
[13:46:30] <kostajh>	 Lucas_WMDE: from the error in https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php72-noselenium-docker/157045/console it looks like it is trying to connect to the default values for db name, host, and server, rather than what is defined by the installer in LocalSettings.php 
[13:47:24] <kostajh>	 Krinkle / James_F / Daimona if you have any ideas on T310255 please let us know, otherwise I think we need to revert https://gerrit.wikimedia.org/r/c/mediawiki/core/+/741970
[13:47:28] <stashbot>	 T310255: Wikibase\Client\Tests\Unit\Api\ApiFormatReferenceTest#testInvalidReferenceJson failure - https://phabricator.wikimedia.org/T310255
[13:47:42] <kostajh>	 Lucas_WMDE: maybe we need https://gerrit.wikimedia.org/r/c/mediawiki/core/+/802597 ?
[13:47:44] <Lucas_WMDE>	 I’m not sure which error you’re referring to
[13:47:47] <Lucas_WMDE>	 I see the socket path you mentioned:
[13:47:50] <Lucas_WMDE>	 Wikimedia\Rdbms\DBConnectionError: Cannot access the database: Access denied for user 'wikiuser'@'localhost' to database 'my_wiki' (localhost:/workspace/db/quibble-mysql-5hjinx97/socket)
[13:48:56] <Daimona>	 No ideas, actually... However, FYI, I also came across another test failure for the CampaignEvents extension caused by the phpunit change, with master being broken. https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CampaignEvents/+/803992 is fixing the failing test, as you can see it has to do with wiki IDs. Unsure if it could be related to the other one
[13:48:58] <kostajh>	 Lucas_WMDE: sorry, you're right, I stopped reading at 'localhost'.
[13:49:57] <Lucas_WMDE>	 kostajh: no idea if that globals patch would help
[13:52:39] <kostajh>	 well, let's see
[13:53:46] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Math, 10MediaWiki-extensions-CentralAuth, 10MediaWiki-extensions-WikibaseClient, and 3 others: Wikibase\Client\Tests\Unit\Api\ApiFormatReferenceTest#testInvalidReferenceJson failure - https://phabricator.wikimedia.org/T310255 (10kostajh) Let's see if https://gerri...
[13:57:43] <James_F>	 kostajh: Eurgh, that's not great. :-(
[13:58:19] <James_F>	 But no ideas beyond "make CA auto-disable in a unit test context".
[14:08:28] <Krinkle>	 kostajh: I support a revert for now. There are now too many workarounds, both merged and unmerged that we don't even understand the reason for. We missed something. Possibly multiple things.
[14:08:45] <Krinkle>	 Let's take it outside the critical path and revisit offline
[14:11:05] <Krinkle>	 And yeah a mostly full revert including the mainatenance->integration bootstrap changes which are presumably the cause
[14:13:27] <Krinkle>	 I'm assuming the extension loader issue Daimona observes is also caused the bootstrap changes
[14:14:46] <Lucas_WMDE>	 I’ll upload a full revert and see if it fixes MediaWiki, at least
[14:14:53] <Lucas_WMDE>	 since the one in https://gerrit.wikimedia.org/r/c/mediawiki/core/+/804305 wasn’t enough, as you pointed out
[14:17:32] <Lucas_WMDE>	 if it fixes *Wikibase, I mean
[14:17:52] <wikibugs>	 10Release-Engineering-Team, 10Data-Persistence (Consultation), 10Security-API-Service, 10Security-Team, and 3 others: Determine CI best practices for service which connects to MySQL - https://phabricator.wikimedia.org/T308789 (10sbassett) @dduvall -   Well, we have a product demo scheduled for this Tuesday...
[14:28:56] <wikibugs>	 10Release-Engineering-Team, 10Scap: Scap's php-fpm restart step "left" counter may be counter-intuitive - https://phabricator.wikimedia.org/T310242 (10dancy) a:03dancy
[14:36:32] <wikibugs>	 (03CR) 10Jforrester: [C: 04-1] Add SearchVue repo zuul config (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/804015 (https://phabricator.wikimedia.org/T309932) (owner: 10Seddon)
[14:47:53] <kostajh>	 I don’t understand why this didn’t get caught in the patch which made the entry point change. Isn’t wikibase a gated extension?
[14:48:51] <Lucas_WMDE>	 I think it might not be, but I’m not sure
[14:51:03] <Lucas_WMDE>	 hm, it’s in gatedextensions in parameter_functions.py at least
[14:52:36] <Lucas_WMDE>	 looks like the full revert makes Wikibase green again
[14:53:05] <Lucas_WMDE>	 not sure why it wasn’t caught in the original patch either
[14:53:19] <Lucas_WMDE>	 Wikibase has some special jobs that mediawiki/core.git wouldn’t have, but the error wasn’t limited to those jobs, right?
[14:53:33] <Lucas_WMDE>	 it also happened in the standard noselenium-docker
[14:54:57] <Lucas_WMDE>	 or at least, it happened in those jobs in Wikibase CI…
[14:59:20] <James_F>	 gated_extensions doesn't recurse, so if the gate has Wikibase and not CA and it passes, but Wikibase has CA it will fail locally for Wikibase.
[15:00:19] <James_F>	 Essentially everything about CI is a hack; we should run different job sets for different concerns (e.g. "Wikimedia-like" vs. "basic third-party" vs. "third-party wikifarm") with different injection sets for each.
[15:01:28] <James_F>	 But running CA in the gate means disabling ~90% of the testing of MW's log-in system, as CA is fundamentally incompatible with it.( When CA is installed it makes all those tests skip; if we added CA to the gate we might as well delete the tests.)
[15:03:59] <wikibugs>	 10Release-Engineering-Team, 10Scap: Scap's php-fpm restart step "left" counter may be counter-intuitive - https://phabricator.wikimedia.org/T310242 (10Krinkle)
[15:05:57] <Lucas_WMDE>	 ah, right, CentralAuth is the part that was missing from the gated exts
[15:06:16] <Lucas_WMDE>	 makes sense
[15:12:46] <Lucas_WMDE>	 alright, the revert made it through gate-and-submit
[15:13:01] <Lucas_WMDE>	 sorry kostajh 😔 I hope the errors can be worked out
[15:15:14] <Lucas_WMDE>	 maybe the CentralAuth test setup (database = false, source tables-generated.sql) should be in a CI bootstrap file instead of (or in addition to?) the onUnitTestsAfterDatabaseSetup hook handler?
[15:15:31] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Math, 10MediaWiki-extensions-CentralAuth, 10MediaWiki-extensions-WikibaseClient, and 3 others: PHPUnit failure in various extensions: Wikimedia\Rdbms\DBConnectionError: Cannot access the database: Access denied for user 'w... - https://phabricator.wikimedia.org/T310255
[15:18:42] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Math, 10MediaWiki-extensions-CentralAuth, 10MediaWiki-extensions-WikibaseClient, and 3 others: PHPUnit failure in various extensions: Wikimedia\Rdbms\DBConnectionError: Cannot access the database: Access denied for user 'w... - https://phabricator.wikimedia.org/T310255
[15:27:47] <Daimona>	 Krinkle: could be related to bootstrap, yes. I didn't really investigate it, but it seems like previously, WikiMap::getCurrentWikiId() would return the empty string for unit tests, whereas now it returns the actual wiki ID. I'm not even sure if this is wanted. At any rate, it most probably has to do with globals
[15:57:46] <wikibugs>	 10GitLab (Project Migration), 10Release-Engineering-Team: Create new GitLab project group: content-transform - https://phabricator.wikimedia.org/T309194 (10brennen) 05Open→03Resolved a:03brennen Created: https://gitlab.wikimedia.org/repos/content-transform  Added everyone on Content Transform I could fin...
[16:07:48] <wikibugs>	 (03PS1) 10Ahmon Dancy: Perform php-fpm restart as a single job [tools/scap] - 10https://gerrit.wikimedia.org/r/804349 (https://phabricator.wikimedia.org/T310242)
[16:11:25] <wikibugs>	 (03PS2) 10Ahmon Dancy: Perform php-fpm restart as a single job [tools/scap] - 10https://gerrit.wikimedia.org/r/804349 (https://phabricator.wikimedia.org/T310242)
[16:13:23] <wikibugs>	 (03PS3) 10Ahmon Dancy: Perform php-fpm restart as a single job [tools/scap] - 10https://gerrit.wikimedia.org/r/804349 (https://phabricator.wikimedia.org/T310242)
[16:15:15] <wikibugs>	 (03PS4) 10Ahmon Dancy: Perform php-fpm restart as a single job [tools/scap] - 10https://gerrit.wikimedia.org/r/804349 (https://phabricator.wikimedia.org/T310242)
[16:19:56] <wikibugs>	 (03CR) 10CI reject: [V: 04-1] Perform php-fpm restart as a single job [tools/scap] - 10https://gerrit.wikimedia.org/r/804349 (https://phabricator.wikimedia.org/T310242) (owner: 10Ahmon Dancy)
[16:20:39] <wikibugs>	 (03PS5) 10Ahmon Dancy: Perform php-fpm restart as a single job [tools/scap] - 10https://gerrit.wikimedia.org/r/804349 (https://phabricator.wikimedia.org/T310242)
[16:32:50] <wikibugs>	 (03CR) 10Ahmon Dancy: [C: 03+2] Mount tmpfs at /run for db,deploy,www containers [tools/train-dev] - 10https://gerrit.wikimedia.org/r/803352 (owner: 10Ahmon Dancy)
[16:33:50] <wikibugs>	 (03CR) 10Ahmon Dancy: [C: 03+2] Remove files related to old git server image [tools/train-dev] - 10https://gerrit.wikimedia.org/r/803351 (owner: 10Ahmon Dancy)
[16:33:57] <wikibugs>	 (03Merged) 10jenkins-bot: Mount tmpfs at /run for db,deploy,www containers [tools/train-dev] - 10https://gerrit.wikimedia.org/r/803352 (owner: 10Ahmon Dancy)
[16:34:35] <wikibugs>	 (03Merged) 10jenkins-bot: Remove files related to old git server image [tools/train-dev] - 10https://gerrit.wikimedia.org/r/803351 (owner: 10Ahmon Dancy)
[16:39:38] <brennen>	 !log gitlab shared runners: re-registering to apply image allowlist configuration
[16:39:40] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[16:55:16] <hashar>	 duesen: apitests are opt in since they need related test files to be written for the extension. From zuul/layout.yaml there is apparently some extensions using them: CampaignEvents, EventBus, FlaggedRevs, GrowthExperiments, Math, Wikibase, WikibaseManifest.  They get the job template `extension-apitests` applied
[16:55:25] <hashar>	 add to them core , vendor and parsoid
[16:57:16] <wikibugs>	 10Release-Engineering-Team, 10Scap, 10Patch-For-Review: Scap's php-fpm restart step "left" counter may be counter-intuitive - https://phabricator.wikimedia.org/T310242 (10dancy) > Frame 0: For the canary restarts, it appears to be doing only 1 at a time, whereas later we do 10 at a time (in-flight never goes...
[16:57:49] <wikibugs>	 (03CR) 10Ahmon Dancy: [V: 03+1] "Tested in prod." [tools/scap] - 10https://gerrit.wikimedia.org/r/804349 (https://phabricator.wikimedia.org/T310242) (owner: 10Ahmon Dancy)
[17:07:05] <ori>	 I need to nuke and recreate the database for wikifunctionswiki on beta. I was going to run `mwscript maintenance/sql.php --wiki=wikifunctionswiki` and then 'drop database wikifunctions', then re-run the addWiki script. Does this sound sane?
[17:10:50] <wikibugs>	 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10Security-Team, 10Security, 10User-brennen: Limit GitLab shared runners to images from Wikimedia Docker registry - https://phabricator.wikimedia.org/T291978 (10brennen) 05Stalled→03Resolved Noting from IRC:  ` <jelto> brennen...
[17:21:16] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Math, 10MediaWiki-extensions-CentralAuth, 10MediaWiki-extensions-WikibaseClient, and 4 others: PHPUnit failure in various extensions: Wikimedia\Rdbms\DBConnectionError: Cannot access the database: Access denied for user 'w... - https://phabricator.wikimedia.org/T310255
[17:22:44] <wikibugs>	 10GitLab (Project Migration), 10Quarry: Move quarry to gitlab - https://phabricator.wikimedia.org/T308978 (10rook) Seems like waiting on this until some of the CI bits of gitlab are better established is recommended. ` <dancy> Ahmon Dancy  Rook:  Sorry for the delay. We're still working on codifying best pract...
[17:39:06] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Math, 10MediaWiki-extensions-CentralAuth, 10MediaWiki-extensions-WikibaseClient, and 4 others: PHPUnit failure in various extensions: Wikimedia\Rdbms\DBConnectionError: Cannot access the database: Access denied for user 'w... - https://phabricator.wikimedia.org/T310255
[18:44:46] <Krinkle>	 thcipriani: dancy: brennen: https://wikitech.wikimedia.org/wiki/Performance/Runbook/Monitor_production_errors
[18:45:00] <Krinkle>	 I've tried to capture most current practices and update some outdated ones
[19:03:37] <brennen>	 Krinkle: nice.
[19:06:16] <Krinkle>	 ori: hm.. good question. probably? I imagine the default user doesn't have those grants, but if it does, then sure.
[19:06:29] <Krinkle>	 as for runnign addWiki.php, see also https://wikitech.wikimedia.org/wiki/Add_a_wiki#Database_creation for some of the parameters and possible other implications
[19:06:32] <brennen>	 changes look good at a glance, but i'll try to give it a more thorough read over when i'm back at work on monday.
[19:07:19] <duesen>	 hashar: my understanding on how the CI jobs work is rather hazy... why you say sounds like extensions can opt into having API tests run in CI for their repo. But What I want to know if API tests for extensions run in CI for the *core* repo... 
[19:16:41] <wikibugs>	 10GitLab (Infrastructure), 10serviceops: Reduce usage of public IPv4 addresses on GitLab hosts - https://phabricator.wikimedia.org/T310265 (10Dzahn) > moving gitlab1001.wikimedia.org to gitlab1001.eqiad.wmnet  This is possible but would require reaching out to dcops to physically connect it to a different netw...
[19:21:48] <wikibugs>	 (03CR) 10Ahmon Dancy: [V: 03+1 C: 04-2] "holding" [tools/scap] - 10https://gerrit.wikimedia.org/r/804349 (https://phabricator.wikimedia.org/T310242) (owner: 10Ahmon Dancy)
[19:23:17] <wikibugs>	 10GitLab (Infrastructure), 10serviceops: Reduce usage of public IPv4 addresses on GitLab hosts - https://phabricator.wikimedia.org/T310265 (10Dzahn) First and foremost though, the reason why gitlab has all public IPs is because we were trying to emulate the gerrit setup. And gerrit has public IPs and is not be...
[19:46:44] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 10SRE, 10serviceops, 10Patch-For-Review: replace doc1001.eqiad.wmnet with a buster VM and create the codfw equivalent - https://phabricator.wikimedia.org/T247653 (10Dzahn) @Krinkle Yep, that summary sounds right to me. That's wha...
[20:01:30] <wikibugs>	 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T308068 (10dduvall)
[20:01:41] <wikibugs>	 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.39.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T308068 (10dduvall) 05Open→03Resolved
[21:07:44] <wikibugs>	 (03PS1) 10Ahmon Dancy: Revert "Add progress reporting to php-fpm-restarts" [tools/scap] - 10https://gerrit.wikimedia.org/r/803924 (https://phabricator.wikimedia.org/T302631)
[21:08:33] <wikibugs>	 (03PS2) 10Ahmon Dancy: Revert "Add progress reporting to php-fpm-restarts" [tools/scap] - 10https://gerrit.wikimedia.org/r/803924 (https://phabricator.wikimedia.org/T302631)
[21:08:58] <wikibugs>	 (03CR) 10Ahmon Dancy: [C: 03+2] Revert "Add progress reporting to php-fpm-restarts" [tools/scap] - 10https://gerrit.wikimedia.org/r/803924 (https://phabricator.wikimedia.org/T302631) (owner: 10Ahmon Dancy)
[21:12:20] <wikibugs>	 (03Merged) 10jenkins-bot: Revert "Add progress reporting to php-fpm-restarts" [tools/scap] - 10https://gerrit.wikimedia.org/r/803924 (https://phabricator.wikimedia.org/T302631) (owner: 10Ahmon Dancy)
[21:13:48] <wikibugs>	 (03Abandoned) 10Ahmon Dancy: Perform php-fpm restart as a single job [tools/scap] - 10https://gerrit.wikimedia.org/r/804349 (https://phabricator.wikimedia.org/T310242) (owner: 10Ahmon Dancy)
[21:25:05] <icinga-wm>	 PROBLEM - Check systemd state on doc1001 is CRITICAL: CRITICAL - degraded: The following units failed: rsync-doc-doc1002.eqiad.wmnet.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
[21:36:48] <wikibugs>	 (03CR) 10Daniel Kinzler: [C: 04-1] "Need to confirm whether api tests for parsoid are run against core patches. Of yes, then we can stop running api integration tests on the " [integration/config] - 10https://gerrit.wikimedia.org/r/802668 (owner: 10Arlolra)
[22:02:17] <wikibugs>	 (03PS3) 10Jeena Huneidi: scap backport: deploy to mwdebug [tools/scap] - 10https://gerrit.wikimedia.org/r/803370 (https://phabricator.wikimedia.org/T308476)
[22:07:20] <wikibugs>	 (03PS4) 10Jeena Huneidi: scap backport: deploy to mwdebug [tools/scap] - 10https://gerrit.wikimedia.org/r/803370 (https://phabricator.wikimedia.org/T308476)
[22:11:29] <wikibugs>	 (03CR) 10CI reject: [V: 04-1] scap backport: deploy to mwdebug [tools/scap] - 10https://gerrit.wikimedia.org/r/803370 (https://phabricator.wikimedia.org/T308476) (owner: 10Jeena Huneidi)
[22:20:47] <icinga-wm>	 RECOVERY - Check systemd state on doc1001 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
[22:22:54] <wikibugs>	 (03PS5) 10Jeena Huneidi: scap backport: deploy to mwdebug [tools/scap] - 10https://gerrit.wikimedia.org/r/803370 (https://phabricator.wikimedia.org/T308476)
[22:49:32] <dancy>	 !log Upgrading scap to 4.9.1-1+0~20220609211227.304~1.gbpe48c42 in beta cluster
[22:49:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[23:07:43] <jinxer-wm>	 (Queue (Jenkins jobs + Zuul functions) alert) firing: Queue (Jenkins jobs + Zuul functions) alert   - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert
[23:12:43] <jinxer-wm>	 (Queue (Jenkins jobs + Zuul functions) alert) firing: (2) Queue (Jenkins jobs + Zuul functions) alert   - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert
[23:13:40] <wikibugs>	 (03PS1) 10Jeena Huneidi: Add medebug container [tools/train-dev] - 10https://gerrit.wikimedia.org/r/804481 (https://phabricator.wikimedia.org/T308476)
[23:16:27] <icinga-wm>	 PROBLEM - Work requests waiting in Zuul Gearman server on contint2001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [400.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10
[23:18:41] <wikibugs>	 (03CR) 10Jeena Huneidi: Add medebug container (031 comment) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/804481 (https://phabricator.wikimedia.org/T308476) (owner: 10Jeena Huneidi)
[23:31:39] <wikibugs>	 10Beta-Cluster-Infrastructure: Create deployment-deploy04 as future secondary/upgrade - https://phabricator.wikimedia.org/T309437 (10Legoktm) What is the goal in building this as bullseye? Shouldn't it aim to match production, which is still buster?
[23:32:43] <jinxer-wm>	 (Queue (Jenkins jobs + Zuul functions) alert) resolved: Queue (Jenkins jobs + Zuul functions) alert   - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert
[23:35:33] <icinga-wm>	 RECOVERY - Work requests waiting in Zuul Gearman server on contint2001 is OK: OK: Less than 100.00% above the threshold [200.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10
[23:41:53] <wikibugs>	 10Beta-Cluster-Infrastructure: Create deployment-deploy04 as future secondary/upgrade - https://phabricator.wikimedia.org/T309437 (10TheresNoTime) >>! In T309437#7993618, @Legoktm wrote: > What is the goal in building this as bullseye? Shouldn't it aim to match production, which is still buster? I admit I didn't...