[09:26:21] morning ;) [09:26:49] kostajh: I have seen your patch to add memcached to Quibble and I am not sure what the benefit would be compared to php in memory cache (APCu) https://gerrit.wikimedia.org/r/c/integration/config/+/757867 [09:27:37] and looking at the Quibble patch it enables `CACHE_MEMCACHED` solely based on the extension being loaded [09:27:39] hashar: good morning [09:27:57] maybe that needs some extra config or maybe mediawiki has some sane default [09:28:07] yeah, so the JJB patch installs the extension + service, and then Quibble would just start using it [09:28:22] the default for memcache + MW is to use localhost:11211, no other config is needed [09:28:32] ah good [09:29:08] it may not have a big impact for Selenium, but maybe it helps as Selenium tests are parallelized or even as all Selenium projects are parallelized (https://gerrit.wikimedia.org/r/c/integration/quibble/+/768354) [09:29:32] so, definitely low priority... but since it's JJB it would be easy to revert, as opposed to needing a new release [09:30:10] i will rebase it :] [09:30:35] and build the images [09:30:50] for Selenium I wanna start a doc to revisit how those tests are being run [09:31:01] will take a bit of time cause I have some other docs to write as well bah [10:15:38] sounds good. you mean a doc for quibble? or on wiki? [10:24:41] kostajh: oh I mean random documents that are pending [10:24:57] I have to write something about tox [10:25:11] and some thing to compare Gerrit vs Gitlab/Github workflows [10:30:09] 10Quibble, 10Patch-For-Review: Use Memcached with Quibble - https://phabricator.wikimedia.org/T300340 (10hashar) ` Successfully published image docker-registry.discovery.wmnet/releng/quibble-buster-php74:1.4.3-s3 Successfully published image docker-registry.discovery.wmnet/releng/quibble-buster-php72:1.4.3-s2... [10:37:04] hashar: do you want me to make a patch to switch the selenium jobs to that image? [10:42:33] kostajh: sure! [10:42:46] I am in a meeting until 2pm, will deploy after that [10:43:10] the `utils/docker-updates` script should list the images that need updates [10:43:22] until I make it to do the change for us :] [10:50:57] that script doesn't work on macOS, unfortunately [10:53:24] ohh [10:54:36] anyway I've got a patch handcrafted :) [10:54:45] maybe cause of `git grep -P` which requires git to be compiled with PCRE [10:55:27] I think it's the `debian.Changelog` python lib [10:56:17] hashar: https://gerrit.wikimedia.org/r/c/integration/config/+/770468/ this bumped everything from s1/s2 to s3. Not sure why all jobs were not already on s2. And I'm not familiar with some of the jobs. So, you'll want to look closely I think :) [10:56:34] that lib might depend on some command line tools [10:59:22] the images installing PHP from sury.org got an extra rebuild so they are on s3 [10:59:33] but the ones having PHP from WMF did not so they are on s2 [10:59:36] I will fix the numbers [11:02:50] if you have a trace about `debian.Changelog` issue I am happy to look at it ;) [11:25:56] (03Restored) 10Hashar: Jenkins job validation (DO NOT SUBMIT) [integration/quibble] - 10https://gerrit.wikimedia.org/r/663557 (owner: 10Hashar) [11:25:57] I have switched the quibble full run jobs to the memcached images, will `recheck` a dummy change [11:26:08] (03CR) 10Hashar: "recheck for memcached in images https://gerrit.wikimedia.org/r/c/integration/config/+/770468/" [integration/quibble] - 10https://gerrit.wikimedia.org/r/663557 (owner: 10Hashar) [11:38:11] hashar: https://gitlab.wikimedia.org/-/snippets/18 about the docker-updates script [11:41:15] ohh of course [11:41:59] the script assumes you have `python-debian` module installed on your machine [11:42:04] it is not wrapped with tox [11:42:25] so theorically: `python3 -m pip install --user python-debian` should install it [11:45:25] (03CR) 10Kosta Harlan: "recheck" [integration/quibble] - 10https://gerrit.wikimedia.org/r/768354 (https://phabricator.wikimedia.org/T226869) (owner: 10Kosta Harlan) [11:48:01] https://gerrit.wikimedia.org/r/c/integration/quibble/+/663557 is a dummy change with the memcached image [11:48:15] looking at the debug log memcached does not seem to be used [11:49:54] hmm [11:50:49] I did a grep of "memcached" on the debug log [11:51:03] will check the debug log of some of the other builds [11:51:10] but for now, I gotta lunch [11:51:12] hashar: I don't seem to be able to pull `docker-registry.wikimedia.org/releng/quibble-buster-php72:1.4.3-s3` locally [11:51:21] k, later [11:51:23] ah [11:51:47] that is 1.4.3-s2 [11:57:13] oh [11:57:24] `php -r "echo extension_loaded( 'memcached' );"` works with `docker-registry.wikimedia.org/releng/quibble-buster-php73:1.4.3-s3` [11:57:42] so then `$wgMainCacheType = CACHE_MEMCACHED;` should be set. [12:00:54] I see `BacklinkCache::partition: got from memcached wikidb:backlinks:ns146%3AL18:templatelinks:300` in the debug log, but I'd expect to see a lot more entries (there's just 8) [12:07:34] or the testsuite explicitly disables the main cache / set it to CACHE_NONE [12:10:01] gotta look at the www log, iirc ew log the list of caches being used [12:56:22] for CLI [objectcache] MainWANObjectCache using store EmptyBagOStuff [12:56:34] and I see a bunch of [localisation] LocalisationCache::isExpired(en): cache missing, need to make one [12:56:49] however I am pretty sure we once made the optiization to keep l10n cached [12:58:29] ah yeah https://gerrit.wikimedia.org/r/c/integration/quibble/+/446840 [12:58:43] which runs rebuildLocalizationCache after update.php to populate the `en` l10n [12:58:52] but if there is no l10n cache .... that is not helpful [14:41:40] so mw cli installer does: [14:41:43] $wgMainCacheType = CACHE_ACCEL; [14:41:43] $wgMemCachedServers = []; [14:42:02] and then Quibble injects $wgMainCacheType = CACHE_MEMCACHED; [14:42:24] I am not sure what happens when there is an empty list of memcached servers configured, most probably no cache is used at all [14:47:09] Oh, hmm. Well, if $wgMemcachedServers is not defined, then the option in DefaultSettings.php would work [14:49:28] (03PS1) 10Kosta Harlan: LocalSettings: Define MemCachedServers when memcached is loaded [integration/quibble] - 10https://gerrit.wikimedia.org/r/770518 (https://phabricator.wikimedia.org/T300340) [20:04:58] hashar: I was expecting https://gerrit.wikimedia.org/r/c/integration/quibble/+/770518 to use 1.4.3-s3, but it looks like it used 1.4.3-s2, did you undeploy the job that changes the image? [20:12:49] hmm [20:14:10] kostajh: what I do is I search for `releng/quibble` in the console output to get the version of the image [20:14:18] then look at the image changelog [20:14:30] releng/quibble-buster-php72:1.4.3-s2 [20:15:06] which has the memcached extension [20:15:17] so I haven't undeployed the patch for the full run jobs [20:19:50] maybe the memcached extension is not loaded [20:21:38] it is [20:24:06] (03CR) 10Hashar: LocalSettings: Define MemCachedServers when memcached is loaded (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/770518 (https://phabricator.wikimedia.org/T300340) (owner: 10Kosta Harlan) [20:24:08] found it [20:24:23] kostajh: the code should be moved below the require_once localsettings-installer.php [20:24:30] to override the settings set by the mediawiki installer [20:25:35] ugh [20:27:41] (03PS2) 10Kosta Harlan: LocalSettings: Move Memcached block and define MemCachedServers [integration/quibble] - 10https://gerrit.wikimedia.org/r/770518 (https://phabricator.wikimedia.org/T300340) [20:28:18] (03CR) 10Kosta Harlan: LocalSettings: Move Memcached block and define MemCachedServers (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/770518 (https://phabricator.wikimedia.org/T300340) (owner: 10Kosta Harlan) [20:32:40] (03CR) 10Hashar: LocalSettings: Move Memcached block and define MemCachedServers (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/770518 (https://phabricator.wikimedia.org/T300340) (owner: 10Kosta Harlan) [20:32:59] I have asked to platform engineering for a way to dump the list of caches and their backend store [20:33:08] some debug log in ObjectCache might do that [20:33:17] or we can write a maintenance script :-] [20:37:19] (03CR) 10Kosta Harlan: LocalSettings: Move Memcached block and define MemCachedServers (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/770518 (https://phabricator.wikimedia.org/T300340) (owner: 10Kosta Harlan) [21:03:37] (03CR) 10Kosta Harlan: "Compared to https://gerrit.wikimedia.org/r/c/integration/quibble/+/770518/2#message-0a9e34ef4a31e48ab8577b3227f570df45862acd the overall t" [integration/quibble] - 10https://gerrit.wikimedia.org/r/770518 (https://phabricator.wikimedia.org/T300340) (owner: 10Kosta Harlan)