[00:26:10] !log lucaswerkmeister@tools-sgebastion-10 tools.lexeme-forms deployed 4106259494 (l10n updates: ht, hu) [00:26:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL [10:38:06] !log tools reboot tools-k8s-worker-nfs-55 got nfs lockup (logrotate in D state) [10:38:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:42:23] hm, I ran a bridgebot restart command, and it says that the pod was deleted, but the command hasn’t finished yet [10:42:34] oh, but apparently it is running again [10:43:36] !log lucaswerkmeister@tools-sgebastion-10 tools.bridgebot Double IRC messages to other bridges [10:43:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bridgebot/SAL [10:44:05] ok there it goes [10:44:19] and I’m sure this is perfectly normal k8s behavior https://tmp.lucaswerkmeister.de/bridgebot.png [10:53:18] filed T358976 which may or may not be related [10:53:18] T358976: kubectl is quite slow the “first time” per user account - https://phabricator.wikimedia.org/T358976 [10:53:34] feel free to retitle / edit it to “something” “better” ;) [11:16:43] i get a "Slim Application Error" for croptool, what does that mean? [11:23:57] Details [11:23:58] Type: ImagickException [11:23:58] Code: 445 [11:23:58] Message: cache resources exhausted `/workspace/public_html/files/b042a8876a88bda9cafcc864ac170b5000f513a2.jpg' @ error/cache.c/OpenPixelCache/4095 [11:23:58] File: /workspace/src/php/File/JpegFile.php [11:23:59] Line: 15 [14:00:51] !log tools.wikibugs Preping for another attempt at migrating to newer codebase (T357851) [14:00:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL [14:00:56] T357851: bd808's big pile of refactoring ideas - https://phabricator.wikimedia.org/T357851 [14:40:13] !log tools.wikibugs Replace redis2irc job with znc + irc jobs (T357851) [14:40:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL [14:40:17] T357851: bd808's big pile of refactoring ideas - https://phabricator.wikimedia.org/T357851 [14:44:34] wikibugs: so... are you just not seeing redis or what? [14:49:43] !log tools.wikibugs Rolled back to redis2irc (T357851) [14:49:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL [14:49:48] T357851: bd808's big pile of refactoring ideas - https://phabricator.wikimedia.org/T357851 [15:09:59] @bd808 re: wikibugs: so... are you just not seeing redis or what? -- are you using a different queue name for the tests? The redis methods used might only send events to a single listener (not entirely sure, I think we use rpush and rpop?) [15:10:47] rpush and blpop, apparently [15:10:58] valhallasw: I didn't notice that the legacy deployment is using REDIS_QUEUE_NAME=libera. My new irc client was looking at an empty list. [15:12:21] Trying to decide between configuring the new stuff with REDIS_QUEUE_NAME=libera, or cutting the feeds over to a new queue name. Using a well known queue name is actually seeming like a poor idea in the long term. [15:12:40] Yes, the queue name should be considered a secret [15:13:20] so I would recommend just generating a random name, or a guid or something like that :-) [15:13:58] :nod: [15:20:42] New plan is to switch the feeds first, pushing to a new queue. I'll verify input with redis-cli and then switch the irc bot last when the current queue is empty. [15:28:03] !log tools.wikibugs Replacing legacy grrrrit job with new gerrit job (T357851) [15:28:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL [15:28:07] T357851: bd808's big pile of refactoring ideas - https://phabricator.wikimedia.org/T357851 [15:32:32] !log tools.wikibugs Replacing legacy wikibugs-phab job with new phorge job (T357851) [15:32:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL [15:35:59] !log tools.wikibugs Replace redis2irc job with znc + irc jobs take 2 (T357851) [15:36:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL [15:36:03] T357851: bd808's big pile of refactoring ideas - https://phabricator.wikimedia.org/T357851 [15:44:28] valhallasw: it's alive! All of the jobs are switched over and data is flowing from both gerrit and phab to the queue and from the queue to irc. [15:44:42] getting all the configuration to match certainly helped. :) [15:48:31] \o/ [15:50:47] I will start on the documentation updates next. [19:22:43] I think I read somewhere that the servers that host WDQS was optimized for DB but I can't find the specs anywhere. Anyone have a link? [19:24:20] Cumulonimbus: you might start looking from https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service [19:25:09] I'm not sure how up to date https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service#Hardware is. Folks in the #wikimedia-search channel during US business hours might have better links for you. [19:31:28] Thanks, found it. The specs were more modest than I though. The cheaper DELLs max out at 128GB which is what they went for I guess. [19:51:50] Page says 6 servers when grafana says 24. Ok, guess it's not completely up to date :) [23:36:41] !log bd808@tools-sgebastion-11 tools.wikibugs Restart phorge task to pick up changes for T358653 [23:36:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL