[03:13:04] (03PS28) 10Krinkle: [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) [03:13:31] (03CR) 10CI reject: [V:04-1] [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [04:20:38] (03PS29) 10Krinkle: [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) [04:20:56] (03CR) 10CI reject: [V:04-1] [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [04:38:57] (03PS30) 10Krinkle: [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) [04:43:56] O_O CI is passing [04:55:40] (03PS31) 10Krinkle: [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) [05:00:25] (03PS1) 10Krinkle: Switch CVNBot2,3,4,5,13,15,20,21 to pycvnbot [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332534 (https://phabricator.wikimedia.org/T327136) [05:04:34] (03CR) 10Krinkle: [C:03+2] Switch CVNBot2,3,4,5,13,15,20,21 to pycvnbot [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332534 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [05:05:08] (03Merged) 10jenkins-bot: Switch CVNBot2,3,4,5,13,15,20,21 to pycvnbot [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332534 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [05:22:21] (03PS1) 10Krinkle: tasks: Update cvnlog command for pycvnbot [labs/countervandalism/cvn-infrastructure] - 10https://gerrit.wikimedia.org/r/1332535 (https://phabricator.wikimedia.org/T327136) [05:22:40] https://grafana.wmcloud.org/d/0g9N-7pVz/cloud-vps-project-board?orgId=1&var-project=cvn&var-instance=$__all&from=now-2d&to=now&timezone=browser [05:22:51] Initial impression is that we're requiring a lot more CPU to do the same work [05:23:15] not sure what'll happen if I switch the big ones over (enwiki/commons/wikidata) [05:52:49] (03PS1) 10Krinkle: Switch CVNBot1,CVNBot14,CVNBot18 to pycvnbot [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332537 (https://phabricator.wikimedia.org/T327136) [05:52:57] (03CR) 10Krinkle: [C:03+2] Switch CVNBot1,CVNBot14,CVNBot18 to pycvnbot [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332537 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [05:53:30] (03Merged) 10jenkins-bot: Switch CVNBot1,CVNBot14,CVNBot18 to pycvnbot [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332537 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [05:53:36] (03CR) 10Krinkle: [V:03+2 C:03+2] tasks: Update cvnlog command for pycvnbot [labs/countervandalism/cvn-infrastructure] - 10https://gerrit.wikimedia.org/r/1332535 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [06:21:12] Okay CPU seems fine on closer look. Might be an unrelated spike [06:21:17] Memory a fair bit lower. [06:21:42] CVNBot18, Wikidata building up a delay of 1min per minute [06:22:07] :07 reported on :14, :10 at :20 [06:22:38] Unclear if due to processing cost or due to feed channel send queue (artificial delay) [06:23:12] That might be worth exposing a a number (send queue size), separate from RCFeed seconds since. [06:24:02] Excess Flood kill [06:26:56] https://libera.chat/guides/faq#are-bots-allowed [06:27:46] "Users and bots with the flags +v [..] within a channel have protection against triggering flood mitigation. [..] this does not exempt bots from [..] rate limits for sending messages, which allow one message to be sent every two seconds." [06:31:13] That having been said, I don't think that's why it's behind. [06:31:52] It's down to a few per minute now but not catching up [06:32:05] So clearly at the intake side [07:48:38] (03PS32) 10Krinkle: [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) [07:49:45] (03PS33) 10Krinkle: [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) [07:49:48] (03CR) 10CI reject: [V:04-1] [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [07:50:39] (03CR) 10CI reject: [V:04-1] [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [08:49:48] (03CR) 10Hashar: "I haven't looked at the code but here are some comments related to tox/flake/pyproject etc." [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [14:51:53] (03PS34) 10Krinkle: [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) [14:52:42] (03PS35) 10Krinkle: [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) [14:52:47] (03PS36) 10Krinkle: [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) [15:03:00] (03PS37) 10Krinkle: [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) [15:07:33] (03CR) 10Krinkle: "Thanks. RE: EventStreams, not in the short term as I have other more immediate things to deal with, including upstream QUnit and other too" [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [15:48:39] (03PS38) 10Krinkle: [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) [17:13:40] (03PS39) 10Krinkle: [WIP] Rewrite CVNBot in python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) [18:43:13] 10CVNBot: Switch MW API queries in CVNBot from format=xml to format=json - https://phabricator.wikimedia.org/T436550 (10Krinkle) 03NEW [18:44:44] 10CVNBot, 13Patch-For-Review: Rewrite CVNBot in Python - https://phabricator.wikimedia.org/T327136#12272362 (10Krinkle) p:05Triage→03High a:03Krinkle [18:44:45] 10CVNBot: Evaluate running from a shared binary - https://phabricator.wikimedia.org/T327131#12272365 (10Krinkle) a:03Krinkle [18:46:58] 10CVNBot: CVNBot: Allow importing more than 500 sysops from wiki - https://phabricator.wikimedia.org/T436552 (10Krinkle) 03NEW [18:47:43] (03PS40) 10Krinkle: Rewrite CVNBot in Python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) [18:49:21] (03PS41) 10Krinkle: Rewrite CVNBot in Python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) [19:23:49] (03PS42) 10Krinkle: Rewrite CVNBot in Python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) [19:23:50] (03PS1) 10Krinkle: ListManager: Optimize classify_editor from 3 queries to 1 query [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1332795 (https://phabricator.wikimedia.org/T327136) [19:25:20] (03PS2) 10Krinkle: ListManager: Optimize classify_editor from 3 queries to 1 query [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1332795 (https://phabricator.wikimedia.org/T327136) [19:31:59] 10Countervandalism-Network, 10Cloud-VPS (Debian Bullseye Deprecation): Upgrade cvn fleet from Debian 11 Bullseye to Debian 12 Bookworm (PHP 7.4 > PHP 8.2) - https://phabricator.wikimedia.org/T395164#12272502 (10AntiCompositeNumber) [19:32:00] 10CVNBot, 13Patch-For-Review: Rewrite CVNBot in Python - https://phabricator.wikimedia.org/T327136#12272503 (10AntiCompositeNumber) [19:32:13] 10CVNBot, 13Patch-For-Review: Rewrite CVNBot in Python - https://phabricator.wikimedia.org/T327136#12272504 (10AntiCompositeNumber) 05Open→03In progress [19:47:05] (03PS3) 10Krinkle: ListManager: Optimize classify_editor from 3 queries to 1 query [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1332795 (https://phabricator.wikimedia.org/T327136) [19:48:31] (03CR) 10Krinkle: [C:03+2] Rewrite CVNBot in Python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [19:49:00] (03CR) 10Krinkle: [C:03+2] ListManager: Optimize classify_editor from 3 queries to 1 query [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1332795 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [19:49:57] (03Merged) 10jenkins-bot: Rewrite CVNBot in Python [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1324908 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [19:50:23] (03Merged) 10jenkins-bot: ListManager: Optimize classify_editor from 3 queries to 1 query [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1332795 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [19:52:54] (03PS1) 10Krinkle: Switch CVNBot_v5 template to /srv/cvn/git/CVNBot directory [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332800 [19:55:23] (03CR) 10Krinkle: [C:03+2] Switch CVNBot_v5 template to /srv/cvn/git/CVNBot directory [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332800 (owner: 10Krinkle) [19:55:59] (03Merged) 10jenkins-bot: Switch CVNBot_v5 template to /srv/cvn/git/CVNBot directory [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332800 (owner: 10Krinkle) [19:56:19] (03PS1) 10Krinkle: Swigtch one bot on cvn-app12 to CVNBot 5 (pycvnbot) [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332801 (https://phabricator.wikimedia.org/T327136) [19:59:36] (03PS2) 10Krinkle: Switch one bot on cvn-app12 to CVNBot 5 (pycvnbot) [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332801 (https://phabricator.wikimedia.org/T327136) [19:59:40] (03CR) 10Krinkle: [C:03+2] Switch one bot on cvn-app12 to CVNBot 5 (pycvnbot) [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332801 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [20:00:15] (03Merged) 10jenkins-bot: Switch one bot on cvn-app12 to CVNBot 5 (pycvnbot) [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332801 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [20:07:00] (03PS1) 10Krinkle: Revert CVNBot6 back [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332803 (https://phabricator.wikimedia.org/T327136) [20:07:08] (03PS2) 10Krinkle: Revert CVNBot6 back [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332803 (https://phabricator.wikimedia.org/T327136) [20:07:20] (03CR) 10Krinkle: [C:03+2] Revert CVNBot6 back [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332803 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [20:07:54] (03Merged) 10jenkins-bot: Revert CVNBot6 back [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332803 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [20:21:53] (03PS1) 10Krinkle: Program: Start global catcher earlier [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1332806 (https://phabricator.wikimedia.org/T327136) [20:21:55] (03PS1) 10Krinkle: IrcClient: Increase send_delay from 0.3s to 0.4s [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1332807 [20:24:37] (03CR) 10Krinkle: [C:03+2] Program: Start global catcher earlier [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1332806 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [20:25:20] (03CR) 10Krinkle: [C:03+2] IrcClient: Increase send_delay from 0.3s to 0.4s [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1332807 (owner: 10Krinkle) [20:26:15] (03Merged) 10jenkins-bot: Program: Start global catcher earlier [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1332806 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [20:27:09] (03Merged) 10jenkins-bot: IrcClient: Increase send_delay from 0.3s to 0.4s [labs/countervandalism/CVNBot] - 10https://gerrit.wikimedia.org/r/1332807 (owner: 10Krinkle) [21:55:17] (03PS1) 10Krinkle: Switch one bot on cvn-app12 to CVNBot 5, again [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332815 (https://phabricator.wikimedia.org/T327136) [21:56:21] (03CR) 10Krinkle: [C:03+2] Switch one bot on cvn-app12 to CVNBot 5, again [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332815 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [21:56:55] (03Merged) 10jenkins-bot: Switch one bot on cvn-app12 to CVNBot 5, again [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/1332815 (https://phabricator.wikimedia.org/T327136) (owner: 10Krinkle) [21:59:04] !log Edit CVNBot.ini on cvn-app12 for all bots to include logsyslog=1 [21:59:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Cvn/SAL [22:26:17] The process just hangs indefinitely [22:26:56] Probably socket.recv without timeout [22:27:19] So the connection drop is insisting from a connection that's quiet. [22:27:27] indistinguishable