[01:56:54] 10WM-Bot, 10Huggle, 10User-MacFan4000: XMLRCs is not functioning - https://phabricator.wikimedia.org/T295487 (10bd808) >>! In T295487#7501291, @Stashbot wrote: > {nav icon=file, name=Mentioned in SAL (#wikimedia-cloud), href=https://sal.toolforge.org/log/FqoAF30Ba_6PSCT9ogVU} [2021-11-13T01:54:16Z] `... [01:58:17] @RC+ meta.wikimedia.org User:billinghurst [01:58:21] System is shutting down, requested by MacFan4000 from #wm-bot [01:58:37] probably best [01:58:41] @RC+ meta.wikimedia.org User:billinghurst [01:58:41] sDrewth: restart required first to reload the module [01:59:00] !ping [01:59:00] Pong. [01:59:05] @RC+ meta.wikimedia.org User:billinghurst [01:59:14] Channel had already feed enabled [01:59:14] @rc-on [01:59:36] @RC+ en.wikipedia.org Tesr [01:59:36] Inserted new item to feed of changes [01:59:48] oh, a spacwe [01:59:51] @RC+ meta.wikimedia.org User:billinghurst [01:59:51] Inserted new item to feed of changes [02:00:02] @RC- en.wikipedia.org Tesr [02:00:02] Deleted item from feed [02:00:32] hmm, didn't do [02:03:05] Channel configuration was reloaded [02:03:05] @reload [02:15:06] DEBUG Exception: Length cannot be less than zero. Parameter name: length st: at System.String.Substring (System.Int32 startIndex, System.Int32 length) [0x0003a] in :0 at libirc.ProcessorIRC.Info (libirc.Network+IncomingDataEventArgs info) [0x0015f] in <9ba4d297e1a34271970c639a70110657>:0 at libirc.ProcessorIRC.Result () [0x00a9c] in <9ba4d297e1a34271970c639a70110657>:0 at libirc.ProcessorIRC.ProfiledResult () [0x00028] in <9ba4d297e1a34271970c639a70110657>:0 at wmib.WmIrcProtocol.ThreadExec () [0x0012f] in <43dd9ad2626747678306d27cbc7f98e3>:0 [02:15:06] DEBUG Exception: Length cannot be less than zero. Parameter name: length st: at System.String.Substring (System.Int32 startIndex, System.Int32 length) [0x0003a] in :0 at libirc.ProcessorIRC.Info (libirc.Network+IncomingDataEventArgs info) [0x0015f] in <9ba4d297e1a34271970c639a70110657>:0 at libirc.ProcessorIRC.Result () [0x00a9c] in <9ba4d297e1a34271970c639a70110657>:0 at libirc.ProcessorIRC.ProfiledResult () [0x00028] in <9ba4d297e1a34271970c639a70110657>:0 at wmib.WmIrcProtocol.ThreadExec () [0x0012f] in <43dd9ad2626747678306d27cbc7f98e3>:0 [02:16:01] hmm [02:16:06] xmlrcs still isn't up [02:16:58] it was started [02:19:57] @RC- meta.wikimedia.org User:billinghurst [02:19:57] Deleted item from feed [02:20:02] @RC+ meta.wikimedia.org User:billinghurst [02:20:02] Inserted new item to feed of changes [02:40:33] yeah, but it's still not up—you can check with huggle [02:41:05] Looking at the logs, there wearing any issues connecting to XMLRCs [02:41:17] it's 100% not a wm-bot thing [02:41:20] it's an xmlrcs thing [02:41:41] wearn’t* [03:00:07] 10WM-Bot, 10Huggle, 10User-MacFan4000: XMLRCs is not functioning - https://phabricator.wikimedia.org/T295487 (10bd808) `name=/opt/xmlrcs/nohup.out Traceback (most recent call last): File "./es2r.py", line 16, in rs.set("es2r.pid", int(os.getpid())) File "/usr/lib/python2.7/dist-packages/redi... [03:00:48] bd808, disk space? [03:01:05] (Also, python 2.7!?) [03:02:16] `df` seems to think there is space, but I'm going to try to find redis logs. And yeah there is still a ton of py2.7 in the Wikimedia tools space. [03:02:30] eep [03:03:19] 10WM-Bot, 10Huggle, 10User-MacFan4000: XMLRCs is not functioning - https://phabricator.wikimedia.org/T295487 (10bd808) `name=/var/log/redis/redis-server.log 11418:M 13 Nov 03:02:37.048 * 1 changes in 900 seconds. Saving... 11418:M 13 Nov 03:02:37.048 # Can't save in background: fork: Cannot allocate memory ` [03:03:36] ohhh, huh [03:03:47] curl https://downloadmoreram.com :D [03:06:11] petan: ^^ [03:07:21] I think what happened is that the redis queue filled up because the consuming script was down. I'm going to reboot the instance and then check back to see what has to be manually started. [03:08:38] that makes sense [03:13:10] http://wm-bot.wmcloud.org/dump/%23wm-bot.htm [03:13:10] @info [03:13:17] ok I am seeing data in a telnet session! Y'all probably need to restart the bot here again [03:13:33] System is shutting down, requested by MacFan4000 from #wm-bot [03:13:47] !ping [03:13:47] Pong. [03:13:59] Change on 12meta.wikimedia.org a page Wm-bot was modified, changed by MacFan4000 link https://meta.wikimedia.org/w/index.php?diff=22328310 edit summary: [03:13:59] Change on 12meta.wikimedia.org a page Wm-bot was modified, changed by MacFan4000 link https://meta.wikimedia.org/w/index.php?diff=22328310 edit summary: [-5] [03:14:06] There we go! [03:14:31] so for next time... https://wikitech.wikimedia.org/wiki/XmlRcs#Maintainer_info [03:14:36] @RC- meta.wikimedia.org User:billinghurst [03:14:36] Deleted item from feed [03:15:01] but wow this is a very "petan made it work but also very custom and manual" thing [03:15:14] it super is :D [03:15:35] bd808 are you going to update that section with what you did, just in case both you and petan get hit by a bus? [03:16:06] I can make that doc slightly better I think... [03:16:19] Thank you kindly :) [03:18:21] 10WM-Bot, 10Huggle, 10User-MacFan4000: XMLRCs is not functioning - https://phabricator.wikimedia.org/T295487 (10Perryprog) 05Open→03Resolved a:03Perryprog From @bd808: "[03:07:21] bd808: I think what happened is that the redis queue filled up because the consuming script was down. I'm going to reboot t... [03:19:11] 10WM-Bot, 10Huggle, 10User-MacFan4000, 10User-bd808: XMLRCs is not functioning - https://phabricator.wikimedia.org/T295487 (10MacFan4000) a:05Perryprog→03bd808 [03:19:33] oops, thank you [03:19:42] I have no clue why it assigned it to me considering that wasn't in the preview bit [03:20:56] Unless you specify otherwise or the task is already assigned, closing a task as resolved auto-assigns it to you [03:21:19] Good to know [03:26:36] Attempted to make https://wikitech.wikimedia.org/wiki/XmlRcs#Maintainer_info more informative with an example [03:27:17] fancy styling [03:27:41] looks like I could follow that, so looks good to me :) [03:28:11] also that should probably be moved to systemd or something instead of a nohup but meh [03:29:50] yeah it 100% needs to be automated to auto-start on boot to have any stability at all. It is sheel luck that this ever works. [03:29:59] *sheer luck [03:33:05] 10WM-Bot, 10Huggle, 10User-MacFan4000, 10User-bd808: XMLRCs is not functioning - https://phabricator.wikimedia.org/T295487 (10bd808) The services DO NOT start on boot, so after rebooting someone needs to do something like: ` $ ssh xmlrcs.huggle.eqiad1.wikimedia.cloud $ sudo su - xmlrcs $ cd /opt/xmlrcs $ .... [03:33:31] agreed [03:34:28] but I think there's no way to get that done in a stable manner without petan themself, so we'll have to wait for them to wake from their cryogenic stasis for that :D [03:44:53] I have added a see also to [[wikitech:wm-bot]] [03:44:53] https://wikitech.wikimedia.org/wiki/wm-bot [03:48:19] thanks bd808 [03:48:30] yw [03:49:48] Though the other issue is that not many active people have access to the huggle project [03:53:41] * perryprog nods [03:54:15] Has petan been active here "recently"? I think the last time I personally remember talking with him was during the Prague hackathon. [03:57:49] I think I've seen him within the last month [03:59:05] I would imagine he would be willing to have some help with the huggle project if any of you want to reach out and offer that. [04:00:20] C# is scary but I can do maintenance stuff [04:00:41] addshore: also has admin access [04:00:41] I'm also confident there's other people better suited than I, but /shrug [04:01:08] addshore has about 150 other side projects too :)