[07:21:10] !log tools deploying toolforge-jobs-framework-cli v7 [07:21:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [07:21:13] !log toolsbeta deploying toolforge-jobs-framework-cli v7 [07:21:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [07:43:35] !log toolsbeta deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (d7d3463) - cookbook ran by arturo@nostromo [07:43:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [07:44:23] !log tools build & push docker image docker-registry.tools.wmflabs.org/toolforge-jobs-framework-api:latest from https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (d7d3463) - cookbook ran by arturo@nostromo [07:44:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [07:52:03] !log tools deployed kubernetes component https://gerrit.wikimedia.org/r/cloud/toolforge/jobs-framework-api (d7d3463) - cookbook ran by arturo@nostromo [07:52:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [09:05:08] Could someone please abandon https://gerrit.wikimedia.org/r/c/labs/tools/graphql/+/743011 as I lack permissions? Thanks in advance! [09:05:57] andre: I don't have either (maybe as it's wip?) [09:06:28] heh. cleanup is hard without having super admin powers, I know :P [09:06:36] I can, but just curious why? [09:07:26] taavi: you still want to understand systems <3 I just want to use them :D :D [10:39:23] help [10:39:45] can you be more specific? (re @MUD882: help) [11:58:23] some network operations happening right now [12:17:22] arturo: yes, thanks! [12:17:29] cool [12:18:15] (we use PAWS to host our Outreachy applicants -- not a big deal to have interruptions but just wanted to make sure it wouldn't be down for a while or they'll get worried) [12:22:40] wm-bridgebot is double-mirroring again (cc bd808) [12:22:59] since isaac.j’s first message (12:15 UTC) [14:28:14] !log wm-bot fix nick of an instance via telnet [14:28:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wm-bot/SAL [15:14:58] seems to be affecting other channels too, at least #wikipedia-abstract / Abstract Wikipedia (re @lucaswerkmeister: wm-bridgebot is double-mirroring again (cc bd808)) [15:17:29] @lucaswerkmeister: when it starts doubling it seems to affect all channels. Just got online to see if I can find anything at all in the logs before restarting. [15:17:42] ok thanks [15:17:53] I hadn’t noticed it in other channels before (but I guess I’m not in a lot of mirrored ones) [15:19:46] !log tools.bridgebot Restarting for double Telegram messages. Did grab the logs from exisitng pod this time to examine before rebooting. [15:19:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bridgebot/SAL [15:20:49] Hi Cloud VPS folks, random question: I'm wondering is there a way to run docker inside a VPS instance? We're working on this visual regression testing tool and it spins up all the selenium chromium dependencies in docker containers https://github.com/nicholasray/pixel [15:21:33] bd808: looks better now, thanks! [15:24:05] jan_drewniak: sure, you will just need to install some docker/container runtime to do it. I have had some good luck with podman on Debian Bullseye. There are some very light docs at https://wikitech.wikimedia.org/wiki/Developer_Portal#Demo_server on what I did to setup podman and configure it to run a single container app recently. [15:44:15] !log tools.wmde-access git -C public_html/ config pull.ff only # squelch a git message, --ff-only mode seems most prudent to me (T305484) [15:44:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wmde-access/SAL [15:45:10] !log tools.wmde-access toolforge-jobs run git-pull --command 'git -C ~/public_html pull' --image tf-php74 --schedule '*/10 * * * *' # T305484 [15:45:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wmde-access/SAL [15:45:46] !log tools.wmde-access crontab -r # T305484 [15:45:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wmde-access/SAL [15:52:17] Getting some hints about the Telegram double message problem from logs. It looks like the bot got disconnected from it's ZNC bouncer and when it reconnected it somehow saw everything double until it was hard restarted. [15:52:49] bd808: the disconnection today could be explained by me doing operations on cloud gateway stuff [15:53:09] but this was just today, the previous days we weren't running any particular network operation [15:54:20] *nod* the disconnect is something that will happen. The more interesting thing is seeing double messages from IRC after reconnecting. [15:54:54] Makes me think that it really isn't disconnecting and ends up with 2 clients connected to IRC at once [15:55:51] that's 100% possible. One connection tuple is temporarily blocked, a new connection happens, then the first one is unblocked, and now you have 2 [15:56:29] the potential solution is to set smaller timeouts, to declare connection as dead earlier [22:55:06] !log tools.bridgebot Restarting with custom matterbridge build to test fix for IRC message duplication (T305487) [22:55:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bridgebot/SAL [22:56:47] Test message from irc after restarting wm-bb [22:57:35] such broken! much wow! [22:58:46] !log tools.bridgebot Restarting to swtich back to official matterbridge-1.25.0-linux-64bit build (T305487) [22:58:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bridgebot/SAL [22:59:40] Test message from telegram after restarting wm-bb [23:02:33] That experiment did not work as hoped. I'm not sure yet if the fault was my lack of golang binary building skills or the patch I wrote to matterbridge.