[00:38:48] I got bold and merged a change to add new listening ports to codesearch, deployed old pending change by Chad [00:39:09] ran puppet on codesearch8.. currently it seems "stuck" at refreshing hound [00:39:24] might just take a little while though [00:40:03] Chad and Legoktm might not be around, are you aware of others who are familiar with codesearch [00:41:21] mutante: should be fine, hound is just very slow [00:41:34] if not, I'll be done with this meeting in a bit and can poke at it then [00:41:37] and thank you :) [00:41:56] legoktm: oh, hi!:) [00:42:17] yea, so I currently see how puppet refreshes hound-discovery and stuff [00:42:27] no error,, just kind of slow [00:42:41] ack:) [00:43:06] and it's done now [00:43:44] legoktm: I now see that docker-proxy is listening on 6099. but not on 6100 [00:44:22] guesses there is a regex somewhere that covers up to 99 [00:44:33] change after I amended is https://gerrit.wikimedia.org/r/c/operations/puppet/+/828057/2/hieradata/cloud/eqiad1/codesearch/common.yaml [00:44:47] I'll look once I'm out of this meeting [00:45:13] search itself works again. thank you [09:25:04] Hi, I am trying to create a developer account on toolforge but when trying to register on https://toolsadmin.wikimedia.org/register/ldap I receive a message that my `username is already in use or invalid`. How should I proceed? [09:25:43] Are you already registered on wikitech? [09:26:17] I am not sure. I had an account on wmflabs previously. How could I check this? [09:26:38] Try using that login on toolsadmin [09:26:45] My username is `ckepper` [09:26:56] on Wikimedia [09:28:24] login with my username on https://toolsadmin.wikimedia.org/auth/login/ fails ("Please enter a correct username and password"). [09:28:49] Would it be possible to reset my password? [09:29:46] You could try to reset your password on wikitech.wikimedia.org using forgot password [09:31:00] Thank you, I will try that 👍 [09:31:46] it worked 👍👍👍 [09:33:17] Thank you for guiding me to the correct URL [09:36:55] Np [10:06:42] !log tools deployed api-gateway 0.0.16, no changes, ci rebuild (T341084) [10:06:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:06:46] T341084: [toolforge] Move all the components to the gitlab ci/cd flow - https://phabricator.wikimedia.org/T341084 [12:46:39] !log toolsbeta deployed builds-admission 0.0.63-20230712120152-2ef80a7c (T341084) [12:46:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [12:46:42] T341084: [toolforge] Move all the components to the gitlab ci/cd flow - https://phabricator.wikimedia.org/T341084 [12:46:46] !log tools deployed builds-admission 0.0.63-20230712120152-2ef80a7c (T341084) [12:46:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:04:10] !log tools.pagepile kubectl rollout restart deployment pagepile (lighttpd was failing to write temp files; “PHP Warning: Unknown: POST data can't be buffered; all data discarded”; cc T341656) [15:04:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.pagepile/SAL [15:14:43] Congrats on 10 years at the WMF, bd808! [15:15:16] andrewbogott: thanks. It seems like only 20 years! ;) [15:15:23] w00t :o congrats! [15:15:30] Congrats bd808 [15:15:46] 🥳congrats! [15:16:01] And also dhinus for tolerating us all for 1 year :) [15:16:18] Oh, and rook! [15:16:36] et moi? [15:16:41] Congrats dhinus [15:16:48] Congrats dhinus [15:16:57] Rook: I think blancadesal just said that you'd also passed a milestone? But I looked away and missed the slide [15:17:06] Rook: you've only been there a year? [15:17:21] Ah 2 [15:17:24] Congrats [15:18:28] 🥳 [15:21:50] Rook and dhinus have fixed tremendously many things since joining and life is so much better with them aboard [15:22:30] !log paws paws deploying from tf in codfw1dev T340657 [15:22:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [15:22:33] T340657: paws deploying from tf in codfw1dev - https://phabricator.wikimedia.org/T340657 [15:23:18] thank you! [16:56:03] Before I start poking people, T337791 — getting `Error while reading greeting packet` / `MySQL server has gone away` [16:56:04] T337791: CopyPatrol error 500 - https://phabricator.wikimedia.org/T337791 [16:59:41] TheresNoTime: I had a similar error at 16:37 on my tool using the replicas - Lost connection to MySQL server during query [17:00:35] hm.. copypatrol tripped my monitor at 16:38 [17:01:03] btullis: ^ [17:01:06] some people are also discussing similar errors in #wikimedia-tech it seems [17:01:12] (ack) [17:01:25] Ah, sorry. [17:02:54] I depooled wikireplicas to apply a view update, but it wasn't supposed to affect any queries in flight. [17:05:19] btullis: both sets of replicas seem to be giving connection errors now, it wasn't just an one-time blip for in-flight ones [17:06:00] Thanks. This seems to be like the LVS error that I saw last time from pybal. [17:08:07] is there a task from last time that would have prevented it from happening again? [17:11:10] (nb. https://copypatrol.toolforge.org/en is back) [17:13:51] I'm afraid I only got as far as writing it up in a Doc (https://docs.google.com/document/d/1yo0pCpOSQ4waAPtWU06xMQUs7UMhM5wnAj2U4NOObj8/edit) and I didn't find out the root cause then. [17:14:36] It required a restart of pybal from the traffic team and I wasn't sure why. We thought that it was coincidental with some concurrent work on the LVS servers, but there was no such work this time. [17:48:04] !log codesearch added new member aokoth [17:48:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Codesearch/SAL [19:41:26] can I keep a 2GB file in toolforge NFS? is there some documentation about the storage limits or something about what we should avoid when working with large files? [21:25:06] danilo: yes [21:25:25] there are tons of tools that have useless log files that are bigger than that [21:28:51] ok, thanks