[04:51:18] <andrewbogott>	 !log tools rebooting tools-sgeweblight-10-27, tools-sgeweblight-10-17 and tools-sgeweblight-10-30; their filesystems seem locked up and I suspect NFS somehow
[04:51:21] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[05:10:23] <legoktm>	 I'm wondering if people have "archived" tools by redirecting them to the wayback machine
[05:11:18] <legoktm>	 I'm looking at https://extreg-wos.toolforge.org/ and I'd like the URL to keep working for historical purposes but I also don't want to have to maintain the running webservice (I think it's just static HTML but still), etc.
[05:14:47] <legoktm>	 ugh, it's a python3.5 webservice :( my point exactly
[06:12:33] <wm-bb>	 <harej> legoktm if it's just static HTML couldn't you just do a dump and then point traffic to that? that's what I did with Wikimedia DC's wordpress
[06:13:25] <legoktm>	 yeah, I'll probably end up doing that
[06:13:28] <wm-bb>	 <harej> This blog has been defunct for almost 9 years but strong is my commitment to preventing linkrot https://blog.wikimediadc.org
[06:13:42] <wm-bb>	 <harej> (And, apparently, the people who took after me long after I stopped doing this)
[11:01:48] <dcaro>	 !log tools rebooting tools-sgeweblight-10-25 due to memory allocation issue (T352753)
[11:01:52] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[11:01:52] <stashbot>	 T352753: [tools-sgeweblight-10-25] puppet throws segmentation fault - https://phabricator.wikimedia.org/T352753
[14:46:12] <Steinsplitter>	 hello, anibody has a hint for me how to kill a stucking grid engine job? My webgrid-lighttpd@tools-sgewebl is stucking in qd status for quite a while now. Thanks
[14:52:00] <JSherman>	 hi there, I've been trying to add/update dns records for the twl project, but I'm getting errors on save. Looks like a bug?
[14:52:01] <JSherman>	 https://phabricator.wikimedia.org/T352713
[15:12:42] <wikibugs>	 !log tools.wikibugs Updated channels.yaml to: 27b1219beab5c5900e7696ccdfd7d37337a715f4 channels: Add commtech-kanban to commtech channel
[15:12:45] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL
[15:15:48] <andrewbogott>	 JSherman: does the error appear during the initial 'create a stub' step, or only when editing?
[15:17:36] <JSherman>	 andrewbogott: only when editing on the stub; when I try to create a full record from the start, I get the error and can't create it.
[15:18:05] <JSherman>	 actually, let me go back and verify rather than depend on memory
[15:18:11] <andrewbogott>	 ok! I'm seeing if I can reproduce it here...
[15:18:48] <Steinsplitter>	 sh: 1: /usr/bin/convert: not found  <-- we have no /bin/convert in the kubernetes images?
[15:19:59] <JSherman>	 create a full record: error, no record created
[15:20:57] <JSherman>	 create stub: no error, record created
[15:21:41] <JSherman>	 update stub with full info: error, no update
[15:22:38] <JSherman>	 I can add in all the info for an spf record if I replace ' ' with '+', but obviously that doesn't do anything useful
[15:23:48] <andrewbogott>	 weird.  ok, thanks for checking
[15:25:11] <JSherman>	 no problem!
[15:36:37] <andrewbogott>	 JSherman: when you update your recordset are you adding an additional record or changing the existing one?
[15:37:41] <andrewbogott>	 (If the latter, can you try adding an additional record instead?)
[15:38:23] <JSherman>	 I think I tried both yesterday, but I only tried the latter today; testing the former now to be sure.
[15:39:26] <JSherman>	 andrewbogott: same result; error, no update when adding a record to the set
[15:39:41] <andrewbogott>	 it's working for me, and I don't know if that's good or bad yet. Mind if I try doing what you're doing in your project?
[15:40:27] <JSherman>	 andrewbogott: go for it; I'm tying to add `v=spf1 a:185.15.56.1 ~all` to that top level txt file
[15:42:43] <andrewbogott>	 ok, so it works if the content is quoted. I'm guessing that doesn't actually work for spf purposes though?
[15:44:29] <TheresNoTime>	 !log `[samtar@tools-sgegrid-master ~ (main u=)]$ sudo qdel -f 3652373` for T352777
[15:44:30] <stashbot>	 TheresNoTime: Unknown project "`[samtar@tools-sgegrid-master"
[15:44:30] <stashbot>	 T352777: Grid engine job 3652373 stucking in dr state - https://phabricator.wikimedia.org/T352777
[15:44:41] <TheresNoTime>	 gr
[15:44:57] <TheresNoTime>	 !log tools.steinsplitter `[samtar@tools-sgegrid-master ~ (main u=)]$ sudo qdel -f 3652373` for T352777
[15:45:00] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.steinsplitter/SAL
[15:45:06] <andrewbogott>	 JSherman: does that agree with what you're seeing? If so that makes me think there's some kind of overly-aggressive input validation happening.
[15:51:55] <JSherman>	 andrewbogott: yep, quoting it worked; when I look at the record with dig, it actually looks correct.
[15:52:16] <andrewbogott>	 ok! So seems like that's an adequate workaround for you.
[15:52:53] <andrewbogott>	 Do you have thoughts about correct behavior?  Should the UI just stealthily quote unquoted txt records, or should there be a warning in the dialog, or... ?
[15:57:06] <JSherman>	 I don't think quotes should be required at all; right now `"v=spf1"`and `v=spf1` result in the same record
[15:57:49] <JSherman>	 cpanel is probably the web dns editor I've used the most, and it doesn't require wrapping records with spacing
[15:58:11] <JSherman>	 I was also having trouble creating cname records, let me see if wrapping fixes that as well.
[16:01:22] <JSherman>	 nope, still can't create a cname record, though it looks like that may be a separate issue. Same result though; 400 error with an empty response. I think I can handle my immediate task without it though.
[16:03:17] <andrewbogott>	 ok. Please open a task about the cname issue as well if you're so inclined.
[16:17:17] <JSherman>	 andrewbogott: I'll do some more exploring on that to see if I can define the problem better; for now, thank you for getting me unstuck!
[19:16:06] <andrewbogott>	 !log tools rebooting tools-sgeweblight-10-26.tools.eqiad1.wikimedia.cloud; can't log in even with root key
[19:16:09] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[19:34:30] <isaacj>	 o/ PAWS is down for me? anyone else? I'm following https://hub-paws.wmcloud.org/hub/login via https://wikitech.wikimedia.org/wiki/PAWS and getting `Service unavailable` on Firefox and Chrome
[19:35:15] <Rook>	 I was running an upgrade, looks like the hub container is taking longer to restart than usual...
[19:35:30] <Rook>	 https://phabricator.wikimedia.org/T310749
[19:36:44] <isaacj>	 ahh excellent just bad timing on my part then - thanks Rook! i'll be patient :)
[19:37:29] <Rook>	 That ticket describes why the issue was visible, though there does appear to be an issue. Looking into it
[19:40:00] <wm-bot>	 !log tools.lexeme-forms <lucaswerkmeister> deployed 95ee032c68 (l10n updates: ca, hno, io, it, pnb, sl, tr; i18n test improvements and fixes)
[19:40:04] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL
[20:14:02] <Rook>	 isaacj: should be back, deployed a new cluster, the old one was having trouble re-assigning the volume
[20:14:11] <Rook>	 Sorry if it breaks again, still tinkering a little
[21:15:40] <isaacj>	 working - thanks!