[00:02:26] danilo: would you like me to try force deleting the job? [00:05:26] bd808: yes, I want to kill the job [00:06:37] !log tools.ptwikis Force killed job 3103931 [00:06:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.ptwikis/SAL [00:06:41] I thought in use 'qdel -f', but I don't know if that kill the job or just remove it from the grid engine list [00:07:00] thanks [00:07:24] danilo: yw. The `-f` flag only works as an admin I believe [04:36:15] I recent got elasticsearch write credentials (https://phabricator.wikimedia.org/T298934). [04:37:06] But the ini file seems to be incorrectly formatted, there's no section headers. [04:37:12] When I try to parse it, I get: [04:37:13] configparser.MissingSectionHeaderError: File contains no section headers. [04:37:29] and I can't fix it myself because the file is owned by root. [08:16:35] !log maps bump quota from 24 -> 26 cores, 48 -> 50 GB RAM, T299585 [08:16:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Maps/SAL [08:16:38] T299585: Request increased quota for maps Cloud VPS project - https://phabricator.wikimedia.org/T299585 [12:34:52] !log toolsbeta removing grid node toolsbeta-sgeexec-1004 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo [12:34:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [12:35:41] !log toolsbeta removing grid node toolsbeta-sgeexec-1003 (depool/drain, remove VM and reconfigure grid) - cookbook ran by arturo@nostromo [12:35:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [12:56:50] !log tools scaling up the grid with 10 buster exec nodes (T277653) [12:56:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:56:53] T277653: Toolforge: add Debian Buster to the grid and eliminate Debian Stretch - https://phabricator.wikimedia.org/T277653 [12:59:44] \o/ [12:59:59] :o [13:10:47] 🎉 [13:35:02] somehow DNS record creation failed for most of the new instances :-/ [14:26:18] hi, anyone that could help me with receiving a +2 for https://gerrit.wikimedia.org/r/c/operations/puppet/+/682259 ? :) [17:05:54] !log tools drop 9 of the 10 buster exec nodes created earlier. They didn't get DNS records [17:05:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:27:11] Southparkfan: I'm not well equipped to test that patch but I can certainly merge it if you're around to test [18:28:22] sure, I am around [18:29:23] the patch is currently applied through a standalone puppetmaster, but after merge I can update the git branch to 'real' (updated) production's [18:32:31] 'k [18:32:50] one more rebase! [18:34:11] thanks [18:34:35] I wish I had learned about https://wikitech.wikimedia.org/wiki/Help:Standalone_puppetmaster#Push_using_a_single_branch sooner, works awesome [18:41:28] looks like it worked :) [18:46:47] no issues found. we should follow up on rollout for projects. thank you! [19:01:15] hi wmcs folks, wikiuser and wikiadmin grants in mysql for wikitech are on three different IPs, 208.80.154.160, 208.80.155.109, 208.80.152.189. All of these needed? [19:02:34] the first 2 make sense, labweb1001 and labweb1002. but the third one does not resolve to anything [19:02:52] possibly was used at some point? [19:02:59] would guess so, yea [19:05:10] I wonder what it was then, apparently not silver or californium [19:05:16] that would be codfw [19:05:18] (per netbox) [19:05:26] maybe the placeholder for cloud in codfw [19:06:19] https://www.irccloud.com/pastebin/7cBdRxSa/ [19:06:27] git log -S on dns brought this [19:06:37] from 2013 [19:07:02] I have no idea what cams is [19:07:06] guess technically you would want labweb2001/labweb2002 there. [19:07:06] cameras? [19:07:12] cameras were literal cameras [19:07:14] a LONG time ago [19:07:27] like to watch the cages in the data center [19:07:28] how that supposed to have access to wikitech database? [19:07:29] Amir1: currently labweb1001/1002 should be all that's needed [19:07:30] actual security cameras [19:08:11] okay then, I just drop that user on s6 [20:46:26] I'm getting a `dschwen@maps-wma2.maps.eqiad1.wikimedia.cloud: Permission denied (publickey).` error for a new instance that was was just this morning able to log in. What could be the reason? (I can log into other instances of that project without issues) [20:47:11] how new? [20:50:01] a day [20:50:14] dschwen82: it's refusing my root key as well, so something is very broken [20:50:29] fuuuuu [20:50:38] is there data on there that's important? [20:50:44] I can try to rescue or you can start over [20:50:57] no, it is all on NFS or the attached volume [20:51:14] I'd love to know what went wrong for future reference [20:52:36] well, depends on whether you can spare the resources to investigate, and if you think figuring out what went wrong might be useful to you, too [20:54:49] error: Unsafe AuthorizedKeysCommand "/usr/sbin/ssh-key-ldap-lookup": bad ownership or modes for directory / [20:56:10] dschwen82: looks like you did a chown on /? That's a good way to break things. Let's see if I can fix [20:56:47] dschwen82: try now [21:01:07] oh lordy, thats.. ..embarassing [21:01:21] we all make typos :) [21:01:35] yeah, that was an rasyng that went to / instead of . [21:01:42] rsync (geez) [21:01:58] thanks, I'm in [21:04:47] Ok, do I file a ticket on phabricator to get this redacted from the public logs? ;-) [21:05:37] No results found. Did you mean "save to https://bash.toolforge.org/"? [21:06:14] :-D [21:07:11] ^ I don't know who that "dschwen82" is [21:08:07] we regret to inform you the call is coming from inside the NAT