[08:09:35] taavi: re: access to alerts.git, we've punted on it for now though if it becomes a serious hinderance in the future we can reevaluate [09:32:17] !log tools cleaned up grid queue errors on tools-sgegrid-master.tools.eqiad1.wikimedia.cloud (T304816) - cookbook ran by arturo@nostromo [09:32:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [14:39:38] if i get this right toolforge-jobs --schedule doesn't use cron, right? [14:40:16] and you can't set the timezone? [14:41:06] gifti: --schedule is a cron job. You can't set the timezone [14:41:15] what [14:41:26] cron job in the context of Kubernetes [14:41:37] so it's not in the crontab? [14:41:56] of the tool user [14:42:05] no, toolfoge-jobs is a thing only for kubernetes, no grid engine, if that's what you mean [14:42:28] is it possible to add timezones in the future? [14:43:00] I guess so... but is not currently in our roadmap. First step would be to open a feature request ticket on phabricator [14:43:29] hm [14:44:11] although per kubernetes upstream docs, timezones are not natively supported https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/ [14:44:56] (bonus point if you detect the link to an enwiki article in that k8s page) [14:44:59] and it wouldn't really be a good idea to add it to toolforge-jobs, since timezones change relative to UTC through the year [15:06:16] redirecting stderr to stdin is also not supported? [15:06:41] --command "... 2>&1" doesn't do the trick [15:07:09] also stderr seems to be output twice [15:35:44] hi folks, not sure if I need to cut a ticket or just ask here, but I have an issue with ml-sandbox.machine-learning.eqiad1.wikimedia.cloud [15:36:43] the VM has a volume attached, and ~ 4 days ago it was rebooted (not by my team, maybe maintenance?). From that time the volume is not visible as device on the OS, and I cannot mount the partition on top of it (that is /srv) [15:37:02] I tried in horizon to detach/attach/etc.. but I get only errors [15:37:20] is there anything that I should check/do to re-attach the volume? [15:53:25] elukey: lots of meeting now, if you open a ticket we will be able to reply async/etc. for now, did you reboot the VM? Stop and start it (so it runs in a different host)? might be interesting debugging it though, see what's the issue (/me back to meeting) [15:55:19] dcaro: yeah tried but can't really get it to work, I'll open a task :) [16:37:01] created https://phabricator.wikimedia.org/T304872 [19:42:11] urbanecm: The blocked issue is happening again. [19:42:15] This time to Harej himself. [19:42:28] He is completely blocked from using the tool on enwiki right now. [19:42:42] Details posted in the ticket. [19:43:07] I am able to edit through the block. Presumably my sysop bit is letting me get around it. [20:03:54] [toolforge-jobs] ERROR: unable to create job: "HTTP 403: likely an internal bug: 403 Client Error: Forbidden for url: https://k8s.tools.eqiad1.wikimedia.cloud:6443/apis/apps/v1/namespaces/tool-giftbot/deployments. k8s [20:03:59] is something down? [20:24:50] Cyberpower678: can Harej reliably reproduce the issue ATM? [20:25:28] if so, perhaps we can do a quick video call so i can check what MW thinks about the requests it sees and compare it with what the webtool thinks it's doing. [20:34:47] they and another editor have reported on the task that it is every attempt [20:43:31] well maybe i can test myself [20:48:32] well, I'd need someone to manually put my unprivileged account into the basicuser group (`Martin Urbanec (public)`) [21:12:54] urbanecm: yes [21:12:58] urbanecm: one sec [21:13:34] urbanecm: I put you into the grou[ [21:13:54] which group where? [21:14:34] basicuser as you requested [21:14:37] oh [21:14:38] righřžt [21:14:41] thanks [21:37:41] urbanecm: were you able to reproduce it? [21:37:47] yes [21:37:53] :-) [21:37:57] Cyberpower678: would it be possible to add more details to the debug comment? [21:38:04] Such as? [21:38:08] full headers of both request and response [21:38:18] Not really [21:38:36] The headers contain the OAuth headers [21:39:05] And the headers are handled on a different layer of the bot's code. [21:39:10] that one can be redacted (the authentication itself seems to work fine) [21:39:54] What are the headers needed for? [21:40:12] to get more information about the issue. it makes less and less sense [21:41:34] Let me see what I can do. I can throw in a hot patch on production, but it will get wiped on the next update. [21:42:04] there is no autoblock set for the IP sees the request from [21:45:33] Cyberpower678: if it makes things any easier, I'm looking especially for the following things: suspicious cookies (the block target's blockID cookie for autoblock would cause this kind of an issue, for instance), X-Forwarded-For header and similar [21:46:27] Yea, I just need to make sure that user sessions cookies aren't being passed back to the output. [21:46:39] #SECURITYFAIL [21:47:56] so long it's _my_ cookies, it shouldn't be an issue, or am i missing sth? [22:18:10] urbanecm: Yea, I just don't want anyone pasting theirs into the Phabricator [22:18:18] got it [22:18:30] let me test a theory i have now... [22:20:14] but it fails [22:23:26] urbanecm: you know what? Screw it. This change is temporary at best and cookies can be invalidated. [22:23:36] I'm not going to bother filtering it out. [22:30:14] :) [22:31:48] urbanecm: give it a trye [22:32:12] Sure, give me a minute [22:34:35] Cyberpower678: this is all i get: `` [22:34:50] Uhhh.. [22:35:14] Oh dear. Fatal error [22:37:44] I'm getting a No route to host error, for the DB connection [22:37:55] Unrelated to what I touched [22:38:38] interesting [22:41:07] Oh wait. I'm not looking at the right lines. [22:41:36] No errors are showing up under error.log [22:43:29] urbanecm: okay, try now [22:45:15] better [22:47:21] Cyberpower678: i can use the tool again. but i don't see the headers anywhere [22:47:44] They should be in the same output. [22:48:24] they're not :( [22:48:29] let me try again [22:48:59] didn't help [22:49:09] They only show up when the error happens. [22:49:15] What are you getting? [22:49:23] this https://usercontent.irccloud-cdn.com/file/LivgZ9lY/image.png [22:49:56] I mean what are you getting when you open up Inspect Element and go to the top? [22:50:11] https://phabricator.wikimedia.org/P23432 [22:50:40] Well shit [22:50:52] That's inconsistent with what I just coded up [22:52:23] Third time's the charm [22:52:40] Give it a minute to update in the webservice and try again [22:53:04] ok [22:56:21] urbanecm: any luck? [22:57:11] sort of [22:57:28] it says ńResponse Headers: ` [22:57:30] but no headers [23:00:36] Okay, I accidentally passed through the wrong place. :/ [23:00:54] Give it another go. If it doesn't, work, I'll cry. [23:05:24] urbanecm: please tell me it worked [23:05:49] Gimme few mins please [23:05:53] ok [23:15:04] finally [23:16:23] urbanecm: yay. Anything useful? [23:16:27] (can't find the request ones though) [23:16:52] There's nothing but the OAuth header being passed in the request. [23:17:53] I don't think that's true. there must be a least host: header (otherwise MW servers don't know which wiki you send the req to) [23:18:36] Well, I'm not passing anything meaningful in there. [23:18:51] So anything generated is done by Curl automatically [23:19:03] what if it's the cookie? :)) [23:19:16] (the cookie autoblock block cookie) [23:19:42] as mentioned at https://phabricator.wikimedia.org/T274050#7676239, that'd result in symptoms we're observing [23:20:35] Let me see if I can fetch the headers [23:20:50] (I vaguely recall that was possible somehow) [23:21:48] at minimum there should be GeoIP, WMF-Last-Access, and WMF-Last-Access-Global cookies as well (after the first request) [23:22:23] yeah [23:27:45] Alright, I added request headers, I think [23:28:35] urbanecm: ^ [23:29:08] it only says `Request Headers: `, [23:29:12] no actual headers present :( [23:29:56] Are the response headers populated? [23:30:00] yes [23:30:30] Then, they're either blank, CURL doesn't want to hand them over, or I messed up...again [23:31:06] they can't be break. not sure how libcurl works though [23:31:57] Oh wait, they might be in array form, rather than a string. Echoing arrays doesn't work [23:33:36] Okay, they should show now [23:33:57] (Can you tell I'm not testing any of this in the slightest? :p) [23:35:03] very much so [23:35:09] and still empty :/ [23:35:14] :D [23:35:15] D: [23:35:38] * Cyberpower678 just can't finish eating his dinner. :p [23:37:42] Nope, it's a string according to the manual [23:38:02] :'-( [23:44:35] urbanecm: can you paste a sample of the current output [23:44:46] I don't know what's going wrong here this time. [23:45:32] https://phabricator.wikimedia.org/P23435 [23:47:41] urbanecm: not fixed, but run it again. I'm var_dumping the variable. [23:48:00] I need to see what it is exactly. [23:52:31]

 /mnt/nfs/labstore-secondary-tools-project/iabot/master/app/src/Core/APII.php:1239:boolean false
[23:52:33] 	 apparently a false
[23:52:47] 	 (but, i think it might be easier if you try it yourself :))
[23:53:22] 	 The data only shows when the error happens.  I really don't want it showing for all web requests.,
[23:53:52] 	 Okay false means the retrieval of the headers from CURL was a failure
[23:54:33] 	 But I don't know why.  There could be some kind of production policy on Toolforge disallowing me from accessing them.
[23:55:10] 	 01:53  The data only shows when the error happens.  I really don't want it showing for all web requests., <== yeah, but at this point, it sounds to show for all requests
[23:55:24] 	 (I'm literally just submitting Prague at enwki from my unprivileged account)
[23:57:00] 	 Ah, I think I see the issue.
[23:58:31] 	 yeah?
[23:58:42] 	 I think so.