[07:16:50] hello folks [08:14:25] I have a systemd override to log DEBUG in celery on ores1001 [08:14:30] and I didn't really see the logs [08:27:31] it would be really nice to have a logstash dashboard with a breakdown of URIs and UA [08:30:27] 10Machine-Learning-Team, 10CFSSL-PKI, 10Infrastructure-Foundations, 10serviceops, 10Patch-For-Review: Extend cfssl-issuer to return the Root CA certificate - https://phabricator.wikimedia.org/T299906 (10JMeybohm) a:03JMeybohm [09:10:42] 10Machine-Learning-Team, 10CFSSL-PKI, 10Infrastructure-Foundations, 10serviceops, 10Patch-For-Review: Extend cfssl-issuer to return the Root CA certificate - https://phabricator.wikimedia.org/T299906 (10JMeybohm) [09:29:06] 10Machine-Learning-Team, 10observability: logstash schema for ORES logs - https://phabricator.wikimedia.org/T299999 (10elukey) [09:39:20] 10Machine-Learning-Team, 10CFSSL-PKI, 10Infrastructure-Foundations, 10serviceops, 10Patch-For-Review: Extend cfssl-issuer to return the Root CA certificate - https://phabricator.wikimedia.org/T299906 (10JMeybohm) [11:26:10] created https://gerrit.wikimedia.org/r/c/operations/puppet/+/756959 that (once ready) should be able to add more structure to our logstash dashboard [11:40:22] * elukey lunch! [15:40:45] o/ [15:41:27] o/ [15:41:49] the api-gateway should now be able to call our endpoints and validate tls certs correctly [15:41:58] \o/ [15:42:01] the per-service ratelimit will be released this week (hopefully) [15:42:04] awesome! [15:43:22] I am wondering about the routing table thought [15:43:26] *though [15:43:52] we need to use the $wiki input to construct a host header to use for our istio routing [15:43:59] but we'll need to sanitize it [15:44:11] otherwise users can pass potentially nasty things [15:44:29] and for example scan pods, and access to backends that we may hide for $reasons [15:44:38] not sure how we can do it [15:44:41] that's a really good point [15:44:51] it's all in yaml right? [15:45:01] it is yes [15:45:09] lol [15:45:17] the config I mean [15:46:17] can we right a regex somewhere? [15:47:04] we can but listing all allowed wikis on a line is horrible :D [15:47:21] ahhh yeah that would not be good lol [15:48:12] if you want to see the current map search for "pathing_map" in helmfile.d/services/api-gateway/values.yaml [15:48:35] there is no restriction afaics [15:48:37] mmm [15:50:12] (I asked to Hugh) [15:50:13] -- [15:50:18] completely different subject - https://gerrit.wikimedia.org/r/c/operations/puppet/+/756959 [15:50:30] I was able to write the logstash filter rule for ORES [15:50:36] lemme know if you like the idea [15:50:58] we should get more granularity to group request fields in the ores dashboard [15:51:05] niiice taking a look [15:54:30] 10Lift-Wing, 10Platform Team Initiatives (API Gateway): Update the API-Gateway k8s config to trust the Root PKI CA - https://phabricator.wikimedia.org/T299550 (10elukey) 05Open→03Resolved a:03elukey Hugh deployed the change to prod! [15:54:32] 10Lift-Wing, 10Machine-Learning-Team (Active Tasks): API Gateway Integration - https://phabricator.wikimedia.org/T288789 (10elukey) [16:01:24] elukey: i think the logstash filter for ORES looks great! [16:03:09] that will be helpful when we do the next deploy for the new nlwiki articlequality model [16:15:14] (need to rework it a little sigh, hope to fix it soon) [16:15:26] I'd love to have break downs for things like UA etc.. [17:50:35] the last change to the logstash filter should be hopefully ok, I'll wait for another review from Cole [17:50:51] after that we should be good in merging and check new data coming in to Logstash [17:51:05] (I'll try to add some breakdowns when it happens) [17:51:40] The user agent field will contain, I think, for the most part the Changeprop UA (that populates the score cache) [17:51:50] and then the bots that use Ores [17:52:10] from those it should be not horribly difficult to figure out who owns them [18:04:40] accraze,chrisalbon - is there a value in having the deployment window for ORES? [18:04:51] if not I created https://gerrit.wikimedia.org/r/c/mediawiki/tools/release/+/757048 [18:05:06] that should remove it from the deployment wikipage [18:05:13] if it has value I'll abandon the CR [18:13:02] elukey: lol i was wondering the same thing [18:14:35] IIRC in the past we did that because there were no SRE on team and we coordinated w/ serviceops [18:15:08] not sure if we need it anymore... [18:19:43] ack I'll add Tyler to the code review to remove it :) [18:20:21] * elukey afk! [18:20:32] have a nice (rest of the) day folks! [18:25:58] have a good one elukey! [20:43:54] accraze: I may have just spent an ungodly amount of money on a first pressing of Pink Floyd's Animals. [21:02:10] klausman: nice! was just listening to my copy the other night (not first press tho) a true classic [21:03:10] i'm too scared to get first pressings, i know i'll damage them lol [21:13:21] In for the penny, in for the pound, I say [21:13:46] It is _allegedly_ still in cellophane, so I am looking forward to actually unpacking it once it arrives [21:13:57] I shall take pictures [21:14:33] ^^ do it! [21:29:42] the ml-sandbox is going offline for a bit [21:32:55] need to rebuild with updated charts [21:42:09] (03PS1) 10Umherirrender: Replace deprecated JobQueueGroup::singleton() [extensions/ORES] - 10https://gerrit.wikimedia.org/r/757109 [22:21:57] (03CR) 10Krinkle: [C: 03+2] Replace deprecated JobQueueGroup::singleton() [extensions/ORES] - 10https://gerrit.wikimedia.org/r/757109 (owner: 10Umherirrender) [23:35:31] (03Merged) 10jenkins-bot: Replace deprecated JobQueueGroup::singleton() [extensions/ORES] - 10https://gerrit.wikimedia.org/r/757109 (owner: 10Umherirrender)