[09:40:03] <wikibugs>	 (03PS1) 10Elukey: editquality: use json.dumps instead of urlencode for EventGate [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/813834 (https://phabricator.wikimedia.org/T301878)
[10:52:13] <wikibugs>	 (03CR) 10Elukey: [C: 03+2] editquality: use json.dumps instead of urlencode for EventGate [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/813834 (https://phabricator.wikimedia.org/T301878) (owner: 10Elukey)
[12:28:37] <klausman>	 Blergh, this heat is killing me
[12:29:41] <klausman>	 At least the humidity has dropped a bit (it's now at 30%RH, but this morning it was well over 50(
[12:30:05] <klausman>	 Or maybe it's just my TBE vaccine shot I got this morning :)
[13:52:02] <ottomata>	 its killing my laptop too!  i can't use google hangouts without an icepack for my lappy
[13:55:35] <klausman>	 I fortunately can place mine lika an A frame for extra ventilation
[13:55:49] <elukey>	 :(
[13:56:55] <ottomata>	 ventilation is not enough anymore, my A frame must be made out of ice
[13:56:58] <klausman>	 Yeah, I heard .it has it even worse than we have it here.
[13:57:37] <elukey>	 ottomata: o/
[13:57:47] <elukey>	 can I ask you a quick thing if you have a min?
[13:59:59] <ottomata>	 yaaaa
[14:00:06] <ottomata>	 actualy meeting startign!
[14:00:08] <ottomata>	 leave q i answer later!
[14:00:11] <elukey>	 I am testing sending an event from liftwing to eventgate main, but it seems that I get an HTTP 500 in return
[14:00:14] <elukey>	 okok!
[14:00:34] <ottomata>	 what' the error say?
[14:00:48] <elukey>	 I am checking https://logstash.wikimedia.org/app/dashboards#/view/AXN5OoJu3_NNwgAUlbUT?_g=h@c823129&_a=h@b0df7bf but it is difficult to grep
[14:00:57] <ottomata>	 i also find logstash near impossible to use
[14:01:09] <elukey>	 I need to add more logging I am afraid, I see only the 500
[14:01:20] <elukey>	 but the event should be well formed, in theory
[14:13:07] <elukey>	 ah interesting!
[14:13:07] <elukey>	 event e1809d3f-db47-493d-bd36-9c0575e63649 of schema at /mediawiki/revision/score/2.0.0 destined to stream unknown encountered an error: Field(s) meta.stream were missing from event and could not be extracted.","time":"2022-07-14T07:25:39.758Z","v":0}
[14:13:55] <klausman>	 Is that just a missing metadata field?
[14:14:22] <chrisalbon>	 Morning all!
[14:19:44] <elukey>	 klausman: in theory yes, but it should have it (at least, all my tests showed it)
[14:19:47] <elukey>	 chrisalbon: o/
[14:23:39] <klausman>	 Weird. I presume this is TLS? Otherwise you could wireshark the traffic and see what's actually going over the wire.
[14:24:11] <elukey>	 it is yes
[14:24:24] <elukey>	 in theory in kafka there should be the events that failed validation
[14:24:35] <elukey>	 I see the topics but can't find the events
[14:34:51] <ottomata>	 okay meeting over
[14:34:52] <ottomata>	 hello
[14:34:59] <chrisalbon>	 hello
[14:35:17] <ottomata>	 elukey: is the failed event body returned in the response?  i can't remember but  i think it is?
[14:35:47] <ottomata>	 elukey:  which k8s cluster are you producing to? staging?
[14:36:14] <elukey>	 ottomata: eventgate-main, the discovery endpoint, codfw in this case
[14:37:59] <ottomata>	 elukey:  ya i think only vaidation errors are produced to the error.validation topic
[14:38:02] <ottomata>	 this is not a validation error
[14:39:16] <ottomata>	 elukey: link to code producing event?
[14:40:25] <elukey>	 ottomata: this was the code change https://gerrit.wikimedia.org/r/c/machinelearning/liftwing/inference-services/+/808247/14/revscoring/editquality/model-server/model.py
[14:40:43] <elukey>	 (I changed it to send json directly instead of url-encoding a dict, but the rest is the same)
[14:41:18] <elukey>	 I tried to test this with a simple http python server "faking" eventgate locally, and  tried to send the event via curl to eventgate, all worked
[14:41:34] <elukey>	 so I suspect that maybe the POST from lift wing to eventgate has some issue
[14:46:42] <ottomata>	 the fact that your error message says  'of schema at /mediawiki/revision/score/2.0.0 ' is strange if that is the case
[14:46:52] <ottomata>	 because clearly the $schema field could be extracted
[14:46:58] <ottomata>	 which means the json parsed okay
[14:47:34] <elukey>	 interesting
[14:47:35] <ottomata>	 ya i dunno that is weird
[14:47:44] <ottomata>	 you are jsut posting to plain on /v1/events, rigth?
[14:47:48] <ottomata>	 no query params or anythning
[14:48:33] <elukey>	 exactly yes
[14:49:08] <elukey>	 and I checked the pod's env variables, I see
[14:49:08] <elukey>	       EVENTGATE_URL:             https://eventgate-main.discovery.wmnet:4492/v1/events
[14:49:11] <elukey>	       EVENTGATE_STREAM:          mediawiki.revision-score-test
[14:50:16] <elukey>	 so in theory the meta.stream field should be populated
[14:56:57] <ottomata>	 is this your event?
[14:56:58] <ottomata>	 https://logstash.wikimedia.org/goto/0e8ff44c30d4c84a9156ce2fca5729f0
[14:57:05] <ottomata>	 maybe that is the one from when you produced with urlencode?
[14:57:13] <ottomata>	 that has failure to extract $schema
[14:57:19] <ottomata>	 but looks like is produced from some tornado UUA
[14:57:21] <ottomata>	 UA
[14:58:47] <elukey>	 yeah the tornado UA makes sense (kserves uses its http client)
[14:59:21] <ottomata>	 where did you find your cannot extract meta.stream error?
[14:59:31] <ottomata>	 https://logstash.wikimedia.org/goto/8dcbb124cb5152ebfc1a39a7a1b33a8e
[14:59:38] <ottomata>	 are the only logs here for eventgate main in the last few hours
[15:00:54] <elukey>	 kubectl logs eventgate-main-production-5f84bcfcbd-f5wwp -n eventgate-main eventgate-main
[15:01:01] <elukey>	 with kube_env codfw
[15:01:41] <elukey>	 ah interesting, it was from the 12th
[15:01:48] <elukey>	 I just realized it
[15:01:54] <ottomata>	 ah, so that is not yours?
[15:02:15] <elukey>	 It should be, probably one of the early attempts
[15:02:18] <elukey>	 today I deployed https://gerrit.wikimedia.org/r/c/machinelearning/liftwing/inference-services/+/813834
[15:02:37] <elukey>	 to avoid the urlencode
[15:02:38] <ottomata>	 okay well good to know that this logstash dashboard is working at least!
[15:02:47] <ottomata>	 you should be able maybe to debug better this way?
[15:03:12] <elukey>	 will try yes! Thanks for the help!
[15:07:37] <elukey>	 going afk folks, talk with you next week (tomorrow I am off)
[15:38:36] <chrisalbon>	 Have a great weekend elukey! Thanks for everything this week
[18:34:05] <wikibugs>	 10Machine-Learning-Team, 10ORES, 10Growth-Team, 10PageCuration: Detect hoaxes - https://phabricator.wikimedia.org/T312841 (10MusikAnimal) > there has also been some very recent progress in comparing Wikipedia statements with their supposed digital sources. Facebook recently built a research prototype for d...