[06:11:25] good morning, the ROCM debs archive key has expired over the weekend, could you please import the latest key to puppet.git? [06:28:50] moritzm: o/ morning, sure I'll do it asap [06:34:03] thanks :-) [06:55:53] moritzm: one qs - IIRC the import of those keys was automated, what shall I do in this case? (I tried checking in the puppet repo but didn't find much) [07:03:14] Puppet takes care of this! You only need to update the key under modules/aptrepo/files/updates-keys [07:03:30] probably the _radeon.gpg one, didn't check in detail [07:03:51] then with the next puppet run on apt1001 it will get picked up [07:12:43] ahhh okok! [07:44:10] * elukey errand, bbl [09:13:51] 10Machine-Learning-Team: Some outlink events are rejected by EventGate - https://phabricator.wikimedia.org/T343002 (10elukey) Schema: https://schema.wikimedia.org/repositories//primary/jsonschema/mediawiki/page/prediction_classification_change/current.yaml ` {'is_redirect': False, 'namespace_id': 0, 'page_id'... [09:20:31] isaranto: morning, when should we deploy https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/942649? [09:20:39] does stuff from hewiki and itwiki work? [09:22:19] o/ Amir1: yep itwiki and hewiki work fine! Any time would be a great time to deploy. I will be on top checking everything etc [09:23:02] awesome. Let's do it now [09:23:05] lemme know when you are available to do it. For me the sooner the better as I will have more hours to verify things work [09:23:12] around to double check as well :) [09:23:14] ack. awesome!! [09:23:35] that includes my home wiki so if it breaks, I'll be really grumpy :P [09:24:23] Amir1: is the exclude list ok (wikidata and enwiki) or should we exclude anything else? [09:24:33] nope, that's good [09:24:50] I don't think we have anything with commons [09:25:05] (03CR) 10Ladsgroup: [C: 03+2] add User-agent header in Lift Wing requests [extensions/ORES] - 10https://gerrit.wikimedia.org/r/941838 (https://phabricator.wikimedia.org/T342605) (owner: 10Ilias Sarantopoulos) [09:27:23] (03Merged) 10jenkins-bot: add User-agent header in Lift Wing requests [extensions/ORES] - 10https://gerrit.wikimedia.org/r/941838 (https://phabricator.wikimedia.org/T342605) (owner: 10Ilias Sarantopoulos) [09:33:52] klausman: o/ We are also ready to proceed with the API gateway changes https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/940945 [09:35:39] oups, I see some traffic from yesterday in a deployment I want to kill (eswikiquotewiki).. [09:36:42] in logstash? [09:38:24] isaranto: I'm not very smart, I didn't save the job running piece we use to test in mwdebug, do you have it? [09:38:39] a yes, I have a new page also [09:38:45] just a sec [09:39:13] https://www.mediawiki.org/wiki/Extension:ORES/Debugging [09:40:47] thanks! [10:04:41] isaranto: something very weird - I see "SampleChangePropInstance" as UA in the ORES dashboard [10:04:50] ~300 messages/min [10:05:06] the only thing that I found is https://github.com/wikimedia/mediawiki-services-change-propagation/blob/master/config.jobqueue.wikimedia.yaml#L152 [10:05:12] but I have no idea [10:05:17] it seems doing /precache as well [10:05:21] yes I've seen that didnt have time to look into it.. [10:06:03] I'll try to dig more into it [10:15:36] wow it comes from outside [10:15:48] somebody is apparently running a changeprop instance [10:15:57] that calls ores to precache :D [10:16:19] https://logstash.wikimedia.org/goto/d616b4f5a215b46be5d14f2baa37e121 [10:16:30] ouch! [10:17:24] it is traffic that we can drop anytime [10:17:39] but reaching out to the owner is difficult, we have the IP address but no contact info [10:43:54] * elukey lunch! [11:30:26] * isaranto lunch as well [12:24:15] elukey: I am seeing some 502s on ruwiki-damaging and by checking at the pods I see that there are some pods terminating and by looking at the events I see that it is because of autoscaling (it is scaling down). I am not sure if the 500s are related to this. For example nlwiki is doing the same but no 500s over there [12:45:11] isaranto: I see around 14 in the past 2 hours, and they are 502s so some connection problems between the istio gateways (reporting the error code, basically the "clients") and the model servers [12:46:08] the autoscaling could be playing a role [12:51:17] (03PS2) 10Ilias Sarantopoulos: add flag for host header [extensions/ORES] - 10https://gerrit.wikimedia.org/r/938267 (https://phabricator.wikimedia.org/T319170) [12:54:45] 10Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10Patch-For-Review: Deployment of Lift Wing usage to all wikis that use ores extension - https://phabricator.wikimedia.org/T342115 (10isarantopoulos) [12:54:49] 10Machine-Learning-Team, 10Wikipedia-Android-App-Backlog: Migrate Machine-generated Article Descriptions from toolforge to liftwing. - https://phabricator.wikimedia.org/T343123 (10Seddon) [12:56:48] 10Machine-Learning-Team, 10Wikipedia-Android-App-Backlog: Migrate Machine-generated Article Descriptions from toolforge to liftwing. - https://phabricator.wikimedia.org/T343123 (10Seddon) p:05Triage→03Low [12:58:17] mmm I checked the gateway logs and I see ""upstream_transport_failure_reason":null" [12:58:34] so it may be a 502 generated from the model servers, in theory [12:59:35] 10Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10Patch-For-Review: Deployment of Lift Wing usage to all wikis that use ores extension - https://phabricator.wikimedia.org/T342115 (10isarantopoulos) [13:00:36] 10Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10Patch-For-Review: Deployment of Lift Wing usage to all wikis that use ores extension - https://phabricator.wikimedia.org/T342115 (10isarantopoulos) At this point Lift Wing usage has been enabled in all of the above wikis except enwiki and wikidata, whic... [15:33:24] 10Machine-Learning-Team: Append PYTHONPATH in blubber - https://phabricator.wikimedia.org/T342273 (10isarantopoulos) a:03isarantopoulos https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/49#note_44989 [15:49:44] going afk for today folks [15:49:48] have a nice rest of the day [16:01:57] good evening! [17:55:10] 10Machine-Learning-Team: Append PYTHONPATH in blubber - https://phabricator.wikimedia.org/T342273 (10CodeReviewBot) dduvall merged https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/49 feat: append pythonpath for python [18:04:14] 10Machine-Learning-Team, 10Wikipedia-Android-App-Backlog: Migrate Machine-generated Article Descriptions from toolforge to liftwing. - https://phabricator.wikimedia.org/T343123 (10Isaac) [18:09:46] 10Machine-Learning-Team, 10Wikipedia-Android-App-Backlog: Migrate Machine-generated Article Descriptions from toolforge to liftwing. - https://phabricator.wikimedia.org/T343123 (10Isaac) Excited to see this moving forward! I did my best to answer the questions in the description. The one that I left un-answere... [18:11:31] 10Machine-Learning-Team, 10Wikipedia-Android-App-Backlog: Migrate Machine-generated Article Descriptions from toolforge to liftwing. - https://phabricator.wikimedia.org/T343123 (10Isaac) Also adding @fkaelin who can help with any questions of ownership while I'm on sabbatical (next week through Sept 8). [18:31:23] 10artificial-intelligence, 10VisualEditor, 10VisualEditor-MediaWiki-References: generate citation error should auto-consult AI, or notify devs, so they can try to improve it - https://phabricator.wikimedia.org/T341420 (10ThurnerRupert) another example which does not work: https://www.ncbi.nlm.nih.gov/pmc/art... [18:31:58] 10artificial-intelligence, 10VisualEditor, 10VisualEditor-MediaWiki-References: generate citation error should auto-consult AI, or notify devs, so they can try to improve it - https://phabricator.wikimedia.org/T341420 (10ThurnerRupert) p:05Triage→03Unbreak! [18:32:58] 10artificial-intelligence, 10VisualEditor, 10VisualEditor-MediaWiki-References: generate citation error should auto-consult AI, or notify devs, so they can try to improve it - https://phabricator.wikimedia.org/T341420 (10ThurnerRupert) p:05Unbreak!→03Triage [18:33:59] 10artificial-intelligence, 10VisualEditor, 10VisualEditor-MediaWiki-References: generate citation error should auto-consult AI, or notify devs, so they can try to improve it - https://phabricator.wikimedia.org/T341420 (10ThurnerRupert) [18:38:12] 10artificial-intelligence, 10VisualEditor, 10VisualEditor-MediaWiki-References: generate citation error should auto-consult AI, or notify devs, so they can try to improve it - https://phabricator.wikimedia.org/T341420 (10ThurnerRupert) as it is an error on the user interface, can this please be a bug report?... [19:03:32] 10artificial-intelligence, 10VisualEditor, 10VisualEditor-MediaWiki-References: generate citation error should auto-consult AI, or notify devs, so they can try to improve it - https://phabricator.wikimedia.org/T341420 (10Izno) [19:05:42] 10artificial-intelligence, 10VisualEditor, 10VisualEditor-MediaWiki-References: generate citation error should auto-consult AI, or notify devs, so they can try to improve it - https://phabricator.wikimedia.org/T341420 (10Izno) No, this is appropriately a feature request. [19:06:03] 10artificial-intelligence, 10Citoid: generate citation error should auto-consult AI, or notify devs, so they can try to improve it - https://phabricator.wikimedia.org/T341420 (10Izno)