[00:07:48] <wikibugs>	 (03CR) 10Kevin Bazira: [C: 03+1] test: liftwing manual testing on deployment server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/884292 (https://phabricator.wikimedia.org/T327787) (owner: 10Ilias Sarantopoulos)
[00:46:58] <wikibugs>	 10Machine-Learning-Team: WikiGPT Experiment - https://phabricator.wikimedia.org/T328494 (10kevinbazira) @isarantopoulos, I could not find Chris' username on Toolforge.  @calbon, please [[ https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Quickstart#Getting_started_with_Toolforge_-_Quickstart | create a Toolfo...
[01:19:37] <wikibugs>	 10Machine-Learning-Team, 10ORES, 10MediaWiki-Special-pages: Highlight edits that are likely to have problems different colors on User contributions page - https://phabricator.wikimedia.org/T328728 (10Aklapper)
[07:20:05] <wikibugs>	 10Machine-Learning-Team, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: codfw row A switches upgrade - https://phabricator.wikimedia.org/T327925 (10Marostegui)
[07:30:58] <wikibugs>	 10Machine-Learning-Team, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: codfw row A switches upgrade - https://phabricator.wikimedia.org/T327925 (10Marostegui)
[08:43:52] <wikibugs>	 10Machine-Learning-Team, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: codfw row A switches upgrade - https://phabricator.wikimedia.org/T327925 (10ayounsi)
[08:49:36] <wikibugs>	 10Machine-Learning-Team, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: codfw row A switches upgrade - https://phabricator.wikimedia.org/T327925 (10MoritzMuehlenhoff)
[08:52:11] <wikibugs>	 10Machine-Learning-Team, 10CirrusSearch, 10Discovery-Search: Add outlink topic model predictions to CirrusSearch indices - https://phabricator.wikimedia.org/T328276 (10achou) > If the outlink model is not enabled on the Draft namespace it might make to leave this problem out for now and not think too much ab...
[08:56:11] <wikibugs>	 (03CR) 10Elukey: "Really great work, I just added some nits here and there for clarity, but the rest LGTM!" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/884292 (https://phabricator.wikimedia.org/T327787) (owner: 10Ilias Sarantopoulos)
[08:58:20] <wikibugs>	 10Machine-Learning-Team: Add a new outlink topic stream for EventGate main - https://phabricator.wikimedia.org/T328899 (10achou)
[09:02:43] <elukey>	 aiko: o/
[09:02:59] <elukey>	 ok if I deploy the new outlink images to prod?
[09:03:07] <aiko>	 elukey: morning luca! 
[09:04:06] <aiko>	 elukey: yeah no problem
[09:04:10] <elukey>	 super thanks :)
[09:05:45] <elukey>	 aiko: done! Also deployed the up-to-date nsfw one. The revert-risk image is updated only in staging, we can do it anytime
[09:08:50] <aiko>	 elukey: nice <3 the revert-risk is fine in staging for now
[09:50:47] <isaranto>	 hey all!
[09:53:36] <klausman>	 morning!
[09:54:45] <elukey>	 hello folks!
[09:57:47] <wikibugs>	 10Machine-Learning-Team, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 12 others: codfw row A switches upgrade - https://phabricator.wikimedia.org/T327925 (10akosiaris) >>! In T327925#8587186, @Marostegui wrote: >>>! In T327925#8587104, @Joe wrote: >> I would suggest that instead of handling...
[10:11:27] <wikibugs>	 10Machine-Learning-Team, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 12 others: codfw row A switches upgrade - https://phabricator.wikimedia.org/T327925 (10Marostegui) Cool! I am going to repool the hosts then :)
[11:36:33] <wikibugs>	 (03PS20) 10Ilias Sarantopoulos: test: liftwing manual testing on deployment server [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/884292 (https://phabricator.wikimedia.org/T327787)
[11:38:07] <wikibugs>	 (03CR) 10Ilias Sarantopoulos: "Made the changes requested. I added a comment on README that httpbb is the preferred way and if something is not supported the recommended" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/884292 (https://phabricator.wikimedia.org/T327787) (owner: 10Ilias Sarantopoulos)
[11:40:17] <wikibugs>	 (03CR) 10Elukey: [C: 03+1] "Thanks!" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/884292 (https://phabricator.wikimedia.org/T327787) (owner: 10Ilias Sarantopoulos)
[11:45:27] <wikibugs>	 10Machine-Learning-Team, 10Data-Engineering, 10Event-Platform Value Stream: Add a new outlink topic stream for EventGate main - https://phabricator.wikimedia.org/T328899 (10Aklapper)
[11:55:32] <isaranto>	 created this patch to remove the drafttopic deployments that aren't deployed on ORES. iirc only en is supported. let me know if I remember well https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/886869
[12:14:17] <wikibugs>	 10Lift-Wing, 10Machine-Learning-Team: Connect Outlink topic model to eventgate - https://phabricator.wikimedia.org/T315994 (10achou) 05Open→03Resolved I just realized I didn't mark this task as RESOLVED. I've created a task T328899 to work on a new stream for the outlink topic model. There is another task...
[12:18:15] * elukey run + lunch!
[12:19:30] <wikibugs>	 10Machine-Learning-Team, 10Data-Engineering, 10Event-Platform Value Stream: Add a new outlink topic stream for EventGate main - https://phabricator.wikimedia.org/T328899 (10achou)
[12:26:04] * klausman lunch
[12:31:13] <wikibugs>	 10Machine-Learning-Team: Remove additional drafttopic deployments - https://phabricator.wikimedia.org/T328916 (10achou)
[12:41:33] <wikibugs>	 10Machine-Learning-Team: Remove additional drafttopic deployments - https://phabricator.wikimedia.org/T328916 (10isarantopoulos) https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/886869
[12:42:35] <aiko>	 isaranto: o/ looks like you added a wrong Phab id for https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/886869
[12:47:32] <wikibugs>	 10Machine-Learning-Team, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 12 others: codfw row A switches upgrade - https://phabricator.wikimedia.org/T327925 (10dcaro)
[12:51:40] <wikibugs>	 10Machine-Learning-Team, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 12 others: codfw row A switches upgrade - https://phabricator.wikimedia.org/T327925 (10Marostegui) I am repooling all the databases since we are going to fully depool codfw for reads.
[12:55:56] <isaranto>	 pff.. and I was wondering why it didn't comment on the task 🤷 . thanks for pointing that out aiko: Fixed it!
[13:00:26] <aiko>	 +1'd :)
[13:11:04] <aiko>	 isaranto: are you gonna deploy the drafttopic change? or I can do it :)
[13:11:20] <aiko>	 not sure if the additional drafttopic deployments will get deleted after we deploy, or they need to be manually deleted by SRE
[13:12:35] <isaranto>	 I can deploy them , we need a +2 from sre first. going for lunch and will check back later
[13:13:13] <aiko>	 ok!
[13:18:11] <klausman>	 isaranto: +2'd you drafttopic cleanup. Jerkins should be doing its thing right now
[13:18:24] <isaranto>	 ty!!
[13:38:21] <wikibugs>	 10Machine-Learning-Team: Remove additional drafttopic deployments - https://phabricator.wikimedia.org/T328916 (10isarantopoulos) a:03isarantopoulos
[14:09:30] <isaranto>	 additional drafttopic deployments have been removed!
[14:09:45] <wikibugs>	 10Machine-Learning-Team, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 12 others: codfw row A switches upgrade - https://phabricator.wikimedia.org/T327925 (10ssingh)
[14:39:38] <wikibugs>	 10Machine-Learning-Team, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 12 others: codfw row A switches upgrade - https://phabricator.wikimedia.org/T327925 (10MatthewVernon) If we're "just" depooling codfw it's worth noting we will still need to depool the affected ms-fe* nodes (since mw alway...
[14:46:35] <chrisalbon>	 Morning all!
[14:48:08] <klausman>	 eyooo
[14:50:32] <elukey>	 o/
[14:53:03] <isaranto>	 ο/
[14:55:50] <aiko>	 \o
[16:19:27] <wikibugs>	 10Machine-Learning-Team, 10Patch-For-Review: Implement new mediawiki.revision-score streams with Lift Wing - https://phabricator.wikimedia.org/T328576 (10elukey) Decided to start with goodfaith, and filed two code reviews: * one to allow autoscaling for enwiki and wikidatawiki, the rest seems not needing the s...
[16:21:06] <elukey>	 the code reviews for the first stream (mediawiki.revision-score-goofaith) are out :)
[16:21:31] <elukey>	 if anybody wants to help in testing etc.. please let me know!
[16:31:48] <isaranto>	 elukey: I am interested in helping, testing etc!
[16:36:06] <isaranto>	 I was reviewing your patches and my laptop restarted...
[16:36:29] <isaranto>	 ¯\_(ツ)_/¯ 
[16:47:27] <elukey>	 ahahahah do you think it is a sign??
[16:51:46] <klausman>	 Mere hardware can not withstand the power of Luca's patches!
[16:53:29] <elukey>	 yeah sure 
[16:53:49] <elukey>	 I meant that it may be a wrong sign and we should revisit our choices :D
[16:54:12] <klausman>	 If nobody time travels from the future to stop you, how bad can it be?
[16:55:46] <isaranto>	 it may be a good sign! 😃
[17:21:53] <wikibugs>	 10Machine-Learning-Team, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 12 others: codfw row A switches upgrade - https://phabricator.wikimedia.org/T327925 (10jbond)
[17:48:48] <klausman>	 elukey: nice work on the presentation!
[17:49:39] <elukey>	 klausman: <3 thanks
[17:51:12] <elukey>	 going afk folks, have a nice rest of the day :)
[17:51:17] <klausman>	 \o
[18:08:12] <isaranto>	 me2, cya tomorrow!
[22:51:49] <wikibugs>	 10Machine-Learning-Team, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 12 others: codfw row A switches upgrade - https://phabricator.wikimedia.org/T327925 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=e0e96453-af13-467f-a75e-ebd1c4122a32) set by bking@cumin2002 for 1 day...