[05:11:37] <Amir1>	 https://phabricator.wikimedia.org/T285608#7515860 🤦‍♂️
[05:37:58] <Amir1>	 djvu clean up 71% done
[06:25:56] <marostegui>	 I have finished upgrading all clouddb* hosts
[06:25:59] <marostegui>	 finally
[07:18:27] <marostegui>	 https://grafana.wikimedia.org/d/000000278/mysql-aggregated?viewPanel=1&orgId=1&from=now-24h&to=now&var-site=eqiad&var-group=core&var-shard=All&var-role=All&refresh=1m
[07:18:28] <marostegui>	 wow
[07:18:58] <marostegui>	 enwiki host: https://grafana.wikimedia.org/d/000000273/mysql?viewPanel=16&orgId=1&from=now-2d&to=now&var-job=All&var-server=db1163&var-port=9104
[07:20:07] <marostegui>	 It sort of matches the release from yesterday
[07:20:18] <marostegui>	 20:43 mwdebug-deploy@deploy1002: helmfile [codfw] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
[07:20:18] <marostegui>	 20:43 jhuneidi@deploy1002: rebuilt and synchronized wikiversions files: all wikis to 1.38.0-wmf.9 refs T293950
[07:20:18] <marostegui>	 20:39 mwdebug-deploy@deploy1002: helmfile [eqiad] Ran 'sync' command on namespace 'mwdebug' for release 'pinkunicorn' .
[07:20:18] <marostegui>	 20:31 jhuneidi@deploy1002: Synchronized php: group1 wikis to 1.38.0-wmf.9 refs T293950 (duration: 01m 03s)
[07:20:18] <marostegui>	 20:30 jhuneidi@deploy1002: rebuilt and synchronized wikiversions files: group1 wikis to 1.38.0-wmf.9 refs T293950
[07:20:18] <stashbot>	 T293950: 1.38.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T293950
[07:20:44] <marostegui>	 Looks like someone has raised this concern already https://phabricator.wikimedia.org/T293950#7515673
[07:20:47] <marostegui>	 I am going to comment there
[07:23:35] <marostegui>	 https://phabricator.wikimedia.org/T293950#7515948
[07:23:38] <marostegui>	 I have set this to UBN
[07:23:39] <marostegui>	 Amir1: ^
[07:24:41] <RhinosF1>	 marostegui: I assume rollback best?
[07:24:55] <marostegui>	 RhinosF1: Yes, I don't want to go like this into the weekend
[07:25:10] <RhinosF1>	 marostegui: I will do some pinging
[07:25:40] <marostegui>	 RhinosF1: thanks - I am fine if someone wants to investigate but if not (or it is not easy to find) I would prefer to revert
[07:26:26] <RhinosF1>	 I copied ^ across
[07:26:48] <marostegui>	 thanks
[07:27:30] <RhinosF1>	 Jeena looking
[07:27:35] <marostegui>	 thanks!
[07:29:03] <RhinosF1>	 Np
[07:29:21] <Amir1>	 back, I was afk
[07:29:57] <Amir1>	 looking
[07:31:04] <RhinosF1>	 Amir1: see -releng too
[07:31:43] <marostegui>	 From what I can see there're lots of SELECT `slot_revision_id` , `s ...  WHERE `slot_revision_id` = ? 
[07:32:58] <Amir1>	 that's pretty generic, it can be basically any function 
[07:32:59] <Amir1>	 ugh
[07:33:07] <marostegui>	 yeah, sys isn't very useful in this case
[07:34:42] <marostegui>	 Lots of SELECT /* MediaWiki\Page\PageStore::loadPageFromConditions  */  page_id,page_namespace,page_title,page_is_redirect,page_is_new,page_touched,page_links_updated,page_latest,page_len,page_content_model,page_restrictions  FROM `page`
[07:34:42] <marostegui>	   WHERE page_id = XXXX  LIMIT 1
[07:35:37] <marostegui>	 224723 of those queries in 10 seconds
[07:36:08] <marostegui>	 I don't know if that is a normal rate for that query, so just throwing the info I am finding 
[07:38:59] <marostegui>	 So from what I can tell it is limited to reads, writes seem to be stable
[07:43:31] <Amir1>	 oh thanks, that's useful 
[10:23:06] <jynus>	 Did you see the "Looking for API Stakeholders" mail? Should we be worried?
[10:30:45] <jynus>	 with the extended maintenance on codfw, I think I am going to pause any codfw media backups and focus on eqiad for now
[10:50:15] <RhinosF1>	 jynus: given the lack of detail on that, I doubt anyone should be worried
[11:23:44] <marostegui>	 jynus: yeah, saw that, but waiting for more details on that same thread