[08:31:38] <jynus>	 there just has been some s2 query spikes: https://grafana.wikimedia.org/d/000000278/mysql-aggregated?orgId=1&refresh=1m&var-site=eqiad&var-group=core&var-shard=s2&var-role=All&from=1639103492641&to=1639125092641
[08:33:34] <marostegui>	 https://grafana-rw.wikimedia.org/d/000000273/mysql?viewPanel=3&orgId=1&refresh=1m&from=now-24h&to=now&forceLogin=true&var-job=All&var-server=db1162&var-port=9104
[08:35:51] <jynus>	 I only see MediaWiki\Revision\RevisionStore::loadSlotRecords at the moment, so "page visits" maybe?
[08:36:06] <jynus>	 to itwiki?
[08:36:13] <marostegui>	 I am also seeing some API queries to itwiki
[08:36:20] <marostegui>	 scanning 500k rows
[08:36:24] <jynus>	 could be a bot
[08:37:31] <marostegui>	 yeah, most of the queries are for itwiki from what I can see
[08:40:31] <jynus>	 A bot called "MediaWikiCrawler-Google/2.0" doing api calls?
[08:41:19] <marostegui>	 pretty much all the queries on db1162 are for itwiki
[08:42:13] <jynus>	 the ip space seems google owned
[08:42:33] <jynus>	 although it could be google cloud
[08:42:39] <marostegui>	 should we ask traffic to block it?
[08:42:51] <jynus>	 let's ask what other know about it
[09:59:09] <jynus>	 db backups are about to be back into a good state, codfw commons media backups are at 25%
[10:18:12] <Amir1>	 jynus: that means parsing (edit or uncached view), that function needs caching T297147
[10:18:12] <stashbot>	 T297147: RevisionStore::newRevisionSlots() needs a cache - https://phabricator.wikimedia.org/T297147
[10:18:22] <Amir1>	 there is already a patch for it :)
[10:29:11] <Amir1>	 I reworked the db error dashboard and added both to logstash home page
[10:30:27] <kormat>	 Amir1: very cool :)
[10:30:41] <Amir1>	 kormat: ^^ <3
[11:08:52] <jynus>	 all backups green now
[11:15:07] <Amir1>	 btw, I'm starting the clean up of old rows of flaggedtemplates. I think this will make it much smaller https://phabricator.wikimedia.org/T296380#7562464
[11:15:25] <Amir1>	 Running it on alswiki now but it's taking a long time :(
[11:16:13] <Amir1>	 I think this will upwards of 1B rows
[11:17:30] <Amir1>	 *will delete
[11:28:26] <Amir1>	 oh and this might cause alerts in backups ^
[11:31:48] <jynus>	 alerts why? you are only doing deletes, right?
[11:41:39] <jynus>	 ah, I get it now, because of size changes
[11:42:04] <jynus>	 that is ok, as long as we see it is intended, it will be easy to spot :-)
[11:42:39] <Amir1>	 :D yup, let's see how much it will delete
[12:23:05] <jynus>	 I will briefly disconnecto to move for lunch. After it, I think I will increase disk size of backup2* hosts.
[12:23:59] <jynus>	 as backup is like 30% done, but disk usage is at 50%, so better increase it so it doesn't alert during Christmas
[14:07:15] <marostegui>	 I will restart mysql on db1123 on Monday to clear that warning about memory usage
[14:14:06] <jynus>	 :-(
[14:28:35] <Amir1>	 I can't believe the clean up of flaggedtemplates for alswiki (a super tiny wiki with 80k pages) is taking five hours now and it's not even half-way done.  
[14:44:57] <kormat>	 huh. i think i have a wmfdb PoC replacement for mysql.py implemented. that was both worse and better than i expected.
[16:08:42] * Emperor offers to declare it "too OOPy" ;-)
[17:32:14] <jynus>	 Now s1 with the QPS 100% increase
[17:34:01] <jynus>	 same query patterns as before
[18:05:03] <jynus>	 seems to be trending down: https://grafana.wikimedia.org/d/000000273/mysql?viewPanel=16&orgId=1&refresh=1m&var-job=All&var-server=db1163&var-port=9104&from=1639148680805&to=1639159480805
[18:06:23] <jynus>	 backup2* maintenance seems to be completed, leaving it here for the week, have a nice weekend