[15:16:57] urandom: done [15:17:21] awesome [15:17:40] confirmed, too: [15:17:44] https://www.irccloud.com/pastebin/ikcWqozJ/ [15:18:28] https://thanos.wikimedia.org/graph?g0.expr=container_memory_rss%7Bsite%3D%22codfw%22%2C%20cluster%3D%22restbase%22%2C%20id%3D~%22%2Fsystem.slice%2Frestbase.service%22%7D&g0.tab=0&g0.stacked=0&g0.range_input=6h&g0.max_source_resolution=auto&g0.deduplicate=1&g0.partial_response=0&g0.store_matches=%5B%5D [15:18:33] what a url that is [15:18:53] ...but memory consumption is now more inline with where it is elsewhere [15:18:59] nemo-yiannis: thanks for taking care of that [15:19:30] \o/ [15:19:30] np [15:19:50] did we win the last deployment of the year award ? [15:20:08] probably. [15:20:32] * Emperor certainly hopes there aren't any more... [15:20:58] unless someone else wants to play roulette in the 11th hour [15:42:32] On a random note, (also not urgent) but urandom do you know if we store the wiki content model in restbase storage? [15:43:06] We would like to see which percentage of all the PCS requests are non wikitext and it might be very useful to query cassandra instad [15:43:09] *instead [15:43:36] (i dont have access to query cassandra myself) [15:44:31] Wiki content model? [15:45:48] yes [15:45:55] like if its wikitext [15:45:57] or json [15:46:01] or css [15:48:02] (mediawiki has a bunch different models for each page) [15:49:36] which tables are we talking about here, mobile-html? [15:56:59] yes [15:57:22] nemo-yiannis: since we're talking about PCS, I assume we're referring to one or more of the 3 tables that we (re)created in T348993. I'm not sure how you'd discern the content type there, and I think that means the answer is 'no'. The data in those tables (other than the associative array of headers) is opaque to Cassandra, so if there is a structured way of determine this type, it must be encoded in that blob [15:57:23] T348993: Create new cassandra table data model for PCS - https://phabricator.wikimedia.org/T348993 [15:57:45] i meant the entries from restbase [15:57:52] but nvm, i dont think we store it either way [15:58:01] was looking locally in my restbase env [15:58:08] yeah, I'm looking at mobile-html, and the blob appears to be... well, html [15:58:49] even if it were in the headers map, that's not indexed [15:59:35] i.e. if you could, it would mean a full table scan [16:00:12] yeah [23:31:42] PROBLEM - MariaDB sustained replica lag on s1 on db1234 is CRITICAL: 2.4 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1234&var-port=9104 [23:36:08] RECOVERY - MariaDB sustained replica lag on s1 on db1234 is OK: (C)2 ge (W)1 ge 0.2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1234&var-port=9104