[12:37:03] some issues with toolforge? Have got bunch of downtime notitications since last Friday and increased response time: https://stats.uptimerobot.com/Z4qDZTPDGw/795623861 [12:40:33] seems specific to this tool, but I was told it's just a static html + js, so not sure why it would be slow [12:46:25] !log jeanfred@tools-sgebastion-10 tools.integraality Deploy 7adc07f [12:46:33] !log jeanfred@tools-sgebastion-10 tools.integraality Deploy ec28a13 [12:46:56] !log jeanfred@tools-sgebastion-10 tools.integraality Deploy 022fa2f (T312727) [12:46:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [12:46:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [12:47:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [13:07:44] Nikerabbit: there has been some interventions going on that might affect response times, but they don't match the times there, so I'm not sure those were the causes [13:22:14] It seems to happen almost always at 18 minutes past the hour [13:42:22] that sounds like a noisy neighbor issue [13:46:54] can I file a complaint against them? :D [13:52:38] Nikerabbit: feel free to open a task to investigate :) [13:54:38] which tags should I use for that? [13:54:42] so at least the front proxy is seeing those failures at :18 as http 499 (which is nginx for "Client Closed Request") [13:55:17] and there number of 499s seems to be fairly stable over time, it does not seem to be a toolforge-wide issue at :18 [13:59:13] and the logs for the kubernetes ingress layer are high-volume enough to only go back a few minutes, we'll need to wait until the next :18 to see if there's anything more useful there [13:59:18] Nikerabbit: #toolforge, probably [17:49:41] !log deployment-prep T355281 updated DNS zones and hiera configs - added *.m.wikipedia.beta.wmcloud.org, *.wikivoyage.beta.wmcloud.org and *.m.wikivoyage.beta.wmcloud.org domains [17:49:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [17:49:45] T355281: Set up some beta cluster wikis with different registrable domain - https://phabricator.wikimedia.org/T355281 [17:56:39] !log deployment-prep Executed "mwscript extensions/WikimediaMaintenance/addWiki.php --wiki=aawiki pl wikivoyage plwikivoyage pl.wikivoyage.beta.wmcloud.org" to add a new wiki - polish wikivoyage on beta.wmcloud.org domain [17:56:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [18:30:57] !log deployment-prep T355281 testing mediawiki-config patch Idcd9cdde4e45950916306f676a07f10f4c68e2a8, executed `scap sync-world` [18:31:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [18:31:01] T355281: Set up some beta cluster wikis with different registrable domain - https://phabricator.wikimedia.org/T355281 [19:55:50] !log lucaswerkmeister@tools-bastion-13 tools.wd-image-positions webservice restart # was unresponsive, idk why [19:55:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-image-positions/SAL [20:15:12] !log lucaswerkmeister@tools-bastion-13 tools.wd-image-positions deployed 72a68851e0 (fix TIFF) [20:15:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-image-positions/SAL [20:23:12] !log lucaswerkmeister@tools-bastion-13 tools.wd-image-positions deployed ea1e8c10e2 (fix language fallback in edit interface) [20:23:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-image-positions/SAL [22:21:36] !log bd808@tools-bastion-12 tools.schedule-deployment Built new image from d178f459 (T366794) [22:21:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.schedule-deployment/SAL [22:22:32] !log bd808@tools-bastion-12 tools.schedule-deployment Restart to pick up new container image (T366794) [22:22:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.schedule-deployment/SAL [23:12:14] is this the right place to request review for https://gerrit.wikimedia.org/r/c/operations/puppet/+/1039850 ? [23:14:03] derenrich: ping t.aavi on the patch [23:14:14] ok thanks [23:15:14] I think he actually wants to build a new prometheus for tenant use, but maybe he will let this one sneak in [23:16:29] we'd appreciate being allowed to sneak in