[09:36:23] !log paws restarted reload-acme-chief-backend.service to ensure certs are refreshed [09:36:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [15:09:36] !log quarry deploying truncation for history entries 9ac970d861c5c12dbbafb065ce45ad1ba4fd5fec [15:09:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [15:17:08] !log quarry deploying entry for blank queries https://gerrit.wikimedia.org/r/c/analytics/quarry/web/+/791779 92de0d5737b4d7a611e4eb635794988f818ce4cd [15:17:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [15:19:53] Hello, my name is Mary and am working with the Abstract Wiki team. Does anyone know if a service running on beta cluster can report metrics via grafana or other portals? Thanks! [15:26:02] !log quarry deploying link to database names https://gerrit.wikimedia.org/r/c/analytics/quarry/web/+/752034 6322dc735d465b9e6e65032a60aabf129d3073bc [15:26:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [15:35:00] There's currently an outage on openstack horizon, preventing users from retrieving the VM details, we are working on it [15:48:41] !log quarry 176506: Remember recent queries filter last used by a user. | https://gerrit.wikimedia.org/r/c/analytics/quarry/web/+/176506 3763507f1f7faa3bb44b196046fc5d153ce03924 [15:48:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [16:08:33] maryyang: sort of, but not well supported. T241285 has some info about the current state of statsd/graphite for the beta cluster project. [16:08:34] T241285: Deployment-prep should host its own statsd/graphite server - https://phabricator.wikimedia.org/T241285 [16:34:16] Thanks for the info! To clarify, it sounds like graphite still contains metrics from beta cluster but they haven't received any updates in months? [16:36:54] maryyang: I think that the configuration was updated and mediawiki specific data is reaching that service, but I have not personally verified that. [16:40:16] I see...I will do a little more digging. Thank you! [17:11:43] openstack-browser stopped showing puppet classes and clicking other links seems slow / working sometimes. should I make a ticket or ongoing work? [17:12:01] f.e. https://openstack-browser.toolforge.org/puppetclass/ seems empty [17:12:40] I see backlog about outage on horizon. so maybe it's that then ack [17:13:17] ok, i'll just check in later again [17:25:20] * dcaro off [17:32:30] !log quarry 788438: Use vars.qrun_id when stopping query | https://gerrit.wikimedia.org/r/c/analytics/quarry/web/+/788438 43f7f0c9f56ac01f1a92b200769ddbe782055381 [17:32:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [17:50:48] mutante: I think we need a bug report for the openstack-browser things. I don't have time to dig in deeper at the moment, but it looks to me like the puppet-enc backend is now requiring auth even for browsing and that is making all the anon queries done by the tool fail. This is possibly unintended fallout of work taavi has been doing to add better auth to that puppet-enc service. [17:51:58] the app is doing things that are functionally `curl https://puppet-enc.cloudinfra.wmcloud.org:8143/v1/roles` -- https://phabricator.wikimedia.org/source/tool-keystone-browser/browse/master/keystone_browser/puppetclasses.py [17:57:11] bd808: ACK! will do that in a bit. thanks [17:59:45] that sounds like the most likely explanation, I thought I already migrated openstack-browser to use keystone authentication with puppet-enc :/ [18:00:40] patches welcome, the proxies.py might have useful code you can almost copy-paste. otherwise will try to find some time to fix that soon-ish, but no promises due to irl priorities [18:05:27] *nod* looks like the `proxy_client` magic from https://phabricator.wikimedia.org/source/tool-keystone-browser/browse/master/keystone_browser/proxies.py is what puppetclasses.py also needs. [19:02:56] !log deployment-prep bking@deployment-elastic06 halted deployment-elastic06 in beta ES cluster; will decom on Friday T299797 [19:03:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [19:03:01] T299797: Deploy new bullseye elastic cluster nodes on deployment-prep - https://phabricator.wikimedia.org/T299797 [19:30:59] !log deployment-prep bking@deployment-elastic07 halted deployment-elastic07 in beta ES cluster; will decom on Friday T299797 [19:31:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [19:31:03] T299797: Deploy new bullseye elastic cluster nodes on deployment-prep - https://phabricator.wikimedia.org/T299797 [19:39:32] I tagged a ticket "cloud-services" but seems like that doesn't make it show up here? [19:39:44] but it's not "beta-cluster-infrastructure" [20:22:42] mutante: any chance you're looking for #wikimedia-cloud-feed ? [20:24:52] andrewbogott: oh, if that's separated then that explains why. personally I'd love to have tickets and chat in the same place but that's irrelevant then. thanks [20:25:22] It was getting too crowded in here for the humans so we banished all the bots. It's a hard balance to get right. [23:30:55] !log tools.svgtranslate-test Fixed up deploy.sh command in jobs.yaml [23:30:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.svgtranslate-test/SAL