[07:32:57] https://youtube.com/shorts/7qK6cXNI4Dw/podcast@BABUCHACKO94944 [08:16:10] Hello [08:17:01] Can someone help me to visualize Administrator timeline with EasyTimeline for a wikimedia project? [11:59:09] ALL: I'm rebooting the cloud-vps bastions in a few minutes, any cloud-vps sessions will be interrupted [12:07:57] !log deployment-prep rebooting all servers for T385264 [12:08:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [12:08:02] T385264: VM live migration failing for many/most VMs - https://phabricator.wikimedia.org/T385264 [12:10:00] https://usercontent.irccloud-cdn.com/file/8xqrjlPa/image.png [12:10:44] xD [12:20:41] !log puppet-diffs hard rebooted 6 workers for T385264 [12:20:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Puppet-diffs/SAL [12:20:43] T385264: VM live migration failing for many/most VMs - https://phabricator.wikimedia.org/T385264 [13:45:14] !log admin cold-migrating all remaining VMs in T385264 except for 'integration' and 'tools' VMs [13:45:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [13:45:19] T385264: VM live migration failing for many/most VMs - https://phabricator.wikimedia.org/T385264 [14:06:50] !log tools cold-migrating tools-proxy-8 for T385264; will cause a brief toolforge outage [14:06:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [14:06:53] T385264: VM live migration failing for many/most VMs - https://phabricator.wikimedia.org/T385264 [14:23:36] !log integration depooling, rebooting, repooling integration workernodes [14:23:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Integration/SAL [14:25:21] fyi the deployment-prep db starts in read only so beta is read only post the reboots [14:28:05] RhinosF1: want to switch it to r/w? Or do I need to do that? [14:28:24] andrewbogott: i do not have the permissions for that [14:29:11] do you know what host name you're talking about? deployment-rdb01? [14:29:58] not that one [14:29:59] one sec [14:31:43] deployment-db11.deployment-prep.eqiad1.wikimedia.cloud andrewbogott [14:32:02] going off https://github.com/wikimedia/operations-mediawiki-config/blob/master/wmf-config/db-labs.php [14:34:14] RhinosF1: better? [14:35:13] andrewbogott: it fixed that issue but it claims session loss [14:35:21] * RhinosF1 tries logging out while cursing beta [14:35:32] oh fun [14:35:38] i can't even log out the normal way [14:36:11] okay ye sessions have died somehow [14:36:35] and keyholder needs arming too [14:39:07] pretty sure I don't know how to do that [14:40:17] https://wikitech.wikimedia.org/wiki/Keyholder is the guide for keyholder [14:40:22] no idea about sessions [14:40:29] i don't even know where to start [14:40:36] except filing a task [14:42:13] I know about keyholder but not where the secret is [14:42:26] (and my access to the pwstore is temporarily broken so I'm not the best candidate to search for it) [14:42:56] i opened https://phabricator.wikimedia.org/T385803 [14:43:07] locations for the secrets is in the doc [14:43:26] but also you 100% shouldn't be doing this, someone should actually manage beta [14:57:43] !log lucaswerkmeister-wmde@tools-bastion-13 tools.phpunit-results-cache install service.template file (based on quickcategories) [15:01:20] !log lucaswerkmeister@tools-bastion-13 tools.stashbot bin/stashbot.sh restart [15:01:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL [15:01:29] !log lucaswerkmeister-wmde@tools-bastion-13 tools.phpunit-results-cache install service.template file (based on quickcategories) [15:01:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.phpunit-results-cache/SAL [15:03:21] !log lucaswerkmeister-wmde@tools-bastion-13 tools.phpunit-results-cache deployed 5a969d593a ("latest" branch, MR !4, T384925, not yet merged into main) for testing [15:03:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.phpunit-results-cache/SAL [15:11:32] RhinosF1: which keyholder specifically do you notice needs arming? The docs don't really correspond to the modern naming of the VMs (for instance, which VM is 'deploy-service '? There aren't any that start with that) [15:12:27] andrewbogott: scap is failing on deployment-deploy04 for the automated jobs [15:14:33] RhinosF1: ok, I can't arm keyholder on that host with the scap password or the deploy-service password or the mwdeploy password [15:15:30] * RhinosF1 copied that to the task [15:26:37] RhinosF1: I think I got it armed now, I didn't realize that one keyholder instance would have multiple passphrases [15:28:55] andrewbogott: scap looks fine now, i still can't login to enwiki beta [15:29:06] i have no clue about where to start with that though [15:29:17] maybe another db that's read-only? [15:30:05] the only host that's shutoff in that project is deployment-restbase-bullseye, surely not related to wiki sessions [15:30:15] no it won't be that [15:31:20] without any monitoring / proper log, i wouldn't be able to guess [15:31:22] tbh [15:33:11] https://grafana.wmcloud.org/dashboards/f/c-w6x7pVk/projects-beta-cluster-deployment-prep has no dashboards [18:19:55] !log bd808@tools-bastion-12 tools.extreg-wos Update public_html/toolinfo.json [18:19:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.extreg-wos/SAL