[13:49:31] !log wikibase-registy `sudo docker kill wikibase-registry_wikibase-update_run_1` kill container not in /root/wikibase-registry/docker-compose.yml - no idea where that came from. T287492 [13:49:33] wm-bot: Unknown project "wikibase-registy" [13:50:04] grr typo [13:50:28] !log wikibase-registry `sudo docker kill wikibase-registry_wikibase-update_run_1` kill container not in /root/wikibase-registry/docker-compose.yml - no idea where that came from. T287492 [13:50:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikibase-registry/SAL [13:53:04] !log wikibase-registry `sudo docker container prune` remove dangling container meta data. T287492 [13:53:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikibase-registry/SAL [13:53:32] !log wikibase-registry `sudo docker image prune` remove dangling images. Freed up 117 MB. T287492 [13:53:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikibase-registry/SAL [13:54:40] ok, now the containers that are running on wbregistry-01 are the ones that are expected to run [14:54:21] !log reading-web-staging rebooting a11y.reading-web-staging.eqiad1.wikimedia.cloud; seems hung [14:54:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Reading-web-staging/SAL [14:57:13] !log wikibase-registry `sudo docker image prune --all` removing all unused docker images. Freed up 5.14 GB. T287492 [14:57:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikibase-registry/SAL [16:11:01] !log wikibase-registry ` sudo docker restart b6f6d2d0dd7a` Restarting wikibase-registry_wdqs-0310_1 to hopefully fix all wdqs queries being 502 [16:11:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikibase-registry/SAL [16:17:55] !log wikibase-registry ` sudo docker restart 6e997bf4a59e` Restarting wikibase-registry_wdqs-proxy_1 to hopefully fix all wdqs queries being 502 [16:17:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikibase-registry/SAL [16:18:37] !log wikibase-registry ` sudo docker restart 873f4b18478d` Restarting wikibase-registry_wdqs-frontend_1 to hopefully fix all wdqs queries being 502 [16:18:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikibase-registry/SAL [16:19:42] ok, now the queries seem to work again [16:21:10] We want to beta test our event logging on the beta cluster. For this we would need access to deployment-eventlog05.eqiad1.wikimedia.cloud What is the process to get that access? [16:25:27] michaelcochez: That hostname is missing a component (probably the project name?) [16:26:00] I took it from this page: https://wikitech.wikimedia.org/wiki/Analytics/Systems/EventLogging/TestingOnBetaCluster [16:26:57] but generally the way you get membership in a project is by requesting it from an existing project admin. Looks like that's in deployment-prep so there are lots [16:26:58] https://openstack-browser.toolforge.org/project/deployment-prep [16:27:57] oh yeah, the hostname is complete in one place and then broken in the others, I'll fix that page [16:29:21] yeah, my own access request seems to have been pretty informal ^^ https://phabricator.wikimedia.org/T188427 [16:29:53] but for some reason I can’t SSH into deployment-eventlog05.deployment-prep.eqiad1.wikimedia.cloud (with project name) either [16:30:04] Lucas_WMDE: that's because it's shut down; I think it was replaced with 08 [16:30:13] But deployment-prep conceals many mysteries [16:30:13] ah, okay [16:30:19] yeah, afaik it's just "convince a project admin to do it" [16:30:22] I looked at openstack browser but 05 was still there [16:30:23] (like, why shut down but not removed?) [16:31:01] I'd otherwise do it but I'm sort of holding my deployment-prep contributions hostage until https://phabricator.wikimedia.org/T215217#6665452 gets done or the code stewardship request gets reopened [16:32:14] yeah, michaelcochez as per majavah's comment, note that deployment-prep is erratically supported and it's not clear who's actually responsible for things [16:33:42] Ok. Should I update the 05 to 08 on the documentation anyway? [16:34:08] We only want to make sure the events we generate are correctly logged before we move to test. [16:36:54] michaelcochez: I’ve added you to the project [16:37:10] does Marta also have a Wikitech account? (if yes, it’s not linked on https://phabricator.wikimedia.org/p/Martaannaj/) [16:37:21] She does. Just a second. [16:39:02] Lucas_WMDE: That should also be Martaannaj [16:39:46] !log devtools rebooting gerrit-prod-1001; seemingly unreachable [16:39:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL [16:40:24] michaelcochez: I'd say that if you try things and they work then you should update the docs [16:40:36] Ok [16:40:46] michaelcochez: alright, Martaannaj also added [16:41:08] Thanks a million. [16:41:29] side note: does anyone know how to go from a wikitech username to the shell name? [16:41:35] I only know the other direction, via the ldap tool [16:41:50] (so I’m glad that Horizon accepted either wikitech username or shell name for project member addition ^^) [16:42:09] Lucas_WMDE: ldapsearch in a bastion [16:42:29] not sure if there are better ways, but that at least works [16:42:30] bd808: is mwv-builder-03.mediawiki-vagrant.eqiad1.wikimedia.cloud good for anything? I can't access it and the console gives a login prompt (which suggests that it never built right in the first place) [16:43:26] andrewbogott: hmmm... it once was useful but may well be sad these days. I'll take a peek [16:43:34] thanks [16:43:34] Lucas_WMDE: I typically use https://contact.toolforge.org/ [16:43:36] majavah: thanks, `ldapsearch -x "cn=$USER"` works [16:43:52] legoktm: oh, even better! thanks! [16:44:37] Is the user known on Horizon as niedzielski here? [16:46:03] he left the WMF a while back, but IIRC might be around ocassionally. Don't see them in this room though [16:46:52] Yeah, looks like Stephen left in November 2020 and his contact email is still the now disabled wmf account. [16:46:54] lots of zombie VMs in reading-web-staging which seems to have been his thing [16:47:13] !log reading-web-staging rebooting nehpets VM, OOM [16:47:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Reading-web-staging/SAL [16:49:00] !log mediawiki-vagrant Hard reboot mwv-builder-03. Unresponsive to ssh and http. [16:49:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Mediawiki-vagrant/SAL [16:52:29] !log wikicommunityhealth rebooting VM 'backend', unreachable and hanging [16:52:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikicommunityhealth/SAL [16:54:00] !log search rebooting relforge-search; it's unreachable and hanging [16:54:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Search/SAL [16:59:46] !log mediawiki-vagrant Adding cinder volume to hold /srv data. [16:59:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Mediawiki-vagrant/SAL [17:14:08] !log mediawiki-vagrant mwv-builder-03: added cinder volume, migrated /srv to it, fixed broken apt state, forced puppet run to ensure things are better now that the / partition has some free space. [17:14:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Mediawiki-vagrant/SAL [17:53:08] !log deployment-prep rebooting deployment-logstash03 as it's in an inconsistent config state [17:53:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [17:56:10] majavah: it looks like you built a new buster logstash cluster in deployment-prep; can I delete the remaining stretch VM? (deployment-logstash03) [18:01:15] I am trying to get the list of users for wikicite-citationgraph-db-1 (a trove instance). I am getting this error: "Error: Unable to get user data. Details [18:01:17] Operation not supported for datastore postgresql. (HTTP 422)" [18:04:30] andrewbogott: yeah, sure. there are still a few references to it around but the logstash install in it doesn't work anymore so deleting it shouldn't matter [18:04:39] thanks! [18:04:55] !log deleting deployment-logstash03; it's replaced with Buster instances [18:04:56] andrewbogott: Unknown project "deleting" [18:11:22] harej, looking [18:11:35] I thought I had set up a username/password for the database on creation, but it didn't work when I tried using it just now, and I have no way of verifying this [18:13:33] harej, it looks like the user/db management integration isn't all there for postgres. If you use the 'manage root access' option you can probably get yourself a psql shell to do what you need [18:13:56] (It will say that root was previously enabled because I just now twiddled it to see if it would throw an error) [18:28:37] So now I am trying to figure out how to create my database and user. psql -h fxul26hriqx.svc.trove.eqiad1.wikimedia.cloud -U root tells me that the database "root" doesn't exist. And I can't specify a database with -d because it doesn't exist yet. I feel like I'm in a Catch 22. [18:32:28] PAWS question, please redirect me if there is a better channel. PAWS is a Jupyter deploy, with additional wikimedia specific bits added in (things like useful environment variables being made available for notebooks)? When PAWS is upgraded, would that typically mean the backing Jupyter code is being upgraded? [18:34:30] harej, where are you running your command? I want to try. [18:59:45] andrewbogott: tools-sgebastion-07 [19:00:13] huh, wouldn't expect it to necessarily work outside of the wikicite project unless you set specific rules for that [19:00:18] but I'll try it! [19:01:56] harej, it doesn't look like you enabled the root account? I'll do that now [19:02:03] I thought I did? [19:02:59] And, I wouldn't be able to execute that command within the wikicite project because I have no VMs in that project, just the DB [19:04:46] oh, ok -- I guess you did, it must want to rotate the password every time [19:18:29] mdipietro: if by "PAWS is updated" you mean you mean upgrading the kubernetes cluster powering it (if not please clarify), then no, kubernetes has a stable backwards-compatible api which means that we can upgrade kubernetes fully independently from the jupyterhub bits running inside the cluster [19:22:50] harej, this is going to take me a while to sort out [19:35:19] I found a workaround: specifying postgres as the database [19:35:51] so the parameters are -U root -d postgres [19:36:18] heh, that's what I just now found :) [19:36:30] I think it just defaults to the same database as the username [19:36:52] harej, a lot of this seems pretty silly but are you able to get what you need now? [19:43:00] Yes, I created the user account I needed and now everything is working [19:48:13] Successful trove implementation!? [19:48:46] 🎉 [20:10:56] I am trying to see events logged from https://wikidata.beta.wmflabs.org/ following https://wikitech.wikimedia.org/wiki/Analytics/Systems/EventLogging/TestingOnBetaCluster . But, it appears no events are logged on the server at all (also not for other services). Does anyone have a pointer where to look? [20:11:31] Tried: the /srv/log/eventlogging folder is empty, mysql is not installed and the kafka client is also not available. [20:12:06] hiya! [20:12:27] interesting that is some old docs! [20:12:49] michaelcochez: what schema are you trying to test? [20:13:03] Server i am using is deployment-eventlog08, not 05. [20:13:38] I want to test this schema: https://gerrit.wikimedia.org/r/c/schemas/event/secondary/+/689152 [20:14:57] ah ok [20:15:06] https://wikitech.wikimedia.org/wiki/Event_Platform/Instrumentation_How_To#In_beta [20:16:24] So easiest seems to look trough the browser... right? [20:16:29] yeah def the easiest [20:16:41] Any idea what the name of the stream is in our case? [20:16:41] are these client side events michaelcochez ? [20:16:46] both [20:16:46] sent via browser js? [20:17:00] some is server side, some client. [20:17:02] ok [20:17:24] i'd watch developer console network to make sure the events are being sent to the event intaek service [20:17:29] and then also watch the eventstreams ui for your stream [20:18:33] How do I know the name of my stream? [20:18:39] oh [20:18:42] you need to declare it! [20:18:57] https://wikitech.wikimedia.org/wiki/Event_Platform/Instrumentation_How_To#Deployment [20:19:12] we probably did, but I don't know where :D [20:19:21] i just looked i don't think i see it [20:19:47] curl 'https://wikidata.beta.wmflabs.org/w/api.php?action=streamconfigs' | jq . [20:19:55] Should I link to the new instructions for events in beta from the old documentation we found? Or should the old documentation be removed altogether? [20:20:05] michaelcochez: i'll take a pass at it [20:20:14] thanks for notifying me [20:20:17] that should have been fixed long ago [20:20:18] (y) [20:21:14] hmm actually i'm going to add an outdated notice and link to eventplatform as you say [20:22:31] done [20:22:31] https://wikitech.wikimedia.org/wiki/Analytics/Systems/EventLogging/TestingOnBetaCluster [20:35:05] ottomata We have both client and server side events. Does it make sense to put them both into the same stream, or is it better to keep them separate and merge later? [20:35:30] same stream is fine [20:35:50] oh [20:35:55] wait you have 2 different schemas, right? [20:35:59] Yes [20:36:04] ok 2 different streams then [20:36:08] Ok [20:36:19] if they were the same schema... [20:36:22] michaelcochez: you could do it that way [20:36:26] i see [20:36:34] i know we already merged this, but if you wanted to have one schema you can have 2 streams with the same schema [20:36:36] or one stream [20:36:44] this is a difference to old eventlogging [20:36:48] a schema is really like a datatype [20:36:55] and a stream is an instance [20:37:35] ore in [20:37:35] https://wikitech.wikimedia.org/wiki/Event_Platform/EventLogging_legacy [20:37:37] balloons: Mostly successful. There are some parts of the Horizon interface that are apparently not fully implemented, and so I had to create my database/user account by command line, but I was having trouble using the command line because I am new to psql and haven't figured out all its mysteries. [20:47:53] So at the moment I am loading a gzipped dump into postgres from a bastion. If I am DoSing the bastion then I apologize. [20:50:48] If one of you killed the process as soon as I said that, nicely played. (Do you know of a way I could connect to the trove from outside the Wikimedia cloud?) [20:51:45] @harej: please make your life easier and just build a tiny host in your project to do your command and control things from. [20:52:32] Okay, fine! [20:53:24] Did someone create an instance named deleteme? [20:56:49] andrewbogott: did you create the deleteme instance when tinkering with trove earlier? [20:59:13] Yes [20:59:44] Can I delete it? [21:00:20] Yes please! [21:03:27] I am trying to make a change to mediawiki-config. (adding my event logging), but when pushing the change for review, I get "(commit 00131bd: invalid committer)". Do I need to be in a certain group or something to commit there? [21:08:17] Has anyone here ever used Qubes? Wild stuff [21:10:25] I solved my above issue. I had configured the email address on git differently from the one on gerrit. [21:19:29] ottomata: could you have a look at the patch created for the event logging? Should I create a separate phabricator ticket for it? [21:29:06] !log tools.admin add mdipietro as maintainer T287287 [21:29:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.admin/SAL [21:29:10] T287287: Onboard Michael DiPietro to Wikimedia Foundation as SRE in Cloud Services - https://phabricator.wikimedia.org/T287287 [21:33:22] !log tools add mdipietro as projectadmin and to sudo policy T287287 [21:33:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [23:44:52] same phab ticket is prob fine MichaelG_WMDE [23:45:49] oops sorry wrong ping