[02:16:03] Hi! I'm from Brazil, and I've working as SRE since 2016. As a Wikitionary lover, I really want to be a volunteer SRE at Wikimedia Foundation (I have 15 hours avaiblable per week). If it's possible, please let me know how I could help. [02:35:48] basis, have you looked at https://wikimediafoundation.org/about/jobs/ already? [02:41:03] gry: At the moment, I want to be a volunteer. [02:43:03] oh! okay. [02:43:59] there is https://www.mediawiki.org/wiki/Wikimedia_Site_Reliability_Engineering [02:44:39] and #wikimedia-sre channel, which may be a great place to ask such a question. probably. [09:05:53] !log toolsbeta testing calico 3.18 upgrade - T280342 [09:05:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [09:05:57] T280342: Upgrade Calico to 3.18 - https://phabricator.wikimedia.org/T280342 [09:17:36] !log tools copying calico 3.18.4 images from docker hub to docker-registry.tools.wmflabs.org T280342 [09:17:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [09:17:41] T280342: Upgrade Calico to 3.18 - https://phabricator.wikimedia.org/T280342 [10:35:46] !log paws undeploy old ingress T266050 [10:35:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [10:35:50] T266050: Build Prometheus service for use by all Cloud VPS projects and their instances - https://phabricator.wikimedia.org/T266050 [10:38:04] oops, wrong task [10:38:22] !log paws correction: undeploy old ingress T264221 [10:38:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [10:38:24] T264221: Upgrade the nginx ingress controller in Toolforge (and likely PAWS) - https://phabricator.wikimedia.org/T264221 [11:56:03] !log puppet-diffs added Jelto as user and admin for access to puppet compilers and syncing facts [11:56:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Puppet-diffs/SAL [12:17:13] !log admin doing ceph outage tests on codfw1 (fyi) [12:17:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [13:36:02] !log tools.vpsalertmanager upgrading karma installation to latest upstream version (0.87) [13:36:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.vpsalertmanager/SAL [14:24:26] majavah: are you still upgrading karma? (vpsalertmanager seems to be down) [14:28:01] dcaro: works for me [14:28:22] oh, refreshed and it works, I was getting 503 proxy error [14:28:36] nm :) [15:29:25] hi, I would like to mount /public/dumps/ in a cloud-vps instance for a research project where we need access to the public dumps. I added more details in this ticket: https://phabricator.wikimedia.org/T286635 could anyone help me figure it out? from what I understand, in principle, this should be possible, but didnt manage so far. thanks in advance. [15:59:13] Did puppet run etc? [16:00:47] Reedy: not sure what that means. do I have to do that manually? [16:00:58] You can... Or wait a little while [16:01:27] `sudo puppet agent --test` (without the ` obviously) to run it manually [16:01:57] I think by default it runs ever 30 minutes [16:04:13] when you login it should tell you when it last ran puppet too [16:13:31] * Reedy fixes his cloud root config [16:15:19] i think nfs needs to be enabled for the project somewhere in prod hiera too [16:15:49] https://github.com/wikimedia/puppet/blob/production/modules/profile/manifests/wmcs/nfsclient.pp seems to have a few conditionals for projects [16:18:24] Though, per majavah I think a patch like https://gerrit.wikimedia.org/r/c/operations/puppet/+/679790/1/modules/labstore/templates/nfs-mounts.yaml.erb is needed [16:18:48] from https://phabricator.wikimedia.org/T279558#7002552 on the linked task :D [16:23:14] patch incoming [16:23:32] Ohh thanks Reedy. I just saw this [16:25:50] balloons: will need someone else to merge though (I can't do it) [16:26:22] https://gerrit.wikimedia.org/r/704381 [16:30:47] Reedy, I gave you my +1, but I too cannot merge [16:30:57] heh [16:31:11] :-) [17:10:43] Reedy: thanks. patch still needs to be merged, right? [17:15:13] Yup [18:03:08] I need to put some time into (semi)automating python package upgrades for my tools [19:26:12] mgerlach: [19:26:13] root@wikinav:~# mount | grep dumps [19:26:13] labstore1006.wikimedia.org:/ on /mnt/nfs/dumps-labstore1006.wikimedia.org type nfs4 (ro,noatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=300,retrans=3,sec=sys,clientaddr=172.16.5.241,local_lock=none,addr=208.80.154.7) [19:26:13] labstore1007.wikimedia.org:/ on /mnt/nfs/dumps-labstore1007.wikimedia.org type nfs4 (ro,noatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=300,retrans=3,sec=sys,clientaddr=172.16.5.241,local_lock=none,addr=208.80.155.106) [19:38:03] hey folks, can anyone approve my toolforge request? *---* [19:39:14] looking [19:41:20] stanglavine: done [19:41:34] thank you majavah :) [19:49:29] thanks majavah! [20:22:29] !log wikicommunityhealth set quota to 600gb, 20 cores, 48g RAM T284687 [20:22:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikicommunityhealth/SAL [20:22:32] T284687: Resource allocation request for the wikicommunityhealth project - https://phabricator.wikimedia.org/T284687 [20:32:05] !log swift quota set --instances 15 --cores 15 --ram 32G T286570 [20:32:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Swift/SAL [20:32:08] T286570: Request increased quota for swift Cloud VPS project - https://phabricator.wikimedia.org/T286570 [21:08:44] !log admin restarting lots of openstack services while trying to resolve T286675 [21:08:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [21:08:54] T286675: Intermittent openstack API failures - https://phabricator.wikimedia.org/T286675 [23:29:12] !log tools mounted nfs on tools-services-05 and backing up aptly to NFS dir T286003 [23:29:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [23:29:16] T286003: NFS backups on aptly aren't set up for tools-services-05.tools.eqiad1.wikimedia.cloud and jessie packages are gone - https://phabricator.wikimedia.org/T286003