[08:39:10] !log wikipathways Increased quota to 80GB space and 1 floating ip (T294395 T294318) [08:39:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikipathways/SAL [08:39:14] T294395: Request increased quota for wikipathways Cloud VPS project (disk) - https://phabricator.wikimedia.org/T294395 [08:39:14] T294318: Request increased quota for wikipathways Cloud VPS project (floating ip) - https://phabricator.wikimedia.org/T294318 [09:14:38] !log tools.integraality Deploy 0a4560a and de296cd (T294570) [09:14:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [09:16:10] !log tools.integraality Install requirements in py37 venv (T294570) [09:16:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [09:28:25] !log tools.integraality Deploy 2260c8f (follow up to T294570) [09:28:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [09:49:04] !log tools.integraality Deploy af49bab (T236590 and T273226) [09:49:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [09:49:51] !log tools.integraality Deploy 40c8cd3 [09:49:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [10:19:46] mutante: RE: PCC for cloud VMs. You need to make facts available from the puppetmaster the VM is using, see `modules/puppet_compiler/files/compiler-update-facts` [15:27:41] !log striker `sudo touch .no-puppet-checks` on striker-uwsgi03 because I don't care that an apt package is not being installed [15:27:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Striker/SAL [16:09:27] anyone know how the user_properties_anon view is supposed to work? [16:10:09] based on maintain-views.yaml I'd expect it to be available for wikis in medium.dblist and small.dblist, but that doesn't appear to be the case (at least not consistently) [16:10:42] https://quarry.wmcloud.org/query/59576 it works on huwiki, which is in large.dblist [16:11:16] but not thwiki (medium, https://quarry.wmcloud.org/query/59573) or thwikinews_p (small, https://quarry.wmcloud.org/query/59578) [16:11:33] or enwiki (large, https://quarry.wmcloud.org/query/59577) [16:15:17] AntiComposite: I think this is the config that builds the view -- https://github.com/wikimedia/puppet/blob/5b9e59a44ed3432c23227e7b1a64562c02047afe/modules/profile/templates/wmcs/db/wikireplicas/maintain-views.yaml#L833-L837 -- but I don't remember what the "limit: 2" part there means. [16:15:34] should be https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/production/modules/profile/files/wmcs/db/wikireplicas/views/maintain-views.py#194 [16:15:52] at least, that's the only place where limit is mentioned in maintain-views.py [16:16:39] *nod* [16:22:49] that size is set to `db_info.get("size", None)` -- https://github.com/wikimedia/puppet/blob/5b9e59a44ed3432c23227e7b1a64562c02047afe/modules/profile/files/wmcs/db/wikireplicas/views/maintain-views.py#L495 which should I think be the value from https://github.com/wikimedia/puppet/blob/5b9e59a44ed3432c23227e7b1a64562c02047afe/modules/profile/templates/wmcs/db/wikireplicas/maintain-views.yaml#L24-L31 depending on which dblist the wikidb was [16:22:49] found in. [16:23:26] So yeah, I agree that in theory this should be small.dblist + medium.dblist dbs [16:25:15] I wonder if it has something to do with stale meta_p data. See https://github.com/wikimedia/puppet/commit/e492c0a9933d553548983d230ce58c49b3e1fce0 for the other bit from the config about the allow list of props to expose [16:25:59] huwiki has been in large.dblist since 2015 https://phabricator.wikimedia.org/rOMWC77560147969945121697192eab093c840f7a8976 [16:26:03] meta_p is local to each wiki replica instance and not replicated from elsewhere [16:26:43] so if wiki replica nodes are added or rebuilt the meta_p data they have can drift or be completely missing [16:27:11] if properties_anon_whitelist was empty, or irrelevant, or whatever, wouldn't the view still get created but with no data available? [16:29:50] maybe? there are several bare returns to exit from do_customview -- https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/production/modules/profile/files/wmcs/db/wikireplicas/views/maintain-views.py#188 -- I haven't reasoned them all out for this case [16:33:12] I'm actually now wondering if we are only provisioning meta_p on the s7 replica nodes in the latest wiki replica cluster design... [16:34:37] https://github.com/wikimedia/operations-cookbooks/blob/84d104d0f2f0246d5a3a23a8357493ce8871840a/cookbooks/sre/wikireplicas/add-wiki.py is the new hotness for maintaining that table and it looks to be restricting it to s7. [16:35:00] This could really be an "action at a distance" problem that we didn't notice when building the latest cluster. [16:36:06] AntiComposite: that's as far into the rabbit hole as I can go right now. sounds like it would be worth a phab task and maybe a cut-n-paste of some of this initial tracing [16:36:11] yup, and since the table is actually missing, it fails check_customview_source and the view doesn't get created [16:36:28] I'll write one up [16:45:24] bd808, https://phabricator.wikimedia.org/T294652 [17:19:45] !log cloudstore Fixed stuck apt configuration on cloudstore-client-1 which was causing puppet failure alerts [17:19:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Cloudstore/SAL [17:25:41] I created a jupyterlab extension to add PAWS-public links https://github.com/toolforge/paws/pull/96 [17:38:36] * bd808 makes grumpy noises about "lab" being in the new extension name ;) [18:12:19] arturo: thank you. But in this case I wasn't using my own local master [23:20:50] !log tools.robokobot deleting giant logfile 'virgule.err' to make space on NFS [23:20:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.robokobot/SAL [23:21:32] !log tools.ifttt deleting ifttt/www/python/src/ifttt.log in order to free up nfs space [23:21:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.ifttt/SAL [23:22:50] !log tools.render deleting error.log in order to free up nfs space [23:22:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.render/SAL [23:23:01] !log tools.render-test deleting error.log in order to free up nfs space [23:23:01] andrewbogott: Unknown project "tools.render-test" [23:23:12] !log tools.render-tests deleting error.log in order to free up nfs space [23:23:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.render-tests/SAL [23:29:55] !log tools.persondata deleting logs/person_bkl2.out in order to free up nfs space [23:29:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.persondata/SAL [23:58:34] !log tools deleting all files older than 14 days in /srv/tools/shared/tools/project/.shared/cache [23:58:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [23:59:54] is there a good way to rotate uwsgi.log files?