[10:28:33] OOI, why does the decom cookbook run swapoff -a ? [given the host is going to end up powered down anyway] [10:30:35] Emperor: because some hosts failed to wipefs without it, see T311593 [10:30:36] T311593: Decommissioning two hosts end up with: Failed to wipe swraid - https://phabricator.wikimedia.org/T311593 [10:31:24] volans: thanks. I have sped things up by getting cumin to run it in parallel on the remaining hosts I'm going to decom :) [10:32:06] if you have alternative solutions feel free to propose them, I was not heavily involved in that task/fix [10:33:03] it seems like a sensible fix, TBH. It's just a bit slow, so when doing a number of nodes at once I get impatient :) but I can do it with cumin in bulk before running the decom for future reference [10:33:24] ack [10:34:12] just as a heads up I will be upgrading another sessionstore host (and more if successful) at around 1330 UTC, depooling the service so as to avoid the badness of last time [11:38:36] https://xkcd.com/2677/ "Your one-stop shop for decentralisation" [11:42:10] Krinkle: Isn't this coinbase's motto? [11:42:14] * claime ducks [11:59:57] fyi all there is a new wmf-laptop-sre package avaliable [12:00:57] <_joe_> jbond: I wouldn't install on my computer something that contains bash scripts in part modified by me [12:01:10] lol :P [12:02:35] What's the stand-out feature of the upgrade? What's in it for me? :-) [12:03:12] <_joe_> (it was a reference to a famous quote by Groucho Marx, cfr https://en.wikiquote.org/wiki/Groucho_Marx) [12:04:15] btullis: no much at all, very minor update to wmf-update-known-hosts-production to grap https://config-master.wikimedia.org/known_hosts instead of https://config-master.wikimedia.org/known_hosts.ecdsa [12:04:59] jbond: Sold! Thanks :-) [12:05:48] <_joe_> jbond: and we added that cryptominer, right? [12:06:33] * jbond whispers to _joe_ shh [12:07:07] <_joe_> ah right [12:17:51] *our cryptocurrency* [12:23:44] Krinkle: The Danish government mandate ID system works by having a central authority store each citizens public and private key. Because people could work the original system where they had keep track of a private certificate. [12:24:12] couldn't work [13:44:39] Do WMF server support domain fronting for Wikimedia sites? [14:07:30] diskdance[m]: short answer: not right now [14:07:53] Oops [14:08:46] I thought we could send fake SNI or simply send no SNI to circumvent censorship [14:09:17] InternetArchiveBot, which runs on Cloud Services, has been encountering 429 errors when making requests to Wikimedia servers. https://phabricator.wikimedia.org/T318065 Do you have logs you could share to help us figure out what's going on? [14:09:27] The bot has a distinctive user agent [14:09:32] But somebody in the Chinese community seemed to manage to do this, how? [14:10:49] hare: it's rather hard to say anything without a sample request/response and the actual user-agent string [14:11:49] user agent is: IABot/2.0 (+https://en.wikipedia.org/wiki/User:InternetArchiveBot) (InternetArchiveBot operated by Cyberpower678. Please contact Cyberpower678 at https://en.wikipedia.org/wiki/User:Cyberpower678 for questions and concerns.) [14:20:05] hare: I see that UA making up to 500-600 req/s at peak to the MW apis, so without futher details I'd say getting some 429s isn't exactly a surprise [14:27:07] <_joe_> hare: hi! yeah without further details it's hard to even say which component is rate-limiting iabot [14:27:43] <_joe_> can you open a task with: * Details of when this started * A typical response (headers and body)? [14:27:59] this is the task: https://phabricator.wikimedia.org/T318065 [14:29:15] It started... in the last few weeks, when I started deploying the bot to more wikis (hence more requests). Cyberpower will need to give specific logs [14:29:47] What I am interested in knowing is if this is a limit that can be adjusted, or if it is a hard limit that applies to everyone [14:30:31] <_joe_> it's not an easy answer sadly [14:31:00] <_joe_> depends a lot on what is requested, if such requests are typically cached, where they're originating from [14:31:14] Odds are good this is an optimization we need to make on our end, but this would help us zero in on what needs to be optimized [14:32:08] Anyways, he may be occupied with other work at the moment, so I'll let you know if I hear more (or he'll comment on the phab ticket). Thank you as always for your help! [14:32:29] <_joe_> I'm wondering if given the volume of requests [14:32:58] <_joe_> we shouldn't just move this functionality to an extension we can activate/deactivate and just generate jobs [14:33:26] <_joe_> I understand there are barriers that go beyond the technical side of things there [14:33:30] We do plan on pursuing our own rewrite that has a different workflow that better fits the scale we now operate at [14:33:37] <_joe_> just thinking out loud [14:33:46] Having this built *into the wiki* would be fantastic but we would need tons of cooperation on your team's part :D [14:33:58] I was in the writing-MediaWiki-extensions-for-production business once, didn't go well [14:34:56] I do sincerely agree that a lot of this would benefit from InternetArchiveBot having closer access to production, and to some extent being a part of it, and if you want to start that conversation I'm interested, but I understand it's not a trivial request. [14:35:27] <_joe_> I would imagine it would be even easier, to respect conway's law, to just emit a specific event stream for added links to pages [14:35:50] <_joe_> that would allow iabot to maintain operational independence but also reduce at least read operations [14:35:50] Like, a subset of the current eventstream? [14:36:06] <_joe_> yeah, I'd have to think about it a bit better though [14:36:29] <_joe_> for now let's try to understand what is wrong there; I posted a question on the task and tagged the traffic team on it [14:36:29] a link-specific stream would be extremely useful for us; we are also looking into wikimedia enterprise for this functionality [14:38:06] right now IABot's workflow is basically going to special:allpages (the db replica/API equivalent) and scanning the external links on each wiki page, which is a lousy workflow but it worked in 2015 [14:38:53] though, separately we consume the eventstream API to identify URLs for archiving [19:38:43] anyone have the ability to grant me push rights to git@github.com:wikimedia/labs-private.git? [19:39:06] why? it's a mirror of a gerrit repo [19:40:34] ah, well then I should use that one, thanks taavi! [19:40:38] jhathaway: https://gerrit.wikimedia.org/r/admin/repos/labs/private is the real repo. The github repo is a read-only mirror [19:41:10] any reason for the mirror? [19:41:38] the same reason for the other ~700 mirrored repos there I guess [19:42:05] >90% of https://github.com/wikimedia/ is mirrored rather than origin [19:43:33] * bd808 will fix the aobut on https://github.com/wikimedia/labs-private [19:44:16] bd808: thanks [19:47:46] ~700 is apparently a vast underestimate. It looks like we have 2459 repos under the @wikimedia github org account. :) [19:48:09] I would be pretty surprised if more than 100 of those were primaries [19:48:13] O.o [19:55:44] bd808: Tyler did an audit a couple years ago (and I wrote an analysis of the mess) https://phabricator.wikimedia.org/T237470 [19:57:03] 153 repos on GitHub (not counting repos in their own organizations or under personal namespaces) [20:19:24] From https://www.mediawiki.org/wiki/Gerrit/GitHub#Projects_on_GitHub it looks like we are slowly creeping more github origin repos over time. I have some hope that this trend will halt when GitLab is a bit easier to setup CI for, but I guess time will tell. [20:20:07] * bd808 was responsible for at least one github origin repo and is not currently sure it was worth the social friction