[00:00:02] jhathaway: yeah I don't either, something for us both to look into on Monday [00:00:04] GenNotability: thanks [00:00:19] GenNotability: +1 [00:00:19] rzl: recommend reconsidering the impact of the UA filter [00:00:26] have we figured out what proxy service this is? [00:00:28] it broke things, but.. [00:00:41] GenNotability: no, doesn't appear to be a proxy :/ [00:00:59] least none of the usual ones [00:01:18] TheresNoTime: yeah I'm starting a patch for it -- I don't want to roll back out the one we used last time, but I should be able to block everything except oauth traffic [00:01:45] not sure if we'll end up using it but I want to have it ready [00:01:47] but with the wide range of ISP & usage types zombies seem more likely [00:02:02] TheresNoTime: spur doesn't flag anything consistently, but I really would like to figure out what we're looking at - unidentified proxy service, botnet, ??? [00:02:16] * GenNotability fires up Shodan [00:02:38] I saw a few residentials I think, I assumed botnet [00:02:54] yeah mix of residential, corp, and hosting [00:03:00] mostly residential [00:03:36] I guess the question is "what's the point" - as DDoS attacks go this is...wimpy [00:03:45] People are weird? [00:03:47] (great, now I've cursed us all) [00:05:30] ...I really want to know why my private global test filter is looking for the phrase "stab an elephant with a nostril" [00:05:43] ohhhh, that LTA [00:05:47] !log pt1979@cumin2002 START - Cookbook sre.hosts.downtime for 2:00:00 on kubernetes2019.codfw.wmnet with reason: host reimage [00:05:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:06:03] the original stuff felt very cartoon villain -- someone scorned by a database taking out their wrath on us [00:06:10] but now it's just nonsense [00:06:56] It feels like on the level of some of the more low-impact vandalism that's usually seen from younger folk so I'd say "scriptkiddie" but that doesn't explain a botnet [00:08:31] setting up an iot botnet is sadly not /that/ difficult [00:08:44] most of the Mirai folks were college students [00:09:04] or even high school ones [00:09:12] !log pt1979@cumin2002 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubernetes2019.codfw.wmnet with reason: host reimage [00:09:15] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:09:18] how can I help? [00:11:06] my experience of these types of botnet attacks (in other contexts) is that they tend to not be persistent. either they'll burn through their network pool, or they'll get bored eventually, usually a few days to a week [00:11:39] its somewhat a waste of zombies tbh [00:13:41] skids with entirely-too-large botnets ain't exactly new either [00:14:00] grabbed enwiki filter 1185, log-only as global filter 297 [00:14:16] sadly global filters can't help with most of the large wikis [00:14:16] AntiComposite: no, but they tend to get bored [00:15:00] GenNotability: you also didn't set it as global :P [00:15:13] TheresNoTime: shaddup [00:15:18] if we need to opt-in some large wikis into global filters on an emergency basis I think we should just do it [00:15:21] zabe: not trying to block, just track [00:15:27] legoktm: +1 [00:15:59] how can I help? [00:16:13] I'm still catching up, does frwiki still need help? [00:16:34] I think they implemented an abusefilter, but that is being throttled, so I guess yes [00:16:41] https://fr.wikipedia.org/wiki/Sp%C3%A9cial:Journal_du_filtre_antiabus?wpSearchUser=&wpSearchPeriodStart=&wpSearchPeriodEnd=&wpSearchTitle=&wpSearchImpact=0&wpSearchAction=any&wpSearchActionTaken=&wpSearchGroup=0&wpSearchFilter=29 no further hits [00:16:42] zabe: just trying to keep an eye out for the botnet jumping to a less-patrolled wiki [00:16:50] turns out, the throttling does not disable blocking [00:16:57] just gives annoying notification spam [00:17:05] legoktm: I can monitor performance of systems [00:17:09] ah okay, we can disable throttling on frwiki [00:17:11] we probably should skip wikidata [00:17:15] GenNotability: fair enough, the big wikis usually have activ reporters [00:17:17] should we just disable af throttling globally? [00:17:29] what can I do for wikidata (WD admin)? [00:17:33] rzl: no, I think we concluded that throttling isn't an issue here [00:17:33] ^ rephrase, I'm inclined to do that but listening for objections :) [00:17:47] the throttle doesn't stop the filters so it doesn't seem worth disabling [00:17:48] +1 [00:17:51] throttling only stops "restricted actions", of which disallow isn't affected [00:17:53] I'm also admin if needed but I'm saying global filters should not be enabled for wikidata [00:18:03] legoktm: oh got it, I misunderstood above, thanks [00:18:10] I got it wrong last night too [00:18:10] the throttling "just" sends echo/emails [00:18:13] I've been watching swviewer for the last 15 mins and nothing has popped up yet 🤷‍♂️ [00:18:16] which, isn't fun [00:18:23] this started on wikidata, but the filter would need to be different anyway [00:18:46] AntiComposite: sigh [00:18:49] let me see [00:18:52] looks like frwiki is still blocking at least https://fr.wikipedia.org/wiki/Sp%C3%A9cial:Journal/block [00:19:06] do you have an example? [00:19:07] !log pt1979@cumin2002 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kubernetes2019.codfw.wmnet with OS bullseye [00:19:12] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:19:12] 10SRE, 10ops-codfw, 10DC-Ops, 10serviceops, 10Kubernetes: (Need By: TBD) rack/setup/install kubernetes20[19|2(012)] - https://phabricator.wikimedia.org/T299470 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2019.codfw.wmnet with OS bullseye comp... [00:19:24] Amir1 on frwiki it's like this https://fr.wikipedia.org/wiki/Spécial:Journal_du_filtre_antiabus/3277035 [00:19:26] also, frwiki has global filters enabled [00:19:54] frwiki admins are blocking old edits. theres nothing ongoing in RC [00:20:08] perryprog: I see. [00:20:17] rephrase, theyre blocking accounts after the attack stopped* [00:20:40] do we have a stew also gblocking the IPs? have they been reusing IPs across wikis? [00:21:22] I randomly looked at a few of the frwikis and didn't see any that were blocked on enwiki (or that hit abusefilter) [00:21:24] I hate to say it but should we ban IP editing in those wikis for a short period of time? [00:21:24] there has been some gblocking, but I haven't seen much reuse [00:21:26] also fwiw, i saw minimal to no ip reuse among not-blocked IPs from the attack yesterday to the one today. [00:21:44] Amir1: you're not going to hear me objecting. [00:21:44] Amir1, wmgEmergencyCaptcha is slightly less drastic and has been effective [00:22:44] there's also still the option of blocking the user agent [00:22:45] we did that in fawiki a while ago (for a week) when some vandalism ended up in national tv and everyone learned they could vandalize wikipedia, fun times [00:22:57] but yeah, captcha is good [00:23:05] it's deployed, what else? [00:23:11] new wikis needed? [00:23:59] It doesn't seem like there's an active attack on any wiki at the moment, from what I can tell. [00:24:18] https://meta.wikimedia.org/w/index.php?title=Special:AbuseLog&wpSearchFilter=297 is empty, so yeah, looks good so far [00:24:42] btw, I don't see anything for wikidata so far https://www.wikidata.org/wiki/Special:RecentChanges?userExpLevel=unregistered&hidebots=1&hidecategorization=1&limit=50&days=30&enhanced=1&urlversion=2 [00:25:41] yeah haven't seen anything on wd today [00:27:53] !log pt1979@cumin2002 START - Cookbook sre.hosts.reimage for host kubernetes2020.codfw.wmnet with OS bullseye [00:27:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:28:00] 10SRE, 10ops-codfw, 10DC-Ops, 10serviceops, 10Kubernetes: (Need By: TBD) rack/setup/install kubernetes20[19|2(012)] - https://phabricator.wikimedia.org/T299470 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2020.codfw.wmnet with OS bullseye [00:29:45] GenNotability: did you end up setting up the global tracking filter? [00:30:08] legoktm: yup, https://meta.wikimedia.org/wiki/Special:AbuseFilter/297 - straight-up rip of enwiki 1185, log-only [00:30:17] have musikbot set to ping me on hits [00:32:26] PROBLEM - k8s API server requests latencies on kubestagemaster1001 is CRITICAL: instance=10.64.16.203 verb={CREATE,UPDATE} https://wikitech.wikimedia.org/wiki/Kubernetes https://grafana.wikimedia.org/dashboard/db/kubernetes-api?viewPanel=27 [00:33:00] (JobUnavailable) firing: Reduced availability for job mjolnir in eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org [00:33:37] awesome [00:34:10] RECOVERY - k8s API server requests latencies on kubestagemaster1001 is OK: All metrics within thresholds. https://wikitech.wikimedia.org/wiki/Kubernetes https://grafana.wikimedia.org/dashboard/db/kubernetes-api?viewPanel=27 [00:36:48] (03PS1) 10RLazarus: varnish: Block a bad UA [puppet] - 10https://gerrit.wikimedia.org/r/763867 (https://phabricator.wikimedia.org/T302047) [00:37:42] I'm mostly around this weekend and next week (with exception of Monday), ping me if you need anything deployed [00:38:59] (03CR) 10Ladsgroup: [C: 03+1] varnish: Block a bad UA [puppet] - 10https://gerrit.wikimedia.org/r/763867 (https://phabricator.wikimedia.org/T302047) (owner: 10RLazarus) [00:40:16] (03CR) 10CDanis: [C: 03+1] varnish: Block a bad UA [puppet] - 10https://gerrit.wikimedia.org/r/763867 (https://phabricator.wikimedia.org/T302047) (owner: 10RLazarus) [00:41:30] ugh, that filter does not work at all on wikidata... [00:43:41] uh [00:43:43] damn [00:43:45] yup, if you give me some vandalism examples, I can build a new one for wikidata [00:44:01] I think they're back and very xwiki, stand by one [00:44:22] https://fr.wikipedia.org/wiki/Colon?diff=prev&oldid=190984836 [00:44:22] https://sl.wikipedia.org/wiki/Sulili?diff=prev&oldid=5652266 [00:44:22] https://lt.wikipedia.org/wiki/Belgijos_futbolo_var%C5%BEybos_1955%E2%80%931956_m.?diff=prev&oldid=6502999 [00:44:23] (03PS2) 10RLazarus: varnish: Block a bad UA [puppet] - 10https://gerrit.wikimedia.org/r/763867 (https://phabricator.wikimedia.org/T302047) [00:44:23] @GenNotability global emergency captcha? [00:44:27] is captcha on? [00:44:33] yes, back xwiki - https://logstash.wikimedia.org/goto/2dafc22a9a031caa6fb1c3bc8c1b65a4 [00:44:36] Confirmed, they're xwiki [00:44:54] I'm going to deploy the captcha now [00:44:57] Seddon: might be needed [00:45:07] wait, can we do the UA block first please? [00:45:16] (03CR) 10Legoktm: [C: 03+1] varnish: Block a bad UA [puppet] - 10https://gerrit.wikimedia.org/r/763867 (https://phabricator.wikimedia.org/T302047) (owner: 10RLazarus) [00:45:21] or at least, one at a time [00:45:28] (03CR) 10Zabe: [C: 03+1] "lgtm now" [puppet] - 10https://gerrit.wikimedia.org/r/763867 (https://phabricator.wikimedia.org/T302047) (owner: 10RLazarus) [00:45:34] one at a time sounds good to me, I'm awaiting varnish tests atm [00:45:54] !log pt1979@cumin2002 START - Cookbook sre.hosts.downtime for 2:00:00 on kubernetes2020.codfw.wmnet with reason: host reimage [00:45:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:46:01] GenNotability: is the FP rate low enough to set the global filter to disallow? [00:46:10] legoktm: uncertain, had one FP on WD [00:46:17] legoktm: I make the patch, keep it stand by [00:46:28] GenNotability: set to warn [00:46:38] that tends to trip up automated things [00:46:41] TheresNoTime: can I also set phasers to stun? [00:46:47] (worked on enwp) [00:46:48] beat me to that joke [00:47:41] (03PS1) 10Ladsgroup: Enable wmgEmergencyCaptcha everywhere [mediawiki-config] - 10https://gerrit.wikimedia.org/r/763870 [00:47:52] new type? https://sk.wikipedia.org/w/index.php?title=Saitama_(Saitama)&diff=prev&oldid=7321289&diffmode=source [00:47:55] GenNotability ^ [00:48:11] I'm seeing it on many many wikis [00:48:12] variation [00:48:16] (03CR) 10Seddon: [C: 03+1] Enable wmgEmergencyCaptcha everywhere [mediawiki-config] - 10https://gerrit.wikimedia.org/r/763870 (owner: 10Ladsgroup) [00:48:18] ffffffff [00:48:20] Amir1: I'd rather try the UA block first and then captcha everywhere, if we can [00:48:27] given how hostile the captcha is to users in some alphabets [00:48:27] lemme flag down a steward real quick to see if they've changed UAs [00:48:35] very very many wikis [00:48:43] rzl: as I said above "I make the patch, keep it stand by" [00:48:46] ah thanks [00:48:59] how are people following/seeing this in real time? [00:49:00] varnish tests done, merging the UA patch now unless any objections? [00:49:07] es, hr, sq, sl, be [00:49:09] :shipit: [00:49:11] legoktm https://swviewer.toolforge.org [00:49:12] !log pt1979@cumin2002 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubernetes2020.codfw.wmnet with reason: host reimage [00:49:15] yep got [00:49:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:49:18] go* [00:49:18] none from me [00:49:22] (03CR) 10RLazarus: [C: 03+2] varnish: Block a bad UA [puppet] - 10https://gerrit.wikimedia.org/r/763867 (https://phabricator.wikimedia.org/T302047) (owner: 10RLazarus) [00:49:23] legoktm: https://logstash.wikimedia.org/goto/2dafc22a9a031caa6fb1c3bc8c1b65a4 for me [00:49:44] for me I have a script to easily find other wikis that an IP has edited in the last 24 hours - once seen on one wiki, check others. https://meta.wikimedia.org/wiki/User:DannyS712/FindIPActivity.js [00:50:00] GenNotability should Q105103969 be added to the filter? [00:50:00] see https://global-search.toolforge.org/?q=%22Banish+%5B%5Bd%3AQ105103969%7CVerified+Handles%5D%5D%22&namespaces=&title= [00:50:09] working on it... [00:50:32] I'm trying to jam two different filters together here [00:50:32] rzl: for checking collateral damage, this would be useful https://grafana.wikimedia.org/d/000000170/wikidata-edits?orgId=1&refresh=1m&from=now-3h&to=now [00:50:41] Wikidata uses a lot of OAuth editing [00:50:53] oauth should not be affected [00:51:03] "should" ;) [00:51:05] GenNotability let me know how I can help, I'll stay out of editing the filters to avoid conflicts unless you want a hand [00:51:16] fair point ;) [00:53:07] * perryprog is no longer seeing it [00:56:07] thank you whoever already cleaned up the "banish" edits [00:56:31] it's back, nvm https://sq.wikipedia.org/w/index.php?title=Amos_Oz&diff=2399780&oldid=2368340&uselang=en&redirect=no [00:57:32] Much larger volume this time. GenNotability, check filter? [00:57:58] perryprog: working on it, boss [00:58:59] !log pt1979@cumin2002 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kubernetes2020.codfw.wmnet with OS bullseye [00:59:01] deployed - irlike failure (left the q capitalized) [00:59:03] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:59:05] 10SRE, 10ops-codfw, 10DC-Ops, 10serviceops, 10Kubernetes: (Need By: TBD) rack/setup/install kubernetes20[19|2(012)] - https://phabricator.wikimedia.org/T299470 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2020.codfw.wmnet with OS bullseye comp... [00:59:59] suggest blocking the qid addition entirely [01:00:24] I hope the filter is private [01:00:31] still seeing it GenNotability https://bg.wikipedia.org/w/index.php?diff=11311591&oldid=11182660&uselang=en&redirect=no&mobileaction=toggle_view_desktop [01:00:43] Should end soon [01:01:11] What does "Banish Verified Handles" even mean? [01:01:23] there's a database called Verified Handles [01:01:24] it's...some database? [01:01:33] aah [01:01:37] apparently they wanted to be included in it, but were rejected [01:01:38] I thought it's the blue tick [01:01:47] lovely [01:01:48] !log pt1979@cumin2002 START - Cookbook sre.hosts.reimage for host kubernetes2021.codfw.wmnet with OS bullseye [01:01:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:01:54] 10SRE, 10ops-codfw, 10DC-Ops, 10serviceops, 10Kubernetes: (Need By: TBD) rack/setup/install kubernetes20[19|2(012)] - https://phabricator.wikimedia.org/T299470 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2021.codfw.wmnet with OS bullseye [01:02:21] GenNotability I'm creating a different filter to just block that qid [01:02:26] DannyS712: go for it [01:02:52] DannyS712: set it to warn, not disallow :) [01:02:59] that seems to be enough* [01:03:01] the UA ban is now on all varnish hosts [01:03:43] if that doesn't cause the edits to drop off sharply I'd love to know :) [01:04:37] checking Amir1's link and it looks like the correct squigglies to me [01:04:46] (wrt collateral damage) [01:05:20] still happening e.g., https://sl.wikipedia.org/w/index.php?title=USS_Reid_(DD-292)&diff=5652307&oldid=5446212&uselang=en&redirect=no [01:05:47] concur [01:06:23] good to know, digging in webrequests to see if the UA changed to something else or if my VCL just failed to catch it [01:06:39] steward around to CU? [01:06:43] that'd be easier [01:06:48] yes please :) [01:07:03] rzl: bad regex i think [01:07:05] created 299 to warn for that addition [01:07:51] seeing a lull [01:08:07] concur [01:08:17] rzl: the regex needs + for UA [01:08:20] right? [01:08:24] ughhhhh [01:08:25] thanks Amir1 [01:08:48] I guess I could do a cu in an extrem emergency, but not sure whether that applies here [01:08:51] though someone beat me to it with disallow, https://meta.wikimedia.org/wiki/Special:AbuseFilter/298 [01:08:55] (what flavour of regex is this?) [01:09:06] global-search gives currently 322 live versions of the edit https://global-search.toolforge.org/?q=%22Banish+%5B%5Bd%3AQ105103969%7CVerified+Handles%5D%5D%22&namespaces=&title= [01:09:27] I'll go clean those up [01:09:36] it's back as https://es.wikipedia.org/w/index.php?diff=141775055&oldid=139481646&uselang=en&redirect=no&mobileaction=toggle_view_desktop [01:09:42] perryprog: how to split it? [01:09:43] PCRE2 mind says something like `^python-requests\/.*$` would be better? [01:09:47] seeing ongoing hits [01:09:53] (03PS1) 10RLazarus: varnish: Block a bad UA, correctly [puppet] - 10https://gerrit.wikimedia.org/r/763873 (https://phabricator.wikimedia.org/T302047) [01:09:59] Amir1 I'm just reverting live edits [01:10:05] er, recent changes edits [01:10:18] (03CR) 10Ladsgroup: [C: 03+1] varnish: Block a bad UA, correctly [puppet] - 10https://gerrit.wikimedia.org/r/763873 (https://phabricator.wikimedia.org/T302047) (owner: 10RLazarus) [01:10:18] TheresNoTime: I considered \W* but I don't want to block a more detailed UA that happens to include python-requests [01:10:21] (03CR) 10CDanis: [C: 03+1] varnish: Block a bad UA, correctly [puppet] - 10https://gerrit.wikimedia.org/r/763873 (https://phabricator.wikimedia.org/T302047) (owner: 10RLazarus) [01:10:26] I did check that it's + and not \+ in this dialect though :) [01:10:27] (03CR) 10Zabe: [C: 03+1] varnish: Block a bad UA, correctly [puppet] - 10https://gerrit.wikimedia.org/r/763873 (https://phabricator.wikimedia.org/T302047) (owner: 10RLazarus) [01:10:31] rzl: ack [01:10:37] that's a lot of +1s [01:10:43] (03CR) 10Jforrester: [C: 03+1] varnish: Block a bad UA, correctly [puppet] - 10https://gerrit.wikimedia.org/r/763873 (https://phabricator.wikimedia.org/T302047) (owner: 10RLazarus) [01:10:52] heh [01:10:53] perryprog: do you want me to create a bot to do it? [01:10:59] (03CR) 10JHathaway: [C: 03+1] varnish: Block a bad UA, correctly [puppet] - 10https://gerrit.wikimedia.org/r/763873 (https://phabricator.wikimedia.org/T302047) (owner: 10RLazarus) [01:11:09] Amir1 DannyS712 might already be on it :) [01:11:13] waiting for the tests, then I'll merge and deploy [01:11:22] someone merge this patch before it breaks +1 record [01:11:41] ah, makes sense [01:11:54] tempted to skip the tests but if there's any file where I'm absolutely not gonna do that, it's text-frontent.inc.vcl.erb :) [01:12:06] *frontend, see?? [01:12:11] (03CR) 10DannyS712: [C: 03+1] varnish: Block a bad UA, correctly [puppet] - 10https://gerrit.wikimedia.org/r/763873 (https://phabricator.wikimedia.org/T302047) (owner: 10RLazarus) [01:12:15] testing is overrated [01:12:17] rzl: you're never going to earn your t-shirt this way [01:12:20] (can't have vandals if all UAs are blocked!) [01:12:33] TheresNoTime: now that's the kind of thinking I'm here for [01:12:40] cdanis: it's reduced to stickers due to budget reasons anyway [01:12:48] wait I figured it out [01:12:55] we just have to make it so that *not* anyone can edit it [01:12:56] Surprised I've only been ratelimited a few times in my reverting [01:12:58] problem solved [01:13:03] "the wiki that nobody can edit" [01:13:10] Wikipedia, the free encyclopedia that cdanis can edit [01:13:27] True, how could no one come up with that until now ;) [01:13:30] hey, I can edit by manually entering records into the database too [01:13:35] good to see you back cdanis :) [01:13:42] $1 fee per edit also edits are stored on a blockchain [01:13:57] perryprog: and people complain about maxlag now... [01:14:07] aaaand I'm unsubscribing from filter notifications [01:14:11] heh [01:14:12] suffice it to say we're getting a lot of hits :P [01:14:20] was wondering how long it would take [01:14:26] I'm probably missing a lot of wikis too since I just have the smallwikis plus a few extra [01:14:38] is the global filter in disallow? [01:14:51] GenNotability: AntiComposite: just put it to warn [01:15:02] that'll probably disrupt a bot enough [01:15:10] still happening at e.g., https://es.wikipedia.org/w/index.php?title=Tupanciretã&diff=141775183&oldid=120322084&uselang=en&redirect=no&diffmode=source [01:15:31] eswiki might be opted out of global filters [01:15:38] certainly [01:15:41] tests passed, merging [01:15:51] (03CR) 10RLazarus: [C: 03+2] varnish: Block a bad UA, correctly [puppet] - 10https://gerrit.wikimedia.org/r/763873 (https://phabricator.wikimedia.org/T302047) (owner: 10RLazarus) [01:16:01] rollout will be a little speedier this time [01:16:08] 1234qwer's is in warn [01:16:50] A while ago I wrote some scripts for globally change js stuff (fix deprecation, etc.) I can reuse most of the code [01:17:18] the only complexity is that it's not based on the global search it just searches each wiki manually [01:17:33] well, automatically, I mean one by one [01:17:47] 6 +1's, not bad. Not sure what the record is. [01:17:59] PROBLEM - Check systemd state on logstash1026 is CRITICAL: CRITICAL - degraded: The following units failed: curator_actions_cluster_wide.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [01:18:01] we're up to about 500 live instances of the phrase assuming there isn't too much lag at the moment [01:19:14] should see a drop off on the filter log now..? [01:19:22] !log pt1979@cumin2002 START - Cookbook sre.hosts.downtime for 2:00:00 on kubernetes2021.codfw.wmnet with reason: host reimage [01:19:25] So far it's looking clear [01:19:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:20:06] no more hits for a minute.. [01:20:17] yep, looks good [01:21:06] I'm clearing the old ones. From global search, run [01:21:06] document.querySelectorAll('td a').forEach( [01:21:06] function ( e ) { e.setAttribute( 'href', e.getAttribute('href') + '?diff=cur&oldid=prev&diffonly=true' ); } [01:21:06] ) [01:21:06] to change each page link to the last diff for easy revert [01:21:33] <3 [01:21:56] !log legoktm@deploy1002 Synchronized private/PrivateSettings.php: T302047 (duration: 00m 49s) [01:22:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:23:20] varnish rollout complete [01:23:44] sorry for the extra delay, wish that hadn't taken two tries to get right :) [01:23:44] rzl: I owe y'all a beer [01:23:52] rzl: nice work! [01:24:29] !log mwdebug-deploy@deploy1002 helmfile [eqiad] START helmfile.d/services/mwdebug: apply [01:24:33] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:24:46] !log pt1979@cumin2002 END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubernetes2021.codfw.wmnet with reason: host reimage [01:24:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:25:44] !log mwdebug-deploy@deploy1002 helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [01:25:45] !log mwdebug-deploy@deploy1002 helmfile [codfw] START helmfile.d/services/mwdebug: apply [01:25:47] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:25:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:25:59] https://grafana.wikimedia.org/d/myRmf1Pik/varnish-aggregate-client-status-codes?viewPanel=2&orgId=1&var-site=codfw&var-site=drmrs&var-site=eqiad&var-site=eqsin&var-site=esams&var-site=ulsfo&var-cache_type=varnish-text&var-status_type=4&var-method=POST&from=now-3h&to=now&refresh=30s [01:26:04] nice spike in 403s [01:26:23] dumb question: does the magic UA filter have enough granularity to block just the edit API? [01:26:38] AntiComposite: nice spike, indeed [01:26:59] !log mwdebug-deploy@deploy1002 helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [01:27:02] GenNotability: it does not [01:27:03] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:27:13] dang [01:27:26] well there was my great idea for the day, have fun kids! [01:27:42] it allows oauth, which should fix most false positives [01:28:01] at least the ones that have been complained about [01:28:53] GenNotability: see what I just posted on the phab ticket [01:28:54] If this doesn't work permanently, and say they work around captcha... what's the next logical deterrent after that [01:29:54] UA again? [01:30:03] Battle of attrition? [01:30:03] Seddon: fairly sure the servers have off switches /s [01:30:13] Seddon: WMF black ops team [01:30:37] perryprog: definitely has limitations [01:30:55] options depend on details I can't see [01:31:34] (details of their requests) [01:32:51] but restricting anon API access is an option, if relevant [01:33:16] !log legoktm@deploy1002 Synchronized private/PrivateSettings.php: T302047 tweaks (duration: 00m 48s) [01:33:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:34:44] !log pt1979@cumin2002 END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kubernetes2021.codfw.wmnet with OS bullseye [01:34:47] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:34:50] 10SRE, 10ops-codfw, 10DC-Ops, 10serviceops, 10Kubernetes: (Need By: TBD) rack/setup/install kubernetes20[19|2(012)] - https://phabricator.wikimedia.org/T299470 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host kubernetes2021.codfw.wmnet with OS bullseye comp... [01:36:03] Anyone that speaks dutch might be able to reply here https://de.wikipedia.org/wiki/Wikipedia:Administratoren/Notizen?diff=cur&oldid=prev&diffonly=true&diffmode=source [01:36:25] perryprog: das ist deutsch [01:36:31] o/ [01:36:54] I was thinking of responding but zabe definitely can do better than mine [01:37:03] what should we respond? [01:37:09] !log mwdebug-deploy@deploy1002 helmfile [eqiad] START helmfile.d/services/mwdebug: apply [01:37:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:37:26] Ack, yes, German not Dutch [01:37:52] i was planning something like "it was a botnet, employees of the wmf are working on it", maybe linking the gerrit change for the techies? [01:38:00] (JobUnavailable) firing: (2) Reduced availability for job mjolnir in eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org [01:38:14] !log mwdebug-deploy@deploy1002 helmfile [eqiad] DONE helmfile.d/services/mwdebug: apply [01:38:15] !log mwdebug-deploy@deploy1002 helmfile [codfw] START helmfile.d/services/mwdebug: apply [01:38:16] well, I learned an important lesson today: when you set up a system that notifies you of every DoS event, it can be very difficult to turn that system back off [01:38:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:38:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:38:36] haha [01:38:53] Problem: Be DDoSd. Solution: Set up notification system in case of DDoS. Problem: Be DDoSd. [01:39:06] zabe: botnet distributed vandalism across wikis? [01:39:11] !log mwdebug-deploy@deploy1002 helmfile [codfw] DONE helmfile.d/services/mwdebug: apply [01:39:15] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:39:18] GenNotability: that's me looking at my pager [01:39:31] yeah sound fair [01:40:22] (JobUnavailable) firing: (2) Reduced availability for job mjolnir in eqiad - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org [01:40:28] And the backlog of "Banish [[d:Q105103969|Verified Handles]]" is officially at one! https://global-search.toolforge.org/?namespaces=&q=%22Banish%20%5B%5Bd%3AQ105103969%7CVerified%20Handles%5D%5D%22&title=&purge=1 [01:40:30] global search now clear (except for the dewiki discussion) [01:40:33] Thanks everyone for y'all's help. [01:40:43] ^^ [01:40:48] !log pt1979@cumin2002 START - Cookbook sre.hosts.reimage for host kubernetes2022.codfw.wmnet with OS bullseye [01:40:49] 👍 [01:40:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:40:51] I go rest, I have a presentation tomorrow [01:40:54] 10SRE, 10ops-codfw, 10DC-Ops, 10serviceops, 10Kubernetes: (Need By: TBD) rack/setup/install kubernetes20[19|2(012)] - https://phabricator.wikimedia.org/T299470 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host kubernetes2022.codfw.wmnet with OS bullseye [01:41:07] what mitigations are we leaving in place over the weekend, and what do we want to think about rolling back? [01:41:12] I'll be around tomorrow ping me in IRC if needed [01:41:29] rzl: CAPTCHA should be rolled back in my opinion as soon as possible [01:41:32] e.g. with the UA block in place do we want to think about switching emergency captchas off? [01:41:34] ye [01:41:35] https://meta.wikimedia.org/wiki/Special:AbuseLog/1578207 [01:41:42] +1 to switching emergencyCaptcha off [01:41:57] and the throttling turned back on [01:42:01] RECOVERY - Router interfaces on cr1-eqiad is OK: OK: host 208.80.154.196, interfaces up: 241, down: 0, dormant: 0, excluded: 0, unused: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [01:42:04] just in case someone makes an edit filter mistake [01:42:06] DannyS712, what's that showing, most of the folks in here can't see private filters [01:42:16] any objections to doing that promptly, or would we rather wait a while to see if we're stable? [01:42:28] "that" = reverting both emergencyCaptcha and AF throttling to normal [01:42:38] rzl: I think we can do it right now [01:42:46] AntiComposite https://el.wikipedia.org/wiki/Χρήστης:ΔώραΣτρουμπούκη trying to add the wikidata id [01:42:51] oh no [01:42:58] rzl objection - looks like maybe accounts too [01:43:21] account compromise? https://el.wikipedia.org/wiki/%CE%A7%CF%81%CE%AE%CF%83%CF%84%CE%B7%CF%82:%CE%94%CF%8E%CF%81%CE%B1%CE%A3%CF%84%CF%81%CE%BF%CF%85%CE%BC%CF%80%CE%BF%CF%8D%CE%BA%CE%B7 [01:43:27] the abuse filter is still constanly triggering with the edit attempts from ips too [01:44:03] DannyS712: I've not seen any IPs recently, just that ^ account [01:44:21] check global filter 297 hits [01:44:30] (03PS1) 10Andrew Bogott: nfs-mounts.yaml: move utrs to a project-local nfs server [puppet] - 10https://gerrit.wikimedia.org/r/763875 (https://phabricator.wikimedia.org/T301280) [01:44:32] (03PS1) 10Andrew Bogott: nfs-mounts.yaml: move wmde-templates-alpha to a project-local nfs server [puppet] - 10https://gerrit.wikimedia.org/r/763876 (https://phabricator.wikimedia.org/T301280) [01:44:44] DannyS712: `01:18, February 19, 2022: User:168.158.155.50` was the last IP? [01:45:18] still +1 on disabling EmergencyCaptcha, since it won't affect autoconfirmed accounts [01:45:38] oh, musikbot is still reporting the hits to me but its from a while ago, I guess it was backlogged [01:45:44] :) [01:45:49] yeah it'll do that :) [01:45:52] objection withdrawn, but can we check that account? [01:46:17] RECOVERY - Router interfaces on cr1-codfw is OK: OK: host 208.80.153.192, interfaces up: 132, down: 0, dormant: 0, excluded: 0, unused: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down [01:47:18]