[08:24:32] hi folks, heads up that later this EU morning I'll be switching swift traffic to eqiad https://phabricator.wikimedia.org/T288458 [08:25:21] though please let me know if you have objections [08:25:45] plan is to leave traffic fully on eqiad for 24/48 hours and start rebalancing the cluster in codfw [08:57:32] ok proceeding [11:17:42] 10netops, 10Infrastructure-Foundations, 10SRE: Move management routers ssh port - https://phabricator.wikimedia.org/T277438 (10ayounsi) I changed the LibreNMS check to exclude management routers. To revert once this task is done: > Processor usage over 85% EXCEPT Management routers [12:08:22] 10Traffic, 10InternetArchiveBot, 10SRE, 10Platform Team Workboards (Clinic Duty Team): IAbot sending a huge volume of action=raw requests (HTTP 415 errors) - https://phabricator.wikimedia.org/T269914 (10Ladsgroup) A month have passed. If it continues to happen. I suggest banning the user. [14:19:44] 10Traffic, 10InternetArchiveBot, 10SRE, 10Platform Team Workboards (Clinic Duty Team): IAbot sending a huge volume of action=raw requests (HTTP 415 errors) - https://phabricator.wikimedia.org/T269914 (10Cyberpower678) The offending task should have long been disabled. Is this still even happening? [14:29:20] 10Traffic, 10InternetArchiveBot, 10SRE, 10Platform Team Workboards (Clinic Duty Team): IAbot sending a huge volume of action=raw requests (HTTP 415 errors) - https://phabricator.wikimedia.org/T269914 (10Cyberpower678) {F34617913} This is the crontab snippet. books.php was the offending program and is cle... [14:36:55] 10Traffic, 10InternetArchiveBot, 10SRE, 10Platform Team Workboards (Clinic Duty Team): IAbot sending a huge volume of action=raw requests (HTTP 415 errors) - https://phabricator.wikimedia.org/T269914 (10Cyberpower678) >>! In T269914#7178112, @Legoktm wrote: > @Cyberpower678 it looks much better now, so tha... [15:40:06] it looks like we don't have updates for the Traffic section for the SRE meeting upcoming. We should probably focus on that a bit for next meetings :) Any updates you can put in right now? [15:40:30] bblack: ema: vgutierrez: sukhe: mmandere: ^ [15:42:34] ema is out today of course [15:49:14] question_mark: adding some, thanks [15:49:21] thank you :) [16:05:10] 10Traffic, 10InternetArchiveBot, 10SRE, 10Platform Team Workboards (Clinic Duty Team): IAbot sending a huge volume of action=raw requests (HTTP 415 errors) - https://phabricator.wikimedia.org/T269914 (10Legoktm) 05Open→03Resolved I don't see any incidents of this in the last 30 days, happy to call this... [16:07:06] 10Traffic, 10 Data-Engineering, 10Analytics, 10Growth-Team, and 4 others: Migrated Server-side EventLogging events recording http.client_ip as 127.0.0.1 - https://phabricator.wikimedia.org/T288853 (10odimitrijevic) p:05Triage→03High [16:32:09] 10Traffic, 10Analytics: Review use of realloc in varnishkafka - https://phabricator.wikimedia.org/T287561 (10odimitrijevic) Adding Traffic team to help assess the patch and push to production if appropriate. [18:05:01] 10Traffic, 10 Data-Engineering, 10Analytics, 10Growth-Team, and 4 others: Migrated Server-side EventLogging events recording http.client_ip as 127.0.0.1 - https://phabricator.wikimedia.org/T288853 (10mforns) a:03Mholloway Just for the record, some discussion has happened in https://gerrit.wikimedia.org/r... [18:11:16] 10Traffic, 10Analytics: Review use of realloc in varnishkafka - https://phabricator.wikimedia.org/T287561 (10odimitrijevic) [21:25:03] 10Traffic, 10SRE: Deploy durum: check service for Wikidough - https://phabricator.wikimedia.org/T289536 (10ssingh) [21:25:24] 10Traffic, 10SRE: Deploy durum: check service for Wikidough - https://phabricator.wikimedia.org/T289536 (10ssingh) [21:25:27] 10Traffic, 10SRE, 10Patch-For-Review: Deploy Wikidough: Experimental DNS-over-HTTPS (DoH) public resolver - https://phabricator.wikimedia.org/T252132 (10ssingh) [21:25:34] 10Traffic, 10SRE: Deploy durum: check service for Wikidough - https://phabricator.wikimedia.org/T289536 (10ssingh) p:05Triage→03High