[02:48:21] 06Traffic, 13Patch-For-Review: [Update DNS Record Request] - wikimedia.org - https://phabricator.wikimedia.org/T408003#11301125 (10BCornwall) 05Open→03In progress p:05Triage→03Low [05:45:03] 06Traffic, 10bot-traffic-requests: FY 25/26 WE 5.4.6 Classify the top 30 spiders by traffic as known bots - https://phabricator.wikimedia.org/T408061 (10Joe) 03NEW [05:45:22] 06Traffic, 10bot-traffic-requests: FY 25/26 WE 5.4.6 Classify the top 30 spiders by traffic as known bots - https://phabricator.wikimedia.org/T408061#11301322 (10Joe) [05:45:46] 06Traffic, 10bot-traffic-requests: FY 25/26 WE 5.4.6 Classify the top 30 spiders by traffic as known bots - https://phabricator.wikimedia.org/T408061#11301324 (10Joe) [05:45:57] 06Traffic, 10bot-traffic-requests: FY 25/26 WE 5.4.6 Classify the top 30 spiders by traffic as known bots - https://phabricator.wikimedia.org/T408061#11301327 (10Joe) [05:55:34] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, 10Thumbor: FY 25/26 WE 5.4.7 Standardize thumbnail sizes - https://phabricator.wikimedia.org/T408062 (10Joe) 03NEW [05:55:45] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, 10Thumbor: FY 25/26 WE 5.4.7 Standardize thumbnail sizes - https://phabricator.wikimedia.org/T408062#11301353 (10Joe) p:05Triage→03High [12:18:08] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Eqiad C/D refresh: 2 x test hosts for config validation - https://phabricator.wikimedia.org/T405560#11302562 (10Jclark-ctr) Sorry for not updating yesterday @elukey 2 servers yesterday we where talking about Getting the same errors... [12:43:29] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Decom Hurricane Electric Transit/Peering circuit eqiad - https://phabricator.wikimedia.org/T407008#11302680 (10Jclark-ctr) 05Open→03Resolved a:03Jclark-ctr removed cable and updated netbox [12:59:49] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Eqiad C/D refresh: 2 x test hosts for config validation - https://phabricator.wikimedia.org/T405560#11302726 (10elukey) Provisioned both nodes with `--no-user --no-dhcp --no-switch` and they worked. Trying to reimage sretest1005 now :) [13:31:39] 10netops, 06Traffic, 06Infrastructure-Foundations: Upgrade End Of Support Junos - https://phabricator.wikimedia.org/T390813#11302841 (10Papaul) @ssingh thanks for the update. I am planning on doing it before Thanksgiving any day during the week of November 17th works for me. Let me know if that woks for you... [13:32:42] 10netops, 06Traffic, 06Infrastructure-Foundations: Upgrade End Of Support Junos - https://phabricator.wikimedia.org/T390813#11302849 (10ssingh) >>! In T390813#11302841, @Papaul wrote: > @ssingh thanks for the update. I am planning on doing it before Thanksgiving any day during the week of November 17th works... [13:53:55] 10netops, 06Infrastructure-Foundations, 10Toolforge, 06tools-infrastructure-team: Plan networking for Toolforge-on-Metal experiment - https://phabricator.wikimedia.org/T407140#11302980 (10Andrew) If we want to attempt the everything-on-containers plan that Alex is promoting, the next step is probably to ma... [14:02:18] 10netops, 06Infrastructure-Foundations, 10Toolforge, 06tools-infrastructure-team: Plan networking for Toolforge-on-Metal experiment - https://phabricator.wikimedia.org/T407140#11303033 (10taavi) The initial networking questions in the task description still need deciding. It seems like we are leaning towar... [14:23:25] TIL IP_FREEBIND [14:40:17] nice [14:40:43] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Eqiad C/D refresh: 2 x test hosts for config validation - https://phabricator.wikimedia.org/T405560#11303180 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmooney@cumin1003 for host sretest1006.eqiad.wmnet... [14:41:04] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Eqiad C/D refresh: 2 x test hosts for config validation - https://phabricator.wikimedia.org/T405560#11303181 (10elukey) The hosts needed to be uefi-provisioned, and https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1194892 needed... [14:41:19] it's one of the very non-standard sockopts though. IP_FREEBIND on Linux, but on the various BSDs the equivalent is any of SO_BINDANY, IP_BINDANY, and/or IPV6_BINDANY [14:44:09] the names on the BSDs are much harder to read as FREEBIRD; not sure if bug or feature [14:46:11] :) [15:19:27] 06Traffic, 06Experimentation Lab: Test the impact of incremental increase in traffic for cache splitting experiments - https://phabricator.wikimedia.org/T407570#11303370 (10JVanderhoop-WMF) [15:26:32] 06Traffic, 06MW-Interfaces-Team, 06serviceops, 07Epic, and 3 others: API Rate Limiting Architecture - https://phabricator.wikimedia.org/T399291#11303425 (10daniel) [15:39:02] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Eqiad C/D refresh: 2 x test hosts for config validation - https://phabricator.wikimedia.org/T405560#11303487 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmooney@cumin1003 for host sretest1006.eqiad.wmnet with... [15:42:28] 06Traffic, 13Patch-For-Review: [Update DNS Record Request] - wikimedia.org - https://phabricator.wikimedia.org/T408003#11303505 (10BCornwall) 05In progress→03Resolved Hi, @JKelsoteel-WMF! Thanks for the detailed ticket - it helps make things clear for us. I've pushed the DNS changes, so I'll go ahead a... [15:42:43] 06Traffic, 10DNS, 13Patch-For-Review: [Update DNS Record Request] - wikimedia.org - https://phabricator.wikimedia.org/T408003#11303509 (10BCornwall) [15:44:38] 06Traffic, 10Hiddenparma, 06SRE: Integrate code from the private repository into the CDN - https://phabricator.wikimedia.org/T404826#11303514 (10BCornwall) This has also caused our varnish test suite to fail as browser-detection.inc.vcl does not exist. [15:52:52] 06Traffic, 10bot-traffic-requests: FY 25/26 WE 5.4.6 Classify the top 30 spiders by traffic as known bots - https://phabricator.wikimedia.org/T408061#11303546 (10Fabfur) @Joe so, just have a better vision over some points: some of these steps must be performed manually, while other should happen in an automate... [16:04:52] 06Traffic, 10bot-traffic-requests: Global block exception for AddDesc app - https://phabricator.wikimedia.org/T407706#11303598 (10Tgr) 05Open→03Invalid Sorry, I was too quick to jump to conclusions. That's a block placed by a wiki administrator and you need to contact them through one of the addresses... [17:05:46] 06Traffic, 10DNS, 06SRE: [Update DNS Record Request] - wikimedia.org - https://phabricator.wikimedia.org/T408003#11303825 (10JKelsoteel-WMF) Thank you @BCornwall - works on my end! [17:07:28] 06Traffic, 10Hiddenparma, 06SRE, 13Patch-For-Review: Integrate code from the private repository into the CDN - https://phabricator.wikimedia.org/T404826#11303839 (10Krinkle) >>! In T407966#11299911, @ssingh merged: > %%%[operations/puppet@production] varnish: add conditional to varnish::common::vcl for bet... [19:36:35] 06Traffic, 10Hiddenparma, 06SRE, 13Patch-For-Review: Integrate code from the private repository into the CDN - https://phabricator.wikimedia.org/T404826#11304332 (10ssingh) >>! In T404826#11303839, @Krinkle wrote: >>>! In T407966#11299911, @ssingh merged: >> %%%[operations/puppet@production] varnish: add c... [19:45:40] 06Traffic, 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-cache-text08 in project deployment-prep - https://phabricator.wikimedia.org/T408148#11304367 (10bd808) `lang=shell-session,counterexample bd808@deployment-cache-text08.deployment-prep.eqiad1:~$ sudo run-puppet-agent... [19:46:33] 06Traffic, 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-cache-text08 in project deployment-prep - https://phabricator.wikimedia.org/T408148#11304371 (10bd808) [19:46:44] 06Traffic, 10Hiddenparma, 06SRE, 13Patch-For-Review: Integrate code from the private repository into the CDN - https://phabricator.wikimedia.org/T404826#11304372 (10bd808) [19:56:39] 06Traffic, 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-cache-text08 in project deployment-prep - https://phabricator.wikimedia.org/T408148#11304390 (10ssingh) Yeah I missed this in the previous fix. I am going to take this tomorrow since now essentially we have to guard... [20:58:30] 06Traffic, 10Hiddenparma, 06SRE, 13Patch-For-Review: Integrate code from the private repository into the CDN - https://phabricator.wikimedia.org/T404826#11304584 (10bd808) > For browser detection, we've created a private gitlab repository to host some code we don't want to be publicly available, to avoid o... [21:19:01] 06Traffic, 10Hiddenparma, 06SRE, 13Patch-For-Review: Integrate code from the private repository into the CDN - https://phabricator.wikimedia.org/T404826#11304609 (10CDanis) >>! In T404826#11304584, @bd808 wrote: >> For browser detection, we've created a private gitlab repository to host some code we don't...