[03:31:57] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 2 others: codfw:frack:servers migration task - https://phabricator.wikimedia.org/T375151#10198601 (10Papaul) [06:16:42] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade Management routers to 22.4R3-S2 - https://phabricator.wikimedia.org/T369504#10198646 (10ayounsi) Let's use the latest recommended, so 23. Thx! [06:16:52] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade Management routers to 23.4R2-S2 - https://phabricator.wikimedia.org/T369504#10198647 (10ayounsi) [06:30:44] elukey: hello! fyi there are pending diff for cr1-eqiad related to aux-k8s*1003 [06:35:11] o/ [06:35:37] I am still not at my desk but there shouldn't be, yesterday I created some new etcd nodes but they shouldn't have any BGP config [06:36:00] and the 1003 Vms that I created should already have a BGP config in place, calico works etc.. [07:42:31] ok properly at my desk [07:43:08] checking homer, it will take a while :D [07:56:05] I have no idea why only cr1 shows diffs, but they look good [07:57:45] show bgp neighbor confirms that no session is established [07:58:02] very weird [07:58:05] anyway, committing [07:58:11] at this point calico is up but not really [08:13:28] perfect I see BGP sessions [08:13:31] thanks for the ping! [08:16:17] no pb! thx for fixing it [10:25:15] 10netops, 06Infrastructure-Foundations, 06SRE: Add link from cloudsw1-e4-eqiad to cloudsw1-f4-eiqad - https://phabricator.wikimedia.org/T372061#10199000 (10cmooney) With the gnmi stats in place we see fairly consistent drops on these links from cloudsw1-d5-eqiad: https://grafana-rw.wikimedia.org/d/5p97dAASz... [10:26:32] XioNoX: I hope/think this is the fix https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1077660 [10:26:44] for sretest2001 [10:27:12] elukey: cool! [11:20:44] 10netops, 06DC-Ops, 10fundraising-tech-ops, 06Infrastructure-Foundations, and 3 others: codfw:frack:rack/install/configuration new switches - https://phabricator.wikimedia.org/T374587#10199081 (10cmooney) >>! In T374587#10160970, @ayounsi wrote: > It would indeed be great to have redundancy for the `fmsw`,... [12:38:51] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE: CloudVPS: IPv6 in codfw1dev - https://phabricator.wikimedia.org/T245495#10199231 (10aborrero) [12:41:31] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE: netbox: create IPv6 entries for Cloud VPS - https://phabricator.wikimedia.org/T374712#10199227 (10aborrero) 05Open→03Resolved [12:41:37] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE: Cloud IPv6 subnets - https://phabricator.wikimedia.org/T187929#10199229 (10aborrero) [12:48:21] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE: cloudsw: codfw: enable IPv6 - https://phabricator.wikimedia.org/T374713#10199236 (10aborrero) Created: * https://netbox.wikimedia.org/ipam/prefixes/1085/ * https://netbox.wikimedia.org/ipam/prefixes/1086/ * https://netbox.... [13:45:25] FIRING: SystemdUnitFailed: conftool2git.service on puppetserver1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:15:25] RESOLVED: SystemdUnitFailed: conftool2git.service on puppetserver1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:23:39] 10Mail, 06Infrastructure-Foundations, 06SRE: Lisa@wikipedia.org is receiving a large number of donor responses - https://phabricator.wikimedia.org/T375643#10199548 (10MBywater-WMF) Hi all, thanks for your help on this! Feel free to ping me if you need any assistance from ITS. [14:29:11] 10Mail, 06Infrastructure-Foundations, 06SRE: Lisa@wikipedia.org is receiving a large number of donor responses - https://phabricator.wikimedia.org/T375643#10199563 (10nisrael) For all contributors to this group I do want to stress that this is an urgent issue. We cannot have Lisa receiving donor responses to... [14:30:16] 10Mail, 06Infrastructure-Foundations, 06SRE: Lisa@wikipedia.org is receiving a large number of donor responses - https://phabricator.wikimedia.org/T375643#10199581 (10jhathaway) @nisrael would it be possible to provide an example raw message, including headers? [14:36:28] 10Mail, 06Infrastructure-Foundations, 06SRE: Lisa@wikipedia.org is receiving a large number of donor responses - https://phabricator.wikimedia.org/T375643#10199623 (10nisrael) Do you mean an example of one of the responses she's been receiving? [14:57:54] 10Mail, 06Infrastructure-Foundations, 06SRE: Lisa@wikipedia.org is receiving a large number of donor responses - https://phabricator.wikimedia.org/T375643#10199728 (10jhathaway) >>! In T375643#10199623, @nisrael wrote: > By this, do you mean an example of one of the responses she's been receiving? yes, exactly [15:03:40] 10Mail, 06Infrastructure-Foundations, 06SRE: Lisa@wikipedia.org is receiving a large number of donor responses - https://phabricator.wikimedia.org/T375643#10199771 (10MBywater-WMF) @nisrael Here are some instructions on hot get email headers: https://support.google.com/mail/answer/29436?hl=en [15:29:46] moritzm: would love some feedback on adding EFI support our patman configs, https://gerrit.wikimedia.org/r/c/operations/puppet/+/1077740 [15:30:01] if anyone else has partman experience or feedback, I would love that as well [15:33:54] jhathaway: o/ he's out today, bank holiday [15:34:10] damn it [15:34:14] thanks elukey [15:34:43] partman is such a crazy world, where few would dare enter [15:34:47] np! Have a good rest of the day, logging off [15:34:50] I totally agree :D [15:34:55] you too [15:36:55] 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade Management routers to 23.4R2-S2 - https://phabricator.wikimedia.org/T369504#10199906 (10Papaul) [15:39:50] jhathaway: you've been fooled, Luca is the leading partman expert at WMF [15:39:55] ;) [15:40:35] I'll add him to the review! [17:20:34] 10Mail, 06Infrastructure-Foundations, 06SRE: Lisa@wikipedia.org is receiving a large number of donor responses - https://phabricator.wikimedia.org/T375643#10200317 (10nisrael) @jhathaway I can attempt to do this, but I don't have access to this inbox and it's getting a bit techy. It may take me a bit to get... [17:28:08] 10Mail, 06Infrastructure-Foundations, 06SRE: Lisa@wikipedia.org is receiving a large number of donor responses - https://phabricator.wikimedia.org/T375643#10200360 (10Dzahn) For the record, the redirect still exists as it did in the past. Our MX server exim alias file for wikipedia.org has ` 45 # Lisa -... [17:45:28] 10Mail, 06Infrastructure-Foundations, 06SRE: Lisa@wikipedia.org is receiving a large number of donor responses - https://phabricator.wikimedia.org/T375643#10200444 (10Dzahn) Is it possible the outgoing fundraising mails have a ` Reply-To:`-header of lisa@wikimedia.org, maybe through a typo somewhere? [18:11:45] 10Mail, 06Infrastructure-Foundations, 06SRE: Lisa@wikipedia.org is receiving a large number of donor responses - https://phabricator.wikimedia.org/T375643#10200559 (10jhathaway) Our logs on our inbound postfix servers show the alias being applied correctly as well: ` 2024-10-03T13:42:05.863401+00:00 mx-in10... [18:15:28] 10Mail, 06Infrastructure-Foundations, 06SRE: Lisa@wikipedia.org is receiving a large number of donor responses - https://phabricator.wikimedia.org/T375643#10200572 (10jhathaway) >>! In T375643#10200317, @nisrael wrote: > @jhathaway I can attempt to do this, but I don't have access to this inbox and it's gett... [18:31:52] 10Mail, 06Infrastructure-Foundations, 06SRE: Lisa@wikipedia.org is receiving a large number of donor responses - https://phabricator.wikimedia.org/T375643#10200651 (10Dzahn) Maybe you can also just send the fundraising email to our inboxes, like treat as if we were the normal recipients. [19:05:00] 10Mail, 06Infrastructure-Foundations, 06SRE: Lisa@wikipedia.org is receiving a large number of donor responses - https://phabricator.wikimedia.org/T375643#10200718 (10nisrael) Oh sure I can do that! @Dzahn just sent you a test. Let me know if there's anyone else I should include. [19:10:18] 10netops, 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 06SRE: cloudsw: codfw: enable IPv6 - https://phabricator.wikimedia.org/T374713#10200735 (10cmooney) >>! In T374713#10199236, @aborrero wrote: > Created: Thanks! I've made some minor edits to them in Netbox btw, just some things... [19:20:41] 10Mail, 06Infrastructure-Foundations, 06SRE: Lisa@wikipedia.org is receiving a large number of donor responses - https://phabricator.wikimedia.org/T375643#10200750 (10jhathaway) please send me one as well, thanks [20:15:56] 10Mail, 06Infrastructure-Foundations, 06SRE: Lisa@wikipedia.org is receiving a large number of donor responses - https://phabricator.wikimedia.org/T375643#10200860 (10nisrael) @jhathaway done! [21:28:05] 10Mail, 06Infrastructure-Foundations, 06SRE: Lisa@wikipedia.org is receiving a large number of donor responses - https://phabricator.wikimedia.org/T375643#10200971 (10jhathaway) @nisrael there is nothing obvious that I see in the email that would indicate why replies are arriving at `lisa@wikimedia.org`. The... [21:29:21] 10Mail, 06Infrastructure-Foundations, 06SRE: Lisa@wikipedia.org is receiving a large number of donor responses - https://phabricator.wikimedia.org/T375643#10200976 (10cscott) People are perhaps manually cut-and-pasting the name from the "from" header, instead of using reply-to? [21:38:10] 10Mail, 06Infrastructure-Foundations, 06SRE: Lisa@wikipedia.org is receiving a large number of donor responses - https://phabricator.wikimedia.org/T375643#10201015 (10jhathaway) >>! In T375643#10200976, @cscott wrote: > People are perhaps manually cut-and-pasting the name from the "from" header, instead of u...