[02:45:06] 10Traffic, 10MediaWiki-General, 10MediaWiki-Platform-Team, 10serviceops, and 4 others: MW returns uncacheable responses for en.wikipedia.org when specific XFF values are sent - https://phabricator.wikimedia.org/T350861 (10sbassett) [08:20:22] 10netops, 10Ganeti, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Investigate Ganeti in routed mode - https://phabricator.wikimedia.org/T300152 (10ayounsi) The above patches should get us as far as DHCP. DHCP is going to be the next big challenge to solve, partly because of the setback of Opti... [11:46:36] 10Traffic: WikiFunctions: Domain Verification for Google Search Console - https://phabricator.wikimedia.org/T355308 (10SCherukuwada) [11:47:00] 10Traffic: WikiFunctions: Domain Verification for Google Search Console - https://phabricator.wikimedia.org/T355308 (10SCherukuwada) [11:49:40] 10Traffic: WikiFunctions: Domain Verification for Google Search Console - https://phabricator.wikimedia.org/T355308 (10SCherukuwada) I have a patch ready if someone would like to review it. https://gerrit.wikimedia.org/r/c/operations/dns/+/991527 [11:52:51] 10Traffic: WikiFunctions: Domain Verification for Google Search Console - https://phabricator.wikimedia.org/T355308 (10Vgutierrez) p:05Triage→03Medium [11:55:16] dr0ptp4kt: :) happy to help when you're around [13:17:45] Thanks vgutierrez . Oh, for the commit message amendment what should be done there? Sorry if I missed it in other comments elsewhere. btullis it looks like we're almost ready on https://gerrit.wikimedia.org/r/c/operations/puppet/+/981352 for deployment (thanks for the additional review BTW). I have the kid morning routine the next couple hours, then meetings, but set a reminder for myself to ping in about three hours in the middle [13:17:45] of a meeting for deployment if that's okay. Bit, on the optimistic side, if it's already done by then m, also good! [13:18:27] * dr0ptp4kt shakes fist at autocomplete [13:29:33] 10netops, 10Infrastructure-Foundations, 10SRE: Verify and Configure ECMP operation for EVPN switches - https://phabricator.wikimedia.org/T334658 (10cmooney) 05Open→03Resolved Closing this. It's a global setting and as per the description we need to keep ports in play to get a load-balance for VXLAN traf... [13:38:22] 10netops, 10Infrastructure-Foundations, 10SRE: Create single Homer BGP group template to cover all variants - https://phabricator.wikimedia.org/T349116 (10cmooney) [13:42:12] 10netops, 10Infrastructure-Foundations, 10SRE: Firewall filter blocking traceroute in underlay QFX5120 EVPN - https://phabricator.wikimedia.org/T348120 (10cmooney) >>! In T348120#9224531, @ayounsi wrote: > Nice rabbit hole! I found this: https://www.reddit.com/r/Juniper/comments/g12qxh/the_right_way_to_allow... [13:46:49] 10netops, 10Infrastructure-Foundations, 10SRE: Re-IP hosts on codfw row A and B to new per-rack vlans/subnets - https://phabricator.wikimedia.org/T354869 (10cmooney) [14:22:41] dr0ptp4kt: as mentioned.. it should state the varnish module.. so "varnish: do X & Y" [15:35:05] Thanks vgutierrez , okay pushed a commit message-only change - https://gerrit.wikimedia.org/r/c/operations/puppet/+/981352 - if you're ready to deploy (heads up btullis ) it would be most appreciated. Thanks! [15:53:21] 10netops, 10Infrastructure-Foundations, 10Prod-Kubernetes, 10SRE, 10serviceops: Test IP-renumbering on kubestage2002.codfw.wmnet - https://phabricator.wikimedia.org/T352883 (10Clement_Goubert) `lang=bash cgoubert@kubestage2002:~$ sudo calicoctl node status Calico process is running. IPv4 BGP status +---... [15:58:07] dr0ptp4kt: ok, moving forward [16:09:13] dr0ptp4kt: got a clean puppet run on cp4037.. I'm reenabling puppet fleet wide now [16:09:39] it will be live in ~30 minutes when puppet runs on the whole fleet [16:09:49] 👓 [16:09:58] ty vgutierrez [16:10:26] dr0ptp4kt: I'd ask for a beer in return but maybe a pack of diapers would be more helpful soon [16:11:02] 🤣 [16:17:51] 10netops, 10Infrastructure-Foundations, 10Prod-Kubernetes, 10SRE, 10serviceops: Test IP-renumbering on kubestage2002.codfw.wmnet - https://phabricator.wikimedia.org/T352883 (10Clement_Goubert) No-op on these nodes, proceeding with the rest. [16:20:02] 10netops, 10Infrastructure-Foundations, 10Prod-Kubernetes, 10SRE, 10serviceops: Test IP-renumbering on kubestage2002.codfw.wmnet - https://phabricator.wikimedia.org/T352883 (10Clement_Goubert) >>! In T352883#9469622, @Clement_Goubert wrote: > `lang=bash > IPv6 BGP status > +-------------------+----------... [16:28:08] 10Traffic, 10Patch-For-Review: WikiFunctions: Domain Verification for Google Search Console - https://phabricator.wikimedia.org/T355308 (10ssingh) 05Open→03Resolved a:03ssingh ` $ dig wikifunctions.org TXT +short "google-site-verification=b5yMq36eaNEQGWpBaatU5KV9s4mjf8m1SoSZ2UmcIII" ` Let us know if the... [16:40:10] 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE, and 2 others: Move lvs2011 primary uplink and connect to new row A/B vlans - https://phabricator.wikimedia.org/T352912 (10cmooney) [16:41:36] 10netops, 10Infrastructure-Foundations, 10Prod-Kubernetes, 10SRE, 10serviceops: Test IP-renumbering on kubestage2002.codfw.wmnet - https://phabricator.wikimedia.org/T352883 (10Clement_Goubert) No-op on the rest of the infra. [16:43:06] 10netops, 10Infrastructure-Foundations, 10Prod-Kubernetes, 10SRE, and 2 others: Update puppet's topology.kubernetes.io/zone logic to take into account the new setup - https://phabricator.wikimedia.org/T352893 (10Clement_Goubert) Summary of deployment from {T352883}: - No-op on all nodes except kubestage200... [17:00:07] the prefetch headers tagging seems to be working pretty well vgutierrez bblack btullis , at least based on kafkacat with some grep and grep -v checks. thank you! [17:04:04] dr0ptp4kt: ack, good stuff. [17:05:28] 10Traffic, 10Data-Engineering, 10Movement-Insights, 10Patch-For-Review: Identify and label prefetch proxy data in our traffic - https://phabricator.wikimedia.org/T346463 (10dr0ptp4kt) It's live and looking good in `kafkacat`. Now we wait a little for stuff to show up in the analytics tables. Thanks @Vgutie... [17:53:45] 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate lvs2011 and lvs2012 to new top-of-rack switches - https://phabricator.wikimedia.org/T348178 (10cmooney) [17:53:54] 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Move lvs2011 primary uplink and connect to new row A/B vlans - https://phabricator.wikimedia.org/T352912 (10cmooney) 05Open→03Resolved Alll done! [17:54:29] 10Traffic, 10Data-Engineering, 10Movement-Insights, 10Patch-For-Review: Identify and label prefetch proxy data in our traffic - https://phabricator.wikimedia.org/T346463 (10dr0ptp4kt) Documentation updated: https://wikitech.wikimedia.org/w/index.php?title=X-Analytics&diff=2140528&oldid=2028273 [17:54:43] 10netops, 10Infrastructure-Foundations, 10SRE: Re-IP hosts on codfw row A and B to new per-rack vlans/subnets - https://phabricator.wikimedia.org/T354869 (10cmooney) [17:54:51] 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate lvs2011 and lvs2012 to new top-of-rack switches - https://phabricator.wikimedia.org/T348178 (10cmooney) [17:54:59] 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE, and 2 others: Move lvs2012 from private1-b-codfw (row) to private1-b2-codfw (rack) vlan - https://phabricator.wikimedia.org/T352918 (10cmooney) [17:55:11] 10netops, 10Infrastructure-Foundations, 10SRE: Re-IP hosts on codfw row A and B to new per-rack vlans/subnets - https://phabricator.wikimedia.org/T354869 (10cmooney) [17:55:19] 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate lvs2011 and lvs2012 to new top-of-rack switches - https://phabricator.wikimedia.org/T348178 (10cmooney) [17:55:27] 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE, and 2 others: Move lvs2011 from private1-a-codfw (row) to private1-a2-codfw (rack) vlan - https://phabricator.wikimedia.org/T352920 (10cmooney) [17:55:42] 10netops, 10Infrastructure-Foundations, 10SRE: Codfw row A/B top-of-rack switch refresh - https://phabricator.wikimedia.org/T327938 (10cmooney) [17:55:53] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Codfw row A-B migration - non-standard device moves - https://phabricator.wikimedia.org/T348128 (10cmooney) 05Open→03Resolved [17:56:04] 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate lvs2011 and lvs2012 to new top-of-rack switches - https://phabricator.wikimedia.org/T348178 (10cmooney) 05Open→03Resolved [17:56:12] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Codfw row A-B migration - non-standard device moves - https://phabricator.wikimedia.org/T348128 (10cmooney) [19:33:22] 10Traffic, 10Patch-For-Review: sre.dns.roll-restart-reboot-wikimedia-dns cookbook sometimes cannot remove downtime - https://phabricator.wikimedia.org/T353779 (10BCornwall) @ssingh I think we shouldn't even bother with pre_action, post_action, and the disable_puppet_on_* at all. We already have the systemd ord... [19:33:39] 10Traffic, 10Patch-For-Review: sre.dns.roll-restart-reboot-wikimedia-dns cookbook sometimes cannot remove downtime - https://phabricator.wikimedia.org/T353779 (10BCornwall) 05Open→03In progress [19:55:46] 10Traffic: ipip-multiqueue-optimizer won't start on server reboot - https://phabricator.wikimedia.org/T355359 (10Vgutierrez) [19:59:50] 10Traffic, 10Data-Engineering, 10Movement-Insights, 10Patch-For-Review: Identify and label prefetch proxy data in our traffic - https://phabricator.wikimedia.org/T346463 (10dr0ptp4kt) It's entering the analytics system based on the following query: ` select http_status, hour, x_analytics_map['prefetch_sec... [20:37:00] 10Traffic, 10Data-Engineering, 10Movement-Insights, 10Patch-For-Review: Identify and label prefetch proxy data in our traffic - https://phabricator.wikimedia.org/T346463 (10fkaelin) Nice! ` pa = spark.table("wmf.pageview_actor").where("""year=2024 and month=1 and day=18 and hour=16""") prefetch_fields = [... [20:53:12] 10Traffic, 10Patch-For-Review: ipip-multiqueue-optimizer won't start on server reboot - https://phabricator.wikimedia.org/T355359 (10BCornwall) 05Open→03Resolved a:03BCornwall Thanks! [23:38:57] 10Acme-chief, 10Traffic: Create automation for DNS registration and related services - https://phabricator.wikimedia.org/T355189 (10BCornwall) 05Open→03In progress p:05Triage→03Medium [23:40:50] 10Acme-chief, 10Traffic: Create automation for registered MarkMonitor DNS and acme-chief/ncredir - https://phabricator.wikimedia.org/T355189 (10BCornwall)