[01:44:21] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade Eqiad row E-F Spines to JunOS 22.2R3 - https://phabricator.wikimedia.org/T366361#9853206 (10OKJ04)
[01:44:27] <wikibugs>	 06Traffic, 06SRE: Anycast NTP and update the list of timeservers for P:systemd::timesyncd - https://phabricator.wikimedia.org/T366360#9853207 (10OKJ04)
[01:45:44] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Include vlans with defined IRB int in device vlans even if no port present - https://phabricator.wikimedia.org/T366348#9853219 (10OKJ04)
[01:51:44] <wikibugs>	 06Traffic, 06SRE: Anycast NTP and update the list of timeservers for P:systemd::timesyncd - https://phabricator.wikimedia.org/T366360#9853309 (10JJMC89)
[01:51:52] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: Upgrade Eqiad row E-F Spines to JunOS 22.2R3 - https://phabricator.wikimedia.org/T366361#9853308 (10JJMC89)
[01:53:05] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Include vlans with defined IRB int in device vlans even if no port present - https://phabricator.wikimedia.org/T366348#9853321 (10JJMC89)
[07:43:31] <wikibugs>	 06Traffic, 06SRE: Anycast ns1.wikimedia.org - https://phabricator.wikimedia.org/T366193#9853619 (10ayounsi) I think the difficult part is where to stop the overengineering, for example it could make sens to use Liberica to healthcheck/advertise one of the NS anycast IP, but it might not be worth using a differ...
[08:09:16] <wikibugs>	 06Traffic, 06SRE: Anycast ns1.wikimedia.org - https://phabricator.wikimedia.org/T366193#9853670 (10ayounsi) > - i.e. one in Germany which will pick ns0 rather than lower latency ns2 Seems like the main one is adguard-dns.com, which picks them randomly.  https://w.wiki/AGmr  We can't really afford to email all...
[12:11:22] <wikibugs>	 06Traffic, 06SRE: Anycast NTP and update the list of timeservers for P:systemd::timesyncd - https://phabricator.wikimedia.org/T366360#9854526 (10ayounsi) Moving the dynamic nature of NTP definition to some automated system instead of human or Puppet is a great idea :) Human as in right now for network devices,...
[12:22:44] <wikibugs>	 06Traffic: Use IPIP encapsulation on lvs<-->text cluster - https://phabricator.wikimedia.org/T366466 (10Vgutierrez) 03NEW
[12:23:30] <wikibugs>	 06Traffic: Use IPIP encapsulation on lvs<-->text cluster - https://phabricator.wikimedia.org/T366466#9854552 (10Vgutierrez) p:05Triage→03Medium
[13:30:38] <wikibugs>	 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: Juniper: use export-format state-data json compact - https://phabricator.wikimedia.org/T362523#9854770 (10ayounsi) 05Open→03Resolved > Our engineering team has now indicated that the compact json is not supported, due to hardware limitations...
[13:36:58] <cdanis>	 vgutierrez: wow IPIP everywhere :D
[13:38:02] <XioNoX>	 it's awesome
[13:39:06] <vgutierrez>	 cdanis: one DC at a time :)
[13:39:12] <vgutierrez>	 just text@ulsfo today
[14:20:00] <wikibugs>	 06Traffic, 06SRE: Anycast NTP and update the list of timeservers for P:systemd::timesyncd - https://phabricator.wikimedia.org/T366360#9855013 (10ssingh) >>! In T366360#9854526, @ayounsi wrote: > Moving the dynamic nature of NTP definition to some automated system instead of human or Puppet is a great idea :) >...
[14:34:26] <wikibugs>	 06Traffic, 13Patch-For-Review: Use IPIP encapsulation on lvs<-->text cluster - https://phabricator.wikimedia.org/T366466#9855095 (10Vgutierrez)
[15:43:43] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw row C/D upgrade racking task - https://phabricator.wikimedia.org/T360789#9855466 (10Papaul)
[15:45:34] <wikibugs>	 06Traffic, 06SRE: Anycast NTP and update the list of timeservers for P:systemd::timesyncd - https://phabricator.wikimedia.org/T366360#9855483 (10ayounsi) > Last time we rolled out this change, it was simply updating modules/install_server/files/autoinstall/common.cfg. Do you have any other place in mind where...
[15:57:53] <wikibugs>	 06Traffic, 06SRE: Anycast NTP and update the list of timeservers for P:systemd::timesyncd - https://phabricator.wikimedia.org/T366360#9855543 (10ssingh) >>! In T366360#9855483, @ayounsi wrote: >> Last time we rolled out this change, it was simply updating modules/install_server/files/autoinstall/common.cfg. Do...
[16:19:09] <wikibugs>	 06Traffic, 06SRE: Anycast ns1.wikimedia.org - https://phabricator.wikimedia.org/T366193#9855670 (10BBlack) >>! In T366193#9853619, @ayounsi wrote: > I think the difficult part is where to stop the overengineering, for example it could make sens to use Liberica to healthcheck/advertise one of the NS anycast IP,...
[16:26:17] <wikibugs>	 06Traffic, 06SRE: Anycast NTP and update the list of timeservers for P:systemd::timesyncd - https://phabricator.wikimedia.org/T366360#9855758 (10BBlack) Re: "same logic" - they're different protocols, different hierarchies, and much different on the client behavior front as well.  It doesn't make sense to shar...
[16:54:35] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw row C/D upgrade racking task - https://phabricator.wikimedia.org/T360789#9855905 (10Papaul)
[16:54:38] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw row C/D upgrade racking task - https://phabricator.wikimedia.org/T360789#9855906 (10Papaul) @cmooney all good on lsw1-d4, lsw1-c2 and lsw1-d8
[18:12:01] <wikibugs>	 06Traffic: Default gadget not loaded yet while Mediawiki:Mobile.css - https://phabricator.wikimedia.org/T366517 (10Nux) 03NEW
[19:21:01] <wikibugs>	 06Traffic: Default gadget not loaded yet while Mediawiki:Mobile.css - https://phabricator.wikimedia.org/T366517#9856777 (10Func) This is because ResourceLoader modules would be inlined into the page HTML, either as a stylesheet link and loaded directly (CSS-only gadgets) or in the `RLPAGEMODULES` array and loade...
[19:39:57] <wikibugs>	 06Traffic: Default gadget not loaded yet while Mediawiki:Mobile.css - https://phabricator.wikimedia.org/T366517#9856856 (10Nux) Yeah, I already heard in T362747 that it's probably 7 days, but might be up to 14 days (maybe).  I think this: ` ... <link rel="stylesheet" href="/w/load.php?lang=pl&amp;modules=ext.fla...
[20:51:23] <wikibugs>	 06Traffic, 06Infrastructure-Foundations, 06SRE: Slowly ramping up traffic to the Brazil data center (magru) and related geo-maps - https://phabricator.wikimedia.org/T359054#9857120 (10CDanis) Results after adding BR.ix are in.  The set of countries that magru improves hasn't changed: BR, CL, AR, UY, PY, BO...
[20:51:31] <wikibugs>	 06Traffic: Craft geo-maps file to create lowest-latency routes from south america - https://phabricator.wikimedia.org/T363722#9857124 (10CDanis) Results after adding BR.ix are in.  The set of countries that magru improves hasn't changed: BR, CL, AR, UY, PY, BO  PE has magru better for many but not nearly all use...
[20:56:36] <wikibugs>	 06Traffic: Craft geo-maps file to create lowest-latency routes from south america - https://phabricator.wikimedia.org/T363722#9857179 (10ssingh) Thanks for the great analysis, Chris!
[22:01:46] <wikibugs>	 06Traffic: Default gadget not loaded yet while Mediawiki:Mobile.css - https://phabricator.wikimedia.org/T366517#9857398 (10Tgr) Gadgets and most other ResourceLoader modules are not inlined. `/w/load.php?lang=pl&modules=startup` will load the module registry (with `Cache-Control: max-age=300`), and the module re...
[22:22:52] <wikibugs>	 06Traffic: Default gadget not loaded yet while Mediawiki:Mobile.css - https://phabricator.wikimedia.org/T366517#9857450 (10Func)
[23:12:47] <wikibugs>	 06Traffic: Default gadget not loaded yet while Mediawiki:Mobile.css - https://phabricator.wikimedia.org/T366517#9857553 (10Func) For this particular case: `lang=html <title>Wikipedia, wolna encyklopedia</title> <script> ... RLSTATE={"skins.vector.user.styles":"ready","ext.gadget.wikiflex":"ready","ext.gadget.inf...
[23:16:04] <wikibugs>	 06Traffic: Newly added default gadget not loaded for anon users for days while migrating away from Common.css and Mobile.css - https://phabricator.wikimedia.org/T366517#9857557 (10Func)
[23:53:15] <wikibugs>	 06Traffic: Newly added default gadget not loaded for anon users for days while migrating away from Common.css and Mobile.css - https://phabricator.wikimedia.org/T366517#9857608 (10Nux) Thanks Func. Yes, I removed infobox styles both from mobile CSS (Mobile.css) and desktop CSS (Common.css). The problem was last...