[01:35:28] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: codfw: Relocate servers racked in U27 in all racks in rowA and rowB - https://phabricator.wikimedia.org/T326564 (10Papaul) p:05Triage→03Medium
[02:17:08] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: codfw: Relocate servers racked in U27 in all racks in rowA and rowB - https://phabricator.wikimedia.org/T326564 (10Papaul)
[07:19:14] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: [eqiad] faulty VC optics - https://phabricator.wikimedia.org/T325803 (10ayounsi) asw2-b-eqiad:fpc1:1/1 is still showing errors...  Next step will be to replace the fiber between the two (already replaced) optics.  @Jclark-ctr let me know when woul...
[08:22:20] <slyngs>	 I'm trying to update the Puppet facts so that PCC will work on my new VM, but I don't have access to pcc-db1001.puppet-diffs.eqiad1.wikimedia.cloud. Can someone run the systemctl start pcc_facts_processor.service command ? Otherwise I'll just wait a bit, it should run in an hour or two anyway
[08:32:32] <moritzm>	 on it
[08:32:52] <moritzm>	 done
[08:35:46] <slyngs>	 It works, wuhuuu :-)
[08:36:04] <RhinosF1>	 moritzm: is it worth just adding slyngs to the project so they can do it in future?
[08:41:04] <moritzm>	 indeed, just added
[08:43:34] <slyngs>	 I also just update the documentation, there's a path from one wikitech page to a script in the puppet repo, which then points to another page in WikiTech. I think is't might be better to just point people to the right documentation from the start
[08:54:48] <wikibugs>	 10Mail, 10Infrastructure-Foundations, 10MediaWiki-Email: Gmail vacation responder sends replies to watchlist notification emails - https://phabricator.wikimedia.org/T326608 (10Aklapper)
[09:14:53] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-OnFire, and 2 others: Upgrade POPs asw to Junos 21 - https://phabricator.wikimedia.org/T316532 (10ayounsi) Same issue with `rcp: /var/run/./vjunos-install.sh: Read-only file system` and then `mount: /dev/ad0s1a : Resource temporarily unavailable`, which...
[09:18:20] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-OnFire, and 2 others: Upgrade POPs asw to Junos 21 - https://phabricator.wikimedia.org/T316532 (10ayounsi) Note that removing ` [edit system] -   internet-options { -       tcp-drop-synfin-set; -       no-tcp-reset drop-all-tcp; -   } ` Is needed otherwi...
[09:52:27] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-OnFire, and 2 others: Upgrade POPs asw to Junos 21 - https://phabricator.wikimedia.org/T316532 (10ayounsi) The pre-upgrade went fine on asw1-eqsin, so I guess the ulsfo issue is a corrupted storage.  The last step for eqsin is a reboot, so I'll maintain...
[14:46:59] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: codfw: Relocate servers racked in U27 in all racks in rowA and rowB - https://phabricator.wikimedia.org/T326564 (10Papaul) @ayounsi @cmooney I have 2 questions  1- I have a total of 17 switches received so 1 is going to be used as the cloudsw in r...
[15:04:55] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-eqiad: [eqiad] faulty VC optics - https://phabricator.wikimedia.org/T325803 (10Jclark-ctr) asw2-b-eqiad: fpc1:1/1   Cleaned fiber and replaced optic
[15:05:36] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: codfw: Relocate servers racked in U27 in all racks in rowA and rowB - https://phabricator.wikimedia.org/T326564 (10ayounsi) 1/ 1 ToR per rack = 8x2 + 1 spare = 17, so indeed 1 dedicated to WMCS  2/ A1 and B1 would make sens, and would match eqiad...
[21:52:54] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team (Kanban): cloud: decide on general idea for having cloud-dedicated hardware provide service in the cloud realm & the internet - https://phabricator.wikimedia.org/T296411 (10nskaggs) 05In progress→03Resolved As https://wikitech.wikimed...