[03:19:56] 10netbox, 10Infrastructure-Foundations: Agree how to document intra-DC patch panels in Netbox - https://phabricator.wikimedia.org/T293221 (10wiki_willy) Thanks @ayounsi and @cmooney for all your feedback and forward thinking suggestions around this. What I'm leaning towards on this is to only document the inf... [04:36:09] Wikimedia sites are down for me. Does anyone have the same issue? [05:01:38] working again^^ [08:29:06] Nahid: what issues did you had? This is an SRE sub-team specific channel, in general is better to report such issues in #wikimedia-operations [10:05:06] 10netbox, 10Infrastructure-Foundations, 10SRE, 10SRE-Access-Requests: Grant cn=nda some sort of read only access to Netbox - https://phabricator.wikimedia.org/T302870 (10ayounsi) 05Open→03Stalled I suggest we first upgrade Netbox to 3.1 (or most likely 3.2 by the time T296452) that will allow us to cle... [10:05:36] 10netbox, 10Infrastructure-Foundations, 10SRE, 10SRE-Access-Requests: Grant cn=nda some sort of read only access to Netbox - https://phabricator.wikimedia.org/T302870 (10ayounsi) [10:05:40] 10netbox, 10Infrastructure-Foundations: Upgrade Netbox to 3.1 - https://phabricator.wikimedia.org/T296452 (10ayounsi) [10:07:06] 10netbox, 10Infrastructure-Foundations, 10SRE: Grant cn=nda some sort of read only access to Netbox - https://phabricator.wikimedia.org/T302870 (10ayounsi) -sre-access-requests for now, to clear the Clinic Duty dashboard. [12:30:23] 10Puppet, 10Infrastructure-Foundations, 10observability, 10cloud-services-team (Kanban): 2 systemctl services failing on cloudcontrol hosts: prometheus-openstack-exporter and logrotate - https://phabricator.wikimedia.org/T303511 (10jcrespo) [12:31:34] 10Puppet, 10Infrastructure-Foundations, 10observability, 10cloud-services-team (Kanban): 2 systemctl services failing on cloudcontrol hosts: prometheus-openstack-exporter and logrotate - https://phabricator.wikimedia.org/T303511 (10jcrespo) Feel free to merge if this is the same as the other ticket I refer... [15:00:52] topranks, XioNoX: FYI no more scap for deploying homer, now that we have all hosts on bullseye it's just the cookbook. I've updated the docs, but maybe you have muscle memory ;) [15:01:56] awesome :) [15:02:03] thanks for the heads up! I'll likely need to do it for the first time in coming days so very timely :) [16:24:46] 10netops, 10Infrastructure-Foundations: Agree how to handle port-block speeds for QFX5120-48Y - https://phabricator.wikimedia.org/T303529 (10cmooney) p:05Triage→03Medium [16:56:42] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q3:(Need By: TBD) rack/setup/install 2 new labstore hosts - https://phabricator.wikimedia.org/T302981 (10Jclark-ctr) @nskaggs Would we be able to Rack these in New Wmcs Dedicated racks? E4 , F4? [16:56:57] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q3:(Need By: TBD) rack/setup/install 2 new labstore hosts - https://phabricator.wikimedia.org/T302981 (10Jclark-ctr) [17:07:10] 10Mail, 10Infrastructure-Foundations: Provide an easier way to drop spam mail - https://phabricator.wikimedia.org/T298110 (10jhathaway) 05Open→03Resolved [17:09:58] 10Mail, 10Infrastructure-Foundations, 10SRE: mx1001.wikimedia.org mail delivery timeouts - https://phabricator.wikimedia.org/T299107 (10jhathaway) 05Open→03Resolved We are no longer seeing the timeouts after setting the sysctl net.ipv4.tcp_fastopen_blackhole_timeout_sec sysctl to 3600 which restores the... [17:10:51] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q3:(Need By: TBD) rack/setup/install 2 new labstore hosts - https://phabricator.wikimedia.org/T302981 (10RobH) Looks like I filed both T302981 & T299610, and T299610 has less recent details, and wasnt linked into T286588, so declining T... [17:18:59] 10Puppet, 10Infrastructure-Foundations: Where to Put Community Modules? - https://phabricator.wikimedia.org/T302423 (10jhathaway) a:03jhathaway [18:21:01] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q3:(Need By: TBD) rack/setup/install 2 new labstore hosts - https://phabricator.wikimedia.org/T302981 (10nskaggs) @Jclark-ctr I would want confirmation from infa foundations that all the necessary network connectivity is present. From w... [18:45:51] 10CAS-SSO, 10Infrastructure-Foundations, 10SRE: idp.wikimedia.org asking twice for YubiKey - https://phabricator.wikimedia.org/T258029 (10MoritzMuehlenhoff) 05Open→03Resolved a:03MoritzMuehlenhoff This was fixed by setting u2f_token_expiry_days to 3650 last year, marking as resolved. [18:46:21] 10CAS-SSO, 10Infrastructure-Foundations, 10SRE: Investigate/enable new actuators for U2F token management - https://phabricator.wikimedia.org/T277837 (10MoritzMuehlenhoff) 05Open→03Declined U2F has been deprecated by Chrome, will be replaced by webauth.