[07:01:59] 10CAS-SSO, 06Infrastructure-Foundations: Login attempts from bd808 get 500 on Debmonitor - https://phabricator.wikimedia.org/T369205#9952727 (10SLyngshede-WMF) It's not something know. We did have an issue on people.wikimedia.org where tokens would get mixed up between applications and users where sent into a... [07:09:05] slyngs: wrt ^^^ the double auth is because debmonitor has never been integrated with SSO [07:09:11] so yes [07:09:14] is expected [07:09:53] Okay, just felt weird and it's always been like that [07:10:23] I do wonder if other have the same issue [07:10:29] debmonitor config requires wmf or nda to access and sets staff and superuser to ops [07:11:25] zgrep 'proxy-server/500' debmonitor.wikimedia.org-access.log* shoes only bryan [07:21:51] Only thing that I can currently think of is the size of the token, I'm just not sure that would even be an issue [07:24:24] We can also set CASDebug On and ask him to try again [07:34:52] be my guest :) [07:34:57] I'm not even here :D [07:35:44] The Ghost of Volans past ... spooky [07:35:54] lol [07:36:00] I'll book Bryan for a debug session :-) [07:38:46] 10CAS-SSO, 06Infrastructure-Foundations: Login attempts from bd808 get 500 on Debmonitor - https://phabricator.wikimedia.org/T369205#9952826 (10SLyngshede-WMF) a:03SLyngshede-WMF @bd808 I think we need to find 20 minutes where we can enable CAS Debugging on Debmonitor and see what is going on. Only guess is... [07:39:41] slyngs: can I leave it to you to delete the saved tokens on the host once they are not needd anymore? [07:39:48] I will forget :) [07:41:03] I'll write it down, otherwise same result [07:41:44] it's on the task description too :) [07:41:48] thx [07:57:00] slyngs: good morning ! if you feel like doing an easy review : https://gerrit.wikimedia.org/r/c/operations/software/netbox-extras/+/1052045 😇 [07:57:38] Sure, btw. More Metrics was released with Netbox 4 support yesterday [08:00:28] nice [08:00:36] awesome! [08:02:16] They didn't use my code and just dropped support for anything older than 4.0, but the official update does actually fix stuff, rather than just work around it [08:03:35] o/ [08:03:55] \o [08:04:00] high five [08:14:20] 10Mail, 06Infrastructure-Foundations: check-dbbackup-time sometimes doesn't send email alerts - https://phabricator.wikimedia.org/T369253#9952897 (10jcrespo) Running it manually it worked every time, so I am confused- it doesn't seem to be a script issue. Could it be a mailing subsystem issue? Checking exim l... [08:16:46] 10Mail, 06Infrastructure-Foundations: Alert email sent from backupmon1001 didn't reach engineer's google inbox (was: check-dbbackup-time sometimes doesn't send email alerts) - https://phabricator.wikimedia.org/T369253#9952907 (10jcrespo) [08:27:44] 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: Capirca setup for routed Ganeti VMs - https://phabricator.wikimedia.org/T367265#9952956 (10ayounsi) 05Open→03Resolved Import script fixed and orphaned IP for the now deleted sretest2005 removed. [09:10:06] slyngs: https://gerrit.wikimedia.org/r/c/operations/software/netbox-deploy/+/1052054 [09:11:21] Nice :-) [09:13:18] mwclient (MediaWiki library for Python) is slowly driving me insane. Copy paste some script I made, with hardcoded values, all is good. Start refactoring, moving the hardcoded variables in settings... BREAKS, but only sometimes [09:21:03] slyngs: that sounds like a lot of fun :) [09:21:21] When it works yes [09:23:30] time to upgrade to Confluence [09:28:23] Confluence is pretty good, but unreasonably expensive [09:28:51] And you need dedicated staff to run the thing [09:58:54] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: No unicast IP ranges announced to peers from eqdfw - https://phabricator.wikimedia.org/T367439#9953260 (10cmooney) Ok change merged, we are now announcing codfw ranges from eqord again: ` cmooney@cr2-eqord> show route advertising-protocol b... [10:40:14] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw row C/D upgrade racking task - https://phabricator.wikimedia.org/T360789#9953434 (10cmooney) >>! In T360789#9941103, @Papaul wrote: > All the cabling is done. I am leaving this task open so when we move the console cables from as... [10:45:39] 10netops, 06Infrastructure-Foundations, 06SRE: Move asw-c-codfw and asw-d-codfw CR uplinks to Spine switches - https://phabricator.wikimedia.org/T366941#9953449 (10cmooney) @papaul sorry I meant to get back to you sooner. I've made decent progress on T369106 and managed to test reimage working ok in one of... [10:45:40] 10netops, 06Infrastructure-Foundations, 06SRE: Move asw-c-codfw and asw-d-codfw CR uplinks to Spine switches - https://phabricator.wikimedia.org/T366941#9953452 (10cmooney) [10:47:21] 10netops, 06Infrastructure-Foundations, 06SRE: Move asw-c-codfw and asw-d-codfw CR uplinks to Spine switches - https://phabricator.wikimedia.org/T366941#9953455 (10cmooney) [11:19:01] 10netops, 06Infrastructure-Foundations, 06SRE: Move IP gateways for codfw row C/D vlans to EVPN Anycast GW - https://phabricator.wikimedia.org/T369274 (10cmooney) 03NEW p:05Triage→03Medium [11:19:31] 10netops, 06Infrastructure-Foundations, 06SRE: Move IP gateways for codfw row C/D vlans to EVPN Anycast GW - https://phabricator.wikimedia.org/T369274#9953512 (10cmooney) [11:19:32] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Codfw row C/D switch installation & configuration - https://phabricator.wikimedia.org/T364095#9953513 (10cmooney) [11:20:02] 10netops, 06Infrastructure-Foundations, 06SRE: Move asw-c-codfw and asw-d-codfw CR uplinks to Spine switches - https://phabricator.wikimedia.org/T366941#9953514 (10cmooney) [11:20:04] 10netops, 06Infrastructure-Foundations, 06SRE: Move IP gateways for codfw row C/D vlans to EVPN Anycast GW - https://phabricator.wikimedia.org/T369274#9953515 (10cmooney) [11:20:05] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Codfw row C/D switch installation & configuration - https://phabricator.wikimedia.org/T364095#9953516 (10cmooney) [11:27:21] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Get test host connected to codfw row c/d lsw's - https://phabricator.wikimedia.org/T367512#9953540 (10cmooney) All is working well on the test-host. Well puppet was giving me a headache but I just skipped all that :) Reimage works an... [11:27:32] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: No unicast IP ranges announced to peers from eqdfw - https://phabricator.wikimedia.org/T367439#9953535 (10cmooney) 05Open→03Resolved All seems good with the policy changes now, closing task. [14:18:02] 10SRE-tools, 10conftool, 06DBA, 06Infrastructure-Foundations, 10Spicerack: Spicerack support for dbctl - https://phabricator.wikimedia.org/T362893#9954117 (10ABran-WMF) [15:31:39] 10SRE-tools, 06Infrastructure-Foundations, 10Puppet-Infrastructure, 10Spicerack, and 2 others: Migrate puppet merges to a cookbook - https://phabricator.wikimedia.org/T366355#9954359 (10elukey) Getting back to this after having read more puppet code/configs, I think I have a clearer idea now :) We can eas... [21:25:17] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Configure QoS marking and policy across network - https://phabricator.wikimedia.org/T339850#9955217 (10cmooney)