[02:10:57] 10netbox, 10Infrastructure-Foundations, 10SRE: Grant cn=nda some sort of read only access to Netbox - https://phabricator.wikimedia.org/T302870 (10wiki_willy) Thanks for checking @ayounsi. My personal opinion on the contacts list is to restrict it if possible. I don't see any issues sharing the generic ven... [07:26:35] 10netbox, 10Infrastructure-Foundations: Netbox: use Custom Model Validation - https://phabricator.wikimedia.org/T310590 (10ayounsi) [07:26:38] 10SRE-tools, 10netbox, 10DNS, 10Infrastructure-Foundations, and 2 others: sre.dns.netbox cookbook dosn't support period terminated domains - https://phabricator.wikimedia.org/T306809 (10ayounsi) [07:32:08] 10netbox, 10Infrastructure-Foundations, 10Patch-For-Review: netbox: drop profile::netbox::active_server parameter - https://phabricator.wikimedia.org/T309034 (10ayounsi) a:05ayounsi→03None [07:37:09] 10netbox, 10Infrastructure-Foundations: Netbox: replace CSV dump with more frequent DB dumps - https://phabricator.wikimedia.org/T310615 (10ayounsi) [07:37:11] 10netbox, 10Infrastructure-Foundations, 10Patch-For-Review: Upgrade Netbox to 3.2 - https://phabricator.wikimedia.org/T296452 (10ayounsi) [07:38:22] 10netbox, 10Infrastructure-Foundations, 10Patch-For-Review: netbox network report improvment - https://phabricator.wikimedia.org/T310299 (10ayounsi) a:05ayounsi→03cmooney [07:45:57] 10SRE-tools, 10netbox, 10DNS, 10Infrastructure-Foundations, and 2 others: sre.dns.netbox cookbook dosn't support period terminated domains - https://phabricator.wikimedia.org/T306809 (10ayounsi) 05Stalled→03Open [08:17:51] 10Puppet, 10Infrastructure-Foundations, 10SRE, 10Traffic-Icebox: Remove old letsencrypt puppet module - https://phabricator.wikimedia.org/T221268 (10taavi) 05Open→03Resolved a:03Andrew https://gerrit.wikimedia.org/r/c/operations/puppet/+/655762 [08:59:10] following the Ganeti cluster change in Netbox I'm now getting a traceback in the decom cookbook when removing a VM, known issue? https://phabricator.wikimedia.org/P30138 [09:05:32] looking [09:07:46] that prevented the VM from being removed, I'll yank it manually with the ganeti CLI tools [09:08:11] and we can use the decom of webperf1002 next week as a test once the cookbook is adapted [09:08:31] ok, I can maybe send a patch, but that would need a spicerack release [09:17:05] there's no hurry, I'm in the process of fixing up the fallout of the webperf2002 decom, we can sort this all out properly with a new spicerack release on Monday [09:19:10] yeah, it's not that trivial as the way the data is structured changed [09:36:09] what did I broke? [09:37:46] ah right... damn [09:40:37] ok so the spicerack side of it I think I can hotfix it for now moritzm, XioNoX [09:40:43] just relaxing things [09:47:46] I knew we should have kept the old FQDN names btw :/ [09:50:27] volans: tech debt :) [09:50:47] (hardcoding urls) [09:51:07] but also if we knew we would have not done it that way :( [09:51:19] sent patch for the decom [09:51:26] could you please have a look and potentially test it? [09:51:43] I have a train to catch in ~1:30 and I will not have my laptop with me [09:51:48] https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/808209 [09:53:00] volans: it's really fine, please go back to your day off, let's fix this on Monday [09:53:19] I'd like if decom/makevm were unblocked at least in the 2 main dcs [09:54:10] do you know if makevm is broken? [09:54:34] no idea, I only ran into it with a decom [09:54:44] from the code I think it might just work [09:55:14] I can test 808209 with the decom of webperf2002 [09:55:17] the above request for the patch was for XioNoX [09:55:30] I don't have time to test it right now, but the change lgtm [09:55:48] it won't just work for half of drmrs and the test cluster [09:59:54] let's just drop a note to the rest of SREs that VM decoms should be postponed to Monday and then fix this for good [10:00:20] even if someone runs into this today, it's straightforward to clean up as well [10:02:39] XioNoX: the comment says that explicitly :D [10:02:58] to unblock eqias/codfw, the other sites don't have decoms often anyway [10:03:06] but yes I'll do a proper fix on monday [10:03:09] really sorry about that [10:03:41] 10SRE-tools, 10Ganeti, 10Infrastructure-Foundations, 10SRE, and 2 others: Create a spicerack cookbook to empty a ganeti node from VMs - https://phabricator.wikimedia.org/T203964 (10MoritzMuehlenhoff) [10:03:47] do you what mutante was referring to in -sre about the dns step of the makevm failing? [10:03:54] is related? [10:04:35] don't hate me but I have to leave it to you if you want to merge the hotfix or not. [10:05:50] 10SRE-tools, 10Ganeti, 10Infrastructure-Foundations, 10SRE: sre.ganeti.makevm: Allow passing a secondary disk - https://phabricator.wikimedia.org/T300046 (10MoritzMuehlenhoff) [10:06:22] 10SRE-tools, 10Ganeti, 10Infrastructure-Foundations: sre.ganeti.makevm cook book only allows specifying RAM size in full gigabytes - https://phabricator.wikimedia.org/T230712 (10MoritzMuehlenhoff) [10:06:32] 10SRE-tools, 10Ganeti, 10Infrastructure-Foundations, 10SRE: Cookbook to failover the Ganeti master - https://phabricator.wikimedia.org/T283320 (10MoritzMuehlenhoff) [10:06:46] 10SRE-tools, 10Ganeti, 10Infrastructure-Foundations: Make Spicerack cookbook to resize ganeti VM - https://phabricator.wikimedia.org/T219454 (10MoritzMuehlenhoff) [10:09:19] 10SRE-tools, 10Infrastructure-Foundations, 10Spicerack: sre.ganeti.makevm NXDOMAIN race condition - https://phabricator.wikimedia.org/T309505 (10jbond) 05Open→03Resolved a:03jbond Thsi is fixed now [10:49:59] no problem at all, enjoy your trip and we'll fix this on Monday. I posted a note to the internal SRE channel that VM decoms need to wait until Monday [11:09:06] 10CAS-SSO, 10DBA, 10Infrastructure-Foundations, 10SRE: Repurpose the "cas" database for webauthn tokens - https://phabricator.wikimedia.org/T311300 (10MoritzMuehlenhoff) [11:26:17] 10CAS-SSO, 10DBA, 10Infrastructure-Foundations, 10SRE: Repurpose the "cas" database for webauthn tokens - https://phabricator.wikimedia.org/T311300 (10Marostegui) p:05Triage→03Medium a:03Marostegui @MoritzMuehlenhoff so you want me to drop or truncate this table?: ` root@db1164.eqiad.wmnet[cas]> show... [11:39:14] moritzm: thanks, not sure if worth mentioning also to embedded sres that are not there [11:40:49] good thought, I'll also post to the other internal channel :-) [11:51:22] thx, and again sorry for the trouble [12:29:52] 10CAS-SSO, 10DBA, 10Infrastructure-Foundations, 10SRE: Repurpose the "cas" database for webauthn tokens - https://phabricator.wikimedia.org/T311300 (10MoritzMuehlenhoff) >>! In T311300#8025518, @Marostegui wrote: > @MoritzMuehlenhoff so you want me to drop or truncate this table?: > ` Please drop the enti... [12:35:34] 10CAS-SSO, 10Infrastructure-Foundations, 10SRE: Enable webauthn in CAS to replace U2F - https://phabricator.wikimedia.org/T311236 (10Marostegui) [12:36:02] 10CAS-SSO, 10DBA, 10Infrastructure-Foundations, 10SRE: Repurpose the "cas" database for webauthn tokens - https://phabricator.wikimedia.org/T311300 (10Marostegui) 05Open→03Resolved I have taken a quick backup from those tables: ` root@cumin1001:/home/marostegui/T311300# ls -lh total 56K -rw-r--r-- 1 ro... [12:41:49] 10Puppet, 10Infrastructure-Foundations, 10SRE, 10User-jbond: document puppet/netbox/hiera interaction - https://phabricator.wikimedia.org/T311304 (10jbond) [12:53:33] I guess we can use the decom of the old netbox servers as test for the fix on monday :) [12:58:28] 10CAS-SSO, 10DBA, 10Infrastructure-Foundations, 10SRE: Repurpose the "cas" database for webauthn tokens - https://phabricator.wikimedia.org/T311300 (10MoritzMuehlenhoff) Thanks!