[08:28:11] 10CAS-SSO, 10Infrastructure-Foundations, 10SRE: Update to CAS 6.4 - https://phabricator.wikimedia.org/T293186 (10MoritzMuehlenhoff) [08:50:12] @all we have an acess requests for a new group which git missed in the I/F meeting and wondered if we could do it async here https://gerrit.wikimedia.org/r/c/operations/puppet/+/728648 (cc moritzm jobo ) for me its a +1 [08:51:01] * volans looking [08:52:41] same here, looks sane [08:53:39] +1 for me, no concerns [08:54:24] also A:otrs matches just one host right now [08:54:55] want a +1 on gerrit/task? [08:56:18] looks good for me as well [08:59:07] cool and yes volans i think worth puppting a plus one with a I/F approves™ [09:00:47] I didn't check the gid [09:01:40] jobo: I guess would be best if you could +1 on gerrit with the I/F approval, if unable I can do that too [09:02:21] the gid is fine [09:04:33] Done [09:04:51] thx [09:16:36] 10SRE-tools, 10Infrastructure-Foundations, 10SRE, 10User-ema: wmf-auto-reimage: 'execution expired' on first puppet run - https://phabricator.wikimedia.org/T201317 (10ema) [09:22:43] 10SRE-tools, 10Infrastructure-Foundations, 10SRE, 10User-ema: wmf-auto-reimage: 'execution expired' on first puppet run - https://phabricator.wikimedia.org/T201317 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by ema@cumin2002 for host cp4021.ulsfo.wmnet with OS buster [10:37:56] 10SRE-tools, 10Infrastructure-Foundations: Netbox check: the Uncommitted DNS changes in Netbox should recover more quickly - https://phabricator.wikimedia.org/T293206 (10Volans) p:05Triage→03Medium [11:02:01] 10SRE-tools, 10Infrastructure-Foundations: Spicerack: add support for Alertmanager - https://phabricator.wikimedia.org/T293209 (10Volans) p:05Triage→03Medium [11:04:34] going to reboot sretest1001/sretest1002 to test a cookbook [11:05:27] +1 for me [11:05:51] 10SRE-tools, 10Infrastructure-Foundations, 10SRE, 10User-ema: wmf-auto-reimage: 'execution expired' on first puppet run - https://phabricator.wikimedia.org/T201317 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by ema@cumin2002 for host cp4021.ulsfo.wmnet with OS buster completed: - cp... [11:33:20] 10SRE-tools, 10Infrastructure-Foundations, 10SRE, 10User-ema: wmf-auto-reimage: 'execution expired' on first puppet run - https://phabricator.wikimedia.org/T201317 (10ema) Trying another reimage as follows: ` root@cumin2002:~# cookbook sre.hosts.reimage --os buster --conftool -t T201317 cp4021 2>&1 | ts |... [11:33:42] 10SRE-tools, 10Infrastructure-Foundations, 10SRE, 10User-ema: wmf-auto-reimage: 'execution expired' on first puppet run - https://phabricator.wikimedia.org/T201317 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by ema@cumin2002 for host cp4021.ulsfo.wmnet with OS buster [11:41:12] 10Puppet, 10Infrastructure-Foundations, 10Machine-Learning-Team, 10ORES: Write puppet for redis-sentinel - https://phabricator.wikimedia.org/T210580 (10jijiki) @Ladsgroup is this work still in progress or abandoned? [11:45:46] 10Puppet, 10Infrastructure-Foundations, 10Machine-Learning-Team, 10ORES: Write puppet for redis-sentinel - https://phabricator.wikimedia.org/T210580 (10Majavah) >>! In T210580#7423631, @jijiki wrote: > @Ladsgroup is this work still in progress or abandoned? I'll note that I ended up puppetizing redis-sent... [12:10:16] 10Puppet, 10Infrastructure-Foundations, 10Machine-Learning-Team, 10ORES: Write puppet for redis-sentinel - https://phabricator.wikimedia.org/T210580 (10Ladsgroup) >>! In T210580#7423631, @jijiki wrote: > @Ladsgroup is this work still in progress or abandoned? definitely abandoned for years. [12:32:22] 10Puppet, 10Infrastructure-Foundations, 10Machine-Learning-Team, 10ORES: Write puppet for redis-sentinel - https://phabricator.wikimedia.org/T210580 (10akosiaris) 05Open→03Invalid I 'll close then. This was specifically for the ORES case, T122676, which hasn't happened. @Majavah has been kind enough to... [13:34:35] 10SRE-tools, 10Infrastructure-Foundations, 10SRE, 10User-ema: wmf-auto-reimage: 'execution expired' on first puppet run - https://phabricator.wikimedia.org/T201317 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by ema@cumin2002 for host cp4021.ulsfo.wmnet with OS buster completed: - cp... [13:57:12] going to reimage sretest1001 to trouble shoot reimage issue (cc moritzm ) [14:05:34] sounds good [14:05:58] actully may not be needed :) will update here if i go ahead [18:15:21] FYI I've released a new debmonitor-client_0.3.1 with a fix for the timeout parameter, deployed and tested on sretest1001, tomorrow I'll rollout it to the fleet [18:15:55] there is also a new version of the debmonitor server side to prevent an issue, to be deployed and tested for performance regression, will try do that too tomorrow [18:16:05] logging off for today [19:03:31] 10netops, 10DNS, 10Infrastructure-Foundations, 10ops-drmrs: setup drmrs mgmt prefix/range - https://phabricator.wikimedia.org/T293294 (10RobH) p:05Triage→03High [19:03:45] 10netops, 10DNS, 10Infrastructure-Foundations, 10ops-drmrs: setup drmrs mgmt prefix/range - https://phabricator.wikimedia.org/T293294 (10RobH) [19:05:26] 10netops, 10DNS, 10Infrastructure-Foundations, 10ops-drmrs: setup drmrs mgmt & private prefixs - question on switch status - https://phabricator.wikimedia.org/T293294 (10RobH) [19:06:02] 10netops, 10DNS, 10Infrastructure-Foundations, 10ops-drmrs: setup drmrs mgmt & private prefixs - question on switch status - https://phabricator.wikimedia.org/T293294 (10RobH) I chatted with @cmooney about this in IRC and we cannot see if there is a set pattern to which ranges are used for mgmt, private1,... [19:10:59] 10netops, 10DNS, 10Infrastructure-Foundations, 10ops-drmrs: setup drmrs mgmt & private prefixs - question on switch status - https://phabricator.wikimedia.org/T293294 (10cmooney) Thanks Rob. Yeah not 100% sure what we should allocate. I'm thinking 10.136.0.0/16 for the site seems logical, with 10.136.12... [19:16:02] 10netops, 10DNS, 10Infrastructure-Foundations, 10ops-drmrs: setup drmrs mgmt & private prefixs - question on switch status - https://phabricator.wikimedia.org/T293294 (10BBlack) CC @MMandere as well once we have a decision on the IP prefixes here!