[08:10:04] 10Puppet, 10Infrastructure-Foundations, 10SRE: Puppet should prune stale entries from sudoers.d - https://phabricator.wikimedia.org/T309268 (10Majavah) [09:08:17] 10Puppet, 10Cloud-VPS, 10Infrastructure-Foundations, 10cloud-services-team (Kanban): Remove prod-specific bits from cloud puppetmasters - https://phabricator.wikimedia.org/T309281 (10Majavah) [09:42:12] 10Puppet, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Puppet should prune stale entries from sudoers.d - https://phabricator.wikimedia.org/T309268 (10jbond) similar to the previous task on [[ https://phabricator.wikimedia.org/T214605#5756945 | apt directories ]], i have queried the repo for ma... [09:49:21] 10Puppet, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Puppet should prune stale entries from sudoers.d - https://phabricator.wikimedia.org/T309268 (10jbond) removing `nagios_long_procs` as it was dropped in https://gerrit.wikimedia.org/r/c/operations/puppet/+/723543/4/modules/base/manifests/mo... [09:59:16] 10Puppet, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Puppet should prune stale entries from sudoers.d - https://phabricator.wikimedia.org/T309268 (10jbond) new updated list with removed nagios_long_procs and also with a fixed file list ` sudo cumin '*' 'ls -1 /etc/sudoers.d/ | grep -Ev "mw-... [10:19:42] 10Puppet, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Puppet should prune stale entries from sudoers.d - https://phabricator.wikimedia.org/T309268 (10jbond) i also slightly update the script io used for apt to hanlde if the path parameter is used and also use the simpler pql syntax ` lang=pyt... [10:41:10] 10Puppet, 10Infrastructure-Foundations, 10SRE, 10User-jbond: Frequent puppet failures - https://phabricator.wikimedia.org/T221529 (10jbond) >>! In T221529#5143984, @jbond wrote: > The error happened as puppet-merge was rolling out changes. I have not looked at how puppet-merge works but this looks like i... [11:02:31] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Agree how to handle port-block speeds for QFX5120-48Y - https://phabricator.wikimedia.org/T303529 (10cmooney) @ayounsi I think based on the above we should proceed with https://gerrit.wikimedia.org/r/c/operations/software/homer/deploy/+/769... [14:38:45] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q3:(Need By: TBD) rack/setup/install 2 new labstore hosts - https://phabricator.wikimedia.org/T302981 (10nskaggs) a:05nskaggs→03Andrew [16:04:15] 10Puppet, 10Infrastructure-Foundations, 10SRE, 10User-jbond: puppet documentation generation is missing some compnets - https://phabricator.wikimedia.org/T271909 (10jbond) [16:49:58] FYI I'm about to reimage sretest1001 [16:50:09] any objection? [16:50:21] noone logged in or process running [16:51:06] going once... [16:52:25] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q3:(Need By: TBD) rack/setup/install 2 new labstore hosts - https://phabricator.wikimedia.org/T302981 (10Andrew) [16:52:46] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q3:(Need By: TBD) rack/setup/install 2 new labstore hosts - https://phabricator.wikimedia.org/T302981 (10Andrew) Note that I'm renaming these two hosts to clouddumps100[12] [16:52:52] ok, sold :D [17:01:56] 10SRE-tools, 10Discovery, 10Infrastructure-Foundations, 10Discovery-Search (Current work), 10IPv6: Some elastic hosts do not have IPv6 DNS records - https://phabricator.wikimedia.org/T271143 (10Volans) Eqiad is also done, pasting only the differences with the above snippet: `lang=python >>> devices = De... [17:22:13] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 3 others: Q3:(Need By: TBD) rack/setup/install 2 new labstore hosts - https://phabricator.wikimedia.org/T302981 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host clouddumps1001.wikimedia.org with... [17:24:40] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: re-label cloudstore101[01] to clouddumps100[12] - https://phabricator.wikimedia.org/T309338 (10Andrew) [17:34:59] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 3 others: Q3:(Need By: TBD) rack/setup/install 2 new labstore hosts - https://phabricator.wikimedia.org/T302981 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host clouddumps1002.wikimedia.org with... [17:49:57] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 3 others: Q3:(Need By: TBD) rack/setup/install 2 new labstore hosts - https://phabricator.wikimedia.org/T302981 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host clouddumps1001.wikimedia.org with OS... [17:58:47] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 3 others: Q3:(Need By: TBD) rack/setup/install 2 new labstore hosts - https://phabricator.wikimedia.org/T302981 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host clouddumps1001.wikimedia.org with... [18:04:47] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 3 others: Q3:(Need By: TBD) rack/setup/install 2 new labstore hosts - https://phabricator.wikimedia.org/T302981 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host clouddumps1002.wikimedia.org with OS... [18:05:03] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 3 others: Q3:(Need By: TBD) rack/setup/install 2 new labstore hosts - https://phabricator.wikimedia.org/T302981 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin1001 for host clouddumps1002.wikimedia.org with... [18:26:00] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 3 others: Q3:(Need By: TBD) rack/setup/install 2 new labstore hosts - https://phabricator.wikimedia.org/T302981 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host clouddumps1001.wikimedia.org with OS... [18:32:31] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 3 others: Q3:(Need By: TBD) rack/setup/install 2 new labstore hosts - https://phabricator.wikimedia.org/T302981 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin1001 for host clouddumps1002.wikimedia.org with OS... [19:06:27] 10Mail, 10Infrastructure-Foundations, 10SRE: [mitigated] Google returning 503 error when delivering to mx1001 and mx2001 - https://phabricator.wikimedia.org/T307873 (10jhathaway) I sent a message to the Exim mailing list, https://www.mail-archive.com/exim-users@exim.org/msg57216.html. Jeremy Harris suggeste... [19:06:38] 10Mail, 10Infrastructure-Foundations, 10SRE: [mitigated] Google returning 503 error when delivering to mx1001 and mx2001 - https://phabricator.wikimedia.org/T307873 (10jhathaway) a:03jhathaway [19:16:41] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Replace labstore100[67] with clouddumps100[12] - https://phabricator.wikimedia.org/T309346 (10Andrew) [19:18:51] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Replace labstore100[67] with clouddumps100[12] - https://phabricator.wikimedia.org/T309346 (10Andrew) a:05Andrew→03ArielGlenn @ArielGlenn these two new servers should be ready; I'm hoping that you have the time to move the data and... [19:19:10] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 3 others: Q3:(Need By: TBD) rack/setup/install 2 new labstore hosts - https://phabricator.wikimedia.org/T302981 (10Andrew) [19:19:17] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 3 others: Q3:(Need By: TBD) rack/setup/install 2 new labstore hosts - https://phabricator.wikimedia.org/T302981 (10Andrew) 05Open→03Resolved [20:12:32] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 3 others: Q3:(Need By: TBD) rack/setup/install 2 new labstore hosts - https://phabricator.wikimedia.org/T302981 (10Kelson) [20:16:24] 10Mail, 10Infrastructure-Foundations, 10SRE, 10Wikimedia-Incident: 2022-05-09 Exim BDAT Errors incident - https://phabricator.wikimedia.org/T309238 (10jhathaway) [20:16:32] 10Mail, 10Infrastructure-Foundations, 10Sustainability (Incident Followup): Determine why Exim BDAT error messages are not counted as bounces by mtail - https://phabricator.wikimedia.org/T309237 (10jhathaway) 05Open→03Resolved @herron I did a little log analysis and from what I can tell the errors from t... [20:18:46] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 3 others: Q3:(Need By: TBD) rack/setup/install 2 new labstore hosts - https://phabricator.wikimedia.org/T302981 (10Kelson) @Andrew Thank you for finally completing this task!