[07:59:03] 10netops, 10Data-Engineering, 10Infrastructure-Foundations, 10SRE, and 2 others: Collect netflow data for internal traffic - https://phabricator.wikimedia.org/T263277 (10JAllemandou) Adding some Data-Engineering considerations, with the assumption that the new `sflow` stream would be comparable to the exis... [08:46:47] 10puppet-compiler, 10Infrastructure-Foundations, 10Cloud-Services-Origin-Team, 10Cloud-Services-Worktype-Project, and 2 others: [pcc] Allow passing the fail-fast option on the jenkins job + cli - https://phabricator.wikimedia.org/T296984 (10dcaro) [10:16:24] 10Puppet, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations: Split mariadb::dbstore_multiinstance into 2 separate roles (backup sources and analytics) - https://phabricator.wikimedia.org/T296285 (10jcrespo) 05Resolved→03Open ` from: SYSTEMDTIMER to: root@cumin2001.codfw.wmnet Output of systemd t... [10:35:47] 10Puppet, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Unused puppet resources audit, 2021 - https://phabricator.wikimedia.org/T272559 (10Ladsgroup) [10:58:09] 10Puppet, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Unused puppet resources audit, 2021 - https://phabricator.wikimedia.org/T272559 (10jcrespo) [10:59:16] 10Puppet, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Unused puppet resources audit, 2021 - https://phabricator.wikimedia.org/T272559 (10jcrespo) [11:01:39] 10Puppet, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Unused puppet resources audit, 2021 - https://phabricator.wikimedia.org/T272559 (10jcrespo) [15:13:01] 10Puppet, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Unused puppet resources audit, 2021 - https://phabricator.wikimedia.org/T272559 (10dcaro) [15:20:20] 10puppet-compiler, 10Infrastructure-Foundations, 10Cloud-Services-Origin-Team, 10Cloud-Services-Worktype-Project, and 3 others: [pcc] Allow passing the fail-fast option on the jenkins job + cli - https://phabricator.wikimedia.org/T296984 (10dcaro) 05Open→03Stalled [18:38:36] 10Puppet, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Unused puppet resources audit, 2021 - https://phabricator.wikimedia.org/T272559 (10Dzahn) icinga::nsca::client is used in fundraising. so there are special cases that can be in use but this audit script doesn't see [18:40:13] 10Puppet, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Unused puppet resources audit, 2021 - https://phabricator.wikimedia.org/T272559 (10Dzahn) something like role::gerrit::migration is not in use but still has value, it is there so you can apply it during the next migration when it's time to... [18:51:28] 10Puppet, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Unused puppet resources audit, 2021 - https://phabricator.wikimedia.org/T272559 (10Dzahn) [18:56:17] 10Puppet, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Unused puppet resources audit, 2021 - https://phabricator.wikimedia.org/T272559 (10Dzahn) re: diamond classes T210993 T212231 [20:43:32] 10Mail, 10Infrastructure-Foundations, 10SRE: MX record issue on mx1001.wikimedia.org - https://phabricator.wikimedia.org/T297017 (10Peachey88) [20:55:16] 10Mail, 10Infrastructure-Foundations, 10SRE: MX record issue on mx1001.wikimedia.org - https://phabricator.wikimedia.org/T297017 (10Majavah) [21:11:14] 10Mail, 10Infrastructure-Foundations, 10SRE: MX record issue on mx1001.wikimedia.org - https://phabricator.wikimedia.org/T297017 (10herron) Hi @bcampbell do you have any examples of the bounce messages including the full raw headers? With that we could trace the messages through the mail logs. A private pa... [21:25:20] 10Mail, 10Infrastructure-Foundations, 10SRE: MX record issue on mx1001.wikimedia.org - https://phabricator.wikimedia.org/T297017 (10bcampbell) Hey @herron thanks. I think I uploaded the eml file privately and added you as a subscriber, but let me know if you don't see it. [21:51:37] 10Mail, 10Infrastructure-Foundations, 10SRE: MX record issue on mx1001.wikimedia.org - https://phabricator.wikimedia.org/T297017 (10herron) Thanks, looking into this I see in the private message: ` Subject: Warning: message 1msZEc-006308-GG delayed 24 hours This message was created automatically by mail de... [22:37:34] 10netops, 10Analytics-Clusters, 10Data-Engineering, 10Infrastructure-Foundations, 10SRE: Automate ingestion of netflow event stream - https://phabricator.wikimedia.org/T248865 (10odimitrijevic) [23:50:20] 10Mail, 10Infrastructure-Foundations, 10SRE: MX record issue on mx1001.wikimedia.org - https://phabricator.wikimedia.org/T297017 (10eliza) Hi Herron What may be the next steps as ITS is receiving more tickets related to this issue. Thanks Eliza [23:57:48] 10Mail, 10Infrastructure-Foundations, 10SRE: MX record issue on mx1001.wikimedia.org - https://phabricator.wikimedia.org/T297017 (10Dzahn) 05Open→03In progress p:05Triage→03High [23:59:22] 10Mail, 10Infrastructure-Foundations, 10SRE: MX record issue on mx1001.wikimedia.org - https://phabricator.wikimedia.org/T297017 (10faidon) @eliza we're looking into this - next update in 15mins.