[09:44:53] 10Mail, 10Infrastructure-Foundations, 10Znuny, 10vrts: Not receiving VRT notification emails - https://phabricator.wikimedia.org/T302139 (10Jonathan5566) We think this is a bigger issue, seem like no user can get agent's reply from VRT (checked at info-zh@wikimedia.org) [09:45:33] 10Mail, 10Infrastructure-Foundations, 10Znuny, 10vrts: Not receiving VRT notification emails - https://phabricator.wikimedia.org/T302139 (10Stang) Did a [[https://ticket.wikimedia.org/otrs/index.pl?Action=AgentTicketZoom&TicketNumber=2022022010002183|quick test]] and seems sender could not receive auto-rep... [09:47:26] 10Mail, 10Infrastructure-Foundations, 10SRE, 10Znuny, 10vrts: Not receiving VRT notification emails - https://phabricator.wikimedia.org/T302139 (10Stang) [09:51:05] 10Mail, 10Infrastructure-Foundations, 10SRE, 10Znuny, 10vrts: Not receiving VRT notification emails - https://phabricator.wikimedia.org/T302139 (10Stang) Also as someone else [[https://vrt-wiki.wikimedia.org/w/index.php?oldid=114912#Email_don't_received|reported]]. [15:07:41] 10Mail, 10Infrastructure-Foundations, 10SRE, 10Znuny, 10vrts: Not receiving VRT notification emails - https://phabricator.wikimedia.org/T302139 (10jbond) I noticed some db errors in the logs, after manually checking the db config was correct, I have restarted otrs-daemon.service and things look healthy... [15:09:38] 10Mail, 10Infrastructure-Foundations, 10SRE, 10Znuny, 10vrts: Not receiving VRT notification emails - https://phabricator.wikimedia.org/T302139 (10GeneralNotability) Confirmed, just got a wave of emails. [15:10:47] 10Mail, 10Infrastructure-Foundations, 10SRE, 10Znuny, 10vrts: Not receiving VRT notification emails - https://phabricator.wikimedia.org/T302139 (10Urbanecm) @jbond Thanks for the quick fix. I confirm emails started flowing back to my mailbox again :). [15:12:45] 10Mail, 10Infrastructure-Foundations, 10SRE, 10Znuny, 10vrts: Not receiving VRT notification emails - https://phabricator.wikimedia.org/T302139 (10Daniuu) I can also confirm that a new waive of emails just started. Thanks for looking into this, @jbond . [15:14:26] 10Mail, 10Infrastructure-Foundations, 10SRE, 10Znuny, 10vrts: Not receiving VRT notification emails - https://phabricator.wikimedia.org/T302139 (10jbond) 05Open→03Resolved a:03jbond awesome will close but please re-open if you see other issues [15:34:09] 10Mail, 10Infrastructure-Foundations, 10SRE, 10Znuny, 10vrts: Not receiving VRT notification emails - https://phabricator.wikimedia.org/T302139 (10akosiaris) >>! In T302139#7724207, @jbond wrote: > I do see the following error in the log but this looks like it can wait for someone more knowledgable about... [15:44:12] 10Mail, 10Infrastructure-Foundations, 10SRE, 10Znuny, 10vrts: Not receiving VRT notification emails - https://phabricator.wikimedia.org/T302139 (10Dzahn) Thank you @jbond ! We did talk about the log entries on Fridays but as you say the DB config looked correct. [15:48:40] 10Mail, 10Infrastructure-Foundations, 10SRE, 10Znuny, 10vrts: Not receiving VRT notification emails - https://phabricator.wikimedia.org/T302139 (10Dzahn) There was a short outage of OTRS on Feb 16th. The config change I deployed was not the original cause but to fix that. Will share more details next week.