[00:45:47] (SystemdUnitFailed) firing: generate_os_reports.service Failed on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:19:32] (SystemdUnitFailed) firing: (2) update-tails-mirror.service Failed on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [01:44:02] 10Mail, 10Infrastructure-Foundations, 10Patch-For-Review, 10User-notice: Stop sending change notification email if edit is done by a bot - https://phabricator.wikimedia.org/T356984 (10Ladsgroup) It should be included ASAP. We are planning to deploy this really soon. [02:19:32] (SystemdUnitFailed) firing: (2) update-tails-mirror.service Failed on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:51:53] 10Mail, 10Infrastructure-Foundations, 10Patch-For-Review, 10User-notice: Stop sending change notification email if edit is done by a bot - https://phabricator.wikimedia.org/T356984 (10UOzurumba) @Ladsgroup thanks. [05:02:37] 10Mail, 10Infrastructure-Foundations, 10Patch-For-Review, 10User-notice: Stop sending change notification email if edit is done by a bot - https://phabricator.wikimedia.org/T356984 (10UOzurumba) @Ladsgroup Is this change happening next week? [06:20:47] (SystemdUnitFailed) firing: generate_os_reports.service Failed on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:00:35] 10SRE-tools, 10Infrastructure-Foundations, 10Puppet-Core, 10SRE, and 5 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619 (10MoritzMuehlenhoff) [10:20:47] (SystemdUnitFailed) firing: generate_os_reports.service Failed on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:46:33] 10SRE-tools, 10Infrastructure-Foundations, 10Puppet-Core, 10SRE, and 5 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619 (10Gehel) [11:37:36] 10Mail, 10Infrastructure-Foundations, 10Patch-For-Review, 10User-notice: Stop sending change notification email if edit is done by a bot - https://phabricator.wikimedia.org/T356984 (10Ladsgroup) Yup [13:53:49] 10CAS-SSO, 10Infrastructure-Foundations: OpenID Connect logout does not log out of IdP on idp.wmcloud.org - https://phabricator.wikimedia.org/T356784 (10SLyngshede-WMF) p:05Triage→03Medium a:03SLyngshede-WMF [14:20:48] (SystemdUnitFailed) firing: generate_os_reports.service Failed on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:59:38] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate hosts from codfw row A/B ASW to new LSW devices - https://phabricator.wikimedia.org/T355544 (10cmooney) [16:00:10] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A3 from asw-a3-codfw to lsw1-a3-codfw - https://phabricator.wikimedia.org/T355862 (10cmooney) 05Open→03Resolved a:03cmooney Thanks everyone for the help on getting this done! [16:12:12] 10Mail, 10Infrastructure-Foundations, 10Trust-and-Safety: Mail from Bishzilla to emergency@wikimedia.org is possibly getting lost - https://phabricator.wikimedia.org/T338032 (10RoySmith) This topic just came up in another forum and I didn't know what to say regarding status. Could I get an update? [16:29:15] 10Mail, 10Infrastructure-Foundations, 10Trust-and-Safety: Mail from Bishzilla to emergency@wikimedia.org is possibly getting lost - https://phabricator.wikimedia.org/T338032 (10jhathaway) @RoySmith did you ever here anything back from the ticket you opened? Also, you said this was mentioned in another forum... [17:16:09] 10Mail, 10Infrastructure-Foundations, 10Patch-For-Review, 10User-notice: Stop sending change notification email if edit is done by a bot - https://phabricator.wikimedia.org/T356984 (10UOzurumba) @Ladsgroup there is a comment in the Tech News: 2024-07 talk page concerning this change after I added it to the... [17:19:58] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate hosts from codfw row A/B ASW to new LSW devices - https://phabricator.wikimedia.org/T355544 (10cmooney) [17:20:48] (PuppetFailure) firing: Puppet has failed on puppetmaster1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [17:49:53] 10Mail, 10Infrastructure-Foundations, 10Trust-and-Safety: Mail from Bishzilla to emergency@wikimedia.org is possibly getting lost - https://phabricator.wikimedia.org/T338032 (10RoySmith) @jhathaway This came up during informal discussions between several Ombuds Commission members earlier today. No, I never... [18:04:18] 10Mail, 10Infrastructure-Foundations, 10Trust-and-Safety: Mail from Bishzilla to emergency@wikimedia.org is possibly getting lost - https://phabricator.wikimedia.org/T338032 (10Dzahn) >>! In T338032#9447464, @Dzahn wrote: > Maybe it's set to only respond to @wikimedia.org emails then. > > edit: submitted a... [18:05:49] (PuppetFailure) resolved: Puppet has failed on puppetmaster1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [18:07:48] 10Mail, 10Infrastructure-Foundations, 10Trust-and-Safety: Mail from Bishzilla to emergency@wikimedia.org is possibly getting lost - https://phabricator.wikimedia.org/T338032 (10Dzahn) >>! In T338032#8899762, @jhathaway wrote: > Thanks @Dzahn, I'll open a ticket with ITS to investigate. @jhathaway Did you o... [18:15:02] 10Mail, 10Infrastructure-Foundations, 10Trust-and-Safety: Mail from Bishzilla to emergency@wikimedia.org is possibly getting lost - https://phabricator.wikimedia.org/T338032 (10jhathaway) >>! In T338032#9530030, @Dzahn wrote: > @jhathaway Did you open one and have the number? I couldn't find a record of me... [18:15:07] 10Mail, 10Infrastructure-Foundations, 10Trust-and-Safety: Mail from Bishzilla to emergency@wikimedia.org is possibly getting lost - https://phabricator.wikimedia.org/T338032 (10Dzahn) I made a new ticket with ITS and I got a confirmation and number: ` Your request (#100088) has been received, and will be... [18:15:17] 10Mail, 10Infrastructure-Foundations, 10Trust-and-Safety: Mail from Bishzilla to emergency@wikimedia.org is possibly getting lost - https://phabricator.wikimedia.org/T338032 (10Dzahn) hehe, I will tell them to merge mine into yours or something :) [18:23:20] 10Mail, 10Infrastructure-Foundations, 10Trust-and-Safety: Mail from Bishzilla to emergency@wikimedia.org is possibly getting lost - https://phabricator.wikimedia.org/T338032 (10RoySmith) Is there some way I can track those zendesk tickets? [18:23:47] (SystemdUnitFailed) firing: generate_os_reports.service Failed on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:30:17] 10Mail, 10Infrastructure-Foundations, 10Trust-and-Safety: Mail from Bishzilla to emergency@wikimedia.org is possibly getting lost - https://phabricator.wikimedia.org/T338032 (10Dzahn) >>! In T338032#9530072, @RoySmith wrote: > Is there some way I can track those zendesk tickets? I am waiting for a response... [18:34:26] 10Mail, 10Infrastructure-Foundations, 10Trust-and-Safety: Mail from Bishzilla to emergency@wikimedia.org is possibly getting lost - https://phabricator.wikimedia.org/T338032 (10Novem_Linguae) One of you that opened a ticket could try replying to it and copying RoySmith. That might add them to the ticket. [18:42:05] 10Mail, 10Infrastructure-Foundations, 10Patch-For-Review, 10User-notice: Stop sending change notification email if edit is done by a bot - https://phabricator.wikimedia.org/T356984 (10Ladsgroup) I responded there, I also added a link to this ticket in tech news. [18:45:54] 10Mail, 10Infrastructure-Foundations, 10Trust-and-Safety: Mail from Bishzilla to emergency@wikimedia.org is possibly getting lost - https://phabricator.wikimedia.org/T338032 (10Dzahn) I tried this (CCed and asked to add the email address I got out of LDAP). [22:24:42] (SystemdUnitFailed) firing: generate_os_reports.service Failed on puppetdb2003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [23:34:45] 10Mail, 10Infrastructure-Foundations, 10MW-1.42-notes (1.42.0-wmf.18; 2024-02-13), 10Patch-For-Review, 10User-notice: Stop sending change notification email if edit is done by a bot - https://phabricator.wikimedia.org/T356984 (10Ladsgroup) 05Open→03Resolved a:03Ladsgroup