[03:11:04] <jinxer-wm>	 FIRING: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting
[03:11:05] <jinxer-wm>	 FIRING: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts
[04:11:04] <jinxer-wm>	 RESOLVED: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting
[04:11:05] <jinxer-wm>	 RESOLVED: NetboxPhysicalHosts: Netbox - Report parity errors between PuppetDB and Netbox for physical devices. - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxPhysicalHosts
[07:29:41] <wikibugs>	 10Mail, 06Infrastructure-Foundations, 10MediaWiki-Email, 07Wikimedia-production-error: failure in Wikipedia's user to user email function - Sendmail exited with non-zero exit code 74 - https://phabricator.wikimedia.org/T401566#11072910 (10Aklapper)
[09:59:44] <wikibugs>	 10Mail, 06Infrastructure-Foundations, 10MediaWiki-Email, 10MediaWiki-extensions-EmailAuth, and 4 others: Could not send confirmation email: Unknown error in PHP's mail() function. - https://phabricator.wikimedia.org/T383047#11073192 (10Tgr) So far: exit code 74 (EX_IOERR): 72 exit code 65 (EX_DATAERR): 7 e...
[12:24:21] <wikibugs>	 10SRE-tools, 10Spicerack: DeprecationWarning: datetime.datetime.utcnow() is deprecated - https://phabricator.wikimedia.org/T401581 (10taavi) 03NEW
[12:50:27] <wikibugs>	 10CAS-SSO, 10WMF-General-or-Unknown, 07Epic, 07Security: All Wikimedia developer services should use single sign-on - https://phabricator.wikimedia.org/T189531#11073545 (10Arendpieter) Do I understand correctly that currently only `bitu` (https://idm.wikimedia.org/) is using `CAS-SSO` (https://idp.wikimedi...
[13:37:15] <wikibugs>	 10CAS-SSO, 10WMF-General-or-Unknown, 07Epic, 07Security: All Wikimedia developer services should use single sign-on - https://phabricator.wikimedia.org/T189531#11073700 (10Aklapper) yes
[13:43:37] <wikibugs>	 10SRE-tools, 06Infrastructure-Foundations, 10Spicerack: Increase the default batch size of puppet.run() - https://phabricator.wikimedia.org/T397687#11073717 (10JMeybohm) >>! In T397687#11000793, @Volans wrote: > @JMeybohm do you have a specific use case that cannot/is hard to solve simply changing the `batch...
[14:24:29] <wikibugs>	 10netops, 10fundraising-tech-ops, 06Infrastructure-Foundations: Move pfw1b-codfw to rack F5 - https://phabricator.wikimedia.org/T401297#11073957 (10cmooney) p:05Triage→03Medium
[14:24:34] <wikibugs>	 10Mail, 06Infrastructure-Foundations, 10MediaWiki-Email, 07Wikimedia-production-error: failure in Wikipedia's user to user email function - Sendmail exited with non-zero exit code 74 - https://phabricator.wikimedia.org/T401566#11073962 (10CDanis) →14Duplicate dup:03T383047
[14:24:43] <wikibugs>	 10Mail, 06Infrastructure-Foundations, 10MediaWiki-Email, 10MediaWiki-extensions-EmailAuth, and 4 others: Could not send confirmation email: Unknown error in PHP's mail() function. - https://phabricator.wikimedia.org/T383047#11073964 (10CDanis)
[14:25:12] <wikibugs>	 10CAS-SSO, 06Infrastructure-Foundations: CAS not letting new Toolsbeta-logging developer account log in - https://phabricator.wikimedia.org/T397651#11073982 (10SLyngshede-WMF) 05Open→03Declined
[14:26:59] <wikibugs>	 10SRE-tools, 06Infrastructure-Foundations, 10SRE Observability (FY2025/2026-Q1): More frequent Puppet runs on the alert hosts? - https://phabricator.wikimedia.org/T398444#11073996 (10jhathaway) p:05Triage→03Medium
[14:30:57] <wikibugs>	 10SRE-tools, 10Observability-Alerting, 10SRE Observability (FY2025/2026-Q1): Cookbook sre.hosts.remove_downtime does not remove silences - https://phabricator.wikimedia.org/T395032#11074018 (10CDanis)
[14:32:45] <wikibugs>	 10Mail, 06Infrastructure-Foundations, 10MediaWiki-Email, 07Wikimedia-production-error: failure in Wikipedia's user to user email function - Sendmail exited with non-zero exit code 74 - https://phabricator.wikimedia.org/T401566#11074028 (10jhathaway) p:05Triage→03Medium a:03jhathaway
[18:44:55] <jinxer-wm>	 FIRING: MaxConntrack: Max conntrack at 80.31% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[18:49:55] <jinxer-wm>	 RESOLVED: MaxConntrack: Max conntrack at 80.31% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[21:21:16] <wikibugs>	 10Mail, 06Infrastructure-Foundations, 10MediaWiki-Email, 10MediaWiki-extensions-EmailAuth, and 4 others: Could not send confirmation email: Unknown error in PHP's mail() function. - https://phabricator.wikimedia.org/T383047#11075860 (10jhathaway) We get the correct exit codes now, which is fabulous. Howeve...