[03:11:05] FIRING: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting [04:11:05] RESOLVED: NetboxAccounting: Netbox - Accounting job failed - https://wikitech.wikimedia.org/wiki/Netbox#Report_Alert - https://netbox.wikimedia.org/core/jobs/ - https://alerts.wikimedia.org/?q=alertname%3DNetboxAccounting [09:34:10] 10netops, 06Infrastructure-Foundations, 06SRE: mr1 port utilization alerts shouldn't mention hash page in their IRC logs - https://phabricator.wikimedia.org/T281055#11171385 (10ayounsi) 05Open→03Resolved Those alerts got moved to AM for the core routers and switches. They are not alerting for managem... [10:07:25] FIRING: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [10:12:25] RESOLVED: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [14:08:14] 10SRE-tools, 06Infrastructure-Foundations: secure-cookbook doesn't allow for --dry-run - https://phabricator.wikimedia.org/T404355 (10ayounsi) 03NEW [14:14:49] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Netbox: Updates for Nokia switch support - https://phabricator.wikimedia.org/T404146#11172893 (10ayounsi) Above patch worked successfully: https://netbox.wikimedia.org/extras/changelog/?request_id=7940ab40-742b-47fb-98c6-fba8e4e2989b Howev... [16:19:59] 10Packaging, 10bacula, 10Data-Persistence-Backup, 10Infrastructure Security, and 3 others: Trixie bacula-fd package incompatible with our bacula installation - https://phabricator.wikimedia.org/T404114#11173651 (10jcrespo) Configuration looks as intended: `lines=10 root@people1005:/etc/apt/sources.list.d$... [16:28:22] 10Packaging, 10bacula, 10Data-Persistence-Backup, 10Infrastructure Security, and 3 others: Trixie bacula-fd package incompatible with our bacula installation - https://phabricator.wikimedia.org/T404114#11173692 (10jcrespo) 05Open→03Resolved a:03jcrespo [16:55:56] FIRING: MaxConntrack: Max conntrack at 84.91% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [17:00:55] RESOLVED: MaxConntrack: Max conntrack at 85.45% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [18:45:42] 10Mail, 06Infrastructure-Foundations, 10MediaWiki-Email, 10MediaWiki-extensions-EmailAuth, and 3 others: Could not send confirmation email: Unknown error in PHP's mail() function. - https://phabricator.wikimedia.org/T383047#11174066 (10Reedy) [19:32:56] FIRING: MaxConntrack: Max conntrack at 80.16% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [19:37:55] RESOLVED: MaxConntrack: Max conntrack at 80.16% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [19:38:56] FIRING: MaxConntrack: Max conntrack at 81.65% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [19:43:55] RESOLVED: MaxConntrack: Max conntrack at 81.35% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [19:44:56] FIRING: MaxConntrack: Max conntrack at 82.42% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [19:49:55] RESOLVED: MaxConntrack: Max conntrack at 81.39% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [20:19:25] FIRING: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag [20:26:56] FIRING: MaxConntrack: Max conntrack at 85.22% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [20:56:56] RESOLVED: MaxConntrack: Max conntrack at 80.1% on krb1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [20:57:44] 10CAS-SSO, 06collaboration-services, 06Infrastructure-Foundations, 10GitLab (Auth & Access): gitlab account maps to two different developer accounts - https://phabricator.wikimedia.org/T384025#11174453 (10bd808) >>! In T384025#10521666, @Jelto wrote: > The linking happens by e-mail afaik. The original "ca... [22:24:25] RESOLVED: MirrorHighLag: Mirrors - /srv/mirrors/ubuntu synchronization lag - https://wikitech.wikimedia.org/wiki/Mirrors - https://grafana.wikimedia.org/d/dbd8a904-eab2-48d1-a3b9-fa1851ef3ed2/mirrors?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DMirrorHighLag