[00:05:55] <jinxer-wm>	 FIRING: MaxConntrack: Max conntrack at 82.54% on cloudvirt1067:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[00:50:55] <jinxer-wm>	 RESOLVED: MaxConntrack: Max conntrack at 80.72% on cloudvirt1067:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[02:57:31] <wmcs-alerts>	 FIRING: ToolsToolsDBReplicationLagIsTooHigh: ToolsDB replication on tools-db-5 is lagging behind the primary, the current lag is 1h 1m 29s - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationLagIsTooHigh
[03:26:39] <wikibugs>	 (03approved) 10anticomposite: Update README.md, footer after fok [toolforge-repos/bullseye] - 10https://gitlab.wikimedia.org/toolforge-repos/bullseye/-/merge_requests/1
[03:26:52] <wikibugs>	 (03merge) 10anticomposite: Update README.md, footer after fok [toolforge-repos/bullseye] - 10https://gitlab.wikimedia.org/toolforge-repos/bullseye/-/merge_requests/1
[04:36:00] <jinxer-wm>	 FIRING: NovafullstackSustainedFailures: Novafullstack tests have been failing for more than 5hours in eqiad - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NovafullstackSustainedFailures - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-nova-fullstack?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DNovafullstackSustainedFailures
[07:01:29] <wmcs-alerts>	 FIRING: PuppetStaleCertificates: Found non-revoked Puppet certificates for 1 deleted instances on gitlab-runners-puppetserver-01 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates
[08:36:15] <jinxer-wm>	 FIRING: NovafullstackSustainedFailures: Novafullstack tests have been failing for more than 5hours in eqiad - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NovafullstackSustainedFailures - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-nova-fullstack?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DNovafullstackSustainedFailures
[08:46:13] <wikibugs>	 10PAWS: Spawn failed: server did not respond in 30 seconds - https://phabricator.wikimedia.org/T398087#10956498 (10Wyslijp16) Same.  **Event log** > Server requested > 2025-06-29T08:44:30.761215Z [Normal] Successfully assigned prod/jupyter--57yslijp16 to paws-127a-m3mctzr7itba-node-2 > 2025-06-29T08:44:31Z [Norm...
[11:51:00] <jinxer-wm>	 RESOLVED: NovafullstackSustainedFailures: Novafullstack tests have been failing for more than 5hours in eqiad - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NovafullstackSustainedFailures - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-nova-fullstack?orgId=1 - https://alerts.wikimedia.org/?q=alertname%3DNovafullstackSustainedFailures
[13:17:55] <jinxer-wm>	 FIRING: MaxConntrack: Max conntrack at 84.17% on cloudvirt1070:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[13:28:55] <jinxer-wm>	 FIRING: MaxConntrack: Max conntrack at 92.93% on cloudvirt1070:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[13:29:07] <wikibugs>	 06cloud-services-team: MaxConntrack Max conntrack at 92.93% on cloudvirt1070:9100 - https://phabricator.wikimedia.org/T398139 (10phaultfinder) 03NEW
[13:33:55] <jinxer-wm>	 RESOLVED: MaxConntrack: Max conntrack at 92.93% on cloudvirt1070:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[13:37:55] <jinxer-wm>	 RESOLVED: MaxConntrack: Max conntrack at 81.41% on cloudvirt1070:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack
[16:54:16] <wikibugs>	 10PAWS: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T394614#10956747 (10Xqt) I don’t know who is responsible for this task but at OAWS Pywikibot is 3 releases behind the current stable one.
[16:58:27] <wikibugs>	 10PAWS: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T394614#10956750 (10RhinosF1) @Rook used to be, I created https://github.com/toolforge/paws/pull/488
[20:22:11] <wikibugs>	 10PAWS: Spawn failed: server did not respond in 30 seconds - https://phabricator.wikimedia.org/T398087#10956801 (10Wyslijp16) Solved on my side.