[00:00:59] PROBLEM - mem161 Puppet on mem161 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:01:14] PROBLEM - rdb151 Puppet on rdb151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:01:48] [02puppet] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/puppet/commit/e565c99b5b44 [00:01:49] 02puppet/03paladox-patch-1 07paladox 03e565c99 Update jvm_prometheus_jmx_exporter.yaml file… [00:01:52] [02puppet] 07paladox opened pull request #4133: Update jvm_prometheus_jmx_exporter.yaml file (03master...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4133 [00:01:58] [02puppet] 07coderabbitai[bot] commented on pull request #4133: --- […] 13https://github.com/miraheze/puppet/pull/4133#issuecomment-2558666901 [00:03:39] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/puppet/commit/418f78b6e2031b7d94ee17b84077cb03a79e7995 [00:03:40] 02puppet/03paladox-patch-1 07paladox 03418f78b Update puppetserver.pp [00:07:36] PROBLEM - mw164 Puppet on mw164 is CRITICAL: CRITICAL: Puppet has 47 failures. Last run 2 minutes ago with 47 failures. Failed resources (up to 3 shown): File[/usr/lib/php/20220829/xhprof.so],File[/opt/prometheus-statsd-exporter_0.26.1-1_amd64.deb],File[/usr/local/bin/fileLockScript.sh],File[/usr/local/bin/foreachwikiindblist] [00:10:46] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 55466MiB (12% inode=99%); [00:15:29] PROBLEM - swiftobject171 Puppet on swiftobject171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:16:57] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:17:20] PROBLEM - mw171 Puppet on mw171 is CRITICAL: CRITICAL: Puppet has 35 failures. Last run 2 minutes ago with 35 failures. Failed resources (up to 3 shown): File[/home/salt-user],File[/etc/logrotate.d/mcrouter],File[/etc/logrotate.d/mediawiki_cron_logs],File[/etc/logrotate.d/mediawiki_debug_logs] [00:18:00] PROBLEM - mw153 Puppet on mw153 is CRITICAL: CRITICAL: Puppet has 66 failures. Last run 2 minutes ago with 66 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/puppetlabs.gpg],File[/usr/local/bin/puppet-enabled],File[/etc/rsyslog.d],File[/etc/rsyslog.conf] [00:18:01] PROBLEM - mw172 Puppet on mw172 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:18:09] PROBLEM - db151 Puppet on db151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:18:14] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CRITICAL: Puppet has 37 failures. Last run 2 minutes ago with 37 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/puppetlabs.gpg],File[/usr/local/bin/puppet-enabled],File[/etc/rsyslog.d],File[/etc/rsyslog.conf] [00:19:28] PROBLEM - mon181 Puppet on mon181 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:20:13] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/puppet/commit/4d29fdf625800ca9aeb2af0d04974ca936732c5b [00:20:13] 02puppet/03paladox-patch-1 07paladox 034d29fdf Update jvm_prometheus_jmx_exporter.yaml.epp [00:23:58] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:24:02] RECOVERY - mw152 Puppet on mw152 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [00:24:56] RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [00:25:06] RECOVERY - mw181 Puppet on mw181 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [00:25:08] RECOVERY - bots171 Puppet on bots171 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [00:28:59] RECOVERY - mem161 Puppet on mem161 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [00:29:14] RECOVERY - rdb151 Puppet on rdb151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:34:19] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/puppet/commit/3bed0b70a59cafece929565232f49ab4c8b6a36e [00:34:19] 02puppet/03paladox-patch-1 07paladox 033bed0b7 Update jvm_prometheus_jmx_exporter.yaml.epp [00:35:34] RECOVERY - mw164 Puppet on mw164 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [00:36:29] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/puppet/commit/6dfd8815b3620ccc459f35e41b5a49283ceb7441 [00:36:29] 02puppet/03paladox-patch-1 07paladox 036dfd881 Update jvm_prometheus_jmx_exporter.yaml.epp [00:40:15] PROBLEM - mw161 Puppet on mw161 is CRITICAL: CRITICAL: Puppet has 66 failures. Last run 2 minutes ago with 66 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/puppetlabs.gpg],File[/usr/local/bin/puppet-enabled],File[/etc/rsyslog.d],File[/etc/rsyslog.conf] [00:40:21] PROBLEM - mw184 Puppet on mw184 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 2 minutes ago with 7 failures. Failed resources (up to 3 shown): File[/home/reception],File[/home/universalomega],File[/home/void],File[/home/macfan] [00:41:00] PROBLEM - phorge171 Puppet on phorge171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:41:04] PROBLEM - mw162 Puppet on mw162 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 3 minutes ago with 11 failures. Failed resources (up to 3 shown): File[/home/agent],File[/home/alex],File[/home/evalprime],File[/home/paladox] [00:41:41] PROBLEM - mwtask171 Puppet on mwtask171 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 3 minutes ago with 11 failures. Failed resources (up to 3 shown): File[/home/agent],File[/home/alex],File[/home/evalprime],File[/home/paladox] [00:43:29] RECOVERY - swiftobject171 Puppet on swiftobject171 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [00:44:57] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:45:58] RECOVERY - mw153 Puppet on mw153 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [00:46:14] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:47:20] RECOVERY - mw171 Puppet on mw171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:47:28] RECOVERY - mon181 Puppet on mon181 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [00:47:58] RECOVERY - mw172 Puppet on mw172 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:48:09] RECOVERY - db151 Puppet on db151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:50:44] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/puppet/commit/5e620898433522017e907755351394431cafc7a6 [00:50:44] 02puppet/03paladox-patch-1 07paladox 035e62089 Update jvm_prometheus_jmx_exporter.yaml.epp [00:52:08] PROBLEM - cp36 Puppet on cp36 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:56:08] RECOVERY - cp36 Puppet on cp36 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [00:58:14] RECOVERY - mw161 Puppet on mw161 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [00:58:21] RECOVERY - mw184 Puppet on mw184 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [00:59:02] RECOVERY - phorge171 Puppet on phorge171 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [00:59:15] RECOVERY - mw162 Puppet on mw162 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:59:42] RECOVERY - mwtask171 Puppet on mwtask171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:00:46] [02puppet] 07paladox merged pull request #4133: Update jvm_prometheus_jmx_exporter.yaml file (03master...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4133 [01:00:46] [02puppet] 07paladox pushed 1 new commit to 03master 13https://github.com/miraheze/puppet/commit/f85f6dd9e8866c15719edc4afd83d329fb2595fb [01:00:46] 02puppet/03master 07paladox 03f85f6dd Update jvm_prometheus_jmx_exporter.yaml file (#4133)… [01:00:48] [02puppet] 07paladox 04deleted 03paladox-patch-1 at 035e62089 13https://github.com/miraheze/puppet/commit/5e62089 [01:01:24] PROBLEM - puppet181 Puppet on puppet181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_puppet] [01:29:24] RECOVERY - puppet181 Puppet on puppet181 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [04:11:09] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [04:11:44] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.94 ms [04:30:46] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 49865MiB (10% inode=99%); [05:41:56] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 31.12 ms [05:43:59] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.11 ms [11:35:59] PROBLEM - mem151 Puppet on mem151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:36:14] PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 3 minutes ago with 11 failures. Failed resources (up to 3 shown): File[/home/agent],File[/home/alex],File[/home/evalprime],File[/home/paladox] [11:36:29] PROBLEM - mw154 Puppet on mw154 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:03:59] RECOVERY - mem151 Puppet on mem151 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [12:04:18] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:04:28] RECOVERY - mw154 Puppet on mw154 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [12:12:23] [02MirahezeMagic] 07translatewiki pushed 1 new commit to 03master 13https://github.com/miraheze/MirahezeMagic/commit/812e2c437ca6fd451fab1d34c871057dbd5b0aa5 [12:12:23] 02MirahezeMagic/03master 07translatewiki.net 03812e2c4 Localisation updates from https://translatewiki.net. [12:24:02] miraheze/MirahezeMagic - translatewiki the build passed. [14:36:28] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 60%, RTA = 31.10 ms [14:38:31] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.13 ms [14:57:02] PROBLEM - wikappedia.cc - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikappedia.cc' expires in 14 day(s) (Tue 07 Jan 2025 02:53:39 PM GMT +0000). [14:57:14] [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/5a2479fbe39ec9f43bc566dfe1e6364adcdcf6d7 [14:57:14] 02ssl/03master 07WikiTideSSLBot 035a2479f Bot: Update SSL cert for wikappedia.cc [15:26:33] RECOVERY - wikappedia.cc - LetsEncrypt on sslhost is OK: OK - Certificate 'wikappedia.cc' will expire on Sun 23 Mar 2025 01:58:38 PM GMT +0000. [19:34:34] [02puppet] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/puppet/commit/abace647536c [19:34:34] 02puppet/03paladox-patch-1 07paladox 03abace64 varnish: Allow caching mobile page [19:34:38] [02puppet] 07paladox opened pull request #4134: varnish: Allow caching mobile page (03master...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4134 [19:34:45] [02puppet] 07coderabbitai[bot] commented on pull request #4134: --- […] 13https://github.com/miraheze/puppet/pull/4134#issuecomment-2560207633 [19:40:29] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/puppet/commit/61cb71f121f9f08d7c936c26024783f309d4fd8a [19:40:29] 02puppet/03paladox-patch-1 07paladox 0361cb71f Update default.vcl [19:51:25] [02puppet] 07paladox commented on pull request #4134: I'm not sure why this was changed in https://github.com/miraheze/puppet/commit/ec1f216#diff-f9b990b998ad48c3119560125bbfb6fL418 […] 13https://github.com/miraheze/puppet/pull/4134#issuecomment-2560223938 [22:22:46] PROBLEM - cp37 Disk Space on cp37 is CRITICAL: DISK CRITICAL - free space: / 27207MiB (5% inode=99%);