[00:00:22] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN [00:02:22] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is WARNING: WARNING - NGINX Error Rate is 50% [00:04:14] RECOVERY - cp36 Disk Space on cp36 is OK: DISK OK - free space: / 19422MiB (21% inode=98%); [00:04:22] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is CRITICAL: CRITICAL - NGINX Error Rate is 67% [00:05:00] Dmehus: that is obsolete now I made it into a browser extension instead [00:05:14] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 19692MiB (22% inode=98%); [00:05:39] RECOVERY - cp41 Disk Space on cp41 is OK: DISK OK - free space: / 22342MiB (23% inode=98%); [00:05:50] CosmicAlpha: what's obsolete? [00:06:09] dmehus: backendinformation.js [00:06:12] https://github.com/miraheze/MediaWikiDebugJS [00:06:22] CosmicAlpha: oh, let me look at that [00:06:38] what it do [00:07:14] Shows backend response time, queued jobs, and what MW and CP server you connect to. [00:07:16] CosmicAlpha: but is that any different than me just adding it to my global.js file? [00:08:14] and, rather, how would I install it? I thought you meant you added it to the Chrome Web Store [00:08:43] Yeah it has better performance, caching for jobs, and supports all major farms also, and has better skin selectors. It supports Fandom, Wikimedia, ShoutWiki, editthis.info, Miraheze, WikiTide, and WikiForge, mostly to allow comparing response times. [00:09:01] You go to chrome://extensions and "Load Unpacked Extension" [00:09:23] I may publish to chrome web store event [00:09:27] *eventually [00:09:29] CosmicAlpha: ah, thanks :) [00:10:18] oh it supports telepedia and wiki.gg also, two I forgot [00:10:33] oh, cool [00:10:37] I'll try it on my home PC [00:10:38] https://github.com/miraheze/MediaWikiDebugJS/blob/e655931a1847a60893e32702ff1511acce3b0790/content_script.js#L3-L23 [00:11:01] I have it working on mobile also using Kiwi browser btw [00:12:03] [02miraheze/MediaWikiDebugJS] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MediaWikiDebugJS/compare/e655931a1847...0e059119fa7c [00:12:06] [02miraheze/MediaWikiDebugJS] 07Universal-Omega 030e05911 - Remove unused matomo.miraheze.org selector [00:14:00] It also automatically pulls PHP version from headers so you don't have to keep updating that or maintain a per-farm list. It covers almost every combination of possibilities from every major farm [00:25:35] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [00:26:13] This value can't be stored [00:26:13] @configure default-link-wiki miraheze [00:26:21] Value miraheze was stored into default-link-wiki in the config [00:26:21] @configure default-link-wiki=miraheze [00:26:26] [[Testing]] [00:26:26] https://meta.miraheze.org/wiki/Testing [00:27:54] dmehus: keep in mind because of how linkie works the mh: prefix will always cause it to link to the Marshallese wikipedia [00:28:10] Should probably add a new admin since dox left [00:29:10] Successfully added .*@WikiTide/Agent-Isai [00:29:10] @trustadd .*@WikiTide/Agent-Isai admin [00:31:47] ty [00:51:02] MacFan4000: ah, yes, that's right, thanks [00:51:46] and thanks for setting the default-link-wiki [00:52:55] > I have it working on mobile also using Kiwi browser btw [00:52:55] CosmicAlpha: cool. though not sure I've used the Kiwi browser [02:23:49] PROBLEM - mw172 HTTPS on mw172 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [02:24:52] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.05, 17.98, 10.39 [02:26:52] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 11.56, 16.83, 11.00 [02:27:49] RECOVERY - mw172 HTTPS on mw172 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 285 bytes in 0.063 second response time [03:00:59] RECOVERY - mon181 Backups Grafana on mon181 is OK: FILE_AGE OK: /var/log/grafana-backup.log is 41 seconds old and 92 bytes [04:27:11] PROBLEM - mw181 Puppet on mw181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[libboost-regex1.74.0-icu72] [04:27:54] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [04:55:11] RECOVERY - mw181 Puppet on mw181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:41:46] PROBLEM - wiki.k2cho.me - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.k2cho.me' expires in 15 day(s) (Wed 10 Apr 2024 05:33:52 AM GMT +0000). [05:55:44] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [07:01:10] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=clovercraftwiki --search-recursively /home/reception/clovercraft --summary=Imported from https://clovercraft.fandom.com (START) [07:01:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:37:41] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=clovercraftwiki --search-recursively /home/reception/clovercraft --summary=Imported from https://clovercraft.fandom.com (END - exit=0) [07:37:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:14:06] RECOVERY - wiki.yumeka.icu - reverse DNS on sslhost is OK: SSL OK - wiki.yumeka.icu reverse DNS resolves to cp36.wikitide.net - CNAME OK [11:26:13] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/ssl/compare/db0b361a8daa...3202a27af0d7 [11:26:15] [02miraheze/ssl] 07Reception123 033202a27 - Create spiceandwolf.wiki [11:26:52] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/3202a27af0d7...26794d5863de [11:26:53] [02miraheze/ssl] 07Reception123 0326794d5 - add spiceandwolf.wiki cert [11:27:11] miraheze/ssl - Reception123 the build has errored. [11:27:47] miraheze/ssl - Reception123 the build has errored. [11:31:59] PROBLEM - puppet181 Puppet on puppet181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_srv-ssl] [11:33:39] PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [11:38:23] PROBLEM - cp27 Puppet on cp27 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [11:38:26] PROBLEM - mw151 Puppet on mw151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [11:38:41] PROBLEM - cp41 Puppet on cp41 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [11:40:27] PROBLEM - mw161 Puppet on mw161 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [11:40:40] PROBLEM - mw162 Puppet on mw162 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [11:47:34] PROBLEM - mw171 Puppet on mw171 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [11:49:50] PROBLEM - mw172 Puppet on mw172 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [11:52:07] PROBLEM - cp36 Puppet on cp36 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [11:55:13] PROBLEM - cp26 Puppet on cp26 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [11:55:25] PROBLEM - mw152 Puppet on mw152 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [11:56:55] PROBLEM - mw181 Puppet on mw181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [12:00:41] PROBLEM - mw182 Puppet on mw182 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [12:01:20] RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [12:01:35] PROBLEM - cp37 Puppet on cp37 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [12:04:24] RECOVERY - mw151 Puppet on mw151 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [12:05:55] RECOVERY - cp27 Puppet on cp27 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [12:06:39] RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:08:21] RECOVERY - mw162 Puppet on mw162 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [12:08:29] RECOVERY - mw161 Puppet on mw161 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [12:13:42] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/190fcc59d0ca...e2b2a3da0e85 [12:13:44] [02miraheze/CreateWiki] 07translatewiki 03e2b2a3d - Localisation updates from https://translatewiki.net. [12:13:47] [02miraheze/ImportDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/ImportDump/compare/ce3669ec1d9d...a99fe914c60c [12:13:49] [02miraheze/ImportDump] 07translatewiki 03a99fe91 - Localisation updates from https://translatewiki.net. [12:13:50] [02miraheze/RequestSSL] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/RequestSSL/compare/5de0765e6a12...46e1a1b849e2 [12:13:51] [02miraheze/RequestSSL] 07translatewiki 0346e1a1b - Localisation updates from https://translatewiki.net. [12:13:54] [02miraheze/DataDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/DataDump/compare/7275fcb9ae55...d6c9729057dd [12:13:55] [02miraheze/DataDump] 07translatewiki 03d6c9729 - Localisation updates from https://translatewiki.net. [12:13:56] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±6] 13https://github.com/miraheze/MirahezeMagic/compare/1b40e38d13d0...02488311ecb0 [12:13:57] [02miraheze/MirahezeMagic] 07translatewiki 030248831 - Localisation updates from https://translatewiki.net. [12:15:25] RECOVERY - mw171 Puppet on mw171 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [12:16:50] RECOVERY - mw172 Puppet on mw172 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [12:17:17] miraheze/MirahezeMagic - translatewiki the build has errored. [12:17:57] miraheze/DataDump - translatewiki the build passed. [12:21:07] RECOVERY - cp36 Puppet on cp36 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:21:48] miraheze/ImportDump - translatewiki the build passed. [12:22:00] miraheze/RequestSSL - translatewiki the build passed. [12:22:28] miraheze/CreateWiki - translatewiki the build passed. [12:23:12] RECOVERY - cp26 Puppet on cp26 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:23:58] !log [@test151] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 [12:23:59] !log [@test151] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [12:24:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:08] !log [@test151] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 [12:24:09] !log [@test151] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [12:24:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:40] PROBLEM - wiki.aetherexplorers.science - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.aetherexplorers.science' expires in 15 day(s) (Wed 10 Apr 2024 12:07:55 PM GMT +0000). [12:25:17] RECOVERY - mw152 Puppet on mw152 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:26:08] RECOVERY - mw181 Puppet on mw181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:28:09] RECOVERY - mw182 Puppet on mw182 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [12:29:19] RECOVERY - cp37 Puppet on cp37 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [12:30:09] RECOVERY - puppet181 Puppet on puppet181 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [12:32:43] !log [@mwtask181] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all [12:32:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:32:53] !log [@mwtask181] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 9s [12:33:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:55:31] !log [alex@test151] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/sql.php --wiki=metawikibeta (END - exit=2) [12:55:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:56:08] ^ was testing checking the existence of wiki requests in the database, no writes made [14:12:49] !log [alex@test151] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/sql.php --wiki=metawikibeta (END - exit=2) [14:12:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:20:52] !log [alex@test151] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/sql.php --wiki=metawikibeta (END - exit=2) [15:20:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:27:24] !log [alex@test151] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/sql.php --wiki=metawikibeta (END - exit=2) [15:27:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:51:56] [02CreateWiki] 07redbluegreenhat opened pull request 03#489: Experimental API for retrieving information related to a wiki request - 13https://github.com/miraheze/CreateWiki/pull/489 [15:55:06] miraheze/CreateWiki - redbluegreenhat the build has errored. [15:55:33] [02CreateWiki] 07redbluegreenhat synchronize pull request 03#489: Experimental API for retrieving information related to a wiki request - 13https://github.com/miraheze/CreateWiki/pull/489 [15:58:01] [02CreateWiki] 07redbluegreenhat synchronize pull request 03#489: Experimental API for retrieving information related to a wiki request - 13https://github.com/miraheze/CreateWiki/pull/489 [15:59:20] miraheze/CreateWiki - redbluegreenhat the build has errored. [16:01:13] miraheze/CreateWiki - redbluegreenhat the build has errored. [16:02:05] [02CreateWiki] 07redbluegreenhat synchronize pull request 03#489: Experimental API for retrieving information related to a wiki request - 13https://github.com/miraheze/CreateWiki/pull/489 [16:05:58] miraheze/CreateWiki - redbluegreenhat the build has errored. [16:07:29] [02CreateWiki] 07redbluegreenhat synchronize pull request 03#489: Experimental API for retrieving information related to a wiki request - 13https://github.com/miraheze/CreateWiki/pull/489 [16:15:25] miraheze/CreateWiki - redbluegreenhat the build passed. [16:19:06] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/527d4b3db5c4...13f7e0b1ef1b [16:19:09] [02miraheze/puppet] 07Universal-Omega 0313f7e0b - changeprop-jobqueue: readd removed comments [16:35:34] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03changeprop-realm [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/9ae66640adb5 [16:35:36] [02miraheze/puppet] 07Universal-Omega 039ae6664 - changeprop: restore realm [16:35:39] [02puppet] 07Universal-Omega created branch 03changeprop-realm - 13https://github.com/miraheze/puppet [16:35:42] [02puppet] 07Universal-Omega opened pull request 03#3790: changeprop: restore realm - 13https://github.com/miraheze/puppet/pull/3790 [16:36:43] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03changeprop-realm [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9ae66640adb5...99c37cf17025 [16:36:45] [02miraheze/puppet] 07Universal-Omega 0399c37cf - Restore [16:36:47] [02puppet] 07Universal-Omega synchronize pull request 03#3790: changeprop: restore realm - 13https://github.com/miraheze/puppet/pull/3790 [16:37:18] [02miraheze/puppet] 07github-actions[bot] pushed 031 commit to 03changeprop-realm [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/99c37cf17025...cf6cd592cb92 [16:37:21] [02miraheze/puppet] 07github-actions 03cf6cd59 - CI: lint puppet code to standards [16:37:21] [02puppet] 07github-actions[bot] synchronize pull request 03#3790: changeprop: restore realm - 13https://github.com/miraheze/puppet/pull/3790 [16:45:50] [02CreateWiki] 07redbluegreenhat edited pull request 03#489: T11931: Experimental API for retrieving information related to a wiki request - 13https://github.com/miraheze/CreateWiki/pull/489 [16:45:57] [02CreateWiki] 07redbluegreenhat edited pull request 03#489: T11931: Experimental API for retrieving information related to a wiki request - 13https://github.com/miraheze/CreateWiki/pull/489 [17:08:46] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/13f7e0b1ef1b...c1c82f21d1ed [17:08:47] [02miraheze/puppet] 07Universal-Omega 03c1c82f2 - eventgate: add topic prefix [17:09:02] [02puppet] 07Universal-Omega closed pull request 03#3790: changeprop: restore realm - 13https://github.com/miraheze/puppet/pull/3790 [17:09:03] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/puppet/compare/c1c82f21d1ed...ddaffa65e5ef [17:09:04] [02miraheze/puppet] 07Universal-Omega 03ddaffa6 - changeprop: restore realm (#3790) [17:09:05] [02miraheze/puppet] 07Universal-Omega deleted branch 03changeprop-realm [17:09:06] [02puppet] 07Universal-Omega deleted branch 03changeprop-realm - 13https://github.com/miraheze/puppet [17:16:15] PROBLEM - cloud17 Puppet on cloud17 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [17:24:53] [02miraheze/puppet] 07Universal-Omega created branch 03eventgate-factory 13https://github.com/miraheze/puppet/commit/ddaffa65e5ef112ff91af20515a432aa508f55f2 [17:24:55] [02puppet] 07Universal-Omega created branch 03eventgate-factory - 13https://github.com/miraheze/puppet [17:25:25] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03eventgate-factory [+1/-0/±2] 13https://github.com/miraheze/puppet/compare/ddaffa65e5ef...e36b3d2291a1 [17:25:28] [02miraheze/puppet] 07Universal-Omega 03e36b3d2 - eventgate: add custom factory module [17:25:37] [02puppet] 07Universal-Omega opened pull request 03#3791: eventgate: add custom factory module - 13https://github.com/miraheze/puppet/pull/3791 [17:32:16] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 9694MiB (10% inode=98%); [17:44:15] RECOVERY - cloud17 Puppet on cloud17 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [17:44:29] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03eventgate-factory [+1/-0/±2] 13https://github.com/miraheze/puppet/compare/e36b3d2291a1...3d643f8fb181 [17:44:30] [02miraheze/puppet] 07Universal-Omega 033d643f8 - Add stream-configs [17:44:31] [02puppet] 07Universal-Omega synchronize pull request 03#3791: eventgate: add custom factory module - 13https://github.com/miraheze/puppet/pull/3791 [17:48:55] PROBLEM - eventgate181 eventgate on eventgate181 is CRITICAL: connect to address 10.0.18.147 and port 8192: Connection refused [17:49:44] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03eventgate-factory [+0/-0/±3] 13https://github.com/miraheze/puppet/compare/3d643f8fb181...7e83929742c2 [17:49:46] [02miraheze/puppet] 07Universal-Omega 037e83929 - Change paths [17:49:47] [02puppet] 07Universal-Omega synchronize pull request 03#3791: eventgate: add custom factory module - 13https://github.com/miraheze/puppet/pull/3791 [17:52:04] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03eventgate-factory [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7e83929742c2...93ca36fb0b66 [17:52:06] [02miraheze/puppet] 07Universal-Omega 0393ca36f - Fix [17:52:07] [02puppet] 07Universal-Omega synchronize pull request 03#3791: eventgate: add custom factory module - 13https://github.com/miraheze/puppet/pull/3791 [17:58:53] [02CreateWiki] 07redbluegreenhat synchronize pull request 03#489: T11931: Experimental API for retrieving information related to a wiki request - 13https://github.com/miraheze/CreateWiki/pull/489 [18:00:49] PROBLEM - cp41 Disk Space on cp41 is WARNING: DISK WARNING - free space: / 10236MiB (10% inode=98%); [18:04:55] RECOVERY - eventgate181 eventgate on eventgate181 is OK: TCP OK - 0.000 second response time on 10.0.18.147 port 8192 [18:05:43] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03eventgate-factory [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/93ca36fb0b66...b29c0d77a0a2 [18:05:46] [02miraheze/puppet] 07Universal-Omega 03b29c0d7 - Fix [18:05:48] [02puppet] 07Universal-Omega synchronize pull request 03#3791: eventgate: add custom factory module - 13https://github.com/miraheze/puppet/pull/3791 [18:06:51] miraheze/CreateWiki - redbluegreenhat the build passed. [18:10:45] [02puppet] 07Universal-Omega closed pull request 03#3791: eventgate: add custom factory module - 13https://github.com/miraheze/puppet/pull/3791 [18:10:46] [02miraheze/puppet] 07Universal-Omega deleted branch 03eventgate-factory [18:10:48] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+2/-0/±2] 13https://github.com/miraheze/puppet/compare/ddaffa65e5ef...4209e60a2d1d [18:10:52] [02miraheze/puppet] 07Universal-Omega 034209e60 - eventgate: add custom factory module (#3791) [18:10:52] [02puppet] 07Universal-Omega deleted branch 03eventgate-factory - 13https://github.com/miraheze/puppet [18:11:48] PROBLEM - cp36 Disk Space on cp36 is WARNING: DISK WARNING - free space: / 9697MiB (10% inode=98%); [18:13:04] [02mw-config] 07redbluegreenhat closed pull request 03#5512: Setup wgPrivilegedGroups - 13https://github.com/miraheze/mw-config/pull/5512 [18:13:05] [02miraheze/mw-config] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/9195d6ea6217...7e1c15bf2f6a [18:13:07] [02miraheze/mw-config] 07redbluegreenhat 037e1c15b - Setup wgPrivilegedGroups (#5512) [18:14:09] miraheze/mw-config - redbluegreenhat the build passed. [18:23:22] !log [@test151] starting deploy of {'config': True} to test151 [18:23:23] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [18:23:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:23:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:27:48] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/4209e60a2d1d...aacb4728e4e2 [18:27:49] [02miraheze/puppet] 07Universal-Omega 03aacb472 - changeprop: restore low_traffic_jobs [18:32:40] !log [@mwtask181] starting deploy of {'config': True} to all [18:32:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:32:50] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 10s [18:32:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:37:15] !log [@mwtask171] starting deploy of {'config': True} to all [18:37:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:37:23] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 8s [18:37:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:43:06] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/26794d5863de...037c844bc9f1 [18:43:07] [02miraheze/ssl] 07Reception123 03037c844 - alphabetical [18:43:58] miraheze/ssl - Reception123 the build has errored. [18:49:50] PROBLEM - ns1 GDNSD Datacenters on ns1 is UNKNOWN: NRPE: Unable to read output [18:51:41] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03jobrunner-lb [+0/-0/±1] 13https://github.com/miraheze/dns/commit/87737a51c44e [18:51:43] [02miraheze/dns] 07Universal-Omega 0387737a5 - Add jobrunner load balancer [18:51:44] [02dns] 07Universal-Omega created branch 03jobrunner-lb - 13https://github.com/miraheze/dns [18:51:46] [02dns] 07Universal-Omega opened pull request 03#502: Add jobrunner load balancer - 13https://github.com/miraheze/dns/pull/502 [18:51:50] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 159.255.146.183/cpweb, 2001:470:25:715::2/cpweb, 63.141.240.6/cpweb2 [18:55:12] PROBLEM - cp26 Puppet on cp26 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [18:56:04] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03jobrunner-lb [+0/-0/±1] 13https://github.com/miraheze/dns/compare/87737a51c44e...9fc926b041c2 [18:56:05] [02miraheze/dns] 07Universal-Omega 039fc926b - Use LB [18:56:06] [02dns] 07Universal-Omega synchronize pull request 03#502: Add jobrunner load balancer - 13https://github.com/miraheze/dns/pull/502 [18:56:06] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_dns] [18:56:18] PROBLEM - mw152 Puppet on mw152 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [18:56:49] [02dns] 07Universal-Omega closed pull request 03#502: Add jobrunner load balancer - 13https://github.com/miraheze/dns/pull/502 [18:56:50] [02miraheze/dns] 07Universal-Omega deleted branch 03jobrunner-lb [18:56:51] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/dns/compare/b0459545c7c8...5a3973607962 [18:56:52] [02miraheze/dns] 07Universal-Omega 035a39736 - Add jobrunner load balancer (#502) [18:56:55] [02dns] 07Universal-Omega deleted branch 03jobrunner-lb - 13https://github.com/miraheze/dns [18:58:08] PROBLEM - mw181 Puppet on mw181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [18:59:53] PROBLEM - mw182 Puppet on mw182 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [19:00:06] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:03:11] PROBLEM - cp37 Puppet on cp37 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [19:03:14] PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [19:05:02] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03mwtask171-cpjobqueue [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/0507f02427af [19:05:03] [02miraheze/puppet] 07Universal-Omega 030507f02 - mwtask171: enable use of cpjobqueue [19:05:05] [02puppet] 07Universal-Omega created branch 03mwtask171-cpjobqueue - 13https://github.com/miraheze/puppet [19:05:06] [02puppet] 07Universal-Omega opened pull request 03#3792: mwtask171: enable use of cpjobqueue - 13https://github.com/miraheze/puppet/pull/3792 [19:07:54] PROBLEM - mw151 Puppet on mw151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [19:08:40] PROBLEM - cp41 Puppet on cp41 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [19:09:39] PROBLEM - cp27 Puppet on cp27 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [19:10:53] [02puppet] 07Universal-Omega closed pull request 03#3792: mwtask171: enable use of cpjobqueue - 13https://github.com/miraheze/puppet/pull/3792 [19:10:54] [02miraheze/puppet] 07Universal-Omega deleted branch 03mwtask171-cpjobqueue [19:10:56] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/aacb4728e4e2...b46dae932174 [19:10:58] PROBLEM - mw162 Puppet on mw162 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [19:10:59] [02miraheze/puppet] 07Universal-Omega 03b46dae9 - mwtask171: enable use of cpjobqueue (#3792) [19:11:01] [02puppet] 07Universal-Omega deleted branch 03mwtask171-cpjobqueue - 13https://github.com/miraheze/puppet [19:11:19] PROBLEM - mw161 Puppet on mw161 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [19:13:56] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/5a3973607962...aa5a335a3b20 [19:13:59] [02miraheze/dns] 07Universal-Omega 03aa5a335 - Use mwtask171 for jobrunner [19:15:27] PROBLEM - mwtask171 JobRunner Service on mwtask171 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [19:17:19] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/7e1c15bf2f6a...cdbaa55c08c1 [19:17:22] [02miraheze/mw-config] 07Universal-Omega 03cdbaa55 - Use EventBus for JobQueue on testwiki [19:18:16] PROBLEM - mw171 Puppet on mw171 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [19:18:22] miraheze/mw-config - Universal-Omega the build passed. [19:18:23] !log [universalomega@mwtask171] starting deploy of {'pull': 'config', 'config': True} to all [19:18:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:18:31] !log [universalomega@mwtask171] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 8s [19:18:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:19:28] PROBLEM - mw172 Puppet on mw172 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [19:22:43] PROBLEM - cp36 Puppet on cp36 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [19:23:16] RECOVERY - cp26 Puppet on cp26 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:23:27] !log [@test151] starting deploy of {'config': True} to test151 [19:23:28] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [19:23:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:23:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:24:07] RECOVERY - mw152 Puppet on mw152 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:24:09] RECOVERY - mw181 Puppet on mw181 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [19:27:38] RECOVERY - mw182 Puppet on mw182 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [19:29:25] RECOVERY - cp37 Puppet on cp37 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [19:30:41] RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [19:32:36] !log [@mwtask181] starting deploy of {'config': True} to all [19:32:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:32:45] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 9s [19:32:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:35:29] RECOVERY - mw151 Puppet on mw151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:36:15] RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [19:36:35] RECOVERY - cp27 Puppet on cp27 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [19:37:45] RECOVERY - mw162 Puppet on mw162 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [19:39:37] RECOVERY - mw161 Puppet on mw161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:45:30] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [19:46:05] RECOVERY - mw171 Puppet on mw171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:47:31] RECOVERY - mw172 Puppet on mw172 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:50:43] RECOVERY - cp36 Puppet on cp36 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:59:26] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03kafka [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/68f396c3effa [19:59:27] [02miraheze/puppet] 07Universal-Omega 0368f396c - kafka: add prometheus exporter [19:59:28] [02puppet] 07Universal-Omega created branch 03kafka - 13https://github.com/miraheze/puppet [19:59:30] [02puppet] 07Universal-Omega opened pull request 03#3793: kafka: add prometheus exporter - 13https://github.com/miraheze/puppet/pull/3793 [19:59:50] [02puppet] 07Universal-Omega synchronize pull request 03#3793: kafka: add prometheus exporter - 13https://github.com/miraheze/puppet/pull/3793 [19:59:53] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03kafka [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/68f396c3effa...38e5fc9cfc8e [19:59:54] [02miraheze/puppet] 07Universal-Omega 0338e5fc9 - - [20:01:35] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03kafka [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/38e5fc9cfc8e...a440910f7ae4 [20:01:38] [02miraheze/puppet] 07Universal-Omega 03a440910 - Add [20:01:39] [02puppet] 07Universal-Omega synchronize pull request 03#3793: kafka: add prometheus exporter - 13https://github.com/miraheze/puppet/pull/3793 [20:25:30] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [20:34:12] PROBLEM - mwtask171 Current Load on mwtask171 is CRITICAL: LOAD CRITICAL - total load average: 98.46, 42.78, 16.24 [20:41:52] [02miraheze/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b46dae932174...2ed2897ec2b7 [20:41:55] [02miraheze/puppet] 07AgentIsai 032ed2897 - jobrunner: up the stats and limits [20:42:12] RECOVERY - mwtask171 Current Load on mwtask171 is OK: LOAD OK - total load average: 0.29, 18.81, 18.18 [20:43:12] [02CreateWiki] 07redbluegreenhat closed pull request 03#489: T11931: Experimental API for retrieving information related to a wiki request - 13https://github.com/miraheze/CreateWiki/pull/489 [20:43:13] [02miraheze/CreateWiki] 07redbluegreenhat pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/e2b2a3da0e85...675caebbf6ba [20:43:15] [02miraheze/CreateWiki] 07redbluegreenhat 03675caeb - T11931: Experimental API for retrieving information related to a wiki request (#489) [20:43:45] !log [alex@test151] starting deploy of {'versions': '1.41', 'upgrade_extensions': 'CreateWiki'} to test151 [20:43:45] !log [alex@test151] finished deploy of {'versions': '1.41', 'upgrade_extensions': 'CreateWiki'} to test151 - SUCCESS in 1s [20:43:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:43:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:49:14] miraheze/CreateWiki - redbluegreenhat the build has errored. [21:18:21] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03kafka [+7/-0/±2] 13https://github.com/miraheze/puppet/compare/a440910f7ae4...85a3b651df22 [21:18:22] [02miraheze/puppet] 07Universal-Omega 0385a3b65 - Add more monitoring [21:18:25] [02puppet] 07Universal-Omega synchronize pull request 03#3793: kafka: add prometheus exporter - 13https://github.com/miraheze/puppet/pull/3793 [21:19:27] [02miraheze/puppet] 07Universal-Omega pushed 034 commits to 03kafka [+8/-0/±4] 13https://github.com/miraheze/puppet/compare/85a3b651df22...30c806addf84 [21:19:28] [02miraheze/puppet] 07Universal-Omega 03b4ee13f - kafka: add prometheus exporter [21:19:30] [02miraheze/puppet] 07Universal-Omega 03309b1ca - - [21:19:31] [02miraheze/puppet] 07Universal-Omega 037506faf - Add [21:19:34] [02miraheze/puppet] 07Universal-Omega 0330c806a - Add more monitoring [21:19:36] [02puppet] 07Universal-Omega synchronize pull request 03#3793: kafka: add prometheus exporter - 13https://github.com/miraheze/puppet/pull/3793 [21:20:06] [02miraheze/puppet] 07github-actions[bot] pushed 031 commit to 03kafka [+0/-0/±7] 13https://github.com/miraheze/puppet/compare/30c806addf84...85c6c8ffd847 [21:20:09] [02miraheze/puppet] 07github-actions 0385c6c8f - CI: lint puppet code to standards [21:20:12] [02puppet] 07github-actions[bot] synchronize pull request 03#3793: kafka: add prometheus exporter - 13https://github.com/miraheze/puppet/pull/3793 [21:21:01] [02CreateWiki] 07redbluegreenhat opened pull request 03#490: Sort `use` statements - 13https://github.com/miraheze/CreateWiki/pull/490 [21:22:30] [02puppet] 07Universal-Omega synchronize pull request 03#3793: kafka: add prometheus exporter - 13https://github.com/miraheze/puppet/pull/3793 [21:22:33] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03kafka [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/85c6c8ffd847...5c10a1217709 [21:22:34] [02miraheze/puppet] 07Universal-Omega 035c10a12 - Fix [21:23:18] PROBLEM - kafka181 Puppet on kafka181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 20 seconds ago with 1 failures. Failed resources (up to 3 shown): Package[prometheus-jmx-exporter] [21:24:16] RECOVERY - kafka181 ferm_active on kafka181 is OK: OK ferm input default policy is set [21:26:27] [02puppet] 07Universal-Omega synchronize pull request 03#3793: kafka: add prometheus exporter - 13https://github.com/miraheze/puppet/pull/3793 [21:26:28] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03kafka [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5c10a1217709...2297019017b9 [21:26:29] [02miraheze/puppet] 07Universal-Omega 032297019 - Fix [21:27:02] [02miraheze/puppet] 07github-actions[bot] pushed 031 commit to 03kafka [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/2297019017b9...04473b180526 [21:27:04] [02miraheze/puppet] 07github-actions 0304473b1 - CI: lint puppet code to standards [21:27:07] [02puppet] 07github-actions[bot] synchronize pull request 03#3793: kafka: add prometheus exporter - 13https://github.com/miraheze/puppet/pull/3793 [21:27:30] [02puppet] 07Universal-Omega synchronize pull request 03#3793: kafka: add prometheus exporter - 13https://github.com/miraheze/puppet/pull/3793 [21:27:32] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03kafka [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/04473b180526...8be4663872de [21:27:34] [02miraheze/puppet] 07Universal-Omega 038be4663 - Fix [21:28:28] [02miraheze/puppet] 07github-actions[bot] pushed 031 commit to 03kafka [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/8be4663872de...5c6d12083f1b [21:28:29] [02miraheze/puppet] 07github-actions 035c6d120 - CI: lint puppet code to standards [21:28:30] [02puppet] 07github-actions[bot] synchronize pull request 03#3793: kafka: add prometheus exporter - 13https://github.com/miraheze/puppet/pull/3793 [21:29:02] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [21:29:13] miraheze/CreateWiki - redbluegreenhat the build passed. [21:29:36] [02puppet] 07Universal-Omega synchronize pull request 03#3793: kafka: add prometheus exporter - 13https://github.com/miraheze/puppet/pull/3793 [21:29:37] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03kafka [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5c6d12083f1b...ce9dc474a22f [21:29:40] [02miraheze/puppet] 07Universal-Omega 03ce9dc47 - Fix [21:30:09] [02miraheze/puppet] 07github-actions[bot] pushed 031 commit to 03kafka [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ce9dc474a22f...6f49fcb1579c [21:30:12] [02miraheze/puppet] 07github-actions 036f49fcb - CI: lint puppet code to standards [21:30:15] [02puppet] 07github-actions[bot] synchronize pull request 03#3793: kafka: add prometheus exporter - 13https://github.com/miraheze/puppet/pull/3793 [21:34:55] [02puppet] 07Universal-Omega synchronize pull request 03#3793: kafka: add prometheus exporter - 13https://github.com/miraheze/puppet/pull/3793 [21:34:56] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03kafka [+1/-1/±0] 13https://github.com/miraheze/puppet/compare/6f49fcb1579c...1f6962351661 [21:34:58] [02miraheze/puppet] 07Universal-Omega 031f69623 - Fix [21:36:16] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03kafka [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/1f6962351661...9823d95512a1 [21:36:17] [02miraheze/puppet] 07Universal-Omega 039823d95 - Fix [21:36:18] [02puppet] 07Universal-Omega synchronize pull request 03#3793: kafka: add prometheus exporter - 13https://github.com/miraheze/puppet/pull/3793 [21:36:47] [02CreateWiki] 07redbluegreenhat closed pull request 03#490: Sort `use` statements - 13https://github.com/miraheze/CreateWiki/pull/490 [21:36:49] [02miraheze/CreateWiki] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/675caebbf6ba...4f47a7ed3c0c [21:36:52] [02miraheze/CreateWiki] 07redbluegreenhat 034f47a7e - Sort `use` statements (#490) [21:37:02] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [21:40:45] !log [alex@test151] starting deploy of {'versions': '1.41', 'upgrade_extensions': 'CreateWiki'} to test151 [21:40:46] !log [alex@test151] finished deploy of {'versions': '1.41', 'upgrade_extensions': 'CreateWiki'} to test151 - SUCCESS in 1s [21:40:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:41:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:41:23] [02puppet] 07Universal-Omega synchronize pull request 03#3793: kafka: add prometheus exporter - 13https://github.com/miraheze/puppet/pull/3793 [21:41:26] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03kafka [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9823d95512a1...d1694dcc0f9b [21:41:27] [02miraheze/puppet] 07Universal-Omega 03d1694dc - Fix [21:42:44] miraheze/CreateWiki - redbluegreenhat the build has errored. [21:43:16] RECOVERY - kafka181 Puppet on kafka181 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [21:44:45] [02puppet] 07Universal-Omega synchronize pull request 03#3793: kafka: add prometheus exporter - 13https://github.com/miraheze/puppet/pull/3793 [21:44:47] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03kafka [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d1694dcc0f9b...e44b71c69977 [21:44:48] [02miraheze/puppet] 07Universal-Omega 03e44b71c - Fix [21:48:06] PROBLEM - kafka181 Kafka Broker Server on kafka181 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'java', args 'Kafka /opt/kafka-2.12-2.4.1/config/server.properties' [21:49:45] [02puppet] 07Universal-Omega synchronize pull request 03#3793: kafka: add prometheus exporter - 13https://github.com/miraheze/puppet/pull/3793 [21:49:47] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03kafka [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e44b71c69977...4e3ecda00395 [21:49:50] [02miraheze/puppet] 07Universal-Omega 034e3ecda - Fix [21:52:39] [02puppet] 07Universal-Omega closed pull request 03#3793: kafka: add prometheus exporter - 13https://github.com/miraheze/puppet/pull/3793 [21:52:41] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+8/-0/±2] 13https://github.com/miraheze/puppet/compare/2ed2897ec2b7...516d12687e33 [21:52:43] [02miraheze/puppet] 07Universal-Omega 03516d126 - kafka: add prometheus exporter (#3793) [21:52:44] [02miraheze/puppet] 07Universal-Omega deleted branch 03kafka [21:52:47] [02puppet] 07Universal-Omega deleted branch 03kafka - 13https://github.com/miraheze/puppet [22:03:01] [02miraheze/CreateWiki] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/4f47a7ed3c0c...d49e4db61cdd [22:03:04] [02miraheze/CreateWiki] 07redbluegreenhat 03d49e4db - Fix CI [22:03:22] !log [alex@test151] starting deploy of {'versions': '1.41', 'upgrade_extensions': 'CreateWiki'} to test151 [22:03:23] !log [alex@test151] finished deploy of {'versions': '1.41', 'upgrade_extensions': 'CreateWiki'} to test151 - SUCCESS in 1s [22:03:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:03:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:08:24] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/516d12687e33...d5e12609004d [22:08:27] [02miraheze/puppet] 07Universal-Omega 03d5e1260 - kafka: fix icinga monitoring [22:09:13] RECOVERY - kafka181 Kafka Broker Server on kafka181 is OK: PROCS OK: 1 process with command name 'java', args 'Kafka /opt/kafka/config/server.properties' [22:09:14] miraheze/CreateWiki - redbluegreenhat the build has errored. [22:20:32] [02CreateWiki] 07redbluegreenhat opened pull request 03#491: temp fix for CI weirdness - 13https://github.com/miraheze/CreateWiki/pull/491 [22:20:38] [02CreateWiki] 07redbluegreenhat closed pull request 03#491: temp fix for CI weirdness - 13https://github.com/miraheze/CreateWiki/pull/491 [22:20:40] [02miraheze/CreateWiki] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/CreateWiki/compare/d49e4db61cdd...0bf54f4234eb [22:20:42] [02miraheze/CreateWiki] 07redbluegreenhat 030bf54f4 - temp fix for CI weirdness (#491) [22:21:22] [02miraheze/CreateWiki] 07redbluegreenhat pushed 031 commit to 03revert-491-fix-ci [+0/-0/±2] 13https://github.com/miraheze/CreateWiki/commit/9e44ebdff48d [22:21:23] [02miraheze/CreateWiki] 07redbluegreenhat 039e44ebd - Revert "temp fix for CI weirdness (#491)" [22:21:25] [02CreateWiki] 07redbluegreenhat created branch 03revert-491-fix-ci - 13https://github.com/miraheze/CreateWiki [22:21:28] [02CreateWiki] 07redbluegreenhat opened pull request 03#492: Revert "temp fix for CI weirdness" - 13https://github.com/miraheze/CreateWiki/pull/492 [22:22:45] [02miraheze/CreateWiki] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/0bf54f4234eb...533855c76862 [22:22:46] [02miraheze/CreateWiki] 07redbluegreenhat 03533855c - Temp fix for CI weirdness [22:22:54] [02CreateWiki] 07redbluegreenhat closed pull request 03#492: Revert "temp fix for CI weirdness" - 13https://github.com/miraheze/CreateWiki/pull/492 [22:24:01] !log [alex@test151] starting deploy of {'versions': '1.41', 'upgrade_extensions': 'CreateWiki'} to test151 [22:24:03] !log [alex@test151] finished deploy of {'versions': '1.41', 'upgrade_extensions': 'CreateWiki'} to test151 - SUCCESS in 1s [22:24:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:24:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:25:38] [02CreateWiki] 07redbluegreenhat deleted branch 03revert-491-fix-ci - 13https://github.com/miraheze/CreateWiki [22:25:39] [02miraheze/CreateWiki] 07redbluegreenhat deleted branch 03revert-491-fix-ci [22:28:13] !log [alex@test151] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/sql.php --wiki=metawikibeta (END - exit=2) [22:28:15] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d5e12609004d...64c5b7507139 [22:28:17] miraheze/CreateWiki - redbluegreenhat the build passed. [22:28:18] [02miraheze/puppet] 07Universal-Omega 0364c5b75 - eventgate: use prometheus metrics [22:28:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:30:35] miraheze/CreateWiki - redbluegreenhat the build passed. [22:31:51] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/64c5b7507139...e5af4ac127e3 [22:31:53] [02miraheze/puppet] 07Universal-Omega 03e5af4ac - changeprop: use prometheus metrics [22:36:43] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e5af4ac127e3...2d55959f0d6d [22:36:45] [02miraheze/puppet] 07Universal-Omega 032d55959 - eventgate: use port 9103 for metrics [22:38:19] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03metrics [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/cf8ce09e611a [22:38:20] [02miraheze/puppet] 07Universal-Omega 03cf8ce09 - prometheus: add cpjobqueue and eventgate metrics [22:38:22] [02puppet] 07Universal-Omega created branch 03metrics - 13https://github.com/miraheze/puppet [22:38:24] [02puppet] 07Universal-Omega opened pull request 03#3794: prometheus: add cpjobqueue and eventgate metrics - 13https://github.com/miraheze/puppet/pull/3794 [22:39:59] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03metrics [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/cf8ce09e611a...fafd626661bd [22:40:02] [02miraheze/puppet] 07Universal-Omega 03fafd626 - Add firewall [22:40:04] [02puppet] 07Universal-Omega synchronize pull request 03#3794: prometheus: add cpjobqueue and eventgate metrics - 13https://github.com/miraheze/puppet/pull/3794 [22:40:41] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03metrics [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/fafd626661bd...4a46631db659 [22:40:42] [02miraheze/puppet] 07Universal-Omega 034a46631 - Add firewall [22:40:44] [02puppet] 07Universal-Omega synchronize pull request 03#3794: prometheus: add cpjobqueue and eventgate metrics - 13https://github.com/miraheze/puppet/pull/3794 [22:41:09] [02puppet] 07Universal-Omega synchronize pull request 03#3794: prometheus: add cpjobqueue and eventgate metrics - 13https://github.com/miraheze/puppet/pull/3794 [22:41:10] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03metrics [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/4a46631db659...408a02778f56 [22:41:11] [02miraheze/puppet] 07Universal-Omega 03408a027 - Fix [22:41:37] [02puppet] 07Universal-Omega synchronize pull request 03#3794: prometheus: add cpjobqueue and eventgate metrics - 13https://github.com/miraheze/puppet/pull/3794 [22:41:40] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03metrics [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/408a02778f56...65ad9d3b8793 [22:41:43] [02miraheze/puppet] 07Universal-Omega 0365ad9d3 - Fix [22:41:57] [02miraheze/puppet] 07Universal-Omega pushed 032 commits to 03metrics [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/65ad9d3b8793...07cd9c3f92ef [22:41:59] [02miraheze/puppet] 07Universal-Omega 0307cd9c3 - Merge branch 'master' into metrics [22:42:00] [02puppet] 07Universal-Omega synchronize pull request 03#3794: prometheus: add cpjobqueue and eventgate metrics - 13https://github.com/miraheze/puppet/pull/3794 [22:42:12] [02miraheze/puppet] 07github-actions[bot] pushed 031 commit to 03metrics [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/07cd9c3f92ef...6b4598eaa69f [22:42:14] [02miraheze/puppet] 07github-actions 036b4598e - CI: lint puppet code to standards [22:42:15] [02puppet] 07github-actions[bot] synchronize pull request 03#3794: prometheus: add cpjobqueue and eventgate metrics - 13https://github.com/miraheze/puppet/pull/3794 [22:57:45] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03metrics [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6b4598eaa69f...9ad3a8ab29cd [22:57:48] [02miraheze/puppet] 07Universal-Omega 039ad3a8a - - [22:57:49] [02puppet] 07Universal-Omega synchronize pull request 03#3794: prometheus: add cpjobqueue and eventgate metrics - 13https://github.com/miraheze/puppet/pull/3794 [23:12:40] [02puppet] 07Universal-Omega synchronize pull request 03#3794: prometheus: add cpjobqueue and eventgate metrics - 13https://github.com/miraheze/puppet/pull/3794 [23:12:43] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03metrics [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9ad3a8ab29cd...64cfa545b191 [23:12:45] [02miraheze/puppet] 07Universal-Omega 0364cfa54 - Update [23:13:14] [02puppet] 07Universal-Omega synchronize pull request 03#3794: prometheus: add cpjobqueue and eventgate metrics - 13https://github.com/miraheze/puppet/pull/3794 [23:13:16] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03metrics [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/64cfa545b191...36dfb7c5fb52 [23:13:17] [02miraheze/puppet] 07Universal-Omega 0336dfb7c - Update [23:13:47] [02puppet] 07Universal-Omega synchronize pull request 03#3794: prometheus: add cpjobqueue and eventgate metrics - 13https://github.com/miraheze/puppet/pull/3794 [23:13:49] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03metrics [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/36dfb7c5fb52...411f4d889ddd [23:13:52] [02miraheze/puppet] 07Universal-Omega 03411f4d8 - Update [23:14:28] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/2d55959f0d6d...f50b5f6e6a7c [23:14:31] [02miraheze/puppet] 07Universal-Omega 03f50b5f6 - Update [23:15:08] [02puppet] 07Universal-Omega synchronize pull request 03#3794: prometheus: add cpjobqueue and eventgate metrics - 13https://github.com/miraheze/puppet/pull/3794 [23:15:11] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03metrics [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/411f4d889ddd...76f5a97c9f3c [23:15:13] [02miraheze/puppet] 07Universal-Omega 0376f5a97 - Update [23:15:42] [02puppet] 07Universal-Omega synchronize pull request 03#3794: prometheus: add cpjobqueue and eventgate metrics - 13https://github.com/miraheze/puppet/pull/3794 [23:15:44] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03metrics [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/76f5a97c9f3c...e207ea75022d [23:15:47] [02miraheze/puppet] 07Universal-Omega 03e207ea7 - - [23:19:04] [02puppet] 07Universal-Omega synchronize pull request 03#3794: prometheus: add cpjobqueue and eventgate metrics - 13https://github.com/miraheze/puppet/pull/3794 [23:19:06] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03metrics [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e207ea75022d...a82894604848 [23:19:08] [02miraheze/puppet] 07Universal-Omega 03a828946 - - [23:19:33] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03metrics [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a82894604848...99681f204646 [23:19:34] [02miraheze/puppet] 07Universal-Omega 0399681f2 - Update [23:19:35] [02puppet] 07Universal-Omega synchronize pull request 03#3794: prometheus: add cpjobqueue and eventgate metrics - 13https://github.com/miraheze/puppet/pull/3794 [23:19:45] [02miraheze/puppet] 07Universal-Omega pushed 032 commits to 03metrics [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/99681f204646...16b7d1070b8c [23:19:46] [02miraheze/puppet] 07Universal-Omega 0316b7d10 - Merge branch 'master' into metrics [23:19:47] [02puppet] 07Universal-Omega synchronize pull request 03#3794: prometheus: add cpjobqueue and eventgate metrics - 13https://github.com/miraheze/puppet/pull/3794 [23:22:34] [02puppet] 07Universal-Omega synchronize pull request 03#3794: prometheus: add cpjobqueue and eventgate metrics - 13https://github.com/miraheze/puppet/pull/3794 [23:22:36] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03metrics [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/16b7d1070b8c...13d14e69d5d0 [23:22:39] [02miraheze/puppet] 07Universal-Omega 0313d14e6 - update [23:29:03] [02puppet] 07Universal-Omega synchronize pull request 03#3794: prometheus: add cpjobqueue and eventgate metrics - 13https://github.com/miraheze/puppet/pull/3794 [23:29:05] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03metrics [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/13d14e69d5d0...e3ece49de506 [23:29:06] [02miraheze/puppet] 07Universal-Omega 03e3ece49 - prometheus [23:31:45] PROBLEM - changeprop151 changeprop on changeprop151 is CRITICAL: connect to address 10.0.15.148 and port 7200: Connection refused [23:35:49] [02puppet] 07Universal-Omega synchronize pull request 03#3794: prometheus: add cpjobqueue and eventgate metrics - 13https://github.com/miraheze/puppet/pull/3794 [23:35:50] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03metrics [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e3ece49de506...7d636498036e [23:35:51] [02miraheze/puppet] 07Universal-Omega 037d63649 - - [23:37:12] [02puppet] 07Universal-Omega synchronize pull request 03#3794: prometheus: add cpjobqueue and eventgate metrics - 13https://github.com/miraheze/puppet/pull/3794 [23:37:15] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03metrics [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7d636498036e...32d29c2555b6 [23:37:17] [02miraheze/puppet] 07Universal-Omega 0332d29c2 - - [23:37:31] [02puppet] 07Universal-Omega edited pull request 03#3794: prometheus: add eventgate metrics - 13https://github.com/miraheze/puppet/pull/3794 [23:37:44] [02puppet] 07Universal-Omega closed pull request 03#3794: prometheus: add eventgate metrics - 13https://github.com/miraheze/puppet/pull/3794 [23:37:46] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/puppet/compare/f50b5f6e6a7c...ddf8cb0f44f2 [23:37:49] [02miraheze/puppet] 07Universal-Omega 03ddf8cb0 - prometheus: add eventgate metrics (#3794) [23:37:52] [02puppet] 07Universal-Omega deleted branch 03metrics - 13https://github.com/miraheze/puppet [23:37:54] [02miraheze/puppet] 07Universal-Omega deleted branch 03metrics [23:37:57] [02miraheze/puppet] 07github-actions[bot] pushed 0320 commits to 03metrics [+0/-0/±20] 13https://github.com/miraheze/puppet/compare/cf8ce09e611a^...32d29c2555b6 [23:37:58] [02miraheze/puppet] 07Universal-Omega 03cf8ce09 - prometheus: add cpjobqueue and eventgate metrics [23:37:59] [02miraheze/puppet] 07Universal-Omega 03fafd626 - Add firewall [23:38:02] [02miraheze/puppet] 07Universal-Omega 034a46631 - Add firewall [23:38:03] [02miraheze/puppet] ... and 17 more commits. [23:38:04] [02puppet] 07github-actions[bot] created branch 03metrics - 13https://github.com/miraheze/puppet [23:38:21] [02miraheze/puppet] 07Universal-Omega deleted branch 03metrics [23:38:23] [02puppet] 07Universal-Omega deleted branch 03metrics - 13https://github.com/miraheze/puppet [23:39:45] RECOVERY - changeprop151 changeprop on changeprop151 is OK: TCP OK - 0.005 second response time on 10.0.15.148 port 7200 [23:45:35] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ddf8cb0f44f2...b3da2bceb00e [23:45:36] [02miraheze/puppet] 07Universal-Omega 03b3da2bc - cpjobqueue: use statsd again [23:55:42] PROBLEM - cp37 Disk Space on cp37 is CRITICAL: DISK CRITICAL - free space: / 5302MiB (5% inode=98%);