[00:00:08] PROBLEM - db142 MariaDB Connections on db142 is CRITICAL: QUERY CRITICAL: Host '2a10:6740::6:505' is not allowed to connect to this MariaDB server [00:00:32] PROBLEM - db121 MariaDB Connections on db121 is CRITICAL: QUERY CRITICAL: Access denied for user 'icinga'@'2a10:6740::6:505' (using password: YES) [00:05:37] [02puppet] 07Universal-Omega synchronize pull request 03#3089: T9478: add icinga monitoring for high MariaDB connections - 13https://github.com/miraheze/puppet/pull/3089 [00:05:38] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0351807df3fe...c3b67e6057a7 [00:05:41] [02miraheze/puppet] 07Universal-Omega 03c3b67e6 - Update commands.conf [00:19:01] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c3b67e6057a7...62b35224e26e [00:19:03] [02miraheze/puppet] 07Universal-Omega 0362b3522 - Update default-grants.sql.erb [00:19:06] [02puppet] 07Universal-Omega synchronize pull request 03#3089: T9478: add icinga monitoring for high MariaDB connections - 13https://github.com/miraheze/puppet/pull/3089 [00:57:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/9314a18560bb [00:57:04] [02miraheze/puppet] 07paladox 039314a18 - mw*: increase php-fpm childs to 32 [00:57:05] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://github.com/miraheze/puppet [00:57:07] [02puppet] 07paladox opened pull request 03#3090: mw*: increase php-fpm childs to 32 - 13https://github.com/miraheze/puppet/pull/3090 [00:57:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9314a18560bb...6198b9552068 [00:57:21] [02miraheze/puppet] 07paladox 036198b95 - Update mw122.yaml [00:57:24] [02puppet] 07paladox synchronize pull request 03#3090: mw*: increase php-fpm childs to 32 - 13https://github.com/miraheze/puppet/pull/3090 [00:57:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6198b9552068...acb0bb15ec04 [00:57:35] [02miraheze/puppet] 07paladox 03acb0bb1 - Update mw131.yaml [00:57:36] [02puppet] 07paladox synchronize pull request 03#3090: mw*: increase php-fpm childs to 32 - 13https://github.com/miraheze/puppet/pull/3090 [00:57:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/acb0bb15ec04...ec4853e50922 [00:57:46] [02miraheze/puppet] 07paladox 03ec4853e - Update mw132.yaml [00:57:47] [02puppet] 07paladox synchronize pull request 03#3090: mw*: increase php-fpm childs to 32 - 13https://github.com/miraheze/puppet/pull/3090 [00:57:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ec4853e50922...bcf772e5042c [00:57:59] [02miraheze/puppet] 07paladox 03bcf772e - Update mw141.yaml [00:58:02] [02puppet] 07paladox synchronize pull request 03#3090: mw*: increase php-fpm childs to 32 - 13https://github.com/miraheze/puppet/pull/3090 [00:58:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/bcf772e5042c...263e4a404e2b [00:58:13] [02miraheze/puppet] 07paladox 03263e4a4 - Update mw142.yaml [00:58:14] [02puppet] 07paladox synchronize pull request 03#3090: mw*: increase php-fpm childs to 32 - 13https://github.com/miraheze/puppet/pull/3090 [00:59:58] [02puppet] 07Universal-Omega synchronize pull request 03#3089: T9478: add icinga monitoring for high MariaDB connections - 13https://github.com/miraheze/puppet/pull/3089 [01:00:01] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/62b35224e26e...e39e2215474b [01:00:03] [02miraheze/puppet] 07Universal-Omega 03e39e221 - Update commands.conf [01:02:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/c89040e5b44b [01:02:25] [02miraheze/puppet] 07paladox 03c89040e - php: increase max requests by 50% [01:02:27] [02puppet] 07paladox created branch 03paladox-patch-3 - 13https://github.com/miraheze/puppet [01:02:30] [02puppet] 07paladox opened pull request 03#3091: php: increase max requests by 50% - 13https://github.com/miraheze/puppet/pull/3091 [01:05:20] See DM on discord CosmicAlpha please [01:08:27] [02puppet] 07Universal-Omega synchronize pull request 03#3089: T9478: add icinga monitoring for high MariaDB connections - 13https://github.com/miraheze/puppet/pull/3089 [01:08:28] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e39e2215474b...81a51013af7b [01:08:29] [02miraheze/puppet] 07Universal-Omega 0381a5101 - Update commands.conf [01:09:58] PROBLEM - db121 MariaDB Connections on db121 is UNKNOWN: /usr/lib/nagios/plugins/check_mysql_query: unrecognized option '--ssl'Usage: check_mysql_query -q SQL_query [-w warn] [-c crit] [-H host] [-P port] [-s socket] [-d database] [-u user] [-p password] [-f optfile] [-g group] [01:09:58] PROBLEM - db112 MariaDB Connections on db112 is UNKNOWN: /usr/lib/nagios/plugins/check_mysql_query: unrecognized option '--ssl'Usage: check_mysql_query -q SQL_query [-w warn] [-c crit] [-H host] [-P port] [-s socket] [-d database] [-u user] [-p password] [-f optfile] [-g group] [01:10:16] PROBLEM - db142 MariaDB Connections on db142 is UNKNOWN: /usr/lib/nagios/plugins/check_mysql_query: unrecognized option '--ssl'Usage: check_mysql_query -q SQL_query [-w warn] [-c crit] [-H host] [-P port] [-s socket] [-d database] [-u user] [-p password] [-f optfile] [-g group] [01:11:08] PROBLEM - db101 MariaDB Connections on db101 is UNKNOWN: /usr/lib/nagios/plugins/check_mysql_query: unrecognized option '--ssl'Usage: check_mysql_query -q SQL_query [-w warn] [-c crit] [-H host] [-P port] [-s socket] [-d database] [-u user] [-p password] [-f optfile] [-g group] [01:11:37] PROBLEM - db131 MariaDB Connections on db131 is UNKNOWN: /usr/lib/nagios/plugins/check_mysql_query: unrecognized option '--ssl'Usage: check_mysql_query -q SQL_query [-w warn] [-c crit] [-H host] [-P port] [-s socket] [-d database] [-u user] [-p password] [-f optfile] [-g group] [01:13:20] [02puppet] 07Universal-Omega synchronize pull request 03#3089: T9478: add icinga monitoring for high MariaDB connections - 13https://github.com/miraheze/puppet/pull/3089 [01:13:21] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/81a51013af7b...294322e72d93 [01:13:23] [02miraheze/puppet] 07Universal-Omega 03294322e - Update commands.conf [01:33:37] [02puppet] 07Universal-Omega synchronize pull request 03#3089: T9478: add icinga monitoring for high MariaDB connections - 13https://github.com/miraheze/puppet/pull/3089 [01:33:39] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/294322e72d93...603b2cc493ae [01:33:42] [02miraheze/puppet] 07Universal-Omega 03603b2cc - Update commands.conf [01:42:52] [02puppet] 07Universal-Omega synchronize pull request 03#3089: T9478: add icinga monitoring for high MariaDB connections - 13https://github.com/miraheze/puppet/pull/3089 [01:42:54] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/603b2cc493ae...308c9ca8c15c [01:42:56] [02miraheze/puppet] 07Universal-Omega 03308c9ca - Update commands.conf [02:05:00] PROBLEM - kiseki.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for kiseki.wiki could not be found [02:24:33] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/308c9ca8c15c...6c22b9339637 [02:24:35] [02miraheze/puppet] 07Universal-Omega 036c22b93 - Create check_mysql_connections.py [02:24:37] [02puppet] 07Universal-Omega synchronize pull request 03#3089: T9478: add icinga monitoring for high MariaDB connections - 13https://github.com/miraheze/puppet/pull/3089 [02:25:45] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6c22b9339637...c0ab9d9d6b76 [02:25:46] [02miraheze/puppet] 07Universal-Omega 03c0ab9d9 - Update init.pp [02:25:47] [02puppet] 07Universal-Omega synchronize pull request 03#3089: T9478: add icinga monitoring for high MariaDB connections - 13https://github.com/miraheze/puppet/pull/3089 [02:32:11] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.63, 3.13, 1.76 [02:34:09] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c0ab9d9d6b76...5a8f8342ccb7 [02:34:09] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 3.13, 3.33, 2.00 [02:34:10] [02miraheze/puppet] 07Universal-Omega 035a8f834 - Update commands.conf [02:34:11] [02puppet] 07Universal-Omega synchronize pull request 03#3089: T9478: add icinga monitoring for high MariaDB connections - 13https://github.com/miraheze/puppet/pull/3089 [02:35:49] PROBLEM - db112 MariaDB Connections on db112 is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_mysql_connections.py", line 13, in import mysql.connectorModuleNotFoundError: No module named 'mysql' [02:36:26] PROBLEM - db142 MariaDB Connections on db142 is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_mysql_connections.py", line 13, in import mysql.connectorModuleNotFoundError: No module named 'mysql' [02:36:29] PROBLEM - db101 MariaDB Connections on db101 is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_mysql_connections.py", line 13, in import mysql.connectorModuleNotFoundError: No module named 'mysql' [02:36:45] PROBLEM - db131 MariaDB Connections on db131 is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_mysql_connections.py", line 13, in import mysql.connectorModuleNotFoundError: No module named 'mysql' [02:37:26] PROBLEM - db121 MariaDB Connections on db121 is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_mysql_connections.py", line 13, in import mysql.connectorModuleNotFoundError: No module named 'mysql' [02:41:16] [02puppet] 07Universal-Omega synchronize pull request 03#3089: T9478: add icinga monitoring for high MariaDB connections - 13https://github.com/miraheze/puppet/pull/3089 [02:41:17] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5a8f8342ccb7...f12870cd602b [02:41:18] [02miraheze/puppet] 07Universal-Omega 03f12870c - Update init.pp [02:44:26] [02puppet] 07Universal-Omega synchronize pull request 03#3089: T9478: add icinga monitoring for high MariaDB connections - 13https://github.com/miraheze/puppet/pull/3089 [02:44:29] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+1/-1/±0] 13https://github.com/miraheze/puppet/compare/f12870cd602b...e3b15f82cd9b [02:44:31] [02miraheze/puppet] 07Universal-Omega 03e3b15f8 - Update and rename check_mysql_connections.py to check_mysql_connections.php [02:44:37] PROBLEM - mon141 Puppet on mon141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [02:45:10] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e3b15f82cd9b...aa00802e9b3c [02:45:13] [02miraheze/puppet] 07Universal-Omega 03aa00802 - Update init.pp [02:45:16] [02puppet] 07Universal-Omega synchronize pull request 03#3089: T9478: add icinga monitoring for high MariaDB connections - 13https://github.com/miraheze/puppet/pull/3089 [02:46:34] [02puppet] 07Universal-Omega synchronize pull request 03#3089: T9478: add icinga monitoring for high MariaDB connections - 13https://github.com/miraheze/puppet/pull/3089 [02:46:37] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/aa00802e9b3c...3d1a9e01c4d2 [02:46:40] [02miraheze/puppet] 07Universal-Omega 033d1a9e0 - Update commands.conf [02:47:54] RECOVERY - db112 MariaDB Connections on db112 is OK: PHP Warning: mysqli::__construct() expects at most 6 parameters, 7 given in /usr/lib/nagios/plugins/check_mysql_connections.php on line 51Warning: mysqli::__construct() expects at most 6 parameters, 7 given in /usr/lib/nagios/plugins/check_mysql_connections.php on line 51PHP Warning: mysqli::query(): Couldn't fetch mysqli in /usr/lib/nagios/plugins/check_mysql_connections.php on line 58War [02:47:54] mysqli::query(): Couldn't fetch mysqli in /usr/lib/nagios/plugins/check_mysql_connections.php on line 58PHP Warning: main(): Couldn't fetch mysqli in /usr/lib/nagios/plugins/check_mysql_connections.php on line 61Warning: main(): Couldn't fetch mysqli in /usr/lib/nagios/plugins/check_mysql_connections.php on line 61Query failed [02:48:17] RECOVERY - mon141 Puppet on mon141 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [02:48:23] RECOVERY - db131 MariaDB Connections on db131 is OK: PHP Warning: mysqli::__construct() expects at most 6 parameters, 7 given in /usr/lib/nagios/plugins/check_mysql_connections.php on line 51Warning: mysqli::__construct() expects at most 6 parameters, 7 given in /usr/lib/nagios/plugins/check_mysql_connections.php on line 51PHP Warning: mysqli::query(): Couldn't fetch mysqli in /usr/lib/nagios/plugins/check_mysql_connections.php on line 58War [02:48:23] mysqli::query(): Couldn't fetch mysqli in /usr/lib/nagios/plugins/check_mysql_connections.php on line 58PHP Warning: main(): Couldn't fetch mysqli in /usr/lib/nagios/plugins/check_mysql_connections.php on line 61Warning: main(): Couldn't fetch mysqli in /usr/lib/nagios/plugins/check_mysql_connections.php on line 61Query failed [02:48:26] RECOVERY - db101 MariaDB Connections on db101 is OK: PHP Warning: mysqli::__construct() expects at most 6 parameters, 7 given in /usr/lib/nagios/plugins/check_mysql_connections.php on line 51Warning: mysqli::__construct() expects at most 6 parameters, 7 given in /usr/lib/nagios/plugins/check_mysql_connections.php on line 51PHP Warning: mysqli::query(): Couldn't fetch mysqli in /usr/lib/nagios/plugins/check_mysql_connections.php on line 58War [02:48:27] mysqli::query(): Couldn't fetch mysqli in /usr/lib/nagios/plugins/check_mysql_connections.php on line 58PHP Warning: main(): Couldn't fetch mysqli in /usr/lib/nagios/plugins/check_mysql_connections.php on line 61Warning: main(): Couldn't fetch mysqli in /usr/lib/nagios/plugins/check_mysql_connections.php on line 61Query failed [02:48:31] RECOVERY - db142 MariaDB Connections on db142 is OK: PHP Warning: mysqli::__construct() expects at most 6 parameters, 7 given in /usr/lib/nagios/plugins/check_mysql_connections.php on line 51Warning: mysqli::__construct() expects at most 6 parameters, 7 given in /usr/lib/nagios/plugins/check_mysql_connections.php on line 51PHP Warning: mysqli::query(): Couldn't fetch mysqli in /usr/lib/nagios/plugins/check_mysql_connections.php on line 58War [02:48:31] mysqli::query(): Couldn't fetch mysqli in /usr/lib/nagios/plugins/check_mysql_connections.php on line 58PHP Warning: main(): Couldn't fetch mysqli in /usr/lib/nagios/plugins/check_mysql_connections.php on line 61Warning: main(): Couldn't fetch mysqli in /usr/lib/nagios/plugins/check_mysql_connections.php on line 61Query failed [02:49:01] RECOVERY - db121 MariaDB Connections on db121 is OK: PHP Warning: mysqli::__construct() expects at most 6 parameters, 7 given in /usr/lib/nagios/plugins/check_mysql_connections.php on line 51Warning: mysqli::__construct() expects at most 6 parameters, 7 given in /usr/lib/nagios/plugins/check_mysql_connections.php on line 51PHP Warning: mysqli::query(): Couldn't fetch mysqli in /usr/lib/nagios/plugins/check_mysql_connections.php on line 58War [02:49:02] mysqli::query(): Couldn't fetch mysqli in /usr/lib/nagios/plugins/check_mysql_connections.php on line 58PHP Warning: main(): Couldn't fetch mysqli in /usr/lib/nagios/plugins/check_mysql_connections.php on line 61Warning: main(): Couldn't fetch mysqli in /usr/lib/nagios/plugins/check_mysql_connections.php on line 61Query failed [02:54:57] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.48, 2.56, 1.77 [02:56:27] [02puppet] 07Universal-Omega synchronize pull request 03#3089: T9478: add icinga monitoring for high MariaDB connections - 13https://github.com/miraheze/puppet/pull/3089 [02:56:30] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/3d1a9e01c4d2...b9d9e322aa13 [02:56:31] [02miraheze/puppet] 07Universal-Omega 03b9d9e32 - Update check_mysql_connections.php [02:56:57] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.46, 2.04, 1.67 [03:08:13] [02puppet] 07Universal-Omega synchronize pull request 03#3089: T9478: add icinga monitoring for high MariaDB connections - 13https://github.com/miraheze/puppet/pull/3089 [03:08:14] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b9d9e322aa13...9f9552560fc5 [03:08:17] [02miraheze/puppet] 07Universal-Omega 039f95525 - Update check_mysql_connections.php [03:09:33] [02puppet] 07Universal-Omega synchronize pull request 03#3089: T9478: add icinga monitoring for high MariaDB connections - 13https://github.com/miraheze/puppet/pull/3089 [03:09:34] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9f9552560fc5...9aa1895cad71 [03:09:36] [02miraheze/puppet] 07Universal-Omega 039aa1895 - Update check_mysql_connections.php [03:14:06] [02puppet] 07Universal-Omega synchronize pull request 03#3089: T9478: add icinga monitoring for high MariaDB connections - 13https://github.com/miraheze/puppet/pull/3089 [03:14:09] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9aa1895cad71...9885bf601def [03:14:11] [02miraheze/puppet] 07Universal-Omega 039885bf6 - Update check_mysql_connections.php [03:17:12] PROBLEM - db101 MariaDB Connections on db101 is CRITICAL: PHP Warning: mysqli::__construct(): (HY000/1045): Access denied for user 'icinga'@'2a10:6740::6:505' (using password: YES) in /usr/lib/nagios/plugins/check_mysql_connections.php on line 51Warning: mysqli::__construct(): (HY000/1045): Access denied for user 'icinga'@'2a10:6740::6:505' (using password: YES) in /usr/lib/nagios/plugins/check_mysql_connections.php on line 51PHP Warning: my [03:17:12] :options(): Couldn't fetch mysqli in /usr/lib/nagios/plugins/check_mysql_connections.php on line 54Warning: mysqli::options(): Couldn't fetch mysqli in /usr/lib/nagios/plugins/check_mysql_connections.php on line 54PHP Warning: mysqli::ssl_set(): Couldn't fetch mysqli in /usr/lib/nagios/plugins/check_mysql_connections.php on line 55Warning: mysqli::ssl_set(): Couldn't fetch mysqli in /usr/lib/nagios/plugins/check_mysql_connections.php on line [03:17:12] nnection failed: Access denied for user 'icinga'@'2a10:6740::6:505' (using password: YES) [03:17:15] PROBLEM - db121 MariaDB Connections on db121 is CRITICAL: PHP Warning: mysqli::__construct(): (HY000/1045): Access denied for user 'icinga'@'2a10:6740::6:505' (using password: YES) in /usr/lib/nagios/plugins/check_mysql_connections.php on line 51Warning: mysqli::__construct(): (HY000/1045): Access denied for user 'icinga'@'2a10:6740::6:505' (using password: YES) in /usr/lib/nagios/plugins/check_mysql_connections.php on line 51PHP Warning: my [03:17:15] :options(): Couldn't fetch mysqli in /usr/lib/nagios/plugins/check_mysql_connections.php on line 54Warning: mysqli::options(): Couldn't fetch mysqli in /usr/lib/nagios/plugins/check_mysql_connections.php on line 54PHP Warning: mysqli::ssl_set(): Couldn't fetch mysqli in /usr/lib/nagios/plugins/check_mysql_connections.php on line 55Warning: mysqli::ssl_set(): Couldn't fetch mysqli in /usr/lib/nagios/plugins/check_mysql_connections.php on line [03:17:15] nnection failed: Access denied for user 'icinga'@'2a10:6740::6:505' (using password: YES) [03:18:17] PROBLEM - db142 MariaDB Connections on db142 is CRITICAL: PHP Warning: mysqli::__construct(): (HY000/1130): Host '2a10:6740::6:505' is not allowed to connect to this MariaDB server in /usr/lib/nagios/plugins/check_mysql_connections.php on line 51Warning: mysqli::__construct(): (HY000/1130): Host '2a10:6740::6:505' is not allowed to connect to this MariaDB server in /usr/lib/nagios/plugins/check_mysql_connections.php on line 51PHP Warning: my [03:18:17] :options(): Couldn't fetch mysqli in /usr/lib/nagios/plugins/check_mysql_connections.php on line 54Warning: mysqli::options(): Couldn't fetch mysqli in /usr/lib/nagios/plugins/check_mysql_connections.php on line 54PHP Warning: mysqli::ssl_set(): Couldn't fetch mysqli in /usr/lib/nagios/plugins/check_mysql_connections.php on line 55Warning: mysqli::ssl_set(): Couldn't fetch mysqli in /usr/lib/nagios/plugins/check_mysql_connections.php on line [03:18:17] nnection failed: Host '2a10:6740::6:505' is not allowed to connect to this MariaDB server [03:18:32] PROBLEM - db112 MariaDB Connections on db112 is CRITICAL: PHP Warning: mysqli::__construct(): (HY000/1045): Access denied for user 'icinga'@'2a10:6740::6:505' (using password: YES) in /usr/lib/nagios/plugins/check_mysql_connections.php on line 51Warning: mysqli::__construct(): (HY000/1045): Access denied for user 'icinga'@'2a10:6740::6:505' (using password: YES) in /usr/lib/nagios/plugins/check_mysql_connections.php on line 51PHP Warning: my [03:18:32] :options(): Couldn't fetch mysqli in /usr/lib/nagios/plugins/check_mysql_connections.php on line 54Warning: mysqli::options(): Couldn't fetch mysqli in /usr/lib/nagios/plugins/check_mysql_connections.php on line 54PHP Warning: mysqli::ssl_set(): Couldn't fetch mysqli in /usr/lib/nagios/plugins/check_mysql_connections.php on line 55Warning: mysqli::ssl_set(): Couldn't fetch mysqli in /usr/lib/nagios/plugins/check_mysql_connections.php on line [03:18:32] nnection failed: Access denied for user 'icinga'@'2a10:6740::6:505' (using password: YES) [03:18:57] PROBLEM - db131 MariaDB Connections on db131 is CRITICAL: PHP Warning: mysqli::__construct(): (HY000/1045): Access denied for user 'icinga'@'2a10:6740::6:505' (using password: YES) in /usr/lib/nagios/plugins/check_mysql_connections.php on line 51Warning: mysqli::__construct(): (HY000/1045): Access denied for user 'icinga'@'2a10:6740::6:505' (using password: YES) in /usr/lib/nagios/plugins/check_mysql_connections.php on line 51PHP Warning: my [03:18:57] :options(): Couldn't fetch mysqli in /usr/lib/nagios/plugins/check_mysql_connections.php on line 54Warning: mysqli::options(): Couldn't fetch mysqli in /usr/lib/nagios/plugins/check_mysql_connections.php on line 54PHP Warning: mysqli::ssl_set(): Couldn't fetch mysqli in /usr/lib/nagios/plugins/check_mysql_connections.php on line 55Warning: mysqli::ssl_set(): Couldn't fetch mysqli in /usr/lib/nagios/plugins/check_mysql_connections.php on line [03:18:57] nnection failed: Access denied for user 'icinga'@'2a10:6740::6:505' (using password: YES) [03:28:12] [02puppet] 07Universal-Omega synchronize pull request 03#3089: T9478: add icinga monitoring for high MariaDB connections - 13https://github.com/miraheze/puppet/pull/3089 [03:28:13] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9885bf601def...40a0b194b241 [03:28:16] [02miraheze/puppet] 07Universal-Omega 0340a0b19 - Update check_mysql_connections.php [03:29:40] PROBLEM - db112 MariaDB Connections on db112 is WARNING: PHP Warning: mysqli_real_connect(): Private key does not match certificate! in /usr/lib/nagios/plugins/check_mysql_connections.php on line 59Warning: mysqli_real_connect(): Private key does not match certificate! in /usr/lib/nagios/plugins/check_mysql_connections.php on line 59Warning connection usage: 2% [03:30:17] RECOVERY - db142 MariaDB Connections on db142 is OK: PHP Warning: mysqli_real_connect(): (HY000/1130): Host '2a10:6740::6:505' is not allowed to connect to this MariaDB server in /usr/lib/nagios/plugins/check_mysql_connections.php on line 59Warning: mysqli_real_connect(): (HY000/1130): Host '2a10:6740::6:505' is not allowed to connect to this MariaDB server in /usr/lib/nagios/plugins/check_mysql_connections.php on line 59Connection failed: H [03:30:17] 2a10:6740::6:505' is not allowed to connect to this MariaDB server [03:30:49] RECOVERY - db131 MariaDB Connections on db131 is OK: PHP Warning: mysqli_real_connect(): Private key does not match certificate! in /usr/lib/nagios/plugins/check_mysql_connections.php on line 59Warning: mysqli_real_connect(): Private key does not match certificate! in /usr/lib/nagios/plugins/check_mysql_connections.php on line 59OK connection usage: 0.4% [03:31:01] PROBLEM - db101 MariaDB Connections on db101 is WARNING: PHP Warning: mysqli_real_connect(): Private key does not match certificate! in /usr/lib/nagios/plugins/check_mysql_connections.php on line 59Warning: mysqli_real_connect(): Private key does not match certificate! in /usr/lib/nagios/plugins/check_mysql_connections.php on line 59Warning connection usage: 3% [03:31:09] PROBLEM - db121 MariaDB Connections on db121 is WARNING: PHP Warning: mysqli_real_connect(): Private key does not match certificate! in /usr/lib/nagios/plugins/check_mysql_connections.php on line 59Warning: mysqli_real_connect(): Private key does not match certificate! in /usr/lib/nagios/plugins/check_mysql_connections.php on line 59Warning connection usage: 2.8% [03:32:14] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1719798148 secs [03:41:03] [02puppet] 07Universal-Omega synchronize pull request 03#3089: T9478: add icinga monitoring for high MariaDB connections - 13https://github.com/miraheze/puppet/pull/3089 [03:41:05] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/40a0b194b241...ec898a4c0c98 [03:41:06] [02miraheze/puppet] 07Universal-Omega 03ec898a4 - Update check_mysql_connections.php [03:42:45] RECOVERY - db112 MariaDB Connections on db112 is OK: PHP Warning: mysqli_real_connect(): (HY000/1045): Access denied for user 'icinga'@'2a10:6740::6:505' (using password: YES) in /usr/lib/nagios/plugins/check_mysql_connections.php on line 42Warning: mysqli_real_connect(): (HY000/1045): Access denied for user 'icinga'@'2a10:6740::6:505' (using password: YES) in /usr/lib/nagios/plugins/check_mysql_connections.php on line 42Connection failed: A [03:42:45] denied for user 'icinga'@'2a10:6740::6:505' (using password: YES) [03:43:01] RECOVERY - db101 MariaDB Connections on db101 is OK: PHP Warning: mysqli_real_connect(): (HY000/1045): Access denied for user 'icinga'@'2a10:6740::6:505' (using password: YES) in /usr/lib/nagios/plugins/check_mysql_connections.php on line 42Warning: mysqli_real_connect(): (HY000/1045): Access denied for user 'icinga'@'2a10:6740::6:505' (using password: YES) in /usr/lib/nagios/plugins/check_mysql_connections.php on line 42Connection failed: A [03:43:01] denied for user 'icinga'@'2a10:6740::6:505' (using password: YES) [03:43:48] RECOVERY - db121 MariaDB Connections on db121 is OK: PHP Warning: mysqli_real_connect(): (HY000/1045): Access denied for user 'icinga'@'2a10:6740::6:505' (using password: YES) in /usr/lib/nagios/plugins/check_mysql_connections.php on line 42Warning: mysqli_real_connect(): (HY000/1045): Access denied for user 'icinga'@'2a10:6740::6:505' (using password: YES) in /usr/lib/nagios/plugins/check_mysql_connections.php on line 42Connection failed: A [03:43:48] denied for user 'icinga'@'2a10:6740::6:505' (using password: YES) [03:44:57] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ec898a4c0c98...989eeff573e2 [03:44:58] [02miraheze/puppet] 07Universal-Omega 03989eeff - Update check_mysql_connections.php [03:44:59] [02puppet] 07Universal-Omega synchronize pull request 03#3089: T9478: add icinga monitoring for high MariaDB connections - 13https://github.com/miraheze/puppet/pull/3089 [03:46:50] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/989eeff573e2...5e49cc6fef81 [03:46:52] [02miraheze/puppet] 07Universal-Omega 035e49cc6 - Update check_mysql_connections.php [03:46:54] [02puppet] 07Universal-Omega synchronize pull request 03#3089: T9478: add icinga monitoring for high MariaDB connections - 13https://github.com/miraheze/puppet/pull/3089 [03:50:23] [02puppet] 07Universal-Omega synchronize pull request 03#3089: T9478: add icinga monitoring for high MariaDB connections - 13https://github.com/miraheze/puppet/pull/3089 [03:50:26] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5e49cc6fef81...f458832e5906 [03:50:27] [02miraheze/puppet] 07Universal-Omega 03f458832 - Update check_mysql_connections.php [03:53:43] PROBLEM - db121 MariaDB Connections on db121 is WARNING: Warning connection usage: 5% [03:54:23] PROBLEM - db112 MariaDB Connections on db112 is WARNING: Warning connection usage: 2.2% [03:55:01] PROBLEM - db101 MariaDB Connections on db101 is WARNING: Warning connection usage: 5.8% [03:58:14] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.07347348332 secs [03:59:27] [02puppet] 07Universal-Omega synchronize pull request 03#3089: T9478: add icinga monitoring for high MariaDB connections - 13https://github.com/miraheze/puppet/pull/3089 [03:59:30] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f458832e5906...0a655d8e3b76 [03:59:31] [02miraheze/puppet] 07Universal-Omega 030a655d8 - Update commands.conf [04:02:39] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0a655d8e3b76...82322c24ec43 [04:02:41] [02miraheze/puppet] 07Universal-Omega 0382322c2 - Update check_mysql_connections.php [04:02:44] [02puppet] 07Universal-Omega synchronize pull request 03#3089: T9478: add icinga monitoring for high MariaDB connections - 13https://github.com/miraheze/puppet/pull/3089 [04:03:02] [02puppet] 07Universal-Omega synchronize pull request 03#3089: T9478: add icinga monitoring for high MariaDB connections - 13https://github.com/miraheze/puppet/pull/3089 [04:03:04] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/82322c24ec43...d789be2c34b4 [04:03:07] [02miraheze/puppet] 07Universal-Omega 03d789be2 - Update default-grants.sql.erb [04:03:51] [02puppet] 07Universal-Omega synchronize pull request 03#3089: T9478: add icinga monitoring for high MariaDB connections - 13https://github.com/miraheze/puppet/pull/3089 [04:03:52] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d789be2c34b4...145c9f7c10e9 [04:03:54] [02miraheze/puppet] 07Universal-Omega 03145c9f7 - Update config.pp [04:09:49] [02puppet] 07Universal-Omega synchronize pull request 03#3089: T9478: add icinga monitoring for high MariaDB connections - 13https://github.com/miraheze/puppet/pull/3089 [04:09:52] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/145c9f7c10e9...c5595cc6ef12 [04:09:54] [02miraheze/puppet] 07Universal-Omega 03c5595cc - Update check_mysql_connections.php [04:10:13] PROBLEM - db142 MariaDB Connections on db142 is CRITICAL: PHP Warning: mysqli_real_connect(): (HY000/1130): Host '2a10:6740::6:505' is not allowed to connect to this MariaDB server in /usr/lib/nagios/plugins/check_mysql_connections.php on line 48Warning: mysqli_real_connect(): (HY000/1130): Host '2a10:6740::6:505' is not allowed to connect to this MariaDB server in /usr/lib/nagios/plugins/check_mysql_connections.php on line 48Connection fail [04:10:13] ost '2a10:6740::6:505' is not allowed to connect to this MariaDB server [04:10:44] [02puppet] 07Universal-Omega synchronize pull request 03#3089: T9478: add icinga monitoring for high MariaDB connections - 13https://github.com/miraheze/puppet/pull/3089 [04:10:45] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c5595cc6ef12...c95f10f282a2 [04:10:48] [02miraheze/puppet] 07Universal-Omega 03c95f10f - Update config.pp [04:12:53] PROBLEM - cp33 Varnish Backends on cp33 is CRITICAL: 5 backends are down. mw121 mw122 mw131 mw132 mw141 [04:13:01] PROBLEM - cp32 Varnish Backends on cp32 is CRITICAL: 3 backends are down. mw131 mw132 mw141 [04:16:19] PROBLEM - cp32 Puppet on cp32 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/ssl/localcerts/dragdown.wiki.crt] [04:16:45] RECOVERY - cp33 Varnish Backends on cp33 is OK: All 14 backends are healthy [04:16:58] RECOVERY - db121 MariaDB Connections on db121 is OK: OK connection usage: 0.6%Current connections: 3 [04:17:03] RECOVERY - db101 MariaDB Connections on db101 is OK: OK connection usage: 4.4%Current connections: 22 [04:17:14] RECOVERY - db112 MariaDB Connections on db112 is OK: OK connection usage: 2.4%Current connections: 12 [04:17:45] RECOVERY - cp32 Varnish Backends on cp32 is OK: All 14 backends are healthy [04:18:57] [02puppet] 07Universal-Omega synchronize pull request 03#3089: T9478: add icinga monitoring for high MariaDB connections - 13https://github.com/miraheze/puppet/pull/3089 [04:18:59] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c95f10f282a2...d7e421a95cb3 [04:19:02] [02miraheze/puppet] 07Universal-Omega 03d7e421a - Update check_mysql_connections.php [04:19:57] [02puppet] 07Universal-Omega synchronize pull request 03#3089: T9478: add icinga monitoring for high MariaDB connections - 13https://github.com/miraheze/puppet/pull/3089 [04:19:59] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d7e421a95cb3...7e4eecfa842d [04:20:02] [02miraheze/puppet] 07Universal-Omega 037e4eecf - Update check_mysql_connections.php [04:40:21] RECOVERY - cp32 Puppet on cp32 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [04:43:44] [02puppet] 07Universal-Omega synchronize pull request 03#3089: T9478: add icinga monitoring for high MariaDB connections - 13https://github.com/miraheze/puppet/pull/3089 [04:43:46] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7e4eecfa842d...32f916aef3b8 [04:43:49] [02miraheze/puppet] 07Universal-Omega 0332f916a - Update check_mysql_connections.php [04:46:42] [02puppet] 07Universal-Omega synchronize pull request 03#3089: T9478: add icinga monitoring for high MariaDB connections - 13https://github.com/miraheze/puppet/pull/3089 [04:46:44] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/32f916aef3b8...e25f168d68c3 [04:46:46] [02miraheze/puppet] 07Universal-Omega 03e25f168 - Update check_mysql_connections.php [04:51:46] [02puppet] 07Universal-Omega synchronize pull request 03#3089: T9478: add icinga monitoring for high MariaDB connections - 13https://github.com/miraheze/puppet/pull/3089 [04:51:49] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e25f168d68c3...881a9ec4e204 [04:51:51] [02miraheze/puppet] 07Universal-Omega 03881a9ec - Update check_mysql_connections.php [05:34:46] [02miraheze/MirahezeDebug] 07Universal-Omega pushed 031 commit to 03manifest-version-3 [+0/-0/±1] 13https://github.com/miraheze/MirahezeDebug/compare/31b94bb724f3...c92793520492 [05:34:49] [02miraheze/MirahezeDebug] 07Universal-Omega 03c927935 - Update new [05:34:50] [02MirahezeDebug] 07Universal-Omega synchronize pull request 03#11: Convert to manifest version 3 - 13https://github.com/miraheze/MirahezeDebug/pull/11 [05:55:10] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.108920455 secs [06:13:10] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.0904507339 secs [06:17:43] [02mediawiki] 07wmfgerrit synchronize pull request 03#12312: Rel1 39 - 13https://github.com/miraheze/mediawiki/pull/12312 [06:24:49] PROBLEM - mon141 Puppet on mon141 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Package[python-is-python3],Package[python] [06:38:42] [02mediawiki] 07wmfgerrit synchronize pull request 03#12312: Rel1 39 - 13https://github.com/miraheze/mediawiki/pull/12312 [06:41:31] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.1701805592 secs [06:54:48] RECOVERY - mon141 Puppet on mon141 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:09:31] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset -0.03264382482 secs [07:13:01] [02CreateWiki] 07amire80 opened pull request 03#358: Add GENDER to log entries - 13https://github.com/miraheze/CreateWiki/pull/358 [07:14:07] [02puppet] 07Universal-Omega closed pull request 03#3089: T9478: add icinga monitoring for high MariaDB connections - 13https://github.com/miraheze/puppet/pull/3089 [07:14:10] [02miraheze/puppet] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [07:14:12] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±3] 13https://github.com/miraheze/puppet/compare/540a329efe62...de627c3a4880 [07:14:14] [02miraheze/puppet] 07Universal-Omega 03de627c3 - T9478: add icinga monitoring for high MariaDB connections (#3089) [07:14:15] [02puppet] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/puppet [07:20:37] miraheze/CreateWiki - amire80 the build passed. [07:52:23] [02ssl] 07Reception123 commented on pull request 03#622: renew revi.wiki cert - 13https://github.com/miraheze/ssl/pull/622#issuecomment-1365687303 [08:18:20] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 7.81, 5.75, 2.73 [08:22:15] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.44, 2.82, 2.20 [08:35:27] PROBLEM - kiseki.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address kiseki.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [08:48:16] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 4.89, 3.40, 2.22 [08:56:15] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 0.73, 3.78, 3.42 [08:58:15] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.57, 2.66, 3.05 [09:08:02] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.00, 3.36, 2.01 [09:10:02] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.80, 3.35, 2.17 [09:12:01] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.68, 2.78, 2.11 [10:10:48] [02mediawiki] 07wmfgerrit synchronize pull request 03#12312: Rel1 39 - 13https://github.com/miraheze/mediawiki/pull/12312 [10:27:10] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1320107579 secs [10:47:10] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.08747622371 secs [13:23:28] [02ssl] 07MacFan4000 commented on pull request 03#622: renew revi.wiki cert - 13https://github.com/miraheze/ssl/pull/622#issuecomment-1365899457 [13:51:02] [02puppet] 07paladox closed pull request 03#3091: php: increase max requests by 50% - 13https://github.com/miraheze/puppet/pull/3091 [13:51:04] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-3 [13:51:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/de627c3a4880...a5a97e99fb8f [13:51:08] [02miraheze/puppet] 07paladox 03a5a97e9 - php: increase max requests by 50% (#3091) [13:51:09] [02puppet] 07paladox deleted branch 03paladox-patch-3 - 13https://github.com/miraheze/puppet [14:59:48] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.05, 3.41, 2.52 [15:01:45] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 0.97, 2.62, 2.34 [15:14:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/2248632f19be [15:14:48] [02miraheze/puppet] 07paladox 032248632 - mail121: increase php childs to 8 [15:14:50] [02puppet] 07paladox created branch 03paladox-patch-3 - 13https://github.com/miraheze/puppet [15:14:51] [02puppet] 07paladox opened pull request 03#3092: mail121: increase php childs to 8 - 13https://github.com/miraheze/puppet/pull/3092 [15:15:00] [02puppet] 07paladox closed pull request 03#3092: mail121: increase php childs to 8 - 13https://github.com/miraheze/puppet/pull/3092 [15:15:02] [02puppet] 07paladox deleted branch 03paladox-patch-3 - 13https://github.com/miraheze/puppet [15:15:05] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-3 [15:15:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a5a97e99fb8f...727ba00b5430 [15:15:10] [02miraheze/puppet] 07paladox 03727ba00 - mail121: increase php childs to 8 (#3092) [15:18:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/abd4d8d11935 [15:18:05] [02miraheze/puppet] 07paladox 03abd4d8d - roundcubemail: switch to static from dynamic [15:18:08] [02puppet] 07paladox created branch 03paladox-patch-3 - 13https://github.com/miraheze/puppet [15:18:10] [02puppet] 07paladox opened pull request 03#3093: roundcubemail: switch to static from dynamic - 13https://github.com/miraheze/puppet/pull/3093 [15:18:18] [02puppet] 07paladox closed pull request 03#3093: roundcubemail: switch to static from dynamic - 13https://github.com/miraheze/puppet/pull/3093 [15:18:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/727ba00b5430...eb3e0f76e651 [15:18:22] [02miraheze/puppet] 07paladox 03eb3e0f7 - roundcubemail: switch to static from dynamic (#3093) [15:18:24] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-3 [15:18:26] [02puppet] 07paladox deleted branch 03paladox-patch-3 - 13https://github.com/miraheze/puppet [15:33:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/a04fa1cbbe10 [15:33:13] [02miraheze/puppet] 07paladox 03a04fa1c - matomo: use the redis role not the redis class [15:33:14] [02puppet] 07paladox created branch 03paladox-patch-3 - 13https://github.com/miraheze/puppet [15:33:17] [02puppet] 07paladox opened pull request 03#3094: matomo: use the redis role not the redis class - 13https://github.com/miraheze/puppet/pull/3094 [15:33:31] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.162145853 secs [15:38:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/eb3e0f76e651...c9799748bdf2 [15:38:45] [02miraheze/puppet] 07paladox 03c979974 - matomo: install redis prometheus exporter [15:45:59] [02ssl] 07Reception123 commented on pull request 03#622: renew revi.wiki cert - 13https://github.com/miraheze/ssl/pull/622#issuecomment-1366000823 [15:46:09] [02ssl] 07Reception123 closed pull request 03#622: renew revi.wiki cert - 13https://github.com/miraheze/ssl/pull/622 [15:46:12] [02miraheze/ssl] 07Reception123 deleted branch 03renew-revi [15:46:14] [02ssl] 07Reception123 deleted branch 03renew-revi - 13https://github.com/miraheze/ssl [15:46:15] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/6ec956645ea2...e659f68ca7c6 [15:46:18] [02miraheze/ssl] 07MacFan4000 03e659f68 - renew revi.wiki cert (#622) [15:49:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/c90f78017dfb [15:49:03] [02miraheze/puppet] 07paladox 03c90f780 - matomo: set a redis password [15:49:05] [02puppet] 07paladox created branch 03paladox-patch-8 - 13https://github.com/miraheze/puppet [15:49:07] [02puppet] 07paladox opened pull request 03#3095: matomo: set a redis password - 13https://github.com/miraheze/puppet/pull/3095 [15:49:36] [02puppet] 07paladox closed pull request 03#3095: matomo: set a redis password - 13https://github.com/miraheze/puppet/pull/3095 [15:49:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c9799748bdf2...b413c07fe9f1 [15:49:38] [02miraheze/puppet] 07paladox 03b413c07 - matomo: set a redis password (#3095) [15:49:39] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-8 [15:49:41] [02puppet] 07paladox deleted branch 03paladox-patch-8 - 13https://github.com/miraheze/puppet [15:53:31] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset -0.01495158672 secs [15:54:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b413c07fe9f1...0534f29f3e72 [15:54:49] [02miraheze/puppet] 07paladox 030534f29 - matomo131: increase php childs to 18 [15:56:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0534f29f3e72...d389b9ac74ba [15:56:26] [02miraheze/puppet] 07paladox 03d389b9a - matomo: upgrade to 4.13.0 [16:14:24] RECOVERY - revi.wiki - PositiveSSLDV on sslhost is OK: OK - Certificate 'revi.wiki' will expire on Fri 29 Dec 2023 23:59:59 GMT +0000. [16:22:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-8 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/42ee9b63f0f7 [16:22:57] [02miraheze/puppet] 07paladox 0342ee9b6 - matomo: experiment not using queued tracking [16:23:00] [02puppet] 07paladox created branch 03paladox-patch-8 - 13https://github.com/miraheze/puppet [16:23:03] [02puppet] 07paladox opened pull request 03#3096: matomo: experiment not using queued tracking - 13https://github.com/miraheze/puppet/pull/3096 [16:24:04] paladox: why not using queued tracking? [16:24:37] I was just about to abandon. I was having a look at what wikimedia do. [16:24:50] [02puppet] 07paladox closed pull request 03#3096: matomo: experiment not using queued tracking - 13https://github.com/miraheze/puppet/pull/3096 [16:24:52] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-8 [16:24:54] [02puppet] 07paladox deleted branch 03paladox-patch-8 - 13https://github.com/miraheze/puppet [16:25:04] Why do we need to mirror the WMF when that was tested and works for us here not too long ago? [16:25:31] plus not mirroring creates a 1:1 request relationship - which overwhelms matomo web services [16:25:47] *not queuing [16:27:13] yeh [16:43:16] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/f253fd9dfb6f [16:43:19] [02miraheze/mw-config] 07Universal-Omega 03f253fd9 - Switch `$wgCompressRevisions` on by default [16:43:22] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [16:43:25] [02mw-config] 07Universal-Omega opened pull request 03#5043: Switch `$wgCompressRevisions` on by default - 13https://github.com/miraheze/mw-config/pull/5043 [16:44:26] miraheze/mw-config - Universal-Omega the build passed. [16:44:43] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 14.44, 10.21, 5.55 [16:46:15] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 4.13, 2.88, 1.64 [16:46:24] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/f253fd9dfb6f...6716aeae4479 [16:46:25] [02miraheze/mw-config] 07Universal-Omega 036716aea - Update ManageWikiSettings.php [16:46:26] [02mw-config] 07Universal-Omega synchronize pull request 03#5043: Switch `$wgCompressRevisions` on by default - 13https://github.com/miraheze/mw-config/pull/5043 [16:47:29] miraheze/mw-config - Universal-Omega the build passed. [16:49:35] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/4d08c727e253 [16:49:38] [02miraheze/puppet] 07Universal-Omega 034d08c72 - mwscript: add `compressOld.php` to longscripts [16:49:40] [02puppet] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/puppet [16:49:43] [02puppet] 07Universal-Omega opened pull request 03#3097: mwscript: add `compressOld.php` to longscripts - 13https://github.com/miraheze/puppet/pull/3097 [16:50:15] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.27, 2.78, 1.95 [16:55:22] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.1478431821 secs [16:56:43] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 5.09, 7.96, 7.02 [16:58:43] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 28.62, 13.72, 9.08 [17:04:44] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 3.90, 6.98, 7.50 [17:08:43] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 4.18, 5.38, 6.70 [17:09:26] [02puppet] 07Universal-Omega synchronize pull request 03#3097: mwscript: add `compressOld.php` to longscripts - 13https://github.com/miraheze/puppet/pull/3097 [17:09:28] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/4d08c727e253...823bc386e131 [17:09:30] [02miraheze/puppet] 07Universal-Omega 03823bc38 - Update mwscript.py [17:16:04] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1029769182 secs [17:18:04] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.0765478313 secs [17:18:26] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03backups [+1/-0/±0] 13https://github.com/miraheze/puppet/commit/96da5ce94fc4 [17:18:27] [02miraheze/puppet] 07JohnFLewis 0396da5ce - Create backup.pp [17:18:29] [02puppet] 07JohnFLewis created branch 03backups - 13https://github.com/miraheze/puppet [17:19:05] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03backups [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/96da5ce94fc4...75c46d3350c8 [17:19:07] [02miraheze/puppet] 07JohnFLewis 0375c46d3 - Update init.pp [17:20:43] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 14.91, 9.41, 7.30 [17:24:48] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03backups [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/75c46d3350c8...017ea542d89d [17:24:49] [02miraheze/puppet] 07JohnFLewis 03017ea54 - Create miraheze-backup.py.erb [17:25:22] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset 0.09047231078 secs [17:25:43] [02puppet] 07JohnFLewis opened pull request 03#3098: T8350: Backups using PCA - 13https://github.com/miraheze/puppet/pull/3098 [17:30:03] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03backups [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/017ea542d89d...feacec3f697d [17:30:06] [02miraheze/puppet] 07JohnFLewis 03feacec3 - backup not backups [17:30:09] [02puppet] 07JohnFLewis synchronize pull request 03#3098: T8350: Backups using PCA - 13https://github.com/miraheze/puppet/pull/3098 [17:34:22] [02puppet] 07paladox closed pull request 03#3098: T8350: Backups using PCA - 13https://github.com/miraheze/puppet/pull/3098 [17:34:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+2/-0/±1] 13https://github.com/miraheze/puppet/compare/d389b9ac74ba...b4f35d28b00b [17:34:25] [02miraheze/puppet] 07JohnFLewis 03b4f35d2 - T8350: Backups using PCA (#3098) [17:34:26] [02miraheze/puppet] 07paladox deleted branch 03backups [17:34:29] [02puppet] 07paladox deleted branch 03backups - 13https://github.com/miraheze/puppet [17:35:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b4f35d28b00b...4131a10e93df [17:35:28] [02miraheze/puppet] 07paladox 034131a10 - Fix [17:37:13] PROBLEM - mw132 Puppet on mw132 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:39:26] PROBLEM - mw141 Puppet on mw141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:50:43] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 3.26, 3.50, 7.56 [17:52:43] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 0.71, 2.46, 6.69 [17:55:26] RECOVERY - mw141 Puppet on mw141 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [17:57:31] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.1459135711 secs [17:58:28] PROBLEM - es131 Puppet on es131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:00:15] PROBLEM - test131 Puppet on test131 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[python3-fabric] [18:03:12] [02mediawiki] 07wmfgerrit synchronize pull request 03#12312: Rel1 39 - 13https://github.com/miraheze/mediawiki/pull/12312 [18:07:13] RECOVERY - mw132 Puppet on mw132 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [18:08:32] [02puppet] 07paladox closed pull request 03#3094: matomo: use the redis role not the redis class - 13https://github.com/miraheze/puppet/pull/3094 [18:08:35] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-3 [18:08:37] [02puppet] 07paladox deleted branch 03paladox-patch-3 - 13https://github.com/miraheze/puppet [18:29:31] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset 0.08947589993 secs [18:33:31] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.1648918688 secs [18:35:22] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.2932775617 secs [18:41:31] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset 0.09063917398 secs [18:49:49] [02puppet] 07paladox closed pull request 03#3090: mw*: increase php-fpm childs to 32 - 13https://github.com/miraheze/puppet/pull/3090 [18:49:52] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [18:49:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±6] 13https://github.com/miraheze/puppet/compare/4131a10e93df...2027d2ae1a6e [18:49:54] [02miraheze/puppet] 07paladox 032027d2a - mw*: increase php-fpm childs to 32 (#3090) [18:49:56] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://github.com/miraheze/puppet [19:01:31] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset -0.1228406131 secs [19:03:31] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset -0.08858555555 secs [19:03:35] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/2027d2ae1a6e...6750ddddb104 [19:03:38] [02miraheze/puppet] 07JohnFLewis 036750ddd - T8350: Automate private and sslkey backups weekly [19:05:22] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset -0.007181495428 secs [19:05:27] [02miraheze/.github] 07dependabot[bot] pushed 031 commit to 03dependabot/github_actions/workflow-templates/actions/cache-3.2.2 [+0/-0/±1] 13https://github.com/miraheze/.github/commit/0a2747c558fb [19:05:30] [02miraheze/.github] 07dependabot[bot] 030a2747c - Bump actions/cache from 3.2.1 to 3.2.2 in /workflow-templates [19:05:33] [02.github] 07dependabot[bot] created branch 03dependabot/github_actions/workflow-templates/actions/cache-3.2.2 - 13https://github.com/miraheze/.github [19:05:36] [02.github] 07dependabot[bot] opened pull request 03#24: Bump actions/cache from 3.2.1 to 3.2.2 in /workflow-templates - 13https://github.com/miraheze/.github/pull/24 [19:05:38] [02.github] 07dependabot[bot] labeled pull request 03#24: Bump actions/cache from 3.2.1 to 3.2.2 in /workflow-templates - 13https://github.com/miraheze/.github/pull/24 [19:18:19] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/6938616bdfe4 [19:18:21] [02miraheze/mw-config] 07paladox 036938616 - Increase scribuntu cpu limit to 10 [19:18:23] [02mw-config] 07paladox created branch 03paladox-patch-3 - 13https://github.com/miraheze/mw-config [19:18:24] [02mw-config] 07paladox opened pull request 03#5044: Increase scribuntu cpu limit to 10 - 13https://github.com/miraheze/mw-config/pull/5044 [19:18:30] [02mw-config] 07paladox closed pull request 03#5044: Increase scribuntu cpu limit to 10 - 13https://github.com/miraheze/mw-config/pull/5044 [19:18:31] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/ea0d9243783d...2bc2ebc700e8 [19:18:33] [02miraheze/mw-config] 07paladox 032bc2ebc - Increase scribuntu cpu limit to 10 (#5044) [19:18:34] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-3 [19:18:37] [02mw-config] 07paladox deleted branch 03paladox-patch-3 - 13https://github.com/miraheze/mw-config [19:18:41] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [19:18:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:18:50] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 9s [19:18:56] !log [paladox@mwtask141] starting deploy of {'config': True} to all [19:18:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:19:02] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 6s [19:19:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:19:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:19:19] miraheze/mw-config - paladox the build passed. [19:19:37] miraheze/mw-config - paladox the build passed. [19:20:38] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/cbd295b3f67f [19:20:41] [02miraheze/mw-config] 07paladox 03cbd295b - Increase maxLangCacheSize to 200 for scribuntu [19:20:42] [02mw-config] 07paladox created branch 03paladox-patch-3 - 13https://github.com/miraheze/mw-config [19:20:45] [02mw-config] 07paladox opened pull request 03#5045: Increase maxLangCacheSize to 200 for scribuntu - 13https://github.com/miraheze/mw-config/pull/5045 [19:21:50] miraheze/mw-config - paladox the build passed. [19:22:15] [02mw-config] 07paladox closed pull request 03#5045: Increase maxLangCacheSize to 200 for scribuntu - 13https://github.com/miraheze/mw-config/pull/5045 [19:22:17] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-3 [19:22:20] [02mw-config] 07paladox deleted branch 03paladox-patch-3 - 13https://github.com/miraheze/mw-config [19:26:44] !log [@test131] starting deploy of {'config': True} to all [19:26:45] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [19:26:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:26:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:51:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6750ddddb104...8bc30a1007d4 [19:51:57] [02miraheze/puppet] 07paladox 038bc30a1 - icingaweb2: switch php-fpm to static [19:54:59] PROBLEM - mon141 Check correctness of the icinga configuration on mon141 is CRITICAL: Icinga configuration contains errors [20:15:24] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/8bc30a1007d4...2c94453f466c [20:15:26] [02miraheze/puppet] 07JohnFLewis 032c94453 - rm - from string [20:20:33] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/2c94453f466c...c9d0f912ed8e [20:20:35] [02miraheze/puppet] 07JohnFLewis 03c9d0f91 - T8350: Backup Phabricator Static every 2 weeks [20:22:22] RECOVERY - mon141 Check correctness of the icinga configuration on mon141 is OK: Icinga configuration is correct [20:23:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/c4bfc20acd46 [20:23:08] [02miraheze/puppet] 07paladox 03c4bfc20 - monitoring: check_icinga_config doesn't take a param [20:23:10] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://github.com/miraheze/puppet [20:23:11] [02puppet] 07paladox opened pull request 03#3099: monitoring: check_icinga_config doesn't take a param - 13https://github.com/miraheze/puppet/pull/3099 [20:23:20] [02puppet] 07paladox closed pull request 03#3099: monitoring: check_icinga_config doesn't take a param - 13https://github.com/miraheze/puppet/pull/3099 [20:23:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c9d0f912ed8e...1e593e6e2b53 [20:23:23] [02miraheze/puppet] 07paladox 031e593e6 - monitoring: check_icinga_config doesn't take a param (#3099) [20:23:26] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://github.com/miraheze/puppet [20:23:28] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [20:24:10] PROBLEM - puppet141 Backups SSLKeys on puppet141 is CRITICAL: FILE_AGE CRITICAL: File not found - /var/log/sslkeys-backup.log [20:25:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/32daa1fc25ae [20:25:15] [02miraheze/puppet] 07paladox 0332daa1f - We install python3-decorator in base/manifests/backup.pp now [20:25:16] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://github.com/miraheze/puppet [20:25:19] [02puppet] 07paladox opened pull request 03#3100: We install python3-decorator in base/manifests/backup.pp now - 13https://github.com/miraheze/puppet/pull/3100 [20:25:28] [02puppet] 07paladox edited pull request 03#3100: prometheus::exporter::elasticsearch: We install python3-decorator in base/manifests/backup.pp now - 13https://github.com/miraheze/puppet/pull/3100 [20:25:34] [02puppet] 07paladox closed pull request 03#3100: prometheus::exporter::elasticsearch: We install python3-decorator in base/manifests/backup.pp now - 13https://github.com/miraheze/puppet/pull/3100 [20:25:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/1e593e6e2b53...636900b0dd63 [20:25:38] [02miraheze/puppet] 07paladox 03636900b - prometheus::exporter::elasticsearch: We install python3-decorator in base/manifests/backup.pp now (#3100) [20:25:39] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [20:25:42] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://github.com/miraheze/puppet [20:26:42] RECOVERY - es131 Puppet on es131 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [20:48:51] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/636900b0dd63...f18b9496fba7 [20:48:53] [02miraheze/puppet] 07JohnFLewis 03f18b949 - T8350: Add SQL backups + custom granularity for non-wiki dbs [20:53:05] PROBLEM - db101 Puppet on db101 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:54:22] PROBLEM - phab121 Backups Phabricator Static on phab121 is CRITICAL: FILE_AGE CRITICAL: File not found - /var/log/phabricator-backup.log [20:58:40] [02miraheze/mediawiki] 07paladox pushed 03333 commits to 03merge_1_39_1 [+18/-0/±1446] 13https://github.com/miraheze/mediawiki/compare/555a1e47cca6^...2cc042309671 [20:58:42] [02miraheze/mediawiki] 07brightbyte 03555a1e4 - exception: Tolerate no service container when trying DB rollback [20:58:44] [02miraheze/mediawiki] 07translatewiki 03f7def1c - Localisation updates from https://translatewiki.net. [20:58:47] [02miraheze/mediawiki] 07translatewiki 03656e597 - Update git submodules [20:58:48] [02miraheze/mediawiki] ... and 330 more commits. [20:58:51] [02mediawiki] 07paladox created branch 03merge_1_39_1 - 13https://github.com/miraheze/mediawiki [20:59:05] [02miraheze/mediawiki] 07paladox deleted branch 03merge_1_39_1 [20:59:07] [02mediawiki] 07paladox deleted branch 03merge_1_39_1 - 13https://github.com/miraheze/mediawiki [21:00:16] [02miraheze/mediawiki] 07paladox deleted branch 03paladox-patch-2 [21:00:18] [02mediawiki] 07paladox deleted branch 03paladox-patch-2 - 13https://github.com/miraheze/mediawiki [21:00:21] [02miraheze/mediawiki] 07paladox deleted branch 03paladox-patch-1 [21:00:22] [02mediawiki] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/mediawiki [21:03:53] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f18b9496fba7...ade79443ea6b [21:03:54] [02miraheze/puppet] 07JohnFLewis 03ade7944 - T8350: Add customer db-backups for misc on db112 [21:03:58] PROBLEM - db131 Puppet on db131 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [21:04:29] PROBLEM - db112 Puppet on db112 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [21:06:21] [02MirahezeMagic] 07dependabot[bot] opened pull request 03#386: Bump actions/cache from 3.2.1 to 3.2.2 - 13https://github.com/miraheze/MirahezeMagic/pull/386 [21:06:23] [02miraheze/MirahezeMagic] 07dependabot[bot] pushed 031 commit to 03dependabot/github_actions/actions/cache-3.2.2 [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/commit/fe0800b77d9e [21:06:25] [02miraheze/MirahezeMagic] 07dependabot[bot] 03fe0800b - Bump actions/cache from 3.2.1 to 3.2.2 [21:06:26] [02MirahezeMagic] 07dependabot[bot] created branch 03dependabot/github_actions/actions/cache-3.2.2 - 13https://github.com/miraheze/MirahezeMagic [21:06:29] [02MirahezeMagic] 07dependabot[bot] labeled pull request 03#386: Bump actions/cache from 3.2.1 to 3.2.2 - 13https://github.com/miraheze/MirahezeMagic/pull/386 [21:06:32] [02MirahezeMagic] 07dependabot[bot] labeled pull request 03#386: Bump actions/cache from 3.2.1 to 3.2.2 - 13https://github.com/miraheze/MirahezeMagic/pull/386 [21:06:43] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ade79443ea6b...46009fd3d3ff [21:06:46] [02miraheze/puppet] 07JohnFLewis 0346009fd - Remove Phabricator dbs [21:06:55] [02.github] 07Universal-Omega closed pull request 03#24: Bump actions/cache from 3.2.1 to 3.2.2 in /workflow-templates - 13https://github.com/miraheze/.github/pull/24 [21:06:56] [02miraheze/.github] 07Universal-Omega deleted branch 03dependabot/github_actions/workflow-templates/actions/cache-3.2.2 [21:06:58] [02miraheze/.github] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/.github/compare/fe12cdb8dd85...b9e44f9aecd7 [21:07:00] [02miraheze/.github] 07dependabot[bot] 03b9e44f9 - Bump actions/cache from 3.2.1 to 3.2.2 in /workflow-templates (#24) [21:07:03] [02.github] 07Universal-Omega deleted branch 03dependabot/github_actions/workflow-templates/actions/cache-3.2.2 - 13https://github.com/miraheze/.github [21:07:18] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/IncidentReporting/compare/eb90e0d80bbd...f0784ceea395 [21:07:20] [02miraheze/IncidentReporting] 07Universal-Omega 03f0784ce - Update workflow [21:07:23] [02miraheze/GlobalNewFiles] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/GlobalNewFiles/compare/68a26d68fec1...0f5cdab016c1 [21:07:24] [02miraheze/GlobalNewFiles] 07Universal-Omega 030f5cdab - Update workflow [21:07:26] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/a8674eeb67fb...835902a4c7b9 [21:07:29] [02miraheze/CreateWiki] 07Universal-Omega 03835902a - Update workflow [21:07:30] [02miraheze/PDFEmbed] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/PDFEmbed/compare/b0fbb0b97b4d...a30226591aad [21:07:31] [02miraheze/PDFEmbed] 07Universal-Omega 03a302265 - Update workflow [21:07:32] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/fa7552a7bacc...94f39f996b60 [21:07:33] [02miraheze/ManageWiki] 07Universal-Omega 0394f39f9 - Update workflow [21:07:34] [02miraheze/CustomHeader] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CustomHeader/compare/fdd595741e54...50ff1b3a05d5 [21:07:35] [02miraheze/CustomHeader] 07Universal-Omega 0350ff1b3 - Update workflow [21:07:38] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/5b6e69a169cc...8bceae3ea7af [21:07:41] [02miraheze/MirahezeMagic] 07Universal-Omega 038bceae3 - Update workflow [21:07:42] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/a99b17ff0468...82961cd7420a [21:07:45] [02miraheze/ImportDump] 07Universal-Omega 0382961cd - Update workflow [21:07:47] [02miraheze/DataDump] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/DataDump/compare/6f8bcb95ee15...cf7d9858b38d [21:07:48] [02miraheze/DataDump] 07Universal-Omega 03cf7d985 - Update workflow [21:07:49] [02miraheze/MatomoAnalytics] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MatomoAnalytics/compare/709676b3086f...d778615294cb [21:07:52] [02miraheze/MatomoAnalytics] 07Universal-Omega 03d778615 - Update workflow [21:07:55] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/RemovePII/compare/590a31358e16...e7df9eca6751 [21:07:56] [02miraheze/RemovePII] 07Universal-Omega 03e7df9ec - Update workflow [21:07:58] [02miraheze/RottenLinks] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/7ca5fcae6a5d...05264fcb3351 [21:08:00] [02miraheze/RottenLinks] 07Universal-Omega 0305264fc - Update workflow [21:08:02] [02MirahezeMagic] 07dependabot[bot] edited pull request 03#386: Bump actions/cache from 3.2.1 to 3.2.2 - 13https://github.com/miraheze/MirahezeMagic/pull/386 [21:08:12] [02MirahezeMagic] 07dependabot[bot] commented on pull request 03#386: Bump actions/cache from 3.2.1 to 3.2.2 - 13https://github.com/miraheze/MirahezeMagic/pull/386#issuecomment-1366182142 [21:08:13] [02MirahezeMagic] 07dependabot[bot] edited pull request 03#386: Bump actions/cache from 3.2.1 to 3.2.2 - 13https://github.com/miraheze/MirahezeMagic/pull/386 [21:08:14] [02MirahezeMagic] 07dependabot[bot] closed pull request 03#386: Bump actions/cache from 3.2.1 to 3.2.2 - 13https://github.com/miraheze/MirahezeMagic/pull/386 [21:08:17] [02miraheze/MirahezeMagic] 07dependabot[bot] deleted branch 03dependabot/github_actions/actions/cache-3.2.2 [21:08:19] [02MirahezeMagic] 07dependabot[bot] deleted branch 03dependabot/github_actions/actions/cache-3.2.2 - 13https://github.com/miraheze/MirahezeMagic [21:09:34] [02miraheze/SpriteSheet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/SpriteSheet/compare/cc0be9db5c2d...6d7ea5f8e021 [21:09:36] [02miraheze/SpriteSheet] 07Universal-Omega 036d7ea5f - Update workflow [21:11:00] [02miraheze/WikiDiscover] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/WikiDiscover/compare/1f625e53c066...245cb80a9550 [21:11:01] [02miraheze/WikiDiscover] 07Universal-Omega 03245cb80 - Update workflow [21:11:38] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/46009fd3d3ff...75b0f09e74b6 [21:11:40] [02miraheze/puppet] 07JohnFLewis 0375b0f09 - Syntax fix for db.pp [21:12:10] RECOVERY - db112 Puppet on db112 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [21:12:16] PROBLEM - db142 Puppet on db142 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [21:12:51] [02miraheze/YouTube] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/YouTube/compare/e9400c98a7f1...ca19af1618a6 [21:12:52] [02miraheze/YouTube] 07Universal-Omega 03ca19af1 - Update workflow [21:15:01] miraheze/MirahezeMagic - dependabot[bot] the build has errored. [21:16:48] miraheze/MirahezeMagic - Universal-Omega the build has errored. [21:20:29] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/75b0f09e74b6...2d6f2c13bb3c [21:20:31] [02miraheze/puppet] 07JohnFLewis 032d6f2c1 - T8350: Backup mhglobal weekly [21:21:37] PROBLEM - db121 Puppet on db121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [21:21:43] miraheze/IncidentReporting - Universal-Omega the build has errored. [21:21:47] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/2d6f2c13bb3c...f39498a0001a [21:21:50] [02miraheze/puppet] 07JohnFLewis 03f39498a - Add default value for hiera lookup for db backup freq [21:22:11] miraheze/RottenLinks - Universal-Omega the build has errored. [21:22:37] miraheze/GlobalNewFiles - Universal-Omega the build passed. [21:22:47] miraheze/MatomoAnalytics - Universal-Omega the build has errored. [21:23:37] PROBLEM - db112 Backups SQL grafana on db112 is CRITICAL: FILE_AGE CRITICAL: File not found - /var/log/sql-grafana-backup.log [21:23:46] PROBLEM - db112 Backups SQL icinga on db112 is CRITICAL: FILE_AGE CRITICAL: File not found - /var/log/sql-icinga-backup.log [21:23:56] PROBLEM - db112 Backups SQL on db112 is CRITICAL: FILE_AGE CRITICAL: File not found - /var/log/sql-backup.log [21:24:00] PROBLEM - db112 Backups SQL roundcubemail on db112 is CRITICAL: FILE_AGE CRITICAL: File not found - /var/log/sql-roundcubemail-backup.log [21:24:13] miraheze/ManageWiki - Universal-Omega the build passed. [21:24:16] PROBLEM - db112 Backups SQL piwik on db112 is CRITICAL: FILE_AGE CRITICAL: File not found - /var/log/sql-piwik-backup.log [21:24:24] PROBLEM - db112 Backups SQL reports on db112 is CRITICAL: FILE_AGE CRITICAL: File not found - /var/log/sql-reports-backup.log [21:24:28] PROBLEM - db112 Backups SQL icingaweb2 on db112 is CRITICAL: FILE_AGE CRITICAL: File not found - /var/log/sql-icingaweb2-backup.log [21:24:30] miraheze/CustomHeader - Universal-Omega the build passed. [21:25:27] miraheze/DataDump - Universal-Omega the build passed. [21:25:51] miraheze/PDFEmbed - Universal-Omega the build passed. [21:26:06] miraheze/RemovePII - Universal-Omega the build passed. [21:26:14] miraheze/SpriteSheet - Universal-Omega the build passed. [21:26:28] miraheze/WikiDiscover - Universal-Omega the build passed. [21:26:58] miraheze/YouTube - Universal-Omega the build passed. [21:27:34] miraheze/CreateWiki - Universal-Omega the build passed. [21:30:58] miraheze/ImportDump - Universal-Omega the build passed. [21:31:01] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f39498a0001a...d7684e87b47e [21:31:02] [02miraheze/puppet] 07JohnFLewis 03d7684e8 - T8350: Add 3-monthly full MediaWiki XML backups [21:36:40] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [21:39:57] PROBLEM - phab121 Current Load on phab121 is CRITICAL: CRITICAL - load average: 2.45, 1.77, 1.05 [21:40:36] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 7 = Critical] [21:56:20] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.1910802722 secs [22:05:38] PROBLEM - phab121 Current Load on phab121 is WARNING: WARNING - load average: 0.37, 1.58, 1.83 [22:07:38] RECOVERY - phab121 Current Load on phab121 is OK: OK - load average: 0.29, 1.18, 1.65 [22:08:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 9.29, 5.76, 3.20 [22:11:19] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.70, 3.01, 2.55 [22:11:27] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 6.82, 5.64, 4.51 [22:11:39] PROBLEM - phab121 Current Load on phab121 is CRITICAL: CRITICAL - load average: 3.96, 3.00, 2.32 [22:12:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.33, 3.82, 3.05 [22:13:15] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 3.21, 3.10, 2.65 [22:13:25] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 6.51, 6.03, 4.80 [22:14:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.47, 3.02, 2.85 [22:19:59] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 4.13, 3.13, 1.76 [22:21:04] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 4.11, 3.68, 3.09 [22:21:53] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.26, 2.48, 1.69 [22:23:01] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.05, 3.44, 3.07 [22:24:58] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 4.01, 3.80, 3.26 [22:28:14] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d7684e87b47e...5067f1a83843 [22:28:16] [02miraheze/puppet] 07JohnFLewis 035067f1a - Integrate phabricator backups into one [22:28:51] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 1.69, 2.92, 3.04 [22:29:57] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset 0.06010824442 secs [22:31:23] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/c598f7e097ad [22:31:24] [02miraheze/puppet] 07Universal-Omega 03c598f7e - Add PHP slowlog to rsyslog [22:31:26] [02puppet] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/puppet [22:31:29] [02puppet] 07Universal-Omega opened pull request 03#3101: Add PHP slowlog to rsyslog - 13https://github.com/miraheze/puppet/pull/3101 [22:35:38] PROBLEM - phab121 Current Load on phab121 is WARNING: WARNING - load average: 0.46, 0.80, 1.83 [22:39:38] RECOVERY - phab121 Current Load on phab121 is OK: OK - load average: 0.70, 0.75, 1.57 [22:43:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/7441b10715df [22:43:16] [02miraheze/puppet] 07paladox 037441b10 - Fix typo [22:43:19] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://github.com/miraheze/puppet [22:43:20] [02puppet] 07paladox opened pull request 03#3102: Fix typo - 13https://github.com/miraheze/puppet/pull/3102 [22:43:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7441b10715df...c7fb4fa0b567 [22:43:39] [02miraheze/puppet] 07paladox 03c7fb4fa - Update db101.yaml [22:43:41] [02puppet] 07paladox synchronize pull request 03#3102: Fix typo - 13https://github.com/miraheze/puppet/pull/3102 [22:43:47] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset -0.1176339388 secs [22:44:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c7fb4fa0b567...482f5c5c648c [22:44:02] [02miraheze/puppet] 07paladox 03482f5c5 - Update db101.yaml [22:44:05] [02puppet] 07paladox synchronize pull request 03#3102: Fix typo - 13https://github.com/miraheze/puppet/pull/3102 [22:44:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/482f5c5c648c...3934f139446a [22:44:31] [02miraheze/puppet] 07paladox 033934f13 - Update db.pp [22:44:32] [02puppet] 07paladox synchronize pull request 03#3102: Fix typo - 13https://github.com/miraheze/puppet/pull/3102 [22:44:42] [02puppet] 07paladox closed pull request 03#3102: Fix typo - 13https://github.com/miraheze/puppet/pull/3102 [22:44:44] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-2 [22:44:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5067f1a83843...afde62e7e544 [22:44:48] [02miraheze/puppet] 07paladox 03afde62e - Fix typo (#3102) [22:44:49] [02puppet] 07paladox deleted branch 03paladox-patch-2 - 13https://github.com/miraheze/puppet [22:45:45] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset -0.08042109013 secs [22:48:00] RECOVERY - db101 Puppet on db101 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:49:22] RECOVERY - db121 Puppet on db121 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [22:50:11] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/59d41770d5f5 [22:50:14] [02miraheze/mw-config] 07paladox 0359d4177 - Add db142 and switch c3 to it [22:50:16] [02mw-config] 07paladox created branch 03paladox-patch-3 - 13https://github.com/miraheze/mw-config [22:50:19] [02mw-config] 07paladox opened pull request 03#5046: Add db142 and switch c3 to it - 13https://github.com/miraheze/mw-config/pull/5046 [22:51:17] miraheze/mw-config - paladox the build passed. [22:52:32] [02mw-config] 07paladox closed pull request 03#5046: Add db142 and switch c3 to it - 13https://github.com/miraheze/mw-config/pull/5046 [22:52:33] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/2bc2ebc700e8...dfbe41191da3 [22:52:35] [02miraheze/mw-config] 07paladox 03dfbe411 - Add db142 and switch c3 to it (#5046) [22:52:37] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-3 [22:52:38] [02mw-config] 07paladox deleted branch 03paladox-patch-3 - 13https://github.com/miraheze/mw-config [22:52:46] RECOVERY - db142 MariaDB on db142 is OK: Uptime: 210944 Threads: 1 Questions: 1280268 Slow queries: 125 Opens: 127194 Open tables: 674 Queries per second avg: 6.069 [22:53:02] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [22:53:09] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 6s [22:53:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:53:09] RECOVERY - db142 MariaDB Connections on db142 is OK: OK connection usage: 0.2%Current connections: 1 [22:53:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:53:34] miraheze/mw-config - paladox the build passed. [22:53:41] PROBLEM - db101 Backups SQL on db101 is CRITICAL: FILE_AGE CRITICAL: File not found - /var/log/sql-backup.log [22:53:55] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/dfbe41191da3...c52397e9ff44 [22:53:56] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [22:53:56] PROBLEM - db101 Backups SQL mhglobal on db101 is CRITICAL: FILE_AGE CRITICAL: File not found - /var/log/sql-mhglobal-backup.log [22:53:57] [02miraheze/mw-config] 07paladox 03c52397e - Take c3 out of displaying an error page [22:54:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:54:02] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 5s [22:54:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:54:11] !log [paladox@mwtask141] starting deploy of {'config': True} to all [22:54:12] PROBLEM - db121 Backups SQL on db121 is CRITICAL: FILE_AGE CRITICAL: File not found - /var/log/sql-backup.log [22:54:17] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 5s [22:54:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:54:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:54:46] paladox: need to remove from L165 also I think. [22:54:58] miraheze/mw-config - paladox the build passed. [22:55:46] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/c52397e9ff44...740abd1b7969 [22:55:48] [02miraheze/mw-config] 07paladox 03740abd1 - Fix [22:55:49] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [22:55:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:55:56] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 6s [22:55:58] !log [paladox@mwtask141] starting deploy of {'config': True} to all [22:56:02] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 4s [22:56:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:56:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:56:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:56:46] miraheze/mw-config - paladox the build passed. [22:57:51] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/740abd1b7969...c25a849fa74b [22:57:53] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [22:57:53] [02miraheze/mw-config] 07paladox 03c25a849 - Take c3 (db142) out of read only [22:57:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:58:00] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 6s [22:58:01] !log [paladox@mwtask141] starting deploy of {'config': True} to all [22:58:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:58:07] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 5s [22:58:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:58:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:58:51] miraheze/mw-config - paladox the build passed. [23:01:09] RECOVERY - db131 Puppet on db131 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [23:01:42] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.83, 3.48, 1.86 [23:03:40] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.70, 2.72, 1.78 [23:10:22] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.30, 7.55, 4.17 [23:10:44] RECOVERY - db142 Puppet on db142 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [23:11:56] PROBLEM - mwtask141 Puppet on mwtask141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [23:16:21] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.38, 3.40, 3.36 [23:24:36] PROBLEM - db142 Backups SQL on db142 is CRITICAL: FILE_AGE CRITICAL: File not found - /var/log/sql-backup.log [23:24:45] PROBLEM - db131 Backups SQL on db131 is CRITICAL: FILE_AGE CRITICAL: File not found - /var/log/sql-backup.log [23:25:02] PROBLEM - test131 Backups MediaWiki XML on test131 is UNKNOWN: NRPE: Command 'check_Backups MediaWiki XML' not defined [23:25:05] PROBLEM - mwtask141 Backups MediaWiki XML on mwtask141 is UNKNOWN: NRPE: Command 'check_Backups MediaWiki XML' not defined