[00:01:39] !log reboot mon1 [00:01:41] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [00:08:15] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/e7e8fe41cfd8...d25a6c1dde2d [00:08:15] [02WikiTideOrg/puppet] 07Universal-Omega 03d25a6c1 - icingaweb2: disable db SSL [00:14:41] PROBLEM - nexttide.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query nexttide.org. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [00:22:19] RECOVERY - db1 MariaDB on db1 is OK: Uptime: 8961 Threads: 9 Questions: 3725611 Slow queries: 188117 Opens: 2528 Open tables: 2474 Queries per second avg: 415.758 [00:22:25] RECOVERY - db1 MariaDB Connections on db1 is OK: OK connection usage: 1.6%Current connections: 8 [00:29:18] RECOVERY - mon1 Puppet on mon1 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [00:30:09] PROBLEM - db1 MariaDB Connections on db1 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Access denied for user 'icinga'@'10.0.0.111' (using password: YES) in /usr/lib/nagios/plugins/check_mysql_connections.php:49Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(49): mysqli_real_connect(Object(mysqli), 'db1.wikitide.ne...', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, false)#1 {main} t [00:30:09] /nagios/plugins/check_mysql_connections.php on line 49Fatal error: Uncaught mysqli_sql_exception: Access denied for user 'icinga'@'10.0.0.111' (using password: YES) in /usr/lib/nagios/plugins/check_mysql_connections.php:49Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(49): mysqli_real_connect(Object(mysqli), 'db1.wikitide.ne...', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, false)#1 {main} thro [00:30:09] gios/plugins/check_mysql_connections.php on line 49 [00:30:09] PROBLEM - db1 MariaDB on db1 is CRITICAL: Access denied for user 'icinga'@'10.0.0.111' (using password: YES) [00:31:41] PROBLEM - mon1 Check correctness of the icinga configuration on mon1 is CRITICAL: Icinga configuration contains errors [00:36:24] RECOVERY - valorantjp.wiki - reverse DNS on sslhost is OK: SSL OK - valorantjp.wiki reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [00:36:40] RECOVERY - db1 MariaDB on db1 is OK: Uptime: 9821 Threads: 8 Questions: 4051453 Slow queries: 204674 Opens: 2617 Open tables: 2563 Queries per second avg: 412.529 [00:36:44] RECOVERY - mon1 Check correctness of the icinga configuration on mon1 is OK: Icinga configuration is correct [00:37:51] RECOVERY - db1 MariaDB Connections on db1 is OK: OK connection usage: 1.8%Current connections: 9 [00:46:39] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/d25a6c1dde2d...9b617c8c0aed [00:46:40] [02WikiTideOrg/puppet] 07Universal-Omega 039b617c8 - Remove duplicate comment [01:36:28] RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [02:07:29] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [02:13:00] RECOVERY - nexttide.org - reverse DNS on sslhost is OK: SSL OK - nexttide.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [06:27:05] RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [06:58:01] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [07:26:47] RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [07:37:26] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [09:28:03] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.403 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [12:51:34] RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [13:33:21] PROBLEM - valorantjp.wiki - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query valorantjp.wiki. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [13:52:55] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [14:03:00] PROBLEM - valorantjp.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - valorantjp.wiki All nameservers failed to answer the query. [14:15:37] PROBLEM - nexttide.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - nexttide.org All nameservers failed to answer the query. [14:32:35] RECOVERY - valorantjp.wiki - reverse DNS on sslhost is OK: SSL OK - valorantjp.wiki reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [14:45:25] PROBLEM - nexttide.org - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [14:52:07] [02landing] 07ivanmarribas starred 03landing - 13https://github.com/ivanmarribas [15:15:09] RECOVERY - nexttide.org - reverse DNS on sslhost is OK: SSL OK - nexttide.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [15:20:08] RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [15:52:16] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [16:21:16] RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [17:22:16] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [18:04:48] [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/14fda65a8e36...fc934daca52c [18:04:51] [02WikiTideOrg/mw-config] 07Universal-Omega 03fc934da - Fix PoolCounter config [18:05:42] WikiTideOrg/mw-config - Universal-Omega the build passed. [18:17:47] !log [@jobrunner1] starting deploy of {'config': True, 'versions': ['']} to [mw1, mw2, jobrunner1] [18:17:49] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [18:17:49] !log [@jobrunner1] finished deploy of {'config': True, 'versions': ['']} to [mw1, mw2, jobrunner1] - SUCCESS in 2s [18:17:51] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [18:18:24] [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03move-wgHooks [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/commit/be9bd5c7f98b [18:18:27] [02WikiTideOrg/mw-config] 07Universal-Omega 03be9bd5c - Move wgHooks to GlobalSettings [18:18:30] [02mw-config] 07Universal-Omega created branch 03move-wgHooks - 13https://github.com/WikiTideOrg/mw-config [18:18:33] [02mw-config] 07Universal-Omega opened pull request 03#21: Move wgHooks to GlobalSettings - 13https://github.com/WikiTideOrg/mw-config/pull/21 [18:19:16] [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03move-wgHooks [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/be9bd5c7f98b...7d63d7b635f0 [18:19:19] [02WikiTideOrg/mw-config] 07Universal-Omega 037d63d7b - Add [18:19:21] [02mw-config] 07Universal-Omega synchronize pull request 03#21: Move wgHooks to GlobalSettings - 13https://github.com/WikiTideOrg/mw-config/pull/21 [18:19:38] [02mw-config] 07Universal-Omega edited pull request 03#21: Move wgHooks to GlobalSettings - 13https://github.com/WikiTideOrg/mw-config/pull/21 [18:20:16] WikiTideOrg/mw-config - Universal-Omega the build passed. [18:20:39] [02mw-config] 07Universal-Omega closed pull request 03#21: Move wgHooks to GlobalSettings - 13https://github.com/WikiTideOrg/mw-config/pull/21 [18:20:40] [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03main [+0/-0/±2] 13https://github.com/WikiTideOrg/mw-config/compare/fc934daca52c...8489bb55a530 [18:20:41] [02WikiTideOrg/mw-config] 07Universal-Omega 038489bb5 - Move wgHooks to GlobalSettings [18:20:43] [02WikiTideOrg/mw-config] 07Universal-Omega deleted branch 03move-wgHooks [18:20:44] [02mw-config] 07Universal-Omega deleted branch 03move-wgHooks - 13https://github.com/WikiTideOrg/mw-config [18:20:59] [02mw-config] 07Universal-Omega commented on pull request 03#7: Update mediawiki/mediawiki-codesniffer requirement from 41.0.0 to 42.0.0 - 13https://github.com/WikiTideOrg/mw-config/pull/7#issuecomment-1837221226 [18:21:02] [02mw-config] 07dependabot[bot] edited pull request 03#7: Update mediawiki/mediawiki-codesniffer requirement from 41.0.0 to 42.0.0 - 13https://github.com/WikiTideOrg/mw-config/pull/7 [18:21:16] [02mw-config] 07dependabot[bot] edited pull request 03#7: Update mediawiki/mediawiki-codesniffer requirement from 41.0.0 to 42.0.0 - 13https://github.com/WikiTideOrg/mw-config/pull/7 [18:21:19] [02WikiTideOrg/mw-config] 07dependabot[bot] pushed 031 commit to 03dependabot/composer/mediawiki/mediawiki-codesniffer-42.0.0 [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/852096f3c9e3...e7e7a0638ca5 [18:21:21] [02WikiTideOrg/mw-config] 07dependabot[bot] 03e7e7a06 - Update mediawiki/mediawiki-codesniffer requirement from 41.0.0 to 42.0.0 Updates the requirements on [mediawiki/mediawiki-codesniffer](https://github.com/wikimedia/mediawiki-tools-codesniffer) to permit the latest version. - [Changelog](https://github.com/wikimedia/mediawiki-tools-codesniffer/blob/master/HISTORY.md) - [Commits](https://github.com/wikimedia/mediawiki-tools-code [18:21:22] [02mw-config] 07dependabot[bot] synchronize pull request 03#7: Update mediawiki/mediawiki-codesniffer requirement from 41.0.0 to 42.0.0 - 13https://github.com/WikiTideOrg/mw-config/pull/7 [18:21:36] WikiTideOrg/mw-config - Universal-Omega the build passed. [18:22:18] WikiTideOrg/mw-config - dependabot[bot] the build passed. [18:22:33] [02mw-config] 07Universal-Omega closed pull request 03#7: Update mediawiki/mediawiki-codesniffer requirement from 41.0.0 to 42.0.0 - 13https://github.com/WikiTideOrg/mw-config/pull/7 [18:22:35] [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/8489bb55a530...6a1725046b41 [18:22:38] [02WikiTideOrg/mw-config] 07dependabot[bot] 036a17250 - Update mediawiki/mediawiki-codesniffer requirement from 41.0.0 to 42.0.0 (#7) Updates the requirements on [mediawiki/mediawiki-codesniffer](https://github.com/wikimedia/mediawiki-tools-codesniffer) to permit the latest version. - [Changelog](https://github.com/wikimedia/mediawiki-tools-codesniffer/blob/master/HISTORY.md) - [Commits](https://github.com/wikimedia/mediawiki-tools [18:22:41] [02WikiTideOrg/mw-config] 07Universal-Omega deleted branch 03dependabot/composer/mediawiki/mediawiki-codesniffer-42.0.0 [18:22:44] [02mw-config] 07Universal-Omega deleted branch 03dependabot/composer/mediawiki/mediawiki-codesniffer-42.0.0 - 13https://github.com/WikiTideOrg/mw-config [18:23:31] WikiTideOrg/mw-config - Universal-Omega the build passed. [18:31:11] !log [@test1] starting deploy of {'config': True, 'versions': ['']} to [test1] [18:31:12] !log [@test1] finished deploy of {'config': True, 'versions': ['']} to [test1] - SUCCESS in 0s [18:31:13] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [18:31:14] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [18:42:37] [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03lists [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/commit/a1e7c4572885 [18:42:38] [02WikiTideOrg/mw-config] 07Universal-Omega 03a1e7c45 - Use unsuffixed DB lists [18:42:40] [02mw-config] 07Universal-Omega created branch 03lists - 13https://github.com/WikiTideOrg/mw-config [18:42:41] [02mw-config] 07Universal-Omega opened pull request 03#22: Use unsuffixed DB lists - 13https://github.com/WikiTideOrg/mw-config/pull/22 [18:43:07] [02WikiTideOrg/mw-config] 07github-actions[bot] pushed 031 commit to 03lists [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/a1e7c4572885...2f194524e1ea [18:43:09] [02WikiTideOrg/mw-config] 07github-actions 032f19452 - CI: lint code to MediaWiki standards Check commit and GitHub actions for more details [18:43:12] [02mw-config] 07github-actions[bot] synchronize pull request 03#22: Use unsuffixed DB lists - 13https://github.com/WikiTideOrg/mw-config/pull/22 [18:43:36] WikiTideOrg/mw-config - Universal-Omega the build passed. [18:44:56] [02mw-config] 07Universal-Omega synchronize pull request 03#22: Use unsuffixed DB lists - 13https://github.com/WikiTideOrg/mw-config/pull/22 [18:44:57] [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03lists [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/2f194524e1ea...964e65ebaa48 [18:44:58] [02WikiTideOrg/mw-config] 07Universal-Omega 03964e65e - Fix [18:45:47] WikiTideOrg/mw-config - Universal-Omega the build passed. [18:47:52] !log [@jobrunner1] starting deploy of {'config': True, 'versions': ['']} to [mw1, mw2, jobrunner1] [18:47:54] !log [@jobrunner1] finished deploy of {'config': True, 'versions': ['']} to [mw1, mw2, jobrunner1] - SUCCESS in 1s [18:47:54] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [18:47:56] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [20:36:13] PROBLEM - valorantjp.wiki - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.402 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [21:05:52] PROBLEM - valorantjp.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - valorantjp.wiki All nameservers failed to answer the query. [21:20:00] RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [21:35:31] RECOVERY - valorantjp.wiki - reverse DNS on sslhost is OK: SSL OK - valorantjp.wiki reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [22:22:25] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [22:26:58] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/9b617c8c0aed...e394c101d575 [22:27:00] [02WikiTideOrg/puppet] 07Universal-Omega 03e394c10 - grafana: disable SSL for database [22:28:35] !log changed Grafana user DB password [22:28:37] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [22:36:50] PROBLEM - valorantjp.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - valorantjp.wiki All nameservers failed to answer the query. [22:41:04] RECOVERY - mon1 HTTPS on mon1 is OK: HTTP OK: HTTP/2 302 - 374 bytes in 0.022 second response time [22:42:14] !log apply grafana grants [22:42:16] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [22:44:07] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/e394c101d575...c1f222a83eb6 [22:44:10] [02WikiTideOrg/puppet] 07Universal-Omega 03c1f222a - Add grafana also [22:53:53] !log fix permissions for some ssl keys on puppet1 [22:53:55] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [22:58:47] !log started Grafana [22:58:49] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [23:06:30] PROBLEM - valorantjp.wiki - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [23:15:13] PROBLEM - nexttide.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - nexttide.org All nameservers failed to answer the query. [23:20:31] RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [23:23:20] [02WikiTideOrg/puppet] 07Universal-Omega deleted branch 03revert-25-mariadb-remove-ssl [23:23:22] [02puppet] 07Universal-Omega deleted branch 03revert-25-mariadb-remove-ssl - 13https://github.com/WikiTideOrg/puppet [23:27:17] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/WikiTideOrg/puppet/compare/c1f222a83eb6...71e356e0f205 [23:27:20] [02WikiTideOrg/puppet] 07Universal-Omega 0371e356e - Revert "mediawiki: stream sitemap files directly" This reverts commit bc531e6. [23:36:09] RECOVERY - valorantjp.wiki - reverse DNS on sslhost is OK: SSL OK - valorantjp.wiki reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [23:44:46] RECOVERY - nexttide.org - reverse DNS on sslhost is OK: SSL OK - nexttide.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [23:51:54] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.