[00:27:55] !log reset www-data crontab on mwtask151, mwtask161, and mwtask171 [00:28:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:28:56] !log [universalomega@mwtask181] starting deploy of {'versions': '1.42', 'upgrade_extensions': 'InterwikiDispatcher'} to all [00:29:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:29:24] !log [universalomega@mwtask181] finished deploy of {'versions': '1.42', 'upgrade_extensions': 'InterwikiDispatcher'} to all - SUCCESS in 28s [00:29:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:32:01] [02ErrorPages] 07AgentIsai pushed 031 commit to 03master [+4/-0/±0] 13https://github.com/miraheze/ErrorPages/compare/1735964a9887...e23656eb57e6 [00:32:03] [02ErrorPages] 07AgentIsai 03e23656e - Prototype experiment with custom CF pages [00:32:46] miraheze/ErrorPages - AgentIsai the build passed. [01:26:56] [02ErrorPages] 07AgentIsai pushed 031 commit to 03master [+2/-0/±3] 13https://github.com/miraheze/ErrorPages/compare/e23656eb57e6...b050dff2484b [01:26:58] [02ErrorPages] 07AgentIsai 03b050dff - Update pages [01:27:40] miraheze/ErrorPages - AgentIsai the build passed. [01:32:33] [02ErrorPages] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ErrorPages/compare/b050dff2484b...de6bf0d0b937 [01:32:35] [02ErrorPages] 07AgentIsai 03de6bf0d - 1000: remove footer [01:33:19] miraheze/ErrorPages - AgentIsai the build passed. [01:34:01] [02ErrorPages] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ErrorPages/compare/de6bf0d0b937...f0750c665c21 [01:34:02] [02ErrorPages] 07AgentIsai 03f0750c6 - 1000: remove header [01:34:44] miraheze/ErrorPages - AgentIsai the build passed. [02:03:36] PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 40.67, 19.37, 7.78 [02:04:05] PROBLEM - db181 PowerDNS Recursor on db181 is CRITICAL: CRITICAL - Plugin timed out while executing system call [02:04:42] man down [02:05:08] PROBLEM - db181 APT on db181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:05:33] PROBLEM - db181 Puppet on db181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:07:37] PROBLEM - db181 Backups SQL on db181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:09:32] RECOVERY - db181 Backups SQL on db181 is OK: FILE_AGE OK: /var/log/sql-backup.log is 411827 seconds old and 81091 bytes [02:09:37] RECOVERY - db181 Current Load on db181 is OK: LOAD OK - total load average: 0.78, 0.23, 0.08 [02:10:01] RECOVERY - db181 APT on db181 is OK: APT OK: 73 packages available for upgrade (0 critical updates). [02:10:25] RECOVERY - db181 PowerDNS Recursor on db181 is OK: DNS OK: 0.057 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [02:10:29] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [02:20:11] [02puppet] 07AgentIsai pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/81384491bdca...d8fe91208842 [02:20:13] [02puppet] 07AgentIsai 03d8fe912 - T12572: Provision db172 [02:21:56] [02dns] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/21583a153082...9ca88f5cb572 [02:21:57] [02dns] 07AgentIsai 039ca88f5 - T12572: Add db172 to hosts [02:22:12] @agentisai what happened to db181? [02:22:25] genuinely have no idea, it just hung [02:22:35] hmm [02:22:35] I thought maybe it was dbbackups but that ran on the 5th [02:22:54] is this bad [02:23:08] it seems to be back now so.. [02:23:20] I rebooted it as it wouldn't let me ssh in [02:23:32] it responded gracefully to the reboot request after like 5 minutes [02:23:57] ah [02:24:17] been a fairly long time since a db crashed. [02:26:58] https://cdn.discordapp.com/attachments/808001911868489748/1282890027231088714/JyfwiPB.png?ex=66e0fff1&is=66dfae71&hm=db242e460590b79e688bbe9eaf4f046a6064c7fcb772d845ea80106fc45ae476& [02:34:04] [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/9ca88f5cb572...99d465cf073b [02:34:06] [02dns] 07Universal-Omega 0399d465c - Fix tabbing [02:39:21] visual studio is so misleading [02:41:42] I use github.dev for a lot of things here lol [02:42:06] I love it but at the same time I hate it because of things like that [02:57:29] I only use VS for my C# development projects lol [03:02:54] PROBLEM - mon181 Backups Grafana on mon181 is WARNING: FILE_AGE WARNING: /var/log/grafana-backup.log is 1382552 seconds old and 93 bytes [03:12:42] PROBLEM - db172 MariaDB Connections on db172 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db172.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connect [03:12:42] n line 66Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db172.wikitide....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66 [03:13:06] PROBLEM - db172 MariaDB on db172 is CRITICAL: Can't connect to server on 'db172.wikitide.net' (115) [03:13:14] PROBLEM - db172 Backups SQL on db172 is CRITICAL: FILE_AGE CRITICAL: File not found - /var/log/sql-backup.log [05:15:19] that’s not good [05:15:34] that is very not good [05:16:50] It's nothing, really [05:17:09] db172 isn't a production database and was just set up so that's expected [05:24:07] ah [05:24:47] do you mind taking a look in #general, someone’s getting a particularly odd error that you might know something about [05:41:50] a task has been opened [05:41:51] https://issue-tracker.miraheze.org/T12571 [09:27:15] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/importDump.php --wiki=jtohwiki jtoh.fandom.com-20240809-history.xml --no-updates (END - exit=0) [09:27:16] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/rebuildall.php --wiki=jtohwiki (START) [09:27:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:27:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:29:59] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/rebuildall.php --wiki=jtohwiki (END - exit=256) [09:30:00] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/initSiteStats.php --wiki=jtohwiki --update (END - exit=0) [09:30:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:30:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:09:39] [02puppet] 07redbluegreenhat created branch 03mediawiki-improve-client-cert-logging - 13https://github.com/miraheze/puppet [10:09:40] [02puppet] 07redbluegreenhat pushed 031 commit to 03mediawiki-improve-client-cert-logging [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/6e4ec77f1243 [10:09:42] [02puppet] 07redbluegreenhat 036e4ec77 - NGINX: Improve logging of the client certificate [10:10:38] [02puppet] 07redbluegreenhat opened pull request 03#3926: NGINX: Improve logging of the client certificate - 13https://github.com/miraheze/puppet/pull/3926 [10:10:46] [02puppet] 07coderabbitai[bot] commented on pull request 03#3926: NGINX: Improve logging of the client certificate - 13https://github.com/miraheze/puppet/pull/3926#issuecomment-2340240535 [10:10:55] [02puppet] 07redbluegreenhat commented on pull request 03#3926: NGINX: Improve logging of the client certificate - 13https://github.com/miraheze/puppet/pull/3926#issuecomment-2340240806 [10:11:05] [02puppet] 07coderabbitai[bot] commented on pull request 03#3926: NGINX: Improve logging of the client certificate - 13https://github.com/miraheze/puppet/pull/3926#issuecomment-2340241234 [10:11:10] [02puppet] 07coderabbitai[bot] commented on pull request 03#3926: NGINX: Improve logging of the client certificate - 13https://github.com/miraheze/puppet/pull/3926#issuecomment-2340241344 [10:12:07] [02puppet] 07coderabbitai[bot] edited pull request 03#3926: NGINX: Improve logging of the client certificate - 13https://github.com/miraheze/puppet/pull/3926 [10:12:51] [02puppet] 07redbluegreenhat pushed 031 commit to 03internal-client-certificate-ca [+0/-0/±3] 13https://github.com/miraheze/puppet/compare/08c2c32f8a9e...e9474d6c3286 [10:12:54] [02puppet] 07redbluegreenhat 03e9474d6 - Trust the internal client certificate CA as well [10:12:55] [02puppet] 07redbluegreenhat synchronize pull request 03#3923: Trust the internal client certificate CA as well - 13https://github.com/miraheze/puppet/pull/3923 [10:13:02] [02puppet] 07coderabbitai[bot] edited a comment on pull request 03#3926: NGINX: Improve logging of the client certificate - 13https://github.com/miraheze/puppet/pull/3926#issuecomment-2340241344 [10:13:55] [02puppet] 07redbluegreenhat pushed 031 commit to 03internal-client-certificate-ca [+0/-0/±3] 13https://github.com/miraheze/puppet/compare/e9474d6c3286...1c8b7dce51a5 [10:13:57] [02puppet] 07redbluegreenhat 031c8b7dc - Trust the internal client certificate CA as well [10:13:58] [02puppet] 07redbluegreenhat synchronize pull request 03#3923: Trust the internal client certificate CA as well - 13https://github.com/miraheze/puppet/pull/3923 [10:14:07] miraheze/puppet - redbluegreenhat the build has errored. [10:15:17] miraheze/puppet - redbluegreenhat the build has errored. [10:21:58] [02puppet] 07redbluegreenhat edited pull request 03#3926: NGINX: Improve logging of the client certificate - 13https://github.com/miraheze/puppet/pull/3926 [10:22:05] [02puppet] 07redbluegreenhat closed pull request 03#3926: NGINX: Improve logging of the client certificate - 13https://github.com/miraheze/puppet/pull/3926 [10:22:08] [02puppet] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d8fe91208842...f9fb5808ad93 [10:22:10] [02puppet] 07redbluegreenhat 03f9fb580 - NGINX: Improve logging of the client certificate (#3926) [10:22:12] [02puppet] 07redbluegreenhat deleted branch 03mediawiki-improve-client-cert-logging [10:22:15] [02puppet] 07redbluegreenhat deleted branch 03mediawiki-improve-client-cert-logging - 13https://github.com/miraheze/puppet [10:41:16] [02mw-config] 07BlankEclair opened pull request 03#5657: T12565: Fully disable transcodes for hommwiki - 13https://github.com/miraheze/mw-config/pull/5657 [10:41:56] [02puppet] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f9fb5808ad93...37c10c91e2c1 [10:41:57] [02puppet] 07redbluegreenhat 0337c10c9 - Phorge: Setup client authentication (#3925) [10:42:00] [02puppet] 07redbluegreenhat closed pull request 03#3925: Phorge: Setup client authentication - 13https://github.com/miraheze/puppet/pull/3925 [10:42:02] [02puppet] 07redbluegreenhat deleted branch 03phorge-authenticated-origin-pull - 13https://github.com/miraheze/puppet [10:42:04] [02puppet] 07redbluegreenhat deleted branch 03phorge-authenticated-origin-pull [10:42:06] miraheze/mw-config - BlankEclair the build passed. [10:45:48] [02puppet] 07redbluegreenhat closed pull request 03#3917: T12559: Workaround resources not loading for non-Cloudflare domains - 13https://github.com/miraheze/puppet/pull/3917 [10:45:49] [02puppet] 07redbluegreenhat commented on pull request 03#3917: T12559: Workaround resources not loading for non-Cloudflare domains - 13https://github.com/miraheze/puppet/pull/3917#issuecomment-2340316134 [11:00:47] [02puppet] 07redbluegreenhat pushed 031 commit to 03ssl-cf-and-internal-class [+1/-0/±1] 13https://github.com/miraheze/puppet/commit/7ccdc5d5dae4 [11:00:48] [02puppet] 07redbluegreenhat 037ccdc5d - SSL: Add a class for pulling the client certificate CAs [11:00:50] [02puppet] 07redbluegreenhat created branch 03ssl-cf-and-internal-class - 13https://github.com/miraheze/puppet [11:01:06] [02puppet] 07redbluegreenhat opened pull request 03#3927: SSL: Add a class for pulling the client certificate CAs - 13https://github.com/miraheze/puppet/pull/3927 [11:01:16] [02puppet] 07coderabbitai[bot] commented on pull request 03#3927: SSL: Add a class for pulling the client certificate CAs - 13https://github.com/miraheze/puppet/pull/3927#issuecomment-2340347876 [11:01:44] [02puppet] 07github-actions[bot] pushed 031 commit to 03ssl-cf-and-internal-class [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7ccdc5d5dae4...962717ab7846 [11:01:46] [02puppet] 07github-actions 03962717a - CI: lint puppet code to standards [11:01:48] [02puppet] 07github-actions[bot] synchronize pull request 03#3927: SSL: Add a class for pulling the client certificate CAs - 13https://github.com/miraheze/puppet/pull/3927 [11:03:36] [02puppet] 07redbluegreenhat pushed 031 commit to 03ssl-cf-and-internal-class [+1/-0/±1] 13https://github.com/miraheze/puppet/compare/962717ab7846...fb167990dd4e [11:03:39] [02puppet] 07redbluegreenhat 03fb16799 - SSL: Add a class for pulling the client certificate CAs [11:03:41] [02puppet] 07redbluegreenhat synchronize pull request 03#3927: SSL: Add a class for pulling the client certificate CAs - 13https://github.com/miraheze/puppet/pull/3927 [11:04:13] [02puppet] 07github-actions[bot] pushed 031 commit to 03ssl-cf-and-internal-class [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/fb167990dd4e...f9d2b8e4ae5f [11:04:16] [02puppet] 07github-actions 03f9d2b8e - CI: lint puppet code to standards [11:04:19] [02puppet] 07github-actions[bot] synchronize pull request 03#3927: SSL: Add a class for pulling the client certificate CAs - 13https://github.com/miraheze/puppet/pull/3927 [11:07:44] [02puppet] 07redbluegreenhat pushed 031 commit to 03ssl-cf-and-internal-class [+1/-0/±1] 13https://github.com/miraheze/puppet/compare/f9d2b8e4ae5f...7c3349d7f8b8 [11:07:46] [02puppet] 07redbluegreenhat 037c3349d - SSL: Add a class for pulling the client certificate CAs [11:07:49] [02puppet] 07redbluegreenhat synchronize pull request 03#3927: SSL: Add a class for pulling the client certificate CAs - 13https://github.com/miraheze/puppet/pull/3927 [11:08:17] [02puppet] 07github-actions[bot] pushed 031 commit to 03ssl-cf-and-internal-class [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7c3349d7f8b8...a2f416110191 [11:08:20] [02puppet] 07github-actions 03a2f4161 - CI: lint puppet code to standards [11:08:21] [02puppet] 07github-actions[bot] synchronize pull request 03#3927: SSL: Add a class for pulling the client certificate CAs - 13https://github.com/miraheze/puppet/pull/3927 [11:08:36] [02puppet] 07redbluegreenhat commented on pull request 03#3927: SSL: Add a class for pulling the client certificate CAs - 13https://github.com/miraheze/puppet/pull/3927#issuecomment-2340363263 [11:08:44] [02puppet] 07coderabbitai[bot] commented on pull request 03#3927: SSL: Add a class for pulling the client certificate CAs - 13https://github.com/miraheze/puppet/pull/3927#issuecomment-2340363525 [11:08:46] [02puppet] 07coderabbitai[bot] commented on pull request 03#3927: SSL: Add a class for pulling the client certificate CAs - 13https://github.com/miraheze/puppet/pull/3927#issuecomment-2340363623 [11:11:02] [02puppet] 07coderabbitai[bot] edited pull request 03#3927: SSL: Add a class for pulling the client certificate CAs - 13https://github.com/miraheze/puppet/pull/3927 [11:12:01] [02puppet] 07coderabbitai[bot] edited a comment on pull request 03#3927: SSL: Add a class for pulling the client certificate CAs - 13https://github.com/miraheze/puppet/pull/3927#issuecomment-2340363623 [11:13:23] [02puppet] 07redbluegreenhat edited pull request 03#3927: SSL: Add a class for pulling the client certificate CAs - 13https://github.com/miraheze/puppet/pull/3927 [11:13:29] [02puppet] 07redbluegreenhat closed pull request 03#3927: SSL: Add a class for pulling the client certificate CAs - 13https://github.com/miraheze/puppet/pull/3927 [11:13:32] [02puppet] 07redbluegreenhat pushed 031 commit to 03master [+1/-0/±2] 13https://github.com/miraheze/puppet/compare/37c10c91e2c1...1efc7708b577 [11:13:35] [02puppet] 07redbluegreenhat 031efc770 - SSL: Add a class for pulling the client certificate CAs (#3927) [11:13:37] [02puppet] 07redbluegreenhat deleted branch 03ssl-cf-and-internal-class [11:13:40] [02puppet] 07redbluegreenhat deleted branch 03ssl-cf-and-internal-class - 13https://github.com/miraheze/puppet [14:20:38] PROBLEM - wiki.mahdiruiz.line.pm - Cloudflare on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [14:49:40] RECOVERY - wiki.mahdiruiz.line.pm - Cloudflare on sslhost is OK: OK - Certificate 'wiki.mahdiruiz.line.pm' will expire on Fri 29 Nov 2024 07:59:27 PM GMT +0000. [15:20:49] PROBLEM - wiki.mahdiruiz.line.pm - Cloudflare on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [15:49:48] RECOVERY - wiki.mahdiruiz.line.pm - Cloudflare on sslhost is OK: OK - Certificate 'wiki.mahdiruiz.line.pm' will expire on Fri 29 Nov 2024 07:59:27 PM GMT +0000. [16:12:30] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/importImages.php --wiki=tf2freakshowwiki images --search-recursively (START) [16:12:31] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/importImages.php --wiki=tf2freakshowwiki images --search-recursively (END - exit=0) [16:12:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:12:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:13:03] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/importImages.php --wiki=tf2freakshowwiki images --search-recursively (START) [16:13:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:13:49] !log [universalomega@mwtask181] starting deploy of {'versions': '1.42', 'upgrade_extensions': 'InterwikiDispatcher'} to all [16:13:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:14:17] !log [universalomega@mwtask181] finished deploy of {'versions': '1.42', 'upgrade_extensions': 'InterwikiDispatcher'} to all - SUCCESS in 28s [16:14:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:22:16] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/importImages.php --wiki=tf2freakshowwiki images --search-recursively (END - exit=0) [16:22:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:51:56] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/importImages.php --wiki=songoftheprairiewiki images --search-recursively (START) [16:52:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:04:18] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/importImages.php --wiki=songoftheprairiewiki images --search-recursively (END - exit=0) [17:04:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:14:50] [02mw-config] 07Universal-Omega pushed 031 commit to 03smw-key [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/1a05d765bfa9 [17:14:51] [02mw-config] 07Universal-Omega 031a05d76 - Avoid SMW upgrade key errors [17:14:52] [02mw-config] 07Universal-Omega created branch 03smw-key - 13https://github.com/miraheze/mw-config [17:14:55] [02mw-config] 07Universal-Omega opened pull request 03#5658: Avoid SMW upgrade key errors - 13https://github.com/miraheze/mw-config/pull/5658 [17:15:51] miraheze/mw-config - Universal-Omega the build passed. [17:24:35] [02mw-config] 07Universal-Omega synchronize pull request 03#5658: Avoid SMW upgrade key errors - 13https://github.com/miraheze/mw-config/pull/5658 [17:24:38] [02mw-config] 07Universal-Omega pushed 031 commit to 03smw-key [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/1a05d765bfa9...a71ac67e7e3a [17:24:40] [02mw-config] 07Universal-Omega 03a71ac67 - +when enabled [17:25:03] [02puppet] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/puppet/compare/1efc7708b577...d362d54ff075 [17:25:05] [02puppet] 07redbluegreenhat 03d362d54 - Trust the internal client certificate CA as well (#3923) [17:25:08] [02puppet] 07redbluegreenhat closed pull request 03#3923: Trust the internal client certificate CA as well - 13https://github.com/miraheze/puppet/pull/3923 [17:25:09] [02puppet] 07redbluegreenhat deleted branch 03internal-client-certificate-ca - 13https://github.com/miraheze/puppet [17:25:13] [02puppet] 07redbluegreenhat deleted branch 03internal-client-certificate-ca [17:25:33] miraheze/mw-config - Universal-Omega the build passed. [17:26:11] miraheze/puppet - redbluegreenhat the build has errored. [17:27:02] [02mw-config] 07Universal-Omega synchronize pull request 03#5658: Avoid SMW upgrade key errors - 13https://github.com/miraheze/mw-config/pull/5658 [17:27:03] [02mw-config] 07Universal-Omega pushed 031 commit to 03smw-key [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a71ac67e7e3a...c02d3efecf85 [17:27:06] [02mw-config] 07Universal-Omega 03c02d3ef - order [17:27:57] miraheze/mw-config - Universal-Omega the build passed. [17:30:19] [02mw-config] 07Universal-Omega synchronize pull request 03#5658: Avoid SMW upgrade key errors - 13https://github.com/miraheze/mw-config/pull/5658 [17:30:21] [02mw-config] 07Universal-Omega pushed 031 commit to 03smw-key [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/c02d3efecf85...1dae8784527f [17:30:22] [02mw-config] 07Universal-Omega 031dae878 - order [17:31:13] miraheze/mw-config - Universal-Omega the build passed. [17:38:34] [02mw-config] 07Universal-Omega reviewed pull request 03#5657 commit - 13https://github.com/miraheze/mw-config/pull/5657#discussion_r1752398389 [17:49:45] [02mw-config] 07Universal-Omega closed pull request 03#5658: Avoid SMW upgrade key errors - 13https://github.com/miraheze/mw-config/pull/5658 [17:49:48] [02mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/2b7c0ec105ad...3b3ec4ae8dc2 [17:49:50] [02mw-config] 07Universal-Omega 033b3ec4a - Avoid SMW upgrade key errors (#5658) [17:49:52] [02mw-config] 07Universal-Omega deleted branch 03smw-key - 13https://github.com/miraheze/mw-config [17:49:54] [02mw-config] 07Universal-Omega deleted branch 03smw-key [17:50:41] miraheze/mw-config - Universal-Omega the build passed. [17:50:55] !log [@test151] starting deploy of {'config': True} to test151 [17:51:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:56:59] !log [@mwtask181] starting deploy of {'config': True} to all [17:57:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:57:26] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 27s [17:57:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:58:41] !log [alex@mwtask171] starting deploy of {'force_upgrade': True, 'versions': '1.42', 'upgrade_extensions': 'MirahezeMagic'} to all [17:58:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:59:06] !log [alex@mwtask171] finished deploy of {'force_upgrade': True, 'versions': '1.42', 'upgrade_extensions': 'MirahezeMagic'} to all - SUCCESS in 25s [17:59:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:05:25] [02mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/3b3ec4ae8dc2...77cb292d705f [18:05:27] [02mw-config] 07Universal-Omega 0377cb292 - Disable Evelution [18:05:35] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [18:05:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:06:03] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 27s [18:06:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:06:20] miraheze/mw-config - Universal-Omega the build passed. [18:07:16] !log [@mwtask171] starting deploy of {'config': True} to all [18:07:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:07:40] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 23s [18:07:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:19:35] [02puppet] 07redbluegreenhat pushed 031 commit to 03fix-mwdeploy-in-test [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/eb852a5c1f8f [18:19:38] [02puppet] 07redbluegreenhat 03eb852a5 - Fix test* not pulling the private key for the mwdeploy cert [18:19:39] [02puppet] 07redbluegreenhat created branch 03fix-mwdeploy-in-test - 13https://github.com/miraheze/puppet [18:19:47] [02puppet] 07redbluegreenhat opened pull request 03#3928: Fix test* not pulling the private key for the mwdeploy cert - 13https://github.com/miraheze/puppet/pull/3928 [18:21:18] [02puppet] 07redbluegreenhat closed pull request 03#3928: Fix test* not pulling the private key for the mwdeploy cert - 13https://github.com/miraheze/puppet/pull/3928 [18:21:21] [02puppet] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d362d54ff075...556775f3bd89 [18:21:24] [02puppet] 07redbluegreenhat 03556775f - Fix test* not pulling the private key for the mwdeploy cert (#3928) [18:21:25] [02puppet] 07redbluegreenhat deleted branch 03fix-mwdeploy-in-test - 13https://github.com/miraheze/puppet [18:21:26] [02puppet] 07redbluegreenhat deleted branch 03fix-mwdeploy-in-test [18:21:32] [02puppet] 07coderabbitai[bot] commented on pull request 03#3928: Fix test* not pulling the private key for the mwdeploy cert - 13https://github.com/miraheze/puppet/pull/3928#issuecomment-2341709631 [18:23:30] !log [@test151] starting deploy of {'config': True} to test151 [18:23:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:27:58] PROBLEM - test151 Puppet on test151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [18:29:18] will be fixed once https://github.com/miraheze/puppet/pull/3928 is applied to test151 [18:37:19] [02ErrorPages] 07MacFan4000 opened pull request 03#44: 1000: don't use fixed position for cferror_details - 13https://github.com/miraheze/ErrorPages/pull/44 [18:37:40] [02ErrorPages] 07coderabbitai[bot] commented on pull request 03#44: 1000: don't use fixed position for cferror_details - 13https://github.com/miraheze/ErrorPages/pull/44#issuecomment-2341742581 [18:38:05] miraheze/ErrorPages - MacFan4000 the build passed. [18:38:24] [02ErrorPages] 07Universal-Omega closed pull request 03#44: 1000: don't use fixed position for cferror_details - 13https://github.com/miraheze/ErrorPages/pull/44 [18:38:25] [02ErrorPages] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ErrorPages/compare/f0750c665c21...35ed97b16e45 [18:38:26] [02ErrorPages] 07MacFan4000 0335ed97b - 1000: don't use fixed position for cferror_details (#44) [18:39:14] miraheze/ErrorPages - Universal-Omega the build passed. [18:53:57] RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [19:07:12] Any ides why everything is breaking today [19:11:41] [1/2] Because we are more active and working on things. It at least seems that always correlates to when things break lol. We also just doubled our resources going into MediaWiki and when we did everything was upgraded so some things broke like extension upgrade. We are working on improving the overall site performance so it takes time to get things perfect and som [19:11:42] [2/2] etimes things accidentally break lol. [19:11:58] Gotcha [19:19:48] suffering from success [19:23:38] PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:25:38] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.0002183616161 secs [19:40:05] It's a day ending y [20:03:37] [02mw-config] 07Universal-Omega closed pull request 03#5651: Remove TAGS - 13https://github.com/miraheze/mw-config/pull/5651 [20:03:40] [02mw-config] 07Universal-Omega deleted branch 03TAGS - 13https://github.com/miraheze/mw-config [20:03:41] [02mw-config] 07Universal-Omega deleted branch 03TAGS [20:06:18] [02mw-config] 07Universal-Omega commented on pull request 03#5649: T12526: Add redundent langs to DisabledLanguages on meta - 13https://github.com/miraheze/mw-config/pull/5649#issuecomment-2341916837 [20:40:34] [02mw-config] 07pixDeVl commented on pull request 03#5649: T12526: Add redundent langs to DisabledLanguages on meta - 13https://github.com/miraheze/mw-config/pull/5649#issuecomment-2341971500 [22:27:50] https://issue-tracker.miraheze.org/T12579 @rhinosf1 when you get a chance ^^ [22:48:00] [02mw-config] 07clarfonthey commented on pull request 03#5649: T12526: Add redundent langs to DisabledLanguages on meta - 13https://github.com/miraheze/mw-config/pull/5649#issuecomment-2342295574 [22:59:10] [02phorge-extensions] 07Universal-Omega closed pull request 03#50: Add MediaWikiLinksApplication - 13https://github.com/miraheze/phorge-extensions/pull/50 [22:59:13] [02phorge-extensions] 07Universal-Omega pushed 031 commit to 03master [+3/-0/±1] 13https://github.com/miraheze/phorge-extensions/compare/a2ae5f748053...e832357917f1 [22:59:16] [02phorge-extensions] 07Universal-Omega 03e832357 - Add MediaWikiLinksApplication [22:59:17] [02phorge-extensions] 07Universal-Omega deleted branch 03mediawikilinks - 13https://github.com/miraheze/phorge-extensions [22:59:20] [02phorge-extensions] 07Universal-Omega deleted branch 03mediawikilinks [22:59:56] miraheze/phorge-extensions - Universal-Omega the build passed. [23:14:36] [02python-functions] 07dependabot[bot] created branch 03dependabot/pip/dot-github/pytest-8.3.3 - 13https://github.com/miraheze/python-functions [23:14:38] [02python-functions] 07dependabot[bot] pushed 031 commit to 03dependabot/pip/dot-github/pytest-8.3.3 [+0/-0/±1] 13https://github.com/miraheze/python-functions/commit/02f2b1ab89c7 [23:14:41] [02python-functions] 07dependabot[bot] 0302f2b1a - Bump pytest from 8.2.2 to 8.3.3 in /.github [23:14:44] [02python-functions] 07dependabot[bot] labeled pull request 03#64: Bump pytest from 8.2.2 to 8.3.3 in /.github - 13https://github.com/miraheze/python-functions/pull/64 [23:14:45] [02python-functions] 07dependabot[bot] labeled pull request 03#64: Bump pytest from 8.2.2 to 8.3.3 in /.github - 13https://github.com/miraheze/python-functions/pull/64 [23:14:46] [02python-functions] 07dependabot[bot] opened pull request 03#64: Bump pytest from 8.2.2 to 8.3.3 in /.github - 13https://github.com/miraheze/python-functions/pull/64 [23:14:49] [02python-functions] 07dependabot[bot] commented on pull request 03#49: Bump pytest from 8.2.2 to 8.3.2 in /.github - 13https://github.com/miraheze/python-functions/pull/49#issuecomment-2342320684 [23:14:50] [02python-functions] 07dependabot[bot] closed pull request 03#49: Bump pytest from 8.2.2 to 8.3.2 in /.github - 13https://github.com/miraheze/python-functions/pull/49 [23:14:51] [02python-functions] 07dependabot[bot] deleted branch 03dependabot/pip/dot-github/pytest-8.3.2 [23:14:52] [02python-functions] 07dependabot[bot] deleted branch 03dependabot/pip/dot-github/pytest-8.3.2 - 13https://github.com/miraheze/python-functions [23:14:53] [02python-functions] 07coderabbitai[bot] commented on pull request 03#64: Bump pytest from 8.2.2 to 8.3.3 in /.github - 13https://github.com/miraheze/python-functions/pull/64#issuecomment-2342320773 [23:22:08] miraheze/python-functions - dependabot[bot] the build passed.