[00:00:46] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 1204MiB (6% inode=94%); [02:02:04] RECOVERY - test1 JobChron Service on test1 is OK: PROCS OK: 1 process with args 'redisJobChronService' [02:06:04] PROBLEM - test1 JobChron Service on test1 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobChronService' [02:29:05] PROBLEM - mw2 NTP time on mw2 is WARNING: NTP WARNING: Offset 0.1332964599 secs [02:55:05] RECOVERY - mw2 NTP time on mw2 is OK: NTP OK: Offset -0.02476498485 secs [04:02:04] RECOVERY - test1 JobChron Service on test1 is OK: PROCS OK: 1 process with args 'redisJobChronService' [04:06:04] PROBLEM - test1 JobChron Service on test1 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobChronService' [07:51:35] PROBLEM - wc.wikitide.org on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:53:29] PROBLEM - wc.wikitide.org on sslhost is WARNING: WARNING - Certificate 'wikitide.org' expires in 13 day(s) (Mon 04 Dec 2023 02:58:50 PM GMT +0000). [11:32:04] RECOVERY - test1 JobChron Service on test1 is OK: PROCS OK: 1 process with args 'redisJobChronService' [11:36:04] PROBLEM - test1 JobChron Service on test1 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobChronService' [13:14:46] PROBLEM - cp3 Disk Space on cp3 is CRITICAL: DISK CRITICAL - free space: / 1148MiB (5% inode=94%); [13:15:35] PROBLEM - wc.wikitide.org on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:17:29] PROBLEM - wc.wikitide.org on sslhost is WARNING: WARNING - Certificate 'wikitide.org' expires in 13 day(s) (Mon 04 Dec 2023 02:58:50 PM GMT +0000). [13:36:32] This is fine [17:08:32] !log [agent@jobrunner1] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases-wikitide.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/CreateWiki/maintenance/setContainerAccess.php (START) [17:08:34] !log [agent@jobrunner1] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases-wikitide.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/CreateWiki/maintenance/setContainerAccess.php (END - exit=2) [17:08:52] !log [agent@jobrunner1] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases-wikitide.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/CreateWiki/maintenance/setContainersAccess.php (START) [17:08:58] !log [agent@jobrunner1] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases-wikitide.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/CreateWiki/maintenance/setContainersAccess.php (END - exit=2) [17:17:26] !log [agent@jobrunner1] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases-wikitide.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/CreateWiki/maintenance/setContainersAccess.php (START) [17:17:29] !log [agent@jobrunner1] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases-wikitide.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/CreateWiki/maintenance/setContainersAccess.php (END - exit=2) [17:18:47] !log [agent@jobrunner1] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases-wikitide.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/CreateWiki/maintenance/setContainersAccess.php (START) [17:18:50] !log [agent@jobrunner1] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases-wikitide.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/CreateWiki/maintenance/setContainersAccess.php (END - exit=2) [17:28:39] !log [agent@jobrunner1] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases-wikitide.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/CreateWiki/maintenance/setContainersAccess.php (START) [17:35:14] (logbot offline) [17:35:50] !log [agent@jobrunner1] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases-wikitide.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/CreateWiki/maintenance/setContainersAccess.php (END - exit=0) [17:36:17] Huh what did happen to the logbot... [17:37:05] PROBLEM - mw2 NTP time on mw2 is WARNING: NTP WARNING: Offset 0.1216629446 secs [17:38:33] Can anyone see when logbot disconnected? [17:38:53] 12:11 ⇐ WikiTideLogbot quit (~WikiTideL@cloud1.wikitide.net) Ping timeout: 255 seconds [17:39:08] Thanks [17:43:03] Seems like there is some 500 errors? [17:44:10] https://www.irccloud.com/pastebin/jOzPrYGM/ [17:46:15] [02WikiTideOrg/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/4b2ffae80335...9b05eaee920d [17:46:16] [02WikiTideOrg/puppet] 07AgentIsai 039b05eae - swift-lb: Correct hostname [17:49:11] !log test [17:49:37] WARNING: Received 500 response: [17:49:45] Still same thing [17:51:01] strange, I can access meta just fine [17:55:46] Huh, I can't seem to find anything beyond a 500 error was given in Graylog also [17:56:06] No indication of why. [17:59:33] !log test [18:03:05] RECOVERY - mw2 NTP time on mw2 is OK: NTP OK: Offset -0.01458665729 secs [18:03:55] !log [agent@swiftobject1] Script: Beginning S3 download of all folders [18:05:23] [02WikiTideOrg/mw-config] 07AgentIsai pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/32184361e3d9...05ed7cedd6ff [18:05:25] [02WikiTideOrg/mw-config] 07AgentIsai 0305ed7ce - Begin file maintenance [18:06:12] !log [agent@jobrunner1] starting deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [mw1, mw2, jobrunner1] [18:06:14] !log [agent@jobrunner1] finished deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [mw1, mw2, jobrunner1] - SUCCESS in 2s [18:06:19] WikiTideOrg/mw-config - AgentIsai the build passed. [18:15:02] !log [agent@jobrunner1] starting deploy of {'config': True, 'force': True, 'versions': ['']} to [jobrunner1] [18:15:03] !log [agent@jobrunner1] finished deploy of {'config': True, 'force': True, 'versions': ['']} to [jobrunner1] - SUCCESS in 0s [18:15:13] !log [agent@jobrunner1] starting deploy of {'config': True, 'force': True, 'versions': ['1.40']} to [mw1, mw2, jobrunner1] [18:15:15] !log [agent@jobrunner1] finished deploy of {'config': True, 'force': True, 'versions': ['1.40']} to [mw1, mw2, jobrunner1] - SUCCESS in 1s [18:15:35] PROBLEM - jobrunner1 MediaWiki Rendering on jobrunner1 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [18:16:34] PROBLEM - mw1 MediaWiki Rendering on mw1 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [18:17:02] PROBLEM - jobrunner1 HTTPS on jobrunner1 is CRITICAL: HTTP CRITICAL: HTTP/2 500 - 239 bytes in 0.025 second response time [18:17:02] PROBLEM - mw2 MediaWiki Rendering on mw2 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [18:17:33] PROBLEM - cp3 HTTPS on cp3 is WARNING: HTTP WARNING: HTTP/2 403 - 2621 bytes in 0.610 second response time [18:17:47] PROBLEM - cp5 HTTPS on cp5 is CRITICAL: HTTP CRITICAL: HTTP/2 500 - 2689 bytes in 1.391 second response time [18:18:36] PROBLEM - mw1 HTTPS on mw1 is CRITICAL: HTTP CRITICAL: HTTP/2 500 - 239 bytes in 0.021 second response time [18:18:43] PROBLEM - cp2 HTTPS on cp2 is CRITICAL: HTTP CRITICAL: HTTP/2 500 - 2688 bytes in 0.245 second response time [18:18:51] PROBLEM - cp6 HTTPS on cp6 is CRITICAL: HTTP CRITICAL: HTTP/2 500 - 2688 bytes in 1.441 second response time [18:18:59] PROBLEM - mw2 HTTPS on mw2 is CRITICAL: HTTP CRITICAL: HTTP/2 500 - 239 bytes in 0.027 second response time [18:19:02] RECOVERY - jobrunner1 HTTPS on jobrunner1 is OK: HTTP OK: HTTP/2 200 - 364 bytes in 0.039 second response time [18:19:09] PROBLEM - cp1 HTTPS on cp1 is CRITICAL: HTTP CRITICAL: HTTP/2 500 - 2689 bytes in 0.035 second response time [18:19:15] PROBLEM - cp4 HTTPS on cp4 is CRITICAL: HTTP CRITICAL: HTTP/2 500 - 2689 bytes in 0.781 second response time [18:19:32] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 50% [18:19:33] PROBLEM - cp3 HTTPS on cp3 is CRITICAL: HTTP CRITICAL: HTTP/2 500 - 2689 bytes in 0.629 second response time [18:19:36] PROBLEM - jobrunner1 MediaWiki Rendering on jobrunner1 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.379 second response time [18:22:50] RECOVERY - cp6 HTTPS on cp6 is OK: HTTP OK: HTTP/2 200 - 2955 bytes in 1.482 second response time [18:22:59] RECOVERY - mw2 HTTPS on mw2 is OK: HTTP OK: HTTP/2 200 - 357 bytes in 0.031 second response time [18:23:03] PROBLEM - mw2 MediaWiki Rendering on mw2 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.380 second response time [18:23:09] RECOVERY - cp1 HTTPS on cp1 is OK: HTTP OK: HTTP/2 200 - 2934 bytes in 0.034 second response time [18:23:15] RECOVERY - cp4 HTTPS on cp4 is OK: HTTP OK: HTTP/2 200 - 2934 bytes in 0.799 second response time [18:23:32] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 32% [18:23:33] RECOVERY - cp3 HTTPS on cp3 is OK: HTTP OK: HTTP/2 200 - 2934 bytes in 0.620 second response time [18:23:48] RECOVERY - cp5 HTTPS on cp5 is OK: HTTP OK: HTTP/2 200 - 2956 bytes in 1.456 second response time [18:24:34] PROBLEM - mw1 MediaWiki Rendering on mw1 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.307 second response time [18:24:36] RECOVERY - mw1 HTTPS on mw1 is OK: HTTP OK: HTTP/2 200 - 357 bytes in 0.033 second response time [18:24:43] RECOVERY - cp2 HTTPS on cp2 is OK: HTTP OK: HTTP/2 200 - 2955 bytes in 0.251 second response time [18:28:17] !log test [18:31:16] !log [@test1] starting deploy of {'config': True, 'versions': ['']} to [test1] [18:31:17] !log [@test1] finished deploy of {'config': True, 'versions': ['']} to [test1] - SUCCESS in 0s [20:18:35] [02WikiTideOrg/landing] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+1/-0/±1] 13https://github.com/WikiTideOrg/landing/commit/e5ca75f83354 [20:18:37] [02WikiTideOrg/landing] 07Reception123 03e5ca75f - make landing page translatable per T71 [20:18:40] [02landing] 07Reception123 created branch 03Reception123-patch-1 - 13https://github.com/WikiTideOrg/landing [20:18:50] [02landing] 07Reception123 opened pull request 03#2: make landing page translatable per T71 - 13https://github.com/WikiTideOrg/landing/pull/2 [20:49:40] PROBLEM - wc.wikitide.org on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:51:43] PROBLEM - wc.wikitide.org on sslhost is WARNING: WARNING - Certificate 'wikitide.org' expires in 12 day(s) (Mon 04 Dec 2023 02:58:50 PM GMT +0000). [21:25:52] hm. all those all caps PROBLEMs don't sound good [22:06:10] [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/05ed7cedd6ff...9d64bcabb408 [22:06:12] [02WikiTideOrg/mw-config] 07Universal-Omega 039d64bca - Fix [22:07:08] WikiTideOrg/mw-config - Universal-Omega the build passed. [22:09:44] [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/9d64bcabb408...0bdb426579de [22:09:47] [02WikiTideOrg/mw-config] 07Universal-Omega 030bdb426 - Update comment [22:10:34] WikiTideOrg/mw-config - Universal-Omega the build passed. [22:14:30] [02mw-config] 07Universal-Omega commented on pull request 03#7: Update mediawiki/mediawiki-codesniffer requirement from 41.0.0 to 42.0.0 - 13https://github.com/WikiTideOrg/mw-config/pull/7#issuecomment-1821787308 [22:14:34] [02mw-config] 07dependabot[bot] edited pull request 03#7: Update mediawiki/mediawiki-codesniffer requirement from 41.0.0 to 42.0.0 - 13https://github.com/WikiTideOrg/mw-config/pull/7 [22:15:06] [02mw-config] 07dependabot[bot] edited pull request 03#7: Update mediawiki/mediawiki-codesniffer requirement from 41.0.0 to 42.0.0 - 13https://github.com/WikiTideOrg/mw-config/pull/7 [22:15:09] [02WikiTideOrg/mw-config] 07dependabot[bot] pushed 031 commit to 03dependabot/composer/mediawiki/mediawiki-codesniffer-42.0.0 [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/a1c86566659e...852096f3c9e3 [22:15:12] [02WikiTideOrg/mw-config] 07dependabot[bot] 03852096f - Update mediawiki/mediawiki-codesniffer requirement from 41.0.0 to 42.0.0 Updates the requirements on [mediawiki/mediawiki-codesniffer](https://github.com/wikimedia/mediawiki-tools-codesniffer) to permit the latest version. - [Changelog](https://github.com/wikimedia/mediawiki-tools-codesniffer/blob/master/HISTORY.md) - [Commits](https://github.com/wikimedia/mediawiki-tools-codesniffer/ [22:15:14] [02mw-config] 07dependabot[bot] synchronize pull request 03#7: Update mediawiki/mediawiki-codesniffer requirement from 41.0.0 to 42.0.0 - 13https://github.com/WikiTideOrg/mw-config/pull/7 [22:16:07] WikiTideOrg/mw-config - dependabot[bot] the build has errored. [22:18:01] !log [@jobrunner1] starting deploy of {'config': True, 'versions': ['']} to [mw1, mw2, jobrunner1] [22:18:05] !log [@jobrunner1] finished deploy of {'config': True, 'versions': ['']} to [mw1, mw2, jobrunner1] - SUCCESS in 3s [22:30:32] [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03remove-checks [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/commit/6178b5745d74 [22:30:35] [02WikiTideOrg/mw-config] 07Universal-Omega 036178b57 - Remove unnecessary checks [22:30:38] [02mw-config] 07Universal-Omega created branch 03remove-checks - 13https://github.com/WikiTideOrg/mw-config [22:30:40] [02mw-config] 07Universal-Omega opened pull request 03#11: Remove unnecessary checks - 13https://github.com/WikiTideOrg/mw-config/pull/11 [22:31:28] !log [@test1] starting deploy of {'config': True, 'versions': ['']} to [test1] [22:31:29] !log [@test1] finished deploy of {'config': True, 'versions': ['']} to [test1] - SUCCESS in 0s [22:31:37] WikiTideOrg/mw-config - Universal-Omega the build passed. [22:32:33] !log [agent@swiftobject1] Script: Finished S3 download of all folders [22:32:59] @agentisai ^ [22:38:55] Thanks for the ping [22:40:12] Np [23:02:52] !log [agent@swiftobject1] Script: Starting upload [23:06:54] !log [agent@swiftobject1] Script: Starting upload [23:11:41] !log [agent@swiftobject1] Script: Starting upload [23:13:30] [02WikiTideOrg/mw-config] 07AgentIsai pushed 031 commit to 03AgentIsai-patch-1 [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/commit/2b04f90811eb [23:13:32] [02WikiTideOrg/mw-config] 07AgentIsai 032b04f90 - T37: Enable FileBackend for Swift [23:13:35] [02mw-config] 07AgentIsai created branch 03AgentIsai-patch-1 - 13https://github.com/WikiTideOrg/mw-config [23:13:41] [02mw-config] 07AgentIsai opened pull request 03#12: T37: Migrate to Swift - 13https://github.com/WikiTideOrg/mw-config/pull/12 [23:13:57] [02WikiTideOrg/mw-config] 07AgentIsai pushed 031 commit to 03AgentIsai-patch-1 [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/2b04f90811eb...5308d5f3de9b [23:14:00] [02WikiTideOrg/mw-config] 07AgentIsai 035308d5f - Don't load AWS anymore [23:14:03] [02mw-config] 07AgentIsai synchronize pull request 03#12: T37: Migrate to Swift - 13https://github.com/WikiTideOrg/mw-config/pull/12 [23:14:32] WikiTideOrg/mw-config - AgentIsai the build passed. [23:14:55] WikiTideOrg/mw-config - AgentIsai the build passed. [23:14:58] PROBLEM - swiftobject1 Current Load on swiftobject1 is CRITICAL: LOAD CRITICAL - total load average: 13.58, 5.39, 2.28 [23:15:15] oh nice [23:15:16] it'll survive [23:17:45] [02WikiTideOrg/mw-config] 07AgentIsai pushed 031 commit to 03AgentIsai-patch-1 [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/5308d5f3de9b...4daeaed7bab1 [23:17:46] [02WikiTideOrg/mw-config] 07AgentIsai 034daeaed - Delete S3 specific GlobalSettings values [23:17:47] [02mw-config] 07AgentIsai synchronize pull request 03#12: T37: Migrate to Swift - 13https://github.com/WikiTideOrg/mw-config/pull/12 [23:18:46] WikiTideOrg/mw-config - AgentIsai the build passed. [23:57:09] !log [agent@swiftobject1] Script: Finished uploading images to Swift [23:57:33] @agentisai that was quick? [23:58:12] Seemed too quick