[00:00:00] !log [www-data@mwtask181] Began automatic backing up [00:00:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261001000001-1d064f0b [00:01:26] [02statichelp] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/statichelp/commit/c6c3683707bf1052edef6adb57d5da9d903caf85 [00:01:26] 02statichelp/03main 07WikiTideBot 03c6c3683 Bot: Auto-update Tech namespace pages 2026-10-01 00:01:23 [00:06:35] PROBLEM - matomo151 HTTPS on matomo151 is CRITICAL: HTTP CRITICAL: HTTP/2 502 - 176 bytes in 0.009 second response time [00:06:35] PROBLEM - db182 Disk Space on db182 is WARNING: DISK WARNING - free space: / 30342MiB (5.1% inode=100%); [00:08:35] PROBLEM - db182 Disk Space on db182 is CRITICAL: DISK CRITICAL - free space: / 24536MiB (4.1% inode=100%); [00:08:37] RECOVERY - matomo151 HTTPS on matomo151 is OK: HTTP OK: HTTP/2 200 - 553 bytes in 2.471 second response time [00:10:35] PROBLEM - db182 Disk Space on db182 is WARNING: DISK WARNING - free space: / 30151MiB (5.1% inode=100%); [00:13:27] PROBLEM - cp201 Disk Space on cp201 is WARNING: DISK WARNING - free space: / 44528MiB (9.8% inode=100%); [00:14:35] PROBLEM - db182 Disk Space on db182 is CRITICAL: DISK CRITICAL - free space: / 25339MiB (4.3% inode=100%); [00:15:27] RECOVERY - cp201 Disk Space on cp201 is OK: DISK OK - free space: / 86541MiB (19.1% inode=100%); [00:18:35] PROBLEM - db182 Disk Space on db182 is WARNING: DISK WARNING - free space: / 30221MiB (5.1% inode=100%); [00:20:35] PROBLEM - db182 Disk Space on db182 is CRITICAL: DISK CRITICAL - free space: / 24355MiB (4.1% inode=100%); [00:21:17] [02puppet] 07Universal-Omega pushed 1 new commit to 03taskbot 13https://github.com/miraheze/puppet/commit/8356cd0ed9e0d5592bdf44e59a72fea916e5ecb5 [00:21:17] 02puppet/03taskbot 07CosmicAlpha 038356cd0 Improvements/fixes [00:22:35] PROBLEM - db182 Disk Space on db182 is WARNING: DISK WARNING - free space: / 31876MiB (5.4% inode=100%); [00:22:38] [02puppet] 07Universal-Omega pushed 1 new commit to 03taskbot 13https://github.com/miraheze/puppet/commit/7937d0ae78086cb8bd88d1b40980f784a501960f [00:22:38] 02puppet/03taskbot 07CosmicAlpha 037937d0a Tests [00:23:19] miraheze/puppet - Universal-Omega the build has errored. [00:23:41] PROBLEM - cp171 Disk Space on cp171 is WARNING: DISK WARNING - free space: / 44341MiB (9.8% inode=100%); [00:23:54] [02puppet] 07Universal-Omega pushed 1 new commit to 03taskbot 13https://github.com/miraheze/puppet/commit/472509232e483cf5eb944675094f78a06f9a963c [00:23:54] 02puppet/03taskbot 07CosmicAlpha 034725092 Update [00:24:13] RECOVERY - bots171 Icinga Task Bot on bots171 is OK: PROCS OK: 1 process with args 'taskbot.py' [00:25:39] RECOVERY - cp171 Disk Space on cp171 is OK: DISK OK - free space: / 86507MiB (19.1% inode=100%); [00:25:50] miraheze/puppet - Universal-Omega the build passed. [00:26:35] PROBLEM - db182 Disk Space on db182 is CRITICAL: DISK CRITICAL - free space: / 28658MiB (4.8% inode=100%); [00:27:37] PROBLEM - bots171 Taskbot Test on bots171 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'sleep', args '31337' [00:28:08] !log [petramagna@mwtask171] sudo -u www-data php /srv/mediawiki/1.46/maintenance/run.php CirrusSearch:ForceSearchIndex --wiki=vocaloidlyricswiki (END - exit=0) (T16224) [00:28:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261001002809-4124b1e2 [00:28:35] PROBLEM - db182 Disk Space on db182 is WARNING: DISK WARNING - free space: / 35749MiB (6.0% inode=100%); [00:29:02] PROBLEM - bots171 Puppet on bots171 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 3 minutes ago with 0 failures [00:29:37] PROBLEM - bots171 Taskbot Test on bots171 is WARNING: PROCS WARNING: 1 process with command name 'sleep', args '31337' [00:30:02] PROBLEM - os201 Disk Space on os201 is WARNING: DISK WARNING - free space: / 44352MiB (10.0% inode=100%); [00:30:42] RECOVERY - bots171 Taskbot Test on bots171 is OK: PROCS OK: 3 processes with command name 'sleep', args '31337' [00:31:27] RECOVERY - speedscope211 Check unit status of speedscope_daily_aggregation on speedscope211 is OK: OK: Status of the systemd unit speedscope_daily_aggregation [00:32:13] PROBLEM - bots171 Icinga Task Bot on bots171 is CRITICAL: PROCS CRITICAL: 3 processes with args 'taskbot.py' [00:42:13] PROBLEM - os151 Disk Space on os151 is WARNING: DISK WARNING - free space: / 44171MiB (9.9% inode=100%); [00:45:27] PROBLEM - speedscope211 Check unit status of speedscope_daily_aggregation on speedscope211 is CRITICAL: CRITICAL: Status of the systemd unit speedscope_daily_aggregation [00:50:13] RECOVERY - os151 Disk Space on os151 is OK: DISK OK - free space: / 46622MiB (10.4% inode=100%); [00:54:02] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/2329ae7da6809b0b3543b53724068d3d7c99c20a [00:54:02] 02puppet/03main 07CosmicAlpha 032329ae7 Update icinga user for Jenny [00:58:57] [02puppet] 07Universal-Omega created 03rename-shell (+1 new commit) 13https://github.com/miraheze/puppet/commit/dd7392a5c415 [00:58:57] 02puppet/03rename-shell 07CosmicAlpha 03dd7392a Rename jenny shell user [00:59:03] [02puppet] 07Universal-Omega opened pull request #5064: Rename jenny shell user (03main...03rename-shell) 13https://github.com/miraheze/puppet/pull/5064 [01:01:44] PROBLEM - puppet181 Check unit status of git_pull_puppet on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit git_pull_puppet [01:02:02] RECOVERY - os201 Disk Space on os201 is OK: DISK OK - free space: / 44699MiB (10.1% inode=100%); [01:03:19] [02puppet] 07Universal-Omega pushed 1 new commit to 03taskbot 13https://github.com/miraheze/puppet/commit/deca4deef2c87dfd1e5ea33f79f2b15b691e670a [01:03:19] 02puppet/03taskbot 07CosmicAlpha 03deca4de Merge branch 'main' into taskbot [01:09:30] RECOVERY - puppet181 Check unit status of git_pull_puppet on puppet181 is OK: OK: Status of the systemd unit git_pull_puppet [01:15:21] PROBLEM - os161 Disk Space on os161 is WARNING: DISK WARNING - free space: / 44551MiB (10.0% inode=100%); [01:27:12] RECOVERY - os161 Disk Space on os161 is OK: DISK OK - free space: / 47201MiB (10.6% inode=100%); [01:30:55] [02puppet] 07Universal-Omega pushed 1 new commit to 03taskbot 13https://github.com/miraheze/puppet/commit/54f4c1432ddc98463f2d1345605f8e676d98b240 [01:30:56] 02puppet/03taskbot 07CosmicAlpha 0354f4c14 Cleanup [01:33:04] [02puppet] 07Universal-Omega pushed 1 new commit to 03taskbot 13https://github.com/miraheze/puppet/commit/dc50798568cd4e5c395ddd8428be0c6085de0506 [01:33:04] 02puppet/03taskbot 07CosmicAlpha 03dc50798 Update [01:40:20] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/8f0a576e293e0bd5c79b577c043ef6317b206ae6 [01:40:20] 02puppet/03main 07CosmicAlpha 038f0a576 Align [01:41:12] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/8c32a3d77f3711081ffab5fbd38d1b537ffcbbc3 [01:41:12] 02puppet/03main 07CosmicAlpha 038c32a3d Align [01:41:20] PROBLEM - phorge171 Disk Space on phorge171 is WARNING: DISK WARNING - free space: / 4954MiB (6.7% inode=95%); [01:42:24] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/bfc4ddd57694609007566ddc63e166621e939d69 [01:42:24] 02puppet/03main 07CosmicAlpha 03bfc4ddd Consistent [01:43:20] PROBLEM - phorge171 Disk Space on phorge171 is CRITICAL: DISK CRITICAL - free space: / 3156MiB (4.3% inode=95%); [01:44:19] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/8ebbefed57122c71374a7d76bb726af441da37b9 [01:44:19] 02puppet/03main 07CosmicAlpha 038ebbefe Consistency [01:46:32] PROBLEM - puppet181 Puppet on puppet181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_puppet] [01:46:46] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/8a25fc21b0b0f8cd88ccc456747fb8508c1a6167 [01:46:46] 02puppet/03main 07CosmicAlpha 038a25fc2 Align [01:47:49] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/de5904fe849412e32b5d5526c96dbc1bf67e5121 [01:47:50] 02puppet/03main 07CosmicAlpha 03de5904f Align + order [01:52:56] PROBLEM - puppet181 Check unit status of git_pull_puppet on puppet181 is CRITICAL: CRITICAL: Status of the systemd unit git_pull_puppet [01:54:19] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/77008a68de11f7b82deb5c5eb740c5290851f092 [01:54:19] 02puppet/03main 07CosmicAlpha 0377008a6 Use define [01:56:14] [02puppet] 07Universal-Omega pushed 1 new commit to 03taskbot 13https://github.com/miraheze/puppet/commit/92e5928bde828023e0b62df630225df1d4bac3ea [01:56:14] 02puppet/03taskbot 07CosmicAlpha 0392e5928 Merge branch 'main' into taskbot [01:58:03] miraheze/puppet - Universal-Omega the build passed. [01:59:11] PROBLEM - os161 Disk Space on os161 is WARNING: DISK WARNING - free space: / 43335MiB (9.7% inode=100%); [02:01:20] RECOVERY - phorge171 Disk Space on phorge171 is OK: DISK OK - free space: / 52781MiB (71.7% inode=95%); [02:04:37] [02puppet] 07Universal-Omega pushed 1 new commit to 03taskbot 13https://github.com/miraheze/puppet/commit/93b6207f1693935155086629f975ef80141bc6d1 [02:04:37] 02puppet/03taskbot 07CosmicAlpha 0393b6207 Update [02:05:42] [02puppet] 07Universal-Omega pushed 1 new commit to 03taskbot 13https://github.com/miraheze/puppet/commit/a7e8cbc404dc8d64892b5a7dede1ef8dab9117b4 [02:05:42] 02puppet/03taskbot 07CosmicAlpha 03a7e8cbc Update [02:06:21] miraheze/puppet - Universal-Omega the build passed. [02:07:35] miraheze/puppet - Universal-Omega the build passed. [02:08:53] [02puppet] 07Universal-Omega pushed 1 new commit to 03taskbot 13https://github.com/miraheze/puppet/commit/9596c7977230abe79ced58fb2dc4dae3a67abb58 [02:08:53] 02puppet/03taskbot 07CosmicAlpha 039596c79 + [02:10:35] miraheze/puppet - Universal-Omega the build passed. [02:39:00] PROBLEM - os161 Current Load on os161 is CRITICAL: LOAD CRITICAL - total load average: 4.32, 3.52, 3.10 [02:42:55] PROBLEM - os161 Current Load on os161 is WARNING: LOAD WARNING - total load average: 3.14, 3.47, 3.19 [02:48:55] RECOVERY - os161 Current Load on os161 is OK: LOAD OK - total load average: 2.06, 3.05, 3.15 [02:52:59] [02puppet] 07Universal-Omega pushed 1 new commit to 03taskbot 13https://github.com/miraheze/puppet/commit/5a3145a1cb1af9f0b11f92e499d220f2afaacbd6 [02:52:59] 02puppet/03taskbot 07CosmicAlpha 035a3145a Add change grace minutes [02:53:52] PROBLEM - os191 Disk Space on os191 is WARNING: DISK WARNING - free space: / 42889MiB (9.6% inode=100%); [02:55:36] [02puppet] 07Universal-Omega pushed 1 new commit to 03taskbot 13https://github.com/miraheze/puppet/commit/a97a5c1e50fe55287c4fd2cf44e975a15110c6ff [02:55:36] 02puppet/03taskbot 07CosmicAlpha 03a97a5c1 Update [02:55:52] RECOVERY - os191 Disk Space on os191 is OK: DISK OK - free space: / 47300MiB (10.6% inode=100%); [02:56:16] [02puppet] 07Universal-Omega pushed 1 new commit to 03taskbot 13https://github.com/miraheze/puppet/commit/981bc1fe51012bf04ae44846262b47bc595ccf2f [02:56:16] 02puppet/03taskbot 07CosmicAlpha 03981bc1f Update [02:57:16] [02puppet] 07Universal-Omega pushed 1 new commit to 03taskbot 13https://github.com/miraheze/puppet/commit/46dcefb76e060b30a8fe886b3cc686b85ede8727 [02:57:16] 02puppet/03taskbot 07CosmicAlpha 0346dcefb Update [02:57:46] [02puppet] 07Universal-Omega pushed 1 new commit to 03taskbot 13https://github.com/miraheze/puppet/commit/1d194f97d4a098639e3f27dae90ac77dbc823f6f [02:57:47] 02puppet/03taskbot 07CosmicAlpha 031d194f9 Update [02:58:29] [02puppet] 07Universal-Omega pushed 1 new commit to 03taskbot 13https://github.com/miraheze/puppet/commit/fbfb74fda08fafb797255d5b1594dde24fde2d94 [02:58:29] 02puppet/03taskbot 07CosmicAlpha 03fbfb74f Update [02:59:49] [02puppet] 07Universal-Omega pushed 1 new commit to 03taskbot 13https://github.com/miraheze/puppet/commit/bd2222d3692bb488385ab0e405d9ea1fa3444c8c [02:59:49] 02puppet/03taskbot 07CosmicAlpha 03bd2222d Update [03:02:50] [02puppet] 07Universal-Omega pushed 1 new commit to 03taskbot 13https://github.com/miraheze/puppet/commit/5d9bf994bc3ed37df6f4c5b17a3d67de1458dae6 [03:02:50] 02puppet/03taskbot 07CosmicAlpha 035d9bf99 - [03:05:14] [02puppet] 07Universal-Omega pushed 1 new commit to 03taskbot 13https://github.com/miraheze/puppet/commit/b3f5e47467d72ebe092989b9adba0330d7fec83e [03:05:14] 02puppet/03taskbot 07CosmicAlpha 03b3f5e47 - [03:06:15] RECOVERY - bots171 Puppet on bots171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [03:08:13] RECOVERY - bots171 Icinga Task Bot on bots171 is OK: PROCS OK: 1 process with args 'taskbot.py' [03:10:56] RECOVERY - puppet181 Check unit status of git_pull_puppet on puppet181 is OK: OK: Status of the systemd unit git_pull_puppet [03:12:49] RECOVERY - puppet181 Puppet on puppet181 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [03:13:40] [02puppet] 07Universal-Omega pushed 1 new commit to 03taskbot 13https://github.com/miraheze/puppet/commit/c7ed3d698c30c20254e27331503cbe127a9de9b2 [03:13:40] 02puppet/03taskbot 07CosmicAlpha 03c7ed3d6 Alignment/order [03:30:21] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/80df28876b1151eba20193254a4cc8d6464d7817 [03:30:21] 02ssl/03main 07WikiTideBot 0380df288 Bot: Auto-update domain lists [03:33:11] [02puppet] 07Universal-Omega pushed 1 new commit to 03taskbot 13https://github.com/miraheze/puppet/commit/67feee566c648d91cbb8930f2634c55025c861ce [03:33:11] 02puppet/03taskbot 07CosmicAlpha 0367feee5 Safer handling [03:33:51] PROBLEM - os201 Disk Space on os201 is WARNING: DISK WARNING - free space: / 43757MiB (9.8% inode=100%); [03:34:12] [02puppet] 07Universal-Omega pushed 1 new commit to 03taskbot 13https://github.com/miraheze/puppet/commit/0f4841a2ea5ef83c879683608cfbc9d525ed3553 [03:34:12] 02puppet/03taskbot 07CosmicAlpha 030f4841a Add [03:34:58] [02puppet] 07Universal-Omega pushed 1 new commit to 03taskbot 13https://github.com/miraheze/puppet/commit/907115009551a037f338737b8f3668f067c04b4b [03:34:59] 02puppet/03taskbot 07CosmicAlpha 039071150 Update [03:37:30] [02puppet] 07Universal-Omega pushed 1 new commit to 03taskbot 13https://github.com/miraheze/puppet/commit/e5d01baab05229c5aa25fc2dd1f7ce1af68146ea [03:37:30] 02puppet/03taskbot 07CosmicAlpha 03e5d01ba - [03:46:45] RECOVERY - os161 Disk Space on os161 is OK: DISK OK - free space: / 45547MiB (10.2% inode=100%); [03:47:35] RECOVERY - os201 Disk Space on os201 is OK: DISK OK - free space: / 47361MiB (10.6% inode=100%); [03:53:50] [02puppet] 07Universal-Omega pushed 1 new commit to 03taskbot 13https://github.com/miraheze/puppet/commit/f0f4808fbbe1f0ad5a3bbe61389da2eec69a431c [03:53:50] 02puppet/03taskbot 07CosmicAlpha 03f0f4808 Update [04:01:47] [02puppet] 07Universal-Omega pushed 1 new commit to 03taskbot 13https://github.com/miraheze/puppet/commit/b4e4247706303fbc74258ee1c67c73e24c58e615 [04:01:47] 02puppet/03taskbot 07CosmicAlpha 03b4e4247 Better downgrade handle [04:03:41] [02puppet] 07Universal-Omega pushed 1 new commit to 03taskbot 13https://github.com/miraheze/puppet/commit/e6fd344a3cd91c8c998a7057dd32c0bcba2e0e61 [04:03:41] 02puppet/03taskbot 07CosmicAlpha 03e6fd344 better tests [04:10:15] [02puppet] 07Universal-Omega merged pull request #5062: monitoring: add taskbot (03main...03taskbot) 13https://github.com/miraheze/puppet/pull/5062 [04:10:15] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/319ae0b93cc7559d00ce0828a430fdcfa4f88b4a [04:10:15] 02puppet/03main 07CosmicAlpha 03319ae0b monitoring: add taskbot… [04:10:17] [02puppet] 07Universal-Omega 04deleted 03taskbot at 03e6fd344 13https://api.github.com/repos/miraheze/puppet/commit/e6fd344 [04:22:45] PROBLEM - os201 Disk Space on os201 is WARNING: DISK WARNING - free space: / 43496MiB (9.8% inode=100%); [04:32:38] PROBLEM - os191 Current Load on os191 is CRITICAL: LOAD CRITICAL - total load average: 4.06, 2.89, 1.84 [04:34:51] PROBLEM - os151 Current Load on os151 is WARNING: LOAD WARNING - total load average: 3.67, 2.96, 1.97 [04:36:06] PROBLEM - os191 Disk Space on os191 is WARNING: DISK WARNING - free space: / 43139MiB (9.7% inode=100%); [04:36:13] [02phorge-extensions] 07Universal-Omega created 03github-link (+1 new commit) 13https://github.com/miraheze/phorge-extensions/commit/3547f98ee186 [04:36:13] 02phorge-extensions/03github-link 07CosmicAlpha 033547f98 Add easier GitHub navigation [04:36:18] [02phorge-extensions] 07Universal-Omega opened pull request #84: Add easier GitHub navigation (03main...03github-link) 13https://github.com/miraheze/phorge-extensions/pull/84 [04:36:51] RECOVERY - os151 Current Load on os151 is OK: LOAD OK - total load average: 3.32, 3.08, 2.14 [04:37:02] miraheze/phorge-extensions - Universal-Omega the build passed. [04:37:08] [02phorge-extensions] 07Universal-Omega pushed 1 new commit to 03github-link 13https://github.com/miraheze/phorge-extensions/commit/6f8615f2e11f0b545ba97d7afe73fa6b7ffdf1d7 [04:37:09] 02phorge-extensions/03github-link 07CosmicAlpha 036f8615f Update [04:37:50] miraheze/phorge-extensions - Universal-Omega the build passed. [04:38:12] [02phorge-extensions] 07Universal-Omega pushed 1 new commit to 03github-link 13https://github.com/miraheze/phorge-extensions/commit/766c544ddd172d920be5c96f02c9a909b0653b89 [04:38:13] 02phorge-extensions/03github-link 07CosmicAlpha 03766c544 Update [04:38:36] PROBLEM - os191 Current Load on os191 is WARNING: LOAD WARNING - total load average: 3.99, 3.92, 2.67 [04:39:00] miraheze/phorge-extensions - Universal-Omega the build passed. [04:39:58] RECOVERY - os191 Disk Space on os191 is OK: DISK OK - free space: / 47406MiB (10.7% inode=100%); [04:42:34] RECOVERY - os191 Current Load on os191 is OK: LOAD OK - total load average: 2.45, 3.13, 2.63 [04:45:49] RECOVERY - os201 Disk Space on os201 is OK: DISK OK - free space: / 45862MiB (10.3% inode=100%); [05:01:09] RECOVERY - mwtask181 Check unit status of mediawiki_job_update-special-pages on mwtask181 is OK: OK: Status of the systemd unit mediawiki_job_update-special-pages [05:02:03] !log [universalomega@test151] Starting deploy of {'versions': 'all', 'upgrade_extensions': 'TabberNeue'} to test151 [05:02:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261001050204-0b7e800c [05:02:14] !log [universalomega@test151] finished deploy of {'versions': 'all', 'upgrade_extensions': 'TabberNeue'} to test151 - SUCCESS in 10s [05:02:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261001050215-4fc8f4db [05:03:27] !log [universalomega@mwtask181] Starting deploy of {'versions': 'all', 'upgrade_extensions': 'TabberNeue'} to all (T16227) [05:03:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261001050328-a5bcac85 [05:04:08] !log [universalomega@mwtask181] finished deploy of {'versions': 'all', 'upgrade_extensions': 'TabberNeue'} to all - SUCCESS in 41s (T16227) [05:04:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261001050409-91025c7d [05:07:02] [02mediawiki-repos] 07Universal-Omega created 03T16227 (+1 new commit) 13https://github.com/miraheze/mediawiki-repos/commit/de8426347f01 [05:07:02] 02mediawiki-repos/03T16227 07CosmicAlpha 03de84263 Update TabberNeue to 4.1.0… [05:07:08] [02mediawiki-repos] 07Universal-Omega opened pull request #171: Update TabberNeue to 4.1.0 (03main...03T16227) 13https://github.com/miraheze/mediawiki-repos/pull/171 [05:14:55] PROBLEM - os191 Disk Space on os191 is WARNING: DISK WARNING - free space: / 44347MiB (10.0% inode=100%); [05:18:31] [02phorge-extensions] 07Universal-Omega merged pull request #84: Add easier GitHub navigation (03main...03github-link) 13https://github.com/miraheze/phorge-extensions/pull/84 [05:18:31] [02phorge-extensions] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/phorge-extensions/commit/b1d340c2e0aaaa840406199ed6d198c429d12ba1 [05:18:31] 02phorge-extensions/03main 07CosmicAlpha 03b1d340c Add easier GitHub navigation [05:18:33] [02phorge-extensions] 07Universal-Omega 04deleted 03github-link at 03766c544 13https://api.github.com/repos/miraheze/phorge-extensions/commit/766c544 [05:19:11] miraheze/phorge-extensions - Universal-Omega the build passed. [05:36:36] PROBLEM - os151 Disk Space on os151 is WARNING: DISK WARNING - free space: / 43753MiB (9.8% inode=100%); [05:56:21] RECOVERY - os151 Disk Space on os151 is OK: DISK OK - free space: / 46515MiB (10.4% inode=100%); [06:00:11] !log [petramagna@test151] Starting deploy of {'folders': '1.46/extensions/VisualEditor'} to test151 (T16228) [06:00:12] !log [petramagna@test151] finished deploy of {'folders': '1.46/extensions/VisualEditor'} to test151 - SUCCESS in 0s (T16228) [06:00:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261001060012-5bf2d4bc [06:00:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261001060017-e82def37 [06:10:54] !log [petramagna@mwtask181] Starting deploy of {'versions': 'all', 'upgrade_extensions': 'VisualEditor', 'debug': True} to all (T16228) [06:10:57] !log [petramagna@mwtask181] finished deploy of {'versions': 'all', 'upgrade_extensions': 'VisualEditor', 'debug': True} to all - SUCCESS in 3s (T16228) [06:10:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261001061055-2dc6ff85 [06:11:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261001061059-fd4698af [06:11:31] !log [petramagna@mwtask181] Starting deploy of {'folders': '1.46/extensions/VisualEditor', 'debug': True} to all (T16228) [06:11:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261001061132-1d4c5710 [06:12:00] !log [petramagna@mwtask181] finished deploy of {'folders': '1.46/extensions/VisualEditor', 'debug': True} to all - SUCCESS in 28s (T16228) [06:12:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261001061202-5c065983 [06:14:06] PROBLEM - os151 Disk Space on os151 is WARNING: DISK WARNING - free space: / 43339MiB (9.7% inode=100%); [06:16:04] RECOVERY - os151 Disk Space on os151 is OK: DISK OK - free space: / 46452MiB (10.4% inode=100%); [06:20:00] PROBLEM - os151 Disk Space on os151 is WARNING: DISK WARNING - free space: / 43310MiB (9.7% inode=100%); [06:21:58] RECOVERY - os151 Disk Space on os151 is OK: DISK OK - free space: / 47625MiB (10.7% inode=100%); [06:22:32] [02mw-config] 07lihaohong6 opened pull request #6572: T15351: add note for RandomSelection about uncached selection (07miraheze:03main...07lihaohong6:03selection) 13https://github.com/miraheze/mw-config/pull/6572 [06:23:37] miraheze/mw-config - lihaohong6 the build passed. [06:30:21] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/666919f0dd57047f34689947e061f43a54ba06ac [06:30:21] 02ssl/03main 07WikiTideBot 03666919f Bot: Auto-update domain lists [06:33:14] RECOVERY - os191 Disk Space on os191 is OK: DISK OK - free space: / 45005MiB (10.1% inode=100%); [06:37:39] [02mw-config] 07lihaohong6 merged pull request #6572: T15351: add note for RandomSelection about uncached selection (07miraheze:03main...07lihaohong6:03selection) 13https://github.com/miraheze/mw-config/pull/6572 [06:37:41] [02mw-config] 07lihaohong6 pushed 1 new commit to 03main 13https://github.com/miraheze/mw-config/commit/2cf25ab96c10fe250513617cc54048218607022a [06:37:42] 02mw-config/03main 07Peter Li 032cf25ab T15351: add note for RandomSelection about uncached selection (#6572) [06:37:51] !log [petramagna@test151] Starting deploy of {'pull': 'config', 'config': True} to test151 [06:37:52] !log [petramagna@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 1s [06:37:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261001063751-862295be [06:37:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261001063756-bc2c8669 [06:38:53] miraheze/mw-config - lihaohong6 the build passed. [06:43:20] !log [petramagna@mwtask181] Starting deploy of {'pull': 'config', 'config': True} to all (T15351) [06:43:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261001064321-d36f2844 [06:43:46] !log [petramagna@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 25s (T15351) [06:43:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261001064347-66be1019 [07:06:06] PROBLEM - os161 Disk Space on os161 is WARNING: DISK WARNING - free space: / 43618MiB (9.8% inode=100%); [07:06:40] PROBLEM - os201 Disk Space on os201 is WARNING: DISK WARNING - free space: / 43389MiB (9.8% inode=100%); [07:16:34] PROBLEM - cp171 Varnish Backends on cp171 is CRITICAL: 2 backends are down. mw153 mw201 [07:22:45] PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 23.57, 20.08, 14.47 [07:24:45] RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 20.24, 20.22, 15.23 [07:28:34] RECOVERY - cp171 Varnish Backends on cp171 is OK: All 34 backends are healthy [07:28:45] PROBLEM - mw183 Current Load on mw183 is CRITICAL: LOAD CRITICAL - total load average: 26.09, 22.75, 17.36 [07:30:34] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 20.63, 19.89, 15.86 [07:30:44] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.58, 21.20, 16.16 [07:30:51] PROBLEM - mw163 Current Load on mw163 is WARNING: LOAD WARNING - total load average: 22.83, 21.07, 16.34 [07:32:34] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 17.64, 19.41, 16.20 [07:32:40] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.45, 21.21, 16.79 [07:32:49] PROBLEM - mw163 Current Load on mw163 is CRITICAL: LOAD CRITICAL - total load average: 24.45, 22.06, 17.24 [07:34:36] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 18.51, 20.34, 17.00 [07:34:46] PROBLEM - mw163 Current Load on mw163 is WARNING: LOAD WARNING - total load average: 20.18, 20.81, 17.34 [07:36:40] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.17, 20.31, 15.70 [07:38:33] PROBLEM - mw202 Current Load on mw202 is WARNING: LOAD WARNING - total load average: 23.74, 21.04, 17.41 [07:38:34] PROBLEM - mw203 Current Load on mw203 is WARNING: LOAD WARNING - total load average: 22.91, 21.18, 17.95 [07:38:40] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 19.18, 20.18, 16.24 [07:39:27] PROBLEM - mw193 Current Load on mw193 is WARNING: LOAD WARNING - total load average: 22.14, 21.34, 18.34 [07:40:45] PROBLEM - mw183 Current Load on mw183 is WARNING: LOAD WARNING - total load average: 18.09, 23.25, 20.88 [07:41:22] RECOVERY - mw193 Current Load on mw193 is OK: LOAD OK - total load average: 13.19, 18.51, 17.68 [07:41:50] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?from=1790836880000&orgId=1&to=1790840510031 [07:42:32] RECOVERY - mw202 Current Load on mw202 is OK: LOAD OK - total load average: 10.80, 17.14, 16.80 [07:42:34] RECOVERY - mw203 Current Load on mw203 is OK: LOAD OK - total load average: 13.96, 18.04, 17.48 [07:42:36] RECOVERY - mw163 Current Load on mw163 is OK: LOAD OK - total load average: 9.76, 16.93, 17.46 [07:42:45] RECOVERY - mw183 Current Load on mw183 is OK: LOAD OK - total load average: 12.71, 19.63, 19.85 [07:56:50] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?from=1790837510000&orgId=1&to=1790841110000 [08:05:38] PROBLEM - os151 Current Load on os151 is CRITICAL: LOAD CRITICAL - total load average: 4.16, 3.55, 2.24 [08:07:35] PROBLEM - os151 Current Load on os151 is WARNING: LOAD WARNING - total load average: 4.00, 3.59, 2.41 [08:08:30] PROBLEM - os151 Disk Space on os151 is WARNING: DISK WARNING - free space: / 43980MiB (9.8% inode=100%); [08:19:19] PROBLEM - os151 Current Load on os151 is CRITICAL: LOAD CRITICAL - total load average: 4.76, 4.17, 3.34 [08:27:09] PROBLEM - os151 Current Load on os151 is WARNING: LOAD WARNING - total load average: 3.58, 3.83, 3.53 [08:33:01] PROBLEM - os151 Current Load on os151 is CRITICAL: LOAD CRITICAL - total load average: 4.60, 3.87, 3.58 [08:36:09] RECOVERY - os151 Disk Space on os151 is OK: DISK OK - free space: / 44950MiB (10.1% inode=100%); [08:36:55] PROBLEM - os151 Current Load on os151 is WARNING: LOAD WARNING - total load average: 3.45, 3.75, 3.62 [08:37:37] RECOVERY - os161 Disk Space on os161 is OK: DISK OK - free space: / 46770MiB (10.5% inode=100%); [08:37:49] RECOVERY - os201 Disk Space on os201 is OK: DISK OK - free space: / 47104MiB (10.6% inode=100%); [08:40:04] PROBLEM - os151 Disk Space on os151 is WARNING: DISK WARNING - free space: / 44390MiB (9.9% inode=100%); [08:40:51] PROBLEM - os151 Current Load on os151 is CRITICAL: LOAD CRITICAL - total load average: 4.04, 3.77, 3.66 [08:44:51] PROBLEM - os151 Current Load on os151 is WARNING: LOAD WARNING - total load average: 3.78, 3.86, 3.74 [08:46:51] PROBLEM - os151 Current Load on os151 is CRITICAL: LOAD CRITICAL - total load average: 4.77, 4.19, 3.87 [08:48:51] PROBLEM - os151 Current Load on os151 is WARNING: LOAD WARNING - total load average: 3.61, 3.89, 3.80 [08:52:51] PROBLEM - os151 Current Load on os151 is CRITICAL: LOAD CRITICAL - total load average: 4.11, 3.92, 3.83 [08:54:51] PROBLEM - os151 Current Load on os151 is WARNING: LOAD WARNING - total load average: 3.59, 3.78, 3.79 [08:56:51] PROBLEM - os151 Current Load on os151 is CRITICAL: LOAD CRITICAL - total load average: 4.86, 4.18, 3.93 [09:24:51] PROBLEM - os151 Current Load on os151 is WARNING: LOAD WARNING - total load average: 2.87, 3.52, 3.90 [09:32:51] PROBLEM - os151 Current Load on os151 is CRITICAL: LOAD CRITICAL - total load average: 4.37, 3.96, 3.94 [10:02:51] PROBLEM - os151 Current Load on os151 is WARNING: LOAD WARNING - total load average: 3.40, 3.67, 4.00 [10:04:51] PROBLEM - os151 Current Load on os151 is CRITICAL: LOAD CRITICAL - total load average: 4.01, 3.85, 4.02 [10:05:58] PROBLEM - db181 mariadb_memory on db181 is WARNING: WARN Memory 94% used. Largest process: mariadbd (1614) = 92.0% [10:12:51] PROBLEM - os151 Current Load on os151 is WARNING: LOAD WARNING - total load average: 3.49, 3.85, 3.99 [10:15:56] [02puppet] 07Universal-Omega created 03revert-4970-mariadb-memory-monitoring (+1 new commit) 13https://github.com/miraheze/puppet/commit/02562a664e8c [10:15:56] 02puppet/03revert-4970-mariadb-memory-monitoring 07CosmicAlpha 0302562a6 Revert "mariadb: add monitoring for memory"… [10:16:04] [02puppet] 07Universal-Omega opened pull request #5065: Revert "mariadb: add monitoring for memory" (03main...03revert-4970-mariadb-memory-monitoring) 13https://github.com/miraheze/puppet/pull/5065 [10:16:38] [02puppet] 07Universal-Omega merged pull request #5065: Revert "mariadb: add monitoring for memory" (03main...03revert-4970-mariadb-memory-monitoring) 13https://github.com/miraheze/puppet/pull/5065 [10:16:40] [02puppet] 07Universal-Omega pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/e63825855a4fc8c78016758e461ea1205183cd7d [10:16:41] 02puppet/03main 07CosmicAlpha 03e638258 Revert "mariadb: add monitoring for memory"… [10:16:43] [02puppet] 07Universal-Omega 04deleted 03revert-4970-mariadb-memory-monitoring at 0302562a6 13https://api.github.com/repos/miraheze/puppet/commit/02562a6 [10:16:51] PROBLEM - os151 Current Load on os151 is CRITICAL: LOAD CRITICAL - total load average: 5.54, 4.31, 4.10 [10:16:55] [02PDFEmbed] 07JimmyvanDelft commented on commit 036995d2a: @Universal-Omega This change bumps the MediaWiki requirement in extension.json from >= 1.43.0 to >= 1.46.0. […] 13https://github.com/miraheze/PDFEmbed/commit/6995d2ad20a4a27d6990d821ee9b013bdbe55c3a#commitcomment-202917001 [10:20:23] [02PDFEmbed] 07Universal-Omega commented on commit 036995d2a: I can branch a REL1_43 if wanted. This was intended because we only test against the latest MediaWiki typically so there are never guarantees it works on earlier versions. 13https://github.com/miraheze/PDFEmbed/commit/6995d2ad20a4a27d6990d821ee9b013bdbe55c3a#commitcomment-202917699 [10:20:51] PROBLEM - os151 Current Load on os151 is WARNING: LOAD WARNING - total load average: 3.09, 3.82, 3.98 [10:24:51] PROBLEM - os151 Current Load on os151 is CRITICAL: LOAD CRITICAL - total load average: 4.37, 3.94, 3.97 [10:28:51] PROBLEM - os151 Current Load on os151 is WARNING: LOAD WARNING - total load average: 1.58, 3.21, 3.72 [10:32:51] RECOVERY - os151 Current Load on os151 is OK: LOAD OK - total load average: 1.33, 2.14, 3.15 [10:45:07] PROBLEM - os201 Disk Space on os201 is WARNING: DISK WARNING - free space: / 43825MiB (9.9% inode=100%); [11:00:35] PROBLEM - os191 Disk Space on os191 is WARNING: DISK WARNING - free space: / 43513MiB (9.8% inode=100%); [11:02:32] RECOVERY - os191 Disk Space on os191 is OK: DISK OK - free space: / 49687MiB (11.2% inode=100%); [11:02:55] RECOVERY - os201 Disk Space on os201 is OK: DISK OK - free space: / 47110MiB (10.6% inode=100%); [11:31:01] PROBLEM - os191 Current Load on os191 is CRITICAL: LOAD CRITICAL - total load average: 4.10, 3.25, 2.70 [11:32:59] RECOVERY - os191 Current Load on os191 is OK: LOAD OK - total load average: 2.95, 3.13, 2.73 [11:40:15] [02PDFEmbed] 07JimmyvanDelft commented on commit 036995d2a: My main concern here is possible security updates in the future. Otherwise I am okay with just using the 'older' version. […] 13https://github.com/miraheze/PDFEmbed/commit/6995d2ad20a4a27d6990d821ee9b013bdbe55c3a#commitcomment-202927621 [11:43:30] PROBLEM - os201 Disk Space on os201 is WARNING: DISK WARNING - free space: / 43506MiB (9.8% inode=100%); [11:56:41] PROBLEM - os191 Current Load on os191 is WARNING: LOAD WARNING - total load average: 3.51, 2.93, 2.61 [12:00:41] RECOVERY - os191 Current Load on os191 is OK: LOAD OK - total load average: 2.74, 3.08, 2.76 [12:15:26] PROBLEM - os191 Disk Space on os191 is WARNING: DISK WARNING - free space: / 42427MiB (9.5% inode=100%); [12:19:22] RECOVERY - os191 Disk Space on os191 is OK: DISK OK - free space: / 45206MiB (10.2% inode=100%); [12:20:34] PROBLEM - os161 Current Load on os161 is WARNING: LOAD WARNING - total load average: 3.48, 2.77, 1.44 [12:23:17] PROBLEM - os191 Disk Space on os191 is WARNING: DISK WARNING - free space: / 42106MiB (9.5% inode=100%); [12:25:15] RECOVERY - os191 Disk Space on os191 is OK: DISK OK - free space: / 45957MiB (10.3% inode=100%); [12:28:34] PROBLEM - os161 Current Load on os161 is CRITICAL: LOAD CRITICAL - total load average: 4.80, 3.92, 2.53 [12:30:34] PROBLEM - os161 Current Load on os161 is WARNING: LOAD WARNING - total load average: 3.26, 3.64, 2.60 [12:32:34] PROBLEM - os161 Current Load on os161 is CRITICAL: LOAD CRITICAL - total load average: 4.20, 3.96, 2.85 [12:34:34] PROBLEM - os161 Current Load on os161 is WARNING: LOAD WARNING - total load average: 3.52, 3.75, 2.91 [12:38:34] RECOVERY - os161 Current Load on os161 is OK: LOAD OK - total load average: 3.33, 3.40, 2.95 [12:42:34] PROBLEM - os161 Current Load on os161 is WARNING: LOAD WARNING - total load average: 3.47, 3.52, 3.12 [12:48:34] RECOVERY - os161 Current Load on os161 is OK: LOAD OK - total load average: 2.84, 3.35, 3.20 [12:56:44] PROBLEM - os161 Disk Space on os161 is WARNING: DISK WARNING - free space: / 44471MiB (9.9% inode=100%); [13:01:30] RECOVERY - os201 Disk Space on os201 is OK: DISK OK - free space: / 47695MiB (10.7% inode=100%); [13:02:44] RECOVERY - os161 Disk Space on os161 is OK: DISK OK - free space: / 46943MiB (10.5% inode=100%); [13:03:22] PROBLEM - os161 Current Load on os161 is WARNING: LOAD WARNING - total load average: 3.64, 3.71, 3.42 [13:07:17] PROBLEM - os161 Current Load on os161 is CRITICAL: LOAD CRITICAL - total load average: 4.29, 3.83, 3.53 [13:11:12] PROBLEM - os161 Current Load on os161 is WARNING: LOAD WARNING - total load average: 3.66, 3.91, 3.64 [13:21:01] PROBLEM - os161 Current Load on os161 is CRITICAL: LOAD CRITICAL - total load average: 5.50, 4.16, 3.77 [13:30:37] !log [wwr@mwtask171] curl -X DELETE https://opensearch-mw.wikitide.net/kodiakwiki_content https://opensearch-mw.wikitide.net/kodiakwiki_general [13:30:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261001133037-51301900 [13:30:51] !log [wwr@mwtask171] sudo -u www-data php /srv/mediawiki/1.46/maintenance/run.php CirrusSearch:UpdateSearchIndexConfig --wiki=kodiakwiki --startOver (END - exit=0) [13:30:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261001133052-118b8973 [13:32:46] PROBLEM - os161 Current Load on os161 is WARNING: LOAD WARNING - total load average: 3.76, 3.95, 3.95 [13:34:18] PROBLEM - os161 Disk Space on os161 is WARNING: DISK WARNING - free space: / 43835MiB (9.8% inode=100%); [13:38:20] !log [wwr@mwtask171] sudo -u www-data php /srv/mediawiki/1.46/maintenance/run.php CirrusSearch:ForceSearchIndex --wiki=kodiakwiki (END - exit=0) [13:38:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log#sal-20261001133821-362e34fc [13:40:37] PROBLEM - os161 Current Load on os161 is CRITICAL: LOAD CRITICAL - total load average: 4.53, 3.83, 3.84 [13:42:35] PROBLEM - os161 Current Load on os161 is WARNING: LOAD WARNING - total load average: 3.16, 3.64, 3.77 [13:50:34] PROBLEM - os161 Current Load on os161 is CRITICAL: LOAD CRITICAL - total load average: 4.26, 3.82, 3.78 [13:56:34] PROBLEM - os161 Current Load on os161 is WARNING: LOAD WARNING - total load average: 3.39, 3.66, 3.74 [14:00:22] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/4d39bd53a5199b4fd4f7ec7b03268694d51abca0 [14:00:23] 02ssl/03main 07WikiTideBot 034d39bd5 Bot: Auto-update domain lists [14:00:34] PROBLEM - os161 Current Load on os161 is CRITICAL: LOAD CRITICAL - total load average: 4.47, 3.77, 3.74 [14:06:34] PROBLEM - os161 Current Load on os161 is WARNING: LOAD WARNING - total load average: 2.78, 3.58, 3.71 [14:12:34] PROBLEM - os161 Current Load on os161 is CRITICAL: LOAD CRITICAL - total load average: 4.02, 3.82, 3.76 [14:14:34] PROBLEM - os161 Current Load on os161 is WARNING: LOAD WARNING - total load average: 3.52, 3.79, 3.76 [14:16:34] PROBLEM - os161 Current Load on os161 is CRITICAL: LOAD CRITICAL - total load average: 5.12, 4.25, 3.93 [14:20:34] PROBLEM - os161 Current Load on os161 is WARNING: LOAD WARNING - total load average: 2.86, 3.87, 3.88 [14:32:34] PROBLEM - os161 Current Load on os161 is CRITICAL: LOAD CRITICAL - total load average: 4.92, 4.07, 3.83 [14:34:34] PROBLEM - os161 Current Load on os161 is WARNING: LOAD WARNING - total load average: 3.64, 3.78, 3.75 [14:46:34] PROBLEM - os161 Current Load on os161 is CRITICAL: LOAD CRITICAL - total load average: 4.64, 3.79, 3.69 [14:46:56] RECOVERY - os151 Disk Space on os151 is OK: DISK OK - free space: / 48710MiB (10.9% inode=100%); [14:48:34] PROBLEM - os161 Current Load on os161 is WARNING: LOAD WARNING - total load average: 3.83, 3.83, 3.72 [14:50:34] PROBLEM - os161 Current Load on os161 is CRITICAL: LOAD CRITICAL - total load average: 4.11, 3.85, 3.74 [14:54:34] PROBLEM - os161 Current Load on os161 is WARNING: LOAD WARNING - total load average: 2.84, 3.73, 3.75 [14:58:34] RECOVERY - os161 Current Load on os161 is OK: LOAD OK - total load average: 1.46, 2.63, 3.30 [16:00:22] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/a4cb331ab6588583e429a1dcedfba9d60f0f9ac5 [16:00:23] 02ssl/03main 07WikiTideBot 03a4cb331 Bot: Auto-update domain lists [16:15:30] PROBLEM - os201 Disk Space on os201 is WARNING: DISK WARNING - free space: / 44154MiB (9.9% inode=100%); [16:17:02] [02puppet] 07Universal-Omega created 03logbot-proxy (+1 new commit) 13https://github.com/miraheze/puppet/commit/293ec12aafa2 [16:17:02] 02puppet/03logbot-proxy 07CosmicAlpha 03293ec12 logbot: support proxy [16:17:11] [02puppet] 07Universal-Omega opened pull request #5066: logbot: support proxy (03main...03logbot-proxy) 13https://github.com/miraheze/puppet/pull/5066 [16:18:47] [02puppet] 07Universal-Omega pushed 1 new commit to 03logbot-proxy 13https://github.com/miraheze/puppet/commit/2bfda1e774a465cb91e8f40e8e283dfecd1fabb0 [16:18:47] 02puppet/03logbot-proxy 07CosmicAlpha 032bfda1e Add [16:20:05] miraheze/puppet - Universal-Omega the build passed. [16:20:14] [02puppet] 07Universal-Omega pushed 1 new commit to 03logbot-proxy 13https://github.com/miraheze/puppet/commit/a0d6d5b2922613f10bb665eb9e34627ba8c84d35 [16:20:15] 02puppet/03logbot-proxy 07CosmicAlpha 03a0d6d5b Update [16:20:56] PROBLEM - os151 Disk Space on os151 is WARNING: DISK WARNING - free space: / 44637MiB (10.0% inode=100%); [16:20:58] [02puppet] 07Universal-Omega pushed 1 new commit to 03logbot-proxy 13https://github.com/miraheze/puppet/commit/a315c72f181328f759cb92c4d0a075a05361d3a4 [16:20:58] 02puppet/03logbot-proxy 07CosmicAlpha 03a315c72 Update [16:21:48] [02puppet] 07Universal-Omega pushed 1 new commit to 03logbot-proxy 13https://github.com/miraheze/puppet/commit/5295c015a6f1887253eababcba4597d149e070b8 [16:21:48] 02puppet/03logbot-proxy 07CosmicAlpha 035295c01 Alignment [16:22:13] miraheze/puppet - Universal-Omega the build passed. [16:22:15] miraheze/puppet - Universal-Omega the build passed. [16:23:07] miraheze/puppet - Universal-Omega the build passed. [16:23:47] [02puppet] 07Universal-Omega pushed 1 new commit to 03logbot-proxy 13https://github.com/miraheze/puppet/commit/2bfda1e774a465cb91e8f40e8e283dfecd1fabb0 [16:23:47] 02puppet/03logbot-proxy 07CosmicAlpha 032bfda1e Add [16:24:34] miraheze/puppet - Universal-Omega the build passed. [16:25:08] [02puppet] 07Universal-Omega created 03ircrcbot-proxy (+1 new commit) 13https://github.com/miraheze/puppet/commit/9edab3efd198 [16:25:08] 02puppet/03ircrcbot-proxy 07CosmicAlpha 039edab3e ircrcbot: support proxy [16:25:14] [02puppet] 07Universal-Omega opened pull request #5067: ircrcbot: support proxy (03main...03ircrcbot-proxy) 13https://github.com/miraheze/puppet/pull/5067 [16:26:31] [02puppet] 07Universal-Omega pushed 1 new commit to 03ircrcbot-proxy 13https://github.com/miraheze/puppet/commit/0afdcbb39056e8676c8205f84b0df494f4264313 [16:26:31] 02puppet/03ircrcbot-proxy 07CosmicAlpha 030afdcbb Update [16:26:44] RECOVERY - os161 Disk Space on os161 is OK: DISK OK - free space: / 47637MiB (10.7% inode=100%); [16:26:56] RECOVERY - os151 Disk Space on os151 is OK: DISK OK - free space: / 47911MiB (10.7% inode=100%); [16:27:25] [02puppet] 07Universal-Omega pushed 1 new commit to 03ircrcbot-proxy 13https://github.com/miraheze/puppet/commit/f40bb2f9fa17d3e416db45960c57089a9b65f7e0 [16:27:26] 02puppet/03ircrcbot-proxy 07CosmicAlpha 03f40bb2f Update [16:27:30] RECOVERY - os201 Disk Space on os201 is OK: DISK OK - free space: / 46363MiB (10.4% inode=100%); [16:28:10] [02puppet] 07Universal-Omega pushed 1 new commit to 03ircrcbot-proxy 13https://github.com/miraheze/puppet/commit/bfc64857979ea0b6fd4189c50f93d580e07d8be3 [16:28:10] 02puppet/03ircrcbot-proxy 07CosmicAlpha 03bfc6485 + [16:29:24] [02puppet] 07Universal-Omega pushed 1 new commit to 03ircrcbot-proxy 13https://github.com/miraheze/puppet/commit/f0ddaab4876cacbb0e7fb0179dec13248ee24f02 [16:29:24] 02puppet/03ircrcbot-proxy 07CosmicAlpha 03f0ddaab + [16:47:30] PROBLEM - os201 Disk Space on os201 is WARNING: DISK WARNING - free space: / 43412MiB (9.8% inode=100%); [16:48:56] PROBLEM - os151 Disk Space on os151 is WARNING: DISK WARNING - free space: / 42969MiB (9.6% inode=100%); [16:50:56] RECOVERY - os151 Disk Space on os151 is OK: DISK OK - free space: / 47479MiB (10.6% inode=100%); [16:51:30] RECOVERY - os201 Disk Space on os201 is OK: DISK OK - free space: / 45700MiB (10.3% inode=100%); [16:55:30] PROBLEM - os201 Disk Space on os201 is WARNING: DISK WARNING - free space: / 43286MiB (9.7% inode=100%); [16:57:30] RECOVERY - os201 Disk Space on os201 is OK: DISK OK - free space: / 47563MiB (10.7% inode=100%); [17:13:30] PROBLEM - os161 Disk Space on os161 is WARNING: DISK WARNING - free space: / 44352MiB (9.9% inode=100%); [17:27:51] [02puppet] 07Universal-Omega pushed 1 new commit to 03ircrcbot-proxy 13https://github.com/miraheze/puppet/commit/635891f9435b6e0c6c7a10ba68032eeeaa742313 [17:27:51] 02puppet/03ircrcbot-proxy 07CosmicAlpha 03635891f Fix SASL [17:59:43] PROBLEM - os151 Disk Space on os151 is WARNING: DISK WARNING - free space: / 43995MiB (9.8% inode=100%); [18:01:28] PROBLEM - mw192 Puppet on mw192 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 4 minutes ago with 0 failures [18:01:30] PROBLEM - mwtask181 Puppet on mwtask181 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 28 minutes ago with 0 failures [18:01:31] PROBLEM - mwtask161 Puppet on mwtask161 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 18 minutes ago with 0 failures [18:01:37] PROBLEM - mw152 Puppet on mw152 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 27 minutes ago with 0 failures [18:01:41] PROBLEM - mw171 Puppet on mw171 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 10 minutes ago with 0 failures [18:01:48] PROBLEM - mw181 Puppet on mw181 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 30 minutes ago with 0 failures [18:01:51] PROBLEM - mw191 Puppet on mw191 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 10 minutes ago with 0 failures [18:02:00] PROBLEM - test151 Puppet on test151 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 28 minutes ago with 0 failures [18:02:00] PROBLEM - mw193 Puppet on mw193 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 14 minutes ago with 0 failures [18:02:00] PROBLEM - mw202 Puppet on mw202 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 22 minutes ago with 0 failures [18:02:04] PROBLEM - mw172 Puppet on mw172 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 21 minutes ago with 0 failures [18:02:05] PROBLEM - mw201 Puppet on mw201 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 28 minutes ago with 0 failures [18:02:08] PROBLEM - mw163 Puppet on mw163 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 4 minutes ago with 0 failures [18:02:15] PROBLEM - mw173 Puppet on mw173 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 18 minutes ago with 0 failures [18:02:17] PROBLEM - mw162 Puppet on mw162 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 23 minutes ago with 0 failures [18:02:18] PROBLEM - mwtask171 Puppet on mwtask171 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 20 minutes ago with 0 failures [18:02:25] PROBLEM - mw182 Puppet on mw182 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 26 minutes ago with 0 failures [18:02:28] PROBLEM - mw153 Puppet on mw153 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 11 minutes ago with 0 failures [18:02:35] PROBLEM - mw203 Puppet on mw203 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 20 minutes ago with 0 failures [18:02:39]