[01:02:38] PROBLEM - phorge171 Current Load on phorge171 is WARNING: LOAD WARNING - total load average: 7.62, 7.04, 5.40 [01:08:38] RECOVERY - phorge171 Current Load on phorge171 is OK: LOAD OK - total load average: 1.69, 5.34, 5.38 [08:07:11] [02python-functions] 07renovate[bot] 04force-pushed 03renovate/all-deps from 037703d17 to 03d0a2c13 13https://api.github.com/repos/miraheze/python-functions/commits/renovate/all-deps [08:07:11] 02python-functions/03renovate/all-deps 07renovate[bot] 03d0a2c13 chore(deps): update all dependencies to v9.1.1 [10:52:04] PROBLEM - matomo151 Check unit status of matomo-archiver-1 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-archiver-1 [14:36:37] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 17.75, 15.47, 13.54 [14:38:34] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 13.89, 14.68, 13.48 [14:54:09] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 16.13, 14.73, 14.13 [14:58:03] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 14.23, 14.71, 14.27 [15:03:54] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 8.16, 11.70, 13.22 [15:31:04] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 13.65, 12.80, 12.41 [15:33:01] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 11.82, 12.13, 12.20 [15:43:20] [02landing] 07renovate[bot] created 03renovate/all-deps (+1 new commit) 13https://github.com/miraheze/landing/commit/6ce33bf53a82 [15:43:20] 02landing/03renovate/all-deps 07renovate[bot] 036ce33bf Update all dependencies to v7 [16:00:04] RECOVERY - matomo151 Check unit status of matomo-archiver-1 on matomo151 is OK: OK: Status of the systemd unit matomo-archiver-1 [16:02:14] [02puppet] 07paladox created 03paladox-patch-2 (+1 new commit) 13https://github.com/miraheze/puppet/commit/e28093c46f7d [16:02:14] 02puppet/03paladox-patch-2 07paladox 03e28093c Set db182 wait_timeout to 28800… [16:02:18] [02puppet] 07paladox opened pull request #4868: Set db182 wait_timeout to 28800 (03main...03paladox-patch-2) 13https://github.com/miraheze/puppet/pull/4868 [16:03:08] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-2 13https://github.com/miraheze/puppet/commit/3aa108f7f9a7f8af79980c8755685d40a47bd614 [16:03:10] 02puppet/03paladox-patch-2 07paladox 033aa108f Add wait_timeout parameter to MariaDB config [16:04:53] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-2 13https://github.com/miraheze/puppet/commit/325b2ad911c610cbab03f003a7cbd9b119e9d02f [16:04:53] 02puppet/03paladox-patch-2 07paladox 03325b2ad Update db.pp [16:05:06] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-2 13https://github.com/miraheze/puppet/commit/141a5bc75c06713c252283c538e1448ff9a81cb8 [16:05:07] 02puppet/03paladox-patch-2 07paladox 03141a5bc Update db182.yaml [16:05:56] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/d10cf9db730e9df39d4f9e0a869391ecc19ab61a [16:05:56] 02puppet/03main 07dependabot[bot] 03d10cf9d build(deps): bump actions/checkout from 6 to 7 (#4867)… [16:05:56] [02puppet] 07paladox merged 07dependabot[bot]'s pull request #4867: build(deps): bump actions/checkout from 6 to 7 (03main...03dependabot/github_actions/actions/checkout-7) 13https://github.com/miraheze/puppet/pull/4867 [16:05:57] [02puppet] 07paladox 04deleted 03dependabot/github_actions/actions/checkout-7 at 03a273d42 13https://api.github.com/repos/miraheze/puppet/commit/a273d42 [16:06:09] [02puppet] 07paladox merged pull request #4868: Set db182 wait_timeout to 28800 (03main...03paladox-patch-2) 13https://github.com/miraheze/puppet/pull/4868 [16:06:10] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/373fb154322765a8d7b1ab014e68603898659971 [16:06:10] 02puppet/03main 07paladox 03373fb15 Set db182 wait_timeout to 28800 (#4868)… [16:06:11] [02puppet] 07paladox 04deleted 03paladox-patch-2 at 03141a5bc 13https://api.github.com/repos/miraheze/puppet/commit/141a5bc [16:07:31] !log restart mysql on db182 to pick up config change [16:07:32] miraheze/puppet - paladox the build passed. [16:07:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:10:23] PROBLEM - matomo151 Check unit status of matomo-archiver-2 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-archiver-2 [16:10:52] PROBLEM - matomo151 Check unit status of matomo-archiver-3 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-archiver-3 [16:11:09] RECOVERY - matomo151 Check unit status of matomo-archiver-2 on matomo151 is OK: OK: Status of the systemd unit matomo-archiver-2 [16:11:12] RECOVERY - matomo151 Check unit status of matomo-archiver-3 on matomo151 is OK: OK: Status of the systemd unit matomo-archiver-3 [16:17:05] PROBLEM - matomo151 Check unit status of matomo-archiver-3 on matomo151 is UNKNOWN: NRPE: Command 'check_Check unit status of matomo-archiver-3' not defined [16:17:07] PROBLEM - matomo151 Check unit status of matomo-archiver-2 on matomo151 is UNKNOWN: NRPE: Command 'check_Check unit status of matomo-archiver-2' not defined [16:17:38] PROBLEM - matomo151 Check unit status of matomo-archiver-1 on matomo151 is UNKNOWN: NRPE: Command 'check_Check unit status of matomo-archiver-1' not defined [16:18:33] [02puppet] 07paladox created 03paladox-patch-2 (+1 new commit) 13https://github.com/miraheze/puppet/commit/d6a761a02d82 [16:18:33] 02puppet/03paladox-patch-2 07paladox 03d6a761a matomo: Reduce concurrent archivers to 2 [16:18:37] [02puppet] 07paladox opened pull request #4869: matomo: Reduce concurrent archivers to 2 (03main...03paladox-patch-2) 13https://github.com/miraheze/puppet/pull/4869 [16:19:11] PROBLEM - matomo151 Puppet on matomo151 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 1 minute ago with 6 failures. Failed resources (up to 3 shown): File[/var/log/matomo/matomo-optimize-last2],File[/var/log/matomo/matomo-optimize-january],File[/var/log/matomo/matomo-queuedtracking-0],File[/var/log/matomo/matomo-queuedtracking-1] [16:19:47] [02puppet] 07paladox merged pull request #4869: matomo: Reduce concurrent archivers to 2 (03main...03paladox-patch-2) 13https://github.com/miraheze/puppet/pull/4869 [16:19:49] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/8ca283ec670b0ef8f7c93ded7581be1499e71723 [16:19:51] [02puppet] 07paladox 04deleted 03paladox-patch-2 at 03d6a761a 13https://api.github.com/repos/miraheze/puppet/commit/d6a761a [16:19:53] 02puppet/03main 07paladox 038ca283e matomo: Reduce concurrent archivers to 2 (#4869) [16:20:37] [02puppet] 07paladox created 03paladox-patch-2 (+1 new commit) 13https://github.com/miraheze/puppet/commit/4624494b4014 [16:20:37] 02puppet/03paladox-patch-2 07paladox 034624494 matomo: Reduce concurrent queuedtracking to 2 [16:20:40] [02puppet] 07paladox opened pull request #4870: matomo: Reduce concurrent queuedtracking to 2 (03main...03paladox-patch-2) 13https://github.com/miraheze/puppet/pull/4870 [16:21:05] PROBLEM - matomo151 Check unit status of matomo-archiver-2 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-archiver-2 [16:21:11] RECOVERY - matomo151 Puppet on matomo151 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [16:21:33] PROBLEM - matomo151 Check unit status of matomo-archiver-1 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-archiver-1 [16:21:57] [02puppet] 07paladox merged pull request #4870: matomo: Reduce concurrent queuedtracking to 2 (03main...03paladox-patch-2) 13https://github.com/miraheze/puppet/pull/4870 [16:21:59] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/4abd3ffee4ba1ce73dd1104e9b737a9230943070 [16:22:01] [02puppet] 07paladox 04deleted 03paladox-patch-2 at 034624494 13https://api.github.com/repos/miraheze/puppet/commit/4624494 [16:22:02] 02puppet/03main 07paladox 034abd3ff matomo: Reduce concurrent queuedtracking to 2 (#4870) [16:23:05] RECOVERY - matomo151 Check unit status of matomo-archiver-2 on matomo151 is OK: OK: Status of the systemd unit matomo-archiver-2 [16:23:30] RECOVERY - matomo151 Check unit status of matomo-archiver-1 on matomo151 is OK: OK: Status of the systemd unit matomo-archiver-1 [16:25:01] PROBLEM - matomo151 Check unit status of matomo-queuedtracking-2 on matomo151 is UNKNOWN: NRPE: Command 'check_Check unit status of matomo-queuedtracking-2' not defined [16:30:21] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/fb7377b1f10580a0707159fe6c5ef009765a7e8b [16:30:21] 02puppet/03main 07paladox 03fb7377b Revert "matomo: Reduce concurrent queuedtracking to 2 (#4870)"… [16:33:01] RECOVERY - matomo151 Check unit status of matomo-queuedtracking-2 on matomo151 is OK: OK: Status of the systemd unit matomo-queuedtracking-2 [16:35:33] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 19.61, 14.99, 12.38 [17:28:50] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 11.84, 13.56, 15.62 [17:36:50] PROBLEM - matomo151 Current Load on matomo151 is CRITICAL: LOAD CRITICAL - total load average: 18.49, 13.95, 14.65 [19:12:50] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 12.06, 13.68, 15.50 [19:13:16] [02puppet] 07paladox created 03paladox-patch-2 (+1 new commit) 13https://github.com/miraheze/puppet/commit/1414a81d8bc3 [19:13:16] 02puppet/03paladox-patch-2 07paladox 031414a81 matomo: Update archiver command to limit concurrent requests [19:13:19] [02puppet] 07paladox opened pull request #4871: matomo: Update archiver command to limit concurrent requests (03main...03paladox-patch-2) 13https://github.com/miraheze/puppet/pull/4871 [19:13:30] [02puppet] 07paladox commented on pull request #4871: Lets see if this will help the load... 13https://github.com/miraheze/puppet/pull/4871#issuecomment-4759725475 [19:13:33] [02puppet] 07paladox merged pull request #4871: matomo: Update archiver command to limit concurrent requests (03main...03paladox-patch-2) 13https://github.com/miraheze/puppet/pull/4871 [19:13:35] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/94a69ec957335acafd48b5bce0deedf33ff90069 [19:13:37] [02puppet] 07paladox 04deleted 03paladox-patch-2 at 031414a81 13https://api.github.com/repos/miraheze/puppet/commit/1414a81 [19:13:39] 02puppet/03main 07paladox 0394a69ec matomo: Update archiver command to limit concurrent requests (#4871) [19:16:21] PROBLEM - matomo151 Check unit status of matomo-archiver-2 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-archiver-2 [19:16:48] PROBLEM - matomo151 Check unit status of matomo-archiver-1 on matomo151 is CRITICAL: CRITICAL: Status of the systemd unit matomo-archiver-1 [19:20:50] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 9.06, 10.55, 13.25 [19:31:35] PROBLEM - matomo151 Puppet on matomo151 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 5 minutes ago with 0 failures [19:34:58] [02puppet] 07paladox created 03paladox-patch-2 (+1 new commit) 13https://github.com/miraheze/puppet/commit/eca7c2a3adb2 [19:34:58] 02puppet/03paladox-patch-2 07paladox 03eca7c2a matomo: Tweek config… [19:35:02] [02puppet] 07paladox opened pull request #4872: matomo: Tweek config (03main...03paladox-patch-2) 13https://github.com/miraheze/puppet/pull/4872 [19:35:26] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-2 13https://github.com/miraheze/puppet/commit/ce3e987b5244ec7095928b8a41e84479e462dbb5 [19:35:27] 02puppet/03paladox-patch-2 07paladox 03ce3e987 Adjust PHP FPM configuration parameters [19:38:58] [02mediawiki-repos] 07renovate[bot] created 03renovate/all-deps (+1 new commit) 13https://github.com/miraheze/mediawiki-repos/commit/38582286cd22 [19:38:58] 02mediawiki-repos/03renovate/all-deps 07renovate[bot] 033858228 chore(deps): update all dependencies to v7 [19:44:32] [02pywikibot-config] 07renovate[bot] created 03renovate/all-deps (+1 new commit) 13https://github.com/miraheze/pywikibot-config/commit/53c45726fa44 [19:44:32] 02pywikibot-config/03renovate/all-deps 07renovate[bot] 0353c4572 Update all dependencies to v7 [20:08:40] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-2 13https://github.com/miraheze/puppet/commit/e0578433903ed82b965f5b65c0550d35554169d3 [20:08:40] 02puppet/03paladox-patch-2 07paladox 03e057843 Update matomo151.yaml [20:25:24] [02puppet] 07paladox merged pull request #4872: matomo: Tweek config (03main...03paladox-patch-2) 13https://github.com/miraheze/puppet/pull/4872 [20:25:24] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/5216ad3cdd4ae93b91583cf4a9fa611073b6dcd6 [20:25:25] 02puppet/03main 07paladox 035216ad3 matomo: Tweek config (#4872)… [20:25:27] [02puppet] 07paladox 04deleted 03paladox-patch-2 at 03e057843 13https://api.github.com/repos/miraheze/puppet/commit/e057843 [20:27:35] RECOVERY - matomo151 Puppet on matomo151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:30:17] [02ssl] 07WikiTideBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/8e0a5b6f4c05c2fb8d018372a4d0eb0ee4b9aa08 [21:30:18] 02ssl/03main 07WikiTideBot 038e0a5b6 Bot: Auto-update domain lists [23:26:48] RECOVERY - matomo151 Check unit status of matomo-archiver-1 on matomo151 is OK: OK: Status of the systemd unit matomo-archiver-1 [23:28:21] RECOVERY - matomo151 Check unit status of matomo-archiver-2 on matomo151 is OK: OK: Status of the systemd unit matomo-archiver-2 [23:40:45] PROBLEM - phorge171 Current Load on phorge171 is CRITICAL: LOAD CRITICAL - total load average: 8.31, 6.53, 4.49 [23:46:16] PROBLEM - phorge171 issue-tracker.miraheze.org HTTPS on phorge171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.013 second response time [23:46:45] PROBLEM - phorge171 Current Load on phorge171 is WARNING: LOAD WARNING - total load average: 7.88, 7.78, 5.74 [23:48:45] PROBLEM - phorge171 Current Load on phorge171 is CRITICAL: LOAD CRITICAL - total load average: 8.45, 8.03, 6.09 [23:51:59] who ate my phorge [23:54:07] RECOVERY - phorge171 issue-tracker.miraheze.org HTTPS on phorge171 is OK: HTTP OK: HTTP/1.1 200 OK - 18207 bytes in 9.317 second response time [23:56:46] oh sorry I didn't realise it was YOUR phorge [23:56:55] look I get a little peckish sometimes [23:56:58] can't a girl eat? [23:58:45] PROBLEM - phorge171 Current Load on phorge171 is WARNING: LOAD WARNING - total load average: 5.59, 7.65, 6.95 [23:59:15] should drop very fast