[00:00:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.98, 3.82, 3.72 [00:01:19] RECOVERY - matomo101 Current Load on matomo101 is OK: OK - load average: 1.15, 1.51, 3.20 [00:01:20] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.16, 3.17, 3.09 [00:03:42] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/2ef4f22a05d4...a7d6a4e83905 [00:03:44] [02miraheze/puppet] 07paladox 03a7d6a4e - matomo: update to 4.9.0 [00:04:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.42, 4.20, 3.90 [00:04:59] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.99, 3.72, 3.19 [00:10:06] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:10:14] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.45, 5.05, 3.84 [00:14:35] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.71, 3.56, 3.45 [00:15:06] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:16:02] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.52, 3.95, 3.72 [00:16:30] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.25, 3.28, 3.37 [00:17:01] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.18, 1.73, 1.26 [00:19:01] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.33, 1.70, 1.31 [00:21:50] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.18, 2.89, 3.33 [00:25:06] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:29:36] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.95, 3.45, 3.44 [00:29:58] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.24, 3.75, 3.38 [00:31:53] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.33, 3.51, 3.33 [00:33:49] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.82, 3.34, 3.29 [00:35:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/211ab94e5aca [00:35:40] [02miraheze/puppet] 07paladox 03211ab94 - base: Fix rsyslog logrotate file [00:35:41] [02puppet] 07paladox created branch 03paladox-patch-9 - 13https://github.com/miraheze/puppet [00:35:43] [02puppet] 07paladox opened pull request 03#2493: base: Fix rsyslog logrotate file - 13https://github.com/miraheze/puppet/pull/2493 [00:36:09] [02puppet] 07paladox synchronize pull request 03#2493: base: Fix rsyslog logrotate file - 13https://github.com/miraheze/puppet/pull/2493 [00:36:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/211ab94e5aca...f64469605f70 [00:36:12] [02miraheze/puppet] 07paladox 03f644696 - Update syslog.pp [00:36:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/f64469605f70...c244298f5f16 [00:36:55] [02miraheze/puppet] 07paladox 03c244298 - Create rsyslog.logrotate.conf [00:36:57] [02puppet] 07paladox synchronize pull request 03#2493: base: Fix rsyslog logrotate file - 13https://github.com/miraheze/puppet/pull/2493 [00:38:22] [02puppet] 07paladox closed pull request 03#2493: base: Fix rsyslog logrotate file - 13https://github.com/miraheze/puppet/pull/2493 [00:38:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/puppet/compare/a7d6a4e83905...3bdbd8142abc [00:38:25] [02miraheze/puppet] 07paladox 033bdbd81 - base: Fix rsyslog logrotate file (#2493) [00:38:26] [02puppet] 07paladox deleted branch 03paladox-patch-9 - 13https://github.com/miraheze/puppet [00:38:28] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-9 [00:39:20] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.26, 3.11, 3.36 [00:40:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.45, 3.35, 3.91 [00:40:06] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:41:43] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.64, 1.82, 1.41 [00:43:38] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.97, 2.14, 1.56 [00:44:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.99, 4.78, 4.31 [00:45:34] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.95, 1.67, 1.45 [00:45:49] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.82, 3.55, 3.18 [00:47:49] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.56, 3.13, 3.07 [00:50:06] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:51:40] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.70, 10.17, 9.07 [00:52:55] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.22, 8.92, 8.30 [00:54:52] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.27, 8.71, 8.28 [00:55:28] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.79, 9.90, 9.25 [00:58:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.38, 3.69, 3.99 [01:00:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.21, 3.82, 4.00 [01:00:31] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.80, 10.73, 9.87 [01:02:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.81, 3.56, 3.89 [01:04:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.72, 3.87, 3.95 [01:05:06] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:08:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.69, 3.61, 3.87 [01:08:26] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.72, 9.13, 9.53 [01:10:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.69, 3.96, 3.96 [01:14:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.14, 3.91, 3.99 [01:18:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.33, 4.73, 4.27 [01:28:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.40, 3.61, 3.98 [01:29:20] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.97, 4.23, 3.28 [01:30:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.87, 4.17, 4.14 [01:31:56] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 4.41, 2.27, 1.51 [01:33:51] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.58, 1.97, 1.49 [01:34:05] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.04, 3.82, 3.20 [01:34:13] paladox, are you around? [01:36:00] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.60, 3.50, 3.17 [01:37:49] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.35, 1.64, 1.46 [01:39:20] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 1.95, 3.52, 3.54 [01:40:06] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:41:49] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.10, 3.66, 3.35 [01:43:20] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.72, 3.12, 3.37 [01:43:49] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.04, 3.09, 3.18 [01:45:06] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:48:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.99, 3.19, 3.97 [01:57:18] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.43, 9.83, 8.98 [01:59:18] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.70, 9.90, 9.12 [02:00:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 8.27, 4.80, 4.07 [02:01:20] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.60, 3.23, 2.99 [02:03:20] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.74, 3.59, 3.14 [02:05:06] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:05:20] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.10, 3.46, 3.15 [02:07:20] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.02, 3.09, 3.07 [02:10:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.85, 3.34, 3.78 [02:10:06] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:14:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.49, 3.53, 3.72 [02:16:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.92, 3.52, 3.69 [02:20:06] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:23:42] PROBLEM - cp30 Disk Space on cp30 is WARNING: DISK WARNING - free space: / 4288 MB (10% inode=96%); [02:25:06] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:26:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.05, 3.67, 3.63 [02:28:01] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.54, 3.48, 3.56 [02:30:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.67, 4.15, 3.81 [02:49:49] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.86, 1.45, 1.16 [02:50:06] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:51:49] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.36, 1.74, 1.30 [02:52:10] [02MirahezeMagic] 07dmehus opened pull request 03#334: Add MirahezeMagic interface messages for requestwikiblocked user group - 13https://github.com/miraheze/MirahezeMagic/pull/334 [02:53:49] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.75, 1.83, 1.39 [02:53:56] [02MirahezeMagic] 07dmehus synchronize pull request 03#334: Add MirahezeMagic interface messages for requestwikiblocked user group - 13https://github.com/miraheze/MirahezeMagic/pull/334 [02:55:06] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:55:35] miraheze/MirahezeMagic - dmehus the build has errored. [02:55:40] [02MirahezeMagic] 07Universal-Omega commented on pull request 03#334: Add MirahezeMagic interface messages for requestwikiblocked user group - 13https://github.com/miraheze/MirahezeMagic/pull/334#issuecomment-1100792393 [02:56:07] [02MirahezeMagic] 07Universal-Omega closed pull request 03#334: Add MirahezeMagic interface messages for requestwikiblocked user group - 13https://github.com/miraheze/MirahezeMagic/pull/334 [02:56:09] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/MirahezeMagic/compare/09d50a0bc5f4...0ba95302c42a [02:56:10] [02miraheze/MirahezeMagic] 07dmehus 030ba9530 - Add MirahezeMagic interface messages for requestwikiblocked user group (#334) [02:56:26] [02MirahezeMagic] 07dmehus commented on pull request 03#334: Add MirahezeMagic interface messages for requestwikiblocked user group - 13https://github.com/miraheze/MirahezeMagic/pull/334#issuecomment-1100792476 [02:57:49] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.65, 2.56, 1.77 [02:58:19] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/0ba95302c42a...b9154b21426f [02:58:20] miraheze/MirahezeMagic - dmehus the build has errored. [02:58:20] [02miraheze/MirahezeMagic] 07Universal-Omega 03b9154b2 - Update qqq.json [02:58:40] [02MirahezeMagic] 07Universal-Omega commented on pull request 03#334: Add MirahezeMagic interface messages for requestwikiblocked user group - 13https://github.com/miraheze/MirahezeMagic/pull/334#issuecomment-1100792680 [03:00:42] miraheze/MirahezeMagic - Universal-Omega the build has errored. [03:01:37] I know miraheze-github.... [03:01:49] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.65, 1.61, 1.57 [03:02:11] We need the upstream fix for the CI to be backported to REL1_38! [03:03:06] miraheze/MirahezeMagic - Universal-Omega the build has errored. [03:03:43] RhinosF1: ^ can you try and talk to upstream about that also? https://gerrit.wikimedia.org/r/c/mediawiki/core/+/778313 is what needs backported I think. [03:06:27] PROBLEM - cp31 Disk Space on cp31 is WARNING: DISK WARNING - free space: / 4284 MB (10% inode=96%); [03:08:09] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.53, 3.09, 2.87 [03:12:01] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.86, 3.57, 3.09 [03:13:57] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.42, 3.42, 3.09 [03:15:06] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:15:53] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.78, 3.21, 3.05 [03:35:06] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:40:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.25, 3.07, 3.96 [03:44:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.71, 3.36, 3.85 [03:46:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.45, 3.21, 3.72 [03:47:49] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.75, 1.75, 1.38 [03:48:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.08, 3.51, 3.75 [03:49:49] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.57, 1.66, 1.39 [03:50:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.84, 3.26, 3.63 [03:58:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.15, 3.67, 3.65 [04:00:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.37, 3.17, 3.47 [04:02:00] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.39, 2.78, 3.27 [04:02:19] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/8d4a676fa598...cc7db235364b [04:02:21] [02miraheze/mw-config] 07Universal-Omega 03cc7db23 - Update LocalWiki.php [04:03:23] miraheze/mw-config - Universal-Omega the build passed. [04:04:10] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/cc7db235364b...73ee5eb5d709 [04:04:12] [02miraheze/mw-config] 07Universal-Omega 0373ee5eb - T8926: add og:image:width meta tag to famedatawiki [04:05:17] miraheze/mw-config - Universal-Omega the build passed. [04:07:40] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/73ee5eb5d709...c0eeb75cd7ed [04:07:42] [02miraheze/mw-config] 07Universal-Omega 03c0eeb75 - Update ManageWikiSettings.php [04:07:59] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/c0eeb75cd7ed...25e9f1073136 [04:08:00] [02miraheze/mw-config] 07Universal-Omega 0325e9f10 - Update ManageWikiNamespaces.php [04:08:15] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/25e9f1073136...bb3241dfe2fe [04:08:16] [02miraheze/mw-config] 07Universal-Omega 03bb3241d - Update ManageWikiExtensions.php [04:08:40] miraheze/mw-config - Universal-Omega the build passed. [04:09:03] miraheze/mw-config - Universal-Omega the build passed. [04:09:12] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/bb3241dfe2fe...56f5bf3c1b3e [04:09:13] [02miraheze/mw-config] 07Universal-Omega 0356f5bf3 - Use // instead of # comments [04:09:20] miraheze/mw-config - Universal-Omega the build passed. [04:10:22] miraheze/mw-config - Universal-Omega the build passed. [04:12:47] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/56f5bf3c1b3e...a34f5a7cd3a2 [04:12:48] [02miraheze/mw-config] 07Universal-Omega 03a34f5a7 - Update GlobalExtensions.php [04:13:03] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a34f5a7cd3a2...29c0b23ed44e [04:13:05] [02miraheze/mw-config] 07Universal-Omega 0329c0b23 - Update GlobalSkins.php [04:13:31] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/29c0b23ed44e...d64416a32f9e [04:13:33] [02miraheze/mw-config] 07Universal-Omega 03d64416a - Add trailing comma [04:13:48] miraheze/mw-config - Universal-Omega the build passed. [04:14:08] miraheze/mw-config - Universal-Omega the build passed. [04:14:21] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/d64416a32f9e...a6b4d3bb28dc [04:14:22] [02miraheze/mw-config] 07Universal-Omega 03a6b4d3b - Use // style comments [04:14:37] miraheze/mw-config - Universal-Omega the build passed. [04:15:18] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a6b4d3bb28dc...51c4edb9b1a7 [04:15:19] [02miraheze/mw-config] 07Universal-Omega 0351c4edb - Add newline to top of file [04:15:30] miraheze/mw-config - Universal-Omega the build passed. [04:16:02] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/51c4edb9b1a7...a7f233ec1022 [04:16:03] [02miraheze/mw-config] 07Universal-Omega 03a7f233e - Use // style comments [04:16:26] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a7f233ec1022...0923e7beb085 [04:16:27] [02miraheze/mw-config] 07Universal-Omega 030923e7b - Format newline [04:16:29] miraheze/mw-config - Universal-Omega the build passed. [04:17:10] miraheze/mw-config - Universal-Omega the build passed. [04:17:28] miraheze/mw-config - Universal-Omega the build passed. [04:18:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.95, 4.31, 3.49 [04:22:21] !log [@mwtask111] starting deploy of {'config': True} to all [04:22:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:22:33] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 12s [04:22:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:24:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.20, 3.41, 3.42 [04:26:00] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.78, 3.15, 3.32 [04:27:15] [02CreateWiki] 07Universal-Omega commented on pull request 03#317: T8885: Support filtering farmer log subtypes - 13https://github.com/miraheze/CreateWiki/pull/317#issuecomment-1100801042 [04:30:06] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:35:06] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:46:34] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.42, 2.97, 2.57 [04:48:30] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.71, 2.84, 2.57 [04:50:06] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:55:06] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:58:53] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.93, 3.27, 1.94 [05:06:33] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.99, 1.63, 1.62 [05:15:06] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:22:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.31, 4.00, 3.49 [05:24:15] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.29, 3.73, 3.45 [05:25:06] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:26:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.86, 4.02, 3.61 [05:32:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.32, 3.64, 3.68 [05:33:49] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.59, 2.48, 1.66 [05:35:49] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.23, 1.90, 1.54 [05:36:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.33, 5.05, 4.21 [05:37:49] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.36, 1.67, 1.50 [05:37:49] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.33, 3.20, 2.64 [05:40:14] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.63, 3.65, 2.81 [05:41:04] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [05:41:49] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.46, 3.24, 2.82 [05:42:10] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.57, 3.14, 2.72 [05:52:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.14, 3.23, 3.95 [05:54:01] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.08, 4.39, 4.29 [05:58:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.09, 3.27, 3.86 [06:00:07] [02mediawiki] 07lens0021 commented on pull request 03#6196: Bump extensions/PortableInfobox from `2a6077a` to `4fbbead` - 13https://github.com/miraheze/mediawiki/pull/6196#issuecomment-1100810916 [06:00:09] [02mediawiki] 07dependabot[bot] commented on pull request 03#6196: Bump extensions/PortableInfobox from `2a6077a` to `4fbbead` - 13https://github.com/miraheze/mediawiki/pull/6196#issuecomment-1100810918 [06:00:53] [02mediawiki] 07Universal-Omega commented on pull request 03#6196: Bump extensions/PortableInfobox from `2a6077a` to `4fbbead` - 13https://github.com/miraheze/mediawiki/pull/6196#issuecomment-1100810989 [06:01:27] Thank you! [06:02:01] No problem! [06:02:06] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [06:02:06] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [06:02:55] Lens0021: I'll try and deploy tomorrow, unless someone else does first. [06:04:00] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.49, 2.39, 3.31 [06:05:42] 👍 [06:09:03] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [06:10:39] [02mediawiki] 07dependabot[bot] edited pull request 03#6196: Bump extensions/PortableInfobox from `2a6077a` to `4fbbead` - 13https://github.com/miraheze/mediawiki/pull/6196 [06:11:23] [02mediawiki] 07dependabot[bot] edited pull request 03#6196: Bump extensions/PortableInfobox from `2a6077a` to `4fbbead` - 13https://github.com/miraheze/mediawiki/pull/6196 [06:11:28] [02mediawiki] 07dependabot[bot] opened pull request 03#6303: Bump extensions/PortableInfobox from `2a6077a` to `5f13d93` - 13https://github.com/miraheze/mediawiki/pull/6303 [06:11:29] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/PortableInfobox-5f13d93 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/22de9be8b847 [06:11:31] [02miraheze/mediawiki] 07dependabot[bot] 0322de9be - Bump extensions/PortableInfobox from `2a6077a` to `5f13d93` [06:11:32] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/PortableInfobox-5f13d93 - 13https://github.com/miraheze/mediawiki [06:11:55] [02mediawiki] 07github-actions[bot] labeled pull request 03#6303: Bump extensions/PortableInfobox from `2a6077a` to `5f13d93` - 13https://github.com/miraheze/mediawiki/pull/6303 [06:11:56] [02mediawiki] 07github-actions[bot] labeled pull request 03#6303: Bump extensions/PortableInfobox from `2a6077a` to `5f13d93` - 13https://github.com/miraheze/mediawiki/pull/6303 [06:14:53] [02mediawiki] 07Universal-Omega closed pull request 03#6196: Bump extensions/PortableInfobox from `2a6077a` to `4fbbead` - 13https://github.com/miraheze/mediawiki/pull/6196 [06:14:56] [02mediawiki] 07dependabot[bot] commented on pull request 03#6196: Bump extensions/PortableInfobox from `2a6077a` to `4fbbead` - 13https://github.com/miraheze/mediawiki/pull/6196#issuecomment-1100812734 [06:14:57] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/PortableInfobox-4fbbead - 13https://github.com/miraheze/mediawiki [06:14:59] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/PortableInfobox-4fbbead [06:20:00] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.60, 2.25, 1.30 [06:22:00] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.37, 1.89, 1.28 [06:24:00] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.09, 1.65, 1.27 [06:30:23] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.53, 3.41, 2.78 [06:30:41] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [06:30:47] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.10, 3.91, 3.18 [06:32:18] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.41, 3.30, 2.81 [06:34:13] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.11, 2.83, 2.69 [06:34:35] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.99, 3.56, 3.22 [06:38:22] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.29, 3.37, 3.22 [06:50:00] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 3.32, 1.90, 1.51 [06:52:00] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.88, 1.95, 1.59 [06:52:03] [02mw-config] 07lens0021 opened pull request 03#4600: Add `$wgPortableInfoboxResponsiblyOpenCollapsed` to ManageWikiSettings.php - 13https://github.com/miraheze/mw-config/pull/4600 [06:53:05] miraheze/mw-config - lens0021 the build passed. [06:54:00] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.14, 1.98, 1.63 [06:55:40] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 3.69, 3.09, 2.47 [06:56:00] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.43, 1.74, 1.59 [06:57:40] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 2.57, 2.91, 2.48 [06:58:00] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.18, 1.52, 1.52 [06:58:58] !log reception@mwtask111:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=obeymewiki --search-recursively /home/reception/Obeyme --user=Miraheze_maintenance_script --comment="Imported from obey-me.fandom.com" [06:59:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:04:56] [02mw-config] 07RhinosF1 reviewed pull request 03#4594 commit - 13https://github.com/miraheze/mw-config/pull/4594#discussion_r851713434 [07:05:38] [02mw-config] 07RhinosF1 reviewed pull request 03#4594 commit - 13https://github.com/miraheze/mw-config/pull/4594#discussion_r851713494 [07:06:00] [02mw-config] 07RhinosF1 reviewed pull request 03#4594 commit - 13https://github.com/miraheze/mw-config/pull/4594#discussion_r851713494 [07:29:23] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.50, 2.89, 2.61 [07:31:20] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.94, 2.90, 2.65 [07:52:37] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.19, 1.83, 1.36 [07:54:32] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.45, 1.69, 1.36 [08:05:06] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:10:06] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:11:20] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.76, 3.40, 2.83 [08:13:20] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.89, 3.20, 2.83 [08:29:06] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.62, 3.45, 3.20 [08:29:20] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.16, 3.49, 2.96 [08:30:03] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.05, 3.51, 2.90 [08:30:59] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.90, 3.38, 3.21 [08:31:20] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.59, 3.13, 2.90 [08:31:58] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.97, 2.92, 2.76 [08:40:28] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.03, 3.51, 3.25 [08:42:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.54, 4.21, 3.50 [08:46:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.22, 3.78, 3.49 [08:48:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.32, 4.32, 3.72 [08:50:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.27, 3.86, 3.62 [08:54:00] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.69, 2.83, 3.26 [09:02:06] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [09:02:06] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [09:27:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.75, 3.20, 2.81 [09:29:12] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.62, 3.04, 2.80 [09:30:41] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [09:46:15] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.44, 4.44, 3.59 [09:48:09] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.99, 3.85, 3.47 [09:52:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.67, 4.45, 3.74 [09:58:00] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.00, 3.59, 3.64 [10:12:00] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.97, 3.03, 3.34 [10:34:55] [02mw-config] 07RhinosF1 reviewed pull request 03#4594 commit - 13https://github.com/miraheze/mw-config/pull/4594#discussion_r851713434 [11:23:23] !log mon111: upgrade grafana [11:23:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:27:28] !log mon111: upgrade icingaweb2 [11:27:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:30:45] PROBLEM - mon111 Puppet on mon111 is CRITICAL: CRITICAL: Puppet has 5 failures. Last run 2 minutes ago with 5 failures. Failed resources (up to 3 shown): Package[python-is-python3],Package[python],Package[icingaweb2-module-monitoring],Package[icingaweb2-module-doc] [11:36:43] RECOVERY - mon111 Puppet on mon111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:38:47] a!log mon111: upgrade icinga2 [11:41:03] RECOVERY - mon111 Puppet on mon111 is OK: OK: Puppet is currently enabled, last run 6 minutes ago with 0 failures [11:44:06] paladox: didn't log [11:44:16] !log mon111: upgrade icinga2 [11:44:17] thanks [11:44:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:44:36] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.79, 3.64, 2.96 [11:48:34] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.58, 3.70, 3.11 [11:50:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.56, 3.89, 3.27 [11:52:32] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.36, 3.28, 3.12 [12:13:07] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:14:27] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 4.76, 4.65, 2.44 [12:14:32] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.55, 4.94, 3.75 [12:18:07] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:18:28] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.71, 3.53, 3.46 [12:20:27] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.75, 1.99, 1.94 [12:22:23] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.53, 3.11, 3.32 [12:24:27] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.35, 1.12, 1.59 [12:39:31] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=aikawiki --report 1 --username-prefix=wikia:aika --no-updates /home/reception/aikafandomcom-20211112-history.xml (START) [12:39:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:51:49] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.48, 4.42, 3.56 [12:52:15] !log DELETE FROM oldimage WHERE oi_archive_name=''; (frozenplumwiki) [12:52:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:58:07] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:03:07] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:04:28] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=aikawiki --report 1 --username-prefix=wikia:aika --no-updates /home/reception/aikafandomcom-20211112-history.xml (END - exit=0) [13:04:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:05:33] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.15, 3.91, 3.95 [13:08:07] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:08:10] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/908e106bea75...87e48af9c9e5 [13:08:11] [02miraheze/dns] 07Reception123 0387e48af - add royal-wiki.org zone [13:09:28] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.47, 4.15, 4.03 [13:11:19] [02miraheze/ssl] 07Reception123 pushed 032 commits to 03master [+2/-0/±3] 13https://github.com/miraheze/ssl/compare/5aa7c81b3c5a...84588eb20297 [13:11:21] [02miraheze/ssl] 07Reception123 03e298808 - add www.agentisai.com cert [13:11:22] [02miraheze/ssl] 07Reception123 0384588eb - add royal-wiki.org cert [13:11:26] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.19, 3.43, 3.78 [13:15:41] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.24, 10.62, 9.09 [13:17:37] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.75, 10.05, 9.06 [13:18:07] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:25:09] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.51, 2.90, 3.35 [13:35:38] [02CreateWiki] 07lens0021 synchronize pull request 03#315: Inject CreateWikiHookRunner - 13https://github.com/miraheze/CreateWiki/pull/315 [13:36:20] [02CreateWiki] 07lens0021 edited pull request 03#315: Update hook calling to use new HookContainer system - 13https://github.com/miraheze/CreateWiki/pull/315 [13:36:48] [02CreateWiki] 07lens0021 edited pull request 03#315: Update hook calling to use new HookContainer system - 13https://github.com/miraheze/CreateWiki/pull/315 [13:36:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.61, 3.27, 2.82 [13:38:44] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.48, 3.85, 3.52 [13:38:55] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.56, 3.65, 3.05 [13:38:58] [02CreateWiki] 07lens0021 commented on pull request 03#310: Update hook calling to use new HookContainer system - 13https://github.com/miraheze/CreateWiki/pull/310#issuecomment-1100877417 [13:38:59] [02CreateWiki] 07lens0021 closed pull request 03#310: Update hook calling to use new HookContainer system - 13https://github.com/miraheze/CreateWiki/pull/310 [13:40:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.28, 3.55, 3.44 [13:40:47] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.86, 3.38, 3.00 [13:40:52] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.69, 3.68, 3.14 [13:41:33] [02CreateWiki] 07lens0021 synchronize pull request 03#315: Update hook calling to use new HookContainer system - 13https://github.com/miraheze/CreateWiki/pull/315 [13:42:49] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.90, 4.03, 3.34 [13:43:07] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:44:46] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.49, 3.60, 3.25 [13:45:11] [02CreateWiki] 07lens0021 commented on pull request 03#315: Update hook calling to use new HookContainer system - 13https://github.com/miraheze/CreateWiki/pull/315#issuecomment-1100878800 [13:45:20] miraheze/CreateWiki - lens0021 the build has errored. [13:45:58] [02CreateWiki] 07lens0021 synchronize pull request 03#315: Update hook calling to use new HookContainer system - 13https://github.com/miraheze/CreateWiki/pull/315 [13:46:43] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.66, 3.30, 3.19 [13:46:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.02, 3.60, 3.50 [13:48:07] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:48:43] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.45, 3.27, 3.40 [13:49:21] miraheze/CreateWiki - lens0021 the build has errored. [13:52:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.03, 3.59, 3.50 [13:53:07] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:53:15] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=aikawiki (START) [13:53:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:54:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.43, 3.36, 3.44 [13:56:43] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.65, 3.17, 3.36 [14:01:13] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 1.84, 3.07, 2.13 [14:03:07] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:07:07] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 0.74, 1.81, 1.89 [14:11:03] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.01, 1.26, 1.64 [14:11:33] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.82, 3.90, 3.51 [14:13:28] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.98, 3.88, 3.55 [14:18:07] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:19:13] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.30, 3.80, 3.57 [14:32:58] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.40, 9.87, 8.70 [14:36:53] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.92, 9.24, 8.73 [14:39:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.39, 3.74, 3.09 [14:41:26] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.85, 3.37, 3.03 [14:48:07] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:49:05] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=aikawiki (END - exit=0) [14:49:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:52:53] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.33, 10.27, 9.49 [14:53:07] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:54:49] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.90, 9.81, 9.42 [14:56:31] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.09, 1.43, 1.06 [14:58:30] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.63, 1.09, 0.98 [15:01:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/7930a9407b3e [15:01:47] [02miraheze/puppet] 07paladox 037930a94 - varnish: tweak systemd unit [15:01:49] [02puppet] 07paladox created branch 03paladox-patch-9 - 13https://github.com/miraheze/puppet [15:01:50] [02puppet] 07paladox opened pull request 03#2494: varnish: tweak systemd unit - 13https://github.com/miraheze/puppet/pull/2494 [15:02:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 1.67, 3.10, 4.00 [15:03:07] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:10:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.33, 4.10, 4.00 [15:12:42] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.50, 10.23, 9.18 [15:13:07] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:13:27] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/9ec148d0839c [15:13:29] [02miraheze/mw-config] 07Universal-Omega 039ec148d - Re-enable CreateRedirect; Disable EditNotify [15:13:30] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [15:13:32] [02mw-config] 07Universal-Omega opened pull request 03#4601: Re-enable CreateRedirect; Disable EditNotify - 13https://github.com/miraheze/mw-config/pull/4601 [15:14:42] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.04, 9.95, 9.22 [15:14:43] miraheze/mw-config - Universal-Omega the build passed. [15:18:07] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:18:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.24, 3.57, 3.85 [15:24:05] !log cp20/21: run varnishadm param.set nuke_limit 1000 [15:24:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:26:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.36, 3.56, 3.70 [15:26:46] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.57, 3.70, 3.18 [15:27:09] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.41, 3.68, 3.13 [15:28:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.60, 3.39, 3.65 [15:28:43] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.82, 3.46, 3.16 [15:29:06] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.71, 3.30, 3.07 [15:30:40] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.98, 2.99, 3.03 [15:34:43] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.19, 2.89, 3.35 [15:38:07] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:38:33] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.38, 1.82, 1.39 [15:40:33] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 14.33, 5.03, 2.51 [15:42:14] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.91, 1.41, 1.14 [15:42:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.93, 3.77, 3.51 [15:44:14] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.23, 1.26, 1.11 [15:46:11] [02mw-config] 07MacFan4000 commented on pull request 03#4594: [#T9071](https://phabricator.miraheze.org/T9071) - 13https://github.com/miraheze/mw-config/pull/4594#issuecomment-1100904217 [15:46:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.22, 4.52, 3.86 [15:48:33] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.99, 1.80, 1.87 [15:48:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.37, 3.63, 3.61 [15:48:55] [02mw-config] 07MacFan4000 reviewed pull request 03#4594 commit - 13https://github.com/miraheze/mw-config/pull/4594#discussion_r851773458 [15:49:02] [02mw-config] 07MacFan4000 reviewed pull request 03#4594 commit - 13https://github.com/miraheze/mw-config/pull/4594#discussion_r851773465 [15:50:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.94, 4.21, 3.82 [15:51:32] PROBLEM - cp30 Disk Space on cp30 is CRITICAL: DISK CRITICAL - free space: / 1993 MB (5% inode=96%); [15:52:33] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 3.96, 3.17, 2.38 [15:52:39] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.09, 3.56, 2.98 [15:53:32] PROBLEM - cp30 Disk Space on cp30 is WARNING: DISK WARNING - free space: / 2452 MB (6% inode=96%); [15:54:21] [02mw-config] 07MacFan4000 reviewed pull request 03#4594 commit - 13https://github.com/miraheze/mw-config/pull/4594#discussion_r851773956 [15:54:39] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.91, 3.19, 2.91 [15:55:15] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.20, 3.92, 3.30 [15:56:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 4.00, 3.95, 3.84 [15:57:15] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.84, 3.36, 3.16 [15:58:07] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:58:33] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.88, 1.51, 1.86 [16:01:32] PROBLEM - cp30 Disk Space on cp30 is CRITICAL: DISK CRITICAL - free space: / 2328 MB (5% inode=96%); [16:02:20] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.42, 10.30, 9.23 [16:03:32] PROBLEM - cp30 Disk Space on cp30 is WARNING: DISK WARNING - free space: / 2420 MB (6% inode=96%); [16:04:18] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.60, 9.63, 9.12 [16:04:33] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.59, 1.08, 1.58 [16:06:39] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.05, 3.51, 3.04 [16:06:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.01, 3.77, 3.73 [16:08:02] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.11, 3.38, 3.15 [16:09:59] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.19, 3.25, 3.13 [16:10:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.36, 3.78, 3.75 [16:12:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/ab2fbfcfcc85 [16:12:10] [02miraheze/puppet] 07paladox 03ab2fbfc - varnish: Set large_objects_cutoff and nuke_limit [16:12:12] [02puppet] 07paladox created branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [16:12:13] [02puppet] 07paladox opened pull request 03#2495: varnish: Set large_objects_cutoff and nuke_limit - 13https://github.com/miraheze/puppet/pull/2495 [16:12:39] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.80, 3.54, 3.25 [16:13:12] [02puppet] 07paladox edited pull request 03#2495: varnish: Set large_objects_cutoff and nuke_limit - 13https://github.com/miraheze/puppet/pull/2495 [16:14:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ab2fbfcfcc85...b797dd2874b2 [16:14:26] [02miraheze/puppet] 07paladox 03b797dd2 - Update varnish.systemd.erb [16:14:28] [02puppet] 07paladox synchronize pull request 03#2495: varnish: Set large_objects_cutoff and nuke_limit - 13https://github.com/miraheze/puppet/pull/2495 [16:14:39] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.63, 2.94, 3.07 [16:14:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.55, 4.04, 3.85 [16:19:40] PROBLEM - cp21 Disk Space on cp21 is WARNING: DISK WARNING - free space: / 4287 MB (10% inode=96%); [16:22:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.94, 3.76, 3.91 [16:23:07] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [16:24:30] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.90, 3.45, 3.24 [16:25:31] PROBLEM - cp30 Disk Space on cp30 is CRITICAL: DISK CRITICAL - free space: / 2315 MB (5% inode=96%); [16:26:27] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.84, 3.29, 3.21 [16:26:28] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.52, 1.76, 1.28 [16:28:28] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.67, 1.37, 1.19 [16:28:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.44, 3.65, 3.77 [16:28:49] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.96, 10.10, 9.24 [16:30:45] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.14, 9.44, 9.11 [16:32:10] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=frozenplumwiki --report 1 --no-updates --username-prefix=mh:dpl3 /home/reception/dpl3wiki_xml_5175727f090296602885.xml (START) [16:32:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:33:15] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.49, 4.18, 3.67 [16:33:34] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [16:33:34] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [16:34:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.52, 3.67, 3.75 [16:34:48] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=frozenplumwiki --report 1 --no-updates --username-prefix=mh:dpl3 /home/reception/dpl3wiki_xml_5175727f090296602885.xml (END - exit=0) [16:34:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:35:15] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.40, 3.91, 3.62 [16:35:17] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=frozenplumwiki (START) [16:35:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:37:15] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.60, 4.12, 3.73 [16:37:57] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=politicsaminowiki --report 1 --username-prefix=wikia:politics-amino --no-updates /home/reception/politicsamino_pages_full.xml (START) [16:38:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:38:39] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=frozenplumwiki (END - exit=0) [16:38:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:39:15] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.43, 3.97, 3.73 [16:40:07] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=politicsaminowiki --report 1 --username-prefix=wikia:politics-amino --no-updates /home/reception/politicsamino_pages_full.xml (END - exit=0) [16:40:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:41:06] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=politicsaminowiki (START) [16:41:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:41:15] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.73, 4.24, 3.85 [16:42:39] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.78, 3.50, 3.07 [16:43:15] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.76, 3.65, 3.68 [16:44:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.71, 3.80, 3.72 [16:45:17] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=politicsaminowiki (END - exit=0) [16:45:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:46:02] [02puppet] 07paladox closed pull request 03#2495: varnish: Set large_objects_cutoff and nuke_limit - 13https://github.com/miraheze/puppet/pull/2495 [16:46:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/3bdbd8142abc...e7575f8bcd9e [16:46:05] [02miraheze/puppet] 07paladox 03e7575f8 - varnish: Set large_objects_cutoff and nuke_limit (#2495) [16:46:07] [02puppet] 07paladox deleted branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [16:46:08] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-10 [16:46:39] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.40, 3.97, 3.35 [16:46:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.62, 3.66, 3.68 [16:48:39] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.87, 3.52, 3.25 [16:48:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.17, 4.27, 3.90 [16:49:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/a5ec1e296d7d [16:49:37] [02miraheze/puppet] 07paladox 03a5ec1e2 - varnish: lower http_req_size [16:49:38] [02puppet] 07paladox created branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [16:49:40] [02puppet] 07paladox opened pull request 03#2496: varnish: lower http_req_size - 13https://github.com/miraheze/puppet/pull/2496 [16:49:51] [02puppet] 07paladox closed pull request 03#2496: varnish: lower http_req_size - 13https://github.com/miraheze/puppet/pull/2496 [16:49:53] [02puppet] 07paladox deleted branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [16:49:54] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-10 [16:49:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e7575f8bcd9e...0a047026c516 [16:49:57] [02miraheze/puppet] 07paladox 030a04702 - varnish: lower http_req_size (#2496) [16:50:39] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.28, 3.68, 3.33 [16:51:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/5d35b8f538fc [16:51:22] [02miraheze/puppet] 07paladox 035d35b8f - varnish: match listen_depth and net.core.soxmaxconn [16:51:24] [02puppet] 07paladox created branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [16:51:25] [02puppet] 07paladox opened pull request 03#2497: varnish: match listen_depth and net.core.soxmaxconn - 13https://github.com/miraheze/puppet/pull/2497 [16:52:39] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.97, 3.07, 3.16 [16:53:19] [02puppet] 07paladox edited pull request 03#2497: varnish: match listen_depth and net.core.soxmaxconn - 13https://github.com/miraheze/puppet/pull/2497 [16:53:26] [02puppet] 07paladox edited pull request 03#2497: varnish: match listen_depth and net.core.somaxconn - 13https://github.com/miraheze/puppet/pull/2497 [16:53:45] [02puppet] 07paladox closed pull request 03#2497: varnish: match listen_depth and net.core.somaxconn - 13https://github.com/miraheze/puppet/pull/2497 [16:53:47] [02puppet] 07paladox deleted branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [16:53:48] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0a047026c516...5607b7f4770c [16:53:50] [02miraheze/puppet] 07paladox 035607b7f - varnish: match listen_depth and net.core.soxmaxconn (#2497) [16:53:51] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-10 [16:55:15] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.04, 2.86, 3.30 [16:58:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/0284b9a0d2a2 [16:58:21] [02miraheze/puppet] 07paladox 030284b9a - varnish: Set TasksMax to infinity [16:58:23] [02puppet] 07paladox created branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [16:58:25] [02puppet] 07paladox opened pull request 03#2498: varnish: Set TasksMax to infinity - 13https://github.com/miraheze/puppet/pull/2498 [16:58:32] [02puppet] 07paladox closed pull request 03#2498: varnish: Set TasksMax to infinity - 13https://github.com/miraheze/puppet/pull/2498 [16:58:33] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-10 [16:58:35] [02puppet] 07paladox deleted branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [16:58:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5607b7f4770c...0fa50bc07eca [16:58:38] [02miraheze/puppet] 07paladox 030fa50bc - varnish: Set TasksMax to infinity (#2498) [16:59:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.53, 9.87, 9.20 [17:02:46] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [17:02:49] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/0f1a9f316f7c [17:02:50] [02miraheze/puppet] 07paladox 030f1a9f3 - varnish: set LimitRTPRIO to infinity [17:02:52] [02puppet] 07paladox created branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [17:02:53] [02puppet] 07paladox opened pull request 03#2499: varnish: set LimitRTPRIO to infinity - 13https://github.com/miraheze/puppet/pull/2499 [17:02:59] [02puppet] 07paladox closed pull request 03#2499: varnish: set LimitRTPRIO to infinity - 13https://github.com/miraheze/puppet/pull/2499 [17:03:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0fa50bc07eca...b7bf1aaddd13 [17:03:02] [02miraheze/puppet] 07paladox 03b7bf1aa - varnish: set LimitRTPRIO to infinity (#2499) [17:03:03] [02puppet] 07paladox deleted branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [17:03:05] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-10 [17:03:16] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.98, 9.50, 9.25 [17:03:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/48d37301aead [17:03:34] [02miraheze/puppet] 07paladox 0348d3730 - varnish: increase LimitMEMLOCK to 90000 [17:03:36] [02puppet] 07paladox created branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [17:03:37] [02puppet] 07paladox opened pull request 03#2500: varnish: increase LimitMEMLOCK to 90000 - 13https://github.com/miraheze/puppet/pull/2500 [17:03:40] [02puppet] 07paladox closed pull request 03#2500: varnish: increase LimitMEMLOCK to 90000 - 13https://github.com/miraheze/puppet/pull/2500 [17:03:41] [02puppet] 07paladox deleted branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [17:03:43] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-10 [17:03:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b7bf1aaddd13...0ece55ae473a [17:03:46] [02miraheze/puppet] 07paladox 030ece55a - varnish: increase LimitMEMLOCK to 90000 (#2500) [17:05:09] PROBLEM - cp31 Disk Space on cp31 is CRITICAL: DISK CRITICAL - free space: / 2242 MB (5% inode=96%); [17:06:39] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.33, 3.55, 3.23 [17:07:09] PROBLEM - cp31 Disk Space on cp31 is WARNING: DISK WARNING - free space: / 2393 MB (6% inode=96%); [17:07:15] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.42, 3.96, 3.58 [17:08:39] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.04, 3.87, 3.40 [17:08:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/c4e426e092f7 [17:08:41] [02miraheze/puppet] 07paladox 03c4e426e - varnish: increase LimitNOFILE to 500000 [17:08:42] [02puppet] 07paladox created branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [17:08:44] [02puppet] 07paladox opened pull request 03#2501: varnish: increase LimitNOFILE to 500000 - 13https://github.com/miraheze/puppet/pull/2501 [17:08:50] [02puppet] 07paladox closed pull request 03#2501: varnish: increase LimitNOFILE to 500000 - 13https://github.com/miraheze/puppet/pull/2501 [17:08:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0ece55ae473a...8fddecf0afb5 [17:08:53] [02miraheze/puppet] 07paladox 038fddecf - varnish: increase LimitNOFILE to 500000 (#2501) [17:08:55] [02puppet] 07paladox deleted branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [17:08:56] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-10 [17:09:15] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.00, 3.60, 3.49 [17:09:30] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.84, 10.68, 9.77 [17:09:57] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/87e48af9c9e5...8912ad3bd2f7 [17:09:59] [02miraheze/dns] 07paladox 038912ad3 - depool gb [17:12:31] !log cp20/21: restart varnish [17:12:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:13:15] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.86, 4.34, 3.77 [17:13:18] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.22, 9.46, 9.49 [17:13:24] [02miraheze/dns] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://github.com/miraheze/dns/compare/8912ad3bd2f7...960408938990 [17:13:26] [02miraheze/dns] 07paladox 0367a4280 - Revert "depool gb" [17:13:27] [02miraheze/dns] 07paladox 039604089 - depool ca [17:14:27] !log cp30/31: restart varnish [17:14:45] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/960408938990...ea691b50c119 [17:14:47] [02miraheze/dns] 07paladox 03ea691b5 - Revert "depool ca" [17:16:39] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:16:47] PROBLEM - cp20 Stunnel HTTP for mw102 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:16:59] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:17:09] PROBLEM - cp31 Disk Space on cp31 is CRITICAL: DISK CRITICAL - free space: / 2344 MB (5% inode=96%); [17:17:16] PROBLEM - cp20 Stunnel HTTP for mw122 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:17:25] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:17:25] PROBLEM - mw102 MediaWiki Rendering on mw102 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:17:25] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:17:29] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:17:30] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:17:38] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:17:40] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:17:54] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:18:09] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:18:10] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:18:14] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:18:18] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:18:20] PROBLEM - cp31 Stunnel HTTP for mw101 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:18:25] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:18:26] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:18:29] PROBLEM - cp30 Stunnel HTTP for mw122 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:18:31] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:18:33] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:18:35] PROBLEM - cp31 Stunnel HTTP for mw122 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:18:49] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:19:28] paladox: expected ^ [17:19:29] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 9.501 second response time [17:19:31] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 7.563 second response time [17:19:36] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:19:42] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 7.081 second response time [17:19:45] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:19:47] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:19:47] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 8.281 second response time [17:19:56] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:20:15] i think that's the cache warming up [17:20:21] RECOVERY - cp31 Stunnel HTTP for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 6.861 second response time [17:20:29] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 4.960 second response time [17:20:41] RECOVERY - cp31 Stunnel HTTP for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 6.528 second response time [17:20:46] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 1.322 second response time [17:21:09] PROBLEM - cp31 Disk Space on cp31 is WARNING: DISK WARNING - free space: / 2349 MB (6% inode=96%); [17:21:33] [02puppet] 07paladox closed pull request 03#2494: varnish: tweak systemd unit - 13https://github.com/miraheze/puppet/pull/2494 [17:21:36] [02puppet] 07paladox deleted branch 03paladox-patch-9 - 13https://github.com/miraheze/puppet [17:21:37] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-9 [17:21:42] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 9.988 second response time [17:21:48] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 5.991 second response time [17:21:51] PROBLEM - cp30 Stunnel HTTP for mw102 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:21:58] PROBLEM - cp21 Stunnel HTTP for mw102 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:22:18] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 8.364 second response time [17:22:19] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 7.479 second response time [17:22:30] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 9.979 second response time [17:22:32] paladox: lovely [17:22:46] PROBLEM - cp31 Varnish Backends on cp31 is CRITICAL: 1 backends are down. mw121 [17:23:09] paladox: maybe let's not depool both sites close together in future [17:23:38] PROBLEM - cp21 Stunnel HTTP for mw122 on cp21 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.799 second response time [17:23:43] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 3.870 second response time [17:23:46] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 1.157 second response time [17:23:50] ok [17:24:19] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.341 second response time [17:24:56] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:25:02] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.303 second response time [17:25:09] PROBLEM - cp31 Disk Space on cp31 is CRITICAL: DISK CRITICAL - free space: / 2339 MB (5% inode=96%); [17:25:20] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.019 second response time [17:25:33] RECOVERY - cp21 Stunnel HTTP for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.017 second response time [17:25:33] RECOVERY - cp20 Stunnel HTTP for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.014 second response time [17:25:57] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:26:06] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:26:08] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:26:12] PROBLEM - cp30 Varnish Backends on cp30 is CRITICAL: 2 backends are down. mw101 mw122 [17:26:14] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:26:15] PROBLEM - cp21 Stunnel HTTP for mw101 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:26:40] PROBLEM - cp20 Stunnel HTTP for mw112 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:26:41] PROBLEM - cp21 Stunnel HTTP for mw112 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:26:53] PROBLEM - cp31 Stunnel HTTP for mw112 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:26:55] RECOVERY - cp30 Stunnel HTTP for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.361 second response time [17:26:55] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.698 second response time [17:27:55] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.307 second response time [17:28:00] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 3.498 second response time [17:28:04] RECOVERY - cp30 Stunnel HTTP for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 3.390 second response time [17:28:05] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 0.316 second response time [17:28:05] RECOVERY - cp21 Stunnel HTTP for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 2.333 second response time [17:28:14] RECOVERY - cp21 Stunnel HTTP for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 0.017 second response time [17:28:45] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 12 backends are healthy [17:29:03] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 4.735 second response time [17:29:28] RECOVERY - cp20 Stunnel HTTP for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 0.042 second response time [17:30:12] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 12 backends are healthy [17:32:03] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=tintinwiki --report 1 --no-updates --username-prefix=wikia:tintin /home/reception/frtintinfr_pages_full.xml (START) [17:32:47] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 9.528 second response time [17:32:53] RECOVERY - cp21 Stunnel HTTP for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 6.182 second response time [17:32:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:32:55] RECOVERY - cp20 Stunnel HTTP for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 5.390 second response time [17:33:05] RECOVERY - cp31 Stunnel HTTP for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 4.058 second response time [17:33:07] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 2.995 second response time [17:38:22] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 5.181 second response time [17:38:38] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 6.488 second response time [17:38:51] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 5.403 second response time [17:39:05] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 5.836 second response time [17:39:37] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 5.690 second response time [18:01:34] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.67, 10.72, 8.93 [18:03:16] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.84, 9.97, 8.78 [18:03:34] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.95, 9.79, 8.82 [18:05:14] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.54, 9.06, 8.60 [18:05:50] PROBLEM - cp20 Disk Space on cp20 is WARNING: DISK WARNING - free space: / 4282 MB (10% inode=96%); [18:08:39] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=tintinwiki --report 1 --no-updates --username-prefix=wikia:tintin /home/reception/frtintinfr_pages_full.xml (END - exit=0) [18:09:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:13:26] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.58, 9.75, 8.89 [18:15:26] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.83, 9.56, 8.94 [18:17:24] !log. change incident reports 51/52 to 50/51 (using db query) [18:20:33] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.41, 1.81, 1.37 [18:20:39] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.01, 11.10, 10.02 [18:21:45] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.93, 10.43, 9.66 [18:22:33] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.29, 1.55, 1.32 [18:22:39] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.31, 11.20, 10.20 [18:23:39] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 10.18, 10.14, 9.63 [18:25:59] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.75, 10.29, 9.33 [18:27:50] paladox, re: Special:IncidentReports/c, do we not have some sort number validation check for incident reports to ensure that they are only numerals 0-9? [18:27:56] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.72, 10.07, 9.37 [18:28:06] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=tintinwiki (START) [18:28:14] i wouldn't know, maybe CosmicAlpha [18:28:17] *knows [18:28:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:29:50] Dmehus: well they aren't numbered manually, they are set to incremented numbers, so I'm not sure what you mean, it should never happen that it isn't a number. [18:31:48] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.16, 10.71, 9.78 [18:33:04] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:33:56] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:34:32] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:34:39] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.97, 9.93, 10.09 [18:35:57] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 6.477 second response time [18:36:34] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 2.671 second response time [18:37:08] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.337 second response time [18:37:11] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/IncidentReporting/commit/e51dad4c7984 [18:37:12] [02miraheze/IncidentReporting] 07Universal-Omega 03e51dad4 - Modernise IncidentReportingOOUIForm [18:37:14] [02IncidentReporting] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/IncidentReporting [18:37:15] [02IncidentReporting] 07Universal-Omega opened pull request 03#40: Modernise IncidentReportingOOUIForm - 13https://github.com/miraheze/IncidentReporting/pull/40 [18:38:26] CosmicAlpha, that's what I thought, too, but why did that incident report show as number `c`? [18:39:04] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.56, 9.88, 9.11 [18:40:00] CosmicAlpha: for clarity, this is what I am referring to: [18:40:00] ```On metawiki Paladox Paladox modified incident report for [[Special:IncidentReports/c]]; https://meta.miraheze.org/wiki/Special:Log/incidentreporting``` [18:40:00] https://meta.miraheze.org/wiki/Special:IncidentReports/c [18:40:00] ^ If that incident report wasn't numbered `c`, then it's an issue in the formatting of the log entries that appear in RC, I think [18:40:02] [url] All public logs - Miraheze Meta | meta.miraheze.org [18:40:52] That's very weird. I'll look shortly. [18:41:04] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.62, 9.53, 9.09 [18:41:21] miraheze/IncidentReporting - Universal-Omega the build passed. [18:41:34] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.67, 9.65, 9.78 [18:41:42] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.49, 11.39, 10.47 [18:43:36] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.86, 11.48, 10.61 [18:43:52] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/IncidentReporting/compare/e51dad4c7984...2e17f8654c25 [18:43:54] [02miraheze/IncidentReporting] 07Universal-Omega 032e17f86 - Update ext.incidentreporting.oouiform.ooui.less [18:43:55] [02IncidentReporting] 07Universal-Omega synchronize pull request 03#40: Modernise IncidentReportingOOUIForm - 13https://github.com/miraheze/IncidentReporting/pull/40 [18:47:24] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.03, 11.57, 10.82 [18:47:59] miraheze/IncidentReporting - Universal-Omega the build passed. [18:49:13] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.76, 11.18, 10.37 [18:49:18] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.84, 11.18, 10.76 [18:51:09] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.10, 10.34, 10.19 [18:51:11] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/IncidentReporting/compare/2e17f8654c25...9f0274b0d4e0 [18:51:13] [02miraheze/IncidentReporting] 07Universal-Omega 039f0274b - Update ext.incidentreporting.oouiform.ooui.js [18:51:14] [02IncidentReporting] 07Universal-Omega synchronize pull request 03#40: Modernise IncidentReportingOOUIForm - 13https://github.com/miraheze/IncidentReporting/pull/40 [18:53:16] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-1/±0] 13https://github.com/miraheze/IncidentReporting/compare/9f0274b0d4e0...7944021f86d7 [18:53:18] [02miraheze/IncidentReporting] 07Universal-Omega 037944021 - Delete ext.incidentreporting.oouiform.edit.ooui.js [18:53:19] [02IncidentReporting] 07Universal-Omega synchronize pull request 03#40: Modernise IncidentReportingOOUIForm - 13https://github.com/miraheze/IncidentReporting/pull/40 [18:54:10] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.33, 9.95, 9.88 [18:55:03] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.33, 9.95, 10.10 [18:56:02] miraheze/IncidentReporting - Universal-Omega the build passed. [18:56:06] miraheze/IncidentReporting - Universal-Omega the build has errored. [18:56:32] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/IncidentReporting/compare/7944021f86d7...f5d5710abf5a [18:56:34] [02miraheze/IncidentReporting] 07Universal-Omega 03f5d5710 - Update extension.json [18:56:35] [02IncidentReporting] 07Universal-Omega synchronize pull request 03#40: Modernise IncidentReportingOOUIForm - 13https://github.com/miraheze/IncidentReporting/pull/40 [18:58:05] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.98, 9.75, 9.84 [18:58:31] CosmicAlpha, cool thanks :) [18:58:40] .give CosmicAlpha cookie [18:59:35] * dmehus thinks MirahezeBot's commands are so unintuitive and we should rethink them as it shouldn't be a sub-command of `.goofy` [19:01:16] miraheze/IncidentReporting - Universal-Omega the build passed. [19:01:26] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/IncidentReporting/compare/f5d5710abf5a...fdf1d3cf71bb [19:01:28] [02miraheze/IncidentReporting] 07Universal-Omega 03fdf1d3c - Update SpecialIncidentReports.php [19:01:29] [02IncidentReporting] 07Universal-Omega synchronize pull request 03#40: Modernise IncidentReportingOOUIForm - 13https://github.com/miraheze/IncidentReporting/pull/40 [19:01:34] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.99, 10.25, 9.51 [19:01:44] dmehus: see PMs [19:01:48] .cookie dmehus [19:01:51] .help [19:01:51] RhinosF1: I've published a list of my commands at: https://sopel.mirahezebots.org/help_prod.html [19:02:14] We should add cookies [19:02:23] And yes I agree goofy is an awful plugin dmehus [19:02:33] Agent: he should also see his from me [19:03:34] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.86, 10.03, 9.51 [19:05:42] miraheze/IncidentReporting - Universal-Omega the build passed. [19:08:42] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.03, 11.07, 10.84 [19:10:32] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.91, 10.03, 9.31 [19:10:42] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.60, 10.99, 10.82 [19:11:14] paladox: ever seen https://phabricator.miraheze.org/T9091 before? [19:11:15] [url] ⚓ T9091 Exception when trying to connect to my account | phabricator.miraheze.org [19:14:23] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.10, 10.12, 9.55 [19:18:39] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.34, 10.62, 9.69 [19:20:39] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.69, 10.08, 9.63 [19:20:42] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/IncidentReporting/compare/fdf1d3cf71bb...a312a4812766 [19:20:44] [02miraheze/IncidentReporting] 07Universal-Omega 03a312a48 - Update IncidentReportingFormFactory.php [19:20:45] [02IncidentReporting] 07Universal-Omega synchronize pull request 03#40: Modernise IncidentReportingOOUIForm - 13https://github.com/miraheze/IncidentReporting/pull/40 [19:20:52] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.71, 1.62, 1.13 [19:22:49] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.19, 1.46, 1.13 [19:24:29] Reception123: I have. [19:24:59] CosmicAlpha: yeah, I just realised it happened to dmehus too before [19:25:04] but I don't think we ever found a real fix? [19:25:29] miraheze/IncidentReporting - Universal-Omega the build passed. [19:25:37] Reception123: try reset the user cache [19:26:18] guess we could try that though last time it seems Doug followed the exact instructions for cache reset and it worked [19:26:31] (because many times just clearing doesn't work, the private browser step also needs following) [19:26:35] Reception123: it is caused by betaheze, if you login there, then production breaks sometimes, and invalidate session for the user will need ran on both metawiki and betawiki. Or at least it is how I fixed it for me last time, and then it allowed me to login correctly to both. [19:26:42] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.82, 9.51, 10.03 [19:26:57] CosmicAlpha: ah, was Betaheze even a thing when it happened to Doug? [19:27:11] Yes. In its early stages I think? [19:27:15] that was a year ago, so I don't think so [19:27:19] It's caused by wired session cache. [19:27:23] Weird. [19:27:39] It's a session issue [19:27:39] So there are other possibilities for causing if that was messed up back then also. [19:27:45] Mediawiki is very stupid [19:28:07] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:28:38] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.09, 1.75, 1.34 [19:29:38] Reception123: you can try invalidating session cache for the specific user. It may work, may not but worth a try I guess. [19:29:49] ok, sure [19:30:35] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.96, 1.82, 1.41 [19:32:33] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.57, 1.66, 1.40 [19:32:39] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.53, 3.68, 3.98 [19:32:43] [02mw-config] 07dmehus reviewed pull request 03#4594 commit - 13https://github.com/miraheze/mw-config/pull/4594#discussion_r851795528 [19:33:52] [02mw-config] 07RhinosF1 reviewed pull request 03#4594 commit - 13https://github.com/miraheze/mw-config/pull/4594#discussion_r851795633 [19:36:39] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.47, 4.23, 4.13 [19:36:51] [02mw-config] 07dmehus reviewed pull request 03#4594 commit - 13https://github.com/miraheze/mw-config/pull/4594#discussion_r851795930 [19:37:42] Reception123: don't think so [19:38:07] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:38:15] [02mw-config] 07RhinosF1 reviewed pull request 03#4594 commit - 13https://github.com/miraheze/mw-config/pull/4594#discussion_r851796037 [19:41:00] [02mw-config] 07dmehus commented on pull request 03#4594: [#T9071](https://phabricator.miraheze.org/T9071) - 13https://github.com/miraheze/mw-config/pull/4594#issuecomment-1100938419 [19:42:42] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.09, 10.15, 9.83 [19:44:42] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.85, 9.68, 9.70 [19:44:43] [02mw-config] 07dmehus reviewed pull request 03#4594 commit - 13https://github.com/miraheze/mw-config/pull/4594#discussion_r851796677 [19:46:33] [02mw-config] 07RhinosF1 commented on pull request 03#4594: [#T9071](https://phabricator.miraheze.org/T9071) - 13https://github.com/miraheze/mw-config/pull/4594#issuecomment-1100939208 [19:47:59] [02mw-config] 07RhinosF1 reviewed pull request 03#4594 commit - 13https://github.com/miraheze/mw-config/pull/4594#discussion_r851797011 [19:48:42] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.47, 11.17, 10.34 [19:50:09] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.34, 9.92, 9.49 [19:51:26] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.69, 10.57, 9.73 [19:53:06] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.20, 1.63, 1.35 [19:53:26] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.58, 10.18, 9.70 [19:54:03] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.02, 9.86, 9.59 [19:54:42] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.18, 9.55, 9.90 [19:56:18] !log delete incident 50 and rename 51 to 50 and also reset auto incremeant [19:56:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:57:59] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.33, 10.12, 9.48 [19:59:57] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.03, 9.54, 9.33 [20:00:03] [02mw-config] 07dmehus commented on pull request 03#4594: [#T9071](https://phabricator.miraheze.org/T9071) - 13https://github.com/miraheze/mw-config/pull/4594#issuecomment-1100940922 [20:00:28] [02mw-config] 07dmehus edited a comment on pull request 03#4594: [#T9071](https://phabricator.miraheze.org/T9071) - 13https://github.com/miraheze/mw-config/pull/4594#issuecomment-1100940922 [20:00:55] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.39, 1.83, 1.63 [20:01:39] [02mw-config] 07dmehus reviewed pull request 03#4594 commit - 13https://github.com/miraheze/mw-config/pull/4594#discussion_r851798196 [20:02:00] [02mw-config] 07dmehus reviewed pull request 03#4594 commit - 13https://github.com/miraheze/mw-config/pull/4594#discussion_r851798223 [20:02:46] [02mw-config] 07RhinosF1 commented on pull request 03#4594: [#T9071](https://phabricator.miraheze.org/T9071) - 13https://github.com/miraheze/mw-config/pull/4594#issuecomment-1100941311 [20:02:52] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.21, 1.62, 1.58 [20:08:07] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:10:30] !log delete IncidentReport #50 related today, no incident or value added. [20:10:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:12:02] Hi [20:13:24] hi, SOUUUP. I will check your DMs shortly [20:13:41] Where [20:13:52] from yesterday or a couple days ago? [20:14:01] U mean my email [20:14:05] i sent one [20:14:08] ok [20:14:15] I can check for that too then [20:14:21] Mk [20:14:26] where dms [20:16:45] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.62, 11.32, 10.21 [20:17:15] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.23, 3.09, 3.85 [20:17:16] [02mw-config] 07dmehus synchronize pull request 03#4594: [#T9071](https://phabricator.miraheze.org/T9071) - 13https://github.com/miraheze/mw-config/pull/4594 [20:17:34] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.55, 10.17, 9.37 [20:18:03] Reception123: [20:18:07] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:18:17] please review ^^ at some point [20:18:31] miraheze/mw-config - dmehus the build passed. [20:18:39] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.14, 3.64, 3.96 [20:18:40] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.98, 11.12, 10.26 [20:19:15] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.21, 3.37, 3.84 [20:19:34] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.25, 9.29, 9.14 [20:20:48] [02mw-config] 07MacFan4000 commented on pull request 03#4594: [#T9071](https://phabricator.miraheze.org/T9071) - 13https://github.com/miraheze/mw-config/pull/4594#issuecomment-1100943885 [20:21:15] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.06, 3.17, 3.71 [20:22:31] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.19, 9.72, 9.87 [20:22:36] [02mw-config] 07dmehus synchronize pull request 03#4594: [#T9071](https://phabricator.miraheze.org/T9071) - 13https://github.com/miraheze/mw-config/pull/4594 [20:23:11] [02mw-config] 07dmehus commented on pull request 03#4594: [#T9071](https://phabricator.miraheze.org/T9071) - 13https://github.com/miraheze/mw-config/pull/4594#issuecomment-1100944229 [20:23:38] [02mw-config] 07MacFan4000 commented on pull request 03#4594: [#T9071](https://phabricator.miraheze.org/T9071) - 13https://github.com/miraheze/mw-config/pull/4594#issuecomment-1100944261 [20:23:41] miraheze/mw-config - dmehus the build passed. [20:23:50] will probably look tomorrow, as there's quite a bit to read.. [20:24:30] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.52, 10.37, 9.91 [20:26:26] chrs: https://github.com/miraheze/mw-config/pull/4594 can be reviewed [20:26:27] [url] [#T9071](https://phabricator.miraheze.org/T9071) by dmehus · Pull Request #4594 · miraheze/mw-config · GitHub | github.com [20:27:15] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.05, 3.42, 3.60 [20:28:26] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.95, 9.97, 9.88 [20:28:39] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.02, 3.48, 3.66 [20:28:41] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=tintinwiki (END - exit=2) [20:28:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:29:15] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.90, 3.29, 3.54 [20:29:26] E [20:29:44] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/invalidateUserSessions.php --wiki=metawiki --user=HeartsDo (END - exit=0) [20:29:45] So long King SOUUUP-er. [20:29:51] !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/invalidateUserSessions.php --wiki=loginwiki --user=HeartsDo (END - exit=0) [20:29:52] M8 [20:29:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:29:58] .kb SOUUUP not helpful [20:29:59] Please wait... [20:30:06] darkmatterman450: please don't [20:30:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:30:39] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.35, 3.46, 3.64 [20:30:56] RhinosF1 Sorry. I was referencing Super Mario 64. :/ [20:31:15] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.12, 2.95, 3.39 [20:31:19] RhinosF1: don't sysadmins have `centralauth-rename` globally? [20:31:39] We do, and I think the plan is to remove that [20:31:55] ah [20:32:10] chrs: I don't think we need it [20:32:17] CosmicAlpha might have in past [20:32:27] darkmatterman450: yeah just don't encourage them [20:33:12] alright so I assume `oathauth-api-all` is just getting removed, what's happening with `oathauth-verify-user`? [20:34:05] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.31, 10.30, 9.96 [20:34:39] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.53, 2.97, 3.40 [20:35:17] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.58, 11.04, 10.37 [20:35:27] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.13, 10.25, 9.90 [20:35:47] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.47, 10.49, 9.63 [20:35:48] chrs, will update you privately [20:36:01] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.07, 9.79, 9.81 [20:36:16] chrs: api-all is dropped [20:36:25] dmehus wanted verify globally [20:36:27] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.31, 10.87, 9.84 [20:36:38] sysadmin is only having it on meta [20:37:15] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.96, 10.04, 10.07 [20:37:44] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.29, 9.45, 9.34 [20:38:25] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.01, 10.13, 9.70 [20:40:39] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.81, 4.03, 3.73 [20:41:15] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.68, 3.78, 3.55 [20:44:56] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.39, 9.12, 9.71 [20:47:15] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.66, 4.25, 3.80 [21:01:19] [02mw-config] 07dmehus synchronize pull request 03#4594: [#T9071](https://phabricator.miraheze.org/T9071) - 13https://github.com/miraheze/mw-config/pull/4594 [21:02:20] miraheze/mw-config - dmehus the build passed. [21:07:49] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.70, 9.62, 9.49 [21:08:39] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.29, 3.46, 3.89 [21:09:44] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.19, 9.37, 9.41 [21:10:32] [02mw-config] 07Universal-Omega synchronize pull request 03#4599: Add SemanticMediaWiki to ManageWikiExtensions - 13https://github.com/miraheze/mw-config/pull/4599 [21:10:33] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/20bff883a6cf...91f8204059fb [21:10:35] [02miraheze/mw-config] 07Universal-Omega 0391f8204 - Update ManageWikiExtensions.php [21:11:39] miraheze/mw-config - Universal-Omega the build passed. [21:12:08] PROBLEM - cloud12 Puppet on cloud12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [21:12:39] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.03, 4.00, 3.96 [21:13:07] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:16:29] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/91f8204059fb...e07f772073f3 [21:16:31] [02miraheze/mw-config] 07Universal-Omega 03e07f772 - Update ManageWikiExtensions.php [21:16:33] [02mw-config] 07Universal-Omega synchronize pull request 03#4599: Add SemanticMediaWiki to ManageWikiExtensions - 13https://github.com/miraheze/mw-config/pull/4599 [21:16:39] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.00, 3.33, 3.74 [21:17:15] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.29, 3.36, 3.97 [21:17:34] miraheze/mw-config - Universal-Omega the build passed. [21:18:07] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:19:15] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.70, 3.69, 4.00 [21:20:39] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.22, 9.93, 9.31 [21:21:15] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.30, 3.59, 3.93 [21:22:39] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.13, 9.34, 9.17 [21:24:39] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.52, 3.32, 3.50 [21:25:15] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.43, 3.59, 3.81 [21:26:06] CosmicAlpha, wow, you've already got SemanticMediaWiki working without any issues, and you're already to the stake of adding it to `ManageWikiExtensions.php`? [21:26:39] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.05, 3.31, 3.48 [21:27:04] dmehus: yep, just running some more tests on it first, but seems to run smoothly without issues on semantic-mediawiki.betaheze.org. [21:27:15] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.49, 3.59, 3.79 [21:29:15] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.03, 4.24, 4.00 [21:31:15] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.16, 3.72, 3.83 [21:31:24] [02mw-config] 07Universal-Omega synchronize pull request 03#4599: Add SemanticMediaWiki to ManageWikiExtensions - 13https://github.com/miraheze/mw-config/pull/4599 [21:31:25] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/e07f772073f3...8ed612251ab7 [21:31:27] [02miraheze/mw-config] 07Universal-Omega 038ed6122 - Remove smw-admin [21:32:23] miraheze/mw-config - Universal-Omega the build passed. [21:33:29] [02mw-config] 07Universal-Omega synchronize pull request 03#4599: Add SemanticMediaWiki to ManageWikiExtensions - 13https://github.com/miraheze/mw-config/pull/4599 [21:33:30] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/8ed612251ab7...663e663cefaf [21:33:32] [02miraheze/mw-config] 07Universal-Omega 03663e663 - Disallow smwadministrator [21:34:34] miraheze/mw-config - Universal-Omega the build passed. [21:34:39] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.26, 4.02, 3.66 [21:35:15] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.88, 4.45, 4.04 [21:36:39] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.08, 3.60, 3.55 [21:36:45] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.64, 10.32, 9.86 [21:38:36] [02mw-config] 07Universal-Omega synchronize pull request 03#4599: Add SemanticMediaWiki to ManageWikiExtensions - 13https://github.com/miraheze/mw-config/pull/4599 [21:38:37] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/663e663cefaf...1f97bfd09150 [21:38:39] [02miraheze/mw-config] 07Universal-Omega 031f97bfd - Fix smw/schema and Rule namespace ids [21:39:38] miraheze/mw-config - Universal-Omega the build passed. [21:40:08] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:40:39] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.32, 4.13, 3.71 [21:44:27] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.97, 9.98, 9.97 [21:45:15] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.08, 3.84, 3.92 [21:46:39] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.63, 3.81, 3.69 [21:47:15] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.84, 4.06, 4.01 [21:50:39] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.21, 10.23, 9.55 [21:53:15] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.89, 3.71, 3.92 [21:53:29] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.32, 10.07, 9.50 [21:54:39] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.25, 3.69, 3.65 [21:55:15] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.63, 4.04, 4.03 [21:55:26] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.08, 9.50, 9.35 [21:58:39] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.02, 9.61, 9.60 [22:04:39] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.56, 3.57, 3.74 [22:10:39] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.53, 4.36, 4.01 [22:14:39] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.03, 3.83, 3.88 [22:17:15] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.59, 3.25, 3.90 [22:18:07] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:18:39] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.70, 4.39, 4.05 [22:19:15] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.24, 4.03, 4.11 [22:28:07] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:28:39] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.83, 3.50, 3.80 [22:30:39] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 3.99, 4.02, 3.97 [22:32:39] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.90, 3.78, 3.90 [22:34:58] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [22:34:58] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [22:39:15] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.17, 3.54, 3.96 [22:40:42] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.01, 9.90, 9.04 [22:41:12] CosmicAlpha, cool, I'll check it out. Wish I could login though :P [22:42:39] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.90, 2.71, 3.39 [22:42:42] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.41, 9.40, 8.95 [22:42:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.48, 3.30, 3.93 [22:42:48] What's the central login wiki for Betaheze? [22:45:15] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.09, 2.56, 3.37 [22:46:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.49, 3.45, 3.80 [22:48:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.98, 3.33, 3.72 [22:52:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.16, 3.85, 3.84 [22:54:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.62, 3.38, 3.67 [22:58:43] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.70, 2.94, 3.40 [23:00:02] !log [@test101] starting deploy of {'l10nupdate': True} to all [23:00:03] !log [@mwtask111] starting deploy of {'l10nupdate': True} to all [23:00:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:04:10] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [23:08:07] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:10:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.90, 3.73, 3.43 [23:12:08] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.81, 1.99, 1.46 [23:12:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.95, 3.56, 3.40 [23:13:13] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.36, 3.60, 3.10 [23:14:14] !log [@test101] finished deploy of {'l10nupdate': True} to all - SUCCESS in 852s [23:14:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:14:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.15, 4.02, 3.58 [23:15:09] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.95, 3.24, 3.02 [23:16:08] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.79, 1.46, 1.38 [23:16:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.19, 3.67, 3.51 [23:18:05] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.88, 3.48, 3.21 [23:18:43] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.86, 3.22, 3.35 [23:20:02] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.25, 3.04, 3.08 [23:23:24] !log [@mwtask111] finished deploy of {'l10nupdate': True} to all - SUCCESS in 1400s [23:23:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:24:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.82, 3.89, 3.61 [23:27:11] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.07, 9.73, 8.82 [23:28:12] dmehus: betawiki. [23:28:22] beta.betaheze.org. [23:28:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.01, 4.07, 3.71 [23:29:10] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.23, 9.55, 8.87 [23:30:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.21, 3.73, 3.63 [23:36:43] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.40, 2.98, 3.33 [23:39:29] CosmicAlpha, ah okay, was worried you'd say login.miraheze.org, in which case I'd say, "uhh..." [23:48:07] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:51:28] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.43, 4.04, 3.49 [23:54:02] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.08, 3.46, 3.09 [23:55:58] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.41, 3.27, 3.06 [23:58:07] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:59:07] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.82, 3.89, 3.73 [23:59:52] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.21, 2.96, 3.01