[00:00:25] miraheze/mw-config - Universal-Omega the build passed. [00:00:35] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:01:12] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 594 failures. Last run 15 seconds ago with 594 failures. Failed resources (up to 3 shown): File[/etc/modprobe.d/nf_conntrack.conf],File[/usr/lib/nagios/plugins/check_conntrack],File[/usr/lib/nagios/plugins/check_ferm],File[/usr/lib/nagios/plugins/check_smart] [00:02:29] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-8 [+0/-0/±1] 13https://git.io/J62hX [00:02:31] [02miraheze/mw-config] 07Universal-Omega 03447a8b0 - Update ManageWikiExtensions.php [00:02:32] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-8 - 13https://git.io/vbvb3 [00:02:49] [02mw-config] 07Universal-Omega opened pull request 03#4168: T8059: Add Flatbox - 13https://git.io/J62jL [00:03:03] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.88, 4.86, 2.93 [00:03:20] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-8 [+0/-0/±1] 13https://git.io/J62j7 [00:03:22] [02miraheze/mw-config] 07Universal-Omega 032d0d5d1 - Update extension-list [00:03:23] [02mw-config] 07Universal-Omega synchronize pull request 03#4168: T8059: Add Flatbox - 13https://git.io/J62jL [00:03:27] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 30.83, 21.59, 16.08 [00:03:41] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.43, 3.58, 2.97 [00:03:59] miraheze/mw-config - Universal-Omega the build passed. [00:04:06] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.76, 6.45, 4.67 [00:04:38] miraheze/mw-config - Universal-Omega the build passed. [00:05:09] PROBLEM - graylog2 Current Load on graylog2 is CRITICAL: CRITICAL - load average: 4.40, 3.05, 1.80 [00:05:13] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 12.80, 7.94, 5.15 [00:05:23] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:05:35] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.83, 3.48, 2.99 [00:05:43] !log [universalomega@mw11] starting deploy of {'extensionlist': True} to all [00:05:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:05:52] !log [universalomega@mw11] finished deploy of {'extensionlist': True} to all - SUCCESS in 9s [00:05:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:06:05] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.61, 6.43, 4.89 [00:06:32] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.45, 7.73, 5.62 [00:07:00] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.06, 4.31, 3.19 [00:07:05] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 2.61, 2.80, 1.86 [00:07:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6aTC [00:07:08] [02miraheze/puppet] 07paladox 03b11fadd - redis: Migrate to ferm [00:07:09] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [00:07:11] [02puppet] 07paladox opened pull request 03#2064: redis: Migrate to ferm - 13https://git.io/J6aT8 [00:07:14] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.82, 7.71, 5.40 [00:07:16] [02puppet] 07paladox closed pull request 03#2064: redis: Migrate to ferm - 13https://git.io/J6aT8 [00:07:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J6aTg [00:07:19] [02miraheze/puppet] 07paladox 037eba139 - redis: Migrate to ferm (#2064) [00:07:21] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [00:07:22] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [00:07:24] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.75, 22.71, 17.86 [00:07:29] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.29, 3.09, 2.91 [00:07:49] PROBLEM - mw12 Puppet on mw12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:08:32] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.87, 6.46, 5.43 [00:09:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J6aLl [00:09:19] [02miraheze/puppet] 07paladox 03a4f11ea - redis: Set notrack for firewall [00:09:23] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.82, 24.31, 19.00 [00:09:46] RECOVERY - mw12 Puppet on mw12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:11:13] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.40, 6.40, 5.38 [00:11:22] RECOVERY - mw13 Puppet on mw13 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [00:11:32] PROBLEM - mwtask1 Puppet on mwtask1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:12:10] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-7 [+0/-0/±1] 13https://git.io/J6aYc [00:12:11] [02miraheze/mw-config] 07Universal-Omega 0387f9039 - Update ManageWikiExtensions.php [00:12:13] [02mw-config] 07Universal-Omega synchronize pull request 03#4167: T5477: Add RatePage - 13https://git.io/J62bn [00:13:13] miraheze/mw-config - Universal-Omega the build passed. [00:15:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J6aGS [00:15:32] [02miraheze/puppet] 07paladox 031f5c8ea - letsencrypt: Migrate to ferm [00:15:47] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.65, 6.54, 4.57 [00:17:32] RECOVERY - mwtask1 Puppet on mwtask1 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [00:19:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J6aWV [00:19:17] [02miraheze/puppet] 07paladox 03a107a85 - jobchron1: Set firewall to ferm [00:19:43] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.68, 5.95, 4.76 [00:20:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J6a8P [00:20:43] [02miraheze/puppet] 07paladox 030382e33 - Migrate redis_exporter to ferm [00:23:31] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.33, 6.55, 5.25 [00:24:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6aEy [00:24:35] [02miraheze/puppet] 07paladox 035ac1b9f - Migrate memcached to ferm [00:24:37] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [00:24:38] [02puppet] 07paladox opened pull request 03#2065: Migrate memcached to ferm - 13https://git.io/J6aE9 [00:25:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6au7 [00:25:22] [02miraheze/puppet] 07paladox 038422e07 - Update memcached_exporter.pp [00:25:24] [02puppet] 07paladox synchronize pull request 03#2065: Migrate memcached to ferm - 13https://git.io/J6aE9 [00:25:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6azL [00:25:38] [02miraheze/puppet] 07paladox 03b83ec50 - Update mem1.yaml [00:25:40] [02puppet] 07paladox synchronize pull request 03#2065: Migrate memcached to ferm - 13https://git.io/J6aE9 [00:25:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6azZ [00:25:46] [02miraheze/puppet] 07paladox 03c1e02fe - Update mem2.yaml [00:25:48] [02puppet] 07paladox synchronize pull request 03#2065: Migrate memcached to ferm - 13https://git.io/J6aE9 [00:26:21] [02puppet] 07paladox closed pull request 03#2065: Migrate memcached to ferm - 13https://git.io/J6aE9 [00:26:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/J6agq [00:26:24] [02miraheze/puppet] 07paladox 03fd1b8b7 - Migrate memcached to ferm (#2065) [00:26:26] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [00:26:27] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [00:28:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J6aay [00:28:15] [02miraheze/puppet] 07paladox 03a4df5be - Set notrack for memcached [00:34:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6aXy [00:34:56] [02miraheze/puppet] 07paladox 03da464ae - Migrate dns to ferm [00:34:58] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [00:34:59] [02puppet] 07paladox opened pull request 03#2066: Migrate dns to ferm - 13https://git.io/J6aXQ [00:35:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6a13 [00:35:19] [02miraheze/puppet] 07paladox 03c9cba2a - Update ns1.yaml [00:35:21] [02puppet] 07paladox synchronize pull request 03#2066: Migrate dns to ferm - 13https://git.io/J6aXQ [00:35:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6a1E [00:35:26] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.58, 5.56, 5.65 [00:35:27] [02miraheze/puppet] 07paladox 03936c47d - Update ns2.yaml [00:35:29] [02puppet] 07paladox synchronize pull request 03#2066: Migrate dns to ferm - 13https://git.io/J6aXQ [00:36:11] [02puppet] 07paladox closed pull request 03#2066: Migrate dns to ferm - 13https://git.io/J6aXQ [00:36:13] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [00:36:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/J6aMw [00:36:16] [02miraheze/puppet] 07paladox 03b411992 - Migrate dns to ferm (#2066) [00:36:17] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [00:39:12] !log [universalomega@test3] starting deploy of {'config': True} to skip [00:39:13] !log [universalomega@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [00:39:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:39:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:40:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6aQD [00:40:47] [02miraheze/puppet] 07paladox 037ceadad - Migrate varnish to ferm [00:40:48] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [00:40:50] [02puppet] 07paladox opened pull request 03#2067: Migrate varnish to ferm - 13https://git.io/J6aQ9 [00:41:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6a5k [00:41:52] [02miraheze/puppet] 07paladox 03ca51ee2 - Update varnish.pp [00:41:53] [02puppet] 07paladox synchronize pull request 03#2067: Migrate varnish to ferm - 13https://git.io/J6aQ9 [00:42:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6adO [00:42:54] [02miraheze/puppet] 07paladox 03116cf10 - Update varnish_prometheus_exporter.pp [00:42:55] [02puppet] 07paladox synchronize pull request 03#2067: Migrate varnish to ferm - 13https://git.io/J6aQ9 [00:43:23] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 11.83, 7.31, 6.15 [00:43:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/J6adh [00:43:35] [02miraheze/puppet] 07paladox 03ff815c7 - Migrate varnish to ferm (#2067) [00:43:37] [02puppet] 07paladox closed pull request 03#2067: Migrate varnish to ferm - 13https://git.io/J6aQ9 [00:43:38] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [00:43:40] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [00:45:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6abA [00:45:10] [02miraheze/puppet] 07paladox 03bde24a5 - Set cp* to use ferm [00:45:12] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [00:45:13] [02puppet] 07paladox opened pull request 03#2068: Set cp* to use ferm - 13https://git.io/J6abh [00:45:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6aNG [00:45:26] [02miraheze/puppet] 07paladox 036d176a2 - Update cp12.yaml [00:45:27] [02puppet] 07paladox synchronize pull request 03#2068: Set cp* to use ferm - 13https://git.io/J6abh [00:45:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6aNB [00:45:34] [02miraheze/puppet] 07paladox 0309443de - Update cp14.yaml [00:45:36] [02puppet] 07paladox synchronize pull request 03#2068: Set cp* to use ferm - 13https://git.io/J6abh [00:45:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6aNi [00:45:49] [02miraheze/puppet] 07paladox 0336adb52 - Update cp15.yaml [00:45:50] [02puppet] 07paladox synchronize pull request 03#2068: Set cp* to use ferm - 13https://git.io/J6abh [00:46:00] [02puppet] 07paladox closed pull request 03#2068: Set cp* to use ferm - 13https://git.io/J6abh [00:46:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/J6aNF [00:46:03] [02miraheze/puppet] 07paladox 03cb241b2 - Set cp* to use ferm (#2068) [00:46:04] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [00:46:06] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [00:47:33] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 2.01, 4.73, 5.39 [00:49:07] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 167.114.2.161/cpweb, 2607:5300:201:3100::1d3/cpweb [00:49:29] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 12.97, 18.93, 23.19 [00:49:36] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 1.75, 3.74, 4.94 [00:49:49] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 167.114.2.161/cpweb, 2607:5300:201:3100::1d3/cpweb [00:50:20] hmm [00:50:55] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:51:06] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:52:35] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.74, 3.60, 3.29 [00:53:26] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/J6VJR [00:53:28] [02miraheze/ManageWiki] 07Universal-Omega 0393f8f32 - Update ManageWikiPermissions.php [00:54:14] [02ManageWiki] 07Universal-Omega opened pull request 03#305: Fix "The unique key array should contain a single unique index" - 13https://git.io/J6VUR [00:54:27] [02ManageWiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vpSns [00:54:50] [02ManageWiki] 07Universal-Omega edited pull request 03#305: Fix "The unique key array should contain a single unique index" - 13https://git.io/J6VUR [00:56:44] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/J6VIw [00:56:46] [02miraheze/ManageWiki] 07Universal-Omega 03aa99886 - Update ManageWikiNamespaces.php [00:56:47] [02ManageWiki] 07Universal-Omega synchronize pull request 03#305: Fix "The unique key array should contain a single unique index" - 13https://git.io/J6VUR [00:57:43] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.85, 16.42, 20.31 [01:00:33] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.94, 3.00, 3.18 [01:02:54] !log [@test3] starting deploy of {'config': True} to skip [01:02:55] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [01:02:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:03:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:03:45] miraheze/ManageWiki - Universal-Omega the build passed. [01:04:01] [02ManageWiki] 07Universal-Omega closed pull request 03#305: Fix "The unique key array should contain a single unique index" - 13https://git.io/J6VUR [01:04:02] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/J6VG8 [01:04:04] [02miraheze/ManageWiki] 07Universal-Omega 033d38158 - Fix "The unique key array should contain a single unique index" (#305) [01:04:28] [02ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vpSns [01:04:29] [02miraheze/ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [01:07:11] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.55, 4.28, 4.09 [01:07:30] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.61, 20.76, 20.03 [01:08:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.74, 3.47, 3.30 [01:09:07] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.58, 3.92, 3.98 [01:09:27] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.60, 18.96, 19.49 [01:10:24] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/J6V8g [01:10:25] [02miraheze/mediawiki] 07Universal-Omega 03548a92b - Update ManageWiki [01:10:27] miraheze/ManageWiki - Universal-Omega the build passed. [01:10:33] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.34, 3.73, 3.41 [01:10:38] !log [universalomega@test3] starting deploy of {'world': True, 'gitinfo': True} to skip [01:10:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:12:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.61, 3.44, 3.35 [01:13:05] !log [universalomega@test3] finished deploy of {'world': True, 'gitinfo': True} to skip - SUCCESS in 147s [01:13:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:14:31] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.28, 3.09, 3.24 [01:14:54] !log [universalomega@test3] starting deploy of {'config': True} to skip [01:14:55] !log [universalomega@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [01:14:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:15:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:19:53] [02mw-config] 07Universal-Omega closed pull request 03#4167: T5477: Add RatePage - 13https://git.io/J62bn [01:19:55] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-7 - 13https://git.io/vbvb3 [01:19:56] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-7 [01:19:58] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/J6Vrf [01:19:59] [02miraheze/mw-config] 07Universal-Omega 0350b6e46 - T5477: Add RatePage (#4167) [01:20:25] !log [universalomega@mw11] starting deploy of {'config': True, 'l10n': True} to all [01:20:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:20:55] miraheze/mw-config - Universal-Omega the build passed. [01:21:13] !log [universalomega@test3] starting deploy of {'config': True} to skip [01:21:14] !log [universalomega@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [01:21:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:21:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:21:32] !log [universalomega@test3] starting deploy of {'config': True, 'l10n': True} to skip [01:21:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:26:17] [02mw-config] 07Universal-Omega closed pull request 03#4168: T8059: Add Flatbox - 13https://git.io/J62jL [01:26:18] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/J6VME [01:26:20] [02miraheze/mw-config] 07Universal-Omega 031a517cc - T8059: Add Flatbox (#4168) [01:26:21] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-8 - 13https://git.io/vbvb3 [01:26:23] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-8 [01:26:29] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.55, 3.59, 3.40 [01:26:44] [02mw-config] 07Universal-Omega closed pull request 03#4166: T7989: Add Mirage - 13https://git.io/J62S5 [01:26:45] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/J6VMx [01:26:47] [02miraheze/mw-config] 07Universal-Omega 03ba64f72 - T7989: Add Mirage (#4166) [01:26:48] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-6 - 13https://git.io/vbvb3 [01:26:50] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-6 [01:27:19] miraheze/mw-config - Universal-Omega the build passed. [01:27:45] miraheze/mw-config - Universal-Omega the build passed. [01:28:28] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.81, 3.05, 3.23 [01:29:06] !log [universalomega@test3] finished deploy of {'config': True, 'l10n': True} to skip - SUCCESS in 453s [01:29:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:30:07] !log [@test3] starting deploy of {'config': True} to skip [01:30:08] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [01:30:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:30:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:30:53] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.86, 18.04, 16.54 [01:32:34] !log [universalomega@mw11] finished deploy of {'config': True, 'l10n': True} to all - SUCCESS in 728s [01:32:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:32:42] !log [universalomega@test3] starting deploy of {'config': True, 'l10n': True} to skip [01:32:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:32:52] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.75, 16.65, 16.19 [01:34:21] !log [@mw11] starting deploy of {'config': True} to all [01:34:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:34:27] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 6s [01:34:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:34:32] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 19.63, 21.81, 18.34 [01:34:46] !log [universalomega@mw11] starting deploy of {'config': True, 'l10n': True} to all [01:34:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:39:30] !log [universalomega@test3] finished deploy of {'config': True, 'l10n': True} to skip - SUCCESS in 407s [01:39:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:42:20] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 15.00, 19.32, 18.85 [01:42:39] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 30.36, 23.08, 18.71 [01:43:41] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://git.io/J6wUD [01:43:42] [02miraheze/mw-config] 07Universal-Omega 032aac23a - Update ManageWikiExtensions.php [01:43:44] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-6 - 13https://git.io/vbvb3 [01:43:55] [02mw-config] 07Universal-Omega opened pull request 03#4169: T8146: Add ArticleCreationWorkflow - 13https://git.io/J6wTT [01:44:50] miraheze/mw-config - Universal-Omega the build passed. [01:44:55] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://git.io/J6wk4 [01:44:56] [02miraheze/mw-config] 07Universal-Omega 03e10d1d6 - Update extension-list [01:44:58] [02mw-config] 07Universal-Omega synchronize pull request 03#4169: T8146: Add ArticleCreationWorkflow - 13https://git.io/J6wTT [01:44:59] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 14.09, 8.44, 5.14 [01:45:56] miraheze/mw-config - Universal-Omega the build passed. [01:46:02] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 6.25, 4.42, 3.21 [01:46:32] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 38.65, 26.26, 21.66 [01:46:39] !log [universalomega@test3] starting deploy of {'config': True} to skip [01:46:40] !log [universalomega@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [01:46:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:46:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:47:00] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:47:44] !log [universalomega@mw11] finished deploy of {'config': True, 'l10n': True} to all - SUCCESS in 778s [01:47:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:47:59] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 4.71, 4.73, 3.49 [01:48:39] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 14.87, 23.04, 20.90 [01:48:51] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 2.45, 5.72, 4.83 [01:50:25] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 17.34, 22.47, 21.37 [01:50:46] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.82, 4.98, 4.67 [01:52:00] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:54:19] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 15.12, 18.31, 19.90 [01:54:30] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 13.87, 19.58, 20.27 [01:54:57] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-6 [+0/-0/±1] 13https://git.io/J6wCG [01:54:58] [02miraheze/mw-config] 07Universal-Omega 0301e06a4 - Update ManageWikiExtensions.php [01:55:00] [02mw-config] 07Universal-Omega synchronize pull request 03#4169: T8146: Add ArticleCreationWorkflow - 13https://git.io/J6wTT [01:55:10] !log [universalomega@test3] starting deploy of {'config': True} to skip [01:55:11] !log [universalomega@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [01:55:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:55:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:55:56] miraheze/mw-config - Universal-Omega the build passed. [01:59:40] [02mw-config] 07Universal-Omega closed pull request 03#4169: T8146: Add ArticleCreationWorkflow - 13https://git.io/J6wTT [01:59:41] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-6 - 13https://git.io/vbvb3 [01:59:43] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-6 [01:59:44] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/J6wRp [01:59:46] [02miraheze/mw-config] 07Universal-Omega 03ebc3be5 - T8146: Add ArticleCreationWorkflow (#4169) [02:00:46] miraheze/mw-config - Universal-Omega the build passed. [02:01:22] !log [@mw11] starting deploy of {'config': True} to all [02:01:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:01:27] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 5s [02:01:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:01:57] !log [universalomega@mw11] starting deploy of {'config': True, 'l10n': True} to all [02:02:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:02:10] !log [universalomega@test3] starting deploy of {'config': True, 'l10n': True} to skip [02:02:18] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.65, 3.24, 3.01 [02:02:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:04:16] [02mw-config] 07Universal-Omega closed pull request 03#4162: Add Citoid sitenotice - 13https://git.io/J6k5A [02:04:18] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-5 [02:04:20] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-5 - 13https://git.io/vbvb3 [02:06:06] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.54, 6.13, 4.46 [02:06:16] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/J6wwy [02:06:17] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.63, 3.29, 3.11 [02:06:17] [02miraheze/mediawiki] 07Universal-Omega 03b6963b7 - Update ManageWiki [02:08:06] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.75, 5.76, 4.55 [02:08:33] !log [universalomega@test3] finished deploy of {'config': True, 'l10n': True} to skip - SUCCESS in 383s [02:08:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:10:14] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.32, 3.47, 3.23 [02:12:14] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.80, 3.12, 3.12 [02:14:46] !log [universalomega@mw11] finished deploy of {'config': True, 'l10n': True} to all - SUCCESS in 769s [02:14:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:15:57] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 26.47, 25.69, 21.85 [02:17:57] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 13.99, 21.35, 20.73 [02:19:20] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 9.16, 5.08, 2.40 [02:19:58] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 14.63, 19.41, 20.10 [02:25:55] !log [universalomega@mw11] starting deploy of {'world': True, 'l10n': True, 'gitinfo': True} to all [02:25:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:27:15] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.38, 1.86, 1.84 [02:31:11] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.16, 1.53, 1.70 [02:35:46] [02puppet] 07Universal-Omega opened pull request 03#2069: prometheus-es-exporter: allow querying by host - 13https://git.io/J6rYN [02:37:13] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.74, 6.86, 5.26 [02:39:11] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.10, 5.94, 5.12 [02:43:25] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.33, 20.29, 18.38 [02:45:05] !log [universalomega@mw11] finished deploy of {'world': True, 'l10n': True, 'gitinfo': True} to all - SUCCESS in 1150s [02:45:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:45:23] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 13.45, 17.77, 17.71 [02:54:16] [02miraheze/DataDump] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/J6rKJ [02:54:17] [02miraheze/DataDump] 07Universal-Omega 034dbec1b - Merge pull request from GHSA-g7rp-6h29-428f [02:59:40] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/J6rDm [02:59:42] [02miraheze/mediawiki] 07Universal-Omega 03ed86cc5 - Update DataDump [03:01:12] !log [universalomega@mw11] starting deploy of {'world': True, 'l10n': True, 'gitinfo': True} to all [03:01:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:01:25] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/J6rSR [03:01:27] [02miraheze/mediawiki] 07Universal-Omega 03b32d34a - Update DataDump [03:01:52] !log [universalomega@test3] starting deploy of {'world': True, 'l10n': True, 'gitinfo': True} to skip [03:01:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:03:48] miraheze/DataDump - Universal-Omega the build passed. [03:06:52] !log [universalomega@test3] finished deploy of {'world': True, 'l10n': True, 'gitinfo': True} to skip - SUCCESS in 299s [03:06:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:07:42] !log [universalomega@mw11] finished deploy of {'world': True, 'l10n': True, 'gitinfo': True} to all - SUCCESS in 389s [03:07:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:08:37] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/J6rAY [03:08:39] [02miraheze/mediawiki] 07Universal-Omega 03d4b4a3b - Add temporary HistoryBlobStub hack [03:08:40] [02mediawiki] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vbL5b [03:08:42] [02mediawiki] 07Universal-Omega opened pull request 03#3802: Add temporary HistoryBlobStub hack - 13https://git.io/J6rAC [03:09:25] [02mediawiki] 07Universal-Omega closed pull request 03#3802: Add temporary HistoryBlobStub hack - 13https://git.io/J6rAC [03:09:27] [02mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vbL5b [03:09:28] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/J6rxZ [03:09:30] [02miraheze/mediawiki] 07Universal-Omega 038c32f82 - Add temporary HistoryBlobStub hack (#3802) [03:09:31] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [03:09:49] !log [universalomega@mw11] starting deploy of {'world': True, 'gitinfo': True} to all [03:09:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:12:40] !log [universalomega@mw11] finished deploy of {'world': True, 'gitinfo': True} to all - SUCCESS in 170s [03:12:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:53:36] PROBLEM - wiki.elgeis.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'wiki.elgeis.com' expires in 7 day(s) (Thu 28 Oct 2021 03:49:16 GMT +0000). [03:57:29] PROBLEM - wiki.park-city.club - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'wiki.park-city.club' expires in 7 day(s) (Thu 28 Oct 2021 03:53:45 GMT +0000). [05:11:46] [02miraheze/ssl] 07Reception123 pushed 034 commits to 03master [+0/-3/±13] 13https://git.io/J666q [05:11:47] [02miraheze/ssl] 07Reception123 03a5bd858 - renew SSL certs [05:11:49] [02miraheze/ssl] 07Reception123 034a523c2 - remove wiki.caker18.com cert (wiki deleted) [05:11:50] [02miraheze/ssl] 07Reception123 03fdcc087 - remove housing.wiki cert (no longer pointing) [05:11:52] [02miraheze/ssl] 07Reception123 039025643 - remove bharatwiki.online cert (no longer pointing) [05:14:03] RECOVERY - mw12 APT on mw12 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [05:14:12] RECOVERY - mw11 APT on mw11 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [05:14:43] RECOVERY - mw13 APT on mw13 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [05:15:08] RECOVERY - mw9 APT on mw9 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [05:15:59] !log sudo apt-get upgrade on mw*/cloud* [05:16:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:16:27] RECOVERY - mw8 APT on mw8 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [05:17:07] RECOVERY - cloud3 APT on cloud3 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [05:17:52] RECOVERY - cloud4 APT on cloud4 is OK: APT OK: 28 packages available for upgrade (0 critical updates). [05:17:53] RECOVERY - services4 APT on services4 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [05:18:23] RECOVERY - services3 APT on services3 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [05:18:23] RECOVERY - cloud5 APT on cloud5 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [05:21:06] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://git.io/J667h [05:21:08] [02miraheze/dns] 07Reception123 03fd7a936 - add adanfime.com zone [05:37:12] RECOVERY - sbs.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'sbs.wiki' will expire on Tue 18 Jan 2022 04:01:54 GMT +0000. [05:38:02] RECOVERY - wiki.fffbr.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.fffbr.org' will expire on Tue 18 Jan 2022 04:01:02 GMT +0000. [05:38:14] RECOVERY - wiki.digitaldesignhq.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.digitaldesignhq.com' will expire on Tue 18 Jan 2022 04:02:54 GMT +0000. [05:38:44] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.54, 2.41, 1.35 [05:39:05] RECOVERY - steamdecklinux.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'steamdecklinux.wiki' will expire on Tue 18 Jan 2022 04:08:10 GMT +0000. [05:39:31] RECOVERY - pj-masks-info.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'pj-masks-info.cf' will expire on Tue 18 Jan 2022 04:04:11 GMT +0000. [05:39:34] RECOVERY - wiki.apico.buzz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.apico.buzz' will expire on Tue 18 Jan 2022 04:05:09 GMT +0000. [05:39:55] RECOVERY - wiki.landev.vn - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.landev.vn' will expire on Tue 18 Jan 2022 03:58:21 GMT +0000. [05:40:00] RECOVERY - wiki.jumpbound.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.jumpbound.com' will expire on Tue 18 Jan 2022 04:06:18 GMT +0000. [05:40:43] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.14, 1.91, 1.29 [05:41:37] RECOVERY - correypedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'correypedia.org' will expire on Tue 18 Jan 2022 03:59:17 GMT +0000. [05:42:32] RECOVERY - biblestrength.net - LetsEncrypt on sslhost is OK: OK - Certificate 'biblestrength.net' will expire on Tue 18 Jan 2022 04:07:10 GMT +0000. [05:42:42] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 0.36, 1.36, 1.16 [06:03:45] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.21, 5.62, 3.79 [06:19:26] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.08, 7.64, 6.61 [06:23:20] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.12, 6.28, 6.30 [06:23:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.41, 3.04, 2.60 [06:25:30] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.87, 2.90, 2.59 [06:42:21] PROBLEM - db11 Disk Space on db11 is WARNING: DISK WARNING - free space: / 48923 MB (10% inode=97%); [06:47:35] RECOVERY - mw10 APT on mw10 is OK: APT OK: 16 packages available for upgrade (0 critical updates). [06:50:51] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.73, 6.93, 6.47 [06:54:51] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 9.23, 7.78, 6.91 [06:56:51] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.59, 7.45, 6.91 [07:00:51] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.72, 6.44, 6.61 [07:12:52] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.30, 7.27, 6.90 [07:14:51] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 4.61, 6.19, 6.55 [07:20:52] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.55, 6.62, 6.55 [07:24:54] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 6.60, 6.62, 6.56 [07:40:19] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J6X9h [07:40:20] [02miraheze/services] 07MirahezeSSLBot 037ed84f2 - BOT: Updating services config for wikis [07:51:01] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03revert-3800-Universal-Omega-patch-1 [+2/-0/±1] 13https://git.io/J61em [07:51:03] [02miraheze/mediawiki] 07Reception123 0304deadb - Revert "Remove Collection and ElectronPdfService (#3800)" [07:51:04] [02mediawiki] 07Reception123 created branch 03revert-3800-Universal-Omega-patch-1 - 13https://git.io/vbL5b [07:51:06] [02mediawiki] 07Reception123 opened pull request 03#3803: Revert "Remove Collection and ElectronPdfService" - 13https://git.io/J61ec [07:51:10] [02mediawiki] 07Reception123 closed pull request 03#3803: Revert "Remove Collection and ElectronPdfService" - 13https://git.io/J61ec [07:51:11] [02mediawiki] 07Reception123 deleted branch 03revert-3800-Universal-Omega-patch-1 - 13https://git.io/vbL5b [07:51:13] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_36 [+2/-0/±1] 13https://git.io/J61eu [07:51:14] [02miraheze/mediawiki] 07Reception123 0309e4649 - Revert "Remove Collection and ElectronPdfService (#3800)" (#3803) [07:51:16] [02miraheze/mediawiki] 07Reception123 deleted branch 03revert-3800-Universal-Omega-patch-1 [07:55:00] !log [reception@mw11] starting deploy of {'config': True, 'world': True, 'l10n': True, 'gitinfo': True, 'extensionlist': True} to all [07:55:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:09:57] !log [reception@mw11] finished deploy of {'config': True, 'world': True, 'l10n': True, 'gitinfo': True, 'extensionlist': True} to all - SUCCESS in 896s [08:10:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:18:51] @Owen: we've had an email from a user saying T&S didn't respond [08:19:53] It's already been raised twice with me [08:23:51] Ok [08:33:10] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://git.io/J61db [08:33:11] [02miraheze/mw-config] 07Reception123 031d6598d - unrestrict collection and citoid [08:33:13] [02mw-config] 07Reception123 created branch 03Reception123-patch-1 - 13https://git.io/vbvb3 [08:33:14] [02mw-config] 07Reception123 opened pull request 03#4170: unrestrict collection and citoid - 13https://git.io/J61Ff [08:34:22] miraheze/mw-config - Reception123 the build passed. [08:34:33] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J61bG [08:34:34] [02miraheze/mw-config] 07Reception123 036ab9349 - unrestrict collection and citoid (#4170) [08:34:36] [02mw-config] 07Reception123 closed pull request 03#4170: unrestrict collection and citoid - 13https://git.io/J61Ff [08:34:37] [02miraheze/mw-config] 07Reception123 deleted branch 03Reception123-patch-1 [08:34:39] [02mw-config] 07Reception123 deleted branch 03Reception123-patch-1 - 13https://git.io/vbvb3 [08:34:54] !log [@mw11] starting deploy of {'config': True} to all [08:34:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:35:00] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 5s [08:35:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:35:37] miraheze/mw-config - Reception123 the build passed. [09:03:18] !log [@test3] starting deploy of {'config': True} to skip [09:03:19] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [09:03:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:03:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:27:16] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.50, 3.87, 3.33 [13:29:16] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.38, 3.60, 3.29 [13:33:13] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.25, 3.23, 3.20 [13:47:52] PROBLEM - mwtask1 Disk Space on mwtask1 is WARNING: DISK WARNING - free space: / 5066 MB (10% inode=89%); [13:49:00] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.58, 3.65, 3.24 [13:52:57] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.99, 3.71, 3.39 [13:56:26] !log [rhinos@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=metawiki lazy.txt -u=RhinosF1 -r=Requested - speedy delete -i=1 (END - exit=256) [13:56:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:56:47] !log [rhinos@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=metawiki lazy.txt -u=RhinosF1 -r=Requested - speedy delete -i=1 (END - exit=256) [13:56:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:56:53] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.08, 3.75, 3.47 [13:58:52] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.36, 3.19, 3.29 [14:23:52] RECOVERY - mwtask1 Disk Space on mwtask1 is OK: DISK OK - free space: / 6028 MB (12% inode=89%); [15:09:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.11, 3.45, 3.35 [15:13:52] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.30, 3.88, 3.54 [15:15:50] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.32, 3.71, 3.51 [15:19:47] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.98, 3.30, 3.39 [15:30:34] JohnLewis: do you know of a way to uncompress a tar.gz file while at the same time deleting that part of the zip? [15:30:48] otherwise it won't be possible to uncompress wiki backups on mwtask1 as there's only like 5GB left [15:31:23] The first question would be, is there sufficient space when uncompressed? [15:31:41] JohnLewis: well yes, because otherwise I wouldn't have been able to generate them in the first place :) [15:31:45] You might need to reclaim space elsewhere unfortunately as 5GB is a very tight margin to work with [15:32:33] hmm, I guess I could delete my older deleted backups but that wouldn't free up too much [15:32:46] definitely nowhere near enough as much as it would require to have both uncompressed and compressed at the same time [15:36:33] JohnLewis: whoops, I had a really large file in my homedir it seems, got 17G now [15:37:00] I should be able to clear some smaller files then and hopefully that should do [15:38:47] https://www.irccloud.com/pastebin/pE171ox3/ [15:38:54] also this is a bit weird [15:39:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6HVV [15:39:04] [02miraheze/puppet] 07paladox 034667c20 - Migrate database servers to ferm [15:39:06] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [15:39:07] [02puppet] 07paladox opened pull request 03#2070: Migrate database servers to ferm - 13https://git.io/J6HVo [15:39:07] not sure what to do about that, I guess I'd just use the .1 file but what's the 23M one then? [15:39:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6Hwf [15:39:23] [02miraheze/puppet] 07paladox 032347c53 - Update db.pp [15:39:24] [02puppet] 07paladox synchronize pull request 03#2070: Migrate database servers to ferm - 13https://git.io/J6HVo [15:40:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6Hrz [15:40:31] [02miraheze/puppet] 07paladox 03c8da926 - Update instance.pp [15:40:33] [02puppet] 07paladox synchronize pull request 03#2070: Migrate database servers to ferm - 13https://git.io/J6HVo [15:43:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6Hif [15:43:29] [02miraheze/puppet] 07paladox 03c5870dc - Update client.pp [15:43:30] [02puppet] 07paladox synchronize pull request 03#2070: Migrate database servers to ferm - 13https://git.io/J6HVo [15:44:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6HiD [15:44:07] [02miraheze/puppet] 07paladox 033417b87 - Update db11.yaml [15:44:09] [02puppet] 07paladox synchronize pull request 03#2070: Migrate database servers to ferm - 13https://git.io/J6HVo [15:44:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6Hid [15:44:14] [02miraheze/puppet] 07paladox 031c3feea - Update db12.yaml [15:44:16] [02puppet] 07paladox synchronize pull request 03#2070: Migrate database servers to ferm - 13https://git.io/J6HVo [15:44:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6HPJ [15:44:22] [02miraheze/puppet] 07paladox 03a395af2 - Update db13.yaml [15:44:23] [02puppet] 07paladox synchronize pull request 03#2070: Migrate database servers to ferm - 13https://git.io/J6HVo [15:44:43] [02puppet] 07paladox closed pull request 03#2069: prometheus-es-exporter: allow querying by host - 13https://git.io/J6rYN [15:44:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J6HPz [15:44:46] [02miraheze/puppet] 07Universal-Omega 037134207 - prometheus-es-exporter: allow querying by host (#2069) [15:46:03] [02puppet] 07paladox closed pull request 03#2070: Migrate database servers to ferm - 13https://git.io/J6HVo [15:46:05] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±6] 13https://git.io/J6H1O [15:46:06] [02miraheze/puppet] 07paladox 03c6cf4f7 - Migrate database servers to ferm (#2070) [15:46:08] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [15:46:09] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [15:48:01] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 34.74, 23.78, 17.62 [15:48:25] PROBLEM - gluster4 Current Load on gluster4 is CRITICAL: CRITICAL - load average: 9.03, 6.02, 3.33 [15:50:35] alerting : [FIRING:1] (!sre MediaWiki Exception Rate yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:50:40] PROBLEM - gluster4 Current Load on gluster4 is WARNING: WARNING - load average: 5.52, 5.93, 3.67 [15:52:03] PROBLEM - mwtask1 Disk Space on mwtask1 is CRITICAL: DISK CRITICAL - free space: / 1758 MB (3% inode=90%); [15:52:25] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.88, 5.16, 3.86 [15:52:37] RECOVERY - gluster4 Current Load on gluster4 is OK: OK - load average: 2.86, 4.90, 3.56 [15:55:35] ok : [RESOLVED] (!sre MediaWiki Exception Rate yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:56:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6HFk [15:56:04] [02miraheze/puppet] 07paladox 034b6714e - Migrate gluster to ferm [15:56:05] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [15:56:07] [02puppet] 07paladox opened pull request 03#2071: Migrate gluster to ferm - 13https://git.io/J6HFm [15:56:53] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6HbT [15:56:54] [02miraheze/puppet] 07paladox 036abde7e - Update gluster.pp [15:56:56] [02puppet] 07paladox synchronize pull request 03#2071: Migrate gluster to ferm - 13https://git.io/J6HFm [15:57:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-1/±0] 13https://git.io/J6Hb4 [15:57:10] [02miraheze/puppet] 07paladox 039c0c962 - Delete firewall.yaml [15:57:12] [02puppet] 07paladox synchronize pull request 03#2071: Migrate gluster to ferm - 13https://git.io/J6HFm [15:58:04] RECOVERY - mwtask1 Disk Space on mwtask1 is OK: DISK OK - free space: / 16058 MB (34% inode=90%); [15:58:18] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.76, 3.98, 3.69 [15:58:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6HN6 [15:58:22] [02miraheze/puppet] 07paladox 03154e91b - Update gluster3.yaml [15:58:23] [02puppet] 07paladox synchronize pull request 03#2071: Migrate gluster to ferm - 13https://git.io/J6HFm [15:58:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6HNH [15:58:30] [02miraheze/puppet] 07paladox 03b85ea68 - Update gluster4.yaml [15:58:32] [02puppet] 07paladox synchronize pull request 03#2071: Migrate gluster to ferm - 13https://git.io/J6HFm [15:59:10] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6HAw [15:59:12] [02miraheze/puppet] 07paladox 03d0ad7fb - Update gluster_exporter.pp [15:59:13] [02puppet] 07paladox synchronize pull request 03#2071: Migrate gluster to ferm - 13https://git.io/J6HFm [15:59:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-1/±5] 13https://git.io/J6HAH [15:59:26] [02miraheze/puppet] 07paladox 03de62272 - Migrate gluster to ferm (#2071) [15:59:27] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [15:59:28] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [15:59:30] [02puppet] 07paladox closed pull request 03#2071: Migrate gluster to ferm - 13https://git.io/J6HFm [16:00:11] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 14.84, 22.78, 21.76 [16:04:11] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.80, 3.18, 3.40 [16:11:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6QL4 [16:11:34] [02miraheze/puppet] 07paladox 03970236d - Migrate puppetserver to ferm [16:11:36] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [16:11:37] [02puppet] 07paladox opened pull request 03#2072: Migrate puppetserver to ferm - 13https://git.io/J6QL0 [16:11:44] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 15.24, 18.53, 20.22 [16:13:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6Qql [16:13:20] [02miraheze/puppet] 07paladox 03a4163ec - Update init.pp [16:13:22] [02puppet] 07paladox synchronize pull request 03#2072: Migrate puppetserver to ferm - 13https://git.io/J6QL0 [16:14:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6QmK [16:14:27] [02miraheze/puppet] 07paladox 03e6f8359 - Update jmx_exporter.pp [16:14:28] [02puppet] 07paladox synchronize pull request 03#2072: Migrate puppetserver to ferm - 13https://git.io/J6QL0 [16:15:37] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 15.08, 19.62, 20.48 [16:15:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6QYA [16:15:41] [02miraheze/puppet] 07paladox 0389cdf35 - Update puppet3.yaml [16:15:43] [02puppet] 07paladox synchronize pull request 03#2072: Migrate puppetserver to ferm - 13https://git.io/J6QL0 [16:15:47] [02puppet] 07paladox closed pull request 03#2072: Migrate puppetserver to ferm - 13https://git.io/J6QL0 [16:15:48] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/J6QOI [16:15:50] [02miraheze/puppet] 07paladox 03e62bf16 - Migrate puppetserver to ferm (#2072) [16:15:51] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [16:15:53] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [16:17:33] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 13.58, 17.75, 19.71 [16:17:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J6QsU [16:17:42] [02miraheze/puppet] 07paladox 036557d4a - Fix duplicates [16:19:11] PROBLEM - puppet3 Puppet on puppet3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:21:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J6QcO [16:21:14] [02miraheze/puppet] 07paladox 03b33b8c2 - Migrate postgresql to ferm [16:22:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.01, 3.58, 3.22 [16:23:14] RECOVERY - puppet3 Puppet on puppet3 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [16:24:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.97, 3.42, 3.21 [16:25:41] PROBLEM - db13 Puppet on db13 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:26:31] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.52, 3.19, 3.15 [16:27:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6QRF [16:27:57] [02miraheze/puppet] 07paladox 03b76a401 - Migrate services to ferm [16:27:59] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [16:28:00] [02puppet] 07paladox opened pull request 03#2073: Migrate services to ferm - 13https://git.io/J6QRA [16:28:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6Q07 [16:28:48] [02miraheze/puppet] 07paladox 032917fac - Update services3.yaml [16:28:49] [02puppet] 07paladox synchronize pull request 03#2073: Migrate services to ferm - 13https://git.io/J6QRA [16:28:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6QEJ [16:28:58] [02miraheze/puppet] 07paladox 0328d0803 - Update services4.yaml [16:28:59] [02puppet] 07paladox synchronize pull request 03#2073: Migrate services to ferm - 13https://git.io/J6QRA [16:29:09] [02puppet] 07paladox closed pull request 03#2073: Migrate services to ferm - 13https://git.io/J6QRA [16:29:11] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/J6QER [16:29:12] [02miraheze/puppet] 07paladox 03e11c7f4 - Migrate services to ferm (#2073) [16:29:14] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [16:29:15] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [16:32:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6Q2i [16:32:19] [02miraheze/puppet] 07paladox 034380655 - Migrate mon2 to ferm [16:32:20] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [16:32:22] [02puppet] 07paladox opened pull request 03#2074: Migrate mon2 to ferm - 13https://git.io/J6Q2H [16:32:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6Qaz [16:32:55] [02miraheze/puppet] 07paladox 03ebe584b - Update icinga2.pp [16:32:57] [02puppet] 07paladox synchronize pull request 03#2074: Migrate mon2 to ferm - 13https://git.io/J6Q2H [16:34:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6Qw6 [16:34:38] [02miraheze/puppet] 07paladox 039610c2c - Update irc.pp [16:34:40] [02puppet] 07paladox synchronize pull request 03#2074: Migrate mon2 to ferm - 13https://git.io/J6Q2H [16:34:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6Qrf [16:35:00] [02miraheze/puppet] 07paladox 03550f6fe - Update matomo.pp [16:35:01] [02puppet] 07paladox synchronize pull request 03#2074: Migrate mon2 to ferm - 13https://git.io/J6Q2H [16:35:49] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6QoJ [16:35:51] [02miraheze/puppet] 07paladox 03b4f7c79 - Update prometheus.pp [16:35:52] [02puppet] 07paladox synchronize pull request 03#2074: Migrate mon2 to ferm - 13https://git.io/J6Q2H [16:36:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6Qox [16:36:34] [02miraheze/puppet] 07paladox 031215b75 - Update mon2.yaml [16:36:35] [02puppet] 07paladox synchronize pull request 03#2074: Migrate mon2 to ferm - 13https://git.io/J6Q2H [16:37:20] [02puppet] 07paladox closed pull request 03#2074: Migrate mon2 to ferm - 13https://git.io/J6Q2H [16:37:21] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [16:37:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±6] 13https://git.io/J6QKH [16:37:24] [02miraheze/puppet] 07paladox 030c02749 - Migrate mon2 to ferm (#2074) [16:37:26] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [16:38:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J6Qie [16:38:18] [02miraheze/puppet] 07paladox 03f4864ae - Fix typo [16:39:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J6QPm [16:39:10] [02miraheze/puppet] 07paladox 03b147c95 - Fix typo [16:40:06] PROBLEM - mon2 Puppet on mon2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:40:25] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.06, 4.05, 3.52 [16:42:04] RECOVERY - mon2 Puppet on mon2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:42:24] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.18, 3.97, 3.57 [16:52:04] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.96, 3.00, 3.32 [16:52:42] RECOVERY - db13 Puppet on db13 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:58:37] [02miraheze/MirahezeMagic] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/J67fO [16:58:38] [02miraheze/MirahezeMagic] 07RhinosF1 03987eced - rebuildVersion: drop --use-staging [16:58:40] [02MirahezeMagic] 07RhinosF1 created branch 03RhinosF1-patch-1 - 13https://git.io/fQRGX [16:58:45] [02MirahezeMagic] 07RhinosF1 opened pull request 03#298: rebuildVersion: drop --use-staging - 13https://git.io/J67fR [16:59:25] JohnLewis: so far this is what's happened :( [16:59:29] sh: 1: mwdumper: not found [16:59:29] mwdumper died with exit status 127 [16:59:41] miraheze/MirahezeMagic - RhinosF1 the build has errored. [17:00:56] [02miraheze/MirahezeMagic] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/J67TJ [17:00:58] [02miraheze/MirahezeMagic] 07RhinosF1 03549ed1b - Update rebuildVersionCache.php [17:00:59] [02MirahezeMagic] 07RhinosF1 synchronize pull request 03#298: rebuildVersion: drop --use-staging - 13https://git.io/J67fR [17:02:00] miraheze/MirahezeMagic - RhinosF1 the build passed. [17:02:20] [02puppet] 07RhinosF1 opened pull request 03#2075: mwdeploy: pin gitinfo to world - 13https://git.io/J67k2 [17:11:03] Reception123: hm? [17:11:35] JohnLewis: here's the full output if that's more useful [17:11:44] https://www.irccloud.com/pastebin/Q4zUdJ6K/ [17:12:31] looking... [17:16:12] https://github.com/wikimedia/mediawiki-tools-mwdumper [17:16:13] [url] GitHub - wikimedia/mediawiki-tools-mwdumper: Github mirror of MediaWiki tool mwdumper - our actual code is hosted with Gerrit (please see https://www.mediawiki.org/wiki/Developer_access for contributi | github.com [17:19:32] Reception123: ^ so you need to follow the install instructions there and then either expand the mwdumper parts in checkStorage.php to be the full java instruction or create a file in /usr/local/bin/mwdumper which starts the java app [17:20:13] JohnLewis: I see, thanks! Sounds fun... [17:20:26] I know right (: [17:22:14] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.19, 3.63, 3.35 [17:24:10] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.73, 3.31, 3.27 [17:37:40] [02puppet] 07Universal-Omega opened pull request 03#2076: Revert "prometheus-es-exporter: allow querying by host" - 13https://git.io/J67M8 [17:43:42] [02puppet] 07paladox closed pull request 03#2076: Revert "prometheus-es-exporter: allow querying by host" - 13https://git.io/J67M8 [17:43:44] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J67Qg [17:43:45] [02miraheze/puppet] 07Universal-Omega 0335c55ec - Revert "prometheus-es-exporter: allow querying by host (#2069)" (#2076) [17:57:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J65vs [17:57:14] [02miraheze/puppet] 07paladox 03ea63ea8 - Migrate graylog to ferm [17:57:15] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [17:57:17] [02puppet] 07paladox opened pull request 03#2077: Migrate graylog to ferm - 13https://git.io/J65vn [17:57:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J65v7 [17:57:53] [02miraheze/puppet] 07paladox 033597331 - Update graylog2.yaml [17:57:55] [02puppet] 07paladox synchronize pull request 03#2077: Migrate graylog to ferm - 13https://git.io/J65vn [17:58:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J65fE [17:58:33] [02miraheze/puppet] 07paladox 03f2f94ad - Update graylog.pp [17:58:34] [02puppet] 07paladox synchronize pull request 03#2077: Migrate graylog to ferm - 13https://git.io/J65vn [18:00:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J65UU [18:00:21] [02miraheze/puppet] 07paladox 0365a565e - Update es_exporter.pp [18:00:23] [02puppet] 07paladox synchronize pull request 03#2077: Migrate graylog to ferm - 13https://git.io/J65vn [18:00:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/J65UV [18:00:54] [02miraheze/puppet] 07paladox 03bbd9000 - Migrate graylog to ferm (#2077) [18:00:55] [02puppet] 07paladox closed pull request 03#2077: Migrate graylog to ferm - 13https://git.io/J65vn [18:00:57] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [18:00:58] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [18:03:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J65I8 [18:03:42] [02miraheze/puppet] 07paladox 0316f926e - Migrate mail to ferm [18:03:44] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [18:03:45] [02puppet] 07paladox opened pull request 03#2078: Migrate mail to ferm - 13https://git.io/J65IB [18:05:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J65Lu [18:05:02] [02miraheze/puppet] 07paladox 036c40262 - Update postfix_exporter.pp [18:05:04] [02puppet] 07paladox synchronize pull request 03#2078: Migrate mail to ferm - 13https://git.io/J65IB [18:05:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J65tk [18:05:33] [02miraheze/puppet] 07paladox 03026df51 - Update roundcubemail.pp [18:05:35] [02puppet] 07paladox synchronize pull request 03#2078: Migrate mail to ferm - 13https://git.io/J65IB [18:06:04] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [18:06:05] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/J65ti [18:06:07] [02miraheze/puppet] 07paladox 036823cd8 - Migrate mail to ferm (#2078) [18:06:08] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [18:06:10] [02puppet] 07paladox closed pull request 03#2078: Migrate mail to ferm - 13https://git.io/J65IB [18:06:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J65qg [18:06:57] [02miraheze/puppet] 07paladox 03351fa7c - Switch mail2 to ferm [18:08:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J65mi [18:08:10] [02miraheze/puppet] 07paladox 03b9bb52a - Migrate phabricator to ferm [18:08:12] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [18:08:13] [02puppet] 07paladox opened pull request 03#2079: Migrate phabricator to ferm - 13https://git.io/J65mD [18:08:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J65YB [18:08:57] [02miraheze/puppet] 07paladox 03ed1b84e - Update phab2.yaml [18:08:59] [02puppet] 07paladox synchronize pull request 03#2079: Migrate phabricator to ferm - 13https://git.io/J65mD [18:09:03] [02puppet] 07paladox closed pull request 03#2079: Migrate phabricator to ferm - 13https://git.io/J65mD [18:09:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/J65Y2 [18:09:06] [02miraheze/puppet] 07paladox 034a524ad - Migrate phabricator to ferm (#2079) [18:09:07] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [18:09:09] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [18:10:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J65OP [18:10:23] [02miraheze/puppet] 07paladox 032945829 - Switch http/https firewall rules to use notrack on mon2 [18:10:24] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [18:10:26] [02puppet] 07paladox opened pull request 03#2080: Switch http/https firewall rules to use notrack on mon2 - 13https://git.io/J65OM [18:10:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J653U [18:10:44] [02miraheze/puppet] 07paladox 033b51d17 - Update icinga2.pp [18:10:46] [02puppet] 07paladox synchronize pull request 03#2080: Switch http/https firewall rules to use notrack on mon2 - 13https://git.io/J65OM [18:10:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J653n [18:10:59] [02miraheze/puppet] 07paladox 03d040c50 - Update grafana.pp [18:11:00] [02puppet] 07paladox synchronize pull request 03#2080: Switch http/https firewall rules to use notrack on mon2 - 13https://git.io/J65OM [18:11:42] [02puppet] 07paladox closed pull request 03#2080: Switch http/https firewall rules to use notrack on mon2 - 13https://git.io/J65OM [18:11:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/J653x [18:11:45] [02miraheze/puppet] 07paladox 033f85f75 - Switch http/https firewall rules to use notrack on mon2 (#2080) [18:11:46] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [18:11:48] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [18:15:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J65ng [18:15:01] [02miraheze/puppet] 07paladox 0339d7951 - Migrate openldap to ferm [18:15:03] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [18:15:04] [02puppet] 07paladox opened pull request 03#2081: Migrate openldap to ferm - 13https://git.io/J65nr [18:15:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J65cc [18:15:47] [02miraheze/puppet] 07paladox 03f8e6bb4 - Update openldap_exporter.pp [18:15:49] [02puppet] 07paladox synchronize pull request 03#2081: Migrate openldap to ferm - 13https://git.io/J65nr [18:16:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J65cK [18:16:09] [02miraheze/puppet] 07paladox 03a48e5cb - Update ldap2.yaml [18:16:11] [02puppet] 07paladox synchronize pull request 03#2081: Migrate openldap to ferm - 13https://git.io/J65nr [18:16:14] [02puppet] 07paladox closed pull request 03#2081: Migrate openldap to ferm - 13https://git.io/J65nr [18:16:15] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [18:16:17] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [18:16:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/J65cy [18:16:20] [02miraheze/puppet] 07paladox 03db4eca3 - Migrate openldap to ferm (#2081) [18:20:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6585 [18:20:19] [02miraheze/puppet] 07paladox 0354faeb6 - Migrate bacula to ferm [18:20:21] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [18:20:22] [02puppet] 07paladox opened pull request 03#2082: Migrate bacula to ferm - 13https://git.io/J658N [18:20:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J654u [18:20:57] [02miraheze/puppet] 07paladox 03ded2aba - Update client.pp [18:20:59] [02puppet] 07paladox synchronize pull request 03#2082: Migrate bacula to ferm - 13https://git.io/J658N [18:21:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J654x [18:21:22] [02miraheze/puppet] 07paladox 03cc780de - Update bacula2.yaml [18:21:24] [02puppet] 07paladox synchronize pull request 03#2082: Migrate bacula to ferm - 13https://git.io/J658N [18:22:13] [02puppet] 07paladox closed pull request 03#2082: Migrate bacula to ferm - 13https://git.io/J658N [18:22:15] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [18:22:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/J65BM [18:22:18] [02miraheze/puppet] 07paladox 033f7357f - Migrate bacula to ferm (#2082) [18:22:19] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [18:27:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J65zh [18:27:20] [02miraheze/puppet] 07paladox 03002b0da - Remove support for ufw in several places [18:27:21] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [18:27:25] [02puppet] 07paladox opened pull request 03#2083: Remove support for ufw in several places - 13https://git.io/J65gU [18:28:05] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J65gP [18:28:06] [02miraheze/puppet] 07paladox 0392eb22a - Update php_fpm.pp [18:28:08] [02puppet] 07paladox synchronize pull request 03#2083: Remove support for ufw in several places - 13https://git.io/J65gU [18:28:30] [02puppet] 07paladox closed pull request 03#2083: Remove support for ufw in several places - 13https://git.io/J65gU [18:28:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/J652O [18:28:33] [02miraheze/puppet] 07paladox 035f5cee1 - Remove support for ufw in several places (#2083) [18:28:34] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [18:28:36] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [18:28:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J6529 [18:29:01] [02miraheze/puppet] 07paladox 037a5a0d8 - Switch firewall to default to ferm [18:29:02] [02puppet] 07paladox opened pull request 03#2084: Switch firewall to default to ferm - 13https://git.io/J6527 [18:29:04] [02puppet] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbiAS [18:29:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J65aY [18:29:21] [02miraheze/puppet] 07paladox 039a6a0c2 - Update cloud3.yaml [18:29:22] [02puppet] 07paladox synchronize pull request 03#2084: Switch firewall to default to ferm - 13https://git.io/J6527 [18:29:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J65al [18:29:29] [02miraheze/puppet] 07paladox 032151b9f - Update cloud4.yaml [18:29:31] [02puppet] 07paladox synchronize pull request 03#2084: Switch firewall to default to ferm - 13https://git.io/J6527 [18:29:32] Is the relay-bot from the #cvt-feed down or something? [18:29:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/J65az [18:29:38] [02miraheze/puppet] 07paladox 034a4ffd4 - Update cloud5.yaml [18:29:40] [02puppet] 07paladox synchronize pull request 03#2084: Switch firewall to default to ferm - 13https://git.io/J6527 [18:29:48] [02puppet] 07paladox closed pull request 03#2084: Switch firewall to default to ferm - 13https://git.io/J6527 [18:29:49] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/J65aD [18:29:51] [02miraheze/puppet] 07paladox 0303c8677 - Switch firewall to default to ferm (#2084) [18:29:53] [02puppet] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbiAS [18:29:54] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-1 [18:29:59] For some reason, it's not even registering. [18:57:44] Can look [18:58:24] paladox: can you check ferm [18:58:33] I think you broke the RC feed [18:58:35] Hmm? [18:58:49] #miraheze-feed has nothing in 2 hours [18:59:18] 17:40:25 was last working paladox [18:59:32] Reception123: did you get things sorted? [18:59:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J6dJ2 [18:59:34] [02miraheze/puppet] 07paladox 034bf15fb - Fix firewall [19:01:18] fixed [19:01:20] Rh [19:01:25] Spookreeeno: [19:02:00] paladox: yep [19:02:16] @DarkMatterMan4500: fixed [19:06:01] Thanks. I see that it's now registering. [19:10:05] JohnLewis: I've asked CosmicAlpha for some assistance and there seem to be some complications [19:10:17] first, we don't have Maven 2 and second we don't have anything for Java [19:11:27] don't have anything for java? [19:12:57] you can install maven [19:13:29] https://packages.debian.org/buster/maven [19:13:30] [url] Debian -- Details of package maven in buster | packages.debian.org [19:16:41] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.88, 3.04, 2.75 [19:18:28] [02puppet] 07Universal-Omega opened pull request 03#2085: mediawiki: install default-jdk and maven - 13https://git.io/J6d8o [19:18:34] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.40, 3.44, 2.93 [19:22:21] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.43, 3.15, 2.95 [19:23:28] I wouldn't personally have them installed permanently for a one-time script [19:23:43] esp java when unused 99.9% of the time as it never has the best security history [19:24:20] JohnLewis: Oh true, how should we do it then, just someone install manually on mwtask1? [19:25:37] yeah [19:28:58] [02puppet] 07Universal-Omega closed pull request 03#2085: mediawiki: install default-jdk and maven - 13https://git.io/J6d8o [19:37:27] !log universalomega@mwtask1:~$ cd /srv/mediawiki && sudo -u www-data git clone https://gerrit.wikimedia.org/r/mediawiki/tools/mwdumper [19:37:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:41:16] !log universalomega@mwtask1:/srv/mediawiki/mwdumper$ sudo -u www-data mvn package [19:41:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:47:53] !log universalomega@mwtask1:/srv/mediawiki/mwdumper$ sudo -u www-data mvn compile; _JAVA_OPTIONS=-Djdk.net.URLClassPath.disableClassPathURLCheck=true sudo -u www-data mvn package [19:48:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:57:00] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.64, 3.46, 3.00 [20:00:48] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.85, 2.93, 2.91 [20:22:18] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.91, 3.44, 3.02 [20:26:18] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.41, 2.97, 2.94 [21:12:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/J6btP [21:12:23] [02miraheze/puppet] 07paladox 0326087bc - Fix firewall rule [21:46:49] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.45, 9.36, 5.88 [21:48:49] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.54, 7.48, 5.62 [21:50:49] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.48, 6.37, 5.43 [22:22:18] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.65, 3.39, 2.73 [22:24:18] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.50, 3.47, 2.84 [22:26:18] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.12, 3.00, 2.75 [22:44:32] JohnLewis: `checkStorage --fix` does not seem to work, been trying for 3 hours, but it never works on any xml file, any ideas? It currently gives: https://www.irccloud.com/pastebin/5UUU9ZbU/ [22:45:20] I have no idea from that error [22:45:54] https://github.com/wikimedia/mediawiki-tools-mwdumper/blob/master/src/org/mediawiki/dumper/input/XmlDumpReader.java#L413 [22:45:55] [url] mediawiki-tools-mwdumper/XmlDumpReader.java at master · wikimedia/mediawiki-tools-mwdumper · GitHub | github.com [22:50:53] JohnLewis: Here is the full output, but it's all I'm given, paladox: any ideas on how to fix it? I'm not sure how it's unable to read timestamps, or stops reading at some point, XML file itself looks intact. https://www.irccloud.com/pastebin/jo4rNN45/ [22:51:21] It's throwing a null when it's not suppose to [22:51:28] hence why it's throwing an exception [22:51:36] https://github.com/wikimedia/mediawiki-tools-mwdumper/blob/master/src/org/mediawiki/dumper/input/XmlDumpReader.java#L413 is the line [22:51:37] [url] mediawiki-tools-mwdumper/XmlDumpReader.java at master · wikimedia/mediawiki-tools-mwdumper · GitHub | github.com [23:05:17] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 10.57, 7.47, 5.02 [23:05:56] PROBLEM - mw10 APT on mw10 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:06:16] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 12.41, 8.76, 5.57 [23:06:41] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.92, 8.88, 6.11 [23:06:49] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 24.55, 21.72, 16.43 [23:06:49] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 10.71, 7.47, 5.08 [23:07:14] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.59, 6.48, 4.95 [23:07:48] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.61, 19.48, 15.44 [23:07:53] RECOVERY - mw10 APT on mw10 is OK: APT OK: 16 packages available for upgrade (0 critical updates). [23:08:14] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.82, 7.57, 5.52 [23:08:40] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 5.52, 8.00, 6.16 [23:08:49] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.32, 19.86, 16.36 [23:09:47] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.04, 21.06, 16.52 [23:10:13] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.59, 6.07, 5.21 [23:10:39] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.64, 6.73, 5.91 [23:10:50] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.39, 7.40, 5.71 [23:11:47] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 20.10, 20.19, 16.73 [23:12:50] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.71, 7.98, 6.13 [23:13:25] paladox: do you think recompiling with a change like this will work then? https://www.irccloud.com/pastebin/ZfoqMSgJ/ [23:13:42] possibly, have no idea tbh [23:13:51] as in i don't know which part is returning null [23:14:02] oh [23:14:04] no [23:14:25] rev.Timestamp doesn't support null so i think that'll throw null [23:14:58] rev.Timestamp takes a Revision instance/class [23:15:14] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 11.26, 8.64, 6.34 [23:15:47] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.98, 21.46, 18.20 [23:16:14] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.09, 8.24, 6.35 [23:17:15] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.12, 7.99, 6.38 [23:17:52] paladox: oh, so then I'm not sure how we will fix this error, and how we will resolve that task... [23:18:14] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 4.32, 6.83, 6.07 [23:18:39] well i'm slightly wrong, it takes a calender [23:18:40] https://github.com/wikimedia/mediawiki-tools-mwdumper/blob/09fe6ae20de9ee99e8acfff3f1c32a793997367b/src/org/mediawiki/dumper/wiki/Revision.java#L33 [23:18:40] [url] mediawiki-tools-mwdumper/Revision.java at 09fe6ae20de9ee99e8acfff3f1c32a793997367b · wikimedia/mediawiki-tools-mwdumper · GitHub | github.com [23:19:16] that hasn't been touched in 16 years :O [23:19:21] PROBLEM - mw12 APT on mw12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:19:47] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.92, 19.96, 18.38 [23:20:14] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.71, 5.92, 5.82 [23:20:34] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.56, 7.12, 6.23 [23:21:14] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.78, 6.77, 6.29 [23:21:21] RECOVERY - mw12 APT on mw12 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [23:21:45] paladox: Yeah it's very outdated... I really have no idea what we do here now then. [23:21:59] i have no idea how to fix that [23:22:33] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.98, 6.64, 6.16 [23:22:59] could try i guess rev.Timestamp = parseUTCTimestamp(bufferContents() == null ? "" : bufferContents()); [23:26:31] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.50, 6.98, 6.46 [23:27:47] PROBLEM - mw12 APT on mw12 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:29:45] RECOVERY - mw12 APT on mw12 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [23:30:30] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 11.77, 9.00, 7.32 [23:35:22] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 14.35, 9.32, 7.28 [23:35:30] paladox: seems to give the same error. [23:35:38] oh [23:35:51] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.13, 21.87, 19.69 [23:35:57] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:35:58] oh i guess you could use that null function [23:36:05] bufferContentsOrNull [23:36:18] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 18.17, 11.15, 7.57 [23:36:30] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:36:51] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 27.00, 22.54, 19.00 [23:37:18] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.02, 7.59, 6.35 [23:37:50] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.24, 23.02, 20.43 [23:37:56] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [23:38:50] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 23.10, 22.24, 19.29 [23:39:14] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.66, 7.29, 6.92 [23:39:14] paladox: Same error [23:39:18] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.17, 6.66, 6.16 [23:39:22] oh, i'm not sure then [23:40:20] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 11.68, 8.76, 6.68 [23:40:32] oh [23:40:33] hold on [23:40:39] check if rev is null [23:40:49] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 24.86, 23.00, 19.92 [23:40:49] if (rev != null ) rev.Timestamp.... [23:40:53] CosmicAlpha: ^ [23:41:14] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.66, 6.61, 6.73 [23:41:30] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:41:53] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.08, 3.78, 3.28 [23:42:49] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 16.25, 20.25, 19.28 [23:42:58] PROBLEM - cp13 Stunnel Http for mw12 on cp13 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.017 second response time [23:43:04] PROBLEM - cp15 Stunnel Http for mw12 on cp15 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.236 second response time [23:43:20] PROBLEM - cp14 Stunnel Http for mw12 on cp14 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.009 second response time [23:43:31] PROBLEM - cp12 Stunnel Http for mw12 on cp12 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.238 second response time [23:43:33] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.014 second response time [23:43:47] paladox: ^ [23:43:47] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 17.91, 19.65, 19.83 [23:43:48] PROBLEM - cp12 Varnish Backends on cp12 is CRITICAL: 1 backends are down. mw12 [23:43:50] PROBLEM - cp15 Varnish Backends on cp15 is CRITICAL: 1 backends are down. mw12 [23:43:54] hmm [23:43:57] PROBLEM - cp13 Varnish Backends on cp13 is CRITICAL: 1 backends are down. mw12 [23:44:00] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:44:27] !log reboot mw12 [23:44:28] PROBLEM - cp14 Varnish Backends on cp14 is CRITICAL: 1 backends are down. mw12 [23:44:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:46:05] paladox: so `if (rev != null ) rev.Timestamp = parseUTCTimestamp(bufferContentsOrNull() == null ? "" : bufferContents());` or `if (rev != null ) rev.Timestamp = parseUTCTimestamp(bufferContents());`? [23:46:20] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.73, 8.00, 6.99 [23:46:39] well i was just thinking of just if (rev != null) rev.Timestamp = parseUTCTimestamp(bufferContents()); [23:46:49] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 2.18, 0.62, 0.22 [23:46:57] RECOVERY - cp13 Stunnel Http for mw12 on cp13 is OK: HTTP OK: HTTP/1.1 200 OK - 15869 bytes in 0.046 second response time [23:46:57] RECOVERY - cp15 Stunnel Http for mw12 on cp15 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.306 second response time [23:47:12] RECOVERY - cp14 Stunnel Http for mw12 on cp14 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.011 second response time [23:47:19] RECOVERY - cp12 Stunnel Http for mw12 on cp12 is OK: HTTP OK: HTTP/1.1 200 OK - 15869 bytes in 0.342 second response time [23:47:28] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 20011 bytes in 0.277 second response time [23:47:33] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.71, 3.98, 3.53 [23:47:39] RECOVERY - cp15 Varnish Backends on cp15 is OK: All 9 backends are healthy [23:47:46] RECOVERY - cp12 Varnish Backends on cp12 is OK: All 9 backends are healthy [23:47:46] RECOVERY - cp13 Varnish Backends on cp13 is OK: All 9 backends are healthy [23:48:16] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 3.85, 6.85, 7.59 [23:48:20] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 3.43, 6.52, 6.59 [23:48:28] RECOVERY - cp14 Varnish Backends on cp14 is OK: All 9 backends are healthy [23:49:00] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:49:15] !log reboot mw11 [23:49:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:49:38] paladox: seemed to fix timestamp, now same error `at org.mediawiki.dumper.input.XmlDumpReader.closeContributor(XmlDumpReader.java:475)` can probably fix though now that we know the issue [23:49:49] ok [23:50:29] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 2.85, 0.84, 0.29 [23:51:21] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.04, 3.14, 3.29 [23:54:14] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.49, 5.66, 6.75 [23:56:21] paladox: Seemed to work! I did not run with checkStorage yet, but the mwdumper worked on its on unlike before. [23:57:29] Great!