[00:00:21] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.61, 1.55, 1.21 [00:00:41] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 2.74, 4.50, 4.87 [00:01:13] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.93, 3.58, 3.29 [00:04:07] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 3.71, 3.73, 5.10 [00:04:53] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 75 failures. Last run 3 minutes ago with 75 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/puppetlabs.gpg],File[/usr/local/bin/puppet-enabled],File[/etc/rsyslog.d],File[/etc/rsyslog.conf] [00:05:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.93, 3.14, 3.21 [00:12:51] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.58, 20.00, 18.20 [00:13:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:15:07] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 2.24, 3.91, 5.88 [00:16:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:19:50] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.04, 3.72, 3.35 [00:21:07] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 22.55, 8.08, 6.63 [00:21:18] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:21:45] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:21:48] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.45, 3.53, 3.33 [00:21:49] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:22:20] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:22:32] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:22:48] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb [00:22:51] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.28, 24.02, 20.94 [00:23:45] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.93, 3.39, 3.30 [00:23:48] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.248 second response time [00:23:51] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 7.481 second response time [00:24:16] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.15, 5.25, 3.92 [00:26:15] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.17, 5.50, 4.20 [00:26:48] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:28:08] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:28:13] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:28:32] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.541 second response time [00:28:41] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 16.00, 9.67, 6.45 [00:28:46] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.055 second response time [00:29:47] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.313 second response time [00:30:03] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 1.340 second response time [00:30:12] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 2.159 second response time [00:30:52] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [00:31:07] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 2.34, 4.51, 5.78 [00:31:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.07, 3.54, 3.26 [00:32:07] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 4.81, 5.23, 4.75 [00:32:51] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.71, 23.27, 22.31 [00:34:07] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 3.71, 4.68, 4.60 [00:35:07] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 1.74, 2.90, 4.81 [00:35:25] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.11, 3.54, 3.34 [00:36:15] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.44, 4.95, 4.62 [00:36:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:36:41] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 4.54, 5.65, 5.77 [00:38:41] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.02, 6.48, 6.04 [00:39:20] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.01, 3.32, 3.29 [00:39:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:40:41] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 3.90, 5.45, 5.71 [00:43:12] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.71, 4.24, 3.66 [00:44:07] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 9.18, 8.70, 6.20 [00:45:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.72, 3.66, 3.52 [00:46:51] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 20.13, 18.97, 20.20 [00:47:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.77, 3.26, 3.38 [00:48:08] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 3.66, 5.82, 5.56 [00:48:41] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 3.51, 4.14, 4.93 [00:53:10] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.80, 3.79, 3.55 [00:54:07] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 2.89, 4.06, 4.85 [00:55:10] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.83, 3.80, 3.58 [00:59:25] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.30, 2.42, 1.62 [01:01:10] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.14, 3.26, 3.40 [01:01:23] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.79, 1.82, 1.49 [01:03:21] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.63, 1.42, 1.38 [01:06:33] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 5.05, 7.36, 6.05 [01:10:30] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 3.76, 5.79, 5.72 [01:15:51] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 6.31, 4.50, 3.78 [01:16:26] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.19, 6.43, 5.89 [01:18:24] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 3.61, 5.44, 5.60 [01:21:43] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.15, 3.85, 3.77 [01:23:07] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 2.34, 5.83, 4.20 [01:24:20] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.61, 6.17, 5.76 [01:25:07] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 2.13, 4.80, 4.04 [01:25:07] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSija [01:25:08] [02miraheze/mw-config] 07Universal-Omega 03505bb41 - Add status to CreateWiki blacklisted subdomains [01:26:18] miraheze/mw-config - Universal-Omega the build passed. [01:26:19] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 5.41, 5.88, 5.70 [01:26:38] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 36.67, 13.37, 7.42 [01:27:36] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.02, 3.82, 3.76 [01:30:16] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 2.93, 4.23, 5.06 [01:31:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.81, 3.55, 3.68 [01:34:12] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 5.70, 6.47, 5.84 [01:36:10] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 4.23, 5.95, 5.74 [01:38:10] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 2.46, 4.87, 5.64 [01:41:50] !log [@mw11] starting deploy of {'config': True} to ovlon [01:41:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:42:06] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 3.04, 4.36, 5.09 [01:42:07] !log [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 17s [01:42:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:46:07] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 1.80, 3.36, 4.81 [01:47:07] PROBLEM - db111 Current Load on db111 is CRITICAL: CRITICAL - load average: 20.78, 8.46, 4.60 [01:47:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.01, 3.64, 3.58 [01:47:21] !log [@test101] starting deploy of {'config': True} to skip [01:47:22] !log [@test101] DEPLOY ABORTED: Canary check failed for localhost [01:47:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:47:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:49:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.76, 3.58, 3.56 [01:50:39] PROBLEM - test101 Puppet on test101 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [01:51:10] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.34, 4.26, 3.81 [01:51:15] !log [@mwtask111] starting deploy of {'config': True} to scsvg [01:51:16] !log [@mwtask111] DEPLOY ABORTED: Canary check failed for localhost [01:51:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:51:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:51:55] [02mw-config] 07dmehus opened pull request 03#4336: Remove rhinosf1wiki - 13https://git.io/JSPBY [01:52:14] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.26, 6.53, 5.56 [01:52:57] miraheze/mw-config - dmehus the build passed. [01:52:58] [02mw-config] 07Universal-Omega closed pull request 03#4336: Remove rhinosf1wiki - 13https://git.io/JSPBY [01:53:00] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSPRs [01:53:01] [02miraheze/mw-config] 07dmehus 03a3ea5ff - Remove rhinosf1wiki (#4336) [01:53:07] PROBLEM - db111 Current Load on db111 is WARNING: WARNING - load average: 2.52, 5.45, 4.54 [01:53:28] PROBLEM - mwtask111 Puppet on mwtask111 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [01:54:03] miraheze/mw-config - Universal-Omega the build passed. [01:54:14] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.89, 6.01, 5.49 [01:54:41] !log [@test3] starting deploy of {'config': True} to skip [01:54:42] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [01:55:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:55:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:55:07] RECOVERY - db111 Current Load on db111 is OK: OK - load average: 1.77, 4.29, 4.23 [01:56:51] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.01, 6.89, 6.31 [01:57:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.44, 3.67, 3.74 [01:58:13] [02mw-config] 07dmehus opened pull request 03#4337: Remove RobLa - 13https://git.io/JSPgU [01:59:12] miraheze/mw-config - dmehus the build passed. [02:00:41] CosmicAlpha: ty [02:00:48] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.14, 6.51, 6.30 [02:01:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.20, 3.97, 3.86 [02:05:10] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.96, 3.99, 3.91 [02:11:32] !log [@mw11] starting deploy of {'config': True} to ovlon [02:11:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:11:43] !log [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 11s [02:11:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:12:07] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 12.03, 7.52, 4.53 [02:14:07] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 2.99, 5.53, 4.16 [02:15:10] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.31, 3.45, 3.59 [02:16:07] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 0.76, 3.91, 3.73 [02:17:51] !log [@test101] starting deploy of {'config': True} to skip [02:17:52] !log [@test101] DEPLOY ABORTED: Canary check failed for localhost [02:18:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:18:14] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.58, 6.75, 5.91 [02:18:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:19:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.84, 3.85, 3.73 [02:21:10] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 3.99, 4.06, 3.83 [02:21:14] !log [@mwtask111] starting deploy of {'config': True} to scsvg [02:21:15] !log [@mwtask111] DEPLOY ABORTED: Canary check failed for localhost [02:21:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:21:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:21:36] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 6.02, 6.53, 5.18 [02:22:14] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.76, 7.12, 6.29 [02:23:10] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.42, 3.71, 3.72 [02:24:14] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.69, 6.76, 6.25 [02:25:11] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.41, 4.00, 3.83 [02:27:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.41, 3.73, 3.75 [02:27:32] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 2.63, 5.33, 5.33 [02:29:30] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 13.85, 8.17, 6.31 [02:33:36] [02mw-config] 07Universal-Omega closed pull request 03#4337: Remove RobLa - 13https://git.io/JSPgU [02:33:37] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSP1p [02:33:39] [02miraheze/mw-config] 07dmehus 03ba2bd6f - Remove RobLa (#4337) [02:34:05] No problem dmehus [02:34:43] miraheze/mw-config - Universal-Omega the build passed. [02:39:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:41:01] !log [@mw11] starting deploy of {'config': True} to ovlon [02:41:10] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.08, 3.56, 3.60 [02:41:13] !log [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 11s [02:41:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:41:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:42:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:45:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.01, 3.68, 3.66 [02:45:19] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 2.91, 4.61, 5.66 [02:47:31] !log [@test101] starting deploy of {'config': True} to skip [02:47:32] !log [@test101] DEPLOY ABORTED: Canary check failed for localhost [02:47:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:47:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:51:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.34, 3.81, 3.68 [02:51:15] !log [@mwtask111] starting deploy of {'config': True} to scsvg [02:51:15] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 5.57, 6.09, 5.95 [02:51:16] !log [@mwtask111] DEPLOY ABORTED: Canary check failed for localhost [02:51:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:51:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:54:19] !log [@test3] starting deploy of {'config': True} to skip [02:54:20] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [02:54:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:54:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:55:10] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.77, 3.91, 3.75 [02:58:07] [02MirahezeMagic] 07dmehus opened pull request 03#309: Add group member interface message for the member group - 13https://git.io/JSPAb [02:59:09] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 2.49, 4.67, 5.74 [03:01:13] [02MirahezeMagic] 07dmehus synchronize pull request 03#309: Add group member interface message for the member group - 13https://git.io/JSPAb [03:03:07] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 2.28, 3.35, 4.95 [03:04:22] miraheze/MirahezeMagic - dmehus the build passed. [03:08:32] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#309 commit - 13https://git.io/JSXJg [03:08:34] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#309 commit - 13https://git.io/JSXJ2 [03:09:03] miraheze/MirahezeMagic - dmehus the build passed. [03:09:13] [02MirahezeMagic] 07dmehus reviewed pull request 03#309 commit - 13https://git.io/JSXUv [03:09:58] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#309 commit - 13https://git.io/JSXUX [03:10:08] [02MirahezeMagic] 07Universal-Omega reviewed pull request 03#309 commit - 13https://git.io/JSXUX [03:10:49] [02MirahezeMagic] 07dmehus synchronize pull request 03#309: Add group member interface message for the member group - 13https://git.io/JSPAb [03:11:39] [02MirahezeMagic] 07dmehus reviewed pull request 03#309 commit - 13https://git.io/JSXTC [03:13:15] [02MirahezeMagic] 07dmehus synchronize pull request 03#309: Add group member interface message for the member group - 13https://git.io/JSPAb [03:13:44] [02MirahezeMagic] 07dmehus reviewed pull request 03#309 commit - 13https://git.io/JSXTz [03:13:56] [02MirahezeMagic] 07Universal-Omega closed pull request 03#309: Add group member interface message for the member group - 13https://git.io/JSPAb [03:13:57] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JSXTg [03:13:59] [02miraheze/MirahezeMagic] 07dmehus 038b32f65 - Add group member interface message for the member group (#309) [03:17:11] CosmicAlpha, thanks again :) [03:17:29] dmehus: no problem. [03:18:37] RECOVERY - test101 Puppet on test101 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [03:18:44] miraheze/MirahezeMagic - dmehus the build passed. [03:19:10] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.74, 3.15, 3.37 [03:21:05] miraheze/MirahezeMagic - dmehus the build passed. [03:22:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:23:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.52, 4.16, 3.76 [03:23:26] RECOVERY - mwtask111 Puppet on mwtask111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:24:41] miraheze/MirahezeMagic - Universal-Omega the build passed. [03:25:10] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.51, 3.76, 3.65 [03:29:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:32:15] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.55, 1.86, 1.35 [03:33:10] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.10, 3.05, 3.37 [03:34:12] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.93, 1.61, 1.33 [03:34:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:37:10] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.04, 3.69, 3.57 [03:39:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.73, 3.37, 3.47 [03:40:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:40:41] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 5.54, 7.87, 5.87 [03:45:11] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.55, 3.97, 3.65 [03:47:10] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.49, 3.81, 3.64 [03:47:33] CosmicAlpha: has puppet been run since the last merge? If so, I'm not seeing the MediaWiki:Group-member-member message where it doesn't exist locally :( [03:47:55] [[:mh:cosmos:MediaWiki:Group-member-member]] [03:47:55] https://meta.miraheze.org/wiki/:mh:cosmos:MediaWiki:Group-member-member [03:48:36] dmehus: MirahezeMagic needs updated first. [03:48:42] oh [03:48:46] how do I do that? [03:49:15] or do you mean it has to wait until the Sunday deploy? [03:49:40] s/deploy/deploy by dependabot [03:49:40] dmehus meant to say: or do you mean it has to wait until the Sunday deploy by dependabot? [03:49:49] dmehus: [[Tech:Updating an Extension]] or wait till Sunday, or I might be able to do it sometime later this week (maybe tomorrow) [03:49:49] https://meta.miraheze.org/wiki/Tech:Updating_an_Extension [03:50:00] CosmicAlpha: oh [03:50:02] looking [03:50:23] [[Tech:Updating an extension]] [03:50:23] https://meta.miraheze.org/wiki/Tech:Updating_an_extension [03:50:41] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 3.32, 5.35, 5.77 [03:50:47] can I do that in GitHub web interface, or only in Git? [03:51:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.49, 4.29, 3.87 [03:51:12] Only git. [03:51:24] oh [03:51:26] ok [03:51:31] I'll wait till Sunday then [03:51:37] or leave for you (if you're bored) [03:53:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.21, 3.90, 3.78 [03:53:35] dmehus: I can update in within like 2 minutes if you absolutely need it. Though probably won't till tomorrow at the earliest, but it takes like 2 minutes to do since I actually have a shell script on my PC I just run —update.sh extensions/MirahezeMagic basically. But I'm mobile right now. [03:54:41] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 2.47, 3.89, 5.08 [03:59:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.60, 4.13, 3.90 [04:00:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:01:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.86, 3.83, 3.81 [04:03:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.86, 4.14, 3.92 [04:04:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:04:41] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 2.92, 6.45, 6.08 [04:04:46] CosmicAlpha, oh no it's not that urgent at all. Just if you want to this week sometime :) [04:05:08] oh that's cool you have written a shell script for it :) [04:05:10] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.00, 3.63, 3.76 [04:06:41] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 3.63, 5.88, 5.93 [04:07:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.52, 3.94, 3.85 [04:10:41] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 1.71, 3.55, 4.95 [04:11:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.91, 3.68, 3.78 [04:14:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:16:41] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 29.69, 11.27, 7.27 [04:18:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:19:10] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.32, 3.53, 3.61 [04:21:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.78, 3.75, 3.69 [04:24:41] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 1.54, 4.88, 5.64 [04:25:10] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.72, 2.78, 3.32 [04:26:41] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 0.21, 3.26, 4.95 [04:33:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:39:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:47:33] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.61, 3.84, 3.49 [04:49:31] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.56, 3.67, 3.47 [04:49:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:51:28] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.28, 4.23, 3.69 [04:53:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.76, 3.93, 3.64 [04:57:20] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.94, 3.22, 3.40 [05:06:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:11:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:19:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.27, 3.42, 3.31 [05:21:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.03, 3.74, 3.45 [05:21:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:23:10] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.41, 3.29, 3.32 [05:26:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:42:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:45:48] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.10, 3.76, 3.44 [05:47:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:47:45] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.62, 3.75, 3.48 [05:53:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:57:31] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.65, 3.18, 3.34 [06:03:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:07:15] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.44, 3.81, 3.47 [06:09:12] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.13, 3.43, 3.37 [06:11:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.59, 3.81, 3.51 [06:13:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.37, 3.23, 3.33 [06:17:10] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.79, 3.75, 3.53 [06:22:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:23:10] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.24, 4.10, 3.70 [06:24:28] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JS1BF [06:24:29] [02miraheze/mediawiki] 07Universal-Omega 033a67bdc - Update MirahezeMagic [06:24:52] !log [universalomega@test3] starting deploy of {'world': True, 'l10n': True} to skip [06:24:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:25:10] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.85, 3.70, 3.61 [06:25:13] !log [universalomega@mw11] starting deploy of {'world': True, 'l10n': True} to all [06:25:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:25:22] dmehus: ^ [06:29:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.29, 3.79, 3.66 [06:32:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:35:11] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.07, 3.42, 3.61 [06:37:00] !log [universalomega@test3] finished deploy of {'world': True, 'l10n': True} to skip - SUCCESS in 727s [06:37:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:37:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.08, 2.93, 3.40 [06:37:52] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.68, 2.89, 1.95 [06:41:48] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.51, 1.64, 1.66 [06:45:22] !log [universalomega@mw11] finished deploy of {'world': True, 'l10n': True} to all - SUCCESS in 1208s [06:45:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:47:22] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JS1VI [06:47:23] [02miraheze/mediawiki] 07Universal-Omega 038326fa6 - Update PortableInfobox [06:47:30] !log [universalomega@mw11] starting deploy of {'world': True} to all [06:47:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:47:37] !log [universalomega@test3] starting deploy of {'world': True} to skip [06:47:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:47:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:51:49] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.92, 5.84, 4.80 [06:52:12] !log [universalomega@test3] finished deploy of {'world': True} to skip - SUCCESS in 275s [06:52:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:52:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.06, 3.58, 3.40 [06:53:26] !log [universalomega@mw11] finished deploy of {'world': True} to all - SUCCESS in 356s [06:53:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:53:48] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.38, 5.21, 4.70 [06:54:53] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.79, 3.25, 3.30 [06:57:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:06:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:11:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:54:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:59:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:23:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:28:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:46:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:51:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:13:41] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.95, 5.60, 4.42 [09:15:41] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 2.95, 4.52, 4.16 [10:10:49] [02MirahezeMagic] 07magogre opened pull request 03#310: Update en.json - 13https://git.io/JSM2Z [12:20:29] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.24, 5.79, 4.49 [12:22:29] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.55, 6.04, 4.74 [12:25:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:35:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:17:33] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.57, 3.43, 3.01 [13:21:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.03, 3.56, 3.18 [13:21:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:31:14] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.46, 3.14, 3.21 [13:51:06] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.19, 7.26, 6.60 [13:51:15] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.28, 7.18, 6.20 [13:51:48] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:52:20] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:53:15] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.62, 7.24, 6.34 [13:53:42] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 0.614 second response time [13:54:16] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.196 second response time [13:57:02] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.71, 6.73, 6.56 [13:58:51] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.97, 6.70, 6.06 [13:59:15] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.56, 6.72, 6.37 [14:00:47] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.66, 6.28, 5.98 [14:06:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:09:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [14:15:10] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.55, 3.77, 3.20 [14:18:06] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.56, 6.97, 6.35 [14:20:03] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.92, 6.44, 6.22 [14:21:10] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.36, 3.80, 3.39 [14:26:33] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.06, 1.57, 1.18 [14:27:10] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.28, 3.36, 3.37 [14:28:31] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.65, 1.57, 1.23 [14:35:58] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.21, 4.09, 3.63 [14:39:53] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 1.91, 3.37, 3.48 [14:47:43] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.89, 3.13, 3.36 [14:55:29] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.97, 3.35, 3.30 [14:59:23] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.01, 3.30, 3.30 [14:59:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:03:15] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.43, 3.97, 3.58 [15:05:12] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.62, 3.80, 3.56 [15:09:10] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.27, 3.58, 3.50 [15:11:12] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.56, 3.14, 3.35 [15:21:10] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.16, 3.81, 3.48 [15:23:10] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.78, 3.85, 3.54 [15:25:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [15:27:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.62, 3.07, 3.33 [15:36:01] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.50, 3.50, 3.38 [15:39:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.32, 3.26, 3.35 [15:41:59] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.81, 6.92, 6.13 [15:42:16] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.76, 5.34, 3.63 [15:42:57] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.79, 20.87, 17.61 [15:43:16] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.42, 7.30, 6.07 [15:43:20] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.81, 7.12, 6.03 [15:45:14] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.78, 6.62, 5.98 [15:45:15] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.79, 6.89, 6.08 [15:45:58] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.38, 7.86, 6.70 [15:46:15] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.46, 4.55, 3.78 [15:46:51] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.78, 19.49, 18.01 [15:47:15] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.53, 6.72, 6.14 [15:47:57] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 5.90, 7.11, 6.57 [15:51:15] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.91, 7.59, 6.64 [15:52:01] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.88, 6.96, 6.34 [15:52:14] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.82, 6.42, 5.77 [15:52:23] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb [15:52:51] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.41, 21.95, 19.54 [15:53:15] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.18, 7.82, 6.85 [15:53:53] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.57, 7.77, 7.04 [15:53:55] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.88, 6.35, 6.20 [15:54:14] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.52, 6.09, 5.73 [15:55:15] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.50, 7.62, 6.88 [15:55:52] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.06, 7.41, 6.98 [15:56:51] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.79, 19.26, 18.98 [15:57:52] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.02, 6.53, 6.72 [15:58:06] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [15:59:15] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.33, 6.44, 6.57 [16:06:31] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JSDIg [16:06:33] [02miraheze/mediawiki] 07paladox 03fc0b8b2 - Update GlobalPreferences [16:07:16] !log [paladox@mw11] starting deploy of {'world': True} to all [16:07:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:08:16] !log [paladox@test3] starting deploy of {'world': True} to skip [16:08:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:10:15] !log [paladox@mwtask111] starting deploy of {'world': True} to all [16:10:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:10:56] !log [paladox@test101] starting deploy of {'world': True} to skip [16:11:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:12:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.56, 3.65, 3.39 [16:12:13] paladox: new DC needs --force until DB exists [16:12:14] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.86, 6.61, 6.13 [16:12:21] oh ok [16:12:28] !log [paladox@test101] DEPLOY ABORTED: Canary check failed for localhost [16:12:33] !log [paladox@test101] starting deploy of {'world': True, 'force': True} to skip [16:12:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:12:48] !log [paladox@mwtask111] DEPLOY ABORTED: Canary check failed for localhost [16:12:53] !log [paladox@mwtask111] starting deploy of {'world': True, 'force': True} to all [16:12:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:13:05] !log [paladox@test3] finished deploy of {'world': True} to skip - SUCCESS in 290s [16:13:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:13:10] !log [paladox@mw11] finished deploy of {'world': True} to all - SUCCESS in 353s [16:13:30] paladox: also it's not all [16:13:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:13:36] You need scsvg [16:13:38] hmm? [16:13:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:13:40] oh [16:13:41] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.50, 6.90, 6.53 [16:13:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:14:00] !log [paladox@mwtask111] starting deploy of {'world': True, 'force': True} to scsvg [16:14:11] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.21, 6.57, 6.18 [16:14:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:15:19] !log [paladox@test101] finished deploy of {'world': True, 'force': True} to skip - FAIL: [0, 0, 65280, 0] in 166s [16:15:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:16:04] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.49, 3.19, 3.27 [16:17:41] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.72, 6.47, 6.41 [16:19:08] paladox: the fail is expected, that'll be gitinfo. It's as successful as it can be. [16:19:16] ok [16:19:23] !log [paladox@mwtask111] finished deploy of {'world': True, 'force': True} to scsvg - FAIL: [0, 0, 65280, 0, 0, 0, 0, 0, 0, 0, 5888, 5888, 5888, 5888, 5888, 5888] in 322s [16:20:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:21:10] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.07, 6.94, 6.21 [16:21:41] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.90, 7.64, 6.99 [16:21:53] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.91, 3.61, 3.47 [16:23:06] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.38, 7.16, 6.36 [16:23:50] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.92, 3.38, 3.40 [16:25:01] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.23, 6.97, 6.37 [16:25:41] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.72, 6.11, 6.53 [16:28:52] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.43, 6.50, 6.36 [16:31:36] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.27, 3.82, 3.50 [16:33:34] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.29, 3.68, 3.49 [16:41:23] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.42, 3.99, 3.68 [16:43:13] miraheze/MirahezeMagic - Universal-Omega the build passed. [16:45:18] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.04, 3.82, 3.71 [16:45:37] PROBLEM - jobchron1 APT on jobchron1 is CRITICAL: APT CRITICAL: 5 packages available for upgrade (4 critical updates). [16:51:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.18, 3.84, 3.73 [16:59:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.02, 3.27, 3.56 [17:00:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:01:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.25, 2.85, 3.36 [17:01:12] [02MirahezeMagic] 07Universal-Omega commented on pull request 03#310: Update en.json - 13https://git.io/JSDqL [17:08:35] PROBLEM - mw122 Puppet on mw122 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/default/ferm] [17:10:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:22:14] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.70, 7.44, 6.23 [17:22:48] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 51.195.220.68/cpweb, 2607:5300:201:3100::929a/cpweb [17:24:12] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.94, 7.22, 6.29 [17:24:48] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [17:30:09] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.51, 6.18, 6.14 [17:31:01] [02puppet] 07Universal-Omega opened pull request 03#2252: mwupgradetool: switch to SCSVG servers - 13https://git.io/JSDmp [17:35:12] RECOVERY - mw122 Puppet on mw122 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [17:38:45] PROBLEM - mw111 Puppet on mw111 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown): Exec[git_pull_JobRunner],Exec[git_pull_mathoid],Exec[git_pull_3d2png] [17:46:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.76, 3.54, 3.32 [17:48:56] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.25, 6.47, 6.05 [17:49:30] Why is puppet failing [17:50:54] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.95, 6.43, 6.09 [17:51:41] Notice: /Stage[main]/Mediawiki::Jobqueue::Runner/Git::Clone[JobRunner]/Exec[git_pull_JobRunner]/returns: fatal: unable to access 'https://github.com/miraheze/jobrunner-service/': Failed to connect to bast.miraheze.org port 8080: No route to host [17:51:42] [url] GitHub - miraheze/jobrunner-service: JobRunner services on Miraheze. Originally a Wikimedia Foundation software available at https://github.com/wikimedia/mediawiki-services-jobrunner | github.com [17:52:14] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.50, 6.98, 6.06 [17:53:59] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.36, 4.12, 3.61 [17:56:49] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.00, 7.09, 6.47 [17:57:54] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.79, 3.73, 3.60 [17:58:14] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.98, 6.50, 6.20 [17:58:48] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.55, 6.14, 6.20 [18:00:34] paladox: is bast ok? [18:03:46] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.57, 4.04, 3.73 [18:05:43] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.65, 3.82, 3.68 [18:06:17] RECOVERY - mw111 Puppet on mw111 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:07:41] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.62, 4.07, 3.79 [18:09:38] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.95, 3.99, 3.79 [18:11:35] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.26, 4.04, 3.83 [18:13:35] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.92, 3.57, 3.68 [18:13:53] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [18:14:48] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 5 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [18:15:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.04, 3.90, 3.80 [18:15:57] PROBLEM - cp21 Stunnel Http for mw10 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:16:09] PROBLEM - cp31 Stunnel Http for mw10 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:16:10] RhinosF1: yes [18:16:32] PROBLEM - cp30 Stunnel Http for mw10 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:16:41] paladox: it recovered so it must be something transient [18:16:49] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:16:53] PROBLEM - mw10 MediaWiki Rendering on mw10 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:17:02] PROBLEM - cp20 Stunnel Http for mw10 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:17:57] RECOVERY - cp21 Stunnel Http for mw10 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.791 second response time [18:18:09] RECOVERY - cp31 Stunnel Http for mw10 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.052 second response time [18:18:35] RECOVERY - cp30 Stunnel Http for mw10 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 4.125 second response time [18:18:56] RECOVERY - mw10 MediaWiki Rendering on mw10 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 8.423 second response time [18:19:10] RECOVERY - cp20 Stunnel Http for mw10 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 8.633 second response time [18:19:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.39, 3.86, 3.82 [18:20:13] PROBLEM - mw13 MediaWiki Rendering on mw13 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:21:25] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.52, 4.10, 3.91 [18:21:40] PROBLEM - cp21 Stunnel Http for mw13 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:21:43] PROBLEM - cp30 Stunnel Http for mw13 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [18:21:53] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:22:20] RECOVERY - mw13 MediaWiki Rendering on mw13 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 9.672 second response time [18:23:23] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.43, 3.74, 3.79 [18:23:45] RECOVERY - cp21 Stunnel Http for mw13 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 6.123 second response time [18:23:45] RECOVERY - cp30 Stunnel Http for mw13 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 5.908 second response time [18:24:48] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::5ebc/cpweb [18:25:51] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb [18:26:48] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:27:17] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.53, 3.70, 3.73 [18:27:50] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:30:12] [02miraheze/puppet] 07JohnFLewis pushed 032 commits to 03master [+0/-0/±3] 13https://git.io/JSD3n [18:30:13] [02miraheze/puppet] 07JohnFLewis 03140d08d - add new db IPs to ferm [18:30:15] [02miraheze/puppet] 07JohnFLewis 036cf6633 - Merge branch 'master' of github.com:/miraheze/puppet [18:31:11] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.82, 3.70, 3.72 [18:32:58] [02miraheze/puppet] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSD38 [18:32:59] [02miraheze/puppet] 07JohnFLewis 031bacd0c - enclose in () [18:33:11] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.76, 4.27, 3.91 [18:37:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.07, 3.62, 3.72 [18:39:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.26, 4.08, 3.89 [18:41:10] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.92, 3.90, 3.85 [18:54:34] PROBLEM - mwtask111 MediaWiki Rendering on mwtask111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:56:39] !log [rhinos@mwtask111] starting deploy of {'world': True, 'l10n': True} to all [18:56:52] JohnLewis: fixed ish [18:56:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:57:13] JohnLewis: I copied ExtMessageFiles [18:57:33] set off a deploy [18:57:39] hopefully that wakes it up [18:57:44] will do test later [18:58:14] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.85, 6.57, 5.84 [19:00:14] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.43, 6.40, 5.86 [19:01:08] I did all [19:01:15] I'll fix it in a sec [19:03:12] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.78, 2.80, 3.28 [19:04:12] !log [rhinos@mwtask111] DEPLOY ABORTED: Canary check failed for localhost [19:04:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:04:27] !log [rhinos@mwtask111] starting deploy of {'world': True, 'l10n': True} to scsvg [19:04:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:07:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JSDsC [19:07:06] [02miraheze/puppet] 07paladox 03a913575 - mediawiki: Temporarily hardcode new mw server ips for varnish to be able to access [19:07:07] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS [19:07:11] [02puppet] 07paladox opened pull request 03#2253: mediawiki: Temporarily hardcode new mw server ips for varnish to be able to access - 13https://git.io/JSDsW [19:07:29] RhinosF1: JohnLewis ^ [19:09:27] [02puppet] 07paladox closed pull request 03#2253: mediawiki: Temporarily hardcode new mw server ips for varnish to be able to access - 13https://git.io/JSDsW [19:09:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSDsE [19:09:30] [02miraheze/puppet] 07paladox 036144eec - mediawiki: Temporarily hardcode new mw server ips for varnish to be able to access (#2253) [19:09:32] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [19:09:32] paladox: Ok [19:09:33] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbiAS [19:10:46] [02puppet] 07Universal-Omega opened pull request 03#2254: logbot: add MirahezeLSBot_ `author_map` alias - 13https://git.io/JSDsV [19:11:11] oh i hardcoded the wrong ips /me facepalm. [19:11:15] supposed to be cps [19:12:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSDs6 [19:12:32] [02miraheze/puppet] 07paladox 0345df414 - mediawiki: Fix wrong ips being hardcoded [19:13:38] PROBLEM - puppet111 Puppet on puppet111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:13:56] RECOVERY - cp21 Stunnel Http for mw112 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 7.249 second response time [19:14:04] RECOVERY - cp20 Stunnel Http for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.057 second response time [19:14:11] !log [rhinos@mwtask111] DEPLOY ABORTED: Canary check failed for localhost [19:14:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:14:24] RECOVERY - cp30 Stunnel Http for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.307 second response time [19:14:27] RECOVERY - cp20 Stunnel Http for mw112 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.020 second response time [19:14:34] RECOVERY - cp31 Stunnel Http for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.318 second response time [19:14:36] RECOVERY - cp31 Stunnel Http for mw101 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.320 second response time [19:14:38] RECOVERY - cp30 Stunnel Http for mw102 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.322 second response time [19:14:38] RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.055 second response time [19:14:49] RECOVERY - cp20 Stunnel Http for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.019 second response time [19:14:50] RECOVERY - cp21 Stunnel Http for mw102 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.014 second response time [19:14:55] RECOVERY - cp31 Stunnel Http for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.325 second response time [19:14:58] RECOVERY - cp31 Stunnel Http for mw122 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.330 second response time [19:15:04] RECOVERY - cp31 Stunnel Http for mw112 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.334 second response time [19:15:10] RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.332 second response time [19:15:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSDsy [19:15:29] [02miraheze/puppet] 07paladox 034282ef7 - Update mediawiki.pp [19:15:32] RECOVERY - cp21 Stunnel Http for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.019 second response time [19:15:36] RECOVERY - cp30 Stunnel Http for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.307 second response time [19:15:36] RECOVERY - cp21 Stunnel Http for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 0.018 second response time [19:15:39] RECOVERY - cp21 Stunnel Http for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.017 second response time [19:15:41] RECOVERY - cp20 Stunnel Http for mw102 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.018 second response time [19:15:47] RECOVERY - cp20 Stunnel Http for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.017 second response time [19:15:50] RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.314 second response time [19:15:52] RECOVERY - cp30 Stunnel Http for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.323 second response time [19:15:57] RECOVERY - cp30 Stunnel Http for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.324 second response time [19:16:00] RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.018 second response time [19:19:17] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.58, 6.51, 5.90 [19:21:15] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.61, 6.07, 5.80 [19:21:47] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.06, 3.74, 3.34 [19:22:45] PROBLEM - mw101 MediaWiki Rendering on mw101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:23:38] RECOVERY - puppet111 Puppet on puppet111 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [19:23:44] !log [paladox@mwtask111] starting deploy of {'l10n': True, 'force': True} to scsvg [19:23:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:25:15] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.38, 7.28, 6.34 [19:25:41] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.99, 3.90, 3.51 [19:27:15] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.65, 6.50, 6.16 [19:28:23] !log [paladox@mwtask111] finished deploy of {'l10n': True, 'force': True} to scsvg - SUCCESS in 279s [19:28:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:28:44] !log db11 -> db101 replication started [19:28:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:29:36] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.34, 3.21, 3.34 [19:33:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.33, 3.45, 3.43 [19:37:15] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.33, 6.92, 6.50 [19:37:22] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.72, 3.28, 3.38 [19:39:15] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.62, 6.33, 6.35 [19:41:13] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.83, 3.74, 3.56 [19:42:16] !log [paladox@mwtask111] starting deploy of {'l10nupdate': True, 'force': True} to scsvg [19:42:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:46:46] !log [rhinos@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki=loginwiki --force --lang=en (END - exit=0) [19:46:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:47:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.00, 3.73, 3.56 [19:47:15] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.64, 6.89, 6.60 [19:47:35] !log [rhinos@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki=loginwiki --force --lang=en --no-database --skip-message-purge (END - exit=0) [19:47:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:49:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.86, 3.37, 3.45 [19:49:31] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.21, 7.01, 6.61 [19:51:10] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.25, 3.60, 3.51 [19:51:30] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.26, 6.39, 6.43 [19:53:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.14, 3.45, 3.47 [19:55:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.55, 3.11, 3.34 [19:59:15] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.08, 6.69, 6.76 [20:00:53] !log db12->db111 replication started [20:00:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:01:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.86, 3.73, 3.48 [20:03:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.08, 3.47, 3.41 [20:05:07] PROBLEM - mw101 Check Gluster Clients on mw101 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [20:05:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.44, 3.01, 3.25 [20:13:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 4.00, 3.41, 3.28 [20:13:15] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.89, 6.37, 5.98 [20:15:03] !log db13->db121 replication started [20:15:10] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.53, 4.00, 3.50 [20:15:15] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.59, 6.47, 6.07 [20:15:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:17:47] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSDnX [20:17:49] [02miraheze/mw-config] 07Universal-Omega 033fa35c3 - Fix memcached to use hostnames for SCSVG [20:18:55] miraheze/mw-config - Universal-Omega the build passed. [20:19:10] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.16, 3.70, 3.51 [20:19:15] !log [paladox@mwtask111] starting deploy of {'config': True} to scsvg [20:19:36] !log [paladox@mwtask111] DEPLOY ABORTED: Canary check failed for mw101 [20:19:43] !log [paladox@mwtask111] starting deploy of {'config': True, 'force': True} to scsvg [20:19:50] !log [paladox@mwtask111] finished deploy of {'config': True, 'force': True} to scsvg - SUCCESS in 7s [20:19:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:20:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:20:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:21:03] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.29, 6.64, 6.11 [20:21:10] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.43, 7.10, 6.48 [20:21:19] RECOVERY - mwtask111 MediaWiki Rendering on mwtask111 is OK: HTTP OK: HTTP/1.1 200 OK - 23686 bytes in 1.879 second response time [20:21:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:22:24] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::5ebc/cpweb [20:23:04] !log [paladox@mwtask111] starting deploy of {'l10n': True, 'force': True} to scsvg [20:23:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.40, 3.23, 3.35 [20:23:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:23:39] !log [paladox@mwtask111] finished deploy of {'l10n': True, 'force': True} to scsvg - SUCCESS in 34s [20:23:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:24:19] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:24:28] !log [@test3] starting deploy of {'config': True} to skip [20:24:29] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [20:24:49] RECOVERY - mw101 MediaWiki Rendering on mw101 is OK: HTTP OK: HTTP/1.1 200 OK - 23670 bytes in 1.398 second response time [20:24:54] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.74, 6.54, 6.21 [20:24:55] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSDnA [20:24:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:24:56] [02miraheze/mw-config] 07Universal-Omega 03071bdfb - Use hostname for jobchron121 in SCSVG [20:25:07] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.95, 6.54, 6.39 [20:25:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:26:04] miraheze/mw-config - Universal-Omega the build passed. [20:27:39] [02puppet] 07paladox closed pull request 03#2254: logbot: add MirahezeLSBot_ `author_map` alias - 13https://git.io/JSDsV [20:27:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSDcf [20:27:42] [02miraheze/puppet] 07Universal-Omega 0338ee9f4 - logbot: add MirahezeLSBot_ `author_map` alias (#2254) [20:31:07] RECOVERY - mw101 Check Gluster Clients on mw101 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [20:33:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.38, 3.60, 3.37 [20:33:31] !log [paladox@mwtask111] starting deploy of {'l10n': True, 'force': True} to scsvg [20:33:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:34:55] PROBLEM - test101 MediaWiki Rendering on test101 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:35:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.18, 3.37, 3.31 [20:36:22] !log [paladox@mwtask111] finished deploy of {'l10n': True, 'force': True} to scsvg - SUCCESS in 170s [20:36:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:36:45] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 23670 bytes in 1.265 second response time [20:36:56] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 23670 bytes in 1.314 second response time [20:36:57] RECOVERY - mw102 MediaWiki Rendering on mw102 is OK: HTTP OK: HTTP/1.1 200 OK - 23670 bytes in 1.232 second response time [20:37:17] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 18 backends are healthy [20:37:17] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 23670 bytes in 1.276 second response time [20:37:29] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 23670 bytes in 1.277 second response time [20:37:35] RECOVERY - cp31 Varnish Backends on cp31 is OK: All 18 backends are healthy [20:38:05] RECOVERY - cp30 Varnish Backends on cp30 is OK: All 18 backends are healthy [20:38:42] RECOVERY - cp21 Varnish Backends on cp21 is OK: All 18 backends are healthy [20:42:02] !log [@mw11] starting deploy of {'config': True} to ovlon [20:42:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:42:17] !log [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 15s [20:42:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:45:32] [02puppet] 07Universal-Omega opened pull request 03#2255: Delete nutcracker module - 13https://git.io/JSDc6 [20:47:45] !log [@test101] starting deploy of {'config': True} to skip [20:47:47] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 2s [20:47:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:48:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:48:39] RECOVERY - test101 MediaWiki Rendering on test101 is OK: HTTP OK: HTTP/1.1 200 OK - 23707 bytes in 1.372 second response time [20:51:33] !log [@mwtask111] starting deploy of {'config': True} to scsvg [20:51:47] !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 14s [20:52:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:52:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:52:11] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb [20:52:12] [02puppet] 07paladox closed pull request 03#2255: Delete nutcracker module - 13https://git.io/JSDc6 [20:52:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-4/±0] 13https://git.io/JSDc5 [20:52:14] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.14, 6.77, 6.24 [20:52:15] [02miraheze/puppet] 07Universal-Omega 0303542e2 - Delete nutcracker module (#2255) [20:53:46] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.09, 6.68, 6.32 [20:54:06] !log [@test3] starting deploy of {'config': True} to skip [20:54:07] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [20:54:14] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.03, 6.35, 6.14 [20:54:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:54:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:55:45] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.38, 6.61, 6.34 [20:56:11] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:01:10] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.20, 3.44, 3.19 [21:10:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:13:10] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.05, 3.32, 3.31 [21:13:41] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.68, 6.81, 6.29 [21:14:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:15:41] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 5.68, 6.48, 6.24 [21:17:10] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.17, 3.89, 3.53 [21:19:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.83, 3.88, 3.57 [21:22:12] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JSDCb [21:22:14] [02miraheze/mw-config] 07Universal-Omega 03b82bdf1 - Use unique keys for databases between ovlon and SCSVG [21:22:15] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [21:22:17] [02mw-config] 07Universal-Omega opened pull request 03#4338: Use unique keys for databases between ovlon and SCSVG - 13https://git.io/JSDCA [21:23:13] RhinosF1: ^ [21:23:26] miraheze/mw-config - Universal-Omega the build passed. [21:23:27] Mind making sure I didn't miss something? [21:27:04] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 3 datacenters are down: 198.244.148.90/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb [21:27:10] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.90, 2.95, 3.27 [21:27:48] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:28:08] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:28:20] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:28:31] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:29:03] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [21:29:20] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [21:31:21] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 1.560 second response time [21:31:45] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 23675 bytes in 0.332 second response time [21:32:11] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.326 second response time [21:32:19] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.069 second response time [21:32:34] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.592 second response time [21:33:10] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.50, 3.72, 3.47 [21:34:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:35:10] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.67, 3.45, 3.39 [21:36:08] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-3 [+0/-0/±1] 13https://git.io/JSDW0 [21:36:09] [02miraheze/mw-config] 07Universal-Omega 03d2a2ba7 - Update Database.php [21:36:11] [02mw-config] 07Universal-Omega synchronize pull request 03#4338: Use unique keys for databases between ovlon and SCSVG - 13https://git.io/JSDCA [21:36:43] CosmicAlpha: lgtm [21:36:51] Not sure why it's doing it [21:36:58] But +1 [21:37:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.41, 4.07, 3.62 [21:37:32] [02mw-config] 07Universal-Omega closed pull request 03#4338: Use unique keys for databases between ovlon and SCSVG - 13https://git.io/JSDCA [21:37:34] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 - 13https://git.io/vbvb3 [21:37:35] miraheze/mw-config - Universal-Omega the build passed. [21:37:35] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSDWE [21:37:37] [02miraheze/mw-config] 07Universal-Omega 03a2b1248 - Use unique keys for databases between ovlon and SCSVG (#4338) [21:37:39] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-3 [21:38:37] miraheze/mw-config - Universal-Omega the build passed. [21:39:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.54, 3.93, 3.63 [21:41:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.59, 4.19, 3.76 [21:41:32] !log [@mw11] starting deploy of {'config': True} to ovlon [21:41:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:41:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:41:54] !log [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 22s [21:42:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:43:26] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.28, 7.20, 6.47 [21:45:25] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.19, 7.09, 6.52 [21:45:59] RECOVERY - db121 MariaDB on db121 is OK: Uptime: 69 Threads: 3 Questions: 90 Slow queries: 12 Opens: 34 Open tables: 30 Queries per second avg: 1.304 [21:47:13] !log [@test101] starting deploy of {'config': True} to skip [21:47:15] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 1s [21:47:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:47:24] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.42, 6.66, 6.42 [21:47:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:49:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.71, 3.79, 3.80 [21:49:58] PROBLEM - db121 MariaDB on db121 is CRITICAL: Access denied for user 'icinga'@'2a10:6740::6:205' (using password: YES) [21:51:29] !log [@mwtask111] starting deploy of {'config': True} to scsvg [21:51:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:51:42] !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 13s [21:52:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:54:49] !log [@test3] starting deploy of {'config': True} to skip [21:54:50] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [21:55:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:55:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:57:00] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.71, 2.27, 1.45 [21:57:10] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.25, 3.70, 3.69 [21:58:22] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSDle [21:58:24] [02miraheze/mw-config] 07Universal-Omega 03cf6f7ae - Add note about why the 'echo' cluster echo's c1 [21:58:58] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.25, 1.83, 1.39 [21:59:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.27, 3.46, 3.60 [21:59:26] miraheze/mw-config - Universal-Omega the build passed. [22:00:44] CosmicAlpha: metawiki isn't on c1 though :) [22:00:57] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.48, 1.39, 1.28 [22:01:19] JohnLewis: Oh. Well then the comment is wrong. It is db11, but not c1 I guess? [22:01:35] db11 is c1 [22:01:39] but meta isn't on c1 :P [22:02:28] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSDlm [22:02:30] [02miraheze/mw-config] 07Universal-Omega 03b0df421 - Fix comment (c1 -> c2) [22:03:05] Thanks JohnLewis! [22:03:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.15, 2.81, 3.31 [22:03:39] miraheze/mw-config - Universal-Omega the build passed. [22:04:10] db11 is c1 and c2 I guess. But yeah that echo cluster is there for metawiki cross wiki echo notifications database, so thought I'd mention that. Fixed. [22:05:23] CosmicAlpha: did att work? [22:05:24] PROBLEM - db121 NTP time on db121 is CRITICAL: connect to address 2a10:6740::6:302 port 5666: No route to hostconnect to host 2a10:6740::6:302 port 5666: No route to host [22:05:30] PROBLEM - db121 SSH on db121 is CRITICAL: connect to address 2a10:6740::6:302 and port 22: No route to host [22:05:48] PROBLEM - db121 Disk Space on db121 is CRITICAL: connect to address 2a10:6740::6:302 port 5666: No route to hostconnect to host 2a10:6740::6:302 port 5666: No route to host [22:06:00] PROBLEM - Host db121 is DOWN: CRITICAL - Destination Unreachable (2a10:6740::6:302) [22:06:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:07:08] RhinosF1: the PR wasn't to fix db121 (on its own) but rather better conventions then the SCSVG and Ovlon sharing keys. [22:08:03] RECOVERY - Host db121 is UP: PING OK - Packet loss = 0%, RTA = 0.76 ms [22:08:11] PROBLEM - db121 Current Load on db121 is CRITICAL: connect to address 2a10:6740::6:302 port 5666: Connection refusedconnect to host 2a10:6740::6:302 port 5666: Connection refused [22:08:11] PROBLEM - db121 conntrack_table_size on db121 is CRITICAL: connect to address 2a10:6740::6:302 port 5666: Connection refusedconnect to host 2a10:6740::6:302 port 5666: Connection refused [22:08:31] PROBLEM - db121 Puppet on db121 is CRITICAL: connect to address 2a10:6740::6:302 port 5666: Connection refusedconnect to host 2a10:6740::6:302 port 5666: Connection refused [22:08:41] PROBLEM - db121 APT on db121 is CRITICAL: connect to address 2a10:6740::6:302 port 5666: Connection refusedconnect to host 2a10:6740::6:302 port 5666: Connection refused [22:08:46] PROBLEM - db121 PowerDNS Recursor on db121 is CRITICAL: connect to address 2a10:6740::6:302 port 5666: Connection refusedconnect to host 2a10:6740::6:302 port 5666: Connection refused [22:08:56] PROBLEM - db121 ferm_active on db121 is CRITICAL: connect to address 2a10:6740::6:302 port 5666: Connection refusedconnect to host 2a10:6740::6:302 port 5666: Connection refused [22:09:10] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.79, 3.65, 3.54 [22:11:34] !log [@mw11] starting deploy of {'config': True} to ovlon [22:11:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:11:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:11:40] !log [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 6s [22:11:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:13:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.37, 3.16, 3.38 [22:15:15] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.51, 6.77, 5.94 [22:17:15] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.59, 6.70, 6.02 [22:17:15] PROBLEM - ping6 on db121 is CRITICAL: CRITICAL - Destination Unreachable (2a10:6740::6:302) [22:17:37] [02puppet] 07Universal-Omega opened pull request 03#2256: deploy-mediawiki: use bast rather than bast101 directly - 13https://git.io/JSDlB [22:17:49] !log [@test101] starting deploy of {'config': True} to skip [22:17:50] !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 1s [22:18:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:18:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:19:15] RECOVERY - ping6 on db121 is OK: PING OK - Packet loss = 0%, RTA = 2.80 ms [22:19:48] [02puppet] 07paladox closed pull request 03#2256: deploy-mediawiki: use bast rather than bast101 directly - 13https://git.io/JSDlB [22:19:49] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSDl0 [22:19:51] [02miraheze/puppet] 07Universal-Omega 03fa51e79 - deploy-mediawiki: use bast rather than bast101 directly (#2256) [22:20:40] CosmicAlpha: att connects to db121 via sql.php [22:21:15] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.70, 6.93, 6.24 [22:21:43] !log [@mwtask111] starting deploy of {'config': True} to scsvg [22:21:56] RhinosF1: even when it said db13 it was db121, the message displayed uses the keys, not the actual server afaik. [22:21:57] !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 14s [22:21:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:21:58] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.41, 3.64, 3.45 [22:22:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:22:40] CosmicAlpha: ok [22:22:51] CosmicAlpha: db111 doesn't work [22:23:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.05, 3.30, 3.34 [22:24:02] !log [@test3] starting deploy of {'config': True} to skip [22:24:03] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [22:24:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:24:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:24:34] RhinosF1: Oh. How so? I didnt think there was a db111 issue. [22:24:48] CosmicAlpha: same as what paladox got on att [22:24:52] Oh [22:24:54] it's just lag [22:25:15] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.66, 6.52, 6.25 [22:25:41] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.21, 1.71, 1.26 [22:26:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:27:26] RECOVERY - db121 SSH on db121 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [22:27:39] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.26, 1.50, 1.24 [22:29:22] !log [rhinos@mwtask111] sudo -u www-data php /srv/mediawiki/w/extensions/CreateWiki/maintenance/generateMissingCache.php --wiki=loginwiki (END - exit=0) [22:29:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:30:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [22:33:34] !log [rhinos@mwtask111] starting deploy of {'folders': 'cache'} to scsvg [22:33:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:33:56] !log [rhinos@mwtask111] finished deploy of {'folders': 'cache'} to scsvg - FAIL: [5888, 0, 0, 0, 0, 0, 0] in 21s [22:34:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:34:58] PROBLEM - db121 MariaDB on db121 is UNKNOWN: [22:35:56] RECOVERY - db121 Disk Space on db121 is OK: DISK OK - free space: / 621142 MB (99% inode=99%); [22:36:12] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 0.88, 1.07, 0.60 [22:36:18] PROBLEM - db121 conntrack_table_size on db121 is UNKNOWN: NRPE: Unable to read output [22:36:49] RECOVERY - db121 APT on db121 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [22:36:52] RECOVERY - db121 PowerDNS Recursor on db121 is OK: DNS OK: 0.413 seconds response time. miraheze.org returns 198.244.148.90,2001:41d0:801:2000::1b80,2001:41d0:801:2000::4c25 [22:36:57] PROBLEM - db121 MariaDB on db121 is CRITICAL: Can't connect to MySQL server on 'db121.miraheze.org' (115) [22:36:59] RECOVERY - db121 ferm_active on db121 is OK: OK ferm input default policy is set [22:37:02] RECOVERY - db121 NTP time on db121 is OK: NTP OK: Offset 0.0008642971516 secs [22:38:15] RECOVERY - db121 conntrack_table_size on db121 is OK: OK: nf_conntrack is 0 % full [22:38:46] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 1.72, 2.61, 1.63 [22:39:28] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.76, 3.61, 3.41 [22:40:35] RECOVERY - db121 Puppet on db121 is OK: OK: Puppet is currently enabled, last run 52 seconds ago with 0 failures [22:42:36] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.43, 1.48, 1.39 [22:45:19] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.74, 3.97, 3.61 [22:47:16] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.78, 3.61, 3.51 [22:51:11] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.21, 3.25, 3.39 [22:51:57] PROBLEM - mwtask1 Disk Space on mwtask1 is WARNING: DISK WARNING - free space: / 5108 MB (10% inode=89%); [22:55:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.65, 3.61, 3.52 [22:59:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.19, 3.04, 3.32 [23:00:01] !log [@test101] starting deploy of {'l10nupdate': True} to skip [23:00:01] !log [@test3] starting deploy of {'l10nupdate': True} to skip [23:00:02] !log [@mwtask111] starting deploy of {'l10nupdate': True} to scsvg [23:00:02] !log [@mw11] starting deploy of {'l10nupdate': True} to ovlon [23:00:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:47] !log [@test101] finished deploy of {'l10nupdate': True} to skip - SUCCESS in 46s [23:00:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:01:36] !log [@mwtask111] finished deploy of {'l10nupdate': True} to scsvg - SUCCESS in 94s [23:01:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:06:01] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.54, 4.02, 3.65 [23:13:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSD8d [23:13:41] [02miraheze/puppet] 07paladox 032a05bef - prometheus: Support bullseye for blackbox [23:16:16] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JSD4v [23:16:18] [02miraheze/puppet] 07paladox 03f951b2d - prometheus: Use mw111 on new cluster [23:16:19] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS [23:16:21] [02puppet] 07paladox opened pull request 03#2257: prometheus: Use mw111 on new cluster - 13https://git.io/JSD4f [23:16:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JSD4U [23:16:58] [02miraheze/puppet] 07paladox 0343f9ba1 - Update blackbox_exporter.pp [23:16:59] [02puppet] 07paladox synchronize pull request 03#2257: prometheus: Use mw111 on new cluster - 13https://git.io/JSD4f [23:17:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JSD4I [23:17:22] [02miraheze/puppet] 07paladox 03a27ce6a - Update blackbox_exporter.yml.erb [23:17:23] [02puppet] 07paladox synchronize pull request 03#2257: prometheus: Use mw111 on new cluster - 13https://git.io/JSD4f [23:17:28] [02puppet] 07paladox closed pull request 03#2257: prometheus: Use mw111 on new cluster - 13https://git.io/JSD4f [23:17:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JSD4L [23:17:31] [02miraheze/puppet] 07paladox 03f4c7cad - prometheus: Use mw111 on new cluster (#2257) [23:17:33] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [23:17:34] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbiAS [23:17:44] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.64, 3.73, 3.79 [23:18:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSD4q [23:18:13] [02miraheze/puppet] 07paladox 03a2be1b8 - Fix [23:20:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:21:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JSD44 [23:21:46] [02miraheze/puppet] 07paladox 03791f976 - prometheus: Fix blackbox [23:21:48] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS [23:21:49] [02puppet] 07paladox opened pull request 03#2258: prometheus: Fix blackbox - 13https://git.io/JSD4B [23:22:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JSD40 [23:22:38] [02miraheze/puppet] 07paladox 038484338 - Update prometheus.yml.erb [23:22:39] [02puppet] 07paladox synchronize pull request 03#2258: prometheus: Fix blackbox - 13https://git.io/JSD4B [23:22:57] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JSD4E [23:22:59] [02miraheze/puppet] 07paladox 03890e656 - Update blackbox_exporter.pp [23:23:00] [02puppet] 07paladox synchronize pull request 03#2258: prometheus: Fix blackbox - 13https://git.io/JSD4B [23:23:16] [02puppet] 07paladox closed pull request 03#2258: prometheus: Fix blackbox - 13https://git.io/JSD4B [23:23:17] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbiAS [23:23:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/JSD4z [23:23:20] [02miraheze/puppet] 07paladox 039844fb1 - prometheus: Fix blackbox (#2258) [23:23:22] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [23:23:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:26:50] PROBLEM - test3 APT on test3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:26:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JSD4X [23:26:57] [02miraheze/puppet] 07paladox 03e8b64a1 - prometheus: Monitor the new mw cluster using the new monitoring host (blackbox) [23:26:58] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 4.78, 6.81, 4.27 [23:27:36] PROBLEM - test3 Current Load on test3 is CRITICAL: CRITICAL - load average: 11.38, 4.88, 2.11 [23:27:55] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 24.58, 20.55, 17.72 [23:28:53] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 3.14, 5.55, 4.10 [23:29:27] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.71, 3.82, 3.64 [23:29:51] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.22, 7.30, 6.46 [23:29:55] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.95, 20.99, 18.26 [23:30:48] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 3.07, 4.82, 4.01 [23:31:00] RECOVERY - test3 APT on test3 is OK: APT OK: 19 packages available for upgrade (0 critical updates). [23:31:25] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.86, 3.52, 3.56 [23:31:49] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.42, 6.94, 6.44 [23:31:55] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 18.94, 19.90, 18.17 [23:33:22] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.77, 3.75, 3.62 [23:33:36] RECOVERY - test3 Current Load on test3 is OK: OK - load average: 1.43, 3.23, 2.39 [23:34:29] !log [@test3] finished deploy of {'l10nupdate': True} to skip - SUCCESS in 2066s [23:34:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:35:41] !log [@mw11] finished deploy of {'l10nupdate': True} to ovlon - SUCCESS in 2139s [23:35:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:35:46] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.70, 6.57, 6.43 [23:37:16] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.84, 3.47, 3.55 [23:38:29] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 4.12, 5.99, 4.95 [23:42:19] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 4.17, 5.07, 4.81 [23:43:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:46:10] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 5.23, 7.24, 5.88 [23:47:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:48:07] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 3.59, 5.95, 5.56 [23:49:10] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.10, 3.15, 3.35 [23:49:15] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.65, 7.14, 6.01 [23:50:14] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.81, 6.23, 5.53 [23:51:15] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.43, 6.71, 5.98 [23:52:07] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 2.09, 3.84, 4.78 [23:52:14] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.75, 5.97, 5.52 [23:53:10] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.16, 3.43, 3.44 [23:57:09] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.68, 3.13, 3.33