[00:00:19] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN [00:01:42] RECOVERY - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is OK: OK - NGINX Error Rate is 25% [00:02:19] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [00:04:06] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 21281MiB (24% inode=98%); [00:04:10] RECOVERY - cp41 Disk Space on cp41 is OK: DISK OK - free space: / 19476MiB (20% inode=98%); [00:05:30] RECOVERY - cp36 Disk Space on cp36 is OK: DISK OK - free space: / 21254MiB (24% inode=98%); [00:05:41] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 50% [00:09:41] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 63% [00:42:53] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:01:10] PROBLEM - graphite151 NTP time on graphite151 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o ] [01:03:02] PROBLEM - graphite151 NTP time on graphite151 is CRITICAL: NTP CRITICAL: Offset 165.987128 secs [01:12:27] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [03:01:15] RECOVERY - db151 Backups SQL on db151 is OK: FILE_AGE OK: /var/log/sql-backup.log is 74 seconds old and 0 bytes [07:10:20] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.84 ms [07:14:24] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.15 ms [07:15:28] PROBLEM - cloud18 Puppet on cloud18 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [07:41:39] RECOVERY - cloud18 Puppet on cloud18 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:34:16] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [08:52:03] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [08:54:28] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 176.65 ms [08:56:29] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 173.73 ms [09:21:46] RECOVERY - wiki.gab.pt.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.gab.pt.eu.org reverse DNS resolves to cp37.wikitide.net - CNAME OK [09:33:06] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp37.wikitide.net - CNAME OK [10:05:51] PROBLEM - glitchcity.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'glitchcity.wiki' expires in 15 day(s) (Sun 28 Apr 2024 09:52:33 AM GMT +0000). [10:06:08] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a30b5ad81f55...b955931670b1 [10:06:09] [02miraheze/ssl] 07WikiTideSSLBot 03b955931 - Bot: Update SSL cert for glitchcity.wiki [10:07:17] PROBLEM - www.glitchcity.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'glitchcity.wiki' expires in 15 day(s) (Sun 28 Apr 2024 09:52:33 AM GMT +0000). [10:32:57] PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [10:35:13] RECOVERY - glitchcity.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'glitchcity.wiki' will expire on Thu 11 Jul 2024 09:06:00 AM GMT +0000. [10:36:33] RECOVERY - www.glitchcity.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'glitchcity.wiki' will expire on Thu 11 Jul 2024 09:06:00 AM GMT +0000. [10:39:46] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.48 ms [10:41:47] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.48 ms [11:01:24] RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [11:12:20] PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 13.38, 23.19, 10.97 [11:16:20] PROBLEM - db181 Current Load on db181 is WARNING: LOAD WARNING - total load average: 0.38, 10.47, 8.50 [11:18:20] RECOVERY - db181 Current Load on db181 is OK: LOAD OK - total load average: 0.24, 7.08, 7.49 [11:21:20] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 206.45 ms [11:23:21] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 186.43 ms [11:39:32] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:13:36] [02CreateWiki] 07Reception123 synchronize pull request 03#508: no dismiss for more details; bump priorities - 13https://github.com/miraheze/CreateWiki/pull/508 [12:13:37] [02miraheze/CreateWiki] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/8b9ff4afa4e9...7b4eef844131 [12:13:38] [02miraheze/CreateWiki] 07Reception123 037b4eef8 - fixes per CA's suggestions [12:16:35] miraheze/CreateWiki - Reception123 the build has errored. [12:25:38] [02CreateWiki] 07Reception123 synchronize pull request 03#508: no dismiss for more details; bump priorities - 13https://github.com/miraheze/CreateWiki/pull/508 [12:25:38] [02miraheze/CreateWiki] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/7b4eef844131...18851d460e1b [12:25:41] [02miraheze/CreateWiki] 07Reception123 0318851d4 - minor syntax [12:33:56] miraheze/CreateWiki - Reception123 the build passed. [12:38:37] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [14:34:32] [02MirahezeMagic] 07amire80 opened pull request 03#487: Consistent CSS and JS spelling in messages - 13https://github.com/miraheze/MirahezeMagic/pull/487 [14:38:08] miraheze/MirahezeMagic - amire80 the build has errored. [15:52:02] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 170.11 ms [15:54:03] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.67 ms [16:33:35] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 226.83 ms [16:37:38] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 183.71 ms [16:38:57] !log [alex@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/namespaceDupes.php --wiki=1d6chanwiki (END - exit=0) [16:39:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:41:49] [02MirahezeMagic] 07Universal-Omega closed pull request 03#487: Consistent CSS and JS spelling in messages - 13https://github.com/miraheze/MirahezeMagic/pull/487 [16:41:50] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/f4712655fea6...74d61e52fe85 [16:41:51] [02miraheze/MirahezeMagic] 07amire80 0374d61e5 - Consistent CSS and JS spelling in messages (#487) [16:45:45] miraheze/MirahezeMagic - Universal-Omega the build has errored. [16:54:03] !log [@test151] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 [16:54:04] !log [@test151] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [16:54:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:54:12] !log [@test151] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 [16:54:13] !log [@test151] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [16:54:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:54:22] !log [@test151] starting deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151 [16:54:23] !log [@test151] finished deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [16:54:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:54:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:54:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:54:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:02:52] !log [@mwtask181] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all [17:03:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:03:02] !log [@mwtask181] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 10s [17:03:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:03:12] !log [@mwtask181] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all [17:03:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:03:22] !log [@mwtask181] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all - SUCCESS in 9s [17:03:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:04:11] PROBLEM - cp41 Disk Space on cp41 is WARNING: DISK WARNING - free space: / 10229MiB (10% inode=98%); [17:07:12] !log [@mwtask171] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all [17:07:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:07:20] !log [@mwtask171] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 8s [17:07:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:07:30] !log [@mwtask171] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all [17:07:38] !log [@mwtask171] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all - SUCCESS in 7s [17:07:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:07:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:21:00] [02mw-config] 07redbluegreenhat opened pull request 03#5539: T11982: Create edittrusteduser protection level in phightingwiki - 13https://github.com/miraheze/mw-config/pull/5539 [17:21:56] miraheze/mw-config - redbluegreenhat the build has errored. [17:22:54] [02mw-config] 07redbluegreenhat edited pull request 03#5539: T11982: Create edittrusteduserprotected protection level in phightingwiki - 13https://github.com/miraheze/mw-config/pull/5539 [17:23:48] [02mw-config] 07redbluegreenhat synchronize pull request 03#5539: T11982: Create edittrusteduserprotected protection level in phightingwiki - 13https://github.com/miraheze/mw-config/pull/5539 [17:24:29] [02mw-config] 07redbluegreenhat synchronize pull request 03#5539: T11982: Create edittrusteduserprotected protection level in phightingwiki - 13https://github.com/miraheze/mw-config/pull/5539 [17:24:31] miraheze/mw-config - redbluegreenhat the build has errored. [17:25:15] [02mw-config] 07redbluegreenhat synchronize pull request 03#5539: T11982: Create edittrusteduserprotected protection level in phightingwiki - 13https://github.com/miraheze/mw-config/pull/5539 [17:25:22] miraheze/mw-config - redbluegreenhat the build passed. [17:26:04] miraheze/mw-config - redbluegreenhat the build passed. [17:28:49] [02mw-config] 07redbluegreenhat closed pull request 03#5539: T11982: Create edittrusteduserprotected protection level in phightingwiki - 13https://github.com/miraheze/mw-config/pull/5539 [17:28:50] [02miraheze/mw-config] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/ec19a75bfd1c...cfb36c7ac088 [17:28:53] [02miraheze/mw-config] 07redbluegreenhat 03cfb36c7 - T11982: Create edittrusteduserprotected protection level in phightingwiki (#5539) [17:29:35] !log [alex@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [17:29:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:29:46] miraheze/mw-config - redbluegreenhat the build passed. [17:29:46] !log [alex@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 11s [17:29:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:36:54] !log [@mwtask171] starting deploy of {'config': True} to all [17:37:02] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 8s [17:37:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:37:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:39:12] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [17:54:03] !log [@test151] starting deploy of {'config': True} to test151 [17:54:04] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [17:54:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:54:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:55:57] [02MirahezeMagic] 07amire80 opened pull request 03#488: Consistent capitalization in a message - 13https://github.com/miraheze/MirahezeMagic/pull/488 [17:59:41] miraheze/MirahezeMagic - amire80 the build has errored. [18:08:44] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [18:11:27] PROBLEM - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [18:40:20] RECOVERY - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mahdiruiz.line.pm' will expire on Fri 14 Jun 2024 04:28:50 PM GMT +0000. [18:50:30] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 171.55 ms [18:56:35] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.04 ms [19:07:46] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [19:37:19] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [20:52:26] [02miraheze/ManageWiki] 07OAuthority pushed 034 commits to 03master [+0/-0/±35] 13https://github.com/miraheze/ManageWiki/compare/26af4bc9e84f...884494362f78 [20:52:29] [02miraheze/ManageWiki] 07OAuthority 03decd0f1 - change phab link [20:52:31] [02miraheze/ManageWiki] 07OAuthority 0308d57f3 - Merge branch 'miraheze:master' into master [20:52:33] [02miraheze/ManageWiki] 07OAuthority 035eb9e9a - Merge branch 'miraheze:master' into master [20:52:35] [02miraheze/ManageWiki] 07OAuthority 038844943 - Merge branch 'master' into master [20:57:02] miraheze/ManageWiki - OAuthority the build passed. [21:35:03] !log [alex@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/cleanupTitles.php --wiki=commonswiki (END - exit=0) [21:35:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:12:06] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 9698MiB (10% inode=98%); [22:15:30] PROBLEM - cp36 Disk Space on cp36 is WARNING: DISK WARNING - free space: / 9708MiB (10% inode=98%); [22:35:14] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/b955931670b1...7120969d89b9 [22:35:16] [02miraheze/ssl] 07WikiTideSSLBot 037120969 - Bot: Add SSL cert for baubles.me [23:06:15] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [23:14:28] !log [macfan@test151] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php --wiki=metawikibeta /srv/mediawiki/1.42/maintenance/archives (END - exit=0) [23:14:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:15:07] !log [macfan@test151] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php --wiki=metawikibeta /srv/mediawiki/1.42/maintenance/archives/patch-user-user_is_temp.sql (END - exit=0) [23:15:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:24:27] !log [macfan@test151] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/sql.php --wiki=testwikibeta /srv/mediawiki/1.42/maintenance/archives/patch-user-user_is_temp.sql (END - exit=256) [23:24:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:41:20] !log [macfan@test151] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/extensions/CentralAuth/maintenance/createLocalAccount.php --wiki=testwikibeta Rodejong (END - exit=0) [23:41:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log