[00:00:38] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/1b3fa737e5e5...0457544b2910 [00:00:39] [02miraheze/dns] 07Universal-Omega 030457544 - miraheze.wiki: remove phab [00:03:06] PROBLEM - cp36 Disk Space on cp36 is CRITICAL: DISK CRITICAL - free space: / 4713MiB (5% inode=98%); [00:04:16] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 20803MiB (23% inode=98%); [00:05:06] RECOVERY - cp36 Disk Space on cp36 is OK: DISK OK - free space: / 20688MiB (23% inode=98%); [00:21:11] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [00:21:40] PROBLEM - db161 SSH on db161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:21:48] PROBLEM - db161 Current Load on db161 is CRITICAL: LOAD CRITICAL - total load average: 82.64, 32.15, 12.26 [00:24:54] PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:25:22] PROBLEM - db161 PowerDNS Recursor on db161 is CRITICAL: CRITICAL - Plugin timed out while executing system call [00:26:23] PROBLEM - db161 Puppet on db161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:27:07] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [00:27:08] PROBLEM - db161 APT on db161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:28:31] CosmicAlpha: ^ [00:29:32] PROBLEM - db161 ferm_active on db161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:29:33] PROBLEM - db161 Backups SQL on db161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:29:38] PROBLEM - db161 Backups SQL mhglobal on db161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:30:25] PROBLEM - db161 conntrack_table_size on db161 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:31:36] oh [00:35:48] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 7 backends are down. mw151 mw152 mw162 mw171 mw172 mw181 mw182 [00:36:35] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 4 backends are down. mw152 mw161 mw181 mw182 [00:36:44] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 3 backends are down. mw161 mw162 mw181 [00:38:55] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 6 backends are down. mw152 mw161 mw162 mw171 mw181 mw182 [00:39:04] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 5 backends are down. mw151 mw152 mw161 mw162 mw182 [00:39:04] PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 6 backends are down. mw151 mw152 mw161 mw162 mw171 mw182 [00:40:35] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 18 backends are healthy [00:40:44] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 18 backends are healthy [00:40:50] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 18 backends are healthy [00:41:04] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 18 backends are healthy [00:41:05] RECOVERY - cp27 Varnish Backends on cp27 is OK: All 18 backends are healthy [00:41:17] RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.451 second response time [00:41:23] RECOVERY - db161 Backups SQL on db161 is OK: FILE_AGE OK: /var/log/sql-backup.log is 173037 seconds old and 113006 bytes [00:41:26] RECOVERY - db161 Backups SQL mhglobal on db161 is OK: FILE_AGE OK: /var/log/sql-mhglobal-backup-weekly.log is 330078 seconds old and 208 bytes [00:41:33] RECOVERY - db161 PowerDNS Recursor on db161 is OK: DNS OK: 0.054 seconds response time. miraheze.org returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [00:41:48] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 18 backends are healthy [00:42:05] RECOVERY - db161 ferm_active on db161 is OK: OK ferm input default policy is set [00:42:07] RECOVERY - db161 conntrack_table_size on db161 is OK: OK: nf_conntrack is 0 % full [00:42:11] RECOVERY - db161 Current Load on db161 is OK: LOAD OK - total load average: 0.71, 0.31, 0.11 [00:42:12] RECOVERY - db161 Puppet on db161 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [00:42:19] RECOVERY - db161 APT on db161 is OK: APT OK: 8 packages available for upgrade (0 critical updates). [00:42:28] PROBLEM - apocrypha.wiki - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [00:42:39] RECOVERY - db161 SSH on db161 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [00:59:19] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:11:10] RECOVERY - apocrypha.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'apocrypha.wiki' will expire on Fri 01 Mar 2024 12:06:59 PM GMT +0000. [01:33:24] [02dns] 07MacFan4000 opened pull request 03#491: remove status.miraheze.wiki - 13https://github.com/miraheze/dns/pull/491 [01:52:46] [02dns] 07Universal-Omega closed pull request 03#491: remove status.miraheze.wiki - 13https://github.com/miraheze/dns/pull/491 [01:52:49] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/0457544b2910...fae26b463725 [01:52:52] [02miraheze/dns] 07MacFan4000 03fae26b4 - remove status.miraheze.wiki (#491) [01:55:51] RECOVERY - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is OK: OK - NGINX Error Rate is 38% [01:57:01] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 41% [02:10:30] PROBLEM - apocrypha.wiki - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [02:11:51] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is WARNING: WARNING - NGINX Error Rate is 40% [02:39:12] RECOVERY - apocrypha.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'apocrypha.wiki' will expire on Fri 01 Mar 2024 12:06:59 PM GMT +0000. [02:57:09] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [03:01:06] PROBLEM - mon181 Backups Grafana on mon181 is CRITICAL: FILE_AGE CRITICAL: /var/log/grafana-backup.log is 1209644 seconds old and 92 bytes [03:01:44] RECOVERY - db162 Backups SQL on db162 is OK: FILE_AGE OK: /var/log/sql-backup.log is 89 seconds old and 1214 bytes [03:06:11] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03relaybot2 [+1/-0/±0] 13https://github.com/miraheze/puppet/commit/81d27f912575 [03:06:12] [02miraheze/puppet] 07Universal-Omega 0381d27f9 - Add second relaybot instance [03:06:13] [02puppet] 07Universal-Omega created branch 03relaybot2 - 13https://github.com/miraheze/puppet [03:06:16] [02puppet] 07Universal-Omega opened pull request 03#3748: Add second relaybot instance - 13https://github.com/miraheze/puppet/pull/3748 [03:07:29] [02puppet] 07Universal-Omega synchronize pull request 03#3748: Add second relaybot instance - 13https://github.com/miraheze/puppet/pull/3748 [03:07:30] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03relaybot2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/81d27f912575...e99242231221 [03:07:31] [02miraheze/puppet] 07Universal-Omega 03e992422 - Update relaybot2.pp [03:10:05] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03relaybot2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e99242231221...d9a5689fe30b [03:10:07] [02miraheze/puppet] 07Universal-Omega 03d9a5689 - Update relaybot.systemd.erb [03:10:09] [02puppet] 07Universal-Omega synchronize pull request 03#3748: Add second relaybot instance - 13https://github.com/miraheze/puppet/pull/3748 [03:21:51] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03relaybot2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d9a5689fe30b...eab9adeb2ae8 [03:21:52] [02miraheze/puppet] 07Universal-Omega 03eab9ade - Update relaybot.pp [03:21:53] [02puppet] 07Universal-Omega synchronize pull request 03#3748: Add second relaybot instance - 13https://github.com/miraheze/puppet/pull/3748 [03:22:09] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03relaybot2 [+0/-1/±0] 13https://github.com/miraheze/puppet/compare/eab9adeb2ae8...d99bd2c3a7b2 [03:22:11] [02miraheze/puppet] 07Universal-Omega 03d99bd2c - Delete modules/irc/manifests/relaybot2.pp [03:22:12] [02puppet] 07Universal-Omega synchronize pull request 03#3748: Add second relaybot instance - 13https://github.com/miraheze/puppet/pull/3748 [03:23:13] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03relaybot2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d99bd2c3a7b2...e3f2ee339a1a [03:23:16] [02miraheze/puppet] 07Universal-Omega 03e3f2ee3 - Update relaybot.systemd.erb [03:23:17] [02puppet] 07Universal-Omega synchronize pull request 03#3748: Add second relaybot instance - 13https://github.com/miraheze/puppet/pull/3748 [03:25:29] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03relaybot2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e3f2ee339a1a...bcfb44cab6cd [03:25:32] [02miraheze/puppet] 07Universal-Omega 03bcfb44c - Update irc.pp [03:25:35] [02puppet] 07Universal-Omega synchronize pull request 03#3748: Add second relaybot instance - 13https://github.com/miraheze/puppet/pull/3748 [03:26:13] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03relaybot2 [+1/-1/±0] 13https://github.com/miraheze/puppet/compare/bcfb44cab6cd...edea8e296da0 [03:26:15] [02miraheze/puppet] 07Universal-Omega 03edea8e2 - Rename config.ini.erb to config-relaybot.ini.erb [03:26:18] [02puppet] 07Universal-Omega synchronize pull request 03#3748: Add second relaybot instance - 13https://github.com/miraheze/puppet/pull/3748 [03:27:06] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03relaybot2 [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/edea8e296da0...c3688291420b [03:27:08] [02miraheze/puppet] 07Universal-Omega 03c368829 - Create config-relaybot2.ini.erb [03:27:10] [02puppet] 07Universal-Omega synchronize pull request 03#3748: Add second relaybot instance - 13https://github.com/miraheze/puppet/pull/3748 [03:34:47] PROBLEM - bots171 Puppet on bots171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [03:36:14] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03relaybot2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c3688291420b...72a95e948a06 [03:36:17] [02miraheze/puppet] 07Universal-Omega 0372a95e9 - Update relaybot.pp [03:36:19] [02puppet] 07Universal-Omega synchronize pull request 03#3748: Add second relaybot instance - 13https://github.com/miraheze/puppet/pull/3748 [03:38:48] RECOVERY - bots171 Puppet on bots171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:40:15] PROBLEM - bots171 IRC-Discord Relay Bot on bots171 is CRITICAL: PROCS CRITICAL: 4 processes with args 'relaybot' [03:42:03] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03relaybot2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/72a95e948a06...0ad18c227680 [03:42:05] [02miraheze/puppet] 07Universal-Omega 030ad18c2 - Update config-relaybot2.ini.erb [03:42:06] [02puppet] 07Universal-Omega synchronize pull request 03#3748: Add second relaybot instance - 13https://github.com/miraheze/puppet/pull/3748 [03:49:56] PROBLEM - mon181 Check correctness of the icinga configuration on mon181 is CRITICAL: Icinga configuration contains errors [03:50:19] [02puppet] 07Universal-Omega synchronize pull request 03#3748: Add second relaybot instance - 13https://github.com/miraheze/puppet/pull/3748 [03:50:22] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03relaybot2 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0ad18c227680...7d88a4e9c6fd [03:50:24] [02miraheze/puppet] 07Universal-Omega 037d88a4e - Update relaybot.pp [03:52:16] RECOVERY - bots171 IRC-Discord Relay Bot on bots171 is OK: PROCS OK: 4 processes with args 'relaybot' [03:53:55] RECOVERY - mon181 Check correctness of the icinga configuration on mon181 is OK: Icinga configuration is correct [03:56:19] PROBLEM - elephants.ggreig.com - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query elephants.ggreig.com. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:58:08] PROBLEM - bots171 IRC-Discord Relay Bot on bots171 is CRITICAL: PROCS CRITICAL: 2 processes with args 'relaybot' [04:02:08] RECOVERY - bots171 IRC-Discord Relay Bot on bots171 is OK: PROCS OK: 4 processes with args 'relaybot' [04:06:00] [02puppet] 07Universal-Omega closed pull request 03#3748: Add second relaybot instance - 13https://github.com/miraheze/puppet/pull/3748 [04:06:03] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+2/-1/±3] 13https://github.com/miraheze/puppet/compare/dc6864972f87...72062a595ab5 [04:06:06] [02miraheze/puppet] 07Universal-Omega 0372062a5 - Add second relaybot instance (#3748) [04:06:08] [02miraheze/puppet] 07Universal-Omega deleted branch 03relaybot2 [04:06:10] [02puppet] 07Universal-Omega deleted branch 03relaybot2 - 13https://github.com/miraheze/puppet [04:25:25] RECOVERY - elephants.ggreig.com - reverse DNS on sslhost is OK: SSL OK - elephants.ggreig.com reverse DNS resolves to cp36.wikitide.net - CNAME OK [05:04:55] PROBLEM - apocrypha.wiki - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [05:33:46] RECOVERY - apocrypha.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'apocrypha.wiki' will expire on Fri 01 Mar 2024 12:06:59 PM GMT +0000. [05:35:04] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/fae26b463725...9b9518c8ed18 [05:35:06] [02miraheze/dns] 07Universal-Omega 039b9518c - Decom WikiTide servers [05:37:48] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/9b9518c8ed18...03c7107ebd49 [05:37:49] [02miraheze/dns] 07Universal-Omega 0303c7107 - Remove cp2 [05:45:41] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[gdnsd-syntax] [05:53:41] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/03c7107ebd49...d5198990000b [05:53:43] [02miraheze/dns] 07Universal-Omega 03d519899 - Remove opensearch CNAME [05:54:20] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[gdnsd-syntax] [05:55:40] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [06:37:05] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [06:53:21] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=klanikwiki --no-updates --username-prefix=wikia:klanik /home/reception/ImportDump/klanikwiki-20240207225911.xml (START) [06:53:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:55:27] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=klanikwiki --no-updates --username-prefix=wikia:klanik /home/reception/ImportDump/klanikwiki-20240207225911.xml (END - exit=0) [06:55:28] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=klanikwiki (START) [06:55:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:55:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:55:55] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=klanikwiki (END - exit=0) [06:55:56] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/initSiteStats.php --wiki=klanikwiki --active --update (END - exit=0) [06:55:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:56:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:14:23] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=tdrwikiwiki --no-updates --username-prefix=jawiki /home/reception/ImportDump/tdrwikiwiki-20240205085226.xml (START) [07:14:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:16:04] !log [agent@db181] Backing up SQL on db181 [07:16:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:19:29] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=tdrwikiwiki --no-updates --username-prefix=jawiki /home/reception/ImportDump/tdrwikiwiki-20240205085226.xml (END - exit=0) [07:19:30] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=tdrwikiwiki (START) [07:19:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:19:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:19:57] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=tdrwikiwiki (END - exit=0) [07:19:58] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/initSiteStats.php --wiki=tdrwikiwiki --active --update (END - exit=0) [07:19:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:20:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:31:51] PROBLEM - apocrypha.wiki - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [07:48:28] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [08:00:39] RECOVERY - apocrypha.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'apocrypha.wiki' will expire on Fri 01 Mar 2024 12:06:59 PM GMT +0000. [08:16:28] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 59% [08:23:35] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/3d04a56f3adb...f6aedb538bd6 [08:23:36] [02miraheze/ssl] 07MirahezeSSLBot 03f6aedb5 - Bot: Add SSL cert for education.gced.in [08:24:27] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [08:25:01] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/f6aedb538bd6...c22350506760 [08:25:02] [02miraheze/ssl] 07MirahezeSSLBot 03c223505 - Bot: Add SSL cert for wiki.chicory.pizza [08:51:38] RECOVERY - storytime.jdstroy.cf - reverse DNS on sslhost is OK: SSL OK - storytime.jdstroy.cf reverse DNS resolves to cp37.wikitide.net - CNAME OK [09:02:14] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 59% [09:29:40] PROBLEM - apocrypha.wiki - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [09:53:54] PROBLEM - storytime.jdstroy.cf - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for storytime.jdstroy.cf could not be found [10:05:12] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 60% [10:28:33] RECOVERY - apocrypha.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'apocrypha.wiki' will expire on Fri 01 Mar 2024 12:06:59 PM GMT +0000. [11:05:34] PROBLEM - wiki.buryland.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.buryland.net All nameservers failed to answer the query. [11:34:59] RECOVERY - wiki.buryland.net - reverse DNS on sslhost is OK: SSL OK - wiki.buryland.net reverse DNS resolves to cp36.wikitide.net - CNAME OK [12:12:32] [02miraheze/ErrorPages] 07translatewiki pushed 031 commit to 03master [+0/-0/±5] 13https://github.com/miraheze/ErrorPages/compare/677ddf88258b...2210250467b8 [12:12:33] [02miraheze/ErrorPages] 07translatewiki 032210250 - Localisation updates from https://translatewiki.net. [12:12:34] [02miraheze/GlobalNewFiles] 07translatewiki pushed 031 commit to 03master [+0/-0/±5] 13https://github.com/miraheze/GlobalNewFiles/compare/ea451998459a...6c9934c84d4a [12:12:35] [02miraheze/GlobalNewFiles] 07translatewiki 036c9934c - Localisation updates from https://translatewiki.net. [12:12:36] [02miraheze/CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±4] 13https://github.com/miraheze/CreateWiki/compare/508187d14b98...62a4fcb6a67b [12:12:37] [02miraheze/CreateWiki] 07translatewiki 0362a4fcb - Localisation updates from https://translatewiki.net. [12:12:40] [02miraheze/ImportDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/609bb01a4d3c...85627784cc56 [12:12:42] [02miraheze/ImportDump] 07translatewiki 038562778 - Localisation updates from https://translatewiki.net. [12:12:43] [02miraheze/landing] 07translatewiki pushed 031 commit to 03master [+0/-0/±12] 13https://github.com/miraheze/landing/compare/62a1bd26b363...a4af7acb7440 [12:12:44] [02miraheze/landing] 07translatewiki 03a4af7ac - Localisation updates from https://translatewiki.net. [12:12:45] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/MirahezeMagic/compare/2f1c73ac9d6d...4fac57279c20 [12:12:47] [02miraheze/MirahezeMagic] 07translatewiki 034fac572 - Localisation updates from https://translatewiki.net. [12:12:49] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/ManageWiki/compare/75510297a32e...8f2d7d3689cf [12:12:50] [02miraheze/ManageWiki] 07translatewiki 038f2d7d3 - Localisation updates from https://translatewiki.net. [12:12:52] [02miraheze/WikiDiscover] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/WikiDiscover/compare/8747207b9efc...4c9aa1a2b8fc [12:12:54] [02miraheze/WikiDiscover] 07translatewiki 034c9aa1a - Localisation updates from https://translatewiki.net. [12:14:00] miraheze/landing - translatewiki the build passed. [12:14:08] miraheze/ErrorPages - translatewiki the build passed. [12:17:19] miraheze/GlobalNewFiles - translatewiki the build passed. [12:17:34] miraheze/ManageWiki - translatewiki the build passed. [12:17:58] miraheze/WikiDiscover - translatewiki the build passed. [12:18:48] miraheze/MirahezeMagic - translatewiki the build passed. [12:18:58] miraheze/ImportDump - translatewiki the build passed. [12:19:53] miraheze/CreateWiki - translatewiki the build passed. [12:23:51] !log [@test151] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 [12:23:52] !log [@test151] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [12:23:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:23:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:01] !log [@test151] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 [12:24:02] !log [@test151] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [12:24:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:33:14] !log [@mwtask181] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all [12:33:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:33:22] !log [@mwtask181] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 8s [12:33:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:50:27] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/d5198990000b...9f1a6a563bc8 [12:50:28] [02miraheze/dns] 07Reception123 039f1a6a5 - add ebcat.me zone [13:06:59] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/c22350506760...fbff1a2059d5 [13:07:00] [02miraheze/ssl] 07MirahezeSSLBot 03fbff1a2 - Bot: Add SSL cert for www.ebcat.me [13:30:49] !log [agent@db181] Finished backing up SQL on db181 [13:30:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:25:28] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [14:44:55] RECOVERY - storytime.jdstroy.cf - reverse DNS on sslhost is OK: SSL OK - storytime.jdstroy.cf reverse DNS resolves to cp36.wikitide.net - CNAME OK [15:15:49] PROBLEM - storytime.jdstroy.cf - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for storytime.jdstroy.cf could not be found [15:23:45] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [16:34:13] !log add MacFan4000 to site-admins on Atlasian account; approve requests by MacFan4000 for Atlas & Statuspage [16:34:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:35:07] MacFan4000: does that work? [16:35:17] Yes [16:35:25] Perfect [16:36:23] MacFan4000: you can delete opsgenie [17:26:13] PROBLEM - cp36 Disk Space on cp36 is WARNING: DISK WARNING - free space: / 9699MiB (10% inode=98%); [17:28:34] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/MirahezeMagic/maintenance/restoreManageWikiBackup.php --wiki=mothergooselibrarywiki --filename /home/reception/mothergooselibrarywiki_managewiki_backup_6c9d3e36673352ac007d.json (END - exit=0) [17:28:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:36:34] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/deleteBatch.php --wiki=deadmau5wiki --r=Requested at T11819 /home/reception/deadmau5del.txt (START) [17:36:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:37:47] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/deleteBatch.php --wiki=deadmau5wiki --r=Requested at T11819 /home/reception/deadmau5del.txt (END - exit=0) [17:37:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:40:59] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 9709MiB (10% inode=98%); [17:52:05] [02puppet] 07MacFan4000 opened pull request 03#3749: relaybot: remove #wikitide-cvt-feed - 13https://github.com/miraheze/puppet/pull/3749 [17:53:13] [02puppet] 07Universal-Omega closed pull request 03#3749: relaybot: remove #wikitide-cvt-feed - 13https://github.com/miraheze/puppet/pull/3749 [17:53:15] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/72062a595ab5...3581a49d9442 [17:53:17] [02miraheze/puppet] 07MacFan4000 033581a49 - relaybot: remove #wikitide-cvt-feed (#3749) [17:53:24] PROBLEM - apocrypha.wiki - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [18:00:51] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/3581a49d9442...66c8a4802fe5 [18:00:52] [02miraheze/puppet] 07Universal-Omega 0366c8a48 - relaybot: remove #wikitide-cvt [18:23:25] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03index.php-1.42 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/19b5d3756d93 [18:23:27] [02miraheze/mw-config] 07Universal-Omega 0319b5d37 - index.php: support MediaWiki 1.42 [18:23:30] [02mw-config] 07Universal-Omega created branch 03index.php-1.42 - 13https://github.com/miraheze/mw-config [18:23:32] [02mw-config] 07Universal-Omega opened pull request 03#5471: index.php: support MediaWiki 1.42 - 13https://github.com/miraheze/mw-config/pull/5471 [18:23:52] CosmicAlpha: thats scary [18:23:58] 1.42 has parsoid for read [18:24:27] miraheze/mw-config - Universal-Omega the build passed. [18:51:14] PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 904 bytes in 0.234 second response time [18:51:55] RECOVERY - apocrypha.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'apocrypha.wiki' will expire on Fri 01 Mar 2024 12:06:59 PM GMT +0000. [18:52:57] !log [@test151] starting deploy of {'config': True} to test151 [18:52:59] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [18:52:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:53:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:06:06] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/4fac57279c20...c05098042302 [19:06:07] [02miraheze/MirahezeMagic] 07Universal-Omega 03c050980 - RebuildVersionCache: add null check [19:11:18] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/f3e643385411 [19:11:21] [02miraheze/mw-config] 07Reception123 03f3e6433 - re-add Foreground T11054 [19:11:23] [02mw-config] 07Reception123 created branch 03Reception123-patch-1 - 13https://github.com/miraheze/mw-config [19:11:26] [02mw-config] 07Reception123 opened pull request 03#5472: re-add Foreground T11054 - 13https://github.com/miraheze/mw-config/pull/5472 [19:11:43] [02mw-config] 07Universal-Omega synchronize pull request 03#5471: index.php: support MediaWiki 1.42 - 13https://github.com/miraheze/mw-config/pull/5471 [19:11:46] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03index.php-1.42 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/19b5d3756d93...e2ab765584cc [19:11:47] [02miraheze/mw-config] 07Universal-Omega 03e2ab765 - Fix [19:12:17] miraheze/mw-config - Reception123 the build passed. [19:12:36] miraheze/mw-config - Universal-Omega the build passed. [19:12:50] miraheze/MirahezeMagic - Universal-Omega the build passed. [19:13:14] RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.390 second response time [19:18:08] [02mw-config] 07Reception123 closed pull request 03#5472: re-add Foreground T11054 - 13https://github.com/miraheze/mw-config/pull/5472 [19:18:09] [02miraheze/mw-config] 07Reception123 deleted branch 03Reception123-patch-1 [19:18:10] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/0f2727ef1694...9edcd8bdbf11 [19:18:13] [02miraheze/mw-config] 07Reception123 039edcd8b - re-add Foreground T11054 (#5472) [19:18:16] !log [reception@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [19:18:16] [02mw-config] 07Reception123 deleted branch 03Reception123-patch-1 - 13https://github.com/miraheze/mw-config [19:18:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:18:25] !log [reception@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 9s [19:18:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:19:06] miraheze/mw-config - Reception123 the build passed. [19:23:04] !log [@test151] starting deploy of {'config': True} to test151 [19:23:05] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [19:23:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:23:06] PROBLEM - apocrypha.wiki - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [19:23:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:32:59] !log [@mwtask181] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all [19:33:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:33:07] !log [@mwtask181] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 8s [19:33:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:41:18] PROBLEM - wiki.vot.gg - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'wiki.vot.gg' expires in 7 day(s) (Fri 16 Feb 2024 07:17:47 PM GMT +0000). [19:44:52] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/9f1a6a563bc8...ce32e134e133 [19:44:54] [02miraheze/dns] 07paladox 03ce32e13 - Re-add analytics to wikitide.net [19:47:26] PROBLEM - eng.archiopedia.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - eng.archiopedia.org All nameservers failed to answer the query. [19:48:42] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/9edcd8bdbf11...4272bc9c41b1 [19:48:44] [02miraheze/mw-config] 07Universal-Omega 034272bc9 - index.php: support MediaWiki 1.42 (#5471) [19:48:47] [02mw-config] 07Universal-Omega closed pull request 03#5471: index.php: support MediaWiki 1.42 - 13https://github.com/miraheze/mw-config/pull/5471 [19:48:48] [02miraheze/mw-config] 07Universal-Omega deleted branch 03index.php-1.42 [19:48:50] [02mw-config] 07Universal-Omega deleted branch 03index.php-1.42 - 13https://github.com/miraheze/mw-config [19:49:37] miraheze/mw-config - Universal-Omega the build passed. [19:52:21] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/4272bc9c41b1...0736aa7ed989 [19:52:22] RECOVERY - apocrypha.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'apocrypha.wiki' will expire on Fri 01 Mar 2024 12:06:59 PM GMT +0000. [19:52:24] [02miraheze/mw-config] 07Universal-Omega 030736aa7 - Fix ParserMigration enable query string config [19:53:20] miraheze/mw-config - Universal-Omega the build passed. [19:57:22] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=arkantyawiki --no-updates --username-prefix=wikia:emeriya-i-kospad /home/reception/ImportDump/arkantyawiki-20240208195228.xml (START) [19:57:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:57:43] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=arkantyawiki --no-updates --username-prefix=wikia:emeriya-i-kospad /home/reception/ImportDump/arkantyawiki-20240208195228.xml (END - exit=0) [19:57:44] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=arkantyawiki (START) [19:57:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:57:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:58:26] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=arkantyawiki (END - exit=35584) [19:58:28] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/initSiteStats.php --wiki=arkantyawiki --active --update (END - exit=0) [19:58:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:58:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:02:09] !log [@mwtask181] starting deploy of {'config': True} to all [20:02:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:02:17] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 8s [20:02:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:16:01] RECOVERY - eng.archiopedia.org - reverse DNS on sslhost is OK: SSL OK - eng.archiopedia.org reverse DNS resolves to cp37.wikitide.net - CNAME OK [20:23:30] [02miraheze/WikiDiscover] 07redbluegreenhat pushed 032 commits to 03master [+0/-0/±2] 13https://github.com/miraheze/WikiDiscover/compare/4c9aa1a2b8fc...267e763a0d74 [20:23:33] [02miraheze/WikiDiscover] 07redbluegreenhat 03267e763 - Merge pull request from GHSA-cfcf-94jv-455f [20:27:54] miraheze/WikiDiscover - redbluegreenhat the build passed. [20:29:29] Agent, CosmicAlpha: https://github.com/miraheze/WikiDiscover/security/advisories/GHSA-cfcf-94jv-455f [20:32:16] Thanks [20:32:23] Orange_Star: any reason no CVE yet? [20:32:37] UO just did [20:32:41] what, you asking me? [20:32:49] Like I know how Mitre bureaucracy works [20:33:07] Has the extension been updated on prod yet? [20:33:10] Orange_Star: have you requested one? [20:33:17] no, that's why I pinged you [20:33:50] RhinosF1: yes [20:33:55] Good [20:33:58] And wtf [20:34:09] Why have we made a GHSA public before fixing our own production [20:34:28] !log [agent@mwtask181] starting deploy of {'versions': '1.41', 'upgrade_extensions': 'WikiDiscover'} to all [20:34:29] !log [agent@mwtask181] finished deploy of {'versions': '1.41', 'upgrade_extensions': 'WikiDiscover'} to all - SUCCESS in 1s [20:34:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:34:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:34:43] because everyone who knows even slightly what that does will know what's going on anyway [20:35:05] ye always patch prod first [20:35:11] hackers knew what happened the moment I merged my private changes [20:35:12] !log [agent@mwtask181] starting deploy of {'force': True, 'versions': '1.41', 'upgrade_extensions': 'WikiDiscover'} to all [20:35:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:35:21] !log [agent@mwtask181] finished deploy of {'force': True, 'versions': '1.41', 'upgrade_extensions': 'WikiDiscover'} to all - SUCCESS in 8s [20:35:24] PROBLEM - grk.archiopedia.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - grk.archiopedia.org All nameservers failed to answer the query. [20:35:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:38:31] Orange_Star: you know you can deploy without a patch being public ? [20:38:53] that's kinda cringe bro, ngl /sarcasm [20:39:02] actually I didn't know [20:39:26] I don't have shells at prod anyway, so not like I can do that on my own [20:44:06] [02TSPortal] 07pixDeVl opened pull request 03#17: Updated text to reflect new owner and juristiction; replace register no. with EIN - 13https://github.com/miraheze/TSPortal/pull/17 [20:44:16] Orange_Star: yes, I mean that's one of the reasons we had the deploy tool built. [20:44:26] So it's really easy to deploy stuff privately [20:44:42] I've added it to the Wikimedia extension security release too [20:44:54] Ping me when there's a CVE and I'll add it [20:45:11] I'll let you know if we have one tomorrow [20:45:41] They are normally pretty quick [20:47:54] [02TSPortal] 07pixDeVl synchronize pull request 03#17: Updated text to reflect new owner and juristiction; replace register no. with EIN - 13https://github.com/miraheze/TSPortal/pull/17 [20:52:31] !log [alex@test151] starting deploy of {'versions': '1.41', 'upgrade_extensions': 'WikiDiscover'} to test151 [20:52:32] !log [alex@test151] finished deploy of {'versions': '1.41', 'upgrade_extensions': 'WikiDiscover'} to test151 - SUCCESS in 0s [20:52:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:52:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:53:09] !log [alex@test151] starting deploy of {'force': True, 'versions': '1.41', 'upgrade_extensions': 'WikiDiscover'} to test151 [20:53:10] !log [alex@test151] finished deploy of {'force': True, 'versions': '1.41', 'upgrade_extensions': 'WikiDiscover'} to test151 - SUCCESS in 1s [20:53:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:53:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:54:01] and that is my first shell session at test151 [20:54:06] weirdly nostalgic, ngl [21:09:57] RhinosF1: WHat is that "Wikimedia extension security release" stuff by the way? [21:10:24] I don't know almost anything regarding Wikimedia tech [21:10:41] Orange_Star: the post release email that lists extension & skin security issues [21:10:47] PROBLEM - grc.repository.archiopedia.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - grc.repository.archiopedia.org All nameservers failed to answer the query. [21:11:05] PROBLEM - grk.repository.archiopedia.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query grk.repository.archiopedia.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [21:11:19] ohh, I know those, I'm subscribed to that list [21:13:26] Orange_Star: wikidiscover will be on the next one [21:15:35] PROBLEM - eng.archiopedia.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query eng.archiopedia.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [21:29:29] PROBLEM - repository.archiopedia.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - repository.archiopedia.org All nameservers failed to answer the query. [21:39:42] RECOVERY - grk.repository.archiopedia.org - reverse DNS on sslhost is OK: SSL OK - grk.repository.archiopedia.org reverse DNS resolves to cp37.wikitide.net - CNAME OK [21:40:25] RECOVERY - grc.repository.archiopedia.org - reverse DNS on sslhost is OK: SSL OK - grc.repository.archiopedia.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [21:44:09] RECOVERY - eng.archiopedia.org - reverse DNS on sslhost is OK: SSL OK - eng.archiopedia.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [21:52:17] PROBLEM - apocrypha.wiki - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [21:59:20] RECOVERY - repository.archiopedia.org - reverse DNS on sslhost is OK: SSL OK - repository.archiopedia.org reverse DNS resolves to cp37.wikitide.net - CNAME OK [22:01:15] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 90%, RTA = 1653.10 ms [22:01:22] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds [22:01:57] PROBLEM - cp41 SSH on cp41 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:03:01] PROBLEM - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:03:28] PROBLEM - cp41 NTP time on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:03:28] PROBLEM - cp41 Nginx Backend for mw152 on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:03:30] PROBLEM - cp41 Nginx Backend for matomo151 on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:03:30] PROBLEM - cp41 Nginx Backend for mw172 on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:03:33] PROBLEM - cp41 Nginx Backend for mw162 on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:03:37] PROBLEM - cp41 Nginx Backend for test151 on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:03:45] PROBLEM - cp41 Nginx Backend for mon181 on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:03:47] PROBLEM - cp41 Nginx Backend for puppet181 on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:03:57] PROBLEM - cp41 Nginx Backend for phorge171 on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:03:58] PROBLEM - cp41 Nginx Backend for mw171 on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:04:03] PROBLEM - cp41 Nginx Backend for mw161 on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:04:05] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:04:09] PROBLEM - cp41 Current Load on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:04:20] PROBLEM - cp41 Nginx Backend for swiftproxy171 on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:04:22] PROBLEM - cp41 Nginx Backend for mwtask181 on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:04:48] PROBLEM - cp41 Puppet on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:04:48] PROBLEM - cp41 Nginx Backend for swiftproxy161 on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:04:51] PROBLEM - cp41 Nginx Backend for mw181 on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:04:54] PROBLEM - cp41 ferm_active on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:04:57] RECOVERY - grk.archiopedia.org - reverse DNS on sslhost is OK: SSL OK - grk.archiopedia.org reverse DNS resolves to cp37.wikitide.net - CNAME OK [22:05:07] PROBLEM - cp41 conntrack_table_size on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:05:10] PROBLEM - cp41 Nginx Backend for mw151 on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:05:12] PROBLEM - cp41 Disk Space on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:05:19] PROBLEM - cp41 APT on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:05:23] PROBLEM - cp41 Nginx Backend for mw182 on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:05:26] PROBLEM - cp41 PowerDNS Recursor on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:06:55] PROBLEM - cp41 Nginx Backend for reports171 on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:21:33] RECOVERY - apocrypha.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'apocrypha.wiki' will expire on Fri 01 Mar 2024 12:06:59 PM GMT +0000. [22:38:06] RECOVERY - cp41 Nginx Backend for mwtask181 on cp41 is OK: TCP OK - 0.000 second response time on localhost port 8160 [22:44:19]