[00:01:22] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [00:01:56] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [00:05:22] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 50% [00:05:22] RECOVERY - cp36 Disk Space on cp36 is OK: DISK OK - free space: / 20661MiB (23% inode=98%); [00:06:11] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 20789MiB (23% inode=98%); [00:07:49] RECOVERY - apocrypha.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'apocrypha.wiki' will expire on Fri 01 Mar 2024 12:06:59 PM GMT +0000. [00:09:05] RECOVERY - wiki.sheepservermc.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.sheepservermc.net' will expire on Fri 10 May 2024 10:40:41 PM GMT +0000. [00:11:22] RECOVERY - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is OK: OK - NGINX Error Rate is 36% [00:11:56] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is WARNING: WARNING - NGINX Error Rate is 41% [00:13:56] RECOVERY - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is OK: OK - NGINX Error Rate is 30% [00:17:59] RECOVERY - wiki.manos.pt - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.manos.pt' will expire on Fri 10 May 2024 10:20:07 PM GMT +0000. [00:38:54] PROBLEM - apocrypha.wiki - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [00:58:02] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:08:06] RECOVERY - apocrypha.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'apocrypha.wiki' will expire on Fri 01 Mar 2024 12:06:59 PM GMT +0000. [01:09:22] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 40% [01:19:56] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is WARNING: WARNING - NGINX Error Rate is 40% [01:20:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [01:27:00] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [01:29:00] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 152.87 ms [01:31:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 153.11 ms [01:39:12] PROBLEM - apocrypha.wiki - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [02:08:25] RECOVERY - apocrypha.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'apocrypha.wiki' will expire on Fri 01 Mar 2024 12:06:59 PM GMT +0000. [02:27:04] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [02:41:00] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 157.62 ms [02:43:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 152.82 ms [02:56:04] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [03:00:26] RECOVERY - mon181 Backups Grafana on mon181 is OK: FILE_AGE OK: /var/log/grafana-backup.log is 9 seconds old and 93 bytes [03:07:00] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 143.36 ms [03:09:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 148.08 ms [03:35:35] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [03:50:24] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=polcompballwiki --no-updates /home/universalomega/pcbwiki.xml (END - exit=0) [03:50:25] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=polcompballwiki (START) [03:50:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:50:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:34:42] PROBLEM - apocrypha.wiki - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [05:42:00] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=polcompballwiki (END - exit=0) [05:42:02] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/initSiteStats.php --wiki=polcompballwiki --active --update (END - exit=0) [05:42:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:42:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:47:00] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 148.50 ms [05:48:25] [02miraheze/mediawiki-repos] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mediawiki-repos/compare/75215eb0388c...7c83ae33598e [05:48:26] [02miraheze/mediawiki-repos] 07AgentIsai 037c83ae3 - Add ReportIncident extension [05:49:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 152.85 ms [05:52:53] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [05:56:05] [02miraheze/mw-config] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/fce2453fc45b...6d607398d283 [05:56:07] [02miraheze/mw-config] 07AgentIsai 036d60739 - Enable support for ReportIncident [05:56:58] miraheze/mw-config - AgentIsai the build passed. [05:57:10] [02miraheze/mw-config] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/6d607398d283...cac81da9af01 [05:57:13] [02miraheze/mw-config] 07AgentIsai 03cac81da - Enable ReportIncident globally [05:58:09] miraheze/mw-config - AgentIsai the build passed. [05:58:31] !log [@mwtask181] starting deploy of {'config': True} to all [05:58:32] !log [@mwtask181] DEPLOY ABORTED: Canary check failed for publictestwiki.com@localhost [05:58:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:58:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:01:26] PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [06:02:04] PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [06:02:13] PROBLEM - mwtask181 HTTPS on mwtask181 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 500 [06:03:18] !log [agent@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [06:03:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:03:26] PROBLEM - mwtask181 MediaWiki Rendering on mwtask181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4716 bytes in 0.269 second response time [06:03:27] !log [agent@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 8s [06:03:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:04:03] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:04:13] RECOVERY - mwtask181 HTTPS on mwtask181 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 285 bytes in 0.080 second response time [06:05:46] PROBLEM - mw162 MediaWiki Rendering on mw162 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 684 bytes in 0.197 second response time [06:05:48] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 684 bytes in 0.224 second response time [06:05:54] PROBLEM - mw152 MediaWiki Rendering on mw152 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 684 bytes in 0.195 second response time [06:06:32] PROBLEM - mw171 MediaWiki Rendering on mw171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 684 bytes in 0.216 second response time [06:06:34] PROBLEM - mw172 MediaWiki Rendering on mw172 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 684 bytes in 0.203 second response time [06:06:48] PROBLEM - mw151 MediaWiki Rendering on mw151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 684 bytes in 0.194 second response time [06:06:51] PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 684 bytes in 0.205 second response time [06:07:03] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 684 bytes in 0.236 second response time [06:07:17] [02miraheze/mw-config] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/cac81da9af01...77cf2a4dceb4 [06:07:20] [02miraheze/mw-config] 07AgentIsai 0377cf2a4 - Disable ReportIncident for now [06:07:27] !log [agent@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [06:07:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:07:36] !log [agent@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 9s [06:07:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:07:46] RECOVERY - mw162 MediaWiki Rendering on mw162 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.270 second response time [06:07:48] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.292 second response time [06:07:54] RECOVERY - mw152 MediaWiki Rendering on mw152 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.287 second response time [06:08:26] miraheze/mw-config - AgentIsai the build passed. [06:08:32] RECOVERY - mw171 MediaWiki Rendering on mw171 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.296 second response time [06:08:35] RECOVERY - mw172 MediaWiki Rendering on mw172 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.283 second response time [06:08:48] RECOVERY - mw151 MediaWiki Rendering on mw151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.256 second response time [06:08:51] RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.268 second response time [06:09:03] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.316 second response time [06:09:26] RECOVERY - mwtask181 MediaWiki Rendering on mwtask181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.433 second response time [06:13:22] PROBLEM - swiftac171 Current Load on swiftac171 is WARNING: LOAD WARNING - total load average: 10.84, 9.54, 5.16 [06:15:18] RECOVERY - swiftac171 Current Load on swiftac171 is OK: LOAD OK - total load average: 3.77, 7.30, 4.85 [06:23:26] !log [@test151] starting deploy of {'config': True} to test151 [06:23:27] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [06:23:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:23:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:31:03] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [06:33:06] RECOVERY - apocrypha.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'apocrypha.wiki' will expire on Fri 01 Mar 2024 12:06:59 PM GMT +0000. [07:00:25] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [07:17:00] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 152.79 ms [07:19:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 148.05 ms [07:45:00] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 151.74 ms [07:47:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 155.37 ms [08:03:00] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 148.10 ms [08:05:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 148.12 ms [08:17:43] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [08:27:00] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 143.46 ms [08:29:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 150.39 ms [08:35:00] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 143.32 ms [08:37:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 157.54 ms [08:45:00] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 152.91 ms [08:47:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 152.84 ms [08:49:00] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 143.37 ms [08:51:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 161.28 ms [09:01:01] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 158.07 ms [09:03:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 148.10 ms [09:56:32] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=bornagainwiki --no-updates --username-prefix=mh:bornagain /home/reception/ImportDump/bornagainwiki-20240211084951.xml (START) [09:56:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:29:34] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=bornagainwiki --no-updates --username-prefix=mh:bornagain /home/reception/ImportDump/bornagainwiki-20240211084951.xml (END - exit=0) [10:29:35] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=bornagainwiki (START) [10:29:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:29:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:35:00] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 153.18 ms [10:35:04] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=bornagainwiki (END - exit=0) [10:35:05] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/initSiteStats.php --wiki=bornagainwiki --active --update (END - exit=0) [10:35:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:35:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:37:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 153.00 ms [10:59:00] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 154.94 ms [11:01:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 150.01 ms [11:09:00] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 143.66 ms [11:11:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 152.88 ms [11:19:00] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 152.96 ms [11:21:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 143.86 ms [11:27:00] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 153.25 ms [11:29:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 153.27 ms [11:35:00] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 152.92 ms [11:37:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 155.71 ms [11:53:00] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 148.29 ms [11:55:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 159.84 ms [11:56:12] PROBLEM - apocrypha.wiki - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [11:57:22] RECOVERY - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is OK: OK - NGINX Error Rate is 37% [12:14:31] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:23:00] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 154.65 ms [12:25:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 148.40 ms [12:25:24] RECOVERY - apocrypha.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'apocrypha.wiki' will expire on Fri 01 Mar 2024 12:06:59 PM GMT +0000. [12:43:31] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [12:53:01] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 152.80 ms [12:55:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 148.09 ms [13:03:00] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 148.27 ms [13:05:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 143.34 ms [13:17:01] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 157.60 ms [13:19:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 148.15 ms [13:33:01] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 148.00 ms [13:37:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 148.07 ms [14:05:01] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 148.10 ms [14:07:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 143.36 ms [14:19:00] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 152.90 ms [14:21:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 152.77 ms [14:23:02] [02mw-config] 07redbluegreenhat opened pull request 03#5476: Set wgKartographerDfltStyle to . - 13https://github.com/miraheze/mw-config/pull/5476 [14:23:59] miraheze/mw-config - redbluegreenhat the build passed. [14:55:00] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 152.78 ms [14:57:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 152.87 ms [14:59:00] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 158.78 ms [15:01:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 157.15 ms [15:07:00] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 153.46 ms [15:09:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 154.38 ms [15:19:00] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 148.47 ms [15:21:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 148.29 ms [15:22:29] PROBLEM - apocrypha.wiki - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [15:25:00] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 148.29 ms [15:27:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 157.64 ms [15:33:22] PROBLEM - cp36 Disk Space on cp36 is WARNING: DISK WARNING - free space: / 9705MiB (10% inode=98%); [15:34:50] !log [paladox@mwtask181] starting deploy of {'versions': '1.41', 'upgrade_skins': 'Tweeki'} to all [15:34:52] !log [paladox@mwtask181] finished deploy of {'versions': '1.41', 'upgrade_skins': 'Tweeki'} to all - SUCCESS in 1s [15:34:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:34:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:34:58] !log [paladox@mwtask181] starting deploy of {'world': True, 'versions': '1.41', 'upgrade_skins': 'Tweeki'} to all [15:35:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:36:00] !log [paladox@mwtask181] finished deploy of {'world': True, 'versions': '1.41', 'upgrade_skins': 'Tweeki'} to all - SUCCESS in 61s [15:36:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:48:11] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 9681MiB (10% inode=98%); [15:51:00] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 156.56 ms [15:54:17] !log [paladox@mwtask181] /usr/local/bin/foreachwikiindblist /srv/mediawiki/standarddialogs.json /srv/mediawiki/1.41/maintenance/sql.php /srv/mediawiki/1.41/extensions/OOJSPlus/vendor/mwstake/mediawiki-component-commonwebapis/sql/mws_title_index_displaytitle_patch.sql [15:54:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:55:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 148.54 ms [16:11:00] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 153.07 ms [16:13:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 154.17 ms [16:20:53] RECOVERY - apocrypha.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'apocrypha.wiki' will expire on Fri 01 Mar 2024 12:06:59 PM GMT +0000. [16:33:01] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 152.93 ms [16:35:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 162.87 ms [16:47:01] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 150.66 ms [16:49:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 148.55 ms [17:07:00] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 148.23 ms [17:09:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 143.37 ms [17:17:00] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 157.60 ms [17:19:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 153.04 ms [17:47:00] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 154.82 ms [17:49:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 162.32 ms [17:57:01] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 148.21 ms [17:59:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 148.88 ms [18:14:30] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [18:15:01] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 153.53 ms [18:17:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 143.52 ms [18:37:00] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 157.67 ms [18:39:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 157.82 ms [18:47:00] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 148.90 ms [18:49:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 155.57 ms [18:57:00] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 148.08 ms [18:59:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 148.74 ms [19:02:34] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03fix-cp28-ping [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/75e2e0190eed [19:02:36] [02miraheze/puppet] 07Universal-Omega 0375e2e01 - monitoring: fix ping and ping6 for cp28 and remove SCSVG [19:02:38] [02puppet] 07Universal-Omega created branch 03fix-cp28-ping - 13https://github.com/miraheze/puppet [19:02:40] [02puppet] 07Universal-Omega opened pull request 03#3752: monitoring: fix ping and ping6 for cp28 and remove SCSVG - 13https://github.com/miraheze/puppet/pull/3752 [19:05:00] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 143.37 ms [19:07:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 148.62 ms [19:08:41] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03remove-apocrypha.wiki [+0/-1/±0] 13https://github.com/miraheze/dns/commit/85f7850b4c1e [19:08:43] [02miraheze/dns] 07Universal-Omega 0385f7850 - Remove apocrypha.wiki zone [19:08:45] [02dns] 07Universal-Omega created branch 03remove-apocrypha.wiki - 13https://github.com/miraheze/dns [19:08:48] [02dns] 07Universal-Omega opened pull request 03#492: Remove apocrypha.wiki zone - 13https://github.com/miraheze/dns/pull/492 [19:09:57] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03remove-apocrypha.wiki [+0/-1/±0] 13https://github.com/miraheze/ssl/commit/d5f07c5716e8 [19:09:59] [02miraheze/ssl] 07Universal-Omega 03d5f07c5 - Remove apocrypha.wiki cert [19:10:00] [02ssl] 07Universal-Omega created branch 03remove-apocrypha.wiki - 13https://github.com/miraheze/ssl [19:10:02] [02ssl] 07Universal-Omega opened pull request 03#777: Remove apocrypha.wiki cert - 13https://github.com/miraheze/ssl/pull/777 [19:11:05] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03remove-apocrypha.wiki [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d5f07c5716e8...677de242cecf [19:11:07] [02miraheze/ssl] 07Universal-Omega 03677de24 - Remove [19:11:10] [02ssl] 07Universal-Omega synchronize pull request 03#777: Remove apocrypha.wiki cert - 13https://github.com/miraheze/ssl/pull/777 [19:12:29] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [19:14:42] [02ssl] 07Universal-Omega closed pull request 03#777: Remove apocrypha.wiki cert - 13https://github.com/miraheze/ssl/pull/777 [19:14:43] [02miraheze/ssl] 07Universal-Omega deleted branch 03remove-apocrypha.wiki [19:14:44] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±1] 13https://github.com/miraheze/ssl/compare/6ebb1220a565...40cba47147d3 [19:14:46] [02miraheze/ssl] 07Universal-Omega 0340cba47 - Remove apocrypha.wiki cert (#777) [19:14:49] [02ssl] 07Universal-Omega deleted branch 03remove-apocrypha.wiki - 13https://github.com/miraheze/ssl [19:14:50] [02dns] 07Universal-Omega closed pull request 03#492: Remove apocrypha.wiki zone - 13https://github.com/miraheze/dns/pull/492 [19:14:52] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/miraheze/dns/compare/6e8d52a5a107...cc3b8ef2baf5 [19:14:53] [02miraheze/dns] 07Universal-Omega 03cc3b8ef - Remove apocrypha.wiki zone (#492) [19:14:56] [02miraheze/dns] 07Universal-Omega deleted branch 03remove-apocrypha.wiki [19:14:57] [02dns] 07Universal-Omega deleted branch 03remove-apocrypha.wiki - 13https://github.com/miraheze/dns [19:21:00] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 148.07 ms [19:23:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 146.65 ms [19:25:55] [02puppet] 07Universal-Omega closed pull request 03#3752: monitoring: fix ping and ping6 for cp28 and remove SCSVG - 13https://github.com/miraheze/puppet/pull/3752 [19:25:57] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/dc68029acbea...1389ecbf7242 [19:25:59] [02miraheze/puppet] 07Universal-Omega 031389ecb - monitoring: fix ping and ping6 for cp28 and remove SCSVG (#3752) [19:26:01] [02miraheze/puppet] 07Universal-Omega deleted branch 03fix-cp28-ping [19:26:03] [02puppet] 07Universal-Omega deleted branch 03fix-cp28-ping - 13https://github.com/miraheze/puppet [19:35:00] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 162.34 ms [19:37:00] PROBLEM - ping6 on cp28 is WARNING: PING WARNING - Packet loss = 0%, RTA = 148.25 ms [19:42:28] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [19:47:11] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 153.12 ms [19:49:12] RECOVERY - ping6 on cp28 is OK: PING OK - Packet loss = 0%, RTA = 148.15 ms [19:54:18] PROBLEM - swiftobject171 NTP time on swiftobject171 is WARNING: NTP WARNING: Offset -0.1111267805 secs [20:00:19] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/6942e8b2c01d...4d9645d6e97c [20:00:21] [02miraheze/ManageWiki] 07Universal-Omega 034d9645d - Fix "PHP Deprecated: explode(): Passing null to parameter #2 ($string) of type string is deprecated" [20:04:34] miraheze/ManageWiki - Universal-Omega the build passed. [20:25:18] RECOVERY - swiftobject171 NTP time on swiftobject171 is OK: NTP OK: Offset -0.05769667029 secs [20:30:08] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/4d9645d6e97c...7bc33a6e641a [20:30:11] [02miraheze/ManageWiki] 07Universal-Omega 037bc33a6 - Fix "PHP Deprecated: json_decode(): Passing null to parameter #1 ($json) of type string is deprecated" [20:34:31] miraheze/ManageWiki - Universal-Omega the build passed. [20:46:02] PROBLEM - swiftobject171 NTP time on swiftobject171 is WARNING: NTP WARNING: Offset 0.1107563972 secs [20:52:14] RECOVERY - swiftobject171 NTP time on swiftobject171 is OK: NTP OK: Offset 0.06857925653 secs [21:11:58] PROBLEM - swiftobject171 NTP time on swiftobject171 is WARNING: NTP WARNING: Offset -0.1360386908 secs [21:16:06] RECOVERY - swiftobject171 NTP time on swiftobject171 is OK: NTP OK: Offset -0.009218454361 secs [21:22:05] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 40% [21:26:30] PROBLEM - swiftobject171 NTP time on swiftobject171 is WARNING: NTP WARNING: Offset -0.2086110413 secs [21:28:20] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03performance [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/e72e42792575 [21:28:22] [02miraheze/mw-config] 07Universal-Omega 03e72e427 - Test some changes for performance [21:28:23] [02mw-config] 07Universal-Omega created branch 03performance - 13https://github.com/miraheze/mw-config [21:28:26] [02mw-config] 07Universal-Omega opened pull request 03#5477: Test some changes for performance - 13https://github.com/miraheze/mw-config/pull/5477 [21:28:34] RECOVERY - swiftobject171 NTP time on swiftobject171 is OK: NTP OK: Offset -0.0972649157 secs [21:28:55] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03performance [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/e72e42792575...0fc7262b9ce0 [21:28:56] [02miraheze/mw-config] 07github-actions 030fc7262 - CI: lint code to MediaWiki standards [21:28:57] [02mw-config] 07github-actions[bot] synchronize pull request 03#5477: Test some changes for performance - 13https://github.com/miraheze/mw-config/pull/5477 [21:29:23] miraheze/mw-config - Universal-Omega the build passed. [21:29:26] PROBLEM - cp36 Disk Space on cp36 is CRITICAL: DISK CRITICAL - free space: / 5305MiB (5% inode=98%); [21:29:39] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03performance [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/0fc7262b9ce0...a77b5af73953 [21:29:40] [02miraheze/mw-config] 07Universal-Omega 03a77b5af - Test [21:29:42] [02mw-config] 07Universal-Omega synchronize pull request 03#5477: Test some changes for performance - 13https://github.com/miraheze/mw-config/pull/5477 [21:30:33] miraheze/mw-config - Universal-Omega the build passed. [21:34:17] [02mw-config] 07Universal-Omega synchronize pull request 03#5477: Test some changes for performance - 13https://github.com/miraheze/mw-config/pull/5477 [21:34:20] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03performance [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a77b5af73953...ed5548ac02d7 [21:34:23] [02miraheze/mw-config] 07Universal-Omega 03ed5548a - Update [21:35:11] miraheze/mw-config - Universal-Omega the build passed. [21:36:54] PROBLEM - swiftobject171 NTP time on swiftobject171 is WARNING: NTP WARNING: Offset -0.1553696394 secs [21:37:01] PROBLEM - test151 MediaWiki Rendering on test151 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [21:37:06] PROBLEM - test151 HTTPS on test151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 500 [21:47:01] PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 3422 bytes in 0.089 second response time [21:47:14] RECOVERY - swiftobject171 NTP time on swiftobject171 is OK: NTP OK: Offset -0.02357521653 secs [21:49:01] RECOVERY - test151 HTTPS on test151 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 285 bytes in 0.071 second response time [21:49:51] PROBLEM - cp37 Disk Space on cp37 is CRITICAL: DISK CRITICAL - free space: / 5293MiB (5% inode=98%); [21:57:02] RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.333 second response time [22:00:54] PROBLEM - test151 HTTPS on test151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 500 [22:01:46] PROBLEM - swiftobject171 NTP time on swiftobject171 is WARNING: NTP WARNING: Offset -0.1401485801 secs [22:03:50] RECOVERY - swiftobject171 NTP time on swiftobject171 is OK: NTP OK: Offset 0.02595296502 secs [22:05:35] !log [paladox@mwtask181] starting deploy of {'world': True, 'versions': '1.41', 'upgrade_extensions': 'LogoFunctions'} to all [22:05:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:06:23] !log [paladox@mwtask181] finished deploy of {'world': True, 'versions': '1.41', 'upgrade_extensions': 'LogoFunctions'} to all - SUCCESS in 48s [22:06:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:08:14] PROBLEM - ping6 on cp28 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 143.80 ms [22:08:50] RECOVERY - test151 HTTPS on test151 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 285 bytes in 0.072 second response time [22:10:15] RECOVERY - ping6 on cp28 is OK: PING OK - Packet loss = 0%, RTA = 154.33 ms [22:15:32] PROBLEM - test151 MediaWiki Rendering on test151 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 1349 bytes in 0.239 second response time [22:17:27] PROBLEM - test151 MediaWiki Rendering on test151 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [22:17:55] [02mw-config] 07Universal-Omega synchronize pull request 03#5477: Test some changes for performance - 13https://github.com/miraheze/mw-config/pull/5477 [22:17:58] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03performance [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/ed5548ac02d7...130718e992b3 [22:18:00] [02miraheze/mw-config] 07Universal-Omega 03130718e - - [22:18:57] miraheze/mw-config - Universal-Omega the build passed. [22:19:22] RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.468 second response time [22:19:47] PROBLEM - test151 HTTPS on test151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 500 [22:26:45] PROBLEM - test151 Puppet on test151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki config] [22:27:47] RECOVERY - test151 HTTPS on test151 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 285 bytes in 0.071 second response time [22:40:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [22:42:24] CosmicAlpha: that is RottenLinksJob on bluepageswiki [22:54:45] RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [23:35:35] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [23:36:23] [02puppet] 07paladox closed pull request 03#3750: reports: upgrade to v16 - 13https://github.com/miraheze/puppet/pull/3750 [23:36:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/1389ecbf7242...b3e910f85324 [23:36:26] [02miraheze/puppet] 07Universal-Omega 03b3e910f - reports: upgrade to v16 (#3750) [23:36:27] [02miraheze/puppet] 07paladox deleted branch 03reports-v16 [23:36:30] [02puppet] 07paladox deleted branch 03reports-v16 - 13https://github.com/miraheze/puppet