[00:18:22] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[01:47:21] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK
[02:02:42] <botifico>	 [02puppet] 07Universal-Omega closed pull request 03#3853: monitoring: update groups for new structure - 13https://github.com/miraheze/puppet/pull/3853
[02:02:44] <botifico>	 [02puppet] 07Universal-Omega deleted branch 03monitoring-org
[02:02:48] <botifico>	 [02puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±16] 13https://github.com/miraheze/puppet/compare/6cbb81db72c6...d2d8d0c261aa
[02:02:49] <botifico>	 [02puppet] 07Universal-Omega 03d2d8d0c - monitoring: update groups for new structure (#3853)
[02:02:52] <botifico>	 [02puppet] 07Universal-Omega deleted branch 03monitoring-org - 13https://github.com/miraheze/puppet
[02:04:01] <botifico>	 [02puppet] 07Universal-Omega pushed 033 commits to 03rename-shell [+0/-0/±34] 13https://github.com/miraheze/puppet/compare/803d150bcecc...34e86f9b6420
[02:04:03] <botifico>	 [02puppet] 07Universal-Omega 0334e86f9 - Merge branch 'master' into rename-shell
[02:04:05] <botifico>	 [02puppet] 07Universal-Omega synchronize pull request 03#3852: Rename shell user - 13https://github.com/miraheze/puppet/pull/3852
[02:19:22] <icinga-miraheze>	 PROBLEM - mon181 Check correctness of the icinga configuration on mon181 is CRITICAL: Icinga configuration contains errors
[02:48:32] <icinga-miraheze>	 PROBLEM - mon181 Puppet on mon181 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 1 minute ago with 0 failures
[02:49:22] <icinga-miraheze>	 RECOVERY - mon181 Check correctness of the icinga configuration on mon181 is OK: Icinga configuration is correct
[02:52:23] <icinga-miraheze>	 RECOVERY - mon181 Puppet on mon181 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures
[04:14:55] <icinga-miraheze>	 PROBLEM - mason.sagan4.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mason.sagan4.org' expires in 15 day(s) (Wed 29 May 2024 04:04:01 AM GMT +0000).
[04:15:08] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/642c3582d5a5...7ba1638dbb91
[04:15:09] <botifico>	 [02ssl] 07WikiTideSSLBot 037ba1638 - Bot: Update SSL cert for mason.sagan4.org
[04:15:57] <icinga-miraheze>	 PROBLEM - gil.wiki - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket
[04:17:23] <icinga-miraheze>	 PROBLEM - meta.sagan4.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'meta.sagan4.org' expires in 15 day(s) (Wed 29 May 2024 04:03:46 AM GMT +0000).
[04:17:33] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/7ba1638dbb91...e80eeeb6f4d6
[04:17:34] <botifico>	 [02ssl] 07WikiTideSSLBot 03e80eeeb - Bot: Update SSL cert for meta.sagan4.org
[04:19:13] <icinga-miraheze>	 PROBLEM - gil.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for gil.wiki could not be found
[05:12:44] <icinga-miraheze>	 RECOVERY - mason.sagan4.org - LetsEncrypt on sslhost is OK: OK - Certificate 'mason.sagan4.org' will expire on Sun 11 Aug 2024 03:15:02 AM GMT +0000.
[05:15:23] <icinga-miraheze>	 RECOVERY - meta.sagan4.org - LetsEncrypt on sslhost is OK: OK - Certificate 'meta.sagan4.org' will expire on Sun 11 Aug 2024 03:17:27 AM GMT +0000.
[11:02:35] <icinga-miraheze>	 PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query franchising.org.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[11:32:29] <icinga-miraheze>	 RECOVERY - franchise.franchising.org.ua - reverse DNS on sslhost is OK: SSL OK - franchise.franchising.org.ua reverse DNS resolves to cp36.wikitide.net - CNAME OK
[13:04:12] <icinga-miraheze>	 PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - franchise.franchising.org.ua All nameservers failed to answer the query.
[13:29:16] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[13:34:07] <icinga-miraheze>	 RECOVERY - franchise.franchising.org.ua - reverse DNS on sslhost is OK: SSL OK - franchise.franchising.org.ua reverse DNS resolves to cp37.wikitide.net - CNAME OK
[14:06:14] <icinga-miraheze>	 PROBLEM - beta.sagan4.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'beta.sagan4.org' expires in 15 day(s) (Wed 29 May 2024 01:43:07 PM GMT +0000).
[14:06:25] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e80eeeb6f4d6...8655cb3357cd
[14:06:26] <botifico>	 [02ssl] 07WikiTideSSLBot 038655cb3 - Bot: Update SSL cert for beta.sagan4.org
[14:27:07] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK
[14:36:10] <icinga-miraheze>	 RECOVERY - beta.sagan4.org - LetsEncrypt on sslhost is OK: OK - Certificate 'beta.sagan4.org' will expire on Sun 11 Aug 2024 01:06:18 PM GMT +0000.
[16:06:31] <botifico>	 [02CreateWiki] 07Universal-Omega pushed 031 commit to 03tags [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/commit/08d00d7711c7
[16:06:32] <botifico>	 [02CreateWiki] 07Universal-Omega 0308d00d7 - Add support for tags
[16:06:34] <botifico>	 [02CreateWiki] 07Universal-Omega created branch 03tags - 13https://github.com/miraheze/CreateWiki
[16:06:37] <botifico>	 [02CreateWiki] 07Universal-Omega opened pull request 03#527: Add support for tags - 13https://github.com/miraheze/CreateWiki/pull/527
[16:13:03] <miraheze-github>	 miraheze/CreateWiki - Universal-Omega the build has errored.
[16:17:33] <botifico>	 [02CreateWiki] 07Universal-Omega pushed 031 commit to 03tags [+1/-0/±0] 13https://github.com/miraheze/CreateWiki/compare/08d00d7711c7...f722c0ac3310
[16:17:35] <botifico>	 [02CreateWiki] 07Universal-Omega 03f722c0a - Add JS
[16:17:37] <botifico>	 [02CreateWiki] 07Universal-Omega synchronize pull request 03#527: Add support for tags - 13https://github.com/miraheze/CreateWiki/pull/527
[16:18:47] <botifico>	 [02CreateWiki] 07Universal-Omega pushed 031 commit to 03tags [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/f722c0ac3310...4078d4a4853b
[16:18:50] <botifico>	 [02CreateWiki] 07Universal-Omega 034078d4a - Add sql
[16:18:52] <botifico>	 [02CreateWiki] 07Universal-Omega synchronize pull request 03#527: Add support for tags - 13https://github.com/miraheze/CreateWiki/pull/527
[16:20:51] <botifico>	 [02CreateWiki] 07Universal-Omega pushed 031 commit to 03tags [+1/-0/±0] 13https://github.com/miraheze/CreateWiki/compare/4078d4a4853b...b16c437b4eba
[16:20:54] <botifico>	 [02CreateWiki] 07Universal-Omega 03b16c437 - Add patch
[16:20:55] <botifico>	 [02CreateWiki] 07Universal-Omega synchronize pull request 03#527: Add support for tags - 13https://github.com/miraheze/CreateWiki/pull/527
[16:23:21] <miraheze-github>	 miraheze/CreateWiki - Universal-Omega the build has errored.
[16:24:28] <miraheze-github>	 miraheze/CreateWiki - Universal-Omega the build has errored.
[16:26:56] <miraheze-github>	 miraheze/CreateWiki - Universal-Omega the build has errored.
[16:27:04] <botifico>	 [02CreateWiki] 07Universal-Omega pushed 031 commit to 03tags [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/b16c437b4eba...53b0e44ece4f
[16:27:05] <botifico>	 [02CreateWiki] 07Universal-Omega 0353b0e44 - Add
[16:27:08] <botifico>	 [02CreateWiki] 07Universal-Omega synchronize pull request 03#527: Add support for tags - 13https://github.com/miraheze/CreateWiki/pull/527
[16:31:37] <botifico>	 [02CreateWiki] 07Universal-Omega pushed 031 commit to 03tags [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/53b0e44ece4f...d3d8220803bf
[16:31:40] <botifico>	 [02CreateWiki] 07Universal-Omega 03d3d8220 - Add
[16:31:42] <botifico>	 [02CreateWiki] 07Universal-Omega synchronize pull request 03#527: Add support for tags - 13https://github.com/miraheze/CreateWiki/pull/527
[16:32:41] <miraheze-github>	 miraheze/CreateWiki - Universal-Omega the build has errored.
[16:37:08] <miraheze-github>	 miraheze/CreateWiki - Universal-Omega the build has errored.
[16:39:40] <botifico>	 [02CreateWiki] 07Universal-Omega pushed 031 commit to 03tags [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/d3d8220803bf...9a2134370d57
[16:39:43] <botifico>	 [02CreateWiki] 07Universal-Omega 039a21343 - Add
[16:39:46] <botifico>	 [02CreateWiki] 07Universal-Omega synchronize pull request 03#527: Add support for tags - 13https://github.com/miraheze/CreateWiki/pull/527
[16:40:57] <botifico>	 [02CreateWiki] 07Universal-Omega pushed 031 commit to 03tags [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/9a2134370d57...5e60913b3e2f
[16:40:58] <botifico>	 [02CreateWiki] 07Universal-Omega 035e60913 - Add
[16:41:01] <botifico>	 [02CreateWiki] 07Universal-Omega synchronize pull request 03#527: Add support for tags - 13https://github.com/miraheze/CreateWiki/pull/527
[16:45:29] <miraheze-github>	 miraheze/CreateWiki - Universal-Omega the build has errored.
[16:46:39] <miraheze-github>	 miraheze/CreateWiki - Universal-Omega the build has errored.
[16:49:43] <botifico>	 [02CreateWiki] 07Universal-Omega pushed 031 commit to 03tags [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/5e60913b3e2f...03178fecea9b
[16:49:46] <botifico>	 [02CreateWiki] 07Universal-Omega 0303178fe - Fix
[16:49:49] <botifico>	 [02CreateWiki] 07Universal-Omega synchronize pull request 03#527: Add support for tags - 13https://github.com/miraheze/CreateWiki/pull/527
[16:52:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[16:57:08] <botifico>	 [02CreateWiki] 07Universal-Omega pushed 031 commit to 03tags [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/03178fecea9b...a1ac70ecde02
[16:57:09] <botifico>	 [02CreateWiki] 07Universal-Omega 03a1ac70e - No Object.entries
[16:57:12] <botifico>	 [02CreateWiki] 07Universal-Omega synchronize pull request 03#527: Add support for tags - 13https://github.com/miraheze/CreateWiki/pull/527
[16:57:19] <miraheze-github>	 miraheze/CreateWiki - Universal-Omega the build has errored.
[16:58:56] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.69, 20.67, 15.34
[17:02:45] <miraheze-github>	 miraheze/CreateWiki - Universal-Omega the build has errored.
[17:02:56] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.52, 23.37, 17.58
[17:04:12] <botifico>	 [02CreateWiki] 07Universal-Omega pushed 031 commit to 03tags [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/a1ac70ecde02...03ad214722cf
[17:04:14] <botifico>	 [02CreateWiki] 07Universal-Omega 0303ad214 - No hasOwnProperty
[17:04:16] <botifico>	 [02CreateWiki] 07Universal-Omega synchronize pull request 03#527: Add support for tags - 13https://github.com/miraheze/CreateWiki/pull/527
[17:08:56] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.35, 23.31, 19.57
[17:10:56] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 22.76, 24.01, 20.33
[17:12:14] <miraheze-github>	 miraheze/CreateWiki - Universal-Omega the build passed.
[17:12:56] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.54, 23.28, 20.53
[17:14:56] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.82, 24.70, 21.35
[17:16:56] <icinga-miraheze>	 PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 18.51, 22.52, 20.96
[17:20:56] <icinga-miraheze>	 RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 11.16, 17.58, 19.42
[17:29:04] <MirahezeRelay>	 <koreirose> [1/2] https://meta.miraheze.org/wiki/MediaWiki:Common.css?diff=prev&oldid=386826
[17:29:05] <MirahezeRelay>	 <koreirose> [2/2] Internal error on meta
[17:29:11] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[17:29:15] <MirahezeRelay>	 <koreirose> [bff74c43c1bbdf2c10a430bf] 2024-05-13 17:28:39: Fatal exception of type "MediaWiki\Revision\RevisionAccessException"
[17:30:44] <MirahezeRelay>	 <rhinosf1, replying to koreirose> That is erm wrong
[17:30:55] <MirahezeRelay>	 <rhinosf1> https://meta.miraheze.org/wiki/MediaWiki:Common.css?action=history is not right
[17:32:01] <MirahezeRelay>	 <koreirose> ?
[17:33:05] <MirahezeRelay>	 <rhinosf1> They are allocated to the wrong page afaics
[17:33:13] <MirahezeRelay>	 <rhinosf1> https://meta.miraheze.org/wiki/MediaWiki:Common.css?diff=386826&oldid=328495 is the broken bit
[17:33:17] <MirahezeRelay>	 <justleafy2003, replying to rhinosf1> [1/2] I wonder what even happened here? :ThinkingHardMH:
[17:33:18] <MirahezeRelay>	 <justleafy2003, replying to rhinosf1> [2/2] https://cdn.discordapp.com/attachments/808001911868489748/1239631568205516830/image.png?ex=6643a05d&is=66424edd&hm=50ca71a0313c36ce9fcf20ca9736eb6a0cfd3f14083a7a15ebee83f25b017779&
[17:33:41] <MirahezeRelay>	 <rhinosf1> @MediaWiki Specialists
[17:33:54] <MirahezeRelay>	 <rhinosf1, replying to justleafy2003> Data corruption
[17:34:03] <MirahezeRelay>	 <justleafy2003> I see
[17:34:11] <MirahezeRelay>	 <justleafy2003> first time I've seen this happen lol
[17:34:22] <MirahezeRelay>	 <koreirose> Same
[17:35:20] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 131.60 ms
[17:36:47] <MirahezeRelay>	 <originalauthority> pretty sure it needs to be fixed in the database
[17:37:23] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 133.39 ms
[17:37:25] <icinga-miraheze>	 [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[17:37:42] <MirahezeRelay>	 <rhinosf1, replying to originalauthority> MW can use sql.php
[17:37:53] <MirahezeRelay>	 <rhinosf1> But I'd like to know why it's broke first
[17:37:58] <MirahezeRelay>	 <originalauthority> 🤷🏼‍♂️
[17:38:25] <MirahezeRelay>	 <bluemoon0332, replying to rhinosf1> a cosmic ray hit the nvme drive of the db server in just the right way
[17:38:36] <MirahezeRelay>	 <rhinosf1, replying to bluemoon0332> One I believe
[17:39:29] <MirahezeRelay>	 <rhinosf1> I'm slightly nervous about just deleting revisions from the database
[17:39:35] <MirahezeRelay>	 <bluemoon0332> maybe solving this bug will allow me to claim that TTC upwarp glitch bounty /s
[17:39:41] <MirahezeRelay>	 <rhinosf1> Maybe infra should back it up first
[17:42:55] <botifico>	 [02CreateWiki] 07Universal-Omega created branch 03namespaces 13https://github.com/miraheze/CreateWiki/commit/ec32b6f0c2b267f527aac1675dc38e7aa1ca9c81
[17:42:56] <botifico>	 [02CreateWiki] 07Universal-Omega created branch 03namespaces - 13https://github.com/miraheze/CreateWiki
[17:43:59] <botifico>	 [02CreateWiki] 07Universal-Omega pushed 031 commit to 03namespaces [+0/-0/±23] 13https://github.com/miraheze/CreateWiki/compare/ec32b6f0c2b2...1c593b253c13
[17:44:00] <botifico>	 [02CreateWiki] 07Universal-Omega 031c593b2 - Use namespaced classes
[17:44:15] <botifico>	 [02CreateWiki] 07Universal-Omega opened pull request 03#528: Use namespaced classes - 13https://github.com/miraheze/CreateWiki/pull/528
[17:46:06] <botifico>	 [02CreateWiki] 07github-actions[bot] pushed 031 commit to 03namespaces [+0/-0/±13] 13https://github.com/miraheze/CreateWiki/compare/1c593b253c13...15b977f9d770
[17:46:07] <botifico>	 [02CreateWiki] 07github-actions 0315b977f - CI: lint code to MediaWiki standards
[17:46:08] <botifico>	 [02CreateWiki] 07github-actions[bot] synchronize pull request 03#528: Use namespaced classes - 13https://github.com/miraheze/CreateWiki/pull/528
[17:49:54] <botifico>	 [02CreateWiki] 07Universal-Omega pushed 031 commit to 03namespaces [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/15b977f9d770...9cc6693f2b61
[17:49:55] <botifico>	 [02CreateWiki] 07Universal-Omega 039cc6693 - Add GlobalVarConfig
[17:49:57] <botifico>	 [02CreateWiki] 07Universal-Omega synchronize pull request 03#528: Use namespaced classes - 13https://github.com/miraheze/CreateWiki/pull/528
[17:52:24] <botifico>	 [02CreateWiki] 07Universal-Omega pushed 031 commit to 03namespaces [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/9cc6693f2b61...8baf64d77bba
[17:52:25] <botifico>	 [02CreateWiki] 07Universal-Omega 038baf64d - Namespace GlobalVarConfig
[17:52:28] <botifico>	 [02CreateWiki] 07Universal-Omega synchronize pull request 03#528: Use namespaced classes - 13https://github.com/miraheze/CreateWiki/pull/528
[17:57:44] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query.
[18:00:55] <miraheze-github>	 miraheze/CreateWiki - Universal-Omega the build passed.
[18:27:06] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[18:53:11] <botifico>	 [02MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/7d8ac7dd3070...f055b027ac63
[18:53:13] <botifico>	 [02MirahezeMagic] 07Universal-Omega 03f055b02 - Use namespaced GlobalVarConfig
[18:53:19] <MirahezeLSBot>	 !log [@test151] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151
[18:53:20] <MirahezeLSBot>	 !log [@test151] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s
[18:53:23] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:53:27] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:53:30] <MirahezeLSBot>	 !log [@test151] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151
[18:53:31] <MirahezeLSBot>	 !log [@test151] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s
[18:53:34] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:53:38] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:53:42] <MirahezeLSBot>	 !log [@test151] starting deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151
[18:53:43] <MirahezeLSBot>	 !log [@test151] finished deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s
[18:53:46] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:53:50] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:56:33] <miraheze-github>	 miraheze/MirahezeMagic - Universal-Omega the build has errored.
[18:57:06] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query.
[18:59:08] <botifico>	 [02RemovePII] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/RemovePII/compare/da063388fa6a...161735e3e658
[18:59:10] <botifico>	 [02RemovePII] 07Universal-Omega 03161735e - Use namespaced GlobalVarConfig
[19:01:14] <botifico>	 [02CreateWiki] 07Universal-Omega pushed 034 commits to 03InsertQueryBuilder [+0/-0/±4] 13https://github.com/miraheze/CreateWiki/compare/41c508f4da16...579110210348
[19:01:15] <botifico>	 [02CreateWiki] 07Universal-Omega 033164b3b - Migrate to use InsertQueryBuilder
[19:01:16] <botifico>	 [02CreateWiki] 07Universal-Omega 032500662 - Migrate
[19:01:18] <botifico>	 [02CreateWiki] 07Universal-Omega 0324ca184 - Migrate
[19:01:21] <botifico>	 [02CreateWiki] 07Universal-Omega 035791102 - Migrate
[19:01:22] <botifico>	 [02CreateWiki] 07Universal-Omega synchronize pull request 03#485: Migrate to use InsertQueryBuilder - 13https://github.com/miraheze/CreateWiki/pull/485
[19:02:10] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket
[19:02:27] <MirahezeLSBot>	 !log [@mwtask181] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all
[19:02:31] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:02:37] <MirahezeLSBot>	 !log [@mwtask181] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 10s
[19:02:41] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:02:49] <MirahezeLSBot>	 !log [@mwtask181] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all
[19:02:52] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:02:58] <MirahezeLSBot>	 !log [@mwtask181] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all - SUCCESS in 9s
[19:03:02] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:03:21] <botifico>	 [02CreateWiki] 07Universal-Omega synchronize pull request 03#485: Migrate to use InsertQueryBuilder - 13https://github.com/miraheze/CreateWiki/pull/485
[19:03:22] <botifico>	 [02CreateWiki] 07Universal-Omega pushed 031 commit to 03InsertQueryBuilder [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/579110210348...89e3ce5f10ee
[19:03:24] <botifico>	 [02CreateWiki] 07Universal-Omega 0389e3ce5 - One row
[19:04:11] <botifico>	 [02CreateWiki] 07Universal-Omega synchronize pull request 03#485: Migrate to use InsertQueryBuilder - 13https://github.com/miraheze/CreateWiki/pull/485
[19:04:14] <botifico>	 [02CreateWiki] 07Universal-Omega pushed 031 commit to 03InsertQueryBuilder [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/89e3ce5f10ee...02aef8acd053
[19:04:16] <botifico>	 [02CreateWiki] 07Universal-Omega 0302aef8a - One row
[19:04:43] <miraheze-github>	 miraheze/RemovePII - Universal-Omega the build has errored.
[19:04:45] <botifico>	 [02CreateWiki] 07Universal-Omega synchronize pull request 03#485: Migrate to use InsertQueryBuilder - 13https://github.com/miraheze/CreateWiki/pull/485
[19:04:46] <botifico>	 [02CreateWiki] 07Universal-Omega pushed 031 commit to 03InsertQueryBuilder [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/02aef8acd053...f8dde035a277
[19:04:48] <botifico>	 [02CreateWiki] 07Universal-Omega 03f8dde03 - One row
[19:05:18] <botifico>	 [02CreateWiki] 07Universal-Omega synchronize pull request 03#485: Migrate to use InsertQueryBuilder - 13https://github.com/miraheze/CreateWiki/pull/485
[19:05:20] <botifico>	 [02CreateWiki] 07Universal-Omega pushed 031 commit to 03InsertQueryBuilder [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/f8dde035a277...bfd918084326
[19:05:22] <botifico>	 [02CreateWiki] 07Universal-Omega 03bfd9180 - One row
[19:07:49] <MirahezeLSBot>	 !log [@mwtask171] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all
[19:07:53] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:07:58] <MirahezeLSBot>	 !log [@mwtask171] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 8s
[19:08:02] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:08:08] <MirahezeLSBot>	 !log [@mwtask171] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all
[19:08:12] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:08:16] <MirahezeLSBot>	 !log [@mwtask171] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all - SUCCESS in 8s
[19:08:20] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:11:28] <miraheze-github>	 miraheze/CreateWiki - Universal-Omega the build has errored.
[19:18:45] <botifico>	 [02CreateWiki] 07Universal-Omega synchronize pull request 03#485: Migrate to use InsertQueryBuilder - 13https://github.com/miraheze/CreateWiki/pull/485
[19:18:48] <botifico>	 [02CreateWiki] 07Universal-Omega pushed 031 commit to 03InsertQueryBuilder [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/bfd918084326...698ee0ea0ead
[19:18:51] <botifico>	 [02CreateWiki] 07Universal-Omega 03698ee0e - Fix
[19:22:06] <botifico>	 [02ci-scripts] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ci-scripts/compare/8eeeb5e04583...b99fa300930e
[19:22:08] <botifico>	 [02ci-scripts] 07Universal-Omega 03b99fa30 - One row
[19:24:23] <miraheze-github>	 miraheze/CreateWiki - Universal-Omega the build has errored.
[19:27:25] <botifico>	 [02CreateWiki] 07Universal-Omega pushed 031 commit to 03InsertQueryBuilder [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/698ee0ea0ead...322e91db0c78
[19:27:28] <botifico>	 [02CreateWiki] 07Universal-Omega 03322e91d - Temp exception
[19:27:31] <botifico>	 [02CreateWiki] 07Universal-Omega synchronize pull request 03#485: Migrate to use InsertQueryBuilder - 13https://github.com/miraheze/CreateWiki/pull/485
[19:30:28] <miraheze-github>	 miraheze/CreateWiki - Universal-Omega the build has errored.
[19:31:34] <botifico>	 [02CreateWiki] 07Universal-Omega pushed 031 commit to 03InsertQueryBuilder [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/322e91db0c78...6a1e667f8f4c
[19:31:37] <botifico>	 [02CreateWiki] 07Universal-Omega 036a1e667 - Temp experimental
[19:31:38] <botifico>	 [02CreateWiki] 07Universal-Omega synchronize pull request 03#485: Migrate to use InsertQueryBuilder - 13https://github.com/miraheze/CreateWiki/pull/485
[19:40:03] <miraheze-github>	 miraheze/CreateWiki - Universal-Omega the build passed.
[19:47:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1
[19:50:29] <botifico>	 [02CreateWiki] 07Universal-Omega synchronize pull request 03#485: Migrate to use InsertQueryBuilder - 13https://github.com/miraheze/CreateWiki/pull/485
[19:50:31] <botifico>	 [02CreateWiki] 07Universal-Omega pushed 031 commit to 03InsertQueryBuilder [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/6a1e667f8f4c...e59e850e2dea
[19:50:33] <botifico>	 [02CreateWiki] 07Universal-Omega 03e59e850 - Fix
[19:57:09] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK
[19:57:54] <botifico>	 [02CreateWiki] 07Universal-Omega synchronize pull request 03#485: Migrate to use InsertQueryBuilder - 13https://github.com/miraheze/CreateWiki/pull/485
[19:57:57] <botifico>	 [02CreateWiki] 07Universal-Omega pushed 031 commit to 03InsertQueryBuilder [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/e59e850e2dea...c4a108b00267
[19:58:00] <botifico>	 [02CreateWiki] 07Universal-Omega 03c4a108b - -
[19:58:11] <botifico>	 [02CreateWiki] 07Universal-Omega synchronize pull request 03#485: Migrate to use InsertQueryBuilder - 13https://github.com/miraheze/CreateWiki/pull/485
[19:58:12] <botifico>	 [02CreateWiki] 07Universal-Omega pushed 031 commit to 03InsertQueryBuilder [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/c4a108b00267...f5e8b9f0d7a3
[19:58:13] <botifico>	 [02CreateWiki] 07Universal-Omega 03f5e8b9f - -
[19:58:44] <miraheze-github>	 miraheze/CreateWiki - Universal-Omega the build passed.
[20:01:31] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mahdiruiz.line.pm' will expire on Fri 14 Jun 2024 04:28:50 PM GMT +0000.
[20:03:34] <botifico>	 [02CreateWiki] 07Universal-Omega pushed 031 commit to 03InsertQueryBuilder [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/f5e8b9f0d7a3...e017d8aa0742
[20:03:36] <botifico>	 [02CreateWiki] 07Universal-Omega 03e017d8a - Temp experimental again
[20:03:37] <botifico>	 [02CreateWiki] 07Universal-Omega synchronize pull request 03#485: Migrate to use InsertQueryBuilder - 13https://github.com/miraheze/CreateWiki/pull/485
[20:04:35] <botifico>	 [02CreateWiki] 07Universal-Omega pushed 031 commit to 03InsertQueryBuilder [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/e017d8aa0742...9c3f03d92837
[20:04:38] <botifico>	 [02CreateWiki] 07Universal-Omega 039c3f03d - Test
[20:04:41] <botifico>	 [02CreateWiki] 07Universal-Omega synchronize pull request 03#485: Migrate to use InsertQueryBuilder - 13https://github.com/miraheze/CreateWiki/pull/485
[20:10:21] <botifico>	 [02CreateWiki] 07Universal-Omega pushed 031 commit to 03InsertQueryBuilder [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/9c3f03d92837...dfa63a2a318e
[20:10:22] <botifico>	 [02CreateWiki] 07Universal-Omega 03dfa63a2 - Test
[20:10:23] <botifico>	 [02CreateWiki] 07Universal-Omega synchronize pull request 03#485: Migrate to use InsertQueryBuilder - 13https://github.com/miraheze/CreateWiki/pull/485
[20:12:55] <botifico>	 [02CreateWiki] 07github-actions[bot] pushed 031 commit to 03InsertQueryBuilder [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/dfa63a2a318e...19305d25d771
[20:12:56] <botifico>	 [02CreateWiki] 07github-actions 0319305d2 - CI: lint code to MediaWiki standards
[20:12:59] <botifico>	 [02CreateWiki] 07github-actions[bot] synchronize pull request 03#485: Migrate to use InsertQueryBuilder - 13https://github.com/miraheze/CreateWiki/pull/485
[20:19:04] <miraheze-github>	 miraheze/CreateWiki - Universal-Omega the build passed.
[20:31:23] <botifico>	 [02CreateWiki] 07Universal-Omega synchronize pull request 03#485: Migrate to use InsertQueryBuilder - 13https://github.com/miraheze/CreateWiki/pull/485
[20:31:24] <botifico>	 [02CreateWiki] 07Universal-Omega pushed 031 commit to 03InsertQueryBuilder [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/19305d25d771...1cd78e1d516f
[20:31:26] <botifico>	 [02CreateWiki] 07Universal-Omega 031cd78e1 - -
[20:31:56] <botifico>	 [02CreateWiki] 07Universal-Omega synchronize pull request 03#485: Migrate to use InsertQueryBuilder - 13https://github.com/miraheze/CreateWiki/pull/485
[20:31:58] <botifico>	 [02CreateWiki] 07Universal-Omega pushed 031 commit to 03InsertQueryBuilder [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/1cd78e1d516f...e0eb3b579580
[20:32:01] <botifico>	 [02CreateWiki] 07Universal-Omega 03e0eb3b5 - -
[20:32:13] <botifico>	 [02CreateWiki] 07Universal-Omega synchronize pull request 03#485: Migrate to use InsertQueryBuilder - 13https://github.com/miraheze/CreateWiki/pull/485
[20:32:15] <botifico>	 [02CreateWiki] 07Universal-Omega pushed 031 commit to 03InsertQueryBuilder [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/e0eb3b579580...c2784a5eb0f5
[20:32:16] <botifico>	 [02CreateWiki] 07Universal-Omega 03c2784a5 - -
[20:40:21] <miraheze-github>	 miraheze/CreateWiki - Universal-Omega the build passed.
[20:47:42] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/8655cb3357cd...f073c37628a5
[20:47:44] <botifico>	 [02ssl] 07WikiTideSSLBot 03f073c37 - Bot: Add SSL cert for wiki.pokrew.net
[20:48:44] <botifico>	 [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/f073c37628a5...67400c5d37e5
[20:48:47] <botifico>	 [02ssl] 07WikiTideSSLBot 0367400c5 - Bot: Add SSL cert for marvelcinematicuniverse.wiki
[20:56:07] <MirahezeRelay>	 <elliethepwincess, replying to rhinosf1> https://meta.miraheze.org/wiki/Tech:Db1?action=history also has broken revisions
[21:00:44] <MirahezeRelay>	 <rhinosf1, replying to elliethepwincess> That would make a lot of sense
[21:02:30] <MirahezeRelay>	 <cosmicalpha> @rhinosf1 Wait so a db server page has database corruption?
[21:03:54] <MirahezeRelay>	 <bluemoon0332> there's also https://meta.miraheze.org/wiki/Miraheze_Meta?oldid=349809
[21:04:02] <MirahezeRelay>	 <rhinosf1, replying to cosmicalpha> And Mediawiki:Common.css
[21:04:17] <MirahezeRelay>	 <cosmicalpha> Oh... this is that thing caused when John compressed revisions the wrong way. The only fix was deletion of affected revisions on hundreds of wikis and created a mess we were still actively ckeaning up a yesr later, apparently longer...
[21:04:47] <MirahezeRelay>	 <rhinosf1> Can you clean meta up?
[21:05:36] <MirahezeRelay>	 <cosmicalpha> If I can remember how I did it last time. I think findBadBlobs or checkStorage or something
[21:06:54] <MirahezeRelay>	 <cosmicalpha> I remember the scripts were broken so I had to upstream patches to fix it and also made my own script to basically import old revisions from archive.org backups to repair but not sure we still have them...
[21:07:26] <MirahezeRelay>	 <cosmicalpha> Its been awhile though.
[21:07:52] <botifico>	 [02mw-config] 07brandon-wm opened pull request 03#5557: T12108: Please alter these DynamicPageList3 variables on metzo.miraheze.org - 13https://github.com/miraheze/mw-config/pull/5557
[21:08:46] <miraheze-github>	 miraheze/mw-config - brandon-wm the build passed.
[21:09:39] <MirahezeRelay>	 <cosmicalpha> https://issue-tracker.miraheze.org/T8184 here was the original task I mass fixed the issues for 150 wikis. And since then many more have popped up also.
[21:12:40] <MirahezeRelay>	 <cosmicalpha> Anyway yeah I'll clean up Meta later @rhinosf1
[21:12:46] <MirahezeRelay>	 <rhinosf1> Thanks!
[21:31:43] <botifico>	 [02mw-config] 07brandon-wm opened pull request 03#5558: T12085: About “Extension:TabberNeue” - 13https://github.com/miraheze/mw-config/pull/5558
[21:32:39] <miraheze-github>	 miraheze/mw-config - brandon-wm the build passed.
[21:35:35] <botifico>	 [02mw-config] 07brandon-wm opened pull request 03#5559: T11757: Disable Upload Wizard Tutorial and Remove Creative Commons selection on aieseattle.miraheze.org - 13https://github.com/miraheze/mw-config/pull/5559
[21:36:18] <MirahezeRelay>	 <cosmicalpha, replying to rhinosf1> hmm it says 0 bad revisions
[22:28:08] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[23:26:17] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query.
[23:47:25] <icinga-miraheze>	 [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1