[02:04:09] [02WikiForge/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/puppet/compare/1e03789cdd02...3f02a4690af4 [02:04:10] [02WikiForge/puppet] 07AgentIsai 033f02a46 - Remove wikiforge.net wildcard [02:06:15] [02WikiForge/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/puppet/compare/3f02a4690af4...ace6632de544 [02:06:16] [02WikiForge/puppet] 07AgentIsai 03ace6632 - Remove monitor statement [02:08:09] RECOVERY - swiftobject1 Puppet on swiftobject1 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [02:09:01] RECOVERY - swiftac1 Puppet on swiftac1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:14:29] [02WikiForge/dns] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/dns/compare/20e7b75da1d2...20067de78a9a [02:14:32] [02WikiForge/dns] 07AgentIsai 0320067de - Add new records [02:15:09] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:18:34] [02WikiForge/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/WikiForge/puppet/compare/ace6632de544...eefea72f63e5 [02:18:37] [02WikiForge/puppet] 07AgentIsai 03eefea72 - Fix references [02:19:53] RECOVERY - mon1 Puppet on mon1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:19:54] PROBLEM - swiftobject1 Current Load on swiftobject1 is CRITICAL: connect to address 10.0.0.121 port 5666: No route to hostconnect to host 10.0.0.121 port 5666: No route to host [02:19:59] WikiForge/puppet - AgentIsai the build has errored. [02:20:00] PROBLEM - swiftobject1 Puppet on swiftobject1 is CRITICAL: connect to address 10.0.0.121 port 5666: No route to hostconnect to host 10.0.0.121 port 5666: No route to host [02:20:03] PROBLEM - swiftobject1 NTP time on swiftobject1 is CRITICAL: connect to address 10.0.0.121 port 5666: No route to hostconnect to host 10.0.0.121 port 5666: No route to host [02:20:10] PROBLEM - swiftobject1 Disk Space on swiftobject1 is CRITICAL: connect to address 10.0.0.121 port 5666: No route to hostconnect to host 10.0.0.121 port 5666: No route to host [02:20:28] PROBLEM - swiftobject1 SSH on swiftobject1 is CRITICAL: connect to address 10.0.0.121 and port 22: No route to host [02:20:30] PROBLEM - swiftproxy1 PowerDNS Recursor on swiftproxy1 is CRITICAL: Domain 'wikiforge.net' was not found by the server [02:21:04] PROBLEM - Host swiftobject1 is DOWN: CRITICAL - Host Unreachable (10.0.0.121) [02:21:07] PROBLEM - swiftproxy1 HTTP on swiftproxy1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:21:32] PROBLEM - swiftproxy1 HTTPS on swiftproxy1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:23:03] RECOVERY - Host swiftobject1 is UP: PING OK - Packet loss = 0%, RTA = 0.38 ms [02:23:27] RECOVERY - swiftobject1 Swift Object Service on swiftobject1 is OK: TCP OK - 0.001 second response time on 10.0.0.121 port 6000 [02:23:52] RECOVERY - swiftobject1 Puppet on swiftobject1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:23:57] RECOVERY - swiftobject1 Current Load on swiftobject1 is OK: LOAD OK - total load average: 0.44, 0.29, 0.12 [02:23:59] RECOVERY - mail1 Puppet on mail1 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [02:24:02] RECOVERY - swiftobject1 Disk Space on swiftobject1 is OK: DISK OK - free space: / 25571MiB (89% inode=95%); [02:24:11] RECOVERY - swiftobject1 NTP time on swiftobject1 is OK: NTP OK: Offset 0.002045631409 secs [02:24:26] RECOVERY - swiftobject1 SSH on swiftobject1 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [02:28:04] RECOVERY - bast1 Puppet on bast1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:34:07] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:37:51] !log [swift@swiftobject1] Script: Beginning S3 download of all folders [02:37:52] !log [swift@swiftobject1] Script: Finished S3 download of all folders [02:38:26] !log [swift@swiftobject1] Script: Beginning S3 download of all folders [02:41:33] [02WikiForge/mw-config] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/mw-config/compare/291d9474fc2b...8028664e0112 [02:41:34] [02WikiForge/mw-config] 07AgentIsai 038028664 - Begin file maintenance [02:42:01] !log [agent@jobrunner1] starting deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [mw1, mw2, mw3, jobrunner1] [02:42:08] !log [agent@jobrunner1] finished deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [mw1, mw2, mw3, jobrunner1] - SUCCESS in 7s [02:42:40] WikiForge/mw-config - AgentIsai the build passed. [02:47:08] [02WikiForge/mw-config] 07AgentIsai pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/WikiForge/mw-config/compare/8028664e0112...e51a1ada8731 [02:47:09] [02WikiForge/mw-config] 07AgentIsai 03e51a1ad - Create FileBackend [02:48:14] WikiForge/mw-config - AgentIsai the build passed. [02:49:51] [02WikiForge/mw-config] 07AgentIsai pushed 031 commit to 03AgentIsai-patch-2 [+0/-0/±1] 13https://github.com/WikiForge/mw-config/commit/d64f35497cc2 [02:49:54] [02WikiForge/mw-config] 07AgentIsai 03d64f354 - Migrate to Swift [02:49:56] [02mw-config] 07AgentIsai created branch 03AgentIsai-patch-2 - 13https://github.com/WikiForge/mw-config [02:50:07] [02mw-config] 07AgentIsai opened pull request 03#141: Migrate to Swift - 13https://github.com/WikiForge/mw-config/pull/141 [02:50:22] [02WikiForge/mw-config] 07AgentIsai pushed 031 commit to 03AgentIsai-patch-2 [+0/-0/±1] 13https://github.com/WikiForge/mw-config/compare/d64f35497cc2...936feed87cd1 [02:50:25] [02WikiForge/mw-config] 07AgentIsai 03936feed - Remove AWS extension [02:50:26] [02mw-config] 07AgentIsai synchronize pull request 03#141: Migrate to Swift - 13https://github.com/WikiForge/mw-config/pull/141 [02:50:54] [02WikiForge/mw-config] 07github-actions[bot] pushed 031 commit to 03AgentIsai-patch-2 [+0/-0/±2] 13https://github.com/WikiForge/mw-config/compare/936feed87cd1...e8476a002d4b [02:50:57] [02WikiForge/mw-config] 07github-actions 03e8476a0 - CI: lint code to MediaWiki standards Check commit and GitHub actions for more details [02:50:59] [02mw-config] 07github-actions[bot] synchronize pull request 03#141: Migrate to Swift - 13https://github.com/WikiForge/mw-config/pull/141 [02:51:24] WikiForge/mw-config - AgentIsai the build passed. [02:51:29] [02WikiForge/mw-config] 07AgentIsai pushed 031 commit to 03AgentIsai-patch-2 [+0/-0/±1] 13https://github.com/WikiForge/mw-config/compare/e8476a002d4b...8fabee36a5dc [02:51:32] [02WikiForge/mw-config] 07AgentIsai 038fabee3 - Add FileBackend to LocalSettings [02:51:34] [02mw-config] 07AgentIsai synchronize pull request 03#141: Migrate to Swift - 13https://github.com/WikiForge/mw-config/pull/141 [02:52:24] WikiForge/mw-config - AgentIsai the build passed. [02:55:12] [02WikiForge/mw-config] 07AgentIsai pushed 031 commit to 03AgentIsai-patch-2 [+0/-0/±1] 13https://github.com/WikiForge/mw-config/compare/8fabee36a5dc...ba6dd700354d [02:55:14] [02WikiForge/mw-config] 07AgentIsai 03ba6dd70 - Add container config [02:55:17] [02mw-config] 07AgentIsai synchronize pull request 03#141: Migrate to Swift - 13https://github.com/WikiForge/mw-config/pull/141 [02:56:19] [02WikiForge/mw-config] 07AgentIsai pushed 031 commit to 03AgentIsai-patch-2 [+0/-0/±1] 13https://github.com/WikiForge/mw-config/compare/ba6dd700354d...bce3ea30eabd [02:56:21] [02WikiForge/mw-config] 07AgentIsai 03bce3ea3 - Disable onExtensionFunctions [02:56:22] [02mw-config] 07AgentIsai synchronize pull request 03#141: Migrate to Swift - 13https://github.com/WikiForge/mw-config/pull/141 [02:56:52] WikiForge/mw-config - AgentIsai the build has errored. [02:58:02] WikiForge/mw-config - AgentIsai the build has errored. [02:59:48] [02WikiForge/mw-config] 07AgentIsai pushed 031 commit to 03AgentIsai-patch-2 [+0/-0/±1] 13https://github.com/WikiForge/mw-config/compare/bce3ea30eabd...92d1e25a4704 [02:59:49] [02WikiForge/mw-config] 07AgentIsai 0392d1e25 - Fix array [02:59:50] [02mw-config] 07AgentIsai synchronize pull request 03#141: Migrate to Swift - 13https://github.com/WikiForge/mw-config/pull/141 [03:01:47] WikiForge/mw-config - AgentIsai the build passed. [03:05:11] !log [agent@jobrunner1] starting deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [jobrunner1] [03:05:14] !log [agent@jobrunner1] finished deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [jobrunner1] - SUCCESS in 2s [03:06:34] !log [agent@jobrunner1] starting deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [mw1, mw2, mw3, jobrunner1] [03:06:37] !log [agent@jobrunner1] finished deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [mw1, mw2, mw3, jobrunner1] - SUCCESS in 3s [03:06:58] [02mw-config] 07AgentIsai closed pull request 03#141: Migrate to Swift - 13https://github.com/WikiForge/mw-config/pull/141 [03:07:00] [02WikiForge/mw-config] 07AgentIsai pushed 031 commit to 03master [+0/-0/±4] 13https://github.com/WikiForge/mw-config/compare/e51a1ada8731...e9f16502f135 [03:07:03] [02WikiForge/mw-config] 07AgentIsai 03e9f1650 - Migrate to Swift (#141) * Migrate to Swift * Remove AWS extension Check commit and GitHub actions for more details * Add FileBackend to LocalSettings * Add container config * Disable onExtensionFunctions * Fix array [03:07:04] [02WikiForge/mw-config] 07AgentIsai deleted branch 03AgentIsai-patch-2 [03:07:07] [02mw-config] 07AgentIsai deleted branch 03AgentIsai-patch-2 - 13https://github.com/WikiForge/mw-config [03:07:08] !log [agent@jobrunner1] starting deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [jobrunner1] [03:07:09] !log [agent@jobrunner1] finished deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [jobrunner1] - SUCCESS in 1s [03:07:59] WikiForge/mw-config - AgentIsai the build passed. [03:24:01] [02WikiForge/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/puppet/compare/eefea72f63e5...11b7107de5b6 [03:24:03] [02WikiForge/puppet] 07AgentIsai 0311b7107 - Add Swift password [03:25:17] [02WikiForge/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/puppet/compare/11b7107de5b6...61343bd96b1a [03:25:19] [02WikiForge/puppet] 07AgentIsai 0361343bd - Add swift_password [03:38:01] [02WikiForge/WikiForgeMagic] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/WikiForgeMagic/compare/17f01d1abb06...41ae483e43ec [03:38:04] [02WikiForge/WikiForgeMagic] 07AgentIsai 0341ae483 - Add Swift support [03:38:25] !log [agent@jobrunner1] starting deploy of {'pull': 'world', 'upgrade_world': True, 'upgrade_vendor': True, 'world': True, 'l10n': True, 'extension_list': True, 'force': True, 'versions': ['1.40'], 'upgrade_extensions': ['3D', 'AJAXPoll', 'AWS', 'AbuseFilter', 'AddHTMLMetaAndTitle', 'AdminLinks', 'AdvancedSearch', 'AntiSpoof', 'ApprovedRevs', 'Arrays', [03:38:26] 'ArticleCreationWorkflow', 'ArticlePlaceholder', 'ArticleRatings', 'ArticleToCategory2', 'AuthorProtect', 'AutoCreateCategoryPages', 'AutoCreatePage', 'Babel', 'BetaFeatures', 'BlogPage', 'Bootstrap', 'CSS', 'Calendar', 'Capiunto', 'Cargo', 'CategoryExplorer', 'CategorySortHeaders', 'CategoryTree', 'CharInsert', 'CheckUser', 'CirrusSearch', 'Cite', [03:38:27] 'CiteThisPage', 'Citoid', 'CleanChanges', 'CodeEditor', 'CodeMirror', 'CollapsibleVector', 'Collection', 'CommentStreams', 'Commentbox', 'Comments', 'CommonsMetadata', 'ConfirmAccount', 'ConfirmEdit', 'ContactPage', 'ContributionScores', 'CookieWarning', 'CountDownClock', 'CreatePage', 'CreatePageUw', 'CreateRedirect', 'CreateWiki', 'CreatedPagesList', [03:38:28] 'DPLForum', 'DarkMode', 'DataDump', 'DataTransfer', 'DeleteUserPages', 'Description2', 'Disambiguator', 'DiscordNotifications', 'DiscussionTools', 'DismissableSiteNotice', 'DisplayTitle', 'DummyFandoomMainpageTags', 'DynamicPageList', 'DynamicPageList3', 'DynamicSidebar', 'Echo', 'EditNotify', 'EditSubpages', 'Editcount', 'Elastica', 'ElectronPdfService', [03:38:29] 'EmailAuthorization', 'EmbedSpotify', 'EmbedVideo', 'EventLogging', 'ExternalData', 'FileStorageMonitor', 'FlaggedRevs', 'FlexDiagrams', 'Flow', 'FontAwesome', 'ForcePreview', 'Gadgets', 'Genealogy', 'GeoCrumbs', 'GeoData', 'GeoGebra', 'GoogleDocs4MW', 'GoogleNewsSitemap', 'Graph', 'GroupsSidebar', 'GrowthExperiments', 'GuidedTour', 'HAWelcome', [03:38:30] 'HeaderTabs', 'HideSection', 'HighlightLinksInCategory', 'HitCounters', 'IPInfo', 'ImageMap', 'ImageRating', 'ImportDump', 'IncidentReporting', 'InputBox', 'Interwiki', 'InterwikiDispatcher', 'InterwikiSorting', 'JSBreadCrumbs', 'JavascriptSlideshow', 'Josa', 'JsCalendar', 'JsonConfig', 'Kartographer', 'LabeledSectionTransclusion', 'LanguageSelector', [03:38:31] 'LastModified', 'LdapAuthentication', 'Lingo', 'LinkSuggest', 'LinkTarget', 'LinkTitles', 'Linter', 'Listings', 'LockAuthor', 'LoginNotify', 'LogoFunctions', 'Loops', 'MagicNoCache', 'ManageWiki', 'Maps', 'MassEditRegex', 'MassMessage', 'Math', 'MatomoAnalytics', 'MediaSearch', 'MediaWikiChat', 'Mermaid', 'MintyDocs', 'MixedNamespaceSearchSuggestions', [03:38:32] 'MobileDetect', 'MobileFrontend', 'MobileTabsPlugin', 'Moderation', 'MsCalendar', 'MsCatSelect', 'MsLinks', 'MsUpload', 'MultiBoilerplate', 'MultimediaViewer', 'MyVariables', 'NamespacePreload', 'NativeSvgHandler', 'NearbyPages', 'NewSignupPage', 'NewUserMessage', 'NewUserNotif', 'NewestPages', 'Newsletter', 'NoTitle', 'Nuke', 'NukeDPL', 'OATHAuth', [03:38:33] 'OAuth', 'OOJSPlus', 'OpenGraphMeta', 'OpenIDConnect', 'OrphanedTalkPages', 'PDFEmbed', 'PWA', 'PageAssessments', 'PageForms', 'PageImages', 'PageNotice', 'PageProperties', 'PageSchemas', 'PageTriage', 'PagedTiffHandler', 'ParserFunctions', 'PdfHandler', 'PipeEscape', 'PluggableAuth', 'Poem', 'PollNY', 'Popups', 'PortableInfobox', 'Preloader', [03:38:34] 'ProofreadPage', 'PropertySuggester', 'ProtectSite', 'ProtectionIndicator', 'Purge', 'QuickInstantCommons', 'Quiz', 'QuizGame', 'RSS', 'RandomGameUnit', 'RandomImage', 'RandomSelection', 'RatePage', 'RealMe', 'RegexFunctions', 'RelatedArticles', 'RemovePII', 'RemoveRedlinks', 'Renameuser', 'ReplaceText', 'Report', 'RevisionSlider', 'RightFunctions', [03:38:35] 'RottenLinks', 'SandboxLink', 'Score', 'Scribunto', 'SearchVue', 'SecurePoll', 'SemanticFormsSelect', 'SemanticMediaWiki', 'SemanticScribunto', 'Share', 'ShortDescription', 'SimpleBlogPage', 'SimpleChanges', 'SimpleSAMLphp', 'SimpleTooltip', 'SkinPerPage', 'SnapProjectEmbed', 'SocialProfile', 'SoftRedirector', 'SpamBlacklist', 'Spoilers', 'SpriteSheet', [03:38:36] 'StandardDialogs', 'StopForumSpam', 'StructuredNavigation', 'SubPageList3', 'SubpageFun', 'SyntaxHighlight_GeSHi', 'TabberNeue', 'Tabs', 'TemplateData', 'TemplateSandbox', 'TemplateStyles', 'TemplateStylesExtender', 'TemplateWizard', 'TextExtracts', 'Thanks', 'Theme', 'TimeMachine', 'TimedMediaHandler', 'Timeline', 'TitleBlacklist', 'TitleKey', 'TocTree', [03:38:37] 'TorBlock', 'Translate', 'TranslationNotifications', 'TreeAndMenu', 'TwitterTag', 'TwoColConflict', 'UniversalLanguageSelector', 'UploadWizard', 'UploadsLink', 'UrlGetParameters', 'UrlShortener', 'UserFunctions', 'UserMerge', 'UserPageEditProtection', 'VEForAll', 'Variables', 'VariablesLua', 'Video', 'VisualEditor', 'VoteNY', 'WSOAuth', 'WebAuthn', [03:38:38] 'WebChat', 'WikiCategoryTagCloud', 'WikiDiscover', 'WikiEditor', 'WikiForgeMagic', 'WikiForum', 'WikiLove', 'WikiSEO', 'WikiTextLoggedInOut', 'Wikibase', 'WikibaseCirrusSearch', 'WikibaseEdtf', 'WikibaseLexeme', 'WikibaseLexemeCirrusSearch', 'WikibaseLocalMedia', 'WikibaseMediaInfo', 'WikibaseQualityConstraints', 'WikidataPageBanner', 'WikimediaIncubator', [03:38:39] 'YouTube', 'cldr', 'mw-ScratchBlocks4', 'wikihiero'], 'upgrade_skins': ['Anisa', 'BlueSky', 'Citizen', 'CologneBlue', 'Cosmos', 'DuskToDawn', 'Femiwiki', 'Gamepress', 'HasSomeColours', 'Mask', 'Medik', 'Metrolook', 'MinervaNeue', 'Mirage', 'Modern', 'Monaco', 'MonoBook', 'Nimbus', 'Nostalgia', 'Pivot', 'Refreshed', 'Splash', 'Timeless', 'Truglass', [03:38:40] 'Tweeki', 'Vector', 'WMAU', 'apex', 'chameleon', 'erudite'], 'ignore_time': True} to [mw1, mw2, mw3, jobrunner1] [03:38:41] !log [agent@jobrunner1] finished deploy of {'pull': 'world', 'upgrade_world': True, 'upgrade_vendor': True, 'world': True, 'l10n': True, 'extension_list': True, 'force': True, 'versions': ['1.40'], 'upgrade_extensions': ['3D', 'AJAXPoll', 'AWS', 'AbuseFilter', 'AddHTMLMetaAndTitle', 'AdminLinks', 'AdvancedSearch', 'AntiSpoof', 'ApprovedRevs', 'Arrays', [03:38:42] 'ArticleCreationWorkflow', 'ArticlePlaceholder', 'ArticleRatings', 'ArticleToCategory2', 'AuthorProtect', 'AutoCreateCategoryPages', 'AutoCreatePage', 'Babel', 'BetaFeatures', 'BlogPage', 'Bootstrap', 'CSS', 'Calendar', 'Capiunto', 'Cargo', 'CategoryExplorer', 'CategorySortHeaders', 'CategoryTree', 'CharInsert', 'CheckUser', 'CirrusSearch', 'Cite', [03:38:43] 'CiteThisPage', 'Citoid', 'CleanChanges', 'CodeEditor', 'CodeMirror', 'CollapsibleVector', 'Collection', 'CommentStreams', 'Commentbox', 'Comments', 'CommonsMetadata', 'ConfirmAccount', 'ConfirmEdit', 'ContactPage', 'ContributionScores', 'CookieWarning', 'CountDownClock', 'CreatePage', 'CreatePageUw', 'CreateRedirect', 'CreateWiki', 'CreatedPagesList', [03:38:44] 'DPLForum', 'DarkMode', 'DataDump', 'DataTransfer', 'DeleteUserPages', 'Description2', 'Disambiguator', 'DiscordNotifications', 'DiscussionTools', 'DismissableSiteNotice', 'DisplayTitle', 'DummyFandoomMainpageTags', 'DynamicPageList', 'DynamicPageList3', 'DynamicSidebar', 'Echo', 'EditNotify', 'EditSubpages', 'Editcount', 'Elastica', 'ElectronPdfService', [03:38:45] 'EmailAuthorization', 'EmbedSpotify', 'EmbedVideo', 'EventLogging', 'ExternalData', 'FileStorageMonitor', 'FlaggedRevs', 'FlexDiagrams', 'Flow', 'FontAwesome', 'ForcePreview', 'Gadgets', 'Genealogy', 'GeoCrumbs', 'GeoData', 'GeoGebra', 'GoogleDocs4MW', 'GoogleNewsSitemap', 'Graph', 'GroupsSidebar', 'GrowthExperiments', 'GuidedTour', 'HAWelcome', [03:38:46] 'HeaderTabs', 'HideSection', 'HighlightLinksInCategory', 'HitCounters', 'IPInfo', 'ImageMap', 'ImageRating', 'ImportDump', 'IncidentReporting', 'InputBox', 'Interwiki', 'InterwikiDispatcher', 'InterwikiSorting', 'JSBreadCrumbs', 'JavascriptSlideshow', 'Josa', 'JsCalendar', 'JsonConfig', 'Kartographer', 'LabeledSectionTransclusion', 'LanguageSelector', [03:38:47] 'LastModified', 'LdapAuthentication', 'Lingo', 'LinkSuggest', 'LinkTarget', 'LinkTitles', 'Linter', 'Listings', 'LockAuthor', 'LoginNotify', 'LogoFunctions', 'Loops', 'MagicNoCache', 'ManageWiki', 'Maps', 'MassEditRegex', 'MassMessage', 'Math', 'MatomoAnalytics', 'MediaSearch', 'MediaWikiChat', 'Mermaid', 'MintyDocs', 'MixedNamespaceSearchSuggestions', [03:38:48] 'MobileDetect', 'MobileFrontend', 'MobileTabsPlugin', 'Moderation', 'MsCalendar', 'MsCatSelect', 'MsLinks', 'MsUpload', 'MultiBoilerplate', 'MultimediaViewer', 'MyVariables', 'NamespacePreload', 'NativeSvgHandler', 'NearbyPages', 'NewSignupPage', 'NewUserMessage', 'NewUserNotif', 'NewestPages', 'Newsletter', 'NoTitle', 'Nuke', 'NukeDPL', 'OATHAuth', [03:38:49] 'OAuth', 'OOJSPlus', 'OpenGraphMeta', 'OpenIDConnect', 'OrphanedTalkPages', 'PDFEmbed', 'PWA', 'PageAssessments', 'PageForms', 'PageImages', 'PageNotice', 'PageProperties', 'PageSchemas', 'PageTriage', 'PagedTiffHandler', 'ParserFunctions', 'PdfHandler', 'PipeEscape', 'PluggableAuth', 'Poem', 'PollNY', 'Popups', 'PortableInfobox', 'Preloader', [03:38:50] 'ProofreadPage', 'PropertySuggester', 'ProtectSite', 'ProtectionIndicator', 'Purge', 'QuickInstantCommons', 'Quiz', 'QuizGame', 'RSS', 'RandomGameUnit', 'RandomImage', 'RandomSelection', 'RatePage', 'RealMe', 'RegexFunctions', 'RelatedArticles', 'RemovePII', 'RemoveRedlinks', 'Renameuser', 'ReplaceText', 'Report', 'RevisionSlider', 'RightFunctions', [03:38:51] 'RottenLinks', 'SandboxLink', 'Score', 'Scribunto', 'SearchVue', 'SecurePoll', 'SemanticFormsSelect', 'SemanticMediaWiki', 'SemanticScribunto', 'Share', 'ShortDescription', 'SimpleBlogPage', 'SimpleChanges', 'SimpleSAMLphp', 'SimpleTooltip', 'SkinPerPage', 'SnapProjectEmbed', 'SocialProfile', 'SoftRedirector', 'SpamBlacklist', 'Spoilers', 'SpriteSheet', [03:38:52] 'StandardDialogs', 'StopForumSpam', 'StructuredNavigation', 'SubPageList3', 'SubpageFun', 'SyntaxHighlight_GeSHi', 'TabberNeue', 'Tabs', 'TemplateData', 'TemplateSandbox', 'TemplateStyles', 'TemplateStylesExtender', 'TemplateWizard', 'TextExtracts', 'Thanks', 'Theme', 'TimeMachine', 'TimedMediaHandler', 'Timeline', 'TitleBlacklist', 'TitleKey', 'TocTree', [03:38:53] 'TorBlock', 'Translate', 'TranslationNotifications', 'TreeAndMenu', 'TwitterTag', 'TwoColConflict', 'UniversalLanguageSelector', 'UploadWizard', 'UploadsLink', 'UrlGetParameters', 'UrlShortener', 'UserFunctions', 'UserMerge', 'UserPageEditProtection', 'VEForAll', 'Variables', 'VariablesLua', 'Video', 'VisualEditor', 'VoteNY', 'WSOAuth', 'WebAuthn', [03:38:54] 'WebChat', 'WikiCategoryTagCloud', 'WikiDiscover', 'WikiEditor', 'WikiForgeMagic', 'WikiForum', 'WikiLove', 'WikiSEO', 'WikiTextLoggedInOut', 'Wikibase', 'WikibaseCirrusSearch', 'WikibaseEdtf', 'WikibaseLexeme', 'WikibaseLexemeCirrusSearch', 'WikibaseLocalMedia', 'WikibaseMediaInfo', 'WikibaseQualityConstraints', 'WikidataPageBanner', 'WikimediaIncubator', [03:38:55] 'YouTube', 'cldr', 'mw-ScratchBlocks4', 'wikihiero'], 'upgrade_skins': ['Anisa', 'BlueSky', 'Citizen', 'CologneBlue', 'Cosmos', 'DuskToDawn', 'Femiwiki', 'Gamepress', 'HasSomeColours', 'Mask', 'Medik', 'Metrolook', 'MinervaNeue', 'Mirage', 'Modern', 'Monaco', 'MonoBook', 'Nimbus', 'Nostalgia', 'Pivot', 'Refreshed', 'Splash', 'Timeless', 'Truglass', [03:38:56] 'Tweeki', 'Vector', 'WMAU', 'apex', 'chameleon', 'erudite'], 'ignore_time': True} to [mw1, mw2, mw3, jobrunner1] - SUCCESS in 0s [03:38:57] !log [agent@jobrunner1] starting deploy of {'pull': 'world', 'upgrade_world': True, 'upgrade_vendor': True, 'world': True, 'l10n': True, 'extension_list': True, 'force': True, 'versions': ['1.40'], 'upgrade_extensions': ['3D', 'AJAXPoll', 'AWS', 'AbuseFilter', 'AddHTMLMetaAndTitle', 'AdminLinks', 'AdvancedSearch', 'AntiSpoof', 'ApprovedRevs', 'Arrays', [03:38:58] 'ArticleCreationWorkflow', 'ArticlePlaceholder', 'ArticleRatings', 'ArticleToCategory2', 'AuthorProtect', 'AutoCreateCategoryPages', 'AutoCreatePage', 'Babel', 'BetaFeatures', 'BlogPage', 'Bootstrap', 'CSS', 'Calendar', 'Capiunto', 'Cargo', 'CategoryExplorer', 'CategorySortHeaders', 'CategoryTree', 'CharInsert', 'CheckUser', 'CirrusSearch', 'Cite', [03:38:59] 'CiteThisPage', 'Citoid', 'CleanChanges', 'CodeEditor', 'CodeMirror', 'CollapsibleVector', 'Collection', 'CommentStreams', 'Commentbox', 'Comments', 'CommonsMetadata', 'ConfirmAccount', 'ConfirmEdit', 'ContactPage', 'ContributionScores', 'CookieWarning', 'CountDownClock', 'CreatePage', 'CreatePageUw', 'CreateRedirect', 'CreateWiki', 'CreatedPagesList', [03:39:00] 'DPLForum', 'DarkMode', 'DataDump', 'DataTransfer', 'DeleteUserPages', 'Description2', 'Disambiguator', 'DiscordNotifications', 'DiscussionTools', 'DismissableSiteNotice', 'DisplayTitle', 'DummyFandoomMainpageTags', 'DynamicPageList', 'DynamicPageList3', 'DynamicSidebar', 'Echo', 'EditNotify', 'EditSubpages', 'Editcount', 'Elastica', 'ElectronPdfService', [03:39:01] 'EmailAuthorization', 'EmbedSpotify', 'EmbedVideo', 'EventLogging', 'ExternalData', 'FileStorageMonitor', 'FlaggedRevs', 'FlexDiagrams', 'Flow', 'FontAwesome', 'ForcePreview', 'Gadgets', 'Genealogy', 'GeoCrumbs', 'GeoData', 'GeoGebra', 'GoogleDocs4MW', 'GoogleNewsSitemap', 'Graph', 'GroupsSidebar', 'GrowthExperiments', 'GuidedTour', 'HAWelcome', [03:39:02] 'HeaderTabs', 'HideSection', 'HighlightLinksInCategory', 'HitCounters', 'IPInfo', 'ImageMap', 'ImageRating', 'ImportDump', 'IncidentReporting', 'InputBox', 'Interwiki', 'InterwikiDispatcher', 'InterwikiSorting', 'JSBreadCrumbs', 'JavascriptSlideshow', 'Josa', 'JsCalendar', 'JsonConfig', 'Kartographer', 'LabeledSectionTransclusion', 'LanguageSelector', [03:39:03] 'LastModified', 'LdapAuthentication', 'Lingo', 'LinkSuggest', 'LinkTarget', 'LinkTitles', 'Linter', 'Listings', 'LockAuthor', 'LoginNotify', 'LogoFunctions', 'Loops', 'MagicNoCache', 'ManageWiki', 'Maps', 'MassEditRegex', 'MassMessage', 'Math', 'MatomoAnalytics', 'MediaSearch', 'MediaWikiChat', 'Mermaid', 'MintyDocs', 'MixedNamespaceSearchSuggestions', [03:39:04] 'MobileDetect', 'MobileFrontend', 'MobileTabsPlugin', 'Moderation', 'MsCalendar', 'MsCatSelect', 'MsLinks', 'MsUpload', 'MultiBoilerplate', 'MultimediaViewer', 'MyVariables', 'NamespacePreload', 'NativeSvgHandler', 'NearbyPages', 'NewSignupPage', 'NewUserMessage', 'NewUserNotif', 'NewestPages', 'Newsletter', 'NoTitle', 'Nuke', 'NukeDPL', 'OATHAuth', [03:39:05] 'OAuth', 'OOJSPlus', 'OpenGraphMeta', 'OpenIDConnect', 'OrphanedTalkPages', 'PDFEmbed', 'PWA', 'PageAssessments', 'PageForms', 'PageImages', 'PageNotice', 'PageProperties', 'PageSchemas', 'PageTriage', 'PagedTiffHandler', 'ParserFunctions', 'PdfHandler', 'PipeEscape', 'PluggableAuth', 'Poem', 'PollNY', 'Popups', 'PortableInfobox', 'Preloader', [03:39:06] 'ProofreadPage', 'PropertySuggester', 'ProtectSite', 'ProtectionIndicator', 'Purge', 'QuickInstantCommons', 'Quiz', 'QuizGame', 'RSS', 'RandomGameUnit', 'RandomImage', 'RandomSelection', 'RatePage', 'RealMe', 'RegexFunctions', 'RelatedArticles', 'RemovePII', 'RemoveRedlinks', 'Renameuser', 'ReplaceText', 'Report', 'RevisionSlider', 'RightFunctions', [03:39:07] 'RottenLinks', 'SandboxLink', 'Score', 'Scribunto', 'SearchVue', 'SecurePoll', 'SemanticFormsSelect', 'SemanticMediaWiki', 'SemanticScribunto', 'Share', 'ShortDescription', 'SimpleBlogPage', 'SimpleChanges', 'SimpleSAMLphp', 'SimpleTooltip', 'SkinPerPage', 'SnapProjectEmbed', 'SocialProfile', 'SoftRedirector', 'SpamBlacklist', 'Spoilers', 'SpriteSheet', [03:39:08] 'StandardDialogs', 'StopForumSpam', 'StructuredNavigation', 'SubPageList3', 'SubpageFun', 'SyntaxHighlight_GeSHi', 'TabberNeue', 'Tabs', 'TemplateData', 'TemplateSandbox', 'TemplateStyles', 'TemplateStylesExtender', 'TemplateWizard', 'TextExtracts', 'Thanks', 'Theme', 'TimeMachine', 'TimedMediaHandler', 'Timeline', 'TitleBlacklist', 'TitleKey', 'TocTree', [03:39:09] 'TorBlock', 'Translate', 'TranslationNotifications', 'TreeAndMenu', 'TwitterTag', 'TwoColConflict', 'UniversalLanguageSelector', 'UploadWizard', 'UploadsLink', 'UrlGetParameters', 'UrlShortener', 'UserFunctions', 'UserMerge', 'UserPageEditProtection', 'VEForAll', 'Variables', 'VariablesLua', 'Video', 'VisualEditor', 'VoteNY', 'WSOAuth', 'WebAuthn', [03:39:10] 'WebChat', 'WikiCategoryTagCloud', 'WikiDiscover', 'WikiEditor', 'WikiForgeMagic', 'WikiForum', 'WikiLove', 'WikiSEO', 'WikiTextLoggedInOut', 'Wikibase', 'WikibaseCirrusSearch', 'WikibaseEdtf', 'WikibaseLexeme', 'WikibaseLexemeCirrusSearch', 'WikibaseLocalMedia', 'WikibaseMediaInfo', 'WikibaseQualityConstraints', 'WikidataPageBanner', 'WikimediaIncubator', [03:39:11] 'YouTube', 'cldr', 'mw-ScratchBlocks4', 'wikihiero'], 'upgrade_skins': ['Anisa', 'BlueSky', 'Citizen', 'CologneBlue', 'Cosmos', 'DuskToDawn', 'Femiwiki', 'Gamepress', 'HasSomeColours', 'Mask', 'Medik', 'Metrolook', 'MinervaNeue', 'Mirage', 'Modern', 'Monaco', 'MonoBook', 'Nimbus', 'Nostalgia', 'Pivot', 'Refreshed', 'Splash', 'Timeless', 'Truglass', [03:39:12] 'Tweeki', 'Vector', 'WMAU', 'apex', 'chameleon', 'erudite'], 'ignore_time': True} to [mw1, mw2, mw3, jobrunner1] [03:39:19] @agentisai I was afraid of that lol [03:39:45] I knew it might happen [03:39:54] @cosmicalpha had to [03:39:55] lol [03:40:06] we need to focus more on WF [03:40:38] I was talking about the logging of every extension lol [03:40:54] yeah, I know [03:41:28] I need to fix the logging on mwdeploy lol but its for another day not concerned right now. But yes I agree we need to focus a bit more here... [03:43:43] !log [agent@jobrunner1] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [03:44:16] WikiForge/WikiForgeMagic - AgentIsai the build has errored. [03:47:37] !log [agent@jobrunner1] starting deploy of {'force': True, 'folders': '../mediawiki-staging/1.40/extensions/WikiForgeMagic', 'versions': ['1.40']} to [mw1, mw2, mw3, jobrunner1] [03:47:40] !log [agent@jobrunner1] finished deploy of {'force': True, 'folders': '../mediawiki-staging/1.40/extensions/WikiForgeMagic', 'versions': ['1.40']} to [mw1, mw2, mw3, jobrunner1] - SUCCESS in 2s [03:48:07] !log [agent@jobrunner1] starting deploy of {'force': True, 'folders': '../mediawiki/1.40/extensions/WikiForgeMagic', 'versions': ['1.40']} to [mw1, mw2, mw3, jobrunner1] [03:48:10] !log [agent@jobrunner1] finished deploy of {'force': True, 'folders': '../mediawiki/1.40/extensions/WikiForgeMagic', 'versions': ['1.40']} to [mw1, mw2, mw3, jobrunner1] - SUCCESS in 2s [03:50:53] PROBLEM - jobrunner1 MediaWiki Rendering on jobrunner1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 764 bytes in 0.171 second response time [03:51:16] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 12 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_checkout_MediaWiki-REL1_41 Femiwiki] [03:51:20] PROBLEM - jobrunner1 PowerDNS Recursor on jobrunner1 is CRITICAL: Domain 'wikiforge.net' was not found by the server [03:51:25] PROBLEM - jobrunner1 APT on jobrunner1 is CRITICAL: APT CRITICAL: 83 packages available for upgrade (19 critical updates). [03:52:43] [02WikiForge/mw-config] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/mw-config/compare/e9f16502f135...44db11740f59 [03:52:45] [02WikiForge/mw-config] 07AgentIsai 0344db117 - Remove onExtensionFunctions [03:53:04] !log [agent@jobrunner1] starting deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [jobrunner1] [03:53:05] !log [agent@jobrunner1] finished deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [jobrunner1] - SUCCESS in 0s [03:53:17] !log [agent@jobrunner1] starting deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [jobrunner1] [03:53:18] !log [agent@jobrunner1] finished deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [jobrunner1] - SUCCESS in 0s [03:53:38] WikiForge/mw-config - AgentIsai the build passed. [03:54:31] [02WikiForge/mw-config] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/mw-config/compare/44db11740f59...e3429221e99e [03:54:33] !log [agent@jobrunner1] starting deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [jobrunner1] [03:54:34] !log [agent@jobrunner1] finished deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [jobrunner1] - SUCCESS in 1s [03:54:34] [02WikiForge/mw-config] 07AgentIsai 03e342922 - Remove onExtensionFunctions [03:54:45] RECOVERY - jobrunner1 MediaWiki Rendering on jobrunner1 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.293 second response time [03:55:25] WikiForge/mw-config - AgentIsai the build passed. [04:20:09] [02WikiForge/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/WikiForge/puppet/compare/61343bd96b1a...0a4d13ee32a8 [04:20:12] [02WikiForge/puppet] 07AgentIsai 030a4d13e - Fix memcached IP [04:40:06] [02WikiForge/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±5] 13https://github.com/WikiForge/puppet/compare/0a4d13ee32a8...a8b7dca55671 [04:40:07] [02WikiForge/puppet] 07AgentIsai 03a8b7dca - Remove bind_ip for Swift [04:41:38] WikiForge/puppet - AgentIsai the build passed. [04:42:39] RECOVERY - swiftac1 Swift Container Service on swiftac1 is OK: TCP OK - 0.000 second response time on 10.0.0.120 port 6001 [04:42:41] RECOVERY - swiftac1 Swift Account Service on swiftac1 is OK: TCP OK - 0.000 second response time on 10.0.0.120 port 6002 [05:27:02] !log [agent@jobrunner1] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases-wikiforge.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/CreateWiki/maintenance/setContainersAccess.php (START) [05:28:36] !log [agent@jobrunner1] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases-wikiforge.json /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/extensions/CreateWiki/maintenance/setContainersAccess.php (END - exit=0) [05:30:53] !log [agent@jobrunner1] starting deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [mw1, mw2, mw3, jobrunner1] [05:30:56] !log [agent@jobrunner1] finished deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [mw1, mw2, mw3, jobrunner1] - SUCCESS in 3s [05:31:02] !log [agent@jobrunner1] starting deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [mw1, mw2, mw3, jobrunner1] [05:31:04] !log [agent@jobrunner1] finished deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [mw1, mw2, mw3, jobrunner1] - SUCCESS in 2s [05:31:29] !log [agent@jobrunner1] starting deploy of {'force': True, 'folders': '1.40/extensions/WikiForgeMagic', 'versions': ['1.40']} to [mw1, mw2, mw3, jobrunner1] [05:31:31] !log [agent@jobrunner1] finished deploy of {'force': True, 'folders': '1.40/extensions/WikiForgeMagic', 'versions': ['1.40']} to [mw1, mw2, mw3, jobrunner1] - SUCCESS in 2s [05:34:00] !log [swift@swiftobject1] Script: Finished S3 download of all folders [05:38:15] !log [swift@swiftobject1] Script: Starting upload [05:38:42] !log [swift@swiftobject1] Script: Starting upload [05:42:50] PROBLEM - swiftobject1 Current Load on swiftobject1 is CRITICAL: LOAD CRITICAL - total load average: 12.93, 7.18, 3.15 [05:45:04] [02WikiForge/dns] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/dns/compare/20067de78a9a...02641af77f26 [05:45:06] [02WikiForge/dns] 07AgentIsai 0302641af - Update records [05:45:31] [02WikiForge/dns] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/dns/compare/02641af77f26...f1a09c1f1536 [05:45:34] [02WikiForge/dns] 07AgentIsai 03f1a09c1 - Update records [05:46:26] [02WikiForge/dns] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/dns/compare/f1a09c1f1536...af4f86c5f9a3 [05:46:28] [02WikiForge/dns] 07AgentIsai 03af4f86c - Update records [05:49:05] [02WikiForge/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/puppet/compare/a8b7dca55671...c5e54d9f9e8f [05:49:08] [02WikiForge/puppet] 07AgentIsai 03c5e54d9 - mw3 -> mwdedi1 [05:50:20] WikiForge/puppet - AgentIsai the build has errored. [05:51:17] !log [agent@jobrunner1] starting deploy of {'force': True, 'folders': '1.40/extensions/WikiForgeMagic', 'versions': ['1.40']} to [mw1, mw2, mwdedi1, jobrunner1] [05:51:26] !log [agent@jobrunner1] finished deploy of {'force': True, 'folders': '1.40/extensions/WikiForgeMagic', 'versions': ['1.40']} to [mw1, mw2, mwdedi1, jobrunner1] - SUCCESS in 8s [05:53:14] !log [agent@jobrunner1] starting deploy of {'force': True, 'folders': '1.41/extensions/WikiForgeMagic', 'versions': ['1.40']} to [mw1, mw2, mwdedi1, jobrunner1] [05:53:16] !log [agent@jobrunner1] finished deploy of {'force': True, 'folders': '1.41/extensions/WikiForgeMagic', 'versions': ['1.40']} to [mw1, mw2, mwdedi1, jobrunner1] - SUCCESS in 2s [06:33:54] !log [swift@swiftobject1] Script: Finished uploading images to Swift [06:34:17] was that so hard? [06:42:50] PROBLEM - swiftobject1 Current Load on swiftobject1 is WARNING: LOAD WARNING - total load average: 1.11, 2.67, 7.51 [06:46:50] RECOVERY - swiftobject1 Current Load on swiftobject1 is OK: LOAD OK - total load average: 0.73, 1.80, 6.06 [07:34:55] PROBLEM - www.theharrypotter.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.40175D51F8140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1586:SSL alert number 40 [07:48:42] PROBLEM - www.harrypotter.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.40D7C30727150000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1586:SSL alert number 40 [07:54:17] PROBLEM - theharrypotter.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.406735A83A150000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1586:SSL alert number 40 [08:05:05] PROBLEM - harrypotter.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.407701BEC6140000:error:0A000410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../ssl/record/rec_layer_s3.c:1586:SSL alert number 40 [08:32:52] PROBLEM - dc.wikiforge.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:35:03] PROBLEM - www.director.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:35:59] PROBLEM - meta.wikiforge.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:37:18] PROBLEM - www.wikiforge.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:51:28] PROBLEM - dcmultiverse.wikiforge.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:52:22] PROBLEM - rippaverse.wikiforge.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:54:57] PROBLEM - avid.wikiforge.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:05:20] PROBLEM - beaconspace.unrestrictedlorefare.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:22:52] PROBLEM - avid.wiki - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [09:22:57] PROBLEM - www.dcmultiversewiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:25:32] PROBLEM - dcmultiversewiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:06:44] PROBLEM - director.wikiforge.net - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:35:24] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:03:24] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:50:50] PROBLEM - mon1 Puppet on mon1 is CRITICAL: CRITICAL: Puppet has 3 failures. Last run 3 minutes ago with 3 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/puppetlabs.gpg],File[/usr/local/bin/puppet-enabled],File[/etc/rsyslog.conf] [12:54:59] PROBLEM - mail1 Puppet on mail1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [13:18:50] RECOVERY - mon1 Puppet on mon1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:22:59] RECOVERY - mail1 Puppet on mail1 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [14:20:50] PROBLEM - mon1 Puppet on mon1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:46:50] RECOVERY - mon1 Puppet on mon1 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [17:34:46] [02puppet] 07Universal-Omega closed pull request 03#219: Use 127.0.0.1 - 13https://github.com/WikiForge/puppet/pull/219 [17:34:49] [02WikiForge/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/puppet/compare/c5e54d9f9e8f...4ca123cda2d3 [17:34:51] [02WikiForge/puppet] 07Universal-Omega 034ca123c - Use 127.0.0.1 [17:34:52] [02WikiForge/puppet] 07Universal-Omega deleted branch 03127.0.0.1 [17:34:53] [02puppet] 07Universal-Omega deleted branch 03127.0.0.1 - 13https://github.com/WikiForge/puppet [17:40:12] [02WikiForge/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/puppet/compare/4ca123cda2d3...073be9c677d6 [17:40:13] [02WikiForge/puppet] 07Universal-Omega 03073be9c - Update [17:48:56] RECOVERY - puppet1 PowerDNS Recursor on puppet1 is OK: DNS OK: 0.113 seconds response time. wikiforge.net returns 2001:41d0:801:2000::2200,51.68.214.246 [17:54:12] RECOVERY - mail1 PowerDNS Recursor on mail1 is OK: DNS OK: 0.173 seconds response time. wikiforge.net returns 158.69.62.132,2001:41d0:801:2000::2200 [17:55:24] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:56:48] RECOVERY - mw1 PowerDNS Recursor on mw1 is OK: DNS OK: 0.170 seconds response time. wikiforge.net returns 2607:5300:201:3100::175d,51.68.214.246 [17:58:09] RECOVERY - bast1 PowerDNS Recursor on bast1 is OK: DNS OK: 0.106 seconds response time. wikiforge.net returns 2001:41d0:801:2000::2200,51.68.214.246 [18:00:56] RECOVERY - swiftproxy1 PowerDNS Recursor on swiftproxy1 is OK: DNS OK: 0.169 seconds response time. wikiforge.net returns 2607:5300:201:3100::175d,51.68.214.246 [18:01:24] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:02:28] RECOVERY - swiftobject1 PowerDNS Recursor on swiftobject1 is OK: DNS OK: 0.182 seconds response time. wikiforge.net returns 158.69.62.132,2001:41d0:801:2000::2200 [18:06:25] RECOVERY - swiftac1 PowerDNS Recursor on swiftac1 is OK: DNS OK: 0.173 seconds response time. wikiforge.net returns 158.69.62.132,2001:41d0:801:2000::2200 [18:08:24] RECOVERY - jobrunner1 PowerDNS Recursor on jobrunner1 is OK: DNS OK: 0.101 seconds response time. wikiforge.net returns 2001:41d0:801:2000::2200,51.68.214.246 [18:08:52] RECOVERY - mw2 PowerDNS Recursor on mw2 is OK: DNS OK: 0.105 seconds response time. wikiforge.net returns 2001:41d0:801:2000::2200,51.68.214.246 [18:18:02] RECOVERY - mon1 PowerDNS Recursor on mon1 is OK: DNS OK: 0.107 seconds response time. wikiforge.net returns 2001:41d0:801:2000::2200,51.68.214.246 [18:19:43] RECOVERY - rippaverse.wikiforge.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wikiforge.net' will expire on Tue 12 Mar 2024 04:50:22 AM GMT +0000. [18:22:02] RECOVERY - avid.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'avid.wiki' will expire on Tue 26 Mar 2024 03:15:43 AM GMT +0000. [18:22:02] RECOVERY - theharrypotter.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'theharrypotter.wiki' will expire on Fri 22 Mar 2024 04:41:00 PM GMT +0000. [18:22:31] RECOVERY - avid.wikiforge.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wikiforge.net' will expire on Tue 12 Mar 2024 04:50:22 AM GMT +0000. [18:22:49] RECOVERY - dcmultiverse.wikiforge.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wikiforge.net' will expire on Tue 12 Mar 2024 04:50:22 AM GMT +0000. [18:25:50] RECOVERY - harrypotter.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'harrypotter.wiki' will expire on Fri 22 Mar 2024 04:41:29 PM GMT +0000. [18:30:42] RECOVERY - dc.wikiforge.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wikiforge.net' will expire on Tue 12 Mar 2024 04:50:22 AM GMT +0000. [18:31:46] RECOVERY - meta.wikiforge.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wikiforge.net' will expire on Tue 12 Mar 2024 04:50:22 AM GMT +0000. [18:32:07] RECOVERY - director.wikiforge.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wikiforge.net' will expire on Tue 12 Mar 2024 04:50:22 AM GMT +0000. [18:32:39] RECOVERY - www.theharrypotter.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'theharrypotter.wiki' will expire on Fri 22 Mar 2024 04:41:00 PM GMT +0000. [18:34:47] RECOVERY - www.dcmultiversewiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'dcmultiversewiki.com' will expire on Thu 21 Mar 2024 09:01:13 PM GMT +0000. [18:34:52] RECOVERY - www.wikiforge.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wikiforge.net' will expire on Tue 12 Mar 2024 04:50:22 AM GMT +0000. [18:41:08] [02WikiForge/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/dns/compare/af4f86c5f9a3...27c44ba19dbb [18:41:09] [02WikiForge/dns] 07Universal-Omega 0327c44ba - Fix ns IPs [18:43:49] RECOVERY - www.harrypotter.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'harrypotter.wiki' will expire on Fri 22 Mar 2024 04:41:29 PM GMT +0000. [18:43:52] RECOVERY - dcmultiversewiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'dcmultiversewiki.com' will expire on Thu 21 Mar 2024 09:01:13 PM GMT +0000. [19:36:57] RECOVERY - cloud1 PowerDNS Recursor on cloud1 is OK: DNS OK: 0.098 seconds response time. wikiforge.net returns 2001:41d0:801:2000::2200,51.68.214.246 [20:02:06] [02WikiForge/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/dns/compare/27c44ba19dbb...a7d7f6688526 [20:02:07] [02WikiForge/dns] 07Universal-Omega 03a7d7f66 - Add acme challenges [20:08:33] [02WikiForge/ssl] 07WikiForgeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/ssl/compare/6068f0f3bb24...2e2a333a949a [20:08:36] [02WikiForge/ssl] 07WikiForgeSSLBot 032e2a333 - Bot: Update SSL cert for wikiforge.net [20:09:25] WikiForge/ssl - WikiForgeSSLBot the build has errored. [20:10:13] [02WikiForge/ssl] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/ssl/compare/2e2a333a949a...07911627598e [20:10:16] [02WikiForge/ssl] 07Universal-Omega 030791162 - Fix [20:15:34] [02WikiForge/ssl] 07WikiForgeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/ssl/compare/07911627598e...92f31bb18aef [20:15:37] [02WikiForge/ssl] 07WikiForgeSSLBot 0392f31bb - Bot: Update SSL cert for wikiforge.net [20:17:43] [02WikiForge/ssl] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/ssl/compare/92f31bb18aef...8466c1640da3 [20:17:45] [02WikiForge/ssl] 07Universal-Omega 038466c16 - Use wikiforge.net [20:37:30] [02WikiForge/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±5] 13https://github.com/WikiForge/puppet/compare/073be9c677d6...b655cf97a0e1 [20:37:33] [02WikiForge/puppet] 07Universal-Omega 03b655cf9 - Use static.wikiforge.net [20:39:23] [02WikiForge/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/puppet/compare/b655cf97a0e1...e59006d660de [20:39:23] WikiForge/puppet - Universal-Omega the build has errored. [20:39:26] [02WikiForge/puppet] 07Universal-Omega 03e59006d - Fix cert [20:50:31] PROBLEM - wikiforge.net - pointing to WikiForge on sslhost is CRITICAL: DNS CRITICAL - expected 'mw-lb.wikiforge.net.' but got '158.69.62.132,2607:5300:201:3100::175d' [21:22:56] [02WikiForge/ssl] 07WikiForgeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/ssl/compare/8466c1640da3...1f67083a301e [21:22:59] [02WikiForge/ssl] 07WikiForgeSSLBot 031f67083 - Bot: Update SSL cert for beaconspace.unrestrictedlorefare.com [21:57:08] RECOVERY - beaconspace.unrestrictedlorefare.com - LetsEncrypt on sslhost is OK: OK - Certificate 'beaconspace.unrestrictedlorefare.com' will expire on Wed 10 Apr 2024 08:22:51 PM GMT +0000.