[07:50:57] !log Cleared Jenkins workspaces on integration-agent-docker-1034 [07:50:59] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:00:42] (03CR) 10Hashar: [C: 03+2] jjb: update Quibble jobs to 1.5.3 [integration/config] - 10https://gerrit.wikimedia.org/r/898807 (https://phabricator.wikimedia.org/T331061) (owner: 10Hashar) [08:01:16] !log Switching Jenkins jobs to Quibble 1.5.3 | https://gerrit.wikimedia.org/r/c/integration/config/+/898807 | T331061, T331621 [08:01:20] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:01:21] T331061: Capture output from failed command and transmit to earlywarningbot - https://phabricator.wikimedia.org/T331061 [08:01:21] T331621: CI: Don't show PHPUnit progress - https://phabricator.wikimedia.org/T331621 [08:02:06] (03Merged) 10jenkins-bot: jjb: update Quibble jobs to 1.5.3 [integration/config] - 10https://gerrit.wikimedia.org/r/898807 (https://phabricator.wikimedia.org/T331061) (owner: 10Hashar) [08:09:13] !log Refreshing all Jenkins jobs to catch up with various changes made in the last few days [08:09:15] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:10:55] ty [08:13:25] 10GitLab, 10serviceops-collab: GitLab restore failed on gitlab1003 - https://phabricator.wikimedia.org/T332114 (10Jelto) [08:13:47] 10GitLab, 10serviceops-collab: GitLab restore failed on gitlab1003 - https://phabricator.wikimedia.org/T332114 (10Jelto) p:05Triageβ†’03Medium [08:33:41] (03CR) 10Hashar: [C: 03+2] Link Groovy doc of PipelineLib [integration/docroot] - 10https://gerrit.wikimedia.org/r/898802 (https://phabricator.wikimedia.org/T222199) (owner: 10Hashar) [08:34:24] (03Merged) 10jenkins-bot: Link Groovy doc of PipelineLib [integration/docroot] - 10https://gerrit.wikimedia.org/r/898802 (https://phabricator.wikimedia.org/T222199) (owner: 10Hashar) [09:00:54] 10GitLab (Infrastructure), 10serviceops-collab: GitLab restore failed on gitlab1003 - https://phabricator.wikimedia.org/T332114 (10Jelto) 05Openβ†’03Resolved The restore failed with a deadlock in the postgres DB restore: ` Mar 15 02:03:46 gitlab1003 gitlab-restore.sh[4113714]: Caused by: Mar 15 02:03:46 git... [09:01:42] !log (deployment-prep) `dwalden@deployment-mwmaint02:~$ mwscript maintenance/refreshLinks.php --wiki enwiki --category='Pages that use Phonos'` for T327708 [09:01:45] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:01:46] T327708: File usage by Phonos should be registered and tracked - https://phabricator.wikimedia.org/T327708 [09:01:59] (03PS1) 10Hashar: dockerfiles: upgrade castor Stretch Bullseye [integration/config] - 10https://gerrit.wikimedia.org/r/899505 (https://phabricator.wikimedia.org/T278203) [09:02:56] (03PS1) 10Hashar: jjb: switch jobs to releng/castor:0.3.0 [integration/config] - 10https://gerrit.wikimedia.org/r/899508 (https://phabricator.wikimedia.org/T278203) [09:04:24] 10Continuous-Integration-Infrastructure, 10Patch-For-Review: Migrate all CI jobs from stretch to buster or later and drop stretch testing support - https://phabricator.wikimedia.org/T278203 (10hashar) [09:11:53] (03PS1) 10Zoranzoki21: Zuul: Archive the CharRangeSpan extension [integration/config] - 10https://gerrit.wikimedia.org/r/899514 (https://phabricator.wikimedia.org/T299258) [09:18:29] 10Continuous-Integration-Infrastructure, 10Quibble, 10Developer Productivity, 10MW-1.40-notes (1.40.0-wmf.26; 2023-03-06): Provide early feedback when a patch has job failures - https://phabricator.wikimedia.org/T323750 (10kostajh) [09:20:14] 10Continuous-Integration-Config, 10Wikimedia-Site-requests: Migrate CI job operations-mw-config-typos-docker job to be inside operations/mediawiki-config - https://phabricator.wikimedia.org/T332121 (10hashar) [09:40:40] (03PS1) 10Hashar: Remove releng/typos image and related job [integration/config] - 10https://gerrit.wikimedia.org/r/899523 (https://phabricator.wikimedia.org/T332121) [09:41:33] 10Continuous-Integration-Infrastructure, 10Patch-For-Review: Migrate all CI jobs from stretch to buster or later and drop stretch testing support - https://phabricator.wikimedia.org/T278203 (10hashar) [09:42:31] (03CR) 10Hashar: [C: 03+2] Zuul: Archive the CharRangeSpan extension [integration/config] - 10https://gerrit.wikimedia.org/r/899514 (https://phabricator.wikimedia.org/T299258) (owner: 10Zoranzoki21) [09:43:41] (03Merged) 10jenkins-bot: Zuul: Archive the CharRangeSpan extension [integration/config] - 10https://gerrit.wikimedia.org/r/899514 (https://phabricator.wikimedia.org/T299258) (owner: 10Zoranzoki21) [10:11:43] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10serviceops-collab, 10Datacenter-Switchover: switch releases.wikimedia.org from eqiad to codfw - https://phabricator.wikimedia.org/T330960 (10hashar) >>! In T330960#8693803, @LSobanski wrote: > .. > I think that makes sense as long as we... [10:18:41] 10Gerrit, 10Upstream: OpenSSH 9 scp does not work with Gerrit due to lack of sftp subsystem - https://phabricator.wikimedia.org/T330740 (10hashar) Looking at upstream repo https://gerrit.googlesource.com/gerrit and the `stable-3.5` branch, the download-commands plugin has been made to always track master and g... [10:38:48] 10Continuous-Integration-Infrastructure, 10Patch-For-Review: Migrate all CI jobs from stretch to buster or later and drop stretch testing support - https://phabricator.wikimedia.org/T278203 (10hashar) [11:09:05] 10GitLab (Auth & Access), 10Release-Engineering-Team, 10Wikimedia-GitHub, 10Security: Support Github login in Gitlab - https://phabricator.wikimedia.org/T325235 (10hashar) We had discussions for Phabricator about supporting GitHub has a third party authentication: * {T16} * {T200} * {T169007} We had one a... [11:13:34] (03PS1) 10Zoranzoki21: Zuul: Archive the BreadCrumbs extension [integration/config] - 10https://gerrit.wikimedia.org/r/899547 (https://phabricator.wikimedia.org/T299261) [11:16:38] 10GitLab (Integrations), 10Release-Engineering-Team: Import Merge Requests from GitHub read-only mirrored repos into GitLab - https://phabricator.wikimedia.org/T325106 (10hashar) When we migrated from Subversion to Git and Gerrit, we had a discussion about auto importing GitHub pull requests: {T37497}. The pus... [11:17:25] (03CR) 10Hashar: [C: 03+2] Zuul: Archive the BreadCrumbs extension [integration/config] - 10https://gerrit.wikimedia.org/r/899547 (https://phabricator.wikimedia.org/T299261) (owner: 10Zoranzoki21) [11:18:36] (03Merged) 10jenkins-bot: Zuul: Archive the BreadCrumbs extension [integration/config] - 10https://gerrit.wikimedia.org/r/899547 (https://phabricator.wikimedia.org/T299261) (owner: 10Zoranzoki21) [12:02:04] 10Release-Engineering-Team, 10mwcli, 10serviceops, 10serviceops-collab: Create /nonexistent directory for nobody user in golang images - https://phabricator.wikimedia.org/T331209 (10Clement_Goubert) Tagging #release-engineering-team and #serviceops-collab teams because it involves CI and gitlab. [12:04:58] 10GitLab (Infrastructure), 10serviceops-collab, 10Sustainability (Incident Followup): Add safeguard flag to gitlab-restore.sh script - https://phabricator.wikimedia.org/T331295 (10Jelto) [12:09:55] 10Release-Engineering-Team, 10mwcli, 10serviceops, 10serviceops-collab: Create /nonexistent directory for nobody user in golang images - https://phabricator.wikimedia.org/T331209 (10Clement_Goubert) p:05Triageβ†’03Medium [12:10:00] 10Release-Engineering-Team, 10mwcli, 10serviceops-collab, 10serviceops-radar: Create /nonexistent directory for nobody user in golang images - https://phabricator.wikimedia.org/T331209 (10Clement_Goubert) [13:13:47] (03PS1) 10Jforrester: Dockerfiles: [php74] Bump to latest PHP 7.4 and cascade [integration/config] - 10https://gerrit.wikimedia.org/r/899621 [13:43:57] 10Continuous-Integration-Infrastructure, 10Quibble, 10Developer Productivity: Capture output from failed command and transmit to earlywarningbot - https://phabricator.wikimedia.org/T331061 (10kostajh) 05Openβ†’03Resolved [13:44:03] 10Continuous-Integration-Infrastructure, 10Quibble, 10Developer Productivity, 10MW-1.40-notes (1.40.0-wmf.26; 2023-03-06): Provide early feedback when a patch has job failures - https://phabricator.wikimedia.org/T323750 (10kostajh) [13:44:15] 10Continuous-Integration-Infrastructure, 10Quibble, 10Developer Productivity, 10MW-1.40-notes (1.40.0-wmf.26; 2023-03-06): Provide early feedback when a patch has job failures - https://phabricator.wikimedia.org/T323750 (10kostajh) [13:48:10] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Yak Shaving πŸƒπŸͺ’), 10SRE, 10Patch-For-Review: Have linters/tests results show up as comments in files on gerrit - https://phabricator.wikimedia.org/T209149 (10kostajh) >>! In T209149#7335003, @Tgr wrote: >>>! In T209149#7333699, @kostajh wr... [14:19:13] 10Continuous-Integration-Infrastructure, 10Quibble, 10Developer Productivity: Run Phan jobs via Quibble - https://phabricator.wikimedia.org/T332178 (10kostajh) [14:20:04] 10Continuous-Integration-Infrastructure, 10Quibble, 10Developer Productivity: Run MediaWiki core/skin/extension mwgate-node16 jobs via Quibble - https://phabricator.wikimedia.org/T332179 (10kostajh) [14:20:32] 10Continuous-Integration-Infrastructure, 10Quibble, 10Developer Productivity: Run MediaWiki core/skin/extension Phan jobs via Quibble - https://phabricator.wikimedia.org/T332178 (10kostajh) [14:25:42] 10Continuous-Integration-Infrastructure, 10Quibble, 10Developer Productivity: Run MediaWiki core/skin/extension Phan jobs via Quibble - https://phabricator.wikimedia.org/T332178 (10Jdforrester-WMF) Is this just {T280990}? [15:16:16] 10Continuous-Integration-Infrastructure, 10Quibble, 10Developer Productivity: Run MediaWiki core/skin/extension Phan jobs via Quibble - https://phabricator.wikimedia.org/T332178 (10kostajh) >>! In T332178#8698283, @Jdforrester-WMF wrote: > Is this just {T280990}? Maybe; that would be one way to solve it but... [15:21:37] 10Scap, 10MW-on-K8s, 10serviceops: Add a flag to scap to force updating the /etc/helmfile-defaults/mediawiki/release/* files - https://phabricator.wikimedia.org/T332187 (10Clement_Goubert) [15:22:06] 10Scap, 10MW-on-K8s, 10serviceops: Add a flag to scap to force updating the /etc/helmfile-defaults/mediawiki/release/* files - https://phabricator.wikimedia.org/T332187 (10Clement_Goubert) p:05Triageβ†’03High [16:44:22] (03CR) 10Hashar: "I have rebuild the quibble images this morning so maybe they already have the latest?" [integration/config] - 10https://gerrit.wikimedia.org/r/899621 (owner: 10Jforrester) [16:47:38] (03CR) 10Hashar: "I have replied too fast." [integration/config] - 10https://gerrit.wikimedia.org/r/899621 (owner: 10Jforrester) [18:17:25] 10Beta-Cluster-Infrastructure: UpdateImplementationsJob not being run on beta cluster - https://phabricator.wikimedia.org/T332207 (10DMartin-WMF) [18:32:17] (03PS2) 10Jforrester: Dockerfiles: [php74] Bump to latest PHP 7.4 and cascade [integration/config] - 10https://gerrit.wikimedia.org/r/899621 [18:38:07] (03CR) 10Jforrester: Dockerfiles: [php74] Bump to latest PHP 7.4 and cascade (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/899621 (owner: 10Jforrester) [18:42:09] 10Beta-Cluster-Infrastructure: deployment-docker-changeprop01: `worker died, restarting` - https://phabricator.wikimedia.org/T332211 (10TheresNoTime) [18:47:42] (03CR) 10Jforrester: [C: 03+2] Dockerfiles: [php74] Bump to latest PHP 7.4 and cascade [integration/config] - 10https://gerrit.wikimedia.org/r/899621 (owner: 10Jforrester) [18:51:30] (03Merged) 10jenkins-bot: Dockerfiles: [php74] Bump to latest PHP 7.4 and cascade [integration/config] - 10https://gerrit.wikimedia.org/r/899621 (owner: 10Jforrester) [19:03:46] (03CR) 10Hashar: "Wonderful :)" [integration/config] - 10https://gerrit.wikimedia.org/r/899621 (owner: 10Jforrester) [19:18:46] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10MW-1.40-notes (1.40.0-wmf.27; 2023-03-13), 10Patch-For-Review, 10Release, and 2 others: 1.40.0-wmf.27 deployment blockers - https://phabricator.wikimedia.org/T330205 (10colewhite) Logstash is experiencing a high rate of `Wikimedia\Rdbms\Platform\SQLPlatfor... [19:22:01] 10Deployments, 10Phabricator, 10Release-Engineering-Team, 10User-brennen: Phabricator deployment 2023-03-15 - https://phabricator.wikimedia.org/T331915 (10brennen) [19:26:32] 10Beta-Cluster-Infrastructure: UpdateImplementationsJob not being run on beta cluster - https://phabricator.wikimedia.org/T332207 (10DMartin-WMF) [19:49:57] 10MediaWiki-Releasing, 10MW-1.40-release, 10Release: Release MW 1.40.0 - https://phabricator.wikimedia.org/T329075 (10taavi) [19:50:40] 10MediaWiki-Releasing, 10MW-1.40-release, 10Patch-For-Review: Add REL1_40 to ExtensionDistributor as the development snapshot - https://phabricator.wikimedia.org/T329085 (10taavi) 05Openβ†’03Resolved a:03taavi [19:52:44] (03PS1) 10Jforrester: jjb: Update php74 jobs to 7.4.33-1+0~20221108.73+debian10~1.gbpa00350a+wmf10u2 [integration/config] - 10https://gerrit.wikimedia.org/r/899764 [19:56:55] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/899764 (owner: 10Jforrester) [19:58:03] (03Merged) 10jenkins-bot: jjb: Update php74 jobs to 7.4.33-1+0~20221108.73+debian10~1.gbpa00350a+wmf10u2 [integration/config] - 10https://gerrit.wikimedia.org/r/899764 (owner: 10Jforrester) [19:59:19] lovely thanks James_F ! [20:53:34] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10MW-1.40-notes (1.40.0-wmf.27; 2023-03-13), 10Patch-For-Review, 10Release, and 2 others: 1.40.0-wmf.27 deployment blockers - https://phabricator.wikimedia.org/T330205 (10thcipriani) I will be out of office Thu, 16 Mar–Tue, 21 Mar For 🚨 [[ https://wikitech.... [20:54:52] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Release, 10Train Deployments: 1.41.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T330207 (10thcipriani) I will be out of office Thu, 16 Mar–Tue, 21 Mar For 🚨 [[ https://wikitech.wikimedia.org/wiki/Deployments/Emergencies#step-by-step | Eme... [21:08:31] 10Beta-Cluster-Infrastructure: deployment-docker-changeprop01: `worker died, restarting` - https://phabricator.wikimedia.org/T332211 (10TheresNoTime) 05Openβ†’03Resolved a:03TheresNoTime Following https://wikitech.wikimedia.org/wiki/Changeprop#To_deployment-prep (thanks @taavi) and getting the config files u... [21:10:41] 10Beta-Cluster-Infrastructure: UpdateImplementationsJob not being run on beta cluster - https://phabricator.wikimedia.org/T332207 (10TheresNoTime) @DMartin-WMF this task (and my T325786) prompted me to go digging, find {T332211} and after much hair-pulling, fix some config files β€” I now see `MediaWiki\Extension\... [21:12:44] 10Beta-Cluster-Infrastructure, 10ChangeProp: changeprop-jobqueue@deployment-prep fails with: getaddrinfo ENOTFOUND cloudmetrics1002.eqiad.wmnet - https://phabricator.wikimedia.org/T326192 (10TheresNoTime) Ah heck, I missed this task and created {T332211} as a dupe β€” issue //seems// to be resolved though! [21:14:46] 10Beta-Cluster-Infrastructure: htmlCacheUpdate job not being triggered on beta cluster - https://phabricator.wikimedia.org/T325786 (10TheresNoTime) 05Openβ†’03Resolved a:03TheresNoTime Resolved per {T332211} [21:19:48] 10Beta-Cluster-Infrastructure, 10Wikimedia-production-error: Could not enqueue jobs for stream mediawiki.job.cirrusSearchCheckerJob - https://phabricator.wikimedia.org/T325594 (10TheresNoTime) 05Openβ†’03Resolved a:03TheresNoTime Going to boldly assume, given this is JobQueue related, that it's allllll dow... [21:23:35] 10Beta-Cluster-Infrastructure: UpdateImplementationsJob not being run on beta cluster - https://phabricator.wikimedia.org/T332207 (10Peachey88) [21:33:46] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10MW-1.40-notes (1.40.0-wmf.27; 2023-03-13), 10Patch-For-Review, 10Release, and 2 others: 1.40.0-wmf.27 deployment blockers - https://phabricator.wikimedia.org/T330205 (10jeena) [21:35:40] 10Phabricator, 10Release-Engineering-Team (Radar), 10Patch-For-Review, 10User-brennen: Show "other assignee" avatar on tasks in workboard - https://phabricator.wikimedia.org/T329974 (10brennen) 05Openβ†’03Resolved a:03brennen This should work now. I was able to see an example on the #release-engineeri... [21:37:07] 10GitLab (Integrations), 10GitLab-Test, 10Phabricator, 10Release-Engineering-Team (Priority Backlog πŸ“₯), and 2 others: Experiment with GitLab-Phabricator integration - https://phabricator.wikimedia.org/T265617 (10brennen) [21:37:15] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team (GitLab V: Event Horizon πŸŒ„), 10Patch-For-Review, 10User-brennen: Build a widget to display GitLab changes on related Phabricator tasks - https://phabricator.wikimedia.org/T324149 (10brennen) 05In progressβ†’03Resolved Deployed. [21:39:20] 10Phabricator, 10good first task: Replace redirect links in Phabricator footer with direct links - https://phabricator.wikimedia.org/T331694 (10brennen) 05Stalledβ†’03Resolved a:03brennen Resolved with {T331915}. [21:52:05] 10Phabricator: Restrict maximum file size for Phabricator file uploads - https://phabricator.wikimedia.org/T151544 (10brennen) [21:52:32] 10Phabricator, 10Release-Engineering-Team, 10Patch-For-Review, 10Upstream: Unclear error message when uploading a larger attachment: "Exception: No configured storage engine can store this file." - https://phabricator.wikimedia.org/T155130 (10brennen) 05Stalledβ†’03Resolved a:03brennen New error messag... [22:00:18] 10Deployments, 10Phabricator, 10Release-Engineering-Team, 10User-brennen: Phabricator deployment 2023-03-15 - https://phabricator.wikimedia.org/T331915 (10brennen) 05In progressβ†’03Resolved Also worked T155130 in here. [22:14:16] 10Phabricator: Milestone workboard view no longer visible - https://phabricator.wikimedia.org/T332234 (10VirginiaPoundstone) [22:16:43] 10Phabricator: Milestone workboard view no longer visible - https://phabricator.wikimedia.org/T332234 (10RhinosF1) p:05Triageβ†’03Unbreak! Fallout of recent deployment [22:18:26] 10Phabricator: Milestone workboard view no longer visible - https://phabricator.wikimedia.org/T332234 (10RhinosF1) This is being investigated. Deployers are aware. [22:18:47] 10Phabricator: Milestone workboard view no longer visible - https://phabricator.wikimedia.org/T332234 (10brennen) > Fallout of recent deployment Yep, investigating. Will roll out a revert shortly if there's not an obvious immediate fix. [22:20:10] 10Phabricator: Milestone workboard view no longer visible - https://phabricator.wikimedia.org/T332234 (10Dzahn) related ticket that introduced "other assignee" field: T329974 [22:22:09] 10Phabricator, 10Release-Engineering-Team (Radar), 10Patch-For-Review, 10User-brennen: Show "other assignee" avatar on tasks in workboard - https://phabricator.wikimedia.org/T329974 (10Dzahn) While this looked fine for us on the dashboard linked above, we are getting reports of issues in another workboard... [22:28:18] 10Phabricator, 10Release-Engineering-Team (Radar), 10Patch-For-Review, 10User-brennen: Show "other assignee" avatar on tasks in workboard - https://phabricator.wikimedia.org/T329974 (10Dzahn) 05Resolvedβ†’03Open revert in progress [22:29:45] 10Phabricator: Milestone workboard view no longer visible - https://phabricator.wikimedia.org/T332234 (10Dzahn) ` AH01071: Got error 'PHP message: [2023-03-15 22:28:58] EXCEPTION: (RuntimeException) Undefined variable: other_assignees at [/src/error/PhutilErrorHandler.php:235]PHP message: arcanist(),... [22:31:31] 10Phabricator: Milestone workboard view no longer visible - https://phabricator.wikimedia.org/T332234 (10Dzahn) @VirginiaPoundstone Workboard is back for me. [22:33:45] 10Phabricator: Milestone workboard view no longer visible - https://phabricator.wikimedia.org/T332234 (10brennen) a:03brennen Looking at code, should be a simple fix. Reverted in production in the meanwhile. [22:33:54] 10Phabricator: Milestone workboard view no longer visible - https://phabricator.wikimedia.org/T332234 (10brennen) p:05Unbreak!β†’03Medium [22:53:45] 10Phabricator: Milestone workboard view no longer visible - https://phabricator.wikimedia.org/T332234 (10brennen) [22:53:53] 10Phabricator, 10Release-Engineering-Team (Radar), 10Patch-For-Review, 10User-brennen: Show "other assignee" avatar on tasks in workboard - https://phabricator.wikimedia.org/T329974 (10brennen) [22:57:21] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10MW-1.40-notes (1.40.0-wmf.27; 2023-03-13), 10Patch-For-Review, 10Release, and 2 others: 1.40.0-wmf.27 deployment blockers - https://phabricator.wikimedia.org/T330205 (10brennen) End-of-workday status: Not rolling back for T321160, but holding at group1. [23:34:25] 10Beta-Cluster-Infrastructure, 10ChangeProp: changeprop-jobqueue@deployment-prep fails with: getaddrinfo ENOTFOUND cloudmetrics1002.eqiad.wmnet - https://phabricator.wikimedia.org/T326192 (10Jdforrester-WMF) >>! In T326192#8700023, @TheresNoTime wrote: > Ah heck, I missed this task and created {T332211} as a d... [23:34:43] 10Beta-Cluster-Infrastructure: deployment-docker-changeprop01: `worker died, restarting` - https://phabricator.wikimedia.org/T332211 (10Jdforrester-WMF) [23:34:52] 10Beta-Cluster-Infrastructure, 10ChangeProp: changeprop-jobqueue@deployment-prep fails with: getaddrinfo ENOTFOUND cloudmetrics1002.eqiad.wmnet - https://phabricator.wikimedia.org/T326192 (10Jdforrester-WMF)