[07:41:48] 10GitLab (Infrastructure), 10serviceops-collab: Let's Encrypt certificate expiration notice for domain gitlab.devtools.wmcloud.org - https://phabricator.wikimedia.org/T335161 (10Jelto) Some more context on the failed let's encrypt renew: Apr 4: In https://gerrit.wikimedia.org/r/c/operations/puppet/+/904843 th... [08:45:32] maintenance-disconnect-full-disks build 485537 integration-agent-docker-1025 (/: 28%, /srv: 96%, /var/lib/docker: 43%): OFFLINE due to disk space [08:55:33] maintenance-disconnect-full-disks build 485539 integration-agent-docker-1025 (/: 28%, /srv: 59%, /var/lib/docker: 43%): RECOVERY disk space OK [08:58:12] 10Project-Admins: Create project tag for Blocked-By-Puppet7-Upgrade - https://phabricator.wikimedia.org/T332866 (10Aklapper) 05Open→03Resolved a:03Aklapper Created: https://phabricator.wikimedia.org/project/view/6511/ [09:15:46] 10Phabricator: Make repository URIs verbose by default - https://phabricator.wikimedia.org/T137036 (10Aklapper) 05Open→03Declined Boldly declining this task as we phase out Diffusion as a canonical code repository hosting venue and as folks should use GitLab/Gerrit/whatever is the actual source. [09:17:58] 10Phabricator, 10User-Bongo-Cat: Can I have volunteer badge? - https://phabricator.wikimedia.org/T286078 (10Aklapper) 05Open→03Resolved a:03Aklapper [09:26:57] 10Diffusion, 10Phabricator: Diffusion's default "HEAD" links make it hard to get to "master" branches - https://phabricator.wikimedia.org/T177785 (10Aklapper) 05Open→03Resolved https://phabricator.wikimedia.org/source/wikibugs/ says `master` now instead of `HEAD` [09:27:21] 10Phabricator: Email sometimes not being sent when a task is created and "other task activity" is not set in user preferences - https://phabricator.wikimedia.org/T182549 (10Aklapper) I have not experienced this for a while. Can anyone still reproduce? [09:29:13] 10Phabricator, 10Release-Engineering-Team (Seen), 10Scap: Refresh Jenkins API token for Harbormaster - https://phabricator.wikimedia.org/T207335 (10Aklapper) Proposing to decline this task, assuming we don't need Harbormaster anymore due to phasing out Differential and canonical hosting in Diffusion. Someone... [09:35:18] 10Phabricator, 10I18n: Typo in Phabricator (Audit) in English - https://phabricator.wikimedia.org/T312053 (10Aklapper) [09:47:57] 10Phabricator (Upstream): Typo in Phabricator (Audit) in English - https://phabricator.wikimedia.org/T312053 (10Aklapper) 05Open→03Declined Thanks for reporting. I have forwarded the issue to upstream in https://we.phorge.it/T15267 Declining this task here in and for Wikimedia Phabricator, as we do not plan... [09:48:02] 10Phabricator (Upstream): 2 simple typos in English for Phabricator (Differential) - https://phabricator.wikimedia.org/T312051 (10Aklapper) 05Open→03Declined Thanks for reporting. I have forwarded the issue to upstream in https://we.phorge.it/T15269 Declining this task here in and for Wikimedia Phabricator,... [09:48:04] 10Phabricator (Upstream): Typo in Phabricator (Arcanist) in English - https://phabricator.wikimedia.org/T312052 (10Aklapper) 05Open→03Declined Thanks for reporting. I have forwarded the issue to upstream in https://we.phorge.it/T15268 Declining this task here in and for Wikimedia Phabricator, as we do not p... [09:51:35] 10Phabricator (Upstream): "Access Denied: Restricted File" strings concatenated without whitespace - https://phabricator.wikimedia.org/T293378 (10Aklapper) 05Open→03Declined Not planning to maintain a downstream custom patch for this thus declining; upstreamed as https://we.phorge.it/T15270 [09:52:38] 10Phabricator, 10Release-Engineering-Team (Seen), 10Traffic: Phabricator search times out - https://phabricator.wikimedia.org/T291775 (10Aklapper) [09:52:54] 10Phabricator: Searching shorter terms ("gerrit", "cors") in global Phabricator search leads to upstream request timeout (too many results?) - https://phabricator.wikimedia.org/T258803 (10Aklapper) [10:06:05] 10Phabricator: "Unknown Object (????)" shown in Status field in saved Phabricator search (due to renaming/removing some status) - https://phabricator.wikimedia.org/T227910 (10Aklapper) 05Open→03Resolved a:03Aklapper I switched W4 from using a custom query to the shared saved query https://phabricator.wikim... [10:10:35] maintenance-disconnect-full-disks build 485554 integration-agent-docker-1033 (/: 28%, /srv: 95%, /var/lib/docker: 39%): OFFLINE due to disk space [10:10:36] maintenance-disconnect-full-disks build 485554 integration-agent-docker-1038 (/: 28%, /srv: 98%, /var/lib/docker: 63%): OFFLINE due to disk space [10:12:54] 10Phabricator: Set "Can Use Application" for Macro to "Public" - https://phabricator.wikimedia.org/T314001 (10Aklapper) 05Open→03Resolved a:03Aklapper Done. [10:14:32] 10Phabricator: Unhandled Exception ("RuntimeException"): Undefined offset: 5 when trying to access T16235 - https://phabricator.wikimedia.org/T284397 (10Aklapper) [10:14:58] 10Phabricator, 10Voice & Tone: "The owner of a task can always view and edit it." is confusing - https://phabricator.wikimedia.org/T279827 (10Aklapper) [10:15:36] maintenance-disconnect-full-disks build 485555 integration-agent-docker-1033 (/: 28%, /srv: 47%, /var/lib/docker: 37%): RECOVERY disk space OK [10:15:37] maintenance-disconnect-full-disks build 485555 integration-agent-docker-1038 (/: 28%, /srv: 71%, /var/lib/docker: 63%): RECOVERY disk space OK [10:25:16] 10Phabricator, 10Release-Engineering-Team, 10Wikimedia-Phabricator-Extensions, 10User-brennen: Custom Phab Reports: "Unhandled Exception ("RuntimeException") Division by zero" - https://phabricator.wikimedia.org/T324319 (10Aklapper) [10:25:31] 10Phabricator, 10Release-Engineering-Team, 10Wikimedia-Phabricator-Extensions: Custom Phab Reports: "Unhandled Exception ("RuntimeException") count(): Parameter must be an array or an object that implements Countable" - https://phabricator.wikimedia.org/T324320 (10Aklapper) [11:07:17] 10Phabricator, 10Release-Engineering-Team, 10serviceops-collab, 10User-brennen: Replace existing aphlict1001 with puppet-managed bullseye host - https://phabricator.wikimedia.org/T333452 (10eoghan) [11:19:03] 10GitLab (Infrastructure), 10serviceops-collab: Define future design of GitLab backups - https://phabricator.wikimedia.org/T330172 (10Jelto) [11:52:53] 10Phabricator (Upstream): If you use “Sign With MFA” on a low-resolution device using Chrome, you won't be able to submit content - https://phabricator.wikimedia.org/T335351 (10I) [12:37:28] 10Phabricator (Upstream), 10Upstream: If you use “Sign With MFA” on a low-resolution device using Chrome, you won't be able to submit content - https://phabricator.wikimedia.org/T335351 (10Aklapper) 05Open→03Stalled > * Select Sign With MFA and make sure that you has not prompted to provide MFA yet. > * Tr... [12:51:04] 10Phabricator (Upstream), 10Upstream: If you use “Sign With MFA” on a low-resolution device using Chrome, you won't be able to submit content - https://phabricator.wikimedia.org/T335351 (10Aklapper) 05Stalled→03Invalid The error message is correct as you have not set up MFA for your account. This is expect... [14:35:38] maintenance-disconnect-full-disks build 485607 integration-agent-docker-1025 (/: 29%, /srv: 97%, /var/lib/docker: 48%): OFFLINE due to disk space [14:40:37] maintenance-disconnect-full-disks build 485608 integration-agent-docker-1025 (/: 29%, /srv: 45%, /var/lib/docker: 44%): RECOVERY disk space OK [15:20:02] Project beta-update-databases-eqiad build #66829: 04FAILURE in 1.9 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/66829/ [15:31:17] Yippee, build fixed! [15:31:17] Project beta-update-databases-eqiad build #66830: 09FIXED in 4 min 58 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/66830/ [16:03:27] 10GitLab (Infrastructure), 10Data-Persistence-Backup, 10serviceops-collab, 10User-brennen: Setup partial backups for GitLab - https://phabricator.wikimedia.org/T316935 (10Jelto) a:03Jelto I've done a benchmark of backing up the different datasources on `gitlab1003`: | datasource | duration | size | | em... [16:04:34] 10Phabricator: Deadline herald rule is broken - https://phabricator.wikimedia.org/T335367 (10Pppery) [16:39:52] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review: Add job definitions to CasC configuration of releases Jenkins - https://phabricator.wikimedia.org/T334669 (10jnuche) 05Open→03Resolved [16:39:54] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team (Priority Backlog 📥), 10serviceops-collab: Automate integration Jenkins deployment and config changes - https://phabricator.wikimedia.org/T319406 (10jnuche) [17:32:42] 10GitLab (Infrastructure), 10serviceops-collab, 10Patch-For-Review: Let's Encrypt certificate expiration notice for domain gitlab.devtools.wmcloud.org - https://phabricator.wikimedia.org/T335161 (10Dzahn) It makes a lot of sense that closing port 80 caused this, as the certbot renewal method uses that. Good... [17:43:49] 10Phabricator: Deadline herald rule is broken - https://phabricator.wikimedia.org/T335367 (10Aklapper) 05Open→03Resolved a:03Aklapper Urgh, thanks for finding this! Trying to edit H296, I only get an `Unhandled Exception: No field with key "herald.custom/std:maniphest:deadline.due" is available to Herald... [17:47:21] 10Phabricator (Upstream), 10Upstream: When a project is not public, it appears to users outside "Visible To" criteria as "Unknown Object", which is confusing - https://phabricator.wikimedia.org/T302525 (10Aklapper) [18:51:47] 10Phabricator, 10Accessibility: Production Error task subtype symbol appearing on tasks, indicating something urgent, but is non-interactive - https://phabricator.wikimedia.org/T256995 (10Aklapper) 05Open→03Resolved Fixed long ago [19:56:08] 10GitLab (Project Migration), 10Release-Engineering-Team (Priority Backlog 📥), 10API Platform, 10Anti-Harassment, and 19 others: Migrate PipelineLib repos to GitLab - https://phabricator.wikimedia.org/T332953 (10Jdforrester-WMF) [20:52:55] 10Gerrit, 10Release-Engineering-Team, 10serviceops-collab, 10Patch-For-Review, 10Sustainability (Incident Followup): Move Gerrit data out of root partition - https://phabricator.wikimedia.org/T333143 (10Dzahn) I updated the code so that we have a new path for lfs data under /srv/gerrit/data, as suggested... [21:00:03] (03PS1) 10Kevin Wilke: Zuul: Add Kevin Wilke to CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/911930 [21:08:07] as always when a prod role gets added to a new machine and puppet is supposed to do intial scap deploy.. it fails: [21:08:07] Package[gerrit/gerrit]/ensure: change from 'absent' to 'present' failed: Execution of '/usr/bin/scap deploy-local --repo gerrit/gerrit -D log_json:False' returned 70 [21:08:34] looks for "tin" deployment server in cache dir [21:22:36] 10Continuous-Integration-Config, 10MediaWiki-Core-Tests, 10Browser-Tests, 10Patch-For-Review, and 2 others: Make MediaWiki Wdio tests less slow (Sept 2019) - https://phabricator.wikimedia.org/T234002 (10Jrbranaa) [21:23:23] !log gerrit1003 - sudo -u gerrit2 /usr/bin/scap deploy-local --repo gerrit/gerrit -D log_json:False (manually it works, but that's the same command that puppet is supposed to run !?) - T257317 T326368 [21:23:27] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:23:27] T326368: gerrit1003 service implementation task - https://phabricator.wikimedia.org/T326368 [21:23:28] T257317: scap deploy --init on deployment server fails on first puppet run - https://phabricator.wikimedia.org/T257317 [21:24:40] :) I got the production gerrit role on gerrit1003 and did not break prod gerrit :) [21:25:38] https://gerrit-new.wikimedia.org/r/ [22:30:13] Nice! [22:38:30] dancy: do you think I should keep the services stopped..so it shows the "nice error page" (like during maintenance).. or just keep it running.. then users can login but confusingly they get redirect to regular prod gerrit after the login .. hmmm [22:39:41] or..what we really want is that the login works but it stays on gerrit-new.. but don't know how yet [22:43:52] Let's leave it running [22:47:00] alright:) that just means I do NOT have to merge a code change to make it optional:) [22:48:07] ah, and after being logged in once.. I can see https://gerrit-new.wikimedia.org/r/dashboard/self now and it shows the state from 2 days ago or so [22:48:13] that's when i rsynced data [22:49:32] since I set "Dark Theme" in user settings in prod gerrit I now have one black and one white gerrit