[01:29:52] 10serviceops, 10PHP 7.3 support, 10PHP 7.4 support, 10Platform Team Workboards (Clinic Duty Team): Rename articles and users to prepare for PHP 7.3 unicode changes - https://phabricator.wikimedia.org/T292552 (10Krinkle) [06:01:57] 10serviceops, 10Infrastructure-Foundations, 10SRE, 10SRE-Access-Requests: allow certain users to disable puppet on mwdebug hosts - https://phabricator.wikimedia.org/T305979 (10jcrespo) This seems to me like a reasonable requests, although as you point out, the details of how to exactly implement it to make... [06:10:11] 10serviceops, 10Math, 10MediaWiki-Categories, 10RESTBase, 10Patch-For-Review: \land – Unclear why the page appears in an error-category - https://phabricator.wikimedia.org/T305613 (10Physikerwelt) OK, I am sending the output of the curl -v in private paste as suggested. Even if it is a cachi... [08:39:13] 10serviceops, 10Release-Engineering-Team, 10Scap: Deploy Scap version 4.6.1 - https://phabricator.wikimedia.org/T305949 (10JMeybohm) a:03JMeybohm [08:46:56] 10serviceops, 10Release-Engineering-Team, 10Scap: Deploy Scap version 4.6.1 - https://phabricator.wikimedia.org/T305949 (10JMeybohm) 4.6.1 rolled out to canaries. `scap pull`works as usual, `cd /srv/deployment/restbase/deploy/; scap deploy --environment dev-cluster` feels way(!) slower then the last times I... [08:47:12] 10serviceops, 10Release-Engineering-Team, 10Scap: Deploy Scap version 4.6.1 - https://phabricator.wikimedia.org/T305949 (10JMeybohm) 05Open→03Stalled [09:17:17] 10serviceops, 10Release-Engineering-Team, 10Scap: Deploy Scap version 4.6.1 - https://phabricator.wikimedia.org/T305949 (10jnuche) @JMeybohm I'll take a look. What host(s) did you run `scap deploy --environment dev-cluster` on? [09:19:12] 10serviceops, 10Release-Engineering-Team, 10Scap: Deploy Scap version 4.6.1 - https://phabricator.wikimedia.org/T305949 (10JMeybohm) >>! In T305949#7850762, @jnuche wrote: > @JMeybohm I'll take a look. What host(s) did you run `scap deploy --environment dev-cluster` on? deploy1002.eqiad.wmnet [09:31:53] 10serviceops, 10Release-Engineering-Team, 10Scap: Deploy Scap version 4.6.1 - https://phabricator.wikimedia.org/T305949 (10jnuche) The deploy to canaries and subsequent rollback took me less than 3mins: ` 09:24:19 Finished deploy [restbase/deploy@627f7d7] (dev-cluster) (duration: 02m 51s) ` How long did it... [09:44:37] 10serviceops, 10Citoid, 10Patch-For-Review: zotero paging / serving 5xxes after CPU spikes - https://phabricator.wikimedia.org/T291707 (10akosiaris) I don't see an alert since Apr 5 in Icinga for either codfw or eqiad. I am gonna re-enabling paging. [10:02:18] 10serviceops, 10Citoid, 10Patch-For-Review: zotero paging / serving 5xxes after CPU spikes - https://phabricator.wikimedia.org/T291707 (10akosiaris) 05Open→03Resolved a:03akosiaris Tentatively resolving this. [11:02:59] 10serviceops, 10Infrastructure-Foundations, 10SRE, 10SRE-Access-Requests: allow certain users to disable puppet on mwdebug hosts - https://phabricator.wikimedia.org/T305979 (10Volans) > let them run "puppet disable/enable" either directly or with a wrapper around it. (the one used by cumin?). Nobody shoul... [11:46:47] 10serviceops, 10Infrastructure-Foundations, 10SRE, 10SRE-Access-Requests: allow certain users to disable puppet on mwdebug hosts - https://phabricator.wikimedia.org/T305979 (10jcrespo) [13:37:32] 10serviceops, 10Prod-Kubernetes, 10Toolhub, 10Kubernetes, 10Patch-For-Review: Maintenance environment needed for running one-off commands - https://phabricator.wikimedia.org/T290357 (10akosiaris) >>! In T290357#7846886, @bd808 wrote: >>>! In T290357#7405708, @akosiaris wrote: >> That leaves the use cases... [13:56:05] 10serviceops, 10Deployments, 10Release-Engineering-Team (Radar), 10Sustainability (Incident Followup), 10User-jijiki: Remove provisioning for 'mwscript', 'foreachwikiindblist' etc from deployment host - https://phabricator.wikimedia.org/T253822 (10Krinkle) p:05Triage→03Medium a:03dancy [15:15:31] 10serviceops, 10Release-Engineering-Team, 10Scap: Deploy Scap version 4.6.1 - https://phabricator.wikimedia.org/T305949 (10jnuche) @JMeybohm so I just learned we are indeed still supporting Scap3, but it hasn't seen any active development in a while. In particular, there haven't been any changes since the la... [15:24:55] 10serviceops, 10Release-Engineering-Team, 10Scap: Deploy Scap version 4.6.1 - https://phabricator.wikimedia.org/T305949 (10JMeybohm) 05Stalled→03In progress >>! In T305949#7851774, @jnuche wrote: > Would you be ok with going ahead with the deploy? We can take a deeper look in the future if the issue pers... [15:24:58] hello, today I have to drive down to L.A.. Most of the day I will be driving and afk. I'll do some work later though during breaks and once I get there [15:30:14] 10serviceops, 10Beta-Cluster-Infrastructure: Serve beta cluster on PHP 7.4 by default - https://phabricator.wikimedia.org/T306042 (10Reedy) [15:37:11] 10serviceops, 10Beta-Cluster-Infrastructure: Serve beta cluster via PHP 7.4 by default - https://phabricator.wikimedia.org/T306042 (10Reedy) [15:41:26] 10serviceops, 10Data-Catalog, 10Data-Engineering, 10SRE, and 2 others: New Service Request: DataHub - https://phabricator.wikimedia.org/T303049 (10BTullis) I'm unsure what else I need to do now to make this new service available. I've successfully deployed the service to staging, eqiad and codfw using `he... [16:04:42] 10serviceops, 10Release-Engineering-Team, 10Scap: Deploy Scap version 4.6.1 - https://phabricator.wikimedia.org/T305949 (10jnuche) @JMeybohm thank you :) [16:22:39] 10serviceops, 10Scap, 10Release-Engineering-Team (Radar): Deploy Scap version 4.6.1 - https://phabricator.wikimedia.org/T305949 (10thcipriani) [16:26:33] 10serviceops, 10MW-on-K8s, 10Kubernetes, 10Patch-For-Review, 10Release-Engineering-Team (Radar): Kubernetes credentials on deployment servers should be available to deployers, not all users - https://phabricator.wikimedia.org/T305729 (10thcipriani) [16:36:52] 10serviceops, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Radar): docker-report-releng failing on multiple image tags because of certificate validation error - https://phabricator.wikimedia.org/T304875 (10thcipriani) [16:42:41] 10serviceops, 10Infrastructure-Foundations, 10Scap, 10Patch-For-Review, 10Release-Engineering-Team (Priority Backlog 📥): Use scap to deploy itself to scap targets - https://phabricator.wikimedia.org/T303559 (10thcipriani) [16:45:30] 10serviceops, 10MW-on-K8s, 10Release-Engineering-Team, 10Scap: Deploy MediaWiki images for kubernetes from the deployment servers - https://phabricator.wikimedia.org/T302539 (10thcipriani) [16:45:36] 10serviceops, 10Infrastructure-Foundations, 10SRE, 10SRE-Access-Requests, 10Release-Engineering-Team (Radar): Need a service account on deploy servers - https://phabricator.wikimedia.org/T303857 (10thcipriani) [16:46:58] 10serviceops, 10MW-on-K8s, 10Scap, 10Release-Engineering-Team (Radar): Deploy MediaWiki images for kubernetes from the deployment servers - https://phabricator.wikimedia.org/T302539 (10thcipriani) 05Open→03Stalled Stalled waiting on the service account [16:47:57] 10serviceops, 10Release-Engineering-Team, 10Scap: Set scap minimum python version to 3.7 - https://phabricator.wikimedia.org/T302086 (10thcipriani) [16:48:57] 10serviceops, 10Scap, 10Release-Engineering-Team (Radar): Set scap minimum python version to 3.7 - https://phabricator.wikimedia.org/T302086 (10thcipriani) 05Open→03Stalled [17:09:57] 10serviceops, 10Gerrit, 10Phabricator, 10Cloud-VPS (Debian Stretch Deprecation): Cloud VPS "devtools" project Stretch deprecation - https://phabricator.wikimedia.org/T306069 (10Dzahn) [17:10:41] 10serviceops, 10Gerrit, 10Phabricator, 10Cloud-VPS (Debian Stretch Deprecation): Cloud VPS "devtools" project Stretch deprecation - https://phabricator.wikimedia.org/T306069 (10Dzahn) p:05Medium→03High raising priority because "may be deleted within 2 weeks" is soon and impactful [17:35:25] 10serviceops, 10Gerrit, 10Phabricator, 10Cloud-VPS (Debian Stretch Deprecation): Cloud VPS "devtools" project Stretch deprecation - https://phabricator.wikimedia.org/T306069 (10Zabe) [17:44:54] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Move kubernetes workers to bullseye and docker to overlayfs - https://phabricator.wikimedia.org/T300744 (10Cmjohnson) [17:44:56] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: setup/install kubernetes10[18-22] - https://phabricator.wikimedia.org/T293728 (10Cmjohnson) [17:51:44] 10serviceops, 10DC-Ops, 10ops-eqiad: Q4: (Need By: TBD) rack/setup/install mw14[57-98] - https://phabricator.wikimedia.org/T306121 (10RobH) [17:51:58] 10serviceops, 10DC-Ops, 10ops-eqiad: Q4: (Need By: TBD) rack/setup/install mw14[57-98] - https://phabricator.wikimedia.org/T306121 (10RobH) [17:52:34] 10serviceops, 10DC-Ops, 10ops-eqiad: Q4: (Need By: TBD) rack/setup/install mw14[57-98] - https://phabricator.wikimedia.org/T306121 (10RobH) [20:13:56] 10serviceops, 10SRE, 10Traffic, 10Platform Team Workboards (Green): MW REST API should be routed to api_appserver MW cluster - https://phabricator.wikimedia.org/T268043 (10BBlack) [20:34:32] 10serviceops, 10MW-on-K8s, 10Scap, 10Release-Engineering-Team (Radar): Deploy MediaWiki images for kubernetes from the deployment servers - https://phabricator.wikimedia.org/T302539 (10dancy) [20:34:37] 10serviceops, 10Infrastructure-Foundations, 10SRE, 10SRE-Access-Requests, 10Release-Engineering-Team (Radar): Need a service account on deploy servers - https://phabricator.wikimedia.org/T303857 (10dancy) [20:34:56] 10serviceops, 10MW-on-K8s, 10Scap, 10Release-Engineering-Team (Radar): Deploy MediaWiki images for kubernetes from the deployment servers - https://phabricator.wikimedia.org/T302539 (10dancy) [20:34:59] 10serviceops, 10MW-on-K8s, 10Kubernetes, 10Patch-For-Review, 10Release-Engineering-Team (Radar): Kubernetes credentials on deployment servers should be available to deployers, not all users - https://phabricator.wikimedia.org/T305729 (10dancy) [20:35:32] 10serviceops, 10Infrastructure-Foundations, 10SRE, 10SRE-Access-Requests, 10Release-Engineering-Team (Radar): Need a service account on deploy servers for automated train pre-sync operations - https://phabricator.wikimedia.org/T303857 (10dancy) [20:38:01] 10serviceops, 10MW-on-K8s, 10Scap, 10Release-Engineering-Team (Radar): Deploy MediaWiki images for kubernetes from the deployment servers - https://phabricator.wikimedia.org/T302539 (10dancy) >>! In T302539#7852342, @thcipriani wrote: > Stalled waiting on the service account Correction: Waiting for T30572... [20:52:21] 10serviceops, 10Performance-Team, 10SRE, 10Traffic: Potential navtiming_responseStart regression as of 13 Mar 2022 - https://phabricator.wikimedia.org/T303782 (10Krinkle) 05Open→03Resolved There seems to be an upward trend that is continying having possibly added around ~25ms (5% of 500ms) on both the... [22:56:58] 10serviceops, 10SRE, 10Thumbor, 10Patch-For-Review, and 2 others: Run latest Thumbor on Docker with Buster + Python 3 - https://phabricator.wikimedia.org/T267327 (10Krinkle)