[09:56:10] 10serviceops, 10Citoid, 10VisualEditor, 10WMSE-Bug-Reporting-and-Translation-2021, and 2 others: Automatic citation generation using ISBN on Wikipedia doesn't work - https://phabricator.wikimedia.org/T294010 (10akosiaris) 05Open→03Resolved a:03akosiaris >>! In T294010#7475352, @Mvolz wrote: >>>! In T... [15:53:41] 10serviceops, 10Community-Tech, 10SRE, 10wikidiff2, 10Platform Team Workboards (Platform Engineering Reliability): Deploy wikidiff2 1.13.0 - https://phabricator.wikimedia.org/T285857 (10hnowlan) The deployment is just finishing up to codfw's API servers in the next few minutes, all others are complete. P... [15:53:48] 10serviceops, 10Community-Tech, 10SRE, 10wikidiff2, 10Platform Team Workboards (Platform Engineering Reliability): Deploy wikidiff2 1.13.0 - https://phabricator.wikimedia.org/T285857 (10hnowlan) [15:54:16] 10serviceops, 10Community-Tech, 10SRE, 10wikidiff2, 10Platform Team Workboards (Platform Engineering Reliability): Deploy wikidiff2 1.13.0 - https://phabricator.wikimedia.org/T285857 (10hnowlan) PHP 7.4 images have been bumped via https://gerrit.wikimedia.org/r/plugins/gitiles/operations/docker-images/pr... [16:14:34] 10serviceops, 10Scap, 10Release-Engineering-Team (Radar): RESTBase deployment fails with scap internal error - https://phabricator.wikimedia.org/T294936 (10thcipriani) ` [thcipriani@deploy1002 ~]$ SSH_AUTH_SOCK=/run/keyholder/proxy.sock ssh -l deploy-service -oIdentitiesOnly=yes -oIdentityFile=/etc/keyholder... [16:14:47] 10serviceops, 10Scap, 10Release-Engineering-Team (Radar): RESTBase deployment fails with scap internal error - https://phabricator.wikimedia.org/T294936 (10Majavah) [[ https://debmonitor.wikimedia.org/packages/scap | Debmonitor ]] reveals restbase2010 (and a bunch of other servers) are still using scap 4.0.0... [16:19:07] 10serviceops, 10Scap, 10Release-Engineering-Team (Radar): RESTBase deployment fails with scap internal error - https://phabricator.wikimedia.org/T294936 (10Legoktm) >>! In T294936#7478496, @thcipriani wrote: > Pinging #serviceops for help: could a serviceopsen ensure that scap is at version 4.0.2 everywhere?... [16:20:39] 10serviceops, 10Scap, 10Release-Engineering-Team (Radar): RESTBase deployment fails with scap internal error - https://phabricator.wikimedia.org/T294936 (10dancy) @Legoktm I will prepare a new release. [16:22:02] 10serviceops, 10Scap, 10Release-Engineering-Team (Doing): RESTBase deployment fails with scap internal error - https://phabricator.wikimedia.org/T294936 (10thcipriani) [16:23:29] 10serviceops, 10Scap, 10Release-Engineering-Team (Doing): RESTBase deployment fails with scap internal error - https://phabricator.wikimedia.org/T294936 (10Legoktm) >>! In T294936#7478503, @Majavah wrote: > I don't have an explanation for the one host running scap 3.17.1 visible on debmonitor, mw2280 has be... [18:13:09] 10serviceops, 10DC-Ops, 10ops-codfw: Q2:(Need By: TBD) rack/setup/install mc20[38-55] - https://phabricator.wikimedia.org/T294962 (10RobH) [18:13:16] 10serviceops, 10DC-Ops, 10ops-codfw: Q2:(Need By: TBD) rack/setup/install mc20[38-55] - https://phabricator.wikimedia.org/T294962 (10RobH) [18:15:48] 10serviceops, 10DC-Ops, 10ops-codfw: Q2:(Need By: TBD) rack/setup/install mc20[38-55] - https://phabricator.wikimedia.org/T294962 (10RobH) a:03jijiki @jijiki, The order task T291998#7461853 didn't have racking details, and it seems you are best to provide this for the servers before they arrive? Addition... [18:23:13] 10serviceops: Productionise mc20[38-55] - https://phabricator.wikimedia.org/T293012 (10RobH) [18:25:17] 10serviceops, 10DC-Ops, 10ops-codfw: Q2:(Need By: TBD) rack/setup/install mc20[38-55] - https://phabricator.wikimedia.org/T294962 (10Joe) @RobH @jijiki is on PTO at the moment. [18:26:37] 10serviceops, 10DC-Ops, 10ops-codfw: Q2:(Need By: TBD) rack/setup/install mc20[38-55] - https://phabricator.wikimedia.org/T294962 (10Joe) a:05jijiki→03Joe [18:28:11] 10serviceops, 10SRE, 10Kubernetes, 10Patch-For-Review: Migrate to helm v3 - https://phabricator.wikimedia.org/T251305 (10Jelto) I migrated all services in `staging-codfw` to helm3 using the snippet https://phabricator.wikimedia.org/P17670. I fixed minor issues in the charts but the general process was quit... [18:54:44] 10serviceops, 10Release-Engineering-Team: Deploy Scap version 4.0.3 - https://phabricator.wikimedia.org/T294966 (10dancy) [18:55:43] 10serviceops, 10Scap, 10Release-Engineering-Team (Doing): RESTBase deployment fails with scap internal error - https://phabricator.wikimedia.org/T294936 (10dancy) [18:55:49] 10serviceops, 10Release-Engineering-Team: Deploy Scap version 4.0.3 - https://phabricator.wikimedia.org/T294966 (10dancy) [18:57:24] 10serviceops, 10Release-Engineering-Team: Deploy Scap version 4.0.3 - https://phabricator.wikimedia.org/T294966 (10Legoktm) a:03Legoktm [19:00:01] 10serviceops, 10Release-Engineering-Team, 10Scap: Deploy Scap version 4.0.3 - https://phabricator.wikimedia.org/T294966 (10dancy) [19:09:58] 10serviceops, 10Patch-For-Review, 10Wikipedia-Android-App-Backlog (Android Release FY2021-22): Create and host assetlinks.json file. (Android 12 deeplinking support) - https://phabricator.wikimedia.org/T294776 (10Dbrant) @Urbanecm thanks for the help with deployment. Just one follow-up question: Would it be... [19:48:03] 10serviceops, 10Wikipedia-Android-App-Backlog (Android Release FY2021-22): Create and host assetlinks.json file. (Android 12 deeplinking support) - https://phabricator.wikimedia.org/T294776 (10Urbanecm) @Dbrant Thanks for asking! That's a very good question, but unfortunately not for me -- the redirect appears... [19:55:50] 10serviceops, 10MediaWiki-General, 10SRE, 10MW-1.35-notes (1.35.0-wmf.28; 2020-04-14), and 5 others: Some pages will become completely unreachable after PHP7 update due to Unicode changes - https://phabricator.wikimedia.org/T219279 (10Urbanecm) @Pchelolo Ping? Can you please review the renames that failed?... [20:30:21] 10serviceops, 10Traffic, 10Wikipedia-Android-App-Backlog (Android Release FY2021-22): Create and host assetlinks.json file. (Android 12 deeplinking support) - https://phabricator.wikimedia.org/T294776 (10Dzahn) [20:33:08] 10serviceops, 10Traffic, 10Wikipedia-Android-App-Backlog (Android Release FY2021-22): Create and host assetlinks.json file. (Android 12 deeplinking support) - https://phabricator.wikimedia.org/T294776 (10Dzahn) Hi Traffic, the question here is.. how come we get a redirect at the caching level from bare domai... [20:35:13] 10serviceops, 10Traffic, 10Wikipedia-Android-App-Backlog (Android Release FY2021-22): Create and host assetlinks.json file. (Android 12 deeplinking support) - https://phabricator.wikimedia.org/T294776 (10Dbrant) (@Dzahn The `spec.yaml` file is also redirecting) [20:42:41] 10serviceops, 10Wikipedia-Android-App-Backlog (Android Release FY2021-22): Create and host assetlinks.json file. (Android 12 deeplinking support) - https://phabricator.wikimedia.org/T294776 (10Dzahn) [20:46:59] 10serviceops, 10Wikipedia-Android-App-Backlog (Android Release FY2021-22): Create and host assetlinks.json file. (Android 12 deeplinking support) - https://phabricator.wikimedia.org/T294776 (10Dzahn) Removed traffic again because I actually found it at the apache level, not at caching level after all. ` 1... [21:15:19] 10serviceops, 10Wikipedia-Android-App-Backlog (Android Release FY2021-22): Create and host assetlinks.json file. (Android 12 deeplinking support) - https://phabricator.wikimedia.org/T294776 (10Dbrant) @Dzahn Thanks for finding that. The more I look into it, the more important this seems to be for Android 12 c... [21:29:35] 10serviceops, 10Wikipedia-Android-App-Backlog (Android Release FY2021-22): Create and host assetlinks.json file. (Android 12 deeplinking support) - https://phabricator.wikimedia.org/T294776 (10Dzahn) @Dbrant This is in the repo `operations/puppet` in `./modules/mediawiki/templates/apache/sites/wwwportals.conf.... [22:47:55] 10serviceops, 10Scap, 10Release-Engineering-Team (Doing): RESTBase deployment fails with scap internal error - https://phabricator.wikimedia.org/T294936 (10Legoktm) @Pchelolo scap is now upgraded on all the restbase hosts. [22:51:42] 10serviceops, 10Scap, 10Release-Engineering-Team (Doing): RESTBase deployment fails with scap internal error - https://phabricator.wikimedia.org/T294936 (10Pchelolo) @Legoktm just tried to deploy again, same result. [22:53:05] 10serviceops, 10Scap, 10Release-Engineering-Team (Doing): RESTBase deployment fails with scap internal error - https://phabricator.wikimedia.org/T294936 (10Pchelolo) Actually, different result, now it's UndefinedError instead of AttributeError.. Feel free to try yourself by running scap deploy from /srv/dep... [23:05:13] 10serviceops, 10Scap, 10Release-Engineering-Team (Doing): RESTBase deployment fails with scap internal error - https://phabricator.wikimedia.org/T294936 (10dancy) {P17672} [23:17:17] 10serviceops, 10Scap, 10Release-Engineering-Team (Doing): RESTBase deployment fails with scap internal error - https://phabricator.wikimedia.org/T294936 (10dancy) I think the UndefinedError is a jinja2 template error. [23:23:49] 10serviceops, 10Release-Engineering-Team, 10Scap: Deploy Scap version 4.0.3 - https://phabricator.wikimedia.org/T294966 (10Legoktm) I rolled back the canaries to 4.0.2 for now. [23:24:59] 10serviceops, 10Scap, 10Release-Engineering-Team (Doing): RESTBase deployment fails with scap internal error - https://phabricator.wikimedia.org/T294936 (10Legoktm) Indeed: {P17675}