[08:11:23] 10serviceops, 10API Platform, 10Growth-Structured-Tasks, 10Image-Suggestions, and 4 others: Exception: Invalid JSON response for page: Espejo - https://phabricator.wikimedia.org/T313973 (10kostajh) >>! In T313973#8194390, @hnowlan wrote: > Argh, my mistake - ignore my last in that case :) > > If this Env... [08:27:26] _joe_, claime: I wrote up the steps to rebuild/extend our PHP 7.4 packages here: https://wikitech.wikimedia.org/wiki/Application_servers#Update_our_PHP_packages [08:27:32] 10serviceops, 10Parsoid, 10Patch-For-Review, 10Performance-Team (Radar): Parsoid migration to php 7.4 - https://phabricator.wikimedia.org/T312638 (10Clement_Goubert) Reverted almost immediately due to https://phabricator.wikimedia.org/T316601 [08:28:16] <_joe_> moritzm: thanks, I think I'll do that myself though for this time, claime has enough stuff to learn for today, but thanks [08:28:25] 10serviceops, 10Parsoid, 10Patch-For-Review, 10Performance-Team (Radar): Parsoid migration to php 7.4 - https://phabricator.wikimedia.org/T312638 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=9b43b97d-2497-4b6c-848a-987b749a898e) set by cgoubert@cumin1001 for 7 days, 0:00:00 on 24 hos... [08:28:28] ok [08:28:39] <_joe_> I will try to follow it with maximum ingenuity and improve the docs [08:28:45] +1 [08:28:50] moritzm: thanks ! [08:29:17] moritzm: would you have a minute for pgp/pwstore stuff ? [08:29:33] sure [08:29:47] ok, moving to pm [09:17:54] 10serviceops, 10Parsoid, 10Patch-For-Review, 10Performance-Team (Radar): Parsoid migration to php 7.4 - https://phabricator.wikimedia.org/T312638 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1001 for host parse1002.eqiad.wmnet with OS buster [09:27:14] 10serviceops, 10Parsoid, 10Patch-For-Review, 10Performance-Team (Radar): Parsoid migration to php 7.4 - https://phabricator.wikimedia.org/T312638 (10Jersione) a:03Jersione [09:28:04] 10serviceops, 10Parsoid, 10Patch-For-Review, 10Performance-Team (Radar): Parsoid migration to php 7.4 - https://phabricator.wikimedia.org/T312638 (10Ladsgroup) a:05Jersione→03None [09:28:43] 10serviceops, 10Parsoid, 10Patch-For-Review, 10Performance-Team (Radar): Parsoid migration to php 7.4 - https://phabricator.wikimedia.org/T312638 (10Clement_Goubert) a:03Clement_Goubert [09:36:37] 10serviceops: Put parse parse100[01-24] in production - https://phabricator.wikimedia.org/T307219 (10Clement_Goubert) a:03Clement_Goubert [10:06:21] 10serviceops, 10Parsoid, 10Patch-For-Review, 10Performance-Team (Radar): Parsoid migration to php 7.4 - https://phabricator.wikimedia.org/T312638 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1001 for host parse1002.eqiad.wmnet with OS buster completed: - parse1002 (*... [12:04:24] 10serviceops, 10Infrastructure-Foundations, 10netbox: Netbox and Redis - https://phabricator.wikimedia.org/T311385 (10Volans) Just for historical context, when Netbox first introduced a requirement of Redis, it was configured to use the shared Redis installation and then moved to a local instance later on in... [12:38:06] 10serviceops, 10Datasets-General-or-Unknown, 10Developer Productivity, 10Growth-Team (Current Sprint): Allow use of wmf's MW CLI scripts on snapshot hosts instead of bypassing - https://phabricator.wikimedia.org/T314697 (10kostajh) p:05Triage→03Low [12:42:00] 10serviceops, 10API Platform, 10Growth-Structured-Tasks, 10Image-Suggestions, and 5 others: Exception: Invalid JSON response for page: Espejo - https://phabricator.wikimedia.org/T313973 (10JMeybohm) Ingress[1] metrics suggest that we have quite some requests with > 10 seconds response time. The timeout con... [13:25:02] 10serviceops, 10Infrastructure-Foundations, 10netbox: Netbox and Redis - https://phabricator.wikimedia.org/T311385 (10akosiaris) >>! In T311385#8181710, @ayounsi wrote: >> Can netbox be configured without the redis cache? After all it just might not be worth it to have it. > Unfortunately not. According to h... [14:16:03] 10serviceops, 10CFSSL-PKI, 10Infrastructure-Foundations, 10Prod-Kubernetes, 10Kubernetes: Update cfssl-issuer to cert-manager 1.8.x - https://phabricator.wikimedia.org/T310486 (10JMeybohm) a:03JMeybohm [14:37:39] 10serviceops, 10Add-Link, 10Growth-Team, 10Prod-Kubernetes, 10Kubernetes: Use ingress for linkrecommendation - https://phabricator.wikimedia.org/T302717 (10jijiki) a:05JMeybohm→03jijiki [19:12:16] 10serviceops, 10Data-Persistence-Backup, 10serviceops-collab, 10GitLab (Infrastructure), and 2 others: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10Dzahn) gitlab2003 before reverting the partman change: ` Filesystem Size Used Avail Use% Mounted o... [19:16:38] 10serviceops, 10Data-Persistence-Backup, 10serviceops-collab, 10GitLab (Infrastructure), and 2 others: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by dzahn@cumin2002 for host gitlab2003.wikimedia.org with OS bullseye [19:18:27] 10serviceops, 10Data-Persistence-Backup, 10serviceops-collab, 10GitLab (Infrastructure), and 2 others: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by dzahn@cumin2002 for host gitlab2003.wikimedia.org with OS bullseye exe... [19:21:42] 10serviceops, 10Data-Persistence-Backup, 10serviceops-collab, 10GitLab (Infrastructure), and 2 others: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by dzahn@cumin2002 for host gitlab2003.wikimedia.org with OS bullseye [19:56:51] 10serviceops, 10Data-Persistence-Backup, 10serviceops-collab, 10GitLab (Infrastructure), and 2 others: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by dzahn@cumin2002 for host gitlab2003.wikimedia.org with OS bullseye com... [20:00:55] 10serviceops, 10Data-Persistence-Backup, 10serviceops-collab, 10GitLab (Infrastructure), and 2 others: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10Dzahn) gitlab2003 has been reimaged. ECDSA key fingerprint is SHA256:hdh4RRTigMf9Q0+fqMKk8e35g0wVUhceRYYlEPLmgf8. now it looks like this... [20:23:38] 10serviceops, 10Data-Persistence-Backup, 10serviceops-collab, 10GitLab (Infrastructure), and 2 others: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by dzahn@cumin2002 for host gitlab2003.wikimedia.org with OS bullseye [20:25:02] 10serviceops, 10DBA, 10Patch-For-Review, 10Performance-Team (Radar): Update wgLBFactoryConf for x2 to register only the local primary - https://phabricator.wikimedia.org/T316482 (10CDanis) @Marostegui I actually implemented this not as a new flavor, but instead as a boolean attribute `omit_replicas_in_mwco... [20:44:36] 10serviceops, 10API Platform, 10Growth-Structured-Tasks, 10Image-Suggestions, and 5 others: Exception: Invalid JSON response for page: Espejo - https://phabricator.wikimedia.org/T313973 (10Eevans) I submitted [[ https://gerrit.wikimedia.org/r/c/generated-data-platform/datasets/image-suggestions/+/828631 |... [20:58:04] 10serviceops, 10Data-Persistence-Backup, 10serviceops-collab, 10GitLab (Infrastructure), and 2 others: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by dzahn@cumin2002 for host gitlab2003.wikimedia.org with OS bullseye com... [21:01:37] 10serviceops, 10Data-Persistence-Backup, 10serviceops-collab, 10GitLab (Infrastructure), and 2 others: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10Dzahn) puppet had not run on apt* yet. I had to repeat the reimaging. But now it looks as intended: ` [gitlab2003:~] $ df -h Filesystem... [21:48:55] 10serviceops, 10Parsoid, 10Patch-For-Review, 10Performance-Team (Radar): Parsoid migration to php 7.4 - https://phabricator.wikimedia.org/T312638 (10Dzahn) @Clement_Goubert Just wanted to share something, it's not a problem at all but Icinga monitoring does alert on hosts "not being in dsh groups". This ma... [21:51:16] 10serviceops, 10Parsoid, 10Patch-For-Review, 10Performance-Team (Radar): Parsoid migration to php 7.4 - https://phabricator.wikimedia.org/T312638 (10Dzahn) P.S. the cookbook was supposed to set the dowtime for that but that failed for some reason, so nothing you were expected to do differently