[00:09:49] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [00:14:49] (TfInfraTestApplyFailed) resolved: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [00:14:53] 10Cloud-VPS, 10cloud-services-team: Rescue DBapp trove instance in glamwikidashboard project - https://phabricator.wikimedia.org/T355138 (10YonatanWMIL) I have stopped the daily service from adding more data to the DB to prevent it from filling up again. I think now, we should rescue the DB again (hopefully ca... [00:31:13] 10Tools, 10Maps (Kartographer): kmlexport can't give a name to items generated from s - https://phabricator.wikimedia.org/T321507 (10Dvorapa) Hi, as mentioned before, KMLexport is just a webscraper. It scans wiki page for geohack links/urls and parses parameters from these urls. One of the geohack url... [02:49:49] (PuppetDisabled) firing: Puppet disabled on cloudservices2005-dev:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=wmcs&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled [02:49:54] 10cloud-services-team: PuppetDisabled Puppet disabled on cloudservices2005-dev:9100 - https://phabricator.wikimedia.org/T355276 (10phaultfinder) [05:41:23] (HAProxyBackendUnavailable) firing: HAProxy service neutron-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [05:46:22] (HAProxyBackendUnavailable) resolved: HAProxy service neutron-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [07:29:23] 10PAWS: Update rstudio - https://phabricator.wikimedia.org/T355288 (10rook) [07:37:19] 10PAWS: Update rstudio - https://phabricator.wikimedia.org/T355288 (10rook) [07:38:59] 10PAWS: Update rstudio - https://phabricator.wikimedia.org/T355288 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/paws/pull/366 [07:39:13] vivian-rook opened https://github.com/toolforge/paws/pull/366 [08:05:29] 10PAWS: Update rstudio - https://phabricator.wikimedia.org/T355288 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/paws/pull/366 [08:05:40] vivian-rook closed https://github.com/toolforge/paws/pull/366 [08:05:45] 10PAWS: Update rstudio - https://phabricator.wikimedia.org/T355288 (10rook) 05Open→03Resolved [08:18:32] (03CR) 10Nikerabbit: "recheck" [labs/tools/wikinity] - 10https://gerrit.wikimedia.org/r/982233 (https://phabricator.wikimedia.org/T310688) (owner: 10Nikerabbit) [10:08:37] 10Toolforge (Toolforge iteration 03): [jobs-api,toolforge-deploy] allow using local harbor instance - https://phabricator.wikimedia.org/T355299 (10dcaro) [10:10:09] 10Toolforge (Toolforge iteration 03): [jobs-api] We are not overwriting the repo when generating the dev helm chart - https://phabricator.wikimedia.org/T355301 (10dcaro) [10:23:52] 10Toolforge (Toolforge iteration 03), 10Patch-For-Review: [jobs-api] We are not overwriting the repo when generating the dev helm chart - https://phabricator.wikimedia.org/T355301 (10CodeReviewBot) dcaro opened https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/27 only_helm_publish: allo... [10:46:02] 10Toolforge (Toolforge iteration 03), 10Toolforge Build Service: apt buildpack (Aptfile support): not installing dependencies of packages already present on the build image - https://phabricator.wikimedia.org/T353847 (10dcaro) [10:46:44] 10Toolforge (Toolforge iteration 03), 10Toolforge Build Service: [apt-buildpack] Not sourcing /layers/fagiani_apt/apt/.profile.d/000_apt.sh - https://phabricator.wikimedia.org/T355214 (10dcaro) 05Open→03In progress [10:48:33] 10Toolforge (Toolforge iteration 03), 10Toolforge Build Service: [apt-buildpack] Not sourcing /layers/fagiani_apt/apt/.profile.d/000_apt.sh - https://phabricator.wikimedia.org/T355214 (10dcaro) If the process is started using the procfile entrypoints, or the launcher the environment is set correctly: ` vagrant... [11:37:32] 10Toolforge: Remove 1 CPU limit for toolforge-jobs - https://phabricator.wikimedia.org/T327748 (10taavi) 05Open→03Resolved a:03taavi The default `pod_cpu` was changed to 3 recently. [12:03:20] 10Toolforge, 10cloud-services-team: setup email for toolforge.org and wmcloud.org like on {tools.,}wmflabs.org - https://phabricator.wikimedia.org/T278109 (10taavi) 05Open→03Resolved a:03taavi [12:21:16] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.remove_grid_node for tools-sgeexec-10-17 [12:24:44] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.add_k8s_node for a worker role in the tools cluster [12:30:24] 10VPS-project-Extdist: extdist should use object storage - https://phabricator.wikimedia.org/T355315 (10taavi) [12:38:08] !log taavi@cloudcumin1001 admin Added a new k8s worker tools-k8s-worker-101.tools.eqiad1.wikimedia.cloud to the cluster [12:38:08] !log taavi@cloudcumin1001 admin END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker role in the tools cluster [12:45:51] 10Toolforge (Toolforge iteration 03), 10Toolforge Build Service: [apt-buildpack] Not sourcing /layers/fagiani_apt/apt/.profile.d/000_apt.sh - https://phabricator.wikimedia.org/T355214 (10LucasWerkmeister) I see, then I probably didn’t get as far as trying the Procfile entries last time I was looking into this…... [12:48:57] 10Toolforge (Toolforge iteration 03), 10Toolforge Build Service: [apt-buildpack] Not sourcing /layers/fagiani_apt/apt/.profile.d/000_apt.sh - https://phabricator.wikimedia.org/T355214 (10taavi) I suspect we should just fix that in `jobs-api` to always use `launcher`? [12:49:46] 10cloud-services-team (FY2023/2024-Q1-Q2): openstack: eqiad1: cleanup leaks from the cloudlb migration - https://phabricator.wikimedia.org/T346630 (10taavi) 05Open→03Resolved [12:49:49] 10cloud-services-team (FY2023/2024-Q1-Q2), 10Goal: cloudlb: eqiad1: change openstack.eqiad1.wikimediacloud.org endpoint to point to cloudlb - https://phabricator.wikimedia.org/T346439 (10taavi) [12:52:35] (03Abandoned) 10Majavah: kubernetes: Use modern weld api [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/929009 (owner: 10Majavah) [12:52:57] (03Abandoned) 10Majavah: Add an option to disable NFS access [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/920259 (https://phabricator.wikimedia.org/T334081) (owner: 10Majavah) [12:53:39] (03Abandoned) 10Majavah: [WIP] Add html webservice type [software/tools-webservice] - 10https://gerrit.wikimedia.org/r/561753 (https://phabricator.wikimedia.org/T241817) (owner: 10Legoktm) [13:01:44] (03PS1) 10Majavah: Stop building buster based images [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/991594 (https://phabricator.wikimedia.org/T287900) [13:08:41] (CloudVPSDesignateLeaks) firing: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:09:16] 10Toolforge (Toolforge iteration 03), 10cloud-services-team, 10Kubernetes: Create Bookworm-based standalone webservice image - https://phabricator.wikimedia.org/T355231 (10taavi) p:05Triage→03Medium a:03taavi I hope most new users use https://wikitech.wikimedia.org/wiki/Help:Toolforge/Build_Service ins... [13:11:26] (03PS1) 10Majavah: Provide a standalone bookworm-web container [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/991595 (https://phabricator.wikimedia.org/T355231) [13:12:38] 10Cloud-VPS, 10cloud-services-team: cloudvirt1051 crashed - https://phabricator.wikimedia.org/T349109 (10taavi) 05Open→03Resolved [13:13:41] (CloudVPSDesignateLeaks) firing: (2) Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:15:31] 10Toolforge (Toolforge iteration 03), 10cloud-services-team, 10Kubernetes, 10Patch-For-Review: Create Bookworm-based standalone webservice image - https://phabricator.wikimedia.org/T355231 (10Count_Count) Unfortunately I can't use the Build Service since https://gitlab.wikimedia.org/countcount/xlinks is a... [13:20:16] 10Toolforge (Toolforge iteration 03), 10cloud-services-team, 10Kubernetes, 10Patch-For-Review: Create Bookworm-based standalone webservice image - https://phabricator.wikimedia.org/T355231 (10taavi) >>! In T355231#9468874, @Count_Count wrote: > AFAICS the [[ https://github.com/heroku/heroku-buildpack-multi... [13:40:53] 10Toolforge (Toolforge iteration 03), 10Patch-For-Review: [jobs-api] We are not overwriting the repo when generating the dev helm chart - https://phabricator.wikimedia.org/T355301 (10CodeReviewBot) dcaro merged https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/27 only_helm_publish: allo... [13:54:28] (PuppetCertificateAboutToExpire) firing: Puppet CA certificate Puppet CA: clouddb-services-puppetmaster-01.clouddb-services.eqiad.wmflabs is about to expire in 27d 23h 58m 43s - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [14:03:21] 10VPS-project-Extdist: extdist should use object storage - https://phabricator.wikimedia.org/T355315 (10Bawolff) I'm not really sure I see the benefit (to be clear, I'm not opposed either, I just don't see what difference it would make for this tool) [14:19:04] 10Tool-Global-user-contributions, 10Stewards-and-global-tools, 10Temporary accounts, 10XTools, and 2 others: [Design] Create user flows for different GUC scenarios - https://phabricator.wikimedia.org/T349902 (10KColeman-WMF) Based on conversations with Legal, this is the latest user flow which includes per... [14:26:58] 10Tool-Global-user-contributions, 10Stewards-and-global-tools, 10Temporary accounts, 10XTools, and 2 others: [Design] Create user flows for different GUC scenarios - https://phabricator.wikimedia.org/T349902 (10KColeman-WMF) [14:48:08] 10Toolforge Build Service: Support monorepos with the Multi Procfile buildpack - https://phabricator.wikimedia.org/T355329 (10Count_Count) [14:50:26] 10Toolforge (Toolforge iteration 03), 10cloud-services-team, 10Kubernetes, 10Patch-For-Review: Create Bookworm-based standalone webservice image - https://phabricator.wikimedia.org/T355231 (10Count_Count) >>! In T355231#9468882, @taavi wrote: > Is there a task for supporting that? Not AFAIK, I created T35... [15:40:05] 10Toolforge (Toolforge iteration 03), 10Toolforge Build Service: [apt-buildpack] Not sourcing /layers/fagiani_apt/apt/.profile.d/000_apt.sh - https://phabricator.wikimedia.org/T355214 (10dcaro) >>! In T355214#9468790, @taavi wrote: > I suspect we should just fix that in `jobs-api` to always use `launcher`? We... [15:41:31] 10Toolforge, 10cloud-services-team: php 8.2 crashes when using XMLReader - https://phabricator.wikimedia.org/T352886 (10taavi) This seems to happen after installing `php-tideways`: `lang=shell-session taavi@taavi-xmltest:~$ php somefile.php 2024-01-18 15:39:59 Start Test XMLReader 2024-01-18 15:39:59 End Test... [15:47:22] (HAProxyBackendUnavailable) firing: HAProxy service neutron-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [15:52:22] (HAProxyBackendUnavailable) resolved: HAProxy service neutron-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [16:54:28] (PuppetCertificateAboutToExpire) firing: Puppet CA certificate Puppet CA: clouddb-services-puppetmaster-01.clouddb-services.eqiad.wmflabs is about to expire in 27d 20h 58m 43s - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [16:55:51] 10Toolforge (Toolforge iteration 03): [jobs-api] We are not overwriting the repo when generating the dev helm chart - https://phabricator.wikimedia.org/T355301 (10dcaro) 05Open→03Resolved [17:03:29] 10Toolforge, 10cloud-services-team: php 8.2 crashes when using XMLReader - https://phabricator.wikimedia.org/T352886 (10Wurgl) @taavi: Great job in finding this problem. Now the mother of all questions: Do we need tideways? Does any tool use its functionality? [17:06:57] 10Toolforge (Toolforge iteration 03): [jobs-api,toolforge-deploy] allow using local harbor instance - https://phabricator.wikimedia.org/T355299 (10dcaro) 05Open→03In progress [17:07:07] 10Cloud-VPS, 10Toolforge (Toolforge iteration 03), 10cloud-services-team: Ensure Toolforge and Cloud VPS comply with Google's new email sender guidelines - https://phabricator.wikimedia.org/T354112 (10dcaro) 05Open→03In progress [17:07:17] 10Toolforge (Toolforge iteration 03), 10cloud-services-team, 10Kubernetes, 10Patch-For-Review: Toolforge k8s: Migrate workers to Containerd and Bookworm - https://phabricator.wikimedia.org/T284656 (10dcaro) 05Open→03In progress [17:07:24] 10Toolforge, 10cloud-services-team, 10Kubernetes: Migrate Toolforge Kubernetes hosts to Debian Bullseye or later - https://phabricator.wikimedia.org/T311908 (10dcaro) [17:07:29] 10Toolforge, 10cloud-services-team: Upgrade Toolforge Kubernetes to version 1.24 - https://phabricator.wikimedia.org/T307651 (10dcaro) [17:07:48] 10Toolforge (Toolforge iteration 03): [dev] Investigate lima-vm as an alternative to Vagrant for lima-kilo - https://phabricator.wikimedia.org/T354406 (10dcaro) 05Open→03In progress [17:07:50] 10Toolforge: [dev] find an alternative to Vagrant - https://phabricator.wikimedia.org/T348960 (10dcaro) [17:07:52] 10Toolforge, 10cloud-services-team: There are some files that I cannot view, delete, or do anything to - https://phabricator.wikimedia.org/T355022 (10taavi) 05In progress→03Resolved a:03taavi I was able to manage those via the NFS server. [17:08:19] 10Tool-spacemedia, 10Toolforge, 10cloud-services-team: NFS broken - Cannot build nor run my tool anymore - https://phabricator.wikimedia.org/T354581 (10taavi) 05Open→03Resolved a:03taavi I was able to delete `logs-old` via the NFS server. [17:28:50] 10Toolforge (Toolforge iteration 03): [jobs-api,toolforge-deploy] allow using local harbor instance - https://phabricator.wikimedia.org/T355299 (10CodeReviewBot) dcaro updated https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/178 jobs-api: use the local harbor for the local env [17:29:02] 10Toolforge (Toolforge iteration 03): [jobs-api,toolforge-deploy] allow using local harbor instance - https://phabricator.wikimedia.org/T355299 (10CodeReviewBot) dcaro updated https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/57 harbor: allow specifying the protocol [17:57:17] 10Tool-Global-user-contributions, 10Stewards-and-global-tools, 10Temporary accounts, 10XTools, and 2 others: [Design] Create user flows for different GUC scenarios - https://phabricator.wikimedia.org/T349902 (10Xaosflux) That flowchart is missing branches here: {F41697819} After Special:GUC it goes to a "... [18:20:37] (03CR) 10Amire80: [C: 04-1] "Thanks! It mostly looks good, but I have some small comments." [labs/tools/Isa] (m2c) - 10https://gerrit.wikimedia.org/r/990674 (https://phabricator.wikimedia.org/T299863) (owner: 10Eugene233) [18:23:21] (03CR) 10Amire80: [C: 04-1] Fix capitalization in some ISA messages (031 comment) [labs/tools/Isa] (m2c) - 10https://gerrit.wikimedia.org/r/990675 (https://phabricator.wikimedia.org/T354920) (owner: 10Eugene233) [18:24:48] (03CR) 10Amire80: [C: 03+1] Suggestions for improvements in some ISA messages (031 comment) [labs/tools/Isa] (m2c) - 10https://gerrit.wikimedia.org/r/990679 (https://phabricator.wikimedia.org/T354921) (owner: 10Eugene233) [18:32:13] (CloudVPSDesignateLeaks) resolved: (2) Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:22:12] (03PS2) 10Eugene233: Backslashes in some ISA messages [labs/tools/Isa] (m2c) - 10https://gerrit.wikimedia.org/r/990674 (https://phabricator.wikimedia.org/T299863) [19:22:14] 10Toolforge, 10cloud-services-team: No connectivity between cloudcontrol1005 and db replica hosts - https://phabricator.wikimedia.org/T355356 (10Andrew) [19:22:36] (03CR) 10Eugene233: Backslashes in some ISA messages (033 comments) [labs/tools/Isa] (m2c) - 10https://gerrit.wikimedia.org/r/990674 (https://phabricator.wikimedia.org/T299863) (owner: 10Eugene233) [19:24:38] (03CR) 10Eugene233: Suggestions for improvements in some ISA messages (032 comments) [labs/tools/Isa] (m2c) - 10https://gerrit.wikimedia.org/r/990679 (https://phabricator.wikimedia.org/T354921) (owner: 10Eugene233) [19:24:48] 10Toolforge, 10cloud-services-team: No connectivity between cloudcontrol1005 and db replica hosts - https://phabricator.wikimedia.org/T355356 (10Andrew) The title of this may be incorrect. I can telnet to 3306 on m5-master just fine, but launching mysql hangs [19:25:02] (03CR) 10Eugene233: [C: 03+2] Suggestions for improvements in some ISA messages [labs/tools/Isa] (m2c) - 10https://gerrit.wikimedia.org/r/990679 (https://phabricator.wikimedia.org/T354921) (owner: 10Eugene233) [19:25:28] (03Merged) 10jenkins-bot: Suggestions for improvements in some ISA messages [labs/tools/Isa] (m2c) - 10https://gerrit.wikimedia.org/r/990679 (https://phabricator.wikimedia.org/T354921) (owner: 10Eugene233) [19:27:09] (03CR) 10Amire80: [C: 03+1] Backslashes in some ISA messages [labs/tools/Isa] (m2c) - 10https://gerrit.wikimedia.org/r/990674 (https://phabricator.wikimedia.org/T299863) (owner: 10Eugene233) [19:27:40] (03CR) 10Amire80: [C: 03+1] "Thanks! <3" [labs/tools/Isa] (m2c) - 10https://gerrit.wikimedia.org/r/990679 (https://phabricator.wikimedia.org/T354921) (owner: 10Eugene233) [19:34:28] 10Grid-Engine-to-K8s-Migration: Migrate sbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320030 (10Tacsipacsi) a:05zhuyifei1999→03None Since the grid jobs were stopped on December 18, most jobs that were running on the grid engine at that time have been migrated to... [19:34:46] (03PS3) 10Eugene233: Fix capitalization in some ISA messages [labs/tools/Isa] (m2c) - 10https://gerrit.wikimedia.org/r/990675 (https://phabricator.wikimedia.org/T354920) [19:36:01] (03CR) 10Eugene233: Fix capitalization in some ISA messages (031 comment) [labs/tools/Isa] (m2c) - 10https://gerrit.wikimedia.org/r/990675 (https://phabricator.wikimedia.org/T354920) (owner: 10Eugene233) [19:41:23] (03CR) 10Amire80: [C: 03+1] Fix capitalization in some ISA messages [labs/tools/Isa] (m2c) - 10https://gerrit.wikimedia.org/r/990675 (https://phabricator.wikimedia.org/T354920) (owner: 10Eugene233) [19:50:12] 10Toolforge, 10cloud-services-team: No connectivity between cloudcontrol1005 and db replica hosts - https://phabricator.wikimedia.org/T355356 (10taavi) This seems to be ToolsDB (`185.15.56.15:3306`) specific and probably due to outdated grants. >>! In T355356#9470535, @Andrew wrote: > The title of this may b... [19:54:28] (PuppetCertificateAboutToExpire) firing: Puppet CA certificate Puppet CA: clouddb-services-puppetmaster-01.clouddb-services.eqiad.wmflabs is about to expire in 27d 17h 58m 43s - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [19:55:07] 10Toolforge, 10cloud-services-team: No connectivity between cloudcontrol1005 and db replica hosts - https://phabricator.wikimedia.org/T355356 (10taavi) >>! In T355356#9470642, @taavi wrote: > This seems to be ToolsDB (`185.15.56.15:3306`) specific and probably due to outdated grants. No, does not seem to be a... [19:58:16] 10Toolforge, 10cloud-services-team: No connectivity between cloudcontrol1005 and db replica hosts - https://phabricator.wikimedia.org/T355356 (10taavi) ` Jan 18 19:56:49 cloudcontrol1005 maintain-dbusers[2707856]: INFO [root._create_accounts_on_host:953] Created account in 185.15.56.15:3306 for user mhurd ` Th... [20:25:53] 10Toolforge, 10cloud-services-team: No connectivity between cloudcontrol1005 and db replica hosts - https://phabricator.wikimedia.org/T355356 (10Andrew) this was a conflict between galera config; fixed by adding --port=3306 to the command line. [20:26:01] 10Toolforge, 10cloud-services-team: Maintain-dbusers troubleshooting for tools.sbot - https://phabricator.wikimedia.org/T355356 (10taavi) [20:51:50] (ProbeDown) firing: Service tools-static-14:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-14:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:56:50] (ProbeDown) resolved: Service tools-static-14:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-14:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:32:54] 10Tool-Global-user-contributions, 10Stewards-and-global-tools, 10Temporary accounts, 10XTools, and 2 others: [Design] Create user flows for different GUC scenarios - https://phabricator.wikimedia.org/T349902 (10KColeman-WMF) Hi @Xaosflux Thank you for spotting that. I've uploaded an expanded flow which sho... [21:34:48] 10Tool-Global-user-contributions, 10Stewards-and-global-tools, 10Temporary accounts, 10XTools, and 2 others: [Design] Create user flows for different GUC scenarios - https://phabricator.wikimedia.org/T349902 (10KColeman-WMF) [22:34:34] 10wikitech.wikimedia.org, 10LDAP: Change my username on Wikitech - https://phabricator.wikimedia.org/T355249 (10Tgr) Doğu ended up registering a new account so I think this can be closed? [22:35:37] 10wikitech.wikimedia.org, 10LDAP: Change my username on Wikitech - https://phabricator.wikimedia.org/T355249 (10Dogu) 05Stalled→03Resolved a:03Dogu [22:54:28] (PuppetCertificateAboutToExpire) firing: Puppet CA certificate Puppet CA: clouddb-services-puppetmaster-01.clouddb-services.eqiad.wmflabs is about to expire in 27d 14h 58m 43s - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [23:05:31] (ToolsToolsDBReplicationLagIsTooHigh) firing: ToolsDB replication on tools-db-2 is lagging behind the primary, the current lag is 3700 - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationLagIsTooHigh [23:09:56] (ProbeDown) firing: Service tools-k8s-haproxy-3:30000 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-3:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:14:56] (ProbeDown) resolved: Service tools-k8s-haproxy-3:30000 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-3:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown