[00:03:39] RESOLVED: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:05:06] FIRING: [2x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_tool_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:10:06] FIRING: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:15:06] RESOLVED: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:47:16] FIRING: [3x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:52:16] FIRING: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:57:16] RESOLVED: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:58:06] FIRING: ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_toolserver_org_redirects_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:02:31] RESOLVED: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [01:07:18] 10Tool-eswstool2: Failed uploads to Commons and the various kinds of errors they throw - https://phabricator.wikimedia.org/T384101 (10Ninovolador) 03NEW [01:07:56] 10Tool-eswstool2: Failed uploads to Commons and the various kinds of errors they throw - https://phabricator.wikimedia.org/T384101#10473477 (10Ninovolador) p:05Triage→03Low [01:11:05] 10Tool-eswstool2: Failed uploads to Commons and the various kinds of errors they throw - https://phabricator.wikimedia.org/T384101#10473485 (10Ninovolador) https://archive.org/details/tesorodelalengua00covauoft https://archive.org/download/tesorodelalengua00covauoft/tesorodelalengua00covauoft.pdf 122.7M After... [01:24:19] 10Tool-eswstool2: Failed uploads to Commons and the various kinds of errors they throw - https://phabricator.wikimedia.org/T384101#10473489 (10Ninovolador) http://bdh.bne.es/bnesearch/detalle/bdh0000207265 https://bdh-rd.bne.es/high.raw?id=bdh0000207265&name=00000001.original.pdf 312M chunk_size = 1024 * 1024 f... [01:26:48] FIRING: [2x] PuppetDisabled: Puppet disabled on cloudcontrol2004-dev:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=wmcs&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled [01:27:08] 10Tool-eswstool2: Failed uploads to Commons and the various kinds of errors they throw - https://phabricator.wikimedia.org/T384101#10473490 (10Ninovolador) https://archive.org/details/obrascompletasyc19ameguoft https://archive.org/download/obrascompletasyc19ameguoft/obrascompletasyc19ameguoft.pdf 122.7M chunk_si... [01:37:03] 06cloud-services-team, 10Toolforge: Toolforge jobs: increased exit code 137 rate since 2024-12-14 - https://phabricator.wikimedia.org/T382865#10473504 (10JJMC89) [01:46:27] 06cloud-services-team, 10Toolforge: [jobs-emailer] duplicate failure emails - https://phabricator.wikimedia.org/T382866#10473505 (10JJMC89) [03:00:39] FIRING: [2x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:05:39] RESOLVED: [2x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:49:06] FIRING: [2x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_tool_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [03:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [03:54:06] RESOLVED: [2x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_tool_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:00:56] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:27:43] FIRING: InstanceDown: Project tools instance tools-k8s-worker-nfs-7 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [04:32:43] RESOLVED: InstanceDown: Project tools instance tools-k8s-worker-nfs-7 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [04:43:05] 06cloud-services-team, 10Toolforge: Toolforge jobs: increased exit code 137 rate since 2024-12-14 - https://phabricator.wikimedia.org/T382865#10473599 (10JJMC89) [05:26:48] FIRING: [2x] PuppetDisabled: Puppet disabled on cloudcontrol2004-dev:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=wmcs&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled [06:07:43] FIRING: InstanceDown: Project tools instance tools-k8s-worker-nfs-7 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [06:12:43] RESOLVED: InstanceDown: Project tools instance tools-k8s-worker-nfs-7 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [06:14:43] FIRING: InstanceDown: Project tools instance tools-k8s-worker-nfs-7 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [06:19:43] RESOLVED: InstanceDown: Project tools instance tools-k8s-worker-nfs-7 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [08:18:35] 10Tools: geohack tool crashing repeately and quickly enough to trigger CrashLoopBackOff - https://phabricator.wikimedia.org/T384092#10473663 (10Bugreporter) [08:49:00] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [08:50:54] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 29350 bytes in 1.644 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [09:26:49] FIRING: [2x] PuppetDisabled: Puppet disabled on cloudcontrol2004-dev:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=wmcs&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled [09:32:43] FIRING: InstanceDown: Project tools instance tools-k8s-worker-nfs-7 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [09:37:43] RESOLVED: InstanceDown: Project tools instance tools-k8s-worker-nfs-7 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [09:44:53] 06cloud-services-team, 10Toolforge: Toolforge jobs: increased exit code 137 rate since 2024-12-14 - https://phabricator.wikimedia.org/T382865#10473682 (10JJMC89) [09:45:01] 10Tool-techcontribs: Add reporting on activity on technical mailing lists - https://phabricator.wikimedia.org/T384112 (10taavi) 03NEW [09:46:00] 10Tool-techcontribs: Add reporting on SAL entries - https://phabricator.wikimedia.org/T384113 (10taavi) 03NEW [09:48:17] (03PS1) 10Majavah: templates: Fix tool repositories not appearing [labs/striker] - 10https://gerrit.wikimedia.org/r/1112323 (https://phabricator.wikimedia.org/T384068) [10:17:46] 06cloud-services-team, 10Openstack-Magnum: CSI Cinder issues causing periodic failures on Magnum cluster - https://phabricator.wikimedia.org/T383560#10473725 (10Proc) The libvert error seems to be fixed by T383583 @rook any ideas on what to do about the main bit of the issue? The cluster still seems to be ino... [10:21:38] 06cloud-services-team, 10Openstack-Magnum: CSI Cinder issues causing periodic failures on Magnum cluster - https://phabricator.wikimedia.org/T383560#10473728 (10Proc) I'm tempted to recreate the cluster, but maybe that wouldn't fix whatever the underlying issue is, and want to get some thoughts before I delete... [10:53:16] 10ToolforgeBundle: Upgrade to Symfony 7 - https://phabricator.wikimedia.org/T361554#10473748 (10Samwilson) [10:53:32] 10ToolforgeBundle: Upgrade ToolforgeBundle to Symfony 7 - https://phabricator.wikimedia.org/T361554#10473749 (10Samwilson) [11:19:48] 10Tool-eswstool2: Failed uploads to Commons and the various kinds of errors they throw - https://phabricator.wikimedia.org/T384101#10473754 (10Ninovolador) https://archive.org/details/A253315 https://archive.org/download/A253315/A253315.pdf59.3M chunk_size = 1024 * 1024 Fails without warnings, with pywikibot.e... [11:29:43] FIRING: InstanceDown: Project tools instance tools-k8s-worker-nfs-7 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [11:34:43] RESOLVED: InstanceDown: Project tools instance tools-k8s-worker-nfs-7 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [12:21:53] (03CR) 10Majavah: [C:03+2] templates: Fix tool repositories not appearing [labs/striker] - 10https://gerrit.wikimedia.org/r/1112323 (https://phabricator.wikimedia.org/T384068) (owner: 10Majavah) [12:23:09] (03Merged) 10jenkins-bot: templates: Fix tool repositories not appearing [labs/striker] - 10https://gerrit.wikimedia.org/r/1112323 (https://phabricator.wikimedia.org/T384068) (owner: 10Majavah) [12:25:57] 10Tools: geohack tool crashing repeately and quickly enough to trigger CrashLoopBackOff - https://phabricator.wikimedia.org/T384092#10473783 (10LucasWerkmeister) I’ve started the service again, and so far it seems to be up. If it’s normal for the tool to receive a lot of traffic, maybe it would make sense to run... [12:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:56:07] 10Striker: Striker (toolsadmin) doesn’t show GitLab repositories of (some?) tools - https://phabricator.wikimedia.org/T384068#10473790 (10taavi) a:03taavi [12:56:36] 10Striker: Striker (toolsadmin) doesn’t show GitLab repositories of (some?) tools - https://phabricator.wikimedia.org/T384068#10473791 (10taavi) 05Open→03Resolved [13:00:06] 10Striker: Striker (toolsadmin) doesn’t show GitLab repositories of (some?) tools - https://phabricator.wikimedia.org/T384068#10473793 (10LucasWerkmeister) Thanks \o/ {F58226535} [13:05:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:10:50] (03PS1) 10Majavah: templates: Fix link to tool pages [labs/striker] - 10https://gerrit.wikimedia.org/r/1112329 [13:11:01] (03CR) 10Majavah: [C:03+2] templates: Fix link to tool pages [labs/striker] - 10https://gerrit.wikimedia.org/r/1112329 (owner: 10Majavah) [13:13:41] (03Merged) 10jenkins-bot: templates: Fix link to tool pages [labs/striker] - 10https://gerrit.wikimedia.org/r/1112329 (owner: 10Majavah) [13:26:49] FIRING: [2x] PuppetDisabled: Puppet disabled on cloudcontrol2004-dev:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=wmcs&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled [14:23:16] (03PS1) 10Majavah: templates: Use Codex for tool project/repo pages [labs/striker] - 10https://gerrit.wikimedia.org/r/1112330 (https://phabricator.wikimedia.org/T380114) [15:03:39] 10Tools: geohack tool crashing repeately and quickly enough to trigger CrashLoopBackOff - https://phabricator.wikimedia.org/T384092#10473824 (10Edgars2007) Not saying this is the best strategy, but adding something like `error_reporting(E_ALL & ~E_NOTICE & ~E_WARNING);` (to not log notices and warnings) in php e... [15:08:07] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-7 [15:12:38] 10Tools: geohack tool crashing repeately and quickly enough to trigger CrashLoopBackOff - https://phabricator.wikimedia.org/T384092#10473826 (10LucasWerkmeister) I disagree. The top notices aren’t difficult to fix: `lang=diff diff --git a/public_html/geohack.php b/public_html/geohack.php index f913292764..02c51... [15:12:55] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-7 [15:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:24:21] 06cloud-services-team, 10Cloud-VPS: CloudVPSDesignateLeaks alert is flapping - https://phabricator.wikimedia.org/T384118 (10taavi) 03NEW [15:24:59] 06cloud-services-team, 10Cloud-VPS: Cloud VPS Designate setup improvements - https://phabricator.wikimedia.org/T340446#10473841 (10taavi) 05Open→03Resolved Closing since there's no open subtasks. [15:35:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:11:02] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [16:11:54] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 29350 bytes in 1.804 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [16:16:03] 06cloud-services-team, 10Toolforge, 06DBA: User has exceeded the 'max_user_connections' (10) on Toolforge DB replicas - https://phabricator.wikimedia.org/T384119 (10MBH) 03NEW [16:16:57] 06cloud-services-team, 10Toolforge, 06DBA: User has exceeded the 'max_user_connections' (10) on Toolforge DB replicas - https://phabricator.wikimedia.org/T384119#10473858 (10MBH) [16:18:53] FIRING: PuppetCertificateAboutToExpire: Puppet CA certificate pontoon-puppet-01.monitoring.eqiad.wmflabs is about to expire in 27d 23h 58m 27s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [17:01:49] FIRING: [2x] PuppetDisabled: Puppet disabled on cloudcontrol2004-dev:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=wmcs&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled [17:16:49] RESOLVED: PuppetDisabled: Puppet disabled on cloudcontrol2004-dev:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=wmcs&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled [17:29:52] 10Tool-ranker, 06translatewiki.net, 10LPL Essential (LPL Essential 2024 Nov-Dec), 07Unplanned-Sprint-Work: Add Ranker to translatewiki.net - https://phabricator.wikimedia.org/T384061#10473888 (10Nikerabbit) [17:29:53] 06cloud-services-team, 10Data-Services, 06DBA: User has exceeded the 'max_user_connections' (10) on Toolforge DB replicas - https://phabricator.wikimedia.org/T384119#10473890 (10JJMC89) [17:29:55] 10Tool-ranker, 06translatewiki.net, 10LPL Essential (LPL Essential 2024 Nov-Dec), 07Unplanned-Sprint-Work: Add Ranker to translatewiki.net - https://phabricator.wikimedia.org/T384061#10473891 (10Nikerabbit) p:05Triage→03Medium [17:31:16] 06cloud-services-team, 06DBA, 10Data-Services (Quota-requests): User has exceeded the 'max_user_connections' (10) on Toolforge DB replicas - https://phabricator.wikimedia.org/T384119#10473893 (10JJMC89) [17:38:28] 06cloud-services-team, 10Toolforge: Toolforge jobs: increased exit code 137 rate since 2024-12-14 - https://phabricator.wikimedia.org/T382865#10473894 (10JJMC89) [17:39:14] 06cloud-services-team, 10Toolforge: [jobs-emailer] duplicate failure emails - https://phabricator.wikimedia.org/T382866#10473895 (10JJMC89) [18:04:32] 10wikitech.wikimedia.org, 10Wikimedia-Site-requests: Drop local 'OAuth administrators' group from Wikitech - https://phabricator.wikimedia.org/T384122 (10taavi) 03NEW [18:17:14] 10wikitech.wikimedia.org, 10Wikimedia-Site-requests, 13Patch-For-Review: Drop local 'OAuth administrators' group from Wikitech - https://phabricator.wikimedia.org/T384122#10473912 (10taavi) a:03taavi [18:57:02] 10wikitech.wikimedia.org: Remove OATH validators user group in Wikitech - https://phabricator.wikimedia.org/T384123 (10Bugreporter) 03NEW [19:25:35] 10wikitech.wikimedia.org, 13Patch-For-Review: Remove OATH validators user group in Wikitech - https://phabricator.wikimedia.org/T384123#10473953 (10taavi) a:03taavi [19:27:43] 10Striker: Add view showing all tools maintained by a user - https://phabricator.wikimedia.org/T384125 (10taavi) 03NEW [19:39:10] 06cloud-services-team, 10Data-Services (Quota-requests): User has exceeded the 'max_user_connections' (10) on Toolforge DB replicas - https://phabricator.wikimedia.org/T384119#10473966 (10Ladsgroup) We (DBAs) don't maintain toolforge databases. One rather easy solution would be not to close the connection but... [19:45:05] 10Tool-eswstool2: Failed uploads to Commons and the various kinds of errors they throw - https://phabricator.wikimedia.org/T384101#10473970 (10Ninovolador) [19:52:25] 10Striker: Add view showing all tools maintained by a user - https://phabricator.wikimedia.org/T384125#10473977 (10Bugreporter) The LDAP tool already provided such information via "service groups" though it is buggy ({T378049}). [20:01:02] 06cloud-services-team, 10Toolforge: maintain-kubeusers should manage tool observer access - https://phabricator.wikimedia.org/T384126 (10taavi) 03NEW [22:24:02] 06cloud-services-team, 10Data-Services (Quota-requests): User has exceeded the 'max_user_connections' (10) on Toolforge DB replicas - https://phabricator.wikimedia.org/T384119#10474079 (10bd808) @MBH A possibly better way to manage the connections you are using is to reuse the same connection for all wikis in... [22:24:58] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10474081 (10IKhitron) |**Wikitech account/LDAP:**| IKhitron| |**SUL account**| IKhitron| |**Account linked on [[ https://idm.wikimedia.org/ | IDM ]]** |N| |**I have visited [[ https://wikitec... [23:20:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks