[00:40:22] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic, 13Patch-For-Review: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10537324 (10Bugreporter) > For a number of these it looks like the migration process led to some folks accidentally creating a n... [01:12:17] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic, 13Patch-For-Review: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10537529 (10bd808) >>! In T161859#10537324, @Bugreporter wrote: >> For a number of these it looks like the migration process led... [01:18:54] Change on 12wikitech.wikimedia.org a page News/2024 Migrating Wikitech Account to SUL was modified, changed by BryanDavis link https://wikitech.wikimedia.org/w/index.php?diff=2271113 edit summary: /* Timeline */ 2025-02-10 renames done; moved "Wikitech config changed to only use Wikimedia SUL accounts." to 2025-02-24 because of notes Taavi gave on config change patch [01:29:36] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic, 13Patch-For-Review: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10537548 (10Bugreporter) >>! In T161859#10468972, @Bugreporter wrote: >>>! In T161859#10467817, @bd808 wrote: >>>>! In T161859#1... [01:29:44] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic, 13Patch-For-Review: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10537549 (10SunAfterRain) Hello, I recall that in November last year, I did attach the Wikitech local account (ID: 30300) to the... [01:39:24] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic, 13Patch-For-Review: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10537553 (10Bugreporter) I have checked accounts of the current 20 crats. Two of them are not yet attached to SUL: * https://wik... [01:40:10] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic, 13Patch-For-Review: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10537554 (10bd808) >>! In T161859#10537548, @Bugreporter wrote: >>>! In T161859#10468972, @Bugreporter wrote: >> Another potenti... [01:43:32] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic, 13Patch-For-Review: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10537555 (10Bugreporter) >>! In T161859#10537554, @bd808 wrote: >>>! In T161859#10537548, @Bugreporter wrote: >>>>! In T161859#1... [02:10:33] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic, 13Patch-For-Review: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10537565 (10Bugreporter) Sysops: * https://wikitech.wikimedia.org/wiki/Special:Contributions/ADenisse-WMF - this user is rename... [02:26:45] FIRING: [2x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_toolserver_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:31:45] FIRING: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [02:35:50] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic, 13Patch-For-Review: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10537571 (10Bugreporter) Autopatrollers: * https://wikitech.wikimedia.org/wiki/Special:Contributions/BBlack_(WMF) - renamed but... [02:36:45] RESOLVED: [4x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [04:05:01] 06cloud-services-team, 10Toolforge: ConnectTimeoutError when trying to pip install - https://phabricator.wikimedia.org/T386059 (10Ederporto) 03NEW [06:23:17] 10wikitech.wikimedia.org: Decide what to do with SUL attached Wikitech accounts that Bitu associates with a different SUL account - https://phabricator.wikimedia.org/T386026#10537714 (10Raymond) Hi, please detach 'Raimond Spekking' from SUL, rename it to 'Raymond', and reattach to SUL. and sorry for confusing... [06:23:18] FIRING: KernelErrors: Server cloudvirt1047 logged kernel errors - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/KernelErrors - https://grafana.wikimedia.org/d/b013af4c-d405-4d9f-85d4-985abb3dec0c/wmcs-kernel-errors?orgId=1&var-instance=cloudvirt1047 - https://alerts.wikimedia.org/?q=alertname%3DKernelErrors [07:20:41] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic, 13Patch-For-Review: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10537757 (10Ameisenigel) I am not sure what has happened with my account now, but it is a complete mess. Months ago I have attac... [07:26:56] 10wikitech.wikimedia.org: Decide what to do with SUL attached Wikitech accounts that Bitu associates with a different SUL account - https://phabricator.wikimedia.org/T386026#10537759 (10Tobi_WMDE_SW) Hi, sorry, seems there was indeed some confusion, so please detach 'Tobias Gritschacher' from SUL, rename it to... [08:05:55] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic, 13Patch-For-Review: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10537785 (10Bugreporter) By reviewing rename log some other apperent errorous matches are: * https://wikitech.wikimedia.org/wiki... [08:37:48] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10537812 (10Zache) |**Wikitech account/LDAP:**| zache-tool| |**SUL account**| Zache| |**Account linked on [[ https://idm.wikimedia.org/ | IDM ]]** | Y |**I have visited [[ https://wikitech.wi... [09:01:19] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Unable to persistently set fs.inotify.max_user_instances and fs.inotify.max_user_watches - https://phabricator.wikimedia.org/T385530#10537851 (10taavi) >>! In T385530#10536491, @Andrew wrote: >>>! In T385530#10536484, @taavi wrote: >> Are the Puppet fai... [09:35:18] 10wikitech.wikimedia.org: Decide what to do with SUL attached Wikitech accounts that Bitu associates with a different SUL account - https://phabricator.wikimedia.org/T386026#10537932 (10Aklapper) Hi, please detach `Aklapper` from SUL, rename it to `AKlapper (WMF)`, and reattach to SUL (per the appreciated [heads... [09:56:29] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic, 13Patch-For-Review: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10537957 (10Aklapper) >>! In T161859#10537555, @Bugreporter wrote: > @Aklapper, are you able to export such a mapping? i.e. A li... [10:23:18] FIRING: KernelErrors: Server cloudvirt1047 logged kernel errors - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/KernelErrors - https://grafana.wikimedia.org/d/b013af4c-d405-4d9f-85d4-985abb3dec0c/wmcs-kernel-errors?orgId=1&var-instance=cloudvirt1047 - https://alerts.wikimedia.org/?q=alertname%3DKernelErrors [10:27:32] FIRING: ToolsToolsDBReplicationLagIsTooHigh: ToolsDB replication on tools-db-5 is lagging behind the primary, the current lag is 110707 - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationLagIsTooHigh [11:23:21] 06cloud-services-team, 10Cloud-VPS, 06DC-Ops, 10ops-eqiad: [Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 4 on cloudvirt1047 - https://phabricator.wikimedia.org/T386083 (10Andrew) 03NEW [11:23:25] 10wikitech.wikimedia.org: Decide what to do with SUL attached Wikitech accounts that Bitu associates with a different SUL account - https://phabricator.wikimedia.org/T386026#10538456 (10Ollie.Shotton_WMDE) Hi, Thanks for flagging that I managed to make a cock up of this migration! 🙈 Please detach `Ollie Shotto... [11:24:03] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1047' (T386083) [11:24:10] T386083: [Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 4 on cloudvirt1047 - https://phabricator.wikimedia.org/T386083 [11:24:31] !log andrew@cloudcumin1001 admin END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1047' (T386083) [11:32:21] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1047.eqiad.wmnet' (T386083) [11:32:27] T386083: [Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 4 on cloudvirt1047 - https://phabricator.wikimedia.org/T386083 [11:40:55] 06cloud-services-team, 10Cloud-VPS, 06DC-Ops, 10ops-eqiad, 06SRE: [Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 4 on cloudvirt1047 - https://phabricator.wikimedia.org/T386083#10538485 (10fnegri) > It has been corrected by h/w and requires no further action Does this mean we s... [11:47:27] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1047.eqiad.wmnet' (T386083) [11:47:34] T386083: [Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 4 on cloudvirt1047 - https://phabricator.wikimedia.org/T386083 [11:50:21] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic, 13Patch-For-Review: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10538508 (10Bugreporter) >>! In T161859#10537957, @Aklapper wrote: >>>! In T161859#10537555, @Bugreporter wrote: >> @Aklapper, a... [12:24:52] 10wikitech.wikimedia.org: ☂ Wikitech account linking and SUL error reporting - https://phabricator.wikimedia.org/T376267#10538594 (10Ladsgroup) >>! In T376267#10537812, @Zache wrote: > |**Wikitech account/LDAP:**| zache-tool| > |**SUL account**| Zache| > |**Account linked on [[ https://idm.wikimedia.org/ | IDM... [12:28:16] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic, 13Patch-For-Review: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10538602 (10Ladsgroup) >>! In T161859#10537957, @Aklapper wrote: >>>! In T161859#10537555, @Bugreporter wrote: >> @Aklapper, are... [12:28:56] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic, 13Patch-For-Review: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10538603 (10Ladsgroup) >>! In T161859#10537757, @Ameisenigel wrote: > I am not sure what has happened with my account now, but i... [12:31:42] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic, 13Patch-For-Review: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10538605 (10Ladsgroup) Fixed. [12:39:10] 10wikitech.wikimedia.org: Decide what to do with SUL attached Wikitech accounts that Bitu associates with a different SUL account - https://phabricator.wikimedia.org/T386026#10538609 (10Ladsgroup) >>! In T386026#10537714, @Raymond wrote: > Hi, > > please detach 'Raimond Spekking' from SUL, rename it to 'Raymond... [12:44:06] 06cloud-services-team, 10Cloud-VPS, 06DC-Ops, 10ops-eqiad, 06SRE: [Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 4 on cloudvirt1047 - https://phabricator.wikimedia.org/T386083#10538613 (10Andrew) >>! In T386083#10538485, @fnegri wrote: >> It has been corrected by h/w and requir... [12:45:32] 10wikitech.wikimedia.org: Decide what to do with SUL attached Wikitech accounts that Bitu associates with a different SUL account - https://phabricator.wikimedia.org/T386026#10538615 (10Ollie.Shotton_WMDE) Thanks! Can accounts be deleted? I notice https://meta.wikimedia.org/wiki/Special:CentralAuth?target=Ollie... [12:46:35] 06cloud-services-team, 10Cloud-VPS, 06DC-Ops, 10ops-eqiad, 06SRE: [Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 4 on cloudvirt1047 - https://phabricator.wikimedia.org/T386083#10538626 (10Andrew) Browsing stack overflow implies that this is likely an impending HW issue [12:47:12] 10wikitech.wikimedia.org: Decide what to do with SUL attached Wikitech accounts that Bitu associates with a different SUL account - https://phabricator.wikimedia.org/T386026#10538628 (10Ladsgroup) >>! In T386026#10538615, @Ollie.Shotton_WMDE wrote: > Thanks! > > Can accounts be deleted? I notice https://meta.wi... [13:13:43] (03open) 10group_203_bot_4866fc124f4b41659f667468a6115cf3: jobs-api: bump to 0.0.349-20250205000226-031f1bc7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/668 [13:17:14] (03PS1) 10CDanis: ceph: add aux-k8s-eqiad-csi user [labs/private] - 10https://gerrit.wikimedia.org/r/1118806 (https://phabricator.wikimedia.org/T380541) [13:29:35] 10wikitech.wikimedia.org: Decide what to do with SUL attached Wikitech accounts that Bitu associates with a different SUL account - https://phabricator.wikimedia.org/T386026#10538747 (10DMburugu) Please detach 'DMburugu' from SUL, rename it to 'DMburugu (WMF) [13:29:57] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic, 13Patch-For-Review: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10538749 (10Bugreporter) A list of not yet attached accounts with edit in Wikitech: P73433 Number of accounts with * 1 edit: 120... [13:39:24] (03PS2) 10CDanis: ceph: add aux-k8s-eqiad-csi user [labs/private] - 10https://gerrit.wikimedia.org/r/1118806 (https://phabricator.wikimedia.org/T380541) [13:39:46] (03PS3) 10CDanis: ceph: add aux-k8s-csi-rbd user [labs/private] - 10https://gerrit.wikimedia.org/r/1118806 (https://phabricator.wikimedia.org/T380541) [13:46:29] (03CR) 10Btullis: [C:03+1] ceph: add aux-k8s-csi-rbd user [labs/private] - 10https://gerrit.wikimedia.org/r/1118806 (https://phabricator.wikimedia.org/T380541) (owner: 10CDanis) [13:47:00] 10wikitech.wikimedia.org: Decide what to do with SUL attached Wikitech accounts that Bitu associates with a different SUL account - https://phabricator.wikimedia.org/T386026#10538813 (10Ladsgroup) >>! In T386026#10538747, @DMburugu wrote: > Please detach 'DMburugu' from SUL, rename it to 'DMburugu (WMF) {{done}} [13:57:32] 06cloud-services-team, 10Striker, 10Bitu, 06Infrastructure-Foundations, 13Patch-For-Review: Move Striker to Bitu username validation API - https://phabricator.wikimedia.org/T364605#10538865 (10Arendpieter) Is this issue still in progress, or has it already been completed? [13:59:17] (03CR) 10CDanis: [V:03+2 C:03+2] ceph: add aux-k8s-csi-rbd user [labs/private] - 10https://gerrit.wikimedia.org/r/1118806 (https://phabricator.wikimedia.org/T380541) (owner: 10CDanis) [14:08:38] FIRING: CloudVPSBastionlessUsers: Detected 1 cloud-vps users not in Bastion project - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Users_not_in_bastion_project - https://logstash.wikimedia.org/app/dashboards#/view/3ef008b0-c871-11eb-ad54-8bb5fcb640c0?_g=h@e78830b&_a=h@21715bf - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSBastionlessUsers [14:09:01] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/5 [14:09:30] 10wikitech.wikimedia.org: Decide what to do with SUL attached Wikitech accounts that Bitu associates with a different SUL account - https://phabricator.wikimedia.org/T386026#10538952 (10dcausse) Please detach 'DCausse' from SUL, rename it to 'DCausse (WMF)', and reattach to SUL. [14:18:38] RESOLVED: CloudVPSBastionlessUsers: Detected 1 cloud-vps users not in Bastion project - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Users_not_in_bastion_project - https://logstash.wikimedia.org/app/dashboards#/view/3ef008b0-c871-11eb-ad54-8bb5fcb640c0?_g=h@e78830b&_a=h@21715bf - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSBastionlessUsers [14:19:30] 10Tool-ranker, 06translatewiki.net, 10LPL Essential (LPL Essential 2024 Nov-Jan), 13Patch-For-Review, 07Unplanned-Sprint-Work: Add Ranker to translatewiki.net - https://phabricator.wikimedia.org/T384061#10539015 (10abi_) 05In progress→03Resolved I've moved the project to the proper namespace - Wi... [14:26:39] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: openstack: keystone may be failing to add users to the bastion project - https://phabricator.wikimedia.org/T379550#10539123 (10Andrew) p:05High→03Medium [14:37:18] 10wikitech.wikimedia.org: Decide what to do with SUL attached Wikitech accounts that Bitu associates with a different SUL account - https://phabricator.wikimedia.org/T386026#10539183 (10Ladsgroup) >>! In T386026#10538952, @dcausse wrote: > Please detach 'DCausse' from SUL, rename it to 'DCausse (WMF)', and reatt... [14:58:59] (03CR) 10Jforrester: "check experimental" [labs/tools/fileprotectionsync] - 10https://gerrit.wikimedia.org/r/1071364 (owner: 10Legoktm) [15:03:19] (03CR) 10Jforrester: "check experimental" [labs/tools/wikiinfo] - 10https://gerrit.wikimedia.org/r/949071 (owner: 10Krinkle) [15:03:36] (03CR) 10Jforrester: "check experimental" [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/1118503 (owner: 10L10n-bot) [15:03:47] (03CR) 10Jforrester: "check experimental" [labs/tools/usage] - 10https://gerrit.wikimedia.org/r/908978 (owner: 10Krinkle) [15:05:11] (03CR) 10Jforrester: "check experimental" [labs/tools/orphantalk] - 10https://gerrit.wikimedia.org/r/1117889 (owner: 10L10n-bot) [15:05:35] (03CR) 10Jforrester: "check experimental" [labs/tools/intuition] - 10https://gerrit.wikimedia.org/r/1118495 (owner: 10L10n-bot) [15:05:38] (03CR) 10Jforrester: "check experimental" [labs/tools/intuition-web] - 10https://gerrit.wikimedia.org/r/1106310 (owner: 10L10n-bot) [15:05:43] (03CR) 10Jforrester: "check experimental" [labs/tools/orphantalk] - 10https://gerrit.wikimedia.org/r/1117889 (owner: 10L10n-bot) [15:06:54] (03CR) 10Jforrester: "check experimental" [labs/tools/force-rebase] - 10https://gerrit.wikimedia.org/r/881434 (owner: 10DannyS712) [15:06:58] (03CR) 10Jforrester: "check experimental" [labs/tools/extjsonuploader] - 10https://gerrit.wikimedia.org/r/1056629 (https://phabricator.wikimedia.org/T370729) (owner: 10Samwilson) [15:07:03] (03CR) 10Jforrester: "check experimental" [labs/tools/coverme] - 10https://gerrit.wikimedia.org/r/1115535 (owner: 10Libraryupgrader) [15:07:06] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge: [toolsdb] mariadb crashing repeatedly on primary host - https://phabricator.wikimedia.org/T385900#10539327 (10fnegri) I checked the slow query logs in the primary, but couldn't find anything conclusive about what caused the crashes. The replica was today... [15:07:35] (03CR) 10Jforrester: "check experimental" [labs/tools/blankpages] - 10https://gerrit.wikimedia.org/r/963431 (owner: 10Krinkle) [15:09:17] (03CR) 10Jforrester: "check experimental" [labs/countervandalism/stillalive] - 10https://gerrit.wikimedia.org/r/882228 (https://phabricator.wikimedia.org/T306066) (owner: 10AntiCompositeNumber) [15:09:32] (03CR) 10Jforrester: "check experimental" [labs/countervandalism/cvn-api] - 10https://gerrit.wikimedia.org/r/879971 (owner: 10Krinkle) [15:10:31] (03CR) 10Jforrester: "check experimental" [labs/tools/sonarqubebot] - 10https://gerrit.wikimedia.org/r/1063008 (https://phabricator.wikimedia.org/T372565) (owner: 10Pwangai) [15:18:20] (03CR) 10Jforrester: "check experimental" [labs/tools/stewardbots] - 10https://gerrit.wikimedia.org/r/1115536 (owner: 10Libraryupgrader) [15:23:52] 10Tool-global-search: 400 - Bad Request on any Global Search - https://phabricator.wikimedia.org/T358541#10539373 (10Gehel) [15:28:00] 10Tool-global-search, 10Internet-Archive: Global-search is showing duplicate results - https://phabricator.wikimedia.org/T359136#10539440 (10Gehel) [15:36:09] 14cloud-services-team (Kanban): cloud: clarify if relforge is still used from within CloudVPS - https://phabricator.wikimedia.org/T268145#10539553 (10Gehel) [15:38:35] 10Cloud-Services, 10Elasticsearch: Create partitioned CirrusSearchElasticaWrite topic - https://phabricator.wikimedia.org/T239135#10539591 (10Gehel) The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ and replace... [15:50:06] 06cloud-services-team, 10Striker, 10Bitu, 06Infrastructure-Foundations, 13Patch-For-Review: Move Striker to Bitu username validation API - https://phabricator.wikimedia.org/T364605#10539800 (10Aklapper) @Arendpieter: See task status and unmerged patches listed above [15:50:55] 10Cloud-Services, 10Elasticsearch, 13Patch-For-Review: Tool Labs elasticsearch cluster broken by production Puppet changes - https://phabricator.wikimedia.org/T131906#10539818 (10Gehel) The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimed... [16:01:47] 14cloud-services-team (Kanban), 10wikitech.wikimedia.org, 10CirrusSearch, 07Wikimedia-production-error: wikitech runJobs.php error - https://phabricator.wikimedia.org/T218935#10540019 (10Gehel) [16:05:28] 10wikitech.wikimedia.org, 10MediaWiki-Search: ns and pageid returned from prefixsearch on wikitech are 0 - https://phabricator.wikimedia.org/T132280#10540068 (10Gehel) [16:06:48] FIRING: PuppetDisabled: Puppet disabled on cloudgw1002:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=insetup&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled [16:08:05] 06cloud-services-team, 10Cloud-VPS, 10CirrusSearch, 10Datasets-Archiving: Make available an XL labs instance with ~350GB available disk space. - https://phabricator.wikimedia.org/T108767#10540109 (10Gehel) [16:08:43] (03open) 10rook: typo [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/225 [16:24:54] 10Cloud-VPS (Quota-requests): Quota increase request for wikidata-query project - https://phabricator.wikimedia.org/T324209#10540395 (10Gehel) [16:27:03] 14cloud-services-team (Kanban), 10Wikidata, 10Wikidata-Query-Service: Dedicated servers on WMCS to test WDQS scalability strategy - https://phabricator.wikimedia.org/T221631#10540449 (10Gehel) [16:32:14] 06cloud-services-team: PuppetDisabled Puppet disabled on cloudgw1002:9100 - https://phabricator.wikimedia.org/T386111 (10phaultfinder) 03NEW [16:34:45] (03approved) 10fnegri: typo [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/225 (owner: 10rook) [16:35:33] 10wikitech.wikimedia.org: Decide what to do with SUL attached Wikitech accounts that Bitu associates with a different SUL account - https://phabricator.wikimedia.org/T386026#10540662 (10cscott) Please detach 'C. Scott Ananian' from SUL, rename it to 'CAnanian (WMF)', and reattach to SUL. [16:36:05] (03open) 10krinkle: Remove reporting of "MediaWiki-Core-Team" to mediawiki-core [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/53 [16:36:22] (03approved) 10krinkle: Remove reporting of "MediaWiki-Core-Team" to mediawiki-core [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/53 [16:36:46] 10wikitech.wikimedia.org: Decide what to do with SUL attached Wikitech accounts that Bitu associates with a different SUL account - https://phabricator.wikimedia.org/T386026#10540693 (10OSefu-WMF) Please detach 'Omari Sefu' from SUL, rename it to 'OSefu-WMF' [16:40:01] (03merge) 10rook: typo [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/225 [16:42:04] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic, 13Patch-For-Review: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10540850 (10SunAfterRain) >>! In T161859#10537549, @SunAfterRain wrote: > Hello, I recall that in November last year, I did atta... [16:51:24] 10wikitech.wikimedia.org: Decide what to do with SUL attached Wikitech accounts that Bitu associates with a different SUL account - https://phabricator.wikimedia.org/T386026#10540888 (10Ladsgroup) >>! In T386026#10540662, @cscott wrote: > Please detach 'C. Scott Ananian' from SUL, rename it to 'CAnanian (WMF)',... [16:57:40] 10Cloud-Services: Support with steps to access Toolforge user data - https://phabricator.wikimedia.org/T386120 (10KCVelaga_WMF) 03NEW The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ and replace it with a more spe... [16:58:09] 06cloud-services-team, 10Toolforge: Support with steps to access Toolforge user data - https://phabricator.wikimedia.org/T386120#10540907 (10KCVelaga_WMF) [16:58:39] 06cloud-services-team, 10Toolforge: Support with steps to access Toolforge user data - https://phabricator.wikimedia.org/T386120#10540909 (10KCVelaga_WMF) p:05Triage→03Medium Assigning this to Bryan after a Slack conversation with him. [17:02:46] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic, 13Patch-For-Review: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10540921 (10Ladsgroup) I check what I can do about it. [17:22:46] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic, 13Patch-For-Review: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10541035 (10Ladsgroup) >>! In T161859#10540850, @SunAfterRain wrote: >>>! In T161859#10537549, @SunAfterRain wrote: >> Hello, I... [17:30:25] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic, 13Patch-For-Review: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10541121 (10Ameisenigel) >>! In T161859#10538605, @Ladsgroup hat geschrieben: > Fixed. Thank you! [17:41:19] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic, 13Patch-For-Review: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10541151 (10SunAfterRain) >>! In T161859#10541035, @Ladsgroup wrote: >>>! In T161859#10540850, @SunAfterRain wrote: >>>>! In T16... [17:41:58] (03CR) 10Andrew Bogott: [C:03+2] add_user_to_project: small changes to usage [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1118573 (owner: 10Andrew Bogott) [17:46:03] (03Merged) 10jenkins-bot: add_user_to_project: small changes to usage [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1118573 (owner: 10Andrew Bogott) [17:53:59] (03merge) 10krinkle: Remove reporting of "MediaWiki-Core-Team" to mediawiki-core [toolforge-repos/wikibugs2] - 10https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/53 [18:13:24] 10cloud-services-team (Hardware), 10Cloud-VPS, 06DC-Ops, 10ops-eqiad, 06SRE: Relocate cloudnet1007-dev and cloudnet1008-dev to new racks and rename - https://phabricator.wikimedia.org/T382412#10541245 (10Andrew) 05Open→03Resolved These moves are done. [18:36:27] 10cloud-services-team (FY2024/2025-Q3-Q4), 10Toolforge: [toolsdb] mariadb crashing repeatedly on primary host - https://phabricator.wikimedia.org/T385900#10541323 (10fnegri) On the subject of what caused the 4 crashes in the primary, I verified that every time it was caused by `innodb_fatal_semaphore_wait_thre... [18:59:15] 10Tool-ranker, 06translatewiki.net, 10LPL Essential (LPL Essential 2024 Nov-Jan), 13Patch-For-Review, 07Unplanned-Sprint-Work: Add Ranker to translatewiki.net - https://phabricator.wikimedia.org/T384061#10541391 (10LucasWerkmeister) >>! In T384061#10512170, @gerritbot wrote: > Change #1115859 had a r... [19:11:45] 06cloud-services-team, 10Toolforge: Support with steps to access Toolforge user data - https://phabricator.wikimedia.org/T386120#10541473 (10bd808) The data you are looking for exists in the LDAP directory that stores [[https://www.mediawiki.org/wiki/Developer_account|Developer account]] information. This is t... [19:16:59] 10Tool-ranker, 06translatewiki.net, 10LPL Essential (LPL Essential 2024 Nov-Jan), 13Patch-For-Review, 07Unplanned-Sprint-Work: Add Ranker to translatewiki.net - https://phabricator.wikimedia.org/T384061#10541494 (10abi_) 05Resolved→03Open >>! In T384061#10541391, @LucasWerkmeister wrote: >>>! In T384... [20:29:01] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/5 (owner: 10l10n-bot) [20:29:06] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/ranker] - 10https://gitlab.wikimedia.org/toolforge-repos/ranker/-/merge_requests/5 (owner: 10l10n-bot) [21:54:15] 06cloud-services-team, 10wikitech.wikimedia.org, 06Infrastructure-Foundations, 07Epic, 13Patch-For-Review: Make Wikitech an SUL wiki - https://phabricator.wikimedia.org/T161859#10542017 (10bd808) >>! In T161859#10538749, @Bugreporter wrote: > A list of not yet attached accounts with edit in Wikitech: P73... [22:32:34] 10wikitech.wikimedia.org: Decide what to do with SUL attached Wikitech accounts that Bitu associates with a different SUL account - https://phabricator.wikimedia.org/T386026#10542118 (10jwang) Thanks for flagging it. Please detach 'Jenniferwang' from SUL, and rename it to 'JWang (WMF)' . [23:04:58] 10wikitech.wikimedia.org: Decide what to do with SUL attached Wikitech accounts that Bitu associates with a different SUL account - https://phabricator.wikimedia.org/T386026#10542181 (10bd808) >>! In T386026#10542118, @jwang wrote: > Thanks for flagging it. > Please detach 'Jenniferwang' from SUL, and rename it... [23:12:29] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Drop support for VMs with .wmflabs FQDNs - https://phabricator.wikimedia.org/T380679#10542188 (10Andrew) note to self: turn on dns query logging and see if those domains are actually being used. [23:13:45] FIRING: ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_toolserver_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:18:45] RESOLVED: ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_toolserver_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:25:39] FIRING: ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [23:30:39] RESOLVED: ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown