[00:07:28] (PuppetAgentStaleLastRun) firing: Last Puppet run was over 24 hours ago on instance tf-infra-test in project tf-infra-test - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [00:12:28] (PuppetAgentStaleLastRun) resolved: Last Puppet run was over 24 hours ago on instance tf-infra-test in project tf-infra-test - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [01:34:56] (03PS2) 10Krinkle: frontend: Implement /_health [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1017384 [01:35:16] (03PS1) 10Krinkle: app.py: redirect _health to frontend, frontend: add link to view JSON [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1017412 [01:35:56] 10Wikibugs, 13Patch-For-Review: Replace Redis queue with custom http solution - https://phabricator.wikimedia.org/T361518#9694406 (10CodeReviewBot) bd808 opened https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/28 Replace Redis queue with custom http solution [01:36:24] 10Quarry, 10Toolforge, 10ChangeProp, 06collaboration-services, and 10 others: Figure out a plan to move forward with regarding Redis License changes - https://phabricator.wikimedia.org/T360596#9694404 (10CodeReviewBot) bd808 opened https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/28... [01:36:57] 10Wikibugs, 13Patch-For-Review: Replace Redis queue with custom http solution - https://phabricator.wikimedia.org/T361518#9694420 (10bd808) [01:37:05] 10Quarry, 10Toolforge, 10ChangeProp, 06collaboration-services, and 10 others: Figure out a plan to move forward with regarding Redis License changes - https://phabricator.wikimedia.org/T360596#9694421 (10bd808) [01:37:41] 10Wikibugs: Wikibugs testing task - https://phabricator.wikimedia.org/T90594#9694424 (10bd808) test [01:37:45] (03CR) 10BryanDavis: [C:04-2] "test" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/1008016 (https://phabricator.wikimedia.org/T90594) (owner: 10BryanDavis) [01:38:21] 10Tool-global-search: Results are not updated on Global Search - https://phabricator.wikimedia.org/T361987 (10T) 03NEW [01:38:25] 10Tool-global-search: Results are not updated on Global Search - https://phabricator.wikimedia.org/T361987#9694452 (10T) [01:40:52] (03PS1) 10Krinkle: app.py: remove DESCRIPTIONS from old UI, add "Hound" to title [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1017413 [01:41:14] (03CR) 10CI reject: [V:04-1] app.py: remove DESCRIPTIONS from old UI, add "Hound" to title [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1017413 (owner: 10Krinkle) [01:41:24] (03PS2) 10Krinkle: app.py: remove DESCRIPTIONS from old UI, add "Hound" to title [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1017413 [01:41:28] (03CR) 10CI reject: [V:04-1] app.py: remove DESCRIPTIONS from old UI, add "Hound" to title [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1017413 (owner: 10Krinkle) [05:17:34] (DiskSpace) firing: Disk space cloudbackup1004:9100:/srv 5.995% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [09:17:34] (DiskSpace) firing: Disk space cloudbackup1004:9100:/srv 5.53% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [09:52:26] 10Data-Services: 14Make af_actor and afh_actor accessible in Wiki Replicas - 14https://phabricator.wikimedia.org/T337921#9694840 (10matej_suchanek) 05Open→03Resolved a:03Zabe [10:02:02] 10Data-Services, 07User-notice: Remove deprecated fields from Wiki Replicas - https://phabricator.wikimedia.org/T361996 (10matej_suchanek) 03NEW [10:11:24] 10Data-Services, 07User-notice: Remove deprecated fields from Wiki Replicas - https://phabricator.wikimedia.org/T361996#9694867 (10matej_suchanek) [13:17:34] (DiskSpace) firing: Disk space cloudbackup1004:9100:/srv 5.164% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [15:26:39] (ProbeDown) firing: (3) Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:31:39] (ProbeDown) resolved: (3) Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:17:34] (DiskSpace) firing: Disk space cloudbackup1004:9100:/srv 4.441% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [19:01:39] (ProbeDown) firing: Service toolsbeta-test-k8s-haproxy-6:30000 has failed probes (http_this_tool_does_not_exist_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#toolsbeta-test-k8s-haproxy-6:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:06:39] (ProbeDown) resolved: Service toolsbeta-test-k8s-haproxy-6:30000 has failed probes (http_this_tool_does_not_exist_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#toolsbeta-test-k8s-haproxy-6:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [19:41:49] 10Wikibugs, 10GitLab (Integrations), 10Release-Engineering-Team (Priority Backlog 📥): Connect WikiBugs IRC bot to Wikimedia GitLab - https://phabricator.wikimedia.org/T288381#9695160 (10bd808) >>! In T288381#9490163, @valhallasw wrote: > This could now presumably be built on top of https://wikitech.wikimedia... [20:16:17] 10Wikibugs: 14Rethink anti-flooding protections - 14https://phabricator.wikimedia.org/T359753#9695182 (10bd808) 05Open→03Resolved a:03bd808 14The bot has been running for a couple of weeks with only the default flood protections from irc3.IrcBot + ZNC mentioned in T359753#9618035. There have not been... [20:18:32] 10Wikibugs: 14Store PhorgeFeedReader.poll_last_seen_chrono_key in Redis - 14https://phabricator.wikimedia.org/T359009#9695186 (10bd808) 05Open→03Declined 14With Gerrit events and the likely implementation of GitLab events both being unable to time travel it doesn't seem important to add this support to... [20:23:09] 10Wikibugs: 14Create function to temporarily mute reports - 14https://phabricator.wikimedia.org/T75900#9695192 (10bd808) →14Duplicate dup:03T231733 [20:23:12] 10Wikibugs: Exclude bulk actions changes from IRC logging - https://phabricator.wikimedia.org/T231733#9695189 (10bd808) [20:28:15] 10Wikibugs: 14wikibugs should handle non-task event types - 14https://phabricator.wikimedia.org/T651#9695194 (10bd808) 05Open→03Declined 14We've gotten by for ~10 years now without irc notifs for pastes, etc and it looks like basically no one has missed that functionality. Declining without prejudice on... [20:41:36] PROBLEM - Disk space on cloudbackup1004 is CRITICAL: DISK CRITICAL - free space: /srv 642730 MB (3% inode=98%): https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space https://grafana.wikimedia.org/d/000000377/host-overview?var-server=cloudbackup1004&var-datasource=eqiad+prometheus/ops [21:09:43] 10Toolforge: [webservice] Allow configuration of Promethus scraping of a specific webservice endpoint for publication in grafana.wmcloud.org - https://phabricator.wikimedia.org/T362012 (10bd808) 03NEW [21:09:59] 10Toolforge: [webservice] Allow configuration of Promethus scraping of a specific webservice endpoint for publication in grafana.wmcloud.org - https://phabricator.wikimedia.org/T362012#9695214 (10bd808) [21:10:03] 06cloud-services-team, 10Toolforge, 10community-labs-monitoring: Establish an internal system or a recommended external system for monitoring user-created Toolforge web services - https://phabricator.wikimedia.org/T53434#9695215 (10bd808) [21:17:34] (DiskSpace) firing: Disk space cloudbackup1004:9100:/srv 3.4% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [23:03:52] (03Abandoned) 10Urbanecm: Reverting: Include git commit --amend --no-edit and git fetch [labs/tools/deploy-commands] - 10https://gerrit.wikimedia.org/r/824505 (owner: 10Urbanecm) [23:04:13] (03Abandoned) 10Urbanecm: do not create users again and again [labs/tools/wiki-importer] - 10https://gerrit.wikimedia.org/r/673269 (owner: 10Urbanecm)