[04:36:53] 06serviceops, 10RESTBase Sunsetting, 07Epic: Replace usage of RESTbase parsoid endpoints - https://phabricator.wikimedia.org/T328559#9682824 (10Pppery) [09:38:52] 06serviceops, 13Patch-For-Review: etcdmirror does not recover from a cleared waitIndex - https://phabricator.wikimedia.org/T358636#9683493 (10Volans) Wow, that was quite an investigation for a `/test` key, thanks for the thorough analysis. As for the `test2` value that could have been me when deploying the sp... [11:01:18] 06serviceops, 06Content-Transform-Team-WIP, 10Page Content Service, 10RESTBase Sunsetting, 13Patch-For-Review: Update mobileapps k8s deployment chart for Cassandra credentials - https://phabricator.wikimedia.org/T350507#9683782 (10Jgiannelos) I am testing things on staging and I am getting this error (an... [11:34:03] 06serviceops, 06Content-Transform-Team-WIP, 10Page Content Service, 10RESTBase Sunsetting, 13Patch-For-Review: Update mobileapps k8s deployment chart for Cassandra credentials - https://phabricator.wikimedia.org/T350507#9683875 (10Jgiannelos) From staging: ` { "status": 500, "type": "internal_error"... [12:10:40] 06serviceops, 06Content-Transform-Team-WIP, 10Page Content Service, 10RESTBase Sunsetting, 13Patch-For-Review: Update mobileapps k8s deployment chart for Cassandra credentials - https://phabricator.wikimedia.org/T350507#9684042 (10hnowlan) It looks like the network setup between staging and cassandra-dev... [12:20:42] 06serviceops, 06Data-Persistence (work done), 10MediaWiki-Parser, 10Parsoid (Tracking): 14CAPEX for ParserCache for Parsoid - 14https://phabricator.wikimedia.org/T263587#9684077 (10daniel) [12:42:50] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Evaluate and enable audit logging for kube-apiserver - https://phabricator.wikimedia.org/T290020#9684179 (10JMeybohm) [12:50:07] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Evaluate and enable audit logging for kube-apiserver - https://phabricator.wikimedia.org/T290020#9684210 (10JMeybohm) #observability-logging could you maybe advice on if/how/where we could potentially store these audit logs to make them mor... [12:55:46] 06serviceops, 06collaboration-services, 06Infrastructure-Foundations, 10Puppet-Core, and 5 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619#9684225 (10MoritzMuehlenhoff) [13:29:43] 06serviceops, 06Content-Transform-Team-WIP, 10Page Content Service, 10RESTBase Sunsetting, 13Patch-For-Review: Update mobileapps k8s deployment chart for Cassandra credentials - https://phabricator.wikimedia.org/T350507#9684391 (10Jgiannelos) I think the problem is on the nodejs cassandra client TLS init... [15:42:07] folks I am running build-production-images on build2001 in a tmux, it is taking ages due to openjdk and spark images [15:42:25] I'll let it run, going afk in a bit but I'll check later on [15:46:00] are 122GB free enough? :-P [15:47:32] ;P [16:44:08] 06serviceops, 07Wikimedia-Incident: 14Helm was left in limbo due to interrupted deployment/rollback - 14https://phabricator.wikimedia.org/T361720#9685222 (10jijiki) [16:44:34] 06serviceops, 07Wikimedia-Incident: 14Helm was left in limbo due to interrupted deployment/rollback - 14https://phabricator.wikimedia.org/T361720#9685218 (10jijiki) 05Open→03Resolved p:05Triage→03Unbreak! a:03jijiki [16:53:12] 06serviceops, 06Release-Engineering-Team, 07Wikimedia-Incident: scap should check if it is running within a tmux/screen - https://phabricator.wikimedia.org/T361724 (10jijiki) 03NEW [16:54:32] 06serviceops, 06Release-Engineering-Team, 07Wikimedia-Incident: scap should check if it is running within a tmux/screen - https://phabricator.wikimedia.org/T361724#9685294 (10jijiki) p:05Triage→03High [17:09:39] 06serviceops, 06Content-Transform-Team-WIP, 10Page Content Service, 10RESTBase Sunsetting, 13Patch-For-Review: Update mobileapps k8s deployment chart for Cassandra credentials - https://phabricator.wikimedia.org/T350507#9685357 (10CodeReviewBot) jgiannelos merged https://gitlab.wikimedia.org/repos/conten... [17:26:35] 06serviceops, 06Release-Engineering-Team, 10Scap, 10Sustainability (Incident Followup): scap should check if it is running within a tmux/screen - https://phabricator.wikimedia.org/T361724#9685429 (10taavi) [17:45:06] 06serviceops, 10MW-on-K8s, 10Release-Engineering-Team (Radar): Helm deployment of MediaWiki now takes 6 minutes - https://phabricator.wikimedia.org/T360403#9685465 (10thcipriani) Moving this to our radar as I don't think #together can do anything about this directly right now—this is just how long it takes t... [20:22:27] 06serviceops, 06Release-Engineering-Team, 10Scap, 10Sustainability (Incident Followup): scap should check if it is running within a tmux/screen - https://phabricator.wikimedia.org/T361724#9685992 (10bd808) > Deployers should always run scap within a server side tmux/screen I think if this is actually the... [21:57:54] 06serviceops: Improve etcdmirror shutdown behavior - https://phabricator.wikimedia.org/T361762 (10Scott_French) 03NEW [21:58:27] 06serviceops: Improve etcdmirror shutdown behavior - https://phabricator.wikimedia.org/T361762#9686396 (10Scott_French) 05Open→03In progress p:05Triage→03Low [22:45:32] 06serviceops, 10MediaWiki-General, 10MediaWiki-libs-Stats, 10observability, and 2 others: MediaWiki Prometheus support - https://phabricator.wikimedia.org/T240685#9686467 (10colewhite)