[09:10:06] 06serviceops, 06collaboration-services, 06Infrastructure-Foundations, 10Puppet-Core, and 5 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619#9696567 (10MoritzMuehlenhoff) [09:54:03] 06serviceops, 06Machine-Learning-Team, 13Patch-For-Review: Rename the envoy's uses_ingress option to sets_sni - https://phabricator.wikimedia.org/T346638#9696700 (10JMeybohm) [09:55:23] 06serviceops, 06collaboration-services, 06Infrastructure-Foundations, 10Puppet-Core, and 5 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619#9696705 (10MoritzMuehlenhoff) [10:24:02] 06serviceops, 06Release-Engineering-Team, 10Scap, 13Patch-For-Review, 10Sustainability (Incident Followup): scap should check if it is running within a tmux/screen - https://phabricator.wikimedia.org/T361724#9696779 (10Clement_Goubert) For this use case the only keybinding you would need to know is how t... [14:01:57] 06serviceops, 06Data-Engineering, 06MediaWiki-Engineering, 10WMF-JobQueue, and 2 others: Could not enqueue jobs: "Unable to deliver all events: 503: Service Unavailable" - https://phabricator.wikimedia.org/T249745#9697632 (10JMeybohm) We had a bunch of error again today. All of them connection errors to ev... [14:06:47] 06serviceops, 06SRE, 10Data Products (Data Products Sprint 11), 07Service-deployment-requests: Commons Impact Metrics AQS 2.0 Deployment to Staging and Production - https://phabricator.wikimedia.org/T361835#9697644 (10mforns) Hi @Scott_French ! > - When do you anticipate having a minimal binary that succe... [14:15:55] 06serviceops, 10Shellbox: shellbox-constraints listener is constantly exceeding the retry limit - https://phabricator.wikimedia.org/T362084 (10JMeybohm) 03NEW [14:18:05] 06serviceops, 10Maps, 06SRE: Move maps/karthoterian to PKI/cfssl - https://phabricator.wikimedia.org/T360778#9697722 (10LSobanski) [14:36:55] 06serviceops, 10ChangeProp, 06collaboration-services, 10GitLab, and 10 others: Figure out a plan to move forward with regarding Redis License changes - https://phabricator.wikimedia.org/T360596#9697832 (10joanna_borun) p:05Triage→03Medium [14:40:47] 06serviceops, 06Data-Engineering, 06MediaWiki-Engineering, 10WMF-JobQueue, and 2 others: Could not enqueue jobs: "Unable to deliver all events: 503: Service Unavailable" - https://phabricator.wikimedia.org/T249745#9697842 (10akosiaris) >>! In T249745#9640217, @Ottomata wrote: > >> This doesn't mean that M... [14:57:34] 06serviceops, 10Shellbox: shellbox-constraints listener is constantly exceeding the retry limit - https://phabricator.wikimedia.org/T362084#9697945 (10JMeybohm) There is a constant stream of 500 errors reported by shellbox-constraints apache as well. I've captured 8 responses, all of them report: `lang=json {... [15:04:45] 06serviceops, 06Infrastructure-Foundations, 10SRE-tools: Switchdc RO/RW: add check to test it editing a real wiki - https://phabricator.wikimedia.org/T163365#9697994 (10joanna_borun) [15:34:27] 06serviceops, 06Content-Transform-Team, 10Page Content Service, 10RESTBase Sunsetting, 13Patch-For-Review: Update mobileapps k8s deployment chart for Cassandra credentials - https://phabricator.wikimedia.org/T350507#9698145 (10MSantos) [15:36:23] 06serviceops, 06Content-Transform-Team, 10Page Content Service, 10RESTBase Sunsetting, 13Patch-For-Review: Update mobileapps k8s deployment chart for Cassandra credentials - https://phabricator.wikimedia.org/T350507#9698151 (10MSantos) Great! From our side this should be tracking now. Please, let us know... [15:37:01] 06serviceops, 06Content-Transform-Team-WIP, 10Page Content Service, 10RESTBase Sunsetting: 14Introduce PCS cache management layer - 14https://phabricator.wikimedia.org/T348995#9698162 (10MSantos) 05Open→03Resolved [15:41:53] 06serviceops, 06Content-Transform-Team, 10Page Content Service, 10RESTBase Sunsetting, 13Patch-For-Review: Update mobileapps k8s deployment chart for Cassandra credentials - https://phabricator.wikimedia.org/T350507#9698208 (10hnowlan) Do we have a plan for when and how we'd like to move this to producti... [16:01:08] 06serviceops, 06SRE, 07Epic: Phase out cergen for ServiceOps services - https://phabricator.wikimedia.org/T360636#9698325 (10akosiaris) I 'll finish parsoid and testreduce in T359387 [16:03:58] 06serviceops, 10Shellbox: shellbox-constraints returning 500 on preg_match error - https://phabricator.wikimedia.org/T362084#9698334 (10JMeybohm) [16:04:36] 06serviceops, 10ChangeProp, 06collaboration-services, 10GitLab, and 10 others: Figure out a plan to move forward with regarding Redis License changes - https://phabricator.wikimedia.org/T360596#9698336 (10CodeReviewBot) bd808 merged https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/28... [17:29:09] 06serviceops, 06SRE, 10Data Products (Data Products Sprint 11), 07Service-deployment-requests: Commons Impact Metrics AQS 2.0 Deployment to Staging and Production - https://phabricator.wikimedia.org/T361835#9698597 (10mforns) [18:36:01] 06serviceops, 10Shellbox, 10Wikibase-Quality-Constraints, 10wmde-wikidata-tech, 07Wikimedia-production-error: shellbox-constraints returning 500 on preg_match error - https://phabricator.wikimedia.org/T362084#9698742 (10Reedy) Pretty sure these examples are from #wikibase-quality-constraints: `lang=php... [18:36:03] 06serviceops, 10Shellbox, 10Wikibase-Quality-Constraints, 10wmde-wikidata-tech, 07Wikimedia-production-error: shellbox-constraints returning 500 on preg_match error - https://phabricator.wikimedia.org/T362084#9698744 (10Reedy) [18:36:58] 06serviceops, 10Shellbox, 10Wikibase-Quality-Constraints, 10Wikidata, and 2 others: shellbox-constraints returning 500 on preg_match error - https://phabricator.wikimedia.org/T362084#9698747 (10Reedy) [18:44:18] 06serviceops, 10Shellbox, 10Wikibase-Quality-Constraints, 10Wikidata, and 2 others: shellbox-constraints returning 500 on preg_match error - https://phabricator.wikimedia.org/T362084#9698752 (10Reedy) Last edit to any of this code was {d7564ee12be34c37a46eda47692077028a42e506} in August 2021... So dependi... [18:56:10] 06serviceops, 10Shellbox, 10Wikibase-Quality-Constraints, 10Wikidata, and 2 others: shellbox-constraints returning 500 on preg_match error - https://phabricator.wikimedia.org/T362084#9698777 (10Reedy) As there's many parts to this... Seemingly you can do "regex validation" in PHP by testing against `null`... [18:57:49] 06serviceops, 10Shellbox, 10Wikibase-Quality-Constraints, 10Wikidata, and 2 others: shellbox-constraints returning 500 on preg_match error - https://phabricator.wikimedia.org/T362084#9698779 (10Reedy) https://www.wikidata.org/wiki/Property:P5504 is `"/^(?:(people|institutions|sources)\\\\/[1-9]\\d*)$/u"` b... [19:01:18] 06serviceops, 06SRE, 10Data Products (Data Products Sprint 11), 07Service-deployment-requests: Commons Impact Metrics AQS 2.0 Deployment to Staging and Production - https://phabricator.wikimedia.org/T361835#9698788 (10Scott_French) Thanks, @mforns! > Does this binary need to implement some of the endpoint... [19:04:26] 06serviceops, 10Shellbox, 10Wikibase-Quality-Constraints, 10Wikidata, and 2 others: shellbox-constraints returning 500 on preg_match error - https://phabricator.wikimedia.org/T362084#9698793 (10Reedy) The other is the text on https://www.wikidata.org/wiki/Q6739335, leading to https://www.wikidata.org/wiki/... [19:10:01] 06serviceops, 10Shellbox, 10Wikibase-Quality-Constraints, 10Wikidata, and 3 others: shellbox-constraints returning 500 on preg_match error - https://phabricator.wikimedia.org/T362084#9698807 (10Reedy) ^ Sticking those two into a test... Lets see what happens [19:45:52] 06serviceops, 10Shellbox, 10Wikibase-Quality-Constraints, 10Wikidata, and 3 others: shellbox-constraints returning 500 on preg_match error - https://phabricator.wikimedia.org/T362084#9698883 (10Reedy) The tests fail as we'd expect/hope: ` 20:14:29 There were 2 errors: 20:14:29 20:14:29 1) WikibaseQuality... [19:57:07] 06serviceops, 06SRE, 07Epic: Phase out cergen for ServiceOps services - https://phabricator.wikimedia.org/T360636#9698910 (10Scott_French) I was planning to migrate etcd to PKI as part of T350565, but can explore this earlier if needed. [19:59:19] 06serviceops: Migrate etcd::tlsproxy Nginx certs and etcd itself to PKI - https://phabricator.wikimedia.org/T352245#9698921 (10Scott_French) a:03Scott_French My initial plan was to move etcd to PKI as part of the v3 API migration (T350565), which is also likely to do away with the TLS proxy. However, the ETA... [22:24:46] 06serviceops, 13Patch-For-Review: etcdmirror does not recover from a cleared waitIndex - https://phabricator.wikimedia.org/T358636#9699259 (10Scott_French) For the package release, my plan is to build and upload the 0.0.11-1 package shortly (but ideally not less than 30m) before a "MediaWiki infrastructure" de... [23:45:46] 06serviceops, 13Patch-For-Review: etcdmirror does not recover from a cleared waitIndex - https://phabricator.wikimedia.org/T358636#9699378 (10Scott_French) As discussed previously, there are two options to consider: * Full reload - i.e., use `--reload` to fully clear and rewrite the keyspace in codfw. * In-pla...