[08:13:54] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1011268 improved my morning (so many "your package is going to be removed from testing" emails) [09:29:50] yep, super fun [09:50:00] so happy with my embedded Intel GPU right now... (VGA compatible controller: Intel Corporation TigerLake GT2 [Iris Xe Graphics] (rev 01)) [11:00:55] We are going to start m1 emergency switchover in a few minutes https://phabricator.wikimedia.org/T309296 [11:01:04] ack [11:04:32] <_joe_> roger [11:24:21] db1128 is no longer m1 master [12:10:28] for people oncall: db1128 has notifications disabled, but I cannot discard: [12:10:38] 1) db1128 to crash again [12:10:54] 2) bacula and dbbackups to notify of backup delays [12:11:04] And db1128 is no longer in use [12:11:29] yes, sorry, it is not in use at all [12:11:58] but a backup is running, which is known to crash servers with bad memory [12:12:47] backups don't page, and I am unsure how they will complain, so I cannot downtime most in advance (I will do it for one case) [12:18:09] jynus: noted [13:24:19] _joe_: moritzm: taavi: i have sent https://gerrit.wikimedia.org/r/c/operations/puppet/+/799982 as a work around for the URI.escape issue let me know if it works/what you think [13:25:10] <_joe_> jbond: probably needs a guard based on ruby version? [13:29:10] jobo: ack updated thanks [13:29:15] _joe_: even [13:30:33] <_joe_> jbond: lgtm, but let me try it with ruby 3 [13:31:30] ack [13:33:38] 😁 [13:40:04] hello jobo! [13:40:41] Hi Chris :) [13:55:40] <_joe_> jobo: I'm sorry :P [13:58:31] Haha no worries Joe. [15:40:58] Just a heads-up, I am about to deploy https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/797354 which will remove https://en.wikipedia.org/wiki/Wikipedia:Administrators%27_noticeboard/Incidents from the restbase rendering deny-list. I'll be keeping an eye on restbase behaviour to make sure nothing blows up [15:45:47] I have a DNS patch failing in jenkins with what seems to be entirely unrelated netbox managed record lint errors -- https://integration.wikimedia.org/ci/job/operations-dns-lint-docker/3884/console -- who can fix such things? [15:46:23] bd808: see -ops, topranks has already replied to you ;) [15:46:48] volans: *nod* saw that just seconds before you responded. Thanks :) [15:47:18] no prob, better two notifications than zero ;) [15:48:18] haha, volans is the one to thank he noticed it and let me know :) [15:48:38] ssshhhh don't tell him all the secrets :-P [15:52:20] * bd808 doesn't have the better call volans image macro handy [15:53:38] haha [15:54:13] bd808: ok you should be good to go now the duplicate is removed [15:54:44] sorry for the hassle :) [15:55:16] thanks topranks. Now I just need to trick^Wconvince a root to review, merge and deploy them. -- https://gerrit.wikimedia.org/r/q/topic:developer-portal+is:open [15:59:59] the changes look ok to me but I'm not sure what the gating process is around new names, I'd be afraid I'm skipping some part of the process [16:02:05] I'm happy to merge and deploy if we can get someone in traffic to give a +1 [16:03:33] hnowlan: cool now we can remove the entire denylist! /j [16:03:51] topranks: the names are based on https://wikitech.wikimedia.org/wiki/Kubernetes/Ingress#DNS_changes and the namespace name for that new service in the k8s cluster. [16:04:30] but I understand not being comfortable with +2 on things you don't normally handle. no worries there [16:06:59] TheresNoTime: let's wait until 5 tomorrow [16:07:09] TheresNoTime: looks good so far anyway! [16:07:15] heheh [16:07:52] yup! thank you for doing that, really appreciate it :) should hopefully provide a useful test for how to phase out the denylist [16:17:54] bd808: You should be good now. I'll ask for forgiveness if I need to :) [16:17:59] https://www.irccloud.com/pastebin/1ml9kXmR/ [16:18:36] topranks: thank you. And tell the haters that it was all my fault if they come for you :) [16:19:34] haha will do :) [16:20:33] `curl https://developer-portal.svc.codfw.wmnet:30443` is alive! Now to figure out what else needs patching to make this "really" live [16:27:17] jbond: nice. I'm off today/tomorrow, will have a look on Monday [16:29:29] ack no rush moritzm :) [16:40:36] bd808: woot! [17:26:55] Is there a best/preferred person to bribe for reviewing and merging changes to the CDN edge<->backend service mappings? https://gerrit.wikimedia.org/r/c/operations/puppet/+/800181 points developer.wikimedia.org (already setup to point to dyna.wm.o) at the developer-portal k8s service. [17:27:59] bd808: #wikimedia-traffic [17:28:10] volans: ack [17:28:18] not everyone is online today [17:29:18] yeah, Feast of the Ascension today right? [20:42:56] bblack: this may be worth re-triaging (no more HTTP 304 possible for upload.wm.o) https://phabricator.wikimedia.org/T256217#7960730 / https://phabricator.wikimedia.org/T295556#7961731