[08:23:53] there will be a few more brief network disruptions in codfw1dev today [08:24:30] ack [09:12:57] dhinus: here's a relatively simple wiki replica view change that can be deployed anytime I think: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1029709 [09:52:49] taavi: I'm not sure if I'm understanding the flow correctly. the person who wrote that patch wants to drop a column in the prod db, realized it would affect wikireplicas, and wrote a patch for dropping the column in wikireplicas first? [09:53:13] is that what normally happens when someone wants to make a change in prod, or were we lucky this time that somebody thought about it? [09:54:00] separate question: where would I check to verify that "the underlying column is not actually used anywhere", as you wrote in the comment? [09:59:01] dhinus: so, in general, when a column is being dropped, the view rules need to be updated before the column is gone from prod databases. in this case yes, the patch author knew that needs to happen, but I would expect that to be noticed at the DBA schema change review phase if the person doing the schema change is not aware of that [10:00:24] for the second question: by looking at the schema. if you have a view definition that selects `NULL AS some_column`, the `some_column` column in the underlying table is never accessed, even if the column created by the view has the same name [10:07:05] ah I see thanks, I misread your comment and thought "not actually used anywhere" referred to the prod db, but you were just talking about the view definition [10:08:27] what would be the next steps to get this live to wikireplicas? merge the patch, then run the update-views cookbook? [10:09:42] more or less that, yes [10:11:37] I can try doing that [10:11:53] I'm confused by https://wikitech.wikimedia.org/wiki/Portal:Data_Services/Admin/Runbooks/Deploy_wiki_replicas_view_change [10:12:11] is it a manual way of doing what the cookbook does? [10:16:53] yes. I've been waiting to get https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1026857 and https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1030055 merged before updating the docs to reference the cookbook instead [10:22:06] I +1d those, the CI is failing on the second one though [10:22:19] yeah, looking at that now [10:37:28] https://wikitech.wikimedia.org/w/index.php?title=Portal:Data_Services/Admin/Runbooks/Deploy_wiki_replicas_view_change&diff=prev&oldid=2176662 [10:39:48] nice thanks! [10:43:21] I'm having lunch shortly, I will run the cookbook after that (and ping -data-persistence first) [11:10:17] I went through the list of toolforge membership requests. Opinions on this one? https://toolsadmin.wikimedia.org/tools/membership/status/1700 [12:08:25] dhinus: fishy, maybe leave it to bryan [12:33:37] arturo: ack, thanks [13:07:10] * arturo food [13:45:11] apparently the toolsbeta prometheus k8s cert just expired [13:52:37] i renewed the toolsbeta cert. the tools cert expires early next month and is tracked as T309782 [13:52:39] T309782: toolforge: Refresh certs that are not controlled by kubeadm (mid 2024 edition) - https://phabricator.wikimedia.org/T309782 [14:01:39] * dcaro got paged (and just now notified), was it the cert expiration? [14:01:43] yes [14:01:57] Ack [14:02:22] My phone did not ring though, I think I have to tweak something somewhere again [14:04:34] Hmmm.. it looks ok on the settings :/ [14:10:32] It decided to use the most silent of notification sounds for some reason... Reverted to default ugly siren [14:10:59] Thanks for taking care of it :) [14:17:32] dhinus: do you have thoughts about the memory alert on clouddb1015? [14:17:44] (I haven't looked) [14:23:32] andrewbogott: looking [14:23:45] (I didn't notice it until now) [14:24:01] thx! [14:27:18] according to Grafana, memory usage has been high for the past 30 days, with no sudden spike [14:27:47] it seems to be increasing constantly and very slowly [14:28:51] it should be safe for at least 1 more week, but I would try restarting the mariadb service on Monday [15:17:15] Happy Friday to you all! [15:18:18] hey balloons! [15:18:44] I hope your new trees are enjoying the spring [15:18:52] o/ [15:21:00] Yes, my trouble is no pollinators have figured out spring starts in the beginning of March now. My trees bloom and don't get pollinated :-( I managed to catch the peach tree, so I will have a few peaches I hope [15:23:53] oh dang [15:24:05] Have to get a stepladder and a paintbrush for next year [15:32:35] Yes, and mark my calendar to go out and check that early! I hope spring has been lovely for you all as well. It's nice to see the earth come alive again. [15:33:47] I'm in a new house with a new yard so every day is a new garden surprise [15:35:15] we also have a new garden, and we are currently dealing with kind of Risk game between our cats and the neighbor's cats, trying to conquer territory from each other [15:36:13] My yard has owls, and crows, and the crows HATE the owls. So a few times a day there's a skirmish and I have to close the windows to get some peace. [15:40:39] hey balloons, happy Friday! [15:41:05] o/ [15:42:09] I didn't know that about crows and owls. Too bad! I like both. It's warbler season here [15:46:20] Oh yeah, very warbler-heavy here too. They're impossible to photograph, too wiggly. [15:49:00] * arturo searches some new vocabulary [16:09:42] * arturo offline