[10:00:01] !incidents [10:00:04] 7525 (UNACKED) db1175 (paged)/MariaDB Replica SQL: s3 (paged) [10:00:04] 7526 (UNACKED) db1170 (paged)/MariaDB Replica SQL: s7 (paged) [10:00:05] 7527 (UNACKED) db1161 (paged)/MariaDB Replica SQL: s5 (paged) [10:00:05] 7528 (UNACKED) db1158 (paged)/MariaDB Replica SQL: s7 (paged) [10:00:05] 7529 (UNACKED) db1157 (paged)/MariaDB Replica SQL: s3 (paged) [10:00:05] 7530 (UNACKED) db1168 (paged)/MariaDB Replica SQL: s6 (paged) [10:00:05] 7531 (UNACKED) db1162 (paged)/MariaDB Replica SQL: s2 (paged) [10:00:06] 7532 (UNACKED) db1174 (paged)/MariaDB Replica SQL: s7 (paged) [10:00:06] 7533 (UNACKED) db1172 (paged)/MariaDB Replica SQL: s8 (paged) [10:00:07] 7534 (UNACKED) db1179 (paged)/MariaDB Replica SQL: x1 (paged) [10:00:07] 7535 (UNACKED) db1185 (paged)/MariaDB Replica SQL: s5 (paged) [10:00:08] 7536 (UNACKED) db1165 (paged)/MariaDB Replica SQL: s6 (paged) [10:00:08] 7537 (UNACKED) db1156 (paged)/MariaDB Replica SQL: s2 (paged) [10:00:09] 7538 (UNACKED) db1188 (paged)/MariaDB Replica SQL: s2 (paged) [10:00:09] 7539 (UNACKED) db1159 (paged)/MariaDB Replica SQL: s5 (paged) [10:00:10] 7540 (UNACKED) db1166 (paged)/MariaDB Replica SQL: s3 (paged) [10:00:10] 7541 (UNACKED) db1193 (paged)/MariaDB Replica SQL: s8 (paged) [10:00:11] 7542 (UNACKED) db1167 (paged)/MariaDB Replica SQL: s8 (paged) [10:00:11] 7543 (UNACKED) db1200 (paged)/MariaDB Replica SQL: s5 (paged) [10:00:12] 7544 (UNACKED) db1192 (paged)/MariaDB Replica SQL: s8 (paged) [10:00:12] 7545 (UNACKED) db1203 (paged)/MariaDB Replica SQL: s8 (paged) [10:00:18] lolsob [10:00:37] I'll ack via the interface [10:01:14] godog: [10:01:16] !ack [10:01:16] <_joe_> marostegui: need help? [10:01:22] !ack [10:01:23] 7562 (ACKED) db2211 (paged)/MariaDB Replica SQL: s5 (paged) [10:01:24] 7563 (ACKED) db2178 (paged)/MariaDB Replica SQL: s5 (paged) [10:01:24] 7564 (ACKED) db2182 (paged)/MariaDB Replica SQL: s7 (paged) [10:01:24] 7565 (ACKED) db2176 (paged)/MariaDB Replica SQL: s1 (paged) [10:01:24] 7566 (ACKED) db2175 (paged)/MariaDB Replica SQL: s2 (paged) [10:01:24] 7567 (ACKED) db2194 (paged)/MariaDB Replica SQL: s3 (paged) [10:01:25] 7568 (ACKED) db2186 (paged)/MariaDB Replica SQL: x1 (paged) [10:01:25] 7569 (ACKED) db2177 (paged)/MariaDB Replica SQL: s3 (paged) [10:01:25] 7570 (ACKED) db2192 (paged)/MariaDB Replica SQL: s5 (paged) [10:01:26] 7571 (ACKED) db2190 (paged)/MariaDB Replica SQL: s3 (paged) [10:01:26] 7572 (ACKED) db2174 (paged)/MariaDB Replica SQL: s1 (paged) [10:01:27] 7573 (ACKED) db2203 (paged)/MariaDB Replica SQL: s1 (paged) [10:01:27] 7574 (ACKED) db1181 (paged)/MariaDB Replica SQL: s7 (paged) [10:01:28] 7575 (ACKED) db2188 (paged)/MariaDB Replica SQL: s1 (paged) [10:01:28] 7576 (ACKED) db1160 (paged)/MariaDB Replica SQL: s4 (paged) [10:01:29] 7587 (ACKED) db2156 (paged)/MariaDB Replica SQL: s3 (paged) [10:01:29] 7588 (ACKED) db2170 (paged)/MariaDB Replica SQL: s1 (paged) [10:01:30] 7589 (ACKED) db2168 (paged)/MariaDB Replica SQL: s7 (paged) [10:01:30] 7590 (ACKED) db2159 (paged)/MariaDB Replica SQL: s7 (paged) [10:01:31] 7591 (ACKED) db2173 (paged)/MariaDB Replica SQL: s1 (paged) [10:01:31] 7592 (ACKED) db2153 (paged)/MariaDB Replica SQL: s1 (paged) [10:01:32] 7593 (ACKED) db2146 (paged)/MariaDB Replica SQL: s1 (paged) [10:01:32] 7594 (ACKED) db2189 (paged)/MariaDB Replica SQL: s2 (paged) [10:01:33] 7595 (ACKED) db2205 (paged)/MariaDB Replica SQL: s3 (paged) [10:01:33] 7596 (ACKED) db2218 (paged)/MariaDB Replica SQL: s7 (paged) [10:01:38] <_joe_> volans: do it in private [10:08:50] <_joe_> !ack [10:08:51] 7656 (ACKED) db2208 (paged)/MariaDB Replica Lag: s7 (paged) [10:08:51] 7657 (ACKED) db2205 (paged)/MariaDB Replica Lag: s3 (paged) [10:08:51] 7658 (ACKED) db2203 (paged)/MariaDB Replica Lag: s1 (paged) [10:08:51] 7659 (ACKED) db2211 (paged)/MariaDB Replica Lag: s5 (paged) [10:08:52] 7660 (ACKED) db1181 (paged)/MariaDB Replica Lag: s7 (paged) [10:08:52] 7661 (ACKED) db1160 (paged)/MariaDB Replica Lag: s4 (paged) [10:08:52] 7662 (ACKED) db1189 (paged)/MariaDB Replica Lag: s3 (paged) [10:08:53] 7663 (ACKED) db1230 (paged)/MariaDB Replica Lag: s5 (paged) [10:09:22] <_joe_> !ack [10:09:23] 7664 (ACKED) db1237 (paged)/MariaDB Replica Lag: x1 (paged) [10:09:23] 7665 (ACKED) db1222 (paged)/MariaDB Replica Lag: s2 (paged) [10:09:23] 7666 (ACKED) db2146 (paged)/MariaDB Replica Lag: s1 (paged) [10:09:23] 7667 (ACKED) db2150 (paged)/MariaDB Replica Lag: s7 (paged) [10:09:24] 7668 (ACKED) db2145 (paged)/MariaDB Replica Lag: s1 (paged) [10:09:24] 7669 (ACKED) db2157 (paged)/MariaDB Replica Lag: s5 (paged) [10:09:24] 7670 (ACKED) db2148 (paged)/MariaDB Replica Lag: s2 (paged) [10:09:25] 7671 (ACKED) db2159 (paged)/MariaDB Replica Lag: s7 (paged) [10:09:25] 7672 (ACKED) db2153 (paged)/MariaDB Replica Lag: s1 (paged) [10:09:26] 7673 (ACKED) db2176 (paged)/MariaDB Replica Lag: s1 (paged) [10:09:26] 7674 (ACKED) db2171 (paged)/MariaDB Replica Lag: s5 (paged) [10:09:27] 7675 (ACKED) db2170 (paged)/MariaDB Replica Lag: s1 (paged) [10:09:27] 7676 (ACKED) db2168 (paged)/MariaDB Replica Lag: s7 (paged) [10:09:28] 7677 (ACKED) db2174 (paged)/MariaDB Replica Lag: s1 (paged) [10:09:28] 7678 (ACKED) db2177 (paged)/MariaDB Replica Lag: s3 (paged) [10:09:29] 7679 (ACKED) db2156 (paged)/MariaDB Replica Lag: s3 (paged) [10:09:29] 7680 (ACKED) db2149 (paged)/MariaDB Replica Lag: s3 (paged) [10:09:30] 7681 (ACKED) db2190 (paged)/MariaDB Replica Lag: s3 (paged) [10:09:30] 7682 (ACKED) db2189 (paged)/MariaDB Replica Lag: s2 (paged) [10:09:31] 7683 (ACKED) db2188 (paged)/MariaDB Replica Lag: s1 (paged) [10:09:31] 7684 (ACKED) db2218 (paged)/MariaDB Replica Lag: s7 (paged) [10:09:45] <_joe_> !ack [10:09:46] 7685 (ACKED) db2180 (paged)/MariaDB Replica Lag: s6 (paged) [10:09:47] 7686 (ACKED) db2193 (paged)/MariaDB Replica Lag: s6 (paged) [10:09:47] 7687 (ACKED) db2217 (paged)/MariaDB Replica Lag: s6 (paged) [10:09:47] 7688 (ACKED) db1173 (paged)/MariaDB Replica Lag: s6 (paged) [10:10:02] <_joe_> !ack [10:10:03] 7689 (ACKED) db1255 (paged)/MariaDB Replica Lag: x3 (paged) [10:10:03] 7690 (ACKED) es1035 (paged)/MariaDB Replica Lag: es7 (paged) [10:10:03] 7691 (ACKED) es1038 (paged)/MariaDB Replica Lag: es6 (paged) [10:10:04] 7692 (ACKED) db2169 (paged)/MariaDB Replica Lag: s6 (paged) [10:10:04] 7693 (ACKED) db2158 (paged)/MariaDB Replica Lag: s6 (paged) [10:10:09] <_joe_> !ack [10:10:10] 7695 (ACKED) db2162 (paged)/MariaDB Replica Lag: x3 (paged) [10:10:47] <_joe_> !ack [10:10:48] 7696 (ACKED) db2224 (paged)/MariaDB Replica Lag: s6 (paged) [10:10:48] 7697 (ACKED) db2214 (paged)/MariaDB Replica Lag: s6 (paged) [10:10:48] 7698 (ACKED) es1037 (paged)/MariaDB Replica Lag: es6 (paged) [10:10:48] 7699 (ACKED) es1036 (paged)/MariaDB Replica Lag: es6 (paged) [10:10:48] 7700 (ACKED) db2187 (paged)/MariaDB Replica Lag: x3 (paged) [10:10:54] <_joe_> !ack [10:10:55] no value provided for parameter incident and no default available [10:10:55] All incidents are already acked. [10:11:00] <_joe_> oh wow [10:11:03] <_joe_> well :P [10:11:50] <_joe_> !ack [10:11:50] no value provided for parameter incident and no default available [10:11:50] All incidents are already acked. [10:16:07] <_joe_> !ack [10:16:08] no value provided for parameter incident and no default available [10:16:08] All incidents are already acked. [10:16:12] <_joe_> !incidents [10:16:12] 7526 (ACKED) db1170 (paged)/MariaDB Replica SQL: s7 (paged) [10:16:13] 7528 (ACKED) db1158 (paged)/MariaDB Replica SQL: s7 (paged) [10:16:13] 7531 (ACKED) db1162 (paged)/MariaDB Replica SQL: s2 (paged) [10:16:13] 7532 (ACKED) db1174 (paged)/MariaDB Replica SQL: s7 (paged) [10:16:13] 7537 (ACKED) db1156 (paged)/MariaDB Replica SQL: s2 (paged) [10:16:13] 7547 (ACKED) db1211 (paged)/MariaDB Replica SQL: x3 (paged) [10:16:14] 7559 (ACKED) db1182 (paged)/MariaDB Replica SQL: s2 (paged) [10:16:14] 7599 (ACKED) es1035 (paged)/MariaDB Replica SQL: es7 (paged) [10:16:14] 7605 (ACKED) es1038 (paged)/MariaDB Replica SQL: es6 (paged) [10:16:15] 7606 (ACKED) es1037 (paged)/MariaDB Replica SQL: es6 (paged) [10:16:15] 7607 (ACKED) es1036 (paged)/MariaDB Replica SQL: es6 (paged) [10:16:16] 7612 (ACKED) db2187 (paged)/MariaDB Replica SQL: x3 (paged) [10:16:16] 7613 (ACKED) db1182 (paged)/MariaDB Replica Lag: s2 (paged) [10:16:17] 7617 (ACKED) db1162 (paged)/MariaDB Replica Lag: s2 (paged) [10:16:17] 7622 (ACKED) db1156 (paged)/MariaDB Replica Lag: s2 (paged) [10:16:18] 7690 (ACKED) es1035 (paged)/MariaDB Replica Lag: es7 (paged) [10:16:18] 7691 (ACKED) es1038 (paged)/MariaDB Replica Lag: es6 (paged) [10:16:19] 7698 (ACKED) es1037 (paged)/MariaDB Replica Lag: es6 (paged) [10:16:19] 7699 (ACKED) es1036 (paged)/MariaDB Replica Lag: es6 (paged) [10:16:20] 7700 (ACKED) db2187 (paged)/MariaDB Replica Lag: x3 (paged) [10:16:20] 7701 (ACKED) db1188 (paged)/MariaDB Replica SQL: s2 (paged) [10:16:25] <_joe_> sigh [13:33:52] * Krinkle updated https://wikitech.wikimedia.org/wiki/Parser_cache#History [13:34:15] Added: Parallel GC, hash stops, incremental GC [13:34:19] _joe_: ^ [13:44:01] elukey: I think we could have an issue with https://gerrit.wikimedia.org/r/c/operations/puppet/+/1247022 [13:44:37] we are positive :) [13:44:46] that wiped the haproxy component on all cp servers [13:45:22] we use ensure_packages => false in profile::cache::haproxy cause ::haproxy takes care of that [13:49:50] I'd say that we need a ensure parameter and not reuse ensure_packages for that purpose [13:52:10] hey folks [13:52:34] so the haproxy component APT config was wiped right? [13:52:44] yes [13:53:04] I didn't understand why you set ensure packages false and you need to keep the component's config [13:53:21] in some setups like trixie we don't need the component [13:53:33] the package is ensured by ::haproxy anyways [13:54:16] maybe add an if based on the OS version? [13:55:11] can I suggest something like https://gerrit.wikimedia.org/r/c/operations/puppet/+/1247606? [13:56:16] I could even do it the other way around.. set $ensure based on ·ensure_packages by default so existing usage of the resource will stay the same [13:56:35] but that would still need a fix of profile::cache::haproxy [13:58:03] I am not 100% convinced this is the best fix, the haproxy profile is the corner case.. It seems that the profile should use either apt::package_from_component or apt::repository depending on the os [13:58:25] that would be a little more clear in my opinion [13:58:55] prior to your patch it was a supported feature.. add the component but not install the packages [13:59:48] and `ensure_packages` documentation is pretty clear IMHO: `If true, the default, also install the packages array with ensure_packages($packages)` [14:00:19] it doesn't say anything about controlling if the component is deployed or removed from the instances [14:00:30] that's what `ensure` should do [14:02:46] not to defend my work, but before my patch we were not cleaning up old apt configs that were probably not needed. Anyway, I see the value of not replicating the configs contained in the define with a bare apt::repository in your case, so [14:03:02] checking your patch [14:03:29] sure, I'm not saying that we shouldn't support cleaning it, but not through ensure_packages but through `ensure` [14:04:01] if you're happy with the general approach I'll amend it to document the new `ensure` parameter [14:04:11] sure sure, seems good [14:04:41] at this point I need to send a patch for kafka [14:07:03] vgutierrez: my only doubt about it is the fact that at line 82 there is a conditional that assumed ensure_packages is a boolean [14:07:19] ensure_package is still a boolean [14:08:49] ah I see $ensure == 'present' [14:08:59] is it allowed? [14:09:07] PCC is happy, linter doesn't complain.. [14:10:04] ok, gimme a sec to file the kafka patch [14:10:10] thx [14:13:13] IIUC https://gerrit.wikimedia.org/r/c/operations/puppet/+/1247612 [14:14:32] looks good as soon as you fix the arrow alignment [14:15:13] I disabled puppet on kafka hosts, you can proceed [14:16:46] thx [14:18:16] (done/merged) [14:18:21] super [14:25:54] thanks to both, I confirm that haproxy component works as expected [15:16:33] on everyone's favorite topic (config mgmt), I recently became aware of a new (and YAML-free!) option: https://github.com/pyinfra-dev/pyinfra [15:23:39] just throwing this out there, interesting link from HN today worth noodling on: https://eversole.dev/blog/we-automated-everything/ [16:02:59] Hi! Can someone add me back to the #wikimedia_security channel? I dropped off for some reason. [16:04:02] dancy: you are on the invite list already and should be able to /join that channel [16:04:45] Maybe I don't have the name right because when I join I'm the only one there. [16:06:11] jelto set me straight. :-) [17:22:31] Amir1: quick check, I assume that https://phabricator.wikimedia.org/T329491#8898529 is still true, is that correct? [17:24:08] yeah, we normalized it and it's much smaller but commons is still no-go [17:24:12] the rest, meh [17:26:14] ok, thanks, appreciated [17:47:23] actually, "meh" == "it's fine" or "I'm not too happy about it"? seems like something I should ask :D [17:58:18] it's fine :D [18:03:46] excellent, thanks, would have been silly if you and I had interpreted it differently :D [21:21:22] If somebody gets bored, it would be nice to have some understanding of what if anything is going on with spamassassin and the wikitech-l list. T386559 is a task about the obvious spam that has been making it though to that list and the apparent lack of x-spam-score headers. [21:21:23] T386559: Some messages on wikitech-l seem to lack an x-spam-score header - https://phabricator.wikimedia.org/T386559