[09:22:20] btullis: The views are all done right? [09:22:27] I just checked the ones for fatwiki and they look good [09:23:11] Yes, I did all of those in the done column, but haven't resolved the tickets yet. Thanks for checking. [09:24:19] coool, I am checking and resolving [12:52:08] moritzm: in flink image I did -wmf0,etc. suffix. whatcha think instead of -s0, etc. [12:52:29] also...should I pin the java version then? right now it'll just install whatever the latest debian has [12:54:28] no, no need to pin! I was only thinking of using it in the image version, so that it's more obvious which JRE is provided [12:54:57] but just a thought, perfectly to simply stick with the previous scheme of random 0.x versions :-) [12:55:08] i like that better too, but [12:56:00] if someone rebuilds the image for some simple reason, I dunno to set some custom defaultt java encryption settings, the java version might change, if upstream debian has updated theirs, right? [12:57:26] that's true, but whoever rebuilds the image can also update the version alongside, right? [12:57:36] but I'm fine either way, just pick what you prefer [12:58:01] i like specific version better. if i were rebuilding i just wouldn't remember to check if the java version changes. i'll add a comment. [12:58:25] moritzm: is -s0 a convention somewhere? I haven't seen that before, and had used -wmf0 elsewhere [13:02:37] going to keep s0 i guess [13:04:58] I don't think there's much of a convention, no idea where s0 is coming from actually [13:05:13] someone started it and the everyone followed suit I suppose :-) [14:04:14] moritzm: okay to merge https://gerrit.wikimedia.org/r/c/operations/docker-images/production-images/+/911905 ? [14:06:10] ship it [14:11:08] Giuseppe added the -s# thing [14:11:52] it's a patch level convention to showcase that the image hasn't changed substantially but got some dependent update [14:12:05] e.g. image X updated because glic vuln on the base image [14:19:14] I am wondering why are we getting this huge increase in queries in eqiad...https://grafana.wikimedia.org/d/000000278/mysql-aggregated?orgId=1&from=now-24h&to=now&viewPanel=1 [14:19:22] The traffic shouldn't have changed that much with active-active [14:19:56] Was eqiad depooled or something? [14:21:06] codfw saw a drop in queries, the same size [14:21:20] so at least we really did just slosh the traffic around, rather than generating new queries somehow [14:21:47] Interesting yeah: https://grafana.wikimedia.org/d/000000278/mysql-aggregated?orgId=1&from=now-24h&to=now&var-site=eqiad&var-group=core&var-shard=All&var-role=All&viewPanel=2 [14:24:57] traffic levels at the appserver side still look correct, GETs are steady in both DCs and POSTs moved as expected [14:25:35] don't forget there's a ton of appserver edits and those all moved from codfw to eqiad, about 3.5 krps -- is that what we're looking at, maybe? [14:25:53] uh sorry, a ton of *api*_appserver edits [14:31:16] one thing I thought is that cloud is nearer the primary now, and that has a small, but significant impact in latency on a number of queries [14:38:50] there is still a net increase in queries in total- e.g. s4 went from 80K ops to 160K ops [14:39:41] ok I think I'm all done [14:44:40] another thing could be the stopped maintenance catching up, it would make sense it affects s4 more [15:14:18] The switchover is again a nice tiny blip https://grafana.wikimedia.org/goto/ROWd_zsVz?orgId=1 [15:24:18] nice work :) [16:05:38] hmmm.. what happened if puppet compiler starts compiling your change on a bunch of hosts but then "HTTP ERROR 403 No valid crumb was included in the request" [16:06:31] oh, maybe it's just putting data in the wrong form field once again :) [21:00:49] hello folks, is there any work being done on wikitech? it is down for me. [21:01:04] Request URL: https://wikitech.wikimedia.org/ [21:01:04] Request Method: GET [21:01:04] Status Code: 502 [21:01:05] Remote Address: 103.102.166.224:443 [21:01:24] Request from 201.196.89.140 via cp5019.eqsin.wmnet, ATS/9.1.4 [21:01:24] Error: 502, Broken pipe at 2023-04-26 21:00:09 GMT [21:01:44] filing a task [21:04:49] https://phabricator.wikimedia.org/T335453