[10:47:44] lunch [11:12:27] Lunch 2 [12:00:11] Hey y'all, I'm preparing the March DC switchover, there's a manual step regarding WDQS lag reporting in https://wikitech.wikimedia.org/wiki/Switch_Datacenter#Manual_steps but the related task seem done. Can it be removed from the instructions? [12:27:01] claime: hey, correct I think we can remove this step [12:27:08] \o/ [12:28:31] Boom, no more manual steps for services switch [12:29:26] nice! [12:31:08] Well at least, no more that I can see x) [13:59:26] o/ [14:46:14] quick errand [17:29:55] hmm, attempting to import mjolnir repo to gitlab says `couldn't get curloptResolve config: resolved address has invalid [FILTERED] address`. I guess i can just push it, but expected that would work. [17:30:54] i think i'm going to have to skip kokkuri for mjolnir, the pipeline has been running 10 minutes now and is still downloading things from archiva, and i can't figure out how to convince kokkuri to mount a folder that gitlab runners cache into the image [17:35:43] they're presenting this at SRE mtg, looks interesting for stream processing https://www.benthos.dev/ [17:55:50] lunch, back in ~1h [19:13:20] even trying to use regular gitlab caching, something is off :S reading suggests it's related to the runners and not the repo specifically: No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted. [19:27:18] back [19:37:18] just in case, the value refresh discussions are available in the staff calendar: wikimedia.org_psfia0h5ak9i988o0bv701iglg@group.calendar.google.com [20:07:32] ebernhardson per https://phabricator.wikimedia.org/T327970 , was wondering about creating the 'instance specific scap repository' [20:07:39] is there an example repo I can crib from? [20:15:40] inflatador: yea, sec [20:15:59] inflatador: in has to be creating in gitlab data-engineering namespace though, so not sure if we can [20:17:13] inflatador: it's https://wikitech.wikimedia.org/wiki/Data_Engineering/Systems/Airflow#Create_the_instance_specific_scap_repository and about the entirety of the repo is to hold scap config. It would look like https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags-scap-analytics [20:22:11] ebernhardson ACK, will take a look [20:57:15] ebernhardson: Hey! I've heard that our cirrus dumps are a fraction of their normal size (no more details than that). [20:59:42] looking at enwiki-content from today vs from Jan 23, it looks like Jan 23 is 150M vs 4G for today. So, something fishy? [21:01:07] gehel: hmm, yea that does sound fishy [21:01:30] looks like we're also missing enwiki-general for today :/ [21:01:42] I invited y'all to the airflow scap repo https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags-scap-search . Take a look around and LMK if it doesn't work for you [21:01:57] gehel: if it's dated 20230206 those shouldn't be done yet, those are in the process of being dumped now [21:02:13] make sense [21:02:13] time to log off. See you tomorrow! [21:03:53] i suppose the reason people could get confused is the `current` symlink is changed when the first set of wikis completes (it dumps by db-shard), rather than when all wikis are complete. It wasn't clear how to have that kind of orcestration to know when all the systemd timers have completed [21:04:16] it would be easy in airflow :P [22:11:56] hmm, maybe replace the current symlink with per-shard symlinks? current-s1, current-s2, etc. Seems spammy though...