[08:52:58] <dcausse>	 wdqs1010 reload status: chunk 1022 out of 1047 and then 11 lexeme chunks to go, we're getting really close
[08:53:11] <gehel>	 Keep your fingers crossed!
[10:50:19] <dcausse>	 lunch
[11:01:37] <pfischer>	 lunch 2
[11:25:04] <gehel>	 Lunch+relocation 
[11:31:36] <pfischer>	 dcausse: Would you have time to look into the airflow deployment with me? https://gerrit.wikimedia.org/r/c/wikimedia/discovery/analytics/+/889579 passes now, BTW.
[12:44:44] <dcausse>	 pfischer: sure, I'm available when you want to do the deploy
[13:41:08] <dcausse>	 gehel: could you subscribe Peter to discovery-alerts?
[13:44:42] <gehel>	 don
[13:44:45] <gehel>	 done
[13:53:16] <dcausse>	 thanks!
[14:08:36] <inflatador>	 <o/
[14:10:00] <inflatador>	 rebooting Mac for updates
[14:13:44] <dcausse>	 o/
[14:33:15] * gehel will skip the unmeeting today, conflicting ERC workshop
[14:40:15] <pfischer>	 dcausse: airflow DAG “wcqs_streaming_updater_reconcile_hourlyschedule” ran successfully :-)
[14:40:25] <dcausse>	 \o/
[14:44:49] <inflatador>	 back
[14:47:33] <gehel>	 pfischer / dcausse: T317202 is in "needs reporting", but the attached CR https://gerrit.wikimedia.org/r/c/schemas/event/primary/+/856507/ is not yet merged. So I'm assuming this should go back to "needs review"?
[14:47:34] <stashbot>	 T317202: Model the update document used by the CirrusSearch Update Pipeline - https://phabricator.wikimedia.org/T317202
[14:48:37] <dcausse>	 hm yes, I think we might want to keep this patch while the pipeline is being worked on as things might change while we implement features
[14:48:52] <dcausse>	 so either in waiting or needs review is fine to me
[14:53:40] <inflatador>	 Looks like the new PostgresDB for the new airflow instance is ready to be created re https://gerrit.wikimedia.org/r/c/operations/puppet/+/889572/ 
[14:54:03] <inflatador>	 we might need some other puppet changes before this works, checking...
[15:07:11] <gehel>	 weekly update posted: https://app.asana.com/0/0/1203994009038960
[15:08:12] <gehel>	 inflatador: I've pinged Olja about getting some support on creating that airflow instance. In an ideal world, I think we should just request a new instance to data engineering and let them provide it to us.
[15:08:27] <gehel>	 we're not in an ideal world yet :)
[15:08:55] <inflatador>	 gehel thanks. I don't know enough to know what I don't know in airflow-land ;)
[15:15:42] <dcausse>	 sigh... the more I look the less I understand how MW can stop refreshing links when there are loops...
[15:19:12] <dcausse>	 seems related to ParserOutput cache time and page.page_links_updated...
[15:48:12] <gehel>	 dcausse: is this blocking your work? Should you just let Platform Engineering sort it out?
[15:49:48] <dcausse>	 gehel: it's me being curious and it somewhats relates to the update pipeline and me trying to better understand how re-renders happen and are triggered in MW
[15:50:28] <gehel>	 make sense!
[15:52:02] <gehel>	 I've pinged the tech mgmt channel on Slack. I'm not sure if anyone really feels ownership of JobQueue these days. Mat will follow up on his side.
[15:53:04] <dcausse>	 I think Timo or Aaron might know what happens (reading the codebase)
[15:56:09] <inflatador>	 working out, will be a little late to unmtg probably
[16:37:03] <inflatador>	 back
[16:38:16] <inflatador>	 I have to give a 5m presentation on my job to my son's class (8 yr olds) today, any suggestions?
[17:39:21] <inflatador>	 lunch, back in ~1h
[18:57:29] <inflatador>	 back
[19:25:00] <inflatador>	 heading to my son's class, back in ~ 1h
[19:32:10] <inflatador>	 I was wrong, it starts in an hour
[20:00:59] <inflatador>	 The data reload cookbook failed. Not sure if the data is corrupt though, what's a good way to check (wdqs1010 is the host)
[20:04:57] <inflatador>	 error message here https://phabricator.wikimedia.org/P44684
[20:07:09] <inflatador>	 Looks like the error probably came from https://github.com/wikimedia/operations-cookbooks/blob/master/cookbooks/sre/wdqs/data-reload.py#L191
[20:07:55] <inflatador>	 ebernhardson I have to leave soon (back in ~90m ) but are you around this afternoon to help finish this up? My gut feeling is that this is a recoverable error
[20:09:45] <inflatador>	 Looks like we set 'timestamp'  at https://github.com/wikimedia/operations-cookbooks/blob/0c38b23886f6e4f352c56f137daa7ade33553f9e/cookbooks/sre/wdqs/data-reload.py#LL94C21-L94C21 but then call 'timestampS` here https://github.com/wikimedia/operations-cookbooks/blob/master/cookbooks/sre/wdqs/data-reload.py#L132 
[20:49:46] <inflatador>	  back
[21:25:59] <dcausse>	 inflatador: seems like a cookbook error indeed, will set the timestamp manually and start the updater
[21:30:55] <dcausse>	 type checks & mypy would have helped here I guess
[21:38:32] <inflatador>	 dcausse excellent! Let me know how it goes re: updater. Is it a matter of wrong data types or are we just using incorrect variable name?
[21:42:10] <dcausse>	 inflatador: I think it's a problem of type, passing str instead of datetime I think
[21:42:41] <dcausse>	 I've set the offsets manually will restart the updater now
[21:56:09] <inflatador>	 OK got it
[22:00:50] <dcausse>	 sigh... used the wrong script to setup offsets, it's consuming 32days of backlog instead of 17days, should be no big deal but will take a bit more time to catchup
[22:08:54] <inflatador>	 Still...this is the closest we've been in awhile!
[22:09:26] <inflatador>	 Also, cloudelastic restart failed due to timeout connecting to omega, I think this is just a transient error...trying again
[22:13:38] <dcausse>	 yes, I think we're getting close, let's wait for the backfill to finish (should take the whole week-end) before claiming victory :)
[22:14:21] <inflatador>	 ACK, have a great weekend!
[22:14:59] <dcausse>	 thx, you too!