[06:57:56] <wikibugs>	 10Data-Engineering-Radar, 10Privacy Engineering, 10Privacy: Privacy review for dataset publishing (Wikidata topic -> pageview data) - https://phabricator.wikimedia.org/T303304 (10Addshore) @Htriedman do you have any timeline estimates for this?
[07:07:55] <wikibugs>	 (03PS5) 10Aqu: Fix: Prevent empty normalized host [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/772027
[08:54:12] <jinxer-wm>	 (VarnishkafkaNoMessages) firing: varnishkafka for instance cp2027:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=codfw%20prometheus/ops&var-source=statsv&var-cp_cluster=cache_text&var-instance=cp2027:9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[08:59:12] <jinxer-wm>	 (VarnishkafkaNoMessages) resolved: varnishkafka for instance cp2027:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=codfw%20prometheus/ops&var-source=statsv&var-cp_cluster=cache_text&var-instance=cp2027:9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[09:10:27] <btullis>	 --^ I am going to have to alter the threshold for statsv throughput for these VarnishkafkaNoMessages alerts.
[09:43:21] <wikibugs>	 10Analytics-Radar, 10SRE, 10observability: Set up cross DC topic mirroring for Kafka logging clusters - https://phabricator.wikimedia.org/T276972 (10fgiunchedi) >>! In T276972#7824672, @Ottomata wrote: > In https://phabricator.wikimedia.org/T304373#7823916 @fgiunchedi wrote >> to clarify my position on T2769...
[10:01:27] <jinxer-wm>	 (HiveServerHeapUsage) firing: Hive Server JVM Heap usage is above 80% on an-test-coord1001:10100 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Alerts#Hive_Server_Heap_Usage - https://grafana.wikimedia.org/d/000000379/hive?panelId=7&fullscreen&orgId=1&var-instance=an-test-coord1001:10100 - https://alerts.wikimedia.org/?q=alertname%3DHiveServerHeapUsage
[10:11:28] <jinxer-wm>	 (HiveServerHeapUsage) resolved: Hive Server JVM Heap usage is above 80% on an-test-coord1001:10100 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Alerts#Hive_Server_Heap_Usage - https://grafana.wikimedia.org/d/000000379/hive?panelId=7&fullscreen&orgId=1&var-instance=an-test-coord1001:10100 - https://alerts.wikimedia.org/?q=alertname%3DHiveServerHeapUsage
[10:57:12] <jinxer-wm>	 (VarnishkafkaNoMessages) firing: varnishkafka for instance cp2035:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=codfw%20prometheus/ops&var-source=statsv&var-cp_cluster=cache_text&var-instance=cp2035:9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[11:06:57] <wikibugs>	 10Data-Engineering, 10Data-Engineering-Kanban, 10Patch-For-Review: Add alert for varnishkafka low/zero messages per second to alertmanager - https://phabricator.wikimedia.org/T300246 (10BTullis) a:05Milimetric→03BTullis Claiming this ticket back again, to reflect what I'm currently working on. Hope that'...
[11:07:12] <jinxer-wm>	 (VarnishkafkaNoMessages) resolved: varnishkafka for instance cp2035:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=codfw%20prometheus/ops&var-source=statsv&var-cp_cluster=cache_text&var-instance=cp2035:9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[11:10:42] <btullis>	 This change removes statsv from the varnishkafka check: https://gerrit.wikimedia.org/r/c/operations/alerts/+/776912 - I haven't added a specific threshold for the statsv itself, because I don't know enough about it yet.
[11:28:49] <btullis>	 This change removes the test hosts from some of the JVM heap tests: https://gerrit.wikimedia.org/r/c/operations/alerts/+/776919
[11:57:02] <ottomata>	 o/
[11:57:24] <btullis>	 Hello ottomata.
[11:59:14] <ottomata>	 hello!
[11:59:34] <ottomata>	 just saw that datahub plans to work on removal of schema reg in 'Q2', whenever that is :)
[11:59:41] <ottomata>	 i guess CY Q2?
[11:59:44] <ottomata>	 so, now?
[12:00:11] <btullis>	 Great. I'm just working on the first staging deploy of datahub now.
[12:00:16] <ottomata>	 cooool
[12:03:04] <btullis>	 Slow and steady wins the race, so they say. :-)
[12:06:50] <ottomata>	 :)
[13:21:10] <mforns>	 ottomata: have you seen my response to your comment in https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/38, it's the first comment. Please, let me know your thoughts, to se if we need more changes! :]
[13:21:14] <mforns>	 *see
[13:27:50] <ottomata>	 mforns:  yes have respoonded
[13:27:59] <mforns>	 thanksss
[13:29:43] <mforns>	 ottomata: DynamicVariable is a bit redundant no? How about VariableConfig
[13:29:44] <mforns>	 ?
[13:30:42] <ottomata>	 Hm , yahhhh hm
[13:31:03] <ottomata>	 mforns:  since it is specifically a collection dict of props
[13:31:06] <ottomata>	 VariableProperties
[13:31:06] <ottomata>	 ?
[13:31:27] <mforns>	 I like VariableProperties too :]
[13:31:30] <ottomata>	 okay!
[13:31:35] <mforns>	 k!
[13:32:19] <ottomata>	 hehe, i had started typing  a comment on the MR about "hm, maybe expecting to be json is too magic, and we should just have a to_dict method"....but then I was  like...wait if we do that then there is no advantage over using a plain ol Variable :)
[13:32:25] <ottomata>	 so, I started to get it :)
[13:46:27] <mforns>	 ottomata: another related question
[13:46:57] <mforns>	 sometimes there are properties that you want to override, that are not primitive types, i.e. datetimes or timedeltas
[13:48:00] <mforns>	 One way of solving that (the current way) is to pass them as strings, and parse them from within the DAG code into datetimes and timedeltas.
[13:48:43] <mforns>	 Another way, would be to make the VariableProperties class deal with them, for instance by checking for tags/prefixes before the value:
[13:50:05] <mforns>	 variable value = '{"prop1": 1, "prop2": "__DATETIME__2022-02-01", "prop3": "__TIMEDELTA__days=5"}'
[13:50:28] <mforns>	 And let VariableProperties parse them into the expected type.
[13:50:52] <mforns>	 orrrrrr.....
[13:51:04] <mforns>	 hmmmm
[13:52:08] <mforns>	 if, with typed python, we can specify the type of a kwargs parameter, and we can inspect it from the class code, then we don't need to indicate it in the json blob...
[13:54:48] <mforns>	 or even without types.. just look at the type of the default value, and use it to try and parse the json.
[14:22:29] <ottomata>	 hm, mforns  what about a meta property that specifies special conversions by key?  
[14:23:04] <ottomata>	 "__conversions__": { "prop2": "datetime", "prop3": "timedelta_days_5" }
[14:23:32] <ottomata>	 with values being static methods in the VariableProperties class
[14:23:46] <ottomata>	 ...hm, and maybe some way for e.g. days=5 parameters?
[15:07:46] <wikibugs>	 10Analytics, 10Data-Engineering, 10Data-Engineering-Kanban, 10Event-Platform, and 2 others: WikipediaPortal Event Platform Migration - https://phabricator.wikimedia.org/T282012 (10Ottomata) 05Open→03Resolved
[15:07:48] <wikibugs>	 10Analytics, 10Data-Engineering, 10Event-Platform, 10Product-Analytics, 10MW-1.36-notes (1.36.0-wmf.18; 2020-11-17): Migrate legacy metawiki schemas to Event Platform - https://phabricator.wikimedia.org/T259163 (10Ottomata)
[15:10:58] <wikibugs>	 10Analytics, 10Data-Engineering, 10Event-Platform, 10Product-Analytics, 10MW-1.36-notes (1.36.0-wmf.18; 2020-11-17): Migrate legacy metawiki schemas to Event Platform - https://phabricator.wikimedia.org/T259163 (10Ottomata)
[15:57:39] <wikibugs>	 10Data-Engineering-Radar, 10Privacy Engineering, 10Privacy: Privacy review for dataset publishing (Wikidata topic -> pageview data) - https://phabricator.wikimedia.org/T303304 (10Htriedman) Hi @Addshore working on this now, hopefully I'll have it done in the next 24h!
[15:58:19] <wikibugs>	 10Data-Engineering-Radar, 10Privacy Engineering, 10Privacy: Privacy review for dataset publishing (Wikidata topic -> pageview data) - https://phabricator.wikimedia.org/T303304 (10Addshore) Amazing! I'll keep an eye out here (also on slack or IRC under the same name) if anything crops up / there are issues
[16:20:35] <wikibugs>	 10Data-Engineering-Kanban, 10Airflow: Fix use of Java LinkedHashMap caching in Spark multi-threaded environment - https://phabricator.wikimedia.org/T305386 (10Antoine_Quhen)
[16:24:46] <mforns>	 ottomata: I think that the only properties that are not primary types in Airflow are datetimes and timedeltas. I don't think we should do something super generic? If the default value is datetime or timedelta, we can try and apply an ISO8601 parsing on the corresponding json property value (both for datetimes and timedeltas). I think that would be enough no?
[16:48:26] <ottomata>	 oh okay
[16:48:36] <ottomata>	 mforns:  sounds good
[16:51:26] <mforns>	 ottomata: 👍
[17:03:50] <joal>	 ottomata, milimetric - meeting?
[17:06:56] <ottomata>	 oh joal i thought it was rescheuld?D
[17:06:57] <ottomata>	 coming!
[18:25:43] <wikibugs>	 (03CR) 10Joal: "Sorry for the delay!" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/775376 (https://phabricator.wikimedia.org/T304623) (owner: 10Snwachukwu)
[18:26:42] <wikibugs>	 (03CR) 10Joal: [V: 03+2 C: 03+2] "Merging for next deploy" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/776269 (https://phabricator.wikimedia.org/T305281) (owner: 10Gerrit maintenance bot)
[18:30:02] <wikibugs>	 (03CR) 10Joal: [C: 03+1] "LGTM :) possibly fix the SonarQ comments?" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/772027 (owner: 10Aqu)
[18:39:44] <wikibugs>	 (03CR) 10Joal: [C: 03+1] "LGTM (one nit) :)" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/774383 (https://phabricator.wikimedia.org/T300039) (owner: 10Aqu)
[18:39:44] <mforns>	 ottomata, milimetric, joal or aqu: Hm, I had a thought about Airflow's VariableProperties (a.k.a. DynamicConfig), anyone available to brainstorm?>??
[18:40:47] <milimetric>	 I'm around mforns but my brain's still not working very well
[18:41:07] <mforns>	 milimetric: please rest if you're not feeling well!!
[19:14:22] <wikibugs>	 (03PS5) 10AGueyte: Add new event action [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/774980 (https://phabricator.wikimedia.org/T296428)
[19:16:10] <wikibugs>	 (03CR) 10AGueyte: Add new event action (032 comments) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/774980 (https://phabricator.wikimedia.org/T296428) (owner: 10AGueyte)
[19:46:47] <ottomata>	 mforns:  ya can brainstorm!
[19:46:53] <mforns>	 yay!
[19:46:56] <mforns>	 bc?
[19:47:08] <ottomata>	 kay!
[20:19:55] <icinga-wm>	 PROBLEM - Check unit status of produce_canary_events on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit produce_canary_events https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[20:31:09] <icinga-wm>	 RECOVERY - Check unit status of produce_canary_events on an-launcher1002 is OK: OK: Status of the systemd unit produce_canary_events https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[20:41:13] <jinxer-wm>	 (VarnishkafkaNoMessages) firing: ...
[20:41:18] <jinxer-wm>	 varnishkafka for instance cp5010:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=eqsin%20prometheus/ops&var-source=eventlogging&var-cp_cluster=cache_text&var-instance=cp5010:9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[20:46:12] <jinxer-wm>	 (VarnishkafkaNoMessages) resolved: ...
[20:46:13] <jinxer-wm>	 varnishkafka for instance cp5010:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=eqsin%20prometheus/ops&var-source=eventlogging&var-cp_cluster=cache_text&var-instance=cp5010:9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[21:13:13] <jinxer-wm>	 (VarnishkafkaNoMessages) firing: varnishkafka for instance cp5011:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=eqsin%20prometheus/ops&var-source=statsv&var-cp_cluster=cache_text&var-instance=cp5011:9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[21:18:12] <jinxer-wm>	 (VarnishkafkaNoMessages) resolved: varnishkafka for instance cp5011:9132 is not logging cache_text requests from statsv - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=eqsin%20prometheus/ops&var-source=statsv&var-cp_cluster=cache_text&var-instance=cp5011:9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[21:50:13] <wikibugs>	 10Data-Engineering-Radar, 10Privacy Engineering, 10Privacy: Privacy review for dataset publishing (Wikidata topic -> pageview data) - https://phabricator.wikimedia.org/T303304 (10Htriedman) Hi @Addshore! Hope you're well — I'm done with my privacy review and am hoping to share it with you soon (I just need y...