[00:25:36] PROBLEM - Check unit status of monitor_refine_eventlogging_analytics on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit monitor_refine_eventlogging_analytics https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [00:37:22] RECOVERY - Check unit status of monitor_refine_eventlogging_legacy on an-launcher1002 is OK: OK: Status of the systemd unit monitor_refine_eventlogging_legacy https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [02:48:32] 10Analytics, 10ContentSecurityPolicy, 10Wikimedia-production-error: Error with Permissions-Policy header: Origin trial controlled feature not enabled: 'interest cohort' - https://phabricator.wikimedia.org/T312823 (10AlexisJazz) [02:49:35] 10Analytics, 10ContentSecurityPolicy, 10Wikimedia-production-error: Error with Permissions-Policy header: Origin trial controlled feature not enabled: 'interest cohort' - https://phabricator.wikimedia.org/T312823 (10AlexisJazz) [04:32:32] PROBLEM - Check unit status of monitor_refine_event_sanitized_analytics_immediate on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit monitor_refine_event_sanitized_analytics_immediate https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [05:27:06] (03CR) 10Aqu: Fix done file path in HDFSArchiver (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/811325 (https://phabricator.wikimedia.org/T310542) (owner: 10Aqu) [05:27:26] (03CR) 10Aqu: [C: 03+2] Fix done file path in HDFSArchiver [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/811325 (https://phabricator.wikimedia.org/T310542) (owner: 10Aqu) [05:37:32] (03Merged) 10jenkins-bot: Fix done file path in HDFSArchiver [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/811325 (https://phabricator.wikimedia.org/T310542) (owner: 10Aqu) [08:10:58] RECOVERY - Check unit status of monitor_refine_eventlogging_analytics on an-launcher1002 is OK: OK: Status of the systemd unit monitor_refine_eventlogging_analytics https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [09:27:48] (03PS3) 10Hashar: Schemas for Gerrit [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/811302 (https://phabricator.wikimedia.org/T311615) [09:28:23] (03CR) 10CI reject: [V: 04-1] Schemas for Gerrit [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/811302 (https://phabricator.wikimedia.org/T311615) (owner: 10Hashar) [09:30:46] (03CR) 10Hashar: "I went to delete the events that have a circular dependency:" [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/811302 (https://phabricator.wikimedia.org/T311615) (owner: 10Hashar) [09:33:32] (03CR) 10Hashar: "Actually /properties/type is missing cause it is marked as a constant:" [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/811302 (https://phabricator.wikimedia.org/T311615) (owner: 10Hashar) [12:04:50] (03PS4) 10Joal: [WIP] Update refine to use Iceberg for event_sanitize [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/811212 (https://phabricator.wikimedia.org/T311739) [12:08:05] (03PS5) 10Joal: [WIP] Update refine to use Iceberg for event_sanitize [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/811212 (https://phabricator.wikimedia.org/T311739) [12:56:31] RECOVERY - Check systemd state on an-launcher1002 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [13:03:49] RECOVERY - Check unit status of monitor_refine_event_sanitized_analytics_immediate on an-launcher1002 is OK: OK: Status of the systemd unit monitor_refine_event_sanitized_analytics_immediate https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [13:05:11] 10Data-Engineering: Check home/HDFS leftovers of dsharpe - https://phabricator.wikimedia.org/T310463 (10Ottomata) Let's discuss on T299315 [14:16:43] (03PS6) 10Joal: [WIP] Update refine to use Iceberg for event_sanitize [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/811212 (https://phabricator.wikimedia.org/T311739) [14:17:03] (03PS7) 10Joal: Update refine to use Iceberg for event_sanitize [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/811212 (https://phabricator.wikimedia.org/T311739) [14:17:28] ottomata: Hello :) I think this ready for review - there have some changes as I'm doing more tests [14:43:47] (03CR) 10Ottomata: "looking good!" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/811212 (https://phabricator.wikimedia.org/T311739) (owner: 10Joal) [14:54:05] (03PS1) 10Aqu: Add Changelog for 0.2.3 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/813249 (https://phabricator.wikimedia.org/T310542) [14:54:12] joal: just left a buncha comments! [14:58:59] 10Data-Engineering: Check home/HDFS leftovers of dsharpe - https://phabricator.wikimedia.org/T310463 (10Ottomata) As discussed at https://phabricator.wikimedia.org/T299315#8072518, the /home/dsharpe/T299315 files have been moved to /home/sbassett/T299315. Proceeding with removal of dsharpe files and dirs. [15:00:30] 10Data-Engineering: Check home/HDFS leftovers of dsharpe - https://phabricator.wikimedia.org/T310463 (10Ottomata) ` sudo -u hdfs kerberos-run-command hdfs hdfs dfs -rm -r /user/dsharpe ` ` sudo cumin 'C:profile::analytics::cluster::client or C:profile::hadoop::master or C:profile::hadoop::master::standby' 'rm -... [15:00:39] 10Data-Engineering: Check home/HDFS leftovers of dsharpe - https://phabricator.wikimedia.org/T310463 (10Ottomata) 05Open→03Resolved [15:10:17] (03CR) 10Ottomata: Schemas for Gerrit (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/811302 (https://phabricator.wikimedia.org/T311615) (owner: 10Hashar) [15:16:57] (03CR) 10Aqu: [C: 03+2] Add Changelog for 0.2.3 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/813249 (https://phabricator.wikimedia.org/T310542) (owner: 10Aqu) [15:25:37] (03Merged) 10jenkins-bot: Add Changelog for 0.2.3 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/813249 (https://phabricator.wikimedia.org/T310542) (owner: 10Aqu) [15:42:40] ottomata: actually, would you have time now? [15:47:59] (03CR) 10Hashar: "In the java code, it considers a constant to be astring by default and thus never set the type. A const is pretty much a syntastic sugar " [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/811302 (https://phabricator.wikimedia.org/T311615) (owner: 10Hashar) [15:48:42] ottomata: oops, I posted my review after you commented :D [15:49:00] `const` is a mystery, the spec does not say much about it [15:49:40] 10Analytics, 10ContentSecurityPolicy, 10Wikimedia-production-error: Error with Permissions-Policy header: Origin trial controlled feature not enabled: 'interest cohort' - https://phabricator.wikimedia.org/T312823 (10AlexisJazz) https://github.com/craftcms/cms/issues/10035 https://amifloced.org/ https://blog.... [15:53:40] 10Data-Engineering-Kanban, 10Data-Catalog, 10Data Engineering Planning (Sprint 01), 10Patch-For-Review: Integrate Superset with DataHub - https://phabricator.wikimedia.org/T306903 (10BTullis) We've decided that we will test the functionality with a manual ingestion run from a stat box - using the patch tha... [16:12:54] 10Data-Engineering-Kanban, 10Data-Catalog, 10Data Engineering Planning (Sprint 01), 10Patch-For-Review: Integrate Superset with DataHub - https://phabricator.wikimedia.org/T306903 (10EChetty) [16:33:39] (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [16:40:27] (VarnishkafkaNoMessages) firing: (7) varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [16:43:57] (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [16:44:17] --^ I'm looking into these VarnishkafkaNoMessages alerts to check that they're not false positives. [16:47:01] (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [16:48:51] (VarnishkafkaNoMessages) firing: (7) varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [16:51:22] Confirmed in #wikimedia-sre that codfw has been depooled after a power incident. [16:51:27] yep [16:51:31] (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [16:51:39] I should have shared here, sorry [16:53:10] (VarnishkafkaNoMessages) firing: (8) varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [16:53:13] Thats' OK, I should have scrolled back further before asking I suppose. [16:55:12] (VarnishkafkaNoMessages) resolved: (2) varnishkafka for instance cp2039:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [17:05:51] (03CR) 10Hashar: Schemas for Gerrit (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/811302 (https://phabricator.wikimedia.org/T311615) (owner: 10Hashar) [17:06:15] ottomata: looks like eventage validates `const` keyword. I added a few tests https://github.com/wikimedia/eventgate/pull/20 [17:06:42] I am guessing jsonschema-tools should learn about that, I can add the addition to the robustness test if that sounds good [17:06:49] well tomorrow, cause it is late ;) [17:06:54] thanks for the quick reviews! [17:17:12] 10Data-Engineering-Kanban, 10Data Engineering Planning (Sprint 01), 10Patch-For-Review: Build and install spark3 assembly - https://phabricator.wikimedia.org/T310578 (10Ottomata) [17:17:58] 10Data-Engineering-Kanban, 10Data Engineering Planning (Sprint 01): Create conda-base-env with last pyspark - https://phabricator.wikimedia.org/T309227 (10Ottomata) [17:28:21] 10Data-Engineering, 10Product-Analytics: Analyze differences between checksum-based and revert-tag based reverts in mediawiki_history - https://phabricator.wikimedia.org/T266374 (10nettrom_WMF) @Isaac has made a preliminary investigation into this for English Wikipedia from May 2022. Adding the query he used a... [17:51:36] (03CR) 10Ottomata: Schemas for Gerrit (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/811302 (https://phabricator.wikimedia.org/T311615) (owner: 10Hashar) [17:55:55] Starting build #108 for job analytics-refinery-maven-release-docker [18:11:57] Project analytics-refinery-maven-release-docker build #108: 09SUCCESS in 16 min: https://integration.wikimedia.org/ci/job/analytics-refinery-maven-release-docker/108/ [18:14:07] hashar: ah justu saw IRC message, i responded on patch [18:14:35] ottomata: see that ;) Does refinery and other tools support enums? [18:14:53] I am can surely had `type: "string"` which will solve our use case [18:15:10] then I am thinking we might be able to support const easily if enum are already supported :] [18:15:32] and kudos on `eventgate` code! [18:21:19] enums are supported, yup! but they are also only used for validation [18:21:23] type is required for enums too [18:21:38] and ya, add type: string and you'll be good! [18:21:42] ty :) [18:22:11] then a const is an enum under the hood :D [18:22:35] but yeah I can get Gerrit to inject the type: string and add a comment about it being used for our platform [18:22:39] okay nice [18:23:19] then the stack would better be sure the event received has the proper constant value, then eventgate validates it just fine so I think that is covered [18:23:27] yup! [18:23:58] joal: i missed your ping! [18:25:07] thanks, I am off again ;-] [19:10:46] Starting build #66 for job analytics-refinery-update-jars-docker [19:10:56] Project analytics-refinery-update-jars-docker build #66: 04FAILURE in 9.9 sec: https://integration.wikimedia.org/ci/job/analytics-refinery-update-jars-docker/66/ [19:47:59] (03CR) 10Joal: "Thanks for the review Andrew. Only a few responses, mostly everything done. I added intentions for the DDL functions containing hard-coded" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/811212 (https://phabricator.wikimedia.org/T311739) (owner: 10Joal) [19:48:57] ottomata: Heya - is now a good moment to talk? [19:49:43] i got 7 mins! [19:50:10] joal: in slack huddle [20:07:16] (03PS8) 10Joal: Update refine to use Iceberg for event_sanitize [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/811212 (https://phabricator.wikimedia.org/T311739) [20:51:58] (03CR) 10Ottomata: [C: 03+1] "Only a couple of remaining nits, but +1 after, merge at will!" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/811212 (https://phabricator.wikimedia.org/T311739) (owner: 10Joal) [21:19:35] I broke refinery :D [INFO] There are 381 errors reported by Checkstyle 8.40 with org/wikimedia/discovery/build/tools/checkstyle/checkstyle.xml ruleset. [22:00:26] (03PS1) 10Hashar: (DO NOT SUBMIT) spark: support "const" in json schema [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/813342 [22:01:21] (03CR) 10Hashar: "I can't run the tests locally, looks like that requires a local spark server, or maybe it is because I use Java 11 or well whatever. Hopef" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/813342 (owner: 10Hashar) [22:04:20] (03CR) 10CI reject: [V: 04-1] (DO NOT SUBMIT) spark: support "const" in json schema [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/813342 (owner: 10Hashar) [22:10:40] /away [22:10:44] ;)