[01:16:25] RECOVERY - Check systemd state on an-launcher1002 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [01:29:11] PROBLEM - Check systemd state on an-launcher1002 is CRITICAL: CRITICAL - degraded: The following units failed: monitor_refine_event.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [07:25:27] 10Data-Engineering-Planning, 10Data Pipelines, 10Discovery-Search (Current work), 10Patch-For-Review: Create airflow v2 instance and supporting repos for search platform - https://phabricator.wikimedia.org/T327970 (10ops-monitoring-bot) Cookbook cookbooks.sre.ganeti.reimage started by bking@cumin1001 for h... [08:13:04] 10Data-Engineering: Check home/HDFS leftovers of mepps - https://phabricator.wikimedia.org/T329820 (10MoritzMuehlenhoff) [08:16:27] (HiveServerHeapUsage) firing: Hive Server JVM Heap usage is above 80% on an-coord1001:10100 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Alerts#Hive_Server_Heap_Usage - https://grafana.wikimedia.org/d/000000379/hive?panelId=7&fullscreen&orgId=1&var-instance=an-coord1001:10100 - https://alerts.wikimedia.org/?q=alertname%3DHiveServerHeapUsage [08:17:25] PROBLEM - puppet last run on an-presto1015 is CRITICAL: CRITICAL: Puppet last ran 1 day ago https://wikitech.wikimedia.org/wiki/Monitoring/puppet_checkpuppetrun [08:17:39] PROBLEM - puppet last run on an-presto1006 is CRITICAL: CRITICAL: Puppet last ran 1 day ago https://wikitech.wikimedia.org/wiki/Monitoring/puppet_checkpuppetrun [08:18:41] PROBLEM - puppet last run on an-presto1007 is CRITICAL: CRITICAL: Puppet last ran 1 day ago https://wikitech.wikimedia.org/wiki/Monitoring/puppet_checkpuppetrun [08:19:17] PROBLEM - puppet last run on an-presto1013 is CRITICAL: CRITICAL: Puppet last ran 1 day ago https://wikitech.wikimedia.org/wiki/Monitoring/puppet_checkpuppetrun [08:19:27] PROBLEM - puppet last run on an-presto1011 is CRITICAL: CRITICAL: Puppet last ran 1 day ago https://wikitech.wikimedia.org/wiki/Monitoring/puppet_checkpuppetrun [08:21:27] (HiveServerHeapUsage) resolved: Hive Server JVM Heap usage is above 80% on an-coord1001:10100 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Alerts#Hive_Server_Heap_Usage - https://grafana.wikimedia.org/d/000000379/hive?panelId=7&fullscreen&orgId=1&var-instance=an-coord1001:10100 - https://alerts.wikimedia.org/?q=alertname%3DHiveServerHeapUsage [08:21:45] PROBLEM - puppet last run on an-presto1009 is CRITICAL: CRITICAL: Puppet last ran 1 day ago https://wikitech.wikimedia.org/wiki/Monitoring/puppet_checkpuppetrun [08:21:45] PROBLEM - puppet last run on an-presto1014 is CRITICAL: CRITICAL: Puppet last ran 1 day ago https://wikitech.wikimedia.org/wiki/Monitoring/puppet_checkpuppetrun [08:21:57] (HiveServerHeapUsage) firing: Hive Server JVM Heap usage is above 80% on an-coord1001:10100 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Alerts#Hive_Server_Heap_Usage - https://grafana.wikimedia.org/d/000000379/hive?panelId=7&fullscreen&orgId=1&var-instance=an-coord1001:10100 - https://alerts.wikimedia.org/?q=alertname%3DHiveServerHeapUsage [08:23:31] RECOVERY - puppet last run on an-presto1006 is OK: OK: Puppet is currently disabled (Create presto cluster for perf testing - T329525 - nfraison), not alerting. Last run 5 minutes ago with 0 failures https://wikitech.wikimedia.org/wiki/Monitoring/puppet_checkpuppetrun [08:31:42] (HiveServerHeapUsage) resolved: Hive Server JVM Heap usage is above 80% on an-coord1001:10100 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Alerts#Hive_Server_Heap_Usage - https://grafana.wikimedia.org/d/000000379/hive?panelId=7&fullscreen&orgId=1&var-instance=an-coord1001:10100 - https://alerts.wikimedia.org/?q=alertname%3DHiveServerHeapUsage [10:39:46] (03PS1) 10Joal: Update hql edit hourly computation script [analytics/refinery] - 10https://gerrit.wikimedia.org/r/889766 [10:55:28] PROBLEM - Kerberos KDC daemon on krb2001 is CRITICAL: PROCS CRITICAL: 9 processes with args /usr/sbin/krb5kdc https://wikitech.wikimedia.org/wiki/Analytics/Systems/Kerberos%23Daemons_and_their_roles [11:02:53] joal: o/ [11:02:55] bonjour [11:03:02] are you folks doing anything with an-presto nodes? [11:03:05] Bonjour! [11:03:11] elukey: we do indeed! [11:03:36] elukey: we were having issues adding the 10 new nodes to the cluster - queries were then failing [11:03:51] We spent time this week with nfraison investigating, and found the issue [11:04:12] elukey: see https://phabricator.wikimedia.org/T329525 [11:04:44] I assume the kerberos error we just saw is related to nfraison and moritzm doing stuff in relation with our issue [11:05:08] sorry for that, I had to restrat the KDCs twice to effect a new setting [11:05:26] okok :) [11:05:28] np moritzm - no big deal on our end, just an alert shooting :) [11:05:39] elukey: May I ask why this question? [11:05:50] HAve you noticed something special? [11:06:10] Except from the various alerts I mean... [11:06:53] joal: nono I noticed the alert and since it is very unusual I asked around :) [11:07:10] ack elukey :) [11:07:22] PROBLEM - Kerberos KDC daemon on krb1001 is CRITICAL: PROCS CRITICAL: 17 processes with args /usr/sbin/krb5kdc https://wikitech.wikimedia.org/wiki/Analytics/Systems/Kerberos%23Daemons_and_their_roles [11:16:30] 10Data-Engineering-Planning, 10Event-Platform Value Stream, 10Growth-Team (Current Sprint), 10MW-1.40-notes (1.40.0-wmf.23; 2023-02-13): Remove CommentFormatter from EventFactory constructor, or otherwise make its usage optional - https://phabricator.wikimedia.org/T327065 (10kostajh) 05In progress→03Res... [11:17:41] (03CR) 10Kosta Harlan: "Thanks for the fix, and sorry for the breakage." [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/888642 (https://phabricator.wikimedia.org/T302925) (owner: 10Aqu) [11:30:50] RECOVERY - Kerberos KDC daemon on krb1001 is OK: PROCS OK: 1 process with args /usr/sbin/krb5kdc https://wikitech.wikimedia.org/wiki/Analytics/Systems/Kerberos%23Daemons_and_their_roles [11:30:58] RECOVERY - Kerberos KDC daemon on krb2001 is OK: PROCS OK: 1 process with args /usr/sbin/krb5kdc https://wikitech.wikimedia.org/wiki/Analytics/Systems/Kerberos%23Daemons_and_their_roles [11:47:14] PROBLEM - Kerberos KDC daemon on krb1001 is CRITICAL: PROCS CRITICAL: 9 processes with args /usr/sbin/krb5kdc https://wikitech.wikimedia.org/wiki/Analytics/Systems/Kerberos%23Daemons_and_their_roles [11:47:24] PROBLEM - Kerberos KDC daemon on krb2001 is CRITICAL: PROCS CRITICAL: 9 processes with args /usr/sbin/krb5kdc https://wikitech.wikimedia.org/wiki/Analytics/Systems/Kerberos%23Daemons_and_their_roles [11:48:09] (03PS5) 10Milimetric: Migrate pageview dumps jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/889532 (https://phabricator.wikimedia.org/T329646) [11:58:08] RECOVERY - Kerberos KDC daemon on krb1001 is OK: PROCS OK: 1 process with args /usr/sbin/krb5kdc https://wikitech.wikimedia.org/wiki/Analytics/Systems/Kerberos%23Daemons_and_their_roles [12:00:08] RECOVERY - Kerberos KDC daemon on krb2001 is OK: PROCS OK: 1 process with args /usr/sbin/krb5kdc https://wikitech.wikimedia.org/wiki/Analytics/Systems/Kerberos%23Daemons_and_their_roles [12:03:12] (03PS1) 10Nmaphophe: GDI Equity Landscape Tables/Scripts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/889512 [12:36:04] (03CR) 10Joal: "Hopefully the final round :)" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/889532 (https://phabricator.wikimedia.org/T329646) (owner: 10Milimetric) [12:56:50] 10Data-Engineering: Add backend field to webrequest Hive table - https://phabricator.wikimedia.org/T257354 (10LSobanski) I don't see anything actionable for #SRE, untagging. [13:28:43] (03PS6) 10Milimetric: Migrate pageview dumps jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/889532 (https://phabricator.wikimedia.org/T329646) [13:29:25] (03CR) 10Milimetric: Migrate pageview dumps jobs (037 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/889532 (https://phabricator.wikimedia.org/T329646) (owner: 10Milimetric) [13:44:06] hm, joal one sec, I wanna change the temp dir path to not include the agent type, it's worse for dags that way [13:45:21] (03PS7) 10Milimetric: Migrate pageview dumps jobs [analytics/refinery] - 10https://gerrit.wikimedia.org/r/889532 (https://phabricator.wikimedia.org/T329646) [13:45:22] ack milimetric - I assume that making the output-dir explicit is clearer in the dag: same value everywhere - makes sense! good catch [13:45:25] done [13:46:00] yeah, exactly, it was just easier to test this way [13:46:05] great [13:49:46] (03CR) 10Joal: [C: 03+1] "LGTM - Thanks milimetric :)" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/889532 (https://phabricator.wikimedia.org/T329646) (owner: 10Milimetric) [14:12:59] 10Data-Engineering, 10Event-Platform Value Stream (Sprint 09): Document Flink job deployment to k8s - https://phabricator.wikimedia.org/T329629 (10lbowmaker) [14:17:02] FYI, I'm rolling out git security updates on the stat* hosts. This update (among other fixes) backports a change https://gerrit.wikimedia.org/r/c/operations/puppet/+/888053 [14:17:03] add an owner check which prevents git operations by a user different [14:17:05] than the one which owns the .git directory within the repository [14:17:47] this unlikely applies to any setup people have on stat*, but if that's the case, a safe.directory config setting can be made [14:18:14] we did the same here for the puppet masters (which use such a setup): https://gerrit.wikimedia.org/r/c/operations/puppet/+/888053 [14:18:50] mostly as a headsup in case someone reports it, I doubt this really affetcs anyone on stat* [14:36:20] 10Analytics-Radar, 10Data-Engineering-Icebox, 10SRE, 10Traffic: Requests to (hard) redirect pages return their target's contents but are counted as pageviews to the redirect page - https://phabricator.wikimedia.org/T125015 (10LSobanski) [14:37:27] ack moritzm - thank you for letting us know [14:39:00] 10Analytics, 10Data-Engineering-Icebox: Augment NEL reports with a computed timestamp-of-generation - https://phabricator.wikimedia.org/T266886 (10LSobanski) [14:44:33] 10Analytics-Radar, 10Data-Engineering-Icebox: Upgrade to Kafka MirrorMaker 2 - https://phabricator.wikimedia.org/T277467 (10LSobanski) I don't see a clear action for #SRE so I'm removing the tag. Please add us back when needed. [15:39:50] 10Data-Engineering-Planning, 10Shared-Data-Infrastructure (Shared-Data-Infra Sprint 08): Upgrade Presto servers to Bullseye - https://phabricator.wikimedia.org/T329361 (10nfraison) Presto service is up but didn't succeed to reach presto coordinator: ` Feb 16 15:35:01 an-test-presto1001 presto-server[540653]:... [15:56:56] 10Data-Engineering-Planning, 10MediaWiki-extensions-EventLogging, 10Metrics-Platform-Planning: Allow JavaScript errors to fail CI builds - https://phabricator.wikimedia.org/T318902 (10kostajh) [16:10:04] 10Data-Engineering-Planning, 10Shared-Data-Infrastructure (Shared-Data-Infra Sprint 08): Upgrade Presto servers to Bullseye - https://phabricator.wikimedia.org/T329361 (10nfraison) Newly created server.p12 file is missing the intermediate certificate. It is the same on an-test-coord1001 On prod cluster the in... [16:11:48] (03PS1) 10Snwachukwu: Update changelog.md with new version 0.2.12 changes. [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/889816 [16:14:35] (03CR) 10Mforns: [V: 03+2 C: 03+2] Update changelog.md with new version 0.2.12 changes. [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/889816 (owner: 10Snwachukwu) [16:18:10] Starting build #117 for job analytics-refinery-maven-release-docker [16:32:17] Project analytics-refinery-maven-release-docker build #117: 09SUCCESS in 14 min: https://integration.wikimedia.org/ci/job/analytics-refinery-maven-release-docker/117/ [16:33:16] (03PS17) 10Aqu: Remove Guava from dependency [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/883118 (https://phabricator.wikimedia.org/T327072) [16:33:25] (03CR) 10CI reject: [V: 04-1] Remove Guava from dependency [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/883118 (https://phabricator.wikimedia.org/T327072) (owner: 10Aqu) [16:36:17] (03PS18) 10Aqu: Remove Guava from dependency [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/883118 (https://phabricator.wikimedia.org/T327072) [16:39:18] (03CR) 10CI reject: [V: 04-1] Remove Guava from dependency [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/883118 (https://phabricator.wikimedia.org/T327072) (owner: 10Aqu) [16:41:55] Starting build #76 for job analytics-refinery-update-jars-docker [16:42:12] (03PS1) 10Maven-release-user: Add refinery-source jars for v0.2.12 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/889515 [16:42:12] Project analytics-refinery-update-jars-docker build #76: 09SUCCESS in 16 sec: https://integration.wikimedia.org/ci/job/analytics-refinery-update-jars-docker/76/ [16:48:11] (03CR) 10Snwachukwu: [C: 03+2] Add refinery-source jars for v0.2.12 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/889515 (owner: 10Maven-release-user) [16:48:41] (03CR) 10Snwachukwu: [V: 03+2 C: 03+2] Add refinery-source jars for v0.2.12 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/889515 (owner: 10Maven-release-user) [16:52:12] (03PS19) 10Aqu: Remove Guava from dependency [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/883118 (https://phabricator.wikimedia.org/T327072) [16:55:41] !log Deployed refinery-source change to remove Github.io from Mediasites definition of referers. [16:55:45] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:03:59] aqu: Heya - I'm finished with Marcel - Do you wish to talk now, or tomorrow? [18:11:55] 10Data-Engineering-Planning, 10Data Pipelines, 10Discovery-Search (Current work): migrate mjolnir application and dag to airflow v2 and spark3 - https://phabricator.wikimedia.org/T329239 (10EBernhardson) >>! In T329239#8618060, @BTullis wrote: > Hello all. Am I right in my thinking that: > > 1) when this ta... [18:22:37] (03PS1) 10Snwachukwu: Bump up the refinery jar version and update record_version of webrequest load job. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/889833 [18:23:02] (03CR) 10Snwachukwu: [C: 03+1] Bump up the refinery jar version and update record_version of webrequest load job. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/889833 (owner: 10Snwachukwu) [18:23:57] (03CR) 10Mforns: [V: 03+2 C: 03+2] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/889833 (owner: 10Snwachukwu) [18:37:05] !log killed webrequest bundle ooze jobs to deploy refinery changes. [18:37:05] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:43:20] (03CR) 10Mforns: [V: 03+2 C: 03+2] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/887371 (https://phabricator.wikimedia.org/T327074) (owner: 10Snwachukwu) [18:44:30] (03CR) 10Snwachukwu: Update Webrequest table to include referer_data column. (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/887371 (https://phabricator.wikimedia.org/T327074) (owner: 10Snwachukwu) [18:45:38] 10Data-Engineering-Planning, 10Data Pipelines, 10Discovery-Search (Current work): Migrate drop_old_data_daily.py from airflow 2 to airflow 3 - https://phabricator.wikimedia.org/T329870 (10EBernhardson) [18:46:11] 10Data-Engineering-Planning, 10Data Pipelines, 10Discovery-Search (Current work): Migrate export_queries_to_relforge.py from airflow 1 to airflow 2 - https://phabricator.wikimedia.org/T329871 (10EBernhardson) [18:46:31] 10Data-Engineering-Planning, 10Data Pipelines, 10Discovery-Search (Current work): Migrate drop_old_data_daily.py from airflow 1 to airflow 2 - https://phabricator.wikimedia.org/T329870 (10EBernhardson) [18:46:58] !log started deploying analytics refinery [18:46:58] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:47:46] 10Data-Engineering-Planning, 10Data Pipelines, 10Discovery-Search (Current work): Migrate glent_weekly.py from airflow 1 to airflow 2 - https://phabricator.wikimedia.org/T329872 (10EBernhardson) [18:48:05] 10Data-Engineering-Planning, 10Data Pipelines, 10Discovery-Search (Current work): Migrate import_cirrus_indexes.py from airflow 1 to airflow 2 - https://phabricator.wikimedia.org/T329873 (10EBernhardson) [18:48:32] 10Data-Engineering-Planning, 10Data Pipelines, 10Discovery-Search (Current work): Migrate import_ttl.py from airflow 1 to airflow 2 - https://phabricator.wikimedia.org/T329874 (10EBernhardson) [18:49:03] 10Data-Engineering-Planning, 10Data Pipelines, 10Discovery-Search (Current work): Migrate incoming_links.py from airflow 1 to airflow 2 - https://phabricator.wikimedia.org/T329875 (10EBernhardson) [18:49:21] 10Data-Engineering-Planning, 10Data Pipelines, 10Discovery-Search (Current work): Migrate ores_predictions.py from airflow 1 to airflow 2 - https://phabricator.wikimedia.org/T329876 (10EBernhardson) [18:49:44] 10Data-Engineering-Planning, 10Data Pipelines, 10Discovery-Search (Current work): Migrate popularity_score.py from airflow 1 to airflow 2 - https://phabricator.wikimedia.org/T329877 (10EBernhardson) [18:50:58] 10Data-Engineering-Planning, 10Data Pipelines, 10Discovery-Search (Current work): Migrate query_clicks.py from airflow 1 to airflow 2 - https://phabricator.wikimedia.org/T329878 (10EBernhardson) [18:51:27] 10Data-Engineering-Planning, 10Data Pipelines, 10Discovery-Search (Current work): Migrate rdf_streaming_updater_reconcile.py from airflow 1 to airflow 2 - https://phabricator.wikimedia.org/T329879 (10EBernhardson) [18:51:52] 10Data-Engineering-Planning, 10Data Pipelines, 10Discovery-Search (Current work): Migrate search_satisfaction.py from airflow 1 to airflow 2 - https://phabricator.wikimedia.org/T329880 (10EBernhardson) [18:52:11] 10Data-Engineering-Planning, 10Data Pipelines, 10Discovery-Search (Current work): Migrate transfer_to_es.py from airflow 1 to airflow 2 - https://phabricator.wikimedia.org/T329881 (10EBernhardson) [18:56:53] 10Data-Engineering-Planning, 10Data Pipelines, 10Discovery-Search (Current work): Migrate rdf_streaming_updater_reconcile.py from airflow 1 to airflow 2 - https://phabricator.wikimedia.org/T329879 (10EBernhardson) [19:07:54] hey mforns and SandraEbele - is the deploy ok for you? [19:08:07] it's going! [19:08:09] I see that oozie jobs got killed a while back and are not yet restarted [19:08:34] We're about to finish the syncing to HDFS, and we'll alter the table [19:08:40] and then restart jobs [19:08:42] perfect :) [19:09:25] yea, it's about 30 mins now that the bundle has been off [19:12:53] joal: BTW, we just received a stacktrace when executing the HDFS sync... [19:13:16] Ah crap [19:13:27] it happened to Steve as well the other day :( [19:13:37] hm [19:14:19] joal: do you know if someone got to a fix for the deployment at lease? [19:14:22] *least? [19:14:46] IIRC after multiple tries it worked [19:16:27] (HiveServerHeapUsage) firing: Hive Server JVM Heap usage is above 80% on an-coord1001:10100 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Alerts#Hive_Server_Heap_Usage - https://grafana.wikimedia.org/d/000000379/hive?panelId=7&fullscreen&orgId=1&var-instance=an-coord1001:10100 - https://alerts.wikimedia.org/?q=alertname%3DHiveServerHeapUsage [19:19:22] mforns: I need to leave - I'll be back in a few hours and check in before going to bed [19:19:31] joal: ok, thank you! [19:51:27] (HiveServerHeapUsage) resolved: Hive Server JVM Heap usage is above 80% on an-coord1001:10100 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Alerts#Hive_Server_Heap_Usage - https://grafana.wikimedia.org/d/000000379/hive?panelId=7&fullscreen&orgId=1&var-instance=an-coord1001:10100 - https://alerts.wikimedia.org/?q=alertname%3DHiveServerHeapUsage [20:45:39] (03PS1) 10Snwachukwu: Update Create webrequest table statement. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/889862 [20:46:01] (03CR) 10Snwachukwu: [C: 03+1] Update Create webrequest table statement. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/889862 (owner: 10Snwachukwu) [20:49:04] https://www.irccloud.com/pastebin/qvVSUhch [20:51:52] (03CR) 10Xcollazo: [C: 03+1] "Looks good to me. Nit pick issue with commit comment." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/889766 (owner: 10Joal) [21:07:21] !log successfully deployed analytics refinery [21:07:22] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [21:09:14] !log Added new field referer_data to wmf.webrequest table using the alter table statement [21:09:15] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [21:10:08] !log restarted oozie webrequest load bundle. [21:10:09] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [21:57:43] (03PS2) 10Joal: Update hql edit hourly computation script [analytics/refinery] - 10https://gerrit.wikimedia.org/r/889766 (https://phabricator.wikimedia.org/T307569) [21:58:30] (03CR) 10Joal: "Thank you @xcollazo :)" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/889766 (https://phabricator.wikimedia.org/T307569) (owner: 10Joal) [22:06:20] (03CR) 10Joal: [V: 03+2 C: 03+2] "Merging !" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/889862 (owner: 10Snwachukwu)