[00:36:50] <icinga-wm>	 PROBLEM - Check unit status of monitor_refine_eventlogging_legacy on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit monitor_refine_eventlogging_legacy https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[00:38:28] <icinga-wm>	 PROBLEM - Check systemd state on an-launcher1002 is CRITICAL: CRITICAL - degraded: The following units failed: monitor_refine_eventlogging_legacy.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
[00:55:00] <icinga-wm>	 PROBLEM - MegaRAID on an-worker1082 is CRITICAL: CRITICAL: 13 LD(s) must have write cache policy WriteBack, currently using: WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring
[02:26:38] <icinga-wm>	 RECOVERY - MegaRAID on an-worker1082 is OK: OK: optimal, 13 logical, 14 physical, WriteBack policy https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring
[03:00:50] <icinga-wm>	 PROBLEM - MegaRAID on an-worker1082 is CRITICAL: CRITICAL: 13 LD(s) must have write cache policy WriteBack, currently using: WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring
[04:55:17] <icinga-wm>	 RECOVERY - MegaRAID on an-worker1082 is OK: OK: optimal, 13 logical, 14 physical, WriteBack policy https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring
[05:29:42] <icinga-wm>	 PROBLEM - MegaRAID on an-worker1082 is CRITICAL: CRITICAL: 13 LD(s) must have write cache policy WriteBack, currently using: WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring
[06:50:54] <wikibugs>	 10Data-Engineering-Kanban, 10Data Engineering Planning: Investigate why airflow sensor tasks fail without sending errors - https://phabricator.wikimedia.org/T311976 (10JAllemandou)
[06:59:00] <wikibugs>	 10Data-Engineering: Check home/HDFS leftovers of mewoph - https://phabricator.wikimedia.org/T311978 (10MoritzMuehlenhoff)
[07:38:41] <icinga-wm>	 RECOVERY - MegaRAID on an-worker1082 is OK: OK: optimal, 13 logical, 14 physical, WriteBack policy https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring
[08:04:15] <elukey>	 !log kill leftover processes of user `mewoph` on stat100x to allow puppet runs
[08:04:16] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[08:10:26] <icinga-wm>	 PROBLEM - MegaRAID on an-worker1082 is CRITICAL: CRITICAL: 13 LD(s) must have write cache policy WriteBack, currently using: WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring
[08:18:13] <RhinosF1>	 btullis: another server in the same batch ^ 1081 has gone before
[08:46:22] <btullis>	 RhinosF1: Many thanks. I will ack it and see if there is anything we can do. We may hva eto ask DC Ops to harvest another battery from a server that is to be decomm'd.
[08:48:15] <RhinosF1>	 btullis: possibly. I imagine they are off today because 4th July
[08:49:34] <wikibugs>	 10Data-Engineering: RAID battery alert in an-worker1082 - https://phabricator.wikimedia.org/T311991 (10BTullis)
[08:50:13] <icinga-wm>	 ACKNOWLEDGEMENT - MegaRAID on an-worker1082 is CRITICAL: CRITICAL: 13 LD(s) must have write cache policy WriteBack, currently using: WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough Btullis Investigating in T311991 https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring
[08:51:40] <btullis>	 RhinosF1: Yep, thanks. It's not super-urgent anyway and I can do a fair bit of investigation myself. I've acked the alert, but if it keeps flapping and comes back than I'll set downtime for it against T311991
[08:51:41] <stashbot>	 T311991: RAID battery alert in an-worker1082 - https://phabricator.wikimedia.org/T311991
[08:54:57] <RhinosF1>	 btullis: thanks
[08:55:33] <wikibugs>	 10Data-Engineering: RAID battery alert in an-worker1082 - https://phabricator.wikimedia.org/T311991 (10BTullis) The battery is not currently being charged and has a very low state of charge. ` btullis@an-worker1082:~$ sudo megacli -AdpBbuCmd -aALL  BBU status for Adapter: 0  BatteryType: BBU Battery State: Unkno...
[09:46:10] <wikibugs>	 10Data-Engineering, 10Data-Persistence (Consultation): Move Mediawiki QueryPages computation to Hadoop - https://phabricator.wikimedia.org/T309738 (10Ladsgroup) >>! In T309738#8045121, @Milimetric wrote: >> do we have a way to handle that? AFAIK we can only handle monthly reports >  > definitely, we can schedu...
[10:11:50] <wikibugs>	 10Data-Engineering-Kanban, 10Data Engineering Planning (Sprint 01): Investigate why airflow sensor tasks fail without sending errors - https://phabricator.wikimedia.org/T311976 (10EChetty)
[10:19:09] <wikibugs>	 10Data-Engineering-Kanban, 10Data Engineering Planning (Sprint 01): Investigate why airflow sensor tasks fail without sending errors - https://phabricator.wikimedia.org/T311976 (10EChetty) p:05Triage→03High a:03JAllemandou
[14:15:48] <icinga-wm>	 RECOVERY - MegaRAID on an-worker1082 is OK: OK: optimal, 13 logical, 14 physical, WriteBack policy https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring
[14:42:36] <wikibugs>	 10Data-Engineering, 10Project-Admins: Archive Analytics tag - https://phabricator.wikimedia.org/T298671 (10JArguello-WMF) > I assume that Analytics-Features is also a candidate to be archived (and removed from Herald rules)?  Yes, please @Aklapper   > For open tasks tagged with Analytics but not with #Data-Eng...
[14:50:16] <icinga-wm>	 PROBLEM - MegaRAID on an-worker1082 is CRITICAL: CRITICAL: 13 LD(s) must have write cache policy WriteBack, currently using: WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring
[14:52:02] <RhinosF1>	 btullis: ^
[14:52:35] <fab>	 btullis regarding the airflow issues on an-airflow1002, the issue is still present as there is no scheduler running. We can start one manually as a workaround, but I am getting an access denied when attempting to properly restart airflow with  `sudo -u analytics-research systemctl restart airflow-scheduler@research`. 
[15:10:43] <btullis>	 fab: I'll try later. Thanks.
[15:11:03] <btullis>	 RhinosF1: Thanks. I'll configure a week's downtime for it.
[15:11:41] <wikibugs>	 10Data-Engineering-Kanban, 10Data Engineering Planning (Sprint 01): Investigate why airflow sensor tasks fail without sending errors - https://phabricator.wikimedia.org/T311976 (10JAllemandou) a:05JAllemandou→03None
[15:15:59] <wikibugs>	 10Data-Engineering-Kanban, 10Airflow, 10Data Engineering Planning (Sprint 01): Investigate why airflow sensor tasks fail without sending errors - https://phabricator.wikimedia.org/T311976 (10EChetty)
[15:17:01] <wikibugs>	 10Data-Engineering-Kanban, 10Airflow, 10Data Engineering Planning (Sprint 01): Investigate why airflow sensor tasks fail without sending errors - https://phabricator.wikimedia.org/T311976 (10EChetty) a:03mforns
[15:19:39] <wikibugs>	 10Analytics: Add help page in wikitech on what the analytics team can do for you similar to release engineering page - https://phabricator.wikimedia.org/T116188 (10JArguello-WMF) 05Open→03Declined
[15:19:48] <wikibugs>	 10Analytics: Make sunburst and stacked-bars resize with window {crow} - https://phabricator.wikimedia.org/T114162 (10JArguello-WMF) 05Open→03Declined
[15:43:39] <fab>	 thanks btullis, that would be great.
[16:14:51] <btullis>	 !log systemctl restart airflow-scheduler@research.service (on an-airflow1002)
[16:14:52] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[16:15:11] <icinga-wm>	 RECOVERY - Checks that the local airflow scheduler for airflow @research is working properly on an-airflow1002 is OK: OK: /usr/bin/env AIRFLOW_HOME=/srv/airflow-research /usr/lib/airflow/bin/airflow jobs check --job-type SchedulerJob --hostname an-airflow1002.eqiad.wmnet succeeded https://wikitech.wikimedia.org/wiki/Analytics/Systems/Airflow
[16:17:49] <btullis>	 fab: I've restarted that airflow-scheduler service. Hope it helps, but please feel free to let me know if you have any other queries.
[16:18:30] <btullis>	 RhinosF1: I've added a week's downtime to that RAID check. Will try to sort it out one way or another within that window, so that it doesn't alert again.
[16:18:50] <RhinosF1>	 ok
[16:51:03] <icinga-wm>	 RECOVERY - MegaRAID on an-worker1082 is OK: OK: optimal, 13 logical, 14 physical, WriteBack policy https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring
[17:50:34] <wikibugs>	 10Data-Engineering, 10Data-Engineering-Kanban, 10Product-Analytics, 10wmfdata-python: conda-create-stacked breaks wmfdata.presto - https://phabricator.wikimedia.org/T301734 (10nshahquinn-wmf) 05Declined→03Open >>! In T301734#8024367, @Ottomata wrote: > Okay, we are aiming to one day replace anaconda-wm...
[17:54:06] <wikibugs>	 10Data-Engineering, 10Product-Analytics, 10wmfdata-python: Support querying a range of hourly data partitions - https://phabricator.wikimedia.org/T294654 (10nshahquinn-wmf) @nettrom_WMF nice, I think this will be very useful! Looking forward to that pull request 😊
[17:56:48] <mforns>	 joal: heya :] remember we troubleshot an airflow problem with the __new__ constructor, related with how the default_args were being translated to actual Operator arguments?
[17:58:11] <joal>	 mforns: Hi! Yessir!
[17:58:35] <joal>	 mforns: similar issue with our dear sensors?
[17:59:04] <mforns>	 joal: don't know... I was just wondering if we changed something in the end?
[17:59:17] <mforns>	 I'm trying to find a root cause
[17:59:24] <joal>	 hm, let me triple think
[17:59:26] <mforns>	 That's a candidate
[18:01:34] <joal>	 mforns: IIRC we haven't changed "global behavior" - I just updated some parameters to make sure they were correctly checked/passed (looking for the CR right now)
[18:01:54] <mforns>	 joal: what I discovered is that since we changed the SLAs to be specified in the Spark task directly, they are not being triggered! I asked, because I wonder if we affected the way default_args are passed?
[18:02:04] <mforns>	 Yea, that's what I remembered
[18:02:22] <joal>	 Ah! that could very well be mforns!
[18:02:47] <joal>	 Of course sensors are regular tasks, and if we don't set SLAs for them, they don't send alerts in case of failure!
[18:02:53] <joal>	 That would make a lot fosense
[18:03:19] <mforns>	 hm.. I think all tasks should send alerts in case of failure
[18:03:41] <mforns>	 the SLA is yet another type of alert no?
[18:04:04] <joal>	 mforns: that's how we had it setup in Oozie - I don't know how airflow handles it :S
[18:05:39] <joal>	 mforns: it's in that CR that I updated some stuff about default_args: https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/62/diffs
[18:05:53] <mforns>	 joal: thank you!
[18:07:00] <joal>	 no major behavioral change - only some checkes for SparkSQL java class
[18:09:10] <joal>	 mforns: Could the problem come from the default value of the "email_on_failure" parameter we define at dag level?
[18:09:30] <mforns>	 joal: agree with no major changes
[18:09:44] <joal>	 mforns: In the main config in wmf_airflow_common, the param is set to false
[18:10:08] <joal>	 But it's not set to true in analytics/config !
[18:10:11] <mforns>	 joal: but if you look at all DAGs configuration in the UI, that value is properly set to true
[18:10:17] <mforns>	 ?
[18:10:22] <joal>	 We do it per dag?
[18:10:30] <joal>	 We possibly could do it in analytics
[18:11:18] <mforns>	 no, no, we do it in general!
[18:11:30] <joal>	 mforns: but were?
[18:11:32] <joal>	 +h
[18:12:10] <mforns>	 joal: in wmf_prod_base_default_args
[18:12:41] <mforns>	 within wmf_airflow_common/config/dag_default_args.py
[18:12:50] <joal>	 Ah my bad mforns - I missed that one
[18:12:51] <joal>	 hm
[18:13:32] <mforns>	 joal: I'm pretty sure it's because the sla is specified within the Spark task, and not at DAG-level via the default_args...
[18:13:55] <mforns>	 We migrated that behavior end of april.
[18:13:56] <wikibugs>	 10Analytics: Could it be that the geo IP matching is not accurate for Africa? - https://phabricator.wikimedia.org/T90240 (10JArguello-WMF) 05Open→03Declined
[18:14:01] <mforns>	 Before that we had slas every week
[18:14:04] <wikibugs>	 10Analytics: Investigate lowering "per-article" resolution data in AQS - https://phabricator.wikimedia.org/T144837 (10JArguello-WMF) 05Open→03Declined
[18:14:14] <wikibugs>	 10Analytics, 10MediaWiki-Action-API: Copy cached API requests from raw webrequests table to ApiAction - https://phabricator.wikimedia.org/T155478 (10JArguello-WMF) 05Open→03Declined
[18:14:21] <wikibugs>	 10Analytics, 10Datasets-General-or-Unknown: Make AQS return 0 instead of no values {slug} - https://phabricator.wikimedia.org/T118402 (10JArguello-WMF) 05Open→03Declined
[18:14:30] <wikibugs>	 10Analytics: Fix the pageview API "top" spec and 404 reporting {slug} - https://phabricator.wikimedia.org/T117018 (10JArguello-WMF) 05Open→03Declined
[18:14:36] <mforns>	 after we only have 1 sla, triggered by the mediarequest job, which was deployed using the "old way" (using default_args).
[18:14:47] <wikibugs>	 10Analytics: reportupdater TLC - https://phabricator.wikimedia.org/T193167 (10JArguello-WMF)
[18:14:49] <wikibugs>	 10Analytics: [reportupdater] add hourly granularity - https://phabricator.wikimedia.org/T193168 (10JArguello-WMF) 05Open→03Declined
[18:14:54] <mforns>	 shortly after deploy, we changed it to pass the sla to the task, and no more SLAs :]
[18:14:56] <wikibugs>	 10Analytics-Kanban: Measuring non pageview requests - https://phabricator.wikimedia.org/T162310 (10JArguello-WMF)
[18:14:58] <wikibugs>	 10Analytics: Measure portal  pageviews - https://phabricator.wikimedia.org/T162618 (10JArguello-WMF) 05Open→03Declined
[18:15:00] <wikibugs>	 10Analytics, 10Data-Engineering, 10Event-Platform, 10EventStreams: EventStreams (via KafkaSSE) does not consume from newly added partitions in topic - https://phabricator.wikimedia.org/T173006 (10JArguello-WMF) 05Open→03Declined
[18:15:03] <wikibugs>	 10Analytics: Webrequest tagging and distribution. Measuring non-pageview requests - https://phabricator.wikimedia.org/T164019 (10JArguello-WMF)
[18:15:05] <wikibugs>	 10Analytics: [EL sanitization] Make WhitelistSanitization support arrays of structs, maps or other arrays - https://phabricator.wikimedia.org/T199230 (10JArguello-WMF) 05Open→03Declined
[18:15:08] <wikibugs>	 10Analytics-Kanban, 10Data-Engineering, 10MediaWiki-extensions-EventLogging: EventLogging sanitization - https://phabricator.wikimedia.org/T199898 (10JArguello-WMF)
[18:15:17] <joal>	 mforns: I hear your point, I still don't understand why we don't emails for task-failures :(
[18:15:21] <wikibugs>	 10Analytics, 10Voice & Tone: Rename geoeditors_blacklist_country - https://phabricator.wikimedia.org/T259804 (10JArguello-WMF) 05Open→03Declined
[18:15:22] <mforns>	 joal: my question is why!?
[18:15:29] <wikibugs>	 10Analytics, 10Analytics-Wikistats, 10Data-Engineering: Add a "number of articles with interwiki links" column in WikiStats tables - https://phabricator.wikimedia.org/T37197 (10JArguello-WMF) 05Open→03Declined
[18:15:31] <joal>	 mforns: maybe task-failure emails have never worked?
[18:15:40] <wikibugs>	 10Analytics, 10Analytics-Wikistats, 10Data-Engineering: Add overall ORES scores to Wikistats - https://phabricator.wikimedia.org/T178019 (10JArguello-WMF) 05Open→03Declined
[18:15:42] <mforns>	 joal: I think we have 2 separate problems
[18:15:42] <wikibugs>	 10Analytics, 10Analytics-Wikistats, 10Data-Engineering: Add UI in mobile view to switch to table view - https://phabricator.wikimedia.org/T191019 (10JArguello-WMF) 05Open→03Declined
[18:15:44] <wikibugs>	 10Analytics: Reload Hive2Druid datasources from already indexed data instead of raw data - https://phabricator.wikimedia.org/T232852 (10JArguello-WMF) 05Open→03Declined
[18:16:14] <mforns>	 joal: I think we might still be receiveing emails for failed Operators, but not for failed Sensors!
[18:16:30] <mforns>	 or at least, not for sensors that failed in certain ways
[18:17:32] <mforns>	 all the errors I could see today, were on sensors, and all of them had at least 1 attempt where Airflow has no record of the execution, and there are no logs!
[18:17:47] <mforns>	 It seems the sensors fail, before Airflow can notice they started.
[18:17:54] <joal>	 meh - this feels wrong :(
[18:18:35] <wikibugs>	 10Data-Engineering, 10API Platform (Product Roadmap), 10Code-Health-Objective, 10Epic, and 3 others: AQS 2.0: Create repository for shared functions - https://phabricator.wikimedia.org/T311541 (10JArguello-WMF)
[18:18:39] <mforns>	 joal: I will test the SLA hypothesis with a test DAG
[18:19:07] <joal>	 ack mforns - works for me - In any case I don't see any issue in having SLAs defined at DAG level
[18:19:09] <wikibugs>	 10Data-Engineering, 10Event-Platform, 10Internet-Archive, 10The-Wikipedia-Library, and 2 others: page-links-change stream is assigning template propagation events to the wrong edits - https://phabricator.wikimedia.org/T216504 (10JArguello-WMF)
[18:19:10] <mforns>	 if it is true, at least we can fix the SLA problem which will cover us, while we fix the sensor problem.
[18:19:20] <wikibugs>	 10Data-Engineering, 10API Platform: Establish testing procedure for Druid-based endpoints - https://phabricator.wikimedia.org/T311190 (10JArguello-WMF)
[18:20:33] <mforns>	 joal: I think SLAs should be defined at DAG level, because they start counting when the DAG is triggered, regardless of which task they belong.
[18:21:00] <mforns>	 joal: the problem is that SLAs are a parameter of the tasks! The default_args is merely passing the slas to all tasks.
[18:21:18] <wikibugs>	 10Data-Engineering: Documentathon - https://phabricator.wikimedia.org/T311413 (10JArguello-WMF)
[18:21:20] <wikibugs>	 10Analytics, 10Code-Health: Cleanup Maven dependencies in analytics/refinery - https://phabricator.wikimedia.org/T258802 (10JArguello-WMF)
[18:21:22] <joal>	 works for me mforns - IIRC I started removing them at DAG level becasue for the examples I had they were defined at task level, and I tried to get consistency
[18:21:24] <mforns>	 that's why I didn't like passing them to the DAG
[18:21:26] <wikibugs>	 10Analytics, 10Code-Health: Integrate SonarCloud analysis as part of the analytics refinery builds - https://phabricator.wikimedia.org/T258800 (10JArguello-WMF)
[18:21:32] <wikibugs>	 10Analytics, 10Code-Health: Introduce various static analysis tools to analytics/refinery - https://phabricator.wikimedia.org/T258699 (10JArguello-WMF)
[18:21:55] <mforns>	 joal: yes, I assume responsibility for this, I pushed for those to be passed to tasks.
[18:22:06] <mforns>	 well, will test
[18:22:15] <wikibugs>	 10Data-Engineering: Documentathon - https://phabricator.wikimedia.org/T311413 (10JArguello-WMF) This parent ticket has not been groomed yet, which is why there is no time frame. The idea is to give the team an example of what type of documentation is missing, therefore the subtasks.
[18:22:30] <joal>	 mforns: we both were in this - let's try dag level SLAs - and they work (I don't see why they wouldn't), at 'east we'll have some type of alerting
[18:22:43] <mforns>	 yea
[18:23:10] <wikibugs>	 10Data-Engineering, 10Documentation: Document our level of support of pyspark-based jobs - https://phabricator.wikimedia.org/T274775 (10JArguello-WMF)
[18:23:33] <mforns>	 joal: I also will test failed Spark Operator, see if it sends me an email
[18:23:37] <mforns>	 I'll let you know
[18:23:42] <wikibugs>	 10Data-Engineering, 10Code-Health: Cleanup Maven dependencies in analytics/refinery - https://phabricator.wikimedia.org/T258802 (10JArguello-WMF)
[18:23:49] <joal>	 Thanks a lot mforns
[18:24:38] <wikibugs>	 10Data-Engineering, 10Code-Health: Integrate SonarCloud analysis as part of the analytics refinery builds - https://phabricator.wikimedia.org/T258800 (10JArguello-WMF)
[18:25:13] <wikibugs>	 10Data-Engineering, 10Documentation: Update Mediawiki Table manuals on wiki (Manual:Logging_table et al) - https://phabricator.wikimedia.org/T179407 (10JArguello-WMF)
[18:28:58] <wikibugs>	 10Data-Engineering, 10Pageviews-API: Provide weekly top pageviews stats - https://phabricator.wikimedia.org/T133575 (10JArguello-WMF)
[18:29:56] <wikibugs>	 10Data-Engineering: Wikistats feature: run http server in npm run dev - https://phabricator.wikimedia.org/T296890 (10JArguello-WMF)
[18:36:07] <wikibugs>	 10Analytics-Wikistats, 10Data-Engineering: Metrics tooltip in detail page is not localized - https://phabricator.wikimedia.org/T287908 (10JArguello-WMF)
[18:37:16] <wikibugs>	 10Analytics-Wikistats, 10Data-Engineering: Change routing to accept a list of wikis in URL - https://phabricator.wikimedia.org/T283596 (10JArguello-WMF)
[18:37:23] <wikibugs>	 10Data-Engineering, 10Data-Engineering-Kanban, 10Product-Analytics, 10wmfdata-python: conda-create-stacked breaks wmfdata.presto - https://phabricator.wikimedia.org/T301734 (10nshahquinn-wmf) Also, is there a task for moving to a non-stacked setup? If not, we should create that as well.
[18:37:46] <wikibugs>	 10Analytics-Wikistats, 10Data-Engineering, 10Epic: Add ability to compare wikis - https://phabricator.wikimedia.org/T283251 (10JArguello-WMF)
[18:37:48] <wikibugs>	 10Analytics-Wikistats, 10Data-Engineering: Change state to store project as an array - https://phabricator.wikimedia.org/T283624 (10JArguello-WMF)
[18:42:12] <wikibugs>	 10Analytics-Wikistats, 10Data-Engineering, 10Inuka-Team, 10Language-strategy, and 2 others: Have a way to show the most popular pages per country - https://phabricator.wikimedia.org/T207171 (10JArguello-WMF)
[19:43:41] <mforns>	 buf joal, I'm not managing to make the dev instance generate an SLA miss...
[19:47:45] <fab>	 thanks btullis, restarting the research airflow instance resolved that issue.
[21:10:04] <wikibugs>	 10Analytics, 10Voice & Tone: Rename geoeditors_blacklist_country - https://phabricator.wikimedia.org/T259804 (10Remagoxer) Just out of curiosity, why was this declined?
[22:41:19] <wikibugs>	 10Analytics-Wikistats, 10Data-Engineering: Wikistats shows 0 views for April when data isn't available yet - https://phabricator.wikimedia.org/T281617 (10JArguello-WMF)
[22:42:05] <wikibugs>	 10Analytics-Wikistats, 10Data-Engineering: Wikistats UI: legend for bar graph not visible on mobile UI - https://phabricator.wikimedia.org/T257530 (10JArguello-WMF)
[22:42:17] <wikibugs>	 10Analytics-Wikistats, 10Data-Engineering: wikistats UI: language menu not usable after making language selection on mobile - https://phabricator.wikimedia.org/T257529 (10JArguello-WMF)
[22:42:23] <wikibugs>	 10Analytics-Wikistats, 10Data-Engineering: Change Wikistats UI language without reloading the page - https://phabricator.wikimedia.org/T251375 (10JArguello-WMF)
[22:44:00] <wikibugs>	 10Analytics-Wikistats, 10Data-Engineering: Wikimedia Statistics - Confusing number formatting/rounding on the vertical axis scale on the "Page to date" statistics graph - https://phabricator.wikimedia.org/T242790 (10JArguello-WMF)
[22:44:13] <wikibugs>	 10Analytics-Wikistats, 10Data-Engineering: Wikistats2 time related bugs - https://phabricator.wikimedia.org/T231248 (10JArguello-WMF)
[22:44:24] <wikibugs>	 10Analytics-Wikistats, 10Data-Engineering: Audit Wikistats unit testing - https://phabricator.wikimedia.org/T192836 (10JArguello-WMF)
[22:46:54] <wikibugs>	 10Analytics-Wikistats, 10Data-Engineering: URLs in description break annotations format - https://phabricator.wikimedia.org/T218845 (10JArguello-WMF)
[22:47:18] <wikibugs>	 10Analytics-Wikistats, 10Data-Engineering: Wikistats 2.0 Remaining reports. - https://phabricator.wikimedia.org/T186121 (10JArguello-WMF)
[22:47:21] <wikibugs>	 10Analytics-Wikistats, 10Data-Engineering: There are two entries for Cantonese language - https://phabricator.wikimedia.org/T215139 (10JArguello-WMF)
[22:47:27] <wikibugs>	 10Analytics-Wikistats, 10Data-Engineering: Audit and address performance issues - https://phabricator.wikimedia.org/T207197 (10JArguello-WMF)
[22:47:47] <wikibugs>	 10Analytics-Wikistats, 10Data-Engineering: Automate the deployment procedure of Wikistats 2 to Production - https://phabricator.wikimedia.org/T274126 (10JArguello-WMF)
[22:47:51] <wikibugs>	 10Analytics-Wikistats, 10Data-Engineering, 10I18n: Fixed time range names are forced to capitalized regardless of locale in sidebar - https://phabricator.wikimedia.org/T287910 (10JArguello-WMF)
[22:48:17] <wikibugs>	 10Analytics-Wikistats, 10Data-Engineering, 10I18n: Fixed time range names are forced to capitalized regardless of locale in sidebar - https://phabricator.wikimedia.org/T287910 (10JArguello-WMF)
[22:49:22] <wikibugs>	 10Analytics-Wikistats, 10Data-Engineering, 10Patch-For-Review: Expand Wikiselector to allow more than one wiki - https://phabricator.wikimedia.org/T285050 (10JArguello-WMF)
[22:57:58] <wikibugs>	 10Analytics-Wikistats, 10Data-Engineering: Offer map and queries by subdivisions of countries - https://phabricator.wikimedia.org/T284294 (10JArguello-WMF)
[22:58:30] <wikibugs>	 10Analytics-Wikistats, 10Data-Engineering: Regional and group evaluations - https://phabricator.wikimedia.org/T264512 (10JArguello-WMF)
[22:58:37] <wikibugs>	 10Analytics-Wikistats, 10Data-Engineering: Wikistats should allow more than one project - https://phabricator.wikimedia.org/T283254 (10JArguello-WMF)
[22:58:43] <wikibugs>	 10Analytics-Wikistats, 10Data-Engineering: Larger system text causes problems on mobile - https://phabricator.wikimedia.org/T258273 (10JArguello-WMF)
[22:58:52] <wikibugs>	 10Analytics-Wikistats, 10Data-Engineering: Search bar from stats.wikimedia.org displays characters poorly on iOS 12 - https://phabricator.wikimedia.org/T269827 (10JArguello-WMF)
[22:59:29] <wikibugs>	 10Data-Engineering, 10API Platform: Establish testing procedure for Druid-based endpoints - https://phabricator.wikimedia.org/T311190 (10BPirkle) Thank you!  For now, I managed to grab some data via the Hue interface. This probably isn't the best way long term, and it probably isn't the right dataset, but it w...