[00:03:05] <icinga-wm>	 PROBLEM - MegaRAID on an-worker1085 is CRITICAL: CRITICAL: 13 LD(s) must have write cache policy WriteBack, currently using: WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring
[00:35:57] <icinga-wm>	 RECOVERY - MegaRAID on an-worker1085 is OK: OK: optimal, 13 logical, 14 physical, WriteBack policy https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring
[01:20:03] <icinga-wm>	 PROBLEM - MegaRAID on an-worker1085 is CRITICAL: CRITICAL: 13 LD(s) must have write cache policy WriteBack, currently using: WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring
[01:54:05] <icinga-wm>	 RECOVERY - MegaRAID on an-worker1085 is OK: OK: optimal, 13 logical, 14 physical, WriteBack policy https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring
[03:21:27] <jinxer-wm>	 (HiveServerHeapUsage) firing: Hive Server JVM Heap usage is above 80% on an-coord1002:10100 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Alerts#Hive_Server_Heap_Usage - https://grafana.wikimedia.org/d/000000379/hive?panelId=7&fullscreen&orgId=1&var-instance=an-coord1002:10100 - https://alerts.wikimedia.org/?q=alertname%3DHiveServerHeapUsage
[03:24:47] <icinga-wm>	 PROBLEM - MegaRAID on an-worker1085 is CRITICAL: CRITICAL: 13 LD(s) must have write cache policy WriteBack, currently using: WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring
[04:10:07] <icinga-wm>	 RECOVERY - MegaRAID on an-worker1085 is OK: OK: optimal, 13 logical, 14 physical, WriteBack policy https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring
[04:56:27] <jinxer-wm>	 (HiveServerHeapUsage) resolved: Hive Server JVM Heap usage is above 80% on an-coord1002:10100 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Alerts#Hive_Server_Heap_Usage - https://grafana.wikimedia.org/d/000000379/hive?panelId=7&fullscreen&orgId=1&var-instance=an-coord1002:10100 - https://alerts.wikimedia.org/?q=alertname%3DHiveServerHeapUsage
[05:06:55] <icinga-wm>	 PROBLEM - MegaRAID on an-worker1085 is CRITICAL: CRITICAL: 13 LD(s) must have write cache policy WriteBack, currently using: WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring
[05:52:19] <icinga-wm>	 RECOVERY - MegaRAID on an-worker1085 is OK: OK: optimal, 13 logical, 14 physical, WriteBack policy https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring
[06:26:21] <icinga-wm>	 PROBLEM - MegaRAID on an-worker1085 is CRITICAL: CRITICAL: 13 LD(s) must have write cache policy WriteBack, currently using: WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring
[06:37:35] <icinga-wm>	 RECOVERY - MegaRAID on an-worker1085 is OK: OK: optimal, 13 logical, 14 physical, WriteBack policy https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring
[07:11:25] <icinga-wm>	 PROBLEM - MegaRAID on an-worker1085 is CRITICAL: CRITICAL: 13 LD(s) must have write cache policy WriteBack, currently using: WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring
[07:22:47] <icinga-wm>	 RECOVERY - MegaRAID on an-worker1085 is OK: OK: optimal, 13 logical, 14 physical, WriteBack policy https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring
[07:26:02] <joal>	 Good morning aqu - I created https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/155 for your review please :)
[07:28:28] <aqu>	 Hello joal ,  thx
[07:28:56] <joal>	 np aqu - we'll still have to fix the 'empty parameter' issue
[07:56:50] <icinga-wm>	 PROBLEM - MegaRAID on an-worker1085 is CRITICAL: CRITICAL: 13 LD(s) must have write cache policy WriteBack, currently using: WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring
[08:08:11] <icinga-wm>	 RECOVERY - MegaRAID on an-worker1085 is OK: OK: optimal, 13 logical, 14 physical, WriteBack policy https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring
[08:14:01] <aqu>	 joal currently working on your branch to fix unit tests + lints
[08:14:20] <joal>	 aqu: ok thanks - I didn't check - please excuse me
[08:14:38] <joal>	 aqu: sending a PR for the apis-graphite issue in minutes
[08:19:48] <joal>	 hm actually aqu, those linting errors are not related to my code - weird
[08:20:30] <aqu>	 Yes, it's on the main branch.
[08:22:36] <joal>	 hm
[08:23:28] <joal>	 have there been a change on linter version? this is weird that it starts failing now
[08:24:30] <aqu>	 I don't think so, but the concerned files were recently changed.
[08:24:35] <joal>	 Ah
[08:24:54] <joal>	 and merged without waiting for checks I guess :(
[08:26:02] <joal>	 I'll tell Marcel and Sandra to not push to master directly
[08:26:47] <joal>	 This is not good practive
[08:26:59] <joal>	 We should even configure gitlab to prevent that
[08:27:08] <wikibugs>	 10Data-Engineering-Operations: RAID battery alert in an-worker1085 - https://phabricator.wikimedia.org/T318659 (10BTullis)
[08:32:20] <btullis>	 > We should even configure gitlab to prevent that
[08:32:20] <btullis>	 Morning. Yes, I agree. At the moment it looks like 'Maintainers' and above can push to the main branch, but not force push. https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/settings/repository
[08:32:29] <btullis>	 Maybe we should change it to 'Owners'
[08:32:44] <joal>	 hm actually I think it should be prevented from now on
[08:32:54] <joal>	 an change to the main branch should go through PR
[08:33:44] <btullis>	 joal: OK, so apply this change, right? 
[08:33:47] <btullis>	 https://usercontent.irccloud-cdn.com/file/CYyGt1DG/image.png
[08:33:48] <joal>	 Cause for instance almost all of analytics is "owner" of the project
[08:34:30] <joal>	 +1 btullis 
[08:34:32] <joal>	 thanks for that
[08:35:04] <btullis>	 Done. 
[08:36:38] <btullis>	 joal: Thanks also for the update about the Cassandra 2 loading. I'll find the ticket about decommissioning aqs100[4-9] and put it on the board. I'll write out the decom plan.
[08:36:46] <joal>	 aqu: I have issues running unit-test on my machine
[08:36:58] <joal>	 Awesome btullis :) thank you for that!
[08:38:06] <joal>	 aqu: I have a lot (29) tests failing, all due to 'No module named ...' with different modules (analytics, wmf_airflow_common.sensors ...)
[08:38:10] <joal>	 aqu: any idea?
[08:39:36] <aqu>	 joal: `export PYTHONPATH=.` ?
[08:39:49] <joal>	 AHHHHHH! thank you  - I forgot about that!
[08:40:14] * joal is going to update the docs right now
[08:40:26] <aqu>	 I'm using `autoenv` to forget about those kind of things
[08:41:00] <aqu>	 I could commit a `.env.example`
[08:41:12] <joal>	 Could be nice
[08:44:04] <joal>	 also aqu, I think it would have been nicer to make a different PR for the artifact addition instead of adding it as a patch to something different
[08:44:44] <joal>	 aqu: nit details, but scope-creep-PR are not great :)
[08:45:37] <joal>	 aqu: nonetheless, thank you a lot for the fix!
[08:45:41] <aqu>	 Actually I was thinking to cherry-pick to main directly before merging your branch. But then you both talked about not pushing to main directly... wrong timing :)
[08:46:08] <joal>	 ehehe :)
[08:46:33] <joal>	 PRs, even for small stuff, and as small patchsets as possible :)
[08:47:11] * joal is becoming a git extremist
[08:51:15] <joal>	 aqu: still a nit on types
[08:51:27] <btullis>	 I also put myself in the git hygiene extremist camp. Sorry :-0
[08:51:30] <btullis>	 I also put myself in the git hygiene extremist camp. Sorry :-)
[09:24:07] <icinga-wm>	 RECOVERY - MegaRAID on an-worker1146 is OK: OK: optimal, 13 logical, 14 physical, WriteBack policy https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring
[10:17:43] <wikibugs>	 10Data-Engineering: Stop ingesting data to the old AQS cluster - https://phabricator.wikimedia.org/T302276 (10BTullis) With the completion of {T306962} this step is now complete. This unblocks the decommissioning of aqs100[4-9].
[10:18:00] <wikibugs>	 10Data-Engineering: Stop ingesting data to the old AQS cluster - https://phabricator.wikimedia.org/T302276 (10BTullis) 05Open→03Resolved
[10:18:02] <wikibugs>	 10Data-Engineering, 10Cassandra, 10Epic, 10Platform Team Workboards (Platform Engineering Reliability): Cassandra3 migration for Analytics AQS - https://phabricator.wikimedia.org/T249755 (10BTullis)
[10:53:37] <aqu>	 joal: https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/157
[10:53:59] <joal>	 reading!
[10:56:46] <joal>	 aqu: one nit about a comment, then I'll merge
[11:11:14] <aqu>	 Done
[11:43:14] <joal>	 Merging aqu - sorry I missed the ping
[11:43:21] <joal>	 actually - merged
[13:01:12] <wikibugs>	 10Data-Engineering, 10API Platform, 10Platform Engineering Roadmap, 10User-Eevans: AQS 2.0: Pageviews: Implement Unit Tests - https://phabricator.wikimedia.org/T299735 (10BPirkle) @FGoodwin , for the final test, "character encoding in article titles is happily handled (should handle per article queries wit...
[13:01:16] <wikibugs>	 10Data-Engineering: Failed to find any Kerberos tgt - https://phabricator.wikimedia.org/T318063 (10bmansurov)
[13:21:22] <wikibugs>	 (03CR) 10Mforns: [C: 03+1] "LGTM! Neil, please feel free to read the session_days field! See inline comment." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/832383 (https://phabricator.wikimedia.org/T312262) (owner: 10Neil P. Quinn-WMF)
[13:25:51] <wikibugs>	 10Data-Engineering: Failed to find any Kerberos tgt - https://phabricator.wikimedia.org/T318063 (10Ottomata) https://wikitech.wikimedia.org/wiki/Analytics/Systems/Kerberos/UserGuide#Run_a_command_as_a_system_user?
[13:30:02] <wikibugs>	 (03PS1) 10Joal: Fix mediawiki-history-denormalize for spark 3 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/835618 (https://phabricator.wikimedia.org/T318589)
[13:31:04] <joal>	 mforns: if you're happy, I think all three reviews I have created this morning are ready to be merged
[13:31:34] <mforns>	 joal: I'm going over a looooot of alerts, can you refresh my memory? :]
[13:33:53] <joal>	 mforns: those 3 PRs areabout fixing all alerts
[13:34:04] <mforns>	 oh
[13:35:30] <wikibugs>	 10Analytics, 10API Platform, 10Code-Health-Objective: Synchronize .gitignore files - https://phabricator.wikimedia.org/T315113 (10VirginiaPoundstone) a:03codebug
[13:40:51] <mforns>	 joal, merged 2 of them, for the other one left a minor comment
[13:40:58] <joal>	 ack - revieing
[13:46:43] <joal>	 update sent mforns - thanks for the reviews :)
[13:46:49] <mforns>	 lookin
[14:05:27] <wikibugs>	 10Data-Engineering, 10API Platform, 10Platform Engineering Roadmap, 10User-Eevans: Create k8s deployment of AQS 2.0 - https://phabricator.wikimedia.org/T288661 (10VirginiaPoundstone)
[14:08:01] <wikibugs>	 10Data-Engineering, 10API Platform, 10Platform Engineering Roadmap, 10User-Eevans: Obtain a security review of AQS 2.0 - https://phabricator.wikimedia.org/T288663 (10VirginiaPoundstone)
[14:08:35] <wikibugs>	 10Analytics, 10API Platform (Product Roadmap), 10Code-Health-Objective, 10Epic, and 3 others: AQS 2.0 - https://phabricator.wikimedia.org/T263489 (10BPirkle)
[14:08:49] <wikibugs>	 10Data-Engineering, 10API Platform, 10Code-Health-Objective, 10Platform Engineering Roadmap, 10User-Eevans: Dashboards for AQS 2.0 - https://phabricator.wikimedia.org/T288667 (10VirginiaPoundstone)
[14:09:01] <wikibugs>	 10Data-Engineering, 10API Platform, 10Code-Health-Objective, 10Epic, and 3 others: Problem details for HTTP APIs (rfc7807) - https://phabricator.wikimedia.org/T302536 (10BPirkle) 05Open→03Resolved a:03BPirkle Done
[14:23:03] <mforns>	 !log deployed Airflow for 3 fixes
[14:23:04] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[14:23:11] <wikibugs>	 10Data-Engineering-Kanban, 10Data Engineering Planning, 10Data Pipelines: Optimization of conda-analytics deb package - https://phabricator.wikimedia.org/T318397 (10EChetty)
[14:23:35] <mforns>	 !log rolled back Airflow
[14:23:36] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[14:24:26] <wikibugs>	 10Analytics, 10Data Engineering Planning, 10Event-Platform Value Stream, 10Patch-For-Review, 10User-Elukey: Port architecture of irc-recentchanges to Kafka - https://phabricator.wikimedia.org/T234234 (10EChetty)
[14:24:55] <wikibugs>	 10Data-Engineering-Operations, 10Data Engineering Planning, 10Mail, 10SRE: Add xcollazo@wikimedia.org to the analytics-alerts mailing list - https://phabricator.wikimedia.org/T315486 (10EChetty) p:05Medium→03High
[14:32:55] <joal>	 hey mforns - what happened?
[14:33:17] <mforns>	 joal: There's a wrong artifact coordinates
[14:33:20] <mforns>	 I think
[14:33:24] <mforns>	 I'm looking into it
[14:34:50] <joal>	 thanks a lot
[14:38:04] <mforns>	 joal: https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/158 
[14:40:06] <joal>	 mforns: merging!
[14:40:13] <mforns>	 thankssss!!
[14:56:31] <mforns>	 !log deployed Airflow (fixed)
[14:56:32] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[14:59:43] <mforns>	 !log re-ran apis_metrics_to_graphite_hourly
[14:59:43] <mforns>	  failed tasks
[14:59:45] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[15:03:00] <mforns>	 !log re-ran cassandra_daily_load failed airflow tasks
[15:03:01] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[15:05:36] <mforns>	 !log re-ran wikidata_metrics_to_graphite_daily failed airflow tasks
[15:05:38] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[15:06:28] <aqu>	 thx mforns
[15:30:36] <wikibugs>	 10Data-Engineering, 10Event-Platform Value Stream, 10Metrics-Platform, 10Patch-For-Review: Incorporate librarized Metrics Platform PHP client into EventLogging - https://phabricator.wikimedia.org/T281762 (10EChetty)
[15:32:02] <wikibugs>	 10Data-Engineering, 10Metrics-Platform, 10Patch-For-Review: Incorporate librarized Metrics Platform PHP client into EventLogging - https://phabricator.wikimedia.org/T281762 (10EChetty)
[16:25:44] <wikibugs>	 (03CR) 10Xcollazo: Fix mediawiki-history-denormalize for spark 3 (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/835618 (https://phabricator.wikimedia.org/T318589) (owner: 10Joal)
[16:30:33] <wikibugs>	 (03CR) 10Joal: Fix mediawiki-history-denormalize for spark 3 (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/835618 (https://phabricator.wikimedia.org/T318589) (owner: 10Joal)
[16:30:46] <joal>	 xcollazo: I added an answer for you there --^ :)
[17:14:12] <wikibugs>	 10Data-Engineering, 10Product-Analytics (Kanban): Superset Date Filter fix needed - https://phabricator.wikimedia.org/T318299 (10mpopov)
[17:14:31] <wikibugs>	 10Data-Engineering, 10Product-Analytics (Kanban): Superset Date Filter fix needed - https://phabricator.wikimedia.org/T318299 (10mpopov) a:03Mayakp.wiki
[17:23:45] <wikibugs>	 10Data-Engineering-Radar, 10Gerrit-Privilege-Requests, 10Release-Engineering-Team (Blocking 🧱): Requesting membership of the analytics group in gerrit for 'snwachukwu' and 'nokafor' - https://phabricator.wikimedia.org/T314592 (10xcollazo) Do we need anything else to move this forward @thcipriani ?  This is b...
[17:24:53] <wikibugs>	 10Data-Engineering-Radar, 10Gerrit-Privilege-Requests, 10Release-Engineering-Team (Blocking 🧱): Requesting membership of the analytics group in gerrit for 'snwachukwu', 'nokafor', and 'xcollazo' - https://phabricator.wikimedia.org/T314592 (10xcollazo)
[18:10:37] <wikibugs>	 10Data-Engineering, 10Product-Analytics (Kanban): Superset Date Filter fix needed - https://phabricator.wikimedia.org/T318299 (10Mayakp.wiki)
[18:18:59] <wikibugs>	 10Quarry: Comment on phabricator task on github update - https://phabricator.wikimedia.org/T318741 (10rook)
[18:19:11] <wikibugs>	 10Quarry: Comment on phabricator task on github update - https://phabricator.wikimedia.org/T318741 (10rook) 05Open→03In progress a:03rook
[18:28:33] <wikibugs>	 10Quarry: Comment on phabricator task on github update - https://phabricator.wikimedia.org/T318741 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/quarry/pull/12
[18:30:57] <wikibugs>	 10Data-Engineering, 10API Platform, 10Platform Engineering Roadmap, 10User-Eevans: AQS 2.0: Editors service - https://phabricator.wikimedia.org/T288305 (10BPirkle)
[18:32:09] <wikibugs>	 10Data-Engineering, 10API Platform, 10Platform Engineering Roadmap, 10User-Eevans: AQS 2.0: Editors service - https://phabricator.wikimedia.org/T288305 (10BPirkle)
[18:32:33] <wikibugs>	 10Data-Engineering, 10API Platform, 10Platform Engineering Roadmap, 10User-Eevans: AQS 2.0: Editors service - https://phabricator.wikimedia.org/T288305 (10FGoodwin)
[18:32:35] <wikibugs>	 10Data-Engineering, 10Product-Analytics (Kanban): Superset Date Filter fix needed - https://phabricator.wikimedia.org/T318299 (10Mayakp.wiki)
[18:34:14] <wikibugs>	 10Data-Engineering, 10API Platform, 10Platform Engineering Roadmap, 10User-Eevans: AQS 2.0: Editors service - https://phabricator.wikimedia.org/T288305 (10FGoodwin)
[18:41:24] <wikibugs>	 10Quarry: Comment on phabricator task on github update - https://phabricator.wikimedia.org/T318741 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/quarry/pull/12
[18:41:48] <wikibugs>	 10Quarry: Comment on phabricator task on github update - https://phabricator.wikimedia.org/T318741 (10rook) 05In progress→03Resolved
[19:26:43] <wikibugs>	 10Data-Engineering-Radar, 10Gerrit-Privilege-Requests, 10Release-Engineering-Team (Blocking 🧱): Requesting membership of the analytics group in gerrit for 'snwachukwu', 'nokafor', and 'xcollazo' - https://phabricator.wikimedia.org/T314592 (10thcipriani) 05Open→03Resolved a:03thcipriani @xcollazo sorry...
[19:46:37] <wikibugs>	 10Data-Engineering-Radar, 10Gerrit-Privilege-Requests, 10Release-Engineering-Team (Blocking 🧱): Requesting membership of the analytics group in gerrit for 'snwachukwu', 'nokafor', and 'xcollazo' - https://phabricator.wikimedia.org/T314592 (10xcollazo) >This should be done now. Confirmed I see I can now do +2...
[20:28:51] <wikibugs>	 (03CR) 10Xcollazo: "Went ahead with a full review, mostly because I want to pickup your changes. 😊" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/835618 (https://phabricator.wikimedia.org/T318589) (owner: 10Joal)
[22:50:14] <wikibugs>	 10Data-Engineering, 10Research, 10Epic: Add more languages to Wikipedia Clickstream - https://phabricator.wikimedia.org/T289532 (10Isaac) Adding some details about the impact of extending the language list to also include all of the languages listed under **Desired state** in the task description (languages...