[00:05:26] RECOVERY - MegaRAID on an-worker1079 is OK: OK: optimal, 13 logical, 14 physical, WriteBack policy https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring [00:21:26] PROBLEM - Check systemd state on an-web1001 is CRITICAL: CRITICAL - degraded: The following units failed: hardsync-published.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [00:31:22] RECOVERY - Check systemd state on an-web1001 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [01:02:46] PROBLEM - MegaRAID on an-worker1079 is CRITICAL: CRITICAL: 13 LD(s) must have write cache policy WriteBack, currently using: WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring [02:00:10] RECOVERY - MegaRAID on an-worker1079 is OK: OK: optimal, 13 logical, 14 physical, WriteBack policy https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring [02:34:30] PROBLEM - MegaRAID on an-worker1079 is CRITICAL: CRITICAL: 13 LD(s) must have write cache policy WriteBack, currently using: WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring [02:52:59] (03PS1) 10Gergő Tisza: mediawiki/client/error: rename 'tags' to 'labels' [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/829295 (https://phabricator.wikimedia.org/T316992) [02:53:32] (03CR) 10CI reject: [V: 04-1] mediawiki/client/error: rename 'tags' to 'labels' [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/829295 (https://phabricator.wikimedia.org/T316992) (owner: 10Gergő Tisza) [02:54:53] (03PS2) 10Gergő Tisza: mediawiki/client/error: rename 'tags' to 'labels' [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/829295 (https://phabricator.wikimedia.org/T316992) [02:55:22] (03CR) 10CI reject: [V: 04-1] mediawiki/client/error: rename 'tags' to 'labels' [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/829295 (https://phabricator.wikimedia.org/T316992) (owner: 10Gergő Tisza) [02:57:20] RECOVERY - MegaRAID on an-worker1079 is OK: OK: optimal, 13 logical, 14 physical, WriteBack policy https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring [03:02:23] (03PS3) 10Gergő Tisza: mediawiki/client/error: rename 'tags' to 'labels' [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/829295 (https://phabricator.wikimedia.org/T316992) [04:17:36] PROBLEM - MegaRAID on an-worker1079 is CRITICAL: CRITICAL: 13 LD(s) must have write cache policy WriteBack, currently using: WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring [04:40:13] (03PS4) 10Gergő Tisza: mediawiki/client/error: rename 'tags' to 'error_context' [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/829295 (https://phabricator.wikimedia.org/T316992) [04:41:23] (03CR) 10Gergő Tisza: "Search showing the field name is not in use: https://logstash.wikimedia.org/goto/13393ce19fdf84bb0228b91ca10b6f99" [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/829295 (https://phabricator.wikimedia.org/T316992) (owner: 10Gergő Tisza) [05:55:34] RECOVERY - MegaRAID on an-worker1079 is OK: OK: optimal, 13 logical, 14 physical, WriteBack policy https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring [06:29:44] PROBLEM - MegaRAID on an-worker1079 is CRITICAL: CRITICAL: 13 LD(s) must have write cache policy WriteBack, currently using: WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring [06:52:18] RECOVERY - MegaRAID on an-worker1079 is OK: OK: optimal, 13 logical, 14 physical, WriteBack policy https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring [07:26:32] PROBLEM - MegaRAID on an-worker1079 is CRITICAL: CRITICAL: 13 LD(s) must have write cache policy WriteBack, currently using: WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring [07:38:02] RECOVERY - MegaRAID on an-worker1079 is OK: OK: optimal, 13 logical, 14 physical, WriteBack policy https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring [08:46:32] PROBLEM - MegaRAID on an-worker1079 is CRITICAL: CRITICAL: 13 LD(s) must have write cache policy WriteBack, currently using: WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough, WriteThrough https://wikitech.wikimedia.org/wiki/MegaCli%23Monitoring [12:49:03] !log pkill remaining processes of user effeietsanders on stat1008 to unblock puppet [12:49:04] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [20:54:21] 10Analytics, 10Analytics-Wikistats, 10Data-Engineering: Get visibility which pages are being heavily edited, plundered, which need patrolling - https://phabricator.wikimedia.org/T315196 (10Aklapper) [20:54:39] 10Analytics-Wikistats, 10Data-Engineering: Feature requests for Active Editors by Country - https://phabricator.wikimedia.org/T304720 (10Aklapper) [20:57:09] 10Analytics-Wikistats, 10Data-Engineering, 10CSS: Metric widget won't stretch when content overflows - https://phabricator.wikimedia.org/T246913 (10Aklapper) {F35509167} {F35509168} [22:22:41] 10Data-Engineering: Fix `refinery-drop-older-than` script for end-of-month/end-of-year - https://phabricator.wikimedia.org/T316746 (10JArguello-WMF) Does this belong to data pipelines? or is it tech debt in general?