[05:16:29] ottomata: do we have a way to query webrequests from Turnilo that includes both HTTP status and useragent map? E.g. % of HTTP 5xx responses by (user,spider,bot) or just agent=user. [05:17:32] (or if not, what'd be the second best option outside Turnilo?) [05:53:43] Krinkle: hive? 'SELECT user_agent_map, count(*) FROM wmf.webrequest WHERE year = ... AND month = ... AND day = ... AND hour = ... AND http_status >= 500 GROUP BY user_agent_map;' [06:58:28] ori: ah yeah I should look there. I misspoke on UA map, the agent bucket is "agent type" but that's indeed not under turnilo either for wmfrequest (it is for turnilo/pageview though but that doesn't have HTTP Status). [06:58:32] https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Traffic/Webrequest [09:43:20] (03PS1) 10Urbanecm: analytics/mediawiki/editgrowthconfig: Add is_registered_user [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/820661 (https://phabricator.wikimedia.org/T312148) [11:31:10] (03PS1) 10Gerrit maintenance bot: Add ig.wikiquote to pageview whitelist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/820705 (https://phabricator.wikimedia.org/T314640) [11:31:37] (03PS1) 10Gerrit maintenance bot: Add ig.wiktionary to pageview whitelist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/820707 (https://phabricator.wikimedia.org/T314646) [11:36:30] (03CR) 10Xcollazo: Add table to compute unique editors per country on a monthly basis. (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/820478 (https://phabricator.wikimedia.org/T314147) (owner: 10Xcollazo) [11:40:10] (03CR) 10Xcollazo: Add table to compute unique editors per country on a monthly basis. (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/820478 (https://phabricator.wikimedia.org/T314147) (owner: 10Xcollazo) [13:02:07] Krinkle: ya i'm not 100% up on what fields are in turnilo (backed by druid), but they are all in hive. [13:28:59] (03CR) 10Vivian Rook: "recheck" [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/816254 (https://phabricator.wikimedia.org/T308362) (owner: 10WelpThatWorked) [13:33:31] (03CR) 10CI reject: [V: 04-1] Escape '|' from wikitable output [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/816254 (https://phabricator.wikimedia.org/T308362) (owner: 10WelpThatWorked) [13:40:37] (03CR) 10Vivian Rook: "Sorry for all the 'recheck' comments. Not sure why jenkins isn't triggering automatically on this review." [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/816254 (https://phabricator.wikimedia.org/T308362) (owner: 10WelpThatWorked) [14:30:15] (03PS5) 10Milimetric: Add world map echart [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/809683 [15:01:30] 10Data-Engineering: Sqoop will fail on 2022-09-01 unless we fix templatelinks query - https://phabricator.wikimedia.org/T314666 (10Milimetric) [15:08:36] PROBLEM - Check unit status of produce_canary_events on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit produce_canary_events https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [15:19:36] RECOVERY - Check unit status of produce_canary_events on an-launcher1002 is OK: OK: Status of the systemd unit produce_canary_events https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [15:29:15] (03PS1) 10Milimetric: Use shaded jar in refine webrequest [analytics/refinery] - 10https://gerrit.wikimedia.org/r/820770 [15:29:41] (03CR) 10Milimetric: [V: 03+2 C: 03+2] Use shaded jar in refine webrequest [analytics/refinery] - 10https://gerrit.wikimedia.org/r/820770 (owner: 10Milimetric) [15:56:34] PROBLEM - Disk space on an-launcher1002 is CRITICAL: DISK CRITICAL - free space: /srv 31 MB (0% inode=98%): https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space https://grafana.wikimedia.org/d/000000377/host-overview?var-server=an-launcher1002&var-datasource=eqiad+prometheus/ops [16:01:15] !log removing airflow logs older than 7 days on an-launcher1002 [16:01:16] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:03:40] (03PS1) 10Milimetric: Remove specific jar versions from comments to enable easily finding what versions we actually use [analytics/refinery] - 10https://gerrit.wikimedia.org/r/820776 [16:05:31] !log force scap deploying refinery [16:05:32] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:16:48] RECOVERY - Disk space on an-launcher1002 is OK: DISK OK https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space https://grafana.wikimedia.org/d/000000377/host-overview?var-server=an-launcher1002&var-datasource=eqiad+prometheus/ops [16:17:16] (03PS2) 10Milimetric: Clean up specific jar versions from comments [analytics/refinery] - 10https://gerrit.wikimedia.org/r/820776 [16:35:58] 10Analytics, 10Analytics-Wikistats, 10Data-Engineering: WikiStats in Uzbek - https://phabricator.wikimedia.org/T314477 (10Malikxan) @Milimetric, can you help with this one? [16:43:42] (VarnishkafkaNoMessages) resolved: ... [16:43:42] varnishkafka for instance cp2035:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=codfw%20prometheus/ops&var-source=eventlogging&var-cp_cluster=cache_text&var-instance=cp2035:9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [17:09:29] (03CR) 10Milimetric: [V: 03+2 C: 03+2] "ah, you need to be in the analytics team gerrit group, I just asked for NJ and Sandra, you can piggyback: https://phabricator.wikimedia.or" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/820478 (https://phabricator.wikimedia.org/T314147) (owner: 10Xcollazo) [17:13:27] 10Data-Engineering-Radar, 10Gerrit-Privilege-Requests, 10Release-Engineering-Team: Requesting membership of the analytics group in gerrit for 'snwachukwu' and 'nokafor' - https://phabricator.wikimedia.org/T314592 (10xcollazo) [21:26:12] (VarnishkafkaNoMessages) firing: ... [21:26:12] varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=codfw%20prometheus/ops&var-source=eventlogging&var-cp_cluster=cache_text&var-instance=cp2027:9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [21:51:12] (VarnishkafkaNoMessages) resolved: ... [21:51:12] varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=codfw%20prometheus/ops&var-source=eventlogging&var-cp_cluster=cache_text&var-instance=cp2027:9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [22:39:09] 10Data-Engineering, 10Event Metrics, 10EventStreams, 10GrowthExperiments-CommunityConfiguration, and 3 others: editgrowthconfig schema: '' should NOT have additional properties, - https://phabricator.wikimedia.org/T314173 (10Etonkovidova) The errors seem to stop - https://logstash.wikimedia.org/goto/6e7542... [23:23:19] (03PS4) 10WelpThatWorked: Escape '|' from wikitable output [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/816254 (https://phabricator.wikimedia.org/T308362) [23:28:15] (03CR) 10WelpThatWorked: Escape '|' from wikitable output (031 comment) [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/816254 (https://phabricator.wikimedia.org/T308362) (owner: 10WelpThatWorked)