[08:04:45] PROBLEM - Disk space on archiva1002 is CRITICAL: DISK CRITICAL - free space: / 1696 MB (1% inode=84%): /tmp 1696 MB (1% inode=84%): /var/tmp 1696 MB (1% inode=84%): https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space https://grafana.wikimedia.org/d/000000377/host-overview?var-server=archiva1002&var-datasource=eqiad+prometheus/ops [08:31:17] 10Data-Engineering, 10Discovery, 10SRE, 10wmde-team-b-tech: archiva1002 is running low on space left in the root partition - https://phabricator.wikimedia.org/T313386 (10hashar) It is erroring out now: PROBLEM - Disk space on archiva1002 is CRITICAL: DISK CRITICAL - free space: / 1696 MB (1% inode=84%) /t... [12:36:47] PROBLEM - Check systemd state on an-launcher1002 is CRITICAL: CRITICAL - degraded: The following units failed: produce_canary_events.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [12:38:15] PROBLEM - Check unit status of produce_canary_events on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit produce_canary_events https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [12:46:59] RECOVERY - Check systemd state on an-launcher1002 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [12:49:49] RECOVERY - Check unit status of produce_canary_events on an-launcher1002 is OK: OK: Status of the systemd unit produce_canary_events https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [15:32:10] 10Data-Engineering, 10Discovery, 10SRE, 10wmde-team-b-tech: archiva1002 is running low on space left in the root partition - https://phabricator.wikimedia.org/T313386 (10BTullis) I'm looking into this issue now, since I'm on leave next week and I would rather not leave it any longer. I will take the advice... [15:32:20] 10Analytics-Wikistats, 10Data-Engineering: Traffic reports: add support for wikihood app - https://phabricator.wikimedia.org/T48279 (10Aklapper) 05Open→03Invalid App is dead since 2014 [15:36:30] !log btullis@ganeti1027:~$ sudo gnt-instance modify --disk add:size=200g archiva1002.wikimedia.org [15:36:31] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [15:37:59] 10Analytics-Wikistats, 10Data-Engineering: Traffic stats: Detect Windows 8 App requests - https://phabricator.wikimedia.org/T48195 (10Aklapper) 05Open→03Invalid App has been dead for years [15:38:47] 10Data-Engineering, 10Discovery, 10SRE, 10wmde-team-b-tech: archiva1002 is running low on space left in the root partition - https://phabricator.wikimedia.org/T313386 (10BTullis) I checked that the primary (ganeti1008) and secondary (ganeti1025) nodes both have plenty of spare disk space: Then I added a 2... [15:47:59] 10Analytics-Wikistats, 10Data-Engineering: Dump stats: further analyze drop in editor counts on English Wikipedia - https://phabricator.wikimedia.org/T48199 (10Aklapper) 05Open→03Declined 9 years old; boldly declining. [15:53:09] 10Analytics-Radar, 10Analytics-Wikistats, 10Data-Engineering: Reasons for decline and deletion on Wikinews - https://phabricator.wikimedia.org/T203090 (10Aklapper) 05Open→03Declined Declining per T203090#4552470 [15:56:40] 10Analytics-Wikistats, 10Data-Engineering: Wikistats report on active editors for all projects (deduplicated) is inconsistent with other Wikistats reports - https://phabricator.wikimedia.org/T149087 (10Aklapper) 05Open→03Declined Closing this ticket as Wikistats version 1 is dead per https://stats.wikimedi... [15:56:43] 10Analytics-Wikistats, 10Data-Engineering: Mail stats showing stats for private (member only) lists? - https://phabricator.wikimedia.org/T130874 (10Aklapper) 05Open→03Declined Closing this ticket as Wikistats version 1 is dead per https://stats.wikimedia.org/Wikistats_1_announcements.htm . In case this tic... [15:56:45] 10Analytics-Wikistats, 10Data-Engineering: Provide a JSON output for wikistat tables - https://phabricator.wikimedia.org/T123742 (10Aklapper) 05Open→03Declined Closing this ticket as Wikistats version 1 is dead per https://stats.wikimedia.org/Wikistats_1_announcements.htm . In case this ticket is still a v... [15:56:47] 10Analytics-Wikistats, 10Data-Engineering: Percentage pageviews from Russia is too low in recent geographical breakdowns in Wikistats - https://phabricator.wikimedia.org/T109582 (10Aklapper) 05Open→03Declined Closing this ticket as Wikistats version 1 is dead per https://stats.wikimedia.org/Wikistats_1_ann... [15:56:49] 10Analytics-Wikistats, 10Data-Engineering: Wikistats report on crawler stats broken - https://phabricator.wikimedia.org/T96372 (10Aklapper) 05Open→03Declined Closing this ticket as Wikistats version 1 is dead per https://stats.wikimedia.org/Wikistats_1_announcements.htm . In case this ticket is still a val... [15:56:51] 10Analytics-Wikistats, 10Data-Engineering: Provide monthly editor counts in SQL databaseLoad - https://phabricator.wikimedia.org/T65071 (10Aklapper) 05Open→03Declined Closing this ticket as Wikistats version 1 is dead per https://stats.wikimedia.org/Wikistats_1_announcements.htm . In case this ticket is st... [15:56:53] 10Analytics-Wikistats, 10Data-Engineering: Enable parallel processing of stub dump and full archive dump for same wiki. - https://phabricator.wikimedia.org/T62826 (10Aklapper) 05Open→03Declined Closing this ticket as Wikistats version 1 is dead per https://stats.wikimedia.org/Wikistats_1_announcements.htm... [15:56:55] 10Analytics-Wikistats, 10Data-Engineering: Only feed data file with complete months to Report Card (or extrapolate for last day). - https://phabricator.wikimedia.org/T69374 (10Aklapper) 05Open→03Declined Closing this ticket as Wikistats version 1 is dead per https://stats.wikimedia.org/Wikistats_1_announce... [15:56:57] 10Analytics-Wikistats, 10Data-Engineering: for Wikidata the report with most active users seems to mistake several bots for humans - https://phabricator.wikimedia.org/T65976 (10Aklapper) 05Open→03Declined Closing this ticket as Wikistats version 1 is dead per https://stats.wikimedia.org/Wikistats_1_announc... [15:56:59] 10Analytics-Wikistats, 10Data-Engineering: wikistats table 'Database records per namespace' is not properly aligned for some wikis - https://phabricator.wikimedia.org/T67153 (10Aklapper) 05Open→03Declined Closing this ticket as Wikistats version 1 is dead per https://stats.wikimedia.org/Wikistats_1_announc... [15:57:01] 10Analytics-Wikistats, 10Data-Engineering: Segregate and document configuration variables - https://phabricator.wikimedia.org/T64566 (10Aklapper) 05Open→03Declined Closing this ticket as Wikistats version 1 is dead per https://stats.wikimedia.org/Wikistats_1_announcements.htm . In case this ticket is still... [15:57:03] 10Analytics-Wikistats, 10Data-Engineering: Mozilla user agent is now 7.21% - https://phabricator.wikimedia.org/T64311 (10Aklapper) 05Open→03Declined Closing this ticket as Wikistats version 1 is dead per https://stats.wikimedia.org/Wikistats_1_announcements.htm . In case this ticket is still a valid bug re... [15:57:05] 10Analytics-Wikistats, 10Data-Engineering: Wikibooks content report has not been updated for long time. - https://phabricator.wikimedia.org/T61824 (10Aklapper) 05Open→03Declined Closing this ticket as Wikistats version 1 is dead per https://stats.wikimedia.org/Wikistats_1_announcements.htm . In case this t... [15:57:07] 10Analytics-Wikistats, 10Data-Engineering: Add wikidata view stats to http://stats.wikimedia.org/wikispecial/EN/TablesPageViewsMonthly.htm - https://phabricator.wikimedia.org/T48380 (10Aklapper) 05Open→03Declined Closing this ticket as Wikistats version 1 is dead per https://stats.wikimedia.org/Wikistats_1... [15:57:09] 10Analytics-Wikistats, 10Data-Engineering: Wikistats Portal: split over-fat index.html into 4 files, do away with jquery tabs - https://phabricator.wikimedia.org/T48214 (10Aklapper) 05Open→03Declined Closing this ticket as Wikistats version 1 is dead per https://stats.wikimedia.org/Wikistats_1_announcement... [15:57:11] 10Analytics-Wikistats, 10Data-Engineering: Dump stats: use last completed month for sitemap page view counts, not current month - https://phabricator.wikimedia.org/T48209 (10Aklapper) 05Open→03Declined Closing this ticket as Wikistats version 1 is dead per https://stats.wikimedia.org/Wikistats_1_announceme... [15:57:13] 10Analytics-Wikistats, 10Data-Engineering: Page View stats: validate monthly forecasts fluctuations - https://phabricator.wikimedia.org/T48203 (10Aklapper) [15:57:15] 10Analytics-Wikistats, 10Data-Engineering: Dump stats: wikisource stats are much too low - https://phabricator.wikimedia.org/T48196 (10Aklapper) 05Open→03Declined Closing this ticket as Wikistats version 1 is dead per https://stats.wikimedia.org/Wikistats_1_announcements.htm . In case this ticket is still... [15:57:17] 10Analytics-Wikistats, 10Data-Engineering: Number formatting controls shouldn't be on some traffic reports - https://phabricator.wikimedia.org/T48061 (10Aklapper) 05Open→03Declined Closing this ticket as Wikistats version 1 is dead per https://stats.wikimedia.org/Wikistats_1_announcements.htm . In case thi... [15:57:19] 10Analytics-Wikistats, 10Data-Engineering: Dump stats: switch to persistent stats rather than monthly regenerated stats - https://phabricator.wikimedia.org/T48198 (10Aklapper) 05Open→03Declined Closing this ticket as Wikistats version 1 is dead per https://stats.wikimedia.org/Wikistats_1_announcements.htm... [15:57:21] 10Analytics-Wikistats, 10Data-Engineering: Dump stats: collect monthly totals more efficiently - https://phabricator.wikimedia.org/T48208 (10Aklapper) [15:57:23] 10Analytics-Wikistats, 10Data-Engineering: Abridged "Wikimedia Stats For All Projects" table is the same as the unabridged table - https://phabricator.wikimedia.org/T89458 (10Aklapper) 05Open→03Declined Closing this ticket as Wikistats version 1 is dead per https://stats.wikimedia.org/Wikistats_1_announcem... [15:57:25] 10Analytics-Wikistats, 10Data-Engineering: Breakdown of traffic by OS does not present Google Chrome OS yet - https://phabricator.wikimedia.org/T53512 (10Aklapper) 05Open→03Declined Closing this ticket as Wikistats version 1 is dead per https://stats.wikimedia.org/Wikistats_1_announcements.htm . In case th... [15:57:27] 10Analytics-Wikistats, 10Data-Engineering: API change for Wikipedia apps since June 2012 may result in double counts - https://phabricator.wikimedia.org/T48278 (10Aklapper) 05Open→03Declined Closing this ticket as Wikistats version 1 is dead per https://stats.wikimedia.org/Wikistats_1_announcements.htm . I... [15:57:29] 10Analytics-Wikistats, 10Data-Engineering: Recent editions of SquidReportUserAgentsTimed.htm have WMF app count 10 x lower than it used to be - https://phabricator.wikimedia.org/T48274 (10Aklapper) 05Open→03Declined Closing this ticket as Wikistats version 1 is dead per https://stats.wikimedia.org/Wikistat... [15:57:31] 10Analytics-Wikistats, 10Data-Engineering: Different traffic reports show different totals for html pages served - https://phabricator.wikimedia.org/T48277 (10Aklapper) 05Open→03Declined Closing this ticket as Wikistats version 1 is dead per https://stats.wikimedia.org/Wikistats_1_announcements.htm . In ca... [15:57:33] 10Analytics-Wikistats, 10Data-Engineering: Make traffic reports account for all events in the relevant scope, also discarded/'other' events - https://phabricator.wikimedia.org/T48268 (10Aklapper) 05Open→03Declined Closing this ticket as Wikistats version 1 is dead per https://stats.wikimedia.org/Wikistats_... [15:57:35] 10Analytics-Wikistats, 10Data-Engineering: Provide ip->geo lookup for X-Forwarded-For header field - https://phabricator.wikimedia.org/T48271 (10Aklapper) 05Open→03Declined Closing this ticket as Wikistats version 1 is dead per https://stats.wikimedia.org/Wikistats_1_announcements.htm . In case this ticket... [15:57:37] 10Analytics-Wikistats, 10Data-Engineering: SquidReportArchive warns about many invalid country codes - https://phabricator.wikimedia.org/T48267 (10Aklapper) 05Open→03Declined Closing this ticket as Wikistats version 1 is dead per https://stats.wikimedia.org/Wikistats_1_announcements.htm . In case this tick... [15:57:39] 10Analytics-Wikistats, 10Data-Engineering: Dump stats: article count wrong for English in regional report - https://phabricator.wikimedia.org/T48213 (10Aklapper) 05Open→03Declined Closing this ticket as Wikistats version 1 is dead per https://stats.wikimedia.org/Wikistats_1_announcements.htm . In case this... [15:57:41] 10Analytics-Wikistats, 10Data-Engineering: Erik Zachte's list of discrepancies for old vs new traffic reports - https://phabricator.wikimedia.org/T48265 (10Aklapper) 05Open→03Declined Closing this ticket as Wikistats version 1 is dead per https://stats.wikimedia.org/Wikistats_1_announcements.htm . In case... [15:57:43] 10Analytics-Wikistats, 10Data-Engineering: Dump stats: one very active veteran user missing from reports - https://phabricator.wikimedia.org/T48211 (10Aklapper) 05Open→03Declined Closing this ticket as Wikistats version 1 is dead per https://stats.wikimedia.org/Wikistats_1_announcements.htm . In case this... [15:57:45] 10Analytics-Wikistats, 10Data-Engineering: Dump stats: collect monthly totals more efficiently - https://phabricator.wikimedia.org/T48208 (10Aklapper) 05Open→03Declined Closing this ticket as Wikistats version 1 is dead per https://stats.wikimedia.org/Wikistats_1_announcements.htm . In case this ticket is... [15:57:47] 10Analytics-Wikistats, 10Data-Engineering: Dump stats: pages with most revisions miscounted? - https://phabricator.wikimedia.org/T48210 (10Aklapper) 05Open→03Declined Closing this ticket as Wikistats version 1 is dead per https://stats.wikimedia.org/Wikistats_1_announcements.htm . In case this ticket is st... [15:57:49] 10Analytics-Wikistats, 10Data-Engineering: Dump stats: rework flow to allow for parallel data gathering multiple wikis - https://phabricator.wikimedia.org/T48206 (10Aklapper) 05Open→03Declined Closing this ticket as Wikistats version 1 is dead per https://stats.wikimedia.org/Wikistats_1_announcements.htm .... [15:57:51] 10Analytics-Wikistats, 10Data-Engineering: Page View stats: validate monthly forecasts fluctuations - https://phabricator.wikimedia.org/T48203 (10Aklapper) 05Open→03Declined Closing this ticket as Wikistats version 1 is dead per https://stats.wikimedia.org/Wikistats_1_announcements.htm . In case this ticke... [15:57:53] 10Analytics-Wikistats, 10Data-Engineering: Traffic/Dump stats: migrate code for page view reports to separate module - https://phabricator.wikimedia.org/T48200 (10Aklapper) 05Open→03Declined Closing this ticket as Wikistats version 1 is dead per https://stats.wikimedia.org/Wikistats_1_announcements.htm . I... [15:57:55] 10Analytics-Wikistats, 10Data-Engineering: Wikistats portal, fix html errors - https://phabricator.wikimedia.org/T48192 (10Aklapper) 05Open→03Declined Closing this ticket as Wikistats version 1 is dead per https://stats.wikimedia.org/Wikistats_1_announcements.htm . In case this ticket is still a valid bug... [15:57:57] 10Analytics-Wikistats, 10Data-Engineering: Dump stats: automated validation of new monthly dump stats - https://phabricator.wikimedia.org/T48197 (10Aklapper) 05Open→03Declined Closing this ticket as Wikistats version 1 is dead per https://stats.wikimedia.org/Wikistats_1_announcements.htm . In case this tic... [15:57:59] 10Analytics-Wikistats, 10Data-Engineering: Vertically Reversed Quarter Data - https://phabricator.wikimedia.org/T57621 (10Aklapper) 05Open→03Declined Closing this ticket as Wikistats version 1 is dead per https://stats.wikimedia.org/Wikistats_1_announcements.htm . In case this ticket is still a valid bug r... [15:58:01] 10Analytics-Wikistats, 10Data-Engineering: Wikinews' users should be "Wikinewsies", not "wikireporters" - https://phabricator.wikimedia.org/T32755 (10Aklapper) 05Open→03Declined Closing this ticket as Wikistats version 1 is dead per https://stats.wikimedia.org/Wikistats_1_announcements.htm . In case this t... [15:59:51] 10Analytics-Wikistats, 10Data-Engineering: Non-mobile UAs on mobile (2g/gprs, etc) IP-blocks - https://phabricator.wikimedia.org/T58628 (10Aklapper) [18:44:16] 10Analytics, 10Analytics-Wikistats, 10Data-Engineering, 10I18n: WikiReportsLocalizations.pm still fetches language names from SVN - https://phabricator.wikimedia.org/T64570 (10Aklapper) >>! In T64570#7756843, @Ottomata wrote: > wikistats 1 bug fix tasks can be declined For the records, I declined about 40... [20:36:45] !log rebooting archiva1002 to pick up new disk [20:36:46] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [20:37:30] 10Data-Engineering, 10Discovery, 10SRE, 10wmde-team-b-tech: archiva1002 is running low on space left in the root partition - https://phabricator.wikimedia.org/T313386 (10ops-monitoring-bot) VM archiva1002.wikimedia.org rebooted by btullis@cumin1001 with reason: Adding disk [20:49:02] 10Data-Engineering, 10Discovery, 10SRE, 10wmde-team-b-tech: archiva1002 is running low on space left in the root partition - https://phabricator.wikimedia.org/T313386 (10BTullis) I had to rename the network interface from ens5 to ens14 in `/etc/network/interfaces` as described here: https://wikitech.wikime... [20:55:23] PROBLEM - Check systemd state on archiva1002 is CRITICAL: CRITICAL - degraded: The following units failed: ifup@ens14.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [20:59:44] 10Data-Engineering, 10Discovery, 10SRE, 10wmde-team-b-tech: archiva1002 is running low on space left in the root partition - https://phabricator.wikimedia.org/T313386 (10BTullis) I've mounted `/dev/vdb1` to `/mnt` temporarily and started an rsync operation with: ` sudo rsync -av /var/lib/archiva/ /mnt ` On... [21:03:03] RECOVERY - Check systemd state on archiva1002 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [21:10:00] !log swapping disks on archiva1002 [21:10:01] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [21:14:53] 10Data-Engineering, 10Discovery, 10SRE, 10wmde-team-b-tech: archiva1002 is running low on space left in the root partition - https://phabricator.wikimedia.org/T313386 (10BTullis) All steps above have now been follwed, except the final removal of the backup in `/var/lib/archiva-bak` The service starts and... [21:17:40] 10Data-Engineering, 10Discovery, 10SRE, 10wmde-team-b-tech: archiva1002 is running low on space left in the root partition - https://phabricator.wikimedia.org/T313386 (10BTullis) The git-fat link service apepars to work without errors: ` Jul 24 21:15:01 archiva1002 systemd[1]: Started Archiva tool to creat... [21:23:05] I've finished the work on archiva now, so the disk space is looking much healthier now. Hopefully there won't be any more problems with it this week while I'm on leave. [21:23:11] https://www.irccloud.com/pastebin/0ZYWnBm6/ [21:25:27] RECOVERY - Disk space on archiva1002 is OK: DISK OK https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space https://grafana.wikimedia.org/d/000000377/host-overview?var-server=archiva1002&var-datasource=eqiad+prometheus/ops