[01:39:18] 10Fundraising Sprint Never gonna git you update, 10Fundraising-Backlog, 10Patch-For-Review, 10acl*WMF-FR: Parse Venmo reports from Braintree and send donations + refunds to Civi - https://phabricator.wikimedia.org/T340801 (10AnnWF) The audit parse will be exactly the same for paypal wrote before, so just a... [02:56:14] PROBLEM - check_mysql on frdb1006 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1374 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [03:01:14] PROBLEM - check_mysql on frdb1006 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1674 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [03:06:14] PROBLEM - check_mysql on frdb1006 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1974 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [03:11:14] RECOVERY - check_mysql on frdb1006 is OK: Uptime: 700768 Threads: 14 Questions: 125956398 Slow queries: 1736627 Opens: 3185 Open tables: 1739 Queries per second avg: 179.740 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [08:10:08] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11676 MB (94% inode=98%): /var 1759 MB (6% inode=79%): /srv 97853 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 20904 MB (78% [08:10:08] ): /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [08:15:08] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11676 MB (94% inode=98%): /var 1759 MB (6% inode=79%): /srv 97853 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 20821 MB (78% [08:15:08] ): /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): /run/user/33 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [08:20:08] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11676 MB (94% inode=98%): /var 1747 MB (6% inode=79%): /srv 97853 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 20806 MB (78% [08:20:08] ): /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [08:25:08] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11676 MB (94% inode=98%): /var 1747 MB (6% inode=79%): /srv 97853 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 20822 MB (78% [08:25:08] ): /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [08:30:08] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11679 MB (94% inode=98%): /var 1747 MB (6% inode=79%): /srv 97853 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 20635 MB (77% [08:30:08] ): /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): /run/user/33 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [08:35:08] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11679 MB (94% inode=98%): /var 1747 MB (6% inode=79%): /srv 97853 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 20732 MB (78% [08:35:08] ): /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [08:40:08] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11677 MB (94% inode=98%): /var 1747 MB (6% inode=79%): /srv 97853 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 20712 MB (78% [08:40:08] ): /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [08:45:08] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11677 MB (94% inode=98%): /var 1747 MB (6% inode=79%): /srv 97840 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 20723 MB (78% [08:45:08] ): /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): /run/user/33 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [08:50:08] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11676 MB (94% inode=98%): /var 1747 MB (6% inode=79%): /srv 97853 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 20611 MB (77% [08:50:08] ): /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [08:55:08] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11676 MB (94% inode=98%): /var 1747 MB (6% inode=79%): /srv 97853 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 20627 MB (77% [08:55:08] ): /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [09:00:08] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11675 MB (94% inode=98%): /var 1747 MB (6% inode=79%): /srv 97853 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 20632 MB (77% [09:00:08] ): /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): /run/user/33 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [09:05:08] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11675 MB (94% inode=98%): /var 1747 MB (6% inode=79%): /srv 97846 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 20473 MB (77% [09:05:08] ): /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [09:10:08] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11675 MB (94% inode=98%): /var 1747 MB (6% inode=79%): /srv 97846 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 20459 MB (77% [09:10:08] ): /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [09:15:08] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11675 MB (94% inode=98%): /var 1747 MB (6% inode=79%): /srv 97846 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 20470 MB (77% [09:15:08] ): /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): /run/user/33 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [09:20:08] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11675 MB (94% inode=98%): /var 1747 MB (6% inode=79%): /srv 97846 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 20454 MB (77% [09:20:08] ): /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [09:25:08] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11675 MB (94% inode=98%): /var 1747 MB (6% inode=79%): /srv 97846 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 20461 MB (77% [09:25:08] ): /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [09:30:08] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11679 MB (94% inode=98%): /var 1747 MB (6% inode=79%): /srv 97846 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 20465 MB (77% [09:30:08] ): /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): /run/user/33 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [09:35:08] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11679 MB (94% inode=98%): /var 1747 MB (6% inode=79%): /srv 97846 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 20370 MB (76% [09:35:08] ): /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [09:40:08] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11676 MB (94% inode=98%): /var 1747 MB (6% inode=79%): /srv 97846 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 20348 MB (76% [09:40:08] ): /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [09:45:08] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11676 MB (94% inode=98%): /var 1747 MB (6% inode=79%): /srv 97837 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 20357 MB (76% [09:45:08] ): /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): /run/user/33 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [09:50:08] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11675 MB (94% inode=98%): /var 1747 MB (6% inode=79%): /srv 97837 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 20331 MB (76% [09:50:08] ): /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [09:55:08] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11675 MB (94% inode=98%): /var 1747 MB (6% inode=79%): /srv 97837 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 20249 MB (76% [09:55:08] ): /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [10:00:13] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11675 MB (94% inode=98%): /var 1747 MB (6% inode=79%): /srv 97837 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 20244 MB (76% [10:00:14] ): /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): /run/user/33 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [10:05:08] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11675 MB (94% inode=98%): /var 1747 MB (6% inode=79%): /srv 97829 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 20194 MB (76% [10:05:08] ): /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [10:10:08] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11675 MB (94% inode=98%): /var 1747 MB (6% inode=79%): /srv 97829 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 20182 MB (76% [10:10:08] ): /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [10:15:08] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11675 MB (94% inode=98%): /var 1747 MB (6% inode=79%): /srv 97829 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 20175 MB (76% [10:15:08] ): /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): /run/user/33 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [10:20:08] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11675 MB (94% inode=98%): /var 1747 MB (6% inode=79%): /srv 97829 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 20157 MB (75% [10:20:08] ): /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [10:25:08] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11675 MB (94% inode=98%): /var 1747 MB (6% inode=79%): /srv 97829 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 20056 MB (75% [10:25:08] ): /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [10:30:08] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11678 MB (94% inode=98%): /var 1747 MB (6% inode=79%): /srv 97829 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 20069 MB (75% [10:30:08] ): /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): /run/user/33 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [10:35:08] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11678 MB (94% inode=98%): /var 1747 MB (6% inode=79%): /srv 97829 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 20074 MB (75% [10:35:08] ): /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [10:40:08] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11676 MB (94% inode=98%): /var 1747 MB (6% inode=79%): /srv 97829 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 19947 MB (75% [10:40:08] ): /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [10:45:08] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11676 MB (94% inode=98%): /var 1747 MB (6% inode=79%): /srv 97829 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 19844 MB (74% [10:45:08] ): /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): /run/user/33 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [10:50:08] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11675 MB (94% inode=98%): /var 1747 MB (6% inode=79%): /srv 97829 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 19930 MB (75% [10:50:08] ): /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [10:55:08] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11675 MB (94% inode=98%): /var 1747 MB (6% inode=79%): /srv 97829 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 19839 MB (74% [10:55:08] ): /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [11:00:08] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11675 MB (94% inode=98%): /var 1747 MB (6% inode=79%): /srv 97829 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 19797 MB (74% [11:00:08] ): /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): /run/user/33 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [11:05:08] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11675 MB (94% inode=98%): /var 1747 MB (6% inode=79%): /srv 97822 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 19793 MB (74% [11:05:08] ): /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [11:10:08] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11675 MB (94% inode=98%): /var 1747 MB (6% inode=79%): /srv 97822 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 19774 MB (74% [11:10:08] ): /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [11:15:08] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11675 MB (94% inode=98%): /var 1747 MB (6% inode=79%): /srv 97822 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 19929 MB (75% [11:15:08] ): /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): /run/user/33 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [11:20:08] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11675 MB (94% inode=98%): /var 1747 MB (6% inode=79%): /srv 97822 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 19902 MB (74% [11:20:08] ): /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [11:25:08] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11675 MB (94% inode=98%): /var 1747 MB (6% inode=79%): /srv 97822 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 19873 MB (74% [11:25:08] ): /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [11:30:08] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11678 MB (94% inode=98%): /var 2167 MB (8% inode=81%): /srv 97822 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 19853 MB (74% [11:30:08] ): /var/log/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): /run/user/571 6392 MB (100% inode=99%): /run/user/33 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [11:35:08] RECOVERY - check_disk on civi1002 is OK: DISK OK - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4830 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12347 MB (99% inode=99%): /tmp 11678 MB (94% inode=98%): /var 7496 MB (28% inode=99%): /srv 97822 MB (33% inode=97%): /var/tmp 422 MB (99% inode=99%): /var/log 19918 MB (75% inode=99%) [11:35:08] g/audit 422 MB (99% inode=99%): /run/user/16759 6392 MB (100% inode=99%): /run/user/571 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [11:41:14] 10Fundraising-Backlog, 10fundraising-tech-ops: Move silverpop_export data directory from /var/lib to /srv - https://phabricator.wikimedia.org/T343071 (10Jgreen) [16:52:05] oh great ^ :)