Moodle MySQL database error

[ndcollegeits]

Help! any Moodlers, MySQL experts out there?

 

Running Moodle 2.6 on Linux debian as a virtual machine on VMware. Yesterday morning got a database error when browsing to Moodle and took the following steps:

 

Rebooted the server and fsck kicked in, took an absolute age to complete the check but then booted ok except failed to run the MySQL service.

 

Logged in and ran df -h command which showed no free space on the disk. 1.4T disk but the previous nights full backup had only been around 500GB?

 

Used FTP client to access and browse the folder structure, found and deleted lots of old moodle sql backups freeing up about 100Gb. Restarted mySQL and all was good for about four hours, successive df -h commands showed that the free space was diminishing rapidly

 

Looked for advice online which suggested it could be a process running on a deleted file, ran lsof | grep "deleted" but didn't find anything. Did a disk clean up to make a bit more space, apt-get clean.

 

Now going to increase the size of the vm disk but this may be futile as I think the disk space will be swallowed up by whatever the underlying cause is.

 

 

Has anyone come across this or similar?

 

Thanks

[pcstru]

Is something spamming a log file? Are you generating binary log files from MySQL?

 

This may be useful info to find which process and file is the culprit.

[ndcollegeits]

Thanks, I have no idea at this stage.

 

I will have a look at the post and see what I can find.

[CAWJames]

When I get massively expanding files I run the following command which searches for files over a certain size:

 

find / -xdev -type f -size +1000M

 

So that looks for all file in the whole file system that are over 1GB, so you could add /var/log instead of just / to search just the log directory, and you can change the 1000M to 10000M to find 10GB files. If you are filling up that fast, you might find a massive file straight away.

Although this is only good if it is one file filling, won't be much good if hundreds and thousands of small files are spawning, so you could use du to give you the top ten biggest directories, if its one directory, you should get a triangle of nested directories pointing to the culprits storage:

 

du -a /var | sort -n -r | head -n 10

 

If you need to drill down more, or show more, just change the 10 to give a bigger amount of results

 

hope that helps, I use both of these, and must admit I found both through google ages ago!

 

James

[ndcollegeits]

Thanks CAWJames that is exactly what I did. Found a freshclam.log file that had grown to 1.1Tb !! I have deleted it and all is well again. I will know what to look for should this happen again.

 

Andy

[CAWJames]

Excellent! It's amazing how big one file can grow! You should also look into why it got so big, and maybe sort out some log rotation with deletion of old logs after a set amount of time.

 

Those 2 commands have saved my bacon a few times over the years. Glad that I could help.

 

James