Very strange server issues

I have been renting and using an OVH Debian dedicated server for almost two years. I had to reboot it today for the first time in ~1.75yrs, because an app had a memory leak that prevented me from SSHing in.

Upon rebooting, all the files in /home/downloads are gone. However, in df, the hard drive still appears full from those files. It doesn't see the files, but it thinks the HDD is full.

What is going on? How do I recover the files? I tried Googling around for a while and filed a support ticket, but I don't think they will be able to help.

Attached: file.png (553x250, 28K)

Other urls found in this thread:

hastebin.com/wurecezota.hs
twitter.com/SFWRedditImages

Not sure if this is much help user, but check the size of /var/log contents.

What exactly am I looking for?
Here is the output: hastebin.com/wurecezota.hs

Try
du -sh /var/log


and compare that to what
df -h

outputs.

Look at /var and see how full it is. Hope that makes sense?

All the space is taken up on /home, which is expected. It's just that all the files that are supposed to be there aren't.

Attached: file.png (486x297, 29K)

syslog, messages and kern logs probably.

Try running a fsck and hopefully you can recover some files in lost+found under /

Not sure user.

Check how free your inodes are as well.

df -ih

Uh, what does this mean?

Attached: file.png (460x184, 18K)

It means you're good. If it was at 100% I'd be worried.

I repairing it, nothing..

Attached: file.png (549x558, 71K)

Crazy issue. Dump the /home partition to another partition. For example dd if=/dev/sda3 of=/srv/http/home.img . Download that img file and mount it on your PC. If you don't see your files, recover them using the photorec utility.

Also tail the last 100 lines of your syslog. See if anything weird shows.

tail -n 100 /var/log/syslog


I assume you're running systemd. Check

systemctl status


and if it reports as "degraded"

It's 1.7TB of data, so this isn't really practical.

ls -a
Check lost and found

Yes it's degraded, I think this might be the issue.

I also found this in /var/log/syslog:
May 21 08:23:51 solidus systemd[1278]: Failed to get udev device from devnum 8:2: Permission denied
May 21 08:24:01 solidus CRON[514]: (root) CMD (/usr/local/rtm/bin/rtm 36 > /dev/null 2> /dev/null)

Attached: file.png (609x207, 31K)

Unmount the /home and run photorec on/dev/sda3

>Not checking lost+found
Come on OP

I don't know OP.

Divide & Conquer.

Troubleshoot the unit and the log messages. Perhaps run photorec as suggested and also do what has suggested.

Where is it going to recover it to if that partition is unmounted?

There is nothing in the lost+found, okay?

How do I fix the "console-setup.service" failing? I tried the following to no avail:
dpkg-reconfigure keyboard-configuration
dpkg-reconfigure console-setup

You need to read why it's failing.

systemctl myfailingunit status


Should have some output suggesting why.

I've had a long day OP. I'm out. Best of luck.

I did, it was getting this error:
May 21 08:34:15 solidus console-setup.sh[1418]: /usr/bin/ckbcomp: Can not find file "symbols/en" in any known directory


I googled and found it can be fixed by reinstalling packages:
console-setup console-setup-linux keyboard-configuration


So I did that and I'm rebooting now after doing
sudo touch /forcefsck

So that fixed the systemctl errors, so it is no longer "degraded". The filesystem issues remain, however.

Any other suggestions apart from photorec?

Please help me :(

what was in the files? pedo shit?
I'll help if you answer

What file system is the server using for the problem partition in question? Output from Mount please.

XFS had this issue where dirty unmounts would fuck with how it allocated space for files. That might be one problem.

Additionally, look into using du (depth 1) (summary) and narrow down where the space went, folder by folder.

I'm not near a Linux machine right now, so I can't remember the flags to do it. Just man du and figure it out.

Attached: OP_Delivers.gif (321x223, 2.01M)