2

Apparently the disappearance of files and directories occurred in all ext4 filesystems, including files and directories usually "read-only" in /var and /usr, like binaries installed for gcc, and several others utilities, like apt-get, aptitude, which we tried to use to do some tests and fixes, but a lot of commands and utilities didn't work anymore, all them were crashed, some of theirs dependencies or files disappeared.

These filesystems were ext4, and this oddly occurred last Friday the 13h (November 13, 2015), at about 4:10pm (GMT-3 DST, Brazil). Some log files of /var also disappeared. But we could get the syslog what started to show a lot of errors just when the users also noted the system anomaly. But these errors were related just to absent of files none explained the event that could cause the disappearance of data or any of kind of hard-disk problems.

As a workaround, we booted with a rescue image, just to try get a copy of the /home data, although corrupted we could recover data form some files. And sure, some data were recovered, but a lot of files and directories disappeared and others were there but truncated (zero size).

There are some known issue that could explain this ? How to prevent this from happening again in the others severs ?

PS: The "storage" was mounted in a hard RAID 1 (between 2 hard-disks)

EDIT - ENVIRONMENT: The machine is down, so I cannot get a lot of info that were there, but it was running Debian: 

uname -a Linux XXXX 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1+deb8u6 (2015-11-09) x86_64 GNU/Linux
  • Under some NATs
  • Internet access to an external (cloud) SVN server (controlled by firewall).
  • Internet access to some packages repositories for manual update (apt-get update/upgrade). No one update or upgrade done for months
  • Access via ssh
Improve this question
4
  • Are you suspecting hacking? give more details, operating system version, connectivity to the Internet, daemons running, service expose to the outside, function of the server Commented Nov 17, 2015 at 13:01
  • No. No hacking suspicion. We have more than 20 servers, some running weak and legacy systems like SCO Unix for decades. This Debian was considerably more strong in settings and access restriction. Commented Nov 17, 2015 at 13:02
  • 1
    As we both know, there can be many causes. What I do recommend for future cases, is what I do here, I send forward to a central syslog server all logs, and also all the commands that are typed at the command line, wether people are sysadmins or regular users. For these cases where you are left without logs, it will leave you an interesting trail to follow. Commented Nov 17, 2015 at 13:09
  • Do you have ECC RAM? If so, did it report any error? If not, have you done a RAM test? Commented Nov 17, 2015 at 23:26

1 Answer 1

Reset to default
1

I actually suspect you were bitten by a much talked ext4 corruption bug in kernel 3 and 4. Have a look at this thread,

http://bugzilla.kernel.org/show_bug.cgi?id=89621 .

There are out there more threads about the bug, I just found this one more interesting.

If the other servers are at the same level of updates and versions, I would suggest a round of security/package updates.

Improve this answer
2
  • I read the thread and others related, it seems to be a different problem. I didn't get recurrent corruptions, but just 1 big issue where after 3 months after system startup, instantly several files and directories disappeared. But sure, the combination Debian + kernel 13.16.0 + ext4, should be re-evaluated. Commented Nov 17, 2015 at 14:41
  • Let´s say in the last line you just agreed there is a high probably for it to be related to it. People are complaining left and right about it, and the way it happened... I would not wish that to my worst enemy. We mostly update the kernel when major advisories come up. We always update, so it a reboot is needed in a urgency, will come up fresh, however we just reboot in purpose when the advisory 1) is serious 2) affects our configuration. Commented Nov 18, 2015 at 7:07

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.