30 July 2019

Filesystem benchmarks

Ok, let's get this straight.
When I choose to use JFS, it's because some years ago I see with my own eyes how JFS is reliable in different scenarios and it still is. Yes, EXT4 is reliable too, but the performance isn't on par with JFS, but both offers a reliable and secure solution. XFS in other hand isn't that secure and reliable (it is to point), but offers a quite good performance. When kernel 5.0 came out, they talked a lot about how BTRFS is good now and, different from most people that rely on "everyone uses, so I'll use too" I want to test myself because I'm not the guy that relies on this  "everyone guy" opinion.
The host used for this test is my main desktop using gentoo linux with the last available kernel (5.2.4-gento0) and last available tools to date (i5-3470 on a gigabyte motherboard, 16gb of ram, sata3 1Tb hdd). The disk is entirely formated with the filesystem being tested. All filesystems are mounted using noatime and using bfq scheduler (bfq offers better performance for rotational disks than mq-deadline)). The IOZone tests was executed with a reboot before creating the new filesystem to test to exclude any possible bias.

Test 1: Creating a 1Gb file with dd=/dev/urandom of=test bs=1024 count=1M (in seconds):

Test 2: Cold reboot during the creation of the 1Gb file from one filesystem to another (5 times)

Auto Fixed?5/55/53/54/54/5
Mounted rw without fixing01241
Wasn't able to fixing00121
Continue working with problems without reporting00121

Test 3: Copy a 1Gb file from one filesystem to another

Test 4: Cold reboot during mariadb heavy worload (5 times)

Auto Fixed?5/55/54/54/54/5
Corrupted databases?NYYYN
Mounted rw with problemsNNYYN
Continue working with problems without reportingNNYYN

Test 5: Boot from EFI to lightdm, sata3 hdd, mounting / and /home (5 times)

After a cold reboot0:421:221:402:112:10

Test 6: Shutdown from lightdm to total shutdown (openrc) 

Test 7: IOZone

My overall opinion:
  • JFS and EXT4 are good, performance wise and secure enough for daily usage.
  • XFS was polished through the years and the most awkward problems was taken off (data corruption with power loss and others), it's a good option but have some issues with performance depending of size and mass of data
  • ZFS is good but you need ram to have a good overall performance, I would suggest start with 16Gb for a desktop (this isn't a problem for a server, of course). Also, keep in mind that the mainline kernel doesn't support ZFS, it takes sometime for ZFS reach the last line (right now, ZOL supports up to 5.1.x). Also, despite the comparison of performance between ZFS and BTRFS seems to be similar, ZFS is far more stable and trustful.
  • By cpu/mem footprint in a copy, JFS is the lighter and ZFS the heavier. Of course, will also depend of how much data are you copying from where to where and kind of storage, but in overall, let's put this way: Copy lots of tiny files JFS > EXT4 > BTRFS > ZFS > XFS. Copy lots of big files: JFS/EXT4 > XFS > ZFS > BTRFS. Of course, the mileage will vary depending on the use case (it can be irrelevant with a full non-virtualized server with lots of ram and a good storage system).
  • Only use BTRFS if you like to restore backups often. To date, I can't trust BTRFS, not after seeing a filesystem having corruption with simple tests. -> http://lkml.iu.edu/hypermail/linux/kernel/1907.1/05873.html

Notes by 20190810:
  • XFS resolved the issues caused by power loss and metadata corruption. 
  • For XFS, the performance can benefit more with mq-deadline than bfs, afaik. Otherwise, use this recommendation in udev and don't forget to use noatime if you don't need it.
  • BTRFS still not ready, it can offer some benefits and performance, but the stability is still far from being acceptable, the auto-healing doesn't work as expected (like zfs, for instance) and scrub doesn't fix as expected in some scenarios. They're getting straight, a lot of this stuff seems to being fixed from 5.2 and beyond.
  • The fact of a filesystem receiving more kernel or userland updates doesn't mean it's more stable, or better, or whatever.
  • You SHOULD do your tests instead of talking bullshit like "everyone uses" or "it's an uncommon use". If the kernel supports it, it's supported. Period.

29 July 2019

How to choose an OS

Here's how to choose an OS that fill your needs.

1) Try on a VM first
Try on a VM. You can test everything on this list inside a virtual machine. Virtualbox is free, but it's your choice.

2) Device drivers
The first thing you need to analyze and pay attention is how the system is supported by hardware and how you can fix it if possible. If you don't have enough knowledge to manually install a required driver, you should rely on the in-house driver installer (like ubuntu, linux mint, manjaro, etc). Even so, pay attention if the drivers was installed correctly and there's no issues, specially during an update.

2) Community Support
If you want to rely on community support (I don't suggest since some communities causes more trouble than help) you have to pay attention closely and don't use it as an unique source. Search on the internet and compare the results, usually the distribution have his own documentation about it and most of time you'll see that some irc/matrix/whatever support USUALLY doesn't rely on their own wiki and in his own standards, specially elitist communities. Let me get and example:

1) You've tested yourself a couple of filesystems and decided to choose, let's say, F2FS or JFS.
2) You had a problem with systemd and ask for help, and paste your dmesg
3) If they say that the problem is F2F2 or JFS without dmesg having something explicit about this, you're dealing with stupid people.

3) Custom configurations
This one is really easy. If the distribution have drivers loaded at boot for an specific filesystem (well supported by the kernel developers) and you can't use it as root (as stated by kernel developers), you're dealing with naive packagers and problems will occur in other places often. For example, some linux distributions doesn't support LVM properly, like.. they will boot correctly if you have /boot and / outside of LVM (?????). This is the result of naive packagers, stay away from that.

4) Elitist community
Over-complicating things to state how they're right based on own defaults without proof; stating age of a processor even if the system have a nice performance compared to a more recent processor (saying that a Intel I7 Haswell is slow because it's old); using "because everyone use it X" as answer; "no one uses this kind of hardware anymore" talking about something like 3 years old hardware. And the list goes on, this kind of crap should be redirected to a black hole.

5) OS implementation
Lack of multilib without any reason; Answering questions of poor implementation saying "because it's professional" instead of the real reason; Trying to hide problems with "because it's better in this way" it's a matter of naive developers.

6) Systemd

7) Documentation
Usually, any wiki should work with any distribution at some point, it'll depend on the implementation. So is good to choose something that is KISS as possible, so you'll have more documentation available if a problem arises. You don't need to use the specific wiki of the OS of your choice if the OS follows the standards of the application maded by the developer, so you can even rely in the application docs to fix something. Unless you're using a trollercoaster OS.

8) Learning curve
You don't need to understand what's happening in the background in a matter of days, but it's probably best for you if you learn how it works at your own pace. The less you rely on other people, the less is the chance of someone giving you a bad advise, and trust me, this will occurs more often than you expect.

9) FHS and POSIX
It seems no one cares about it, but FHS and POSIX standards are important. No, isn't a matter of "get used", it's a matter of having standards. I accept changes when it benefits, changes that came form stupid nature should be discarded ASAP (for example, linking /bin to /usr/bin). I suggest avoid OS maded by brainless people that take decisions without a sane reason.