vuxu.org: Hecate

Hecate is a self-built workstation featuring a i5-2400 (3.1Ghz, 4 cores, VT-x, VT-d, AES-NI), 16GB RAM and a 1TB SSD (Samsung EVO 870), hosting a 2x4TB HDD RAID.

The naming

Hecate is a chthonic Greco-Roman goddess associated with magic, witchcraft, necromancy, and crossroads.

Egyptian-inspired Greek esoteric writings connected with Hermes Trismegistus, and in magical papyri of Late Antiquity she is described as having three heads.

Components

ArticlePriceWhere from
Lancool PC-K5863€Arlt
i5-2400166€CSV
Scythe Katana 319€CSV
Scythe 2.5” Twin Mount8€Schwanthaler
Gigabyte GA-Z68XP-UD3109€CSV
Crucial m4 64GB99€Schwanthaler
G.Skill DIMM Kit 8GB33€CSV
G.Skill DIMM Kit 8GB33€CSV
Sapphire Radeon HD 545025€CSV
be quiet! Straight Power (E8-400W/BN153)57€NB-Computer
(Shipping)5€CSV
Seagate ST3000DM001 3TB154€Schwanthaler
Seagate ST3000DM001 3TB154€Schwanthaler
SAMSUNG SSD 840 Series Basic 250 GB (MZ-7TD250BW)139€PIXmania.com
TOTAL1064€

Mistakes

  • First, I used a OCZ Vertex Plus 60GB (for 65€). Absolutely don’t buy this model: the controller reproducably locked up (“failed command: FLUSH CACHE”) after a few days of continous usage, and required a hard power-cycle to even be detected by the BIOS again. I have replaced it with a 64GB Crucial m4, which runs fine.

  • I planned 2x2TB Samsung SpinPoint F4 EcoGreen as a RAID1, but the hard disk prices went to the roof (55€ to 95€, then to 130€) one week before I bought the machine. Five months later, prices are still high, but I now bought 2x3TB Seagate Barracuda 7200 (154€ each), which also are plenty fast.

  • 64GB SSD is too small for / and /home. I bought an additional 250GB SSD later (luckily, prices have dropped a lot).

Rough setup

  • Flash BIOS version 1.19. Get it here. You can use 7z x to unpack it. Flash with Q-Flash from a USB stick.

  • Boot archlinux-2011.08.19.

  • Partition scheme: I partitioned manually with fdisk, which automatically 2k-aligned the partitions.

       sda1 128mb /boot ext2 label ssd-boot
   sda2 rest dm_crypt named cryptsdd, -c aes-xts-plain -y -s 512 --align-payload=8192
     lvm volume group "ssd"
       20g / ssd-root ext4 label root
       35g /home ssd-home ext4 label home

Base configuration

  • grub.conf:

       kernel /vmlinuz-linux root=/dev/mapper/ssd-root cryptdevice=/dev/sda2:ssd printk.time=1 ro loglevel=6

  • Make ondemand default scheduler in /etc/conf.d/cpufreq.

  • Extend logrotate keep times.

  • In /etc/pacman.conf, make tar.gz packages instead of tar.xz. (I will directly install them mostly, the compression is a waste of time.)

  • Copy old passwd/group/shadow/gshadow.

  • adduser, groups: adm,wheel,log,games,audio,video,power,scanner,plugdev

  • /etc/fstab: Add relatime to /, /home.

  • Add coretemp, acpi_cpufreq, it87, kvm-intel, pcrypt to MODULES.

  • Install lots of stuff with packer --noconfirm. Bitch for broken PKGBUILDs.

RAID setup

  • Preparing the disks, /dev/sdb and /dev/sdc:

    smartctl -a /dev/sdb >initial-sdb
smartctl -a /dev/sdc >initial-sdc
smartctl -t short /dev/sdb
smartctl -t short /dev/sdc
badblocks -v -s -w -o /root/badblocks-sdb /dev/sdb
badblocks -v -s -w -o /root/badblocks-sdc /dev/sdc

    (I stopped after one pass because badblocks takes a long time.)

  • Install a GPT-compatible MBR in case we ever need to boot from these disks:

    dd if=/usr/lib/syslinux/gptmbr.bin of=/dev/sdb
dd if=/usr/lib/syslinux/gptmbr.bin of=/dev/sdc

  • Partition disks, keeping a bit free for a potential OS installation:

    # gdisk /dev/sdb
...
Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048       209717247   100.0 GiB   0700  Linux/Windows data
   2       209717248      5860533134   2.6 TiB     FD00  Linux RAID

    Same for /dev/sdc.

  • Designate /dev/sdb as master device for now.

  • Set up the RAID:

    mdadm -v --create /dev/md/hecate-data --homehost=hecate -n 2 -l raid1 /dev/sdb2 missing

  • Shred the RAID data as preparation for disk encryption. (Why shred the RAID and not the disks? Because we need the same data on both disks, and the random data will be copied during RAID resync.)

    shred -n1 -v /dev/md/hecate-data

  • Setup the dm-crypt:

    cryptsetup luksFormat /dev/md/hecate-data --cipher aes-xts-plain64 -y -s 512 --use-random

    Note that we need aes-xts-plain64 since the device is bigger than 2 TB.

    cryptsetup luksOpen /dev/md/hecate-data data

  • Create a file system:

    mkfs.ext4 -m0 -E lazy_itable_init=0 -L data /dev/mapper/data

    (I did not pass the lazy_itable_init, but I absolutely recommend it: else, your RAID will be writing for days until the initialization really finished. Better wait 20min the mkfs takes longer.)

  • You can now copy data on the RAID.

  • Complete the RAID:

    mdadm /dev/md/hecate:hecate-data --add /dev/sdc2

  • Wait for the rebuild, speed it up by allowing more I/O:

    echo 1000000 >/proc/sys/dev/raid/speed_limit_max

  • Check /proc/mdstat for progress. If its done, you can trigger a manual check for consistency later by:

    echo check >/sys/block/md127/md/sync_action

  • If you didn’t use lazy_itable_init, you can speed up itable generation by:

    mount -o remount,init_itable=0 /data

  • Allow the disks to go to sleep when they are not in use for a few minutes:

    hdparm -S 80 /dev/sdb
hdparm -S 80 /dev/sdc

    (80 is 6 minutes, 40 seconds.)

RAID disk failure

ArticlePriceWhere from
Seagate ST3000DM001 3TB111€Schwanthaler

One of the RAID disks went bad (suddenly had 40 reallocated sectors and was slow as hell). To remove it from the RAID:

# mdadm --fail /dev/md127 /dev/sdc2

After replacing the disk (done offline here), repeat steps from above to setup the new disk:

# smartctl -a /dev/sdc >initial-new-sdc
# smartctl -t short /dev/sdc
# dd if=/usr/lib/syslinux/gptmbr.bin of=/dev/sdc
# gdisk /dev/sdc
... see above ...

And add the disk back:

# mdadm /dev/md/hecate:hecate-data --add /dev/sdc2
# echo 1000000 >/proc/sys/dev/raid/speed_limit_max

Hacks

  • it87 doesn’t support the Mainboard officially yet, use

    options it87 force_id=0x8720

  • Enable power-saving for the Radeon HD 5450 (doesn’t save much, but it runs cooler).

    echo auto >/sys/class/drm/card0/device/power_profile

Headphones

For Hi-Fi audio listening, I use a Superlux HD-668 B (29€) with AKG K-240/270/271 Velour Pads (17€). Definitely best sound for that money.

Triple-Monitor Setup

I have acquired two 1600x1200 screens (Lenovo L201p and Samsung 214T, both from 2006) that I run on hecate together with a 1920x1080 screen (Samsung, 2011). The latter is used for watching movies and gets its own X server, so windows will never pop up there. See xorg.conf.triplehead for details.

Second SSD

This was pretty straight forward. Again, I added space for a potential /boot partition.

e2label /dev/mapper/ssd-home old-home
fdisk /dev/sdb
  128mb sda
  rest sdb
cryptsetup luksFormat -c aes-xts-plain -y -s 512 --align-payload=8192 /dev/sdb2
cryptsetup luksOpen /dev/sdb2 cryptssd2
pvcreate /dev/mapper/cryptssd2
vgcreate ssd2 /dev/mapper/crypthome
lvcreate -l 100%FREE -n home ssd2
mkfs.ext4 -m0 -L home -E lazy_itable_init=0 /dev/mapper/ssd2-home
mount LABEL=home /mnt
rsync -avHAXx /home/ /mnt/

I pondered various ways to move the LV between PV, but decided to keep it simple and used rsync.

Void Linux

I reinstalled the machine with Void Linux, reusing the space of the old home partion on the old SSD for the root partition.

RAID replacement

Both Seagate ST3000DM001 showed logged read errors, so I decided to replace the RAID with:

ArticlePriceWhere from
Western Digital Blue WD40EZRZ 4TB82.90€Mindfactory
Western Digital Blue WD40EZRZ 4TB82.90€Mindfactory
TOTAL165.80€

I decided to replace both disks of the RAID at once, by growing it to four disks, then removing the two old disks. This is well explained here: Migrating existing RAID1 volumes to bigger drives.

# mdadm --grow /dev/md127 --bitmap none
mdadm: no bitmap found on /dev/md127

# mdadm --add /dev/md127 /dev/sda2
mdadm: added /dev/sda2
# mdadm --add /dev/md127 /dev/sdb2
mdadm: added /dev/sdb2
# mdadm --grow --raid-devices=4 /dev/md127
raid_disks for /dev/md127 set to 4

(wait 5h)

# mdadm --fail /dev/md127 /dev/sde2
mdadm: set /dev/sde2 faulty in /dev/md127
# mdadm --remove /dev/md127 /dev/sde2
mdadm: hot removed /dev/sde2 from /dev/md127
# mdadm --fail /dev/md127 /dev/sdc2
mdadm: set /dev/sdc2 faulty in /dev/md127
# mdadm --remove /dev/md127 /dev/sdc2
mdadm: hot removed /dev/sdc2 from /dev/md127

-bash-5.0# mdadm --grow --raid-devices=2 /dev/md127
raid_disks for /dev/md127 set to 2

(poweroff, remove old disks)

# mdadm --examine /dev/sdc2
# mdadm --examine /dev/sdd2
# mdadm --grow /dev/md127 --size=max

(wait 2h)

# e2fsck -f /dev/mapper/cryptdata
# resize2fs /dev/mapper/cryptdata 
resize2fs 1.45.6 (20-Mar-2020)
Resizing the filesystem on /dev/mapper/cryptdata to 950539217 (4k) blocks.
The filesystem on /dev/mapper/cryptdata is now 950539217 (4k) blocks long.
    # mount -o init_itable=0 /data

The new disks are far more silent, but also slower (5400 RPM).

Third SSD

After about 10 years, the original Crucial M4 SSD failed. I replaced it with a Samsung EVO 870.

ArticlePriceWhere from
Samsung 870 Evo MZ-77E500B/EU 500G65.87€Böttcher AG

As it only contained the OS, this was essentially reinstalling Void.

For some reason, mdadm assembly in dracut was broken until I added:

add_dracutmodules+=" mdraid "
add_drivers+=" raid1 "

to, say, /etc/dracut.conf.d/mdraid.conf.

Fourth SSD

Unfortunately, above Samsung EVO 870 failed on 2023-08-22 during a system upgrade. After some research, it turned out the disk had the broken firmware SVT01B6Q and hasn’t been updated to SVT02B6Q in time… unfortunately, updating after the fact does not recover it.

I therefore replaced it with a new, bigger one, and also moved /home there:

ArticlePriceWhere from
Samsung SSD 870 EVO 1TB53.99€Mediamarkt

This disk already has firmware SVT03B6Q.

Luckily, most of the disk was readable, so no reinstall was required:

# cryptsetup luksFormat -c aes-xts-plain -y -s 512 --align-payload=8192 /dev/sde2
# cryptsetup luksOpen /dev/sde2 cryptssd1
# pvcreate /dev/mapper/cryptssd1
# vgcreate ssd1 /dev/mapper/cryptssd1
# lvcreate -L 64G -n root ssd1
# lvcreate -L 512G -n home ssd1
# mkfs.ext4 -E lazy_itable_init=0 -L root /dev/mapper/ssd1-root
# mount -o remount,rw /home
# pv < /dev/mapper/ssd2-home > /dev/mapper/ssd1-home
... finally copy root ...

At first I forgot to install Grub on the new disk, but then everything worked fine.

10 Gigabit Ethernet

I installed a 10GB ethernet card:

ArticlePriceWhere from
Intel X520-DA2 incl. 2x SFP+ 10GB-SR-LC68.90€eBay
YuanLey YS25-0402 4x2.5Gbit + 2x SFP+ Switch80.99€Amazon
FLYPROFiber OM3 4x2m LC29.99€Amazon
TOTAL179.88€

The other part of the network is connected to tanit.

I gave the NIC persistent names using /etc/iftab and ifrename:

ix0 mac 90:e2:ba:xx:xx:xx
ix1 mac 90:e2:ba:xx:xx:xy

I removed the Sapphire Radeon HD 5450 video card because I mostly use the machine headless and it’s just using up power.

The switch uses ~3 W when only loaded with the 2 SFP+ cages and full wirespeed usage (measured using a REV Ritter EMT717ACTL).

Benchmarks

  • Power Usage (desktop configuration)
    • Idle: 41 W, Full load: ~85 W

ChangeLog

  • 16jan2012: Update SSD firmware to 0309, using grub4dos like with the BIOS updates for juno.

  • 29mar2012: RAID setup.

  • 30oct2012: Triple head.

  • 21feb2013: Second SSD.

  • 03sep2013: RAID disk failure.

  • 12sep2020: reinstall with Void Linux.

  • 18sep2020: RAID replacement.

  • 11may2021: Third SSD.

  • 24aug2023: Fourth SSD.

  • 25sep2023: 10 Gigabit Ethernet.

Created: Tue, 01 Nov 2011 20:09:13 +0100
Updated: Wed, 22 Nov 2023 22:41:01 +0100
Copyright © 2007–2021 .