vuxu.org: Hecate

Hecate is a self-built workstation featuring a i5-2400 (3.1Ghz, 4 cores, VT-x, VT-d, AES-NI), 16GB RAM and a 64GB SSD (Crucial m4) + 250GB SSD (Samsung 840), hosting a 2x3TB RAID.

The naming

Hecate is a chthonic Greco-Roman goddess associated with magic, witchcraft, necromancy, and crossroads.

Egyptian-inspired Greek esoteric writings connected with Hermes Trismegistus, and in magical papyri of Late Antiquity she is described as having three heads.

Components

ArticlePriceWhere from
Lancool PC-K5863€Arlt
i5-2400166€CSV
Scythe Katana 319€CSV
Scythe 2.5” Twin Mount8€Schwanthaler
Gigabyte GA-Z68XP-UD3109€CSV
Crucial m4 64GB99€Schwanthaler
G.Skill DIMM Kit 8GB33€CSV
G.Skill DIMM Kit 8GB33€CSV
Sapphire Radeon HD 545025€CSV
be quiet! Straight Power (E8-400W/BN153)57€NB-Computer
(Shipping)5€CSV
Seagate ST3000DM001 3TB154€Schwanthaler
Seagate ST3000DM001 3TB154€Schwanthaler
SAMSUNG SSD 840 Series Basic 250 GB (MZ-7TD250BW)139€PIXmania.com
TOTAL1064€

Mistakes

  • First, I used a OCZ Vertex Plus 60GB (for 65€). Absolutely don’t buy this model: the controller reproducably locked up (“failed command: FLUSH CACHE”) after a few days of continous usage, and required a hard power-cycle to even be detected by the BIOS again. I have replaced it with a 64GB Crucial m4, which runs fine.

  • I planned 2x2TB Samsung SpinPoint F4 EcoGreen as a RAID1, but the hard disk prices went to the roof (55€ to 95€, then to 130€) one week before I bought the machine. Five months later, prices are still high, but I now bought 2x3TB Seagate Barracuda 7200 (154€ each), which also are plenty fast.

  • 64GB SSD is too small for / and /home. I bought an additional 250GB SSD later (luckily, prices have dropped a lot).

Rough setup

  • Flash BIOS version 1.19. Get it here. You can use 7z x to unpack it. Flash with Q-Flash from a USB stick.

  • Boot archlinux-2011.08.19.

  • Partition scheme: I partitioned manually with fdisk, which automatically 2k-aligned the partitions.

       sda1 128mb /boot ext2 label ssd-boot
       sda2 rest dm_crypt named cryptsdd, -c aes-xts-plain -y -s 512 --align-payload=8192
         lvm volume group "ssd"
           20g / ssd-root ext4 label root
           35g /home ssd-home ext4 label home
    

Base configuration

  • grub.conf:

       kernel /vmlinuz-linux root=/dev/mapper/ssd-root cryptdevice=/dev/sda2:ssd printk.time=1 ro loglevel=6
    
  • Make ondemand default scheduler in /etc/conf.d/cpufreq.

  • Extend logrotate keep times.

  • In /etc/pacman.conf, make tar.gz packages instead of tar.xz. (I will directly install them mostly, the compression is a waste of time.)

  • Copy old passwd/group/shadow/gshadow.

  • adduser, groups: adm,wheel,log,games,audio,video,power,scanner,plugdev

  • /etc/fstab: Add relatime to /, /home.

  • Add coretemp, acpi_cpufreq, it87, kvm-intel, pcrypt to MODULES.

  • Install lots of stuff with packer --noconfirm. Bitch for broken PKGBUILDs.

RAID setup

  • Preparing the disks, /dev/sdb and /dev/sdc:

    smartctl -a /dev/sdb >initial-sdb
    smartctl -a /dev/sdc >initial-sdc
    smartctl -t short /dev/sdb
    smartctl -t short /dev/sdc
    badblocks -v -s -w -o /root/badblocks-sdb /dev/sdb
    badblocks -v -s -w -o /root/badblocks-sdc /dev/sdc
    

    (I stopped after one pass because badblocks takes a long time.)

  • Install a GPT-compatible MBR in case we ever need to boot from these disks:

    dd if=/usr/lib/syslinux/gptmbr.bin of=/dev/sdb
    dd if=/usr/lib/syslinux/gptmbr.bin of=/dev/sdc
    
  • Partition disks, keeping a bit free for a potential OS installation:

    # gdisk /dev/sdb
    ...
    Number  Start (sector)    End (sector)  Size       Code  Name
       1            2048       209717247   100.0 GiB   0700  Linux/Windows data
       2       209717248      5860533134   2.6 TiB     FD00  Linux RAID
    

    Same for /dev/sdc.

  • Designate /dev/sdb as master device for now.

  • Set up the RAID:

    mdadm -v --create /dev/md/hecate-data --homehost=hecate -n 2 -l raid1 /dev/sdb2 missing
    
  • Shred the RAID data as preparation for disk encryption. (Why shred the RAID and not the disks? Because we need the same data on both disks, and the random data will be copied during RAID resync.)

    shred -n1 -v /dev/md/hecate-data
    
  • Setup the dm-crypt:

    cryptsetup luksFormat /dev/md/hecate-data --cipher aes-xts-plain64 -y -s 512 --use-random
    

    Note that we need aes-xts-plain64 since the device is bigger than 2 TB.

    cryptsetup luksOpen /dev/md/hecate-data data
    
  • Create a file system:

    mkfs.ext4 -m0 -E lazy_itable_init=0 -L data /dev/mapper/data
    

    (I did not pass the lazy_itable_init, but I absolutely recommend it: else, your RAID will be writing for days until the initialization really finished. Better wait 20min the mkfs takes longer.)

  • You can now copy data on the RAID.

  • Complete the RAID:

    mdadm /dev/md/hecate:hecate-data --add /dev/sdc2
    
  • Wait for the rebuild, speed it up by allowing more I/O:

    echo 1000000 >/proc/sys/dev/raid/speed_limit_max
    
  • Check /proc/mdstat for progress. If its done, you can trigger a manual check for consistency later by:

    echo check >/sys/block/md127/md/sync_action
    
  • If you didn’t use lazy_itable_init, you can speed up itable generation by:

    mount -o remount,init_itable=0 /data
    
  • Allow the disks to go to sleep when they are not in use for a few minutes:

    hdparm -S 80 /dev/sdb
    hdparm -S 80 /dev/sdc
    

    (80 is 6 minutes, 40 seconds.)

RAID disk failure

ArticlePriceWhere from
Seagate ST3000DM001 3TB111€Schwanthaler

One of the RAID disks went bad (suddenly had 40 reallocated sectors and was slow as hell). To remove it from the RAID:

# mdadm --fail /dev/md127 /dev/sdc2

After replacing the disk (done offline here), repeat steps from above to setup the new disk:

# smartctl -a /dev/sdc >initial-new-sdc
# smartctl -t short /dev/sdc
# dd if=/usr/lib/syslinux/gptmbr.bin of=/dev/sdc
# gdisk /dev/sdc
... see above ...

And add the disk back:

# mdadm /dev/md/hecate:hecate-data --add /dev/sdc2
# echo 1000000 >/proc/sys/dev/raid/speed_limit_max

Hacks

  • it87 doesn’t support the Mainboard officially yet, use

    options it87 force_id=0x8720
    
  • Enable power-saving for the Radeon HD 5450 (doesn’t save much, but it runs cooler).

    echo auto >/sys/class/drm/card0/device/power_profile
    

Headphones

For Hi-Fi audio listening, I use a Superlux HD-668 B (29€) with AKG K-240/270/271 Velour Pads (17€). Definitely best sound for that money.

Triple-Monitor Setup

I have acquired two 1600x1200 screens (Lenovo L201p and Samsung 214T, both from 2006) that I run on hecate together with a 1920x1080 screen (Samsung, 2011). The latter is used for watching movies and gets its own X server, so windows will never pop up there. See xorg.conf.triplehead for details.

Second SSD

This was pretty straight forward. Again, I added space for a potential /boot partition.

e2label /dev/mapper/ssd-home old-home
fdisk /dev/sdb
  128mb sda
  rest sdb
cryptsetup luksFormat -c aes-xts-plain -y -s 512 --align-payload=8192 /dev/sdb2
cryptsetup luksOpen /dev/sdb2 cryptssd2
pvcreate /dev/mapper/cryptssd2
vgcreate ssd2 /dev/mapper/crypthome
lvcreate -l 100%FREE -n home ssd2
mkfs.ext4 -m0 -L home -E lazy_itable_init=0 /dev/mapper/ssd2-home
mount LABEL=home /mnt
rsync -avHAXx /home/ /mnt/

I pondered various ways to move the LV between PV, but decided to keep it simple and used rsync.

Benchmarks

  • Power Usage (desktop configuration)
    • Idle: 41 W, Full load: ~85 W

ChangeLog

  • 16jan2012: Update SSD firmware to 0309, using grub4dos like with the BIOS updates for juno.

  • 29mar2012: RAID setup.

  • 30oct2012: Triple head.

  • 21feb2013: Second SSD.

  • 03sep2013: RAID disk failure.