Con­struc­tion and com­pon­ents

The in­form­a­tion on this device avail­able across the CWWK’s web­site and Ali­ex­press stores is an­em­ic, so it is pretty hard to know what to ex­pect. Open­ing the server up is not a stream­lined ex­per­i­ence. 6 Phil­lips screws hold the top panel in place. With rack ears in­stalled (ini­tially in the ac­cessor­ies box), there are 4 ex­tra on each side as well. I was greeted with this view:

Start­ing at the top left, the power sup­ply in­cluded with the server is two units of what looks to be a Mean­well EPS-65-12-C. These sup­plies feed into what looks to be a fail-over or per­haps a bal­an­cing board for re­dund­ant sup­ply. From there a 12V and GND wires are con­nec­ted to the mother­board. Look­ing at the spe­cific­a­tions for the power sup­ply sug­gests it is quite high qual­ity, but at the same time I do not have ac­cess to tool­ing to meas­ure power qual­ity prop­er­ties such as its out­put ripple. Re­gard­less, I’m glad this power sup­ply is from a reput­able brand and not a no-­name grab bag of fire haz­ard clear­ance caps.

The mother­board and the ex­ten­sion board are both cus­tom designs by CWWK. CWWK does sell re­place­ment ex­ten­sion boards, at least right now, but it seems un­likely that re­pla­cing them with a stand­ard PCIe device would be pos­sible. For most in­tents and pur­poses this device is as­-de­livered.

The SFP+ ad­don board is based on the old In­tel 82599ES con­trol­ler that’s known to have some is­sues with ASPM, but that turned out to be easy to work-around for me in prac­tice, so I was­n’t par­tic­u­larly wor­ried.

All of the fans in the device are 3-pin. The CPU fan is plugged into the SYS header and the ex­haust fans in the back are plugged into the CPU head­er. Though this does­n’t mat­ter as the board can­not con­trol DC voltage for the fans any­way – they run at 100% all the time. The fan on the SFP board is very aud­ible even to my hard-of-­hear­ing ear. The mother­board has an IT8613 Su­per I/O chip for con­trolling the fan speeds, but this only works for PWM fans. Re­pla­cing these 3-­pins with PWM fans is go­ing to be both very ex­pens­ive and clunky due to their form factor. An al­tern­at­ive would be to keep the fans and add some­thing like the Noc­tua NA-FH1, PH-P­WHUB_02 or a Phobya trans­former which can con­trol DC fans based on the PWM sig­nal. It is­n’t clear to me if any of these are able to stop the fans en­tirely, though. An­other al­tern­at­ive is to 3D print some air­flow guides and re­place the small fans with a 120mm PWM fan in­stalled side­ways.

Be­sides the rack ears men­tioned earli­er, the ac­cess­ory box also con­tains two of the chosen (US or EU) power cables, rub­ber feet and cables to use SATA. The SATA power cable is wired for +12V and +5V only. And yet you would be wrong if you thought that you would be able to use your left-over 2.5 inch SATA disk here. The board is wired up to use the mSATA con­nector only. To the best of my know­ledge the CF card will not work either. Sup­posedly there is some res­istor some­where on the board that you should be able to desolder and re­place on dif­fer­ent pads to switch over to the SATA con­nect­or. For me – it was easier to grab a used mSATA SSD off the mar­ket­place for 15€. Had I been aware of this, I would have ordered the server with stor­age (and memory) in­cluded. Re­gard­less, no space for re­dund­ant stor­age here. Though, if you want to go su­per cheap, you could use the in­ternal USB (2.0?) port and boot off some stor­age plugged into it.

Board also ap­pears to have a SIM slot and a Mini PCIe con­nector for a WWAN. However, much like with SATA, I find it un­likely it is ac­tu­ally go­ing to work: out of the 9 PCIe lanes avail­able out of the CPU, 8 of them are al­loc­ated between the ad­don board (4x) and the cop­per nics (1x each.) While there is 1 lane still avail­able that could be routed to this slot, lspci only re­veals 5 PCI bridges: all of them already oc­cu­pied.

The GPIO pins avail­able on the board are prom­in­ently mar­keted in spec­sheets found on the Ali­Ex­press list­ings, but these are routed to the IT8613 Su­per I/O chip rather than the CPU it­self. While you can use­fully use these GPIO pins as an out­put, they are worth­less as an in­put – there is no way to re­ceive an in­ter­rupt on the pin state change. This foiled my plan to add a PPS GPS time device to this serv­er. I’ll ex­per­i­ment with GPS re­gard­less, but the PPS will have to come through the con­veni­ently avail­able USB port in­side the case. An­other thing to keep in mind is that IT8613 is not sup­por­ted out­-of-the-box on Linux. You need to force a mod­ule load with a dif­fer­ent ID.

The board ap­pears to have head­ers not only for VGA that’s ex­posed to out­side the chassis, but also HDMI and Dis­play­Po­rt. Fig­ur­ing out the pin-out and up­grad­ing the port on the chassis might be an in­ter­est­ing fu­ture pro­ject (espe­cially with the JetKVM com­ing my way.)

An over­arch­ing theme here is that doc­u­ment­a­tion for the most part is woe­fully ab­sent. All you get is what’s prin­ted on the silk­screen of the mother­board: some de­scrip­tions of the jumper pin func­tions (e.g. there’s a jumper for whether the server should turn on after power loss) & brief names of the head­ers is all you get.

Out of the box ex­per­i­ence

Turn­ing the server on for the first time I was greeted by AMI Ap­tio sug­gest­ing there is no boot­able me­dia avail­able (see notes on SATA above.) Op­tions ex­posed here are a fairly ran­dom grab-bag of the most com­mon op­tions and op­tions that aren’t par­tic­u­larly in­ter­est­ing in this type of a device. Some ex­amples of not par­tic­u­larly use­ful op­tions: you can modify the fan curve (des­pite the device ship­ping with 3-pin fans), some GPU over­clock­ing op­tions (in a net­work ap­pli­ance?), power loss be­ha­viour (des­pite it be­ing con­trolled by a jumper on the board in­stead.) At the same time many of the op­tions that would oth­er­wise be very use­ful were ab­sent: no PCIe con­fig­ur­a­tion what­so­ever (and no ASPM con­trol to go with it,) the CPU power sav­ing op­tions are not ex­posed and even ba­sic set­tings like turn­ing off In­tel HD Au­dio aren’t pos­sible either! Upon con­tact­ing with CWWK folks I was in­formed that there is cur­rently (2025-01-01) only this ver­sion of the firm­ware avail­able.

Boot­ing into a bare-­bones NixOS sys­tem off a USB stick I was meas­ur­ing power con­sump­tion at idle to be around 22W. The CPU cores were prop­erly en­ter­ing the deep­est C10 C-state, but the pack­age could­n’t go be­low C2, un­for­tu­nately. Out of these 22W, the fans are con­sum­ing 3W. Re­mov­ing the SFP+ card and the fans would drop the idle power con­sump­tion down to 10W for the CPU and I226-V NICs.

Spe­cify­ing the pcie_aspm=force pcie_aspm.policy=powersave ker­nel ar­gu­ments, run­ning the powertop --auto-tune & the enable-aspm script for everything pos­sible would drop the power con­sump­tion slightly, but the pack­age C-state re­mained locked at C2. An­other thing I tried was to up­grade the ACPI tables dur­ing initrd in or­der to change the “ASPM not sup­por­ted” flag from 1 to 0, but in the end it had no ef­fect on power con­sump­tion what­so­ever.

Any at­tempts to go around the Ap­tio con­fig­ur­a­tion to change firm­ware set­tings would fail as well – the set­tings stor­age is write pro­tec­ted in UEFI and on­wards. This left me with the only re­main­ing way for­ward: get firm­ware fixed.

Modi­fy­ing the firm­ware

Turns out firm­ware modi­fic­a­tion for AMI sys­tems is par­tic­u­larly straight­for­ward nowadays, with num­ber of tools to aid the ef­fort. While it took me some time try­ing to ob­tain the flash con­tents in the first place (the sup­port would not send me any “BIOS”, un­for­tu­nately,) in the end it turned out to be an ex­tremely straight­for­ward pro­cess:

1. Ob­tain a firm­ware up­date for some other N100-­based device made by CWWK. I used the most re­cent avail­able for their NAS miniPC;
2. From this firm­ware up­date grab the fpt.efi tool and put it on a EFI par­ti­tion along with the EFI shell (boot.loader.systemd-boot.edk2-uefi-shell.enable = true; in my NixOS con­fig was suf­fi­cient, also avail­able pre­com­piled from the edk pro­ject);
3. Boot into the EFI shell and dump the cur­rent firm­ware us­ing fpt.efi -d flash.bin. I then ran this com­mand twice more to dif­fer­ent files to make some du­plic­ates (to avoid ac­ci­dental si­lent cor­rup­tion.)
4. Verify the dumps (sha256hash same between all cop­ies?) and save them in a safe place;
5. Fol­low the in­struc­tions in the README of UE­FI-Ed­itor;
   • On NixOS the UE­FITool can be had with nix-shell -p uefitoolPackages.old-engine for the 0.28 ver­sion and nix-shell -p uefitool for the new ver­sion;
   • ifrextractor is at nix-shell -p ifrextractor-rs;
6. In­side the UE­FI-Ed­itor ex­pose ASPM and other in­ter­est­ing set­tings by set­ting Ac­cess Level to 05. This has to be done re­curs­ively. So for in­stance to en­able HD Au­dio con­fig­ur­a­tion it is ne­ces­sary to set the Ac­cess Level to 05 both for the “HD Au­dio Con­fig­ur­a­tion” page as well as the “HD Au­dio” op­tion (other­wise HD Au­dio Con­fig­ur­a­tion page does­n’t ap­pear any­way due to be­ing “empty”);
   • Touch­ing Suppress If broke the Ap­tio Setup for me eas­ily, re­quir­ing to go back to the be­gin­ning mean­while Ac­cess Level modi­fic­a­tions worked without fuss;
   • I gen­er­ally found that things worked great so long as the UE­FI-Ed­itor modi­fic­a­tions would modify just the Setup­Data sec­tion and not the oth­ers;
7. Once the ad­jus­ted firm­ware is con­struc­ted by fol­low­ing the UE­FI-Ed­itor in­struc­tions, flash it back from the UEFI shell with fpt.efi -f newflash.bin.

By modi­fy­ing the firm­ware this way, over 20 re­vi­sions or so later, I had en­abled a ma­jor­ity of the op­tions I was af­ter: PCI(e) set­tings (in­clud­ing ASP­M), CPU power ef­fi­ciency op­tions, Si0x set­tings and such. Then I promptly en­abled the power sav­ing fea­tures, dis­abled the su­per­flu­ous hard­ware and was happy to find that both ASPM and CPU set­tings were now set up cor­rectly. Time to get Linux take ad­vant­age of all this.

En­abling everything power­-sav­ing on Linux

As far as ASPM is con­cerned, most of the hard­ware had ASPM en­abled by the firm­ware. The only ex­cep­tion be­ing the 82599ES SFP+ board. For 82599ES I have his­tor­ic­ally found that al­though the chip does not ad­vert­ise sup­port­ing the L1 ASPM state, it nev­er­the­less will work well with it for­cibly en­abled through its PCI re­gisters. To do this I set up a couple of udev rules:

# Enable L1 and L0s for 82599ES devices
ACTION=="add", SUBSYSTEM=="pci", ENV{PCI_ID}=="8086:10FB", RUN+="setpci -s $kernel B0.b=3:3"
# Enable L1 and L0s for all pcieports
ACTION=="add", DRIVER=="pcieport", ENV{PCI_ID}=="8086:54B8", RUN+="setpci -s $kernel 50.b=3:3"

NOTE: PCI_IDs or setpci com­mands might need chan­ging for you, you can find out the PCI_ID with udevadm info /sys/class/net/enp1s0f1/device/ and udevadm info /sys/bus/pci/devices/* for entries with DRIVER=pcieport. I de­rived the setpci com­mands from the enable-aspm script.

In ad­di­tion, the fol­low­ing rules im­ple­ment most of the powertop --auto-tune:

# The following rule is IMPORTANT!
SUBSYSTEM=="pci", ATTR{power/control}="auto"
ACTION=="add", SUBSYSTEM=="usb", TEST=="power/control", ATTR{power/control}="auto"
ACTION=="add", SUBSYSTEM=="scsi_host", KERNEL=="host*", ATTR{link_power_management_policy}="med_power_with_dipm"

Fi­nally add the fol­low­ing op­tions to the ker­nel com­mand line (pcie_aspm=force is no longer ne­ces­sar­y):

pcie_aspm.policy=powersave nmi_watchdog=0 i915.disable_display=1 xe.disable_display=1

I found that set­ting pcie_aspm=powersupersave would in­crease laten­cies to over 1ms and af­fect mdev sig­ni­fic­antly as well, while at the same time sav­ing very little in terms of power con­sump­tion. Mean­while pcie_aspm=powersave has al­most no eff­fect.

I226-V hangs

Ap­ply­ing ag­gress­ive PCIe link power sav­ing has caused some is­sues with the I226-V NICs for me. In par­tic­u­lar in one out of three boots the NICs would re­fuse to link up with the peer any­more. Some of the is­sues with I225 and I226 series of NICs are some­what doc­u­mented on the in­ter­net, with the usual sug­ges­tion be­ing to dis­able any power sav­ing set­tings. For­tu­nately in my case I quickly found out that a simple rmmod igc; modprobe igc would bring the NIC back to life and keep it rock stable af­ter­wards. How­ever this will kill all 4 ports for a brief while, mak­ing this way big­ger of a can­non for the small tar­get stand­ing a feet away than it needs to be. After read­ing through the igc driver source code I found an­other way to re­set the net­devs in­di­vidu­ally – toggle the En­ergy Ef­fi­cient Eth­er­net op­tion for the net­dev.

Set­ting up the fol­low­ing script and sys­temd unit solved this prob­lem for good (and prob­ably achieved fur­ther power sav­ings to boot):

#!/usr/bin/env bash
set -e
for path in /sys/module/igc/drivers/pci:igc/*/net/; do
    for dev in "$(ls $path)"; do
        if cat "/sys/class/net/$dev/operstate" | grep up > /dev/null; then
            ethtool --set-eee "$dev" eee on
        else
            ethtool --set-eee "$dev" eee off
            ethtool --set-eee "$dev" eee on
        fi
    done
done

[Unit]
Description=Check that i226 managed to link up, reset netdev otherwise

[Service]
ExecStart=/nix/store/fqq3s50g8n2j4mqppjxzk5x63pn55ky9-unit-script-igc-check-start/bin/igc-check-start
Restart=always
RestartMode=direct
RestartSec=30s
Type=exec

Eval­u­ation and Fu­ture Im­prove­ment

With these changes I have man­aged to get the CPU pack­age drop all the way down to the C8 state. Not quite C10 that is achiev­able without the SFP card plugged in, but the pack­age power con­sump­tion dif­fer­ence between the two states seems to be min­im­al. Even at C8 the CPU re­ports just 0.15~0.20 watts for the pack­age at idle which is a big im­prove­ment over 2.5 Pkg­Watt at C2!

$ turbostat -s 'POLL%,C1E%,C6%,C8%,C10%,CPU%c1,CPU%c6,CPU%c7,Pkg%pc2,Pkg%pc3,Pkg%pc6,Pkg%pc8,Pkg%pc10,PkgWatt,CorWatt'
<snip>
POLL%  C1E%  C6%   C8%   C10%   CPU%c1  CPU%c6  CPU%c7  Pkg%pc2  Pkg%pc3  Pkg%pc6  Pkg%pc8  PkgWatt  CorWatt
0.00   0.24  0.14  0.04  99.25  0.24    0.21    99.92   2.70     0.01     0.03     94.96    0.16     0.01
0.00   0.84  0.02  0.14  98.76  0.83    0.11    99.48   2.70     0.01     0.03     94.96    0.16     0.01
0.00   0.06  0.19  0.00  99.46  0.06    0.23    100.13
0.00   0.04  0.34  0.04  99.11  0.04    0.43    99.72
0.00   0.02  0.02  0.00  99.69  0.02    0.09    100.33

Meas­ur­ing at the wall the device con­sumes just 11W, and goes down to as low as 6W with the SFP card re­moved. Not quite 5W you can get with RB5009, but still pretty darn good! And much bet­ter than e.g. the num­bers re­por­ted for QO­TOM model in the STH re­view (20W); though I would­n’t be sur­prised if Patrick from STH does­n’t spend 2~3 days hack­ing at BIOS and co­er­cing Linux to do un­usual things.

In terms of NIC per­form­ance, the SFP board is some­what lim­ited by the PCIe 2.0 4x link. I was able to meas­ure it max­ing out at 24G­bps total through­put hand­ling bi­d­irec­tional traffic across both ports. I find this meas­ure­ment some­what weird as it is more than what the PCIe link is sup­posed to be able to handle – maybe the NIC is­n’t send­ing the full packet up to the CPU? Either way this is good enough for me for the time be­ing. The cop­per NICs work as you’d ex­pect. The ad­ded latency av­er­ages at around 0.5ms. The CPU cores are reach­ing about 50% util­iz­a­tion hand­ling some very ba­sic fire­wall rules at max­imum through­put.

Once I have ad­ded my own SSD (15€) and memory (50€), the total price of the device comes out to roughly 450€. Sub­stan­tially more com­pared to RB5009 or the N100 mini PCs. How­ever for this price I got a re­dund­ant power sup­ply and a 1U en­clos­ure, and I was well cog­niz­ant of there be­ing a rack tax when eval­u­at­ing my op­tions. I wish I did­n’t need to fiddle for a few days to get the power con­sump­tion down to ac­cept­able levels, es­pe­cially when it in­volves modi­fic­a­tions to the low-­level firm­ware. This is reas­on­ably fix­able by CWWK through re­leas­ing a firm­ware up­date, though, and I hope they go through get­ting it out sooner rather than later.

Adding a fan con­trol­ler for the fans (~30€ shipped) would be an­other sig­ni­fic­ant ad­di­tion to this device I wish I did­n’t have to make. Adding one such has a good chance of mak­ing the device pass­ive most of the time and fur­ther re­duc­tions to power con­sump­tion are en­ti­cing. That said I am hold­ing off for a pur­chase of a 3D print­er. Among many other pro­jects I have in mind, I feel it should be pos­sible to print some air­flow guides and use a PWM 120mm fan in­stalled side­ways to gen­er­ate enough air­flow to cool the com­pon­ents well – at a much lower price. This is an­other in­stance where CWWK could prob­ably have achieved a per­-unit BOM re­duc­tion at a slight ad­di­tional en­gin­eer­ing time ex­pense.

With that said, I’m quite sat­is­fied with my choice and this product. Maybe one of the other devices con­sidered would have given me more ports (that I don’t need & at a power over­head) or bet­ter hard­ware. But to me this CWWK model is ex­actly the right amount of net­work­ing for a hard-to-­beat price and hard to com­plain about power ef­fi­ciency.

Other use­ful notes

This hard-to-find thread on L1T was a par­tic­u­larly use­ful dump of in­form­a­tion about ASPM, LTR and sim­ilar top­ics as well as has given me a hint to look into /sys/kernel/debug/pmc_core/.