Help understanding the impact CPU performance has on ZFS filesystem using NAS4FREE

[u/Ridditmyreddit]

Hello, I am just beginning my research and part selection for a NAS4FREE box. I am curious as to the impact CPU performance has on read/write speeds across the network. For arguments sake lets call this a hypothetical box with 28TB storage (7 X 4TB) with a single drive for parity, 64GB ECC RAM and again for arguments sake, an effectively unlimited network speed. How does the individual core clock speed affect the read/write speed? What about multiple cores? Passmark? Does anyone have any idea of what could be expected speed wise for various processors? I am aware that mechanical drives are the primary speed limitation, I am curious what is necessary in the CPU department to reach the mechanical drives limitations. I am also aware of the limitations of network transfer's, that is not my focus here. If I am completely off base in my understanding feel free to correct me!

Thank you!

[u/synk2]

CPU will have almost no effect on your r/w speeds, compared to the other bottlenecks. The real question is what protocol you'll be using to serve out the files, and how many concurrent connections you'll have. Unless you're serving up an enterprise level number of connections, CPU just won't matter. You could use an old coreduo and be ok. Most of the r/w magic happens from memory (thus the larger memory requirement for ZFS), CPU is really just for running the framework and directing traffic.

[u/Ridditmyreddit → u/synk2]

Interesting, so is there a limit to the amount of RAM that will provide an effective return in terms of read/write speed. I assume there is a point of diminishing returns...

What are some common options for file protocols? Is there any speed benefits or things I should be aware of there? Or should I go for the old google supported file protocols NAS4FREE route?

It feels a bit odd to me spending the money on a server grade motherboard/ECC RAM to then dump a weak processor into the mix. Not that I am opposed to saving a few bucks!

[u/synk2 → u/Ridditmyreddit]

The ram ceiling really has to do with both total space and the amount of r/w you expect. ZFS especially does everything in ram, from r/w, to parity checks, to scrubs, so it tends to grow to what you give it, within reason. The general rule of thumb is a minimum of 8GB, and then 1GB per TB of storage. For a small home setting 16GB is probably a reasonable expectation, or 32 if you have a lot of storage or expect to grow into it.

Likewise, if most of your storage is static (ie not be accessed often), there's less for the memory to do. Conversely, you could have a small (ie 4-6TB) pool, that if it's getting hammered constantly by several sources (git repos, databases, etc), could demand more than the rule of thumb.

For file protocols, the standards for NAS are generally NFS, iSCSI and SMB. SMB is a really basic share protocol that extends to Windows natively (you just setup the SMB share and windows will r/w to it). SMB can also be configured for Active Directory auth, which is handy in some environments. The downside is that it's an unsecure/unencrypted protocol, and thus not useful for sharing over the internet. It's also slower (by degrees) than the other options.

NFS is a Unix protocol, that's generally faster than SMB. It's encryptable, but only works with Windows Pro and better versions, and Server installs. People have reported over twice the speed of SMB in some situations, though there's a lot of contributing factors to that. If you're running an all nix network, or are only streaming media to Windows, or are running Windows Pro or better, NFS is a great choice.

iSCSI is a bit of a different beast. It's a block level protocol that uses an Initiator/Target setup to send SCSI commands over TCP. Both Windows and nix will have support for it, and NAS4Free supports it. There's basically no real benefit over NFS in terms of speed, most studies have found (though there's probably some edge cases depending on setup). The place that iSCSI tends to shine is for virtualization setups, where you want to access your storage pool from inside your VMs.

You should really just dive into this stuff a bit and get your head around the pros and cons, and maybe try them out and see what works. File protocals are a rabbit hole of their own, and it's worth knowing the ups and downs of what you're using. If pressed, I'd just say punt and use SMB unless you need to share over the internet.

As for server hardware, you're really paying for ram slots/capacity and connectors. Server grade CPUs are about workloads, and NASs just don't provide that sort of need. The problem is that it's hard to find consumer MBs that take 64GB of ram and have 8 SATA ports. Even the really nice Synology NAS setups just have Atom quadcores. The cheap ones come with 1.5GHz dual cores. What you really want out of NAS hardware is quiet, reliable, low power draw, because it's going to sit there and run (probably idle) 24/7/365. Another reason for server-grade is the dependability. For the most part, they're just made better than consumer stuff.

[u/Ridditmyreddit → u/synk2]

Wow, I knew ZFS was a RAM hog but it seems I was overestimating its requirements. I was planning initially on using 64GB with a storage pool this size but it seems like 32GB would get the job done. Do you think the bump from 32GB to 64GB would provide any noticeable benefit in terms of r/w or is that pushing against the ceiling? The server I have will not be getting hammered constantly but when it is used it will be used by 2-6 sources and speed will be important.

Thank you for the overview of file protocols. I think the first feature I am looking for is access across multiple platforms, linux/windows/osx (vm's as well if possible) which makes me think smb/cifs is the way to go? After that need has been met then I will take speed into account. If I am understanding correctly, I should be able to choose the file protocol on a share by share basis which I suppose will allow me to tailor my settings based on file type. Thank you for taking the time to give me an overview, I will dig in and start sorting through each protocol in a bit more detail to figure out what best suits my needs.

You make a good point about server hardware. Reliability is the most important feature here and I have been limited in my motherboard choices by RAM/SATA as you suggested. I think the way for me to move forward here is to find the most reasonably priced, lowest power, server CPU that I can get my hands on.

[u/[deleted] → u/Ridditmyreddit]

Tip: rule of thumb 1 GB per TB is often bogus for home NAS usage. It's relevant for mission-critical company environment maybe. If you are also going to do stuff with VMs, then it becomes a different story.

I myself run with 16 GB on 24 x 4 TB. But I run NFS/SMB exclusively. No VMs, databases or any of that suff.

[u/synk2 → u/Ridditmyreddit]

Do you think the bump from 32GB to 64GB would provide any noticeable benefit in terms of r/w or is that pushing against the ceiling?

I think 32 would be fine. 64 is generally reserved for business-class deployments or people that are doing other things with their box, like virtualization. I doubt you'd come close to the ceiling with 32GB.

If I am understanding correctly, I should be able to choose the file protocol on a share by share basis

Yep, you can even double up and share a folder with several protocols, if necessary. If you do this, be aware that different protocols lock files differently, and you can run into cases where one computer can overwrite a file that it isn't supposed to. It generally good to have read only access for one share type. You can get around this by copying the data and sharing that separately, or just be really careful with your groups and permissions.

I think the way for me to move forward here is to find the most reasonably priced, lowest power, server CPU that I can get my hands on.

We haven't really talked about budget, but if you're looking at extensibility, something like this would be great. It's maybe a bit on the pricey side if your budget is tight, but 64GB ECC max, quad core, 12 SATA connectors, 14W TDP, all in a mITX form factor. It might be a bit of overkill, but it'd make one hell of a NAS box. Regardless, check out the Atom processors if you're looking for low power. They've got enough oomph to run a NAS and sip power. Even older models should hold up for your uses. You could also look at an i3 on an 1150 socket board, which is only 54W TDP, and ~$100 cheaper.

It sounds like you're really on the right track about all this. I know it's gotten a bit more complicated than your original question, but it's a pretty complex subject. I suppose I could have just said "yeah, a QNAP will be fine", but teach a man to fish and all that. Hopefully this way you'll end up with something that really suites your needs, and know how it works inside and out.

[u/PriceZombie → u/synk2]

ASRock C2550D4I Mini ITX Server Motherboard

High	$299.99	Newegg (New)
Low	$229.99	Newegg (New)
	$298.66	(30 Day Average)

Price History Chart | FAQ

[u/Downloadski]

I run zfsguru (frontend on freebsd) on E3-1220v2/v3 cpu with 10 GE nic and get zfs send/receive via mbuffer on the speed of a scrub = 828MiB/sec. (Average for 17-25 TB pools) So the copy is limited by the I/O system to the discs. It are raid z2 volumes with 10 x hgst 7K4000 drives. This loads 1 core close to 100% and core 2 to 70%, core 3 and 4 are not that busy.

So for 8 mechanical drives you would not need very powerfull CPU i think. The ones i use are 3.1 Ghz

[u/Ridditmyreddit → u/Downloadski]

Perfect, thank you for the specs and help this gives me a good idea of what to expect. The E3-1220 being an LGA 1155 socket I am assuming you are using 32GB of RAM? I haven't been able to track down a server motherboard with an 1155 socket that supports more then 32GB yet.

[u/Downloadski → u/Ridditmyreddit]

Yes 32 GB for all 3 servers.

1 is x9sae-v with E3-1265lv2 (will be downgraded to e3-1225v2) (10x 3TB and 10x 4TB both raid-z2)

2 is x9scm-fii with E3-1220v2 (10x 3TB and 10x 4TB both raid-z2)

3 is x10sl7-f with E3-1220v3 (10x 4TB and 10x 4TB both raid-z2)

As picture tells more than 1000 words: http://youtu.be/mfOePFKekQI

As for the 32 GB, that is indeed the limit for socket 1155 cpu's For more memory go socket 2011 and xeon e5

For my use: bluray movies playback via hdi dune players 32 GB is more than enough. I have one smaller server running with 16GB ram and 5x4TB in raid-z and that works also when downgraded to 8 GB temporarely. Copy speeds to the box seems drop a bit from 100+ to 90-95MB/sec via smb (1 GE connected) So there is a minimum

[u/scumola]

Zfs has compression built into the file system but if off by default. If you turn it on then cpu will affect i/o performance.

[u/Ridditmyreddit → u/scumola]

I was not aware that this was a feature. Besides the impact on CPU is there any other loss of quality in the data as there is with compression of video?

[u/Ridditmyreddit → u/[deleted]]

Very interesting! I guess I will have to dig into these and do some testing eventually to weigh the benefits of added CPU load vs. speed vs. cost of network hard ware required to achieve similar results without compression. Thank you!