A Custom Machine Learning Box

Building a Machine Learning box can be a lot of fun and save a lot of money over cloud solutions. For developers that want to dive deep into machine learning and small companies it offers a lot of flexibility, customizability, security and performance for the dollar. Building your own box may not be an option for larger enterprise level companies that require warranties and service, but they usually have their own methods of acquiring hardware and servers for their on premise data centers. For these companies making a between cloud solutions and on premise servers requires weighing a lot of factors such as security, data center capability and room, sysadmin availability and price. However a small company or developer can usually easily set up a box or two and then supplement their needs with a cloud solution if necessary to get the most for their money.

The four most important parts: GPU, CPU, Storage and Memory

For those who haven’t built a computer before they may be surprised to find out there are only really 8 components required to put together a computer: GPU, CPU, Storage, Memory, CPU Cooler, Motherboard, Power Supply and Case. While all the parts do depend on each other (so don’t try to cheap out on the Power Supply or else you may have issues!) the ones that matter the most for Machine Learning are the GPU, CPU, Storage and Memory. The GPU will do most of the work with the machine learning model so you’ll want it to be as fast as possible and have enough on-board memory to hold the model and data. The CPU has to manage all the GPUs and can be a bottleneck if it is too slow. Another common bottleneck is the memory and storage, they need to be big enough and fast enough to feed the datasets to the GPUs.

AMD CPUs vs Intel CPUs

We’ll start with the CPU as it dictates a lot of the rest of the system, such as the motherboard. Currently AMD CPUs offer far more performance per dollar than Intel CPUs, especially when it comes to CPU threads. An AMD Threadripper 1920X is $199.99 and offers 12 Cores/24 Threads, a 3.5 Ghz Clock and 60 PCIe lanes. An Intel 9900X has 10 Cores/20 Threads a 3.5 Ghz Clock and only 44 PCIe lanes and costs $597.87. In case you were wondering why these CPUs, read on!

Make sure your CPU can handle 4 GPUs

It’s hard to be sure how many GPUs you’ll end up with when you’re building your first box. Starting with 1 GPU can save a lot of money, but if your models take too long to train you’ll want to be able to upgrade. For Machine Learning a GPU takes 8 PCIe lanes. Your M.2 SSD storage will take 4 PCIe lanes and you’ll need another 4 PCIe lanes for your Gigabit ethernet. So at a minimum you’ll need 40 PCIe lanes.

You’ll also want enough threads to run 4 experiments per GPU, so you’ll need 16 threads to handle the 4 GPUs possible.

Picking the right GPU

As mentioned before, the model and dataset need to be able to fit in the on-board memory of your GPU. Too little memory and you just won’t be able to run your model at all. Nvidia GPUs have an edge over AMD when it comes to machine learning support and tools, so that narrows the choices down considerably. You’ll also want to make sure the GPU is a blower style, where the fans blow the hot air out of case instead of into it. This is especially important if you’re going to have 4 GPUs as there will be very little room for air flow inside the case with that many GPUs.

I would suggest a 2080 Ti (11 GB of RAM, $1079) for a high end build, a 2070 Super (8 GB, $499) for a mid tier build and a 2060 Super (8 GB, $372) for a low end build. You can mix and match GPUs, though remember, if your model and dataset require more than 8 GB you won’t be able to run them on the lower end GPUs.

NOTE: Remember to get a blower style GPU, otherwise the heat will be vented right on top of your other GPUs, causing them to overheat.

Memory and Storage

One of the big bottlenecks in the machine learning cloud solutions is the Input/Output from storage to the GPU instances. We don’t want that to happen in our custom box so we’re going to make sure the Memory and Storage is fast enough. For Storage I recommend a NVMe SSD. NVMe uses PCIe instead of SATA. A SATA3 SSD typically has a read/write speed up to 550MB/second while a NVMe SSD is up to 3500MB/second. That’s a pretty large difference for the money (only around $20 more if you compare otherwise similar drives).

CPU Cooler, Motherboard, Power Supply and Case

You’ll want a good quality CPU Cooler to keep your CPU cool, an air cooler is cheaper and easy to deal with than a water cooler and will more than do the job. Your Motherboard needs to support 40+ PCIe lanes and work with your CPU. For the Power Supply, make sure it provides enough power for up to 4 GPUs and the CPU working at max. You’ll want 1600 watts to supply 4 x 250W GPUs, the 180W CPU, and 150W for the rest of the system. You’ll notice that 1600 watts is more than that, that’s because Power Supplies don’t deliver 100% of the power they are rated for, usually it’s around 80%-90%. The case should be large enough to have 8 expansion slots. I would suggest a full size tower to give yourself the most room and air flow to work with, but some mid size towers will also work.

PC Part Picker

I suggest using PC Part Picker to pick out your parts. Not only will it help you find the best prices, it will help ensure that all your parts are compatible with each other.

Here are two builds I put together as examples. The only difference is the GPU.
The first one is a budget build with a low end single GPU (2060 SUPER) build for a total of $1650.83. The second one is a high tier single GPU (2080 Ti) build for a total of $2380.83. The GPU obviously makes up a large portion of the budget, especially when you scale it up to 4 GPUs.

PCPartPicker Part List

PCPartPicker Part List

Summary

Once you have your parts picked out it's easy to use the links provided by PC Part Picker to order all of your parts. Once your parts arrive it's a fairly simple matter to put them together. I suggest looking up a few videos on Youtube to give you a good idea on what to do. This video by Linus Tech Tips is fairly informative. Put the parts together and install Linux and enjoy your custom Machine Learning box!