A little Nutanix CE 2.1 Science Project S&M

Well folks, I am pleased to announce that my science project with Nutanix CE has concluded with an in-depth exploration of CE 2.0 and 2.1 on various AMD based platforms including single node clusti, 3 node clusters, 4 node clusters and 5 node clusters using AMD Ryzen 5900X, 9900X and Threadripper 5000 series CPU.

If you're considering setting up a home lab or replacing your free VMware 3-node setups, I must say that I can only recommend Nutanix CE 2.0 with AOS 6.5.5.7 and use just Prism Element if your RAM per node is 64, 128 or 192GB for managing your VM's and replication if you have multiple clusters.

Prism Central isn't suitable for mainstream use yet, and specifically for home lab use, its memory and core requirements are too high to be a practical option for home labbers.

In addition, CE 2.1 has a lot of issues identifying SATA SSD and NVMe SSD from the less popular manufacturers and even Samsung and their work arounds do not fix this issue for single node clusters.

The Install script kept identifying SATA SSD as HDD and whenever I typed in the fixes, Stargate took a big shit and I got rewarded with total instability.

In fact, the support for CE just sucks in general and the forums are not very useful at all if you want to get into fixing the many issues you will encounter.

You also have to make sure your networking is all either Intel or Mellanox 10GbE or better and I advise you use the same Samsung 1.92TB SATA SSD Nutanix use in their NX-1175S-G7/8/9 nodes and also use SMC based AMD EPYC armed servers with SMC RAM if you want to get serious about a CE 3 node cluster as a bare minimum starting point.

If you want to see LCM work properly though build a 4 node cluster, 3 nodes are a problem with any sort of workloads running on them when it comes to lifecycle management and upgrades.

If you do want to be tormented and use Prism Central you will need 384GB per Node to pull it off and assign 128GB RAM to the Prism Central VM.

This means desktop PC platforms are not on the menu for you and you will have to use SuperMicro or older Dell and HPE fare.

Trust me when I tell you that Prism Central is not well baked software at all.

I also observed high wear rates on RAM as well as NVMe SSD from Samsung, WD, SANDISK and a whole horde of others in single node mode so be aware that you will be replacing your main NVMe drive every 12 months if you use single node clusters for your VM's.

I myself have transitioned to Verge.io because it is quicker and significantly more stable for my personal requirements.

I say this after having worked at Nutanix for 5 years myself.

Another advantage of verge.io is that you only pay for the processor, not the cores.

I built me a 5 node SuperMicro cluster with AMD EPYC 24 Core CPU and 256GB RAM per node using Mellanox 25GbE on two now discontinued SN2010 switches and this is my setup.

I am experimenting with some innovative database technology on it and have equipped the cluster with NVIDIA GPU per node. It now serves as my AI engine cluster, where I explore various LLM builds and experiment with Grok3, ChatGPT 4.0, Perplexity, and numerous other AI tools using my collection of A100 GPUs.

I am developing code for three platforms and will begin a fourth with Linux. My goal is to create the world's first commercially stable Chess Engine and GUI for Linux variants, starting with Mint Linux, priced at just $29.99 each.

Currently, these are merely hobbies because I am employed at a Japanese storage OEM and thoroughly enjoying every moment of it FOR THE FIRST TIME IN A LONG TIME!

This isn't my first experience as a storage Architect, and my current outfit remain the best-kept secret in the storage platform industry.

I am actively assisting in displacing Pure, Nutanix, and HPE at key accounts by concentrating on VMware 8, Redhat Openshift, and Linux-focused clients effectively.

Happy days are here again!

If you're searching for the earlier Nutanix CE posts, a keyboard mishap on my Mac while using WIX caused the entire post to be lost, and I didn't have the motivation to rewrite it.

I will likely conclude my WIX phase as the expenses have become unmanageable, and I plan to return to hosting my own site, as I did for 15 years with NetMax Linux. I have an abundance of servers and robust network security with Infoblox to support it.

If you're seeking a Nutanix expert with impressive skills on CE, consider reaching out to a guy named Brian Holmes on social media. He is one of the few at Nutanix who truly knows his stuff. However, he is currently swamped by eccentric management individuals who are earnestly competing for the headless chicken VIP of the millennium award.

I've left that world behind, but best of luck with your HCI endeavors no matter which vendor you go for!

Nutanix single node clusters are suitable if you are going to use Prism Element, but for performance needs, consider using verge.io instead on a two node setup.

In terms of buying Nutanix should your CE adventures tempt you into paying cash, you need to stay far away from all the Nutanix NX nodes that come with Intel CPU unless it is the NX-8170 all NVMe platform.

For all the security and performance features you can only go with AHV as the Hypervisor, those features are not in ESXi and never will be.

I myself advise only the NX-8155A-G9 with a suitable 16 Core or more CPU pair with 1024GB RAM in 64GB DIMMS (DDR5), Mellanox NICs and make bloody sure no Toshiba SSD are in it!

Most Nutanix Commercial reps are not very good at the art of selling or negotiating and they keep all their deals in what they call the green band with a 25% off MSRP discount to keep their greedy DM's happy.

You can also tell your Nutanix AE you do not want Prism Central or their Pro license for NCI or NCM as it's a rip off and Prism Central ain't worth shit! (You do need PC for license activation - but tell them anyways).

Demand another 15-19% off whatever they present you with which will be either 20% or 25% off MSRP or tell them you will buy verge.io instead.

They can go to 46% off by the way but every dog and his flea at Nutanix has to sign off on it at that level.

Unless you are using Nutanix for VDI, also demand ZERO files or Object. Some reps love putting in a token 3 TB for files and Object on NCI and NCM vanilla configs because their commission goes up a lot if the deal includes Pro licenses for NCI and NCM with a token amount of files and Object in the mix that gets the reps the kicker accelrators on their commission checks that they want.

Nutanix reps by and large don't give a rats ass about what's good for the customer just what is good for their commission check.

Also know they have a much cheaper NCI license for VDI use cases called NCI-VDI and this has files built into it that cannot be removed but it is very attractive from a pricing point of view.

Request standard licenses for Non-VDI use clusters (NCI+NCM), but note that there is a package called NCP Starter that includes NCI Pro and NCM Pro, which you do not require that the more shady reps will put in the quotes, so be very careful with your quote and ask the rep what versions of NCI and NCM licenses you have exactly.

What they don't disclose about NCM is that you receive the standard version for free, and that's usually all you require.

Just pay for the NCI Pro if you feel you need it (most do not).

There is also no license required for asynchronous replication so resist their up-sell antics for the full replication package as less than 1% of Nutanix customers actually use it and bitch about it a whole bunch when they realize they wuz hoodwinked four months into running the clusters.

Nutanix do not downgrade any licenses after the fact by the way so be very careful before you purchase that you are getting what you need and not what the reps commission check needs.

You should also know commercial and enterprise groups in Nutanix have different price points so if your buds at Wells Fargo tell you what they paid for it, you cannot draw comparisons to the Commercial class stuff.

If you are wanting to run SQL or Oracle on your Nutanix cluster all I can say is don't as verge.io is substantially faster and they only charge per CPU not per core.

This means you can buy a nice fat high GHz, high core count CPU and only worry about the license cost per core from the database vendors.

The only real good use case for Nutanix IMHO is for VDI with their VDI licensing at NCI for VDI at Pro level or for general purpose VM's on NCI and NCM at Standard or starter licenses.

Do not run Databases on Nutanix nodes unless it is the NX-8170 or NX-8155A platform with a strong processor and remember you need a bunch of RAM and Cores for Prism Central so if you buy an 8 Core CPU thinking this is all for your Database you would be wrong - there is overhead to account for!

Nutanix CVM's need 10 Cores and 128 GB of RAM and the Prism Central VM needs 4 cores and 128GB RAM.

I had a few cases with SAP Hana where I had to give the CVM 14 cores and 128GB RAM to work in the sweet spot.

You still need cores and RAM for your database and other VM's!

The Nutanix sizing tool says 4 cores and 32GB RAM is OK for CVM's and this is true if you are not going to run any VM's on the cluster.

As this is unlikely, you will need to adjust the CVM accordingly for real world operations.

Nutanix fired their sizing Guru Wally back in 2023 and there is nobody that knows what Wally knew about Nutanix sizing.

I worked with him for 7 months on a banking customers environment and was pretty well schooled by ol Wally in what's what with CVM's and Prism Central resource needs.

The real bare minimum is 6 cores and 48GB for general purpose VM's, enter any Database workloads and you will need 10-14 cores and 96-128GB RAM per CVM.

Be aware that tuning a Nutanix CVM for database action will adversely affect the performance of general purpose VM workloads in a very bad way which is why it is common sense and best practice to have a cluster dedicated to just database ops and another cluster for the rest of it.

Also do not mix VDI workloads with general purpose or database workloads within the same cluster.

They will insist it can handle it but the truth is it cannot. Sure, it will run, but it will not run well at all.

Also be advised if you want to build a big Nutanix files NAS cluster that they have a 500,000 object limit per NAS instance which is about as useful as tits on a bull.

Files is only useful for VDI workloads.

When it comes to backups, we had a long fight internally to set RF to One for clusters that are backup targets. This means it can tolerate ZERO failures, but the costs are acceptable.

The minute you build a Nutanix cluster for file and Object at RF2 you are immediately on the far dark side of fantasy when it comes to cost for backup purposes and they also make you pay for files twice at the remote and the target site which also contributes to the stoned on acid fantasy proposal for backup targets from a pricing point of view.

Cohesity and others just charge for one side.

The Nutanix ERA tool is also a mixed bag and has no replication to another cluster HA features built into it but it does offer a lot of time savings for database Admins and features zero space cloning capability which is pretty huge in my opinion.

I would buy it just for that feature alone for devops guys use.

Nutanix workshops for PS are also expensive and a huge fat waste of time and money by the way.

Also be aware most Nutanix SE's are clueless with the art of sizing a nutanix cluster so if you see 3 node clusters in the solution know you are dealing with a clown, and probably one that hails from Egypt somewhere to boot.

Nutanix nodes need to be at N+2 redundancy levels for storage, RAM and CPU when it comes time to use LCM to upgrade AOS, Firmware and AHV.

Any customer IT architect who deploys a Nutanix 3 node cluster in production needs to be replaced with someone who knows what they are doing.

While 3 node clusters are the technical starting point, you need the other node for the LCM upgrades as the process takes a node out of the cluster to upgrade in a serial cyclical fashion.

The other node should also ensure you are near N+2 levels in normal operation mode at all times.

In the 5 years I was at Nutanix I was the only SE with ZERO CSAT issues and this is because I know how to architect solid and reliable Nutanix HCI solutions.

Make sure when you examine your Nutanix cluster solution proposal documents that you can see everything at N+2 and the box for what happens when one node fails is ticked and everything there is also at N+2 levels of redundancy IN THE ONE NODE FAILED display in the proposal document.

If you see a 3-node cluster solution thank them for wasting your valuable time and show them the door as you cannot take such a proposal or the people that made it seriously.

Also make sure each CPU is at the 16 Core minimum.

8 core and 10 Core CPU on Nutanix clusters cater to people with serious sado-masochistic psychological problems and are brain damaged from sniffing too much glue when they were doing their psychology or poly-sci class while at college.

Most of these types also sport beards and look like unkempt sasquatches who escaped from the Yosemite park at some point or whose fathers did not beat them up enough when they were growing up at various California air bases.

The truth is out there!!

Happy HCI'ing!