I was originally supposed to have a rundown in March but life got the better of me and I couldn’t finish a decent write-up, alas here we are.
Without further ado here is the new, cleaner and up to date diagram for MuffNet:
As you can see when comparing my previous rundown there is a lot more happening at a glance. I’ll try and go through this step by step for those of you interested in the internal workings of my lab.
For a while now I’ve been looking for a new, larger rack to upgrade from my 12U. It’s been pretty good to me and I love the thing but my rate of expansion has been pretty fast and I have needed more space for a while. Recently, I found a very reasonably priced 25U on eBay with all its sides and doors, so I grabbed it and moved into the new rack. I now have a lot more room for expansion and future projects (currently have a SAN project in the works for example) and just generally feel a lot more free in my labbing, I was constantly having to plan around the 12U making sure everything can fit. Not anymore.
Do I feel like I’ll grow out of the 24U one day? Yes, but, that will be the point I look at the lab and start consolidating because having over 25Us of lab equipment is pretty ridiculous, impractical and expensive. So on the flip side, no, I may fill the thing up but I don’t think I’m ever going to go bigger because that would be insane, especially since I live in London and most of your bathrooms are probably bigger than my living space.
Here are some pictures of the move into the new rack, it took a good few hours and I was ill at the time which probably didn’t help.
As you may notice in the following pictures (click below to view) I have kept some space below the bottom most server, this is a 2U space I have reserved for a future UPS I am hoping to install.
If any of you are paying attention to MuffNet you’ll have noticed that there is no longer an R710 in my environment, and you’d be correct. I switched my R710 for a DL380 G7 as my preference for servers swings to the HP side of things. I generally find their stuff a bit more robust, quieter, a slight bit more efficient and more compatible with other hardware, but that’s just me. MUFFHOST01 has 92GB of memory expandable to however much I like as I have the DIMMS to completely fill this thing to the maximum of 288GB. For the sake of power have only populated it to 92GB. As you can see in the diagram, this host has 2x x5690s which are currently being extremely underutilised unfortunately, but I plan to change this in the months to come. The storage is, currently, local with 8x 300GB SAS drives in a RAID10 which provides nice and fast storage but lacking space, although I plan to remedy this soon.
MUFFHOST02 which was previously my test lab for a completely contained lab that sits next to my desk still exists but is now MUFFHOST04. This still sits in the same case and location as last time and I will look into putting it into a 2U case now I have the space for non-prod equipment in the rack. If I have a bit if spare £££ I may look into changing this into a custom dual 2011 system as the price for those parts has fallen drastically but since this is only a test environment I may not bother. It is only powered on when I want to play around with virtualizing large scale environment so the power usages aren’t great and the actual hardware is more than enough.
MUFFHOST02 is now a DL360 G7 which has been introduced into prod. My plan is to have these two hosts in a cluster but there is currently no shared storage, but more on that later. Currently, MUFFHOST02 is hosting my pfSense, again, more on that later, and secondary services, such as my second domain controller. This host currently has 1x E5640 which I have found to be the right balance between power and performance, along with 64GB of memory. Like MUFFSTORE01 it currently has 8x 300GB SAS drives in a RAID10.
MUFFHOST03 now exists and is offsite. The pfSense machine at my mother’s house died due to a PSU failure and since she needed a CCTV system for her business I thought this was the best way. It’s a small Optiplex 7010 which works great for pfSense and the CCTV server which runs in a VM and has a WD Purple drive given to it for DVR needs.
The diagram currently has a placeholder, this is where my Docker host used to live which was hosted on Debian on one of my R210 IIs but after messing around with Docker, which was very cool, I decided it wasn’t the best for me as I have dedicated virtualisation hosts which are already underutilised as it is; so that spot is currently empty possibly for another host when I get my shared storage up and running and can take advantage of VMware’s more advanced clustering features.
I currently have an R210 and R210 II, these have been put into projects before and then wiped. They are currently not doing anything but I do have some plans for them in the future….
Switching & Routing.
Switching is still handled by my trusty HP 1810-24G. I love this switch as it’s managed, silent and pulls next to nothing in terms of power. I have a few Gigabit 48p L3 Cisco switches about but keep coming back to this one due to power and noise. I am looking to upgrade this in the near future however but there is no urgent need to right now.
A lot of people complain that I mount my switches at the front which is a valid point but for me, in this environment, it makes life a whole lot easier when needing to reconfigure things and as a visual aid to see what’s going on in my network. It may not be the prettiest thing in terms of patching, but it works and I like it.
I have also now properly mounted by Brocade 5100 switch of which I have 4. This will be used in my upcoming SAN project as a SAN switch with the ability to handle 8Gb connections, this should be sweeeeet for back-end storage of my virtualization cluster. All 48 ports are fully licensed and as you can see I have more than enough SFPs so I am really looking forward to get stuck into my SAN project and using this thing. I have messed around with it a little so far but will properly get stuck in when the time calls for it.
Routing has moved from my dedicated pfSense box to a VM as one day my entire network started going absolutely haywire. After a bit of panicking and investigation I found out that my pfSense machine was flapping its network ports due to most likely a dead network controller. This was causing the NICs to drop almost 90% of the packets which was causing serious issues. I moved pfSense to the Dell R210 temporarily before moving it to a VM. This will probably change in the future.
My VPS has now been added to the IPSEC site to site I have allowing me to connect to the services hosted on that from my local VLANs. Any of my ‘sites’ can be accessed via each other with my house being the core. I’m not too worried about communication between my VPS and mothers house if my site goes down as there really isn’t much important traffic that needs to traverse that link on a day to day basis.
Another notable change to my switching infrastructure is that I now use a Meraki switch that I use for my room. I really do like this thing but the lack of console access really does bug me sometimes.
Storage has changed a little from last time too. MUFFSTORE01, currently my Plex server remains the same, with a little more storage than last time as my media has grown quite a bit, even from before. My media share currently sits at about 70TB. This is the 4U case I have at the bottom of the rack.
MUFFSTORE02, the file server directly above this, I built recently and blogged about it here. This is mainly my backup server which backs up the entirety of MuffNet nightly, including the offsite locations using Veeam. Using Veeam I am also backing up my workstations and with the use of compression and deduplication I have plenty of space to spare. I love this backup infrastructure and it really has been great knowing I have full backups of all my stuff. This machine also hosts my storage shares such as my software repo, FTP backups of websites (like this one), Music etc, it’s been great. These shares also backup to the main backup repository, just in case.
The HP Microserver sitting above these servers remains my FreeNAS box and holds my personal photos and videos. This was shown in this project. MUFFSTORE03 syncs nightly to MUFFSTORE04 nightly which is great in case of anything either failing in this machine, theft or destruction from whatever means. I’ve actually really enjoyed using these as FreeNAS machines and they work flawlessly, I’ve yet to encounter any issue with it and I am able to fully saturate gigabit when working on photos in Lightroom and Photoshop.
AD! So I have been messing with active directory and domains for a long time now, both at work and in my labs but it wasn’t implemented in my actual prod network, but I finally pulled the trigger.
One day I finally decided to centralise a lot of my management and put everything in a domain to make life easier for myself, so I went ahead and created a domain controller on MUFFHOST01 and MuffNet.xyz was born! All my machines are now domain connected wherever they can be and almost all my authentication is LDAP and SSO where possible which has made my life so much easier. I can how login to vCenter without entering a single thing using SSO, sweet!
Share permissions are also a lot more straightforward now as I no longer need to login to shares when logged into my domain account and GPOs have streamlined a lot of the things I would otherwise have to enter manually on a load of machines. My GPO to set the correct firewall settings and enable RDP by default is a godsend, for example. I also have some GPOs to fix some of the Windows 10 annoyances I have.
Windows DNS is now my main DNS for the machines in MuffNet moving from pfSense and DHCP the same. Management has gone from easy to just plain lazy now, which is great.
I have a backup domain controller on MUFFHOST02 and also one in the cloud in MUFFCLOUD01 which is a VPS I own hosted in America. The purpose of this is to allow everything domain related to continue to function if my hosts go down for whatever reason/I need to work on them.
So there are a lot more VMs running now than there were before, and generally a lot more going on.
Active Directory is now syncing between 3 instances, as previously explained. I have 2 BIG-IP load balancers virtualized that handle things like DNS requests, internal web requests and AD load balancing between the 3 instances, with it weighted strongly towards the 2 local instances.
I have also setup an SQL cluster which replicate between hosts which has been a cool learning experience and pretty practical as I run a few things that have a database backend hosted here.
There are others which are on the diagram above which are pretty self-explanatory. There are actually a lot more than I have documented but the documented ones are the ones I use 24/7, there are others which I use to mess around with and test various apps.
What’s In The Works?
So there are a few things I’m working on and planning on changing in MuffNet, and they would be the following:
- SAN! I am in the process of planning out a SAN build for my VM infrastructure so I can have some shared storage and cluster my ESXi hosts. This is being blogged about so keep an eye out for that! I am planning on using 8Gb fibre between the hosts so some exciting times lay ahead.
- Moving Plex. Currently, as previously linked, my Plex server runs on MUFFSTORE01 but I have since looked at my main virtualization host and realised that it is being heavily underutilised so I am planning on moving Plex to a VM. In this move I will change the hardware of MUFFSTORE01 to a much more barebones, low power setup as it will solely be a file server and this should net me some power savings overall. I have yet to do this move as I have a lot of metadata, well over 500GB which I can’t move to MUFFHOST01 due to space on the RAID10, the SAN should fix this issue. Once my SAN has been setup and I have enough storage for these types of VMs I will start to move my Plex over to a VM, with the metadata hosted on the SAN, I am not comfortable with having the metadata over the network.
- I need a UPS. Like, badly. As stated previously I have left 2Us of space at the bottom to install a UPS, I held off on this previously because of the space constraints of my 12U but now I have the space I really need to look into getting one, the whole rack has been unexpectedly turned off more times than I would like in the past so this is quite a high priority.
- Move pfSense to its own hardware, again. Having pfSense in a VM is currently temporary as my previous box had network controller issues one day, I am still deciding what to buy/build for this, but for now, it’s working and keeping me online just fine.
Some of you always ask how loud this enviroment is with the amount that’s going on. I pay extreme attention to noise as I live in a small flat in London and this thing lives in my living room. The noise it generates is definatly noticable but you are easily able to drown it out if you don’t focus on it. Everything I do in my lab I do with noise and power in mind, so here is a little example of how loud this is with everything running normally.
And that concludes my rundown of 2016s MuffNet! Until next time folks…