"Managed Services"? Think twice....

Michael’s MSP Rant

It seems like the entire IT industry is talking about (and promoting and selling) “Managed Services”. The idea that for one (1), “low”, fixed monthly price, all of your IT woes go away.

Outsourcing your IT to an MSP isn't “always” a bad idea; It’s just “mostly” a bad idea.

The supposed list of “benefits” reads like;

  • Knowing your monthly IT bill in advance, will allow you to budget reliably

  • There is an “all you can eat” option, call us for everything/anything

  • Your IT will finally do what you want it to do

  • We will manage everything in the background

  • Something about “reliability….”

  • Everyone is doing it

  • As your MSP (“Managed Service Provider”), having predictable income allows us to hire the perfect people to manage your IT

  • Blah, blah, blah...etc, etc, etc.

Sounds great right? But, you’re gut instinct is “Wow! This sounds to good to be true.”.

Odds are, you’re right. It is to good to be true. And while not always a bad idea, it’s a bad idea, most of the time (and we will get to when it actually works below)…

Perhaps your unhappy with your current IT solutions or maybe you’ve been approach by someone promising to make things better. Maybe it is the cost of hiring. Maybe it just doesn’t feel like your getting the value of the dollars spent. Maybe you’ve heard about Managed Services, and your just MSP curious.

So what’s the catch?

  • In actually, you’ve inadvertently financial motivating someone to NOT service your business.

What?

  • The math looks like this;

    • They promise the moon, AND give you a great (lower?) predictable price. That is what starts the cycle (see below).

    • They spend some number of hours at your site “loading agents”, “upgrading your anti-virus” software, patching servers, changing equipment to “there solution”, labeling things, and leaving you with a sense of value for your money.

    • As expected, they send you the bill, and you happily pay them, month after month.

    • Then, what starts out as apparently great service, starts to slowly turn.

Why? Think about it this way;

    • They have your money. They have your money this month, and next month, and the next month.

    • Replacing them means

      • finding someone else (your time)

      • the new persons implementing there solution (the entire companies time and money)

    • If the MSP did little (or nothing), they still get the monthly service fee. 100% profit (minus what they pay for those agents they loaded)

    • Worse still (it gets worse?), EVERYTHING they do;

      • answering the phone

      • replying to an email

      • sending someone to your office

      • spending time fixing a problem, etc.

      • ALL OF IT, eats into what would otherwise be 100% profit for the MSP.

    • In effect, you’ve paid someone to NOT service your business

    • In this relationship, you have incentivized them to do the least amount of work possible;

      • If they ignore the problem, and your staff “work around it”, they make (rather, get to keep) more money

      • After your on-boarded, “support” might be outsourced to another company (and often another country)!. They make more money

      • They wait till is gets bad enough that a “special project” is needed to fix it? They make more money.

      • Getting the picture?

    • So, since the service isn’t what it used to be, and you’re back to spending a fortune, you go looking for a new (and often cheaper) MSP. More of your time and money.

    • So the next MSP comes along, and they would “be very happy to charge less”; for also doing nothing, and ultimately the same (terrible) result.

    • This leads to either a high-turnover or a putting up with, “race to the bottom”. Who can sell it (and not deliver it) cheaper

“Ouch! Thanks Michael, is there any good news?”

I’m glad you asked, and “Yeah, some good-ish news...”

I’ve actually worked with, and client’s have found, the “MSP Right-size-sweet-spot”.

The MSP “sweet spot” is an MSP who has grown enough (but not to big) to have dedicated, properly utilized, not over or under worked staff for particular IT roles. People are challenged, and winning is possible. The right amount of staff, such that the MSP doesn’t have a microscope on expenses, they pay for top people, the work gets done, and the team delivers reliably.

For example, they have a full time person whose sole mission and focus in life is desktop optimization (patching, tuning, software delivery, documentation for other staff, etc)

“So, how do I find such an MSP”

I’ve hired (and/or worked with) a few GREAT MSPs over the last 5 years, and here is what they mostly have in common for staffing;

      • 2 or 3 “helpdesk” / Tier I administrators

      • 2 Tier II IT Administrators

      • 2 Tier III IT Administrators / special projects

      • 1 or 2 Network / WiFi specialist

      • 2 Sales people / account managers

      • 1 project manager

      • Optional “Mobile phone” expert

      • Optional “Web site creator/administrator”

Larger than this, and you become a small fish. Smaller than this, and they don’t reliably deliver.

How to select a good MSP

      • Ask where there help desk is based out of. Make sure it is US based.

      • insist in the contract that if the MSPs staff becomes outsourced (another company or off-shore) you have a no penalty cancellation clause

      • Insist that they store system passwords in such a way that you always have access (“you can always just call us..” doesn’t cut it).

      • Ask for “the” phone number you’ll be calling for support, BEFORE you sign up. Not the salesman's phone number. Not the “on boarding team” phone number. “The” support phone number. The one you’re to call after you’ve been on-boarded. If they aren’t willing to share it, then don’t share you dollars.

      • When they do share the “support” number, make a few test calls.

        • See what the response time is (number of rings).

        • Confirm if this is the support number for company “X”.

        • Ask the person who answers how long have they been with company X

        • Ask the person where they are working from. Which state?

        • How long have they been in IT?

        • How do they react to someones call that they cant find in there database?

        • Politely thank them for there time.

      • Make certain your clear what is/isnt in the MSPs deliverables

      • Ask for an annual estimated “Out of contract” work to be done, for both “on-boarding” and an ongoing annual “project” budget

Personally, having experimented with offering MSP solutions, and at the end of the day, I don’t offer MSP solutions. The field is crowded, with a lot of bad apples messing it all up, and, suprisingly, I find it more cost effective for the client to hire someone who is great at what they do, and capable of letting the customer know if/when they are not.

So don't overlook the old fashioned way, of hiring someone hourly. If they don’t get the work done. Fire them. Get another, and repeat until you find the right one. After all, not only is that cheaper, it might require less time (and fewer headaches) than you’d have spent going through multiple MSPs in a race to the bottom.

VMWare to HyperV architecture, planning, and migration

Sometimes, even the best planning needs to be tweaked on the fly.

In this case, what appeared to be a 'simple' 9TB, 6 Server, VMWare to HyperV migration, morphed into an "opportunity to get creative."

Issue: Replace aging server hardware, and incorporate a more robust (time to recovery) disaster recovery architecture. 

On the surface, this was a slam dunk.  Migrate the VMWare virtual machines to new hardware running HyperV, setup HyperV replication, and configure a second Exchange Server for DAG (Database Availability Group). 

While the architecture and planning seemed solid (we had done a lot of this type of migration), what was unexpected was the throughput of the existing network, the speed of the older server hardware, and having filenames/paths that exceeded the 255 character limit (didn't see that coming!). 

The first 4 machines migrated without a hitch (although slower than expected), inside the outage windows, and ran great on the new HyperV platform.

Regarding the migration speed, it turns out that the hardware drivers (Dell Servers) on the old Vmware servers was the out of the box, VMWare versions, and not the Dell optimized versions. The client never noticed this "opening a single file", or when "checking email" on the Exchange Server. 

We, on the other hand, were trying to grab multiple Terabytes of data from the drives (old controller driver) across the network (old network drivers).

The first big gotcha was existing corruption of the VMWare VMDK files.  Anytime we attempted to convert a copied out VMDK file to VHD(x), it failed. We tried all the different ways to copy it out, and multiple converters.  No luck.

Here is the first (and unsuccessful) clever part: we built a temporary, virtual, Windows 2012R2 Server, on the new HyperV server, and then joined the temporary server to the AD domain. Next, we started a robocopy from the old virtual machines D: drive to the temporary machines D: drive; basically the same setup as any physical to physical migration.

Sounds like a no-brainer, right?   

Not quite. The next issue was long file names.  Windows Server has a limit of 255 characters for the path and filename.  While not recommended, this isn't a problem when mapping a drive to a folder lower in the directory, but a big problem for RoboCopy (and as it turns out, the migration problem for our usual assortment of VMDK to VHD(x) migration tools)!

Here is the 2nd, and in this case, successful, clever part.   Robocopy clearly wasnt going to work on the long filenames.  So, we used the backup software to backup the old D: drive, and then recover the D: drive to the temporary server (on a Wednesday).  All of the files the client was updating where not in the 255+ range, so we ran Robocop every night, to sync the changes.  

Then on Friday night, for the migration, we ran one last robocopy to get a solid D: drive. All that was left was to shut down the old server, copy out the VMWare VHDK file of the C: drive, convert it to a VHDX file, and create a new virtual machine on the HyperV server with it.

We started the virtual server in its new home, on the HyperV Server.  After everything looked good, we shut it down, as well as the temporary server.  We removed the temporary server's D: drive and attached it to the server in its new location. After starting up, the server now saw its D: drive, and purred like a kitten! 

Now out of the bushes, we configured the HyperV servers (3 total) to replicate between each other, as part of the new disaster recovery plan (plain vanilla certificate-based replication). 

After deploying the new solution, the network speed and server response is incredibly faster, the disaster recovery window is much smaller (10 minutes), and the server footprint went from 16u to 6u.

Not the easiest migration we've done, but certainly one of the most educational!

Wireless network for 500 simultaneous users, wireless roaming, with multiple SSIDs

Wireless networks get complex when the use case includes roaming, high densities, multiple SSID's, VLANs, or interference.  

In this case, the client needed to provide a more reliable wireless network to church attendees, volunteers and employees. 

Issue:  Wireless users where getting dropped due to access point congestion (SonicPoint Access Points). 

After an onsite wireless assessment, we spec'd the model and location of the new access points, and ordered hardware (Ruckus wireless Access Points and Controller in this case). 

While waiting for hardware, we assisted the client in tested the existing Internet connection (50mbps) and configured the firewall with an additional isolated network.

After deploying the new solution, there are no dropped connections or lag, and the public/private networks are separated.

It's here! It's here! Server 2016 and Containers!

At App Gap, we do a lot of virtualization.  That includes P2V (physical 2 virtual), V2V (virtual to virtual), tuning, replication, backup, etc...you get the picture. 

It isn't unusual to P2V six servers onto 1 physical Hyper-V server. 

But Containers!! Well, "Containers" are THE next evolution of virtualization.  Containers will allow for EVEN MORE efficient use of hardware! Think dozens of Containers (virtual machines) on 1 physical server!

So what does that mean for my Windows Server environment?  Well... guess what! Container support is built into Windows Server 2016!

Microsoft has a great 5 minute read on Containers, here:

https://msdn.microsoft.com/en-us/virtualization/windowscontainers/about/about_overview

Then, take another minute, and at least check out the picture on this page, and read the paragraph above it, to get a quick understanding:

https://azure.microsoft.com/en-us/blog/containers-docker-windows-and-trends/

And for you hard core types, 'Docker' is what started it all:

https://www.docker.com/whatisdocker