ESXi Security Service Configurator (PowerCLI)

4 08 2012

Personally, I get annoyed when I have to dig through the vSphere Client GUI to turn on or off certain ESXi services on a regular basis. Since admins are generally on top of it in terms of following good security standards, I see Lockdown Mode on and SSH off by default on their ESXi hosts in many environments. When troubleshooting issues or configuring certain VMware-integrated products (such as HyTrust Appliance), it is sometimes necessary to temporarily undo this setup (enable SSH and disable Lockdown Mode).

The tool linked below can be used to turn on or off SSH and/or Lockdown Mode for a single host or all hosts in the environment. As usual, feel free to use all, some, or none of the code. I’m hoping to add additional services to it in the future, but these two are consistently needing to be toggled…

What it looks like in action:

Find virtual machine snapshots with PowerCLI

2 10 2011

Run from a PowerCLI session connected to a vCenter environment to find and list all of the snapshots (and users  who took them, which Get-VM | Get-Snapshot won’t do) on your managed ESX/ESXi hosts:

$myVMs = Get-VM
$VMsWithSnaps = @()
foreach ($vm in $myVMs) {
    $vmView = $vm | Get-View
    if ($vmView.snapshot -ne $null) {
        Write-Host "VM $vm has a snapshot"
        $SnapshotEvents = Get-VIEvent -Entity $vm -type info -MaxSamples 1000 | Where { 
            $_.FullFormattedMessage.contains("Create virtual machine snapshot")}
        try {
        $user = $SnapshotEvents[0].UserName
        $time = $SnapshotEvents[0].CreatedTime
        } catch [System.Exception] {
            $user = $SnapshotEvents.UserName
            $time = $SnapshotEvents.CreatedTime
        $VMInfo = “” | Select "VM","CreationDate","User"
        $VMInfo."VM" = $vm.Name
        $VMInfo."CreationDate" = $time
        $VMInfo."User" = $user
        $VMsWithSnaps += $VMInfo
$VMsWithSnaps | Sort CreationDate

Implementing the HyTrust Appliance – Part 3

16 08 2011

Roles, Rules, & Constraints (Oh my.)

IMO, the flagship features of the HyTrust Appliance (HTA) are the additions to the default vCenter Server security mechanisms through a more granular set of access controls to the virtual environment.  As always, a layered approach to security is ideal, and use of the appliance provides just that. For more background, check out my previous posts on the HTA:

Implementing the HyTrust Appliance – Part 1 (deployment, configuration considerations)

Implementing the HyTrust Appliance – Part 2 (compliance templates, root password vaulting)

Read the rest of this entry »

Stopped VDR backup leaves behind hidden virtual machine snapshots

16 06 2011

Earlier this week a long-running VMware Data Recovery (VDR) job was stopped due to concerns over space usage on the source datastore.  While the job seemed to stop cleanly, it logged this error in the VDR logs:

error -3942 (Delete snapshot failed)

At first, this didn’t seem like a big deal – I simply went to delete all snapshots using the vSphere Client… and there were none listed.  Seeing the same result when pointing the vSphere Client directly to the host the virtual machine was running on, I started looking for a non-GUI way to do this and came across this KB article:

Committing snapshots on ESX/ESXi host from command line

These steps did eventually work to remove the snapshots, but it should be noted that I had to create another snapshot before the snapshot.get command would list anything.  Once the extra snapshot was in place the snapshot.removeall command did the trick.

Implementing the HyTrust Appliance – Part 2

29 03 2011

Last time I went into our deployment of the HyTrust Appliance (HTA), some configuration considerations, and setting up the appliance for centralized authentication using Active Directory.  For this post I will talk a bit about our use of host compliance templates and, my favorite feature, root password vaulting. Read the rest of this entry »

Killing a virtual machine in ESXi 4.1

26 02 2011

* I’m pretty sure at least one step I take in this post is unsupported by VMware, so please use at your own risk.  If you don’t feel entirely comfortable with any part of this, and have a support contract – use it.

I was re-IPing a virtual machine (VM) today when it locked up and showed no signs of life (zero CPU usage, disk I/O, etc.). I went in through the host that it runs on using the vSphere Client and tried to hard reset it… which stuck at 95%. I restarted the management agents on the host, which successfully killed the reset command.

Now the VM showed as invalid, so I tried unregistering the VM, then selecting “Add to Inventory” by right-clicking on the <vmname>.vmx file in the VM’s folder on the datastore.  While this made the host recognize the virtual machine correctly again, it didn’t help with booting the VM up – I was now getting an error while booting that told me the virtual machine swap file (<vmname>.swp) was locked.

I’d seen similar errors before, and have a good history with the tried-and-true hard kill method for resolving these:

ps aux | grep <vmname>  –> which gave me a series of process IDs (pid) for the VM’s world
kill -9 <pid>

which resulted in:

cannot kill pid 123123: No such process

Uh-oh.  At this point I began contemplating contacting all of the customers whose VMs (about 40 of them) are on this host for an emergency reboot of their systems so I can reboot the physical host.  Then I found some threads that talked about the vm-support command and how it can hard crash the VM world for you. Huh.  Cool.  Here’s how I used that to make my day much better:

vm-support -x   (this lists out all of the VM worlds running on your host)

When I found the numeric world ID (wid) for my VM, I then ran this command and allowed it to kill the VM:

vm-support -X  <wid>

This process took a while (10-15 minutes), but it worked!  Here’s a good KB article on the use of vm-support and other means of killing virtual machines that just won’t stop:

Implementing the HyTrust Appliance – Part 1

25 02 2011

It happened for our Windows infrastructure several years ago, and now it’s happening at the hardware virtualization layer – we’re too big for our britches and lack a solid methodology for monitoring, securing, and maintaining our vSphere systems as we continue to expand.  Originally, I approached HyTrust regarding their HyTrust Appliance (HTA) primarily as a means to implement two-factor authentication for the virtual infrastructure via RSA SecureID.  Our RSA/two-factor project is in limbo, but I quickly realized that HyTrust had much more to offer us than just two-factor authentication.  Their implementation of host configuration templates alone has made it worth the purchase, not to mention the granular access policies and auditing, and root password vaulting (my favorite).  Over a couple of posts, I hope to relay some of the implementation trials, tribulations, and successful milestones as we roll the HTA into production and start to get a better handle on our environment.

Read the rest of this entry »