This new service will allow companies who have systems in the AWS cloud to leverage a lower cost solution to store data that can easily be retrieved from another location or re-processed from source data.  You can now design your application to request data from S3 RRS, and if the data object does not exist, your application can retrieve it from another location or recreate the object (maybe from source data on the more resilient S3 storage).   This may be a great solution for companies that are using EC2 just for computing or processing large data sets, S3 RRS could reduce your monthly costs by quite a bit.

The only problem with this new storage service is that the cost is still pretty high for what you get.   Also, AWS did not decrease the cost for the data transfer to the RSS service (it’s the same as the standard S3), so they can potentially lose your data and the worst of it is you have to pay them to replace it.  But with a 99.99% reliability over the course of the year we’re probably not talking about a lot of data.

My company makes its living by storing an unlimited amount of our customers’ data indefinitely.  We can do this because we can leverage the highly resilient and low cost S3 storage to keep email, IM’s and social media messages on our system forever without making huge capital outlays in storage systems, colo’s, etc…   I think it’s great that AWS has brought in this lower tier of storage to differentiate itself with the other up and coming and established cloud computing providers by not forcing everyone to use the same “expensive” long term storage.  But for my money, it’s worth it to pay such a small amount more (5c per GB) for such a large improvement in resiliency.

Email is a management nightmare for companies.  People want to keep their emails forever, they don’t want mailbox quotas, and they have decided that email is the best place to store all their documents.   While working as a technology consultant, many people I spoke with wanted to outsource their email problem, and you see this as large organizations such as the City of LA move to Google Apps.   Now, most companies are not there yet when it comes to outsourcing all their email, but outsourcing archiving is something that many companies are steadily moving towards.  They don’t want to spend thousands on storage and servers for an archiving solution that is only used from time to time.  This is where Sonian comes in.  We can take your emails that you’ve been saving for 5 or 10 years, bring them in to our cloud based service and keep them securely, forever.  And no, we won’t charge you huge amounts to do this, just a simple flat monthly fee per user, per month.  We’ll also provide you a  web based interface to search and categorize your emails for compliance or investigatory reasons, and well as end user acesss.

So, it is a pretty exciting time here at Sonian, with almost 1900 customers in the past couple years we must on to something. And our explosive growth here only confirms that “The Cloud” is here, and people are ready to push their data up to it.

Those stats are equally impressive, since being able to achieve an almost 20% CPU decrease while increasing performance means more density per virtual host.  This further allows companies to squeeze more resources from their virtual infrastructure without needing to purchase more hardware.  And in this economy, everyone is trying to get their money’s worth when it comes to their infrastructure capital spending.

Since PVSCSI adapters are not supported for boot devices (they work, just not supported by VMware), you will need to add a 2nd hard drive to use the PVSCSI adapter.   When setting up a new virtual environment on vSphere for a client, it wasn’t clear where exactly that option is located.  It seemed when adding a 2nd hard drive, it just used the existing SCSI adapter.  On the VMware KB site, I found KB article 1010398 which talks about the steps to set that up.  Below are the details from the VMware KB site.  The most important step is #12, you NEED to select a SCSI adapter that starts from SCSI (1:0) through SCSI (3:15).  Selecting the next available SCSI interface, eg.  SCSI (0:1), uses the boot volume SCSI adapter.

When you select SCSI (1:0) or higher, you’ll see the new SCSI controller added.

1. Launch a vSphere Client and log in to an ESX host system.
2. Select a virtual machine, or create a new one.
3. Ensure a guest operating system that supports PVSCSI is installed on the virtual machine.  Currently:
   Windows Server 2008
   Windows Server 2003
   Red Hat Enterprise Linux (RHEL) 5

   Note: Booting from a disk attached to a PVSCSI adapter is not supported. The system software must be installed on a disk attached to an adapter that does support bootable disk.

4. In the vSphere Client, right-click on the virtual machine and click Edit Settings.
5. Click the Hardware tab.
6. Click Add.
7. Select Hard Disk.
8. Click Next.
9. Choose any one of the available options.
10. Click Next.
11. Specify the options your require. Options vary depending on which type of disk you chose.
12. Choose a Virtual Device Node between SCSI (1:0) to SCSI (3:15) and specify whether you want to use Independent mode.
13. Click Next.
14. Click Finish to finish the process and exit the Add Hardware wizard. A new disk and controller are created.
15. Select the newly created controller and click Change Type.
16. Click VMware Paravirtual and click OK.
17. Click OK to exit the Virtual Machine Properties dialog.
18. Power on the virtual machine.
19. Install VMware Tools. VMware Tools includes the PVSCSI driver.
20. Scan and format the hard disk.

I hope this helps anyone who wants to use the PVSCSI adapter, but is having trouble locating how exactly you add it to your virtual machine.  This vSphere update and the blog article referenced above from the VMware website is just another example that shows if you size your virtual environment correctly, you can virtualize even your highest demanding I/O enterprise applications.

VMware Fault Tolerance is leading edge technology that provides continuous availability for applications in the event of server failures,  by creating a live shadow instance of a virtual machine that is in virtual lockstep with the primary instance. By allowing instantaneous failover between the two instances in the event of hardware failure, VMware Fault Tolerance eliminates even the smallest of data loss or disruption.

At VMworld 2008 they let us play with a demo of VMware FT, and it really is an amazing technology.  Almost like watching your first VMotion (“You mean the VM moved from this server to that server?”).   VMware FT will allow you to have two running versions of the same virtual machine.  If you lose a host, the VM will continue running with no dataloss and minimal downtime (technically just a couple pings drop, but your users would not be likely to notice a disruption of service).  VMware FT does this by sending the same CPU instructions to both CPU’s via a FT logging NIC, which is a dedicated gigabit or better ethernet NIC on your vSphere hosts.

With any software that gives you that kind of power, there are some caveats and requirements to make FT work in your environment.   I felt it was a good idea to start a blog post that I could update with the various requirements for the use of FT with vSphere.  This list is my no means all-inclusive, but simply a place where I can keep track of the needs and caveats of FT.  Read more for my listing of requirements that I’ve found thus far.

Host Requirements:

CPU

• AMD Barcelona (Series 13xx, 23xx, 83xx)
• Intel Harpertown (Intel 31xx, 33xx, 52xx, 54xx, 74xx)
• Specifically a Hardware Virtualization (HV) enabled CPU
• Intel VT or AMD-V enabled in the BIOS
• Disable any power management in the BIOS (recommended)
• Disable Hyper Threading (recommended)

Network

• 2 FT Logging NIC (suggested)
• 1Gbps or better

Storage

• Shared
• Fiber Channel, iSCSI, or NAS

ESX

• Same build version of ESX on each host.
• VMware HA must be enabled on the primary and secondary hosts in the cluster.

Guest Requirements:

• All ESX supported guest OS’s, 32bit or 64bit
• One (1) vCPU  – vSMP is not yet supported.
• Thin-provisioned disks are not supported (they will be converted to thick)
• Paravirtualization is not supported
• Physically attached CD-ROM, Floppy not supported
• Physical RDM’s (Raw Device Mappings) not supported – Virtual RDM is supported.

Caveats:

• Storage VMotion not supported
• N-Port ID Virtualization (NPIV) not supported
• Need to have no single points of failure in any part of the environment (not required, but defeats the point of FT if your environment is not redundant).
• DRS can not be enabled on the protected VMs (You can still run manual VMotion’s)
• Hot add of of devices to the protected VMs is not supported
• Snapshots are not supported (must be deleted before protecting)
• VM Hardware must be at v7
• Remove 3rd party clustering solutions prior to enabling FT.

Anything else that I’m missing?  Let me know in the comments, and I’ll keep the above info updated.

To make the snapshot management easier, I store the VM configuration files on a separate LUN, and where you store the VM configuration files is where the snapshot deltas are created.  This lets us keep the main VMDK’s LUN’s fairly static, without worry of snapshots filling up our available space.   When looking at a specific VM today, I noticed that the data stores listed only showed the Snapshot LUN.  This had meant that there was a snapshot taken which had not been removed.  This particular VM was not currently replicating, so I knew that snapshot should not have existed.  Normal operation should show both the snapshot LUN and the VMDK LUN.

This VM:

Working VM:

When going to the Snapshot Manager, I was not able to see any snapshots on that virtual machine.

I accessed the Datastore Browser to see if there were any delta VMDK’s on the disk; there ended up being 2 delta’s on my datastore.

I wanted to confirm that the virtual machine was indeed running off the delta disk.  To check that, I simply went to edit the settings of this virtual machine, and looked at the virtual disk object.  In this instance it was accessing the disk “exch-000002-delta.vmdk”, which was one of my delta disks.

There is a command you can run on the service console to try to remove snapshots if you are unable to with the VI Client.

/usr/bin/vmware-cmd <path to VM cfg> removesnapshots

When I ran this command, I received the following error:

VMControl error -3: Invalid arguments: Virtual machine has no snapshots

Doing some research on the VMware communities website, I found a recomendation to create a new snapshot excluding the VM memory, and then removing the snapshot.  When I created a new snapshot on my virtual machine, I saw something very interesting.   I saw an additional snapshot called “Consolidate-Helper-0″

At this point I deleted all the snapshots from the VM, and waited for the process to finish.  A couple of my snapshots were pretty large, so vCenter timed out before they finished.  I waited an hour, and then confirmed they were gone by checking the virtual disk resource in the VM settings.

Disclaimer – Before doing anything like this, make sure you have adequate backups and understand that this is probably not supported by Microsoft or VMware – I take no responsibility for any damage to your systems

If you would like to grow a virtual disk the “safe” way (read: much, much slower), you can use VMware’s Converter tool, which will grow the volume for you on conversion, the downside is that it will require the VM to be shutdown during the conversion.   The steps below can significantly speed up the process, especially if you have a very large VMDK which needs to be extended.

To grow a system volume, you are going to need a 2nd virtual machine that will act as the helper VM to mount the virtual disk you would like to extend.  I will be using 2 virtual machines, both are Windows 2003 standard.

The disk on the VM that I need to extend is currently 20GB, I would like to make this 80GB.  I have already made sure that I have enough space on my VMFS formatted storage partition to support this larger VMDK.

Next, I need to shutdown the VM to extend the disk.  There are 2 different ways to extend a VMDK.  The easiest way to grow the VMDK disk is to right-click the virtual machine in the VI Client, and select “Edit Settings”

Then Click on the virtual disk object, and enter in the new size of the VMDK (in my case I would like to extend this to 80GB), then click OK.

Power on the VM to make sure the OS can see the additional space; right-click “My Computer” >> “Manage” >> “Disk Management”.  It will not be able to use the space until we tell the OS to extend the partition to fill the remaining space.  You should see something similar to the image below.

Now, shut the VM down again, and also shutdown the helper VMh, the virtual machine that you are going to use to mount and extend the disk.   Once both virtual machines are shutdown, we are going to remove the virtual disk we are growing from our VM, and attach it to our helper VM.  To do this, right click the VM we are growing and select “Edit Settings”, click the virtual disk resource, and click “Remove”.

Click the option to “Remove from Virtual Machine”, do NOT click the other option, this WILL delete your virtual disk.

Now right click our helper VM, select “Edit Settings”, then select “Add” toward the bottom of the screen.

Select “Hard Disk”, next, and select the option to use an existing virtual disk.

Next, you will need to browse for the VMDK file that we are extending.  Since we have already grown the virtual disk to 80GB you will notice that in the size column for this VMDK.

After selecting the full path to your virtual disk, click thru the last couple pages, and you will see the virtual disk added to the VM’s hardware list.  After clicking OK, power-on the virtual machine, logon, and you will see the other VM’s disk connected at D:\ (or the next available drive letter).

Now, we can finally run diskpart to extend our volume.  Click “Start” >> “Run” >> type “cmd” >> type “diskpart”.  You should see the diskpart command prompt.

Type “list volume” to display a list of all the attached volumes:

DISKPART> list volume

Volume ###  Ltr  Label        Fs     Type        Size     Status     Info
———-  —  ———–  —–  ———-  ——-  ———  ——–
Volume 0     C                NTFS   Partition     20 GB  Healthy    System
Volume 1     D                       CD-ROM          0 B  Healthy
Volume 2     E                NTFS   Partition     20 GB  Healthy

We want to select the system volume 2, the boot volume is the disk for our helper VM.

DISKPART> select volume 2

Volume 1 is the selected volume.

Then we extend:

DISKPART> extend

DiskPart successfully extended the volume.

If you launch disk management, you will see the volume is now a single 80GB partition.

Finally, we shutdown our helper machine, detach the disk, and reattach it to our original VM.  When you power on the original with our extended disk, you will immediately see the additional space.  You may be prompted to reboot one time, so that the operating system will be able to work with the larger re-added disk.

Enjoy your additional space!

Expanded Support for Enhanced vmxnet Adapter — This version of ESX Server includes an updated version of the VMXNET driver (VMXNET enhanced) for the following guest operating systems:

• Microsoft Windows Server 2003, Standard Edition (32-bit)
• Microsoft Windows Server 2003, Standard Edition (64-bit)
• Microsoft Windows Server 2003, Web Edition
• Microsoft Windows Small Business Server 2003
• Microsoft Windows XP Professional (32-bit)

The reason this is such a great update, is because of the support for jumbo frames within this new group of OS’s.  Previously the enhanced vmxnet driver was only supported on Windows Enterprise and Datacenter versions.  When using certain iSCSI arrays which support application consistant data snapshots, you need to install the iSCSI initiator within the virtual machine.  This is the only way for the EqualLogic toolkit to take application level snapshots of Microsoft SQL or Exchange Server.   Additionally you will require a NIC which supports jumbo frames.   Previously this was done via a dirty (and unsupported) hack if you were running the non-supported OS.  Now that VMware supports (for data traffic only) jumbo frames for the standard version of Windows OS’s, this can decrease the CPU usage for the guest VM, while not having to spend 4x as much for the enterprise version of the OS.   So it’s still “technically” not supported for iSCSI traffic, but works great and can lower the guest CPU usage during high data IO operations.

Additionally, as a old school linux guy and Ubuntu fan, I’m glad they are adding support for newest version of Ubuntu desktop and server, 8.10 – Intrepid Ibex