About a month ago my Jira board grew with more and more stories asking for new or monitors, bug fixes and additional metrics for Sensu. We’ve had a few large projects start recently with came with tight deadlines and large resource needs, so I didn’t expect to complete these stories for at least a few months. Based on the schedule, I thought if we can have everyone spend one day to work on some of these stories/tickets, the larger projects shouldn’t be delayed.

We have engineering-wide codefests during our company meetups three times a year. These are fantastic opportunities for team members across all parts of our engineering teams (devs and non-devs alike) to work on and present new solutions and ideas to the entire company. We needed a day to hack on a specific project (our Sensu monitors and metrics), and I didn’t want to waste codefest on that. I needed a separate day, just to hack on a specific project.

I presented the idea to the head of engineering, who signed off on our experiment as long as we had some clearly defined goals we were working towards. I wanted specific tasks defined before we started so that we wouldn’t waste time planning when we should be hacking. So, I started planning for our hackday about 3 weeks in advance. I made sure to coordinate with all the rest of the teams leads to make sure they were aware and would send any requests directly to myself for the day. I would then act as the buffer for any 3rd level support or other assistance our DevOps teams provides.

Over the next 3 weeks, everyone on the DevOps team added ideas to a wiki document; things that were keeping them up at night, code they wanted to refactor, etc… Two days before our hackday, we reviewed the list of items and assigned ownership to all the tasks (Pairing was encouraged). This allowed the geographically diverse team to be able to hit the ground running when they logged on for the day (Since everyone starts working at different times). On the day of our hackfest - everyone had their task list and began working on the various fixes we had lined up. We had a quick standup mid-way thru to check on status, blockers, etc, and to make sure we were making progress.

So - what were the results?

Comparing the diff for our branch on github - we saw the following: 47 changed files with 689 additions and 478 deletions

Now - what did we really get done?

• We created 7 new monitors for things we had not tracked in the past
• We converted the last few of our nagios checks over to use the sensu-plugin. Huzzah! - no more Nagios anything!
• We consolidated 5 monitors down to one which simplifies our codebase while still providing full insight into our systems.
• We cleaned up our metrics - removing nearly 20 separate ones that we found did not provide us with actionable information. And since we use Librato - this is real $$$ saved. But instead of saving that money - we decided to reinvest and increase the metric check interval - so our metrics could capture more data more often.
• We added new metrics - creating about 12 new metrics for our indexing clusters - everything from memory and load as well as application level data, health etc…

Overall - the team enjoyed taking a break from our big projects to fix things that keep us up in the middle of the night (literally). And I enjoyed a much smaller backlog of work as well as personally fixing and adding new sensu checks.

We were able to complete about a week’s worth of work in a single day. We added in a few hours of testing and validation, and deployed the code out to production the next day (Oh, the joys of DevOps). The success of this project has already sparked the interest of other engineering teams who are considering doing similar hackdays. My team has already begun to discuss doing this again once every couple months with different focuses each time.

Feel free to ask my team if you want their input on the success of our hackday. (Decklin, Sean,Justin, Josh, and Sam.

Sonian is continually committed to Sensu as an open source project - which is why we share many of our checks with the rest of the community - you can check out many of the new and refactored checks (the result of our hackday) by heading over to Sensu’s community plugins page.

One of our AWS accounts is a non-production account where we spin up and down test systems to support new feature testing and other activities to support development. Our build cluster (which lives in a separate AWS account) needs access some S3 buckets that live in our non-production account. The problem is that this account also contains some of our more sensitive stacks, like QA and UA systems, which are locked down from a moderate to a production level of access. So, in this case, I needed to create an IAM user on our build cluster account that could access specific buckets in our Non-Prod account. This is to ensure we’re not giving out keys that could be used to potential cause data destruction.

Here is how you can do it if you are looking for the same level of Amazon cross-account access for S3 buckets (with granular per-bucket IAM level permissions).

This guide assumes that you already have an IAM user on the account you want to enable access from (Our build cluster account). And you’ll need to know the S3 bucket in question (This will be a bucket called my-shared-bucket-20932 in our non-production account). I’m also going to assume you have a basic understanding of IAM (How to create users, groups, etc…)

First step is to get your AWS account ID number - this is a 12 digit number. You can find it in your account activity report.

Next - go to the IAM user you are going to be granting access for to grab the User ARN. You will need this when you create the bucket policy on the bucket in the other account you’re granting them access to.

After you have both the AWS account ID number 1234567890 and your User ARN arn:aws:iam::1234567890:user/test.user, access your S3 bucket properties for the bucket you are going to allow access to. My example is going to use the bucket my-shared-bucket-20932.

Now you are going to need to paste in a bucket policy in JSON format - here is an example one below to enable “Get” access to anything in the bucket my-shared-bucket-20932. If you wanted to allow full access, you could change the s3:Get* to s3:* or even s3:Put* if you only wanted to allow uploading files.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

{
  "Version": "2012-02-02",
  "Statement": [
      {
          "Sid": "NonProdAccountAccess",
          "Effect": "Allow",
          "Principal": {
              "AWS": "arn:aws:iam::1234567890:user/test.user"
          },
          "Action": "s3:Get*",
          "Resource": "arn:aws:s3:::my-shared-bucket-20932/*"
      }
  ]
}

Now that you have your S3 bucket policy in place - it’s time to create a new group that your user(s) will be a member of to allow them the ability to access the bucket in your other account.

Access your IAM tab in your AWS console - Create a group - and when your group is created - create a policy with the following information

1
2
3
4
5
6
7

{
   "Statement":{
      "Effect":"Allow",
      "Action":"s3:Get*",
      "Resource":"arn:aws:s3:::my-shared-bucket-20932/*"
   }
}

It’s going to look something like this:

Finally - make your IAM user a member of the group you just created - any they will now have access to your S3 bucket in a separate AWS account.

In addition - I’m working to increase my knowledge of Ruby as we use Ruby for all our tools and applications. Coming from a non-computer science background, that will be a challenge in itself. I’ve starting by reading the fantastic book written by Chris Pine - Learn to Program.

I’ll be writing more about some of the challenges of managing big data and large system at scale in the cloud - where you have no control over network and disk I/O (among other things we can’t control), and discuss how we deploy, monitor, scale and secure our systems.

This new service will allow companies who have systems in the AWS cloud to leverage a lower cost solution to store data that can easily be retrieved from another location or re-processed from source data.  You can now design your application to request data from S3 RRS, and if the data object does not exist, your application can retrieve it from another location or recreate the object (maybe from source data on the more resilient S3 storage).   This may be a great solution for companies that are using EC2 just for computing or processing large data sets, S3 RRS could reduce your monthly costs by quite a bit.

The only problem with this new storage service is that the cost is still pretty high for what you get.   Also, AWS did not decrease the cost for the data transfer to the RSS service (it’s the same as the standard S3), so they can potentially lose your data and the worst of it is you have to pay them to replace it.  But with a 99.99% reliability over the course of the year we’re probably not talking about a lot of data.

My company makes its living by storing an unlimited amount of our customers’ data indefinitely.  We can do this because we can leverage the highly resilient and low cost S3 storage to keep email, IM’s and social media messages on our system forever without making huge capital outlays in storage systems, colo’s, etc…   I think it’s great that AWS has brought in this lower tier of storage to differentiate itself with the other up and coming and established cloud computing providers by not forcing everyone to use the same “expensive” long term storage.  But for my money, it’s worth it to pay such a small amount more (5c per GB) for such a large improvement in resiliency.

Email is a management nightmare for companies.  People want to keep their emails forever, they don’t want mailbox quotas, and they have decided that email is the best place to store all their documents.   While working as a technology consultant, many people I spoke with wanted to outsource their email problem, and you see this as large organizations such as the City of LA move to Google Apps. Now, most companies are not there yet when it comes to outsourcing all their email, but outsourcing archiving is something that many companies are steadily moving towards.  They don’t want to spend thousands on storage and servers for an archiving solution that is only used from time to time.  This is where Sonian comes in.  We can take your emails that you’ve been saving for 5 or 10 years, bring them in to our cloud based service and keep them securely, forever.  And no, we won’t charge you huge amounts to do this, just a simple flat monthly fee per user, per month.  We’ll also provide you a  web based interface to search and categorize your emails for compliance or investigatory reasons, and well as end user acesss.

So, it is a pretty exciting time here at Sonian, with almost 1900 customers in the past couple years we must on to something. And our explosive growth here only confirms that “The Cloud” is here, and people are ready to push their data up to it.

Those stats are equally impressive, since being able to achieve an almost 20% CPU decrease while increasing performance means more density per virtual host. This further allows companies to squeeze more resources from their virtual infrastructure without needing to purchase more hardware.  And in this economy, everyone is trying to get their money’s worth when it comes to their infrastructure capital spending.

Since PVSCSI adapters are not supported for boot devices (they work, just not supported by VMware), you will need to add a 2nd hard drive to use the PVSCSI adapter.   When setting up a new virtual environment on vSphere for a client, it wasn’t clear where exactly that option is located.  It seemed when adding a 2nd hard drive, it just used the existing SCSI adapter.  On the VMware KB site, I found KB article 1010398 which talks about the steps to set that up.  Below are the details from the VMware KB site.

The most important step is #12, you NEED to select a SCSI adapter that starts from SCSI (1:0) through SCSI (3:15).  Selecting the next available SCSI interface, eg. SCSI (0:1), uses the boot volume SCSI adapter.

When you select SCSI (1:0) or higher, you’ll see the new SCSI controller added.

Note: Booting from a disk attached to a PVSCSI adapter is not supported. The system software must be installed on a disk attached to an adapter that does support bootable disk.

1. Launch a vSphere Client and log in to an ESX host system.
2. Select a virtual machine, or create a new one.
3. Ensure a guest operating system that supports PVSCSI is installed on the virtual machine.  Currently: _Windows Server 2008 Windows Server 2003 Red Hat Enterprise Linux (RHEL) 5
4. In the vSphere Client, right-click on the virtual machine and click Edit Settings.
5. Click the Hardware tab.
6. Click Add.
7. Select Hard Disk.
8. Click Next.
9. Choose any one of the available options.
10. Click Next.
11. Specify the options your require. Options vary depending on which type of disk you chose.
12. Choose a Virtual Device Node between SCSI (1:0) to SCSI (3:15) and specify whether you want to use Independent mode.
13. Click Next.
14. Click Finish to finish the process and exit the Add Hardware wizard. A new disk and controller are created.
15. Select the newly created controller and click Change Type.
16. Click VMware Paravirtual and click OK.
17. Click OK to exit the Virtual Machine Properties dialog.
18. Power on the virtual machine.
19. Install VMware Tools. VMware Tools includes the PVSCSI driver.
20. Scan and format the hard disk.

I hope this helps anyone who wants to use the PVSCSI adapter, but is having trouble locating how exactly you add it to your virtual machine.  This vSphere update and the blog article referenced above from the VMware website is just another example that shows if you size your virtual environment correctly, you can virtualize even your highest demanding I/O enterprise applications.

VMware Fault Tolerance is leading edge technology that provides continuous availability for applications in the event of server failures,  by creating a live shadow instance of a virtual machine that is in virtual lockstep with the primary instance. By allowing instantaneous failover between the two instances in the event of hardware failure, VMware Fault Tolerance eliminates even the smallest of data loss or disruption.

At VMworld 2008 they let us play with a demo of VMware FT, and it really is an amazing technology.  Almost like watching your first VMotion (“You mean the VM moved from this server to that server?”).   VMware FT will allow you to have two running versions of the same virtual machine.  If you lose a host, the VM will continue running with no dataloss and minimal downtime (technically just a couple pings drop, but your users would not be likely to notice a disruption of service).  VMware FT does this by sending the same CPU instructions to both CPU’s via a FT logging NIC, which is a dedicated gigabit or better ethernet NIC on your vSphere hosts.

With any software that gives you that kind of power, there are some caveats and requirements to make FT work in your environment.   I felt it was a good idea to start a blog post that I could update with the various requirements for the use of FT with vSphere.  This list is my no means all-inclusive, but simply a place where I can keep track of the needs and caveats of FT.  Read more for my listing of requirements that I’ve found thus far.

Host Requirements

CPU

• AMD Barcelona (Series 13xx, 23xx, 83xx)
• Intel Harpertown (Intel 31xx, 33xx, 52xx, 54xx, 74xx)
• Specifically a Hardware Virtualization (HV) enabled CPU
• Intel VT or AMD-V enabled in the BIOS
• Disable any power management in the BIOS (recommended)
• Disable Hyper Threading (recommended)

Network

• 2 FT Logging NIC (suggested)
• 1Gbps or better

Storage

• Shared
• Fiber Channel, iSCSI, or NAS

ESX

• Same build version of ESX on each host.
• VMware HA must be enabled on the primary and secondary hosts in the cluster.

Guest Requirements

• All ESX supported guest OS’s, 32bit or 64bit
• One (1) vCPU  - vSMP is not yet supported.
• Thin-provisioned disks are not supported (they will be converted to thick)
• Paravirtualization is not supported
• Physically attached CD-ROM, Floppy not supported
• Physical RDM’s (Raw Device Mappings) not supported - Virtual RDM is supported.

Caveats

• Storage VMotion not supported
• N-Port ID Virtualization (NPIV) not supported
• Need to have no single points of failure in any part of the environment (not required, but defeats the point of FT if your environment is not redundant).
• DRS can not be enabled on the protected VMs (You can still run manual VMotion’s)
• Hot add of of devices to the protected VMs is not supported
• Snapshots are not supported (must be deleted before protecting)
• VM Hardware must be at v7
• Remove 3rd party clustering solutions prior to enabling FT.

Anything else that I’m missing?  Let me know in the comments, and I’ll keep the above info updated.

To make the snapshot management easier, I store the VM configuration files on a separate LUN, and where you store the VM configuration files is where the snapshot deltas are created.  This lets us keep the main VMDK’s LUN’s fairly static, without worry of snapshots filling up our available space.   When looking at a specific VM today, I noticed that the data stores listed only showed the Snapshot LUN.  This had meant that there was a snapshot taken which had not been removed.  This particular VM was not currently replicating, so I knew that snapshot should not have existed.  Normal operation should show both the snapshot LUN and the VMDK LUN.

This VM:

Working VM:

When going to the Snapshot Manager, I was not able to see any snapshots on that VM.

I accessed the Datastore Browser to see if there were any delta VMDK’s on the disk; there ended up being 2 delta’s on my datastore.

I wanted to confirm that the virtual machine was indeed running off the delta disk.  To check that, I simply went to edit the settings of this virtual machine, and looked at the virtual disk object.  In this instance it was accessing the disk “exch-000002-delta.vmdk”, which was one of my delta disks.

There is a command you can run on the service console to try to remove snapshots if you are unable to with the VI Client.

/usr/bin/vmware-cmd removesnapshots

When I ran this command, I received the following error:

VMControl error -3: Invalid arguments: Virtual machine has no snapshots

Doing some research on the VMware communities website, I found a recommendation to create a new snapshot excluding the VM memory, and then removing the snapshot.  When I created a new snapshot on my virtual machine, I saw something very interesting.   I saw an additional snapshot called “Consolidate-Helper-0”

At this point I deleted all the snapshots from the VM, and waited for the process to finish.  A couple of my snapshots were pretty large, so vCenter timed out before they finished.  I waited an hour, and then confirmed they were gone by checking the virtual disk resource in the VM settings.

Disclaimer - Before doing anything like this, make sure you have adequate backups and understand that this is probably not supported by Microsoft or VMware - I take no responsibility for any damage to your systems
If you would like to grow a virtual disk the “safe” way (read: much, much slower), you can use VMware’s Converter tool, which will grow the volume for you on conversion, the downside is that it will require the VM to be shutdown during the conversion.   The steps below can significantly speed up the process, especially if you have a very large VMDK which needs to be extended.

To grow a system volume, you are going to need a 2nd virtual machine that will act as the helper VM to mount the virtual disk you would like to extend.  I will be using 2 virtual machines, both are Windows 2003 standard.

The disk on the VM that I need to extend is currently 20GB, I would like to make this 80GB.  I have already made sure that I have enough space on my VMFS formatted storage partition to support this larger VMDK.

Next, I need to shutdown the VM to extend the disk.  There are 2 different ways to extend a VMDK.  The easiest way to grow the VMDK disk is to right- click the virtual machine in the VI Client, and select “Edit Settings”

Then Click on the virtual disk object, and enter in the new size of the VMDK (in my case I would like to extend this to 80GB), then click OK.

Power on the VM to make sure the OS can see the additional space; right-click “My Computer” >> “Manage” >> “Disk Management”.  It will not be able to use the space until we tell the OS to extend the partition to fill the remaining space.  You should see something similar to the image below.

Now, shut the VM down again, and also shutdown the helper VMh, the virtual machine that you are going to use to mount and extend the disk.   Once both virtual machines are shutdown, we are going to remove the virtual disk we are growing from our VM, and attach it to our helper VM.  To do this, right click the VM we are growing and select “Edit Settings”, click the virtual disk resource, and click “Remove”.

Click the option to “Remove from Virtual Machine”, do NOT click the other option, this WILL delete your virtual disk.

Now right click our helper VM, select “Edit Settings”, then select “Add” toward the bottom of the screen.

Select “Hard Disk”, next, and select the option to use an existing virtual disk.

Next, you will need to browse for the VMDK file that we are extending.  Since we have already grown the virtual disk to 80GB you will notice that in the size column for this VMDK.

After selecting the full path to your virtual disk, click thru the last couple pages, and you will see the virtual disk added to the VM’s hardware list. After clicking OK, power-on the virtual machine, logon, and you will see the other VM’s disk connected at D:\ (or the next available drive letter).

Now, we can finally run diskpart to extend our volume.  Click “Start” >> “Run” >> type “cmd” >> type “diskpart”. You should see the diskpart command prompt.

Type “list volume” to display a list of all the attached volumes:

1

DISKPART> list volume

Volume      Ltr  Label        Fs     Type        Size     Status     Info
----------  ---  -----------  -----  ----------  -------  ---------  --------
Volume 0     C                NTFS   Partition     20 GB  Healthy    System
Volume 1     D                       CD-ROM          0 B  Healthy Volume 2
E                              NTFS   Partition     20 GB  Healthy

We want to select the system volume 2, the boot volume is the disk for our helper VM.

1
2
3

DISKPART> select volume 2

Volume 1 is the selected volume.

Then we extend:

1
2
3

DISKPART> extend

DiskPart successfully extended the volume.

If you launch disk management, you will see the volume is now a single 80GB partition.

Finally, we shutdown our helper machine, detach the disk, and reattach it to our original VM.  When you power on the original with our extended disk, you will immediately see the additional space.  You may be prompted to reboot one time, so that the operating system will be able to work with the larger re- added disk.

Enjoy your additional space!

From the VMware site:

Expanded Support for Enhanced vmxnet Adapter — This version of ESX Server includes an updated version of the VMXNET driver (VMXNET enhanced) for the following guest operating systems:

• Microsoft Windows Server 2003, Standard Edition (32-bit)
• Microsoft Windows Server 2003, Standard Edition (64-bit)
• Microsoft Windows Server 2003, Web Edition
• Microsoft Windows Small Business Server 2003
• Microsoft Windows XP Professional (32-bit)

The reason this is such a great update, is because of the support for jumbo frames within this new group of OS’s.  Previously the enhanced vmxnet driver was only supported on Windows Enterprise and Datacenter versions.  When using certain iSCSI arrays which support application consistant data snapshots, you need to install the iSCSI initiator within the virtual machine.  This is the only way for the EqualLogic toolkit to take application level snapshots of Microsoft SQL or Exchange Server. Additionally you will require a NIC which supports jumbo frames.   Previously this was done via a dirty (and unsupported) hack if you were running the non- supported OS.  Now that VMware supports (for data traffic only) jumbo frames for the standard version of Windows OS’s, this can decrease the CPU usage for the guest VM, while not having to spend 4x as much for the enterprise version of the OS.   So it’s still “technically” not supported for iSCSI traffic, but works great and can lower the guest CPU usage during high data IO operations.

Additionally, as a old school linux guy and Ubuntu fan, I’m glad they are adding support for newest version of Ubuntu desktop and server, 8.10 - Intrepid Ibex