The VMware Factor

One of the big takeaways from the EqualLogic sessions this morning was just how many of us are using VMware.  During the tools session, the question was asked, “How many of you are using hyper-v?” Of probably 40 people in the room, not a single person.  No one. Nada. Zilch. 

However, almost everyone in the room was using VMware.  Why does this matter?  Well, for one thing, we all have gotten different answers over time from vendors about what is best for our virtualized environment.  We are all learning the hard way the best practices for deploying iSCSI initiators (VMware says to use their initiator instead of Microsofts, etc), how to do MPIO (right now using the Vsphere MPIO option for round robin seems to be the only solution), etc. 

What is clear is that this is a moving target.  The rules are changing constantly and new tools are coming from VMware and from EqualLogic that we will need to think through.  With anything there will be tradeoffs.  What I would love to see happen is a combined effort from VMware and EqualLogic to prepare a consistent message about best practices.  Certainly documents exist today about best practices, but if they are more than two months old, are they the best resource today?

Hoping that the sessions dedicated to VMware this afternoon will answer some of these questions.  And I’d hate to be running a session in the same timeslot as the VMware ones, because I gotta feeling that the VMware sessions will be standing room only.

Dell Strategic Focus in a New Era

A few notes from Praveen Asthana’s Keynote…

Dell is focused on Open, Capable, and Affordable solutions.  Many vendors try to do one or two of these, but very few if any are able to do all three.  In the 2010s Dell hopes to lead in the Virtual Era.  Dell wants to have a holistic approach that creates efficiency across the spectrum of devices (from handhelds to clients), be open, and create a world class customer-tailored solution… meaning specialized sales people who have deep knowledge of products.  Also working towards not changing account representatives every six months.  (This is huge in my book!)

EqualLogic is currently up to 14,000 customers.  Dell claims 50& less time to manage, is 50% less expensive, and has a longer than expected useful life greater than 5 years.  EqualLogic is at the heart of Dell’s transformation. 

1) Own intellectual property
2) Software is important
3) Great channel partners
4) Virtualization changes everything
5) Continue a holistic solution focus

Dell still remains customer focused.  They want to be a solutions company but not lose focus on the customer.

Why I STILL Love Our EqualLogic SAN

IMG_0278As a follow up to the post I wrote several months back (see Why I Love our EqualLogic SAN), I wanted to talk about my experience yesterday with the Dell crew.

I had the great pleasure yesterday to meet with Jeff Sullivan (@sanpenguin) from Dell Tech Center here at Watermark.  We were invited to be part of a case study talking about our choices of technology, specifically around storage and EqualLogic.  Jeff also brought two other individuals with him, Kristin Storer and Stephen Sheppeard to produce and shoot the video.  The greatest asset that Dell has, in my opinon are the people who work there.  Jeff, Kristin, and Stephen were awesome to work with and made our time together a lot of fun and low stress.

We spent most of our time talking through all of the reasons why we love the technology choices that we have made.  As IT leaders, we often don’t take the time to reflect on the good choices we’ve made, but rather get caught up in the day-to-day firefighting and “what’s next.”  Having a camera pointed at you is a great way to make you stop and think, “If I were to do this over again, would I do anything differently?”  Honestly, around our choices with storage, I can say that we made the best choice for our organization.  The bottom line is SIMPLICITY. If you are a small business, a church, or have a limited IT staff… EqualLogic is a great way to go for storage. Even if you are a large organization who has spent millions of dollars in the past on Fibre Channel SAN solutions, you’d be doing yourself a disservice by not taking a hard look at iSCSI and EqualLogic.  When we compared our storage needs with everything out there (direct attached, cloud storage, fibre channel, etc) the numbers just put EQL in first place.  And it’s easy.  And we don’t have to go to training to use it, and it works.  Everytime.

Dell is a big company, but they are a great ministry partner for us.  We are certainly a mixed technology shop… we have Dell, Apple, VMWare, Extreme Networks, Aruba, Palo Alto, Juniper and others playing specific roles in our technology stack.  I can honestly say that Dell has been one of the most active in pursuing us, working with us, and engaging us in discussions about how their products work.

Thanks guys for a really fun day talking about technology.

P.S. We also talked a lot about how we use social media.  That being said, you can find Jeff on Twitter at @sanpenguin, myself at @watermarkgeek, and Watermark Community Church on both Facebook and Twitter

Why I Love our EqualLogic SAN

Last night the IT team at Watermark completed the addition of a new EqualLogic array into our existing SAN solution.  There are many things that I love about EqualLogic, but first and foremost is the simplicity.  Here at Watermark, we really aren’t an IT shop.  We have two full time IT people (myself included) and a part time helpdesk person.  We are currently supporting a ton of users, a large campus, Windows servers, Mac OS, Web Development, VMware ESX, etc.  The last thing I need to worry about is allocating hours of time to support our storage solution.  That’s where EqualLogic has been a huge win for us.


We purchased our first EqualLogic array one year ago from VR6 Systems (whom I HIGHLY recommend you talk to if you are looking to purchase).  It is a sixteen drive, 8 Terabyte solution meant to drive the storage for our VMware ESX implementation project.  Since that time, we have moved our Exchange 2007, Sharepoint, File Serving, Print, SQL servers, a domain controller, etc over to VMware with the data stores on the SAN.  We’ve experienced great performance with few issues at all.  A perfect result for a small-to-midsize IT team.

We have also seen our data needs grow tremendously over the past year. We began to look at cloud storage with Amazon S3 and others for some of these growing needs but at the end of the analysis, the amount of time uploading these files, storing them, downloading them again for reuse was going to become more costly over 3 years than looking at new EQL storage solution not even considering the speed benefits of local storage.

Last night we upgraded the firmware on our new array and the existing one to the latest 4.1.4 version.  No major issues on either and it took very little time.  At that point we ran the wizard provided by EqualLogic, which found the new uninitialized array.  A series of maybe 4 questions later and the new array was added to the existing group.  Just… Like… That…  It really is that easy.  Adding the new 16TB of storage to the existing pool and right away, that space was available.  Within minutes, the (now one large) array was moving data over to the new spindles.  Over the next week, it will automatically move the data to the right number of disks to give us the best performance.  We did notice last night that one of the iscsi connections from VMware had also automatically starting using the NICs on the new controller, so once again, we are starting to see the benefit of adding more gigabit connections to our data.

I love that we get to enjoy working with such great technology, especially when doing it for ministry.  It’s a blessing.

Snapshots in VMware and how to survive them.

vmwareWe survived a rather scary day with one of our main file servers on Sunday.  This server was one that we had only virtualized several weeks before and contains a large amount of critical data. I’m a huge fan of VMware and their products have transformed the way we do IT at Watermark, but yesterday was not fun.

Sunday morning I received a call that the server wasn’t responding, and on further review noticed that the server’s data store was completely out of space.  The server would start for a few minutes, but then error with “There is no more space in the redo log for servername-00002. You may be able to continue by freeing disk space on the relevant partition.”  This was the beginning of our lesson on VMware snapshots.  ***side note, we have Gold Support for VMware, which you think would be good, but no… if you want support outside the hours of 6am-6pm M-F, you need platinum support… Nice***  But I digress.

Last Saturday we had taken a snapshot which we had subsequently forgotten about.  When you take a snap in VMware, the system puts the original VMDK (virtual disk file) into a “holding pattern” and begins to write changes to a new virtual disk file, in our case the servername-00002 file.  The best practice of course, is to do a snapshot, make your changes, and then immediately delete the snapshot; at which point all of the changes will be written back into the original “holding pattern” VMDK and all is well.  Unfortunately, the system doesn’t do anything to remind you that the snap still exists if you forget to do this.  At the time of our discovery, the new 00002 file had grown to the size of 21 gigabytes and had filled up all of the available disk space.  This to me seems like something VMware should implement, a reminder that snaps are growing like crazy and about to take you out at the knees.

So our immediate course of action was to keep the server stopped (it wouldn’t run for more than a few minutes anyway before falling over), and get a complete copy of our file system from the SAN as a backup.  After copying nearly 60 gig from the SAN to a different location, it was time to attempt removal of the snapshot.

We went into Virtual Center, under Snapshots, and snapshot manager and saw the snapshot from last Saturday that we wanted to remove, and promptly removed it.  The task started and then hung at 95% for about fifteen minutes, at which we received a message that the “Operation Timed Out.”

Now, here is the kicker. You would think that a message like that would be a prompting to try again, but after lots of research it appears that the process has not really timed out at all.  Because the “tracking changes” VMDK is so large, it has lots of data to roll back into the original.  So in reality, the process is still running in the background and you just need to give it time to finish.  In fact, many people have said that reissuing the “remove snapshot” command will in-fact kill your data.  Not good VMware.

Fortunately, we found this information out before trying to remove the snap again.  Surely enough, two hours later, the process finished running and we were back to our original VMDK file.  The server started up with plenty of storage available again and all is well.

Like I said earlier, I absolutely love VMware, but I will not be using the snapshot capability in the future.  I think I will stick with EqualLogic snapshots, which seem to be faster AND safer.

I’d love to hear your comments on what we did right/wrong and how VMware has worked for you.