My New Gig: Panzura

This week I am giddy as a kid in a sweet shop.  Not just any sweet shop, but a Willy Wonka style emporium of sweet with unlimited possibilities and the landlord telling me that there was going to be a lock-in that night and I’m invited. That giddy.

I have recently taken a role with Panzura as the EMEA Solutions Architect and will be working closely with their Cloud Service Providers to promote Panzura, develop joint content, be a technical liaison for their sales and technical teams and generally spread the good word throughout Europe, Middle East and Africa in my usual inimitable way.

Simply put, Panzura works with a number of Cloud Storage Providers (EMC Atmos, HP Openstack, Google Cloud, Amazon AWS, Cleversafe, Nirvanix and HDS) as a Cloud Storage Controller to deliver cost effective  storage, for file services or archival purposes and most importantly, to me at least, as a Globally Distributed Filesystem.

And that’s what began to get me excited.  The ability to create a global unified namespace, dispersed geographically around the world with global deduplication, global-write and file-locking.  There are a million different use cases for this type of technology and even though I have only started to learn the intricacies of the product, it’s more than enough to know that this is a very appealing technology to an awful lot of people.

Panzura scales out infinitely through the implementation of a Quicksilver Appliance at each location and connects seamlessly into the public or private Cloud, comfortably delivering PetaBytes and even ExaBytes of data thanks to PanzuraOS and it’s 128-bit Panzura file system, CloudFS.  The benefits to the customer are immediately obvious, aside from the standard cloud messaging… You have an appliance that allows a true, global centralisation of your file servers, and one that delivers a global unified namespace with a familiar tree infrastructure for an easy user adoption.  The huge advantage comes when you realise that all of the files are being stored as objects in the cloud and being deduplicated on the way up there.

Your users also get LAN speeds for file access and directory browsing thanks to Panzura decoupling the metadata from the file itself, and storing in tabular form on the Quicksilver appliance in every location.  That’s huge (and really not that easy to do!). It’s bad enough waiting for a local file structure to open because the directory size is massive, but if you’re pulling it from the other side of the world, that could become impossible to cope with and a major reason that more organisations haven’t attempted a truly centralised NAS infrastructure.

Without wanting this blog post to become too long or extolling every single virtue that Panzura has, I’ll save some of the other abilities for future posts and you’ll have to check back with me, or see me at one of the shows that we all end up at from time to time, to hear the rest of the story.

As for me, I’m going to go and carry on drilling down into this superb technology.

Flash Storage Jargon Explained – Part 3 – Wear Levelling

In Part 2 I looked at Garbage Collection and the part that it plays in Flash Storage, and Wear Levelling is a key link between the two processes.

Wear Levelling is the process by which the drive itself (in an individual SSD) or the operating system of the array manages the write process as data is committed to the cells.  The overall purpose of Wear Levelling is to ensure that the Write Cliff is never reached by ensuring that the data is spread evenly over the cells in the array over the lifespan of the array.

Wear Levelling can be managed in 2 forms, Static and Dynamic.

Static

Static Wear Levelling can be best described as the process of moving immovable cells.  Operating systems will create static blocks which it commits to disk, and manages the location of, through the LBA (Logical Block Address) Table.  Since these blocks are seldom re-written, they tend to hit a cell once and stay there, effectively holding the wear count at a low level.  This isn’t a problem, so much as it is hugely inefficient and it will dramatically shorten the life of your array because you are reducing the number of cells that can be written to.

Static Wear Levelling references the LBA Table hunting for static blocks and moves them to cells with higher wear counts, effectively freeing up the cell with a lower count into the write lifecycle of the array.

Dynamic

Dynamic Wear Levelling is the process whereby rapidly changing data is updated in the LBA Table as invalid and subsequently written to a different cell to which it was originally assigned.  The intelligence comes in arrays that manage this dynamic process by retaining information on the cell write count for the drives in the array.  Naturally, it is impossible to retain the count on all cells, so a representative sample is maintained.

How Does It Affect Me?

This process is integral for extending and maintaining the life of your array.  Put in mathematical terms, it can be displayed below.

Assuming that an MLC drive will have a lifespan of 10,000 write/erase cycles, and there are 81,920 blocks of 2MB each in a 160GB SSD Drive

160GB x 1024 = 163,840MB

163,840 / 2MB = 81,920 Blocks

Approximately 20% of an unmanaged SSD drive is reserved for systems management, buffer space for wear levelling and garbage collection and other system processes, so factoring that in, we have 65,536 Blocks available for the write process.

If we take those 65,536 Blocks and work on the basis of the write process being 15% efficient and that we write to each individual block twice every second then we are left with the following calculation.

65,536 Blocks x 10,000 Writes x 15% Efficiency = 98,304,000 Total Writes

98,304,000 / (2x 60 x 60 x 24 x 365) =  1.56 Years

Which is fairly short term in the scheme of IT purchases.  I have seen evidence of this, and heard plenty of stories in the wild of this happening, and the figure of 10,000 writes depends on your drive lithography and manufacturer, as all drives are certainly not created equal.

If we then compare this to a fully managed array, and assuming that the wear levelling efficiency isn’t perfect and operates at 80% we see the calculation change

65,536 Blocks x 10,000 Writes x 80% Efficiency = 524,288,000 Total Writes

524,288,000 / (2x 60 x 60 x 24 x 365) =  8.31 Years

This is largely down to the fact that you are using software to manage this process, and also the compute power in an array is far far in excess of that of a single, unmanaged SSD drive.  When you combine other software managed aspects like Block Alignment and TRIM (The subject of Part 4), then you will find that the lifespan of your SSD array will far exceed 8.31 years.

Flash Storage Jargon Explained – Part 2 – Garbage Collection

Following on from the previous post in this series, where I discussed The Write Cliff, this post sets out to elaborate further on Garbage Collection, which is linked closely to the write cliff.

Garbage Collection is the process by which the SSD management layer works across the array/drive to identify, collate and redistribute the data from partially filled cells, delete invalid data (data due for deletion/stale data) so that a block can be reclaimed in full and also provide a degree of wear levelling (Explained in Part 3) across the drive.  It is handily explained by this diagram from Wikipedia

GC

 

Garbage Collection is a vital process in any SSD or Flash Array, as write performance will be impacted if too many partially filled cells exist, or there are no flashed or totally empty cells.  This is due to the way in which Flash Cells are written to, in their Read-Flash-Write process, where any existing data that is to be retained has to be buffered while the block is erased and then written back.

When Will This Affect You?

Garbage Collection becomes an issue when the array/SSD gets to around 85% full.  It’s important to pay attention to the best practices of the vendor, as they will have slightly different ways of designing a solution.  For example, WHIPTAIL recommend never going above 80% capacity on the array, or GC can start to make an impact.

What Can You Do About It?

Simply, you can allow the GC process to complete and resume “normal performance”, but it is worthwhile ensuring that you are below the upper capacity level set out by the vendor.  If you are permanently running near the GC limit, and it is unavoidable, I would suggest working with the vendor’s support team to schedule a full GC pass on a regular maintenance schedule.  They can be created as a cron job to run at a low impact time and will keep the array running smoothly.

Flash Storage Jargon Explained – Part 1 – The Write Cliff

The Write Cliff

Having spent almost 2 years working in the SSD/Flash Storage market, I have had a number of people ask the same questions over and over regarding the technologies involved and some FUD that floats around the market from time to time.  Over the course of this series (I have no idea how long it’ll be) I’ll aim to explain what the terms mean, whether they are myth or reality and how the current Flash Vendors avoid those that actually exist.

The first term to approach is The Write Cliff.  This is the phenomenon where the performance of an SSD drive drops dramatically, and suddenly over time, as illustrated by the representative chart below.

Image

The Write Cliff is not a myth; it exists but is largely a defunct issue in most SSD storage arrays.  It occurs when all of the cells within an individual drive have been written to at least once.  This is down to the way that SSD’s are written to, undergoing a Read-Erase-Write process where any incoming data is buffered until the cell it is destined to be written to has been “flashed” and the new data (along with any data being retained if the page is partially filled) re-written back to the cell.  It is important to clarify that while an individual cell can be written to, an entire page must be deleted.

This degradation in write performance, and often overall performance (depending on your queue depth) is most commonly seen in unmanaged SSDs once they’ve been in situ for a prolonged period of time (in my experience it can be anywhere from 9 to 18 months).

Modern arrays such as WHIPTAIL, Violin Memory et al. have algorithms in place to manage this process, ensuring that an (often significant) proportion of the reserved array space is held for Garbage Collection (GC) (the process by which an array is “tidied up” and the subject of another post), and that the Garbage Collection process is running almost continuously as a background process.

So how does this affect you, the consumer?

In reality, it doesn’t, IF you are purchasing a recognised SSD Array/PCIE Flash Solution and have sized it according to the vendors specifications.  But it is often thrown out there by these vendors in competitive situations as FUD in the form of disparaging the performance of the competitors array over its lifespan.

It may happen, and subsequently affect performance, if you fill your array to 90%+ of its capacity and the Garbage Collection process is having to work harder and more frequently than expected.  In this situation, you will see an increase in the CPU and Memory usage within the array and a drop in the write and read performance until the process has completed.  It is often wise to schedule maintenance time for the GC process to be run in it’s entirety (The best way to do this is to arrange it with the vendor’s support team, as they will often have a script to initiate a full GC run) as you’ll extend the life of the array and maintain the sparkling performance that you purchased it for!

It will most certainly occur if you decide that an appropriate solution is to simply purchase a number of SSDs and pop them into your server, SAN and see how it goes.

It’d be great to hear any real life stories involving falling off the edge of the Write Cliff.

8 Business Travel Tips & Tricks

I have travelled for work a lot over the last 4 years, both in Europe and in the USA.  I will regularly do a small tour of a region, travelling to multiple different cities inside a short period of time, which often makes packing much more difficult.  Over the last 4 years, I have pretty much refined my business travel routine, so I thought that I would share some interesting techniques that I have picked up.

Compression Sacs 

There are several choices for compression bags here, but I generally go with Eagle Creek Compression Sacs because they have a meshed technology that makes them a lot more robust than the cheaper bags you get in any British supermarket.  The last thing you want is for your compression sac to split while you’re away and be left with an impossible situation of trying to get your clothes to fit into your carry on.

For EMEA travels, where I am only taking carry on luggage, I will take 1 medium and 2 small bags and the space savings you gain are absolutely brilliant.  There is some wrinkling and creasing of your shirts, non-suit trousers but with most hotels having irons/ironing boards (or an Ironing Room in a lot of Scandinavian hotels) and combined with Tip 2, it rarely poses a problem.

For longer term, US travels, I tend to take 2 larger bags in my main suitcase along with 2 smaller bags, and then a medium bag in my carry on luggage.  I find this a useful arrangement, as I can separate my casual weekend clothes into one sac and my work attire into the other.

Steaming

Steam is a good thing.  Despite hotel rooms often having their thermostats set to volcano, there are few ways to effectively get creases out of your clothes, especially when you have spent the day travelling in them.  A suit, or sports jacket for example.  Simply shut all of the doors (and windows if your lucky enough to have them) in the bathroom, turn the hot tap and shower onto full heat (make sure the plugs are removed!) and hang your shirts/jackets/trousers from the highest point you can.  be careful not to hang it from the smoke detector though… They’re far more sensitive than they look!

Spare Shirt/Underwear/Socks In Laptop Bag

I used to travel to St Petersburg in Russia a few times a year, and the airline I travelled with was notoriously bad for losing luggage.  Since it happened to me twice consecutively, I decided to implement this as a contingency plan.

I chose my laptop bag with this in mind.  I tend to prefer the rucksack style laptop bags because I have a silly amount of hardware that I lug around with me on my travels (more on those later) and the extra pockets are useful, especially for this purpose.  I keep the spare clothes hidden away in a centre compartment so that there is no danger of my superhero boxer shorts falling out when I’m grubbing around in my laptop bag for a particular dongle, cable or whiteboard marker.

AppleTV

This isn’t so much use when I’m doing a rapid, multi-city trip, as I’m never really in a hotel for long enough to set it up or use it.  The AppleTV is much more useful for longer trips where I am in the same place for the best part of a week or more.

Most hotel rooms now have TV’s with HDMI output ports, or in some of the better hotels, they have full AV panels in the wall.  The AppleTV takes very little time to set up, and as a device is a mere 10cmx10cmx2cm so takes up very little space in the laptop bag.  I understand that a lot of people wouldn’t be impressed by the wireless speeds in hotels (I’m regularly incensed by how bad they are), I tend to use the AppleTV for its Airplay Mirroring capabilities.  This means that I can display my iPad Mini, iPhone, Macbook Air onto the TV in the hotel and either work, or more likely watch a movie or TV show on a sensible screen size and in much more comfort.  I also have another trick in this area as well.

Netflix Subscription

I also have a NHL Gamecenter subscription, but that’s because I’m a hockey nut.  I chose Netflix over Lovefilm, or any of the other media providers as they cover an awful lot of countries and I prefer watching entire TV series, rather than movies.  I find this particularly useful in Non-English speaking countries where the English TV channels are limited at best, and also in those that only have Freeview.

Hotels vs Suites

This is a huge one for me as I’ve recently hit my 30’s and 4 years of travelling are starting to take their toll on my waistline!  This particular tip applies more to America than it does to Europe.

When I am in the US, I generally prefer to use the Suite style hotels to a standard hotel room.  The prices are often comparable and the extra space makes it feel a lot more like home, even for a couple of nights.  The biggest bonus for me is the kitchenette that often comes with them.  With portion sizes being somewhat larger than those I’m used to in England, it is often a great way to help your waistline to have a decent lunch and take the leftovers home with you (or vice versa) and have them for dinner.  I find that this fills me more than adequately, and if you don’t fancy eating out, the Suite hotels generally tend to be close to a supermarket or 7-11 at the very worst that you can pick up simple, healthy food for any meals.

Loyalty Programmes

There is a lot of debate over whether you should join one or two loyalty schemes and remain loyal to them, or whether you should join them all and take whatever option is cheapest.  My personal opinion is that you should have 2 tiers of loyalty scheme in your possession.

Tier 1 – Should be the programmes that you most regularly use, and I would advise getting one for hotels and one for airlines.  On the airlines, there are really only 2 alliances, OneWorld (BA, American etc) and Star Alliance (Lufthansa, Delta etc) and one tip I’d give here is to look at all of the airlines that fly their own flights from your local airport and compare the levels required to achieve status on those airlines.

For example; because BA has decided to align themselves with the Avios points scheme, it is much harder to achieve OneWorld status with BA than it is with American Airlines.  Unless you have an Avios credit card, it seems pointless to sign up for BA and struggle towards status.

For hotels, it is a little harder and depends on whether you get to keep the points that you earn.  It also is worth weighing up the benefits you get once you achieve that status.

Personally I use Hilton as their status members get Free Breakfasts, Free Wireless Access, Zero Blackout Dates (Very useful for last minute meetings or overnight flight delays when the airline is being recalcitrant on where they want to put you!) and Free Room Upgrades.  All of these combined can save you £20-£50 a night when you book, depending where you are in the world.

Tier 2 – This should be the secondary alliance from your local airport, and often used for price comparisons on routes.  If you can gain a couple of status levels that’s an added bonus as I find that lounge access in whatever airport you’re in is invaluable (unless it’s such a regional airport that it doesn’t have a lounge!)

As far as hotels goes, it’s often a good trick to find a regionalised hotel chain in a country that you spend a lot of time in.  I was spending a lot of time in Scandinavia, so Radisson was a natural choice for me and I gained a very large number of points very quickly.

If you do get to keep the points, and earn enough points to be useful, I’d recommend looking at the large international chains with holiday resorts.  I’ve had several holidays where I’ve not had to pay for anything besides incidentals through my business travelling.

 Tripit – Travel Application

If you are travelling to multiple cites in a short space of time, then the itineraries can be ridiculously complicated.  Tripit is a fantastic application for managing your travel plans.  You simply forward your booking confirmation to their email address and they update your application remotely with all of your travel plans, linked together and in chronological order.

Whether you upgrade to Tripit Pro or not is up to you.  I found it useful for the loyalty points tracker, but with them dropping a lot of schemes, it has become less valuable.

I hope that you found some of these tips and tricks useful, and if you have any more, please feel free to leave them in the comments below!

Welcome to TheFlashStorageGuy Blog

Welcome!

Thanks for taking the time to hunt out and read my blog, or thanks for responding to my shameless plug on Twitter/LinkedIN/Facebook/Anything Else. Over my life in the Virtualisation and Tech Communities I have met a wealth of interesting people, read countless interesting and useful blog posts and experienced a number of interesting events, situations and occurrences that may even make worthy blog posts themselves.

So, in homage to those interesting and often entertaining people and events, I have decided to create this blog. It’s primary aim is not to be a mine of technical, deep dive information on a narrow subject, nor a how to guide on day to day admin tips and tricks. There are an infinite number of more qualified people out there to deliver that content (although I will definitely link to it where it helps my posts), but more as a conceptual, discussion-based blog looking at cool technologies, interesting ideas and if they’ll let me, some Q&A with industry titans.

Who knows, it may even be amusing on occasion.

Either way; I hope that you enjoy it, find it useful and that I remember to keep it up to date.

Cheers

Mike