Only two things really matter when picking an agency.
If you are a major marketer, the type of agencies you’ll consider all have the same basic approach and capabilities. Their processes, for better or worse, are all about the same as well. Many claim to have proprietary tools or processes, but that’s just not the case. They may have different names and different labels for what they do, but the breadth and delivery of services is essentially the same at any holding company agency or major independent. Pricing is also not different. More accurately, if there is a pricing difference, agencies are quick to match their competitors in order to win or retain a client.
If the basic capabilities are the same at most every agency, what distinguishes them? In my experience there are two factors that really matter. The first factor is the quality of the people working directly on your business. A great team at a mediocre agency will tend to do great work, and the opposite is true as well.
The second factor is the standards an agency sets for itself – standards for creativity, professionalism, and integrity. The agencies with higher standards make it harder for bad work to get out the door, and are quicker to realize when their work is falling short of what it should be.
If I had to choose a new agency as efficiently as possible, I would do three things. First, I would look at all their work. Not just their highlights or major clients, but everything they’ve done in the past 6-12 months so I could judge their overall standard of work. For example, on a big retail account there is often a lot of little stuff that has to get churned out quickly and cheaply, like tent cards or shelf talkers. Do they just jam their print ads into another format, or do they actually take a little time to design it for the environment it’s in.
Second, I would talk to each of their clients to see if they had more than the usual compliments and gripes so I could assess their standards of professionalism and integrity. No client-agency relationship is without its spats and hiccups, but I’d listen to see if problems get addressed, or if the same spats and hiccups keep recurring
Third, I would meet directly with the people who would be working on my business. I’d not only assess whether I like and trust their work, but also to sense if they are people who will champion my business back at their place. Big agencies have lots of people competing priorities and opinions, so I want someone who is going to advocate for me when I’m not there. Oh, and you can have them do some work too since most agencies are giving it away for free anyway.
The typical pitch process gets around to accomplishing these things indirectly, but generally takes a lot more money and time.
The Ryan Report 
Doug, Your comments and suggestions are definitely “on point”. I would add two specifics which fit within your observations: one, when looking over work of a candidate agency, look for the IDEAS behind what you see and discuss those with the agency: what was the context or problem/opportunity to be addressed, etc; and, two, assess the CHEMISTRY your folks have with the agency people who would be working on your business. If those two factors are in sync for you, then you have your new agency.
Those are excellent builds Wilder. The ideas behind the work show you how they will think about both their business and yours.
Nothing would be more valuable than gathering insights from marketers that worked with the agency previously–your second point. To that end, we have just launched a platform the makes this process immensely easy. Take a drive-by http://www.marmaloo.com. I eagerly await your feedback.
Cheers. Joe VR
Joe, although my first reaction was to look askance at someone peddling their wares in the comments, your platform is actually quite relevant. It looks like a Yelp for agencies. If you can get a large enough base of comments that there not just the malcontents and/or the adorers, it could be very useful. It has the potential to bring some much needed transparency to agency and client alike. Is there anything in it for the reviewer besides the venue to have an opinion voiced?