Every Etsy Shop is going to treat Custom Orders differently. But based on my 5+ years of experience selling on Etsy, here are my main recommendations on how to treat Custom Orders:
If you feel like a customer is asking for something strange or unreasonable, politely decline
Once in a while, you'll get someone crazy asking for some ridiculous custom order for something that has nothing to do with your Etsy Shop. They'll show you a crazy picture of some puppy jumping over a pit of fire, in neon pink and green, and ask them if you can sew it onto their favorite grandmother's blouse. But you sell black leather purses! What? How did this happen? Just politely say, "I'm so sorry but I think that another Etsy Shop would be better at creating that than I am!" Use your gut when selecting who you want to do custom orders with. If their customizations seam realistic and reasonable, and they ask in a polite way, you can move on to the next step.
If your customer is asking for a customization very similar to what you offer, you'll most likely have a good experience.
This is not always the case. Let's say you sell something on Etsy that is black. And a customer wants a custom alteration for a Dark Blue design instead. Most Etsy customers will be reasonable, and understand that there are many different shades of dark blue. But once in a while you'll get someone picky who has a very, very particular shade of dark blue in mind. Most of the time, however, customers who only want minor alterations of items you already make are a good way to start a relationship and create a custom listing.
Show your Etsy Client concepts and revisions before making the final product
This is the best way to ensure the customer is happy with their product. Show them a sketch, mock-up, or half-finished product to make sure they are happy with how it is coming out. This ensures that your customers know what's going on with the custom order process, and have an opportunity to say something. If they say "This looks great!" then they are less likely to complain once receiving the idea, because they had a chance to correct any issues before getting the final products.
Expect around a 10% dissatisfaction rate, but a 90% satisfaction rate
When you weed out your custom orders by following the steps above, most of your Etsy Customers are going to be very happy with the final results. Etsy Customers are some of the gracious customers I know when it comes to selling online. Most people love to support local, small business artists and love to be gracious with them. Even if they do not absolutely love the final product, they'll still give you a 5-star review, or often times not leave a review at all if they were unhappy, they just feel bad about leaving a negative review since you are a small business.
But what I find is that 1-2% of customers will be unhappy, and retaliate against you. Often times it has nothing to do with you and your business. Most of the time, it really is that person is generally unhappy with their life and they are retaliating against you. Don't take it personally. Just expect that some of your customers, whether they are Custom orders or not, will be unhappy.
In these cases I am prepared to offer a full refund, no need to send it back (unless you sold something crazy expensive like over $200 USD). Otherwise, I refund the order. Because I don't want a bad review. And a bad review means that my Etsy Shop's future can be potentially hurt.
I don't care if I lose some time and money, because I am thinking of the 90% to 99% of customers who are super happy with my products & services. And I am willing to deal with those 1 to 10% of customers who are unhappy in order to fulfill my dream of selling my artwork on Etsy.
Always take the high road with your customers - always treat them with respect, and don't argue with them
If a customer is unhappy, and it is not even your fault at all, just own up to it anyways, by saying something like:
"I am so sorry that you had a bad experience with my Etsy Shop. As an apology, I am fully refunding your order. I am so sorry for the inconvenience."
And leave it at that. Nine times out of 10, the customer leaves a 5-star review for your great customer service.