A cardigan is a the perfect addition to complete a fall or winter business casual outfit. Pairing a navy blue cardigan and tie can make the outfit feel more professional, and add some personality. If you do choose to wear this outfit, personalize it further by rolling up the sleeves. This outfit looks great with a watch. Leather watches, especially in a work environment can influence how you are perceived in a very positive way. Matching the leather of the watch, belt and shoes makes this outfit work very well.
This is a basic idea and I don't feel comfortable making blanket rules for everyone. It's a very personal thing for everyone and what makes you the best guy is what fits and what fits you. If you think a cardigan is going to do you good, then wear it.
If you are a casual guy who feels that a cardigan is a little too much, it could be that you're more comfortable in the shirt you're wearing. If that is the case, then don't wear anything. In fact, if you feel that nothing works for you then do wear something that does. Just make sure you're comfortable with the shirt you're wearing.
If you're more comfortable in a long sleeve shirt that looks good on you then I would recommend wearing that if you have it in your wardrobe. But in general, if you find it too casual, then I would recommend a long sleeve shirt on you. However, if you are comfortable with an all over cardigan, I would recommend something more formal like a suit or a t-shirt