When people tell us that they want to see the Prague Castle and the Charles Bridge, we always recommend visiting these two marvelous sights either early in the morning or later in the evening to beat the crowds. Alternatively, you can visit the Castle during the day to enter the St. Vitus Cathedral and have a stroll in the Castle gardens (which is highly recommended), and then come back in the evening to truly enjoy the Castle's exteriors.
To illustrate what we mean, we took a short walk to the Castle yesterday (our kitchen is located within walking distance of the Castle) and took these photos for you. These shots were snapped at about 9:00 p.m. There was hardly anyone there, except for the attendants of a BMW party (yes, you can rent the halls of the Prague Castle for corporate events).