Even in our own language we sometimes have problems understanding other people’s speech. Generally the more we listen to a specific accent, the easier it is to understand.
Whenever possible, get yourself into situations where you can practice your listening skills. You could also do some systematic listening practice: exploit the internet to practice listening to accents that you have difficulty with. Keep in mind that as much as 40% of the words used in speech are not important for the understanding of the message. Try to pick out chunks of language and notice regularities in the dialects used by different speakers. Experiment with listening by turning off subtitles, or use a transcript if one is provided.