If your audiences are too small, you won't serve many impressions.
A good rule of thumb is that Retargeting Audiences need to contain at least 1,000 unique people to serve meaningful impressions on a consistent basis. 1,000 people might seem like a lot, but it's not. On any given day, there are people who won't do a whole lot on the internet, making it hard to serve them ads. Also, the people in your list are in OTHER companies' marketing Audiences as well and you might have to compete for their impressions.
The people in your Audience may not be loading Facebook or web pages in countries that have a lot of available real-time ad inventory. For instance, there are more than 5 billion daily ad impressions available for retargeters in the United States. In other parts of the world that number could be much lower. If your audience is primarily outside the U.S., retargeting can get trickier.
Finally, make sure that if you are excluding Retargeting Audiences in your campaign settings, that you are not excluding so many people that your campaigns won't target a meaningful number. That can happen, too. If you "include" a segment with 50,000 people, but "exclude" a segment with 49,500 people, your campaign won't serve very many impressions.
To check the size of your Retargeting Audiences, go to the Manage-> Audiences section of your dashboard and look at the numbers in the "30d Size" column in the table.
To increase the size of your Audiences, let them build over time as more people visit your web site or buy traffic from services like Google Adwords.