There is no such thing as a bad incoming link. Google operates on the idea that you have control over who you link to, but not who links to you. So even sites with PR's of 0 linking to you won't hurt your PR. However, if you link out to such sites, then you're hurting your Google reputation (PageRank). Try to do effective link exchanges and be very mindful of both your incoming and outgoing links for quality. In all likelihood, the PR calculation never reaches zero, so even the worst of links coming in to you will count for something positive to your pagerank.

I recommend adoing 3-way link exchanges. These can be more difficult to set up, but you're protecting yourself when Google does away with counting traded links for PR. *I am of the opinion that this already takes place to some degree.*

Time also seems to be a factor in your PR. The older and more established your site, the more likely you are to have good PR. You can't control the time factor, but be ever mindful to not drastically restructure your site or move around directories. If you absolutely must move directories or domains, be sure to 301 redirect the pages to their new locations.

The structure of your site is also contributory. Try to have pages no more than 2 directories deep, with each and every page linking back to the front page in some fashion.

Last, if you're not submitting your site to Google via a sitemap, I recommend you get setup to do so. At a minimum, you're letting Google know which of your pages you deem as most significant.