Social Security still makes up a large chunk of the retirement income pie (about 35% for the average American). It may come as a surprise to some that up to 85% of a person’s benefit could indeed be added to your taxable income. In fact, about 40% of Americans pay taxes on their Social Security benefit. So how do you determine if your Social Security payments are taxable?

In general, you’ll only have to pay federal income taxes on your Social Security benefits if you have a substantial amount of income from other sources, such as wages from a job, taxable pension benefits, distributions from qualified retirement plans, income from a business you own, or investment/dividend income.

When determining how much of your Social Security income may be subject to income tax, the IRS uses your “combined income,” which consists of your adjusted gross income (AGI), nontaxable interest, and half of your Social Security benefits.

For individual (single) tax filers, if your combined income is between $25,000 and $34,000, 50% of your Social Security benefits may be subject to income tax. If your combined income is more than $34,000, up to 85% of your benefits could be taxable.

For joint filers, the income thresholds increase to a range of $32,000 to $44,000 for 50% taxation, and $44,000 and above for 85% taxation. Under no circumstances is more than 85% of your Social Security benefit subject to taxation, no matter how much income you have.

Social Security income might also be taxed at the state level. An analysis of state taxation of retirement benefits by the tax and accounting firm Wolters Kluwer found that 14 states tax Social Security payments for some retirees, including Colorado, Connecticut, Iowa, Kansas, Minnesota, Missouri, Montana, Nebraska, New Mexico, New Jersey, North Dakota, Rhode Island, Vermont and West Virginia.

Keeping in mind that your Social Security might be taxable, it’s smart to undergo a little bit of planning when it comes to generating your retirement income to potentially minimize the impact.

If Social Security is your main source of retirement income, chances are you won’t have to worry about its taxability. However, if you draw additional funds from you investments to supplement your retirement income, the probability of your benefit being taxed increases.

A key strategy that provides the needed flexibility for tax planning when it comes to Social Security benefits is setting yourself up prior to retirement so that you have diversified sources of income from a tax standpoint (i.e. having money stashed in taxable, tax-deferred and tax free accounts). If you can decrease your adjusted gross income by tapping taxable and tax free accounts, you may be able to control just how much, if any, of your Social Security benefit is taxed. The tricky part comes when you have to start taking required minimum distributions from your tax deferred accounts, as this may automatically boost your adjusted gross income.

Thus, another strategy to minimize Social Security taxes may be to draw down your pretax 401(k) and IRA balances before signing up for Social Security, which will get you higher Social Security payments due to delayed claiming and lower or no taxes on the benefit. The thinking is this money will have to come out at some point, so you could take your 401(k) or IRA funds out first and then start taking Social Security.

In short, knowing your options and being armed with different resources to tap to minimize the taxability of your Social Security payout may help to keep Uncle Sam out of your pocket.