A passport is a wonderful privilege that opens up the opportunity for international travel — but it’s very much a privilege, not a right. That means it can be taken away from you, which could happen if you don’t pay your taxes.

According to the Internal Revenue Service (IRS), “Generally, the State Department will not issue passports to taxpayers after receiving their delinquent debt certification from the IRS. The State Department may also deny a taxpayer’s passport application or revoke their current passport.”

Of course, there are many stipulations here — being behind on taxes doesn’t outright ban you from international travel. “It does not happen overnight, and taxpayers need to realize that,” Morris Armstrong, an enrolled agent licensed by the Treasury, tells Travel + Leisure. “The IRS must have filed a Notice of Intent to Levy, and the case worked its way through the administrative system. If you have filed your most recent tax return, and there is a large balance due, you are a year away from the threat of revocation.”

But first, we need to back up even further. Diving into the law that allows the IRS and State Department to revoke an existing passport or deny a passport application, a person must have “seriously delinquent tax debt.” So, what does that mean? “The threshold for the debt is $62,000 and that includes interest and penalties,” says Amstrong. (That number changes annually due to inflation, however, so check the specifics with the IRS if you’re concerned about your debt.)

It’s not a black-and-white number though, as there are exceptions.

“Fortunately, the IRS does not include in the definition of serious delinquent debt the following: child support, people paying through an IRS-approved installment agreement, those taxpayers who have had an offer in compromise accepted, debts where a Collection Due Process hearing is scheduled (this is a step taxpayers can take after receiving the Notice of Intent to Levy), and those taxpayers who have filed for innocent spouse relief,” says Armstrong.

Armstrong also notes that “the IRS will not certify to the State Department any debt that is currently non-collectible,” whether due to hardship, identity theft, bankruptcy, or living in a federally declared disaster area. Plus, “military personnel serving in a combat zone are also not subject to certification,” he adds.

So, in short, it’s actually somewhat difficult to meet the qualifications to have your passport revoked. “As you can see, you can still have debt with the IRS, but you must be in an active payment plan or similar arrangement,” says Armstrong. Once you work out some sort of accepted plan with the IRS, your passport should be fine.

It’s also important to note that this is all related to federal tax, not state tax. “Since passports are issued by the federal government, owing state back taxes does not affect your ability to obtain, use, or renew your passport,” certified public accountant Logan Allec, owner of tax relief company Choice Tax Relief, tells T+L.“However, since states are in charge of other things, such as driver’s licenses and professional licenses, owing state back taxes can result in those state-granted privileges being revoked.”

Can your passport be revoked if you don’t pay taxes?

Yes, if you have “seriously delinquent tax debt,” your passport can be revoked — and that can even apply if you’re out of the country. Since modern passports have electronic components, your passport will be updated to become invalid, barring you from any international travel. “The IRS is not without heart, though, and a one-time passport can be issued to allow your re-entry to the U.S.,” says Armstrong.

Can you be denied a new passport if you haven’t paid taxes?

Yes — but again, you must have “seriously delinquent tax debt.” “If you attempt to apply for or renew your passport after your tax debt has been certified to the State Department, the State Department will send you a letter stating that you are ineligible to receive passport services at this time,” says Allec. “However, they give you a 90-day grace period in which to reach a resolution with the IRS concerning your tax debt, after which you can resume your application.”

Note: Travel + Leisure recommends readers talk to legal or tax experts for more detailed answers when it concerns their own situations.