For a variety of reasons, a taxpayer may believe that the IRS penalty is unreasonable. But what grounds of defence would be valid?

For a variety of reasons, a taxpayer may believe that the IRS penalty is unreasonable. But what grounds of defence would be valid?