I was in prison and I had 9 books and each only had unbreaking 3, I tried to auction them and it said "you do not have enough of this item" when I tried to auction 9 at one time, but It allowed me to auction 1 book at a time, any help?
Are you sure you're doing the command correctly? Do /auc 9. Whilst holding the book, it will auction all 9 of them in your inventory.
If you're not currently an A-Prisoner or higher you won't be able to auction. If you are A-Prisoner and still can't apply make sure you did what @Kyleep said above.