Ideally, you don't have pay the joining fee if you cancel the card without activation.
However, make sure you complete the cancellation before due date.
I was recently missold SBI Elite as LTF & was charged joining fee after card issuance. In my case, since the card cancellation happened after due date, SBI had already reported to CIBIL that I missed my payment and that resulted in my CIBIL score dropping by >70 points.