Whenever I try to click on my character's inventory while playing this quest, the game freezes on me!

That happened to me too a couple of times, but I was finally able to play by starting the game and quickly leaving the town. I tried to open the inventory in Ardyl and it was working fine so I ran back to town and it didn't crash after that! The quest is awesome and requires a lot of patience!  I hope this helps!!