I don't think he's afraid of the Hammerites. I think he came here for something else. Something else that's beneath the temple, and the reason the temple was built. I wonder if it has anything to do with that message I read under his mansion, something about a compass.