Agi is faster earlier on but vit is faster later on. Agi is also nice if you're poor while vit can cost a lot of money in HP/SP pots. Smiths can't flee to save their lives so I only recommend an agi build for the aspd. You'd probably really enjoy a hybrid build. If you're just going to PvE, you don't need nearly as much dex as other melee classes, so you could throw your extra points from that into agi or vit to round off your build.
But if you get any less than 99 base str we're going to have a talking to. :stir: