Use xctool. When you compile your project on the command line it will emit the time taken to compile each file to the console. e.g:
✓ Compile MySwiftFile.swift (12067 ms)
Additionally, for Swift 1.2 at least there is a flag debug-time-function which can be passed to the Swift compiler to show you problematic functions.