The question of performance and what can benefit from additional work optimizing code (or art) is a very complex problem. To grossly oversimplify the questions of performance, it is worth looking at a few of the terms used to describe what involved in the actual measure of performance. Usually when talking with people about a game’s performance in simplistic terms they will focus on fps (frames-per-second) but there are many other considerations memory usage, texture memory usage, cpu usage, gpu usage, network usage (if applicable) or things like load time, amount of loads, and more. To measure any of these it is worth focusing on testing different types of components, and the platform and/or engine and getting minimum specs for what you need (size of art assets, number of triangles, animations, size of textures, number of lights, number of postprocess fx, complexity of shaders, etc.), the list goes on and on.
One thing that is often talked about CPU bound vs GPU bound, meaning that one of these systems is a bottleneck for the other.
CPU bound could be for a lot of processing, physics calculations (on the CPU) or a lot of other complex calculations (searching or sorting large amounts of data or complex AI calculations like pathfinding).
GPU bound if a lot of polygons, high resolution textures, or dealing with a lot complex shaders (often with multiple lights and shadows).
The best approach is to find the first largest bottleneck and try to minimize it, then the next largest and so on. While it is sometimes worthwhile to look at the code and try and optimize it (things like replacing divides, multiplications ect. with bitshifts and attempting to precalculate as much as possible before actually hitting runtime), this won’t solve the problem if the game falls over from huge art assets that the hardware does not easily support. It is often considered that premature optimization is a huge mistake, since it is often difficult to know if you are optimizing the right part of the code (and often the code becomes significantly more complex and more difficult to maintain if it is optimized). It is still worthwhile to make as efficient code as possible while writing it (considering order of magnitude algorthims (big O notation)), but if you don’t actually test with real data, you may be wasting huge amounts of time optomizing the wrong thing.
In terms of performance and optimizations, I almost always come back to an example that a colleague of mine gave me that he saw on a previous project. The project was on a 3D handheld and the team wanted to add a depth of field postprocess render fx to the game, but were worried that the performance hit would be too expensive. They decided that the look was so cool, they had to try it and see what it would cost. What they found was that by adding this fx they actually got better performance! On this particular 3D handheld both the CPU and GPU shared the same bus, so by adding this postprocess fx the bus was freed up to handle a lot of CPU processing, while the GPU was dealing with this more complex rendering. Not an easy thing to estimate or plan for 😛
There are a few useful tools in helping debug performance, PerfStudio for AMD http://developer.amd.com/gpu/PerfStudio/Pages/default.aspx and PerfHUD for NVIDIA http://developer.nvidia.com/object/nvperfhud_home.html are great tools for seeing the low level API calls to the graphics card as well as debugging tools for texture visualization and overrides which are great for shader debugging, checking mipmap level etc.
One of the things that really impressed me in the UDK was the use of performance and memory built right into the editor, as well as things like mipmap levels, shader complexity, that are great ways of helping artists debug scenes and texture sizes. Unity3D offers a few tools (like the stats window and profiler) to help see where some of the memory is being used, but I haven’t seen it at the level of UDK anywhere (and will still need to look at XNA, but without an editor, it may be difficult to provide any of these kinds of tools).
In the end, it all comes down to trade offs, and what you can live with for performance for your game, usually at the expense of something else, and hopefully not too much of the visual look. It is worthwhile to develop some tests that check for significant changes in memory usage, video memory usage, frame rate, etc. on every commit (or at least everyday) to help catch these memory issues as soon as possible (especially on a large project).
Best of luck optimizing,