iPhoto 5.01 has several performance issues. Adjusting thumbnail and photo display sizes seems to help some people. I was surprised, however, to read that one of these might work in iPhoto 4 (requires plist editing):
There was another hint which mentioned keyboard shortcuts to change the size of the thumbnail view.
If you press 0,1,2 you will see different sizes of thumbnails.
0 - Minimum size
1 - Maximum Size
2 - 'Natural' thumbnail size
So pressing 2 should give you the same size as this tip without having to edit a .plist. This definitely speeds up iPhoto and worked well in iPhoto 4 (but had to be done by editing the plist).