Turns out there's an official way (with one catch) that works in 10.4 and 10.5:
Mac 101: Remove unwanted System Preference panes - The Unofficial Apple Weblog (TUAW) To uninstall System Preference panes, just right-click (or control click if you have a one button mouse) on the preference pane icon and select 'remove x preference pane.'The catch is, you need to be logged in as an administrative user! If you're a non-admin user you don't see anything; there's not even a "gray menu" (tested in 10.4). That's why I never noticed it -- I don't run as an admin user. I'm used to OS X asking for a privilege escalation for admin tasks.
So this is not well implemented in 10.4. It should be a menu item for non-admin users, and ask for a un/pw when it's selected. I wonder if Apple fixed that in 10.5?