When GrowlMail gets the body of a message, it also loads images, even if you have turned off "Display remote images" in Mail's preferences.
The bug is known and is slated to be addressed in version 2.0.
The workaround is to select "Show a summary or received emails" in GrowlMail's preferences. (That means GrowlMail won't show the text of messages, of course.)
The issue is discussed at http://groups.google.com/group/growldiscuss/browse_thread/thread/9ea4179d02aa968a/c62811c448ae6c0d
If it matters to you, you can express that by starring the ticket: http://code.google.com/p/growl/issues/detail?id=89