Firstly, install Subversion (currently 1.4.4) from downloads.open.collab.net - this version is compiled with all the language bindings, so you don't have to mess about with building SWIG by yourself. The `git-svn` integration depends on having the Perl bindings available.
Next, I downloaded and built git 126.96.36.199 by hand. Unless you have the Expat XML parser installed, you'll want to define the `NO_EXPAT` environment variable to prevent the part of git that depends on it failing to compile.
Finally, and this is the trick, you need to add `/usr/local/lib/svn-perl` to `$PERL5LIB` so that `git-svn` can find the Perl bindings to Subversion. If you're getting errors about "can't find SVN/Core.pm" or some such, this is what you're missing.