In reply to CrushUnit:
Hi - I currently use a crop Canon body and have both the 17-40 L (been shooting with it for a year or so) and 24-105 L (which I only recently got). I've also has the 70-200 L for quite a while.
Of them all, I really love the 17-40. As you don't need a particularly fast lens at that focal range, I've not struggled with the f4 outside. Inside the low light it is more of an issue. Ratcheting up the ISO helps of course but you will start to get more noise of course and, if you are taking portraits for example, you can't produce the really shallow depth of field you sometimes want. However, that aside, this lens covers a very nice wide angle (I rarely want to go wider) to almost portrait lens range. I used this on a trip to the US last Autumn and didn't need anything else really.
If, however, you want to get in close to take shots of people from a distance or pick out bits of a landscape, or a flexible lens for portraits, you will need more. Which is ultimately why I got the 24-105. Other than at the wide end (and it isn't really wide enough on a crop for much of what I do with landscapes), it is a very useful focal range. If I am being honest, however, I am a little bit disappointed by the optical performance of the 24-105. It is slightly softer generally than the 17-40 and 70-200 (or at least my version is) but edge to edge (at least on a crop) it is pretty consistent, the distortion is fine etc. The IS is handy for indoor shooting in low light. I am still getting used to what it can do and do like it a lot so I can see why it is considered the workhorse for many full frame users (where the focal range makes a little more sense if you have only one lens).
With a crop body, you get the advantage (if you consider it that) with the L's that you don't use the edges and experience the drop off in IQ at the edges. However, the focal ranges aren't ideal and you end up buying more lenses than perhaps you could get away with. Which is your real dilemma it seems.
At the moment I have the 24-105 stuck to my camera as I get used to it. It's a very useful range as I say but, somehow, I still love the 17-40 more. If you look at my (small) gallery on here, there's a colourful one from the USA using the 17-40 which I couldn't have taken with the 24-105, just not wide enough. It depends what your shooting preferences are pehaps.
I know I am a good one to ask this, having that line up of lenses (and one or two others) with a crop, but have you considered the lenses with focal ranges perhaps more suited for crop bodies? Like the canon 17-55 f2.8 IS which seems spot on for you in many ways ? I decided I didn't need the f2.8 and have intentions to move up to full frame at some point so considered the L's an investment but, in hindsight, I might have been better off getting the 17-55 as an all-rounder if only in the medium term. As someone on here rather wisely said in an old thread, maybe buy the lens for the camera body you have now than the one you "might" get? If that's part of your thinking.
Hope my ramblings helps a little.