Yeah, I definitely agree with you about Spike trying to goad Buffy in to killing him in that scene. I think he thought of the worst thing he could possibly imagine that might prompt her to do so, & the fact that it was raping young girls, means that he thinks that's the worst thing he could possibly do. I think the chances of him actually having done so are less than slim-to-none; it just doesn't fit with what we know of his character.

I by no means think he has a healthy relationship with sex & violence (as evidenced by most of the above, not to mention his relationship with Drusilla), but I definitely think rape would be crossing a line for him. And outside of that moment, when it's obvious Spike's motivating factor there is to get Buffy to kill him, there's never been any indication of his having crossed that line before.