You’re spot on. There’s a mental image I’ve read (and I’m going to butcher this because I don’t remember it clearly) of a moral gradient (I’m 60% sure that’s what it’s called). The way I fit this concept in my head is that we each have a sort of a topographical map of something hill-shaped. This map represents the moral weight we put on others and ourselves. We’re at the center, and the points furthest away are those strangers from far away that we will never meet in our lives. Different areas may represent different groups of people. Even better, instead of a topographic map, you’re forming it with a finite amount of playdough. Some people might have more or less playdough than others, but they’re usually pretty similar.
Say one person’s hill is shaped like Grinch mountain (incredibly steep) he holds himself in far higher esteem than anyone else, even those very close to him. Someone whose map is a flat plain (or plateau, but it’s hard to tell, isn’t it?) is messiah-like in her even-handedness with humanity; every person is as valuable to her as her mother. Both of these are very different from normal, and we like to see other people be normal. Sometimes quirkiness is acceptable, and I suppose we might admire someone whose map looks like a ridge representing her strong devotion to the children of an African village she visits every year as well as animals of all sorts.
When I compare my moral gradient to the people around me, I notice some important differences. I put a much higher weight on strangers and foreigners. I still probably put a lower weight on the poor than a typical democrat. But that effect is swamped when you account for how much more I care about strangers and foreigners. So if I care more about the world’s poor, then who’s missing out on love? From whence came this playdough? I’m pretty sure it’s my girlfriend’s coworkers. I honestly can’t keep them straight and I can’t piece together the stories I hear about them into anything but the most abstract people.
I think my moral gradient might be pretty typical for a libertarian. Like you, I don’t want to put people out and that can appear stand-offish. But that’s really just me saying, “I don’t know you, but I believe you aren’t simply a solipsist delusion left here for my abuse or neglect.” But other-oriented sentiments come at a cost. I consider people in the abstract, where non-libertarians consider people on a case-by-case basis. Each one is special, but only a few of them actually count. Most people incorporate Dunbar’s number as both the limit of their social network and a limit to the number of people they actually care about. For me, I basically recognize “family”, “friend”, “human being”. I don’t have “second cousin, twice removed” or “friend’s ex-boyfriend’s cool cousin that I still hang out with some times.”
That’s all a very long winded way of saying “egh, their measure for love might have been focusing on love for those in-between strangers whose names I just can’t remember for the life of me.”
Here is the wiki for Dunbar’s numbers (Robin Dunbar is a British anthropologist). Here is more from Rick on the topic. The ‘moral gradient’ Rick speaks of probably has to do with the research of the economists Sandra Peart and David Levy (I’d start here, if you’re interested), but I’m just wagering a guess.