I'm not going to speak for America, so I'll just speak for me. I don't know any folks from Pakistan. And I'm not sure I know any racists sophisticated enough to be able to tell if someone is Pakistani, Indian, Arab or anyone else from that billion sq. miles of real estate. Therefore, I'm not sure I've ever heard the term itself, let alone used as a slur. I will however, take your word for it.
I blame the map
https://www.google.com/search?client=ms-android-att-us&q=pakistani+communities+in+usa&spell=1&sa=X&ved=0ahUKEwit1PzzxZfQAhXh64MKHWoZDEkQvwUIFigA&biw=360&bih=518