I did that to get Astrid, so I think so. I wasn't too far in my previous town (despite that Robin and I were pretty close), but I barely made any process on that town, so I really didn't lose much. To me, Animal Crossing isn't the same when a villager you love is not in your town.
It's good to do it when you haven't made much progress.
"On a scale from 1 to 10, these children are 15."
- Nanny Stella