You don't have to be the one that's lagging to experience lag. Someone else can have the bad connection, the host can have the bad connection, or you can be in that sweet spot where you can't really tell there's lag but there is.
Lag is funny like that.