I think when people hear 'Control our Borders', they think 'reduce immigration'.
I know most leave voters didn't vote for zero immigration, but they surely wanted a drastic cut, perhaps to 'the tens of thousands'.
Irrational hatred of foreigners has been whipped up for decades by a large part of the Brexit supporting press (who you rightfully described as poison) so the Kippers and their less savoury allies are used to blameing every ill on immigrants and muslims.
The brexit promises are gradually being dismantled one by one, this is a decision that the country will surely come to regret.
To quote Jayan Ganesh, of the FT:
Brexit is an idea whose only effective rebuttal is its own implementation.
There is more chance of Britain leaving and then rejoining in stages than there is of Britain never leaving in the first place ... It is a matter of steering the evolution of British laws and institutions towards the EU norm, until the gap between membership and non-membership withers. This is not just possible, it is probable. The best argument against exit was never the steep downside so much as the measly upside. Sovereignty is a dream: the gravitational pull of a unified Europe on our medium-sized nation is too strong.