I’m sorry to hear you’re having trouble with your son’s school. If the school are concerned with other children ‘noticing he’s different’, what are they doing at the school to celebrate difference, support friendships, and raise awareness of disability and equality? The Public Sector Equality Duty (under the Equality Act 2010) states that all public sector bodies - including schools - must:
• Eliminate unlawful discrimination, harassment and victimisation and other conduct prohibited by the Act.
• Advance equality of opportunity between people who share a protected characteristic and those who do not.
• Foster good relations between people who share a protected characteristic and those who do not.
The school's reason that "his peers are starting to notice that he is different to them" seems incompatible, in particular, with their duty to encourage good relations between those with protected characteristics (which includes disability) and other children. Instead, as part of their duty they should be taking proactive steps to help ensure your son can be integrated into the school and with his peers - they certainly shouldn't be using it as an excuse for him to be removed from the school.
You might be interested to read the following from the SEND code of practice, which explains how schools and other public bodies must implement the Children and Families Act:
1.26 As part of its commitments under articles 7 and 24 of the United Nations Convention of the Rights of Persons with Disabilities, the UK Government is committed to inclusive education of disabled children and young people and the progressive removal of barriers to learning and participation in mainstream education. The Children and Families Act 2014 secures the general presumption in law of mainstream education in relation to decisions about where children and young people with SEN should be educated and the Equality Act 2010 provides protection from discrimination for disabled people.
1.27 Where a child or young person has SEN but does not have an EHC plan they must be educated in a mainstream setting except in specific circumstances […].
You can see the circumstances mentioned above explained on page 25 onwards of the code here: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/398815/SEND_Code_of_Practice_January_2015.pdf.
As Dave mentions, for some children a special school could be the best option, but your son’s school should do everything they can to offer him the support he needs to remain with his peers. The only basis on which the impact on his peers should be taken into account is if he's impacting on their learning (disruption, etc). From your description of your son, this seems unlikely.
You should feel confident in challenging your son’s school about this issue, because from your description, they could be breaking the law. You might find the following resource useful: https://www.ipsea.org.uk/what-you-need-to-know/disability-discrimination.
Have you spoken to your local IAS? They might be able to offer some support, and you can find their details by looking for your local authority here: https://councilfordisabledchildren.org.uk/information-advice-and-support-services-network.
Best of luck,