One man spoke of how a “mob” turned on him, throwing bricks at his family and their home in Swansea.