Brick strength is measured in ability to withstand deformation under load and as far as I know there isn't any popular "plastic" that can do comparably.
For a single story house this probably doesn't make any difference, but for a larger structure the load could cause bricks to loose shape, creep and cause the structure to fail.
Also, plastics tend to get soft rapidly at temperatures. I don't think plastic bricks exposed to sun (Kenya?) would fare for long.
Another issue is that plastics tend to deteriorate in UV. That could be fixed with a good paint but it also costs.
(Edit: for the pavement this is horrible idea as the plastic is going to be broken down and dispersed in environment in large amounts)