We do not really know how many bodies are present in the 100 meter to few kilometer size range. That's why astronomers such as Eugene Shoemaker and his colleagues have finally convinced NASA to support a systematic search for bodies in this size range. It is these that could do severe ecological damage. I think the estimates are that there are something like a few thousand bodies in this size range, and we only know of a few percent of these. They are very dim, and you only seem to spot them when they are less than a few weeks out from the Earth because their angular speeds in the sky have to be high enough to see them move against all the other background clutter. A recent computer simulation of how much time we get to predict an exact impact site by a colliding body is pretty sad. If we pick it up a week or more out, we can forecast the longitude range of the impact but not the latitude to a precision better than the entire diameter of the Earth. By a few days from impact, we can localize the impact to about a 100 mile foot print on the Earth.
We know the orbits of about 7000 minor bodies in the solar system, and these run to sizes as small as a few kilometers across. But their orbits can and do get perturbed over decades and centuries. Some of these which end up on orbits intersecting the Earth called 'Apollo-Amor' asteroids have been know for decades. But it is cometary bodies launched from out by the orbit of Jupiter and beyond which are troublesome. Hyakutake and IRAS-Alcock discovered in 1983 were nearly upon us before they were detected at all. Comets are a real problem, and can do nearly as much damage as asteroids. Hale-Bopp is one of the largest and is 20 - 50 kilometers across! We have, however, known about it for several years now because it is big enough and its orbit is optimal enough to have been tracked beyond the orbit of Mars.
This is serious business, and the extent to which we all live on a knife edge is very unsettling.