How do you account for life being so common, but no aliens visiting us here on Earth?

We don't really know how common life is beyond Earth. We will only have a good way to estimate that after we have surveyed enough stars and detected the first planet whose atmosphere has free oxygen - a well known life sign on a planetary scale. Then we can estimate from our search just how many planets in our Milky Way have living systems.

Intelligent life is far more difficult to assess by remote sensing from 100 light years away or more! Right now, if you were to use something like the Drake Equation to estimate how many planets are likely to have intelligent life, you can cook the numbers any way you want and get numbers any where from 1 civilization ( us) to millions. The search by SETI during the last 10 years has been rather disappointing. I think this says that technological civilizations like ours are not that common, and are probably separated by at least 1000 light years, or millions of years of time.

Interstellar travel is so difficult that I cannot believe any civilization would be stupid enough to try it. There are far more ways of exploring the universe than stuffing your body in a can and sending it out to the stars. All of these other ways are inexpensive in comparison.

Interstellar travel is HARD! Only 100 years of science fiction writing has made us believe that it ought to be easy.