Should i be worried about Colon cancer at 23?

I'm 23 and in good health, i actually had a physical last week and the doctor said i was healthy. On saturday i started feeling pain in my side and stomach. Its has lasted 4 days, so I went to the doctor. I told him my symptoms and he said it was IBS. I stressed to him i had lost 6-8 pounds in 4 days, but he said it wasn't colon cancer due to my age, my health, and that no one in my family has had colon cancer. I should tell tou that I always stress over this type of stuff, but should I worry? They took blood test, can they rule this out with a blood test?