I think that some people make fun of the ICW, the "Ditch", in order to feel superior to those who choose to make their way south in it. I loved our trips, both the sailboat trip and the powerboat trip. For our powerboat trip, up and back down, I've a blog at http://www.sailblogs.com/member/mvmelon/
I would second KaptainKen's comments. We left Boston in SV Watermelon in 1986, came down through Long Island Sound, (you would miss that with the canals into the Hudson River). Sailed offshore past the NJ coast into the Delaware River and through the Cheseapeake & Delaware Canal into Chesapeake Bay - we spent a lot of time in the Chesapeake and loved it.
Second trip down in our MV Watermelon in 2006 we could go faster and chose to stay offshore down Delaware because it cuts many miles and two days off the trip. We still had to wait for favorable weather because the power boat isn't as sea kindly as our sailboat was.
We stayed on the ICW from Norfolk to south of Cape Hatteras (one of the worst places for sudden bad weather and more wrecks than most places), then went offshore for two days to Charleston, SC because bad weather descended on us and we decided to wait it out rather than sail through it. The Gulf Stream isn't that close to the coast here.
We made a big mistake in not checking the weather carefully enough when we left Charleston for Florida. it was wild and windy sail (in October) and had we been smarter we would have waited for better weather. However, no problems, just entering Mayport, FL (Jacksonville) at night which is safe but still anxiety-ridden for us then - relative newbies. I would do it again, just be more cautious about weather.
We stayed inside the ICW this trip with the power catamaran, but it was late, the weather was unseasonably nasty so early. the power cat is much faster than a sailboat motoring, but the last part of it was still a bit boring.
In a sailboat I would encourage you to sail as much of it as you can - the ICW is full of power boats roaring past with many shoal spots that cause a lot of anxiety. However, as KaptK said, going in and out, daysailing for chunks, can be a lot of fun. The people on the ICW are boater friendly and there's lots to see.