You're planning on leaving the land of the pasty, of course you're crazy!
Seriously though, I'm pretty certain that if you had jobs lined up you wouldn't be worried at all. After all, worst case scenario: you have a wonderful little adventure involving all the American things you like, and then decide to move back to the UK. You will be able to keep your family fed, sheltered, and educated the entire time so it would be a petty mistake, not a life-shattering one.
Personally I didn't go through this, because I had no choice in the move. But it seems it's a common feeling for people after they get the visa: .
Of course, just because DH has his green card doesn't mean you have to move. But put everything you have into it and it will work out well. Keep focused on the positives and you'll be fine.