From my perspective, it doesn't damage the brain at all, quite the opposite, actually. Obviously there are extreme cases where it can be damaging, but that is with everything!
During my age of 2 and about 16 (up until I became more interested in girls, you could say, heh), I was quite a gamer. Not a hardcore one, even though 4 to 5 hours a day I wouldn't say is average (maybe now it is, I'm not really in the game world anymore). I developed great reflexes. Once I sat in an F-16 Flight Simulator, and I was quite good at it, or solving those thinking puzzles, or shooting ranges, or sniping around. And working with computers in general. I 'get' the system, the virtual/imaginary system.
Games back then (80's up until beginning 90's) were often a lot more abstract than now, with a lot clearer set of rules, which made the thinking very abstract.