Audio synchronisation with RAW?

First of all I am so so sorry if this is posted in the wrong place. I am not sure where I should post this exactly? I just know that there are good people here that may possibly point me in the right direction. Again I apologise in advance.
I have now had several experiments trying to synchronise audio with the RAW video in post.
I am used to using the traditional method of recording audio on a external sound recorder (ZoomH2) and using a clapper board to get an audio/visual sync.
I have used this technique to get greatly improved sound when compared to the 5DMKII's inbuilt mic.
I have found it to be pretty reliable and with a visual cue (the clapper board) its pretty easy to align them in most decent NLE's
However I have tried aligning the separate audio with the RAW video and there is something that's not quite right.
A few days ago I just recorded some video with separate audio using clapper board and aligned them but in the video there was nothing else for me to see if the sync was staying in sync. I just made the assumption that everything was fine. Although I could never be sure exactly how good the sync was it just looked ok because all the sounds were happening off screen.
Anyway today I decide to test a little more thoroughly. I just set up a simple shot where I would use the clapper and then I would tap a little beat on a surface and obviously keep my fingers in the frame (so I could see if it was all in sync). I recorded a few takes of the same kind of thing with both RAW video being recorded and my ZoomH2 recording the sound.
I got them on the computer processed the RAW files and import them into After Effects. I then dropped the audio into After Effects and went in to align them. I aligned the sound of the clapper at the frame where the clapper shuts. This is something I have done numerous times before (with the old H264 video) and would usually be confident at this stage, that when I hit play everything will be perfectly (or very nearly perfectly) in sync.
Well this is not so. I am not sure whats happening. Where I have aligned the clap it's perfect but it seems that it must be drifting.
My only conclusion is that there is some mismatch somewhere in the timing of the two. The problem is where would that be?
One thing I would really like to know is that should the 23.976fps be processed as 24fps? (I have tried both and they both drift somehow).
In fact this one has really got me because I can't seem to get the sync whatever I do?
I am wondering maybe if I will have to do a slight pitch shift on the audio or maybe some kind of re-conform of the video. I don't want the additional processing involved but if I could somehow get some workflow that 'worked' then maybe that would help?
I must say at this stage I am lost with this one.
I was wondering if anyone else had any experience of audio synchronisation with the RAW footage?
Any help would be greatly appreciated.