In such phrases the speaker will tend to choose which sounds more natural. The problem (actually I don't think it's a problem at all) is that people differ in what they think sounds natural.
I don't think you will get a reply which will say that me is wrong and my is right (or vice versa).
Added: If anyone tries to tell you that me is wrong and my is right, refer them to Language Log March 3, 2009, which points out that the construction has been in used for over 300 years and also gives lots of examples of the use of the possessive (my) and the accusative (me). You can find this on the internet (http://languagelog.ldc.upenn.edu/nll/?p=1200).