3a^2b-{2a^2b+[9a^b-(6a^2b+4a^2)]-(3a^2b-8a^2)}=

3a^2b-{2a^2b+[9a^2b-(6a^2b+4a^2)]-(3a^2b-8a^2)}
=3a^2b-2a^2b-[9a^2b-(6a^2b+4a^2)]+(3a^2b-8a^2)
=a^2b-9a^2b+(6a^2b+4a^2)+3a^2b-8a^2
=-5a^2b+6a^2b+4a^2-8a^2
=a^2b-4a^2