It really can't be done, the custom bbcodes are for admins to set up codes the members use when posting, what you're asking for is something that would take up a lot of resources to code up for the network since image editing is resource heavy. I wish I had a better answer for you smwon. But Custom BbCodes don't work like that and I ran it by BbBoy to confirm it wasn't possible, so I'm sorry this isn't something we can help with. You can ask members to resize their images using an online editor like
http://www.myimager.com or something, and then post them, but unfortunately many people will ignore that request for whatever reason.