You might want to consider the STM32F103 -- they're programmable over serial in much the same way that traditional ATMEGA chips are, are arduino-compatible, can be found for roughly the same price as the 1284p, are a ton faster, and most versions have a ton more memory and flash.
You can see the full lineup of chips here: http://www.st.com/en/microcontrollers/stm32f103.html?querycriteria=productId=LN1565 -- I've personally used the STM32F103CB and STM32F103RE in a handful of projects lately.
@NeverDie -- the motors used in 1610 CNC mills are generally what are called a "775 Motor". You might be able to find other specs, but the ones I've found suggest that at 24V and no load , they claim 7kRPM -- http://linksprite.com/wiki/index.php5?title=File:Motor_performance_parameter.png; I'm not sure how much slower we could expect it to be while milling. As far as actually measuring this, there are devices you could buy, but you could pretty easily fabricobble your way to an answer if you wanted to make a project out of it: http://www.instructables.com/id/Measure-RPM-DIY-Portable-Digital-Tachometer/.
I have the same mill as you, and swapped for one of these https://www.amazon.com/gp/product/B074FVKRZM/ and have had much better results so far.
@alexsh1 I'm afraid I've only used STM32F103 variants. Although the Arduino core I've used (https://github.com/rogerclarkmelbourne/Arduino_STM32) is STM32F103 specific, it does look like the official core has support for that chip, though: https://github.com/stm32duino/Arduino_Core_STM32. Just keep in mind that different chips have differing capabilities as far as programming is concerned; I do think the STM32F103 is the most limited of them, though -- you can find details about the bootloader version for each chip here: http://www.st.com/content/ccc/resource/technical/document/application_note/b9/9b/16/3a/12/1e/40/0c/CD00167594.pdf/files/CD00167594.pdf/jcr:content/translations/en.CD00167594.pdf .
I actually haven't had much trouble with vibration, so I can't really speak to that. The problems I've had have mostly been around runout, occasional arcing in the old spindle causing enough EMI to reset the microcontroller, Z-Axis backlash, the super-slow milling speeds necessary when spinning at such a low RPM, copper flakes everywhere, etc. Having the faster (and, maybe unfortunately, heavier) brushless spindle has been great for all of those things.
I happened to notice you looking into tool changers; if you're using bCNC, you might not have noticed the tool change workflow -- it's actually pretty elegant and will automatically re-zero your Z-Axis. It does require having a limit switch on at least the top of that axis, though. I used to hate tool changes given how tedious they were, but now they're pretty effortless.
@neverdie Congratulations on your purchase! I think you'll find having a real spindle will help a ton.
If you haven't designed your own mount, you might want to check out the part I posted on Thingiverse the other day: https://www.thingiverse.com/thing:2817974 .
@neverdie Hrm; honestly I haven't monitored it very closely, and almost always run the spindle at full speed; so I can't really say for sure if it would attempt to adjust to make up for higher friction. Also, I have the NVBDL rather than the NVBDH version, so I'm not even sure if it is able to tell if the motor is spinning slower than intended.
If either of us have a chance of that being a possibility, it's probably you with the slightly better spindle driver.
