Looks I succeed. I had to change the LOW fuses to correct value. I modified the APM Optiboot bootloader to accept 4MHz external oscilator. Here is the correct settings which need to be add to the boards.txt file:
##############################################################
apm96_4MHz.name=APM Optiboot external 4MHz noBOD 38400baud
apm96_4MHz.upload.tool=avrdude
apm96_4MHz.upload.protocol=arduino
apm96_4MHz.upload.maximum_size=32256
apm96_4MHz.upload.speed=38400
apm96_4MHz.bootloader.tool=avrdude
apm96_4MHz.bootloader.low_fuses=0xfd
apm96_4MHz.bootloader.high_fuses=0xde
apm96_4MHz.bootloader.extended_fuses=0x07
apm96_4MHz.bootloader.path=optiboot_v50
apm96_4MHz.bootloader.file=atmega328_1a_4MHz.hex
apm96_4MHz.bootloader.unlock_bits=0x3F
apm96_4MHz.bootloader.lock_bits=0x2F
apm96_4MHz.build.mcu=atmega328p
apm96_4MHz.build.f_cpu=4000000L
apm96_4MHz.build.core=arduino
apm96_4MHz.build.variant=standard
##############################################################
I had to change following lines:
apm96_4MHz.upload.speed=38400
apm96_4MHz.bootloader.low_fuses=0xfd
apm96_4MHz.build.f_cpu=4000000L
I also changed the file name to: apm96_4MHz.bootloader.file=atmega328_1a_4MHz.hex but the file is exactely the same as the original one. I also changed the name in the beginning of each line.
Currently looks it is working well. I will add more info later if all runs ok after some testing.
And here is also nice FUSES configurator.