Your going to have to drop to command line... The GUI isnt giving you enough control
From the logs you can see
C:\Program Files\arduino-1.6.9\hardware\tools\avr/bin/avrdude -CC:\Program Files\arduino-1.6.9\hardware\tools\avr/etc/avrdude.conf -v -patmega1284p -cstk500v1 -PCOM5 -b19200 -e -Ulock:w:0x3F:m -Uefuse:w:0xFD:m -Uhfuse:w:0xDE:m -Ulfuse:w:0xFF:m
but its erroring out
This is just trying to set the fuses (hardware options on the cpu)
I find I have to add in some delays with the -B flag
Last time I tryed this I needed - B 8, but you may need to try a few numbers (its different for each pc)
eg C:\Program Files\arduino-1.6.9\hardware\tools\avr/bin/avrdude -B 8 -CC:\Program Files\arduino-1.6.9\hardware\tools\avr/etc/avrdude.conf -v -patmega1284p -cstk500v1 -PCOM5 -b19200 -e -Ulock:w:0x3F:m -Uefuse:w:0xFD:m -Uhfuse:w:0xDE:m -Ulfuse:w:0xFF:m
Once this succeeds you need to move on to the actual boot loader
which would be something like
C:\Program Files\arduino-1.6.9\hardware\tools\avr/bin/avrdude -B 8 -CC:\Program Files\arduino-1.6.9\hardware\tools\avr/etc/avrdude.conf -v -patmega1284p -cstk500v1 -PCOM5 -b19200 -e -U flash:w:{your path}/ATmegaBOOT_1284P.hex
From the logs you can see
C:\Program Files\arduino-1.6.9\hardware\tools\avr/bin/avrdude -CC:\Program Files\arduino-1.6.9\hardware\tools\avr/etc/avrdude.conf -v -patmega1284p -cstk500v1 -PCOM5 -b19200 -e -Ulock:w:0x3F:m -Uefuse:w:0xFD:m -Uhfuse:w:0xDE:m -Ulfuse:w:0xFF:m
but its erroring out
This is just trying to set the fuses (hardware options on the cpu)
I find I have to add in some delays with the -B flag
Last time I tryed this I needed - B 8, but you may need to try a few numbers (its different for each pc)
eg C:\Program Files\arduino-1.6.9\hardware\tools\avr/bin/avrdude -B 8 -CC:\Program Files\arduino-1.6.9\hardware\tools\avr/etc/avrdude.conf -v -patmega1284p -cstk500v1 -PCOM5 -b19200 -e -Ulock:w:0x3F:m -Uefuse:w:0xFD:m -Uhfuse:w:0xDE:m -Ulfuse:w:0xFF:m
Once this succeeds you need to move on to the actual boot loader
which would be something like
C:\Program Files\arduino-1.6.9\hardware\tools\avr/bin/avrdude -B 8 -CC:\Program Files\arduino-1.6.9\hardware\tools\avr/etc/avrdude.conf -v -patmega1284p -cstk500v1 -PCOM5 -b19200 -e -U flash:w:{your path}/ATmegaBOOT_1284P.hex