OK, 2.2c again, and this is a rough guide as I haven't read the gauges in detail.
Going from a heavy wire gauge to one of lighter gauge is not necessarily an issue, as long as the lighter gauge does not carry the same current over a long run. Having said that, fires can happen due to heat generated by the induction at the joins.
Having the windlass running only while the engine is running makes much sense. You can in fact use a VSR to control this -- if the battery terminal voltage drops below a certain voltage, cut the power to the windlass. Then again, if your battery voltage drops too low while the windlass is running the windlass won't work anyway.
Having the windlass running off the engine battery is also a smart move, as the engine battery is specifically designed to handle that sort of high current load for short bursts. However for a 1200W windlass it shouldn't be a major bother, provided you run the engine / genset / battery charger for a good long while after using the windlass. I have a 2000W windlass running off my house batteries over a longer cable run than above, and it's not a bother. Anything over 2400W and you really do want to be running off the engine battery, as otherwise you're pulling 200A from your house batteries, not good.
Some things to do:
Calculate the voltage drop of the heavier longer run. This will be a percentage. e.g. 1.6%. Subtract this from 100, e.g. 100 - 1.6 = 98.4, and divide that by 100 = 0.984. Remember that as number 1.
Calculate the voltage drop of the lighter shorter run. This will also be a percentage, e.g. 1.2%. Do the same calculation as above which will give you another number (like 0.988 in my example), which is number 2.
Calculate the voltage drop of any cable joins (splices, terminal points or blocks, etc). This is best done by measuring the resistance in ohms of the cable join itself (which may be a fraction of an ohm) and comparing that to the resistance per meter of the lightest cable you're using (which can be obtained from the manufacturer, or from a chart, I have one in mm2 but not in AWG sizes as I'm outside the USA). Divide the resistance of the joiner by the resistance per meter of the cable and you have a length in meters which you should pretend that the joiner is. e.g. if the joiner resistance is 0.2 ohms, and your cable is 0.02 ohms per meter, then pretend your joiner is a cable of 10 meters. Calculate the voltage drop of this length of cable, do the same subtract-from-then-divide-by-100 calculation as above, and you have number 3.
Multiply numbers 1, 2, and 3 together. The end result will be a number like 0.97745 ... example.
This number must not be less than 0.97, or your attempt will be a failure. In fact it shouldn't even be all that close to 0.97 for a 1200W windlass. If number 3 is close to 0.97 by itself (or even lower than it) then you need to get heavier cable joints in place, preferably tin plated copper all around.
Once it's all together you have to make sure there's insufficient inductance in the joints to cause a problem. Probably the best way to do this is to get two long lengths of wire and connect them from your multimeter to each end of each cable run (positive first, then negative). e.g. run from + at the battery to + at the multimeter, then from + at the windlass to - at the multimeter. Set the multimeter to read ohms, you want to measure the resistance of this wire. Fire up the windlass and look for any spikes in the resistance -- if the resistance jumps up by more than about 100% as you fire up the windlass (say from 2 ohms to 5 ohms or more) then you have inductance in your cable run, probably at the cable join, and need to start straightening things out. Preferably you should have cable coming into the join and going out of the join at 180 degrees opposite each other, e.g. this ----+---- and not making a 90 degree bend or similar at the join.
Hope that helps.