1. Start by making the custard. Beat together the egg yolks and sugar until pale. Mix the cream, milk and vanilla in a pan and warm to just before boiling point. Pour into the beaten egg, blending well. Return to the pan over a low heat and stir continuously until thickened. Transfer to a bowl and cool.
2. Roughly break up the sponge, if large, and spread with the jam. Place in the bottom of a large glass bowl or trifle dish.
3. Roughly crush the biscuits and scatter over the jammed sponge.
4. Mix the sherry and whisky (or Drambuie) and pour over the biscuit and sponge. Leave to soak.
5. Whisk the double cream and caster sugar to thick peaks and fold in the whisky if using.
6. Pour the cooled custard over the sponge. Top with a layer of raspberries and banana slices, if using, and spoon over the thickened cream. Finally, decorate with a scattering of flaked almonds.