Combine sugar, cornstarch and salt in a medium-sized saucepan (off heat). Whisk until combined and no lumps of cornstarch remain.
Add egg yolks, whisk until combined then add both the whole milk, evaporated milk, pumpkin purée, dark rum and spices. Whisk mixture until completely smooth and everything has dissolved.
Transfer saucepan to the stove and cook over medium heat, whisking constantly, until liquid thickens and comes to a bubble. The pudding will first begin to thicken on the bottom of the pan where it is in contact with heat. You must continually remove the thickened pudding from the bottom of the pan or it will scorch. Don’t worry, just stir constantly, and don’t forget about the corners of the pan!
Once the first bubble rises to the surface, reduce the heat to low and continue to cook for exactly 1 minute, whisking vigorously and constantly.
Remove the pudding from the heat and whisk in vanilla and butter until completely smooth and well-combined. Transfer pudding to a medium-sized mixing bowl then cover with plastic wrap. Place plastic wrap directly in contact with the pudding to prevent a thick skin from forming on the surface.
Refrigerate for at least 1 hour or until completely chilled. Fold in whipped cream then refrigerate for 2 more hours.
For each serving, make a layer of graham cracker crumbs, chopped pecans, then pumpkin mousse. Repeat each layer then top with whipped cream, a few crumbs and more nuts.
If you need to save time, you can definitely use whipped topping instead of making your own whipped cream. Just know that doing so will increase the sweet factor substantially and this mousse is plenty sweet already.