Preheat oven to 350 degrees. Generously grease two 9x5 inch loaf pans and sprinkle with four. (or line with parchment paper)
In a large bowl cream together butter and sugar until very light and fluffy (2-3 minutes with an electric mixer). Add eggs, one at a time, mixing after each until incorporated. Mix in vanilla and nutmeg.
Alternate mixing in flour and milk until both are incorporated (do not overmix).
Spread batter evenly in prepared pans. Bake for 55-70 minutes until a toothpick inserted in the center comes out mostly clean (a few crumbs are okay but no wet batter).
Allow to cool completely before frosting the cake.
To prepare the frosting, cream butter for 2-3 minutes until very light and fluffy. Mix in powdered sugar. Add eggnog 1 tablespoon at a time until frosting is smooth and spreadable (but not runny). Stir in nutmeg.
Spread frosting on cooled pound cakes. Cut into slices and serve or store in airtight container up to 3 days.