Preheat the oven to 350°F (175°C) and grease a 9x5-inch loaf pan or line it with parchment paper.
In a medium bowl, beat together the softened cream cheese, sugar, egg, flour, and vanilla extract for the cream cheese filling until smooth. Set aside.
In a large bowl, whisk together flour, baking soda, baking powder, salt, cinnamon, nutmeg, ginger, and cloves for the bread batter.
In another bowl, combine granulated sugar, brown sugar, pumpkin puree, oil, eggs, milk, and vanilla extract. Mix well.
Gradually fold the dry ingredients into the wet ingredients, being careful not to overmix.
Pour half of the batter into the loaf pan, then spread the cream cheese mixture evenly over it. Pour the remaining batter on top and smooth the surface.
Bake for 50-65 minutes, until a toothpick inserted into the bread comes out clean. If the top browns too quickly, cover it loosely with foil in the last 15 minutes.
Let the bread cool in the pan for 15 minutes before transferring to a wire rack to cool completely.