Preheat the oven to 350°F (175°C). Grease a 9x5-inch loaf pan or line it with parchment paper.
In a medium bowl, beat the softened cream cheese, sugar, egg, flour, and vanilla extract until smooth and creamy. Set aside.
In a large bowl, whisk together the flour, baking soda, baking powder, salt, cinnamon, nutmeg, ginger, and cloves.
In another bowl, combine the granulated sugar, brown sugar, pumpkin puree, oil, eggs, milk, and vanilla extract. Mix well.
Gradually fold the dry ingredients into the wet ingredients, mixing just until combined. Avoid overmixing.
Pour half of the pumpkin batter into the prepared loaf pan, smoothing it out evenly. Spoon the cream cheese filling over the batter, spreading it carefully. Pour the remaining pumpkin batter on top and smooth the surface.
Bake for 50-65 minutes, or until a toothpick inserted into the bread (not the cream cheese layer) comes out clean. If the top browns too quickly, loosely cover it with aluminum foil during the last 15-20 minutes.
Let the bread cool in the pan for 15 minutes before transferring it to a wire rack to cool completely.