Thank you Ofnuts for getting me on the right track! The issue was that I didn't make the alpha mask opaque. My code for replicating color-erase posterize is below:
Code:
groupLayer = pdb.gimp_layer_group_new(image)
srcLayer = pdb.gimp_layer_copy(image.layers[0], False)
eraseLayer = pdb.gimp_layer_copy(image.layers[0], False)
#Insert layer group with two layers inside
pdb.gimp_image_insert_layer(image,groupLayer, None, -1)
pdb.gimp_image_insert_layer(image, srcLayer, image.layers[0], -1)
pdb.gimp_image_insert_layer(image, eraseLayer, image.layers[0], -1)
#Posturize erase layer, then set paint mode to Color Erase
pdb.gimp_layer_set_mode(eraseLayer, 57)
pdb.gimp_drawable_posterize(eraseLayer, amount)
#transfer grouplayer alpha to new mask, then fill alpha with white
mask = pdb.gimp_layer_create_mask(groupLayer, 3)
pdb.gimp_layer_add_mask(groupLayer, mask)
pdb.gimp_drawable_edit_fill(mask, 2)
# Apply mask and merge group
pdb.gimp_image_merge_layer_group(image, image.layers[0])
pdb.gimp_layer_remove_mask(image.layers[0], 0)