I'm curious to know how this method is working, are you sure that you should do your RGB to Indexed in the same image?
I would do the conversion in a new/other image (in a different GIMP tab), then once it's clean, convert back to RGB and only then drag n' drop to the normal other image and use it as a mask for example.
Because in the same image and if I understood your workflow (it would be nice if you post 2 images before and after for me to get an idea):
Indexed use 256 colors,
RGB ~16.7million
You are transforming an image with 16.7 million colors to an image with 256 colors (16 colors setting is even worse), then try to get back the original quality by re-transposing your image in 16.7 million colors, but you've lost 16.5 million colors while in indexed colors...
So yes you went back to RGB, but with only 256 colors spreaded thru the 16.7 million colors
Having said that, and if it's about pixels at the edge of your subject, click on the mask then Filters > Blur > Median blur (median blur is very good at that)
I would do the conversion in a new/other image (in a different GIMP tab), then once it's clean, convert back to RGB and only then drag n' drop to the normal other image and use it as a mask for example.
Because in the same image and if I understood your workflow (it would be nice if you post 2 images before and after for me to get an idea):
Indexed use 256 colors,
RGB ~16.7million
You are transforming an image with 16.7 million colors to an image with 256 colors (16 colors setting is even worse), then try to get back the original quality by re-transposing your image in 16.7 million colors, but you've lost 16.5 million colors while in indexed colors...
So yes you went back to RGB, but with only 256 colors spreaded thru the 16.7 million colors
Having said that, and if it's about pixels at the edge of your subject, click on the mask then Filters > Blur > Median blur (median blur is very good at that)