Previous studies have shown the existence of misrepresentation regarding various religious identities in Indonesian media. Misrepresentations of other marginalized identities in natural language processing (NLP) datasets have been recorded to inflict harm against such marginalized identities, in cases such as automated content moderation, and as such must be mitigated. In this paper, we analyze, for the first time, several Indonesian NLP datasets to see whether they contain unwanted bias and the effects of debiasing on them. We find that two, out of three, datasets analyzed in this study contain unwanted bias, whose effects trickle down to downstream performance under the form of allocation and representation harm. The results of debiasing at the dataset level, as a response to the biases previously di scovered, are consistently positive for the respective dataset. Nevertheless, depending on the dataset and embedding used to train the model, they vary highly at the downstream performance level. In particular, the same debiasing technique can decrease bias on a combination of datasets and embedding, yet increase bias on another, particularly in the case of representation harm. © 2023, Ikatan Ahli Informatika Indonesia. All rights reserved.