In part three, chapter 1, right after O'Brien reveals himself to Winston Smith in the Ministry of Love, he says, "You knew this Winston. Don't deceive yourself. You did know it - you have always known it."
Really? I thought Winston was completely convinced that the Brotherhood existed and that O'Brien was a part of it. When did he ever doubt that?

And also, does this mean that the Brotherhood was something fabricated by the Party to try and catch everyone who commits thoughtcrime or do the Brotherhood (and Goldstein) actually exist and O'Brien and other Party members just pretend to be a part of it for the same reason?

That is one of the unanswered questions in the novel. He never gives the reader any undeniable proof that the Brotherhood, Goldstein, or Big Brother for that matter actually exist. O'Brien uses doublethink so you don't really know what's the real "truth" or not, and it's not supposed to matter, because regardless, the Party manufactures everyone's reality.

Actually, O'brien later states that the Brotherhood <i>was</i> a device to ensnare would be rebels. I don't have the page or the section, but I believe that Winston asks him about the Brotherhood, and O'brien <s>enlightens</s> crushes his every dream.