It's hard to sustain fannishness for just one thing. And yet, we're only as deeply fannish as we can be when it is for just one thing.
That's how it's been for me with HP. (More specifically HP slash, but let's allow the source material to stand for the smaller focus.) For the most part, I still eat, sleep, and breathe it, but it's a different eat/sleep/breathe than it was at the beginning. At first I was devoted to writing fic, reading fic, viewing art, commenting on fic/art, meta and responding to meta. Over the years it's become something else: reading fic, viewing art, commenting on fic/art, interacting with the friends I've made through HP (and with whom HP is still a large part of our discussions), reacting to disillusionment over the canon, and thinking about writing fic more than the actual writing of it. ^_^
You'd look at that evolution of my obsession and you might say that a lot of that is not HP; surfing livejournal is not merely HP; sympathising with someone over their grandmother's illness is not at all HP; discussing the 300 trailer is not at all HP. But I'd respond, I would not have a livejournal if not for HP. Not just that HP brought me to livejournal; I mean that if HP were not in my life I would be unlikely to want to sustain the journal. When I talk to someone about Torchwood or Pet Shop of Horrors or their cat, there's still this sense of connection that we became interactive friends because we share or once shared that fondness for HP. There's this bit from an old French TV show where Sherlock Holmes's character quotes something, and then says, "Shakespeare." Watson says, wait, that's not Shakespeare, that's Dickens. Holmes quips, "Tout est Shakespeare. Même Dickens," (Everything's Shakespeare. Even Dickens). (ETA: Or is that "Tout sont Shakespeare"? Here I admit my monolingualness, despite all that study.)
All through that, yes, I have had other interests. You might even call them fannish; you might even call them fandoms. I'm not sure I would, because I think fandom implies major dedicated interest. Yes, even as I adore HP fandom, I do watch television, see movies, read books. Stories, in prose and in audio and in visual media, are my life. I do not ever want to have children because that would cut into all of that and I'd effing resent that. I've never missed an episode of 24, never missed one of the new Doctor Who. I have even scribbled a ficlet for each of those sources, and have read a bit of fanfiction for them. Am I in their fandom? I don't really feel I am. I'm a fan, yes. But there needs to be more, needs to be a sense of, "I love this thing, I am really dedicated to thinking about these characters/situations, am highly needy to know more about their off-screen lives and how they'd be in situations that I dream up. I want to see pictures of that and read stories about that and you know what, I have the stories in my head I want to see and I know no one will write them precisely the way I want to see them written so I gotta do it, in fact I can hardly wait to do it, where's my Microsoft Word icon, new document, yes, yes, yes..."
No, I don't think that "in fandom" must equal "fanfiction writer/fanartist" for everyone. Some people have no need to create in that way, and so can't be held to those criteria. But I think you have to want more than the canon, have to think about what's beyond the canon a lot, be needy about it. If you don't die of squee a bit when you discover that someone has drawn a picture of your OTP from the canon, I don't think you really have a deep enough devotion to be "in" the fandom. For me, that latter's a really good criterion; my heart pulses faster when I see the words Harry/Lucius, Harry/Draco, Harry/Sirius on a link in hp_art_daily. To this day it does, oh yeah.
Some of that is because I really, really, really, really don't want another fandom than HP--I'm spoiled by its vastness. I read Lois McMaster Bujold's The Sharing Knife some months back, freakin' loved it, searched the 'net for any fansites devoted to it. I really wanted me some fanart of the characters. There isn't s**t out there for it. So I squelched my disappointment and thanked my lucky stars again for having been thunderbolted into HP, the fandom which has entire communities devoted to skirtporn and SnapeSlash. Am I eager to stay in this fandom, to actively resist being sucked into another? Hell, yes. Will that matter, will my character be any less if it does happen one day? Hey, you try hiding from a determined thunderbolt.