Ex Google employee Doug Edwards is credited with coining this phrase for ex-Googlers. He's also credited with Nooglers for new Google hires. Doug Edwards worked for the company from 1999 to 2005, which took Google from a scrappy startup to to a publicly held company that dominates the Web.
Xooglers also refers to the blog he started, xooglers.blogspot.com, which talks about his experiences working for Google and now includes YouTube videos of Google in the early days. He abandoned the blog at one point and has since revived it to publicize his his book,I'm Feeling Lucky: The Confessions of Google Employee Number 59, which is scheduled for publication in July of 2011.