The Special Episodes are usually Christmas Episodes, which is a common thing to have happen in the UK, and the don't count as parts of any series, typically.  I imagine that's the case, so... you're going to be waiting for a while before you get any more new Downton.

I kept thinking about whether I should pick up the new season or not.