You can go into the park anytime you want
I for one often walked through the park to get to the train station after work (since the Entertainment department is located near Its a small world). No one has ever said that you cant or are not allowed to do this- or go through the cast member entrance and into the park on your days off.
It is true that you don't walk through the maingate though. The special passes you get are for you and 2 other people.
But yes- you can go into the park as much as you want with your ID. No problem
I assume alot of people would be upset if they suddenly banned this- and at the same time they would have to put guards at every castmember entrance/exit. Alot of people also go backstage on their days off to eat, sort paperwork, dance class, whatever.