You would roll backwards. This is because the wall is stationary while with you having the roller skates on makes you unstationary. The force you put on the wall does nothing to the wall but instead goes back to push you. Again this is because the wall is stationary and is not going anywhere.
If you were wearing roller skates and pushed as hard as you could against a wall you would be slung backwards. This happens from you pushing yourself & from gravity .
