This is generally a problem with hand grinders and leads to uneven grind. One way I have seen other similar grinders resolve it is by having a metal frame at the bottom the helps stop the twist. Yours doesn’t have one so there is probably no easy way to fix this issue.