I just tried another DVD with the same settings, and everything works fine. So something that has changed recently in x264 causes this error (AV seems related to threading, but I didn't read about any changes in threading), but because it's just in this single dvd, I don't really want to waste more time. Thanks to all of you for trying to help.