Love the idea! We're doing something similar to parse rubrics and student submissions at https://automark.io - great to see an open source library exploring the space more! Like you said, I think iteratively adding explicit layers of LLM understanding to the raw extraction will allow a lot more control over what information gets extracted. Also interested to see an integration with GPT-4V as an additional aid. I'd love to chat sometime if you have time - my email is in my bio.