Abstract
The development of Question Answering (QA) systems has become important because it reveals research issues that require insight from a variety of disciplines, including Artificial Intelligence, Information Extraction, Natural Language Processing, and Psychology. Our goal here is to develop a QA approach to answer yes/no questions relevant to civil laws in legal bar exams. A bar examination is intended to determine whether a candidate is qualified to practice law in a given jurisdiction. We have found that the development of a QA system for this task provides insight into the challenges of formalizing reasoning about legal text, and about how to exploit advances in computational linguistics. We separate our QA approach into two steps. The first step is to identify legal documents relevant to the exam questions; the second step is to answer the questions by analyzing the relevant documents. In our initial approach described here, the first step has been already solved for us: the appropriate articles for each question have been identified by legal experts. So here, we focus on the second task, which can be considered as a form of Recognizing Textual Entailment (RTE), where input to the system is a question sentence and its corresponding civil law article(s), and the output is a binary answer: whether the question sentence is entailed from the article(s). We propose a hybrid method, which combines simple rules and an unsupervised learning model using deep linguistic features. We first construct a knowledge base for negation and antonym words for the legal domain. We then identify potential premise and conclusion components of input questions and documents, based on text patterns and separating commas. We further classify the questions into easy and difficult ones, and develop a two-phase method for answering yes/no questions. We answer easy questions by negation/antonym detection. For more difficult questions, we adapt an unsupervised machine learning method based on morphological, syntactic, and lexical semantic analysis on identified premises and conclusions. This provides the basis to compare the semantic correlation between a question and a legal article. Our experimental results show reasonable performance, which improves the baseline system, and outperforms an SVM-based supervised machine learning model.
Keywords
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Merkl, D., Schweighofer, E.: En route to data mining in legal text corpora: clustering, neural computation, and international treaties. In: Proceedings of International Workshop on Database and Expert Systems Applications, pp. 465–470 (1997)
Satoh, K., Asai, K., Kogawa, T., Kubota, M., Nakamura, M., Nishigai, Y., Shirakawa, K., Takano, C.: PROLEG: an implementation of the presupposed ultimate fact theory of Japanese civil code by PROLOG Technology. In: Bekki, D. (ed.) JSAI-isAI 2010. LNCS (LNAI), vol. 6797, pp. 153–164. Springer, Heidelberg (2011)
Jikoun, V., de Rijke, M.: Recognizing textual entailment using lexical similarity. In: Proceedings of the PASCAL Challenges Workshop on RTE (2005)
MacCartney, B., Grenager, T., de Marneffe, M.-C., Cer, D., Manning, C.D.: Learning to recognize features of valid textual entailments. In: Proceedings of HLT-NAACL (2006)
Sno, R., Vanderwende, L., Menezes, A.: Effectively using syntax for recognizing false entailment. In: Proceedings of HLT-NAACL (2006)
Das, D., Smith, N.A.: Paraphrase identification as probabilistic quasi-synchronous recognition. In: Proceedings of ACL-IJCNLP (2009)
Schubert, L.K., Durme, B.V., Bazrafshan, M.: Entailment inference in a natural logic-like general reasoner. In: Proceedings of the AAAI 2010 Fall Symposium on Commonsense Knowledge (2010)
MacCartney, B., Galley, M., Manning, C.D.: A phrase-based alignment model for natural language inference. In: Proceedings of EMNLP (2008)
MacCartney, B., Manning, C.D.: Natural logic for textual inference. In: Proceedings of Workshop on Textual Entailment and Paraphrasing at ACL (2007)
Ohno, S., Hamanishi, M.: New Synonym Dictionary. Kadokawa Shoten, Tokyo (1981)
Kim, M.-Y., Kang, S.-J., Lee, J.-H.: Resolving ambiguity in inter-chunk dependency parsing. In: Proceedings of 6th Natural Language Processing Pacific Rim Symposium, pp. 263–270 (2001)
Walas, M.: How to answer yes/no spatial questions using qualitative reasoning? In: Proceedings of the International Conference on Computational Linguistics and Intelligent Text Processing, pp. 330–341 (2012)
Bdour, W.N., Gharaibeh, N.K.: Development of yes/no arabic question answering system. Int. J. Artif. Intell. Appl. 4(1), 51–63 (2013)
Kouylekov, M., Magnini, B.: Tree edit distance for recognizing textual entailment: estimating the cost of insertion. In: Proceedings of the Second PASCAL Challenges Workshop on RTE (2006)
Vanderwende, L., Menezes, A., Snow, R.: Microsoft research at RTE-2: syntactic contributions in the entailment task: an implementation. In: Proceedings of the Second PASCAL Challenges Workshop on RTE (2006)
Nielsen, R.D., Ward, W., Martin, J.H.: Toward dependency path based entailment. In: Proceedings of the Second PASCAL Challenges Workshop on RTE (2006)
Zanzotto, F.M., Moschitti, A., Pennacchiotti, M., Pazienza, M.T.: Learning textual entailment from examples. In: Proceedings of the Second PASCAL Challenges Workshop on RTE (2006)
Harmeling, S.: An extensible probabilistic transformation-based approach to the third recognizing textual entailment challenge. In: Proceedings of ACL PASCAL Workshop on Textual Entailment and Paraphrasing (2007)
Marsi, E., Krahmer, E., Bosma, W.: Dependency-based paraphrasing for recognizing textual entailment. In: Proceedings of ACL PASCAL Workshop on Textual Entailment and Paraphrasing (2007)
Penas, A., Forner, P., Sutcliffe, R., Rodrigo, A., Forascu, C., Alegria, I., Giampiccolo, D., Moreau, N., Osenova, P.: Overview of ResPubliQA 2009: question answering evaluation over european legislation. In: Proceedings of the Cross-Language Evaluation Forum Conference on Multilingual Information Access Evaluation: Text Retrieval Experiments, pp. 174–196 (2009)
Monroy, A., Calvo, H., Gelbukh, A.: NLP for shallow question answering of legal documents using graphs. In: Gelbukh, A. (ed.) CICLing 2009. LNCS, vol. 5449, pp. 498–508. Springer, Heidelberg (2009)
Mai, Z., Zhang, Y., Ji, D.: Recognizing text entailment via syntactic tree matching. In: Proceedings of NTCIR-9 Workshop Meeting (2011)
Arya, D.A., Yaligar, V., Prabhu, R.D., Reddy, R., Acharaya, R.: A knowledge based approach for recognizing textual entailment for natural language inference using data mining. Int. J. Comput. Sci. Eng. 2(6), 2133–2140 (2010)
Bench-capon, T.: What makes a system a legal expert? In: Legal Knowledge and Information Systems: JURIX 2012, pp. 11–20 (2012)
Alberti, M., Gomes, A.S., Gonçalves, R., Leite, J., Slota, M.: Normative systems represented as hybrid knowledge bases. In: Leite, J., Torroni, P., Ågotnes, T., Boella, G., van der Torre, L. (eds.) CLIMA XII 2011. LNCS (LNAI), vol. 6814, pp. 330–346. Springer, Heidelberg (2011)
Lundstrom, J.E., Aceto, G., Hamfelt, A.: Towards a dynamic metalogic implementation of legal argumentation. In: Proceedings of ICAIL, pp. 91–95 (2011)
Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., Witten, I.H.: The WEKA data mining software: an update. SIGKDD Explor. 11(1), 10–18 (2009)
Acknowledgements
This research was supported by the Alberta Innovates Centre for Machine Learning (AICML) and the iCORE division of Alberta Innovates Technology Futures.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Kim, MY., Xu, Y., Goebel, R., Satoh, K. (2014). Answering Yes/No Questions in Legal Bar Exams. In: Nakano, Y., Satoh, K., Bekki, D. (eds) New Frontiers in Artificial Intelligence. JSAI-isAI 2013. Lecture Notes in Computer Science(), vol 8417. Springer, Cham. https://doi.org/10.1007/978-3-319-10061-6_14
Download citation
DOI: https://doi.org/10.1007/978-3-319-10061-6_14
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-10060-9
Online ISBN: 978-3-319-10061-6
eBook Packages: Computer ScienceComputer Science (R0)