"Do you understand what I'm signing to you?" This is the title of your project, what do you want to achieve with it?
My first goal is to develop algorithms (also called models) for recognizing signs of Polish Sign Language (PJM). This leads to the development of sign language translators, which do not have a commonly used written form. The second goal is to build a database. Initially, I planned to ask only Deaf people for help, for whom PJM is a native language. In reality, however, it is not easy to collect the necessary amount of data with their help only, because artificial intelligence easily adapts to the shape of the hand, thus not coping with a previously unseen hand. Therefore, from many different people. For this reason, I asked my family and friends for help, for whom PJM was new. I wanted to check if there is a chance to train a model with the help of people who have never used sign language but will be constantly learning signs and showing them - and we have many more such people.
What are the prospects for the development of this research?
Broad, because it is a niche topic, and at the same time difficult. Currently, I am dealing with a single aspect, which is the recognition of signs in isolation. By this I mean that only 1 sign is shown to the model at the same time. This greatly simplifies the task and although it is an important stage, in reality signing looks completely different. Above all, signs are shown in dynamic sequences and due to the non-linearity of the language, several pieces of information can be conveyed at the same time. This requires the development of models that can divide them. There are also problems with data collection. When we have a single sign, people learn them in a moment. In the case of whole sentences, it is not that simple.
Where will the solution you are developing be applied?
The first area is sign language translators. Although my research is focused around PJM, the solution will probably be applicable to any sign language. However, it will not be that simple, because due to differences in signs or grammar (each sign language is a separate language) a separate database will have to be collected for each of the languages, which is one of the most difficult parts of the project. The next area is gesture control.
Although from a linguistic perspective gestures and sign language signs are two separate concepts, in both cases the movement and shape of the hand matter, so from the perspective of artificial intelligence these are very similar issues. Sign languages, due to their complexity, are a good experimental field. Of course, there are already solutions on the market that allow for gesture control (motion games or cameras in phones), but these are not systems that allow for complex sequences.
The E2Top program gives a chance to develop under the guidance of a mentor. Who did you choose?
Piotr Duch, PhD whom I met in the 1st semester. During these classes he proved to be a person who easily conveys knowledge. When I found out that he would be one of the guardians and in the subject in which I already wanted to develop at that time, I immediately decided to send an application. After over 1.5 years of cooperation with Dr. Duch, I am even more convinced of these features.
What inspires you in scientific work?
On the one hand, I love exploring the world and discovering how different things work in it. Thanks to scientific work, with each passing day I learn more and more about the possibilities of artificial intelligence. Realizing that even the most complicated models are based on simple operations is amazing.
On the other hand, I am inspired by the fact of developing something new and solving subsequent problems. Of course, sometimes it is frustrating, but all bad emotions are compensated by seeing that the model is starting to work better and better.
Interviewed by: Agnieszka Garcarek-Sikorska