Final post

You should write a 1000 word commentary in which you reflect about different ways of combining different methods in order to answer complex research questions.

Simply diving into complex research question is not trivial, there are many aspects to consider and thoroughly think through before being able to make real progress. One of the first steps you have to take before getting started with your own research is to try and get a good grasp on what has been done within the area of your choice. A great way of going about this is to gain knowledge by reading research papers that are relevant to the area, and try to find strengths and weaknesses with the methods used in the papers you read. By doing so, one might find that some methods are better for the kind of research you intend to carry out. As is the same with a lot of research, the most important kind of knowledge to gain before being able to start answering complex research questions is a posteriori - the kind of knowledge that is known on the basis of experience. Relating to previous research is very important when trying to dig deeper into their theories and results. This, of course, is rather hard to do if the research you want to conduct is at the forefront of the field, or perhaps in a brand new research area. Something that is excellent for that kind of research is a case study. A case study does not have the main purpose of confirming or adding to already existing theories, but rather to spawn new theories or find new research questions more suitable for another form of method.

Building on the theme of creating new research question and spawning new theories, using qualitative methods in the form of a case study is a great way to gain new insights. Going with the method of having semi-structured interviews gives the researcher the opportunity to "go with the flow" so to speak, as the interviewer has every possibility to go onto a side-track that they find interesting during the interview. This action alone may be inspiration enough to spawn a new research question, as you never know just what your interviewee may answer to your questions. An open environment is very helpful for this sort of research, and seeing how very important it is to formulate a research question, this is a great method of exploring a new research field. One thing to think of when conducting exploratory research as described is to try to refrain from doing lots of quantitative research (which may prove challenging at this stage) and instead focus your studies on a smaller number of test subjects, going deep into their experiences and thoughts to form new theories.

When the "pre-studies" have been made and you have formed a suitable research question, thinking about the way to go about your contribution to knowledge is always a good idea. Combining different methods is usually a great way to go. Both qualitative and quantitative studies can be used here. By collecting multiple sources of data during the studies, building explanations and comparing across cases, the validity of the research is increased according to Yin (2009). This could mean that you gather some data that is purely quantitative. Gathering news headlines over a set time period or tracking the number of times a website's links are visited are examples of this sort of data. Data is also only what it is until some form of analysis has been done. A theory can not be formed with only data. We can do lots of tests and gather tons of data, but only by explaining why the tests end like they do, we can create a theory according to Sutton (1995). Furthermore, combining quantitative and qualitative methods can help the researcher in forming a respectable and valid theory. Qualitative methods are often used to give some sort of explanation to the data gathered from the quantitative methods used. For instance, a study could be made based on the interaction by users as they use a smartphone application. A simple study could be formed by creating two versions of the application and letting two user groups use them separately. The difference could be based on a different design, or perhaps an added functionality. This method is known as A/B-testing or split testing and is meant to give insight into what works best for a certain scenario, webpage or application. Quantitative data could then be gathered by logging the users' activity within the applications, to see if the difference actually made a difference in the usage pattern for the users. By doing so, you may find that one version of the application was used to a greater extent than the other one, which then could be the quantitative data of the study. At this point, the research is not really research as there is no explanation as to why one version was more used than the other. In order to understand the data, one could use qualitative methods, perhaps in the form of interviews with the users. Before the interview, you could take a look at your quantitative data first, to try and understand it. By doing to, you may find it easier to form the questions you want answered during your interview. Although keep in mind that it may be a good idea to do some form of pilot study before conducting the interviews that will be the base of your theory-forming. It is not easy to know exactly what it is you want answered, although hopefully you may find out what that is during the pilot study.

One last point I want to make on this topic of complex research questions is that you have to keep in mind that what works in some fields of research may not be at all suitable in another one. By doing the mandatory literature studies before you start your own research, you will find which types of methods are used within your research area and may be able to build upon it in order to conduct a valid study.

After: Theme 6

This week's theme was the final one, Qualitative and case study research. The qualitative part was again a big part of the research on methods we did for the bachelor's thesis, so I did not really have any problems understanding the concept of qualitative studies. However, I did get some new knowledge during the seminar this time, especially in relation to case studies which is a form of qualitative research.

This brings us to the paper that I chose for this week's assignments. Luckily I found one that actually fit both parts of the assignment as it was a case study. The first thing I reacted to was how large the pilot study actually was. The case study only had nine participants, and the pilot study had five, more than half of the actual study! Naturally, the results of the pilot study was not used in the research, but was only used to give the final shape of the case study. I had no idea that the pre-work, or the pilot study part of qualitative research (perhaps all research?) was such a large part of it. During the seminar, Ilias explained that this is not uncommon but rather the norm when it comes to qualitative research - something that I did not know before.

He also had a form of a mini lecture on the differences between case studies and regular qualitative research. During that lecture, I learned that case studies are usually used for spawning new theories, or coming up with a research question, while qualitative research usually tends to confirm or add to an already existing theory.

I also learned a few things about some qualitative methods, especially interviews. Something that I had not really thought about before was how much information an interviewer is able to gather from an interviewee just by observing the way that they answer a question and not just by reading the answer. To be able to see their face and hear their voice gives a much deeper understanding to the answers, and also gives the interviewer the chance to steer and customize the interview in a semi-structured interview. In a semi-structured interview, this is great as you normally do not prepare as much since the interview is more free-flowing and gives the interviewer a lot more freedom in which questions he or she asks.

This is a common aspect of qualitative research as well, being able to customize the research method. I have learned that qualitative methods are really flexible. The researcher can design the methods almost any way they want, as long as they can motivate the form of their method. Otherwise, the research may not be taken seriously.

After: Theme 5

The theme for this week was design research and was a bit related to the first themes on knowledge, as we discussed whether or not practical design work in itself can be considered a "knowledge contribution". If I recall correctly, the first question we discussed on the seminar was regarding the empirical data in the two papers we read. I found it quite interesting that the author of one of the papers actually held the seminar so it was really interesting to hear his opinions as well as everyone else's regarding the tasks for this week, mostly for the reason that I suppose his opinions on his own paper was a bit more educated than our own.

Regarding how practical design work can be considered "knowledge contribution", we had some difficulties reaching some kind of consensus, however we all agreed that the contribution is extremely specific both regarding the time, what tools were used, in what kind of area the research had been conducted, et cetera. In my initial post, I wrote that practical design work of course can be considered knowledge contribution. However, I had not given much though to exactly how specific the knowledge contribution actually is. After the seminar, I do stand by my statement that it can be knowledge contribution, although very specific and it may not mean much in a couple of years. Although, if the research do not reach any conclusions about the concept itself by doing it practically, at least the contribution to knowledge is the way you carried out your design work and that by itself could be of great help for other researchers and/or designers when they want to try out similar concepts.

During the seminar, we discussed something called grounded theory and how tightly connected it is to practical design research. This was very obvious as Anders Lundström described the concept of grounded theories and how it is very much based on empirical data retrieved by case studies, prototype testing and other forms of empirical data collection. These methods are often used design research, something that I learned during the seminar.

Building on the previous paragraph, we also came back a bit to another earlier theme that was about what a theory is and what it is not. What I am thinking of was a discussion we had with Anders where we talked about how the empirical data retrieved during practical design research always is analyzed to form a theory, which also makes the research actually research and not just presentation of data, which is something that I had not reflected upon before the seminar.

Before: Theme 6

For this assignment, I looked around a lot to find a suitable paper to base the questions on. I had a hard time finding a paper that would work for this assignment as I wasted a lot of time just trying to find one that actually used qualitative methods. Finally, I found the paper Game-Play Breakdowns and Breakthroughs: Exploring the Relationship Between Action, Understanding, and Involvement, published in the journal Human–Computer Interaction, Volume 30, 2015 - Issue 3-4 which has an impact factor of 3.7.

Which qualitative method or methods are used in the paper? Which are the benefits and limitations of using these methods?
In the abstract, the following sentence is found: Data included interviews, multiple observations of game-play, postplay cued interviews, and diary entries. The methods used are described quite well in the sentence. The authors first developed a set of theoretical claims representing suggested relationships between involvement and learning on the basis of previous literature, and later on assessed them though a critical examination of the data set retrieved though the previously mentioned methods. As the authors set out to find the relationship between user involvement and learning in games, the main benefit from using these methods is that the authors can get a deeper understanding of the relationship by not just collecting quantitative data which by itself does not really say much about why the data looks like it does. One of the main drawbacks is that it is quite hard to generalize the data as the participant count was rather low for the study, as only nine participants took part. However, it seems the participants differed in terms of age, gender and how they identified as gamers, which is great to try and make some form of generalization. Worth noting is that the aim with a case-study approach is not to make statistical generalization but to make analytical generalizations that expand theories.


What did you learn about qualitative methods from reading the paper?
I learned that by collecting multiple sources of data during the studies, building explanations and comparing across cases validity is increased, according to Yin (2009). I also learned that qualitative studies is not very common when it comes to HCI research, as I did look through a bunch of papers before I found this one, all of which only used quantitative methods. The authors did pilot studies with five participants before the actual study started in order to ensure that the lab and data collection were set up appropriately. The results of those sessions were not included in the main study. I had not really thought of pilot studies that much before, so that was new to me.


Which are the main methodological problems of the study? How could the use of the qualitative method or methods have been improved?
Of course, the main problem here is that the number of participants was quite low which means that the results could have been skewed as it is hard to generalize the results of a population using such a small subset of participants.


Luckily, the paper that I chose for the first part of this assignment actually uses case studies, so I will continue the rest of this blog post with the same paper.

Briefly explain to a first year university student what a case study is.
A case study is something that is used to research to understand a certain situation instead of using loads of statistical data to try and form theories. It is useful for trying out theoretical models by using them in real world situations. It is not used to answer a question or explain a situation in its whole, but rather get more knowledge into the area in order to allow further elaboration and hypothesis creation on a subject.


Use the "Process of Building Theory from Case Study Research" (Eisenhardt, summarized in Table 1) to analyze the strengths and weaknesses of your selected paper.
I found the table from the paper to be overly complex in order to analyze the strengths and weaknesses of the paper I selected. Going through every step and analyzing the paper would also likely be way more than would fit into the 800 words of which I only have a few remaining. I suggest that this assignment could be moved into a single blog post in the future, were the task to remain. Regarding the strengths and weaknesses of my selected paper, see the previous questions in this blog post.

After: Theme 4

This week's theme has been a bit different since we have not yet had the seminar to discuss what we have read and the theme of the week, which makes this reflection a bit harder to write when compared to the other themes.

During this week, a lot of the concepts that was part of the theme was also used extensively when I wrote my bachelor's thesis, and I believe everyone who is taking this course and did their bachelor at KTH would feel the same. Comparing and discussing pros and cons when it comes to qualitative and quantitative methods is very much a big part of the bachelor's thesis as you research methods and compare the two, so there was not really that much new material to soak up this week.

Basically, we all know that it is hard to get objective data, and that choosing a method very much depends on the research you are trying to conduct as all methods do not fit all research. Often, it is advantageous to combine both qualitative and quantitative methods to try and make the end result as reliable as possible.

By reading the paper on immersive virtual reality, I got some analytical practice when it comes to analyzing a paper with drawbacks and advantages of the methods used. I would not say that I in any way have changed my understanding of the two concepts since I wrote my bachelor's thesis, however this has been a good refreshment of my knowledge, as I will soon embark upon the journey that is going to be my master's thesis during the spring.

Not having discussed anything as the seminar was cancelled sort of makes this reflection a bit empty and shorter than usual, but I hope that is okay.