my papers

Getting a paper accepted into an academic journal requires passage through the often opaque process of peer-review. Scientists make a big deal of peer-review, because it is supposed to be the filter that separates mere opinions from trusted, citable sources. However, the peer-review process in science has close analogs in any “old-media” field, such as TV or radio. Like academic journals, these are mediums of limited capacity, and there are always more submissions (or ideas for submissions) than there are openings.  Selecting content worthy enough for distribution is made by the field’s establishment, which effectively silences what they don’t choose. This is especially true of peer-review as practiced in prestigious journals, defined as the ones that get their contributors faculty jobs.

He goes on with explaining how GitHub changed the way Open Source Software is developed and how contributing gems to the Rails community gives the contributor prestige and reputation.

Similarly, anyone can publish a gem to GitHub, and published gems are prioritized by the numbers of developers “watching” for updates or “forking ” new development lines.  This is the social web at work, where the audience gets to decide what and whom to pay attention to all by itself, without requiring assistance from all-powerful editorial committees.  One can complain that lowering barriers to publication leads to content that on average is of lower quality.  But the abundance of non-significant projects in GitHub does not detract from its usability, because those projects are never brought to anyone’s attention.

Finally, he also talks about an important factor for developers to participate in Open Source Software development and contributions to the community.

Gems on GitHub are not just code.  They also have authors whose relative contributions are automatically catalogued by Git, as shown in this impact graph for the popular and open source jQuery project.  If you’ve visited a web application recently, chances are you’ve benefitted from jQuery, which makes it easy for a web engineer to turn static web sites into responsive web applications (think interactions with buttons instead of navigation through links).  This impact graph can let you know precisely which developers are responsible for this awesome-ness. In this way, GitHub acts as an efficient, incorruptible “central bank” of the prestige supply. Furthermore, unlike in Google, great contributions in GitHub bring prestige to their creators, not their domain names.  If you wanted to hire a contractor to work on a web application, GitHub can let you know who has publicly demonstrated the skills you’d need.  It’s thus not surprising that GitHub profiles are supplanting traditional resume items, such as a CS degree, for discerning employers looking to hire top talent.

I’m wondering if we see some of those GitHub features in OpenScience in the next years. Maybe the STELLAR dataTEL theme team will come up with an usable solution for sharing TEL data sets, that others could reuse and thus give reputation to the contributors of the data set. Maybe we will see real Open Archives of publications that will provide Open Data of the usage metrics. And finally I hope to see a real change in how scholarly communication as a whole is taking place – without content sitting behind paywalls but still being considered for your “professional record”. For me it’s a shame that it’s still the fact that

[...] at the beginning of an academic’s career, before citations accumulate, reputation rests largely on what journals they have published in.

In my case I was interested in the topic “open science” and this is what came out. Cool, right?

Tonight we started the closed beta of ginkgo, a cool new tool for the management of scientific events.

ginkgo is no classical conference management system. It rather combines conference management functions with those known from social networking sites. The goal of ginkgo is to raise awareness of researchers, make the organization of scientific events easier and more fun. We are currently working hard on getting the last pieces together and providing you with a fresh approach to the management of scientific events pretty soon. In the meantime you might want to join the beta list, follow ginkgo on Twitter or Facebook or follow Julian and me on Twitter for latest updates.

Researchers are the archetype of knowledge workers.

For me this so obvious that it is even hard to argument for it. If you look at the typical job descriptions of researcher positions, they have to create new knowledge using a multitude of methods. Researchers are carrying out non-linear and non-routine tasks that require a lot of cognitive effort, combination and recontextualization. From my point of view that’s exactly how I would describe knowledge workers.

What is your take on this? Do you agree to the statement or do you think this is pure rubbish?

For the analyses shown in this article we used Artefact-Actor-Networks (Reinhardt, Moi & Varlemann, 2009, AANs) for obtaining and storing data from Twitter and Gephi for calculating metrics and visualizing the resulting networks. Gephi is an Open Source Software tailored for graph and network analysis. In AANs we distinguish between actor networks that represent each user in a learning service as node. In artifact networks each digital object (e.g. a tweet) is represented as a node. For every learning service that is used we analyze the interactions. If we are interested in a more holistic view about the ongoing interactions, consolidated artifact and actor networks can be analyzed. Moreover, with the AAN approach artifact and actor networks can be combined for even more sophisticated analyses.

Artifact-level analysis

To track interaction and communication in the seminar we asked the participants to use the tag #fsln10 for all course-related issues on Social Media channels. In total we analyzed 431 tweets, 384 Delicious bookmarks, 14 SlideShare documents and 61 documents on Mendeley. In this article we focus solely on the analysis of the social network that arose from the usage of Twitter during the seminar. From the 431 tweets we extracted over 80 keywords using semantic analysis services. Moreover, we extracted all other hashtags that were used. For analyzing the semantic relations of artifacts and actors we are able to apply different measures: the connectedness via shared keywords and hashtags as well as the semantic similarity of the artifacts. Figure 1 shows the artifact network of tweets that are connected via shared keywords or hashtags.

Figure 1: Artifact network of tweets connected via shared keywords and hashtags

The color and size of the node reflect the degree of the respective artifact: the darker and larger the node, the more connections it has to other tweets. The light and small nodes at the bottom left corner have no keyword connection to the other artifacts. Digging deeper in the artifact network, we can identify several clusters of artifacts that deal with special seminar topics. Using the semantic connections between the respective artifacts it is easy to find out how those clusters are formed: clusters between Twitter artifacts arise when they share common keywords or hashtags or are replies or retweets to each other. Figure 2 shows an example visual analysis of tweets connected via the keywords ‘mendeley’, ‘dropbox’, ‘mixxt’, and ‘skype’ which represent learning services used in the seminar. Figure 2 also depicts vertices with the labels ‘stellarnet’ and ‘aan’ referring to projects that were associated to the seminar. Finally, the Figure also shows that artifacts are connected with vertices labeled ‘hasReply’ and ‘isReply’ that indicate the existence of a reply connection.

Figure 2: Detailed inspection of connections between tweets

In its current version Gephi is not able to visualize multiple vertices between two nodes, leading to the impression that tweets are always only connected with one relation. Figure 3 tackles this problem and visualizes the artifact network based on the semantic similarity between tweets. An edge between two tweets thus represents their similarity – the thicker and more colored an edge, the higher the semantic similarity between the tweets. Compared to Figure 2 we do not consider replies and retweets for the calculation of semantic similarity as shown in Figure 3.

Figure 3: Artifact network of tweets connected via semantic similarity

What becomes obvious from the artifact visualizations is the fact that most of the tweets have no structural or semantic connection to other tweets (79% of the tweets are isolated). While this can partly be explained with the process of topic extraction that we applied in this study (it is difficult to extract keywords and meaningful content from only 140 characters), the missing structural connections point to an improvable interaction structure within the educational setting. Analyzing the social relations in actor networks allows us to apply metrics of classical social network analysis (SNA) to the seminar and to investigate the interaction patterns more directly.

Actor-level analysis

We will now investigate the actor network that was formed by the Twitter users that used the #fsln10 hashtag. In this analysis there is a connection between two actors if they have written similar tweets. The more similar the tweets are that they have written, the higher the weight of the edge between them. Regarding this setup we found 21 actors that could be connected in the actor network (9 seminar participants, 2 supervisors, 10 external discussion participants). One seminar participant does not show up in the actor network as he did not write tweets that were similar to others. If we had applied other measures for building up the network (e.g. the following relations, replies or retweets) he would have been included. The network has a density of 0,276, which is above average compared to other Twitter networks.

Figure 4: Actor network of Twitter users connected via semantic similarity

Considering the degree centrality in the actor network, the actor wollepb (a supervisor) is most central. wollepb is called a local hero, as the degree centrality calculates centrality only based on local network properties. A local hero attracts many direct connections on himself; in the context of the seminar he published many semantically similar tweets. The closeness centrality measure (Figure 5a) additionally incorporates indirect connections on shortest paths into the calculation. wollepb still is the most central actor but looses centrality with this measure. Finally, when considering the betweenness centrality (Figure 5b) the centrality of those actors that are on only few shortest paths in the network is declining. Users who act as mediators in the network are becoming more central with this measure. Two participants of the seminar (rolwi and AlexanderPLN) are thus also connecting the other seminar participants.

Figure 5a: Applying different centrality measures on the actor network (closeness centrality)

Figure 5b: Applying different centrality measures on the actor network (betweenness centrality)

Discussion

The impact of our evaluations is that for one we believe we have found a best practice example of how students can be confronted with state-of-the-art technologies and projects while still at university in a way that fosters skills and competencies they will need in the workplace. This type of interconnectedness between institutional, informal and workplace learning is focus of this endeavor. Further we see the possibility to integrate students with their fresh, creative and sometimes unconventional ideas in real-world research and industry projects, which can provide added value for both the students as well as the involved projects.

On the other hand our analyses show that we need to further engage students to use Social Media tools as a means for interacting within a seminar. Realizing that a supervisor is the most central person in a seminar is not surprising; the gap between the Twitter adoption of supervisors and students should give us food for thought on how to better motivate students to use the tools. Noticeably, the students used other tools that seem to be less public and without the need for direct interactions far more frequently. The students were heavily bookmarking web pages in Delicious and uploaded references to Mendeley. Also they used Skype for one-to-one interactions on a regular basis.

Also we see that the artifact networks are not very dense (meaning that the published tweets are rather broad in their contents). While this is partly because of the applied analysis method, it also points to the fact that it might be a good approach to narrow down the seminar topics to espouse a more lively topical exchange. Moreover, the analyses presented in this article were done subsequently to the seminar. In order to react in a timelier manner about the discussed phenomena we would need tools for instant social media monitoring in educational settings. Those tools should then obtain, analyze and visually present the results. Exploring the interactions and topical connectedness of students’ work will help us to better supervise learning designs that make extensive use of Social Media.

The course design has caught much attention during its first round of implementation (see the number of external persons that were involved in communication) and allowed to broaden the scope of the seminar. The seminar is being carried out again in the fall semester of 2010. In the summer term 2011 we will integrate a third partner from another German university into the seminar and focus on the design and development of a Social Media Monitoring tool for educational contexts as described above.

References

Jonassen, D.H. (1991). Objectivism versus constructivism: do we need a new philosophical paradigm? Educational Technology Research and Development, 39(3):5–14.

Jonassen, D.H. (1999). Designing constructivist learning environments. Instructional design theories and models: A new paradigm of instructional theory, 2:371–396.

W. Reinhardt, M. Moi, and T. Varlemann. Artefact-Actor-Networks as tie between social networks and artefact networks. In Proceedings of the 5th International ICST Conference on Collaborative Computing: Networking, Applications and Worksharing (CollaborateCom 2009), November 2009.

Acknowledgment

Thanks go to my graduate student Matthias Moi for visualizing the networks and applying SNA metrics to them.

As for the creation of HITs in MTurk I happily link to the blog post of Andy Daio. I followed is step-for-step instructions, reused his template HIT template and learned how to embed a Flash-based audio player that helps your workers to listen to the audio.

Altogether I had 13 interviews with about 11 hours of audio that should be transcribed. The interviews were in English and most of the interviewees were non-native speakers. I split the audio files in equally long parts (between 4:45 and 5:25) so that all workers had to process the same amount of the interview. Furthermore, I remove some organizational chatter at the beginning (“can you hear me?” “yes I can hear you” “thanks very much ….”) and the chatter at the end (“please keep me updated about the results” “sure Iwill, by the way how is your project xyz …”). I uploaded the audio files to a local web server and created the HITs as described in Andy’s post.

I assumed that the workers would need around 2 hours for a 5 minute audio file to transcribe (actually they were much faster) and payed $2 per completed assignment. I let the assignments be automatically approved after 7 days (you might want to change this).

My 120 audio files were transcribed in only about 14 hours. As I mentioned earlier, I’m more than satisfied with the results BUT there are some things that you should consider when doing this yourself:

• Decide on whether you want a verbose (including um, eh, pauses, duplicate words and so on) or non-verbose transcription. If you don’t tell the worker how to transcribe they will do want they like to.
• Give your workers some background about the audio they are to transcribe (it is only a part of an interview, the topic is XYZ and so on)
• When you split your audio files, try to create the files with 2 seconds of overlap, so you don’t miss any words that are right on the cut.
• Check your transcriptions as soon as they’re ready. I had a very busy week and couldn’t check the transcripts right away and only saw some very bad transcripts after they’ve been automatically accepted.
• Be aware that the transcripts may require a lot of manual polishing. Especially if you interview topic is not about public domain knowledge you should consider that the workers may not know what you are talking about and “guess” the right words.
• Workers will try to cheat. They will start to transcribe but not finish their tasks. Whenever you spot this, reject the worker’s transcription.
• Transcripts have varying quality. MTurk worker used to be Americans with an above-average level of university degree. With Amazon’s move to allow workers from other countries doing the job there are some indicators that the transcription quality is decreasing. You will notice those differences in one interview. While the first three parts of an interview may be perfectly transcribed, the next part could be full of [xx] or ?? where the worker didn’t understand the interviewer or just didn’t know the word.

Altogether I paid around $260 (€200) for the transcription of 11 hours of interviews. The whole process with splitting, creating the HITs, the workers’ transcription, my review and creation of aggregated transcripts took me about one week (you could be faster when you check the HITs more regularly and create your aggregated transcripts faster). Now I will need to rehear the interview and do the polishing of the transcripts.

Would I recommend using MTurk in your research? Oh yes, I’d do it every time again.

***************************************
Call for papers and demos

1st International Workshop on Awareness and Reflection in Research Communities (ARRC11)
to be held in conjunction with CSCW 2011, Hangzhou (China)

March 19-23, 2011

http://thales.cs.upb.de/arrc11

***************************************

*** Background ***
Traditionally the concept of awareness is used in the research field of CSCW to re-establish awareness conditions of face-to-face situations with visual cues showing for example, who is online or working on a document. In the field of perception psychology, awareness is the state or ability to focus on certain stimuli of the environment while ignoring others. Here, being aware of something does not necessarily mean to understand it. In marketing, awareness usually relates to the degree consumers knows about a certain product. Generating public awareness is seemed as a task of the media to establish topics the public should know about.
This workshop will discuss and build an interdisciplinary understanding of implicit and explicit structures researchers build through their practices using the Web. Considering the multitude of views on awareness we ask: “What does awareness mean for technology-enhanced learning (TEL)? How does it relate to reflection as one of the key abilities of researchers? While researchers are aware of their relations in their networks, technology establishes related activities which are beyond the individuals focus of attention. This is possible due to two advantages of computer-mediated communication: (1) selectively mirroring information that would be visible in face-to-face situations, and (2) mirroring information that would not be visible in face-to-face situations, but that can improve the working or learning process. This information could help researchers to become aware of their constantly changing connections and interactions beyond their individual context and to reflect this. This workshop aims to challenge the understanding of awareness and reflection in technology-enhanced learning.

*** Topic of interest ***
The ARRC workshop focuses on current research trends in the field of awareness and reflection in the domain of research communities. The workshop seeks to attract quality research papers that propose solutions to the issues identified above. The workshop also welcomes papers that comment how the application of social media can impact on real life experiences in such communities. It aims to bring together scientists, designers and engineers who work on designing and/or developing the above mentioned solutions, as well as practitioners who use and evaluate them in diverse authentic environments. We encourage authors to present their interdisciplinary results and such that focused on technical, psychological and/or pedagogical challenges.

# Different views on awareness
# Social awareness
# Knowledge awareness
# Artefact awareness
# Process awareness
# Context awareness
# Awareness-support in research communities
# Awareness in organizational learning
# Semantic Web and awareness
# Orchestration of awareness and reflection
# Awareness and reflection in online research environments
# Bridging formal and informal research environments and their consequences for awareness and reflection
# Activation of reflection processes
# Awareness of pivotal events and their reflection support

Authors are invited to submit original unpublished work. Including demonstrations is explicitly encouraged. The following contributions are possible:

# Short papers (6-10 pages) that state the position of the authors within the scope of the workshop and describe solution concepts and work in progress in a premature state.
# Full papers: (12-16 pages) that describe problems, needs, novel approaches and frameworks within the scope of the workshop. Empirical evaluation papers and industrial experience reports are welcome for submission.
# Posters and demo papers (1-2 pages) that summarize preliminary work results

All submitted contributions will be peer-reviewed by at least two members of the program committee for originality, significance and quality. The workshop proceedings will be published in CEUR-WS.org a publication series with ISSN. Furthermore selected papers will be part of a Special Issue in a well-known international Journal (currently in negotiations).

*** Workshop format ***
Papers presented at the workshop will need to engage participants in discussions on the presented key topics. To achieve this aim, all accepted papers will be clustered according to their topics in (2-4 clusters). Each such cluster will contain not more than 4 papers, each of which will be presented by one of its authors. The presentation of each paper will be 20 minutes. Authors will be asked to follow a structured template for their presentation (focusing on the problem addressed in their paper, the methodology followed, and the results of their work). After the short presentations, each cluster will have 90 minutes for discussion and preparation of a final round table discussion with all clusters. This closing session will summarize the outcomes of the single workshop sessions and determining future activities in the research field of awareness and activation of reflection in learning and research communities.

*** Dates ***
# December 17, 2010 >> deadline submission of 2 pages abstract
# December 30, 2010 >> paper acceptance notification
# January 07, 2011 >> Early bird registration for ARRC11 (please register for the workshop until this date).
# January 21, 2011 >> deadline for the full papers
# February 11, 2011 >> results of the review for the full papers
# February 25, 2011 >> deadline for camera-ready copies
# April 2011 >> publication of selected papers in a Journal’s Special Issue (currently in negotiations with several journals

*** Organizers ***
Wolfgang Reinhardt (University of Paderborn, Germany)
Nina Heinze, (Knowledge Media Research Center, Germany)
Thomas Ullmann, (Open University of the UK)

*** Program committee ***
# Ilona Buchem, Beuth University of Applied Sciences (Germany)
# Cristina Costa, University of Salford (UK)
# Erik Duval, KU Leuven (Belgium)
# Tanja Engelmann, Knowledge Media Research Center (Germany)
# Jean-Marie Favre, University of Grenoble (France)
# Angela Fessel, Know Center Graz (Austria)
# Mark Hahnel, Imperial College (UK)
# Jim Hensman, Coventry University (UK)
# Marco Kalz, Open University (The Netherlands)
# Reinhard Keil, University of Paderborn (Germany)
# Peter Kraker, Know Center Graz (Austria)
# Peter Scott, Open University (UK)
# Paulo Simões, Air Force (Portugal)
# Peter Sloep, Open University (The Netherlands)
# Carsten Ullrich, Shanghai Jiao Tung University (China)
# Riina Vuorikari, European Schoolnet (Belgium)
# Steve Wheeler, University of Plymouth (UK)
# Fridolin Wild, Open University (UK)
# Katrin Wodzicki, Knowledge Media Research Center (Germany)
# Martin Wolpers, FIT (Germany)

Today I submitted the first audio of an interview to the service. I allowed the workers two days of time for doing the transcription (52 minutes altogether, split in chunks of 5 minutes). I followed the superb tutorial of Andy Baio, reused his templates and even found out that there is a reusable MP3 player in Google Reader that I used to provide the workers with a Flash-based player for the audio chunks. If you ever wondered about the demographics of the service, the folks  someone from PARC provides a detailed overview about it (here is an earlier version of the demographics data).

I’ll tell you what happens, how satisfied I am and what it cost me in the end in a follow-up post…

Driven from the fact that many people in my community are collecting data from social media (you remember all of our Twitter analysis and the cool Delicious visualizations, don’t you?) and always struggling with the question: where to get clean data from, the idea of soharc was born.

What you can do in soharc is fairly simple: you create your online profile with the handles you possess and we’ll create a dedicated page for you. You can create “scientific” events or projects and join them, so everybody who’s interested can see what you are doing. In order to support scientific awareness and solve the problem of how to get clean data, we’ll be soon offering a SWRC export of your social media profile and those of an event’s or project’s participants. This will allow the aggregation of social media data for analyses with multiple tools, done by different people and organizations based on a well-defined and user-generated dataset.

Cool, right? Let me know what you think about the social handle archive in the comments. I’d like to thank Peter Kraker and Gonzalo Parra for the chats we had on soharc and their alpha tests of the service.

NB: SWRC export is not yet enabled as we are discussing a new version of the ontology together with researchers from STELLAR and FZI.

One of our preparation tasks was to prepare a mini Pecha Kucha talk consisting of 6 slides where each of the slides is shown exactly 30 seconds – no matter if you finished what you wanted to say or not. Cristina has uploaded her slides already here.

I hope people will understand what I’m about with Artefact-Actor-Networks and ask good questions. Here are my slides: