ERC Consolidator Award: Future Social Interaction in XR

The future of XR is social.

There are no words for how happy I am to announce I’ve been awarded an ERC Consolidator Grant called FUSION with €1.98 million in funding to pursue my vision for the future of social XR.  FUSION proposes new techniques to fuse social signals between individuals and across realities, hypothesizing that interaction stability in XR is a signal we can measure and manipulate.  If we can make social interaction in XR feel good, it will forever change the way we connect, collaborate, and share our lives with each other.   

Behind every success is a long line of failures, of ideas lovingly crafted and thrown away, and many stressful days.  Every academic will tell you to keep applying and be resilient, but that is easier said than done and comes at great personal cost. In the ACM Code of Ethics and Professional Conduct, the statement that always speaks to me is “respect the work required to produce new ideas.”  When it’s crunch time with deadlines, when we’re doing peer review, and when we’re forced to work in a resource constrained environment, it can be a challenging practice to respect the work it takes to produce new ideas.  Be kind to your colleagues, be kind to anonymous peers whose work you review, and be kind to yourself.  

I want to celebrate today by also sharing and celebrating the failures. I’ve submitted seven rejected fellowship applications since beginning my academic career in 2014, and more rejected standard grant applications than I can count on two hands. I see a significant progression in the quality of my ideas and work over these years, but I also see the care and love that went into the early ideas, the sadness of throwing them away, and the struggle of going back to the drawing board and trying something new.

As part of my dedication to open science, I am sharing here all the past fellowship proposals that led to this success, so the invisible work of failure is visible with this happy announcement.

ERC Consolidator Grant 2023 – Accepted after Interview

The ideas behind FUSION have been the focus of my work since my first projects using social signal processing to model behaviour around public displays in 2014. This transformed after a serendipitous connection through volunteering, where my experience in social signal processing could be applied to modelling behaviour in virtual environments. I made significant revisions to my proposal after interviewing for ERC in 2022. I learned how much more concrete and detailed the work plan needed to be, which must be balanced against clear research questions. Conceptual integrity, leading to an ambitious and single outcome, was the biggest challenge in writing this successful proposal.

EPSRC Open Fellowship 2023 with Plus Component – Rejected after Interview

SocialXR was another revision of the rejected 2022 ERC submission, and I was excited to add the Plus Component, which focused on open science and inclusive publication policy. The Plus Component should be an opportunity for more diverse contributions to science to be recognised, but I was frustrated with the panel feedback that my Plus Component should have only focused on my specific discipline, and should not have aspired to go beyond computing. I was very discouraged by the process around the Plus Component, and I do not think I would apply for this again or recommend others to either.

ERC Consolidator Grant 2022 – Rejected after Interview

FUSION was the first interview I was granted for a fellowship application. Although I was unsuccessful, I learned so much through the process and had a really good experience. In preparing for the interview, it became clear my work plan was too high level. I had a clear vision of “what” I wanted to do, but not “how.” I knew I could revise this work and make it better. I knew I was finally on the right track and really had an idea worth picking myself back up for.

EPSRC Fellowship 2021 – Rejected without Interview

FIX was a clear progression from the first immersive technologies proposals, but I can see that some of the ideas are still somewhat unformed and in places half-baked. I still believe in these key concepts, particularly around mediated perception and the ethics of immersion, but these ideas were not so well developed yet.

ERC Starting Grant 2019 – Rejected without Interview

V-Real tried to further refine the first immersive ideas, but in hindsight it’s clear there’s no core concept tying it all together yet. I threw together input techniques, my favourite concept around manipulated perception, and my track record on social acceptance into one proposal. There’s low conceptual integrity, but it was a step in the right direction.

EPSRC Fellowship 2018 – Rejected without Interview

SituatedVR was the beginning of my shift towards immersive technologies. The work was not well received, and I was still very new to the field. Since finishing my PhD, I had changed topics frequently, and struggled to find a good fit and establish myself in any one area. Until I found immersive technologies, which give me such intrinsic joy I’ve never looked back.

ERC Starting Grant 2016 – Rejected without Interview

EMIT was my first ERC Grant application, and I was still working in public displays and public interaction. This was not well received, but writing was a good exercise.

EPSRC Fellowship – Rejected without Interview

DIPS was my first fellowship application, and my first big application after being successful in the EPSRC First Grant Scheme. The First Grant Scheme at that time was capped at £120,000, so it was a small project. Even though I was technically “on the ladder” at this point, it would be many years until I was able to translate my First Grant Scheme success into a successful full-size grant application.



Fellowship Applications History by Julie Williamson is licensed under CC BY-NC-ND 4.0

Being Social in XR with Stable Interpersonal Realities

If we do not solve the challenge of maintaining stability in interpersonal relationships in XR, we will not realise the potential of social XR in the future.

This talk, given at TU Delft during Prof Pablo Cesar’s Inaugural Lecture celebration, discusses my previous work and current open challenges in creating stable interpersonal realities.

Future Immersive Interaction @CHI2022

We have a busy CHI planned with a full papers, poster paper, and alt.chi paper published from the group this year. I’ve also been busy as Papers Chair for 2022, and looking forward to continuing this job in 2023.

We evaluated a consensus seeking task in a virtual environment accessed via an HMD or a desktop PC.

With colleagues from University of Glasgow, CWI, Toyota Research, and Northeastern University, we published our latest work on digital proxemics, available Open Access in the ACM DL. Code and Data are also available on GitHub.

Digital Proxemics: Designing Social and Collaborative Interaction in Virtual Environments

April 2022pp 1-12 https://doi.org/10.1145/3491102.3517594 Behaviour in virtual environments might be informed by our experiences in physical environments, but virtual environments are not constrained by the same physical, perceptual, or social cues. Instead of replicating the properties of physical spaces, ..

Laura will be presenting her poster on “reality anchors,” research looking at what anchors users need from reality to contextualise virtuality.

“Reality Anchors”: Bringing Cues from Reality into VR on Public Transport to Alleviate Safety and Comfort Concerns

April 2022pp 1-6 https://doi.org/10.1145/3491101.3519696Virtual Reality (VR) headsets have the unique capability to create private virtual content anywhere around the user, going beyond the capacities of traditional devices, but are not widely used while travelling, due to safety and comfort concerns. …

Ross will be presenting his speculative project on the future of always on XR in the alt.chi track, in collaboration with the Neil McDonnell in the School of Philosophy.

When Virtuality Surpasses Reality: Possible Futures of Ubiquitous XR

April 2022pp 1-8 https://doi.org/10.1145/3491101.3516396 Is our future heading towards enhancing the human experience with computer-mediated reality? Immersive technology is unique, existing between the world and our senses, letting users traverse wholly virtual environments (i.e. distant places or fantasy …

VR for Public and Social Settings

I was excited to give this talk for Professor Tony Tang at University of Toronto. In this lecture, I discuss my vision for the future of always-on immersive devices and the projects I’ve been working on for VR for social and public settings.

Sources On How to Chair Meetings and Develop Agendas

I’ve spent a lot a time lately thinking about how to improve my abilities as a meeting chair and facilitator of boardroom-style discussions. I have had some great discussions with trusted colleagues and mentors, thank you to everyone who’s given me their time and thoughts. I’ve also been reading some interesting sources:

How to Be a Better Meeting Chairman – It might be an older article, but the advice is as relevant as ever. Harvard Business Review, 1969, Limited Number of Articles Available Free with Login.
Making Things Happen – Focused on project management, but with a lot of insights into modern organisations (author Scott Berkun has extensive experience at Microsoft). Book available widely, list price $39.99.
How to Win Friends and Influence People – Unfairly teased as a self help book, it’s amazingly relevant and practical. A classic text which I enjoyed as an audio book, and the voice of Andrew MacMillan was fabulous.
My recent experiences have really put me to the test, including chairing my first meeting as the SIGCHI VP for Publications and chairing a discussion at the ACM Publications Board on the Policy for Name Changes in the Digital Library. 

The former was challenging because ACM publications are currently undergoing significant changes, and organising a productive meeting on this topic and establishing an infrastructure to ease these changes has not been trivial.  The latter was challenging because there are competing needs on this issue, and I knew that there would be passionately opposed viewpoints around the table.   I found facilitating this discussion was also challenging because of my own strong opinions (in favour of a liberal name change policy) and my position as a comparatively junior member of the board.

 

Opinion: Current Thoughts on the Open Letter Opposing Zero Embargoes on Publicly Funded US Research

This is how I wish the ACM had responded to the community about the open letter opposing zero embargoes on publicly funded US research.

(Note: This opinion is my own and in no way represents an official statement from ACM.  I speak from my own experience as a volunteer in ACM SIGCHI, the ACM Publications Board, the ACM Future of Computing Academy, the ACM Europe Council.)

ACM leadership apologises to the community for signing the open letter to the Office of Science and Technology Policy (an office of the US Executive Branch headed by a political appointee) without wider consultation from the ACM community. In the future, ACM will avoid signing open letters without consultation with ACM Boards and Councils.  This is especially true when signatures involve coalitions with for-profit publishers.

First and foremost, we would like to emphasise that ACM is very supportive of open access and open science initiatives. As a non-profit organisation and professional society, ACM supports a wide range of open access initiatives and is already preparing for a financially sustainable future where an increasing majority of the ACM Digital Library is available as gold open access.  An especially exciting initiative currently being developed is ACM OPEN, which supports Gold Open access.

The key reason ACM signed this letter is to oppose an Executive Order in the United States which would enforce zero embargo periods for access to publicly funded American research.  This is seriously concerning for two reasons.  Firstly, a zero embargo period would interfere with or make impossible some of ACM’s ongoing open access initiatives and threaten the financial stability of ACM as a non-profit publisher. Secondly, enforcing such an initiative through an Executive Order does not require public discourse and, in this case, has not considered how such an order would affect non-profit publishers or consulted with these stakeholders.

For the ACM community, the scope of this letter is crucially important.  This letter concerns specific implementation details of open access policy for publicly funded US research only. The content of this letter should be considered within this context only, and not interpreted to apply to other contexts or other regions. For example, ACM is working with policy-makers to develop a financially sustainable way to implement Plan S and open access research in Europe.

Considering the context of this letter, we accept that some aspects of this letter are deeply problematic.  Based on feedback from the community, we recognise that many of the statements in this letter go well beyond our specific concern around enforcing zero embargo periods through an Executive Order.  We hope that this statement makes clearer our original intentions when signing this letter, and hope that we can continue to work together with our author community towards a sustainable open access publishing model.

(Note: This opinion is my own and in no way represents an official statement from ACM.  I speak from my own experience as a volunteer in ACM SIGCHI, the ACM Publications Board, the ACM Future of Computing Academy, the ACM Europe Council.)

If you are interested, here is some additional material about this issue.

The Open Letter

ACM’s Official Response

Commentary from Robert Harrington 

Content Light Homepage of the OSTP

 

Performative Interaction @ CHI 2019

The research group has social acceptability, group experiences, and virtual reality on the mind at ACM CHI 2019.

Along with colleagues from Ulm University, LMU Munich, Universität Hamburg, and NYU, we organised a workshop on the challenges for using immersive headsets in public and social settings.  As part of the workshop, we set out in Glasgow to get some first hand experience.

Workshop participants try Oculus Go in a busy restaurant.

Workshop participants try Oculus Go in a busy restaurant.

You can access our workshop abstract here:

ACM DL Author-ize serviceChallenges Using Head-Mounted Displays in Shared and Social Spaces

Jan Gugenheimer, Christian Mai, Mark McGill, Julie Williamson, Frank Steinicke, Ken Perlin
CHI EA ’19 Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, 2019

We also presented our work on the social acceptance using virtual reality headsets while travelling and presented new techniques for improving user comfort and acceptance of these devices using mixed reality techniques.

VRComposite

You can access the full text here:

ACM DL Author-ize servicePlaneVR: Social Acceptability of Virtual Reality for Aeroplane Passengers

Julie R. Williamson, Mark McGill, Khari Outram
CHI ’19 Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 2019

FET Open Levitate

Scientists at the University of Glasgow, UK, have managed to suspend little polystyrene particles in mid-air, supported only by ultrasonic acoustic waves. This is levitation. The technology may lead to new kinds of displays to command machines and hence revolutionise human-machine interactions. The study runs under the Levitate project, supported by the research programme on Future and Emerging Technologies of the European Commission.

Familiar Stranger Army @ The Fringe

A new performance piece melding live theatre and virtual reality has just opened as part of Army@TheFringe.

Familiar Stranger brings together live acting and virtual reality to tell the story of an Iraq War veteran returning to civilian life.

The 45-minute show is a collaboration between The University of Glasgow Department of Computing Science and Glasgow-based artist and coder collective RealRealReal.

Hosted in the Hepburn House Army Reserve Centre (Venue 210) in East Claremont Street, Edinburgh, it offers an insight into a veteran’s attempts to reintegrate into everyday life.

It opens with a monologue performed by career soldier Sergeant Major Garry Worrall, after which audiences are introduced to the Oculus Go VR headsets that plunge them into a virtual space, and the veteran’s inner life.

It ranges through his home and then into his memories of deployment – an experience that is simultaneously familiar and strange.

Afterwards the audience meet Garry again and have the chance to talk to him about his experiences in and out of the Army – opening up the space between the artists’ ideas of army life and his first hand knowledge.

Dr Julie Williamson, Lecturer in Human Computer Interaction at the university’s School of Computing Science, developed the technical set up and collaborated on the script.

She said:“Virtual reality is often considered a solitary activity, but I’m interested in exploring how we can use virtual spaces to expand shared experiences.

“Working with Dennis Reinmuller and Debbie Moody from RealRealReal and Army@TheFringe has given us a great opportunity to explore how theatre can be melded with VR to create an experience that can’t be delivered any other way.”

Familiar Stranger features the voice of Louise Oliver as The Magazine Soldier, guiding the viewer through the fictional veteran’s memory. The music is created by Sarah J Stanley of HQFU together with RealRealReal.

Army@TheFringe is presented by Army Headquarters Scotland as a way of engaging with wider society through the arts and initiating discussion about soldiering.

The venue, which runs from 10 to 25 August, is staffed by soldiers who run the bar and front of house services, and who mingle with the public before and after shows.

Familiar Stranger if supported by the University of Glasgow’s Dean’ Fund.

It runs until 24 August with performances at 1pm, 3.45pm and 6.45pm daily.

Public and Performative Interaction @ CHI 2018

We are presenting two papers at ACM CHI 2018.  The links below will provide an open access copy of these papers.

Object Manipulation in Virtual Reality Under Increasing Levels of Translational Gain

Room-scale Virtual Reality (VR) has become an affordable consumer reality, with applications ranging from entertainment to productivity. However, the limited physical space available for room-scale VR in the typical home or office environment poses a significant problem. To solve this, physical spaces can be extended by amplifying the mapping of physical to virtual movement (translational gain). Although amplified movement has been used since the earliest days of VR, little is known about how it influences reach-based interactions with virtual objects, now a standard feature of consumer VR. Consequently, this paper explores the picking and placing of virtual objects in VR for the first time, with translational gains of between 1x (a one-to-one mapping of a 3.5m*3.5m virtual space to the same sized physical space) and 3x (10.5m*10.5m virtual mapped to 3.5m*3.5m physical). Results show that reaching accuracy is maintained for up to 2x gain, however going beyond this diminishes accuracy and increases simulator sickness and perceived workload. We suggest gain levels of 1.5x to 1.75x can be utilized without compromising the usability of a VR task, significantly expanding the bounds of interactive room-scale VR.

Acoustic levitation enables a radical new type of human-computer interface composed of small levitating objects. For the first time, we investigate the selection of such objects, an important part of interaction with a levitating object display. We present Point-and-Shake, a mid-air pointing interaction for selecting levitating objects, with feedback given through object movement. We describe the implementation of this technique and present two user studies that evaluate it. The first study found that users could accurately (96%) and quickly (4.1s) select objects by pointing at them. The second study found that users were able to accurately (95%) and quickly (3s) select occluded objects. These results show that Point-and-Shake is an effective way of initiating interaction with levitating object displays.

ACM DL Author-ize servicePoint-and-Shake: Selecting from Levitating Object Displays

Euan Freeman, Julie Williamson, Sriram Subramanian, Stephen Brewster
CHI ’18 Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, 2018