COST287-ConGAS WG2 Meeting No. 3

June 1-2, 2005, Leeds, U.K

Minutes

(draft 1, 6 June 2005)

 

Present: Kia Ng (in the chair), Nicola Bernardini, Thomas Moeslund, Andy Hunt

 

WG2_3.0 Welcome

The chair welcomed all participants to Leeds and to ICSRiM and the WG2 meeting started at 2pm on June 1^st, and continued at 9.00am on June 2^nd.  This was the 3^rd WG2 meeting:

§         1^st in Genova

§         2^nd in Leeds during GIMS 2004

§         3^rd in Leeds WG2 2005 (THIS MEETING)

§         4^th Leeds GIMS+FUSE2006 (potential next meeting)

 

WG2_3.1 Apologies

Apologies were received from:

Gunnar Johannsen, Loic Kessous, Marcelo Wanderley, Andy Hunt (for June 1^st)

 

WG2_3.2 Leeds FUSE 2006

A presentation on Leeds FUSE 2006 plan and organisation was given by

Kate Burrough, Project Development Manager of the Leeds City Council.

·        FUSE2006: 6-14 May 2006

·        10 May London Symphonietta concert

·        Potential GIMS2006 schedule

o       9 May 2006: paper sessions

o       10 May 2006: interactive workshops/demos/performances in bar-area, and foyer

o       To run from early PM until pre-concert time for the London Symphonietta concert in the evening, in the Yorkshire Playhouse

·        Issues raised

o       Access and availability to the Yorkshire Playhouse and the Colleague of Music (next door to the Yorkshire Playhouse) with a hall called “The Venue”

o       Action KB to keep KN with the development of the programme and confirm the booking

o       KN to send KB the logos and ConGAS full name for acknowledgement in all promotional/publicity materials

o       Research that can make use of the event to collect user interaction and other studies

o       Interactive Installation

o       Concerts/performances

o       GIMS Symposium: papers/talks/lectures

o       Interaction with the public

o       Hands-on workshop

 

WG2_3.3 Approval of Minutes

–        Last WG2 meeting 30 March 2004 which was co-located with AISB and GIMS2004

–        Matters arising:

•         Nicola asked IRCAM for permission to put the “Trends in Gestural Control of Music” on the ConGAS web-site but no success

•         Electronics links are now maintained under the WG2 website

 

WG2_3.4 WG2 Progress Report

KN gave a Progress Report with extended background and information from NB

–        Presented the events attended and events planned

–        Discuss the problem of WG2 earlier meeting (Jan 2005) which was cancelled

–        Discuss the requirements of focused activities particularly for WG2

–        NB explained the background of ConGAS WGs

–        Discussed the functions of WG and MC, parallel or individual

 

WG2_3.5 ConGAS Library (NB)

–        This is a good focused activity for WG2

–        NB proposed a general and open library to offer low level direct communication to all forms of input devices (e.g. USB, serial, parallel)

–        Write a library that consists of some layers to generalise gesture capture to PD.

–        Action: NB is to investigate if there is such a library for PD or not

–        Action: NB is to start the design and prototype in August

–        Action: ALL To propose protocol and spec to NB

–        STSMs could be used to help this activity by bringing people and devices together for effective development and evaluations

–        NB to investigate within the PD world on the existing of such modules

–        NB to draft a prototype during August

–        TM + KN + ALL: to provide/suggest protocols; machines that we have; open protocols; 

 

 

WG2_3.6 Visit to Yorkshire Playhouse and “The Venue”

The aim of this visit was to find out the space available for ConGAS GIMS2006 to be co-located with Leeds FUSE2006.

•         Playhouse main Hall is > 800 sites

•         Playhouse small hall is around 380 sites

•         Many coffee bar areas and foyer areas usable for interactive demo etc

•         “The Venue” College of Music – 350 sites

•         CONTACT: Henry Duckworth, Julie Rebbeck – Yorkshire Playhouse, Tel: 0113 2137800

 

WG2_3.7 Activity Plan

• Salerno SMC (2 days 22/23 Nov 2005, 1 day WGs, 1 day MC)
• Barcelona ICMC inc. Reactable Workshop, sponsored by ConGAS
• FUSE + ConGAS_GIMS2006 – 8-10 May 2006, + MC meeting
• MC May 8^th or 11^th (Action: Kia to confirm)
• GIMS 9^th and possibly 10^th
• Possible WGs on 10^th (a.m.) followed by event/installation

(NIME is at IRCAM in 1-3 June 2006)

 

AISB2006: agreed not to do this; replaced by FUSE.

 

It is good to have projects which focus the Working Groups

For WG2 this could be the PD ConGAS input library.

 

STSMs could be used to carry out the ConGAS PD Library project – for example gathering protocols from devices in use by ConGAS members.

 

 

WG2_3.8 Paper to MUSICNETWORK 2005

This paper was finalised and sent.

 

WG2_3.9 Scientific and Technical co-operation

• EC IST projects

COST projects are trying to demonstrate that good IST projects can come out of the action. So we need to report collaboration with other EU projects.  Please can people inform Nicola when a project is put together, or if specific collaboration takes place with other EU projects.

• Contacts with SMEs

It’s important that we get in contact with the manufacturers of the interface devices, and particularly with those of A-D devices and different interface sensors.

• PD ConGAS library – provides a good context for contacting SMEs
• Controller survey
• NIME – they also want to involve SMEs, so we could collaborate here. KN to collaborate with NIME.
• Medical Applications

Nicola stressed how important it is to report medical/social outcomes & plans to him & Alessandra, as well as reporting this in the book.

• National Interests

We need to compile a questionnaire, and this would be made available on the ConGAS website, and handed out at forthcoming conferences in this area. Completed forms will be passed to National Representatives for checking.

 

WG2_3.10 ConGAS Book 2

 

Suggested deadlines:

• Dec 2005: Suggested first Draft
• Dec 2006: Suggested Book Complete
• June 2007: End ConGAS

 

It became clear that there is plenty of scope for a book dedicated to the translation of human control gestures via capture devices and signal processing into standards for storage and transmission. (Some people has expressed doubt that the 3-book structure would work, because every application has gestures, conversion & ‘application’).  At the end of a long discussion we felt very much that a dedicated WG2-led book would be a very useful contribution to the research world. We restructured the existing structure to fit a better ‘flow’ and to link in the offering received from potential authors.

 

Listed below are the full notes from the discussion about WG2’s contribution to the book:

 

First, our ‘informal working titles’ for each book. 

 

Second, a summary of the offers of contribution received by Kia.

 

Thirdly, an edited suggestion of the Book 2 structure.  This has a better ‘flow’ from human operator, through physical sensors, via signal conversion, via signal processing and conditioning, into formats for storage and transmission.

 

Finally, some other suggestions and offerings from people which haven’t yet been fully integrated.

 

Book 1: What is Gesture? How to define Gesture?

 

Book 2: How to capture gesture?

 

Book 3: How to use Gesture?

 

Book 2 structure (version 2 June 2005) based on the original draft as proposed at the Oslo Meeting. New contributions are marked in red.

 

1        Hardware and Software Platforms for Gesture Acquisition

1.1  Sensing Systems

1.1.1        Introduction to the type of sensors (e.g. audio, visual, haptic etc...)

1.1.2        categories of physical quantities measurable (direct or indirect measurements) (Frederic Bevilacqua and Emmanuel Flety, …)

1.1.2.1  position (absolute and relative), velocity, acceleration, force, pressure, ... (Frederic Bevilacqua and Emmanuel Flety, …)

1.1.2.2  physiological quantitites (electrophysiology,respiration, heart beat, ....) (Frederic Bevilacqua and Emmanuel Flety, …)

1.1.3        sensors/technology commonly available: accelerometers, FSR, ultrasound, optical sensors, vision system (just brief mention here), ... (Frederic Bevilacqua and Emmanuel Flety, Kia Ng, …)

1.1.3.1  FSR

1.1.3.2  Ultrasound…

1.1.3.3  Vision system (Frederic Bevilacqua, Emmanuel Flety, Kia Ng, …)

1.1.3.3.1        camera (standard, specialised, high speed, …)

1.1.3.3.2        video card and protocol (firewire, usb, ....)

1.1.3.3.3        image analysis principles

1.1.3.3.4        3d motion capture (VICON, stereo imaging…)

1.1.3.3.5        dedicated software: eyesweb, ....

1.1.4        Combinations of Sensors: Contact Devices (Kia à to Daniel Arfib)

1.1.5        Challenges and perspectives

1.1.6        Data Acquisition (Frederic Bevilacqua and Emmanuel Flety, …)

1.1.6.1  principles of AD conversion and data transmission

1.1.6.2  wired interfaces

1.1.6.3  wireless interfaces

1.1.6.4  Digitisers (Kia, …)

1.1.6.4.1        AtoMIC

1.1.6.4.2        I-CubeX

1.1.6.4.3        etc

1.1.7        Differentiate what is sensed and what is returned (Sergi Jorda chapter 3, Jean-Michel Couturier 2.3, suggested by Daniel Arfib) (Is this the semantic of the numbers from the acq devices?)

1.2  New Instruments and Controllers

1.2.1        imitative instruments (suggested by Daniel Arfib)

1.2.2        alternative devices (e.g. gloves could be an important subchapter (Kessous) (suggested by Daniel Arfib)

1.2.3        Virtual Instruments (Kia’s MvM, Antonio’s EyesWeb, ALMA project ?)

1.2.4        "graphical objects as a source of sound" and "tangible objects" (Couturier chapter 3 and Jorda) (suggested by Daniel Arfib)

1.2.5        reacTable survey (Kà Martin Kaltenbrunner)

1.2.6        Sensor, wearable projector performance (Kà Bert Bongers, )

1.2.7        'Distributed Interfaces' to the controller section, and Laser pointer (Kristoffer Jensen )

System Architectures

1.2.8        Taxonomy

1.2.9        Input & Output considerations

1.3  Data Conditioning and Processing

1.3.1        Real-time issues

1.3.1.1  Latency, jitter

1.3.2        pre-treatment of data: filtering, derivative, ... (Frederic Bevilacqua and Emmanuel Flety, …)

1.3.3        integrate mapping and feedback in some way (Jorda,
Kessous, Couturier, Arfib) (suggested by Daniel Arfib)

1.3.4        Hardware/software

1.4  Standards

1.4.1        midi protocol

1.4.2        network protocol

1.4.3        OSC

1.4.4        PD

1.4.5        MPEG

1.4.6        ...

2        Research Methodologies

2.1  Design of Sensors and Architectures

2.1.1        Affordances, Conventions & Constraints

2.1.2        Practical considerations

2.1.3        Testing

2.2  Multi-modal Measurement of Gestures

2.2.1        Movement & audio & Haptic Tracking

2.2.2        Movement Quantification

3        Experiments, annotation and evaluation

3.1  Setting up Experiments

3.1.1        Experimental Paradigms

3.1.2        Overview of Research Done

3.2  Annotation of Gestures

3.2.1        Techniques of Annotation

3.2.2        Examples of Annotation

3.2.3        Representation and annotation of gesture [Ng, Leman, Gibet, Volpe, Moeslund] (move to Book 1???)

3.3  Evaluation of Gestural Research

3.4  Evaluation of gesture data capture

3.4.1        integrate musical evaluation (e.g. learning, pedagogy) as well as HCI evaluation (kessous cahpter III and Jorda chapter 7) (suggested by Daniel Arfib)

Contributions/ideas/proposals Received and to be integrated into the book proposal

 

·        Suggestions from Daniel Arfib:

o       Also "audio systems" should be treated at the same level as gesture control, and not as a byproduct. In a general way, we should take care to come back to the notion of "gesture-controlled" and not of "gesture control of". The difference being between a holistic system (gesture in the loop) and not a linear one (gesture commands everything).

o       By the way some old stuff like the one of Bill Buxton or Paradiso published on the web could be excellent in order not to reinvent the wheel. Marcelo Wanderley knows the bibliography webography well. Jorda, couturier and kessous also did a good job.

 

·        Conductors' Gestures, their Acquisition and Mapping to Sound Synthesis

-         Gunnar Johannsen

-         Maybe this is best to go in book 3 as it might be predominantly to do with the application of conducting?

·        Contact Devices

-         Daniel Arfib

·        Sensors, Interfaces and Vision system

-         Frederic Bevilacqua and Emmanuel FLETY

·        Virtual Instruments

-         Kia’s MvM, Antonio’s EyesWeb, ALMA project (?)

·        'Distributed Interfaces' to the controller section, and Laser pointer,

-         Kristoffer Jensen

-         Distributed is when there is no clear link between the position of the body and the position of the interaction While the original idea has to do with local physical distribution, such as laser pointer to a screen with camera detection, electronic and digital distribution on the internet and such should also be covered.

·        reacTable survey

-         Martin Kaltenbrunner

·        Sensor, wearable projector performance @ FUSE2006 (?)

-         Bert Bongers,

·        Gestural Bi-manual Controls of Sound Processes, PhD Thesis, http://loickessous.free.fr/loicthese.html

-         Loic Kessous

·        ?

-         Sergi Jordà,

·        ConGAS Library for PD

-         NB

 

It was noted that many parts of the contributions from Teemu Maki-Patola () can be adopted in Book 2, particularly on the human-machine comparisons, e.g. Human sensory system in contrast with Electronics Sensors, etc. This will be used to verify the flow and structure of the Book 2 proposal:

·        Introduction to physical gesture applications (instruments, sound environments, installations, etc.)

-         System parts (book 2)

-         brain, body, user interface, feedback (book 2)

-         What happens in the system / how does the system function (book 2)

-         Instruments (book 2)

§         Aim + inner prediction model / What the performer wants to accomplish

§         Causality in order to allow to learn, predict and plan

§         Introduction to the human perception-action-loop

·        action -> perception of feedback -> refinement -> ...

§         Sound environments, artistic installations, etc.

§         Interaction methods

§         Gestural perspectives

·        Introduction to musical/sound/imagined gesture (pitch, timbre and other features may be the result of the gesture).

·        Introduction to human perception (book 2)

-         sensory system (book 2)

-         Perception of separate modalities (book 2)

-         Properties of sound, touch and visual perception (book 2)

-         Accuracy (book 2)

-         psycho-physical results (book 2)

-         psychological issues (book 2)

·        Attention span (book 1)

-         how many object are we capable of being aware of in the same time

-         Acoustics, perceptual center, masking, discrimination

-         Briefly on perception of grouping/note separation (in terms of what auditory gestures are audible and not audible)

-         Tables, JND, psycho-physical curves (equal loudness contours, webers law, ...), time, timbre, ..

-         Rhythm perception

-         Time precision (conscious and subconscious)

·        Multisensory perception (multisensory feedback) (book 1 or 2?)

-         Timing anomalies in human perception-action

-         Tapping ahead of time etc. as an introduction to the

-         neural models for the perception of simultaneity

·        Review of the essential human physiology

-         Nerve conduction times

-         Auditory feedback reaches the brain fastest, then touch, then visual

-         Reaction times to different stimuli

-         Table with comparison of reaction times and performance

-         Nerve conduction hypothesis

-         Simultaneity is decided at the inner representational level of the brain

-         Sensory accumulator model

-         Stronger sensory excitation reaches the brain faster (processes faster too?)

-         Pulfrich illusion (darker view processes slower. 3D in horizontal motion)

-         tapping tests (25ms vs 60ms asynchronies)

·        Actions

-         Motor system

-         Issuing motor commands slightly ahead of time, prediction in all actions

-         Resolution, limits, endurance

-         multi-mode action

·        Coordination

-         action-perception links, Fitts law, 2/3 power law, ...

-         Mechanical learning, attention span, reaction and action possibilities

-         Multi-mode coordination

·        Human performance

-         Spatial accuracy

-         Temporal accuracy

-         Performance in musical situations

§         Synchronization with external events

§         Synchronization with other output modes

-         Behavioral differences between skilled and unskilled performers

-         incentive for movement and coordination in music

-         other issues of live interaction, like perceived gestural effects in audio spatialization

·        Conclusions of the chapter:

-         Summary of the main issues for perception-action-coordination

-         The big picture in short

-         Some boundary constraints resulting from the presented properties

-         Emphasize consequences for gesture music

·        Contents in consideration:

-         Introduction to gestures in control, interaction and manipulation