Addend- 08: Marysol - proposes analyzing emotion in voice spectra:

Note from Marysol:

hello Michael, hello Bill, hello Frank,
I send you the graphics that I sent to Dan about the harmonic signature and the voice emotion and intention,to coneect this function in real time to the heart with the Heartuner.

Look at the video so you see this in movement in www.YouTube.com/marysolsol the video named HeartVoice and let me know if you understand what I am saying that this will teach the person how the voice is expressing emotion and intention in the last harmonics rise, exactly where the Harmonic Signature is found,,, from 3khz to 7khz or more in men and from 4khz to 8khz or more in wimen.... www.YouTube.com/marysolsol the video called HeartVoice. I now have this to see it and listen to the voice after from the recording, not real time.  So the idea is to have my software programmers convert it to real time and from the main octave from the voice decide the area of Action from their main octave in the voice of the fundamental frequencies of the voca chords and from there adjust to the voice of the individual person, having the first area detected and individualized then the areas of Thoughts and Emotions will also be established for each person.  And this hooked to the HearTuner will be even more awareness for the person using it as to how spoken word helps in coherence, emotion and intention.  The whole thing is to change the software I now have to be real time... I cannot ask my programmers to do it, and maybe if he sees more involvement form other persons and organizations, or some funding comes up, we will get this done for not much money, because most of the work is already done,,, only do it real time.
ok, hope you understand what I am saying, it seems that Dan did not understand this much.

Love
MarySol

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Voice analysis Software website: www.biosonic.org :Programa de analisis de voz
PROFESSIONAL BLOG on WAVEFORM of the VOICE: http://ONDAdeFORMAdelaVOZ.blogspot.com
Curriculum vitae: www.sacrocraneal.org/marysol
PERSONAL BLOG on Psychomagic art events: http://ARTE-PSICOMAGICO.blogspot.com
Casa ahabah, camino de Cueva Negra 14, Apartado Postal 1406, Mojacar, Almeria, Spain,
Tel: (34) 950168735,  (34) 619228048,  (34) 699987118

---

re: Sentics..

Subject:	When it comes to emotional signatures, why reinvent the wheel?--Re: voice fft
Date:	Wed, 16 Jan 2008 17:41:39 -0700
From:	Michael Heleus <mheleus1@cox.net>
To:	biosonica@retemail.es
CC:	williamdonavan@alltel.net, frank@heartcoherence.com, danwinter@goldenmean.info

Hi Marysol,
Let's make this easier if possible, even tho it may look more complex at first by considering Manfred Clynes essentic emotional signatures as the best reasonably available emotional signatures--Dan alread knows a lot about them, and Clynes' friend Andrew Junker even helped Dan and Frank develop the cepstrum nested FFT software for the tuners.I am saying anyone's voice signature will reflect a characteristic blend of emotions found by superimposing the essentic signatures each with its own fundamental frequency, phase with respect to the others with one chosen as fiduciary against which the others are compared, and its own amplitude, exactly as if they were an individual frequency. That may be oversimplifying, and each form might be needed to be treated as if it were a timbre.There are 8 main emotional states and Clynes originally found about 12 more, which could be appropriate here.See
http://microsoundmusic.com/home.htmSentic Emotional Forms

---

Manfred CLYNES

Sentic ( Essentic )Forms (reprinted from rexresearch.com

Sentic Forms
Manfred Clynes: Sentic Cycles --- The Passions At Your Fingertips; Psychology Today (May 1972)
M. Clynes: Time-Forms, Nature's Generators and Communicators of Emotion; Proc. IEEE Intl. Workshop on Robot & Human Comm. ( Tokyo, Sept. 1992 )
Manfred Clynes: US Patent # 3,691,652 ~ Programmed System for Evoking Emotional Responses
M. Clynes: USP # 3,995,492 ~ Sound-Producing Isometric Exerciser
M. Clynes: USP # 4,704,682 ~ Computerized System for Imparting an Expressive Microstructure to...  a Musical Score
M. Clynes: USP # 4,763,257 ~ Computerized System for Imparting an Expressive Microstructure...
M. Clynes: USP # 4,999,773 ~ Technique for Contouring Amplitude of Musical Notes...
M. Clynes: USP # 5,195,895 ~ Sentic Cycler Unit
M. Clynes: USP # 5,305,423 ~ Computerized System for Producing Sentic Cycles...
M. Clynes: USP # 3,755,922 ~ System for Producing Personalized Sentograms

Sentic Forms

Psychology Today (May 1972)

Sentic Cycles --- The Passions At Your Fingertips

by

Manfred Clynes

Specific emotions --- anger, hate, grief, love, sex, joy and reverence --- produce distinct muscle movement. They’re the same in Mexico, Japan, Indonesia, and upstate New York. When you run through a cycle, expressing each of the seven with your fingertip, you feel better for it.

In 1967 I took part, as pianist, in Pablo Casals’ master classes in San Juan, Puerto Rico. One day when Casals was teaching Haydn’s "Cello Concerto", he asked a participant, a young master in his own right, to play the theme from the third movement. His playing was expert, sure and graceful. But for Casals something was missing.

The master stopped the performance. “No, no!”, he said, waving his hands. “That must be graceful!”.

He took up his own cello and played the same passage. And it was graceful, a hundred ties more graceful than we had just heard. Yes --- it seemed as though we had never heard grace before. We had experienced one of the least understood forms of human communication --- a powerful and clear transmittal f feeling without words, a feeling that penetrated our defenses and transformed our states of mind.

Casals played the same notes, and at similar speed. But the muscles of his hands and arms acted precisely together with his cello according to his very clear idea of grace.

How was his possible? How, precisely, was Casals’ expression different from the student’s? And how did the sound of his cello carry the idea and feeling of grace from his mind to ours?

Action --- These and similar questions that I thought about for many years led me to record and measure the precise motions of expressive action. My experiments on impulses from the pressure of a single finger, precisely expressing various fantasized emotional states have shown that there is a specific dynamic form of action underlying the expression of each emotion, and that the dynamic character of this action-form probably is universal, unlearned, and genetically programmed.

Emotions may be experienced in various aspects: (1) in a real situation, or (2) through fantasy --- as when one imagines being with a loved person. Also one may experience emotion trough empathy with another person who is (1) either really experiencing the emotion, or (2) experiencing it as fantasy --- as when one is watching a play or movie.

It is hard to study the quantitative effects of emotion in a real situation. In most experiments it is difficult to specify situation that reliably produce a given emotion. Repeated experiments are difficult to carry out under the same conditions. Emotions, in real situations, may today be less amenable to scientific study than fantasized emotions are. I have found a way of generating and expressing fantasy emotions that allows precise, repeated measurement.

Signs --- How can one know what another person is feeling? Psychologists and other scientists have tried to identify outward, physical signs that correspond to each emotion. Perspiration, pupil-diameter, skin conductance, heart rate --- all have been used as objective, observable indications of internal states. But these are not uniformly related to experience --- anger turns some persons’ faces red, others, pale.

And yet, artists, musicians, dancers and actors are aware of the precision of emotions that may be communicated. They may communicate through movement of hands, legs, mouths, eyes, the whole body, and through tone of voice. The precise way that one uses his body to express an emotion is more important than the part of the body one sues [see “Body Talk --- A Game” by Layne Longfellow, Psychology Today, October 1970].

Anger --- We may consider that there is a common brain program for specific emotion that determines the character of the movement and its precise time course, regardless of the particular body movement that expresses it. Or, example, in expressing anger no matter what part of the body one uses, the brain program that determines the character of its time course is revealed. And this is in turn what we notice when we watch the movement and perceive anger.

In this method of generating and expressing fantasy emotions by a succession of single, expressive, appropriately timed acts, the fantasy emotion increases with each expression, until it reaches a peak that may be maintained for a time and then gradually dissipates. This dissipation takes place even though one continues to perform the expressive acts.

Finger --- In view of this, I decided to use the expressive pressure movement of one finger as a standardized basic measure of expressed fantasized emotion. In my experiments, the subject sits in a straight-back chair and rests the middle finger of his right hand on a finger rest. I ask him to fantasize a given emotion (say, anger or love) for the next few minutes. Whenever he hears a signal --- a soft click --- he is to express that emotion as precisely as possible through the single, transient pressure of the finger. The clicks come at unpredictable intervals that vary by several seconds. The finger-rest is mounted with two pressure transducers that produce two graphic tracings of finger pressure during the two seconds immediately following the click. One tracing measures the finger’s vertical pressure; the other measures its horizontal pressure, toward the body and away from it.

Trials --- Most subjects find it easy to express a fantasy emotion with a single finger pressure. About 70 percent can do it on the first set of trials.

To get a stable overall measurement, I usually have a subject express each emotion 50 times, then I feed the data into a computer of average transients (CAT) that averages the vertical tracings for a given emotion into one common vertical form and, likewise, extracts a common form from the horizontal tracings. I have found that the more a subject practices a clearly separate fantasy emotion, the more his individual expression tends to approach the common form for that emotion.

Usually I have a subject express anger, hate, grief, love, sex, joy, reverence, and a state of no emotion.

Expressing no emotion in this method is like the mechanical movement of typewriting: primarily downward, and slightly outward, away from the body. Anger is a similar emotion, but with reversed emphasis: outward movement is more pronounced than downward movement. Anger is a brief expression --- the finger returns to its original position in less than a second. Love is slower, and takes two seconds or more. Changes in pressure during love are gradual and smooth, and the horizontal tracings often show an inward, embracing movement. The form for sex is distinct from love.

Measurements of electrical activity in the muscles show that there is a secondary delayed pressure that begins after the expression has started. Such delayed muscular activity also occurs in hate --- another passionate emotion. Hate, like anger, involves a push away from the body. Grief is slow, like love, but is flatter and slightly outward. In joy, after an initial downward push, the finger pressure rebounds above its starting position, as if one were jumping for joy. Reverence is similar to love, but lacks the inward pull ad follows a longer tie scale --- the full expression of reverence may take three or four seconds. I first included fear among emotions that a subject was to fantasize, but I found that fear implied withdrawal and inhibition of expression and our technique could not measure this.

Oxygen --- In many cases I have measured additional physiological variables during an experiment, and these measurements confirm both a specific pattern for the expression of each emotion, and the persistent physiological changes that accompany sustained fantasy emotions. During an expression by this method, the electrical activities in separate muscle groups (the forearm, upper arm, front shoulder and back) show reliable, identifiable patterns). Respiration also tends to follow a specific pattern: a subject tends to exhale as he expresses hate or grief, for example, and to inhale when he expresses joy.

Heart rate and oxygen consumption show definite, characteristic changes while a particular emotion is fantasized. Oxygen consumption appears to be highest in the states of hate and sex, lowest in love and reverence.

Stability --- Emotional expressive forms measured in this way are stable and apparently universal. A subject will give essentially the same tracing for a single emotion on different occasions, and different subjects from different cultures produce remarkably similar tracings for a given emotion. We can state the degree of correspondence between any two measurements as a correlation in which zero indicates no relationship between one measure and the other and 1.0 indicates a perfect match. Between two measurements of the same emotion in one person, the vertical tracings usually correlate above 0.90, whereas cross-correlations between different emotions are generally lower than 0.30. The forms are also remarkably consistent between individuals --- correlations between two persons’ expressions of an emotion are generally above 0.80. Dramatic differences can occur, and the discrepancy can be instructive. For example, when I compared my expressions to those of another subject, I found that our tracings were similar for most emotions (correlations of 0.80 or higher), but that for anger they differed sharply, with a negative correlation of 0.22. We learned that we interpreted the word anger differently. I had expressed an irritable, ready-to-strike-out anger, whereas my colleague’s anger was of the slow, burning type. The tracings seems to detect the word anger designates two different emotions.

Test --- The observation raises an interesting question. Suppose two persons expressing an emotion (joy, for example) produce slightly different waveforms. Does this mean that they express the same emotion in different ways, or does the slight shape discrepancy imply a corresponding difference in the way the two feel joy? We cannot answer, but my research indicates that when there are large differences in form, there are large differences in the emotions experienced.

For example, I have tried to train subjects to express one emotion with the expressive form of another emotion, but they cannot learn to do it. I asked subjects to fantasize anger and try to experience this with the patern associated with love, and vice versa. When a subject’s tracings approached the love form, re received praise; when he reproduced anger-shaped waves, he was warned. But no matter ow hard he tried, no subject could generate and express anger by finger pressure resembling love.

With repeated expression, a subject’s fantasies became more and more intense. He may shed tears while he expresses grief, or become aroused.

Several investigators have shown that human beings can learn to control their bodies to an extent previously considered impossible. Peter Lang finds that, through immediate visual feedback about what some of his organs are doing, a subject can learn to control these as he would learn to drive a car [see "Automatic Control or Learning to Play the Internal Organs", by Peter Lang, Psychol. Today, October 1970].

We gave our subjects no feedback about the shapes they were producing --- and yet their tracings became more accurate and precise as the trials went on. Clearly, something different from instrumental learning is involved here. It is as if one were discovering within himself those precise emotional expression programs that were there all along. This uncovering of the forms, and their resistance to change, suggest that the different basic emotional expressive forms are inborn --- not culturally learned --- a hypothesis that gains support from my finding that the forms are much the same from one culture to another. (The effect of culture may often be to suppress access to these forms, at some sage of development.)

I tested subjects in Mexico and Japan, and in Bali, Indonesia, and their basic shapes were the same as those of Americans. The few cross-cultural inconsistencies could be traced to language differences. Indonesians have no word for hate, and in Mexico alegria (happiness) was the closest approximation I could find for joy. Disparity between alegria tracings and the typical joy shape sowed just how inaccurate the translation was.

Rubicon --- The brain programs the entire course of a single brief movement before it acts. Once the decision to move has been made --- a swing of a bat, or an eye movement, for example --- it must continue; for 200 milliseconds, one cannot change the movements of a limb or muscle by another decision because of limits in the nervous system. The existence of specific, universal brain programs corresponding to certain basic elements of experience is not a new discovery. In 1965, Michael Kohn and I measured the electrical activity originating from the different parts of subjects’ brains while the subjects looked at various colors. We found that with the help of a computer we could identify more than 100 separate brain responses to specific visual stimuli --- and the patterns had consistent physiologic code elements in all the subjects we tested. We could thus tell what color a subject was looking at from the pattern of electrical activity in his brain.

The spectrum of our emotions, like our perception of color, is precisely programmed by the brain. This programming is different for different emotions. We call a single programmed movement having a clear beginning and end, together with the decision giving rise to it, an acton. The emotion-seeking expression modulates actons into different E-actons for each emotion. E-actons are so precisely programmed into the brain that we have been able to find a differential equation that can be used to simulate these human forms of expression on a computer.

To understand how the idea of an emotion directs the body’s movements, consider what happens when a pitcher throws a ball at a target. He must have (1) a clear idea of where he wants to hit the target and (2) a precise execution. The idea of the target modulates his throwing motion, so that eventually one may choose any object within a certain range, think of hitting it, and a spatio-temporal form will direct the exact movements of the arm. As many a major-league pitcher has demonstrated, practice can refine the accuracy with which the idea is executed.

A similar process determines the expression of emotions. Effective emotional communication depends on (1) a clear idea of the emotion one wants to express and (2) a precise execution of the muscular acts involved --- finger movements, gestures, tone of voice, etc. The capacity to develop a clear idea of an emotion (which I call an idiolog) is as much a part of human nature as the ability to perceive red or sweet or hot. The idiolog accurately dictates the specific expressive movement (if it is permitted to do so).

Animals --- Any number of bodily movements can express a given emotional idiolog. The specific brain program for anger, which can turn an innocuous arm-raising into an angry threat, also can direct angry movements of the foot, or the mouth, or the tone of voice. In successful communication the specific brain program effectively commands a movement --- with no inhibitions to block the expression. We generally interpret direct, unhindered expression of emotion as faithful or sincere.

When one perceives an emotional expression, the nervous system recognizes the form, and decodes it into a corresponding emotional idiolog. As the receiver of such messages, the nervous system is programmed to interpret the shape of movements, and there is little we can do to change this program. We even attribute appropriate characteristics to animals whose movements remind us of human qualities we are programmed to recognize (for example, a graceful antelope or an uncouth hippopotamus).

Cycle --- In my first exploratory studies, I usually expressed no emotion 50 times in sequence and then expressed each of the seven emotions 50 times. The entire process takes about 30 minutes. I call this a sentic cycle, according to the terms in my formal theory, in which the specific expression of an emotion is an essentic form, and the emotion brain programs that produces the form is a sentic state.

Although at first a subject may like to imagine various scenes to help him fantasize the emotion, he soon finds out that he can express an emotion without directing it at a specific person or imagining a specific scene. He learns to experience the emotion in itself, as in music, without needing specific provokers or recipients for the emotion. This pure emotion does not imply lack of consideration for the individual: when we understand and experience emotion in its most general sense, we are also most able to become genuinely concerned about and close to a particular person, that is, to develop empathy.

Peace --- Often I go through several sentic cycles at a sitting. (A straight-backed chair and a correct arm position, I have found, a crucial to performing sentic cycles reliably and without fatigue). When I first began doing sentic cycles for several hours at a stretch, I was surprised to find that I was neither bored nor tired, but refreshed and satisfied, and that I required less sleep than usual the following night.

At first, I attributed these effects to enthusiasm and curiosity about a new discovery and to the satisfaction of completing a good day’s experiment. But I soon found that others reported similar feelings of well-being and satisfaction from repeated sentic cycles.

In further, systematic observation, I had subjects go through one-hour sessions made up of two sentic cycles. Most subjects reported that after the second cycle (which they often experienced more fully than the first), they felt calm, content --- some compared the experience to a marijuana high.

Other researchers have confirmed the general observation: performing sentic cycles lessened anxiety for 3 to 24 hours. And, together with calmness, they often showed marked increases in mental energy. It is not necessary to record the expressive movements to reap these benefits. Practicing sentic cycles in the home, in the proper position and with the finger rest, works as well as performing in the laboratory.

Release --- Many persons work off anger by punching at a wall or chopping wood. They say that the physical activity makes them feel better --- it releases the anger, "lets off steam". But it may not be the amount of energy expended that is effective, but the quality of anger that is expressed repeatedly. One can work off anger by appropriate and repeated pressing of a finger as effectively as by chopping a pile of wood.

Most of us tend to suppress emotions in our daily lives, but in sentic cycles one can express a spectrum of emotions freely, without embarrassment or fear of social censure. This freedom to discover and to be what is natural contributes to the satisfactory results of a sentic session. In addition, there is the satisfaction of finding that one can summon up various emotions at will. This creates a condition of sentic fluidity --- as compared with the rigidity found in emotional disturbances.

Whatever their cause, the beneficial effects of sentic cycles have many applications. In fantasizing emotions one experiences relief from daily emotional tensions. Fearful or anxious persons have reported that sentic cycles help relieve their symptoms.

Psychiatrist Alfred P. French and Joe Tupin have found that sentic cycles provide "immediate and dramatic relief of symptoms of depression", in some patients. Other researchers have evidence that they may also relieve psychosomatic disorders, perhaps because during sentic cycles one naturally expresses emotions that might otherwise be shunted to various parts of the system causing long-term internal stress.

Touch --- After experience with sentic cycles, one comes to appreciate the language of touch: he becomes more sensitive to the emotional signals in another person’s touch, and more aware of the emotions he communicates through his own touch.

Learning to express and control emotions in the way I have described may help drama and music students learn to be better, more convincing communicators. Feedback in the form of tracings can show them when their expressions approach the true sentic form. The measurements might shed light on what we call natural talent.

Training a person to express fantasy emotions in this manner, to be in touch with the spectrum of emotions, may help in the treatment of emotionally disturbed, neurotic, or psychopathic personalities. It also seems to lend itself to rechanneling of anxiety-driven aggressiveness to a creative energy, in which the expressive act itself gives satisfaction. We should be able to learn more about the basic human emotions, and perhaps someday, possibly with help from geneticists, discover new ones better than we have yet experienced. Indeed the experience of the sentic cycle itself is a step in this direction.

---

Proceedings of the IEEE International Workshop on Robot and Human Communication ( Tokyo, Japan, Sept. 1992 )

Time-Forms, Nature's Generators and Communicators of Emotion

Keynote paper, IEEE International Workshop on Robot and Human Communication, Tokyo, Japan, Sept. 1992.

Manfred Clynes

CNMAT
University of California, Berkeley
1750 Arch Street, Berkeley 94709, and
Microsound International Ltd. Box 143, Sonoma Ca 95476

Abstract --- Dynamic forms, called sentic forms, are described as language elements of a natural, biologically evolved language of communicating and generating emotions. These forms are genetically programmed into the central nervous system, and can be stored and recognized by computers and robots, and can serve as basis of real time emotional communication between them and humans. Music and art also utilize these forms to store and embody emotional meaning. A double-stream theory of music is outlined - including two principles of unconscious musicality which can realised on a computer - allowing one to create first-rate meaningful interpretations without manual performance or dexterity. New social opportunities and dangers are discussed arising from simulation and virtual reality which may exceed the average human in emotional eloquence. Real-real time is introduced as a concept to include human timeconsciousness.

I. INTRODUCTION

This paper is a short review of new thinking and practical applications arising from the discovery and study of time-forms and their biologically evolved meanings in communicating and generating emotion.

In communicating between men and machines, the use of discursive symbols has been general practice. Symbols come in two kinds: discursive and nondiscursive [1]. Discursive symbols have one-to-one relationship with what they denote. In non-discursive symbols human imagination is brought into play; such a symbol can act to evoke meanings and indirectly feelings. It used to be thought that music and the arts made use largely of such non-discursive symbols [1,2,3].

A major step forward was achieved in realizing that the evolution of nature has designed specific dynamic forms which act not in a symbolic way but directly on the central nervous system. [4,5,6] These forms, like laughter and yawning, require no symbolic translation. They constitute the "words" of a natural language of emotional communication. Over two decades of studies have been devoted to isolate these forms and elucidate their function [7,8,9,10,11]

Human communication systems use zeros and ones, or dots and dashes, to transmit information. But nature's own emotion communication system has codesigned the sender, receiver, and message units with meaning evolved by nature: the message units themselves (the sentic forms) have analog form which act like keys in locks of our nervous system.

The specific dynamic forms - "words" - are produced through appropriate, prewired modulation by the sender, as analog forms. The receiver has demodulation filters that recognise these forms, like keys in locks. In our brain, the amygdala, a special structure of the midbrain, acts as a "gatekeeper" both to modulate and demodulate the specific dynamic forms, or "words"[12]. Several output-input modes may be chosen for this process, moreover: the auditory, visual, tactile or motor systems.

The biologically evolved communication system inherently encodes and decodes emotional meaning - with specific dynamic forms (in the range of 1-10 sec duration). Without this remarkable symbiotic design, we would exist in emotional isolation. Through it we can touch one another emotionally in the present: we can share our emotions in our stream of life, and outside this stream through art and music.

This auto- and cross-communication system a has developed its own vocabulary. There is a class of qualities of experience which can be communicated inherently by specific dynamic forms and are contagious. These may be regarded as basic emotions. The emotional quality is transmitted from one individual to the other, in whom it is generated in turn. Such transmitting of emotional qualities may be observed in animals also (cf. innate release mechanisms), in the behavior of flocks or herds. In humans it may be seen for example, in crowd behavior, political oratory by demagogues, concerts of music, theatre, as well as in intimate behavior.

The basic dynamic forms, called sentic forms, may be evident in a gesture, in a tone of voice, in a musical phrase or dance step. It is the character of the form, not the particular output modality that determines its emotional meaning. We have isolated the dynamic forms for a number of emotions especially, anger, hate, grief, love, sex, joy, and reverence [4,5,6]. Like laughter and yawning these forms themselves appear to be largely universal and can not be arbitrarily learned - one can only discover what is already there inherently, "hard-wired".

Societies differ in the degree of suppression, the frequency of use, and the choice of output modality for communicating these emotional qualities using their biologically given dynamic forms .

Precision of Dynamic Form

A consequence of the biologic design of key-lock relationship is that the power of transmission becomes a function of the precision with which the form is realized. A deviation from the biologically designed form will tend to diminish its power to generate the emotion. If the deviation is sufficiently large, no recognition will take place at all - the key will not fit.

An interesting question is raised as a consequence: what kind of distortions are acceptable to the process, and what kind of distortions suppress the emotional meaning? This becomes one of the central questions to elucidate in a theory of emotional language transmission. Jamming of the emotional quality in the transmission process may be produced by one class of distortions, but not by another. Thus, the concepts of noise and of signal to noise ratio need to be looked at in a different light from that of man- made transmission systems. A new kind of mathematics needs to be developed that can distinguish between interfering and non-interfering types of distortion and noise.

The gain of the transmission of emotionally meaningful form is a form function. The degree of the perfection of the form governs the power of generation. It is therefore not sufficient to have a form that is conventionally 'similar' to a desired expressive quality. It is necessary to redefine the meaning of 'similar' in this context.

Such a form could be a caricature, and can also be perceived as mimicking. Mimicry may be useful when no real emotion communication is desired [7,9], i.e. no real-real time emotion communication (for the meaning of real-real time, see later in this paper). Then the form merely acts to remind a person of the emotional quality in a relatively "untouching" way. If one wants to have genuine emotional communication, the forms need to be precisely realized. These forms thus operationally define what we sense as a 'sincere' or "from the heart" expression of emotion. The conditions for this are biologically given.

Therefore by simply being faithful to this precision of dynamic form we can achieve a mode of communication between man and machine, and between machine and man, which will be felt as not machine-like.

(That function defines as well exaggeration which also acts as impediment to 'sincere' sharing.)

A human tends to be very quickly seduced to project an entity into a machine that communicates in this way. One needs in fact to develop a resistance to avoid being "sucked in". Continuing and varied interaction between man and machine with truly expressive gestures and tones of voice on the part of the machine, or of genuine musical expressiveness, make it mountingly difficult to maintain one's awareness that it is "only a machine".

As it is increasingly becoming possible, by incorporating the dynamic forms now known to be truly and 'sincerely' expressive of specific emotional qualities, to program a machine to interact with the human so that the expressions are experienced as not being machine-like but "living", we need therefore to look as far as we can now, with some urgency, to consider the differences between a man and such a machine - especially emotionally.

Anyone who has seen the newest little toy dogs walk, jump and beg may have had the almost irresistible response to feel: "isn't he cute!" Applying that paradigm to the far more sophisticated dialogues and simulations that are possible with this new knowledge, it is not hard to realize that we will need to 're-arm' ourselves emotionally to maintain our human identity. This will not be easy to do because we are biologic prisoners of those forms, and cannot escape the feelings of livingness associated with the spontaneous production of such forms, and especially the more so in dialogue.

It is not so much the ability of such a machine to "think" that would cause this problem, but the implied and varied feelings which it would express so well. The emphasis is on so well . It becomes an artistic achievement, so to speak, to program a machine so that it communicates in emotionally meaningful dynamic forms, indistinguishible from those produced by humans - and by humans moreover who are most effective in their emotional communication.

Until now, it has been the province of great art to provide a "living" storage of emotionally meaningful form. The great performer of music Pablo Casals could express more emotional meaning in shaping one single note than another performer perhaps in an entire concert. Inwardly perceived first, the high resolution of time-form required in all the variables concerned in producing the musical meaning and sound, was the province of his art.

That power of pure biologic form, transferred into a computer and machine can potentially result in more powerful communication of emotional qualities than the average human can produce most of the time.

This is a prospect that we need to face as the Pandora's box of sentic forms is now open. Society needs to learn to use them in new creative ways (see footnote 1) and to live with their now conscious knowledge - as well as with the results produced by ever-better "mechanization" of incorporating these living forms in dead machines. Looking at a mannequin may intrigue us to a degree. But when that mannequin can variously and expressively converse with us and even sing with more eloquent and "living" dynamic shapes than most humans can produce, our feelings towards the mannequin will change almost irresistably. To force ourselves to regard such a "living" mannequin as a machine may after a while tend to make us wary of real humans to whom we might tend to apply the same filter, as if they were such mannequins. This, if successful, would recreate an isolation and alienation from which nature had rescued us millions of years ago when it developed these very forms. If we cannot trust these forms any more, what can we trust?

A new kind of emotional education will be required. We know too little yet to be able to say how such new emotional communication will turn out.

The most difficult programming eventually will be to incorporate true empathy, as distinct from sympathy, in the dynamic expressions of the machine [see 7, 9 for distinctions between empathy and sympathy]. Such expressions are not commonly found in theatre. They need to be based on a knowledge that the machine has of the human that far exceeds the current question the machine is being asked; they need to be based on insight which comprises his potentiality. Once that is achieved however, it will be very difficult to avoid feeling an entity, a benevolent, perhaps even loving entity within the machine.

Conversely, a hostile, malevolent persona can be programmed, for purposes that may be devilish, or perhaps merely protective against intrusion.

Personality structure could in fact eventually be introduced which would govern the types of emotional responses produced in specific dialogue and situations.

Furthermore, even the personality structure could be altered (or switched) in response to different questioners , in a preprogrammed way, so that "strangers", for example may be treated entirely differently from "friends", within a whole field of discourse. This could be experienced as two (or more than two) different people residing in the machine. And these different personalities could even function simultaneously, from the same machine, independently of one another - being friendly, and hostile, and indifferent simultaneously to different users or customers!

It thus becomes increasingly important to attempt to elucidate the real distinctions between man and machine, especially in view of

1) the largely prevelant concept that increasing complexity of machines might be the required prelude to their becoming conscious and

2) the ability of computers and machines to simulate not just human thinking but emotional communication in the biological language devised by nature with a precision and power that will not merely equal that of humans but may exceed it.

TIME FORMS

The central nervous system has evolved the capacity to perceive transient time forms with great precision - an ability that is used in the communication of emotional qualities. Because of the transient nature of these forms they rarely receive recognition in human language by specific words (some exceptions: sigh, caress). Nevertheless they represent real entities to the nervous system, as do visible objects which are frequently named. The time forms concerned are in the general range of about 1 to 10 seconds, and the precision may be as high as 1 part in 500. Small changes in the transient form affect the meaning. The dynamic forms behave in parametric space like functions with a number of troughs or minima (solutions for the parametric values for which the forms are most powerful (singularities)) separated by saddle regions. As the shape changes from its optimal form, by varying the parameters, it becomes gradually less meaningful and powerful. If we transform the form gradually from one basic emotion to another, it first becomes meaningless before it starts to express the second emotion. The dynamic forms for specific emotions may thus be described as islands of meaning in a sea of meaninglessness.

The parametric description of such forms associated with its meaning and the effect of distortions requires a new mathematical treatment, or a new kind of mathematics for its appropriate representation. We have described [6,7] the averaged expressive forms for each of the emotions studied in terms of Laplace transforms as responses to unit impulses as follows:

While this description gives a good representation of the transient form it does not portray the changes in meaning and in the power of communication as the form changes. We are presently working on a mathematical formulation that can reflect this important part of its natural behavior. It can be said that techniques of least square differences and other currently used curve-fitting approaches are not suitable for this.
 
 

PROPERTIES OF SENTIC TIME FORMS

We may list the remarkable properties of these dynamic forms, that act as multiple body-mind windows [25], and their action in the central nervous system as follows:

   1. Its power of communication depends on its faithfulness to the characteristic form.
   2. There is a one-to-one correspondence between the quality to be expressed and the particular dynamic form. There is an inherent coherence between meaning and dynamic form. This coherence is biologically determined and genetically conserved.
   3. The expressive time forms are preprogrammed by the brain as single entities (trajectories) for each act of expression.
   4. Iteration of the form tends to augment the intensity of the emotion state it generates over a certain number of repetitions to a peak from which intensity will recede with continued repetition. Repetition is most effective when it is not strictly predictable, i.e.not 'mechanical', but rather when the intervals between repetitions vary somewhat, in an unpredictable, quasi-random manner. This variation is of the order of 5-10% of the repetition rate.
   5. Mixed, or compound emotions are expressed not through an algebraic addition of the expressions of the emotions of which it is composed, but through a telescoping of two separate expressions in time: the two component emotion expressions are seamlessly spliced somewhere in the middle (at a variable point). The expression of melancholy, for example, a compound emotion, begins with love, then is spliced to end in sadness. The love expression cannot complete itself and turns into a sadness expression, somewhere about half-way through the expression; however this is a single preprogrammed entity of 5-6 seconds duration, not two separate expressions.
   6. Each emotion expression needs to be completed, otherwise a frustration is felt. Interupted expressions produce frustration regardless of whether they express positive or negative emotions.
   7. Humans can express only one sentic form (single or compound, see 5 above) at any one time "sincerely" - regardless of which and how many output modes they may employ - ie. sentic expression is a single channel sytem. If several output forms are used simultaneously, they will tend to embody the same sentic expressive form. A human cannot, for example, express anger with the voice and love with a gesture simultaneously, and both sincerely.

A computer however is under no such restriction since for it "sincerity" is not bound to a single channel.

Different classes of distortions will be differently effective to degrade meaning. Some types of distortions are quite tolerable and affect meaning little. Other types are highly detrimental. It is important to be able to distinguish the different types of distortion. In the visual arts, for example, small missing gaps in lines can be readily supplied by the eye, or more precisely, by the central nervous system, while even small distortions in the shape of the lines will be felt as serious detriments. Similar distinctions apply to the dynamic forms. Thus, meaning suffers less from limitations of high frequency response between 10,000 and 20,000 Hz say, than through the effects of wow and flutter, and the relative shaping of individual musical phrases, i.e. sub-audiofrequencies. These sub-audiofrequencies (ie. transient shapes) are demodulated by the amygdala, a different brain structure than is involved in auditory perception per se.

While the effects of hi-fi have been greatly promoted, relatively little attention has been paid to the concept of fidelity of meaning. Within a relatively restricted frequency band it is yet possible to have a highly faithful rendition in regard to meaning, as well as a highly degraded one. Thus the tone of voice, for example, the minute timing and loudness inflections of the voice can carry vast ranges of meaning, even in a limited frequency domain. The extensions of the frequency band may change the range of meanings only comparatively little. (Arthur Schnabel's interpretations of Beethoven gained very little when the frequency response was improved from the 78 rpm reproductions). Even whispered speech can carry a large range of emotional meaning in its inflections.

There is thus a need to develop a more extensive theory of communication of qualities through dynamic forms [25]. The work which we have conducted over the last two decades attempts only to make a beginning in this.

APPLICATIONS

We have identified and isolated specific dynamic transient forms, called sentic forms, specific for each emotion, that naturally serve to auto- and cross- communicate basic human emotions, such as anger, love, grief, or sexual desire. They also serve to generate emotions through repeated expression.

These time domain forms are distinct from facial expression, which has been predominantly investigated by researchers in a static setting (though animation by film artists has been an active contribution). The forms work expressively in the various modes of spatio-temporal expression [26]. For example, they can communicate and generate emotion over the telephone as sound shapes, and through touch expression, gesture, dance, and music.

Measured through touch, sentographically (see footnote 2), they can be transformed into sounds expressing the same emotion conserving the dynamic form. Such expressive sounds derived from touch, have been tested on hundreds of subjects inter-culturally, and will be demonstrated.

Sentic forms can be stored in computers and used to modulate a variety of outputs which thereby become emotionally expressive. They can also be used to modulate the tone of voice.

The relation of dynamic form to meaning has also been most particularly studied in great detail with music. The microstructure of music has been shaped in accordance with meaningful forms so as to convert the dead notes of a score into living performances [13,14,15,16,17,21].

The generating power of repeated expression of these forms have been used to create a sequence of emotions for an individual, each expressed repeatedly called sentic cycles, that has therapeutic benefits and benefits to well-being[7, 9, 18]. This programmed sequence of about half hour duration can also be stored on the computer, to guide the timing of expressions. The measured expressive shapes can also be stored, and displayed on the screen [19].

Now that we know sentic forms, we can communicate with emotional meaning between man and computer, or robot, in the time domain. While the robot will not feel the emotion, it will be able to discriminate among emotions and in turn, send dynamic signals to humans in the biologically based language devised and used by nature in communicating and generating emotion - in sharing emotion among individuals.

COMMUNICATION THROUGH TIME FORMS IN MUSIC

In music the precision of time forms in communicating qualities is exemplified par excellence. It provides an excellent laboratory for the study of small changes in time form and their effects.

Time forms in music appear to function through two streams, simultaneously produced or perceived by the central nervous system.

One stream is the pulse, a repetitive phenomenon which is automatized after the first instance utilizing the property of the Central Nervous System we have called Time Form Printing. A dynamic form or movement of the order of one second duration may be repeated automatically by the nervous system without subsequent specific attention being paid to the form - we can put it into 'automatic repeat mode' by a simple act of will! For example, one may make a movement pattern with the arm say, in a triangular shape, or in an ellipse, and once begun, one may reiterate this pattern an indefinite number of times, without paying further attention to it. The shape of this movement will tend to be conserved throughout the repetitions. Stopping such a movement however, or changing it to a new form requires a momentary attention, to direct it to move with the new form; and it will then continue to repeat with the changed form, again without requiring further attention. While we do such a repetitive movement with the arm, we can talk and tell a story, a second stream, without one stream interfering with the other.

Such a two-stream process goes on in music and appears to be of the essence. In Western music of the late eighteenth and nineteenth centuries the pulse seems to represent the intimate identity of who is telling the story, that is, the composer - as was noted by Becking [20]. Along with this repetitive stream there is a second stream, the unfolding emotional story of the music. The two streams are perceived in parallel.

Different styles of music emphasize the two streams differently. Rock and roll has mostly the pulse, Gregorian Chant mostly the story, classical music of the 19th century a balance between the two. Notably, much of present day avant-garde music has lost the pulse as a separate stream. (This tendency seems to have originated to some degree with the impressionistic (vs. expressionistic) music of Debussy.) In folk music and most of ethnic music there also is a fine balance between the two streams. The pulse here expresses the ethnic identity, rather than the composer's, and the microstructure of the ethnic pulse appears to be related to the rhythmic fine structure of the spoken language.

In one of the great drawbacks of an otherwise toweringly invaluable music tradition, Western notated music is dead if performed as written. The musical thought of the composer cannot be notated in detail, only as a skeleton. The performer has to flesh out the skeleton with meaningful musical microstructure in order to re-create the living musical thought.

Two principles which convert the dead musical notes to living music have been discovered [12].

   1. The Pulse,
   2. Predictive Amplitude Shaping.

Each applies to one of the streams described.

1. The Hierarchic Pulse.

The pulse is a repetitive combined time and amplitude warp applying throughout the music to groups of notes forming the pulse matrix, typically a group of four nominally equal notes.

We have determined the microstructure for the pulse for a number of composers as a pulse matrix combining specific time and amplitude warps [13,14,15,16,21,24]. This pulse matrix functions hierarchically on several levels, in a relatively attenuated mode for the higher levels. By applying the hierarchal pulse structure to a musical composition we can obtain performances of considerable quality and distinction in terms of musical meaning. Such a performance of Mozart's Sonata K330 will be demonstrated at this meeting. The attenuation levels at the various hierarchical levels and the hierarchical form of the pulse structure need to be chosen with musical judgment, for each composition. The pulse is therefore applied with musical judgment, not in a purely mechanized way. But the application of the pulse and the choice of parameters occurs generically to the entire piece. Of the several levels of the pulse, typically three, the lowest two are composer specific, the highest piece specific. For piano music the pulse alone can provide much of the musicality required to bring music to life. Other interpretive variables which need to be entered include the balancing of voices, specific dynamic indications prescribed by the composer in the score, micropauses at the boundaries of sections and occasionally elsewhere, choices of tempo, and of changes in general tempo.

Like the gait of a person, or handwriting, the pulse represents the unique personality of a composer and provides his 'point of view'. or 'presence'. It can be readily realized with MIDI. Pulse matrix values are now known for Beethoven, Mozart, Haydn, Schubert, Schumann, Mendelssohn and some other composers. The pulse is also highly applicable to popular music, enhancing its livingness.

2. Predictive Amplitude Shaping

The second, parallel, stream of the music process is the unfolding of the musical story as a chain of melody and is continually changing and developing as the story unfolds. In this stream the amplitude contour shape of each separate note contributes to the musical meaning. A singer, string player or wind player can and does shape each note distinctively and differently according to the musical meaning. How the amplitude envelope shapes of each note are related to the melodic structure was discovered and is described by the principle of Predictive Amplitude Shaping, a second principle underlying the microstructure of music. It too, describes an unconscious element of musical thought, involved in musicality, not notated.

Where musical instruments allow it, every tone of a piece of music is shaped differently according to expressive needs. In a good performance the shapes of the tones are by no means uniform, but what guides their shape has not been systematically described either by music interpreters or theorists.

By using computers to generate melodies using sinusoidal sound only, and varying the amplitude envelope shapes and the durations of the tones, a principle was discovered which we call Predictive Amplitude Shaping, which appears to mirror functions of musicality which govern the shapes employed in meaningful interpretations [13].

This principle makes the amplitude shape of the present note deviate from a basic shape in a specific manner depending on what the next note is going to be. The envelope shape is skewed forward in proportion to the tangent of the pitch-time curve at that note. This means that the shape of the present note is skewed forward if the next note is going to be higher in pitch, and it is skewed backwards if the next note will be at a lower pitch, depending also on the time when the next note occurs.

The principle relates the shapes organically to melodic structure, so that the shape of the present tone implicitly presages what tone will follow. This gives a feeling of continuity and a continuity of feeling to the performance. It markedly enduces musicality into the phrasing of melodies, and appears to apply to melodies in general, i.e., it is not composer specific. The mean form however from which the skewing takes place seems to have composer-specific aspects.

In creating the shapes of envelopes for the amplitude of the musical tones we have departed from the industry standard of specifying attack, decay, sustain and release. These parameters were derived from the properties of a piano or keyboard tone, they reflect the musical machinery of a piano. We note however, that musical thought does not think in terms of such discontinuities. Rather, the musical thought of a tone is shaped as a continuous curve. Such continuous curves of amplitude contour are naturally produced by the human voice, string instruments, wind instruments, wherever the tone can be continously shaped. This ability is in fact a source of superiority of such instruments over the comparatively fixed amplitude contours produced by pianos (and even more so harpsichords and organs which cannot even readily accomodate the amplitude warps of the composers' pulse matrix - but not so the clavichord!).

The shapes of individual tones, and their range of variation in music can be largely described through only two parameters of a beta function [13,21]. We have used this method to calculate customized shapes for each tone based on the melodic structure, and the above principle. The basic shape from which skewing takes place applies to all notes, short or long (except for very long notes eg. pedalpoints).

These organically varied shapes seem to represent musical thought better the conventional methods. They make even scales sound more musical, and often result in phrasing as specially indicated by composers.

They relate to the inner gestures that govern the emotional expression of the music and allows the form of the music to correspond to the inner desired shape.

Vibrato structure and changes in timbre within each note can also be guided by related principles in an integral way, customizing every note.

This second principle shall be of great use especially in the next technologic generation.

It is not readily realised through MIDI, in contrast to the Hierarchic Pulse, but will be of great value in the next technologic generation for the general user to achieve living musicality.

APPLICATIONS

Both the pulse and predictive amplitude shaping can readily be incorporated into a computer program which can perform music in a living manner as a consequence, with emotional meaning as desired by the user of the program. This allows any user who has no manual dexterity from ages 8 to 80 to interpret great music using only their own sense of musicality.

Composers can now specifiy in a micorscore the way they wish their music to be performed.

Using these principles teaches one how to improve one's musicality in a profound and subtle way. It is a powerful pedagogic means.

The knowledge that we have obtained through the realization and application of these principles of unconscious musical thought, through computers, is significant to AI [27], to musicians and to the general public. In a sense it is the achievement of the virtual reality of the music performer - the simulation of music interpretation and performance, so that every smallest detail is known (through the global adjustment of the principles, using one's musicality. Less than 2 % of the notes need 'manual, individual' adjustment). It has implications also to the subtleties of rhythms and inflections of speech in conveying shades of emotional meaning.

Appendix A gives details of the demonstrations of its function, given as part of this presentation.

EMOTIONAL COMMUNICATION WITH A COMPUTER OR ROBOT

TIME CONSCIOUSNESS: REAL-REAL TIME

More than any sense perception, such as visual, auditory or tactile experience, time consciousness is intimately linked to consciousness. Without seeing, hearing, touching, or other sensory inputs, consciousness is still engaged with time consciousness. Mental events are ordered in time, apart from sensory inputs. One may think, for an obvious example, of a musical rhythm. The tempo of such a rhythm for a given meaning depends on our time consciousness.

Time consciousness has different aspects for short term, intermediate term and long term periods. In communicating emotions and qualities with dynamic forms we are concerned with shorter periods, times under 10 seconds, as entities or units communicating of emotional meaning, and periods of the order of one hour for the experience of communicative structures ('stories') built from multiple units (e.g. a symphony). In these regions we have observed stabilities of timing of the order of one part in 500 [22, 23]. Such stable timing has been observed regardless of time of day, temperature of the environment, body temperature, including fever of 2-3 degrees above normal, and to a degree, variable acoustics. Studies of timings of performances of musical pieces well known to the performers on different occasions over several years have documented this. Experiments with tapping and mentally rehearsing portions of the same piece, over many years by the same person has illustrated similar stability [22].

These findings suggest that if the intended meaning attributed to the musical piece does not change (i.e. the concept of the music is not altered), a high degree of stability is observed in the performance timing and consequently, we may say, in the individual rate of time consciousness. Moreover, that a particular piece of music, or dynamic expressive form, will tend to have similar meaning to different persons at a given tempo suggests that their relative time consciousnesses cannot be too widely different. One could conjecture at most differences of the order of 5% between individuals, but it could be considerably less than that. This rate of time consciousness does not appear to change with age over an adult lifetime, and may be even similar at an earlier age. Clearly, there are neurobiologic clocks operating in the brain which are involved in this stability, and most likely they are of molecular rather than neuronal character [23].

A quite different aspect of our experience with time is found in relation to our sense of boredom and excitement. Here periods of the order of hours may seem relatively long, or short, depending on our engagement in activity and its degree of fascination. On this scale the experience of passage of time appears to be also highly age dependent: For a child an hour may seem an interminable period, especially when it is filled with an unwanted activity or condition. As one gets older an hour seems to shorten progressively, and days and weeks appear to go by considerably quicker, with a ratio for a sixty year old of perhaps as much as 3:1 compared with childhood. That this should happen although the time consciousness involved in the expression of dynamic expressive forms (eg. music) remains unchanged is very remarkable and appears to attest to the existence of different, less precise processes in this range.

A third aspect of time consciousness is found in the estimate of longer periods, of the order of a few hours to one day, and involve functions which make it possible, for example, to wake up at a specific time without the benefit of alarm clocks, simply by an act of will, i.e. a pre-determined period entered by the mind into itself. This ability does not seem to degrade with increasing age, and can also be remarkably precise.

In our memories experiences are ordered in time, both in short term memory and long term memory, although the method of tagging may well be different for each. (If one could not tell the order in time of two consecutive events, all thought, logical or otherwise would be impossible.) Thinking, like a computer program, is not reversible in time. Time consciousness is the only solid scientific evidence we have for the direction of time. Physical laws do not really define it (they work equally for both directions) and the laws of thermodynamics allow us only statistical inferences. (see footnote 3)

Furthermore, time consciousness occurs in the present. The contents of consciousness changes, or, we can say, the content of the present changes, but the present (itself) remains the same, unchanging.

Physicists have brainwashed us into a habit of regarding time as a straight line, going from left to right in which the present is considered as a mathematical point which moves along this line at an unspecified rate. It is, in fact, meaningless to physics to ask what the rate of movement of this point is, along the line. What events take place are described by a change from T1 to T2 ("as T goes from T1 to T2"!). The present does not enter at all. Physics cannot deal with the present.

But the past is gone and the future is not yet here. All that is here is the present. Physics gives us an insufficient view of what exists, leaving out the present - which is all that really exists - and in which the eternal (ie.at some level unchanging) laws of physics operate.

But not so, time consciousness. Through it we are aware of the present moment and may distinguish it from both the past and the future. How it does that is unknown. We can say, however, that time consciousness is entirely relative to the species in which it is found. This relativity of time consciousness has nothing to do with the theory of relativity. It concerns rather the concept that individuals from a different galaxy might for example experience time in such a way that for them, night and day might appear as a flicker. Or, a fly on earth may well have a different time consciousness from a human. There is nothing absolute about time consciousness.

For a stone which may be assumed to have no time consciousness, it has no meaning to say that a given time period is either long or short. Thus, a billion years for a stone are not long and a microsecond not short, as it has no time consciousness.

For simplicity we propose that we denote human real time as real-real time when considering communicating with computers and robots.

HOW DO COMPUTERS AND ROBOTS EXIST IN TIME?

Clearly, computers and robots have no time consciousness. If and when they do acquire time consciousness, this would have to be especially engineered to be scaled according to human time consciousness or real-real time, for compatibility with humans. Materially, a computer is not essentially different from a stone. Unlike a stone, however, it operates with a succession of logical operations. Each operation is actually carried out between the ticks of the computer clock. Changing the rate of the computer clock, or even making it uneven has no effect on the outcome of its calculations. The numbers by which successive operations are given time tags are no different from any other series of numbers which the computer may store and handle.

At each tick of a computer clock the computer is in fact totally stationary as far as its calculating functions are concerned. It is, in fact, "dead". Only between ticks does it actually "work". And here the time taken is not known, and does not enter into the picture as long as its fast enough so it can be completed before the next tick occurs.

How does the computer therefore relate to the present?

Can we consider the computer to operate according to the image of time which we have learned from physicists? Its activities may be considered to span a period of from T1 to T2. The result of the calculation depends only on the sequence of logical operations or instructions carried out and is itself totally independent of how long it might have taken to complete them.

When a computer operates in real time it harmonizes the sampling rate of input and output so as to reproduce or produce output behavior corresponding to the physical changes which it attempts to model. Its calculations merely have to be fast emough to keep up with the sampling rate.

It is not concerned with human time consciousness nor that of any other species, nor with the human experience of the present.

But when a computer performs a time-form such as emotionally expressive music or speech, its real time operations need to reflect the real-real time experience of humans. In that domain a 1% difference in the time scale will already noticeably alter meaning. A 10% change is a considerable change for the meaning of music, for example. In speech a 10% change will not change the meaning of the words but will alter the effectiveness of the emotional "overtones" which the message carries. Accordingly therefore when communicating emotional qualities to humans the computer cannot choose its own convenient scaling but must match the output to human time consciousness, or real-real time. This requirement is unnecessary when a computer prints out verbal messages or presents visual information such as emotionally expressive faces, for example, as long as those faces are not in expressive motion.

Departure from real-real time results in the emotional qualities acquiring a Mickey Mouse character. They are still recognizable as representing particular emotional qualities, but lose most of their emotional impact. To understand this better we need to consider that the emotions as transmitted by these dynamic forms, which we have called sentic forms, also contain and imply cognitive substrates . Thus, we have shown experimentally that with the feeling of love, there is an openess and guilelessness as part of the inherent biological program [11]. Even a small lie effectively blocks the experience of love at that time. Similarly, grief affects memory function: the ability to learn, and short term memory are diminished by the feeling of grief, and the consequent loss of interest in interaction with the environment. The cognitive substrates are the first to disappear when the dynamic forms are distorted [24]. Thus the Mickey Mouse characteristics imply that the emotions denoted are robbed of their cognitive substrates. But it is precisely the cognitive substrates that provide the elements of profoundity which may be experienced through really true emotional expressions and of its sequences. Thus a piece of music may seem profound not merely because it contains a sequence of emotions but because the sequence of emotions implies a series of cognitive substrates which give it a widely and deeply probing story.

In the use of computers and robots to communicate emotions to humans, one therefore has a choice to communicate merely signs a la Mickey Mouse or to represent genuine, "sincere" human expressions and emotions. The particular applications will decide which approach may be more appropriate. If it is decided to use "sincere" expressions we have then the power in real-real time to make these expressions more powerful and more convincing than the average human would tend to produce under most average conditions. We have at our fingertips the knowhow to make these expressions powerfully communicative, contagious, and seductive in the manner of the very best that any human can do. (see footnote 4) We can optimize it beyond the abilities of the average human: we can optimize it to the degree of which our most powerful art is capable. Whether we chose to do so and for what reasons and needs, surely will comprise a new branch of social ethics which badly needs to be developed.

One positive way of using the new powers opened up by this Pandora's Box opened by the findings of our research is to provide great musical performances. How this can be done we will attempt to demonstrate with a performance of the Mozart Sonata K330, which will be brought into conjunction with the best available performances on CD by great artists for the first time. That this can be done now is only the first step in an endeavor that has many further possibilities, not easy to fathom.

Appendix A

Demonstrations of the Meaning and Precision of Time Forms

A Computer Interpretation of Mozart's Piano Sonata K330, which utilizes the composer-specific hierarchic pulse principle will be heard, along with 6 other performances of the greatest recordings available on CD of this Sonata. The music panel and the audience will be asked to rate the performances, and to pick which one is the computer performance. This test will compare the real-time performane of a computer in terms of musicality with the best real-real time human efforts.

Also played will be a computer interpretation of Bach's Air on the G string, for four independent voices, employing both Predictive Amplitude Shaping, and the Hierarchic Pulse.

The purpose is to demonstrate the degree of understanding of the principles of unconscious musical thought, of musicality.

In a second presentation, emotionally expressive sounds will be presented, which were transformed from touch expressions of the same emotion, by a transform that conserves the dynamic form - to illustrate that the nature of a particular emotion expression depends on the dynamic form, and not on the output mode, ie. is largely independent of the output mode. White urban touch expressions of specific emotions transformed to sound expressions were tested on Australian Aboriginees, for additional cross-cultural validation [13] .
Appendix B

A Short Note on the Development of Consciousness

In the context of this paper some remarks on the natural development of consciousness may be permitted. Experience of emotion is not possible without consciousness. Also, sensory experience, such as red for example, has continuity in time, and stability over long time (in addition to its unique quality), which are so far difficult to ground on the discontinuous events in time and in space of the multiple neuronal events accompanying the experience. Accordingly some thoughts on how consciousness may have arisen in nature may be assayed, in view of our developing knowledge of molecular biology. As a newly evolved phenomenon, it is of such importance that it seems not unlikely that its potentiality is provided for in the laws of nature, ie. that like water, say, it appeared in the evolution of the universe not as a total surprise. Contrary to a prevelant view according to which consciousness is deemed to become possible only when complexity increases to a sufficiently high degree, i.e. that consciousness is essentially complexity- related, the author puts forward a different view of what may be required for consciousness to arise. This approach is outlined briefly here, and will be described elsewhere more fully.

In exploring such a view, one should also consider the pervasive unconscious mental functioning, dreaming, and especially the unexplored questions of the boundaries between the unconscious and the autonomic, and whether unconscious mental function predates consciousness.

   1. Content of Consciousness --- Humans are able to see, hear, smell and touch simultaneously without notable interference of one sensory experience with the other. This defines in size, and in also complexity, a minimum capacity of consciousness even without considering other mental functions.

   2. Animals, even relatively primitive animals, appear to share this capacity of consciousness with humans in that they too can simultaneously see, hear, smell, touch and so on. Moreover, many primitive animals can sense some of these variables with higher resolution than can humans. Consequently, with regard to these functions one cannot take the view that animals have a smaller capacity of consciousness. The argument that such animals have no consciousness at all and act as reflex automatons is rejected (animals clearly appear to make decisions based on their sensory experience involving many of these variables, aneasthetics are used to eliminate consciousness in these animals, etc.).

   3. One of the earliest qualities of experience developed in evolution is hunger. Experience of hunger - a remarkable "invention" of nature, replaced chemotaxis and served as a fount of knowledge for the animal concerning what to eat, when to eat, and how much to eat - all of which is encapsulated in the experiential entity, "hunger". But hunger, as well as experience of sexual attraction which also developed at an early stage, can exist and function only if there is consciousness.

   4. It is suggested therefore as plausible that consciousness itself may have developed through newly evolved genes at a relatively low stage of evolution. It would be supposed that these genes for consciousness produce certain proteins or other gene products which cause consciousness in the brain. According to this view, it is thus not complexity per se but these specific genes that would account for the emergence of consciouness in evolution. (If this is so, the capacity for being a little bit conscious would appear somewhat like a little bit pregnant.)

      As progress is being made in the human genome project it is possible that such genes for consciousness could be identified in decades to follow. As we share so may of our genes with animals we may also share such consciousness creating genes with them.

   5. In achieving consciousness through the interaction of proteins and other gene products in a totally unknown way it could be that a physical law would be invoked which is as yet unknown - a law which would in effect provide for the establishment of a Leibnitzian monad, as a result of specific molecular interaction. This kind of creation of one from many, similar to the creation of the oneness of the molecule from so many atoms (the potentiality of which preexists, predictable through the laws), and in some ways in effect reminescent of a field which at any one point automatically and necessarily summates the effects of many contributing sources in space, but obeying relationships as yet quite unknown would seem, according to this view, to be necessarily involved in the phenomenon of consciousness.

      Accordingly, machines that do not possess the gene functions for consciousness would not become conscious no matter how complex they might become.

      This does not preclude that those effective gene functions might be reproduced by an alternative molecular realisation, so that consciousness could then also be produced by a different configuration of matter than in natural evolution on this planet. But that would depend on the specific nature of the interaction and functions; it could also be that the solution realised by nature which we observe on earth is the only one possible.

References

[1] Langer, S. (1946) Philosophy in a New Key.

[2] Langer, S. (1953) Feeling and Form, New York, Scribner

[3] Piechowski, M. (1981). "The logical and the empirical form of feeling", J. of Aesthetic Ed., 15,1,31-53

[4] Clynes, M. (1969). "Precision of essentic form in living communication," in Information Processing in the Nervous System, edited by K.N. Leibovic and J.C. Eccles, (Springer, New York), pp. 177-206.

[5] Clynes, M. (1970). "Towards a view of Man", in Biomedical Engineering Systems, edited by M. Clynes and J. Milsum , (McGraw-Hill, New York), pp. 272-358.

[6] Clynes, M. (1973). "Sentics: biocybernetics of emotion communication", Annals of the New York Academy of Sciences, 220, 3, 55-131.

[7] Clynes, M. (1977). Sentics, the Touch of Emotion. (Doubleday Anchor, New York. New edition, Prism Press, Avery Publishing, New York, London, 1989.

[8] Clynes, M. (1980). "The communication of emotion: theory of sentics", in Theories of Emotion, Vol. 1, edited by R. Plutchik and H. Kellerman, (Academic Press. New York) pp. 171-216.

[9] Clynes, M. (1988). "Generalised emotion, how it is produced and sentic cycle therapy" in Emotions and Psychopathology edited by M.Clynes and J. Panksepp, (Plenum Press, New York) pp. 107-170.

[10] Clynes M., and Nettheim, N., (1982). "The living quality of music, neurobiologic patterns of communicating feeling", in Music, Mind and Brain: the Neuropsychology of Music, edited by M.Clynes, (Plenum, New York) pp. 47-82.

[11] Clynes, M., S. Jurisevic, and M.Rynn (1990). "Inherent cognitive substrates of specific emotions: Love is blocked by lying but not anger", Perceptual and Motor Skills, 70, 195-206.

[12] Aggleton, J.P. and Mishkin, M. (1986). "The amygdala: sensory gateway to the emotions", in Emotion: Theory, Research and Experience, edited by R. Plutchik and H. Kellerman, Vol 3, (Academic Press, New York).

[13] Clynes, M. (1983). "Expressive microstructure linked to living qualities" in Publications of the Royal Swedish Academy of Music, No. 39 edited by J. Sundberg pp.76-181.

[14] Clynes, M. (1985a). "Secrets of life in music" in Analytica, Studies in the description and analysis of music in honour of Ingmar Bengtsson. Publication of the Royal Swedish Academy of Music, No 47 pp. 3-15.

[15] Clynes, M. (1985b). "Music beyond the score", Communication and Cognit., 19, 2, 169-194.

[16] Clynes, M., (1987). "What a musician can learn about music performance from newly discovered microstructure principles, P.M. and P.A.M", in Action and Perception of Music, edited by A. Gabrielsson (Publications of the Royal Swedish Academy of Music, No. 55, Stockholm) pp. 201-233.

[17] U.S Patents 4,704,682, 4,763,257, 4,999,773, Jap. Pat. Pend., EEC.Patent.

[18] Patent pending.

[19] Patent pending.

[20] Becking, G. (1928). Der musikalische Rhythmus als Erkenntnisquelle (Filser, Augsburg, Germany).

[21] Clynes, M. (1986). "Generative principles of musical thought: Integration of microstructure with structure", Comm. and Cognition, CCAI, Vol. 3, 185-223.

[22] Clynes, M., and Walker, J., (1982). "Neurobiologic functions of rhythm, time and pulse in music", in Music, Mind And Brain: the Neuropsychology of Music edited by M. Clynes, (Plenum New York) pp. 171-216.

[23] Clynes,M. and J. Walker (1986) "Music as Time's Measure", Music Perception 4,1,85-120.

[24] Clynes, M. "Composer's pulses are liked best by the best musicians", in press.

[25] Clynes, M. (1990). Mind-body windows and music", Musikpaedagogische Forschung, Vol 11, 19-42 Verlag Die Blaue Eule, Essen, Germany

[26] Hama, H. and Tsuda, K., (1990). "Finger-pressure waveforms measured on Clynes' sentograph distinguish among emotions". Perceptual and Motor Skills, 70, 371-376

[27] Minsky, M., (1987), The Society of Mind, Simon and Schuster, NewYork

Footnotes

Footnote 1 --- One such way is Sentic Cycles, a simple art form of touch leading to emotional balance and increased joy of living, which has been developed by the author[ 9, 7]. Touch ExPress(tm) is a self-contained hand-held version of the Sentic Cycle kit .

Footnote 2 --- The sentograph is an instrument measuring transient pressure (or force, more precisely) in two dimensions independently; pressure of the middle finger is used for expressing the sentic forms, as voluntary actions ('voluntary' here is a technical term meaning deliberate, conciously initiated by an act of will, transmitted by the voluntary muscle system, just like the movement of the arm in throwing a ball to hit a target.)

Footnote 3 --- When we see a car moving say at about 20 miles an hour, or a person walking, ie. at moderate speeds we do not see a series of stroboscopic pictures, nor a blurred image, and we can estimate the speed of the car, yet a camera will either show it still if the shutter speed is fast enough, or with a blur. From a single photo we can judge the speed only from the degree of blur. We cannot tell directly from a sharp photo whether the object is moving or not. (Nor can physics tell the momentum of an object from an instantaneous view-apart from the relativity contraction, it will look just like a stationary object.) The human (and an animal, probably) does it without the need for blurring (blurring occurs only at higher speeds, exceeding the system's capacity), using his or her time consciousness-through a perceptual quality, aided by ratesensitive receptors that are submerged from the successive images that would appear on a photograph. Thus he can tell the direction of movement, and so of time, without needing a blur, nor multiple images. How much can we still learn about dataprocessing from nature!

Footnote 4 --- People with the clinical condition of hypomania often have an abnormally effective and powerful production of sentic forms in speech and gesture; this helps them to convince and seduce others - specific biochemical causes of this heightened expressity are not known.

---

Manfred Clynes

US Patent # 3,691,652

Programmed System for Evoking Emotional Responses
 

---

rexresearch.com

---

reprinted from (rexresearch.com reprint of Dr Clynes work)

US Patent # 4,704,682

( US. Cl. 84/622; 984/309; 984/377; 984/DIG1; Intl. Cl. G10F 001/00; G10H 007/00 )
November 3, 1987

Computerized System for Imparting an Expressive Microstructure to Succession of Notes in a Musical Score

Manfred E. Clynes

Abstract ~

A computerized system into which is fed the nominal values of a musical score, the system acting to process these values with respect to the amplitude contour of individual tones, the relative loudness of different tones in a succession thereof, changes in the duration of the tones and other deviations from the nominal values which together constitute the microstructure of the music notated by the score. The system yields the specified tones of the score as modified by the microstructure, thereby imparting expressivity to the music that is lacking in the absence of the microstructure.

References Cited ~
U.S. Patent Documents:
3,881,387 ~ May., 1975 ~ Kawakami ~ 84/1
3,956,960 ~ May, 1976 ~ Deutsch ~ 364/419
3,972,259 ~ Aug., 1976 ~ Deutsch ~ 364/419
4,022,097 ~ May., 1977 ~ Strangio ~ 84/1
4,026,180 ~ May., 1977 ~ Tomisawa, et al. ~ 84/1
4,058,043 ~ Nov., 1977 ~ Shibahara ~ 84/464
4,177,706 ~ Dec., 1979 ~ Greenberger ~ 364/718
4,329,902 ~ May., 1982 ~ Love ~ 84/1
4,344,347 ~ Aug., 1982 ~ Faulkner ~ 364/722
4,378,720 ~ Apr., 1983 ~ Nakada, et al. ~ 84/1
4,391,176 ~ Jul., 1983 ~ Niinomi, et al. ~ 84/1
4,417,494 ~ Nov., 1983 ~ Nakada et al. ~ 84/1
4,476,766 ~ Oct., 1984 ~ Ishii ~ 84/1
4,492,142 ~ Jan., 1985 ~ Oya et al. ~ 84/1

Other References:
Ward, Brice, Electronic Music Circuit Guidebook, TAB Books, Blue Ridge Summit, PA, 1975, 27-29.
Clynes, Manfred, "Secrets of Life in Music," Analytica, Mar. 1985, 3-15.
Clynes, Manfred, "Music Beyond the Score," Somatics, vol. 1, No. 5, Autumn-Winter 1985, 4-14.
Beyer, William H. (Editor), CRC Standard Mathematical Tables, 26th Edition, The Beta Function, CRC Press, Inc., Boca Raton, Fla., 1981, 400.

Description

BACKGROUND OF INVENTION

1. Field of Invention:

This invention relates generally to a technique for manipulating the nominal notational values of a musical score with respect to the amplitude contour of individual tones, the relative loudness of different tones, slight changes in tone duration and other deviations from the nominal values which together constitute the expressive microstructure of music. More particularly, the invention deals with a computerized system capable of manual or automatic operation for impressing an expressive microstructure on a musical score inputted therein in terms of nominal notational values, the system being usable for composing music that includes microstructure.

Music has been defined as the art of incorporating intelligible combinations of tones into a composition having structure and continuity. A melody is constituted by a rhythmic succession of single tones organized as an aesthetic whole. The standard system of notation employs characters to indicate tone, the duration of a tone (whole, half, quarter, etc.) being represented by the shape of the character and the pitch of each tone by the position of the character on the staff. In such notation, a melody is a musical line as it appears on the staff when viewed horizontally.

While the notation of a musical score gives the nominal values of the tones, in order for a performer to breathe expressive life into the composition, he must read into the score many subtilties or nuances that are altogether lacking in standard notation. Some expressive subtilties are introduced as a matter of accepted convention, but most departures from the nominal values appearing in the score depend on the interpretive power of the performer.

Thus, a musical score, while it may indicate whether a section of the score is to be played loudly (forte) or softly (piano), does not generally specify the relative loudness of component tones either of a melody or of a chord with anything approaching the degree of discrimination required by the performer. The performer decides for himself how loudly specific notes are to be played to render the music expressive.

Even more important to an effective performance is the amplitude contour of each tone in the succession thereof. To satisfy musical requirements, the amplitudes of the tones must be individually shaped. Though, in general, amplitude contours are completely unspecified in standard notation, each performer, such as a singer or violinist, who has the freedom to shape tones, does so in actual performance to impart expressivity thereto. Indeed, with those instruments that lend themselves to tone shaping, variations in the amplitude shapes of the tones constitute a principal means of expression in the hands of an expert performer.

Another factor which comes into play in the microstructure of music are subtle deviations from the temporal values prescribed in the score. Thus, in actual performance, to avoid temporal rigidity which dehumanizes music, the performer will in actual practice amend the nominal duration values indicated by standard notation. These nominal values are arithmetic ratios of simple whole numbers such as 1/2, 1/3, 1/4, 1/16, etc.

Yet another expressive component of music which is unspecified in the score is the timbre to be imparted to each tone; that is, the harmonic content thereof. A performer of a string instrument, by varying the pressure and velocity of the bow on the string, can give rise, not only to variations in the loudness of the tone, but also variations in its tonal timbre independently of loudness. On a wind instrument, the performer can achieve similar effects by changes in lip pressure and wind velocity.

In short, the macrostructure of a musical composition is defined in the score by standard notation. If, therefore, one executes this score by being assiduously faithful to its macrostructure, the resultant performance, however expertly executed, will be bereft of vitality and expression. The term "microstructure" as used herein encompasses all subtle deviations from the nominal values of the macrostructure in terms of amplitude shaping, timing, timbre and all other factors which impart expressiveness to music.

It has been found that the measure of expressivity that can be attained using only sinusoidally-generated tones whose amplitude is shaped is quite surprising, despite the absence of harmonics which enrich the tones. Our ears appear to be highly sensitive to changing amplitudes and shapes, and our memories can effortlessly detect relative amplitudes and amplitude-shapes sounded in sequence in a musical context, even when the corresponding tones are sounded up to 10 or 15 seconds apart with many other tones in between.

This faculty of short term memory for comparing tone amplitudes and tone shapes makes it possible for the typical listener to distinguish between identical forms that are mostly perceived as mechanical and monotonous, and slightly varying forms; for the latter, played even a few seconds apart, are perceived in relationship to one another and can produce varied meaning and vitality. Thus, the relationships which constitute the microstructure in music, though not explicit in the score, are vital to its appreciation.

Essential also to an understanding of controllable microstructure in a system in accordance with the invention are A essentic forms; that is, the dynamic expressive forms of specific emotions; and B, the inner pulse of composers. These will now be separately considered.

As explained in an article by Clynes and Nattheim (pp47-82) included in Music, Mind and Brain: The Neuropsychology of Music, M. Clynes (ed.), Plenum Press, New York (1982), touch expressions of specific emotions such as love, grief and hate, can be transformed into sound expressions of like emotions; i.e., the nature of the transforms was found so that the sound expresses the same emotional quality as the touch expression from which it is transformed.

The touch expressions are measured by recording the transient forms of finger pressure when these are voluntarily expressed. The instrument enabling this measurement to be made is called the Sentograph; it measures both the vertical and horizontal components of finger pressure independently as vector components varying with time. The sentographic forms obtained are stored in a computer memory and can be reproduced at will. (See Clynes patent No. 3,755,922, "System for Producing Personalized Sentograms" which discloses in greater detail the nature of essentic forms and how sentographs are produced.)

In transforming the sentograms for touch expression of specific emotions into corresponding sound expressions it was found that the dynamic form (essentic form) of the touch was preserved to become the frequency contour of the sound. The sound is a frequency and amplitude modulated sinusoid. The amplitude modulation also was related to the dynamic touch form but needed to be passed through an imperfect differential network.

These expressive sound shapes of "continuous" frequency modulation are related to melodies which represent a form of "discrete" frequency modulation. As spelled out in the above-identified Clynes and Nattheim article, it is possible to create musical melodies which express an emotional quality similar to dynamic, expressive sound forms in such a way that the melodic steps of the created melody act to outline the frequency contour of the "continuous" form, and the amplitude contour of the expressive sound is preserved. As different melodic steps are chosen, the durations would be constrained so as to conform to the "continuous" frequency modulation wave. (essentic form).

The inner pulse of specific composers such as Beethoven, Mozart and Schubert, are expressed by sentographic forms obtained by having an individual think the music of a selected composer in his mind and by concurrently expressing the pulse by "conducting the music on a sentograph with finger pressure. The resultant sentograms indicate that major composers, such as those previously identified, impart individual pulse forms to their music which characterize their creativity identity or personal idiom. This inner pulse characteristic of each composer is, to a degree, analogous to individualistic brush strokes which distinguish one painter from another, regardless of the subject matter of their paintings. (See M. Clynes, "Sentics, The Touch of Emotions" - published by Doubleday - 1977.)

SUMMARY OF INVENTION

In view of the foregoing, the main object of this invention is to provide a computerized system for processing the nominal values of a musical score to impart an expressive microstructure thereto.

More particularly, an object of the invention is to provide a system of the above type which is operable in a manual mode in which the values representing the microstructure are entered by the user, or in the automatic mode wherein microstructure values are calculated from a shaping function responsive to the melodic contour and by means of pulse matrices having certain values relating to the amplitude and duration of pulse component tones stored in the system.

Among the significant advantages of a system in accordance with the invention is that it makes it possible to enliven a musical score and render it highly expressive, the system thereby acting to deepen the user's understanding and appreciation of music and its structure.

Briefly stated, these objects are attained in a computerized system into which is fed the nominal values of a musical score, the system acting to process these values with respect to the amplitude contour of individual tones, the relative loudness of different tones in a succession thereof, changes in the duration of the tones and other deviations from the nominal values which together constitute the microstructure of the music notated by the score. The system performs the specified tones in the score as modified by the microstructure, thereby imparting expressivity to the music that is lacking in the absence of the microstructure. The microstructure may include changes in pitch or vibrato.

OUTLINE OF DRAWINGS

For a better understanding of the invention as well as other objects and further features thereof, reference is made to the following detailed description to be read in conjunction with the accompanying drawings, wherein:

FIG. 1 illustrates a family of curves generated from one set of beta fraction values;
 
 
 

FIG. 2 illustrates a family of curves generated from a second set of beta function values;
 
 
 

FIG. 3 illustrates a family of curves generated from a third set of beta function values;
 
 
 

FIG. 4 illustrates a family of curves generated from a fourth set of beta function values.
 
 
 

FIG. 5 illustrates a family of curves generated from a fifth set of beta function values.\
 
 
 

FIG. 6 illustrates a family of curves generated from a sixth set of beta function values;
 
 
 

FIG. 7 illustrates the notes of a Mozart theme below which are three graphs representing different microstructural aspects of the theme;
 
 
 

FIG. 8 similarly illustrates the notes of a Chopin theme;
 
 

FIG. 9 similarly illustrates the notes of a Beethoven theme;
 
 
 

FIG. 10 is a block diagram of a computerized system in accordance with the invention which is operable in the manual mode;
 
 
 

FIG. 11 is a graph illustrating how an amplitude shaped tone is obtained from a succession of digital values yielded by the calculator included in the system;
 
 
 

FIG. 12 shows both a sinusoidal wave and a differentiated square wave which, when combined, produce a non-sinusoidal wave rich in harmonics; and
 
 
 

FIG. 13 illustrates in block form a portion of a system operating in the automatic mode.
 
 
 

DESCRIPTION OF INVENTION

Microstructure and Expressiveness

Consideration will first be given to the extent to which detailed expressiveness is possible without vibrato and timbre, using only sinusoidal tones with individually-shaped amplitude envelopes, representing the notes of the music, with duration modification. Such amplitude modulation appears to be the most basic mode of dynamic expression and thus the understanding of this should precede the analysis of the expressive effects of vibrato, timbre and of timbre variation. It has been found that a significant degree of expressiveness is indeed possible with these very simple means, and that many of the subtlest nuances can be realized.

In doing this, we shall also point out the relationship existing between the amplitude shapes of individual tones and the shape of the melody (i.e. essentic form). We shall show that a simple relationship holds between the shapes of individual tones of a melody and the form of its time course. Its shape within a continuum of upright ("assertive") and forward sloping ("plaintive") form is seen to be a function of the slope of the essentic form. (Assertive will hereafter be represented by the letter A and plaintive by the letter P) This means in practice that the shape of a tone is strongly and predictably influenced by the pitch of the next tone and by the time when it occurs. The shape of individual tones has a musically predictive function contributing to continuity, and thus, we may say, to musical logic.

We shall also demonstrate how the individual pulse characteristics of a number of composers, in particular Mozart, Beethoven and Schubert, systematically modulate the pattern of tones both in amplitude and temporally, in characteristic ways which will be precisely given. In the course of these considerations we shall also outline how certain microstructure functions, involving durations, silences, and amplitude shapes, relate to various aspects of phrase and larger structure.

Beta Function for Calculating Amplitude Shapes

In order to produce convenient shapes for amplitude-modulating individual tones, we have used a mathematical means, briefly called the Beta Function. This term derives from a similar-named function in mathematical statistics. The Beta Function permits us to create a wide variety of shapes with the aid of only two parameters (P.sub.1 & P.sub.2). It has a considerably wider applicable scope than the use of two exponentials would have, for example.

In electronic generation of musical sounds, it has heretofore been conventional to specify tones using parameters of rise time, decay time, sustain time, release time and final decay, or some subset of these. These parameters, natural to the electronic engineer, do not really have a musical function of like aptness. Amplitude shapes of musical tones often need to be convex rather than concave (or vice versa) in particular portions of their course (e.g., convex in their termination), and hardly ever have sustained plateaux. Moreover, separation of the termination of a tone into a decay and a release is generally the result of the mechanical properties of keyboard instruments and not a musical requirement.

We have found that the varied rounded forms available through the Beta Function allow a simple and time-economical realization of the multitude of nuances of musical tone amplitude forms. Beta Function is defined as:

x.sup.p.sbsp.1 (1-x).sup.p.sbsp.2 for 0 .ltoreq.x .ltoreq.1, Equation 1

and is normalized for a maximum amplitude of 1 by dividing by a constant. ##EQU1## for a particular set of values of p.sub.1 and p.sub.2. p.sub.1, p.sub.2 have values .gtoreq.0.

The resulting shape is multiplied by a parameter G to give the amplitude size of the particular tone. The shape stretches over a number of points determined by the duration of the tone.

By choosing suitable values of p.sub.1 and phd 2, a shape may be selected from families of shapes such as the ones shown in FIGS. 1 to 6. Choosing the value 1 for both parameters gives rise to a symmetrical, rounded form, and 0.89 for both parameters produces a form very close to a sine half wave. Smaller values of p.sub.1 result in steeper rises; zero being a step function. Larger values than 1 for either p.sub.1 or p.sub.2 make the curve concave, at the corresponding regions. A combination of zero and 1 results in a sawtooth.

Most commonly used p values for musical tones generally lie within the region of 0.5 to 5, and most frequently in the region of 0.7 to 2. Where required, a second or several more Beta Functions may be added to produce the desired shape--this is seldom necessary, however.

FIGS. 1 to 4 illustrate different families of Beta Function shapes, showing some of the kinds of shapes readily obtained by choosing appropriate p.sub.1 and p.sub.2 values. In each group of curves, pairs of p.sub.1, p.sub.2 values are given starting from the leftmost curve. Maximum amplitude is normalized as 1.

FIGS. 5 and 6 are Beta Function shapes illustrating degrees of skewness, starting from a symmetrical (1, 1) form, with pairs of p values forming a series as shown. These are types of shapes used for the amplitude of many musical tones (called A [left group] and P [right group] types).

Computer Program for Shaping and Playing Melodies:

The Beta Function is used in a computer program that calculates individual tone shapes. In this program, the amplitude character of a tone is specified by three numbers, respectively denoting the amplitude magnitude G, and parameters p.sub.1 and p.sub.2. We use a linear scale for the magnitude with a resolution of 1 part in 4096. This in accordance with our findings that such transient sound phenomena may often be better understood by changes in amplitude on a linear rather than logarithmic scale. It also presents a visually more tractable aspect. Silences of various durations in the millisecond and centisecond range can also be readily inserted between tones.

The duration of each tone is specified by the number of points it occupies over which the Beta Function is calculated. The calculation for each tone is done without affecting the duration and number of points of other tones. The temporal resolution of tones is usually better than 1 millisecond. In practice, the amplitude contour may be constituted by a 12 bit DA Converter and modulates a voltage-controlled amplifier (VCA) of linearity better than 0.1% over a dynamic range of 1 to 4096. The frequency of the tones is set by another channel of a DA converter which modulates a voltage-controlled oscillator (VCO).

In practice, the invention may be realized with D to A converters having 8 to 16 bits. One may also use a digitally-controlled VCA and VCO which can, in effect, be integrated within a digital synthesizer, in which event there is no need for a D to A converter to operate the VCA and VCO. In that case, the ultimate sampling rate per channel must be 44 kHz or higher to obtain good quality results.

The FORTRAN IV program was run on a PDP 11-23 compurer. The tempo can be varied over a very wide range. Parameters of any tone can be readily varied and the changed result listened to in a few seconds, typically 2-10 seconds. Any desired portion of the music can be listened to. The maximum length of the piece to be played is only limited by the length of the microscore that can be stored on the disc. In practice, many tens of thousands of tones can be stored.

Manually-Created Microstructure:

Before considering examples of manually-produced microstructures, some general comments may be appropriate.

(i) It may take a musical person a day or two to perfect such a theme by gradually improving the values of the parameters, and repeatedly listening until it is refined to a degree where he no longer is sure what change would be an improvement.

(ii) As one listens repeatedly and changes parameter values for a time, one eventually experiences satiation or lessening of sensitivity to particular aspects and/or the totality of the expression. It appears that one's sensitivities in this regard are dulled with repeated exposure, and need some time to recover, to regain their spontaneous freshness. A few minutes is sometimes sufficient, but after many hours of work, several hours or overnight may be required to revitalize one's listening.

(iii) Working on a theme in this manner in fact sharpens one's hearing and attention to detail in regard to expressive qualities, and can be considered invaluable for that reason alone.

(iv) A further phenomenon occurs as one continues working with a theme in this manner: the theme teaches one its own nature. As one repeatedly interacts, many new aspects, relationships and meanings become clear. One becomes more and more involved with the theme, and it gains more vitality and clarity in one's mind. The goal becomes clearer; yet its form also develops as one works towards it. The process becomes self-refining, a systematic interaction with a stable limit; assymptotic and not seemingly finite, but stable. When one finally reaches the point of not being able to improve it any further, one feels this is not because it could not be improved, but rather that the required changes are so subtle, that greater understanding than can be summoned at the time for A-B comparison would be required in order to continue. But still, there is joy and satisfaction in having created something vital, particularly on hearing it freshly at a later time.

This method of sculpturing tones and melodies allows a musical artist, then, to perfect the expression in much the same manner as a painter or a sculptor can, working with a painting for long periods of time, gradually perfecting the forms so that they correspond to his inner vision, a vision which itself becomes more perfect as the interaction grows. At what stage to say "it is enough" depends on a higher level of integration where another vision and its realization interact in a like manner.

It is also in many ways similar to the process of practicing for a musical performance, in the course of which the artist, enamoured with the piece, refines his performance and understanding through repeated reciprocal interaction-feedforward and feedback.

Resolution of Parameter Values:

Sensitivity in discriminating different shapes of tones is typically of the order of 0.01-0.02 in the p values in the range of 0.5 to 2 (most commonly used). For larger values it is correspondingly greater. The limen of discrimination of the magnitude of amplitude peaks within a melody is of the order of 2% or about 1/4dB. This means that the ear is more sensitive to the shape than to peak amplitude. For example, a difference in shape resulting in 2% deviation of critical portions of the shape (referred to peak amplitude) will be considerably more noticeable than a 2% change in overall amplitude Concerning relatively critical portions of the shape of a tone with respect to sensitivity, see Clynes and Walker, supra.

Mozart Quintet

An example of a theme embodied by this method is the first eight bars of the Mozart Quintet in G minor, as shown in FIG. 7.

In this figure, the stave on top shows the notes of the melody.

Graph 1 below this represents the pitch of the sinusoidal tones. Small markers on this graph indicate repetition of the same note and micropauses.

Graph 2 is the amplitude contour (in linear scale from 0 to 4096, the 200 point being thus equivalent to -26 dB referred to as the loudest level; 1000 being played at about 50 dB above threshold normally).

Graph 3 represents the temporal deviations from the nominal values for each tone, in percent, upward deflection being slower. (Micropauses are often included in the representation.) The time marker at the bottom of all figures represents 1 second.

The digital printout prints only every sixth point of the functions--actual resolution is this six times greater than that shown in the illustration.

The table shown below for the Mozart Quintet and similar ones gives a list of all the tones and rests of the computer realization, specifying for each:

1. the duration of the tone, or rest, in points

2. amplitude size

3. amplitude shape (p.sub.1 , p.sub.2 )

In some tables, where Beta functions are used to span only part of the duration of the tone, as may occur for staccato tones, the sound duration of staccato tones is given in parenthesis next to the total tone duration. Micropauses are indicated as "P," Rests as "R." When more than one Beta function is used for a tone amplitude, they are listed in vertical sequence for that tone. Metronome mark for the nominal unit used (often a subdivision of a quarter note) is also given.
 

    ______________________________________
    MICROSCORE
    MOZART: QUINTET IN G MINOR, 1ST MOVEMENT
    Duration (sec): 8.73
    MM (100 pt tomes/min): 230.00
                     DUR
    T#    NOTE       (PTS)     AMPL    P1   P2
    ______________________________________
    1     D4         100       800     1.00 1.00
    2     G4         123       1580    1.10 0.88
    3     B4.music-flat.
                      88       480     1.20 0.80
    4     D5         100       950     0.85 3.40
    5     D5         112       430     1.20 4.60
    6     D5         223       660     0.96 0.83
    7     R           80        0      0.00 0.00
    8     G5         105       920     1.10 1.85
    9     G5         133       2100    1.15 1.70
    10    F5#        102       600     0.97 0.80
    11    F5         101       850     0.80 3.20
    12    F5         110       400     1.00 3.60
    13    E5         231       380     1.45 1.40
    14    R           83        0      0.00 0.00
    15    E5.music-flat.
                     105       580     1.20 0.65
    16    E5.music-flat.
                     139       1550    1.28 1.65
    17    D5         103       430     0.85 1.00
    18    D5         105       380     0.85 0.90
    19    E5.music-flat.
                     108       620     0.80 1.10
    20    E5.music-flat.
                     135       800     1.80 1.20
    21    D5          95       200     0.80 1.30
    22    D5         113       400     0.85 1.90
    23    C5          98       240     0.80 1.80
    24    C5         215       530     1.55 0.90
    25    B4.music-flat., C, B.music-flat.
                      62, 40, 38
                               245     0.83 0.80
    26    B4, B4.music-flat.
                      31, 39   210     0.80 0.70
    27    B4         227       235     1.80 1.20
    ______________________________________

The chosen unit of time in the melody (in this case, an eighth note) is assigned 100 points duration nominally, so that a quarter note becomes 200 points nominally, a half note 400 points, and so forth. The actual duration of each tone is modified from these so that a particular eighth tone may have a duration of 96, say, a half tone 220; and so on, depending on its position and expressive requirements.

In this example, all parameters 4 for each tone: duration, peak amplitude, p.sub.1 and p.sub.2) were chosen by trial and error; that is, by repeated listening and gradually improving the values as dictated by "the ear."

The program allows us to play any portion or the entire theme, and will repeat as many times as desired (with a short pause between repetitions). The metronome mark entered (i.e 230 refers to the nominal chosen unit of duration (in the present example, 100 points for an eighth note). If an actual tone has more or fewer number of points than 100, it will have a correspondingly different duration. Minute tempo variations within the theme arise from the differences from nominal values in the number of points for each of the tones. In this example, we may note the following:

1. Amplitude Relationship within a Four-tone Group (One Pulse)

The amplitude relationship between the four eighth notes of the first bar shows that the second and fourth tones are much smaller, the fourth one being a little less than the second. The third tone is considerably larger in amplitude than the second and fourth but less than the first. A similar pattern is repeated in the third bar, but the accentuation of the first tone is even greater. Throughout the theme, the first tone of each bar is considerably larger in amplitude.

2. Peak Amplitudes Outline Essentic Form

The peak amplitudes form a descending curve from bar 3 to bar 4. The form of this descending curve combines with the frequency contour to produce an essentic form related to grief (this form may well be considered to be a mixed emotion: predominantly sad, with aspects of loneliness, anxiety, and perhaps regret). Bars 1 and 2 provide similar forms of diminishing amplitude, but in bar 1 combined with rising frequency. Pain and sadness are implicit in bar 2. Bar 1 suggests a resigned view, accepting fate, without the quality of hope; "this is how it is; there is nothing that can be done about it." The combined effect is a combination of grief with a stoical, strong acceptance of what is; without defiance or rebellion.

3. Individual Tone Shapes

The shapes of each individual tone are governed by their place in the melodic context. The shorter tones may seem similar in shape on the graph, but in fact they are varied, as can be seen from the p values in Table 3. (Small changes in the p values noticeably affect the quality of the sound.) In the longer tones, such as the fifth tone of bars 1 and 2, first and fifth tones of bar 4, the shape of the termination of the tone is as important as its rise, for appropriate expression. For the shorter tones, the termination phase of the tone relates to the degree of legato that is achieved. Smaller p values result in greater legato. It is not generally necessary to include a DC component to maintain a legato between successive tones. The momentary drop in amplitude between tones shaped by the Beta Function is not perceived by the ear if it is quite short, as is the case for appropriately low p values. Staccato tones are produced in the first few examples by choosing appropriate Beta Functions with high p.sub.2 values.

4. Duration Deviations

We may note systematic time deviations from the given note values. First notes in each bar are lengthened, the second shortened. Hardly any tones correspond in duration to the actual note value. Some tones are lengthened by as much a 39% (first tone bar 3). Specially lengthened are first notes of beats 5, 9, that correspond to accentuated dissonant tones, which like suspensions are resolved in the following, second tone of the bar. Such prolongations induce a lamenting quality in the expression.

Further Observations

(a) When working with such a theme, one notices fairly quickly that when one changes the amplitude, or duration, of one note, it affects the balance of all notes: that is, the theme is an organic entity. Increasing the loudness of the fifth tone of bar 3, for example, even by only, say, 5%, or one dB, will affect its relationship with first tones of bars 3 and 4. Also, it affects its prominence compared to its adjacent tones, the sixth tone of bar 3 and the fourth tone of bar 3. But further, the altered amplitude contour now constituted by the peaks of the first and fifth tones of bar 3, and the first tone of bar 4 affects how bars 1 to 2 balance with bars 3 to 4.

Similar organic behavior is evident in changing the duration of a tone. Additionally, changing the duration of a tone also affects its relative emphasis. For example, lengthening the upbeat duration will tend to give greater prominence and energy to the following main beat, which then in turn will affect its relation to the other main beats.

The appropriate way to realize the trill, in bar 4, seemed to be to group the trill in terms of 3 and 2 notes (the latter being its termination), each group under a single amplitude contour.

(c) When a phrase is repeated such as in bar 3, the amplitude relationships may be similar, but never the same. Whether they tend to be augmented or diminished in successive repetitions depends on the specific piece, the structural design, and the nature of the composer. In the present theme, the second presentation is less emphatic than the first.

Chopin Ballade

As illustrated in FIG. 8, the opening theme of Chopin's Ballade Number 3 in A flat shows how a melody written for piano, an instrument of limited ability to vary amplitude shapes, can be expressed by varying amplitude shapes according to its character and not in violation of it. In this example, amplitude tone shapes are realized that are implicit in the melody, and are heard inwardly even when they are not actually produced. One thinks this melody with these shapes. Notable in this example is the strong rubato in the second part of the first bar. This quickening reaches its maximum extent on the fourth eighth note and is counterweighed by a slowing down in much of the second bar. The microscore for this Chopin piece is as follows:
 

    ______________________________________
    CHOPIN: BALLADE NO 3 IN A FLAT, OP 47
    Duration (sec): 5.81
    MM (100 pt tones/min): 145.00
                     DUR
    T#      NOTE     (PTS)   AMPL     P1   P2
    ______________________________________
    1       E4.music-flat.
                     224     840      0.80 0.73
    2       F4       180     800      1.20 0.67
    3       G4        95     800      1.00 1.50
    4       A4.music-flat.
                      80     1150     1.00 0.80
    5       B4.music-flat.
                      98     1680     1.00 0.90
    6       C5       276     2170     1.70 2.53
    P       --        9       0       0.00 0.00
    7       F5       151     810      0.95 2.36
    8       E5.music-flat.
                     340     660      0.85 1.55
    ______________________________________

Beethoven Piano Concerto

FIG. 9 illustrates the first movement, second subject of the Beethoven piano concerto no. 1. The microscore for this piece which illustrates the automatic mode of operation, using both pulse matrix and automatic amplitude and shape calculations (P.sub.1 & P.sub.2 values) is as follows:
 

    ______________________________________
    BEETHOVEN: PIANO CONCERTO NO 1 1ST MOVT
    Duration (sec): 6.44
    MM (100 pt tones/min): 261.00
    Base P1, P2: 1.18 0.82
                     DUR
    T#      NOTE     (PTS)     AMPL    P1   P2
    ______________________________________
    1       D5       400       1000    1.16 0.83
    2       C5#      107       1000    1.30 0.75
    3       D5       89        399     1.46 0.66
    4       E5       96        799     0.96 1.00
    5       D5       108       799     1.18 0.82
    6       D5       107       1000    0.98 0.99
    7       C5       89        899     1.06 0.91
    8       B4       96        799     0.96 1.00
    9       A4       108       799     0.98 0.99
    10      G4       107       850     1.07 0.90
    11      F4#      89        340     0.95 1.01
    12      E4       96        680     0.96 1.00
    13      D4         108(99) 650     1.18 0.82
    P       --       17         0      0.00 0.00
    14      G4       107       1000    1.18 0.82
    15      R        89         0      0.00 0.00
    16      G4       86        800     1.18 0.82
    17      R        108        0      0.00 0.00
    18      A4, G4     24,83   1000    1.07 0.90
    19      F4#      89        400     1.31 0.74
    20      G4       96        800     1.77 0.55
    21      B4       107       800     0.89 1.09
    P       --        7         0      0.00 0.00
    22      G4#      210       1000    1.24 0.78
    23      A4       170       280     1.00 1.00
    ______________________________________
     1 pulse considered to be 1 half note

Automatic Linking of Amplitude Shapes of Individual Tones to Melodic Form

Having observed and worked with a large number of examples in the manner illustrated in the previous section in connection with a Mozart Quintet and considered the interaction between amplitude shapes of individual tones and the music, we were enabled to posit a certain relationship between the amplitude shapes and the melodic course of the music.

A and P Classes of Amplitude Shapes

Let us consider these shapes to belong to a continuum between two classes and intermediate shapes between the two extremes. We may conveniently call these classes (A) assertive and (P) plaintive or pleading, respectively--without wishing at all thereby to tie the musical expression to such categories, of course.)

We can then consider an actual tone shape as placed somewhere along this continuum--and consider the nature of the influence that displaces it from a neutral position (the base shape) to the place on this continuum where it needs to be.

We can describe forms along this continuum by pairs of p values, starting from a base shape (say 1,1) such that as the values of p.sub.1 increase those of p.sub.2 decrease in proportion, and vice versa. Thus, say, for example (0.9, 1.11) , (0.8, 1.25), (0.7, 1.42) etc. will give a series of shapes shifting gradually towards class (A) which has a realtively sharp rise time). The inverse series (1.11, 0.9), 1.25, 0.8), 1.42, 0.7) etc. will tend more and more towards class (P) (of gradual rise time).

We then can consider the influence which causes the shift to be the slope of the essentic form, as measured by the slope of the pitch contour.

More particularly, the deviation from a base shape for a particular tone is seen to be a function of the slope of the pitch contour (essentic form) at that tone: both pitch and duration determine the deviation in such a way that:

(a) Downward (-ve) steps in pitch deviate the shape towards A, upward steps toward P, in proportion to the number of semitones between the tone and the next tone.

(b) Deviations are affected by the duration of the tone so that the longer the tone, the smaller the deviation (since the slope is correspondingly smaller).

Further, in practice, the slope is measured from the beginning of the tone considered to the following tone. In measuring the slope between the tone and the next tone, rather than the previous tone, the amplitude shape acquires a predictive function. The first derivative of a function has a predictive property (lead in phase). The amplitude shape associated with a particular slope leads us to expect a melodic step in accordance with it. The movement of the melodic line is thus prepared.

This appears to match well with its actual function in music, relating the present tone to what is to follow, and may be considered to be in accordance with musical logic. It gives a sense of continuity both musically and in terms of feeling.

Experience shows that the proportionality constant needs to be an approximately 10% change in the p values per semitone, for tones of 250 msec duration. The shift is, of course, to be expected to be linear only over a limited range; a degree of nonlinearity for both the duration and pitch factors existing over a broader range.

In order to see how the duration factor and the pitch factor may obey different power laws, the equation is put into the form: ##EQU2## where s=number of semitones to next tone

b=const. of p.sub.1,2 by frequency

a=const. of modulation p.sub.1,2 by duration

T=duration of tone in milliseconds .sup.P 1(i)' .sup.p 2(i)=base (initial values of p.sub.1 and p.sub.2

Experience has shown that preferred values are in the region of

a=0.00269 b=0.20

for such music.

It is to be understood that a simplified or linear equation may be substituted for equation (3) having approximately the same behavior within the range considered.

Choice of Base Values

There remains to consider the choice of the base form, which we have nominally put at 1,1.

In music of different composers, and of different types, it would seem that certain preferred base values apply. Values in the vicinity of (1.2, 0.8) appear as an appropriate choice for much of Beethoven--giving a greater legato and more gradual attack. For Mozart, base values around (0.9, 1.1) give a more rapid decaying sound, and a somewhat sharper attack. For Schubert, base values in the vicinity of (1.15, 0.9) are seen to be appropriate.

These values are influenced by the type of instrumental sound that the style of the composer appears to require, and may also be linked to historical consideration of the instruments in use at the time. They may also be related to how the inner pulse affects the microstructure, which we will consider in the next section.

Equation 3 relates to the use of Beta Functions to obtain the desired shift in amplitude shape. In practice, when one chooses not to employ Beta Functions, the amplitude shape may be modified as a function of the slope of melodic form using traditional expedients for individually shaping the component tones of a melody; that is, such expedients as attack, decay, sustain and release, which are appropriately varied.

Other Implications

Silence before Downward Leaps

The rule implies that within a "legato" melody for a tone of given duration, the larger the upward leap in pitch to the tone that follows, the more should the tone have a (P) form, and the larger the downward leap, the more marked the (A) form. In the case of a downward leap in moderate and fast tempo this can (depending on the size of the leap and the duration of the tone starting the leap) in a momentary silence--microsilence (due to the tail of the (A) form) before the low tone. This is felt as appropriate in larger downward leaps. In upward leaps, silence will tend to occur before the tone starting the leap.

Expressiveness of Scales

Also implied is that tones of scales have somewhat more (P) shape going up, and going down more (A) (the exact deviation depending on the tempo as given by equation (3)). In fact, scales become considerably more musical when this rule is appropriately observed.

(We may also relate the amplitude shape deviations to the type of gestures with which one would conduct the music; often a downward gesture goes with greater (A)--assertive; an upward gesture with more (P)--plaintive, pleading).

Limitations of the Equation

By the nature of the rule, it cannot predict the amplitude shape of the last tone of a melody (since it measures slope to the next note)--this must be entered manually. (It is possible, however, that a rule referring to last notes of phrases may be formulated which would also take into account aspects of larger structure.) Further, it does not apply to tones which require more than one Beta function.

The basic thought behind this formulation is: if we can consider the available shapes to lie along a continuum between (A) and (P) forms, then the slope of the essentic form at that tone determines where in that continuum a particular tone lies. The particular merit of this formulation may be seen in that it harmoniously integrates the behavior of single tones to that of the larger whole, for different kinds of music. In this, it seems to show a surprising elegance and power--and musicality.

The Composer's Specific Pulse Expressed in Microstructure

In the previous section, we have been how the amplitude shape of individual tones can be derived from the form of the melody. We can now proceed and ask, how much of the remaining unaccounted microstructure can be attributed to the function of the inner pulse? It seems an unexpectedly large amount.

The inner pulse of a given composer is not the same as the rhythm or meter of a piece; it is found in slow movements, in fast movements; in duple time, in triple time, or compound time. The tempo of the pulse has generally been considered to be in the range of 50-80 per minute. In slow movements, one pulse may correspond to an eighth note, even a sixteenth in a very slow movement; in a fast movement a half note; and in a moderate movement a quarter note.

The inner pulse as a specific signature of a composer became established in Western music around the middle of the eighteenth century and continued until the advent of music in whose rhythmic motion there no longer was interfused an intimate revelation of the personality of the composer. In the music of Mozart, Haydn, Beethoven, Schubert, Schumann, Chopin, Brahms, for example, we find a clear and unique personal pulse which the composer has impregnated successfully into his music (the knowledge of which we ultimately acquire from the score). Indeed, because of the time course of the inner pulse, (0.7-1.2 sec. approx. per cycle), the matrix of its wave form is most prominently expressed in microstructure.

In considering the nature of the beat, and of the inner pulse, the property of the nervous system called Time Form Printing is very relevant. This function enables the human organism to decide on a particular form of movement to be repeated, and the rate, and then to repeat the movement at that rate without further attention--until a separate decision is made to stop or to alter the form or the rate of the movement. Once conceived as a repetitive movement and initiated, the movement will continue in "automatic" fashion until stopped. This process takes place mentally when thinking the beat, and the inner pulse.

In practice, this means that once initiated, the inner pulse can carry through the musical piece without need for further specific initiation of form, although the rate will be caused to vary to a degree. Small pauses can momentarily suspend the pulse, and act as punctuation, as it were, in the musical phrasing.

In any composition, certain parts of the score embody the pulse more clearly and obviously than others. However, through Time Form Printing, the pulse will tend to carry through all parts, if it has been well initiated at the beginning of the piece. Scale passages, for example, can become characteristic of the composer through the pulse: the same scale passage can be played with a Mozartian pulse or with a Beethovenian pulse, for example, and will sound appropriate correspondingly. (This does in fact happen in satisfying performances.) Thus "neutral" passages acquire from the pulse the characteristic "flavor" of the composer. This, of course, is precisely the province of microstructure, and we shall show how this occurs.

By experimenting with such neutral structures, and combining the results with knowledge of the inner pulse forms as expressed through touch, we can arrive empirically at an answer to the question: What does the inner pulse do to the specific tones in a piece of music?

Concerning the Derivation of Pulse Microstructure Values

The following considerations have aided the derivation of these values.

Characteristics of the pulse matrix may be in part connected to the kinesthetic "feel" of the pulse, observed in recording it sentographically. We have seen that different degrees of inertia, damping, can be experienced for pulses of various composers, and for some pulses different kinds of tensions occurring at specific phases.

These show up indirectly in the recorded sentographic form; but appear to be significant clues to the deviations in amplitude and duration that constitute the matrix. Different degrees of inertia, or sluggishness, are brought about by different modes of tensions by agonist and antagonist muscle groups (as a result of which greater or lesser massiveness is displayed by the arm--e.g., more of the upperarm mass is involved). This massiveness, a mental program, translates to the degree of flexibility of modulation within a pulse. The Beethoven pulse has high inertia, the Mozart pulse low inertia. Accordingly, we would expect to see much greater differences between the amplitude of component tones for Mozart than for Beethoven. Indeed, there is a relative effortlessness in changing the loudness level in Mozart within a pulse, that is not present in Beethoven.

The massiveness has an influence also on the duration deviations, since each pulse cycle contains an initiating point at a particular phrase (near the time of the upbeat). The deviation values can also be looked at in the light of rhythm studies that relate the energy of a beat to the duration and amplitude of the upbeat, in relation to a given downbeat. High inertia would accordingly tend to be accompanied by a longer duration upbeat.

Pulse Matrix of Mozart, Beethoven and Schubert

A two-fold effect may be observed: The inner pulse affects

1. Relative amplitude sizes of its component tones.

2. Duration deviations of its component tones. Both 1 and 2 must occur. Either alone is insufficient. Accordingly, the influence of a composer's pulse is stated for a particular meter by a matrix that specifies (1) amplitude ratio, (2)duration deviations.

The following matrices specify the influence of the inner pulse for Mozart, Beethoven and Schubert, respectively, the 4/4/ meter. For each tone, two numbers are given. One specifies the amplitude size ratio, referred to the first tone as 1. The other gives the duration referred to as 100 as a mean duration for the 4 components.
 

    ______________________________________
           Tone No.
           1    2        3        4
    ______________________________________
    Amp. Ratio
             1      .39      .83    .81    Beethoven
                    -7.8 db  -1.8 db
                                    -2.0 db
    Duration 106    89       96     111
    Amp. Ratio
             1      .21      .51    .23    Mozart
                    -14.4 db -6.6 db
                                    -13.5 db
    Duration 105    95       105    95
    Amp. Ratio
             1      .65      .40    .75    Schubert
                    -3.1 db  -7.7 db
                                    -1.9 db
    Duration 98     115      99     91
    ______________________________________

Salient Features of the Pulse Matrices

The Mozart Pulse

Large amplitude difference between first tone and the others--archness in articulation, second tone very slightly softer than the fourth tone, the third tone having a subsidiary accent, subdividing the four tones.

Duration deviations are rather symmetrical, first and third tones moderately longer.

The Beethoven Pulse

Amplitude ratios generally more even, fourth tone not softer than third. Second considerably softer than first. Durations: First tone considerably extended

Fourth tone considerably extended.

The extended duration of the fourth tone combined with its relatively high amplitude is a cardinal feature of the Beethoven pulse. The third tone is not extended in duration, and is more nearly equal to the first tone in amplitude than for Mozart. A more even, less arched articulation, with a special aspect to the fourth tone. (A resistance is displayed against "excessive" amplitude modulation--experienced often as a kind of "ethical restraint.") First and second tone taken together are contracted compared with the third and fourth tone together.

The Schubert Pulse

Low amplitude of the third tone--no subdivision; higher amplitude and short duration of the fourth tone. No duration extension of the first tone (or third tone) but a considerably extended duration of the second tone. Third and fourth tones, taken together, are contracted compared with first and second tones together.

The unusually extended duration, without accent, of the second tone of the Schubert pulse can be linked to the following:

1. The special tension that occurs in the corresponding part of the Schubert pulse, a pulling upward, uniquely characteristic of the Schubert pulse. Its initial downward phase acts on low inertia. During the course of a rebound, however, a tension is added, increasing the resistance and slowing down what otherwise would be a carefree bounce, into an entirely different character, an impression of being pulled upwards.

2. It can be seen in the scores of Schubert's music that there is a predilection for special treatment melodically, or in other ways, of the weak tone following the first tone of the bar. In performance and in thought, these deviations are not experienced as unevenness, but appear as an appropriate flow.

Pulse matrix values for triple meters are as follows for the three composers illustrated:
 

    ______________________________________
    Duration     Amplitude Ratio
    ______________________________________
    BEETHOVEN
    105          1
    88           .46
    107          .75
    MOZART
    106          1
    97           .33
    97           .41
    SCHUBERT
    97           1
    106          .55
    98.5         .72
    ______________________________________

Values for duple meter are derived simply from the quadruple values by adding the duration of tones 1 and 2, and of tones 3 and 4, respectively, to obtain the duration proportions, and keeping the amplitude values for tones 1 and 3, which now become 1 and 2.

Thus the matrix values for duple meter are:
 

    ______________________________________
    Duration     Amplitude Ratio
    ______________________________________
    BEETHOVEN
    97.5         1
    103.0        .83
    MOZART
    100          1
    100          .51
    SCHUBERT
    106          1
    94.5         .40
    ______________________________________

Pulse matrices for compound meters can be derived from the above as follows:

The two matrices are combined so that ##EQU3## where A.sub.c (i,j) is the compound pulse amplitude D.sub.c (i,j) is the compound pulse duration

A.sub.1 (i), A.sub.2 (j) are the simple pulse amplitudes and

D.sub.1 (i), D.sub.2 (j) are the simple pulse durations respectively

for the i.sup.th and j.sup.th tone of the simple pulses.

To allow for different degrees of hierarchical dependence attenuation factors are introduced that allow the effectiveness of the subordinate pulse structure to be de-emphasised or emphasised, with the parameters n and m, so that when n=1 and m=1, a full hierarchical effect is obtained, and for smaller values the duration and/or amplitude effects of the subordinate pulse structure are relatively more attenuated. Thus ##EQU4## values ot n and m in the range of 0.7 to 0.8 are found to be often appropriate.

For example, the 6/8 pulse for Beethoven is
 

    ______________________________________
    for n = 1, m = 1
            102  1
             86  .46
            104  .75
            109  .83
             91  .38
            111  .62
    for n = .8, m = .7
            101  1
             88  .58
            103  .82
            108  .83
             94  .48
            109  .68
    ______________________________________

The effect is that each group of 3 tones constitutes a small 3-pulse, and the two groups of 3 tones form a 2-pulse.

Applying the Pulse to a Melody Having Various Note Values

When a melody has a combination of notes of different values, as is generally the case (some larger and some smaller than the component values), the following appears to apply:

1. Duration deviations are proportioned according to the component tones of the pulses; e.g., a dotted quarter has the duration deviation of one tone plus half that of the next.

2. The amplitude is taken as that prevalent at the beginning of the tone; i.e., is not averaged; e.g., the dotted quarter has the same amplitude as the quarter would have had without the dot.

While it appears that the range of 50-80 per minute is an approximately useful guideline in applying the pulse, some pieces may present alternate possibilities of a larger or smaller frame for the pulse.

An example of automatic operation of the system using a pulse matrix and using automatic calculations of amplitude shape (p values) is given in FIG. 9.

In general, it should be emphasized that the pulse and its effects in microstructure as described herein is in no way to be considered a binding Procrustes bed, but rather as a level from where fine artistic realization of the music can be more readily attempted, taking into account the individual concept of the piece, and personal interpretive preferences.

Additional Expressive Functions And Properties Considered

Further properties of melodic relationships relevant to manual and computer realizations are:

1. Microstructural

In working with the various examples manually, we may readily note that:

(a) A tone of smaller amplitude following a larger amplitude tone will sound more legato than in the reverse order, for given amplitude shapes (p values).

(b) A tone will sound softer after a tone of greater amplitude than before it (a masking effect, the degree depends on the tempo).

(c) Micropauses often need to be inserted between phrases, and longer Luftpauses between major sections. They are appropriate for all composers. The placement of these is important, and not automatic. These pauses tend to be placed at the end of pulses (before the next upbeat tone, where there is one). They also occur immediately before a subito piano.

At times the required pauses are actually provided for by the p values, as calculated by the rule given, by means of the tail of the amplitude shape--this means that in such a place, the composer's indication of phrasing indicates the amplitude shape required by the music. Also, they may be appropriate between separate bowing marks (as distinguished from phrasing signs--an often difficult distinction), but not always.

2. Structural

(a) Pitch Crescendo. It can be noted that there is a general tendency within a melody to increased loudness with higher pitch. This tendency is not found to the same degree, however, for different music and different composers. For Beethoven, it is appropriate to keep the amplitudes in terms of voltage similar. This insures a small degree of crescendo, since a given voltage amplitude is perceived as louder with increasing pitch. For Mozart, however, this crescendo tends to be greater--approximately an additional 4 db/octave. For Schubert, it can be even more than 6 db/octave.

If the same melodic crescendo is applied to Beethoven, the effect tends to sound exaggerated. There is a resistance to such crescendo in Beethoven, a restraint that (as in the modern modulation of tone amplitude heights by the Beethoven pulse) appears to translate into the strength of an ethical constraint.

(b) Alternation of "heavy" and "light" bars. This is a general structural property (not explicit in the score) which can be produced on the computer by scaling down the "light" bars. For Mozart, a suitable amplitude proportion may be 0.72, and bars often (but of course not always) alternate as Heavy-Light, Heavy-Light. This adds considerably to the sense of musical balance and logic, where appropriate; it presents larger units to the mind, allowing a greater overview.

In Beethoven, other patterns tend to dominate and often special dramatic devices, such as crescendo and subito piano. Crescendi and Diminuendi specially noted by the composer in the score have, of course, to be included in the computer realizations.

A Preferred Procedure for Shaping A Melody

We may concisely summarize the steps involved in controlling microstructure for the artistic shaping of a melody. Steps 7 to 9 are not always required. Some of these steps can be combined into single, automatic operations:

1. Enter tones and nominal durations according to score.

2. Choose tempo.

3. Apply pulse matrix for the particular composer, and particular meter (choose length of pulse) (B, M, or S for Beethoven, Mozart and Schubert, respectively).

4. Insert micropauses of appropriate duration at appropriate places (end of phrases and other special points).

5. Choose base p.sub.1, p.sub.2 values and let the program calculate p. values for amplitude shapes of individual tones.

6. Adjust last tones of phrases by "ear" (both amplitude and p values).

7. Apply pitch cres-dim function as appropriate, and any cres or dim or special accents directed by the score.

8. As required, modify amplitude and/or duration of one or several key tones to shape a particular essentic form.

9. If phrases repeat, subtly change the repetition as desired.

The first tone only of the theme is generally to be lengthened in duration by about 5%, at the beginning of a piece (but not on subsequent reintroductions of the theme).

A few minutes are generally sufficient to carry out these steps. It is, of course, useful to listen to the result after each step is taken. Micropauses p.sub.1, p.sub.2 base values or other parameters can be adjusted at any time to improve the result, as desired.

The Stability of Pulse Matrices

The pulse matrix values given previously are subject to further refinement. The salient features of the specific pulse matrices hold over a wide range; but their degree of prominence is likely to be influenced to some degree by the tempo (with the range given; 50-80 per minute), and by pitch height of the entire theme (i.e., transposition). These factors may modify the elements within the pulse matrices as a second order effect. Other second order changes in the pulse forms may occur with variables such as variables related to specific pieces and the composer's age.

Relation of Timbre to Microstructure

Timbre is essential for melodic expressiveness when there is more than one melodic line. Several sinusoids tend to coalesce and fuse--distinctness of voice leading and contrast sounds with dynamic proportions of harmonics are required. Secondly, we can now, in the light of the present studies with sinusoids, investigate how timbre variations are of help in improving expressiveness of a single melogic line.

Within each tone it is possible to add timbre (and also vibrato) various dynamic ways to the expressive forms already determined, and to see how such individually shaped time varying functions of timbre for each tone (time-shaped harmonic content--not only in relation to the attack) may augment or interfere with expressiveness. One must take into account the complication that addint timbre (and/or vibrato) can modify the perception of amplitude shapes, in part because the overtones require their own amplitude shaping (not according to the amplitude shapes of the fundamental), as well as for psychophysiologic reasons related to persistence of hearing and dynamic masking.

Summary of Findings

We have found that:

1. Expressiveness and Microstructure can be fruitfully studied with amplitude-modulated sinusoidal sounds. Musical meaning in its subtlety can be largely expressed by this means for single melogic lines.

2. A systematic relation appears to exist between individual amplitude shapes of tones of a melody and its course, so that the shape can be predicted from the slope; and thus is heard to presage the next tone of a melody--forging an organic link, and making it possible for the living qualities inherent in the melodic shape to cast a presence within each tone.

3. In music which incorporates a personal pulse, this is shown to systematically affect both amplitude and duration of component tones in a way characteristic to that personal pulse, as also an individual's signature bears the continuing stamp of his person. The realization of the pulse and its effects is seen to be necessary for the life, power and beauty of such music.

Among the advantages of imparting microstructure to a musical score in a computerized system in accordance with the invention are the following:

1. It can improve one's artistic understanding and output.

2. Far from being "mechanical," it can infuse life and livingness into music.

3. It tends to give us a degree of understanding of the very nature of that livingness, as bound to iconic and unique form.

4. It allows us to use imagination and creative insight from a higher hierarchic point of view, using as units of thought entities that before needed to be specially constructed every time from constituent parts.

The practical applications of a computerized system in accordance with the invention to music education are obvious and therefore need not be detailed. In an age of personal computers, the programs developed therefor can give access to creative interpretation to all so inclined without the need to acquire physical musical skills or manual dexterity.

The invention is also useful to a serious or amateur composer, for it allows the composer to incorporate his own realization of microstructure into the macrostructure of his own composition, and it also allows him to experiment with inner pulse forms. The final product thereby reflects the imagination, feeling and discernment of the individual who shapes a musical composition. The computerized system serves as a tool which assists its user in thinking musically in a manner somewhat analogous to the relationship of an electronic calculator to a mathematical concept.

The Computerized System

Referring now to FIG. 10, there is shown a manually-operated computerized system in accordance with the invention based on the technique disclosed hereinabove for processing the nominal tones of a raw score entered therein to impart an expressive microstructure thereto.

The system includes a beta function calculator 10. In the calculator is entered by way of a computer keyboard or means performing a like function represented by entry station 11, the successive tones of a music score in terms of their nominal pitch and duration values expressed in alpha-numeric terms. Thus the pitch of a given note, which depends on its position on the staff, is represented by an appropriate value, as is the duration of the same note. In practice, an electronic piano keyboard may be used. By depressing a selected key, there is produced an appropriate value for entry into the calculator. In that case, the tone duration will have to be normalized.

Also entered manually into calculator 10 is the desired microscore of each nominal note. By "microscore" is meant digital values representing desired deviations from the nominal values of the note necessary to its processing to impart a microstructure thereto.

Thus the p.sub.1 and p.sub.2 values required by the beta function to shape the amplitude contour of each note is entered as well as digital values representing changes in the duration of each note and values representing the relative amplitude of successive notes in the score. Also entered are micropauses and whatever other variables are to be processed by the system, such as timbre.

From entry station 11, there are two channels C.sub.1 and C.sub.2 leading into calculator 10, channel C.sub.1 conveying the amplitude and timing data, and channel C.sub.2 the pitch and timbre data for each note.

Calculator 10 is programmed to process the data supplied thereto and to yield a series of equi-spaced digital values V during a specified interval, as shown in FIG. 11, that represent the successive amplitude levels in the contoured tone T whose microstructure duration is interval P. In the absence of the microstructure impressed on the nominal tone, its form would be represented by a square wave of constant amplitude and predetermined duration, which depends on whether it is a whole note, a half note or whatever else is notated.

The series of digital values V, which outline the amplitude shape, are applied to a D-to-A converter 12 to yield an analog voltage A.sub.1 reflecting the amplitude contour or envelope of the processed note. The digital data derived from entry station 11, which represents the frequency of each note, is also fed to D/A converter 12 to yield an analog voltage A.sub.2, reflecting the pitch of the tone to be played.

Analog voltage A.sub.1 representing the amplitude contour is applied to a voltage-controlled amplifier 13 (VCA) whose output is applied to a loudspeaker 14. Analog voltage A.sub.2 representing the pitch is applied to a voltage-controlled oscillator 15 whose output frequency is in accordance with the pitch of the tone. The sinusoidal output of this oscillator may be applied directly to amplifier 13, in which event the reproduced tone is without a harmonic content but has the desired microstructure impressed thereon. Alternatively, the oscillator output may be applied to the amplifier through a timbre network 16 which changes the sinisoidal wave shape so that the resultant tone is rich in harmonics and therefore has a timbre depending on its harmonic content.

Any known means may be used for introducing a varying harmonic content. One approach is to combine a sinusoidal wave SW, as shown in FIG. 12, with the differentiated form DW of a square wave having the same period, the resultant sharp pulses being adjustably clipped and rectified to provide sharp peaks which, when summed with the sinusoidal wave, produce a nonsinusoidal wave having a desired harmonic content that depends on the adjustment of clipping and rectification. The resultant sum may be further variably rectified to provide preponderantly even or odd harmonics.

With each tone, the timbre is varied through a number of D to A control channels, typically up to 4 channels, each output of which is shaped by beta functions or equivalent means. All of these functions can also be carried out in an entirely digital manner in a digital synthesizer.

It is to be understood that among the means usable for producing a varying harmonic content are additive or subtractive synthesizers, wave shaping and other means, realized either digitally or through analog means.

It is to be understood that while calculator 10 is advantageously operated in accordance the beta function disclosed herein requiring only two parameters (p.sub.1 and p.sub.2), in order to produce a desired amplitude contour, any known electronic means to effect amplitude shaping in response to applied digital parameters may be used for the same purpose.

In operating a system in accordance with the invention in an automatic mode, all of the digital values with respect to amplitude and duration necessary to impart a microstructure to the nominal note values of the raw musical score entered therein may be stored, as shown in FIG. 13, in a pulse matrix 17 which in one output channel A.sub.3 yields the amplitude and timing data required to process each note, and in another output channel a.sub.4 yields the necessary pitch data for each note. The digital data from channel A.sub.3 is allied to an amplitude-shaped calculator 18, while digital data from channel A.sub.4 is applied to a tim

------