Contents

Diphone Studio

Copyright © 1997-1999 Fluency/Van Dale Lexicografie BV, Utrecht/NL

All rights reserved

Introduction

License agreement

Reference

Menu options

Using the toolbar

Using the mouse

Using the Edit window

Keyboard shortcuts

User guide

Preparation of a diphone database

Recording the corpus

Segmentation of diphones

Testing diphones

Exporting a diphone database

Email support:  support@fluency.nl


Introduction

Diphone Studio is a tool for developing and maintaining a set of diphones for use in speech synthesis. A diphone is the transition between two speech sounds, obtained from natural speech. A diphone synthesizer creates new utterances by concatenation of diphones, modifying pitch and duration of each sound according to some specification.

A diphone set is created in a number of steps:

1.	Specification of a phoneme inventory (the sounds of a language)

2.	Creation of a corpus (the words/phrases from which the diphones will be extracted)

3.	Recording the corpus

4.	Segmentation of the corpus

5.	Preliminary evaluation

6.	Post-processing

Diphone Studio is designed to facilitate steps 3 through 5. The program was specifically designed for the creation of diphone databases for the MBROLA synthesizer, developed by Thierry Dutoit and colleagues at the Faculté Polytechnique de Mons. For more information about the MBROLA synthesizer see

http://tcts.fpms.ac.be/synthesis/mbrola.html

on the World Wide Web.


License agreement

License agreement 	Part A

Diphone Studio is freely available for evaluation and non-commercial use. For commercial use, the user is required to obtain Part B of this license agreement. For more information, contact:

Surface mail:

Fluency/Van Dale Lexicografie BV

P.O. Box 19232

NL-3501 DE  Utrecht

Netherlands

Email:

sales@fluency.nl


Menu options

File

New	create new diphone database
Open	open diphone database
Save	save diphone database
Save as	save diphone database under a different name
Export	prepare database for further processing
Exit	quit the program

Remark:

The information contained in a diphone database is stored as an ASCII text file (default file extension: .dat). See: Preparation of a diphone database

Edit

First	go to first diphone
Last	go to last diphone
Previous	go to previous diphone
Next	go to next diphone
Next to edit	go to first diphone which is not fully specified
Goto diphone	find diphone by name
Zoom full out (switch)	checked: show entire waveform  unchecked: show part of waveform (use scrollbars for navigation)
Edit utterance	edit target utterance for current diphone
Clear markers	remove segmentation markers for current diphone

Sound

Play all	play entire waveform
Play with break at phoneme boundary	play entire waveform with a 500 ms pause at Phoneme boundary
Play diphone	play waveform segment between Left and Right boundary
Record waveform	record new waveform for current diphone
Undo recording	remove recording for current diphone

Remark: Waveform segments can also be played by clicking the right mouse button. See: Using the mouse.

Test

Set input text	set input for diphone concatenation, with spaces between phonemes
Test input	play concatenation of current input

Options

Find zero crossings (switch)	checked: place boundary markers at first upward zero crossing in waveform to the right of mouse pointer (default) unchecked: place boundary markers at mouse pointer

Help

Contents	Show this help file
About	Display About box


Using the toolbar

You can quickly select the most common Diphone Studio commands using the buttons on the toolbar. For example, to open a diphone database, click the Open button on the toolbar.

	Open diphone database
	Save diphone database
	Play waveform
	Record waveform
	Help
	Play test

Remark: When the mouse is pointed at a toolbar button, the function of the button appears in the status bar at the bottom-left of the screen.


Using the mouse

In the Edit window the left mouse button is used to insert segmentation markers for a diphone. The right mouse button is used to play the various parts of a waveform.

To set boundary markers:

1.	Point the mouse at the position where you want to place a boundary marker

2.	Press the left mouse button. Select from the following menu:

Left boundary

Right boundary

Phoneme boundary

Cancel

To reposition boundary markers for the diphone, choose a new location for the boundary, following the instructions above. The old marker will be removed.

To play a waveform segment:

1.	Point the mouse within the segment you want to hear.

2.	Click the right mouse button.

This will playback the part of the waveform between the nearest markers, if any.


Using the Edit window

Below, a picture is shown of Diphone Studio displaying the diphone e-n for Dutch.

The Edit window shows part of a waveform, in this case the Dutch word prenataal, recorded from a female voice.

The red, vertical lines mark boundaries in the e-n diphone, as follows:

1.	the first marker represents the start of the diphone

2.	the second marker represents the boundary between the two phonemes in the diphone

3.	the third marker represents the end of the diphone

These markers are set by pressing the left mouse button at the appropriate location in the waveform. (See: Using the mouse)

The horizontal scrollbar is used to scroll through the waveform.

The vertical scroll bar is used to obtain a more or less detailed view of the waveform by zooming in or out.

At the bottom of the screen, the Status line displays the following information about the diphone diplayed in the Edit window:

1.	filename of the waveform

2.	name of the diphone

3.	transcription of the utterance used for recording the waveform

4.	durations of the parts of the diphone as indicated by the boundary markers

5.	a check mark (X) if Options | Find zero crossings is selected


Keyboard shortcuts

Left Arrow	scroll left through waveform
Right Arrow	scroll right through waveform
Shift+Up Arrow	zoom out small step
Shift+Down Arrow	zoom in small step
Shift+Page Up	zoom out large step
Shift+Page Down	zoom in large step

Remark: the above keys do not work if Edit | Zoom full out is checked.

Ctrl+PgUp	go to previous diphone
Ctrl+PgDn	go to next diphone
Ctrl+N	go to first diphone which is not fully specified
Ctrl+G	find diphone by name
Ctrl+Z	zoom full out (switch)
Ctrl+E	edit target utterance for current diphone
Ctrl+C	clear markers
Ctrl+F	find zero crossings (switch)
Ctrl+P	play entire waveform
Ctrl+B	play entire waveform with a 500 ms pause at Phoneme boundary
Ctrl+D	play waveform segment between Left and Right boundary
Ctrl+R	record new waveform for current diphone
Ctrl+S	set input for diphone concatenation
Ctrl+T	play concatenation of current input
Ctrl+U	remove recording for current diphone
F1	show this help file
F2	save diphone database
F7	create new diphone database
F8	open diphone database
Alt+X	quit program


Preparation of a diphone database

Before Diphone Studio can be used to create a new diphone set, the following preparations need to be made:

1. Specification of phoneme inventory

The phoneme inventory lists all phonemes of the target language, as well as any allophones that are needed for accurate pronunciation of all words (and word transitions) in the language.

For example, suppose the target language admits only a single vowel, /a/, and just two consonants, /p/ and /l/, our phoneme inventory is: /a/, /p/, /l/. However, suppose also that the pronuciation of /l/ depends on whether it occurs before or after a vowel (as is the case in English or Dutch). In that case, we need two allophones for /l/, which we could distinguish by using [l] to indicate /l/ before a vowel (as in [papla]), [L] to indicate /l/ following a vowel (as in [paLpa]). Now, our complete inventory is: [a], [p], [l], [L] and silence (usually represented by the underscore symbol), a grand total of 5.

2. Derivation of diphones

For automatic derivation of a diphone set from a list of phonemes/allophones, see below.

In principle, a grand total of 5 phonemes/allophones requires 25 diphones. That is, a list of diphones is obtained by writing out the square product of the set of phonemes. However, there may be combinations of phonemes which do not occur in the language, so the actual list is usually somewhat smaller.

In the above example, [L] cannot occur without a preceding vowel, and as a result cannot be used to start an utterance. Therefore, there is no need to include a diphone [_-L] for the transition from silence to [L].

3. Creation of corpus

For each diphone, we need a word or short phrase which contains the required phoneme sequence. For example, the word palpa may be used to represent the [L-p] diphone in the corpus.

4. Creation of Diphone Studio data file

Before Diphone Studio can be used for (recording and) segmentation of the diphone corpus, a data file must be created, listing the diphones, and the names of the corresponding waveform files from which the diphones are extracted. This data file is an 8-bit ASCII text file. It can be created e.g. with Windows Notepad, or generated automatically by Diphone Studio. The default file extension is .dat.

For automatic generation of Diphone Studio data file:

1.	List your phoneme inventory in a text file: designate each phoneme/allophone in the format recognized by the target synthesis software, separate phonemes/allophones by newlines

2.	Save this file as a .lst file

3.	Select File | New or F7

4.	Type the name of your .lst file in the file name box (or choose from list)

5.	Click Open to create a .dat file, which will be placed in the same directory as the .lst file

The first line of the data file defines the sampling frequency used for the waveform files, preceded by an exaclamation mark. For example, the following specification sets the sampling frequency to 22050 samples per second:

!22050

If this line is omitted, a default sampling frequency of 16000 is assumed.

All following lines define diphones. The syntax for a diphone definition is:

Left  Right  Wavefile  Utterance  [Start  Boundary  End]

The fields in each definition are separated by TAB characters; the definition ends with CR/LF.

Left	Name of the lefthand phoneme in a format recognized by the target synthesis software
Right	Name of the righthand phoneme in a format recognized by the target synthesis software
Wavefile	Path and filename of the .wav file containing the recording from which the diphone is extracted. If Diphone Studio is used to record the waveform, it will be saved under the filename specified here
Utterance	Transcription of the utterance used to record the diphone
Start	Position in Wavefile marking the beginning of the diphone, in samples. This field is filled in by Diphone Studio
Boundary	Position in Wavefile marking the phoneme boundary within the diphone, in samples. This field is filled in by Diphone Studio
End	Position in Wavefile marking the end of the diphone, in samples. This field is filled in by Diphone Studio

Example:

…

L	p	d0008.wav	palpa

…

For an example of a fully specified data file, load the file dstudio\example\example.dat into the Notepad editor.


Recording the corpus

Diphone Studio uses 16-bit waveform files recorded in the Windows Wave format. There are two ways to obtain these waveforms:

1.	using existing recordings for further processing with Diphone Studio

2.	using the recording facility of Diphone Studio

In each case, make sure that these recordings are of a sufficiently high quality:

1.	use the highest gain possible, but avoid clipped sounds

2.	make sure ambient noise is as little as possible; preferably, make the recordings in a sound-proof room

3.	make sure each diphone is surrounded by a sufficient amount of context (100 ms at the very least)

4.	the speaker should aim to produce all target utterances at a constant pitch, with as little intonation as possible; also, the speaker should attempt to maintain a constant rhythm (not too slow, not too fast) and loudness

5.	if the recordings are made in more than one session, make sure the same settings are used throughout (distance from microphone, gain)

To record a diphone corpus from within Diphone Studio, follow the following steps:

Create data file (See: Preparation of a diphone database)

Open data file

1.	select File|Open

2.	select the .dat file that defines the diphone database

Record utterance

1.	goto the diphone for which you want to record an utterance

2.	select Sound|Record

3.	a dialogue box appears, specifying the current diphone and target utterance

4.	press Start

5.	record target utterance

6.	press Stop to stop recording

The new recording is written to a waveform file using the extension .rec. When the database is saved, .rec files replace any previously existing .wav files of the same name, making the new recordings permanent.

Undo recording

As long as the diphone database is not saved, any recordings made during a session can be "undone". Select Sound|Undo recording. This removes the .rec file for the current diphone. If a .wav file for this diphone previously existed, it is restored.

Save database

Select File|Save. The data file is saved to disk. Also, any new recordings made during this session, are now made permanent.


Segmentation of diphones

To segment diphones, proceed as follows:

1. Open a database

Select File|Open to load a diphone database. An example of a small diphone database can be found in dstudio\example\example.dat. This database defines the 20 diphones needed to synthesize the numerals 1, 2, 3 in Dutch (pronounced [en], [twe], [dri]), in any order.

For each diphone, follow steps 2 through 3:

2. Select a diphone

Select Edit|Goto Diphone, and enter the diphone description into the dialog box. Alternatively, scroll through the database using the Next, Previous, Last, First, or Next to Edit options under the Edit menu. The corresponding waveform is shown in the Edit window. The name of the diphone appears bottom-left in the Status line.

3. Define diphone boundaries

In order to define a diphone, indicate start and end of the diphone in the waveform, as well as the transition point between the two phonemes. This is done by setting boundary markers, as follows:

First, use the horizontal scroll bar to scroll through the waveform until the section containing the diphone is displayed in the Edit window. Use the vertical scroll bar or the Zoom in/out option in the Edit menu to obtain the desired level of detail. (See: Using the Edit window)

Next, set boundary markers:

1.	Point the mouse at the position where you want to place a boundary marker

2.	Press the left mouse button. Select from the following menu:

Left boundary

Right boundary

Phoneme boundary

Cancel

Positions for the boundary markers are determined by visual inspection, as well as by auditory evaluation.

Playback

1.	select Sound|Play all to play te complete waveform

2.	select Sound|Play with break at phoneme boundary to play the complete waveform with a 500 ms pause at the Phoneme boundary

3.	select Sound|Play diphone to play the diphone, that is, to play the waveform segment between the Left and Right boundaries.

4.	use the right mouse button to play back any part of the waveform

Zero crossings

It is preferable to place boundary markers at at the beginning of a pitch period. To facilitate this, Diphone Studio automatically searches for upward zero crossings in the signal. To disable this facility, uncheck Options|Find zero crossings.

Zooming

To obtain the necessary amount of detail for visual inspection:

1.	use Edit|Zoom full out to zoom in or out

2.	use the vertical scroll bar to select the amount of detail shown when zoomed in


Testing diphones

Listening to a rough sequencing of diphones is a good way to test the quality of the segmentation.

To test diphones:

1.	select Test|Set test input text

2.	type a phoneme string, with spaces between phonemes, then select OK. The test string appears in the text box top-right

3.	press the [image] button next to the text box (or press Ctrl+T) to play the sequence

The results of changes to the segmentation can easily be evaluated this way.

Example

An example of a small diphone database can be found in dstudio\example\example.dat. This database defines the 20 diphones needed to synthesize the numerals 1, 2, 3 in Dutch (pronounced [en], [twe], [dri]), in any order.

1.	Open the file example.dat

2.	select Test|Set test input text and enter the following phoneme sequence:

_ d r i t w e _

3. Press Ctrl+T to play the test string. You should hear the Dutch equivalent of "three two"

If the segmentation is correct, the utterance should sound natural enough (except for lack of intonation), without clicks or other unwanted noises.


Exporting a diphone database

Diphone Studio was specifically designed for the creation of diphone databases for the MBROLA synthesizer, developed by Thierry Dutoit and colleagues at the Faculté Polytechnique de Mons.

Before a diphone database can be used with MBROLA, it must be further processed by the MBROLA team. To prepare the database for such further processing, select File|Export. This creates a new directory called export below the directory where the diphone database is stored. In the export directory, the following files are created:

export.dat 	data file

dxxxx.raw	waveform files

The file export.dat is an ASCII text file, with the following data for each diphone:

Wavefile	Left	Right 	Start 	End	Boundary

The fields in each definition are separated by TAB characters; the definition ends with CR/LF.

Wavefile	Filename of a headerless 16-bit waveform file containing the sampled data which make up a diphone. Each diphone is cut from the original recording with some 100 ms extra on both sides, to provide context for the MBROLA analysis software
Left	Name of the lefthand phoneme
Right	Name of the righthand phoneme
Start	Position in Wavefile marking the beginning of the diphone, in samples
End	Position in Wavefile marking the end of the diphone, in samples
Boundary	Position in Wavefile marking the phoneme boundary within the diphone

Example:

d0.raw	_	e	1360	3205	2621

...

The files in the export directory can be stored in a .zip file and sent to Mons for further processing. For more information about this, see

http://tcts.fpms.ac.be/synthesis/mbrola.html

on the World Wide Web.