Progress Report
Reihaneh Rabbany
Presented for NLP Group
Computing Science Department
University of Alberta
April 2009
Agenda
•Project Proposal for Guiding Agent by Speech
•Many to Many Alignment by BayesianNetworks
–Letter to Phoneme Alignment
–Evaluation of phylogenetic trees
2
Quick RL overview
•An agent interacting with environment
–perceives state
–performs actions
–receive rewards
•Agent
–Computes the value of each action in each state
•long term reward obtainable from this state by performing thisaction
–Performs action selection by choosing the best action orsometimes a random action
•exploration-exploitation
3
Project Proposal for Guiding Agent bySpeech
•Accelerate learning using speech
–The emotion in speech signal has considerableamount of side information
–Happiness or anger of a speech signal can providea shaping reinforcement signal
•Developing tools and methods to extractemotion from speech and designing amethodology to use it as a shaping signal
4
Approaches to use speech signal as aguide for learning
•Extracting prosodic features from speech
•Associating meaning to these features
–Supervised learning-based approach
•data-set of (prosodic features, emotion) pairs
–excited, happy, upset, sad, bored
•Assigns a reward to the recognized emotion
–Pure RL approach
•inspired by the learning process of the parent-infant
–The infant gradually learns to associate value to perceivedspeech and how to use it to guide her exploration of the world
5
RL Approach
•Two ways for developing this idea
–Augmenting the observation space to include theprosodic features
•Emotion will become state-dependent
–Learns a separate instructor module
•Estimates the value of prosodic features
•Instructions (learnt instructor values) would affect theagent's action selection
6
Instructions
•Different ways that these instructions (learnt instructorvalues) could affect the agent's action selection
–Balancing the exploration-exploitation
•When the speaker is not happy with what the agent is doing and itshould explore other actions
–Use it directly in action selection by some weights
•Motivates the agent to keep its previous action if the instructor issatisfied with its current action
–Use it as a shaping reward to define a new reward functionby adding it to the actual reward received from theenvironment
7
Agenda
•Project Proposal for Guiding Agent by Speech
•Many to Many Alignment by BayesianNetworks
–Letter to Phoneme Alignment
–Evaluation of phylogenetic trees
8
Many to Many Alignment by BayesianNetworks
•Finding Alignment between two sequences
–Assuming the order is preserved
•I’ve applied it into two applications
–Letter to phoneme alignment
•Aligning for a given dictionary
–Evaluating Phylogenetic trees
•Shows how compatible the tree is with the giventaxonomy
9
Agenda
•Project Proposal for Guiding Agent by Speech
•Many to Many Alignment by BayesianNetworks
–Letter to Phoneme Alignment
–Phylogenetic trees evaluation
10
Model
•Word:
–sequence of letters
•Pronunciation:
–sequence of phonemes
• Alignment:
–sequence of subalignments
•Problem: Finding the most probable alignment
•Assumption: sub alignments are independent
11
Many-to-Many EM
1. Initialize prob(SubAlignmnets)
// Expectation Step
2. For each word in training_set
2.1. Produce all possible alignments
2.2. Choose the most probable alignment
// Maximization Step
3. For all subalignments
3.1. Compute new_p(SubAlignmnets)
12
Dynamic Bayesian Network
•Model
•Subaligments : hidden variables
•Learn DBN by EM
li
Pi
ai
13
Context Dependent DBN
•Context independency assumption
–Makes the model simpler
–It is not always a correct assumption
–Example: P(<h,h>) in Chat and Hat
•Model
li
Pi
ai
ai-1
14
Agenda
•Project Proposal for Guiding Agent by Speech
•Many to Many Alignment by BayesianNetworks
–Letter to Phoneme Alignment
–Evaluation of phylogenetic trees
15
Evaluation of Phylogenetic Trees
•Phylogenetic Trees
–Show the evolution of species
•Taxonomy
–Caninae; True dogs; Canis; Coyote
–…
–Caninae; True foxes; Vulpes; Kit Fox
–Caninae; True foxes; Vulpes; Fennec Fox
–…
–Caninae; Basal Caninae; Otocyon ; Bat-eared Fox
–...
16
Tree Evaluation
•Labeling the inner nodes in the tree
•For each species
–A path in the tree
• sequence of inner node labels
–A taxonomy description
• taxonomy sequence
–There should be a many to many alignmentbetween these two sequences
17
Tree Evaluation (Cont.)
•Finding alignment between these sequencesfor all the species
–Finding the most probable alignments
•Measuring the mean probability of thesealignment
–How probable is this tree given this taxonomy
18
•Taxonomy and Trees
•Aligned result
19
Discussion
20