Worksheet: Event Extraction

Background

You are annotating data for an AI system that extracts events from news text.

An event consists of:

A trigger (the word or phrase that signals the event)
One or more arguments (participants, locations, times, etc.)

Event Type: ATTACK
An action involving physical or violent harm against a person, group, or location.

Annotation Labels

Trigger labels:

B-ATTACK beginning of attack trigger

I-ATTACK continuation of trigger

O not part of a trigger

Argument labels:

B-AGENT / I-AGENT who carried out the attack

B-TARGET / I-TARGET who/what was attacked

B-PLACE / I-PLACE where it happened

Part 1: Warm-up

Q1

Which of the following could be valid attack triggers? (Check all that apply.)

attacked killed shot explosion clash fire

Briefly explain one difficult case:

Part 2: Sequence Labeling

Sentence 1

"The militants launched an attack on the military base in the city."

The

militants

launched

an

attack

on

the

military

base

in

the

city

Q1

Which word(s) did you choose as the event trigger? Why?

Part 3: Boundary Ambiguity

Sentence 2

"The militants carried out a deadly attack in the capital."

The

militants

carried

out

a

deadly

attack

in

the

capital

Q1

Is the trigger:

"attack" "deadly attack" "carried out a deadly attack"

Why?

Part 4: Trigger Ambiguity

Sentence 3

"Fighting erupted near the border late Monday."

Fighting

erupted

near

the

border

late

Monday

Q1

Does this sentence contain an ATTACK event?

Yes No Unclear

Explain your reasoning:

Part 5: Argument Ambiguity

Sentence 4

"The army shelled rebel positions outside the town."

The

army

shelled

rebel

positions

outside

the

town

Q1

Which argument roles were hardest to decide?

AGENT TARGET PLACE

Why?

Part 6: Reflecting on Disagreement

Looking back at your annotations in Parts 2–5, consider where other annotators might make different choices.

Q1

Where do you think annotators would disagree most?

Trigger boundaries Trigger existence Argument boundaries Argument roles

Give one concrete example of a likely disagreement:

Q2

Which guideline change would most improve agreement?

More trigger examples Stricter boundary rules Allow multiple valid triggers Add confidence/uncertainty labels More annotator training

Explain your choice:

Part 7: Why Event Extraction Is Hard

Q1

Why is event extraction harder than sequence labeling tasks like NER? (Check all that apply.)

Triggers can be implicit Boundaries are unclear Arguments depend on syntax/semantics Multiple interpretations valid All of the above

Part 8: Aspectual Verbs and Event Boundaries

Aspectual words describe the beginning, continuation, or end of an event, rather than the event itself.

Common aspectual words: began, started, continued, stopped, resumed, ended

Whether these should be included as part of an event trigger is often unclear and guideline-dependent.

Sentence 5

"The army began shelling rebel positions near the town."

The

army

began

shelling

rebel

positions

near

the

town

Q1

Which word(s) did you label as the ATTACK trigger?

shelling began shelling began No attack trigger

Explain your choice:

Q2

What does "began" contribute to the meaning of the event?

Indicates the start of the attack Is part of the attack itself Only provides temporal/aspectual information Makes the event uncertain

Explain briefly:

Sentence 6

"The fighting continued throughout the night."

The

fighting

continued

throughout

the

night

Q3

Does this sentence describe:

A new ATTACK event The continuation of a previous ATTACK event No clear ATTACK event

Why?

Part 9: Aspectual Decisions

Q1

Should aspectual words like began, continued, ended be:

Included in the event trigger Excluded from trigger but annotated separately Ignored entirely Treated differently depending on context

Explain your reasoning:

Q2

What guideline rule would reduce disagreement the most?

"Only label the core action verb as the trigger" "Include aspectual verbs if they directly modify the action" "Annotate aspect separately from event type" "Allow multiple valid trigger spans"

Explain your choice:

Part 10: Aspect, Intent, and Event Reality

Aspect vs. Core Event

Sentence 7

"The army attacked rebel positions near the town."

Sentence 8

"The army was preparing to attack rebel positions near the town."

Q1

Which sentence(s) contain an ATTACK event?

Sentence 7 only Sentence 8 only Both Neither

Explain your reasoning:

Q2

What is the key difference between these sentences?

One describes an event, the other describes intent One is completed, the other is hypothetical One is observable, the other is inferred They are equivalent for event annotation

Continuation vs. New Events

Sentence 9

"The fighting stopped after international pressure."

Sentence 10

"The fighting resumed the next morning."

Q3

How many ATTACK events are described here?

One continuous event Two separate events No clear event Depends on guidelines

Explain:

Part 11: Guideline Design and Modeling Consequences

Q1

Which rule would you adopt for this project?

Only annotate concrete, realized events Annotate aspectual verbs as part of the trigger Separate event type from event status (started, ongoing, ended) Allow annotators to choose freely

Explain your choice and one tradeoff:

Q2

Suppose you include began, continued, stopped as part of ATTACK triggers. Which outcomes are likely? (Check all that apply.)

Longer trigger spans Lower inter-annotator agreement Models conflate events with temporal structure Better temporal reasoning Harder evaluation

Explain one effect in detail:

Q3

If instead you exclude aspectual words from triggers, what might the model fail to learn?