Note

columns

  1. ordinary: the ordinary grammar (using neither unknown word handling nor bridge rules)
  2. unk: unknown word handling (the YY mode (ZhongYYMode))
  3. br: robust parsing with bridge rules
  4. unk+br: unknown word handling and robust parsing with bridge rules
  5. generation: using the result of [1] as the input MRSes (# of generated items / # of parse items)
  6. end-to-end: # of generated items / # of input sentences

numbers

After using gTest, the numbers are described in the following format. Before that, only the total coverages are presented.

coverage-of-grammatical (readings-or-grammatical) / coverage-of-ungrammatical (readings-or-ungrammatical)

For example, 4.94 (1.5) / 7.89 (2.6667) means the coverage for grammatical items is 4.94%, the coverage for ungrammatical items is 7.89%, the average number of readinfs for grammatical items is 1.5, and the average number of readings for ungrammatical items is 2.6667.

Not all datasets have ungrammatical items. If there is no ungrammatical item in the dataset, the part after “/” is missing.

Note that with the numbers for grammatical items, the higher the better, while with those for ungrammatical ones the lower the better.

abbreviations

  • auto: lexical items automatically constructed from the Sinica treebank
  • punct: robust text handling, such as punctuations, symbols, interjection
  • 16K+pu: Zhenzhen’s 16K lexical items and improvement in handling punctuations and symbols
  • spr+ditr: decreasing spuriousity and ditranstive variations
  • nydva: generic lexical rules for numbers and year, ditransitive variations, vv-compounds, A-not-A

options

  1. nbest: 5
  2. timeout: 5 sec.
  3. max-chart-megabytes: 512mb (previously 256mb) for parsing, 2048mb for generation
  4. max-unpack-megabytes: 512mb (previously 256mb), 2048mb for generation

Development Data

mrs

               
Changes Date ordinary unk br unk+br generation end-to-end
relatives 2015-07-17 88.79 (3.41) 88.79 (3.45) 100.0 (5.0) 100.0 (5.0) 90.53 80.38
frag+512mb 2015-04-24 94.39 (3.52) 94.39 (3.56) 100.0 (5.0) 100.0 (5.0) 95.05 89.72
NP+PP 2015-04-20 94.39 (3.55) 94.39 (3.59) 99.07 (5.0) 99.07 (5.0) 95.05 89.72
prep+lzg+gle 2015-04-13 91.59 (2.88) 91.59 (2.91) 100.0 (5.0) 100.0 (5.0) 97.96 89.72
nydva 2015-04-10 91.59 (2.71) 91.59 (2.74) 100.0 (5.0) 100.0 (5.0) 97.96 89.72
spr+ditr 2015-04-02 95.33 (3.4) 95.33 (3.41) 100.0 (5.0) 100.0 (5.0) 99.02 94.4
16K+pu 2015-03-27 95.33 (4.05) 95.33 (4.05) 100.0 (5.0) 100.0 (5.0) 99.02 94.4
gTest 2015-03-20 93.46 (3.27) 93.46 (3.27) 100.0 (5.00) 100.0 (5.00) 99.0 92.53
AABB 2015-03-13 93.5 88.8 100 94.4 N/A N/A
interjections 2015-03-13 93.5 88.8 100 94.4 N/A N/A
auto 2015-03-12 93.5 88.8 100 94.4 N/A N/A
w/o auto 2015-03-12 38.3 38.3 97.2 93.5 N/A N/A
change YYYY-MM-DD ordinary unk br unk+br generation end-to-end

ntumc

               
Changes Date ordinary unk br unk+br generation end-to-end
relatives 2015-07-17 2.71 (3.83) / 0.0 (0.0) 12.42 (4.09) / 20.0 (5.0) 10.88 (4.94) / 6.67 (5.0) N/A 64.13 1.74
frag+512mb 2015-04-24 3.05 (3.91) 14.56 (4.23) 10.01 (4.93) 47.16 (4.89) 75.77 2.31
NP+PP 2015-04-20 2.4 (4.09) 11.08 (4.39) 8.95 (4.89) 37.07 (4.83) 63.1 1.51
prep+lzg+gle 2015-04-13 1.8 (3.71) 9.44 (3.93) 7.99 (4.84) 42.81 (4.82) 70.69 1.27
nydva 2015-04-10 1.65 (3.63) 8.46 (3.84) 8.01 (4.84) 47.24 (4.84) 78.05 1.29
spr+ditr 2015-04-02 1.52 (3.32) 6.77 (3.7) 7.59 (4.82) 37.96 (4.8) 60.0 0.91
16K+pu 2015-03-27 1.58 (3.42) 6.85 (3.77) 7.59 (4.83) 38.51 (4.8) 79.66 1.26
gTest 2015-03-20 0.32 (1.92) 4.04 (4.14) 1.13 (4.49) 9.7 (4.39) 100.0 0.32
AABB 2015-03-13 0.8 1.5 1.0 3.0 N/A N/A
interjections 2015-03-13 0.8 1.5 1.0 2.7 N/A N/A
auto 2015-03-12 0.8 1.5 1.0 2.8 N/A N/A
w/o auto 2015-03-12 0.0 1.4 0.0 1.9 N/A N/A
change YYYY-MM-DD ordinary unk br unk+br generation end-to-end

pctb-dev (the first 5,000 sentences)

               
Changes Date ordinary unk br unk+br generation end-to-end
relatives 2015-07-17 0.68 (4.68) 5.56 (4.65) 10.34 (4.97) 46.62 (4.98) 44.12 0.3
frag+512mb 2015-04-24 0.8 (4.75) 6.92 (4.74) 10.18 (4.97) 44.06 (4.97) 45.0 0.36
NP+PP 2015-04-20 0.74 (4.78) 6.02 (4.73) 9.66 (4.97) 35.08 (4.96) 43.24 0.32
prep+lzg+gle 2015-04-13 0.5 (4.44) 4.84 (4.53) 9.78 (4.97) 41.82 (4.96) 60.0 0.3
nydva 2015-04-10 0.36 (4.17) 4.12 (4.46) 10.56 (4.97) 51.7 (4.97) 61.11 0.22
spr+ditr 2015-04-02 0.38 (4.32) 9.92 (2.73) 3.28 (4.9) 35.74 (4.97) 63.16 0.24
16K+pu 2015-03-27 0.38 (4.21) 9.96 (2.75) 3.3 (4.9) 36.1 (4.97) 68.42 0.26
gTest 2015-03-20 0.04 (1.50) 3.08 (4.68) 0.06 (5.00) 7.3 (4.84) 100.0 0.04
AABB 2015-03-13 0.0 2.5 0.1 4.1 N/A N/A
interjections 2015-03-13 0.0 2.5 0.1 4.1 N/A N/A
auto 2015-03-12 0.0 2.5 0.1 4.1 N/A N/A
w/o auto 2015-03-12 0.0 2.3 0.0 3.0 N/A N/A
change YYYY-MM-DD ordinary unk br unk+br generation end-to-end

sinica-dev (the first 5,000 sentences (nltk_data))

               
Changes Date ordinary unk br unk+br generation end-to-end
relatives 2015-07-17 3.76 (3.78) 34.86 (4.34) 6.54 (4.65) 65.3 (4.94) 78.72 2.96
frag+512mb 2015-04-24 3.56 (3.79) 37.74 (4.52) 6.34 (4.57) 64.22 (4.92) 80.9 2.88
NP+PP 2015-04-20 3.3 (3.77) 35.38 (4.54) 6.32 (4.56) 62.62 (4.92) 77.58 2.56
prep+lzg+gle 2015-04-13 3.02 (3.32) 31.22 (4.15) 6.28 (4.53) 63.3 (4.91) 80.13 2.42
nydva 2015-04-10 2.96 (3.15) 28.96 (4.1) 6.36 (4.53) 63.74 (4.91) 81.08 2.4
spr+ditr 2015-04-02 2.9 (2.9) 23.64 (3.96) 6.24 (4.55) 56.6 (4.9) 81.38 2.36
16K+pu 2015-03-27 3.0 (3.01) 24.08 (4.05) 6.24 (4.56) 56.6 (4.9) 82.0 2.46
gTest 2015-03-20 1.36 (3.06) 23.06 (4.70) 2.8 (4.46) 30.86 (4.83) 95.59 1.3
AABB 2015-03-13 1.2 8.7 2.5 11.1 N/A N/A
interjections 2015-03-13 1.2 8.6 2.5 11.1 N/A N/A
auto 2015-03-12 1.2 8.6 2.4 11.1 N/A N/A
w/o auto 2015-03-12 0.1 8.1 0.7 10.0 N/A N/A
change YYYY-MM-DD ordinary unk br unk+br generation end-to-end

Held-out Data

jec

               
Changes Date ordinary unk br unk+br generation end-to-end
relatives 2015-07-17 10.43 (3.69) 32.77 (3.74) 27.19 (5.0) 79.85 (5.0) 72.88 7.6
frag+512mb 2015-04-24 11.92 (3.77) 37.71 (3.87) 26.6 (5.0) 78.96 (5.0) 93.08 11.1
NP+PP 2015-04-20 11.97 (3.81) 37.78 (3.89) 26.17 (5.0) 77.06 (5.0) 90.48 10.83
prep+lzg+gle 2015-04-13 10.07 (3.26) 32.33 (3.24) 24.49 (4.99) 77.51 (4.99) 95.88 9.66
nydva 2015-04-10 9.01 (3.13) 28.96 (3.13) 24.06 (4.99) 77.87 (4.99) 96.03 8.65
spr+ditr 2015-04-02 8.43 (2.79) 26.04 (2.84) 22.45 (4.99) 69.16 (4.99) 96.42 8.13
16K+pu 2015-03-27 8.54 (3.05) 26.21 (3.05) 22.45 (4.99) 69.19 (4.99) 95.54 8.16
gTest 2015-03-20 2.77 (2.29) 21.66 (3.88) 6.26 (5.00) 37.03 (5.00) 87.5 2.42
AABB 2015-03-13 2.7 20.5 4.5 31.1 N/A N/A
interjections 2015-03-13 2.5 20.2 4.3 30.4 N/A N/A
auto 2015-03-12 2.5 20.2 4.3 30.6 N/A N/A
w/o auto 2015-03-12 0.0 18.3 0.1 22.7 N/A N/A
change YYYY-MM-DD ordinary unk br unk+br generation end-to-end

fu-berlin

               
Changes Date ordinary unk br unk+br generation end-to-end
relatives 2015-07-17 22.22 (2.56) / 39.47 (2.67) 77.78 (2.84) / 68.42 (2.77) 22.22 (4.89) / 55.26 (5.0) 97.53 (4.97) / 100.0 (5.0) 81.82 18.18
frag+512mb 2015-04-24 22.22 (3.11) / 39.47 (3.07) 80.25 (3.12) / 68.42 (3.0) 22.22 (4.89) / 55.26 (5.0) 97.53 (4.97) / 100.0 (5.0) 90.91 20.2
NP+PP 2015-04-20 11.11 (3.0) / 18.42 (4.14) 67.9 (3.27) / 42.11 (3.44) 22.22 (4.89) / 55.26 (5.0) 97.53 (4.97) / 100.0 (5.0) 81.25 9.03
prep+lzg+gle 2015-04-13 8.64 (1.29) / 15.79 (2.0) 54.32 (2.0) / 34.21 (2.15) 16.05 (4.92) / 42.11 (4.75) 97.53 (4.99) / 100.0 (4.89) 100.0 8.64
NP+PP 2015-04-20 11.11 (3.0) / 18.42 (4.14) 67.9 (3.27) / 42.11 (3.44) 22.22 (4.89) / 55.26 (5.0) 97.53 (4.97) / 100.0 (5.0) 81.25 9.03
nydva 2015-04-10 8.64 (1.29) / 15.79 (1.83) 46.91 (1.74) / 31.58 (1.92) 16.05 (4.92) / 42.11 (4.75) 97.53 (4.99) / 100.0 (4.89) 100.0 8.64
spr+ditr 2015-04-02 8.64 (1.29) / 15.79 (1.83) 45.68 (1.68) / 26.32 (1.9) 16.05 (4.92) / 42.11 (4.31) 79.01 (4.98) / 78.95 (4.63) 100.0 8.64
16K+pu 2015-03-27 8.64 (2.14) / 15.79 (2.67) 43.21 (2.26) / 26.32 (2.5) 16.05 (4.92) / 42.11 (4.63) 79.01 (4.98) / 78.95 (4.8) 100.0 8.64
gTest 2015-03-20 4.94 (1.5000) / 7.89 (2.67) 37.04 (2.67) / 21.05 (2.38) 12.35 (4.10) / 36.84 (3.50) 74.07 (4.85) / 76.32 (4.28) 100.0 4.94
AABB 2015-03-13 4.9 39.5 9.9 65.4 N/A N/A
interjections 2015-03-13 4.9 40.7 9.9 65.4 N/A N/A
auto 2015-03-12 4.9 40.7 9.9 65.4 N/A N/A
w/o auto 2015-03-12 1.2 48.1 3.7 64.2 N/A N/A
change YYYY-MM-DD ordinary unk br unk+br generation end-to-end

mcg-smallest

               
Changes Date ordinary unk br unk+br generation end-to-end
relatives 2015-07-17 51.35 (3.16) / 61.11 (3.82) 57.66 (3.22) / 61.11 (3.82) 84.68 (5.0) / 94.44 (5.0) 100.0 (5.0) / 100.0 (5.0) 94.12 48.33
frag+512mb 2015-04-24 54.95 (3.34) / 61.11 (3.73) 61.26 (3.4) / 61.11 (3.73) 83.78 (5.0) / 94.44 (5.0) 99.1 (5.0) / 100.0 (5.0) 95.83 52.66
NP+PP 2015-04-20 55.86 (3.16) / 61.11 (3.73) 62.16 (3.23) / 61.11 (3.73) 83.78 (5.0) / 94.44 (5.0) 99.1 (5.0) / 100.0 (5.0) 95.89 53.56
prep+lzg+gle 2015-04-13 42.34 (2.7) / 38.89 (3.29) 50.45 (2.75) / 50.0 (2.78) 77.48 (5.0) / 72.22 (5.0) 98.2 (5.0) / 94.44 (5.0) 100.0 42.34
nydva 2015-04-10 36.94 (2.56) / 38.89 (3.29) 41.44 (2.67) / 50.0 (2.78) 77.48 (5.0) / 72.22 (5.0) 98.2 (5.0) / 94.44 (5.0) 100.0 36.94
spr+ditr 2015-04-02 34.23 (2.08) / 27.78 (3.4) 37.84 (2.12) / 27.78 (3.4) 70.27 (5.0) / 66.67 (5.0) 87.39 (5.0) / 72.22 (5.0) 100.0 34.23
16K+pu 2015-03-27 35.14 (2.49) / 27.78 (3.6) 38.74 (2.51) / 27.78 (3.6) 70.27 (5.0) / 66.67 (5.0) 87.39 (5.0) / 72.22 (5.0) 100.0 35.14
gTest 2015-03-20 24.32 (1.89) / 16.67 (2.33) 38.74 (2.81) / 22.22 (2.00) 45.05 (5.00) / 33.33 (5.00) 67.57 (5.00) / 44.44 (5.00) 100.0 24.32
AABB 2015-03-13 27.9 44.1 44.1 64.0 N/A N/A
interjections 2015-03-13 26.1 42.3 42.3 62.2 N/A N/A
auto 2015-03-12 26.1 42.3 42.3 62.2 N/A N/A
w/o auto 2015-03-12 4.5 46.8 7.2 56.8 N/A N/A
change YYYY-MM-DD ordinary unk br unk+br generation end-to-end

mcg-wxl

               
Changes Date ordinary unk br unk+br generation end-to-end
relatives 2015-07-17 54.75 (3.62) / 63.0 (3.1) 63.45 (3.56) / 63.0 (3.1) 82.44 (5.0) / 98.0 (5.0) 99.37 (5.0) / 100.0 (5.0) 91.69 50.2
frag+512mb 2015-04-24 56.17 (3.79) / 62.0 (3.29) 65.35 (3.77) / 62.0 (3.29) 80.7 (5.0) / 95.0 (5.0) 97.63 (5.0) / 97.0 (5.0) 93.05 52.27
NP+PP 2015-04-20 56.49 (3.74) / 62.0 (3.29) 65.35 (3.73) / 62.0 (3.29) 80.38 (5.0) / 95.0 (5.0) 97.31 (5.0) / 97.0 (5.0) 95.08 53.71
prep+lzg+gle 2015-04-13 43.99 (3.37) / 48.0 (2.81) 53.8 (3.36) / 53.0 (2.64) 75.47 (5.0) / 82.0 (5.0) 96.04 (5.0) / 95.0 (5.0) 98.77 43.45
nydva 2015-04-10 39.08 (3.19) / 48.0 (2.73) 43.99 (3.21) / 51.0 (2.63) 75.16 (5.0) / 82.0 (5.0) 96.04 (5.0) / 95.0 (5.0) 98.98 38.68
spr+ditr 2015-04-02 36.23 (2.71) / 43.0 (2.05) 43.2 (2.69) / 50.0 (2.26) 68.2 (5.0) / 72.0 (5.0) 87.18 (5.0) / 81.0 (5.0) 98.16 35.56
16K+pu 2015-03-27 35.92 (2.97) / 46.0 (2.52) 42.88 (2.91) / 53.0 (2.66) 68.2 (5.0) / 72.0 (5.0) 87.18 (5.0) / 81.0 (5.0) 99.19 35.63
gTest 2015-03-20 21.36 (2.21) / 23.0 (1.40) 34.97 (3.06) / 31.0 (2.19) 45.89 (5.00) / 45.0 (5.00) 70.09 (5.00) / 55.0 (5.00) 100.0 21.36
AABB 2015-03-13 23.9 37.8 44.5 66.3 N/A N/A
interjections 2015-03-13 23.4 37.3 43.8 65.5 N/A N/A
auto 2015-03-12 23.4 37.3 43.8 65.5 N/A N/A
w/o auto 2015-03-12 3.8 46.6 5.5 56.2 N/A N/A
change YYYY-MM-DD ordinary unk br unk+br generation end-to-end

Test Data

pctb-test (the second 5,000 sentences)

               
Changes Date ordinary unk br unk+br generation end-to-end
relatives 2015-07-17 0.88 (4.7) 6.26 (4.47) 10.76 (5.0) 44.26 (4.98) 45.45 0.4
frag+512mb 2015-04-24 1.06 (4.81) 7.88 (4.59) 10.68 (5.0) 41.6 (4.98) 40.0 0.42
NP+PP 2015-04-20 0.9 (4.53) 6.84 (4.59) 10.12 (5.0) 32.9 (4.97) 53.33 0.48
prep+lzg+gle 2015-04-13 0.7 (4.17) 5.68 (4.24) 10.14 (5.0) 39.46 (4.97) 60.0 0.42
nydva 2015-04-10 0.66 (4.12) 5.12 (4.22) 10.76 (5.0) 50.36 (4.98) 63.64 0.42
spr+ditr 2015-04-02 0.4 (4.25) 10.42 (2.6) 3.64 (5.0) 34.06 (4.97) 70.0 0.28
16K+pu 2015-03-27 0.4 (4.4) 10.44 (2.64) 3.62 (5.0) 34.22 (4.97) 75.0 0.3
gTest 2015-03-20 0.0 3.68 (4.41) 0.08 (5.00) 7.38 (4.89) 0.0 0.0
AABB 2015-03-13 0.0 8.5 0.0 9.9 N/A N/A
interjections 2015-03-13 0.0 8.5 0.0 9.9 N/A N/A
auto 2015-03-12 0.0 8.5 0.0 9.9 N/A N/A
w/o auto 2015-03-12 0.0 9.2 0.0 8.5 N/A N/A
change YYYY-MM-DD ordinary unk br unk+br generation end-to-end

sinica-test (the second 5,000 sentences (nltk_data))

               
Changes Date ordinary unk br unk+br generation end-to-end
relatives 2015-07-17 0.06 (4.0) 11.82 (4.87) 0.18 (4.56) 39.08 (4.96) 0.0 0.0
frag+512mb 2015-04-24 0.06 (4.0) 15.3 (4.93) 0.18 (4.56) 37.16 (4.96) 0.0 0.0
NP+PP 2015-04-20 0.04 (4.5) 13.92 (4.93) 0.18 (4.56) 29.3 (4.95) 50.0 0.02
prep+lzg+gle 2015-04-13 0.02 (5.0) 12.84 (4.85) 0.18 (4.56) 35.16 (4.96) 100.0 0.02
nydva 2015-04-10 0.02 (5.0) 11.42 (4.85) 0.18 (4.56) 36.92 (4.96) 100.0 0.02
spr+ditr 2015-04-02 0.02 (5.0) 7.76 (4.77) 0.16 (4.0) 29.4 (4.95) 100.0 0.02
16K+pu 2015-03-27 0.02 (5.0) 7.92 (4.79) 0.18 (3.67) 29.46 (4.95) 100.0 0.02
gTest 2015-03-20 0.0 5.14 (4.95) 0.02 (5.00) 3.74 (4.68) 0.0 0.0
AABB 2015-03-13 0.0 1.8 0.1 1.5 N/A N/A
interjections 2015-03-13 0.0 1.7 0.1 1.5 N/A N/A
auto 2015-03-12 0.0 1.7 0.1 1.5 N/A N/A
w/o auto 2015-03-12 0.0 1.7 0.0 1.1 N/A N/A
change YYYY-MM-DD ordinary unk br unk+br generation end-to-end

Last update: 2015-07-20 by SanghounSong [edit]