|
Germanic Lexicon Project
Message Board
|
|
|
Author: Keith Briggs
Date: 2004-11-09 06:47:45
Subject: Re: Probabilistic correction
> That's a good idea. There would need to be some way to handle the problem the text has a lot of errors (so a trigram of Latin words might contain two good words and one bad word, but you could probably still use that noisy information to make a good guess).
> ...
> The general approach seems to make a lot of sense. It's certainly worth a try. Sometimes I try things like this and they work beautifully, and other times they are a disappointment. I don't know which one this will be. There's the non-trivial groundwork of getting a Latin corpus assembled from the stuff on the web.
> ...
> There is also the issue of the abbrevations. You get a one, two, or three word abbreviation, followed by number comma number (then optional: space, letter). These could be treated as another "language". The difference in this case is that we have a complete list of the abbreviations, since the authors give them to us. There's a nice XML file I made containing the B/T abbreviations; it is on the B/T page under "Texts".
Lots to think about here! At the moment I have time for only a few simple tests, but I'm thinking more and more that we need to initiate a proper research project on this. I might be able to organize something next year with some Master's program I am involved with - if this works, we could have a student work for 3 months next summer. But would this be too late?
Meanwhile, I trained on a Latin text - I picked Pliny Liber II at random (it's about 2200 words). Then, I split the BT gehelmian entry into 11 lines. Three lines are Latin, all with errors. Two of these are the only lines dbacl gives a score for the Latin category much greater than zero, although they still score higher as c (=corrected BT). It fails to recognize "corSnasti nos" - not surprising as the Pliny text contains no instances of an "asti" termination. Still, this is quite promising - I think by training on bigger corpora we should be able to get very reliable language recognition. Remember that category c contains mixed languages, so it's really a bad model.
> db="dbacl -c c -c latin -vN"
> echo "ge-helmian ; p. ode, ede; pp. od, ed" | ${db}
c 100.00% latin 0.00%
> echo "To cover with a helmet, crown;" | ${db}
c 100.00% latin 0.00%
> echo "galeSre, coronare" | ${db}
c 93.32% latin 6.68%
> echo ":-- ÐÚ gehelmodest us" | ${db}
c 100.00% latin 0.00%
> echo "corSnasti nos," | ${db}
c 100.00% latin 0.00%
> echo "Ps. Spl. 5, 15. Of wuldre and weorþmynt ðú" | ${db}
c 100.00% latin 0.00%
> echo "gehelmedest hine" | ${db}
c 99.99% latin 0.01%
> echo "de gloria et hondre coronasti eum," | ${db}
c 99.79% latin 0.21%
> echo "Ps. Spl. T. 8, 6. Gehelmod " | ${db}
c 100.00% latin 0.00%
> echo "gáleátus, Ælfc. Gr. 43; Som. 45, II." | ${db}
c 100.00% latin 0.00%
> echo "[Laym, i-helmed : O. H. Ger. gehelmot.]" | ${db}
c 100.00% latin 0.00%