Help needed - bulk extraction of words

Previous Topic Next Topic
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Help needed - bulk extraction of words


I did word collection several times using different sources( web sources)

I use linux, but these tools are also available
for windows as gnu tools.

I used awk, like:
 for (i = 1; i <= $NF; i++)
   print $i;

This prints each word in a single line.

Then I sorted the file using sort <infile > outfile
and then used further awk scripts to get rid of word endings,
this is probably much easier for Danish, than for Hungarian.

Good luck! Eleonora

[lingu-dev] Help needed - bulk extraction of words

Hi all,
The Danish project has been so fortunate to receive a bunch of articles
from a news magazine. These are odt files and we would like to extract
the words from these documents. We have programs for this purpose, but
we usually get donations one document at the time. This time we have
several thousand documents and I believe it would take about a year to
load these documents one by one.

Do any of you have a program that can extract words from several documents ?

The words will be loaded into our workflow for linguistic processing and
at the end be a part of the Danish spelling directory.

Thanks in advance.

Med venlig hilsen - best regards,

Leif Lodahl

To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]