45 lines
1.2 KiB
TypeScript
45 lines
1.2 KiB
TypeScript
// pipeline.ts pseudo code
|
|
|
|
/*
|
|
step 1: discover source lists
|
|
|
|
this will give us an array of objects with this schema
|
|
sourceLanguage: 'en' | 'de' | 'es' | 'fr' | 'it'
|
|
pos: 'noun' | 'verb' | 'adjective' | 'adverb'
|
|
words: string[];
|
|
filePath: string;
|
|
|
|
the terminal output should be something like:
|
|
|
|
found 5 source lists:
|
|
|
|
de: noun
|
|
en: noun
|
|
|
|
and so on
|
|
|
|
later on, it will also contain it: noun, verb, adjective etc
|
|
*/
|
|
|
|
/*
|
|
|
|
step 2: validating source lists
|
|
|
|
a small script that trims whitespaces, removes duplicated words etc
|
|
|
|
terminal output: summary of how many words per pos per language were found
|
|
|
|
*/
|
|
|
|
/*
|
|
|
|
step 3: writing to database?
|
|
|
|
my thought: ill restart the pipeline several times during testing, and when adding more wordlists with other pos or extending the exisiting noun lists
|
|
eventually the lists will contain tens or hundreds of thousands of words
|
|
how do we prevent reading and processing the same words multiple times?
|
|
if we read and validate+normalize the wordlists and write them to the database, we could then read from the database fill the missing translations etc
|
|
and not read the same words from the same text files multiple times?
|
|
|
|
if we do this, we have to adjust the database schema because there are several notNull() rows inside
|
|
*/
|