Data Preprocessing Pipeline for Brazilian Portuguese NLP Model Training
A robust pipeline for collecting, cleaning, and preparing Brazilian Portuguese text data for NLP model training.
Build a reproducible data pipeline that transforms raw data into high-quality datasets ready for Brazilian Portuguese NLP model training.
At a glance
Access
Free prompt
Open to copy — no account or payment needed.
Prompt objective
Build a reproducible data pipeline that transforms raw data into high-quality datasets ready for Brazilian Portuguese NLP model training.
Real use case
A Brazilian social media analytics company wants to train a sentiment analysis model for social media comments. They have 500,000 Instagram and Twitter comments, but 40% are spam, 15% contain emojis that confuse the tokenizer, and 10% mix Portuguese with English and regional slang.
Customize these fields first
Replace the placeholders with your own context before you run the prompt. That usually improves the first output more than adding more instructions later.
Prompt
Create a data preprocessing pipeline for [PROJECT NAME], preparing [TEXT DATA TYPE] in Brazilian Portuguese for training a [TASK: sentiment classification/NER/summarization/QA] model. **Context:** - Raw data volume: [NUMBER] records - Sources: [LIST: social media, reviews, documents, chat logs] - Known issues: [LIST: spam, emojis, slang, typos, etc.] - Language: Brazilian Portuguese (with possible excerpts in [OTHER LANGUAGES]) - ML Framework: [HUGGING FACE/SPACY/PYTORCH] **1) Collection and Ingestion:** - Collection scripts per source (API, web scraping, database export) - Rate limiting and retry logic for APIs - Deduplication: hash-based (exact) + MinHash LSH (near-duplicate) - Storage format: Parquet (efficient) or JSONL - Metadata per record: source, timestamp, author_id, language **2) Text Cleaning:** - Encoding normalization (UTF-8) - HTML/markdown tag removal - URL handling: remove, replace with [URL], or keep domain - Mention (@user) and hashtag handling - Emojis: [REMOVE/CONVERT TO TEXT/KEEP] — justify for the task - Whitespace and punctuation normalization - Corrupted encoding fixes (mojibake) - PII removal: emails, phone numbers, national IDs (regex + NER) - Complete reusable Python script with functions **3) Quality Filtering:** - Language detection (langdetect/fasttext-lid): filter non-Portuguese content - Length filter: minimum [X] tokens, maximum [Y] tokens - Spam/bot filter: repetitive patterns, excessive links - Toxic content filter (if relevant) - Perplexity-based filtering (remove incoherent text) - Before/after statistics for each filter **4) Linguistic Processing (Brazilian Portuguese):** - Tokenization with target model tokenizer - Brazilian regional slang normalization (mapping) - Common abbreviation handling (vc, tb, pq, q, cmg, blz → voce, tambem, porque, que, comigo, bele) - Stemming vs. Lemmatization: when to use each - Named Entity Recognition for PII masking - Light spell checking (for obvious errors, not slang) **5) Annotation:** - Annotation strategy: [MANUAL/SEMI-AUTOMATIC/WEAK SUPERVISION] - Annotation guidelines (document for annotators) - Inter-annotator agreement (Cohen's Kappa target: > 0.8) - Tools: [LABEL STUDIO/PRODIGY/CUSTOM] - Active learning: prioritize examples where the model has most uncertainty - Augmentation: back-translation, paraphrasing, synonym replacement **6) Dataset Split and Versioning:** - Stratified split: train/val/test maintaining class distribution - Time-based split (for temporal data): avoid data leakage - DVC (Data Version Control) for dataset versioning - Dataset card: statistics, class distribution, limitations, biases - Export in Hugging Face Datasets format **7) Reproducible Pipeline:** - Orchestration: [MAKE/PREFECT/AIRFLOW/DVC PIPELINE] - YAML config for parameterization - Logging per step with statistics - Reproducible: fixed seed, pinned dependencies - Processing time and cost estimates Provide complete Python code with input/output examples for each step.
Open directly in an AI — the text is pre-filled:
How to use this prompt
- 1Replace the key placeholders first: PROJECT NAME, TEXT DATA TYPE, TASK: sentiment classification/NER/summarization/QA, NUMBER.
- 2Replace any bracketed placeholders like [this] with your own context.
- 3Add extra background information when you want more tailored results.
- 4Combine multiple prompts in one conversation when you need a richer output.
- 5Save your best-performing prompts so they are easy to reuse later.
Next best step
Open the guide first, then branch only if you still need more.
A guide for technical builders choosing between prompts, coding workflows, and agent-based implementation.
If this prompt is close but not quite right, generate variants next. If the job is recurring, move into the course library after the guide.
Related prompts
View allFine-tuning LLMs with Custom Data Using LoRA and QLoRA
Complete guide to fine-tuning language models with efficient parameter adaptation techniques.
Best for
Fine-tune a large language model for a specific domain, minimizing computational costs with PEFT (Parameter-Efficient Fine-Tuning) techniques.
RAG Pipeline (Retrieval-Augmented Generation) with Embeddings and Vector Database
Complete RAG implementation covering chunking, embeddings, semantic search, and augmented generation.
Best for
Build a document-based Q&A system that reduces hallucinations and delivers answers grounded in your actual data.
Advanced Prompt Engineering with Chain-of-Thought and Function Calling
Advanced prompt engineering techniques to maximize LLM performance in production environments.
Best for
Master prompt engineering techniques that improve the quality, consistency, and reliability of LLM responses in production applications.
Complete MLOps Pipeline with Model Training, Versioning, and Deployment
MLOps infrastructure to manage the full lifecycle of ML models in production.
Best for
Implement an MLOps pipeline that automates model training, evaluation, versioning, and deployment with continuous monitoring.
Explore other prompt categories
Move sideways into adjacent libraries when the current category is not the full answer.
Every prompt here is free. The course teaches the thinking behind them.
Copy as many prompts as you like. When you want to move from single prompts to a repeatable AI workflow, Learn AI in 30 Days walks through it, one day at a time.
Buy the course once ($15/$20 by length), or go all-access for $10/mo with a verifiable certificate.