Mario Pérez (@mape_code) header
Mario Pérez (@mape_code) profile picture

Mario Pérez

@mape_code

Data Engineer

 Spain
Joined June 2020
  • Tweets1.1K
  • Following550
  • Followers74
18Media
Photo by Mario Pérez (@mape_code)Photo by Mario Pérez (@mape_code)
Content-addressable deduplication at scale: MD5 hash all columns (minus metadata) → find hash collisions → save duplicates to _log table → keep one record. Implemented as BigQuery stored procedures. Track everything, delete nothing permanently. Easy rollback if needed.
1
35
more replies
MD5(COALESCE(CONCAT( COALESCE(sanitize_text(column1), ''), COALESCE(sanitize_text(column2), ''), COALESCE(sanitize_text(column3), '') )))
1
22
Key design decisions: sanitize_text() normalizes whitespace, case, accents Inner COALESCE converts NULL → empty string Outer COALESCE on CONCAT handles all-NULL rows to_hex() converts binary MD5 to readable string NULL-safe hashing = consistent results.
51
Using LLMs for data standardization at scale: Send messy province names to Claude with an official list → get back a mapping dictionary → validate programmatically → quarantine unmappable records to separate tables.
1
102
more replies
Traditional ETL: Write 50+ regex rules, maintain edge case dictionary, manual QA team reviews thousands of records. LLM-powered ETL: One prompt template, automatic fuzzy matching, quarantine pattern for edge cases, 95%+ automation rate
1
46
The AI doesn't replace human judgment - it triages at scale. Humans focus on the 5% that truly needs expertise, not the 95% that's tedious pattern matching. This is the future of data cleaning: semantic understanding + programmatic validation + comprehensive audit trails.
11
AGI?
Anthropic (@AnthropicAI) profile pictureAnthropic
@AnthropicAI29 Oct 2025
New Anthropic research: Signs of introspection in LLMs. Can language models recognize their own internal thoughts? Or do they just make up plausible answers when asked about them? We found evidence for genuine—though limited—introspective capabilities in Claude.
An example in which Claude Opus 4.1 detects a concept being injected into its activations.

ALT An example in which Claude Opus 4.1 detects a concept being injected into its activations.

19
Interesante entrevista sobre Codeoscopic, compañía que ha creado varias soluciones SAAS en el ecosistema como avant2, bcover, versus y tesis.
1
135
more replies
Un gran parte del revenue de la compañía proviene de vender datos a aseguradoras. Una demostración más de la importancia del data en este sector.
1
117
youtube.com/watch?v=b0gkSA0f…
120
Load more