maciejpi/nordabiz
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

fix(scraper): Dodano domeny paywall do SKIP_DOMAINS

Browse Source 
- wyborcza.pl - paywall Gazety Wyborczej
- rp.pl - paywall Rzeczpospolitej
- wykop.pl - agregator bez oryginalnej treści
- reddit.com - agregator

Te domeny zwracają cookie dialog zamiast treści artykułów

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
Maciej Pienczyn 2026-01-16 20:48:26 +01:00
parent 21586f6b91
commit 1e42c4fbd8
1 changed files with 7 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
zopk_content_scraper.py
View File

@ -140,12 +140,19 @@ ELEMENTS_TO_REMOVE = [
# Domains that are not scrapeable (paywalls, dynamic content, etc.)
SKIP_DOMAINS = [
# Social media
'facebook.com',
'twitter.com',
'x.com',
'linkedin.com',
'youtube.com',
'instagram.com',
# Paywalled news sites (require login, return cookie dialogs)
'wyborcza.pl', # Gazeta Wyborcza paywall
'rp.pl', # Rzeczpospolita paywall
# Aggregators (no original content)
'wykop.pl', # Social news aggregator
'reddit.com',
]