Files
DomGod/app/__pycache__/replicate_ai.cpython-311.pyc

99 lines
13 KiB
Plaintext
Raw Normal View History

fix: AI worker crash-proof + GDPR/hosting/accessibility analysis AI worker fixes (root cause of "nothing reaches Replicate"): - Worker task died silently — no exception handler around while loop - Added try/except around entire loop body with exc_info logging - Added watchdog task that restarts dead workers every 10 seconds - ensure_workers_alive() called on every /api/ai/assess/batch POST - _assess_one() is now a top-level function (not closure) — avoids subtle scoping bugs with async inner functions in while loops - /api/ai/debug endpoint: shows worker alive status, task exception, last 10 queue entries — browse to /api/ai/debug to diagnose - /api/ai/worker/restart endpoint + UI button - "Restart AI worker" button + "Debug AI queue" link in enrichment tab site_analyzer.py — new signals: - IP resolution + ip-api.com for ASN, org, ISP, host country - EU hosting detection (27 EU + EEA + adequacy countries) - GDPR: detects Cookiebot, OneTrust, CookiePro, Osano, Iubenda, Borlabs, CookieYes, Complianz, Usercentrics + text signals - Privacy policy and GDPR text presence - Accessibility: html lang missing, images without alt count, skip nav link, empty links, inputs without labels Gemini prompt additions: - Hosting section: IP, ASN, org/ISP, EU vs non-EU flag - GDPR section: cookie tool, notice, privacy policy - Accessibility section: all quick-scan results - New output fields: hosting_notes, gdpr_compliance, accessibility_issues[] Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-13 18:01:34 +02:00
<EFBFBD>
q<0F>ig<00><00>,<00>UdZddlZddlZddlZddlZddlZddlmZddlZej e
<EFBFBD><00>Z ej dd<05><00>Z dZeej dd<08><00><00><00>Zdaeejed <d
ejfd <0B>Zd ed
efd <0A>Zded
efd<0F>Zded
efd<11>ZdS)u8Replicate / Gemini integration — deep site assessment.<2E>N)<01>Optional<61>REPLICATE_API_TOKEN<45>(r8_7I7Feai78f9PzMOs20y5GVFKiLkgUWP463vZOzChttps://api.replicate.com/v1/models/google/gemini-3-pro/predictions<6E>AI_CONCURRENCY<43>3<>_ai_sem<65>returnc<00>P<00>t<00>tjt<00><00>atS)N)r<00>asyncio<69> Semaphorer<00><00><00>-/home/malin/c0ding/DomGod/app/replicate_ai.py<70>_semrs<00><00><0E><EFBFBD><19>#<23>N<EFBFBD>3<>3<><07> <12>Nr<00>ac<00><> <00>g}|<00>d<01><00>r9|<01>dd<03>|ddd<05><00><00><00><00><02><00>|<00>d<06><00>r9|<01>dd<03>|ddd<05><00><00><00><00><02><00>|<00>d<08><00>r9|<01>d d<03>|ddd
<EFBFBD><00><00><00><00><02><00>|<00>d <0B><00>r9|<01>d d<03>|d dd <0A><00><00><00><00><02><00>d<0E>|<01><00>pd}d<0E>d<10>|<00>d<11><00>pgD<00><00><00><00>pd}d<03>|<00>d<13><00>pg<00><00>pd}d<03>|<00>d<15><00>pg<00><00>pd}d<03>|<00>d<16><00>pg<00><00>pd}d<03>|<00>d<17><00>pg<00><00>pd}|<00>d<18><00>pddd<1A>}d<19>gd<1B>|<00>d<1C><00><00><00>d<1D>|<00>d<1E><00><00><00>d<1F>|<00>d <20><00><00><00>d!<21>|<00>d"<22><00><00><00>d#<23>|<00>d$<24><00><00><00>d%<25>|<00>d&<26><00><00><00>d'<27>|<00>d(<28><00><00><00>d)<29>|<00>d*<2A><00>pd+<2B><00>d,<2C>|<00>d-<2D><00><00><00>d.<2E>|<00>d/<2F><00><00><00>d0<64>|<00>d1<64><00><00><00>d2<64>|<00>d3<64><00><00><00>d4<64>|<00>d5<64><00><00><00>d6<64>|<00>d7<64><00><00><00>d8<64>|<00>d9<64><00>pd:<3A><00>d;<3B>|<00>d<<3C><00>pd:<3A><00>d=<3D>|<00>d><3E><00>pd:<3A><00>d?<3F>|<00>d@<40><00>pdA<64><00>dB<64>|<00>dC<64><00><00><00>dD<64>|<00>dE<64><00><00><00>dF<64>|<00>dG<64><00><00><00>dH<64>|<04><00>dI<64>|<05><00>dJ<64>|<00>dK<64><00><00><00>dL<64>|<06><00>dM<64>|<00>dN<64><00><00><00>dL<64>|<07><00>dO<64>|<00>dP<64><00><00><00>dQ<64>|<03><00>dR<64>|<02><00>dS<64>|<08><00>dT<64><01><00>S)Uz7Build the Gemini prompt from a full site analysis dict.<2E>emailsz Emails: z, N<><00>phonesz Phones: <20>whatsappz WhatsApp: <20><00> social_linksz Social: <20><00>
z None foundc3<00> K<00>|] }d|<01><00>V<00><00>
dS)z - Nr )<02>.0<EFBFBD>ss r<00> <genexpr>z _build_prompt.<locals>.<genexpr>$s(<00><00><00><00>P<>P<>a<EFBFBD>z<EFBFBD>a<EFBFBD>z<EFBFBD>z<EFBFBD>P<>P<>P<>P<>P<>Pr<00>kit_digital_signalsz None detected<65>analytics_present<6E>none<6E>webmaster_verified<65> lorem_matches<65>placeholder_matches<65>visible_text_snippet<65>i<>z<>You are a senior web consultant and IT sales analyst reviewing a Spanish SME website.
=== TECHNICAL SNAPSHOT ===
Domain: <20>domainz
Reachable: <20> reachablez | Status: <20> status_codez | Load time: <20> load_time_msz ms
Final URL: <20> final_urlz
Page size: <20> page_size_kbz KB | Server: <20>serverz
| CMS: <20>cms<6D>unknownz
SSL valid: <20> ssl_validz | SSL expires in: <20>ssl_expiry_daysz days
Mobile viewport: <20>has_mobile_viewportz
Word count: <20>
word_countz | Images: <20> image_countz | Scripts: <20> script_countz4
=== SEO & INDEXING SIGNALS ===
Page title: <20>
page_title<EFBFBD>missingz
H1: <20>h1_textz
Meta description: <20>meta_descriptionz
Canonical URL: <20> canonical_urlznot setz
Sitemap.xml: <20> has_sitemapz
Robots.txt: <20>
has_robotsz | Blocks Googlebot: <20>robots_disallows_googlez
Analytics: z
Webmaster verified:z-
=== CONTENT QUALITY ===
Lorem ipsum found: <20>has_lorem_ipsumu → matches: z
Placeholder text: <20>has_placeholderul
=== KIT DIGITAL (Spanish gov digitalization grant — sites must display EU logos) ===
Detected: <20> kit_digitalz
Signals:
z
=== CONTACT CHANNELS ===
z.
=== PAGE TEXT SAMPLE (first 2000 chars) ===
u<EFBFBD>
=== TASK ===
Analyse this site for IT services upsell potential. The client sells:
web design/redesign, SEO, hosting migration, SSL renewal, security audits,
maintenance contracts, Google Ads, and AI-assisted tools for SMEs.
Respond ONLY with valid JSON — no markdown, no text outside the JSON object:
{
"summary": "2-3 sentence executive summary of the site's current state",
"site_quality_score": <0-10 integer>,
"content_issues": ["list of specific content problems found — lorem ipsum, broken sections, placeholder text, etc."],
"performance_notes": "comment on load time, page size, mobile readiness",
"seo_status": "brief SEO assessment — indexing signals, missing elements",
"kit_digital_confirmed": true/false,
"kit_digital_reasoning": "1 sentence — why confirmed or not",
"is_local_sme": true/false,
"lead_quality": "HOT|WARM|COLD",
"lead_reasoning": "1-2 sentences on why",
"best_contact_channel": "email|phone|whatsapp|social|web_form|unknown",
"best_contact_value": "the actual value to use (email address, phone number, URL) or empty string",
"all_contacts": {
"emails": [],
"phones": [],
"whatsapp": [],
"social": []
},
"pitch_angle": "One concrete opening sentence in Spanish for cold outreach",
"services_needed": ["service1", "service2"],
"urgency_signals": ["list of specific urgent issues — expiring SSL, lorem ipsum live, no GA, blocked robots etc"],
"outreach_notes": "Key context for the sales rep"
})<03>get<65>append<6E>join) r<00>contacts_block<63> contacts_str<74>kd_str<74> analytics_str<74> webmaster_str<74> lorem_str<74>placeholder_str<74> text_snippets r<00> _build_promptrLsB<00><00><17>N<EFBFBD><08>u<EFBFBD>u<EFBFBD>X<EFBFBD><EFBFBD><EFBFBD>^<5E>><3E>0<>0<>1]<5D><14><19><19>1<EFBFBD>X<EFBFBD>;<3B>WY<57>XY<58>WY<57>?<3F>A[<5B>A[<5B>1]<5D>1]<5D>^<5E>^<5E>^<5E><08>u<EFBFBD>u<EFBFBD>X<EFBFBD><EFBFBD><EFBFBD>^<5E>><3E>0<>0<>1]<5D><14><19><19>1<EFBFBD>X<EFBFBD>;<3B>WY<57>XY<58>WY<57>?<3F>A[<5B>A[<5B>1]<5D>1]<5D>^<5E>^<5E>^<5E><08>u<EFBFBD>u<EFBFBD>Z<EFBFBD><18><18>`<60>><3E>0<>0<>1_<31><14><19><19>1<EFBFBD>Z<EFBFBD>=<3D>Y[<5B>Z[<5B>Y[<5B>K\<5C>A]<5D>A]<5D>1_<31>1_<31>`<60>`<60>`<60><08>u<EFBFBD>u<EFBFBD>^<5E><1C><1C>g<>n<EFBFBD>3<>3<>4f<34>D<EFBFBD>I<EFBFBD>I<EFBFBD>a<EFBFBD>P^<5E>N_<4E>`b<>ab<61>`b<>Nc<4E>Dd<44>Dd<44>4f<34>4f<34>g<>g<>g<><17>9<EFBFBD>9<EFBFBD>^<5E>,<2C>,<2C>><3E><0E>L<EFBFBD> <11>Y<EFBFBD>Y<EFBFBD>P<>P<>A<EFBFBD>E<EFBFBD>E<EFBFBD>2G<32>,H<>,H<>,N<>B<EFBFBD>P<>P<>P<> P<> P<> e<>Te<54>F<EFBFBD><18>I<EFBFBD>I<EFBFBD>a<EFBFBD>e<EFBFBD>e<EFBFBD>$7<>8<>8<>><3E>B<EFBFBD>?<3F>?<3F>I<>6<EFBFBD>M<EFBFBD><18>I<EFBFBD>I<EFBFBD>a<EFBFBD>e<EFBFBD>e<EFBFBD>$8<>9<>9<>?<3F>R<EFBFBD>@<40>@<40>J<>F<EFBFBD>M<EFBFBD><18>I<EFBFBD>I<EFBFBD>a<EFBFBD>e<EFBFBD>e<EFBFBD>O<EFBFBD>4<>4<>:<3A><02>;<3B>;<3B>E<>v<EFBFBD>I<EFBFBD><1A>i<EFBFBD>i<EFBFBD><01><05><05>&;<3B> <<3C> <<3C> B<><02>C<>C<>M<>v<EFBFBD>O<EFBFBD><15>E<EFBFBD>E<EFBFBD>0<>1<>1<>7<>R<EFBFBD><15>$<24><15>?<3F>L<EFBFBD>A <06>A <06>A <06>A <06>A <06>A <06><16>E<EFBFBD>E<EFBFBD>(<28>O<EFBFBD>O<EFBFBD>A <06>A <06>A <06>A <06><16>E<EFBFBD>E<EFBFBD>+<2B>&<26>&<26> A <06>A <06>A <06>A <06>67<36>U<EFBFBD>U<EFBFBD>=<3D>5I<35>5I<35> A <06>A <06>A <06>A <06>\]<01>[`<60>[`<60>ao<61>[p<>[p<> A <06>A <06>A <06>A <06>
<16>E<EFBFBD>E<EFBFBD>+<2B>&<26>&<26> A <06>A <06>A <06>A <06> <16>E<EFBFBD>E<EFBFBD>.<2E>)<29>)<29> A <06>A <06>A <06>A <06> <=<3D>5<EFBFBD>5<EFBFBD><18>?<3F>?<3F> A <06>A <06>A <06>A <06> WX<01>V[<5B>V[<5B>\a<>Vb<56>Vb<56>Vo<56>fo<66> A <06>A <06>A <06>A <06><16>E<EFBFBD>E<EFBFBD>+<2B>&<26>&<26>A <06>A <06>A <06>A <06>>?<3F>U<EFBFBD>U<EFBFBD>CT<43>=U<>=U<>A <06>A <06>A <06>A <06><16>E<EFBFBD>E<EFBFBD>/<2F>0<>0<>A <06>A <06>A <06>A <06><16>E<EFBFBD>E<EFBFBD>,<2C>'<27>'<27>A <06>A <06>A <06>A <06>78<37>e<EFBFBD>e<EFBFBD>M<EFBFBD>6J<36>6J<36>A <06>A <06>A <06>A <06>[\<01>Z_<5A>Z_<5A>`n<>Zo<5A>Zo<5A>A <06>A <06>A <06>A <06><16>E<EFBFBD>E<EFBFBD>,<2C>'<27>'<27>4<>9<EFBFBD>A <06>A <06>A <06>A <06><16>E<EFBFBD>E<EFBFBD>)<29>$<24>$<24>1<> <09>A <06>A <06>A <06>A <06><16>E<EFBFBD>E<EFBFBD>,<2C>-<2D>-<2D>:<3A><19>A <06>A <06>A <06>A <06><16>E<EFBFBD>E<EFBFBD>/<2F>*<2A>*<2A>7<>i<EFBFBD>A <06>A <06>A <06>A <06> <16>E<EFBFBD>E<EFBFBD>-<2D>(<28>(<28>!A <06>A <06>A <06>A <06>"<16>E<EFBFBD>E<EFBFBD>,<2C>'<27>'<27>#A <06>A <06>A <06>A <06>"AB<01><05><05>F_<46>@`<60>@`<60>#A <06>A <06>A <06>A <06>$"<22>%A <06>A <06>A <06>A <06>&"<22>'A <06>A <06>A <06>A <06>,<16>E<EFBFBD>E<EFBFBD>+<2B>,<2C>,<2C>-A <06>A <06>A <06>A <06>,?H<01>-A <06>A <06>A <06>A <06>.<16>E<EFBFBD>E<EFBFBD>+<2B>,<2C>,<2C>/A <06>A <06>A <06>A <06>.?N<01>/A <06>A <06>A <06>A <06>4<16>E<EFBFBD>E<EFBFBD>-<2D>(<28>(<28>5A <06>A <06>A <06>A <06>8<08>9A <06>A <06>A <06>A <06>><0E>?A <06>A <06>A <06>A <06>D<0E>EA <06>A <06>A <06>A <06>A <06>A <06>Ar<00>rawc<00><><00>tjdd|<00><00><00><00><00><00>d<03><00><00><00><00>}tjd|<01><00>}|r= t j|<02>d<05><00><00><00>S#t
j$rYnwxYwt<00>
d|dd<07><00><00>|dd<08>d d
dd d <0C>S) Nz ```(?:json)?r&<00>`z \{[\s\S]+\}rz+Could not parse Gemini JSON output, raw: %s<>,i<><00>COLDr/T)<05>summary<72> lead_quality<74>best_contact_channel<65>best_contact_value<75> parse_error) <0B>re<72>sub<75>strip<69>rstrip<69>search<63>json<6F>loads<64>group<75>JSONDecodeError<6F>logger<65>warning)rM<00>text<78>ms r<00> _parse_outputrdps<><00><00> <0A>6<EFBFBD>/<2F>2<EFBFBD>s<EFBFBD> +<2B> +<2B> 1<> 1<> 3<> 3<> :<3A> :<3A>3<EFBFBD> ?<3F> ?<3F> E<> E<> G<> G<>D<EFBFBD>
<EFBFBD> <09>.<2E>$<24>'<27>'<27>A<EFBFBD><08><11> <11><17>:<3A>a<EFBFBD>g<EFBFBD>g<EFBFBD>a<EFBFBD>j<EFBFBD>j<EFBFBD>)<29>)<29> )<29><><13>#<23> <11> <11> <11> <10>D<EFBFBD> <11><><EFBFBD><EFBFBD>
<EFBFBD>N<EFBFBD>N<EFBFBD>@<40>#<23>d<EFBFBD>s<EFBFBD>d<EFBFBD>)<29>L<>L<>L<><16>t<EFBFBD><03>t<EFBFBD>9<EFBFBD><1E> )<29> <20><1B>  <06> <06>s<00>&&B <00> B<03>B<03>analysisc
<00><00>K<00>t<00><00>4<00>d{V<00><03>dt|<00><00>ggddddd<07>i} tjd<08> <09><00>4<00>d{V<00><03>}|<02>t
d
t <00><00>d d d <0A>|<01><0E><00><00>d{V<00><03>}|<03><00><00>|<03><00><00>}ddd<01><00><00>d{V<00><03>n#1<00>d{V<00><03>swxYwY|<04> dd<10><00>}t|t<00><00>rd<10> |<05><00>}t|<05><00>}t<00>d|<00> d<12><00>|<06> d<13><00>|<06> d<14><00><00><00>|cddd<01><00><00>d{V<00><03>S#t $rg}t<00>d|<00> d<12><00>|<07><00>t%|<07><00>dd<16>dddd<19>cYd}~cddd<01><00><00>d{V<00><03>Sd}~wwxYw#1<00>d{V<00><03>swxYwYdS)zCCall Gemini with the full site analysis. Returns parsed assessment.N<>inputg<74><67><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>?g<><67><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>?<3F>lowi)<07>prompt<70>images<65>videos<6F>top_p<5F> temperature<72>thinking_level<65>max_output_tokens<6E>x)<01>timeoutzBearer zapplication/json<6F>wait)<03> Authorizationz Content-Type<70>Prefer)<02>headersr\<00>outputr&uAI %s → %s (quality %s)r'rS<00>site_quality_scorezReplicate error %s: %srPrQr/)<04>errorrSrTrU)rrL<00>httpx<70> AsyncClient<6E>post<73>REPLICATE_MODEL<45>REPLICATE_TOKEN<45>raise_for_statusr\rA<00>
isinstance<EFBFBD>listrCrdr`<00>info<66> Exceptionrx<00>str)re<00>payload<61>client<6E>resp<73>datarv<00>result<6C>es r<00> assess_domainr<6E><00>s<><00><00><00><00><13>v<EFBFBD>v<EFBFBD>*<0E>*<0E>*<0E>*<0E>*<0E>*<0E>*<0E>*<0E> <13>'<27><08>1<>1<><1D><1D><1E>"<22>"'<27>%)<29><0E><0E>

<EFBFBD><07> <0E><1C>(<28><13>5<>5<>5<> #<23> #<23> #<23> #<23> #<23> #<23> #<23><16>#<23>[<5B>[<5B>#<23>)D<>?<3F>)D<>)D<>);<3B>)/<2F><16><16>
!<21>)<29><12><12><12><12><12><12><12><12><04><15>%<25>%<25>'<27>'<27>'<27><1B>y<EFBFBD>y<EFBFBD>{<7B>{<7B><04> #<23> #<23> #<23> #<23> #<23> #<23> #<23> #<23> #<23> #<23> #<23> #<23> #<23> #<23> #<23> #<23> #<23> #<23> #<23> #<23> #<23> #<23> #<23><><EFBFBD><EFBFBD> #<23> #<23> #<23> #<23><1A>X<EFBFBD>X<EFBFBD>h<EFBFBD><02>+<2B>+<2B>F<EFBFBD><19>&<26>$<24>'<27>'<27> )<29><1B><17><17><16><1F><1F><06>"<22>6<EFBFBD>*<2A>*<2A>F<EFBFBD> <12>K<EFBFBD>K<EFBFBD>3<> <20> <0C> <0C>X<EFBFBD>.<2E>.<2E><06>
<EFBFBD>
<EFBFBD>><3E>0J<30>0J<30>F<EFBFBD>J<EFBFBD>J<EFBFBD>Wk<57>Ll<4C>Ll<4C> n<01> n<01> n<01><19>C*<0E>*<0E>*<0E>*<0E>*<0E>*<0E>*<0E>*<0E>*<0E>*<0E>*<0E>*<0E>*<0E>*<0E><>F<19> <0E> <0E> <0E> <12>L<EFBFBD>L<EFBFBD>1<>8<EFBFBD><<3C><<3C><08>3I<33>3I<33>1<EFBFBD> M<> M<> M<>'*<2A>1<EFBFBD>v<EFBFBD>v<EFBFBD>d<EFBFBD>s<EFBFBD>d<EFBFBD>|<7C>'-<2D>(1<>')<29> <0E><0E> <0E> <0E> <0E> <0E> <0E>K*<0E>*<0E>*<0E>*<0E>*<0E>*<0E>*<0E>*<0E>*<0E>*<0E>*<0E>*<0E>*<0E>*<0E><><EFBFBD><EFBFBD><EFBFBD>F <0E><><EFBFBD><EFBFBD>G*<0E>*<0E>*<0E>*<0E>*<0E>*<0E>*<0E>*<0E>*<0E>*<0E><><EFBFBD><EFBFBD>*<0E>*<0E>*<0E>*<0E>*<0E>*sa<00>G5<03>F<02> AB7<05>%F<02>7
C <09>F<02>C <09>B)F<02>
G2<05> A
G-<05>G2<05>G5<03>-G2<05>2G5<03>5
G?<07>G?)<1A>__doc__r r\<00>logging<6E>osrW<00>typingrry<00> getLogger<65>__name__r`<00>getenvr}r|<00>intrrr <00>__annotations__r<00>dictr<74>rLrdr<>r rr<00><module>r<>sM<00><01>><3E>><3E>><3E><0E><0E><0E><0E> <0B> <0B> <0B> <0B><0E><0E><0E><0E> <09> <09> <09> <09> <09> <09> <09> <09><1B><1B><1B><1B><1B><1B> <0C> <0C> <0C> <0C> <1A><17> <1A>8<EFBFBD> $<24> $<24><06><1B>"<22>)<29>1<>3]<5D>^<5E>^<5E><0F>W<><0F><15>#<23>i<EFBFBD>b<EFBFBD>i<EFBFBD> 0<>#<23>6<>6<>7<>7<><0E>'+<2B><07><18>'<27>#<23> $<24>+<2B>+<2B>+<2B><13>g<EFBFBD><1F><13><13><13><13>R<06>T<EFBFBD>R<06>c<EFBFBD>R<06>R<06>R<06>R<06>j<06>s<EFBFBD><06>t<EFBFBD><06><06><06><06>$,<0E>$<24>,<0E>4<EFBFBD>,<0E>,<0E>,<0E>,<0E>,<0E>,r