Java Developer with Web Crawler Experience Job at Axiom Software Solutions Limited, Austin, TX

a1FWQkpibkR1YTZ0dDhuUFZ5enlGS2hhVmc9PQ==
  • Axiom Software Solutions Limited
  • Austin, TX

Job Description

Job Description

Job Description

Role: Java Developer with Web Crawler Experience

Location: Austin TX(Hybrid)

Responsibilities:

1.Web Crawler Development: Design and implement efficient and scalable web crawlers in Java to collect data from various online sources.

2.Data Extraction: Develop and maintain systems for structured data extraction, handling various data formats (HTML, JSON, XML, etc.).

3.Data Storage and Processing: Design data storage and processing pipelines, ensuring extracted data is clean, structured, and easily accessible.

4.Performance Optimization: Optimize web crawling processes for speed, efficiency, and accuracy, while ensuring minimal impact on source websites.

5.Error Handling and Logging: Implement error-handling mechanisms and logging systems to detect and resolve issues during crawling operations.

6.Data Integrity and Compliance: Ensure data collection practices are ethical, legal, and compliant with relevant regulations (e.g., robots.txt, copyright laws).

Requirements:

Proficiency in Java and experience with Java-based web scraping libraries (e.g., Jsoup, Apache

Knowledge of web crawling frameworks and tools, such as Scrapy, Selenium, or Puppeteer.

Strong understanding of HTML, CSS, JavaScript, and web data structures.

Familiarity with data parsing and handling techniques for JSON, XML, and other common formats.

Experience with database technologies (SQL, NoSQL) to store and manage scraped data.

Knowledge of protocols, headers, proxies, and load handling.

Job Tags

Similar Jobs

Haven a boutique salon

Hair salon assistant/ jr. stylist Job at Haven a boutique salon

 .... Opportunity to grow! MUST have nys cosmetology license, prior salon experience is a plus. Instagram portfolio a plus. Jr. Stylist...  ...treatments and blowdrys while training for additional commission. Hair extension certification optional. Color lines used: Goldwell,... 

TalentBurst, Inc

Local Contract Pharmacy Technician - $25 per hour Job at TalentBurst, Inc

 ...Job Description TalentBurst, Inc is seeking a local contract Pharmacy Technician for a local contract job in Philadelphia, Pennsylvania. Job Description & Requirements ~ Specialty: Pharmacy Technician ~ Discipline: Allied Health Professional ~ Start Date:... 

Mobile Wellness Solutions

Mobile Phlebotomist Job at Mobile Wellness Solutions

 ...Mobile Wellness Solutions is looking for a part-time Mobile Phlebotomist. Job Details: The candidate/phlebotomist is responsible...  ...venipuncture via the vacutainer method. The candidate will travel to the customer's location for appointments. Paid per event... 

Raytheon Careers

Principal Linux Systems Administrator Job at Raytheon Careers

 ...Onsite Principal Linux Systems Administrator- El Segundo, CA -Must have a TS/SCI active clearance, we offer relocation! At Raytheon, the foundation of everything we do is rooted in our values and a higher calling to help our nation and allies defend freedoms and... 

Diverse Lynx

Development Product Owner Job at Diverse Lynx

Job Description Job Title Development Product Owner Relevant Experience(in yrs) 8-10 Work Location (State, City and Zip) Texas - San Antonio...  ...-78288 Technical/Functional Skills P&C Insurance Scrum and Agile development methodologies SAFe Agile Jira...