Integrated Web Scraping And Data Analysis Pipeline Tokopedia Github

Leo Migdal

-Nov 17, 2025, 5:18 AM

integrated web scraping and data analysis pipeline tokopedia github

Developed an automated dual-source web scraping and data integration pipeline combining GraphQL API extraction (Tokopedia) and pseudo REST API HTML parsing (TurnBackHoax.id). The unified dataset supports text mining, misinformation detection (NLP), and e-commerce trend analysis. GraphQL API Integration (Tokopedia) Built a dynamic scraper that sends GraphQL payloads to Tokopedia’s https://gql.tokopedia.com/graphql/SearchProductQueryV4. Extracted detailed product attributes such as name, price, rating, shop, and city through a single GraphQL endpoint. Pseudo REST API Scraping (TurnBackHoax.id) Implemented an HTML parser using BeautifulSoup to simulate REST-style HTTP GET requests, extracting structured data fields — title, date, category, and fact-check results — from TurnBackHoax.id articles. Data Integration Unified both sources into a structured Pandas DataFrame, enabling direct use for NLP preprocessing, EDA, or BI dashboards.

Automated Export & Visualization Implemented an automated .xlsx export pipeline for live storage and used Matplotlib to visualize price clustering and misinformation trends. Welcome to my portfolio! Here you’ll find a curated selection of data projects, dashboards, and analyses demonstrating my experience in data analytics, automation, and visualization. Created comprehensive finance dashboards (AR/AP/CF/Budget) to automate performance monitoring for the FAT department. Automated summary generation and streamlined weekly review. Role / Tools: Automation & Reporting Analyst — Excel Impact: Simplified cross-department financial monitoring; improved visibility and accuracy.

Created an Excel-based dashboard to automate Balance Sheet and Profit & Loss reporting, aggregate COA, and surface five bank-aligned financial ratios for monthly monitoring. Role / Tools: Automation & Reporting Analyst — Excel, Power Query Impact: Simplified reporting and monitoring; improved visibility of financial position and key ratios. This notebook focuses on performing web scraping to collect data from the Tokopedia website, followed by data preprocessing, analysis, and visualization. The goal is to extract meaningful insights from the scraped data, which can be used to understand market trends, product popularity, and other relevant metrics on Tokopedia. For details and to view the notebook, check out this Jupyter Notebook. Proyek ini bertujuan untuk melakukan web scraping atau crawling data dari situs e-commerce Tokopedia untuk mengumpulkan informasi produk secara otomatis.

Data yang diambil termasuk nama produk, harga, rating, jumlah ulasan, dan informasi lainnya yang berguna untuk keperluan riset atau analisis lebih lanjut. Pastikan Anda telah mengunduh dan mengkonfigurasi WebDriver untuk Selenium (misalnya, ChromeDriver untuk Google Chrome). Pastikan WebDriver ada di PATH Anda atau disimpan di lokasi yang diketahui. Jalankan script utama untuk memulai proses crawling: Anda dapat menyesuaikan URL dan kategori produk yang ingin diambil dengan mengedit parameter di script. Setelah proses crawling selesai, data yang terkumpul akan disimpan di folder dataset/ dalam format CSV.

This repository is an automated web scraping method, to scrape or collect data from a famous e-commerce in Indonesia, Tokopedia. The scrape method is build using Python language, with the help of Python's library: (Note : The program is build using Python version >=3.8 and the browser that I'm using is Chrome version 10.) Here is the variables that you can change to your preference: Here is the list of column name and definition in dataset.csv file: The program aimed to extract product data from the Tokopedia marketplace website based on specified keywords using web scraping techniques.

Selenium with JavaScript-enabled selectors was utilized to extract the data due to the dynamic elements on the website. The extracted data included product name, price, location, rating, number of items sold, and details link, which were essential for data analysis and market research. The data was saved in both CSV and JSON formats for further processing and analysis. Selenium Documentation: https://www.selenium.dev/documentation/ This code is intended for educational purposes. Please respect privacy, copyright and code and data terms of use.

Read more about Tokopedia here. Download dan install python3 di OS kamu jika belum menginstall. Cek: Download pyhton Clone atau download repository tokped scraper via. git clone https://github.com/rahmatalhakam/tokopedia-scraper.git Install library python yang dibutuhkan.

pip install requests beautifulsoup4 pandas Setting keyword pencarian di file config.json Tambahkan url toko yang ingin di file tokopedia_shops.csv. Contoh: Tokopedia, one of Indonesia’s biggest e-commerce platforms has 90+ million active users and 350 million monthly visits. The platform has a wide range of products, from electronics, fashion, groceries to personal care.

For businesses and developers, scraping Tokopedia data can give you insights into product trends, pricing strategy, and customer preference. Tokopedia uses JavaScript to render its content; the traditional scraping method doesn’t work. Crawlbase Crawling API helps by handling JavaScript-rendered content seamlessly. In this tutorial, you’ll learn how to use Python and Crawlbase to scrape Tokopedia search listings and product pages for product names, prices, and ratings. Scraping Tokopedia data can be beneficial for businesses and developers. As one of Indonesia’s biggest e-commerce platform, Tokopedia has a lot of information about products, prices and customer behavior.

By extracting this data, you can get ahead in the online market. There are many reasons why one would choose to scrape data from Tokopedia: In the next section we will see what we can scrape from Tokopedia.

Integrated Web Scraping And Data Analysis Pipeline Tokopedia Github

People Also Search

Developed An Automated Dual-source Web Scraping And Data Integration Pipeline

Automated Export & Visualization Implemented An Automated .xlsx Export Pipeline

Created An Excel-based Dashboard To Automate Balance Sheet And Profit

Data Yang Diambil Termasuk Nama Produk, Harga, Rating, Jumlah Ulasan,

This Repository Is An Automated Web Scraping Method, To Scrape