Summary The data is stored in javascript variable in 51job website. The crawling could not get right data directly from html, but javascript. Using json plugin to achive the goal. Website: https://search.51job.com/ Search result Html source The data is not in the page html, but javascript code in json format. Xpath Using xpath to look for data on the page is fine, but in the code it returns nothing, because the data is in javascript variable. contentx = le.HTML(content) rets = contentx.xpath('//div[@class="e"]//p[@class="t"]') # returns nothing Crawling with python Using regular expression to parse out the jason data in javascript. ret = re.findall('window.__SEARCH_RESULT__ = (.*?)</script>', content_str)[0] Use json plugin to load ret into json object. d = json.loads(ret) Finally, get the data from json array. Complete code import requests import lxml.etree as le import re import json content = requests.get( url='https://search.51job.com/list/010000,000000,0000,00,9,99,python,2,1.html?lang=c&postchannel=0000&workyear=99&cotype=99°reefrom=99&jobterm=99&companysize=99&ord_field=0&dibiaoid=0&line=&welfare=', ).content content_str = content.decode('gbk', 'ignore') # <meta http-equiv="Content-Type" content="text/html; charset=gbk"> ret = re.findall('window.__SEARCH_RESULT__ = (.*?)</script>', content_str)[0] d = json.loads(ret) # <class 'dict'> {'top_ads': [], 'auction_ads': [], 'market_ads': [], 'engine_search_result': [{'type': 'engine_search_result', results = d['engine_search_result'] for result in results: print(result)
Problem The second line contains pingying and alias. html code: Zhōngfǔ 肺之募穴 First try get data roughly: alias = p.Select(Selectors.XPath("//span[@style]"))?.Value.Replace(" ", " "); Result: Zhōngfǔ 肺之募穴 Split the alias column in MySQL: UPDATE `xuewei` set pingying=substring_index(alias,' ',1); UPDATE `xuewei` set bieming=substring_index(alias,' ',-1); Split the column in process: alias = p.Select(Selectors.XPath("//span[@style]"))?.Value.Replace(" ", " "); var arr = alias.Split(' '); pingying = arr[0]; alias = arr[arr.Length-1]; //get last item in the array for the space in between is uncertain. Save to database: await conn.ExecuteAsync( $"INSERT IGNORE INTO xuewei (name,pingying,alias,position,anatomy,indication,operation,imagegeneral,imagepoint) VALUES " + $"('{data.Name}', '{data.Pingying}', '{data.Alias}', '{data.Position}', '{data.Anatomy}', '{data.Indication}', '{data.Operation}', '{data.ImageGeneral}', '{data.ImagePoint}');");
Content data on webpage: It's html code looks like this: Parsing data using regular expression with named group. Use foreach loop to avoid the order change on page. Do not use the text value from DOM function/property, because the text value does not contain html tag, which gives you the necessary mark on data.
Analysis url: https://www.luscious.net/albums/cassie-laine_348936/ load images with ajax. It returns json file url: https://api.luscious.net/graphql/nobatch/?operationName=AlbumListOwnPictures&query=+query+AlbumListOwnPictures%28%24input%3A+PictureListInput%21%29+%7B+picture+%7B+list%28input%3A+%24input%29+%7B+info+%7B+...FacetCollectionInfo+%7D+items+%7B+...PictureStandardWithoutAlbum+%7D+%7D+%7D+%7D+fragment+FacetCollectionInfo+on+FacetCollectionInfo+%7B+page+has_next_page+has_previous_page+total_items+total_pages+items_per_page+url_complete+%7D+fragment+PictureStandardWithoutAlbum+on+Picture+%7B+__typename+id+title+description+created+like_status+number_of_comments+number_of_favorites+moderation_status+width+height+resolution+aspect_ratio+url_to_original+url_to_video+is_animated+position+tags+%7B+category+text+url+%7D+permissions+url+thumbnails+%7B+width+height+size+url+%7D+%7D+&variables=%7B%22input%22%3A%7B%22filters%22%3A%5B%7B%22name%22%3A%22album_id%22%2C%22value%22%3A%22348936%22%7D%5D%2C%22display%22%3A%22rating_all_time%22%2C%22page%22%3A1%7D%7D Coding json structure below. We need the highest resolution images. import requests import json from urllib.request import urlretrieve def download(url): header = { "User_Agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36" } html = requests.get(url, headers=header).text #return json file jobj = json.loads(html) items = jobj['data']['picture']['list']['items'] for item in items: img = th['url_to_original'] filename = img[img.rfind("/") + 1:] filepath = 'd:/img/'+filename print(filepath) urlretrieve(img, filepath) url = "https://api.luscious.net/graphql/nobatch/?operationName=AlbumListOwnPictures&query=+query+AlbumListOwnPictures%28%24input%3A+PictureListInput%21%29+%7B+picture+%7B+list%28input%3A+%24input%29+%7B+info+%7B+...FacetCollectionInfo+%7D+items+%7B+...PictureStandardWithoutAlbum+%7D+%7D+%7D+%7D+fragment+FacetCollectionInfo+on+FacetCollectionInfo+%7B+page+has_next_page+has_previous_page+total_items+total_pages+items_per_page+url_complete+%7D+fragment+PictureStandardWithoutAlbum+on+Picture+%7B+__typename+id+title+description+created+like_status+number_of_comments+number_of_favorites+moderation_status+width+height+resolution+aspect_ratio+url_to_original+url_to_video+is_animated+position+tags+%7B+category+text+url+%7D+permissions+url+thumbnails+%7B+width+height+size+url+%7D+%7D+&variables=%7B%22input%22%3A%7B%22filters%22%3A%5B%7B%22name%22%3A%22album_id%22%2C%22value%22%3A%22348936%22%7D%5D%2C%22display%22%3A%22rating_all_time%22%2C%22page%22%3A1%7D%7D" download(url) print('done!') Result