IT Blog - Page 3 of 16 - Technical notes in IT

Add some useful extension methods for string manipulations in C#. Make coding more elegent, interesting and productive. string.ToBoolean(); string.ToInt(); string.RemoveSufix(string suffix); string.RemovePrefix(string prefix); string.Remove(string toBeRemoved); string.RemoveSpace(); string.RemoveWhiteSpace(); /// <summary> /// Accept string value: "true", "t", "1" as true, all others will be evaluated as false. /// </summary> /// <returns></returns> public static bool ToBoolean(this string value) { if (string.IsNullOrEmpty(value)) return false; switch (value.Trim().ToLower()) { case "true": return true; case "t": return true; case "1": return true; case "0": return false; case "false": return false; case "f": return false; case "": return false; default: return false; //throw new InvalidCastException("You can't cast that value to a bool!"); } } /// <summary> /// Convert a string to int with default value /// </summary> /// <returns></returns> public static int ToInt(this string value) { if (string.IsNullOrEmpty(value)) return 0; int.TryParse(value, out int i); return i; } /// <summary> /// Remove string suffix /// </summary> /// <param name="suffix"></param> /// <returns></returns> public static string RemoveSufix(this string value, string suffix) { if (value.EndsWith(suffix)) { value.Substring(0, value.Length - suffix.Length); } return value; } /// <summary> /// Remove string prefix /// </summary> /// <param name="prefix"></param> /// <returns></returns> public static string RemovePrefix(this string value, string prefix) { if (value.StartsWith(prefix)) { value.Substring(prefix.Length); } return value; } /// <summary> /// Remove all spaces from a string /// </summary> /// <returns></returns> public static string RemoveSpace(this string value) { return value.Replace(" ", string.Empty); } /// <summary> /// Remove all white spaces from a string, including " ", \t, \r, \n /// </summary> /// <returns></returns> public static string RemoveWhiteSpace(this string value) { return value.Replace(" ", string.Empty) .Replace("\t",…

2020-09-24 0 Comments 914 Views 0 Like IT Team Read more

Zoom meeting recorded video and saved in mp4 format file. But it is not allowed to download. In order to overcome "access denied" error, we have to add hread informations: user agent, cookie, and referer. Here is the source code to download the video using python. import requests url="https://ssrweb.zoom.us/cmr/replay/2020/09/21/2189503564/CD309625-D877-4B35-B99F-34CA4529F0CD/GMT20200920-233653_Tom-Ren-s-_1920x900.mp4?response-content-type=video%2Fmp4&response-cache-control=max-age%3D0%2Cs-maxage%3D86400&data=b87bffbf0d63fc51fb5b53caa6a6e3be04d1fc8b436a138418c4cfce8ff64b49&s001=yes&cid=aw1&fid=eT5a1fFKL6E57JgRK8F681dHTUjCtkgCRQSIhogfoc991lO-o1SPZgBwVzAnFJVok-qWSQ3oTrbpuF_L.01AJKlG7IPx983bD&s002=ly1zSoH_nrGCD_5BpiDHTfumcztfOw0ymT3bgpFePYuEvjANCyhpAsfNHw.bIBlHettrxUuB0Ga&Policy=eyJTdGF0ZW1lbnQiOiBbeyJSZXNvdXJjZSI6Imh0dHBzOi8vc3Nyd2ViLnpvb20udXMvY21yL3JlcGxheS8yMDIwLzA5LzIxLzIxODk1MDM1NjQvQ0QzMDk2MjUtRDg3Ny00QjM1LUI5OUYtMzRDQTQ1MjlGMENEL0dNVDIwMjAwOTIwLTIzMzY1M19Ub20tUmVuLXMtXzE5MjB4OTAwLm1wND9yZXNwb25zZS1jb250ZW50LXR5cGU9dmlkZW8lMkZtcDQmcmVzcG9uc2UtY2FjaGUtY29udHJvbD1tYXgtYWdlJTNEMCUyQ3MtbWF4YWdlJTNEODY0MDAmZGF0YT1iODdiZmZiZjBkNjNmYzUxZmI1YjUzY2FhNmE2ZTNiZTA0ZDFmYzhiNDM2YTEzODQxOGM0Y2ZjZThmZjY0YjQ5JnMwMDE9eWVzJmNpZD1hdzEmZmlkPWVUNWExZkZLTDZFNTdKZ1JLOEY2ODFkSFRVakN0a2dDUlFTSWhvZ2ZvYzk5MWxPLW8xU1BaZ0J3VnpBbkZKVm9rLXFXU1Ezb1RyYnB1Rl9MLjAxQUpLbEc3SVB4OTgzYkQmczAwMj1seTF6U29IX25yR0NEXzVCcGlESFRmdW1jenRmT3cweW1UM2JncEZlUFl1RXZqQU5DeWhwQXNmTkh3LmJJQmxIZXR0cnhVdUIwR2EiLCJDb25kaXRpb24iOnsiRGF0ZUxlc3NUaGFuIjp7IkFXUzpFcG9jaFRpbWUiOjE2MDA3NTA5MzB9fX1dfQ__&Signature=J-GEzf4ZyLk~Jlt~05nwpyWnVlw1~KTHasbnBpYaUV5H36d7Vijx~arr0VHNjUvG6jMCqQvjZBht5OnGmKLNKwLCC8sgJsY~dK~~LHyWXtOeSmKCN5OeIftk6YITItvnpyWYujuggXECzcGqis1K~gxa9krlax0J5~OosoKZat33JZISttY3RPAbfO~GeRqfJkDwiTVS1sKIyFdFbc~m-haPN4SlDx1ZDKpDWowYtRIkd4u692cclyp7fgxeIRy9OKyKt3Wfkw-39fTq5TcDJ5u1Yq~o4u0qbeLzASSFDgw7DRpZVnMqODupnhXp1QrZ8886s~ij7GU2oY7rXKzRHA__&Key-Pair-Id=APKAJFHNSLHYCGFYQGIA" content = requests.get(url, headers={ "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.102 Safari/537.36", "Cookie": "_zm_ssid=aw1_c_rWPCvlX3Tcmc8iikrsu0Bg; _zm_mtk_guid=b2c4961722954e1890f35c83dc55fc01; zm_oauth_cluster=aw1; zm_general_guid=""5Rog1fA9Dxwqucr9rINhlQzwlU/sFl0FdC6GZtoSZp4=""; _zm_lang=en-US; zm_gnl_guid=ab1cGxWMcSkw4h2H6yO/DOPyREzINR14Ppkyl7FBOlG1eU5nI2Qnwdltx4ESb0I1odSd; zm_gnl_ruid=udw1HTAmShCudkTwo4ToCA; _zm_csp_script_nonce=JG_kdyI9T-CxO2Y4shUwHA; _zm_launcher=1597953850725; zm_gnl_guid_v2=dffcVktj7guFTpRscXSNCeINKl8xSIlozAf16PavwVWL29lhYi78Ym1sSg3l9Js/VWHPWMBeaBr7Ap6D0q0Z00gLsZb2SrCjP5C8Y=; _zm_page_auth=aw1_c_wCHnZvcWSvuMw-h46T8FIg; _zm_ctaid=51pkTNQpTz-uUZmjm4Rulw.1600739087485.c798cb33478fb73a92bac81a5f6662e2; _zm_chtaid=811; _zm_currency=CAD", "referer": "https://zoom.us/rec/play/P4ZLJUcpCuFLJBpjP7wRiBRHquR8PNIK91C3FRHSPGJ0XX39e6e88ZYmzRLhXq9djw90IHTTTKv0cxfI.QAryZuRQmCV5viBv?continueMode=true&_x_zm_rtaid=51pkTNQpTz-uUZmjm4Rulw.1600739087485.c798cb33478fb73a92bac81a5f6662e2&_x_zm_rhtaid=811" }).content # print(content) counter = 0 with open('./video/第二讲加拿大房屋保险.mp4', 'wb') as f: # f.write(content) # f.close() for buffer in content.iter_content(chunk_size=4096): f.write(buffer) counter += len(buffer) print(counter) 495 total views

2020-09-21 0 Comments 220 Views 0 Like IT Team Read more

Unicode expressed like "\u4a44". Chinese words located from 0x3400 to 0x9fa5, including simplified and traditional words. Online tool: Unicode to Chinese covnerter C# Stores unicode data as a string, for each unicode in a string lookup the string by hex or integer number to get the right characters to replace them in the string. string ZHTable = "㐀㐁㐂㐃㐄㐅㐆㐇㐈㐉㐊㐋㐌㐍㐎㐏㐐... ..."; private string GetChar(string unicode) { unicode = unicode.Replace("\\u", ""); var code = Convert.ToInt32(unicode, 16); return ZHTable[code - 13312].ToString(); } var matches = Regex.Matches(text, @"\\u[a-f0-9]{4}"); string result = text; if (matches.Count > 0) { foreach (var m in matches) { var key = m.ToString(); result = result.Replace(key, GetChar(m.ToString())); } } public string ToChinese(string text) { var matches = Regex.Matches(text, @"\\u[a-f0-9]{4}"); string result = text; if (matches.Count > 0) { foreach (var m in matches) { var key = m.ToString(); result = result.Replace(m.ToString(), GetChar(key)); } } return result; } PHP Stores unicode data in a dictionary like array, for each unicode in a string look up the dictionary to get the right characters to replace them in the string. $unicodedata = [ ... ... 0x3437 => '㐷', 0x3438 => '㐸', 0x3439 => '㐹', 0x343a => '㐺', 0x343b => '㐻', 0x343c => '㐼', ... ... ]; for number calculate as power of 16; for a-f, convert to ascii code - 87, so as a=10, f=15, then calculate as power of 16; for hex letter only has a-f, other than that throw exception. if($this->is_number($u[$i])){ $val += $u[$i] * pow(16, 3-$i); } elseif($this->is_hexletter($u[$i])){ $val += ord($u[$i])-87 * pow(16,3-$i); } to match all unicode, which…

2020-09-18 0 Comments 361 Views 0 Like IT Team Read more

install Flask pip install flask Get Template Writing code Idea: Based on the proper visualization template, feed data from python program to the template, then render the result in the template. 1. Passing data from python to template through javascript. In Python file: from flask import Flask, render_template app = Flask(__name__) @app.route('/') def root(): divdata = '草莓,乌梅,蓝莓,红莓,白梅,话梅,绿莓'; areadata = '555,666,777'; return render_template('bigdata.html', divdata=divdata, areadata=areadata, ... ) if __name__ == '__main__': app.run(debug=True) 2. In template html file Render data as an html element, using js to convert it into js varaibles. <div id='divdata' style="display:none">{{ divdata }}</div> <div id='areadata' style="display:none">{{ areadata }}</div> <script> var divdata = document.getElementById('divdata').innerText; var datadiv = divdata.split(','); var areadata = document.getElementById('areadata').innerText; var dataarea = areadata.split(','); </script> 3. In chart js configuration file: //initialize echarts, and configure the chart var myChart = echarts.init(document.getElementById('echart1')); option = { ... xAxis: [{ type: 'category', data: datadiv, axisLine: { var data = [ { name: '海门', value: dataarea[0]}, { name: '鄂尔多斯', value: dataarea[1]}, { name: '招远', value: dataarea[2]}, { name: '舟山', value: dataarea[3]}, Result 321 total views

2020-09-13 0 Comments 229 Views 0 Like IT Team Read more

Error handling in sql server was a pain before SQL Server 2005. Starting from Sql Server 2005 try ... catch blocks was introduced. It was not only giving us a beauty of coding, but greatly improved our development performance. It comes with the following build-in functions: ERROR_NUMBER() ERROR_SEVERITY() ERROR_STATE() ERROR_PROCEDURE() ERROR_LINE() ERROR_MESSAGE() When these functions is called in a CATCH block, they will returns the error information. After working on many projects, I found a best pattern for error handling in Sql Server as the following steps. 1. Create a log table to store error information catch by try...catch blocks, CREATE table sys_log( id int identity primary key, procName varchar(255), essionID int, errorNumber int, severity smallint, errorState smallint, errorLine int, description nvarchar(255), info nvarchar(max), mslangid smallint default 1033, --1033 English, 2052 Chinese timeStemp datetime default getdate()) GO 2. create a stored procedure to catch and log error information: CREATE proc sys_LogError @procname sysname, @info nvarchar(max)='' AS INSERT INTO sys_log(ProcName,SessionID,ErrorNumber,Severity,ErrorState,ErrorLine,Description,info) values ( @procname, @@spid, ERROR_NUMBER(), ERROR_SEVERITY(), ERROR_STATE(), ERROR_LINE(), ERROR_MESSAGE(), @info ) 3. in all stored procedures that need to handle exepctions in transaction using this template: CREATE proc proc_name AS BEGIN tran BEGIN try -- do something print 1/0 commit tran END try BEGIN catch rollback tran declare @info nvarchar(max)='@table='+@table+', @urls='+@urls declare @procname sysname = OBJECT_SCHEMA_NAME(@@PROCID) + '.' + OBJECT_NAME(@@PROCID) exec sys_LogError @procname,@info ---- re-throw exeception if you need --declare @msg nvarchar(2048) = error_message() --declare @severity smallint = error_severity() --declare @state smallint = error_state() --raiserror (@msg, @severity, @state) END catch ERROR_MESSAGE() can be referenced with MASTER.dbo.sysmessages table, mslangid is for identifying…

2020-09-03 0 Comments 84 Views 1 Like IT Team Read more

Based OO principles the private fields should not be accessd by anything outside of the object. However, for some reason we have to access the private fields, a good example is unit testing. With dotnet reflection we could access any fields, members in an object. The accessibility is controled by BindingFlags, specify proper bindingFlags we can access all private fields in a class. To access the private fields in base class we need to add one more level of the type with BaseType. Here is the extension method to get the value of a public and private field in an instance. public static T GetFieldValue<T>(this object obj, string name) { var bindingFlags = BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Instance; var field = obj.GetType().GetField(name, bindingFlags); if (field == null) field = obj.GetType().BaseType.GetField(name, bindingFlags); //father else if (field == null) field = obj.GetType().BaseType.BaseType.GetField(name, bindingFlags); //granpa else if (field == null) field = obj.GetType().BaseType.BaseType.BaseType.GetField(name, bindingFlags); //father of granpa return (T)field?.GetValue(obj); } 525 total views, 5 views today

2020-09-03 0 Comments 439 Views 0 Like IT Team Read more

Post_name in wordpress is urlencoded, and not showing Chinese chars. urlencode DELIMITER ; DROP FUNCTION IF EXISTS urlencode; DELIMITER | CREATE FUNCTION URLENCODE(str VARCHAR(4096) CHARSET utf8) RETURNS VARCHAR(4096) CHARSET utf8 DETERMINISTIC CONTAINS SQL BEGIN -- the individual character we are converting in our loop -- NOTE: must be VARCHAR even though it won't vary in length -- CHAR(1), when used with SUBSTRING, made spaces '' instead of ' ' DECLARE sub VARCHAR(1) CHARSET utf8; -- the ordinal value of the character (i.e. ñ becomes 50097) DECLARE val BIGINT DEFAULT 0; -- the substring index we use in our loop (one-based) DECLARE ind INT DEFAULT 1; -- the integer value of the individual octet of a character being encoded -- (which is potentially multi-byte and must be encoded one byte at a time) DECLARE OCT INT DEFAULT 0; -- the encoded return string that we build up during execution DECLARE ret VARCHAR(4096) DEFAULT ''; -- our loop index for looping through each octet while encoding DECLARE octind INT DEFAULT 0; IF ISNULL(str) THEN RETURN NULL; ELSE SET ret = ''; -- loop through the input string one character at a time - regardless -- of how many bytes a character consists of WHILE ind <= CHAR_LENGTH(str) DO SET sub = MID(str, ind, 1); SET val = ORD(sub); -- these values are ones that should not be converted -- see http://tools.ietf.org/html/rfc3986 IF NOT (val BETWEEN 48 AND 57 OR -- 48-57 = 0-9 val BETWEEN 65 AND 90 OR -- 65-90 = A-Z val BETWEEN 97 AND 122 OR -- 97-122 =…

2020-08-30 0 Comments 282 Views 0 Like IT Team Read more

Summary The data is stored in javascript variable in 51job website. The crawling could not get right data directly from html, but javascript. Using json plugin to achive the goal. Website: https://search.51job.com/ Search result Html source The data is not in the page html, but javascript code in json format. Xpath Using xpath to look for data on the page is fine, but in the code it returns nothing, because the data is in javascript variable. contentx = le.HTML(content) rets = contentx.xpath('//div[@class="e"]//p[@class="t"]') # returns nothing Crawling with python Using regular expression to parse out the jason data in javascript. ret = re.findall('window.__SEARCH_RESULT__ = (.*?)</script>', content_str)[0] Use json plugin to load ret into json object. d = json.loads(ret) Finally, get the data from json array. Complete code import requests import lxml.etree as le import re import json content = requests.get( url='https://search.51job.com/list/010000,000000,0000,00,9,99,python,2,1.html?lang=c&postchannel=0000&workyear=99&cotype=99&degreefrom=99&jobterm=99&companysize=99&ord_field=0&dibiaoid=0&line=&welfare=', ).content content_str = content.decode('gbk', 'ignore') # <meta http-equiv="Content-Type" content="text/html; charset=gbk"> ret = re.findall('window.__SEARCH_RESULT__ = (.*?)</script>', content_str)[0] d = json.loads(ret) # <class 'dict'> {'top_ads': [], 'auction_ads': [], 'market_ads': [], 'engine_search_result': [{'type': 'engine_search_result', results = d['engine_search_result'] for result in results: print(result) 388 total views

2020-08-28 0 Comments 326 Views 0 Like IT Team Read more

Based on the official knowledge base, currently there is no “out of the box” way to control events visibility — no plugin setting, plugin filter, etc. But with a bit of custom coding, you can get pretty close. 1. Hide event from month view if the use is not logged in. Ref: https://theeventscalendar.com/knowledgebase/k/making-a-members-only-calendar/ 1. create a event category: Agent Assign the event to the category. 2. add stylesheet, e.g. Custom CSS & JS plugin. .type-tribe_events.tribe-events-category-agent { display: none; } body.logged-in .type-tribe_events.tribe-events-category-agent { display: inline; } 2. disable popup tip window. ref: https://theeventscalendar.com/support/forums/topic/disabling-the-calendar-pop-up-views/ body .tribe-events-calendar .tribe-events-tooltip { visibility:hidden; } @media (max-width: 600px) { body .tribe-events-calendar .tribe-events-tooltip { visibility:hidden; } } 128 total views

2020-08-20 0 Comments 90 Views 0 Like IT Team Read more

Python Code import json import requests # 3rd party import jsonpath from pyecharts.charts import Map from pyecharts import options as opts from src.covid19_data import nameMap url = 'https://api.inews.qq.com/newsqa/v1/automation/foreign/country/ranklist' resp = requests.post(url).text data = json.loads(resp) # get contry names and confirmed cases $：outer {}， ..: fuzzy match + key name = jsonpath.jsonpath(data, "$..name") # ['美国', '巴西', '印度', '俄罗斯', ... ] confirm = jsonpath.jsonpath(data, "$..confirm") pairs = list(zip(name, confirm)) # [('美国', 5321520), ('巴西', 3109630), ('印度', 2329638), ...] # 2. data visualization map plotting - size,title,color,data map_ = Map(opts.InitOpts(width='1200px', height='600px')) .add(series_name="Global Covid-19 Confirmed Cases", data_pair=pairs, # data entry - country : confirmed maptype="world", # world map name_map=nameMap, # country name mapping is_map_symbol_show=False) # 不显示标记点 # set series options map_.set_series_opts(label_opts=opts.LabelOpts(is_show=False)) map_.set_global_opts(title_opts=opts.TitleOpts(title="Global Covid-19 Pandemic (Chinese)"), visualmap_opts=opts.VisualMapOpts(max_=6000000, is_piecewise=True)) map_.render('../result/covid19.html') covid19_data nameMap = { 'Singapore Rep.': '新加坡', 'Dominican Rep.': '多米尼加', 'Palestine': '巴勒斯坦', 'Bahamas': '巴哈马', 'Timor-Leste': '东帝汶', 'Afghanistan': '阿富汗', 'Guinea-Bissau': '几内亚比绍', "Côte d'Ivoire": '科特迪瓦', 'Siachen Glacier': '锡亚琴冰川', "Br. Indian Ocean Ter.": '英属印度洋领土', 'Angola': '安哥拉', 'Albania': '阿尔巴尼亚', 'United Arab Emirates': '阿联酋', 'Argentina': '阿根廷', 'Armenia': '亚美尼亚', 'French Southern and Antarctic Lands': '法属南半球和南极领地', 'Australia': '澳大利亚', 'Austria': '奥地利', 'Azerbaijan': '阿塞拜疆', 'Burundi': '布隆迪', 'Belgium': '比利时', 'Benin': '贝宁', 'Burkina Faso': '布基纳法索', 'Bangladesh': '孟加拉国', 'Bulgaria': '保加利亚', 'The Bahamas': '巴哈马', 'Bosnia and Herz.': '波斯尼亚和黑塞哥维那', 'Belarus': '白俄罗斯', 'Belize': '伯利兹', 'Bermuda': '百慕大', 'Bolivia': '玻利维亚', 'Brazil': '巴西', 'Brunei': '文莱', 'Bhutan': '不丹', 'Botswana': '博茨瓦纳', 'Central African Rep.': '中非共和国', 'Canada': '加拿大', 'Switzerland': '瑞士', 'Chile': '智利', 'China': '中国', 'Ivory Coast': '象牙海岸', 'Cameroon': '喀麦隆', 'Dem. Rep. Congo': '刚果（金）', 'Congo': '刚果（布）', 'Colombia': '哥伦比亚', 'Costa Rica': '哥斯达黎加', 'Cuba': '古巴', 'N. Cyprus': '北塞浦路斯', 'Cyprus': '塞浦路斯', 'Czech Rep.': '捷克', 'Germany': '德国', 'Djibouti': '吉布提', 'Denmark': '丹麦', 'Algeria': '阿尔及利亚',…

2020-08-12 0 Comments 512 Views 0 Like IT Team Read more

1 234 5…16

Extend some useful string functions in C#

Download video from zoom

Unicode to Chinese conversion notes

Data visualization with Flask

SQL Server Transaction Exception Handling Best Practice

Accessing private fields in C#

MySQL urlencode/urldecode that support chinese chars

Crawling 51job in Python

WordPress EventCalendar visibility

Python Data Visualization Practise