Webscraping of JavaScript Pages using a combination of Selenium & Beautiful Soup

Why Selenium?

Beautiful Soup makes web scraping easy by traversing the DOM (Document Object Model). But it handles only Static Scraping and is not capable of handling JavaScript. Essentially Beautiful Soup fetches webpage from the server without the help of a browser. We get what we see in the "view page source". If the data that we are looking for is available in the "view page source", then Beautiful Soup is sufficient for web scraping. But if we  need data that gets rendered only upon clicking a JavaScript link, then we need to use dynamic web scraping methods In such a situation we first use Selenium to automate the browser and click on the JavaScript link, wait for the elements to load and then use Beautiful Soup to extract the elements.

Intro to Python logging module - An alternative to print() function

 Logging is generally used to debug our Python code. Many of us (like me) have a habit of using the print() function extensively during the development of a program. Nothing bad in that per se, just that when we want to promote this program to the production environment we must necessarily remove the print() statements. 

There are other reasons too why the print statement is not recommended and its better to use the logging module:

• The print statement works only if we have access to the console
• We have no mechanism to write the print statements to a text file for further perusal

pdb - The Python Debugger / breakpoint

breakpoint versus pdb

Normally we import pdb and then to use the PDB in the program we have to use one of its method named set_trace(). This is how the code goes:

import pdb
pdb.set_trace()

An easier alternative (starting from Python version 3.7 onwards) is to use the built-in breakpoint() function. This helps us to do away with explicit import

breakpoint() # in python 3.7 and above

When we call the breakpoint() function, the default implementation of the breakpoint() function will call sys.breakpointhook(), which in turn calls the pdb.set_trace() function. This is why we do not need to import pdb and call pdb.set_trace() explicitly ourselves. 

Create a Watchdog in Python

 What is Watchdog in Python

Watchdog is a python module that can be used to monitor file system changes. As the name suggests this module observes the given directory and can notify if a file is created or changed.

Step 1: Import some Stuff

import os
import time
from watchdog.observers import Observer
from watchdog.events import PatternMatchingEventHandler

Step 2: Create the event handler

if __name__ == "__main__":
    # patterns = "*" # the file patterns we want to handle
    patterns = "*.csv" # choose only csv files
    ignore_patterns = ""
    ignore_directories = True
    case_sensitive = True
    # Create the event handler
    my_event_handler = PatternMatchingEventHandler(patterns, ignore_patterns, ignore_directories, case_sensitive)

Installation of WordPress on localhost

Why install wordpress.org locally on your computer?

We can do all our testing on our local computer , one that doesn’t affect our live WordPress site.

Note: If you install WordPress locally on Windows, then the only person who can see that site is you. If you want to make a website that is available to the public, then you need to have a domain name and web hosting.

What do I need for running wordpress.org locally on my computer?

For Windows users, WampServer is the best way to start off with installation of wordspress.org on your computer? WAMP, is a compilation of Apache web server, PHP and MySQL bundled for Windows computers.

A Primer to ib_insync - Interactive Brokers Part 8

Why ib_insync ?

While retrieving stock data for multiple symbols asynchronously from the TWS API provided by IB, I found the going a little steep. And here was ib_insync which made the task so very simple for me.

Pls. note that ib_insync is very different from the TWS API provided by IB. The TWS API is the official API created by IB while  ib_insync is created by a team of individuals headed by Guru Ji Ewald de Wit

Developing a Trading System for TWS API - Interactive Brokers Part 7

What is a trading system?

It is a set of rules that can be based on technical indicators, quantitative methods, fundamental analysis or simply a mix of all or some of them.

What are the elements of a full fledged trading system that can be deployed in Production?

• An Entry Point
• A Stop Loss / Trailing Stop Loss
• An Exit Point