Webscraping of JavaScript Pages using a combination of Selenium & Beautiful Soup

Why Selenium?

Beautiful Soup makes web scraping easy by traversing the DOM (Document Object Model). But it handles only Static Scraping and is not capable of handling JavaScript. Essentially Beautiful Soup fetches webpage from the server without the help of a browser. We get what we see in the "view page source". If the data that we are looking for is available in the "view page source", then Beautiful Soup is sufficient for web scraping. But if we  need data that gets rendered only upon clicking a JavaScript link, then we need to use dynamic web scraping methods In such a situation we first use Selenium to automate the browser and click on the JavaScript link, wait for the elements to load and then use Beautiful Soup to extract the elements.

Advanced Techniques in TWS API - Interactive Brokers Part 3

In the first part of the blog on the TWS API of Interactive Brokers, I have dealt with the Installation of TWS API on a Windows machine for Anaconda distro.

In the second part of the blog, titled Fundamentals of TWS API, I have dealt with the essentials to get started.

Now in this third part I seek to explore the concepts like using a scanner, retrieving the OHLCV data and writing it to a CSV file, Hedging a Futures Contract with Options and many others including a fully functional sample trading system.

Table of Contents :

• Optionable contracts in NSE Using Scanner

• Write OHLCV data to Pandas and CSV files

• Web Scrap IB site for Single Stock Futures (SSF)

• Creating a Stock Universe

• Write OHLCV Data of the Stock Universe to CSV file

• Intraday Trading System: OpenEQ

• Codebase on GitHub

Optionable Contracts in NSE using the Scanner

Let's get to the point straight - The TWS API is just an interface to the TWS. If you are having problems defining a scanner viz the API , always make sure you can create a similar scanner using the TWS or the Mosaic Market Scanner.

Facebook Cleaner - Automate deletion of FB Posts/Likes/Comments

The Selenium WebDriver API is a great choice when it comes to automating browsers for auto / semi-auto trading. It also has some other quick fix applications like automating the deletion of facebook posts/likes/comments etc. To give a rough analogy -  As CCleaner is to the PC, this Facebook Cleaner is to your facebook  profile.

For the specific purpose of automating Facebook,  using the  Facebook SDK for Python is another  good option for consideration. However at the time of the development of this script,  the  Graph API Version 2.7 was the latest released version for the  facebook SDK for Python. But the official Graph API Version of facebook was 2.10. And so Selenium it was !!!!

Program on Github